IS LABEL SMOOTHING TRULY INCOMPATIBLE WITH KNOWLEDGE DISTILLATION: AN EMPIRICAL STUDY
Year: 2021

Abstract:
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation (Müller et al., 2019). We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample’s representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness.1

heading: 1 INTRODUCTION
text: Label smoothing (Szegedy et al., 2016) and knowledge distillation (Hinton et al., 2015) are two commonly recognized techniques in training deep neural networks and have been applied in many state-of-the-art models, such as language translation (Vaswani et al., 2017; Tan et al., 2019; Zhou et al., 2020), image classification (Xie et al., 2019; He et al., 2019) and speech recognition (Chiu et al., 2018; Pereyra et al., 2017; Chorowski & Jaitly, 2017). Recently a large body of studies is focusing on exploring the underlying relationships between these two methods, for instance, Müller et al. (Müller et al., 2019) discovered that label smoothing could improve calibration implicitly but will hurt the effectiveness of knowledge distillation. Yuan et al. (Yuan et al., 2019) considered knowledge distillation as a dynamical form of label smoothing as it delivered a regularization effect in training. The recent study (Lukasik et al., 2020) further noticed label smoothing could help mitigate label noise, they showed that when distilling models from noisy data, the teacher with label smoothing is helpful. Despite the massive and intensive researches, how to use label smoothing as well as knowledge distillation in practice is still unclear, divergent, and under-explored. Moreover, it is hard to answer when and why label smoothing works well or not under a variety of discrepant circumstances. View of incompatibility between label smoothing and knowledge distillation. Recently, Müller et al. proposed a new standpoint that teachers trained with label smoothing distill inferior student compared to teachers trained with hard labels, even label smoothing improves teacher’s accuracy, as the authors found that label smoothing tends to “erase” information contained intra-class across individual examples, which indicates that the relative information between logits will be erased to some extent when the teacher is trained with label smoothing. This rising idea is becoming more and more dominant and has been quoted by a large number of recent literatures (Arani et al., 2019; Tang et al., 2020; Mghabbar & Ratnamogan, 2020; Shen et al., 2020; Khosla et al., 2020). However, this seems reasonable observation basically has many inconsistencies in practice when adopting knowledge distillation with smoothing trained teachers. Thus, we would like to challenge whether this perspective is entirely correct? To make label smoothing and knowledge distillation less mysterious, in this paper, we first systematically introduce the mechanism and correlation
1Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html.
Training w/o LS Training w/ LS Validation w/o LS Validation w/ LS
1 1
2 2
1
2
1
2 !"
!! !!
!"
!! !!
• What will actually determine the performance of a student in knowledge distillation? From our empirical study, we observe if the student architecture is settled, the dominating factor in knowledge distillation is the quality of supervision, i.e., the performance of a teacher network. A higher-accuracy teacher is particularly successful in distilling a better student, regardless it is trained with or without label smoothing. This observation is partially against the conclusion in (Müller et al., 2019) which stated “a teacher with better accuracy is not necessary to distill a better student”. • When will the label smoothing indeed lose its effectiveness for learning deep neural networks? Long-tailed class distribution and increased number of classes are two scenarios we observed wherein label smoothing will lose or impair its effectiveness. We empirically verify the findings on iNaturalist 2019 (Van Horn et al., 2018), Place-LT (Liu et al., 2019) and curated ImageNet (Liu et al., 2019).

heading: 2 BACKGROUND
text: In this section, we first introduce the background of label smoothing and knowledge distillation through a mathematical description. Given a dataset D = (X,Y ) over a set of classes K, X is the input data and Y is the corresponding one-hot label with each sample’s label y ∈ {0, 1}K , where the element yc is 1 for the ground-truth class and 0 for others. Label smoothing replaces one-hot hard label vector y with a mixture of weighted y and a uniform distribution:
yc = { 1− α if c = label, α/(K − 1) otherwise. (1)
where α is a small constant coefficient for flattening the one-hot labels. Usually, label smoothing is adopted when the loss function is cross-entropy, and the network uses softmax function to the last layer’s logits z to compute the output probabilities p, so the gradient of each training sample with
respect to z will be:∇H(p,y) = p−y = K∑ c=1 (Softmax(zc)− yc), whereH(p,y) = − K∑ c=1 yclogpc is the cross-entropy loss and zc is c-th logit in z. Effects of label smoothing on loss
To further understand the effects of label smoothing on loss function, Fig. 3 illustrates correction effects of smoothing on the binary cross-entropy loss (K = 2). We can observe that the standard logistic loss (α = 0) vanishes for large and confident positive predictions, and becomes linear for large negative predictions. Label smoothing will penalize confident predictions and involve a finite positive minimum as it aims to minimize the average perclass. Generally, larger α values will produce larger loss values rebounding at positive predictions. This is also the underlying reason that smoothed loss can flatten the predictions of a network.
In knowledge distillation, we usually pre-train the teacher model Tw on the dataset in advance. The student model Sw is trained over the same set of data, but utilizes labels generated by Tw. More specifically, we can regard this process as learning Sw on a new labeled dataset D̃ = (X, Tw(X)). Once the teacher network is trained, its parameters will be frozen in the whole distillation.
The student network Sw is trained by minimizing the similarity between its output and two parts: the hard one-hot labels and the soft labels generated by the teacher network. Letting pTwc (X) = Tw(X)[c], pSwc (X) = Sw(X)[c] be the probabilities assigned to class c in the teacher model Tw and student model Sw. The distillation loss can be formulated as λH(pSw ,y) + (1 − λ)H(pSw/T ,pTw/T ) where T is the temperature scaling factor and λ is the trade-off coefficient to balance the two terms.

heading: 3 THE “ERASE INFORMATION” EFFECT BY LABEL SMOOTHING
text: This section aims to explain the erasing information effect more thoroughly. We start by reproducing the visualization of penultimate layer’s activations using the same procedure from (Müller et al., 2019). We adopt ResNet-50 trained with hard and smoothed labels on ImageNet. As shown in Fig. 1, we obtain similar distributions as (Müller et al., 2019). Since examples in training set are the ones used for distillation, we mainly analyze the visualization from the training data. The core finding in (Müller et al., 2019) is that if a teacher is trained with hard labels, representations of examples are distributed in broad clusters, which means that different examples from the same class can have different similarities (D1 and D2) to other classes. For a teacher trained with label smoothing, they observed the opposite behavior. Label smoothing encourages examples to lie in tight equally separated clusters, so each example of one class has very similar proximities (D1 is closer to D2) to examples of the other classes. Our re-visualization also supports this discovery. The authors derive the conclusion that a teacher with better accuracy is not necessarily to distill a better student. This seems reasonable as the broad clusters can enable different examples from the same class to provide different similarities to other classes, which contains more information for knowledge distillation.
However, if refocusing on the two semantically similar classes, when label smoothing is applied, the clusters are much tighter because label smoothing encourages each example is to be equidistant from all other class’s templates, while, the tight cluster substantially promotes different class representations to be separate, i.e., the distance of clusters Dc increases, which further indicates that different class examples obtain more distinguishable features. This phenomenon is crucial as these difficult classes are the key for boosting classification performance. Generally, it is not necessary to measure “how much a poodle is a particularly similar to a tench” since we have enough evidence to classify them, but it is critical to have information “how different is a toy poodle to a miniature poodle”.
Visualizations of teacher predictions. We further visualization the mean distribution of different classes crossing examples, as shown in Fig. 4. We average all the probabilities after softmax layer if the examples belong to the same category, and show the first 100 classes in ImageNet. Usually, the probabilities have a major value (the bars in Fig. 4 (1)) that represents model’s prediction on category and other small values (i.e., minor predictions in Fig. 4 (2)) indicate that the input image is somewhat similar to those other categories, some discussions about minor predictions are given in Appendix F. Our purpose of this visualization is to make certain of what label smoothing really calibrates in a network and shed light on how it affects the network predictions. We can observe in this figure that a model trained with label smoothing will generate more softened distributions, but the relations across different classes are still preserved. We conjecture the softened supervision is also the reason why teachers with label smoothing produce larger training loss during knowledge distillation. Consequently, label smoothing will both decrease the variance (verified by following stability metric) and mean predictive values within a class, but will not impair the relations crossing different classes.

heading: 3.1 A SIMPLE METRIC FOR MEASURING THE DEGREE OF ERASED INFORMATION
text: Different from the visualization scheme (Müller et al., 2019) of finding an orthonormal basis of the plane that only studies this problem qualitatively, we further address the “erasing” phenomenon through a statistical metric that is simple yet effective, and can measure the degree of erasing operation quantitatively. Our motivation behind it is straight-forward: If label smoothing erases relative information within a class, the variance of intra-class probabilities will decrease accordingly, thus we can use such variance to monitor the erasing degree, since this metric evaluates the fluctuation
of the representations, we can also call it the stability metric. The definition is as follows:
SStability = 1− 1
K K∑ c=1 ( 1 nc nc∑ i=1 ||pTw{i,c} − p Tw {i,c}|| 2) (2)
where i is the index of images and nc is the #image in class c. pTw{i,c} is the mean of p Tw in class c. This metric utilizes the probabilities of intra-class variance to measure the stability of a teacher’s prediction. The results on various network architectures are shown in Sec. 5 and a PyTorch-like code for calculating this metric is given in Appendix C.
Such metric has at least two advantages: 1) It can measure the degree of erased information quantitatively and further help discover more interesting phenomena, e.g., we observe that data augmentation method like CutMix (Yun et al., 2019) together with longer training erases the relative information on logits dramatically and can further be reinforced by label smoothing. 2) We found that the proposed metric is highly aligned with model accuracy, thus such metric can be used as a complement for accuracy to evaluate the quality of teacher’s supervision for knowledge distillation.

heading: 4 A CLOSE LOOK AT LABEL SMOOTHING AND KNOWLEDGE DISTILLATION
text: A few recent studies (Shen & Savvides, 2020; Shen et al., 2019) suggested supervised partH(pSw ,y) (i.e. hard labels) is not necessary as soft prediction is adequate to provide crucial information for students, meanwhile, removing supervised part can avoid involving incorrect labels caused by random crop, multi-object circumstance or false annotations by humans. Thus, here we only consider the soft partH(pSw/T ,pTw/T ) with the commonly used Kullback-Leibler divergence similarity. KL-divergence measures the similarity of two probability distributions. We train the student network Sθ by minimizing the KL-divergence between its output pSθc (X) and the soft labels pTθc (X) generated by the teacher network. Following (Müller et al., 2019; Hinton et al., 2015) we set T =1 as the
temperature constant and it is omitted for simplicity, thus our loss function will be: DKL(Tw‖Sw) = Ex∼Tw [ − log Sw(X)
Tw(X) ] = Ex∼Tw [− logSw(X)]−H(Tw(X))
(3)
Here, Ex∼Tw [− logSw(X)] is the cross-entropy between Sw and Tw (denoted H(pSw ,pTw) ) . The second termH(Tw(X)) = Ex∼Tw [− log pTw(x)] is the entropy of teacher Tw and is constant with respect to Tw. We can remove it and simply minimize the loss as follows:
H(pSw ,pTw) = − K∑ c=1 pTwc (X) logp Sw c (X). (4)
We can observe that Eq. 4 is actually a standard cross-entropy loss. Then, we have:
Property 1. If not consider hard labels in knowledge distillation, distillation loss and cross-entropy loss with label smoothing have the same optimizing objective, i.e., DKL(Tw‖Sw) =H(pSw ,pTw). This property shows that label smoothing and knowledge distillation have the same optimization objective, the sole difference between them is the mechanism of producing the soft labels. Therefore, except for the neural machine translation, in this paper all of our knowledge distillation experiments are conducted without the hard labels, which means our student solely relies on the softened distribution from a teacher without the one-hot ground-truth. This may challenge the common practice in knowledge distillation (Hinton et al., 2015; Romero et al., 2015) that adopted both hard and soft labels with the cross-entropy loss for distillation, while our surprisingly good results and previous studies (Shen & Savvides, 2020; Shen et al., 2019; Bagherinezhad et al., 2018) indicate that knowledge distillation enabled by soft labels solely is not only an auxiliary regularization (Yuan et al., 2019) but can be the dominating supervisions, which further inspires us to carefully revisit the role of knowledge distillation and design better supervision/objective in training deep neural networks.

heading: 5 EMPIRICAL STUDIES
text: Metric Evaluation. Our results of stability metric are shown in Table 1, the second and third columns are results without label smoothing and the last two are with it. We study the metric crossing a variety of different network architectures. The gaps of SStability using the same architecture measure the degree of erasing relative information. We can observe that the variances (1-SStability) with label smoothing always have lower values than models trained without label smoothing, this proves that label smoothing will erase information and enforce intra-class representations of samples being similar. Generally, the accuracy and stability have a positive correlation between them. But the stability can even overcome some outliers, for example, Wide ResNet50 with label smoothing has lower accuracy, but the stability is still consistent with the tendency of predictive quality. Moreover, models trained with more epochs and augmentation techniques like CutMix (Yun et al., 2019) can dramatically increase the stability, this means relative information will be erased significantly by more augmentation together with longer training. We emphasize that this discovery cannot be observed by the qualitative visualization method (Müller et al., 2019). A PyTorch-like code is in Appendix C.
Image Classification. We verify our perspective through investigating the effectiveness of knowledge distillation with label smoothing on the image classification tasks. We conduct experiments on three
datasets: ImageNet-1K (Deng et al., 2009), CUB200-2011 (Wah et al., 2011a) and iMaterialist product recognition challenge data (in Appendix D). We adopt ResNet-{50/101} as teacher networks and ResNet-{18/50/101} as students, respectively. More experimental settings are in Appendix A. Results. The visualizations of our distillation training and testing curves are shown in Fig. 5. A more detailed comparison is listed in Tables 2 and 7. From the visualization we found two interesting phenomena: On training set, the loss of teacher networks that trained with label smoothing is much higher than that of without label smoothing. While on validation set the accuracy is comparable or even slightly better (The boosts on CUB is greater than those on ImageNet-1K, as shown in Table 2). We also provide the experiments of combining hard and soft labels in Appendix E, the results still show the effectiveness of better teachers with label smoothing, which distill better students. To make it clearer why this happens in distillation, we visualize the supervisions from teacher networks in Fig. 4 and the discussion is shown there. It indicates that label smoothing flattens teacher’s predictions which causes the enlarged training loss, while the student’s generalization ability is still preserved.
Binary Neural Networks (BNNs). We then examine the effectiveness of knowledge distillation on Binary Neural Networks. BNN aims to learn a network that both weights and activations are discrete values in {-1, +1}. In the forward pass, real-valued activations are binarized by the sign function:
Ab = Sign(Ar) = { −1 if Ar < 0, +1 otherwise. where Ar is the real-valued activation of the previous layers, produced by the binary or real-valued convolution operations. Ab is the binarized activation.
The real-valued weights are binarized by: Wb = ||Wr||l1
n Sign(Wr) = { − ||Wr||l1n if Wr < 0, + ||Wr||l1
n otherwise. where Wr is the real-valued weights that are stored as latent parameters to accumulate the small gradients. Wb is the binarized weights. We update binary weights through multiplying the sign of real-valued latent weights and the channel-wise absolute mean ( 1n ||Wr||l1). Training BNNs is challenging as the gradient of optimization is approximated and the capacity of models is also limited.
We perform experiments on ImageNet-1K and results are shown in Fig. 6. Withal, the teacher network trained with one-hot labels (blue curve) is over-confident as the loss value is much smaller, which
means that the teacher trained with label smoothing can prevent distillation process from being overconfident on the training data, and obtain slightly better generalization and accuracy (63.108% vs. 63.002%) on the validation set. These results still support our conclusion on knowledge distillation.
Neural Machine Translation (NMT). Finally, we investigate our hypothesis of knowledge distillation on the English-to-German translation task using the Transformer architecture (Vaswani et al., 2017). We utilize the distillation framework of (Tan et al., 2019) on IWSLT dataset, and the pretraining/distillation curves are shown in Fig. 7. A consistent setting is imposed on all the two comparison experiments, except the teacher is trained with and without label smoothing. We choose α = 0.1 for label smoothing as suggested by (Vaswani et al., 2017; Szegedy et al., 2016;
Müller et al., 2019), we use Adam (Kingma & Ba, 2014) as the optimizer, lr with 0.0005, dropout with drop rate as 0.3, weight-decay with 0 and max tokens with 4096, all of these hyper-parameters are following the original settings of (Tan et al., 2019). Our results of Fig. 7 deliver two important conclusions: First sub-illustration (left one) proves the statement of Vaswani et al. (Vaswani et al., 2017) that label smoothing (α = 0.1) boosts the BLEU score of language model despite causing worse perplexity if comparing to a model is trained with one-hot/hard labels. Second sub-illustration (right one) implies that on the machine translation task, a stronger teacher (trained with label smoothing) will still distill a higher BLEU student. That is to say, label smoothing may not suppress the effectiveness of knowledge distillation in the NMT task.

heading: 6 WHAT IS A BETTER TEACHER IN KNOWLEDGE DISTILLATION?
text: Better Supervision is Crucial for Distillation. We further explore the effects of teacher’s accuracy on the student through fixing the student structure and switching different teachers. We perform two settings for this ablation study: using the same teacher structure with different training strategies and different teacher architectures. All teacher models are re-trained with label smoothing. The results are shown in Fig. 8 (Left) and Table 8, generally, teachers with higher accuracies can distill stronger students, but they are not linear related and are limited to the capability of the student itself. To further
support the argument that better teachers usually distill better students, we choose the state-of-the-art FixRes model (Touvron et al., 2019) for both the teacher and student and perform our distillation training via Eq. 4. The results are shown in Fig. 8 (Right) and our method is slightly better than the baseline and FixRes. Note that the compared FixRes is already the state-of-the-art with ResNet and ResNeXt architecture, so our result (under ResNet family) is a fairly competitive single-crop accuracy to date on ImageNet-1K.

heading: 7 WHAT CIRCUMSTANCES INDEED WILL MAKE LS LESS EFFECTIVE?
text: Long-Tailed Distribution. We found the long-tailed datasets with imbalanced samples across classes will indeed suppress the effectiveness of label smoothing (LS) and hurt model performance. The results on long-tailed ImageNet-LT, Places365-LT and iNaturalist 2019 are shown in Table 3. Our results on these three datasets empirically verify this observation and support this conclusion. Since LS has weight shrinkage and regularisation effects (Lukasik et al., 2020), we have following conjecture:
Conjecture 1. Weight shrinkage effect (Regularisation) enabled by label smoothing is no longer effective on the long-tailed recognition circumstance and will further impair the performance.
The weight shrinkage effect (label smoothing regularization) has been proven in (Lukasik et al., 2020) (Theorem 1) on the linear models. As label smoothing will assign probabilities equally to all minor classes (i.e., (1− α) · Ipos + αL−1 · Jneg), this operation may be biased to many-shot classes in the long-tailed scenario. We derive the conclusion empirically that label smoothing is inapplicable when the class distribution is long-tailed.
More #Class. This is another circumstance we found will impair the effectiveness of label smoothing. The results are shown in Table 4 and Fig. 9, we average the gains of label smoothing across two different network architectures and compare the boosts between curated ImageNet-100/500 and 1K. Generally, more classes will reduce the improvement produced by label smoothing.
Table 3: Teacher results on the long-tailed ImageNet-LT, Places365-LT and iNaturalist 2019 val set.
ImageNet-LT Place-LT iNaturalist 2019 Teacher w/ LS Acc. (Top1/Top5) Acc. (Top1/Top3) Acc. (Top1/Top3)
ResNet-18 8 39.975/64.645 26.479/47.233 67.195/83.465 4 39.115/63.655 25.877/46.260 66.700/83.432 ResNet-34 8 41.150/66.205 27.329/48.753 70.165/86.304 4 40.965/65.850 26.863/48.110 69.406/86.073 ResNet-50 8 40.985/66.030 27.384/48.740 73.729/88.845 4 39.965/65.195 27.562/47.945 72.904/87.954 ResNet-101 8 –/– 28.096/50.164 74.389/88.416 4 –/– 27.466/48.781 73.597/88.779
Table 4: Teacher results on the curated ImageNet dataset when increasing the number of classes.
ImageNet-100 ImageNet-500 ImageNet-1K Teacher w/ LS Acc. (Top1/Top5) Acc. (Top1/Top5) Acc. (Top1/Top5)
ResNet-18 8 82.380/95.520 73.521/91.642 69.758/89.076 4 82.740/95.440 74.123/92.004 69.606/89.372
ResNet-101 8 82.000/94.340 81.712/95.080 77.374/93.546 4 83.400/95.300 82.020/95.300 77.836/93.662 Average (↑) ↑0.880/0.440 ↑0.455/0.291 ↑0.155/0.206 100 500 1K
#class
1.6
0.8
0
-0.8
Bo os t
Figure 9: Acc. downtrend.

heading: 8 CONCLUSION
text: We empirically demonstrated that label smoothing could both decrease the variance (i.e., erase relative information between logits) and lower mean predictive values (i.e., make prediction less confident) within a category, but it does not impair the relation distribution across different categories. Our results on image classification, binary neural networks, and neural machine translation indicate that label smoothing is compatible with knowledge distillation and this finding encourages more careful to understand and utilize the relationships of label smoothing and knowledge distillation in practice. We found through extensive experiments and analyses that the indeed circumstances label smoothing will lose its effectiveness are long-tailed distribution and increased number of classes. Our study also suggests that, to find a better teacher for knowledge distillation, accuracy of teacher network is one factor, the stability of supervision from teacher network is also an alternative indicator.

heading: A DATASETS AND EXPERIMENTAL SETTINGS
text: A.1 FOR TRAINING TEACHER AND KNOWLEDGE DISTILLATION MODELS TO VERIFY THE COMPATIBILITY BETWEEN LABEL SMOOTHING AND KNOWLEDGE DISTILLATION
Standard ImageNet-1K Classification (Deng et al., 2009) ImageNet-1K contains ∼1.28 million images in 1000 classes for training and 50K images for validation. For training teacher networks, we follow the standard training protocol (He et al., 2016; Goyal et al., 2017), i.e., total training epoch is 90, initial learning rate is 0.1 and decayed to 1/10 with every 30 epochs. For distillation, as the supervision is a soft distribution and will dynamically change, we train with 200 epochs and the learning rate is multiplied by 0.1 at 80 and 160 epochs. All models are trained from scratch.
Fine-grained Recognition on CUB200-2011 (Wah et al., 2011a) This dataset contains 200 bird species and 11,788 images, and is a wildly-used fine-grained classification benchmark. Following (Wah et al., 2011a), we use standard split with ∼30 samples of each breed for both training and testing. As CUB200-2011 has limited training data, we fine-tune our model from the ImageNet pre-trained parameters. Both teachers and students are trained with 200 epochs and the learning rate is multiplied by 0.1 at 80 and 160 epochs.
Noisy Product Recognition (iMaterialist Challenge)2. The training data consists of 1,011,532 images from 2,019 categories. The validation data has 10,095 images (around 5 for each category). The training data is collected from the Internet and contains some noise (∼30% of incorrect labels), the validation data has been cleaned by human annotators. The training protocol is following CUB2002011. While, as the training images in this dataset is sufficiently enough, all our models are trained from scratch.
A.2 FOR TRAINING TEACHERS ONLY TO EXPLORE THE EFFECTIVENESS OF LABEL SMOOTHING
Long-tailed iNaturalist 2019 (Van Horn et al., 2018). The iNaturalist Challenge 2019 dataset contains 1,010 species, with a combined training and validation set of 268,243 images, wherein, the validation set has a uniform distribution of three images in each category. While, the training set is constructed with a long-tailed distribution of image numbers. We conduct experiments using the
2https://sites.google.com/view/fgvc6/competitions/imat-product-2019? authuser=0.
same training protocol as the CUB200-2011 dataset. we evaluate our results on validation set using Top-1/Top-3 metrics. Also, this protocol is applied to the Place-LT dataset.
Place-LT (Liu et al., 2019). The training set of this dataset is constructed by a Pareto distribution (Reed, 2001) with a power value α = 6. It contains a total number of 184.5K images from 365 categories. There are 4,980∼5 images per class from the 365 classes of Places-365 dataset (Zhou et al., 2017). 20 images per class are randomly selected from the original Places-365 training set as a validation set. The training protocol is the same as iNaturalist 2019, we also evaluate our results on validation set using Top-1/Top-3 metrics.
ImageNet-LT (Liu et al., 2019). The long-tailed ImageNet is constructed by sampling a subset from the standard ImageNet (Deng et al., 2009) following the Pareto distribution with the power value α = 6. The whole dataset contains 115.8K images from 1000 categories, with maximal 1280 images and minimal 5 images per class. 20 training images per class are randomly selected from the origin training set of ImageNet as a validation set. All our models are trained from scratch.
Curated ImageNet-100/500/1K. We construct this dataset by randomly selecting a subset of 100/500/1K categories in the standard ImageNet (Deng et al., 2009). We use this dataset to explore the impact of #class for label smoothing. We observe that label smoothing is more effective on the dataset with fewer numbers of classes.

heading: B WHAT HAPPENS IF TEACHERS WITH LABEL SMOOTHING ARE INFERIOR?
text: In the main paper, we discussed that the superior teachers (even trained with label smoothing) could distill better students. Also, as we noted that long-tailed distribution will suppress the effectiveness of label smoothing. In this section, we would like to explore what happens if we use these inferior teachers for knowledge distillation? We conduct experiments on the long-tailed iNaturalist 2019 data and show more evidence to prove our conclusion that the quality of supervision is more crucial for knowledge distillation than the ways of training teachers (with or without label smoothing). The results are shown in Table 6, it can be observed that if teachers with lower accuracies, the distilled students will also have poorer performance, regardless of the teacher is trained with label smoothing or not.
C A PYTORCH-LIKE CODE FOR CALCULATING STABILITY METRIC
Algorithm 1 PyTorch-like Code for Calculating Stability Metric.
# x, target: input images and the corresponding labels # single_std: std value for each class # N: batchsize (we use N=50 on ImageNet for ease of implementation as it has 50 val images
in each class) # model_path: model path that you want to calculate stability
# load model checkpoint = torch.load(model_path) model.load_state_dict(checkpoint[’state_dict’]) # initialize for (x, target) in loader: # load a minibatch x with N samples (we choose N=50 for ImageNet)
# make sure shuffle = False in dataloader
# compute output output = model(x)
# compute softmax probabilities softmax_p = nn.functional.softmax(output, dim=1) predict = softmax_p[:, target[0]] # obtain target probability
# compute stability single_std = predict.std(dim=0) all_std += single_std # aggregate all std values from each class
return 1 - all_std/1000.0 # return computed stability

heading: D TWO DETAILED TABLES OF RESULTS

heading: E RESULTS OF KNOWLEDGE DISTILLATION WITH HARD AND SOFT LABELS
text: Here we verify whether label smoothing is still effective when adopting hard labels in knowledge distillation. As we mentioned above, the traditional distillation loss can be formulated as
λH(pSw ,y) + (1 − λ)H(pSw/T ,pTw/T ). We use ResNet-50 as the teacher and ResNet-18 as the student and conduct experiments with three different ratios of λ: 0.3, 0.5, 0.7. Our Top-1/5 results are given in Table 9. We can see the teacher networks with label smoothing still distills better students than the teacher without label smoothing. Also, with a higher probability of hard labels, the performance declines.

heading: F MINOR PROBABILITIES IN TEACHER NETWORK
text: The visualization of minor probabilities is shown in Fig. 4 (2), we have two interesting observations in it: 1) In the enlarged area of Fig. 4 (2), we can observe that several classes have values are close to zero, this phenomenon means in ImageNet dataset, there are several categories that are very unique to other classes and the model will barely predict other classes to them; 2) We can also observe more “red” areas gather at the bottom of the bars. It reflects that the probabilities from the model trained with label smoothing will be assigned to more classes with smaller values, this is also consistent with what the label smoothing operation actually does.

heading: G SUPPLEMENTARY METRICS OF INTER-CLASS/ENTIRE VARIATIONS
text: To better understand the behavior of erased information by label smoothing across different classes, we also introduce the inter-class stability metric as follows:
S interStability = 1− 1
K K∑ c=1 (||pTw{c} − p Tw {all}|| 2) (5)
where K is the number of classes. pTw{all} is average of probability across the entire dataset. p Tw {c} is the mean of pTw in class c. S interStability can be regarded as a supplement for our intra-class stability metric to measure the inter-class stability. The average probability within each class pTw{c} is defined as:
pTw{c} = 1
nc nc∑ i=1 pTw{i,c} (6)
where i is the index of images and nc is the #image in class c. The average probability across entire dataset pTw{all} is defined as:
pTw{all} = 1
N N∑ i=1 pTw{i} (7)
where N is the number of samples in the entire dataset. This metric utilizes the probabilities of inter-class variance to measure the stability of a teacher’s prediction. The results on various network architectures are shown in Table 10.

Review summaries:
The authors re-analyze and re-confirm the relationship between label-smoothing and knowledge distillation, which is firstly argued by Muller et al. (“When Does Label Smoothing Help?“, NerurIPS 2019.). This paper shows that the previous argument, "label smoothing is not helpful for knowledge distillation", does not always hold, and carefully re-visits the missing points of the previous analysis by Muller et al. Based on this analysis, label smoothing can be helpful for knowledge distillation and can be explained using the intra-class variation and between-class distance within similar classes. The authors have empirically verified the arguments of the paper with various experiments. 

Recent literature proposed that even label smoothing improves the teacher model, it will hurt the distillation training of student models due to the information erasing. Although this idea dominated more and more literature, this paper argued that this observation is not entirely correct. In order to clarify this idea, the paper systematically discussed the correlation between knowledge distillation and label smoothing. Comprehensive experiments well support the claims in this paper, i.e. label smoothing is compatible with knowledge distillation. The correlation between label smoothing and knowledge distillation remains an open question to date, and this paper made a breakthrough regarding this question. Besides the main purpose (clarify previous ideas), this paper also provided multiple interesting empirical conclusions, e.g. a better teacher always leads to a better student by producing more informative distillation labels, the distillation itself can provide enough regularization for training and the hard-label classification loss is no more needed. 

This paper is mainly based on the prior work by Muller et al., which suggests that label smoothing is incompatible with knowledge distillation. Firstly, this paper provides an explanation of this incompatibility---label smooth tends to erase relative information among different classes, and provide a way to qualitatively measure the degree of erased information. Then, this paper argues that label smoothing actually is compatible with knowledge distillation, and show several empirical results as evidence. Lastly, this paper suggests that the performance of the teacher model is a more directly related factor for determining the performance of the student model.