Year: 2021

Abstract:
This work investigates the compatibility between label smoothing (LS) and knowledge distillation (KD). Contemporary findings addressing this thesis statement take dichotomous standpoints. Specifically, Müller et al. (2019) claim that LS erases relative information in the logits; therefore a LS-trained teacher can hurt KD. On the contrary, Shen et al. (2021b) claim that LS enlarges the distance between semantically similar classes; therefore a LS-trained teacher is compatible with KD. Critically, there is no effort to understand and resolve these contradictory findings, leaving the primal question − to smooth or not to smooth a teacher network? − unanswered. In this work, we establish a foundational understanding on the compatibility between LS and KD. We begin by meticulously scrutinizing these contradictory findings under a unified empirical consistency. Through our profound investigation, we discover that in the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes. This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective. We show this systematic diffusion qualitatively by visualizing penultimate layer representations, and quantitatively using our proposed relative distance metric called diffusion index (η). Importantly, our discovered systematic diffusion was the missing concept which is instrumental in understanding and resolving these contradictory findings. Our discovery is comprehensively supported by large-scale experiments and analyses including image classification (standard, fine-grained), neural machine translation and compact student network distillation tasks spanning across multiple datasets and teacher-student architectures. Finally, we shed light on the question − to smooth or not to smooth a teacher network? − in order to help practitioners make informed decisions.

heading: 1 INTRODUCTION
text: This paper deeply investigates the compatibility between label smoothing (Szegedy et al., 2016) and knowledge distillation (Hinton et al., 2015). Specifically, we aim to explain and resolve the contradictory standpoints of Müller et al. (2019) and Shen et al. (2021b), thereby establishing a foundational understanding on the compatibility between label smoothing (LS) and knowledge distillation (KD). Both LS and KD involve training a model (i.e.: deep neural networks) with soft-targets. In LS, instead of computing cross entropy loss with the hard-target (one-hot encoding) of a training sample, a soft-target is used, which is a weighted mixture of the one-hot encoding and the uniform distribution. A mixture parameter α is used in LS to specify the extent of mixing. On the other hand, KD involves training a teacher model (usually a powerful model) and a student model (usually a compact model). The objective of KD is to transfer knowledge from the teacher model to the student model. In the most common form, the student model is trained to match the soft output of the teacher model. The success of KD has been attributed to the transference of logits’ information about resemblances between instances of different classes (logits are the inputs to the final softmax which produces the soft targets). In KD (Hinton et al., 2015), a temperature T is introduced to
facilitate the transference: an increased T may produce more suitable soft targets that have more emphasis on the probabilities of incorrect classes (or equivalently, logits of the incorrect classes).
To smooth or not to smooth? Recently, a fair amount of research has been conducted to understand the relationship between LS and KD (Müller et al., 2019; Shen et al., 2021b; Lukasik et al., 2020; Yuan et al., 2020). One of the most intriguing and controversial discussion is the compatibility between LS and KD. Particularly, in KD, does label smoothing in a teacher network suppress the effectiveness of the distillation?
Müller et al. (2019) are the first to investigate this topic, and their findings suggest that applying LS to a teacher network impairs the performance of KD. In particular, they visualize the penultimate layer representations in the teacher network to show that LS erases information in the logits about resemblances between instances of different classes. Since this information is essential for KD, they conclude that applying LS for a teacher network can hurt KD. • ‘If a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective.” (Müller et al., 2019) • “Label smoothing can hurt distillation” (Müller et al., 2019) The conclusion of Müller et al. (2019) is widely accepted (Khosla et al., 2020; Arani et al., 2021; Tang et al., 2021; Mghabbar & Ratnamogan, 2020; Shen et al., 2021a). However, very recently, this is questioned by Shen et al. (2021b). In particular, their work discussed a new finding: information erasure in teacher can actually enlarge the central distance between semantically similar classes, allowing the student to learn to classify these categories easily. Shen et al. (2021b) claim that this benefit of using a LS-trained teacher outweighs the detrimental effect due to information erase. Therefore, they conclude that LS in a teacher network does not suppress the effectiveness of KD. • “Label smoothing will not impair the predictive performance of students.” (Shen et al., 2021b) • “Label smoothing is compatible with knowledge distillation” (Shen et al., 2021b)
LS and KD compatibility remains mysterious. We were perplexed by the seemingly contradictory findings by Müller et al. (2019) and Shen et al. (2021b). While the latter has shown empirical results to support their own finding, their work does not investigate the opposite standpoint and contradictory results by Müller et al. (2019). Critically, there is no effort to understand and resolve the seemingly contradictory arguments and supporting evidences by Müller et al. (2019) and Shen et al. (2021b). Consequently, for practitioners, it remains unclear as to under what situations LS can be applied to the teacher network in KD, and under what situations it must be avoided.
Our contributions. In this work, we conduct an empirical investigation to establish a foundational understanding on the compatibility between LS and KD. We begin by meticulously scrutinizing the opposing findings of Müller et al. (2019) and Shen et al. (2021b). In particular, we discover that in the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes. This systematic diffusion essentially curtails the benefits (as claimed by Shen et al. (2021b) ) obtained by distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective. We perform largescale distillation experiments using ImageNet-1K to comprehensively demonstrate this systematic diffusion in the student qualitatively using penultimate layer visualizations, and quantitatively using our proposed relative distance metric called diffusion index (η).
Our finding on systematic diffusion is very critical when distilling from a LS-trained teacher. Particularly, we argue that this diffusion maneuvers the penultimate layer representations learnt by the student of a given class in a systematic way that targets in the direction of semantically similar classes. Therefore, this systematic diffusion directly curtails the distance enlargement (between semantically similar classes) benefits obtained by distilling from a LS-trained teacher. Our qualitative and quantitative analysis with our proposed relative distance metric (η) in Sec 4 aims to establish not only the existence of this diffusion, but also establish that such diffusion is systematic.
We further conduct extensive experiments using fine-grained image classification (CUB200-2011), neural machine translation (English to German, English to Russian translation using IWSLT) and compact student network distillation (using MobileNetV2) tasks to support our key finding on systematic diffusion. Importantly, using systematic diffusion analysis, we explain and resolve the contradictory findings by Müller et al. (2019) and Shen et al. (2021b), thereby establishing a foundational understanding on the compatibility between LS and KD. Finally, using our discovery on systematic diffusion, we provide empirical guidelines for practitioners regarding the combined use of LS and KD. We summarize our key findings in Table 1. The key takeaway from our work is:
• In the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes. This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective. Specifically, systematic diffusion was the missing concept that is instrumental in explaining and resolving the contradictory findings of Müller et al. (2019) and Shen et al. (2021b), thereby shedding light on whether to smooth or not to smooth a teacher network.
Paper organization. In Sec 2, we review LS and KD. In Sec 3, we review key findings of Müller et al. (2019) and Shen et al. (2021b) to to emphasize the research gap. Our main contribution is Sec 4, where we introduce our discovered systematic diffusion, conduct qualitative, quantitative and analytical studies to verify that the diffusion is not isotopic but systematic towards semanticallysimilar classes, and therefore it directly curtails the benefits of using a LS-trained teacher. In Sec 5, we perform rich empirical studies to support our main finding on Systematic Diffusion. In Sec 6, we provide our perspective regarding the combined use of LS and KD as empirical guidelines for practitioners, and finally conclude this study.

heading: 2 PREREQUISITES
text: Label Smoothing (LS) (Szegedy et al., 2016): LS was formulated as a regularization strategy to alleviate models’ over-confidence. LS replaces the original hard target distribution with a mixture of original hard target distribution and the uniform distribution characterized by the mixture parameter α. Consider the formulation of LS objective with mixture parameter α as follows: Let pk,wk represent the probability and last layer weights (including biases) corresponding to the k-th class. Let x, yk, yLSk represent the penultimate layer activations, true targets and LS-targets where yk = 1 for the correct class and 0 for all the incorrect classes1. xT is the transpose of x. Then for a classification network trained with LS containing K classes, we minimize the cross entropy loss between LS-targets yLSk and model predictions pk given by LLS(y,p) = ∑K k=1 −yLSk log(pk),
where pk = exp(xTwk)/ ∑K l=1 exp(x Twl) and yLSk = yk(1− α) + αK .
Knowledge distillation (KD) Hinton et al. (2015): KD uses a larger capacity teacher model(s) to transfer the knowledge to a compact student model. The success of KD methods is largely at-
1x is concatenated with 1 at the end to include bias as wk includes biases at the end.
tributed to the information about incorrect classes encoded in the output distribution produced by the teacher model(s) (Hinton et al., 2015). Consider KD for a classification objective. Let T indicate the temperature factor that controls the importance of each soft target. Given the k-th class logit xTwk, let the temperature scaled probability be pk(T ). For KD training, let the loss be LKD. For LKD, we replace the cross entropy loss H(y,p) with a weighted sum (parametrized by β) of H(y,p) and H(pt(T ),p(T )) where pt(T ),p(T ) correspond to the temperature-scaled teacher and student output probabilities. That is, pk(T ) = exp(x Twk T )/ ∑K l=1 exp( xTwl T ) and LKD = (1 − β)H(y,p) + βT 2H(pt(T ),p(T )). Following Hinton et al. (2015) T 2 scaling is used for the soft-target optimization as T will scale the gradients approximately by a factor of T 2. Following Müller et al. (2019); Shen et al. (2021b), we set β = 1 for this study since we primarily aim to isolate and study the effects of KD. β = 1 achieves good performance (Shen et al., 2021b).

heading: 3 A CLOSER LOOK AT LS AND KD COMPATIBILITY
text: In this section, we review the contradictory findings of Müller et al. (2019) and Shen et al. (2021b) from the perspective of information erase in LS-trained teacher. This discussion is a necessary preamble to discuss our main finding, Systematic Diffusion in the student in Sec 4.
Information Erase in LS-trained teacher. LS objective optimizes the probability of the correct class to be equal to 1 − α + α/K, and incorrect classes to be α/K. This directly encourages the differences between logits of the correct class and incorrect classes to be a constant (Müller et al., 2019) determined by α. Following Müller et al. (2019), the logit xTwk can be approximately measured using the squared Euclidean distance between penultimate layer’s activations and the template corresponding to class k. That is, xTwk can be approximately measured by ∥x−wk∥2. This allows to establish 2 important geometric properties of LS (Müller et al., 2019): With LS, penultimate layer activations 1) are encouraged to be close to the template of the correct class (large logit value for the correct class, therefore small distance between the activations and the correct class template), and 2) are encouraged to be equidistant to the templates of the incorrect classes (equal logit values for all the incorrect classes). This results in penultimate layer activations to tightly cluster around the correct class template compared to the model trained with standard cross entropy objective. We demonstrate this clearly in Figure 1 Observation 1. With LS applied on the ResNet-50 model, we observe that the penultimate layer representations become much tighter. As a result, substantial information regarding the resemblances of these instances to those of other different classes is lost. This is referred to as the information erase in LS-trained network (teacher) (Müller et al., 2019).
Claim 1: Information erase in LS-trained teacher cause LS and KD to be Incompatible (Müller et al., 2019): Müller et al. (2019) are the first to investigate this compatibility, and they argue that the information erasure effect due to LS (shown in Figure 1 Observation 1) can impair KD. Given the prominent successes in KD methods being largely attributed to dark knowledge/ inter-class information emerging from the trained-teacher (Hinton et al., 2015; Tang et al., 2021), the argument by Müller et al. (2019) that LS and KD are incompatible due to information loss in the logits is generally convincing and widely accepted (Khosla et al., 2020; Arani et al., 2021; Tang et al., 2021; Mghabbar & Ratnamogan, 2020; Shen et al., 2021a). This is also supported by empirical evidence.
Claim 2: Information erase in LS-trained teacher provides distance enlargement benefits between semantically similar classes, resulting in LS and KD to be Compatible (Shen et al., 2021b): Recently an interesting finding by Shen et al. (2021b) argue that LS and KD are compatible. Though they agree that information erasure generally happens with LS, their argument focuses more on the effect of LS on semantically similar classes. They argue that information erase in LStrained teacher can promote enlargement of central distance of clusters between semantically similar classes. This allows the student network to easily learn to classify semantically similar classes which are generally difficult to classify in conventional training procedures. We show this increased separation between semantically similar classes with LS in Figure 1 Observation 1. It can be observed that the central distance between the clusters of standard poodle and miniature poodle increases with using LS on the ResNet-50 teacher. In our work, we further extend to show that this property is inherited by the ResNet-18 student as well in Observation 2. We remark that this inheritance is not shown by Shen et al. (2021b). This finding by Shen et al. (2021b) is largely supported by experiments and quantitative results. Though Shen et al. (2021b) claim that the benefit derived from larger separation between semantically similar classes outweigh the drawbacks due to information erase, thereby making LS and KD compatible, their investigation does not address the contradictory findings and empirical results obtained by Müller et al. (2019).
Research Gap: Studied in isolation, both these contradictory arguments are convincing and are well supported empirically. This has caused serious perplexity among the research community regarding the combined use of LS and KD.

heading: 4 SYSTEMATIC DIFFUSION IN STUDENT
text: Through profound investigation, we discover an intriguing phenomenon occurring in the student called systematic diffusion when distilling from a LS-trained teacher at higher T . Particularly, this diffusion maneuvers the penultimate layer representations learnt by the student of a given class in a systematic way that targets in the direction of semantically similar classes. This systematic
diffusion is critical as it directly curtails the distance enlargement benefits between semantically similar classes when distilling from a LS-trained teacher.
Penultimate layer visualization as evidence of systematic diffusion. We follow Müller et al. (2019), and use their visualization method based on linear projections of the penultimate layer representations. See Figure 1 for visualization (We discuss Figure 1 deeply in Sec 5). Particularly, our discovery on systematic diffusion affects the distance between semantically similar classes in the student when distilled from a LS-trained teacher at higher T . This systematic diffusion can be clearly observed by visualizing the penultimate layer representations of the student. We include the visualization algorithm and Numpy-style code in Supplementary E.
Given that the increased cluster center separation between semantically similar classes being the reason for the compatibility claim between LS and KD (Shen et al., 2021b), we discover that this cluster center separation is affected by the degree of systematic diffusion in the student. Importantly, systematic diffusion is instrumental in explaining and resolving the contradictory findings of Müller et al. (2019) and Shen et al. (2021b), thereby establishing a foundational understanding on the compatibility between LS and KD.
Formulation of Diffusion index (η) to measure systematic diffusion. To comprehensively support our discovery, we formulate a novel metric called diffusion index (η) to quantitatively measure this systematic diffusion. Given that the interpretation of ‘semantics’ is rather subjective, we carefully construct this metric to support our discovery. The basic idea of this metric is to quantify the distance change between clusters in the student network when distilled from a LS-trained teacher at higher T . Critically, the design of the metric is to verify that the diffusion is systematic: i.e. at higher T , intercluster distance decreases for semantic similar classes and increases (relatively) for the remaining classes. As explained in the Introduction, this systematic behaviour is critical in our study. There are important considerations in formulating this metric discussed below.
• A target class π can be characterized by the centroid of the penultimate layer representations of samples belonging to π. Let the centroid of class π be cπ .
• Consider the sets S1, S2 where S1 contains |S1| semantically similar classes to π and S2 contains |S2| semantically dissimilar classes to π. |S| indicates the number classes in the set S. For easier understanding, consider 2 classes p, q where p ∈ S1, q ∈ S2.
• The proximity of cπ to cp can approximately measure the semantic similarity between class π and p. Though this proximity can be directly measured by Euclidean distance between centroids, it requires some careful thought on normalization. The reason is as follows: What we are interested is how close is centroid of class π to class p compared to class q. In other words, we are interested in the relative distance between centroids of classes (π, p) and (π, q). Hence to measure this relative distance we normalize the distance by the sum of pairwise distance from cπ to centroids of all other classes in S.
• Do note that the location of the centroids will change with temperature. In fact, we are interested in the change of centroids with increased T to measure this systematic diffusion. We formulate the following diffusion index η to measure the average percentage change in distances between semantically similar classes and semantically dissimilar classes with respect to a target class.
Given a class π and its centroid cπ . Let the centroid of a class k be represented by ck, k ∈ S1, S2. Let the temperature be T . We quantify the relative distance between classes π and k:
d(cπ(T ), ck(T )) = ∥cπ(T )−ck(T )∥2 R , where R = ∑ p∈S1 ∥cπ(T ) − cp(T )∥ 2 + ∑ q∈S2 ∥cπ(T ) − cq(T )∥2 (normalization constant). The diffusion index η measures the average percentage change in distance between a target class π and classes in the set S when temperature is changed from T1 to T2 defined as follows:
η(T1, T2;π, S) = 1 |S| ∑ k∈S d(cπ(T2), ck(T2))− d(cπ(T1), ck(T1)) d(cπ(T1), ck(T1))
(1)
Substituting S1, S2 into S of Eq. 1, we have: i) η(T1, T2;π, S1) measures the change in relative distance between class π and its semantically similar class in S1. ii) η(T1, T2;π, S2) measures the change in relative distance between class π and its semantically dissimilar class in S2.
To give more intuition on η, consider the 3 class example (Fig. 1): miniature poodle (as π class), standard poodle (as p ∈ S1 class and |S1| = 1), submarine (as q ∈ S2 class and |S2| = 1). As T increases from T1 = 1 to T2 = 3, the relative distance between miniature poodle and standard poodle will reduce due to diffusion (Fig. 1), therefore d(cπ(T2), cp(T2)) < d(cπ(T1), cp(T1)). From Eq. 1, it is clear that the numerator will be negative. We normalize by the reference distance to calculate the percentage change. As a result, the average percentage change over S1 will give a negative value, indicating the diffusion towards semantically similar classes. Similarly when measured over S2, the average percentage change between miniature poodle and submarine will be positive (because d(cπ(T2), cq(T2)) > d(cπ(T1), cq(T1)) as we observe in Fig. 1) indicating diffusion away from the target class.
Why is this diffusion systematic and not isotopic? We revisit discussion from Hinton et al. (2015) to motivate the intuition behind this systematic diffusion. Hinton et al. (2015) introduce T to scale the logits at the final softmax in order to produce soft targets that are more suitable for transfer. As argued by Hinton et al. (2015) on MNIST classification, a sample of ‘2’ may be assigned a probability of 10−6 of being a ‘3’ and 10−9 of being a 7. The resemblance between ‘2’ and ‘3’ is valuable information, but a probability of 10−6 has negligible influence on the loss when distilling to student. Hinton et al. (2015) introduce a temperature T to emphasize the probabilities of such incorrect classes: during KD, their T -scaled counterparts have more noticeable effects on the student. On the other hand, the effect of T scaling on the probability of 10−9 is negligible; consequently, the T -scaled counterparts of such probabilities remain to have unnoticeable effects on the student.
In particular, for a given sample of ground-truth class k∗, we let ptk∗ represent the probability of the correct class output by the teacher, ptm represent the probability of one of the K − 1 incorrect classes. Among these K − 1 ptm, one or a few could be significantly larger than the other; we refer such probability as ptml (i.e.: probability of being a 3 in the above example). In particular, the class ml is usually a semantically similar class of class k∗, therefore ptml is not negligible for a class k ∗ sample (See Figure 2). For the rests of ptm which are almost zero (noise level), we refer them as ptms (e.g., probability of being a 7 in the above example). Therefore, {ptm} = {ptml} ∪ {ptms}. Usually, we have ptml ≫ ptms and ptms ≈ 0. We remark that {ptm} are not all the same and can be observed even for a LS-trained teacher. It is because logits’ information is not completely erased (see Figure 2).
When KD of an increased T is used, the soft output of the teacher is scaled and becomes pt(T ). In particular, the effect of T scaling is to bring ptml closer to p t k∗ , i.e., p t ml(T ) is closer to p t k∗(T ) relatively. Consequently, with soft target pt(T ), student is encouraged to produce a penultimate representation of a class k∗ sample that is closer to the incorrect class ml. This results in systematic diffusion of representations of class k∗ towards the incorrect class ml. This can be observed in Figure 1 Observation 3 for standard poodle activations (here class ml being miniature poodle), and similarly for miniature poodle activations. On the other hand, because ptms is negligi-
bly small, even with T scaling ptms(T ) remains negligible and has unnoticeable effect for student’s penultimate representation. Therefore, the diffusion due to an increased T is not isotopic but towards semantically similar classes (class ml). We provide more detailed discussion in Supplementary D.
We remark that this systematic diffusion can sometimes be observed when using a teacher without LS, see Figure 1, row 2 subplot 1 and row 3 subplot 1. For a teacher without LS (i.e. without information erase), this systematic diffusion could in fact be advantageous in some cases, as it improves generalization of the student network using the rich logits’ information about instance resemblances. However, we focus on our thesis statement: compatibility between LS and KD. In our case, systematic diffusion in student due to KD at an increased T curtails the distance enlargement (between semantically similar classes) benefits of using a LS-trained teacher, rendering KD ineffective.

heading: 5 EMPIRICAL STUDIES
text: In this section, we conduct large-scale KD experiments (classification) using ImageNet-1K. We remark that LS and KD are compatible when with all the other factors fixed (including T ), student distilled from a LS-trained teacher outperforms the student distilled from a teacher trained without LS. We use ResNet-50 teacher and ResNet-18, ResNet-50 students similar to Shen et al. (2021b). Results are shown in Table 2.
Penultimate layer visualization analysis. We show this systematic diffusion in ResNet-18 student using Figure 1 Observation 3. We focus on the two semantically similar classes: miniature poodle, standard poodle. Given the same LS-trained ResNet-50 teacher and using the exact distillation process, we observe that at increased temperatures (T = 1 to T = 3), the above semantically similar classes start to diffuse. We also observe that class submarine diffuses towards another class which is semantically similar to submarine (not shown in the figure). Because of this systematic diffusion, the central cluster distances between miniature poodle and standard poodle reduces with increased T in the presence of LS-trained teacher. Consequently, this systematic diffusion results in detrimental performance in the student causing an accuracy drop of 5.05% as shown in Table 2 A. ResNet-50 student visualization is included in Supplementary A.
Analysis using diffusion index (η). We quantitatively illustrate systematic diffusion in the ResNet18 student using η for 10 target classes in Table 3. We clearly observe that η(T1 = 1, T2 = 3;π, S1) < 0 and η(T1 = 1, T2 = 3;π, S2) > 0 for all these 10 target classes, thereby quantitatively showing that the penultimate layer representations are diffused towards semantically similar classes when distilled from a LS-trained teacher at a larger temperature. This systematic diffusion results in detrimental performance of the student resulting in an accuracy drop of 5.05% as shown in Table 2 A. We show similar analysis for ResNet-50 student in Supplementary A.
Resolving the contradictory claims using systematic diffusion. The seemingly contradictory findings of Müller et al. (2019) and Shen et al. (2021b) can be resolved using our discovery on
systematic diffusion as follows: Müller et al. (2019) make the incompatibility claim between LS and KD due to observing students distilled from LS-trained teacher performing inferior to students distilled from teacher trained without LS at higher T . On the contrary, Shen et al. (2021b) make the compatibility claim between LS and KD due to observing students distilled from LS-trained teacher performing superior to students distilled from teacher trained without LS at lower T (i.e.: T = 1). Critically, our main finding shows that in the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes. This systematic diffusion essentially curtails the distance enlargement (between semantically similar classes) benefits of distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective. More specifically, in the presence of a LS-trained teacher, the degree of systematic diffusion is low when distilling at lower T thereby making LS and KD compatible. On the other hand, the degree of systematic diffusion is relatively higher when distilling at higher T , thereby making LS and KD incompatible. Our findings are summarized in Table 1. Importantly, we remark that systematic diffusion was the missing concept that is instrumental in resolving the contradictory claims of Müller et al. (2019) and Shen et al. (2021b).
Extended experiments to further support main finding. We perform extensive experiments using fine-grained image classification, neural machine translation and compact student network distillation. These results and analysis further support our main finding and are reported in Supplementary B.1, B.2 and B.3 respectively.

heading: 6 DISCUSSION AND CONCLUSION
text: To smooth or not to smooth? Based on our study, we provide our perspective on this question: “To smooth or not to smooth a teacher network?” While increased T is believed to be a helpful empirical trick (Also observed in some of our experiments when distilling from a teacher trained without LS) to produce better soft-targets for KD, we convincingly show that in the presence of LS-trained teacher, an increased T causes systematic diffusion of penultimate layer representations towards semantically similar classes in the student. This systematic diffusion directly curtails the distance enlargement (between semantically similar classes) benefits of a LS-trained teacher, thereby rendering KD ineffective at increased T . As a rule of thumb, we suggest to use lower T (i.e.: T = 1) for KD in the presence of a LS-trained teacher to avoid systematic diffusion.
Conclusion. Focusing on the compatibility between LS and KD, we have conducted an empirical study to investigate the seemingly contradictory findings of Müller et al. (2019) and Shen et al. (2021b). Through comprehensive scrutiny of these works, we discover an intriguing phenomenon called systematic diffusion: That is in the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes. This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective. We showed this systematic diffusion both qualitatively and quantitatively using extensive analysis. We also supported our findings with large scale experiments including image classification (standard, finegrained), neural machine translation and compact student network distillation tasks. Critically, using our discovery on systematic diffusion, we resolve the contradictory findings of Müller et al. (2019) and Shen et al. (2021b), thereby establishing a foundational understanding regarding the compatibility between LS and KD. Finally, based on our new finding, we discussed our viewpoints on the question: to smooth or not to smooth a teacher network.

heading: 7 REPRODUCIBILITY STATEMENT
text: Code Submission Our submission includes Pytorch code to allow for research reproducibility. Refer README.txt for specific instructions. The submitted code contains the following:
• ImageNet LS and KD code to reproduce Table 2 in the main paper ( src/imagenet/train teacher.py, src/imagenet/train student.py ).
• Fine-grained classification/ Compact neural network distillation using CUB200-2011 dataset to reproduce Tables 5 and 6 ( src/cub/train teacher.py, src/cub/train student.py, ).
• Penultimate layer visualization code to reproduce all visualizations in the main paper. ( src/visualization/alpha-LS-KD imagenet centroids.py ).
• We provide clear bash file execution points to train all our models. (See /bash scripts)
Pre-trained models submission Our submission includes all pretrained models for image classification using ImageNet-1K, fine-grained classification using CUB200-2011, neural machine translation using IWSLT and compact student distillation. We submit the pretrained models for both teachers and students. All these models can be downloaded at this Google Drive Link. All our claims reported in Main paper Table 2 and Supplementary tables 5, 6, 7, 8, 9 can be reproduced using the submitted models.
Docker information : To allow for training in containerised environments (HPC, Super-computing clusters), please use nvcr.io/nvidia/pytorch:20.12-py3 container.

heading: Experiment details and hyper-parameters
text: ImageNet-1K: For ImageNet experiments, we follow similar setup as Shen et al. (2021b) and use ILSVRC2012 version. For training LS networks, we train for 90 epochs with initial learning rate 0.1 decayed by a factor of 10 every 30 epochs. For KD experiments, we train for 200 epochs with initial learning rate 0.1 decayed by a factor of 10 every 80 epochs. We conducted a grid search for hyper-parameters as well. For all experiments, we use a batch size of 256 and SGD with momentum 0.9 . For data augmentation, we use random crops and random horizontal flips. All experiments were repeated 3 times. For visualization of penultimate layer representations, we use 150 samples for training set and 50 samples for validation set.
Fine-grained classification and compact student distillation. We follow similar setup as Shen et al. (2021b). For training both LS and KD networks, we train for 200 epochs with initial learning rate 0.01 decayed by a factor of 10 every 80 epochs. We conducted a grid search for hyper-parameters as well. For all experiments, we use a batch size of 256 and SGD with momentum 0.9 . All experiments were repeated 3 times. For data augmentation, we use random crops, random rotation, color jitter and random horizontal flips. For visualization of penultimate layer representations, we use all samples for training and validation sets.
Neural Machine Translation (NMT) We use IWSLT dataset. We follow similar setup as Shen et al. (2021b). We use Adam as the optimizer, lr with 0.0005, dropout with drop rate as 0.3, weight-decay with 0 and max tokens with 4096, all of these hyper-parameters are following settings of Shen et al. (2021b). These hyper-parameters were used for both translation tasks (English to German, English to Russian). We use the code here similar to Shen et al. (2021b).

heading: SUPPLEMENTARY MATERIALS

heading: CONTENTS OF THIS SUPPLEMENTARY
text: This Supplementary provides additional experiments and results (penultimate layer visualization and η analysis) to further support our main finding on Systematic diffusion. The Supplementary materials are organized as follows:
• Section A: Visualizations and η analysis for ResNet-50 Student (ImageNet-1K) • Section B: Extended experiments and Analysis
– Section B.1: Fine-grained classification – Section B.2: Neural machine translation – Section B.3: Compact student network distillation
• Section C: Standard Deviation of ImageNet-1K experiments • Section D: Additional Discussion: Why this diffusion is systematic and not isotopic? • Section E: Algorithm for Projection and visualization of penultimate layer representations • Section F: Semantically similar / dissimilar classes
– Section F.1: Using standard, pre-defined ImageNet knowledge graph as a prior – Section F.2: Using distance in the feature space
• Section G: Case study: Smoothness of targets are insufficient to determine KD performance – Section G.1: Case study at lower T with same degree of smoothness – Section G.2: Case study at moderately higher T with same degree of smoothness – Section G.3: Case study at very high T with same degree of smoothness
• Section H: Class-wise accuracy for target classes • Section I: Additional Exploration of α and T • Section J: Alternative characterization of cluster distance • Section K: Additional References
A VISUALIZATIONS AND η ANALYSIS FOR RESNET-50 STUDENT (IMAGENET-1K)

heading: B EXTENDED EXPERIMENTS

heading: B.1 FINE-GRAINED IMAGE CLASSIFICATION
text: We conduct fine-grained image classification experiments using CUB200-2011 dataset (Wah et al., 2011) similar to Shen et al. (2021b). Similar to Shen et al. (2021b), we use ResNet-50 teacher and ResNet-18, ResNet-50 students. The results are shown in Tables 5 and 6 respectively. Similar to ImageNet-1K, we select two semantically similar classes (Great Grey Shrike and Loggerhead Shrike) and one semantically dissimilar class (Black footed Albatross) in CUB200-2011 dataset to clearly demonstrate this systematic diffusion. We show the systematic diffusion using penultimate layer visualization for CUB200-2011 in Figure 4. These results also comprehensively support our main finding: In the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes.
This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective.

heading: B.2 NEURAL MACHINE TRANSLATION
text: We conduct neural machine translation experiments using IWSLT-2014 dataset. We perform translation on English - German task (Similar to Shen et al. (2021b)) and English - Russian task. The results are shown in Tables 7 and 8 respectively. These results also comprehensively support our main finding: In the presence of a LS-trained teacher, KD at higher temperatures systematically diffuses penultimate layer representations learnt by the student towards semantically similar classes. This systematic diffusion essentially curtails the benefits of distilling from a LS-trained teacher, thereby rendering KD at increased temperatures ineffective.

heading: B.3 COMPACT STUDENT NETWORK DISTILLATION
text: KD is very widely explored in neural network compression applications. Specifically, we use KD to transfer knowledge from a large and powerful teacher (i.e.: ResNet-50, DenseNet121 etc) to compact student networks (i.e.: MobileNetV2). In this section, we conduct KD experiments using ResNet-50
teacher and MobileNetV2 student on the fine-grained classification task (using CUB200-2011). The results are shown in table 9

heading: C STANDARD DEVIATION OF IMAGENET-1K EXPERIMENTS

heading: D ADDITIONAL DISCUSSION: WHY THIS DIFFUSION IS SYSTEMATIC AND
text: We provide more perspective into why this diffusion is systematic and not isotopic. We use the LS-trained ResNet-50 teacher (same one in Figure 2) trained on ImageNet-1K to numerically show more evidence as to why this diffusion is systematic and not isotopic. Particularly we show that only very few classes (out of the 1000 classes in ImageNet-1K) have probabilities significantly larger than others. We examine the output probability for 3 classes: standard poodle samples, golden retriever samples and thunder snake samples (We choose this classes randomly, similar analysis can be done for other classes as well).
For each class, we compute the average output probability for 1300 training samples, and observe following: Let p1 be the largest probability which is also probability of the correct class.
• For the average probability of standard poodle samples, the second largest probability, p2 (miniature poodle) is at least 100x larger than 976 other probabilities (out of 999 probabilities)
• For the average probability of golden retriever samples, the second largest probability, p2 (Labrador retriever) is at least 100x larger than 924 other probabilities (out of 999 probabilities)
• For the average probability of thunder snake samples, the second largest probability, p2 (ringneck snake) is at least 100x larger than 964 other probabilities (out of 999 probabilities)
Can this support the diffusion is systematic? We use results of standard poodle for discussion. When KD of an increased T is used, these probabilities are scaled, and p2 is brought closer to p1, see Figure 2. Consequently, student is encouraged to produce penultimate layer representations of standard poodle samples that are closer to miniature poodle. This results in diffusion of penultimate layer representations of standard poodle towards miniature poodle, curtailing the distance enlargement benefit of distilling from a LS-trained teacher. For the 976 classes which have probabilities at least 100x smaller than that of miniature poodle, even with T scaling, the probabilities remain negligible. They have no influence on the representation of standard poodle. Therefore diffusion of standard poodle will be towards miniature poodle and several semantically similar classes but there is no diffusion towards these 976 classes. Therefore, the diffusion is systematic and is not isotopic.
In this discussion, we use 100x to mean significance/insignificance. If a probability pi is 100x smaller than another probability pj , then even with T scaling pi remains insignificant compared to pj .

heading: E ALGORITHM FOR PROJECTION AND VISUALIZATION OF PENULTIMATE LAYER REPRESENTATIONS
text: Algorithm 1 Projection and visualization of penultimate layer features Input: 1⃝ High dimensional (h) features (X,Y ) of three classes extracted from penultimate layers of the trained model f 2⃝ Model weight w of the final layer of f Output: The projected 2-D features X ′
Compute the othonormal basis as w′ = qr-decomposition (w) # dim = (h, 3) for all samples do
Obtain the projected features on new basis via dot product: proj(X) = np.dot(X , w′) # dim = (∗, 3) Dimension reduction from 3-D to 2-D via PCA(proj(X)) # dim = (∗, 2)
end for return 2-D features: PCA(proj(X))
Algorithm 2 NumPy-style pseudo-code of the visualization algorithm
1 # Inputs 2 # weights_path: weights path of the final layer of your trained model 3 # feature_path: feature path of the penultimate layer high dimension features extracted by your trained model 4 5 # Outputs 6 # 2-D features of each class 7 8 # ------------------------------------------------------------------- # 9 # Step 0. Init settings and select the class to visualize
10 CLASSES = [’miniature_poodle’, ’standard_poodle’, ’submarine’] 11 color = [’r’, ’g’, ’b’] 12 model = ’resnet18’ # the student model 13 14 # Step 1. Compute the orthonormal basis 15 weights = np.load(weights_path) # load the final layer weights 16 basis, _ = np.linalg.qr(weights.T) # dim=(*, 3) 17 18 # Step 2. Load the extracted features 19 num_sample = 150 # We sample 150 images per class 20 output_feature = np.load(feature_path) 21 22 # Step 3. Project the high dimension features to the new 3-D subspace 23 output_project = np.dot(output_feature, basis) 24 25 # Step 4. Dimension reduction from 3-D to 2-D using PCA 26 pca = PCA(n_components=2) 27 pca.fit(output_project) 28 output_array = pca.transform(output_project) 29 30 # Step 5. Plot the features in a 2-D plane 31 for i, subclass in enumerate(CLASSES): 32 plt.scatter(output_array[i * num_sample:(i + 1) * num_sample, 0], 33 output_array[i * num_sample:(i + 1) * num_sample, 1], 34 c=color[i], label=subclass)

heading: F SEMANTICALLY SIMILAR / DISSIMILAR CLASSES
text: Given a target class π, let the set of semantically similar and dissimilar classes be S1,, S2 respectively. In this section, we discuss 2 important methods for identifying S1, S2 for the target class π.

heading: F.1 METHOD 1: USING STANDARD, PRE-DEFINED IMAGENET KNOWLEDGE GRAPH AS A PRIOR
text: We use ImageNet hierarchy derived from WordNet (Fellbaum, 1998) to select semantically similar classes and semantically dissimilar classes to quantify systematic diffusion. WordNet (Fellbaum, 1998) is a laboriously hand-coded lexical database linking words into semantic relations including synonyms, hyponyms, and meronyms 2. Do note that ImageNet is organized using WordNet hierarchy. A web browser version of the ImageNet hierarchy can be accessed at this link (You can click any node to browse images that correspond to the associated synset)
We use this ImagNet hierarchy to select semantically similar classes and semantically dissimilar classes for the target class π. This way, we ensure the selection of semantically similar classes (S1) and semantically dissimilar classes (S2) is based on a strong prior (knowledge graph) to support our main finding.

heading: F.2 METHOD 2: USING DISTANCE IN THE FEATURE SPACE TO QUANTITATIVELY DEFINE SEMANTICALLY SIMILAR / DISSIMILAR CLASSES
text: This method is a quantitative approach for defining semantically similar / dissimilar classes. Specifically, we consider the official ResNet-50 model trained on ImageNet-1K (classification). We use the validation set of ImageNet-1K and extract the penultimate layer representations for all the samples. For each class, we consider the centroid of the penultimate layer representations as the class prototype and calculate the centroid-centroid distance between all the classes (This will give a symmetric matrix of 1000 x 1000).
For selecting S1: Next, for the target class π, we identify the closest 1% of classes (10 out of 999 classes) using the centroid-centroid distances. These would be the semantically similar classes to the target class as they have the smallest distances to the centroid of the target class.
For selecting S2: Next, for the target class π, we identify the distant 90% of classes (900 out of 999 classes) using the centroid-centroid distances discussed above. These would be the semantically dissimilar classes to the target class as their centroids lie much far away from the centroid of the target class.
Consistency measurements between the 2 methods: Let the semantically similar and dissimilar classes identified using method 1 be S1,qualitative, S2,qualitative respectively. Let the semantically similar and dissimilar classes identified using method 1 be S1,quantitative, S2,quantitative respectively. In this section, we measure the consistency between qualitative selection of S1,qualitative, S2,qualitative (method 1) and the quantitative definition of S1,quantitative, S2,quantitative (method 2). This consistency measurements are shown for all the target classes in the Table 12. As one can clearly observe both method 1 and method 2 agree 85% on average for semantically similar classes and 94% on average for semantically dissimilar classes. Do note that we use pre-defined knowledge graph for ImageNet-1K as prior (method 1) to select the semantically similar / dissimilar classes for our η computation in Table 3.
2https://en.wikipedia.org/wiki/WordNet

heading: G CASE STUDY: SMOOTHNESS OF TARGETS ARE INSUFFICIENT TO DETERMINE KD PERFORMANCE
text: An interesting perspective is whether the degree of smoothness of targets produced by an LS-trained teacher can determine the KD performance (of the student). We acknowledge that smoothness of targets produced by the teacher at different temperatures is important. However, we quantitatively show that the degree of smoothness cannot adequately explain the KD performance in the presence of an LS-trained teacher. More specifically, we show that the KD performance in the presence of LS-trained teachers can be explained by our discovered systematic diffusion and not directly using the degree of smoothness. The detailed study is discussed below.
Our view: The degree of smoothness of targets is rather unable to explain the performance of KD. We show this using 3 comprehensive case studies comprising 7 counterexamples.
Measuring smoothness of targets: To perform a quantitative study to support our view, we measure the smoothness of the targets produced by the teacher. The target produced for every training sample by the teacher for KD is a discrete probability distribution. To measure the smoothness of this target, we can use entropy which is a very popular method. Entropy of a discrete probability distribution with N classes can be indicated by H(p) = ∑N i −piln(pi) where pi indicates the probability assigned to the ith class. The maximum entropy/smoothness will be equal to Hmax(p) = ln(N) which corresponds to the uniform probability distribution over all classes. The key idea here is higher the entropy, smoother the target. We measure the average entropy for the training set (since this is the set used for distillation) to approximate the smoothness of the targets. Do note that the average entropy is measured using the targets produced by the teacher at different T .
Table 13 shows the average entropy/ smoothness of the targets for the ResNet-50 teachers used in our CUB200-2011 experiments. Higher entropy indicates that the targets are over-smoothed. Do note that the maximum average entropy for CUB200-2011 (Wah et al., 2011) is ln(200) ≈ 5.298.

heading: G.1 CASE STUDY AT LOWER T WITH SAME DEGREE OF SMOOTHNESS
text: Consider a lower T .
As shown in Table 13, the entropy / smoothness of targets produced by LS-trained teacher (α = 0.1) at T = 1 is approximately equal to the entropy/ smoothness of targets produced by normally-trained teacher (α = 0.0) at T = 1.481375. If smoothness of targets can determine the KD performance, then we expect comparable performances in both the instances above as they have the same degree of smoothness.
But using 2 counterexamples shown in Table 14, we show that even at the same degree of smoothness, distilling from LS-trained teachers produces better students compared to distilling from normally-trained teachers at lower T due to lower degree of systematic diffusion (LS and KD are compatible). Through these counterexamples we show that whether or not LS was used during training of teacher is very important in determining the performance of distillation even at the same degree of smoothness, thereby showing that the degree of smoothness is insufficient/ unreliable in determining the performance of distillation.

heading: G.2 CASE STUDY AT MODERATELY HIGHER T WITH SAME DEGREE OF SMOOTHNESS
text: Consider a moderately higher T .
As shown in Table 13, the entropy / smoothness of targets produced by LS-trained teacher (α = 0.1) at T = 3 is approximately equal to the entropy/ smoothness of targets produced by normally-trained teacher (α = 0.0) at T = 5.638. If the smoothness is the most important factor, then we expect comparable performances in both the instances above as they have the same degree of smoothness.
But using 2 counterexamples shown in Table 15, we show that even at the same degree of smoothness, distilling from LS-trained teachers produces poorer students compared to distilling from normally-trained teachers at moderately higher T due to increased degree of systematic diffusion (LS and KD are incompatible). Through these counterexamples we show that whether LS was used during training of teacher or not is very important in determining the performance of distillation even at the same degree of smoothness, thereby showing that the degree of smoothness is insufficient/ unreliable in determining the performance of distillation.

heading: G.3 CASE STUDY AT EXTREMELY HIGH T WITH SAME DEGREE OF SMOOTHNESS
text: Consider a very high T .
As shown in Table 13, the entropy / smoothness of targets produced by LS-trained teacher (α = 0.1) at T = 64 is approximately equal to the entropy/ smoothness of targets produced by normallytrained teacher (α = 0.0) at T = 64 since at very high T both these models produce a probability distribution that is very close to the uniform distribution. If the smoothness is the most important factor, then we expect comparable performances in both the instances above as they have the same degree of smoothness.
But using 3 counterexamples shown in Table 16, we show that even at the same degree of smoothness, distilling from LS-trained teachers produces poorer students compared to distilling from normally-trained teachers at extremely higher T due to extreme degree of systematic diffusion (LS and KD are incompatible). Through these counterexamples we show that whether LS was used during training of teacher or not is very important in determining the performance of distillation even at the same degree of smoothness, thereby showing that the degree of smoothness is insufficient/ unreliable in determining the performance of distillation.
Conclusion regarding smoothness: Through these 3 quantitative case studies comprising of 7 counterexamples, we show that whether or not LS was used during training of teacher is very important in determining the performance of distillation even at the same degree of smoothness, thereby showing that the degree of smoothness is insufficient/ unreliable in determining the performance of distillation.
Another way to intuitively think about this is that smoothness of targets can be characterized using the probability output of the teacher at different temperatures. But systematic diffusion is a phenomenon happening exclusively in the student. This is precisely the reason why we quantify the
degree of systematic diffusion using penultimate layer representations of the student, as these student representations are more indicative of the resulting student performance. That is, in all our 28 experiments, increased systematic diffusion definitely indicates lower performance of students whereas the degree of smoothness of targets does not give reliable insights as shown in the case studies G.1, G.2, G.3.

heading: H CLASS-WISE ACCURACY FOR TARGET CLASSES
text: This section contains class-wise accuracy for all the target classes used in the paper.
Given that we use the training set for distillation, let us consider both the training set and the validation set for this analysis. There are 1300 training and 50 validation samples for each class in ImageNet-1k. We use an exhaustive list of T values for this analysis, T = 1, T = 2, T = 3, and use the exact LS-trained teacher (ResNet-50, α = 0.1) reported in Table 2. There are 13 target classes used: 3 classes for the visualization in Figure 1, and 10 classes in Table 3. We show the complete class wise accuracies for both the training and validation set at T = 1, T = 2, T = 3. For each set we also compute the average accuracies to show the general trend to support our main findings. The results are shown in Tables 17, 18 and 19. As one can observe in Tables 17, 18, 19 , in the presence of an LS-trained teacher, KD at higher temperatures causes systematic diffusion thereby rendering KD ineffective. We can see this for most classes at increased temperatures shown below. That is, in the presence of a LS-trained teacher as we increase the temperature from T = 1, the accuracies for most of these classes drop due to systematic diffusion. This can be seen in both training and validation sets.
I ADDITIONAL EXPLORATION OF α AND T
Given that label smoothing was originally formulated as a regularization strategy to alleviate models’ overconfidence, most works spanning different learning problems use a smaller α = 0.1, including work closely related to our study. The intuition is that a larger α can introduce too much regularization that may subsequently hurt the model performance.
To show this, here we conduct additional experiments using larger α (α = 0.2) for compact student network distillation. We use CUB200-2011 dataset for these experiments.
The results are shown in Table 20. These additional results further support our findings on systematic diffusion.
In particular, we can make two important observations here: (i) larger α (α = 0.2) results in a weaker ResNet-50 teacher. We emphasize that it is reasonable to expect such behaviour, and this suggests why most works use α = 0.1 as in our main experiments. (ii) As one can clearly observe, with α = 0.2, KD at higher T causes systematic diffusion, thereby rendering KD substantially ineffective.
These experiments further support our main finding, and we emphasize that our findings can be generalized to larger values of α (α = 0.2).

heading: J ALTERNATIVE CHARACTERIZATION OF CLUSTER DISTANCE
text: Here we discuss an alternative characterization of cluster distance based on pairwise distances.
While our proposed η (Table 3) to use centroids to characterise distance between clusters should be very robust, here we discuss an alternative.
In this alternative, we propose to replace centroid-centroid distance with average pairwise distance between the projected penultimate layer representations. Note that this alternative is more computationally expensive.
We perform additional experiments using this alternative pairwise distance metric. We show that diffusion index based on this alternative distance, ηpairwise, for all the 10 target classes used in the paper with this pairwise distance below.
As one can clearly observe, using this alternative (pairwise distances) we obtain consistent findings for all 10 target classes as that in the paper Table 3: negative ηpairwise for S1, positive ηpairwise for S2.

heading: K ADDITIONAL REFERENCES
text: LS: LS: Many state-of-the-art models have leveraged on LS to improve the accuracy of deep neural networks across multiple tasks including image classification (He et al., 2019; Real et al., 2019; Zoph et al., 2018; Huang et al., 2019), machine translation (Vaswani et al., 2017) and speech recognition (Chorowski & Jaitly, 2017; Chiu et al., 2018; Pereyra et al., 2017).
KD: Recently KD methods have been widely used in visual recognition (Zhang et al., 2020; Peng et al., 2019; Lopez-Paz et al., 2016), NLP (Hu et al., 2018; Jiao et al., 2020; Nakashole & Flauger, 2017), speech recognition (Shen et al., 2020; Kwon et al., 2020; Perez et al., 2020) and selfsupervision (Fang et al., 2021).

Review summaries:
The paper provides a new perspective for the compatibility of label smoothing and knowledge distillation. A new definition named "systematic diffusion" is proposed to analyze the contradiction between the claims of [1] and [2]. By observing the visualization of the penultimate layer representations, the authors claim that KD is compatible with LS under the setting of lower temperature T. The paper conducts a large-scale evaluation of the impact of increasing temperature T in KD in the perspective of the systematic diffusion. The findings are interesting and attractive. The visualization looks good.  [1] Rafael Muller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in Neural Information Processing Systems, 2019 [2] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. International Conference on Learning Representations, 2021

This paper investigated whether a label smoothing-trained teacher is compatible with knowledge distillation. It started by reproducing the results of two previous studies, i.e., (Muller et al. 2019) claimed that LS erases information in the logits and therefore is harmful to KD, while (Shen et al. 2021b) argued that LS-trained teacher increases the distance between semantically similar classes and therefore is compatible with KD. After that, the authors tried to draw a unified conclusion from the above contradictory claims by introducing experiments with different temperatures: with an LS-trained teacher, a higher temperature introduces systematic diffusion in the representations of the semantically similar classes, which hurts KD. It is the different temperatures used in (Muller et al. 2019) and (Shen et al. 2021b) that resulted in different conclusions. Experiment results supported the main claims.

This paper investigates the contradiction of previous findings: (1) LS can hurt KD and (2) the benefit of using LS outweighs its disadvantage for KD. The authors provide ample experiments to verify their claim of "LS is useful for KD with a moderate (low) value of temperature". They interpret the negative or positive influence from LS with respect to systematic diffusion. The conclusion is using LS with a low T (e.g., 1) for KD. To support this argument, the authors analyze the diffusion phenomenon qualitatively (maybe t-SNE) and quantitatively (diffusion index).