Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
Authors: Zhiqiang Shen
Zechun Liu
Dejia Xu
Zitian Chen
Kwang-Ting Cheng
Marios Savvides
Venue: ICLR 2021 Poster (2021)
Keywords: label smoothing
knowledge distillation
image classification
neural machine translation
binary neural networks

Abstract: This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness.

Published as a conference paper at ICLR 2021
Is LABEL SMOOTHING TRULY INCOMPATIBLE WITH KNOWLEDGE DISTILLATION: AN EMPIRICAL STUDY
Zhiqiang Shen CMU
Zechun Liu CMU & HKUST
Dejia Xu
Zitian Chen UMass Amherst
Peking University
Kwang-Ting Cheng HKUST
Marios Savvides
CMU
ABSTRACT
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation (Müller et al., 2019). We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness.
INTRODUCTION
Label smoothing (Szegedy et al., 2016) and knowledge distillation (Hinton et al., 2015) are two commonly recognized techniques in training deep neural networks and have been applied in many state-of-the-art models, such as language translation (Vaswani et al., 2017; Tan et al., 2019; Zhou et al., 2020), image classification (Xie et al., 2019; He et al., 2019) and speech recognition (Chiu et al., 2018; Pereyra et al., 2017; Chorowski & Jaitly, 2017). Recently a large body of studies is focusing exploring the underlying relationships between these two methods, for instance, Müller et al. (Müller et al., 2019) discovered that label smoothing could improve calibration implicitly but will hurt the effectiveness of knowledge distillation. Yuan et al. (Yuan et al., 2019) considered knowledge distillation as a dynamical form of label smoothing as delivered a regularization effect in training. The recent study (Lukasik et al., 2020) further noticed label smoothing could help mitigate label noise, they showed that when distilling models from noisy data, the teacher with label smoothing is helpful. Despite the massive and intensive researches, how to use label smoothing as well as knowledge distillation in practice is still unclear, divergent, and under-explored. Moreover, it is hard to answer when and why label smoothing works well or not under a variety of discrepant circumstances.
View of incompatibility between label smoothing and knowledge distillation. Recently, Müller al. proposed new standpoint that teachers trained with label smoothing distill inferior student compared to teachers trained with hard labels, even label smoothing improves teacher's 's accuracy, as the authors found that label smoothing tends to "erase" information contained intra-class across individual examples, which indicates that the relative information between logits will be erased to some extent when the teacher is trained with label smoothing. This rising idea is becoming more and more dominant and has been quoted by large number of recent literatures (Arani et al., 2019; Tang et al., 2020; Mghabbar& & Ratnamogan, 2020; Shen et al., 2020; Khosla et al., 2020).
However, this seems reasonable observation basically has many inconsistencies in practice when adopting knowledge distillation with smoothing trained teachers. Thus, we would like to challenge whether this perspective is entirely correct? To make label smoothing and knowledge distillation less mysterious, in this paper, we first systematically introduce the mechanism and correlation
Project page: http://hhiiiaageeeeeooooom////// wtteeeen..cc//po////eeeee
htgp:////./////gggggggn..o...oor linar//soooerttttsssssnanenna// KI// ndex. html.

---

Published as a conference paper at ICLR 2021
D2
D1
D1
Training w/o
Training w/ LS
Validation w/o LS
Validation w/ LS
Figure I: Illustrations of the effects of label smoothing on penultimate layer output. The figure is plotted ImageNet with ResNet-50 following (Müller et al., 2019), we also choose two semantically similar classes (toy poodle and miniature poodle, in green and yellow) and one semantically different class (tench, in purple). b is the discovery observed by Müller et al. that label smoothing will enforce each example to be equidistant to its template, i.e., erasing the relative information between logits. D, and D2 are the degree of measuring "how much tench is similar poodle". @ is our new finding in this paper that "erasing" effect enabled by label smoothing actually promotes to enlarge relative information on those semantically similar classes, i.c., making them have less overlap on representations. De is the distance between the semantically similar "toy poodle" cluster and the "miniature poodle" cluster. More details can be referred to Sec. 3.
between these two techniques. We then present novel connection of label smoothing to the idea of "erasing" relative information. We expose the truth that factually the negative effects of erasing relative information only happen on the semantically different classes. Intuitively, those classes easy to classify as they have obvious discrepancies. Therefore, the negative effects during distillation are fairly moderate. On those semantically similar classes, interestingly, can enforce two clusters being away from each other and actually enlarge the central distance of clusters between classes, which means makes the two categories easier for classifying, as shown in Fig. 1. These classes in traditional training procedure are difficult to distinguish, so generally, the benefits of using label smoothing on teachers outweigh the disadvantages when training knowledge distillation. Our observation in this paper supplements and consummates prior Müller et al.'s discovery essentially, demonstrates that label smoothing is compatible with knowledge distillation through explaining the erasing logits information on similar classes. We further shed light on understanding the behavior and effects when label smoothing and knowledge distillation are applied simultaneously, making their connection more interpretable, practical and clearer for usage.
observe that erasing phenomenon
How to prove that their discovery is not judgmatic? We clarify such widely accepted idea through the following exploratory experiments, and exhaustively evaluate our proposed hypothesis: (i) Stan- dard ImageNet- 1K (Deng et al., 2009), fine-grained CUB200-201 (Wah et al., 20)1b) and noisy iMaterialist product recognition: (ii) Binary neural networks (BNNs) (Rastegari et al., 201 6): (iii) Neural machine translation (NMT) ( (Vaswani et al., 2017). Intriguingly. we observe that if the teacher is trained with label smoothing, the absolute values of converged distilling loss training set much larger than that the teacher is trained with hard labels, whereas, as we will discuss in detail later in Fig. 5 and 6, the accuracy on validation is still better than that without label smoothing. We ex- plain this seemingly contradictory phenomenon through visualizing the teachers' output probabilities with and without label smoothing, it suggests that the suppression of label smoothing for knowledge distillation only happens on training phase as the distributions from teachers with label smoothing is more flattening, the generalization ability of networks on validation set is still learned during opti- mization. That is to say, the dynamical soft labels generated by teacher networks can prevent learning process from overfitting to the training data, meanwhile, improving the generalization on the unseen test data. Therefore, we consider this erasing relative information function within class from label smoothing as merit to distinguish semantically similar classes for knowledge distillation, rather than drawback. Moreover, we also propose a stability metric to evaluate the degree of erased information by label smoothing, we found the proposed metric is highly aligned with model's s accuracy and can be regarded as supplement Finally, we discuss several intriguing properties of label smoothing we observed on the long- category distribution and rapidly-increase
alternateness to identify good teachers for knowledge distillation.
#class scenarios, as provided in Sec. 7. More specifically, this paper aims to address the following questions:
Does label smoothing in teacher networks suppress the effectiveness of knowledge distillation? Our answer is No. Label smoothing will not impair the predictive performance of students. Instead, observe that a smoothing trained teacher can protect the student from overfitting on the training set, which means that with smoothing trained teachers in knowledge distillation, the training loss is always higher than that without smoothing, but the validation accuracy is still similar or even better.
2

---

Published as a conference paper at ICLR 2021
cat
Yis (1 yee 2-
Smoothing
Engut
Teacher
Dux le pl ((r))og
Koss
Pln g00
Input
Hip, 21- 90 (lg((e))
CE
Stadent
Knowledge Distillation
Net
Label Smoothing
Figure 2: Knowledge distillation (KD) and label smoothing (LS) overview. Both the KD and LS adopt softened distributions for learning the target networks. The KD differs from LS the generation of these distributions and the objectives for optimization. KD chooses to utilize pre-trained teacher to produce the supervision dynamically, while LS uses a constant uniform distribution for training. In the figure, the black lines are the forward pass and the red lines are the gradient propagation direction.
What will actually determine the performance of a student in knowledge distillation. From our empirical study, we observe if the student architecture is settled, the dominating factor in knowledge distillation is the quality of supervision, i.e., the performance of a teacher network. higher-accuran teacher is particularly successful in distilling a better student, regardless it is trained with or without label smoothing. This observation partially against the conclusion in (Müller et al., 2019) which stated "a teacher with better accuracy is not necessary to distill a better student".
When will the label smoothing indeed lose its effectiveness for learning deep neural networks? Long-tailed class distribution and increased number of classes are two scenarios we observed wherein label smoothing will lose or impair its effectiveness. We empirically verify the findings on iNaturalist 2019 (Van Horn et al., 2018), PTace-LT (Liu et al., 2019) and curated ImageNet (Liu et al., 2019).
2 BACKGROUND
In this section, we first introduce the background of label smoothing and knowledge distillation through mathematical description. Given a dataset D = (X, Y) over set of classes K, X is the input data and Y is the corresponding one-hot label with each sample's label y E {0, 1]. where the element Ye is 1 for the ground-truth class and 0 for others. Label smoothing replaces one-hot hard label vector with mixture of weighted y and a uniform distribution:
ye = {
1 1 a/(K
if = label, otherwise.
where a is small constant coefficient for flattening the one-hot labels. Usually, label smoothing is adopted when the loss function is cross-entuue wross-entropy. and the network uses softmax function to the last layer's logits z to compute the output probabilities p, the gradient of each training sample with K
K
respect to will be: VH(pp y) p the cross-entropy loss and z is e-th logit in z.
(Softmax(zc) Vc). where H(p, y)
Σ logg.
Effects of label smoothing on loss
Loss
0=0.0
To further understand the effects of label smooth- ing on loss function, Fig. illustrates correction effects of smoothing on the binary cross-entropy loss (K 2). We can observe that the standard lo- gistic loss (a = 0) vanishes for large and confident positive predictions, and becomes linear for large negative predictions. Label smoothing will penalize confident predictions and involve a finite positive minimum it aims to minimize the average per- class. Generally, larger a values will produce larger loss values rebounding positive predictions. This also the underlying reason that smoothed loss can flatten the predictions of network.
a=0.1
0=0.2
0=0.3
0=0.4
Y Ypred
Figure 3: Correction effects of label smoothing on logistic loss with different a. Black dotted line presents the standard logistic loss and other colored lines are imposed label smoothing.
3

---

Published as a conference paper at ICLR 2021
In knowledge distillation, we usually pre-train the teacher model Tw on the dataset in advance. The student model Sw is trained over the same set of data, but utilizes labels generated by Tw More specifically, we can regard this process as learning Sw on a new labeled dataset D = (X, (w (X)). Once the teacher network is trained, its parameters will be frozen in the whole distillation.
The student network Sw is trained by minimizing the similarity between its output and two parts: the hard one-hot labels and the soft labels generated by the teacher network. Letting P.. (X) Tw (X) [c), p (X) Sw (x) [c] be the probabilities assigned to class e in the teacher model Tw and student model Sw- The distillation loss can be formulated as XH(pSw, yy (1 A)H(pS /T, pp=" /T) where T is the temperature scaling factor and À is the trade-off coefficient to balance the two terms.
3 THE "ERASE INFORMATION EFFECT BY LABEL SMOOTHING
This section aims to explain the erasing information effect more thoroughly. We start by reproducing the visualization of penultimate layer's activations using the same procedure from (Müller et al., 2019). We adopt ResNet-50 trained with hard and smoothed labels on ImageNet. As shown in Fig. 1, we obtain similar distributions as (Müller et al., 2019). Since examples in training set are the ones used for distillation, we mainly analyze the visualization from the training data. The core finding in (Müller et al., 2019) that a teacher is trained with hard labels, representations of examples distributed in broad clusters, which means that different examples from the same class can have different similarities (D1 and D2) to other classes. For a teacher trained with label smoothing, they observed the opposite behavior. Label smoothing encourages examples lie in tight equally separated clusters, so each example of one class has very similar proximities (D1 is closer to D22 to examples of the other classes. Our ce-viuualization also supports this discovery. The authors derive the conclusion that a teacher with better accuracy is not necessarily to distill a better student. This seems reasonable as the broad clusters can enable different examples from the same class to provide different similarities to other classes, which contains more information for knowledge distillation.
However, if refocusing on the two semantically similar classes, when label smoothing applied, the clusters are much tighter because label smoothing encourages each example is to be equidistant from all other class's 's templates, while, the tight cluster substantially promotes different class representations to be separate, i.e., the distance of clusters D. increases, which further indicates that different class examples obtain more distinguishable features. This phenomenon is crucial as these difficult classes are the key for boosting classification performance. Generally, it is not necessary to measure "how much a poodle is a particularly similar to a tench" since we have enough evidence to classify them, but it is critical to have information "how different is a toy poodle to miniature poodle".
Visualizations of teacher predictions. We further visualization the mean distribution of different classes crossing examples, shown in Fig. 4. We average all the probabilities after softmax layer the examples belong to the same category, and show the first 100 classes in ImageNet. Usually, the probabilities have major value (the bars in Fig. 4 (1)) that represents model's prediction on category and other small values (i.e., minor predictions in Fig. 4 (2)) indicate that the input image somewhat similar those other categories, some discussions about minor predictions are given in Appendix F. Our purpose of this visualization is to make certain of what label smoothing really calibrates in a network and shed light on how it affects the network predictions. We can observe in this figure that a model trained with label smoothing will generate more softened distributions, but the relations across different classes are still preserved. We conjecture the softened supervision is also the reason why teachers with label smoothing produce larger training loss during knowledge distillation. Consequently, label smoothing will both decrease the variance (verified by following stability metric) and mean predictive values within class, but will not impair the relations crossing different classes.
3.1 A SIMPLE METRIC FOR MEASURING THE DEGREE OF ERASED INFORMATION
Different from the visualization scheme (Müller et al., 2019) of finding an orthonormal basis of the plane that only studies this problem qualitatively, we further address the "crasing" phenomenon through a statistical metric that is simple yet effective, and can measure the degree of erasing operation quantitatively. Our motivation behind it is drraightfoowwadd If label smoothing erases relative information within a class, the variance of intra-class probabilities will decrease accordingly, thus can use such variance to monitor the erasing degree, since this metric evaluates the fluctuation
4

---

Published as a conference paper at ICLR 2021
Tench
Night snake
Goose
Tench
Night snake
Goose
training set
valset
1.0
0.8
Probabiliyy
0.6
0.4
0.2
0.2
0.0
100 0.0
20
40
80
20
40
60
80
100
(1) Major
training
val
1.0
1.0
0.9
Minor Probability
0.8
0.6
0.4
0.00
0.0 ImageNet Class ID
ImageNet Class ID
(2) Minor
without label smoothing
with label smoothing
Figure 4: Probability distributions with/without label smoothing ResNet-50. We show the first 100 categories in ImageNet. The red/green bars are distributions with/without label smoothing. respectively. "Minor probability" denotes the small probabilities predicted by networks when the outputs are used as supervisions in knowledge distillation.
of the representations, we can also call it the stability metric. The definition as follows:
K
Sstubility
P (f.c)
Th Pir
nc
where i is the index of images and n. is the #image in class ((,c) is the mean of Tw in class c. This metric utilizes the probabilities of intra-class variance to measure the stability of a teacher's prediction. The results on various network architectures are shown in Sec. 5 and Py PyTorch-lite code for caleulating this metric given in Appendix C.
Such metric has at least two advantages: 1) It can measure the degree of erased information quantita- tively and further help discover more interesting phenomena, e.g., observe that data augmentation method like CutMix (Yun et al., 2019) together with longer training erases the relative information on logits dramatically and can further be reinforced by label smoothing. 2) We found that the proposed metric is highly aligned with model accuracy, thus such metric can be used as a complement for accuracy to evaluate the quality of teacher's supervision for knowledge distillation.
4 A CLOSE LOOK AT LABEL SMOOTHING AND KNOWLEDGE DISTILLATION
few recent studies (Shen & Savvides, 2020; Shen et al., 2019) suggested supervised part H(pS~ yy) (i.e. hard labels) is not necessary as soft prediction is adequate to provide crucial information for students, meanwhile, removing supervised part can avoid involving incorrect labels caused by random crop, multí-object circumstance or false annotations by humans. Thus, here only consider the soft part H(pS5 TM /T) with the commonly used Kollbback-eeible divergence similarity. KL-divergence measures the similarity of two probability distributions. We train the student network S by minimizing the KL-divergence between its output p.S (X) and the soft labels p (X) generated by the teacher network. Following (Müller al., 2019; Hinton al., 2015) set T =1 as the
5

---

Published as a conference paper at ICLR 2021
temperature constant and it is omitted for simplicity, thus our loss function will be:
log Sw (X)
D (er.(Twlllw EnnTe.
Tw (X)
(3)
= =E log Sw (X)] - H(T (X))
Here, 1T log S is the cross-entropy between Sw and Tw (denoted ((ppS.p7T)) The second term H(T (X)) = EEw [- log TT (x)) is the entropy of teacher Tw and is constant with respect to Tw. We can remove it and simply minimize the loss as follows:
K
H(p pTw)= = - - vew (X) logp (X).
(4)
We can observe that Eq. 4 is actually a standard cross-entropy loss. Then, we have:
Property 1. not consider hard labels in knowledge distillation, distillation loss and cross-entropy loss with label smoothing have the same optimizing objective, i.e., (KI((T](Sw)) = H(p "ip"",p"" .p'"p"" ."")..
This property shows that label smoothing and knowledge distillation have the same optimization objective. the sole difference between them is the mechanism producing the soft labels. Therefore, except for the neural machine translation, in this paper all of our knowledge distillation experiments are conducted without the hard labels, which means our student solely relies on the softened distri- bution from teacher without the one-hot ground-truth. This may challenge the common practice in knowledge distillation (Hinton et al., 2015; Romero et al., 2015) that adopted both hard and soft labels with the cross-entropy loss for distillation, while our surprisingly good results and previous studies (Shen & Savvides, 2020; Shen et al., 2019; Bagherinezhad et al., 2018) indicate that knowl- edge distillation enabled by soft labels solely not only an auxiliary regularization (Yuan al., 2019) but can be the dominating supervisions, which further inspires us to carefully revisit the role of knowledge distillation and design better supervisionnoohective in training deep neural networks.
EMPIRICAL STUDIES
Metric Evaluation. Our results of stability metric are shown in Table 1, the second and third columns are results without label smoothing and the last two are with it. We study the metric crossing variety of different network architectures. The gaps of Sstabiiity using the same architecture measure the degree of erasing relative information. We can observe that the variances (1-Sstabltt)) with label smoothing always have lower values than models trained without label smoothing, this proves that label smoothing will erase information and enforce intra-class representations of samples being similar. Generally, the accuracy and stability have positive correlation between them. But the stability can even overcome some outliers, for example, Wide ResNet50 with label smoothing has lower accuracy, but the stability is still consistent with the tendency of predictive quality. Moreover, models trained with more epochs and augmentation techniques like CutMix (Yun et al., 2019) can dramatically increase the stability, this means relative information will be erased significantly by more augmentation together with longer training. We emphasize that this discovery cannot be observed by the qualitative visualization method (Müller et al., 2019). A PyTorch-like code is in Appendix C. Table 1: Accuracy and stability results with and without label smoothing on ImageNet- 1K. Here we show (1-Sstabilit), which denotes the aggregated intra-class variance (the lower the better). Red numbers are the quantitative values of the erased information by label smoothing.
Netowrks
Acc. (%) w/o LS
(1-Ssuhalt) w/o LS
Acc. (%r w/ LS
(1-Suubaay) w/ LS
18 (He et 2016)
69.758/89.077
0.3359
69.774/89.122
0,3358 (-0.0001)
ResNet-50 (He al., 2016)
75. 88889266662
0.3217
76.139992.936
0,3106 (-0.0111)
ResNet- 101 (He al., 2016)
77.37493.556
0.3185
77.726/938800
0.3070 (-0.0115)
MobileNet v2 (Sandler et 2018)
71.878/90.286
0.3341
DenseNet121 (Huang et 2017)
144434/1.972
0.3243
ResNeXX50 32 (Xie et al, 2017)
77.618/336698
0.3229
77.77493.642
0.3182 (-0.00147)
Wide ResNer50 (Zagomyko & Komodakis, 2016)
78.468/94.086
0.3201
77.808/93 682
0.3155 (-0.IM146)
ResNeX1101 32x8d (Xie et al., 2017)
79.312/94.526
0.3177
79.698/94.768
0.3116 (-0.0061)
ResNer50+LLng
76.526/3.070
0.3222
77.106993.340
0.3090 (-0.0132)
ResNetS0+Lon-=Cullis (Yun al., 2019)
76.874/93.500
0.2999
77.274/93.304
0.2890 (-0.0109)
Image Classification. We verify our perspective through investigating the effectiveness of knowledge
distillation with label smoothing the image classification tasks. We conduct experiments three

---

Published as a conference paper at ICLR 2021
Table 2: Image classification results on ImageNet-IK, CUB200-2011 and iMaterialist product recog- nition (in Appendix D). The teacher networks with label smoothing are denoted by "v"". We report average over runs for all the teacher network training and student distillation.
ImugoNet-1K (Slanford);
Tencher
w/TS
Aec: (ToptrTion))
Stndent
Acc. (Top1/Tops))
76.056 19/92.791 0.106
B8xNet118
71.425
0.086 /90.195 0.075
ResNet-50
ResNer-50
76.325 0.068/922.55 + 0.043
76.128 0.069/92.977 /92.977 030
18
71816
0. 0.0/99 466 0.0774
RosNet-50
77,052 0.030/93 376 0..15
CUR200-2011 (Fine-gninedl;: Teacher
(Top1/Top5)
Stadent
(Topl/Top5)
x
79.931 1 0.007996.700 0.064
ResiNe--13 18
71.116 a 0,088 55.221 = 0,108
ResNet-50
ResNet-50 50
80,910
0..03/91.733 0.11d
81.497 0.035/955..00 0.112
ResNec-18
78.382 0..09/998 /13 621 0.120
ResNet 50
82.355 050/95.0045400 0.075
datasets: ImageNet- IK (Deng et al., 2009), 2UB200-2011 (Wah et al., 2011a) and iMaterialist product recognition challenge data (in Appendix D). We adopt ResNet-(50/101) as teacher networks and DesNeet((18/00111 as students, respectively. More experimental setrings are in Appendix A.
The visualizations of our distillation training and testing curves are shown in Fig. 5. A more detailed comparison is listed in Tables 2 and 7. From the visualization we found two interesting phenomena: On training set, the loss of teacher networks that trained with label smoothing is much higher than that of without label smoothing. While on validation set the accuracy is comparable or even slightly better (The boosts on CUB is greater than those ImageNet- IK, shown in Table 2). We also provide the experiments of combining hard and soft labels in Appendix E, the results still show the effectiveness of better teachers with label smoothing, which distill better students. To make it clearer why this happens in distillation, we visualize the supervisions from teacher networks in Fig. 4 and the discussion is shown there. It indicates that label smoothing flattens teacher's predictions which causes the enlarged training loss, while the student's generalization ability is still preserved.
Results.
Acc: 77.237
Avc: 81 (26
Acs: 78.497
Acc: 82.263
25
200
10
103
15 Ln
T: ResNer50 S: ResNet18
ResNet101 S: ResNet50
tran liss wiTuucs L5
tostigg enu. wibhut metting ever
train los: without 15
beeting smrrr without testigg enon with
tain less with
train kosi will L3
Acc: 80,933
Acc: 81.572
Acc:
Ape:
of spoots
of spochs
of spoccp Fo
of specta
T: ResNet50 S: ResNetS0
T: ResNet101 S: ResNet101
Figure 5: The training and testing curves of knowledge distillation on CUB200-201 when teachers trained w/ and w/o lahel smoothing. The specific teacher and student architectures are given below each subfigure, therein, T indicates the teacher architecture and S indicates the student.
Binary Neural Networks (BNNs). We then examine the effectiveness of knowledge distillation on Binary Neural Networks. BNN aims to learn a network that both weights and activations are discrete values in {-1, +. In the forward pass, real-valued activations are binarized by the sign function:
-1 if A, < <, where A, is the real-valued activation of the previous Au = (Arr = +1 otherwise.
layers, produced by the binary or real-valued convolution operations. Ay, is the binarized activation. [|-lli if W, < O, [WW otherwise.
The real-valued weights are binarized by: W,- | Sign (W.)
where W, is the real-valued weights that are stored as latent parameters to accumulate the small gradients. W is the binarized weights. We update binary weights through multiplying the sign of real-valued latent weights and the channel-wise absolute mean ( W, ). Training BNNs is challenging as the gradient of optimization is approximated and the capacity of models is also limited.
We perform experiments on ImageNet- IK and results are shown in Fig. 6. Withal, the teacher network trained with one-hot labels (blue curve) is over-confident as the loss value is much smaller, which
7

---

Published as a conference paper at ICLR 2021
means that the teacher trained with label smoothing can prevent distillation process from being over- confident on the training data, and obtain slightly better generalization and accuracy (63.108% vs. 63.002%) the validation set. These results still support conclusion knowledge distillation.
55
5.0
train loss: without LS
testing emrrr without 1S with
LS
test loss: without LS
10
with
20
Acc. (5) without 63302/942244
3.0
Resling
Ace. (8) with IS: 84,366644
255
29
20
50 03
100
220
40
62
of opochs
# epochs
Figure 6: Left is the averaged train/test loss curves in distillation, right is the testing error w/ best 1/5 accuracy. We linear learning rate decay following other binary network training protocol (Martinez al., 2020; Liu et al., 2018). Our teacher networks are ResNet-50 with and without label smoothing which have similar performance. The student is the state-of-the-an ReActNet (Liu et al., 2020) with ResNet-18 backbone. We can observe that when the teacher is trained with label smoothing, the distillation loss is much higher, but the accuracy of student is still better.
Neural Machine Translation (NMT). Finally, we investigate our hypothesis of knowledge distil- lation on the English-to-Genn -er-ma translation task using the Transformer architecture (Vaswani et al., 2017). We utilize the distillation framework (Tan et al., 2019) on IWSLT dataset, and the pre- training/disttilationn curves are shown in Fig. 7. consistent setting is imposed on all the two comparison experiments, except the teacher is trained with and without label smoothing. We choose a = 0.1 for label smoothing as suggested by (Vaswani al., 2017; Szegedy et al., 2016; Müller et al., 2019), we use Adam (Kingma & Ba, 2014) as the optimizer, Ir with 0.0005, dropout with drop rate as 0.3, weight-decay with 0 and max tokens with 4096, all of these hyper-parametto are following the original settings of (Tan et al., 2019). Our results of Fig. 7 deliver two important conclusions: First sub-illustration (left one) proves the statement of Vaswani et al. (Vaswani et al., 2017) that label smoothing (a 0.1) boosts the BLEU score of language model despite causing worse perplexity if comparing to a model is trained with one-hot/hard labels. Second sub-illustration (right one) implies that on the machine translation task, a stronger teacher (trained with label smooth- ing) will still distill a higher BLEU student. That is to say, label smoothing may not suppress the effectiveness of knowledge distillation in the NMT task.
without LS
without
10K
20K
30K
40K
50K
JOK
20K
30K
40K
50c
step
step
Figure 7: Illustrations of BLEU score curves for teacher pre-training and student distillation. The left figure is teachers' pre-training with and with- out label smoothing. The right one is the distilla- tion process of students.
6
WHAT IS A BETTER TEACHER IN KNOWLEDGE DISTILLATION?
Teache (Tpp
86 KZ
Bascline
Studert Acc. (Top-2)
Distilled
Acc. 1Top-33
82-
Network Index
ResNet-50
ResNeXt-101 32V483
Figure 8: Left the accuracy relationship between teachers and students, wherein, all teachers are trained with label smoothing. Right is the accuracy of knowledge distillation by using strong teacher to fine-tune the student, FixRes (Touvron et al., 2019) is adopted in both teacher and student networks.
Better Supervision is Crucial for Distillation. We further explore the effects of teacher's accuracy on the student through fixing the student structure and switching different teachers. We perform two settings for this ablation study: using the same teacher structure with different training strategies and different teacher architectures. All teacher models re-trained with label smoothing. The results are shown Fig. 8 (Left) and Table 8, generally, teachers with higher accuracies can distill stronger students, but they are not linear related and are limited the capability of the student itself. To further
8

---

Published as a conference paper at ICLR 2021
support the argument that better teachers usually distill better students, we choose the state-of-the thetthe-rt FixRes model (Touvron et al., 2019) for both the teacher and student and perform our distillation training via Eq. 4. The results baseline and FixRes. Note that the compared FixRes is already the state-of-the-arr with ResNet and ResNeXt architecture, our result (under ResNet family) is a fairly competitive single-crop accuracy to date on ImageNet-1KK
shown in Fig. 8 (Right) and our method is slightly better than the
WHAT CIRCUMSTANCES INDEED WILL MAKE LS LESS EFFECTIVE?
Long-Tailed Tailed Distribution. We found the long-tailed datasets with imbalanced samples across classes will indeed suppress the effectiveness of label smoothing (LS) and hurt model performance. The results long-tailed ImageNet-LT, Places365-LT and iNaturalist 2019 are shown in Table 3. Our results on these three datasets empirically verify this observation and support this conclusion. Since LS has weight shrinkage and regularisation effects (Lukasik al., 2020), we have following conjecture:
Conjecture 1. effective on the long-taiil recognition circumstance and will further impair the performance.
Weight shrinkage effect (Regularisation) enabled by label smoothing is no longer
The weight shrinkage effect (label smoothing regularization) has been proven in (Lukasik et al., 2020) (Theorem 1) on the linear models. As label smoothing will assign probabilities equally all minor classes (i.e.. (1 a) pos + I-I J neg)), this operation may be biased to many-shot classes in the long-tailed scenario. We derive the conclusion empirically that label smoothing is inapplicable when the class distribution is long-tailed.
More #Class. This is another circumstance we found will impair the effectiveness of label smoothing. The results are shown in Table 4 and Fig. we average the gains of label smoothing across two different network architectures and compare the boosts between curated ImageNet-100/571 and 1K. Generally, more classes will reduce the improvement produced by label smoothing.
Table 3: Teacher results on the long-tailed ImageNet-LT, Places365-LD LT and iNaturalist 2019 val set.
ImageNet-LT Acc. (TopI/Top5)
Place-LT LT
iNaturalist 2019
Teacher
w/ LS
Acc. (Top 1/Tp3/
Acc. (Top1/Top3)
X
39.975/64.645
26.479/47.238
67.195/83,465
ResNett-1
V
39. 15111166665
25.877/46. 260)
66.701083432
X
41.150/66.20 205
27.329/48.753
70.165886.304
ResNet-34
V
40. 965/666566600 850
26.863/48.1 110
69.406/86.073
X
40.985/66.034
27.384/48.740
73.729/888845
ResNet-50
V
39. 99.965/65.11 1195
27.562/47, 945
72.904/87.05 .54
28.096/50.166
74.389/88.416
ResNet-101
27.466/48. 781
73.597/88.779
Table 4: Teacher results the curated ImageNet dataset when
1.6
Top-1
increasing the number of classes.
Top-5
ImageNet-100
ImageNet-500
ImageNet-IK
Boost
0.8
Teacher LS
Acc. (Topl/Top5)
Acc. (Topl/Top5)
Acc. (Topl/Toph)
82.380/95.520
73.521/91.66 1.642
69.758/89 076
ResNet-18
V
82.740/95.440
74.123/92.004
69, 60.666/89.72
82.000/94.
81.712/95.088 0000
77.374/93.546
RosNet-101
83.400/95.300
82.020/95.300
77.836/93.662
Average (*)
*0.880/0.440
10.455/0.291
15.155/0.206
-0.8 100
500
#class
Figure 9: Acc. downtrend.
8 CONCLUSION
We empirically demonstrated that label smoothing could both decrease the variance (i.e., erase relative information between logits) and lower mean predictive values (i.c., make prediction less confident) within category, but it does not impair the relation distribution across different categories. Our results on image classification, binary neural networks, and neural machine translation indicate that label smoothing is compatible with knowledge distillation and this finding encourages more careful to understand and utilize the relationships of label smoothing and knowledge distillation in practice. We found through extensive experiments and analyses that the indeed circumstances label smoothing will lose its effectiveness are long-tailed distribution and increased number of classes. Our study also suggests that, to find a better teacher for knowledge distillation, accuracy of teacher network is one factor, the stability of supervision from teacher network is also alternative indicator.

---

Published as a conference paper at ICLR 2021
REFERENCES
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Improving generalization and robustness with noisy collaboration knowledge distillation. arXiv preprint arXiv: 19l0:090005 2019.
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint arXiv: 100::88000000 02641, 2018. 6
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, al. State-of-the-art speech recognition with Sequence-to-sequee Acoustics, Speech and Signal Processing (ICASSP), pp. 4778-4778. IEEE, 2018.
models. In 2018 IEEE International Conference on
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence sequence models. Proc. Interspeech 2017, pp. 523-527, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
2009. 2, 7, 13, 14
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint (tti:::170.0000000 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 6, 13, 17
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503005500.2531 2015. 1, 1,5,6
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017. 17
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11362. 2020. 1
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.698) 2014. 8
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp-
722-737, 2018. 8
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. arXiv preprint XXX::2003.03440 2020. 8
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019. 3, 14
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.028/9 2020. 1,
10

---

Published as a conference paper at ICLR 2021
Brais Martinez, Jing Yang. Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural net- works with seal-to-binary convolutions. In International Conference on Learning Representations, 2020. 8
Idriss Mghabbar and Pirashanth Ratnamogan. Building a multi-domain neural machine translation model using knowledge distillation. arXiv preprint arXiv:2004.07322, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019. 2, 3, 5, 6,
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06544 2017. 1
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542, 2016. 2
William J. Reed. The pareto, zipf and other power laws. Economics Letters, 2001. 14
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 6
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo-
bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018. 6, 17
Zhiqiang Shen and Marios Savvides. Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks. arXiv preprint arXtv:2009.08453 2020. 5,
Zhiqiang Shen, Zhankui He, and Xiangyang Xue. Meal: Multi-model ensemble via adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4886- 4893, 2019. 6,
Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing, Un- mix: Rethinking image mixtures for unsupervised visual representation learning. arXiv preprint arXiv:2003. 05438, 2020. 1
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 281 2818-2826, 2016. 1,8
Xu Tan, Yi Ren, Di He, Tao Qin, and Tie- Yan Liu. Multilingual neural machine translation with knowledge distillation. International Conference on Learning Representations, 2019. 1, 8
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Under- standing and improving knowledge distillation. arXiv preprint arXiv:2002.03532, 2020. 1
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems (NeurIPS). 2019. 8, 9
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018. 3, 13, 14
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017. 2,
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds 200-s-200-201 Dataset. Technical report, 201 la. 7, 13
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 201 lb. 2
11

---

Published as a conference paper at ICLR 2021
Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoe V Le. Self- Self-training with noisy student improves imagenet classification. arXiv preprint orXiv:1911.04252, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017. 6, 17
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: teacher-free framework. arXiv preprint arXiv: 1/23,, 2019. 6
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6023-6032, 2019. 6,
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07144 2016. 6, 17
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2017.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation non- autoregressive machine translation. In International Conference Learning Representations. 2020.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
In this appendix, we provide details omitted in the main text, including:
Section A: A introduction of datasets and experimental settings (Sec. 5 "Empirical Studies" of the main paper.)
Section B: A discussion about what happens if teachers with label smoothing are inferior? (Sec. 6 "What a Better Teacher Knowledge Distillation?" of the main paper.)
Section C: A PyTorch-like code for calculating stability metric (Sec. 3.1 "A Simple Metric" of the main paper.)
Section D: (1) Table 7 is detailed version of our Table 2 (Sec. "Empirical Studies" of the main paper.) (2) Table 8 is a detailed results for visualization of Fig. 8 (Sec. 6 "What a Better Teacher in Knowledge Distillation?" of the main paper.)
Section E: Results of knowledge distillation by adopting both hard and soft labels (Sec. 5 "Empirical Studies" of the main рaper.)
Section F: Discussions of minor probabilities from teacher network. (Sec. 3 "The Erase Information Effect by Label Smoothing" of the main paper.)
Section G: Supplementary metric of inter-class variation. (Sec. 5 "Metric Evaluation" of the main рaper.)
A DATASETS AND EXPERIMENTAL SETTINGS
A.1 FOR TRAINING TEACHER AND KNOWLEDGE DISTILLATION MODELS TO VERIFY THE COMPATIBILITY BETWEEN LABEL SMOOTHING AND KNOWLEDGE DISTILLATION
Standard ImageNet-IK Classification (Deng et al., 2009) ImageNet- IK contains ~ 1.28 million images in 1000 classes for training and 50K images for validation. For training teacher networks, we follow the standard training protocol (He et al., 2016; Goyal et al., 2017), i.e., total training epoch 90, initial learning rate 0.1 and decayed to 1/10 with every 30 epochs. For distillation, as the supervision is soft distribution and will dynamically change, we train with 200 epochs and the learning rate multiplied by 0.1 80 and 160 epochs. All models are trained from scratch.
Fine-grained Recognition on CUB200-2011 (Wah et al., 201 10) This dataset contains 200 bird species and 1 11,788 images, and is a wildly-used fine-grained classification benchmark. Follow- ing (Wah et al., 201 la), we use standard split with ~30 samples of each breed for both training and testing. As CUB200-222 2010000011111 has limited training data, we munetune our model from the ImageNet pre-trained parameters. Both teachers and students are trained with 200 epochs and the learning rate is multiplied by 0.1 at 80 and 160 epochs.
Noisy Product Recognition (iMaterialist Challenge)?. The training data consists of 1,01 1,532 images from 2,019 categories. The validation data has 10,095 images (around 5 for each category). The training data is collected from the Internet and contains some noise (~300% of incorrect labels), the validation data has been cleaned by human annotators. The training protocol is following CUB200- 2.11. While, the training images in this dataset is sufficiently enough, all models trained from scratch.
A.2 FOR TRAINING TEACHERS ONLY TO EXPLORE THE EFFECTIVENESS OF LABEL SMOOTHING
Long-tailed iNaturalist 2019 (Van Horn et al., 2018). The iNaturalist Challenge 2019 dataset contains 1,010 species, with a combined training and validation set 268,243 images, wherein, the validation set has a uniform distribution of three images in each category. While, the training set constructed with long-tailed distribution of image numbers. We conduct experiments using the
httpps: //si tes. google boogle.co com/ vi ew http:///ccopttteo t cioos//ii htaimmmatpppp matt-product-20199 authuser=0.
13

---

Published as a conference paper at ICLR 2021
same training protocol as the CUB200-2011 dataset. evaluate our results on validation set using Top-1/Top-3 metrics. Also, this protocol is applied to the Place-LT dataset.
Place-LT (Liu et al., 2019). The training set of this dataset is constructed by Pareto distribu- tion (Reed, 2001) with a power value a 6. It contains a total number of 184.5K images from 365 categories. There are 4,980~5 images per class from the 365 classes of Places-365 dataset (Zhou et al., 2017). 20 images per class are randomly selected from the original Places- training set as a validation set. The training protocol the same as iNaturalist 2019, we also evaluate our results on validation set using Top-1/Top-3 metrics.
ImageNet-LTT (Liu et al., 2019). The long-tailed ImageNet is constructed by sampling subset from the standard ImageNet (Deng et al., 2009) following the Pareto distribution with the power value 6. The whole dataset contains 115.8K images from 1000 categories, with maximal 1280 images and minimal 5 images per class. 20 training images per class are randomly selected from the origin training set of ImageNet as a validation set. All our models are trained from scratch.
Curated ImageNNet1000/500/K We construct this dataset by randomly selecting a subset of 100/500/1 categories the standard ImageNet (Deng et al., 2009). We use this dataset to ex- plore the impact of #class for label smoothing. We observe that label smoothing is more effective on the dataset with fewer numbers of classes.
Table 5: Overview of six datasets used in our experiments. a indicates the Pareto distribution value.
Dataset
#class
Property
Long-tailed tailed Head class size Tail class size #Training set #Testing
ImageNet-1K K
1000
Standard
N/A
N/A
N/A
1,281,167
50,000
CUB200-2011
200
Fine-grained
N/A
N/A
N/A
5,994
5,794
iMaterialist
2019
Noisy (~30%)
N/A
N/A
N/A
~1
~ -0K
iNaturalist 2019
1010
Long-tailed
N/A
250
211
265,213
3,030
ImageNet-LT
1000
Long-tailed
6
389
611
115,846
20,000
Place-LT
365
Long-tailed
6
~00
100
184,500
7,300
B WHAT HAPPENS IF TEACHERS WITH LABEL SMOOTHING ARE INFERIOR
In the main paper, we discussed that the superior teachers (even trained with label smoothing) could distill better students. Also, as noted that long-tailed distribution will suppress the effectiveness of label smoothing. In this section, we would like to explore what happens we use these inferior teachers for knowledge distillation? We conduct experiments on the long-tailed iNaturalist 2019 data and show more evidence to prove our conclusion that the quality of supervision is more crucial for knowledge distillation than the ways of training teachers (with or without label smoothing). The results are shown in Table 6, it can be observed that if teachers with lower accuracies, the distilled students will also have poorer performance, regardless of the teacher is trained with label smoothing or not.
Table 6: Distillation results using inferior teachers on the long-tailed iNaturalist 2019 (Van Horn et al., 2018).
NNatrallist 2019 (Long-tailed)
Teacher
w/ LS Acc. (TopI/Top3)
Student
Acc. (Topl/Top3)
ResNet-18
67.756/84.521
x
73.729/88.845
ResNet-50
74.125/88.944
5e
ResNet-18
67.228/84.587
72.904/87.954
ResNet-50
72.838/88. 152
ResNet-50
73.894/89.142
x
74.389/88.416
ResNet-101
74.488/89.100
Res
ResNet-50
72.409787.66 .657
73.597/88.777
Res
73.234/88.647
C A PYTORCH-LIKE CODE FOR CALCULATING STABILITY METRIC
14

---

Published as a conference paper at ICLR 2021
Algorithm PyTorch-like Code for Calculating Stability Metric.
4 x, target: imput images and the cozresponding labels
single_std: std value for each class
batotaize twe use Neb0 on ImageNet for ease of implemenzation as it has 50 vall images in each class)
padel_path: model path thet you
to esleulate stability
lcad model checkpoint madel 1oadll.oodd_deeee oad_ssatee sirteecctttt (chackpot (M [ # Initialize
load (model
for 1x, target) in loader: 1 load minibatch x with N samples (we choose N-50 for Imagellet) make sure shuffle Palse in dataloades
compute output output = model (X]
compute softmax probabilities
softmax__ nn. funntiioaal..tt..mm poutput, dim=1)
predict softmax_pli, target[0]] obtain target probability
computo stability
aingle_atd predict.staddinn
all_std single_std # aggregate all snd values from each class
all_std/1000.0 return computed stabllity
D TWo DETAILED TABLES OF RESULTS
Table 7: Image classification results ImageNet- IK, CUB200-2011 and iMaterialist product recog- nition. The teacher networks with label smoothing are denoted by "V". We report average over runs for ResNet-50 as the teacher network on both teacher training and knowledge distillation.
ImageNar-IA IK (Standard):
Teacher
(Topl/TopS)
(Topl/Top5)
71.425
0.038/90.1K
x
76.056 0. 0,119/92.799 791 0.106
ResNet-50
ResNet-50
76.325
0.068/92
76.128 0. 0.069/99.977 7 0.030
ResNet-18
71.816
0.011/99..060
ResNet-50
77.052
0.003//93330
0.015
ResNet-50
77.428/9..112
77.374/93.566
ResNet-101
ReaNet-101
78.270994.52
ResNet.50
77.624/938622
77.836/33.62
ResNet-101
78.476/94.008
CUR200-2011 (Fine-grained);:
Teacher
w/ 1.8
Acc. (Topl/Top5)
Student
Acc. (Topl/Top3)
ResNet-18 18
77.116 1.086/93.241 0,108
79.931 4: 0.037/94.370 4: 0.064
ResNet-50
ResNot-50
80,910 003/99997777 0.114
ResNet-18
78.382
0.099 621 1 0.120
81.497 + 0 0.0355/95.00 0443 0.1
ResNet-50
82.355
0.050/UG, (40-94544440 0.075
ResNer-50
81.261/94.905
80.380/94499
ResNet-101
ResNet1111
81.572/95-711
ResNet-50
82,263/95.38
82.333//44.90
ResNet-101
82.522/95. 199
iMateriafist-20P9 P (Noisy):
Teacher
Acc. (Top1/Top3)
Student
Acc. (Top1/Topp)
ResNet-18
65.250090.244
x
66.241/91.013
ResNet-50
67.420/92.155
5c
ResNet-18 18
65.359/90.530
66.525/91.669
ResNet-50
67.528/92.551
ResNet-50
67.905/92.448
x
66.726/91.266
ResNet-101
68.281/92.580
ResNet- 101
ResNet-50
67.925/92.789
67.370/91877
ResNet- 101
68.618892.997
E RESULTS OF KNOWLEDGE DISTILLATION WITH HARD AND SOFT LABELS
Here we verify whether label smoothing is still effective when adopting hard labels in knowl- edge distillation. As mentioned above, the traditional distillation loss can be formulated as
15

---

Published as a conference paper at ICLR 2021
Table 8: ImageNet results on the same student structure with different teachers. "Long" indicates we
train with more budget (160 epochs), the default is 90. "R50 and R18" are ResNet-50018 respectively.
Teacher (same arch)
(Top-1)
Student Acc. (Top-1)
Teacher (different archs) Acc. (Top-1)
Student
Acc. (Top-1)
R50
76.056
R18
71.478
MobileNet V2
71.878
R18
70.054
R50+LS
76.212
R18
71.866
DenseNet-121
74.894
R18
70.922
R50+LS+Long
77.106
R18
72.024
Wide ResNet-50-2
77.808
RI8
72.252
R50+LS+Long+
77.274
R18
72.172
ResNeXI-101-32x8d
79.698
R18
72.412
Table Distillation results with different ratios of the combination with hard labels and soft labels.
The teacher network is ResNet-50 and the student is ResNet-18.
ImageNet
Ratio (hard label soft label)
w/ LS
Acc. (TopI/Top5)
X
71.592/90.386
0.3 0.5
71.752/90.412
X
71.484/90.211
0.5
71.748/90.454
X
71.164/90.19
0.3
71.314/90.200
XH(pS~, y) + (1- A)H(pS0/ /T. We use ResNet-50 as the teacher and ResNet- 18 as the student and conduct experiments with three different ratios of A: 0.3, 0.5, 0.7. Our Top- 1/5 results are given in Table 9. We can see the teacher networks with label smoothing still distills better students than the teacher without label smoothing. Also, with higher probability of hard labels, the performance declines.
F MINOR PROBABILITIES IN TEACHER NETWORK
The visualization of minor probabilities is shown in Fig. 4 (2), we have two interesting observations in it: 1) In the enlarged area of Fig. 4 (2), we can observe that several classes have values are close to zero, this phenomenon means in ImageNet dataset, there are several categories that are very unique to other classes and the model will barely predict other classes to them; 2) We can also observe more "red" areas gather at the bottom of the bars. It reflects that the probabilities from the model trained with label smoothing will be assigned to more classes with smaller values, this is also consistent with what the label smoothing operation actually does.
G SUPPLEMENTARY METRICS OF INTER-CLASS/ENTIR VARIATIONS
To better understand the behavior of erased information by label smoothing across different classes, we also introduce the inter-class stability metric as follows:
K
cinter Stability
1
Σ
(IPTT (c)
Plauy113
(5)
K
where K the number of classes. Tw is average of probability across the entire dataset. pTaj P(auy the mean of Tw in class Sinter be regarded supplement for intra-class stability metric to measure the inter-class stability. The average probability within each class pTa is defined as:
Σ
(t.c)
(6)
ne
where is the index of images and n. is the #image in class c. The average probability across entire dataset P Tw is defined as:
Piatty
1 N
p
The P(au)
Σ
P(
(7)
N
i=1
16

---

Published as a conference paper at ICLR 2021
Table 10: Accuracy and inter-class stability results with and without label smoothing on ImageNet- IK.
Here we show (1 (1-Stnter Stability which denotes the aggregated inter-class variance (the lower the better). Red numbers are the quantitative values of the erased information by label smoothing.
Netowrks
Acc. (96 w/o LS
(1-S Nuahiiy w/o LS
Acc. (%) w/ LS
(1-S Stailly w/ LS
ResNet-18 (He 2016)
69.758/89078
0.1858
69.774/89.122
0.1724 (-0.0134)
ResNet-50 2016)
75.888/92.66 642
0.1733
76.13099.9936
0.1610 (-0.0123)
ResNet-101 (He al., 2016)
77.374/3.556
77.726993830
0.1646 (-0.0025)
MobileNet v2 (Sandler et al., 2018)
71.878/90.266
0.1797
0.1726 (-0.00771)
DenseNet121 (Huang al., 2017)
74.434/91.97
0.1763
0.1666 (-0.00997)
ResNeXt50 32x4d (Xie et al., 2017)
77.618/93.00 698
0.1658
77.774/9.6622
0,1729 (+0.0071)
Wide ResNar50 (7agonykk & Komodakis, 2016)
78.468/940000
0.1602
77.808/93 682.
168888 (+0.0086)
ResNeXr101 32x8d (Xie al,, 2017)
79.312/94.526
0.159
79.698/94.768
0.1677 (+0.0081)
where N is the number of samples in the entire dataset. This metric utilizes the probabilities of inter-class variance to measure the stability of teacher's s prediction. The results on various network architectures shown in Table 10.
17


Reviews:
[{"review_id": "v-hAiDcUaY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is mainly based on the prior work by Muller et al., which suggests that label smoothing is incompatible with knowledge distillation. Firstly, this paper provides an explanation of this incompatibility---label smooth tends to erase relative information among different classes, and provide a way to qualitatively measure the degree of erased information. Then, this paper argues that label smoothing actually is compatible with knowledge distillation, and show several empirical results as evidence. Lastly, this paper suggests that the performance of the teacher model is a more directly related factor for determining the performance of the student model.\n\n\nPros:\n\n(1) This paper performs a set of careful diagnoses on showing the effects of information erasing (caused by using label smoothing) at the category-level, and observes an interesting phenomenon: erasing relative information only cause negative effects to semantically different classes, but will help the classification for semantically similar classes. Both qualitative and quantitative evidence is provided to confirm this observation.\n\n(2) A simple and novel metric is proposed to facilitate the measurement of the degree of erased information. \n\n(3) Extensive experiments on the image classification task and the neural machine translation task are provided to confirm that label smoothing is indeed compatible with the knowledge distillation framework.\n\n\n\nCons (please address them during the rebuttal):\n\n(1) The presentation of this paper needs to be improved. In all abstract, introduction and conclusion sections, this paper highlights that \"we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness\". Nonetheless, the reviewer CANNOT find any related discussions in the main paper. The only related discussion is provided in the appendix. The reviewer does not think it is a good way for presenting the paper, as the appendix is mainly used for explaining some not very important details. Putting the entire discussion of an important contribution of this paper in the appendix is inappropriate.\n\n(2) Though the reviewer agrees that removing the hard label part in knowledge distillation can facilitate the analysis of this paper, the authors should also provide a brief discussion on whether adding this hard label part back will still lead to the same conclusions. For example, in table 2, with the hard label part, if the teacher model with label smoothing can still help the student model.\n\n(3) One minor question is that, as shown in Table 1, ResNet-50+long substantially outperforms ResNet-50 in terms of accuracy, but their stability measurements are nearly the same. Can the authors provide any explanation of this \"counter-intuitive\" phenomenon?", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jpg8a2QZV3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper Summary**\nThe authors re-analyze and re-confirm the relationship between label-smoothing and knowledge distillation, which is firstly argued by Muller et al. (“When Does Label Smoothing Help?“, NerurIPS 2019.). This paper shows that the previous argument, \"label smoothing is not helpful for knowledge distillation\", does not always hold, and carefully re-visits the missing points of the previous analysis by Muller et al. Based on this analysis, label smoothing can be helpful for knowledge distillation and can be explained using the intra-class variation and between-class distance within similar classes. The authors have empirically verified the arguments of the paper with various experiments. \n\n**Pros**\n1. Introduced an interesting analysis for improved knowledge distillation via label smoothing. \n2. The paper is well written and the contributions are clearly explained by comparing against the previous work (Muller et al.)\n\n**Cons**\n1. The main analysis is based on the original knowledge distillation paper (Hinton et al., 2015), therefore, it seems to be difficult to apply to the recent knowledge distillation. For example, the proposed analysis cannot explain the recent works (Relational Knowledge Distillation (CVPR 2019), or Contrastive Representation Distillation (ICLR 2020) which are based on the \"relation\". \n2. The stability metric $S_{stability}$ is based on the intra-class variation, but the sample variation of the entire dataset is not considered. Since the intra-class variation is proportional to the variation of the entire samples, the increase of the stability metric might come from the decrease of the entire samples' variation. For example, LDA (linear discriminant analysis) utilizes both intra- and between-class variation. Thus the stability metric needs to be improved and the experiment's scores need to be measured again. \n3. Indeed, the knowledge distillation was originally proposed using cross-entropy [1][2], so section 4 (explaining that distillation loss is the same as cross-entropy loss) is not a new observation and it is not necessary for the paper's flow. \n[1] Distilling the Knowledge in a Neural Network (NeurIPS workshop 2015)\n[2] Fitnets : FITNETS: HINTS FOR THIN DEEP NETS (ICLR 2015)\n4. Figure 4 needs to be improved. It is hard to see and capture the meaning of \"other soft predictions\" in the figure. It would be nice to improve the figure 4 so that it clearly expresses what it is intended.\n5. In appendix D, does the \"loader\" utilize standard ImageNet data augmentation (e.g., random-resize-cropping)? \n\n**Comments after the author response**\n- The authors have answered all my questions and they revised the manuscript as well, so I will keep my initial rating (weak accept). \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CNQY2PvnNJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recent literature proposed that even label smoothing improves the teacher model, it will hurt the distillation training of student models due to the information erasing. Although this idea dominated more and more literature, this paper argued that this observation is not entirely correct. In order to clarify this idea, the paper systematically discussed the correlation between knowledge distillation and label smoothing. Comprehensive experiments well support the claims in this paper, i.e. label smoothing is compatible with knowledge distillation. The correlation between label smoothing and knowledge distillation remains an open question to date, and this paper made a breakthrough regarding this question. Besides the main purpose (clarify previous ideas), this paper also provided multiple interesting empirical conclusions, e.g. a better teacher always leads to a better student by producing more informative distillation labels, the distillation itself can provide enough regularization for training and the hard-label classification loss is no more needed. \n\nTo conclude, the paper overturns the previous perspective with convincing explanations, discussions, and experimental results. Several empirical discoveries are introduced, which are expected to have high impacts on the tasks of knowledge distillation. The major contributions of this paper can be concluded as:\n1) The paper empirically confirmed that label smoothing is well compatible with knowledge distillation, overturning previous dominant ideas. This is an important finding because it can prevent subsequent research from being misled.\n2) It further explained the phenomenon of relative informative erasing, which only happens on the semantically different classes. Thus previous lopsided ideas (label smoothing hurts knowledge distillation) can be well explained.\n3) The paper claimed that the dominating factor in knowledge distillation is the performance of the teacher and further proposed a stability metric to measure the quality of supervision. This metric is crucial in the tasks of knowledge distillation since it provides a simpler and faster way to measure distillation quality. The paper also claimed that the distillation loss itself can provide enough regularization, which also inspires me a lot.\n\nIt's quite a good empirical paper and I really enjoy reading it. I think there's no significant weakness on it, so I recommend a clear acceptance for this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7OOzCVlcrs7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper empirically discusses the relationship between Label Smoothing (LS) and Knowledge Distillation (KD). It designs a  stability metric to measure the degree of erasing information and finds that LS can be compatible with knowledge distillation except in long-tailed distribution and increased number of classes.\n\nStrengths:\n\n1. Adopting stability metric to measure the degree of erasing information quantitatively is straightforward and effective. \n2. Extensive experiments from image classification to NMT are conducted to reveal the relationship between LS and KD and validate the idea of the work.\n\nWeaknesses:\n1. Even this work discusses the detailed relationship between LS and KD, but for me, the finding of this work is not enough to reach the bar of the top-tier conferences. \n2. The paper has many claims (Bold or Italic), some of that are well-known by the KD community, but too many claims make readers lose what is the paper's key insight. \n3. I am still confused that how do the finds in the paper can guide the KD community?  I think \"better supervision is crucial for distillation\" or \"better teachers usually distill better students\" should be a well-known and practical skill for KD, and I think many researchers in the KD community don't agree with the conclusion that \"a teacher with better accuracy is not necessary to distill a better student\" in [1]. \n\n[1]. When does label smoothing help? In Advances in Neural Information Processing Systems, Muller, et al.\n\nAfter reading the response from the authors, I would like to increase my rating to 6: Marginally above acceptance threshold.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
This paper studies the effect of label smoothing on knowledge-distillation. A previous work on this topic (Muller et al.) has claimed that label smoothing can hurt the performance of the student model in knowledge-distillation. The rationale behind this argument is that label smoothing erases information encoded in the labels. This work shows that such claimed effect does not necessarily happen. Specifically, by a comprehensive study on image classification, binary neural networks, and neural machine translation, the authors show that label smoothing can be compatible with knowledge distillation. However, they conclude that label smoothing will lose its effectiveness with long-tailed distribution and increased number of classes.

Overall ratings of this paper are all on the positive side, and R2 finding this paper an important step toward understanding the interaction between knowledge-distillation and label smoothing. I concur with the reviewers about the importance of this research direction and I think this submission provides a reasonable empirical evidence to change our earlier perspectives. I recommend accept.

While the paper specifically studies the effect of label smoothing on knowledge-distillation, I think providing a bigger context and reviewing some of the recent demystifying efforts on understanding knowledge-distillation could allow paper to communicate with a broader audience. I hope this can be accommodated in the final version.

Decision: Accept (Poster)