Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy?
Authors: Md Arafat Sultan
Venue: EMNLP 2023 Main (2023)
Keywords: knowledge distillation
label smoothing
regularization
interpretation of knowledge distillation
TL;DR: Knowledge distillation has been claimed by some to be a form of label smoothing regularization; we present evidence against that claim in this paper.

Abstract: Originally proposed as a method for knowledge transfer from one model to another, some recent studies have suggested that knowledge distillation (KD) is in fact a form of regularization. Perhaps the strongest argument of all for this new perspective comes from its apparent similarities with label smoothing (LS). Here we re-examine this stated equivalence between the two methods by comparing the predictive confidences of the models they train. Experiments on four text classification tasks involving models of different sizes show that: (a) In most settings, KD and LS drive model confidence in completely opposite directions, and (b) In KD, the student inherits not only its knowledge but also its confidence from the teacher, reinforcing the classical knowledge transfer view.

Knowledge Distillation ~ Label Smoothing: Fact or Fallacy?
Md Arafat Sultan IBM Research Al arafat.sultan @ ibm.com
Abstract
designed "for encouraging the model to be less confident" (Szegedy et al., 2016) by imposing a uniform prior on the target distribution; the result- ing new distribution is intended to simply prevent overfitting without revealing any new details about the task. KD, on the other hand, was designed to explicitly teach the student better inter-class rela- tionships for classification, example, may only have a very small chance of be- ing mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot" " (Hinton et al., 2014).
Originally proposed as a method for knowl- edge transfer from one model another, some recent studies have suggested that knowledge distillation (KD) fact form regulariza- tion. Perhaps the strongest argument of all for this new perspective comes from its apparent similarities with label smoothing (LS). Here re-examine this stated equivalence between the two methods by comparing the predictive confidences of the models they train. Experi- ments on four text classification tasks involving models of different sizes show that: (a) most settings, KD and LS drive model confidence in completely opposite directions, and (b) In KD the student inherits not only its knowledge also its confidence from the teacher, reinforcing the classical knowledge transfer view.
that "a BMW, for
Such different motivations of the two methods prompt us to re-examine the claim of their equiva- lence in this paper, where we find the underlying argument to be essentially flawed. With growing dependence in key areas like NLP on increasingly large models, KD and other model compression methods are becoming more relevant than ever be- fore (Wang et al., 2023; Taori et al., 2023; Dettmers et al., 2023). It is therefore crucial that also in- terpret such methods correctly help steer their future research the right direction, which is a primary goal of this work.
1 Introduction
Knowledge distillation (KD) was originally pro- posed as a mechanism for small and lightweight student model to learn perform a task, e.g., clas- sification, from a higher-capacity teacher model (Hinton et al., 2014). In recent years, however, this view of KD as a knowledge transfer process has come into question, with number of studies finding that it has features characteristic of regu- larizer (Yuan et al., 2020; Zhang and R. Sabuncu, 2020; Dong et al., 2019). Perhaps the most direct argument for this new position comes from Yuan al. (2020), who show that KD has interesting sim- ilarities with label smoothing (L.S) (Szegedy et al., 2016). One of their main conclusions, that "KD is a type of learned LS regularization." has sub- sequently been reiterated by many (Menon et al., 2021; Chen et al., 2021a; Shen et al., 2021; Chen et al., 2b) and remains important part of our understanding of KD this day.
KD and LS do indeed share the common prop- erty that they both substitute one-hot training labels with soft target distributions. The key question for us, however, is whether the soft teacher estimates in KD simply regularize training like the smoothed la- bels in LS do, or if they present detailed and finer- grained view of the task, as originally intended. Yuan et al. (2020) argue in favor of the former based observations such as (a) similarities in the cost functions of KD and LS, and (b) strong gener- alization of distilled image classifiers independent of the level of expertise of their teachers.
Here we first analyze their arguments and point out why they may be inadequate for a "KD Ls" assertion ($2, 3). Then we present empirical evi- dence on four text classification tasks from GLUE (Wang et al., 2019) that KD lacks the defining prop-
This is a rather curious suggestion, however, given the starkly different purposes for which the two methods were originally developed: LS was

---

erty of LS in most settings. Concretely, relative to standard empirical risk minimization (ERM) over one-hot expert labels, LS by design increases the uncertainty (entropy) in the trained model's poste- rior over the associated categories. In other words, it always reduces the model's confidence in its pre- dictions, as expected. The effect of KD on the student's 's posterior, the other hand, is a function of the relative capacities of the teacher and the stu- dent: In the more common setting where a larger teacher is distilled into a smaller student and also in dintliisillation, KD almost always reduces entropy over ERM, while also improving accuracy. It is only when the student is larger than its teacher that we see small increase in entropy-still much less than in LS-along with smaller gain in accuracy. These results clearly show that unlike LS, KD does not operate by merely making the trained model less confident in its predictions.
the same cross-entopy loss LERM that is typically minimized in standard empirical risk minimization (ERM) training of classifiers.
KD, on the other hand, is defined directly as minimizing the following cost function:
LKD (1- + aDRIld p) (2)
where p² (k((z)) the posterior computed by pre- trained teacher t given x. The similarity in the forms of Eqs. and is one of the main reasons why Yuan et al. (2020) interpret KD "as a special case of LS regularization."
A second argument stems from their observa- tion on a number of image classification datasets that a "teacher-free free KD" method, which employs a smoothed target distribution as a virtual teacher, has similar performance as standard KD. Finally, some of their empirical observations, where KD improves student accuracy even when the teacher is weak or poorly trained model, provide support for the more general view of KDD ularization. Investigation of this general question can be found in others' work as well, e.g., in the context of multi-generational self-KD (Zhang and R. Sabuncu, 2020; Mobahi et al., 2020) and early stopping (Dong et al., 2019).
The following a summary of our contributions:
We study the inner workings of KD in the rel- atively under-explored modality of language, specifically on text classification tasks.
form of reg-
We show that KD lacks the defining property of LS regularization, rebutting the claim that the former is a form of the latter.
We demonstrate how the specific selection of the teacher model determines student confi- dence in KD, providing strong support for its classical knowledge transfer view.
Finally, related line of work view KD sspeefice-specffic label smoothing", where the amount of smoothing is not uniform across cat- egories, but a function of the input approxi- mated using a pre-trained network (Zhang and R. Sabuncu, 2020; Lee et al., 2022). We note that this is a fundamentally different assertion than KD being form of LS with a uniform prior, since such methods do in fact bring new task-specific knowl- edge into the training process, with the pre-trained network essentially serving as a teacher model.
2 The Argument for KD R LS
Let us first take look at the main arguments, pri- marily from Yuan al. (2020), for why KD should be considered a form of LS. Given a training in- stance x, let (klzz) be the corresponding one-hot ground truth distribution over the different classes k E {1, K). LS replaces (kk(z) with the fol- lowing smoothed mixture as the target for training:
3 A Closer Look at the Arguments
(klz)) = (1 ()k(k(=)))))) + au(k)
As already stated, the focus of this paper is on the specific claim by Yuan et al. (2020) that KD is a form of LS with a uniform prior (Szegedy et al., 2016), a primary argument for which is that their cost functions in Eqs. I and 2 have similar form. A careful examination of the two functions, how- ever, reveals key difference.
where u(k) = 1/K is the uniform distribution over the K classes. Yuan et al. (2020) show that optimizing for qi. is equivalent to minimizing the following cost function:
LLS (a)H(()))))) + aDKI(u,P)
(1)
where p(k)z)) is the model output for x, and HI and DKL are the cross-entropy and the Kullback- Leibler divergence between the two argument distri- butions, respectively. Note that the H H(g. p) term is
To elaborate, let [I(.) represent the entropy of a probability distribution; it then follows that Vp E ["] H()) > Hp),, where A is a prob- ability simplex, a is the uniform distribution and pa

---

different distribution on A Accordingly, minimiz- ing DRz(u, p) in Eq. unconditionally increases H (p), training higher-entropy models than ERM, a fact that we also confirm empirically in $4.
Hyperparameter
Values
Learning rate # of epochs a (Eqs. and 2)
(1, 7 x 10
(1,2, 10}
(.1,2,
Table I: Hyperparameter grid for model selection.
The same is not true for D Dx (p', p) in Eq. 2. however, since p² is dependent on the specific se- lection of the teacher model I, and H (p²) can take any value between 0 and H (u). Crucially, this means that I1 (p') can be both higher lower than H (p) if the two respective models were trained separately to minimize LERM- Unlike LS, therefore cannot be guaranteed KD that mini- mizing ( .)) would increase (p)). In $4, we show how the selection of the teacher model affects the predictive uncertainty of the student, for different combinations of their sizes. Based on the same results, we also comment the teacher-free KD framework of Yuan et al. (2020).
Batch size is kept fixed at 32. We train with mixed precision on single NVIDIA A100 GPU. An AdamW optimizer with warm-up is used. In Appendix A.2, provide the hyperparameter con- figuration that optimal on validation and is sub- sequently evaluated on test for each model and task. We use the Transformers library of Wolf et al. (2020) for implementation.
The experiments reported in this section use KD temperature т effectively disallowing any artificial smoothening of the target distribution. In Appendix A.3, we also show that treating as a hyperparameter during model selection does not qualitatively change these results.
4 Experimental Results
Setup. We use four text classification tasks from GL.UE (Wang al., 2019) in our experiments:
Method
MRPC
QNLI
SST-2
MNL1
MRPC (Dolan and Brockett, 2005) asks for a yes/no decision if two given sentences paraphrases of each other.
ERM
.1123
.1419
.1101
.6988
LS
.6896
.6752
.447
.803
Standard KD Self-KD
.0548
.1210
.0594
.1887
.0738
.1341
.0977
.2758
QNLI provides a question and a sentence from SQUAD (Rajpurkar et al., 2016) and asks if the latter answers the former (yes/no).
Table 2: Test set posterior entropies of different DistilBERT classifiers. LS increases entropy over ERM as expected, while KD almost always reduces it.
SST-2 (Socher et al., 2013) is a sentiment detec- tion task involving positivelnegaative classifica- tion of movie reviews.
Results. Table compares the average entropy of posteriors over classes predicted by LS models with those by standard and self-KD students, measured on the four test sets. As expected, increases model uncertainty over ERM, fact quite signif- icantly (~3-6xx). Both KD variants, on the other hand, bring entropy down; the only exception to this is self-KD on MNLI, where we see very slight increase. These results have direct implications for the central research question of this paper, as they reveal that the typical effect of KD is reduction in predictive uncertainty over ERM, which the exact opposite of what LS does by design.
MNLI (Williams et al., 2018) is 3-way (entail- ment/contrraicciionneeutt
classification task asking for whether premise sentence entails hypothesis sentence.
For each task, we use a train-validation--tes split for training, model selection, and evaluation, re- spectively. See Appendix A.1 for more details. To examine the effect of KD in depth, like Yuan al. (2020), we run experiments under three dif- ferent conditions: Standard KD distills from fine- tuned BEse-base (Devlin et al., 2019) (BERT hence- forth) classifier into a DistilBERT base (Sanh et al., 2019) (DistilBERT henceforth) student: Self-KD dis- tills from DiStilBERT teacher into a DistilBERT stu- dent; Reverse KD distills from DistilBERT teacher into BERT student. In all cases, the teacher is trained using ERM with cross-entropy loss.
Figure illustrates how each training method regulates model uncertainty at real time during training. Both KD variants cause rapid drop in entropy carly in training, and entropy lower than that of both ERM and LS is maintained throughout. The comparisons are also very similar between the training and the test set, indicating that the smooth- ing effects (or lack thereof) of cach of these meth- ods generalize well from training to inference.
For every model evaluated under each condi- tion, we perform an exhaustive search over the hyperparameter grid of Table (when applicable).

---

10
entropy
62
40
100
66 % training
15 training
training
training
QNLI
SST-2
(d)
Figure 1: Change in posterior entropy of different DiTT classifiers training progresses.
82
60
40
100
62
Udc
160
training
training
training
tratning
QNLI
SST-2
(d) MNLI
Figure 2: Change in posterior entropy of different BERT classifiers as training progresses.
Method
MRPC
QNLI
8ST-2
MNLI
Method
MRPC
QNLI
88T-2
MNLI
ERM
81.0
88.5
93.8
81.3
.0678
.1288
.0124
.1845
80.3
88.4
95.0
81.0
.5122
.6288
.4376
.470
Standard KD
82.0
89.2
94.6
81.8
Reverse KD
.0954
.0834
.0954
.2011
Self-KD
82.3
89.0
94.7
82.2
Table 4: Test set posterior entropies of BERT classifiers. KD increases entropy in three out of four tasks, but the LS indindd increases are far greater in magnitude.
Table 3: Test set accuracy of DistilBERT fine-tuned with ERM, KD and I.S. KD has the best performance profile.
In Table 3 we report the test set accuracy of each method on the four tasks. Both KD variants do gen- erally better than LS, as the latter also lags behind the ERM baseline in three out of four evaluations. Interestingly, self-KD outperforms standard KD in most cases. While Yuan al. (2020) explain sim- ilar successes of weaker teacher models in image classification with the equivalence of KD and LS, an alternative explanation is that these results are simply an outcome of the standard-KI -DD student not adequately learning to mimic its higher-capacity teacher due limited amounts of training data. Others (Liang et al., 2021; Sultan al., 2022) have shown that such teacher-student knowledge gaps can be reduced by using additional synthetic data during distillation.
also observe a similar difference in runtime training entropy, depicted in Figure 2.
The contrast in the results of Tables 2 and 4 paints clear and important picture: the predic- tive uncertainty of the student in KD is a direct function of that of the teacher, and not of the KD process. To elaborate, we first point the reader to row 1 of both tables, which show that when trained using FRM, BERT classifiers have lower posterior entropy than their smaller DistiLBERT counterparts. Consequently, in Table 2, standard KD yields lower- entropy students than self-KD. For the same reason. reverse KD in Table also generally trains higher- entropy models than ERM.
The above results have key implications for the general question concerning the true nature of KD, namely, is it knowledge transfer or regularization? As student learns to mimic its teacher in KD, see in the above results that also inherits its confidence-an integral aspect the data-driven knowledge of both models. This is clearly unlike methods such as LS or teacher- faeceer-rre KD (Yuan
Next we turn attention to reverse KD from DiStilBERT into BERT. In Table LS uncondition- ally increases posterior entropy before. The effect of reverse KD, on the other hand, is quali- tatively different than standard and self-KD, as also increases entropy in most cases over ERM. We

---

Method
MRPC
QNLI
SST-2
MNLI
Tim Dettmers. Artidoro Pagnoni. Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient
ERM
84.0
91.0
94.5
83.7
LS
84.1
89.8
94.6
83.7
Finetuning of Quantized LLMs. arXiv preprint
Reverse KD
83.4
91.3
94.9
83.9
arXiv:2305. 14314.
Table 5: Test set accuracy of BERT classifiers fine-tuned with different training methods. KD and I. again have generally dissimilar performance profiles.
Jacob Devlin, Ming-Was Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In NAACL.
et al., 2020), which unconditionally reduce confi- dence in a data-agma aggoosic way to limit overfitting. With a weaker teacher model, reverse KD yields a smaller overall gain in accuracy than before over ERM and L. (see Table 5). It is interesting, nev- ertheless, that a lower inductive bias along with strong language modeling capabilities enables the reverse-KD student outperform its teacher.
William B Dolan and Chris Brockett. 2005. Auto- matically Constructing Corpus of Sentential Para- phrases. In Proceedings of the International Work- shop on Paraphrasing.
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. 2019. Distillation & Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network. arXiv prePrinft arXiv: 191001155
5 Conclusion
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the Knowledge Neural Network. In NeurlPS Deep Learning Worksop.
Knowledge distillation is a powerful framework for training lightweight and fast yet high-accuracy models. While still topic of active interest, any misinterpretation, including false equivalences with other methods, can hinder progress for the study of this promising approach going forward. this paper, we present evidence against its inter- pretation as label smoothing, and more generally, regularization. We hope that our work will inspire future efforts to understand distillation better and further improve its utility.
Dongkyu Lee, Ka Chun Cheung, and Nevin L.Zhang. 2022. Adaptive Label Smoothing with Self- Knowledge in Natural Language Generation. In EMNLP.
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021. MixKD: Towards Efficient Distillation of Large-scale Language Models. ICLR.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. 2021. A Statisti- cal Perspective on Distillation. In ICML.
Limitations
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. 2020. Self- Suft-Disilllaion Amplifies Regular- ization in Hilbert Space. NeurIPS.
To manage the complexity of our already large- scale experiments involving (a) four different classi- fication tasks, and (b) hyperparameter grids contain- ing up to 1,960 search points, we ran experiments with single random seed. While this is sufficient for exploring our main research question involv- ing the posterior entropy of different models for which we only need to show one counterexample, i.e., one where entropy decreases with knowledge distillation the exact accuracy figures would be more reliable averaged over multiple seeds.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev abd, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In ACI..
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistiBBERT, Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. In EMC at NeuriPS.
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. 2021. Is Label Smoothing Truly Incompatible with Knowl- edge Distillation: An Empirical Study. In ICLR.
References
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning. Andrew Ng. and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality over a Sentiment Tree- bank. In EMNLP.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang. Zhe Wang. and Chun Chen. 202 Cross-Layer Distillation with Semantic Calibration. In AAAI.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. 2021b. Robust Overfitting may Mitigated by Properly Learned Smoothening. In ICLR.
Md Arafat Sultan, Avirup Sil, and Radu Florian. 2022. Not to Overfit or Underfit the Source Domains? An Empirical Study of Domain Generalization Ques- tion Answering. In EMNLP.

---

Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jonathon Shlens, and Zbigniew Wojna. 2016. Re- thinking the Inception Architecture for Computer Vision. In CVPR.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A Strong, Replicable Seruuctio-FFollwinng Model.
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019. GLUE: A Multi-Task Benchmark and Analysis Plat- form for Natural Language Understanding. ICLR.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alísa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self- Geff-Generated Instructions. In ACL.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A Broed-Coverage Challenge Corpus for Sentence Understanding through Inference. In NAACI.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, emm Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-Aee Natural Language Processing. In EMNLP (System Demonstrations).
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. 2020. Revisiting Knowledge Distillation via Label Smoothing Regularization. In CVPR.
Zhilu Zhang and Mert R. Sabuncu. 2020. Self- Distillation Instance-Specifie Label Smoothing. NewrIPS.

---

A Appendix
only LS, but also ERM. In Table KD again has better overall test accuracy than LS. Across all these results, the comparisons between standard and self-KD also remain generally unchanged.
A.1 Datasets Used in Our Experiments
We collect all datasets and their splits from the Hugging Face Datasets library'. Among the four tasks, only MRPC has labeled test set. For the other three datasets, split the train set into two parts. QNLI and MNLI experiments, the two parts are used for training and validation, and the original validation set used as blind test set. For SST-2. -2 since the validation set is small, we also use it for model selection; the train set is therefore split into train and test sets. For MNLI, we evaluate only on the test-matched set, which consists of unseen in-domain examples. Table shows the number of examples in each dataset.
The reverse KD results- both predictive un- certainty in Table 10 and Figure 4, and test set performance in Table 1l-also show very similar trends those described in $4. Altogether, the results presented in this section with KD in three different settings demonstrate that even at high tem- peratures, our original findings and conclusions hold true.
The optimal hyperparameter combinations for the above experiments are provided in Table 12.
Task MRPC
Train
Validation
Test
3,668
408
1,725
QNI.
99,743
5,000
5,463
SST-2
65,549
872
1,800
MNLI
382,887
9,815
9,815
Table 6: Size statistics for our datasets.
A.2
Model Selection
Table 7 contains the optimal configurations for the models evaluated in $4 as observed on the respec- tive validations sets.
A.3 Distillation at Higher Temperatures
We present results for the same experiments in $4 here, with two main differences: First, for KD, we include the temperature the hyperparameter search and re-define the domain for a as follows:
Hyperparameter
Values 8,2,4,8, 16, 32, 64) {.25, 55,
f a (Eq. 2) Leaming rate # of epochs
f1,2,.., 10
...,...00
yielding a 4D grid consisting of 1.960 points. Note that for this more general experiment, we allow a = 1, which only uses the teacher's soft targets for training. Second, to keep the grid size the same (1,960) for I.S, re-define the domain of for contain 28 equally-distaneee points ranging from to .9911.
In Table 8 and Figure 3, we report posterior en- tropies of DistilBERT models measured at inference and during training, respectively. Similar to the experimental results in $4, both KD variants gen- erally train models with lower entropy than not
'https: http://ccc.cccccc/ttatttttttti/s

---

Task
Model
Training
Learning Rate
# Epochs
ERM
7e-5
BERT
LS
7e-5
Reverse KD
4e-5
MRPC
ERM
5c-5
1.5
9
4e-5
DistilBERT
Standard KD
60-5
Self-KD
8
7e-5
ERM
4e-5
BERT
L.
77
2c-5
Reverse KD
3
20-5
10
QNII
ERM
36-3
3
LS
8
20-5
8
DiSTiLBERT
Standard KD
8
3e-5
Self-KD
7
2c-5
10
ERM
Te-5
BERT
LS
3
2e-5
Reverse
7
le-5
SST-2
ERM
le.3
DistilBERT
LS
3
6e-5
Standard Self-KD
6c-5
7
Se-5
ERM
3e-5
BERT
LS
3e.5
Reverse KD
3e-5
MNLI
ERM
3e.5
L.5
36-5
DistilBERT
Standard KD
3e-5
Self-KD
9
36-5
Table 7: Optimal hyperparameter combinations (r=l) for the $4 experiments.
SePVC
Sanco
62
20
46
300
28
300
64
81
20
260
training
training
training
training
MRPC
(b) QNLI
SST-2
(d) MNLI
Figure 3: Change in posterior entropy for different DistilBERT classifiers as training (at optimal r) progresses.
Method
MRPC
QNI.I
SST-2
MNLI
Method
QNLI
SST-2
ERM
.1233
.1419
.1101
.2698
ERM
.0678
.1218
.0124
6916
2949
.6802
.5955
Standard KD Self-KD
.067
.1044
.0107
.1481
LS
.3778
.6512
.4891
.770
Reverse KD
.1058
.140
.0995
.2163
.0927
.1179
.0853
2476
Table 10: Test set posterior entropies of BERT classifiers fine-tuned with different training methods. While KD does increase entropy over ERM on three out of four tasks, LS yields increases that are much higher.
Table Test set posterior entropies of DistilBERT clas- sifiers fine-tuned with different training methods. Label smoothing (LS) increases entropy over ERM expected, but distillation (KD) induces drop on tasks.
Method
MRPC
QNLI
SST-2
MNLI
Method
QNLI
SST-2
ERM
81.0
88.5
93.8
81.3
ERM
84.0
91.0
94.5
83.7
78.4
88.8
95.2
80.8
Standard KD
80.2
89.6
94.6
82.7
LS
83.9
90.8
94.5
83.7
Reverse KD
84.3
91.3
94.9
84.0
Self-KD
82.1
89.5
94.8
82.6
Table 11: Test set accuracy of BERT classifiers fine-
Table Test set accuracy of DistiBBERT classifiers fine- tuned with different training methods. has the hest overall performance profile.
tuned with different training methods, KD clearly out- performs LS.

---

Rovere
entropy
entropy
SPM
entroo
% training
training
training
55 training
(a) MRPC
(b) QNLI
SST-2
(d) MNLI
Figure 4: Change in posterior entropy for different BERT classifiers as training (at optimal r) progresses.
Task
Model
Training
Learning Rate
Epochs
ERM
7c-5
2332
7c-5
Reverse KD
4e-5
MRPC
ERM
3e-5
DistilBERT
LS
.9255
3e-5
Standard KD Sclf-KD
3e-5
7c-5
ERM
4e-5
BERT
LS
.6944
4e-5
3
Reverse KD
.25
4e-5
2
QNLI
ERM
3e-3
DistilBERT
LS
.1333
3c-5
4
Standard KD Self-KD
.75
64
3c-5
9
64
3c-5
ERM
le-5
9
BERT
LS
3664
2e-5
3
Reverse KD
.75
1e-5
5
SST-2
ERM
le-5
2
I.S
8326
5e-5
DistiBBERT
Standard KD
2e-5
Self-KD
.25
5e-5
ERM
3e-5
3e-5
Reverse
2c-5
FRM
3e-5
DistilBERT
LS
.1666
3e-5
Standard
4e-5
10
Self-KD
5e-5
Table 12: Optimal hyperparameter combinations for the $A.3 experiments.


Reviews:
[{"review_id": "iZrlyGdXJr", "reviewer": "Reviewer_39Vd", "summary": "", "questions": "Question A: Line 151. Why does minimizing D_KL(u, p) in Eq. 1 increases H(p)?\nAs D_KL(u,p) = H(u,p) - H(u), there is no guarantee that minimizing D_KL(u,p) necessarily results in an increase of H(p).", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 39Vd for their valuable feedback. We respond to their questions and concerns below.\n\n**RR1:** \"The claim is grounded in empirical findings and does not provide a solid mathematical foundation.\"\n\n**Response:** We use a combination of mathematical arguments (Section 3) and empirical analysis (Section 4) to refute the existing claim that knowledge distillation (KD) is a form of label smoothing (LS) regularization. Reviewer 39Vd is correct to point out that our argument has empirical components, but it is unclear why that makes a scientifically flawed approach, as proof by counterexamples is a well-established method used to refute inductive claims such as KD ≈ LS.\n\n**RR2:** \"... KD can be viewed as a special form of LS. This is particularly true when the teacher network is uniformly distributed and the temperature is set at 1, then LS and KD are equivalent.\"\n\n**Response:** Since a strictly uniform teacher does not provide any useful training signal, we assume that Reviewer 39Vd meant a smoothened teacher. Since Reviewer 39Vd also referred to a \"teacher network\", we assume that LS is being applied to smoothen the output of a teacher network previously trained on labeled data. Such a teacher would combine two separate notions: (a) sourcing the target distribution from a trained network as in KD, and (b) smoothing that distribution with LS. We must recognize that this is a combination of KD and LS, not LS playing the role of KD. \n\n**RR3:** \"The authors only compared one of the existing works in this area and did not sufficiently address related works. Here are some related works for LS and KD: ...\"\n\n**Response:** One of the papers Reviewer 39Vd lists (“Revisit Knowledge Distillation: a Teacher-free framework”) is actually an earlier version of our baseline paper by Yuan et al. For the rest, it is unclear to us what exactly to compare with from these papers. We have a very specific goal: to examine the accuracy of the assertion that KD and LS are *equivalent*, and not any other relation between the two methods or how such relations can be exploited to develop new algorithms such as \"teacher-free distillation\". While these papers generally explore KD and LS/regularization in some way, none of them directly investigate the question of whether LS and KD are equivalent. Crucially, these papers also do not examine the properties of what is arguably the most common form of KD — distilling a bigger teacher model into a smaller student model — where we find the differences between KD and LS to be the starkest. We are generally aware of these studies, and will cite them and more as related work in the final version given the additional space.\n\n**Q(A):** \"... Why does minimizing D_KL(u, p) in Eq. 1 increases H(p)? As D_KL(u,p) = H(u,p) - H(u), there is no guarantee that minimizing D_KL(u,p) necessarily results in an increase of H(p).\"\n\n**Response:** The uniform distribution u has the highest entropy H(u) among all distributions on the respective probability simplex. Accordingly, as p becomes smoother and H(p) increases, H(u, p) goes down, also lowering D_KL(u, p) = H(u, p) - H(u) [Note that H(u) is a constant]. The following table illustrates this for the Bernoulli distribution (the distribution associated with binary classification tasks) as an example:\n\n| p(0)  | p(1) | H(p) | H(u, p) | D_KL(u, p) |\n| :-: | :-: |:-: | :-: | :-: |\n| .1 | .9 | .141 | .523 | .222 |\n| .2 | .8 | .217 | .398 | .097 |\n| .3 | .7 | .265 | .339 | .038 |\n| .4 | .6 | .292 | .310 | .009 |\n| .5 | .5 | .301 | .301 | 0 |\n| .6 | .4 | .292 | .310 | .009 |\n| .7 | .3 | .265 | .339 | .038 |\n| .8 | .2 | .217 | .398 | .097 |\n| .9 | .1 | .141 | .523 | .222 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The experimental findings, which suggest that the student inherits the teacher's predictive uncertainty, are somewhat interesting.\n\n- The experiment is well organized, showing the changes in entropy and accuracy throughout training and testing.", "reasons_to_reject": "- The claim is grounded in empirical findings and does not provide a solid mathematical foundation.\n\n-  Although I acknowledge that KD and LS are not identical, I believe KD can be viewed as a special form of LS. This is particularly true when the teacher network is uniformly distributed and the temperature is set at 1, then LS and KD are equivalent.\n\n- The authors only compared one of the existing works in this area and did not sufficiently address related works. \nHere are some related works for LS and KD:\n\nLee, Dongkyu, Ka Chun Cheung, and Nevin Zhang. \"Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\nZhang, Zhilu, and Mert Sabuncu. \"Self-distillation as instance-specific label smoothing.\" Advances in Neural Information Processing Systems 33 (2020): 2184-2195.\n\nLi Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. \"Revisit knowledge distillation: a teacher-free framework.\" arXiv preprint arXiv:1909.11723, 2019.\n\nYun, Sukmin, et al. \"Regularizing class-wise predictions via self-knowledge distillation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n", "questions_to_authors": "Question A: Line 151. Why does minimizing D_KL(u, p) in Eq. 1 increases H(p)?\nAs D_KL(u,p) = H(u,p) - H(u), there is no guarantee that minimizing D_KL(u,p) necessarily results in an increase of H(p).", "paper_topic_and_main_contributions": "The paper revisits the connection between knowledge distillation (KD) and label smoothing (LS). Previous studies suggested that KD could be interpreted as a form of LS. Both KD and LS introduce soft labels instead one-hot targets. By minimizing the KL divergence between the student's predictions and soft labels, KD acts as a regularizer similar to how LS does with its smoothed labels. However, in this study, the authors argue that KD and LS are not equivalence. The authors evaluate this through four text classification tasks from the GLUE benchmark. By analyzing the entropies of LS and KD, the experimental results present evidence against KD's interpretation as label smoothing.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Lee, Dongkyu, Ka Chun Cheung, and Nevin Zhang. \"Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\nYun, Sukmin, et al. \"Regularizing class-wise predictions via self-knowledge distillation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020."}}, {"review_id": "ccdj6tFPVv", "reviewer": "Reviewer_qYKw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer qYKw for their valuable feedback. We respond to their questions and concerns below.\n\n**RR1:** \"The paper will be stronger if the analysis is extended to other machine learning tasks such as image classification in CV.\"\n\n**Response:** While we agree that results on more tasks and modalities would be interesting to look at, all that is needed to refute an inductive claim such as KD ≈ LS is a single counterexample. The goal of this short paper submission to EMNLP is to present a counter-perspective to the above claim in the modality of language, which we do using experiments on four different text classification tasks. We hope that our work will inspire future efforts to understand KD better, including on other tasks and modalities. Please also see our response to Reviewer FFtC for a related discussion.\n\n**RR2:** \"Though this paper shows the advantage of KD over LS, the results have been known by most researchers about KD (since they can simply get the conclusions by the model performance). I wonder whether there are any suggestions for readers to design a better knowledge distillation model in application.\"\n\n**Response:** We would like to first clarify that our primary goal is not to show that KD is a superior model supervision technique to LS, but that their underlying mechanisms are different. This is part of the more general question of correctly interpreting KD, the importance of which we discuss in Sections 1 and 5. Given this goal, developing better KD algorithms is out of the scope of this short paper submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clear and well-written. This paper shows an investigation of the relationship between knowledge distillation and label smoothing, which can help the community to better understand the strength of knowledge distillation. The experimental results provide valuable insights into the behavior of these techniques in text classification tasks and contribute to a better understanding of knowledge distillation.\n", "reasons_to_reject": "1. The paper will be stronger if the analysis is extended to other machine learning tasks such as image classification in CV. \n2. Though this paper shows the advantage of KD over LS, the results have been known by most researchers about KD (since they can simply get the conclusions by the model performance). I wonder whether there are any suggestions for readers to design a better knowledge distillation model in application.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about the interpretation of knowledge distillation (KD) in machine learning and its relationship to label smoothing (LS). The paper addresses the problem of misinterpretation of KD as a form of regularization, and the false equivalences made between KD and LS. The main contribution of the paper is to provide evidence against the interpretation of KD as LS, and to clarify the fundamental differences between the two techniques. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8I0xg1NlLN", "reviewer": "Reviewer_FFtC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer FFtC for their valuable feedback. We respond to their questions and concerns below.\n\n**RR1:**  The experimental tasks in this paper is multiple text classification tasks, while the task in Yuan et al., 2020 is image classification. I was wondering whether the different conclusions drawn from this paper and Yuan et al., 2020 are due to the totally different tasks.\n\n**Response:** An inductive claim such as KD ≈ LS must hold across different tasks and modalities. The fact that in text classification we observe very different characteristics in the models trained by the two methods already sufficiently refutes the equivalence assertion. That said, it is also not difficult to run a thought experiment on image classification where we can expect to see similar results: First, LS by design would always train high-entropy models irrespective of task and modality. All that is now needed to complete this thought experiment is an overconfident image classification teacher model (already quite common among large neural networks, but can also be created artificially using techniques such as temperature downscaling) for KD -- distilling such a teacher at low temperatures, when successful, should yield students that are also peaky. Thus we strongly believe that the different conclusions reached by Yuan et al. and us can be attributed to the different levels of detail in which the two studies explore the KD ≈ LS claim and are not due to our divergent selection of tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper gives a different point of view against the previous studies that \"KD is a form of LS regularization\". The authors give analyses that KD and LS are different, and provide empirical results to support their arguments. \n- The reverse KD experiments is also interesting, and show that KD is more about transferring knowledge from a teacher model to a student, rather than a regularization process like LS.\n- This paper may give others a better understanding of KD in future research.", "reasons_to_reject": "- The experimental tasks in this paper is multiple text classification tasks, while the task in Yuan et al., 2020 is image classification. I was wondering whether the different conclusions drawn from this paper and Yuan et al., 2020 are due to the totally different tasks. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the relationshiop between Knowledge Distillation (KD) and Label Smoothing (LS). The authors re-examine the claim that \"KD is a type of learned LS regularization\" and claim that KD and LS work in different ways and are therefore not equivalent. They empirically show that KD and LS are only similar in terms of the loss function form. Experiments demonstrate that LS largely increases the model uncertainty over standard empirical risk minimization while KD tends to decrease it. This paper provides a different view of KD vs. LS, which may lead to better understanding of KD in future research.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L41: Maybe an explanation for NLP before using the abbreviation."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides empirical evidence that knowledge distillation and label smoothing are not equivalent, contradicting previous work which has argued that KD could be interpreted as LS. The reviewers are generally in agreement that the empirical results are interesting and convincing, and that the experimental setup is well-executed. However R1 believes that the paper could benefit from a theoretical treatment to support the empirical evidence in order to be more convincing, as well as addressing more of the literature on the topic. Additionally, the generality of the results could be improved by incorporating data with different modalities e.g. vision. Overall though, the reception is quite positive and the paper would make a good contribution to the conference.
Decision: Accept-Main