Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper is mainly based on the prior work by Muller et al., which suggests that label smoothing is incompatible with knowledge distillation. Firstly, this paper provides an explanation of this incompatibility---label smooth tends to erase relative information among different classes, and provide a way to qualitatively measure the degree of erased information. Then, this paper argues that label smoothing actually is compatible with knowledge distillation, and show several empirical results as evidence. Lastly, this paper suggests that the performance of the teacher model is a more directly related factor for determining the performance of the student model.


Pros:

(1) This paper performs a set of careful diagnoses on showing the effects of information erasing (caused by using label smoothing) at the category-level, and observes an interesting phenomenon: erasing relative information only cause negative effects to semantically different classes, but will help the classification for semantically similar classes. Both qualitative and quantitative evidence is provided to confirm this observation.

(2) A simple and novel metric is proposed to facilitate the measurement of the degree of erased information. 

(3) Extensive experiments on the image classification task and the neural machine translation task are provided to confirm that label smoothing is indeed compatible with the knowledge distillation framework.



Cons (please address them during the rebuttal):

(1) The presentation of this paper needs to be improved. In all abstract, introduction and conclusion sections, this paper highlights that "we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness". Nonetheless, the reviewer CANNOT find any related discussions in the main paper. The only related discussion is provided in the appendix. The reviewer does not think it is a good way for presenting the paper, as the appendix is mainly used for explaining some not very important details. Putting the entire discussion of an important contribution of this paper in the appendix is inappropriate.

(2) Though the reviewer agrees that removing the hard label part in knowledge distillation can facilitate the analysis of this paper, the authors should also provide a brief discussion on whether adding this hard label part back will still lead to the same conclusions. For example, in table 2, with the hard label part, if the teacher model with label smoothing can still help the student model.

(3) One minor question is that, as shown in Table 1, ResNet-50+long substantially outperforms ResNet-50 in terms of accuracy, but their stability measurements are nearly the same. Can the authors provide any explanation of this "counter-intuitive" phenomenon?
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Paper Summary**
The authors re-analyze and re-confirm the relationship between label-smoothing and knowledge distillation, which is firstly argued by Muller et al. (“When Does Label Smoothing Help?“, NerurIPS 2019.). This paper shows that the previous argument, "label smoothing is not helpful for knowledge distillation", does not always hold, and carefully re-visits the missing points of the previous analysis by Muller et al. Based on this analysis, label smoothing can be helpful for knowledge distillation and can be explained using the intra-class variation and between-class distance within similar classes. The authors have empirically verified the arguments of the paper with various experiments. 

**Pros**
1. Introduced an interesting analysis for improved knowledge distillation via label smoothing. 
2. The paper is well written and the contributions are clearly explained by comparing against the previous work (Muller et al.)

**Cons**
1. The main analysis is based on the original knowledge distillation paper (Hinton et al., 2015), therefore, it seems to be difficult to apply to the recent knowledge distillation. For example, the proposed analysis cannot explain the recent works (Relational Knowledge Distillation (CVPR 2019), or Contrastive Representation Distillation (ICLR 2020) which are based on the "relation". 
2. The stability metric $S_{stability}$ is based on the intra-class variation, but the sample variation of the entire dataset is not considered. Since the intra-class variation is proportional to the variation of the entire samples, the increase of the stability metric might come from the decrease of the entire samples' variation. For example, LDA (linear discriminant analysis) utilizes both intra- and between-class variation. Thus the stability metric needs to be improved and the experiment's scores need to be measured again. 
3. Indeed, the knowledge distillation was originally proposed using cross-entropy [1][2], so section 4 (explaining that distillation loss is the same as cross-entropy loss) is not a new observation and it is not necessary for the paper's flow. 
[1] Distilling the Knowledge in a Neural Network (NeurIPS workshop 2015)
[2] Fitnets : FITNETS: HINTS FOR THIN DEEP NETS (ICLR 2015)
4. Figure 4 needs to be improved. It is hard to see and capture the meaning of "other soft predictions" in the figure. It would be nice to improve the figure 4 so that it clearly expresses what it is intended.
5. In appendix D, does the "loader" utilize standard ImageNet data augmentation (e.g., random-resize-cropping)? 

**Comments after the author response**
- The authors have answered all my questions and they revised the manuscript as well, so I will keep my initial rating (weak accept). 

review_title: Interesting paper, but need some clarification

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper empirically discusses the relationship between Label Smoothing (LS) and Knowledge Distillation (KD). It designs a  stability metric to measure the degree of erasing information and finds that LS can be compatible with knowledge distillation except in long-tailed distribution and increased number of classes.

Strengths:

1. Adopting stability metric to measure the degree of erasing information quantitatively is straightforward and effective. 
2. Extensive experiments from image classification to NMT are conducted to reveal the relationship between LS and KD and validate the idea of the work.

Weaknesses:
1. Even this work discusses the detailed relationship between LS and KD, but for me, the finding of this work is not enough to reach the bar of the top-tier conferences. 
2. The paper has many claims (Bold or Italic), some of that are well-known by the KD community, but too many claims make readers lose what is the paper's key insight. 
3. I am still confused that how do the finds in the paper can guide the KD community?  I think "better supervision is crucial for distillation" or "better teachers usually distill better students" should be a well-known and practical skill for KD, and I think many researchers in the KD community don't agree with the conclusion that "a teacher with better accuracy is not necessary to distill a better student" in [1]. 

[1]. When does label smoothing help? In Advances in Neural Information Processing Systems, Muller, et al.

After reading the response from the authors, I would like to increase my rating to 6: Marginally above acceptance threshold.
review_title: Review of AnonReviewer4

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Recent literature proposed that even label smoothing improves the teacher model, it will hurt the distillation training of student models due to the information erasing. Although this idea dominated more and more literature, this paper argued that this observation is not entirely correct. In order to clarify this idea, the paper systematically discussed the correlation between knowledge distillation and label smoothing. Comprehensive experiments well support the claims in this paper, i.e. label smoothing is compatible with knowledge distillation. The correlation between label smoothing and knowledge distillation remains an open question to date, and this paper made a breakthrough regarding this question. Besides the main purpose (clarify previous ideas), this paper also provided multiple interesting empirical conclusions, e.g. a better teacher always leads to a better student by producing more informative distillation labels, the distillation itself can provide enough regularization for training and the hard-label classification loss is no more needed. 

To conclude, the paper overturns the previous perspective with convincing explanations, discussions, and experimental results. Several empirical discoveries are introduced, which are expected to have high impacts on the tasks of knowledge distillation. The major contributions of this paper can be concluded as:
1) The paper empirically confirmed that label smoothing is well compatible with knowledge distillation, overturning previous dominant ideas. This is an important finding because it can prevent subsequent research from being misled.
2) It further explained the phenomenon of relative informative erasing, which only happens on the semantically different classes. Thus previous lopsided ideas (label smoothing hurts knowledge distillation) can be well explained.
3) The paper claimed that the dominating factor in knowledge distillation is the performance of the teacher and further proposed a stability metric to measure the quality of supervision. This metric is crucial in the tasks of knowledge distillation since it provides a simpler and faster way to measure distillation quality. The paper also claimed that the distillation loss itself can provide enough regularization, which also inspires me a lot.

It's quite a good empirical paper and I really enjoy reading it. I think there's no significant weakness on it, so I recommend a clear acceptance for this paper.
review_title: Official Blind Review #2

Meta-review: 
This paper studies the effect of label smoothing on knowledge-distillation. A previous work on this topic (Muller et al.) has claimed that label smoothing can hurt the performance of the student model in knowledge-distillation. The rationale behind this argument is that label smoothing erases information encoded in the labels. This work shows that such claimed effect does not necessarily happen. Specifically, by a comprehensive study on image classification, binary neural networks, and neural machine translation, the authors show that label smoothing can be compatible with knowledge distillation. However, they conclude that label smoothing will lose its effectiveness with long-tailed distribution and increased number of classes.

Overall ratings of this paper are all on the positive side, and R2 finding this paper an important step toward understanding the interaction between knowledge-distillation and label smoothing. I concur with the reviewers about the importance of this research direction and I think this submission provides a reasonable empirical evidence to change our earlier perspectives. I recommend accept.

While the paper specifically studies the effect of label smoothing on knowledge-distillation, I think providing a bigger context and reviewing some of the recent demystifying efforts on understanding knowledge-distillation could allow paper to communicate with a broader audience. I hope this can be accommodated in the final version.

Decision: Accept (Poster)