Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduced a new data-centric perspective on the phenomenon of 'larger teacher, worse student’.  They search for the classes that disrupt the distillation process, they defined them as the undistillable classes, and then exclude them to improve the performance of the small student network when distillation; this framework is called 'Teach Less Learn More (TLLM)’. Specifically, to find the undistillable classes, they suggest two criteria; per class accuracy and agreement prediction agreement. Various ablation studies support their observation and experimental results on multiple datasets with varying networks validate the effectiveness of TLLM. 
strengths_and_weaknesses: ### Strengths
- The observations are interesting and well supported experimentally. 

- The method is conceptually simple and seems to work well across the board. 

- Experiments are conducted with several neural network architectures and various datasets. 

### Weaknesses
- The concept and approach of the proposed method are different from ESKD[3] but somewhat similar in methodology for improving the KD performance; ESKD also halts distillation from teacher to student at the last phase. A direct comparative experiment with ESKD will enhance the completeness of this paper. 

- The proposed method, TLLM, is potentially impactful, but it should use significantly more computational resources and time for finding undistillable classes. Thus, this point is not useful from a practical point of view. It would be great if they could suggest an efficient way to find undistillable classes. 

- They claim that unstillable classes are universes, but there is no basis for that. It is necessary to analyze the characteristics of the classes and why they interfere with distillation. 

- Minor point 
> Line127, replace j with M  
> Line163-5, please check the suitability of examples  
> Line166, what is the meaning of crystal  
> Figure 4, explain the meaning of the dashed line  
questions: I was wondering if there is a practical way to apply TLLM over the new dataset. 
limitations: Please check the weaknesses and questions; in terms of practicality and analysis.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper propose that some classes is not distillable to students, and by removing these classes, the distillation performance would be improved.
strengths_and_weaknesses: Pros:
  1. The paper is well written and easy to follow.
  2. The idea is simple and effective in practice. The observation that some classes are not distillable to the student is novel.
  3. Experiment result is competitive against some mainstream KD methods.

Cons:
  1. The paper lacks deep insight into these bad distillable classes. The method of this paper is similar to "On the efficacy of knowledge distillation". Both learn teachers at the beginning, and then gradually start to learn one-hot labels. The difference is that this paper argues that certain classes are the cause of the problem. So why do hard examples come in the form of categories, are all samples of these certain classes hard to distill?
questions: Would the author analyze why some undistillable classes are consistent across multiple experiments, while others are not? And in those consistent classes, is the same group of instances undistillable?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper aims to study the problem of “larger teacher, worse student” in knowledge distillation (KD). 
From a data-centric perspective, this paper claims that undistillable classes are the cause of the inefficacy of large teachers. Furthermore, this paper proposes a new KD framework called Teach Less, Learn More (TLLM) to address the issue. In detail, TLLM tries to identify the undistillable classes during training with a moving window and lets the student learn these classes directly from the ground truth labels. 
strengths_and_weaknesses: Strengths

- The data-centric perspective is novel and interesting in the study of “larger teacher, worse student”, which understands the problem much better than the previous capacity mismatch explanation. Moreover, To the best of my knowledge, the per-class accuracy has not been explored in KD before. It deserves to be studied more in the future since the undistillable classes are universal. 

- Experiments are comprehensive and the results are convincing. This paper considers many teacher-student pairs and even some advanced architectures such as vision transformers (ViTs). Moreover, this paper considers both output-based KD and feature-based KD. 


Weaknesses

- A clearly mathematical definition of the “undistillable class” is missing. 
From section 2.1, we know that the “undistillable classes” are classes with bad distillability. However, a specific mathematical distillability measurement ||c is not given, which makes it difficult to understand the detailed implementation in this paper.  As in Figure 5, we know that the undistillable class is a class whose accuracy after KD is lower than that before KD. However, it is unclear whether this is a necessity and sufficiency of undistillable class. It is also unclear about what specific criterion this paper uses to select samples in Figure 1. To this end, this paper should give a clear definition of it in the rebuttal letter. 


- More studies of the undistillable classes should be discussed. 
This paper mainly studies the proportion of undistillable classes, which is insufficient. More studies about the distribution of these classes should be explored. For example, whether these undistillable classes are challenging classes (i.e., classes with relatively low accuracy before KD), and whether these classes are network-independent.   

questions: The explanation from per-class accuracy in general novel. Nevertheless, the above weaknesses refrain me to give a better score. I would be happy to improve my rating as long as the authors address the above weaknesses in the rebuttal.   
limitations: Yes, this paper clearly discusses the limitations and potential negative social impacts in the supplementary.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The major contributions of this paper are 2-fold:
1) The thesis of this paper is to study the “larger teacher, worse student” phenomenon in knowledge distillation (KD) from a data-centric approach. The authors show that the above phenomenon happens due to the presence of undistillable classes in the teacher. This observation is shown across different KD methods, datasets and teacher-student architectures.

2) The authors propose a simple, yet effective fix -- Teach Less, Learn More (TLLM) -- to identify and discard undistillable classes during student training, thereby improving the final accuracy of students.

strengths_and_weaknesses: **Strengths:**
1) This paper is written / presented clearly (especially Fig 5, bottom Middle). It is easy to follow.
2) The proposed TLLM framework shows noticeable improvements on CIFAR-100 (across different KD methods, teacher-student architectures). 


**Weaknesses:**
1) Although the work is technically sound, do you think the explanation / definition of undistillable classes is a bit overkilled? Can the following be a more simpler explanation (I will use ImageNet experiment as example): The teacher network is not perfect. That is the top1 accuracy of ResNet-50 teacher is 76.2% (Table 3). Therefore, the teacher tends to make a noticeable amount of errors. If class-wise analysis is performed on the teacher based on misclassified samples, it corresponds to a subset of classes which can be treated as undistillable classes. Therefore, the proposed TLLM avoids such situations by eliminating classes where the teacher's predictions are largely incorrect and replacing them with ground-truth labels (one-hot vectors). Some analysis I’d like to see are:
- What is the average accuracy of ResNet-50 teacher (Table 3) for regular and undistillable classes?
- Can you decompose the improvements obtained ($\Delta$) in Tables 2, 3 for regular and undistillable classes?

2) ImageNet results are insufficient to assess the efficacy of proposed TLLM: What is the reason for using only KR (Chen et. al) as the baseline for ImageNet experiments? Can the authors produce a table similar to Table 2 for ImageNet to show the improvements of the proposed TLLM scheme?

3) The Baseline performance of MobileNetV2 in Table 3 (ImageNet) seems to be very low. top1 accuracy should be ~72.154% whereas the reported accuracy is only 68.900%. Please clarify. Link to public pytorch models with top1 accuracies: https://pytorch.org/vision/stable/models.html

4) If the authors use label smoothing for undistillable classes (Lines 66-68) in the student, it is not reasonable to compare with reported Baselines as label smoothing will further improve accuracy (by alleviating models’ overconfidence [1, 2, 3]). A more reasonable comparison will be to include the accuracies of students trained with label smoothing. Can you report these results for Tables 2, 3. I suspect that a noticeable amount of improvement can be obtained by using label smoothing, especially if undistillable classes are semantically similar classes (See [2, 3]). 

5) CUB200 KD results are missing (Not available in Supplementary as well). 

This is an interesting paper. In my assessment, I feel that the weaknesses of this paper outweigh the strengths. But I’m happy to change my opinion based on the rebuttal. 


====================

**Post-Rebuttal:**

Thank you authors for the great effort on the rebuttal. 

I have increased my recommendation. Authors have addressed most of my concerns although concrete understanding of undistillable classes still remains unclear. 

As discussed, please include details on the effect of label smoothing on the proposed TLLM framework.

==================

[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." Advances in neural information processing systems 32 (2019).

[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K. T., & Savvides, M. (2021). Is label smoothing truly incompatible with knowledge distillation: An empirical study. In ICLR

[3] Chandrasegaran, K., Tran, N. T., Zhao, Y., & Cheung, N. M. (2022). Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?. ICML

questions: Please see Weaknesses section above for a list of questions. Further please consider answering the following questions:

1) What is the final optimization / loss function (preferably in equation form)?
2) What is the mixture parameter ($\alpha$) used for label smoothing in undistillable classes?

limitations: Limitations / potential societal impacts discussed in Supplementary Section A.
review_title: null

Meta-review: metareview: This paper makes an interesting observation on knowledge distillation such that excluding certain undistillable classes improves performance. This observation is quite interesting and potentially impactful for a better understanding of knowledge distillation. The authors use this observation to consistently improve the existing knowledge distillation methods in the experiments. One weakness is that the explanation of "why" it is beneficial to exclude certain classes is not very satisfactory. 

Nevertheless, the strength of this paper outweighs the weakness. All the reviewers are positive about this paper and I also recommend acceptance. 
confidence: Certain
Decision: Accept