Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper provides a way to research the effect of soft labels in knowledge distillation from the perspective of sample-wise bias-variance tradeoff.  Based on some observations about regularization samples, the authors propose the weighted soft labels to handle the tradeoff.  Experiments on standard datasets show that the proposed method can improve the standard knowledge distillation.

The motivation and logic of the article are clear. Based on the form of rewriting distillation loss into a regularization loss adding the direct training loss, this work introduces the sample-wise bias-variance tradeoff, which help to understand the regularization effect of KD.  Despite the weighted soft label method is heuristic, given the simplicity and effectiveness of the proposed method, I think this paper is satisfactory.

Still, I would like to give the authors some suggestions:
1.Section 3.4 should do more experiments about the intermediate state between excluding regularization samples and only on regularization samples to clearly show the tradeoff.
2.Can the authors give some quantitative results in the Table 5 experiments to show how weighted soft labels tune sample-wise bias-variance?
3.Can the weighted soft labels be applied to the variants of KD?  It would be better to show some results about this respect. 
review_title: A new perspective to understand the effect of soft labels in KD

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper shows a new perspective of tackling the knowledge distillation problem. The author(s) have decomposed the expected student's training error into the bias, variance, and irreducible noise parts. This decomposition is further rewritten as two parts: one for bias reduction and another for variance reduction. The motivation is clearly explained and the experimental results show that this new approach can improve the model training performance of the student on both CIFAR100 and Imagenet. 

pros:
- A novel perspective of performing knowledge distillation.
- The motivation is clear and the framework is straightforward to understand.
- The code is provided, therefore, I believe the reproducibility is high.

cons:
- The definition of variance is flawed. The formulation doesn't match with the figure illustration and code implementation.
- The observed phenomenon regarding the regularization examples is not well explained.
- The experimental details are not sufficient.

concerns:
- My major concern comes from the definition of variance and its corresponding implementation. From my understanding, the variance should be estimated from multiple teachers, just as shown in Fig 1, which matches my thoughts. But it seems like the author(s) are trying to fade this concept intentionally. First, the expectation's definition for Equation 1 is not clear. It should take the expectation over the space of infinite teachers, or at least empirically multiple teachers with a substantial number. Second, after checking on the code, I only find one teacher is utilized in the implementation. Would you please elaborate on why this is the design? 

- Second concern is that the mathematical reasons behind the not-too-big or not-too-small weights for these regularization examples are unclear. This reweight scheme is purely empirically, and it would be a much solid paper with mathematical interpretations. 

- The bias-variance decomposition on classification has been well studied in multiple papers, e.g., [Pedro Domingos 2002], and [Giorgio Valentini 2004]. The relation (similarity and difference) between your employed BVD from [Heskes 1998] and [Pedro Domingos 2002] is not discussed. The latter [Giorgio Valentini 2004] is not even cited, which seems inappropriate. Also, comparing with the variance definition of [Zitong Yang, 2020], have you tried multiple loss functions and observe the same phenomenon? 

- minor issue: the sample-wise error for one example x should be explicitly listed and the notation should be consistent through the whole paper, y(x), and y should not be mixed together. I know the original [Heskes 1998] paper did the same, but you should be consistent on this notation. 

- other minor things: Could you please elaborate a bit more of how you preprocess the imagenet data into 1.2 million images and how many classes are remained? This piece of information seems missing and the code doesn't reveal much about it. 
review_title: a new perspective of performing knowledge distillation using bias-variance decomposition 

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors studied the soft labels for knowledge distillation from a bias-variance tradeoff perspective. Specifically, the authors first provide a mathematically descriptions of the bias-variance decomposition in knowledge distillation. Then, based on the theoretically analysis and experiments, the authors proposed an novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff.

Strength:
1. The paper is well written and easy to follow.

2. The authors provide many mathematically proof in the paper, which could serve as a theoretically foundation of this topic. 

3. The authors claim that their code is available in the supplementary, which makes it easier for other researchers to reproduce this work.

Weakness:
1. In addition to computer vision, Knowledge distillation is also very popular in NLP area recently. Maybe the authors can add more experiments for NLP to prove that the solution can be widely-adopted.

2. I am not very familiar with computer vision area, but it seems that ResNet-50/ResNet-34 has a huge gap with the current SOTA (Ref: https://paperswithcode.com/sota/image-classification-on-imagenet). Experimenting with stronger baselines would make the conclusion even stronger.  

Overall comments:
I think this is a good paper and I'd like to see it to be accepted.
review_title: Official Blind Review #4

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper analyzes the distillation from the bias-variance perspective. Beyond this, the regularization samples affect the performance. Based on the observation, a novel weighted mechanism is proposed to distill knowledge from teacher networks.

Strengths:

+) This paper is clear and easy to follow, the organization is good. The bias-variance analysis, the regularization samples, the weighted soft labels all make sense. I felt comfortable when I was reading this paper.

+) The analysis is clear and reasonable. The deduction seems correct. The figures are clear.

+) The experiments are enough to examine the effectiveness of the proposed weighted distillation (see below).

+) The code is submitted to contribute to the community. I appreciate the submission.


Weaknesses & Concerns:

-) Sec. 3, first paragraph, $T(x, \tau)$ -> $\hat{y}^t = T(x, \tau)$, $S(x, \tau)$ -> $\hat{y}^s = S(x, \tau)$ to make it more clear. 

-) ' For loss function, we set α = 2.25 for distillation on CIFAR and α = 2.5 for ImageNet via grid search.' How many $\alpha$s have been tested? What are the results? The main concern is that the grid search is costly in practice.  Therefore, I appreciate the analysis in this paper that helps us understanding KD better. However, the grid searched hyper-parameters makes Sec. 4 costly in practice.

Based on the quality of the paper, I select 6 as the initial score.
review_title: Review

Meta-review: The paper investigates the effect of soft labels in knowledge distillation from the perspective of sample-wise bias-variance tradeoff. They observe that during training the bias-variance tradeoff
varies sample-wisely. and under the same distillation temperature setting, we
 distillation performance is negatively associated with the number of regularization samples. But removing them altogether hurts the performance (the authors show empirical evidence of this). Based on some observations about regularization samples, the authors propose the weighted soft labels to handle the tradeoff. Experiments on standard datasets show that the proposed method can improve the standard knowledge distillation.

pros.
-the paper is written clearly.
-through the review period the authors added additional experiments suggested by the reviewers and enhances experimental results. The experiment results are convincing and the authors have now added explanations on hyperparameter choices.
-the mathematical setting is now clear after incorporating reviewer's comments.
-the missing related work as suggested by reviewers is added

cons.
-comparison with results of Zitong Yang et al 2020[1] is missing.

I thank the authors for incorporating the changes requested by reviewers. Please add comparison with result of [1] in the final version.

[1] Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, Yi Ma

Decision: Accept (Poster)