Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the knowledge distillation problem by formulating it as a transfer learning problem. It finds that even if the teacher is trained with balanced training data, the transferred information could be imbalanced and it hurts the performance of KD. Therefore, this paper proposes Inverse Probability Weighting Distillation (IPWD) as an improvement of typical knowledge distillation loss, and extensive experimental results show the effectiveness of the proposed IPWD.
strengths_and_weaknesses: + The paper investigates KD from a transfer learning perspective, which is interesting compared to the typical dark knowledge. 
+ The structure and writing are good. 
+ The experimental results are comprehensive. 

- Figure 1 shows that there seems long-tailed property of teacher predictions. I wonder whether it could be fixed by some techniques from long-tailed classification. 

- Formulation of propensity in Section 4.2 seems sort of heuristic. I do not really get how to understand it in a principal and mathematical way. 

- IPDW is a logits-based KD method. Would it be better if combined with the feature-based method? Necessary discussions and experimental comparisons are needed.  

- This paper claims that knowledge distillation will work if there is a valid transfer gap. However, recent work argues that if the teacher becomes way too strong, the KD seems to degrade. Will the proposed IPDW handle this issue? If so, experiments about a strong teacher should be included. A recent work DIST investigates a similar setting, which should be helpful if the authors want to verify the transfer gap issue in terms of a huge or strong teacher. 

Knowledge Distillation from A Stronger Teacher, https://arxiv.org/abs/2205.10536




questions: Please refer to the "Weaknesses". 
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new KD method named IPWD. The authors state that the teacher's knowledge is imbalanced due to the imbalanced soft labels in teacher's predictions, and propose to use inverse probability weighting (IPW) to balance the weight of each sample in KD loss. Experiments on CIFAR-100 and ImageNet are provided to show the superiority of IPWD.
strengths_and_weaknesses: Strengths:
1. This paper gives an interesting domain transfer perspective of KD, in which the distributions of ground-truth labels and teacher's soft labels are different, this conflict makes KD less effective. Based on this, the authors propose a simple KD loss to balance the teacher knowledge using IPW, which gains significant improvements over the KD baseline.

Weaknesses:
1. The authors should discuss the impact of imbalanced teacher knowledge in conventional KD methods, to show the necessity of balanced knowledge. For example, compare the predictions of under-represented class of student trained with or without KD.
2. In Table 3, IPWD performs worse than WSLD on ResNet-18 student (71.88% vs. 72.04%).
questions: 1. The distribution of teacher's soft labels would be varied a lot according to different training strategies. Can IPWD adapt better to these distribution shifts? For example, some works [1, 2] observed that KD performs poorly when with label smoothing.

I will raise my rating if the authors can well address my concerns.


[1] Müller, R., Kornblith, S. and Hinton, G.E., 2019. **When does label smoothing help?**. Advances in neural information processing systems, 32.  
[2] Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K.T. and Savvides, M., 2020, September. **Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study**. In International Conference on Learning Representations.
limitations: Both limitations and potential negative societal impact were discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new loss function for teacher-student learning by observing the fact that the label predictions from a teacher model has a distribution shift compared with ground-truth labels. It is pointed out that this distribution mismatch can cause a gap during transfer due to over-representation and under-representation of examples, e.g. the existence of context-equivariance and context-invariance information in some image datasets. Specifically, inverse probability weighting is used in the loss function to assign larger weights to the KL divergence loss of under-represented examples. The weight is implemented by comparing the cross-entropies of both classification outputs and knowledge distillation outputs to the ground-truth label. Finally, empirical evaluations are conducted on some image datasets and results show that the proposed method can achieve improved performance over some baselines.
strengths_and_weaknesses: Strengths: The motivation of solving the distribution mismatch problem of the teacher model is reasonable and novel. This observation is of practical importance in knowledge distillation as this problem is expected to occur in many teacher student learning datasets. The use of inverse probability weighting (IPW) for KD in the proposed method seems to be helpful for this problem according to the results of experiments and ablation study.

Weaknesses: Although the aforementioned problem is well clarified, the use of IPW is not well motivated or explained. Many explanations of IPW and its motivation relate to the problem of non-IID in the model dataset. However, my concern is that this does not exactly fit or target the problem of distribution mismatch problem. Please see the questions below for more details. Moreover, the method of IPW has been well-established before. The technical contribution is limited.

questions: Q1: It is claimed several times in the paper that the dataset generated by the teacher model is no longer IID (e.g. L142). I don't understand why this is the case. Once we have a teacher model, the dataset is generated from the distribution induced by this teacher model. Given the parameter $\theta^t$, individual examples $(x,y^t)$ are also independent from each other. As introduced in Section 2, IPW is practically suitable to non-IID observations. The resulting distribution is likely to be long tailed but the examples still seem to be independent, which weakens the justification of the use of IPW. Could the authors please explain more on this?

Q2: L146: "Thanks to the causality-based theory, we can use the IPW technique to overcome the confounding effect brought by the transfer gap". In my opinion, this claim is not well supported by current writing. Which part of the causality-based theory is relevant to the problem described in this paper? In this paragraph only references are mentioned but detailed explanation is missing. For the second half of the sentence, I don't understand what is the confounding effect brought by the transfer gap. How does the confounding effect occur in the dataset of teacher model?

Q3: In Fig.1, the blue dashed line refers to the histogram of different labels. The ImageNet dataset (ILSVRC 2012) has 1,000 classes with varying number of images per class ranging from around 732 to 1300. Its training set is imbalanced. I don't see why in the right subfigure the label distribution is a straight line. How is the average probability on the y-axis calculated for labels or predictions from a teacher model?

Q4: In experimental comparisons, the proposed method is compared with some baselines including CRD and WSLD. I'm not sure whether the reported results are SOTA on knowledge distillation benchmark. Are there any other published baselines?

limitations: See weakness and questions.
review_title: null

Meta-review: metareview: This paper analyzes the way in which most previous knowledge distillation methods violate IID assumptions and it aims to address the drop in performance on student models through this analysis. The paper proposes an Inverse Probability Weighting Distillation (IPWD) technique, derived in part through a causal analysis of the distillation setting. Results are mainly presented for CIFAR-100, but some ImageNet results are given and these result show that the proposed approach does indeed outperform a wide variety of prior work for distillation. The review scores for this paper place it right at the borderline of acceptance, with two weak accepts and one weak reject. 

Given the paper was at the borderline of numerical acceptance and the signals from reviews and subsequent discussions were not conclusive, the Area Chair also read this paper and found the underlying idea to be quite interesting and novel. The application of causal analysis to the problem in this way does a nice job of brining together an important branch of machine learning (causal analysis) with deep learning and knowledge distillation. The AC also judged that the experimental work in this paper was substantial. Given that the method also yields better results than many other prior methods, AC recommends accepting this paper.

confidence: Less certain
Decision: Accept