Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses an interesting concern : how can we transfer knowledge from a large model (master) to a tiny one (student) taking into account possible bias in the learning process of the teacher
The framework considered is that of  image classification, where the context contained is the image can yield some bias in the classification task (for e.g. garden when distinguishing a dog from a cat)
The paper considered two kinds of knowledge transfer
* one may want to transfer representation of each class
* one may also want to transfer the classifier in itself

The approach proposed in this paper to circumvent the bias problem is based on causal  intervention.  The claim is that there is an underlying graph where the output of the classifier may have as parent in the graph, both the true representation of the object to identify and the context
The idea is to perform causal intervention to remove the influence of the context in the classification
Numerical experiments are performed in the image context. The task is object clasiffication and several classical datasets are considered
main_review: * I am quite convinced by the relevance of the problem which is addressed
* The approach in itself : first separation of the two kinds of knowledge that we want to transfer and second use of do calculus to remove influence of the context in classification in the teacher model is smart. I like it
* I wonder how difficult the objective function defined through do calculus is difficult to solve. I have no idea of possible properties of this function. It is probably non convex, I don't know if it is smooth. Can you give some details about that?
* In numerical experiments, the compression rate is significant and the results seems better than other state-of-art approaches. Results are mainly given hen the teacher is a ResNet or a WRN. Other architectures are mentionned but the resulats are not given. Can you give some details?
* What about robustness to noise of the approach? Are the performance of the student similar if the images are noisy or if there is noisy labels? One can think context may help when the images are mislabeled. Is there any drawback of this approach? What do we miss when we remove the impact of context?
 
limitations_and_societal_impact: OK from this point of view
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new knowledge distillation method called CIS that captures both sample and class representations from the teacher while removing the bias with causal intervention. Experiments on several benchmark datasets validate the effectiveness of the proposed method.

main_review: Strengths:
1. This paper is well written and easy to follow.
2. The motivation is clear. I think it is the first attempt to investigate the bias problem in teacher's prediction.
3. Experiments are done on different benchmarks, and comparison covers SOTA methods.

Weakness:
The class representation part is unclear to me. As is described in (2). The class representation dimension is related to the number of images in the current category. So it will inevitably increase the training time. However, this paper did not provide training time in the current version. Besides, the class representation idea is similar to [33]. Although [33] only considers within batch comparison, but it considers the inter-class comparison while this paper only considers the intra-class comparison. The advantage over [33] should be discussed and analysed. 

Post-rebuttal:

Thanks to the authors for addressing my concerns and questions regarding the class representation part. I keep my previous score.

limitations_and_societal_impact: This paper has addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on comprehensive knowledge distillation to transfer the class representation. The proposed method aims to improve the performance of the student model by considering the transferring of class representation. It is a novel view, especially with a causal intervention approach. The experiments are conducted to demonstrate the advantages of the proposed method.
main_review: Considering class representation or category structure into knowledge distillation is not a novel approach to improve the performance of knowledge  distillation. There are some works done with the same idea such as Chen et.al.  But the introducing of causal intervention is novel in knowledge distillation.  The writing and organization of this paper are clear, and the motivation of comprehensive representation distillation is also meaningful and significant.  The experiments and analysis are sufficient. 

Post-rebuttal:

Many thanks to the authors for addressing my concerns. After reading the rebuttal, I have updated my score.
limitations_and_societal_impact: - the motivation of the interventional distillation is unclear. What is the confounder or bias that affected the comprehensive distillation? How to guarantee the prior knowledge K is not the same as the How to ensure that the prior knowledge K in interventional distillation and knowledge in comprehensive representation distillation are not in conflict.
- how to know the prior knowledge K is a confounder of X and Y? it is against our intuition. Which type of prior knowledge?
- in the introduction, the authors claim that "good class representations are beneficial to sample representation learning, since they can shape the sample representation distribution." It is hard to understand. How do the class representations shape the sample representation distribution?
-the idea of considering class representation or category structure into knowledge distillation is not novel. Please compare with the work with Chen. et.al.
Chen et.al., Improving Knowledge Distillation via Category Structure. ECCV 2020.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors have proposed a novel framework (CID) for knowledge distillation that is able to capture sample and class representations and reduce knowledge bias induced by the context prior through causal intervention. There are two major differences between the proposed approach and previous work: first, CID is able to transfer class representations. Second, it uses soften logits as sample context information. Authors have provided extensive experimental results to show the effectiveness of their approach.
main_review: The paper is well-written and easy to follow. The idea is novel although there has been some usage of causal inference for knowledge distillation for the question answering application. The major concern that I have is about the experimental results. Some of the cited papers that are recently published in top venues (e.g.  the SSKD approach: Knowledge distillation meets self-supervision from ECCV 2020) are not included in the tables. In addition, there are discrepancies  between numbers reported here vs. numbers reported in previous works. As an example, the performance of AT is reported 71.78 in the Table 3 of SSKD paper for (ResNet-56, ResNet-20); while it is reported as 71.08±0.34 in the submitted work. Also, there are some (teacher, student) configurations that are missing in this paper compared to the previous works. 

I am more than happy to increase my score if authors can provide a fair explanation about issues in the experimental results.

----------------
Post-rebuttal:

Thanks to authors for addressing my concerns and questions regarding experimental results. I have updated my score.
limitations_and_societal_impact: Authors have adequately discussed limitations and societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an improved knowledge distillation process named Comprehensive, Interventional Distillation (CID). It adds a class representation term to the loss function of a knowledge distillation model to enforce the student and the teacher to learn not only similar sample representations but also class representations, where a class is represented by the similarity matrix among all of its samples. It further modifies the loss function term of the student to remove the bias learned by the teacher from the context information in training data, based on casual intervention. Experimental results on four real datasets confirm the effectiveness of the proposed model in improving the classification accuracy of the student model. 
main_review: This paper studies an interesting problem, i.e.,  to transfer class representations and to remove biases with causal intervention in knowledge distillation. The paper is motivated with a figure (Figure 1) to highlight the importance of removing biases. However, it then makes an assumption that " the training data used by the teacher and those used by the student are from the same distribution". The importance of removing biases under such an assumption needs further justification. 

Removing biases with causal intervention in knowledge distillation is new and interesting. The derivation of Equations (4) - (7) is  solid. The idea of learning and transferring class representations is also interesting but this is mainly a heuristic without performance guarantees. 

The experiments are run on multiple real datasets of different sizes which is also good. 

The paper is well written overall and is an interesting exploration. There are a few issues that need to addressed to further strengthen the paper as detailed in the "Limitations And Societal Impact" section. 

Post-rebuttal:

I appreciate the authors' efforts to address my comments. I am not fully convinced by the response to Q2, as I am still unclear why learning the bias by the teacher is an issue given the assumption that the training data used by the teacher and those used by the student are from the same distribution (i.e., they may share the same bias). Since this is a basic assumption of the paper, I cannot boost my score. 

And update after further author response:

I appreciate the further discussion, which clears the confusion. No further question or objection to acceptance from my end.  

limitations_and_societal_impact: The experimental results are not very strong. The claim "the performances drop significantly without anyone of these terms" for Fig. 6 seems to be an overstatement. The performance drop for w/o CR and w/o INV are just about 0.5%. Similar claims appear in the discussions on the rest of the experimental results such as "outperforms all the baselines significantly", "beats all the SOTA approaches substantially", "outperforms these competitors significantly", while the performance difference between the proposed model CID and the best baseline is less than 1% in many cases. What is the definition of "outperforms significantly/substantially"?

Given the claim "Since the bias knowledge in the teacher is caused by the training data, we assume that the training data used by the teacher and those used by the student are from the same distribution", is the bias induced by the context learned by the teacher still a problem?

Are there concrete examples in the test datasets that cause classification errors because of the context bias learned by the teacher model in the baseline models, which are correctly classified by the proposed CID model? 

What is the implication in terms the extra training introduced by the CID model?

Typo: "but also server as a tool"
review_title: null

Meta-review: The paper is concerned with the distillation / compression of a large teacher model. Besides the transfer of the sample representation, the originality of the approach lies in the transfer of the class representation, and the use of a causal intervention (CID) aimed to prevent the transfer of the teacher's biases to the student (that can indeed be harmful even when the training and test distributions are the same). 

Formally, CID proceeds by aligning $P(outcome | do (input))$ for the teacher and student, using the backdoor adjustment formula to account for the confounders (context of the different classes). 

The merits of the approach are supported by extensive empirical comparisons with the state of the art, considering various architectures and datasets, and the relative importance of the three aspects (transferring the sample representation, transferring the class representation and using interventions to prevent the transfer of the biases) is shown using lesion studies. It is a bit disappointing that the interventional term, that is the most innovative and theoretically grounded one, is found to be the less important one after the lesion studies.
 
The authors did a very good job of answering the reviews, justifying the choice of the considered architectures, examining the impact of noise, clarifying the difference w.r.t. related works (e.g., Improving Knowledge Distillation via Category Structure) and  explaining the discrepancies with the results reported for the baselines in previous papers, depending on the use of pre-trained models. 

In summary, this paper tackles a relevant issue in an innovative way, and it is viewed as above the acceptance threshold by all reviewers.
Decision: Accept (Poster)