Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 1) The main contribution of this paper is a new knowledge distillation (KD) framework called "skeptical student" that can circumvent the "immunity (undistillability)" provided by a "nasty (defensive/secretive)" teacher. 

The concept of "nasty" teacher was proposed in [A] to prevent stealing of a teacher model by a malicious student. This paper proposes a countermeasure that borrows ideas from self-distillation, where knowledge is transferred to shallow subsections of the student model, each with its own auxiliary classifier (AC). The proposed skeptical student performs knowledge distillation both to the shallow sub-sections and to the full student model to overcome the impact of a nasty teacher.

[A] Ma et al., "Undistillable: Making a nasty teacher that cannot teach students", ICLR 2021

2) The paper also includes an algorithm for training skeptical students in a data-free KD scenario (the original data used to train the teacher model is not available). 

3) Experimental results on three datasets indicate that the proposed skeptical student can learn well both from the normal and nasty teachers. 
main_review: 1) The paper is moderately original. It uses existing concepts such as self-distillation and zero-shot knowledge transfer in a clever way to achieve the goal of training skeptical students that can learn even from defensive teachers.

2) The main problem with the paper is its motivation. Under what real-world circumstances are defensive teachers necessary, especially when the training data is assumed to be known? Even in the data-free scenario, an attacker who wishes to steal IP information of the target teacher model (including data used to train the model) can carry out direct model extraction attacks, rather than relying on the knowledge distillation framework. A simple and straightforward approach to mitigate such attacks is to add random noise to the softmax values (if they ever need to be released), rather than train a defensive teacher. If the need for defensive teachers itself is not strong enough, the motivation for skeptical students is very weak.

3) The key observation of the paper is that the proposed skeptical student can learn well from both normal and defensive teacher models. If that is the case, why not treat the skeptical student as a new knowledge distillation framework and analyze its properties more comprehensively purely from the knowledge distillation perspective (e.g., theoretical analysis on why this approach is better and/or experiments on more datasets to determine its strengths and limitations).

4) As explained earlier, if there is likely to be any use case for defensive teachers and skeptical students, it will be only under the data-free KD scenario. However, this scenario is treated very casually both in terms of explanation and experimental results. For example, the paper claims to include attention transfer loss, which is not at all possible unless one has white/gray-box access to the target teacher model. If someone has white/gray-box access to the teacher model, why would he/she attempt to steal the IP that is already available to him/her? Even with the attention transfer loss and access to 20% of the original training data, the proposed skeptical student performs only marginally better than the regular/normal student under the data-free scenario. This further illustrates the futility of this whole approach.

5) If my understanding of Figure 2 and Section 3.2 is correct, an auxiliary classifier trained with only level 1 features (shallow sub-section) of the student network works as well as the teacher model. In that case, why not simply go for a shallow student network with only level 1 features? In general, what is the impact of the depth of the student model on the proposed approach. In traditional knowledge distillation, one usually goes for shallower/simpler student architectures compared to the teacher. But if the goal is to steal the IP of the teacher, will going for shallower or deeper student networks help in any way?

6) The overall organization of the paper is very confusing and does not flow well. For example, what is the need for Section 3.1 (transferability impact of nasty teachers) if it is not used anywhere else in the paper.

limitations_and_societal_impact: The paper can be enhanced by including strong motivating examples to justify the need for this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Knowledge distillation (KD) can be used by an adversary to obtain sensitive information from a teacher model presenting privacy risks. Nasty teachers were proposed to handle this problem where students were unable to learn or distill knowledge from nasty teachers. In this paper, the authors show that skeptical students using a combination of KD to intermediate shallow subsections of their model together with Self Distillation (SD) can extract information from a nasty teacher and perform well on classification tasks.
main_review: The paper provides an attack against the nasty teacher-based defense proposed in prior work. 

Some of my concerns about the work are as follows:
a. The authors motivate the idea using knowledge leakage and then introduce nasty teacher as a proposed defense. However, the paper does not provide any explicit threat model corresponding to the defense. It is important to assess if the adversary is able to extract the information while adhering to the constraints of the threat model. 

b. Along similar lines, the metric used for showing the effectiveness of the attack is the classification accuracy. However, it is not clear what the exact leakage is due to the distillation. There is a need to decouple the two and show that indeed some sensitive information is being leaked. 

c. In Table 2, when a normal student is used, the drop in accuracy from the baseline is not very significant (other than when MobileNetV2 model is used for CIFAR-100 and Tiny-ImageNet). This indicates two problems: First, the baseline protection mechanism using a nasty teacher is itself not very strong. A normal student is able to still distill most of the information from the nasty teacher. Second, for models where we observe a drop in accuracy, I think there is a need to isolate the reason - is it due to the nasty teacher or the model architecture being used? The authors report improved performance with skeptical students, but from a privacy standpoint, I am not convinced this really means much. The system seems to be already leaking a lot of information. This further goes back to the point made in (b) above.

d. I really like the use of self-distillation in the data-free distillation formulation. However, the improvement in test accuracy is not very significant as reported in Fig. 8. I agree that the gap is a little larger when only 20% of training data are available. Maybe, representing in tabular form will help bring out the improvement.

Writing Issues:
The paper is well-written. Some minor issues are pointed below:
Line 126-127: The sentence, "One the other hand.." is unclear. Moreover, this form of sentence structure is used to contrast with another observation. There is no such contrasting observation presented.

Line 148: The notation used in Eqn (2) is confusing. How is \Phi_{S}^{j} and  \Phi_{S}^{'} related?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors investigate the knowledge distillation immunity of Nasty teachers. A nasty teacher is a model that maintains good prediction performances but significantly degrades the performances of student models that distill knowledge from it. In this paper, the authors proposed a hybrid distillation that neutralizes the protection of nasty teachers, allowing the training of accurate student models.
main_review: Overall, the paper is addressing an important problem and the experiments conducted by the authors demonstrated that the proposed approach can effectively bypass the immunity of nasty teachers. 

My main concern with the paper is that the ability to steal an IP from the nasty teacher is solely evaluated with the accuracy. However, accuracy can be misleading due to predictive multiplicity. That is, both teacher and student models can exhibit high accuracy while having different properties. So it would be interesting to report more details on the difference between students distilled from normal teachers and those distilled from nasty teachers. 

Minor remark:
The word "privacy" in the title can be misleading as it can give the impression that the paper is about the robustness of nasty teachers against privacy inference attacks, which is not the case. 

Section 2 has some readability issues and will benefit from a proofreading.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a skeptical student distillation framework that diminishes the effect of a nasty teacher. By placing auxiliary classifiers at different depths of the student model to form a shallow subsection and introducing a new self-distillation term, the resulting skeptical student clearly outperforms a vanilla student in all data-available, data-limited, and data-free settings when learning from a nasty teacher. The proposed KD method (especially the ensembled outcome) also provides similar performance as a vanilla student when learning from a normal teacher, thus being applicable in most cases. Quantitative analysis and visualizations are provided to show that the method works as expected. 

main_review: Originality: how to effectively use KD to learn from a protected nasty teacher model is an interesting new question. The self-distillation technique is not new but is applied in the new setting. Related works are well cited.

Quality: The submission is technically sound and most claims are supported by empirical results. The work is complete and the comments for the evaluation experiments are fair.

Clarity: The paper is clearly written and should be detailed enough for reproducing the result. However, some major prior works (e.g. self-distillation network structure, details about using auxiliary classifier) are not clearly discussed and more details on those preliminaries can help readers to better understand the topic without jumping through references.

Significance: The paper discusses an interesting technique that to some extent compromises an existing model IP protection mechanism. There might be follow-up works on how to defend against such skeptical student model stealers. As also mentioned by the authors, the paper might motivate further investigations on model privacy protection.

limitations_and_societal_impact: Yes, the author clearly addressed them in the paper.
review_title: null

Meta-review: Previous work defined a *nasty teacher* to be a specially trained network that yields nearly the same performance as a normal one; but if used as a teacher model, it will significantly degrade the performance of student models that try to imitate it (with the goal of protecting against model stealers). In this paper, the authors proposed an attack that neutralizes the protection of nasty teachers, allowing the training of accurate student models. The reviewers agree that this is an interesting paper, all of them supporting acceptance.
Decision: Accept (Poster)