Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper reveals and studies a new problem, that KD is posing a potential risk of intruding the intellectual property (IP) of released ML models or their training data. Even trained ML models are released only in “black boxes” (e.g., as executable or APIs, no open-sourcing codes), their functionalities can be largely replicated by KD through imitating input-output behaviors. I find this problem and idea very interesting, and of both social and legal importance.

Pros:
- The self-undermining training method seems to be a variant of adversarial training. It maximizes the K-L divergence between the nasty teacher logits and those produced by another adversarial network. The authors used a similar idea to born-again NNs and self-training KD, to generate adversarial attacks from another pretrained network of the same architecture; thus no extra model is necessary. The training method is conceptually simple but effective.
- The authors conducted many experiments on several datasets to demonstrate that the proposed method is effective on both standard KD and data-free KD. Especially, they show the KD-immunity w.r.t various student models from small to large.

Cons:
- Even the authors presented an ablation study, it remains unclear to me why this self-attack is a good choice? Is convenience the only reason, and might there be more benefits (e.g., are the attacks generated by the same architecture more “specialized”)? 
- “While our nasty teacher targets to degrade the performance of all kinds of students networks through KD”, I would suggest the authors to seriously tone down claims like this (or present substantially more experiments)
- We need more understanding why the nasty teacher works. Looking at Figure 2, I cannot get any useful information why the nasty features are more detrimental to distillation, and why this is not reflected on t-SNE? 
review_title: Interesting idea

final_score: rating: 7: Good paper, accept
review_content: This paper’s main idea is refreshing and attractive: proposing a defensive method called nasty teacher, to avoid knowledge leaking or cloning through KD. A nasty teacher model is a specially trained network that yields nearly the same performance itself, while significantly degrading the performance of student models learned by imitating it. The standing point for machine learning IP protection is novel and hasn’t seen many discussions before. The introduction section motivated the study clearly and nicely. The method and experiments demonstrate a promising first step towards protecting machine learning IPs. 

The writeup has a smooth logic but demands much more serious proofreading. Specifically, my initial expectation after reading the abstract and introduction parts was quite high (those seem to have been well revised and mature). However, starting from Section 2 the readability drops to a rather poor level. There are many typos that hamper the paper:
-	Sec 1, “maximizing difference between nasty teacher” – missing “the”. Section 3.1, “a αclose to 1” – “a: should be “an”. The authors need carefully check their usage of the articles in general: “a” versus “an”, missing “a/an/the”, or largely confusing them.
-	Sec 2, “Teacher-free KD framework (Yun et al., 2020) train - “train” should be “trains”. There are a few more missing “s”/”es”
-	Section 3.1, “The higher α is” – “higher” should be “larger”. The authors seem to have confused “high””large””big” and abused them randomly throughout the paper, etc. 
-	Section 3.2, “we hope to build a kind of logit whose argmax is still the correct answers” – this sentence is grammatically wrong from end to end …
-	“the distribution of incorrect answers varies as much as possible to the real distribution over the dataset” – I guess that is in contrary to what you wanted to say. “varies as much as…” means “correlates well with”, but your method should expect them to differ from each other.
… and I could have placed a longer list. I urge the authors to maintain the same high standard of revision beyond the point of Section 1.

Another issue is while the authors stressed their training algorithm called self-undermining KD in the abstract and introduction, that algorithm is only very briefly introduced in the last paragraph of section 3.2 -- which I nearly missed. I strongly suggest the authors to use an algorithm table to make your main algorithm clearer and more visible. 

Besides, the text in Section 3.2 also needs to be considerably re-organized and revised; currently it is hard to figure out the right focus and lacks clarity. For example, I cannot see whether the adversarial network is also updated during training.

From the draft, it is also unclear whether the authors will release their codes and models for reproducibility

Overall, this paper is technically novel and interesting, and can potentially generate a positive impact. But the draft quality is currently unsatisfactory. Careful proofreading and thorough revision are requested from the authors; otherwise my final recommendation cannot be positive. 

review_title: Weak Accept

final_score: rating: 7: Good paper, accept
review_content: Summary: 
This paper explores an interesting and novel research problem: how to make a teacher model undistillable. This can be a promising countermeasure to model extraction/stealing. The proposed Nasty Teacher approach is a two stage method, which first trains a good teacher network, then utilizes a self-undermining KD strategy to further distill the good teacher network to a bad one. Overall, this  Nasty Teacher approach can reduce the performance of the student network by ~5% in most cases. Although the performance decrease is not huge, the proposed approach is promising and can be a very useful baseline for this new research direction. A set of ablation and understanding experiments have also been conducted to support the effectiveness of  Nasty Teacher. 

Strong points:
1. A new security scenario, and a novel defense approach against model stealing.
2. The proposed idea is well motivated and described.
3. The effectiveness of the proposed approach is substantiated by thorough experiments.

Weak points:
1. Shown in Fig.3 and Fig.4, the performance decrease of the student network is not very significant, and is slightly sensitive to the parameters. By finding the right parameter, the student may still achieve ok performance.  What would happen if the student uses temperature \tau < 4 in Fig.4 (a)?
2. In Eq. (1), isn’t the second P_T should be P_S? The cross entropy is defined on p?, but I thought p are the logits? Why no \sigma in the XE? 
3. Eq. (1) indicates the distillation needs logits. It is ok to train the nasty teacher, but how about the student network? In the intro the authors argued that protecting “backbox” APIs is one of the main motivations of the proposed method, but this was not tested in the experiments. What would happen if the student is distilled with probabilities (not logits)? I believe the effectiveness will be even better if the student can only access probabilities. 
4. It seems like the distillation experiments were all done on 100% training data? This setting is somewhat less suitable from model stealing since the attacker may not have 100% training data. What would happen if the attacker has only 10% of training data? In Table 5: Data-free KD, I think a performance decrease of ~6% may still be a successful stealing, considering the huge amount of money and effort it takes to train the teacher network. 
6. How well is the proposed method compared to other defense methods against model stealing? Prior works in this field should also be reviewed, if there are any.

review_title: Novel research direction, interesting method, some counter model stealing experiments will be helpful

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a new method that prevent the teacher model from being learned by other models. The paper  introduces and investigates a concept called Nasty Teacher: a specially trained teacher network that yields nearly the same performance as a normal one, but would significantly degrade the performance of student models learned by imitating it.

Pros:

- This is overall a meaningful direction of knowledge distillation, that makes a neural network 'undistillable'.

- The method is simple yet effective by disturbing the output distribution of the teacher network.

- The experimental results show that the proposed method can succesfully prevent student networks from learning knowledge from teacher network both with original KD method and with data-free KD method.

Cons:

- Since this is a paper that focus on the real-world application problem, I would wonder whether the results are acceptable on ImageNet dataset. I notice that a 2% degrad appears on ResNet-50 on Tiny-ImageNet, which is almost unacceptable on real-world applications. I believe that there will be a larger gap between nasty teacher and original teacher on ImageNet in order to achieve the same undistillable results.

- The paper only discuss the influence of the final layer. What will happen if the student network learns from the feature maps deriving from the middle layers of teacher network? It seems that the loss function can only disturb the final output distribution, but we can still distill knowledge from the other layers. Experiments should be conducted for further explaination.

- It seems that the \alpha in Eq.(1) will also influence the student performance when learning from nasty teacher. The paper only shows that a large \alpha (=0.9 in the paper) will degrad the student performance. However, in practice the weight is always set to multiple values and the best among them is picked. I think the author should make an ablation study on \alpha.

review_title: This is overall an ok submission.

Meta-review: The paper argued some viewpoint about knowledge distillation quite interesting to me: the technically good KD might surprisingly be socially bad in helping outsiders "stealing" commercial models, even if the models are released as black boxes. Then the paper proposed a way called self-undermining KD in order to turn a well trained model into a "nasty teacher" (i.e., an undistillable model), and by this way the commercial models and the corresponding intellectual properties for training them from insiders can be nicely protected.

Overall, the quality is quite high. The argument is very conceptually novel and the method is still technically novel. The idea of the method is simple but works for the purpose --- that's great! Although the experimental significance seems not too impressive, the paper opens a door to a new world concerning model privacy instead of data privacy, and hence it is of social significance. In my opinion, the paper should have a potentially huge social impact to DL practitioners (and company owners), because KD is being used almost everywhere in the Internet industry to provide the standalone mode of Apps without clouds on personal devices. Based on the quality and the impact, I recommend to accept the paper as a spotlight presentation.
Decision: Accept (Spotlight)