Undistillable: Making A Nasty Teacher That CANNOT teach students
Authors: Haoyu Ma
Tianlong Chen
Ting-Kuei Hu
Chenyu You
Xiaohui Xie
Zhangyang Wang
Venue: ICLR 2021 Spotlight (2021)
Keywords: knowledge distillation
avoid knowledge leaking

Abstract: Knowledge Distillation (KD) is a widely used technique to transfer knowledge from pre-trained teacher models to  (usually more lightweight) student models. However, in certain situations, this technique is more of a curse than a blessing. For instance, KD poses a potential risk of exposing intellectual properties (IPs): even if a trained machine learning model is released in ``black boxes'' (e.g., as executable software or APIs without open-sourcing code), it can still be replicated by KD through imitating input-output behaviors. To prevent this unwanted effect of KD, this paper introduces and investigates a concept called $\textit{Nasty Teacher}$: a specially trained teacher network that yields nearly the same performance as a normal one, but would significantly degrade the performance of student models learned by imitating it. We propose a simple yet effective algorithm to build the nasty teacher, called $\textit{self-undermining knowledge distillation}$. Specifically, we aim to maximize the difference between the output of the nasty teacher and a normal pre-trained network. Extensive experiments on several datasets demonstrate that our method is effective on both standard KD and data-free KD, providing the desirable KD-immunity to model owners for the first time. We hope our preliminary study can draw more awareness and interest in this new practical problem of both social and legal importance. Our codes and pre-trained models can be found at: $\url{https://github.com/VITA-Group/Nasty-Teacher}$.

Published as a conference paper at ICLR 2021
UNDISTILLABLE: MAKING A NASTY TEACHER THAT CANNOT TEACH STUDENTS
Haoyu Ma', Tianlong Chen?, Ting-Kuei Hu', Chenyu You', Xiaohui Xie', Zhangyang Wang' UUiierrsity of Califormia, Irvine. Universityy of Texas at Austin,
Teexs A&M University, $ *Yale University [hacyum3, x bociieduu (tianlong. chen, atlasvangloutexsseetb t khu@tamu. edu, chenyu, chenyu.you@yalle you@yale. edu
ABSTRACT
Knowledge Distillation (KD) is widely used technique transfer knowledge from pre-trained teacher models to (usually more lightweight) student models. However, in certain situations, this technique is more of a curse than a blessing. For instance, KD poses potential risk of exposing intellectual properties (IPs): even if a trained machine learning model is released in "black boxes" (e.g.. as exe- cutable software or APIs without ppen-iour code), it can still be replicated by KD through imitating input-output behaviors. To prevent this unwanted effect of KD, this paper introduces and investigates a concept called Navty Teacher: spe- cially trained teacher network that yields nearly the same performance as a normal one, but would significantly degrade the performance of student models learned by imitating it. We propose a simple yet effective algorithm to build the nasty aim
teacher, called self-undermining knowledge distillation. Specifically, maximize the difference between the output of the nasty teacher and a normal pre- trained network. Extensive experiments on several datasets demonstrate that our method is effective on both standard KD and data-free KD, providing the desir- able KD-immunity to model owners for the first time. We hope our preliminary study can draw more awareness and interest in this new practical problem of both social and legal importance. Our codes and pre-trained models can be found at https: / / thub. com Hatm//VTAA-rrour
Teacher
INTRODUCTION
Knowledge Distillation (KD) (Hinton et al.] 2015) aims to transfer useful knowledge from teacher neural network to student network by imitating the input-output behaviors. The student model im- itates logit outputs or activation maps from the teacher by optimizing a distillation loss. The efficacy of leveraging teacher knowledge to boost student performance has been justified in many applica- tion fields (Wang et al., 2019 Liu et al. 2019 Chen et al.) 2020b a). yielding high performance and often weghth student models. Typically KD requires learning the student model over the same training set used to train the teacher. However, recent studies (Lopes et al. [017] Chen et al.) 2019) have demonstrated the feasibility of the data-free knowledge distillation, in which the knowl- edge is transferred from the teacher to the student without accessing the same training data. This is possible because the training data are implicitly encoded in the trained weights of deep neural nets. The data-free knowledge distillation is able to inversely decode and re-synthesize the training data from the weights, and then clone the onput-output mapping of the teacher network.
With many practical benefits brought by the KD technique, this paper looks at KD's unwanted severe side effect, that it might pose risks to the machine learning intellectual property (IP) protection. Many machine learning models will be only released as executable software or APIs, without open- sourcing model configuration files or codes, e.g., as "black boxes". That can be due to multifold reasons, such (1) those advanced models might take huge efforts and resources for the model owners to develop. who would need to keep this technical barrier; or (2) those trained models have involved protected training data other information, that are legally or ethically prohibited to be openly shared. However, KD techniques might open a loophole to unauthorized infringers to clone the IP model's functionality, by simply imitating the black box's input and output behaviors (leaked

---

Published as a conference paper at ICLR 2021
knowledge). The feasibility of data-free KD (Chen et al. 2019 Yin et al. 2020) eliminates the necessity of accessing original training data, therefore making this cloning more practically feasible. Even worse, those techniques point reverse-engineering ways (Yin et al. 2020) to recover the (potentially private) training data from black-box models, threatening the owners' data privacy and security (Yonetani et al.; 2017 Wu et et. 2018).
To alleviate the issue, this paper introduces a defensive approach for model owners, called Nasty Teacher. A nasty teacher is a specially trained network that yields nearly the same performance as normal one; but if used as a teacher model, it will significantly degrade the performance of student models that try to imitate attack on deep learning systems (Chen et al. (017)). which creates model to fit "adversarial" goals an "imperceptible" way. However, while backdoor attacks aim to manipulate or damage the performance of the poisoned model itself when triggered by specific inputs, the goal of the nasty teacher is to undermine the performance of any student network derived from it. The primary objective of constructing nasty teacher is for model protection a novel motivation and setting that have not been explored before. Our contributions are summarized as follows:
In general, the concept of nasty teacher related to the backdoor
We introduce the novel concept of Nasty Teacher, a defensive approach to prevent knowl- edge leaking and unauthorized model cloning through KD without sacrificing performance. We consider a promising first step towards machine learning IP and privacy protection.
We propose simple yet efficient algorithm, called self-undermining knowledge distilla- tion, to directly build a nasty teacher through self-training. requiring no additional dataset nor auxiliary network. Specifically, the model is optimized by maximizing the difference between the nasty teacher (the desired one) and a normally trained counterpart.
We conduct extensive experiments on both standard KD and data-free KD approaches, and demonstrate that nasty teacher trained by eeff-nndermining KD can achieve nearly the same accuracy their original counterpart (less than 1 1% accuracy gap), while the student model learned from will degrade accuracy by up to over 10% or even diverge during training.
2 RELATED WORK
2.1 KNOWLEDGE DISTILLATION
Knowledge distillation aims to boost the performance of light-weight models (students) under the guidance of well-trained complicated networks (teachers). is firstly introduced in (Hinton et al.) 2013). where the student directly mimics the soft probabilities output produced by the well pre- trained teacher. The following researchers explore the knowledge transferal from either interme- diate features (Romero et al. 2014; Zagoruyko & Komodakis 2016 Passalis & Tefas 2018 Ahn al. 2019 Li al. (2020), or logit responses (Park al., [2019] Mirzadeh et al. [2019] Chen al.
2021a by Ma et al. (221). Recent studies have also shown that, instead of distilling from a compli-
cated teacher, the student networks can even be boosted by learning from its own pre-trained version
(Furlanello et al. 2018 Zhang et al. 2019 Yun et al. 2020: Yuan et al.) 2020).
Several recent works also focus on data-free knowledge distillation, under which settings students are not able to access the data used to train teachers. In (Lopes et al.) (017)), the author attempts to reconstruct input data by exploring encoded meta-data lying in the pre-trained teacher network. In the following work, the author of (Chen et al., 2019) proposes a learning scheme, called "Data-Free Learning" (DAFL), which treats the teacher as a fixed discriminator, and jointly trains a generator to synthesize training examples so that maximum responses could be obtained the discriminator. The latest work "DeepInversion" (Yin et al., 2020) directly synthesizes input images given random noise by "inverting" a trained network. Specifically, their method optimizes the input random noise into figh-fi images with a fixed pre-trained network (teacher).
2.2 POISONING ATTACK ON NEURAL NETWORK
The typical goal of poisoning attack is to degrade the accuracy of models by injecting poisoned data into training set (Xiao et al., 2015 Moosavi- Dezfovi-Dezfooli et al. 2016). On the contrary, backdoor attack intends to open a loophole (usually unperceived) to the model via inserting well-crafted malicious data into training set (Chen et al. 2017
Gu al. 2017 Kurita al. 2020). The goal of back-
2

---

Published as a conference paper at ICLR 2021
door attack is to make the poisoned model perform normally well for most of the time, yet failing specifically when tttacker-triggeeee signals are given.
Our proposed seff-nndermining knowledge distillation aims to create special teacher model (i.e., an undistillable model), which normally performs by itself but "triggers to fail" only when being mimiced through KD. The motivation looks similar to backdoor attack's s at the first glance, but differs in the following aspects. Firstly, the backdoor attack can only be triggered by pre-defined patterns, while our nasty teachers target at degrading any arbitrary student network through KD. Secondly, backdoor attack tends to poison the model itself, while our nasty teacher aims to undermine other student networks while preservingits own performance. Thirdly, our goal is to prevent knowledge leaking in order to protect released IP, as defensive point of view, while the backdoor attack tends to break down the system by triggering attacking signals, as an attacking point of view.
2.3 PROTECTION OF MODEL IP
Due to the commercial value, IP protection for deep networks has drawn increasing interests from both academia and industry. Previous methods usually rely watermark-basee (Uchida et al. 2017 Zhang et al. 2020a) or passport-based (Fan et al. 2019 Zhang et al. (0200) ownership verification methods to protect the IP. Nevertheless, these methods can only detect IP infringement but remain ineffective to avoid model cloning.
A few recent works also explore defensive methods against model stealing (Kariyappa & Qureshi 2020 Juuti et al., [219 Orekondy et al.] 2020). Typically. they assume attackers obtain pseudo- labels on their own synthetic data or surrogate data by querying the black-box model, and train a network on the new dataset to clone the model. However, none of these defense approaches have explored the KD-based model stealing, which is rather a practical threat.
3 METHODOLOGGY
3.1 REVISITING KNOWLEDGE DISTILLATION
Knowledge distillation (Hinton et al. 2013) helps the training process of "student" networks by dis- tilling knowledge from one or multiple well-trained "teacher" networks. The key idea to leverage soft probabilities output of teacher networks, of which incorrect-class assignments reveal the way how teacher networks generalize from previous training. By mimicking probabilities output, student networks are able to imbibe the knowledge that teacher networks have discovered before, and the performance of student networks is usually better than those being trained with labels only. In what follows, we formally formulate the learning process of knowledge distillation.
Given pre-trained teacher network for (-) and student network foe (.), where of and 0g denote the network parameters, the goal of knowledge distillation is to force the output probabilities of foe (-) to be close to that of fo,, (-). Let (æz, ys) denote a training sample in dataset X' and pf, (x;) indicate the logit response of x; from fa(-). the student network Jeg could be learned by the following:
min Ag
Σ
aT₃ (p (2:)), o+, (P10g (a:))) + (1 a)XE((((P))))))))))) (x))).
(1)
where -C((-)) and X are Kullback-Leibler divergence (K-L divergence) and cross-entroon loss. respectively. The introduced "softmax temperature" function o,, (-) (Hinton et al. 2015) pro- duces soft probabilities output when large temperature т. (usually greater than 1) is picked, and it decays to normal softmax function af.) when equals 1. Another
a is also
introduced to balance between knowledge distillation and cost minimization.
3.2 TRAINING NASTY TEACHERS: RATIONALE AND IMPLEMENTATION
Rationale The goal of nasty teacher training endeavors to create a special teacher network, of which performance is nearly the same its normal counterpart, that any arbitrary student networks cannot distill knowledge from it. To this end, we propose a simple yet effective algorithm, dubbed Self-Undermining Knowledge Distillation, while maintaining its correct class assignments, maxi- mally disturbing its in-correct class assignments so that no beneficial information could be distilled from it, as described next.
3

---

Published as a conference paper at ICLR 2021
Let (+) and fea (-) denote the desired nasty teacher and its adversarial learning counterpart, the uelf-nnnerni training aims to maximize the K-L divergence between the adversarial network and the nasty teacher one, that a false sense of generalization could be output from the nasty teacher. The learning process of the nasty teacher could be formulated as follows,
min Or
Σ XE(o(PIo, (4:)), v.) WTC (P (z+)), T. Tra (P1s (ax))), (ES
where the former term aims maintain the accuracy of the nasty teacher by minimizing the cross en- tropy loss, and the latter term achieves the "undistillability" by maximizing KL divergence between the nasty teacher and the adversarial one. Similarly in equation D TA denotes the temperature for self self-undermining, and w balances the behavior between normal training and adversarial learning.
Implementation We naturally choose the same network architecture for (or(.) and fax(t) (yet different group of network parameters) since no additional assumption network architecture is made here. We provide a throughout study with respect to the selection of architectures in 4.3 revealing how the architecture of fo (-) influences the adversarial training. As for the update rule, the parameter of fea (+) is typically normally pre-trained advance and fixed during adversarial training, and only the parameter of fez (-) is updated. Note that the selected temperature TA does not to be necessary the same as Ts in equation[] | and we provide a comprehensive study with respect to Ta in 4.3 Once upon finishing adversarial training, the nasty teacher for (-) could be released within defense of KD-based model stealing.
4
EXPERIMENTS
4.1 NASTY TEACHER ON STANDARD KNOWLEDGE DISTILLATION
To evaluate the effectiveness of our nasty teachers, we firstly execute self-undermining training to create nasty teachers based on equation 2 Then, given an arbitrary student network, we evaluate the performance of nasty teachers by carrying out knowledge distillation from equation I to recognize how much nasty teachers go against KD>based model stealing.
4.1.1 EXPERIMENTAL SETUP
Network. We explore the effectiveness of our nasty teachers on three representative datasets, i.e., CIFAR-10, C0F,R-100, and Liny-ImageNet Firstly, we consider ResNet-18 18 (teacher network) and 5-layer plain CNN (student network) as our baseline experiment in CIFAR- 10, and replace the stu- dent network with two simplified ResNets designed for CIFAR-10 10 (He et al. 2016), i.e., ResNetC-20 and ResNetC-32,, to explore the degree of impact with respect to the capacity of student networks. For both CIFAR-100 and Tiny- ImaaeNma,,,,, we follow the similar setting in Yuan et al. 2020), where three networks from ResNet family, i.e., ResNet-18, ResNet-50 and ResNeXt-29, 29, are considered as teacher networks, and three widely used light-weigh, networks, i.e., MobileNetV2 (annderr al., 2018). ShuffleNetV2 (Ma et al.] 2018)) and R8sNet-18, are served as student networks. Following the "self-KD" setting (Yuan al. 2020), an additional comparative experiment, dubbed "Teacher Self", is provided, where the architectures of the student and the teacher are set to be identical.
Training. The distilling temperature TA for self-undermining training is set to for CIFAR-10 and 20 for both CIFAR-100 and Tiny-ImageNe as suggested in (Yuan et al.) (020)). For the selection of w, 0.004, 0.005, and 0.01 are picked for CIFAR-IO, 10, CIFAR-IO0, and Tiny-ImageNet, respectively. For the plain CNN, we train it with a learning rate of 1e-3 for 100 epochs and optimize it by Adam optimizer (Kingma & Ba 2014). Other networks are optimized by SGD optimizer with momentum 0.9 and weight decay 5e-4 4. The learning rate is initialized as 0.1. Networks are trained by 160 epochs with learning rate decayed by a factor of 10 at the 80th and 120th epoch for CIFAR-10, and 200 epochs with learning rate decayed by a factor of at the 60th, 120th and 160th epoch for CIFAR- 100 and Tiny- ImageNet. Without specifically mentioned, the temperature т. the same as TA. which is used for self-undermining training.
4.1.2 EXPERIMENTAL RESULTS
Our experimental results on CIFAR-10, CIFAR-100, and Iiny-ImageNet are presented in Table Table 2 and Table respectively. Firstly, we observe that all nasty teachers still perform similarly
4

---

Published as a conference paper at ICLR 2021
as their normal counterparts by at most ~ 2% accuracy drop. Secondly, in contrast to normally trained teacher networks, from which the accuracy of student networks could be boosted by at most 4% by distilling, no student network can benefit from distilling the knowledge of nasty teachers. It indicates that our nasty teachers could successfully provide a false sense of generalization to student networks, resulting in decreases of accuracy by 1.72% 67.57%. We also notice that weak student networks (e.g. MobilenetV2) could be much more poisoned from distilling toxic knowledge than stronger networks (e.g., ResNet-18), since light-weight networks intend to rely more on the guidance from teachers. It terms out that KD-based model stealing is no longer practical if the released model is "nasty" in this sense. Additional experiments, dubbed "Teacher Self", are also provided here. Opposite to the conclusion drew in (Yuan et al., 2020), the student network still cannot be profited from the teacher network even their architectures are exactly the same. The aforementioned experiments have justified the efficacy of the proposed self-undermining training.
Table 1: Experimental results CIFAR-10.
Teacher network
Teacher
Students performance after KD
performance
CNN
ResNeIC-20
ResNeiC-32
ResNet-18
Student baseline
86.64
92.28
93.04
95.13
ResNet-18 (normal)
95.13
87.75 (+1.11)
92.49 (+0.21)
93.31 (+0.27)
95.39 (+0.26)
ResNet-18 (nasty)
94.56 (-0.57)
82,46 (-4.18)
88.01 (-4.27)
89.69 (-3.35)
93.41 (-1.72)
Table 2: Experimental results on CIFAR-100.
Teacher
Teacher
Students performance after KD
network
performance
Shufflenet22
MobilenetV2
ResNet-18
Teacher Self
Student baseline
71.17
69.12
77.44
ResNet-18 (normal)
77.44
74.24 (43.07)
73.11 (43.99)
79.03 (+1.59)
79.03 (+1.59)
ResNet-18 (nasty)
77.42(-0.02)
64.49 (-6.68)
3.45 (-65.67)
74.81 (-2.63)
74.81 (-2.63)
Re (normal)
78.12
74.00 (+2.83)
72.81 (+3.69)
79.65 (+2.21)
80.02 (+1.96)
ResNet-50 (nasty)
77.14 (-0.98)
63.16 (-8.01)
3.36 (-65.76)
71.94 (-5.50)
75.03 (-3.09)
ResNeXt-29 (normal)
81.85
74.50 (43.33)
72.43 (43.31)
80.84 (+3.40)
83.53 (+1.68)
ResNeXt-29 (nasty)
80.26(-1.59)
58.99 (-12.18)
1.55 (-67.57)
68.52 (-8.92)
75.08 (-6.77)
Table 3: Experimental results on Tiny-ImageNet
Teacher network
Teacher
Students performance after KD)
performance
Shufflenetv2
MobilenetV2
ResNet-18
Teacher Self
Student baseline
55.74
51.72
58.73
ResNet-18 (normal)
58.73
58.09 (+2.35)
55.99 (+4.27)
61.45 (+2.72)
61.45 (+2.72)
ResNet-18 (nasty)
57.77 (-0.96)
23.16 (-32.58)
1.82 (-49.90)
44.73 (-14.00)
44.73 (-14.00)
ResNet-50 (normal)
62.01
58.01 (+2.27)
54.18 (+2.46)
62.01 (+3.28)
63.91 (+1.90)
ResNet-50 (nasty)
60.06 (-1.95)
41.84 (-13.90)
1.41 (-50.31)
48.24 (-10.49)
51.27 (-10.74)
ResNeXt-29 (normal)
62.81
57.87 (+2.13)
54.34 (+2.62)
62.38 (43.65)
64.22 (+1.41)
ResNeX129 (nasty)
60.21 (-2.60)
42.73 (-13.01)
1.09 (-50.63)
54.53 (-4.20)
59.54 (-3,27)
4.2
QUALITATIVE ANALYSIS
We present visualizations of our nasty ResNet-18 and the normal ResNet-18 on CIFAR-10 to qual- itatively analyze the behavior of nasty teachers. Figure [ visualizes the logit responses of normal ResNet- 18 and its nasty counterpart after "softmax temperature" function. We notice that the logit response of nasty ResNet-18 consists of multiple peaks, where normal ResNet-18 consistently out- puts single peak. We observe that the class-wise multi-peak responses might be un-correlated
5

---

Published as a conference paper at ICLR 2021
to the sense of generalization. For instance, the class output of bird and dog might both spond actively, and it will give false sense of generalization to student networks. We hypothesize that the multi-peak logits misleads the learning from the knowledge distillation and degrades the performance of students.
We also present the visualizations of t-Distributed Stochastic Neighbor Embedding (t-SNE) for both feature embeddings and output logits, as illustrated in Figure 2 It is observed that the feature-space inter-class distance of nasty ResNet-18 and the normal ResNet-18 behaves similarly, which aligns with our goal that nasty teachers should perform similarly to their normal counterparts. Meanwhile, we also observe that the logit response nasty ResNet-18 18 has been heavily shifted, and it entails that our method mainly modifies the weights in the final fully-connected layer.
Figure 1:
The visualization of logit responses after "temperature softmax" function. Each row represents two examples from CIFAR-10. The sampled images are shown in the 1st and the 4th columns. The 2nd and the 5th columns summarize the scaled output from the normal teacher. The 3rd and the 6th columns represent the scaled output from nasty teacher
Normel ResNet18
Nasly ResNe:118
automobile
cat
deer
frog
ship
ISNE of feature embeddings before fully-comnected layer. The dimension of feature embeddings is 512.
Normel ResNet18
Nasty ResNe:18
airolane
automebile
cat
dog
frog
norsa
ship
truc
(b) ISNE of output logits
Figure 2:
Visualization of (SNEs for both normal and nasty ResNet18 CIFAR- 10. Each dot represents
data point.

---

Published as a conference paper at ICLR 2021
4.3 ABLATION STUDY
Adversarial Network. Instead of choosing the same architecture for both nasty teacher and ad- versarial network, we vary the architecture of the adversarial network to measure the consequent influence with respect to different network structures. As illustrated in Table 4 our training method shows the generality to various architectures (e.g., ResNet-18, ResNext-29, etc), and we also no- tice that weak networks (e.g., Plain CNN) might lead to less effective nasty teachers. However, although stronger networks contribute to more effective nasty teachers, we observe that the trade- off accuracy saturated quickly and converges to uelff-nnermmningg" ones. Thus, we consider the "self-unddrmining"" training as convenient fallback choice.
Table 4:
Ablation study w.r.t the architecture of the adversarial network (.) on CIFAR-1O.
Teacher
Teacher performance
Students after KD
network
CNN
ResNetC20
ResNetC32
ResNet18
Student baseline
86.64
92.28
93.04
95.13
ResNet1
95.13
87.75 (+1.11)
92.49 (+0.21)
93.31 (+0.27)
95.39 (+0.26)
ResNec18(ResNee113)
94.56 (-0.57)
82.46 (-4.18)
88.01 (-4.27)
89.69 (-3.35)
93.41 (-1.72)
ResNer18(CNN)
93.82 (-1.31)
77.12 (-9.52)
88.32 (-3.96)
90.40 (-2.64)
94.05 (-1.08)
ResNet18(RcsNexxt229
94.55 (-0.58)
82.75 (-3.89)
88.17 (-4.11)
89.48 (-3.56)
93.75 (-1.38)
Students Network. In practice, the owners of teacher networks have no knowledge of student's architecture, and it is possible that the student is more complicated than the teacher. As the Reversed KD in Yuan et al.] (2020), the superior network can also be enhanced by learning from a weak network. To explore the generalization ability of our method, the reversed KD. In detail, we consider the ResNet-18 18 as teacher, and ResNet-50 and ResNeX29 as students. From Table 5] these two sophisticated students can still be slightly improved by distilling from a normal ResNet-18 8 in most cases, while be degraded by distilling from a nasty ResNet-18. This implies that our method also effective for the reversed KD setting.
further conduct experiments on
Table 5: Ablation study w.r. the architecture of the student networks.
Dataset
CIFAR-10
CIFAR-I00
Student network
ResNet-50
ResNeXt-29
ResNet-50
ResNeXt-29
Student baseline
94.98
95.60
78.12
81.85
KD from ResNet-18 (normal)
94.45 (-0.53)
95.92 (+0.32)
79.94 (+1.82)
82.14 (+0.29)
KD from ResNet-18 (nasty)
93.13 (-1.85)
92.20 (-3.40)
74.28 (-3.84)
78.88 (-2.97)
Weight w. As illustrated in Figure 3 we vary the weight w from to 0.1 on CIFAR-10 and from O to0.01 on CIFAR-100. We show that nasty teachers can degrade the performance of student networks no matter what w is selected. By adjusting w, we could also control the trade-off between performance suffering and nasty behavior. In other words, a more toxic nasty teacher could be learned by picking a larger w at the expense of more accuracy loss.
Temperature T.. By default, the Ta. used for knowledge distillation, is the same as TA, used in the uelf-unndrnining training. To explore the impact of temperature selection, we vary the temperature т, from to 20, and Figure 4 presents the accuracy of students after knowledge distillation with the given We show that nasty teachers could always degrade the performance of students no matter what T, is picked. Generally, with a larger T., the performance of student networks would be degraded more by the nasty teacher since larger temperature тe usually lead to more noisy logit outputs. Note that our nasty teacher is still effective even if the student directly distills knowledge from probabilities (T,=1).
7

---

Published as a conference paper at ICLR 2021
CIFAR-10 10
CIFAR-100
96
80
94
60
Accuracy
92
90
40
T:ResNet18
T:ResNet18
88
ReseNeCC20
20
S:MovileNetv2
86
S:ResNetC32
S:ShuffleNetv2
0.00
0.02
0.04
0.06
0.08
0.10
0.000
0.002
0.004
0.006 0.008 0.010
@
Ablation study warl on CIFAR-10 10 and CIFAR-100. The initials "T" and "S" in the legend represent teacher networks and student networks, respectively. The dash-line represents the accuracy that the model normally trained.
Figure 3:
Baseline
KD from normal
KD from nasty ResNetC32
Daseline MobileNetV2
KD from normal
KDD from nasty
ResNetCC0
ShuffNetV2
Accuracy
Accuracy
3
Accuracy
13 16
4 10 19
10 16
19
13 16 19
student temperature T
Student temperature
student temperature T
student temperature T
(a) Nasty teacher with Ta on CIFAR-10
(b) Nasty teacher with Тд on CIFAR-100
Figure 4: Ablation study w.r. temperature Ta- The architecture of teacher networks are ResNet-18 for both CIFAR-10 10 and CIFAR-100 experiments. Each figure presents accuracy curves of student networks under the guidance of the nasty normal ResNet-18 with various temperature
Balance Factor
to 0.9 the common practice in (Hinton et al. 2015) To explore the effect of a, we conduct the experiments by varying a from 0.1 to 1.0, and the ex- perimental results were summarized in Figure In general, our nasty teachers could degrade the performance of student networks regardless of what a is selected. We also observe that a small a can help student networks perform relatively better when distilling from the nasty teacher. However, a small a also makes the student depend less on the teacher's knowledge and therefore benefit less from KD itself. Therefore, the student cannot easily get rid of the nasty defense while still mincing effectively through KDD, by simply tuning smaller.
We by default
Bascline
2332 mosly
Baseline
nommal
from muyy
ResNetC20
ResNetC20
ResNetC32
Accuracy
Accuracy
Accuracy
Accuracy
0.2
0.4
0.6
0.8
0.2
1.0
105
40
60
Percent of training data / %
Percent training
(a)
(b)
Figure 5: Ablation study with respect to various a (a) and different percentage of training samples (b). Both experiments are conducted under the supervision of either normal nasty ResNet-18.
Percentage of Training Samples. In practice, stealers (student networks) may not have full access to all training examples. Thus, to reflect the practical scenario of model stealing, we conduct the experiments by varying the percentage of training examples from 10% to 90% and keep other hyper- parameters the same. As illustrated in Figure ((6) comparing to normally trained teachers, the nasty teachers still consistently contribute negatively student networks.
8

---

Published as a conference paper at ICLR 2021
4.4 NASTY TEACHER ON DATA-FREE KNOWLEDGE DISTILLATION
Instead of getting full access to all training samples, KD without accessing any training sample is considered a more realistic way to practice model stealing. To reflect the practical behavior of steal- ers, we evaluate our nasty teachers on two state-of-the-ar data-free knowledge distillation methods, i.., DAFL (Chen et al. 2019) and DeepInversion (Yin et al. 2020), where students only have ac- cess to the probabilities produced by teacher networks. For fair comparison, we strictly follow the setting in DAFL and adopt ResNet-34 and ResNet-18 as the teacher-student pair for further knowl- edge distillation. The experiments are conducted
both CIFAR-10 10 and CIFAR-100, where nasty ResNet-34 is trained by respectively setting w and Тд as 0.04 and 4. The experimental results with regard to DAFL are summarized in Table 6 We show that the nasty ResNet-34 largely detriments the accuracy of student networks by more than 5%, in contrast to that under the supervision of normal ResNet-34. Based on DeepInversion, we also present the visualizations in Figure 6 where the images are generated by reverse engineering both nasty and normal ResNet-34, We demon- strate that images generated from normal ResNet-34 enable high visual fidelity, while images from nasty ResNet-34 consist of distorted noises and even false category-wist features. This visualization showcases how nasty teachers prevent illegal data reconstruction from reverse engineering.
Table 6: Data-free KD from nasty teacher on CIFAR-IO and CIFAR-100
dataset
CIFAR-10
CIFAR-100
Teacher Network
Teacher Accuracy
DAFL
Teacher Accuracy
DAFL
ResNet34 (normal)
95.42
92.49
76.97
71.06
ResNet34 (nasty)
94.54 (-0.88)
86. (-6.34)
76.12 (-0.79)
65.67 (-5.39)
(a) Normal Teacher
(b) Nasty Teacher
Figure 6: Images generated by inverting a normal ResNet34 and a nasty ResNet34 trained on CIFAR-10 with DeepInversion. For each image, each column represents one category.
4.5 DISCUSSION
In practice, the owner of IP can release their sophisticated network defended by self self-undermining training, at the cost of acceptable accuracy loss. As the aforementioned experiments show, even if third-party company owns the same training data, they are not able to leverage knowledge distillation to clone the ability of the released model, since the performance of theirs would be heavily degraded, instead of being boosted as usual. Furthermore, we also show that stealers would suffer more than 5% drop of accuracy if data-free knowledge distillation is performed, of which performance drop is not acceptable in highly security deuritt-demanding applications, such autonomous driving.
To sum up, our self-undernining training serves as a general approach to avoid unwanted cloning or illegal stealing, from both standard KD and data-free KD. We consider our work first step to open the door to preventing the machine learning IP leaking, and we believe that more future work needs to be done beyond this current work.
5 CONCLUSION
In this work, we propose the concept of Nasty Teacher: a specially trained teacher network that per- forms nearly the same as normal one but significantly degrades the performance of student models that distill knowledge from it. Extensive experiments on several datasets quantitatively demonstrate that our nasty teachers are effective under either the setting of standard knowledge distillation or data-free one. We also present qualitative analyses by both visualizing the output of feature embed- ding and logits response. In the future, we will seek other possibilities to enlarge the current gap so that the proposed concept could be generally applied in practice, for which our current work just lays the first cornerstone.

---

Published as a conference paper at ICLR 2021
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 9163-9171, 2019.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3514-3522, 2019.
Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. arXiv preprint arXiv:2003.03519 2020a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets lifelong learning. In International Conference on Learning Representations, 2021a. URL https: http:p::///ppeevvviiie net/ http://aaaaeieeee.eeereeraaaaaaa
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In International Conference Learning Representations, 2021b.
net/ wet.:/ooommidda/yyypaea
Xinghao Chen, Yiman Zhang, Yunhe Wang, Han Shu, Chunjing Xu, and Chang Xu. Optical flow distillation: Towards efficient and stable video style transfer. arXiv preprint urXiv:2007.05 2020b.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 2017.
Lixin Fan, Kam Woh Ng. and Chee Seng Chan. Rethinking deep neural network ownership veri- fication: Embedding passports to defeat ambiguity attacks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv:1805.04770 2018.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 178.00677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint / 2015.
Mika Juuti, Sebastian Szyller, Samuel Marchal, and Asokan. Prada: protecting against dnn model stealing attacks. In 2019 IEEE European Symposium Security and Privacy (EuroS&P), pp. 512-527. IEEE, 2019.
Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adap- tive misinformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 770-778, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv: 1412.6412.880 2014.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks pre-trained models. arXiv preprint arXiv:2004.06666 2020.
Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin, Wei Zhang, Jiashi Feng, and Tong Zhang. Residual distillation: Towards portable deep neural networks without shortcuts. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Published as a conference paper at ICLR 2021
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2604-2613, 2019.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535. 2017.
Haoyu Ma, Tianlong Chen, Ting-Kuel Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv,
2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116,131, 2018.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Has- san Ghasemzadeh. Improved knowledge distillation via teacher assistant. arXiv preprint arXiv:1902.03393, 2019.
Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method fool deep neural networks. In Proceedings of the IEEE conference on com- puler vision and pattern recognition, pp. 2574-2582, 2016.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stealing attacks. In International Conference on Learning Representations, 2020. /ttptttss://ppereeeete
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268 268-284, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin ichi Satoh. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM International Conference Multimedia Retrieval, pp. 269-277, 2017.
Tao Wang. Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 4933-4942, 2019.
Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. Towards privacy-preserving visual recognition via adversarial training: A pilot study. In Proceedings of the European Conference Computer Vision (ECCV), pp.
2018.
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is feature selection secure against training data poisoning? In International Conference on Machine Learning. pp. 1689-1698, 2015.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8715-8724, 2020.
11

---

Published as a conference paper at ICLR 2021
Ryo Yonetani, Vishnu Naresh Boddeti, Kris M Kitani, and Yoichi Sato. Privacy-preserce visual learning using doubly permuted homomorphic encryption. Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 2040-2050, 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911. 2020.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowlede distillation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13876-13885, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv:
2016.
Jie Zhang, Dongdong Chen, Jing Liao, Han Fang, Weiming Zhang, Wenbo Zhou, Hao Cui, and Nenghai Yu. Model watermarking for image processing networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12805 12812, 2020a.
Jie Zhang, Dongdong Chen, Jing Liao, Weiming Zhang, Gang Hua, and Nenghai Yu. Passport-aware normalization for deep model protection. Advances in Neural Information Processing Systems, 33, 2020b.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3713-3722, 2019.
12


Reviews:
[{"review_id": "pu_3XJoCfNw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper explores an interesting and novel research problem: how to make a teacher model undistillable. This can be a promising countermeasure to model extraction/stealing. The proposed Nasty Teacher approach is a two stage method, which first trains a good teacher network, then utilizes a self-undermining KD strategy to further distill the good teacher network to a bad one. Overall, this  Nasty Teacher approach can reduce the performance of the student network by ~5% in most cases. Although the performance decrease is not huge, the proposed approach is promising and can be a very useful baseline for this new research direction. A set of ablation and understanding experiments have also been conducted to support the effectiveness of  Nasty Teacher. \n\nStrong points:\n1. A new security scenario, and a novel defense approach against model stealing.\n2. The proposed idea is well motivated and described.\n3. The effectiveness of the proposed approach is substantiated by thorough experiments.\n\nWeak points:\n1. Shown in Fig.3 and Fig.4, the performance decrease of the student network is not very significant, and is slightly sensitive to the parameters. By finding the right parameter, the student may still achieve ok performance.  What would happen if the student uses temperature \\tau < 4 in Fig.4 (a)?\n2. In Eq. (1), isn’t the second P_T should be P_S? The cross entropy is defined on p?, but I thought p are the logits? Why no \\sigma in the XE? \n3. Eq. (1) indicates the distillation needs logits. It is ok to train the nasty teacher, but how about the student network? In the intro the authors argued that protecting “backbox” APIs is one of the main motivations of the proposed method, but this was not tested in the experiments. What would happen if the student is distilled with probabilities (not logits)? I believe the effectiveness will be even better if the student can only access probabilities. \n4. It seems like the distillation experiments were all done on 100% training data? This setting is somewhat less suitable from model stealing since the attacker may not have 100% training data. What would happen if the attacker has only 10% of training data? In Table 5: Data-free KD, I think a performance decrease of ~6% may still be a successful stealing, considering the huge amount of money and effort it takes to train the teacher network. \n6. How well is the proposed method compared to other defense methods against model stealing? Prior works in this field should also be reviewed, if there are any.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iQs76W8ppl", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new method that prevent the teacher model from being learned by other models. The paper  introduces and investigates a concept called Nasty Teacher: a specially trained teacher network that yields nearly the same performance as a normal one, but would significantly degrade the perfor\u0002mance of student models learned by imitating it.\n\nPros:\n\n- This is overall a meaningful direction of knowledge distillation, that makes a neural network 'undistillable'.\n\n- The method is simple yet effective by disturbing the output distribution of the teacher network.\n\n- The experimental results show that the proposed method can succesfully prevent student networks from learning knowledge from teacher network both with original KD method and with data-free KD method.\n\nCons:\n\n- Since this is a paper that focus on the real-world application problem, I would wonder whether the results are acceptable on ImageNet dataset. I notice that a 2% degrad appears on ResNet-50 on Tiny-ImageNet, which is almost unacceptable on real-world applications. I believe that there will be a larger gap between nasty teacher and original teacher on ImageNet in order to achieve the same undistillable results.\n\n- The paper only discuss the influence of the final layer. What will happen if the student network learns from the feature maps deriving from the middle layers of teacher network? It seems that the loss function can only disturb the final output distribution, but we can still distill knowledge from the other layers. Experiments should be conducted for further explaination.\n\n- It seems that the \\alpha in Eq.(1) will also influence the student performance when learning from nasty teacher. The paper only shows that a large \\alpha (=0.9 in the paper) will degrad the student performance. However, in practice the weight is always set to multiple values and the best among them is picked. I think the author should make an ablation study on \\alpha.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TlAS1rxcM3_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper’s main idea is refreshing and attractive: proposing a defensive method called nasty teacher, to avoid knowledge leaking or cloning through KD. A nasty teacher model is a specially trained network that yields nearly the same performance itself, while significantly degrading the performance of student models learned by imitating it. The standing point for machine learning IP protection is novel and hasn’t seen many discussions before. The introduction section motivated the study clearly and nicely. The method and experiments demonstrate a promising first step towards protecting machine learning IPs. \n\nThe writeup has a smooth logic but demands much more serious proofreading. Specifically, my initial expectation after reading the abstract and introduction parts was quite high (those seem to have been well revised and mature). However, starting from Section 2 the readability drops to a rather poor level. There are many typos that hamper the paper:\n-\tSec 1, “maximizing difference between nasty teacher” – missing “the”. Section 3.1, “a αclose to 1” – “a: should be “an”. The authors need carefully check their usage of the articles in general: “a” versus “an”, missing “a/an/the”, or largely confusing them.\n-\tSec 2, “Teacher-free KD framework (Yun et al., 2020) train - “train” should be “trains”. There are a few more missing “s”/”es”\n-\tSection 3.1, “The higher α is” – “higher” should be “larger”. The authors seem to have confused “high””large””big” and abused them randomly throughout the paper, etc. \n-\tSection 3.2, “we hope to build a kind of logit whose argmax is still the correct answers” – this sentence is grammatically wrong from end to end …\n-\t“the distribution of incorrect answers varies as much as possible to the real distribution over the dataset” – I guess that is in contrary to what you wanted to say. “varies as much as…” means “correlates well with”, but your method should expect them to differ from each other.\n… and I could have placed a longer list. I urge the authors to maintain the same high standard of revision beyond the point of Section 1.\n\nAnother issue is while the authors stressed their training algorithm called self-undermining KD in the abstract and introduction, that algorithm is only very briefly introduced in the last paragraph of section 3.2 -- which I nearly missed. I strongly suggest the authors to use an algorithm table to make your main algorithm clearer and more visible. \n\nBesides, the text in Section 3.2 also needs to be considerably re-organized and revised; currently it is hard to figure out the right focus and lacks clarity. For example, I cannot see whether the adversarial network is also updated during training.\n\nFrom the draft, it is also unclear whether the authors will release their codes and models for reproducibility\n\nOverall, this paper is technically novel and interesting, and can potentially generate a positive impact. But the draft quality is currently unsatisfactory. Careful proofreading and thorough revision are requested from the authors; otherwise my final recommendation cannot be positive. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QSb4iHR86T7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper reveals and studies a new problem, that KD is posing a potential risk of intruding the intellectual property (IP) of released ML models or their training data. Even trained ML models are released only in “black boxes” (e.g., as executable or APIs, no open-sourcing codes), their functionalities can be largely replicated by KD through imitating input-output behaviors. I find this problem and idea very interesting, and of both social and legal importance.\n\nPros:\n- The self-undermining training method seems to be a variant of adversarial training. It maximizes the K-L divergence between the nasty teacher logits and those produced by another adversarial network. The authors used a similar idea to born-again NNs and self-training KD, to generate adversarial attacks from another pretrained network of the same architecture; thus no extra model is necessary. The training method is conceptually simple but effective.\n- The authors conducted many experiments on several datasets to demonstrate that the proposed method is effective on both standard KD and data-free KD. Especially, they show the KD-immunity w.r.t various student models from small to large.\n\nCons:\n- Even the authors presented an ablation study, it remains unclear to me why this self-attack is a good choice? Is convenience the only reason, and might there be more benefits (e.g., are the attacks generated by the same architecture more “specialized”)? \n- “While our nasty teacher targets to degrade the performance of all kinds of students networks through KD”, I would suggest the authors to seriously tone down claims like this (or present substantially more experiments)\n- We need more understanding why the nasty teacher works. Looking at Figure 2, I cannot get any useful information why the nasty features are more detrimental to distillation, and why this is not reflected on t-SNE? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper argued some viewpoint about knowledge distillation quite interesting to me: the technically good KD might surprisingly be socially bad in helping outsiders "stealing" commercial models, even if the models are released as black boxes. Then the paper proposed a way called self-undermining KD in order to turn a well trained model into a "nasty teacher" (i.e., an undistillable model), and by this way the commercial models and the corresponding intellectual properties for training them from insiders can be nicely protected.

Overall, the quality is quite high. The argument is very conceptually novel and the method is still technically novel. The idea of the method is simple but works for the purpose --- that's great! Although the experimental significance seems not too impressive, the paper opens a door to a new world concerning model privacy instead of data privacy, and hence it is of social significance. In my opinion, the paper should have a potentially huge social impact to DL practitioners (and company owners), because KD is being used almost everywhere in the Internet industry to provide the standalone mode of Apps without clouds on personal devices. Based on the quality and the impact, I recommend to accept the paper as a spotlight presentation.
Decision: Accept (Spotlight)