Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
Authors: Souvik Kundu
Qirui Sun
Yao Fu
Massoud Pedram
Peter Anthony Beerel
Venue: NeurIPS 2021 Poster (2021)
Keywords: Model confidentiality
Data Privacy
DNN Poisoning
Knowledge Distillation
Model IP protection
Data-free distillation

Abstract: Knowledge distillation (KD) has recently been identified as a method that can unintentionally leak private information regarding the details of a teacher model to an unauthorized student. Recent research in developing undistillable nasty teachers that can protect model confidentiality has gained significant attention. However, the level of protection these nasty models offer has been largely untested. In this paper, we show that transferring knowledge to a shallow sub-section of a student can largely reduce a teacher’s influence. By exploring the depth of the shallow subsection, we then present a distillation technique that enables a skeptical student model to learn even from a nasty teacher. To evaluate the efficacy of our skeptical students, we conducted experiments with several models with KD on both training data-available and data-free scenarios for various datasets. While distilling from nasty teachers, compared to the normal student models, skeptical students consistently provide superior classification performance of up to ∼59.5%. Moreover, similar to normal students, skeptical students maintain high classification accuracy when distilled from a normal teacher, showing their efficacy irrespective of the teacher being nasty or not. We believe the ability of skeptical students to largely diminish the KD-immunity of potentially nasty teachers will motivate the research community to create more robust mechanisms for model confidentiality. We have open-sourced the code at https://github.com/ksouvik52/Skeptical2021

Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
Souvik Kundu, Qirui Sun, Yao Fu, Massoud Pedram, Peter Beerel
Electrical and Computer Engineering University Southern California Los Angeles, CA 90089
{souvikku, qiruisun, yaof, pedram, pabeerel) edu
Abstract
Knowledge distillation (KD) has recently been identified as a method that can unintentionally leak private information regarding the details of teacher model to an unauthorized student. Recent research developing undistillable nasty teachers that can protect model confidentiality has gained significant attention. However, the level of protection these nasty models offer has been largely untested. In this paper, we show that transferring knowledge student can largely reduce a teacher's influence. By exploring the depth of the shallow subsection, student model to learn even from a nasty teacher. To evaluate the efficacy of our skeptical students, we conducted experiments with several models with KD under both training lata-available and data-free scenarios for various datasets. While distilling from nasty teachers, compared to the normal student models, skeptical students consistently provide superior classification performance of up to ~55.55% Moreover, similar to normal students, skeptical students maintain high classification accuracy when distilled from normal teacher, showing their efficacy irrespective of the teacher being nasty or not. We believe the ability of skeptical students largely diminish the KD-immunity of potentially nasty teacher will motivate the research community to create more robust mechanisms for model confidentiality. We have open-sourced the code at fithub.ccmm/ee http.////.oooii/////psppi.aalll
shallow sub-section of
then present distillation technique that enables a skeptical
1 Introduction
Knowledge distillation (KD) 10 aims to transfer the useful knowledge of a trained model (the
teacher) to another model (the student). KD has found success in various applications 29 3 17
and is particularly useful for resoucce-constrainee IoT applications where the compute budget is limited and compute-efficient models are required. Generally. KD requires the student model to be trained over the same data-set that is used to train the teacher. However, recent research [31] 20 2 has shown the efficacy of KD even under the "data-free" scenario where the training data may not be available for the student get trained.
Over the past few years various forms of distillation have been proposed, including distillation from the student itself (36) and via an ensemble of students [37]. However, recently, reference (18] has highlighted the fact that many forms of distillation may unintentionally leak the IP of a teacher model. In particular, some teacher models contain significant IP associated with the arduous effort of both data collection and model training, which motivates their release as "black-box" executable pieces of software. Morcover, these trained models may even enable safeguarding the training data (for example, sensitive medical images (28) and company proprictary information [27 30)
well as their performance on the secure data. Use of KD under these circumstances. sometimes
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

80
Base Pr
Dust od mode
Iook
Dicblled
2
28 220
-60
250
440
30
30
200
20
10
10
Teacher: Suato 5eu50
Res
R6S50
Raste Ras30
MoVZ
Ocction
Figure 1: Distillation from a nasty ResNer50 to (a) normal students, (b) proposed skeptical students, on CIFAR-I00. In particular, for MobileNetV2 (MbV2) which is a reduced parameter model, the proposed distillation method can improve the accuracy by 59.49 59.49%. (c) Impact of transferring knowledge various depth of a ResNet18 from nasty teacher. BB represents basic-block layer.
under the "data-free" condition, may enable training of an unauthorized student to yield comparable performance as the teacher. To mitigate this issue, reference [88] has proposed the idea of a nasty teacher that prevents knowledge leaking to a student and thereby reduces the student's classification performance. Fig. T a) depicts the the success of nasty ResNet50 in degrading various student models' classification performances compared to their respective baseline performances]
Earlier evaluations [8 have shown that nasty teachers can retain their efficacy under various settings of two key hyperparameters, namely the weight of the distillation loss (a) and the softmax temperature (r) of the distillation loss. However, a comprehensive evaluation of the efficacy of undistillable nasty teachers has yet to be completed. Towards this goal, we investigate the performance of KD on distillation at different depths of the student model, and in particular, how the teacher's ' influence changes when transferring knowledge to an intermediate shallow section of the student through an auxiliary classifier (AC). We find that the impact of the nasty teacher drastically reduces as we transfer knowledge to a shallow subsection of the student (Fig. (c).
Our contributions:
Based on these findings, we present skeptical student that uses intermediate shallow auxiliary classifier to transfer the information derived through the soft probabilities of the teacher's 's output classes We further propose novel hybrid distillation scheme to improve learnability of the student by distilling both from a teacher and the student itself. Our approach has some similarity with self- distillation (SD) (36] because both approaches use auxiliary classifier for the knowledge transfer. However, the goal of SD is to show the efficacy of a model distilling from itself, contrasting our goal of analyzing the possible presence of potential model stealer who extract knowledge from an undistillable nasty teacher. More importantly, the proposed hybrid distillation is effective in stealing a model's IP even under "data-free" scenario, contrasting SD which is only applicable for students who have access to training data.
We conduct extensive experiments using both standard KD with available training data CIFAR-10, CIFAR- 100, and Tiny-ImageNet, and data-free KD on CIFAR-I0 testset. Experimental results show that compared to normal ones, skeptical students exhibit improved performance of up to ~5.5% and ~5.8% for data-available and data-free KD, respectively, when distilled from nasty teachers. This exposes significant limitation of nasty teachers attempting protect model IP. Moreover, proposed students perform similar to normal student models while distilled from normal teachers, demonstrating their efficacy irrespective of the teacher being nasty or not.
Preliminaries
2.1 Knowledge Distillation
Knowledge distillation, similar to the goal of various model compression techniques 13 7 5 14 16] embeds the rich information of compute-heavy model into model that generally requires fewer computations. The traditional KD [0 relies on information transfer through a Kullback-Leibler (KL) divergence measure between the soft logits the output classifier layers of the teacher and the student. Apart from this, over the past few years, various other efficient distillation methods have been proposed, including distilling from hints provided by the teacher (25). distilling via attention transfer [34], and other approaches (23) 21]. To reduce the distillation based training time and avoid the requirement of a separate teacher, reference (36] has proposed self-distillation. The authors partition
'The baseline models are trained with only cross-entropy (CE) loss.

---

Table 1: Performance of student (ResNet18)
77
under transferability test on CIFAR- 100.
366
Teacher
[Teacher type Teacher Ace % Student Ace brase
774
T. Kesty Restiet50
ResNet50|
Nasty
76.57
72.47
-5.08
772
T. Normal Reshiet50
Baseine
ResNet18| Distilled
72.47
70.99
-6.56
ResNe150
Normal
78.04
+1.84
Figure 2: A ResNet18 student's 's performance on CIFAR-100 dataset.
Aux. classifier location
ResNer18|
Distilee
79.39
79.47
+1.92
the student model into several shallow sections, each having its own auxiliary classifier, to which the final classifier transfers the soft-logits to enhance the model's classification performance. Several recent studies [32 33] have also shown the efficacy of KD to a student from its own pre-trained variant as its teacher. Similar to [88, in this work, assume to have access to the teacher model'ss s intermediate features. Rather, we focus on the information leaking to an unauthorized outsider and thus rely on standard KD.
2.2 Model IP protection
Protection model IP has drawn significant interest primarily due to the massive human resource and financial costs required for large model training. Earlier works explored various defense strategies. including adaptive misinformation against model performance stealing [II] and passport- -based defense [35] In most of these cases, the stealer is assumed have access only synthetically generated data. However, also of interest is when a portion of training data is unintentionally leaked.
2.3 Poisoning of Neural Network Models
attacker can degrade the performance of a neural network by simply injecting poisoned data [2] into the training set. Adversarial-attee -ttackk generated images [6] in degrading a model's performance. Backdoor attacks 4 insert crafted malicious data into the
19 [5] have proven to be effective
training set that apparently trains the model to perform well until such time that the attacker sets up signal that degrades the model performance drastically. The Bit flip attack (24) corrupts selective bits of the trained DNN weights to lower its performance. As mentioned earlier, reference (18) has recently proposed to poison neural network model through training, such that the model retains its classification performance, but loses its ability to be used as efficient teacher, thus referred to as a nasty teacher. These nasty teachers are believed to protect the model IP This work analyzes the degree of IP protection that such teachers provide and, in particular, presents skeptical moudent-basee distillation technique that diminishes the effect of their nastiness, as detailed in the next Section.
Motivational Case Study
To motivate our skeptical students, this section presents an empirical analysis that explore the efficacy of nasty teacher models under two distinct KD scenarios
3.1 Transferability of the Impact of Nasty Teachers
Definition 1. Secondary student: We define a secondary student as a model that is trained via KD from trained model which earlier trained via distillation from teacher model. In this context, we refer to the student that is distilled from the original teacher model, as the primary student. We measure the transferability impact (TT) of teacher as the performance improvement (or degradation) of secondary student with respect to its baseline (Abase). A negative Amsse signifies the success of teacher's privacy confidentiality preserving effort [18]
"In so far as teacher is trying to protect its private IP from inquiring or even intrusive students, we believe a better phrase characterize him/her is "defensive teacher" in the worst case "secretive teacher". In our view, there is nothing "nasty" about what teacher is attempting to do. However, because the prior art has consistently used the term "nasty teacher," we will also use that phrase in this paper.
The description of training hyperparameters is given in Section 5.1 for all the experiments in this section.
3

---

Student's loss
Selt Oatt-aaiioion lon lose
Distilatioon loss from teacher
Shelow saction (8r))
Student (0g) (Potantiol model
Inpur
Stesles)
Trainable parameters
Student's sucilary branch
Pre-trainad teacher model
Figure 3: Skeptical student distillation framework. Note the arrow of the distillation loss components directed from teacher to student for the corresponding KL-divergence computation.
We first trained both normal as well as nasty variants of ResNet50 on CIFAR- 100. We then used two ResNet18 8 students to distill knowledge from these two teachers. Finally, to test TI we used the distilled ResNet18 8 models teachers secondary students with identical model architectures.
Observation 1. The TI of a nasty teacher is negative, meaning the nastiness of a teacher transfers to its student.
Table T shows the classification performance of secondary student is 6.56% lower than the baseline while using a distilled ResNet18 teacher. Interestingly, this implies that the false sense of generalization that the primary student inherits makes become a nasty teacher. Thus simple redistillation from primary to secondary student will not evade the nastiness of teacher.
3.2 Transferring Knowledge to Shallow Subsection of the Student
We computed the weighted KL-divergence based distillation loss at an auxiliary classifier (AC) of the student, and used its final classifier (C) to compute the weighted CE-loss only. We evaluated the impact of the auxiliary classifier placed at different depths of the student model. In particular, for the ResNet18 student we performed four experiments by placing the AC after the n BB, n ranging from to 4, where the model's 's final classifier is located at the end of 4th block. We follow similar procedure as (36 to design the AC branches.
Observation 2. The influence of a teacher (both nasty and normal) on a student's performance reduces distill knowledge to a shallow subsection of the student model.
Fig. 2 shows the accuracy of the student model approaches the baseline accuracy as we distill from the teacher shallower depths (n 1). Interestingly, this trend be observed for both nasty and normal teachers. In particular, from a nasty ResNet50, the student ResNet18 has test accuracy of up to 77.19% while distilled AC 1, in contrast the accuracy is 72.47% when distilled at the final classifier. On the other hand, ResNet18 8 can have classification accuracy reduction of as much as 1.56% when distilled shallow depths.
We leverage these two observations in our hybrid distillation on the skeptical students and diminish the transferring of teachers' nastiness, presented in the next section.
4 Skeptical Students
Let us consider student model 4s. which distills knowledge from pre-trained teacher 47 where Ios (-) and (.)(() (.) are the functions describing student and teacher models, respectively. Let (x, be the vectorized pairs of inputs and corresponding output labels used train these models. For teacher-bart based traditional KD [10], the training loss for the student may be written as
CKD= = (1- (1- * Les (g (x.y))) + a (xc (x.y), r), (go, (x)
(1)
where Lee represents the student's cross-entropy (CE) loss and Lxy represents the KL divergence loss aimed at transferring knowledge. Here, (.)) is the softmax function and is the softmax
'I may also be interesting to note that, as shown in Table the TI of normal teacher follows similar trend and remains positive.

---

temperature, both of which are used to compute soft probabilities. is set to 1 for the CE loss. The hyperparameter a acts as a balancing factor between the two loss terms. Based on the observations in Section 3 we propose a skeptical student that can largely diminish its nastiness at the very first round of distillation. particular, skeptical students are those models that always transfer the teacher's KL.-div driven knowledge at a shallow section ($'g), as depicted in Fig. [] Thus the teacher driven loss, LT can be formulated same as the LKD with Y replacing 2 in both the CE and KD loss components. To train the complete student model Ds we rely on the CE-loss applied at the final classifier. However, due to reduced influence of the teacher, such students can hardly get any benefit of KD from normal teacher. Hence, to improve these models' performance, motivated by the idea of self siffdistillatiion (SD), we introduce a third loss term that allows distillation to shallow ACs from the student's 's final classifier,
LSD
E{(L- 9) Lce Ceeeeeeleeee (o( (x,y))) + B Lxce (ol99) (x.y), o (gos (x,y), (2) JEJ
Note that, J E AC, where, N > i> isg Here N and ioy represent the total number of sub blocks
in the model and sub-block index at which the AC of 8's is placed, respectively. Finally, our hybrid distillation loss is given by,
(3) where the last term corresponds to the CE loss of the complete student Dg. Here, n, 72, and 7 are hyperparameters to balance the KD loss of the auxiliary classifier, self-distillation loss, and the CE loss of the student. It is noteworthy that we use the auxiliary classifier sections during training and that these auxiliary sections may removed during inference, nullifying any extra inference parameter cost.
= + 12CDD + raCcs (rggs((x)))
Skeptical students for data-free KD As described earlier, the skeptical students primarily use Therefore, teaching the remaining
an intermediate auxiliary classifier to distill knowledge. part of the network ($g B's) is, in particular, difficult for data-free KD because there is no CE-loss to train the whole network g. To mitigate this issue we propose an auxiliary self distillation loss to distill knowledge to the final classifier from the intermediate auxiliary classifier. Note that, as depicted in Fig. 4 the same auxiliary classifier works as a student to learn from a teacher under the data-free scenario. To evaluate this, we use recently proposed zero-shot knowledge transfer [20] with the skeptical students using a loss function enhanced by the auxiliary self KD,
Aundilary anlf
Figure 4: Data-free distillation from a teacher to skeptical student.
fSeF Lxc (1(90, (x,y), of
+ Lxx (x ( (x,y), + Yat CT
(4)
The first term takes care of knowledge transfer from the teacher, while the second term helps train the final classifier. Similar the original paper (20), we also use an ceenttio-transfer loss AT-
Nasty teacher training, As proposed in 18 we use self-undermining knowledge distillation to design the nasty teacher ΦT. Specifically, we train ọ7 via distillation from pre-trained model & with the same network architecture, minimizing the following loss
L Lee (xo, (x. ON T Lx (a (x.y), TN), (x.y), Tx))
where the CE loss terms helps retain 4r's classification performance. The second term maximizes the KL divergence between A and Φf allowing Kr to learn a false form of generalization that plays a key role in its undistillability [18]. Here, TN is the softmax temperature, similar to traditional KD, and a controls the severity of the aaff-undermniing distillation of the nasty teacher.
5
Experimental Results
5.1 Experimental Setup
Models and Datasets. To evaluate the efficacy of our hybrid distillation approach, we performed detailed experiments on three popular datasets, CIFAR-1Ó, CIFAR-100 [12, and Tiny-Ima g
'To minimize the auxiliary layer computation overhead during training, in this paper we use 1 AC to transfer the teacher knowledge and 1 AC for SD.
5

---

Table 2: Performance of normal vs. skeptical student when distilled from a nasty teacher.
Dataset
r
45 Acc. (%)
4g Base- line Acc. (%)
Student Acc. (%)
Agce
Normal (aco.)
Skeptical (aer.)
SSeetical E. (
ResNet18
94.67
ResNet18
95.15
94.13(10.18)
95.09(
0.15)
94.77CL 0.05)
+0.96
MobilcNerV2
88.13(=+0.13)
90.37(
0.25)
90.21C 0.18)
+2.24
CIFAR -10
94.28
ResNet18
95.15
94.38(:10.18)
95.16/
0.01)
95.021:: 0.01)
+0.78
ResNet50
ResNei50
94.9
94.211:00:04)
95,48( 1: 0.14)
95,48C 0,14)
+1.27
MobileNetV2
90.12
88.76(==0.14)
91.021 + 0.09)
90.880 0.23)
+2.26
ResNer18
77.55
ResNet18
77.58
75.00(1::0:14 14)
77.33
0.21)
76,38C 0.1)
+2.33
MobileNetV2
7.171
66.62(11 0.30)
64,26r I 0.64)
+59,49
CIFAR -100
76.57
ResNet18
77.55
72.28(==0.277)
77.25(
75.48:+00.54
+4.97
ResNet50
ResNer50
78.04
74.14(1:0.855
78.651 0.29)
77.61(=011)
+4.52
MobileNelV2
69.24
7.72(4:1.57)
66.38( 0.50)
62.93(100755
+58.66
Tiny- ImageNet
ResNet18
62.08
ResNet18
63.07
53.60(1:0004)
65.76(+:0.83)
60.63(+:0007)
+12.16
MobileNetV2
57.01
4.81(1 0.19)
54.74(1:0.84)
54.27(12.94)
+49.93
with R ResNet50 (9] and MobileNetV2 [26 models. We used PyTorch API to define and train our models on an Nvidia RTX 2080 Ti GPU.
Training hy typerparameters. We used standard data augmentation techniques (horizontal flip and random crop with reflective padding) and the SGD optimizer for all training. To create a nasty teacher, we first trained network 4 for 160 epochs Tiny-ImageNet with an initial learning rate (LR) of 0.1 for all. For CIFAR-10, we reduced the LR by factor of 0.1 after 80 and 120 epochs. For CIFAR-I00 and Tiag-em the LR decayed at 60, 120, and 160 epochs by factor of 0.2. We then trained the nasty ọ of same architecture with the same epochs and LR hyperparameters. We chose a' as 0.04, 0.005 and 0.005, for CIFAR-IO. CIFAR- 100, and Tmne-lmageNet, respectively [18]. Similar to [18]. we chose TN to be 4, 20, and 20 for the three datasets. For the distillation training to 4s (both normal and skeptical), we trained for 180 epochs with a starting LR of 0.05 that decays by a factor of 0.1 after 120, 150, and 170 epochs. Unless stated otherwise, we kept 7 the same as TN and chose a and We placed the skeptical students' auxiliary classifiers after the 2nd (0's for KD from the teacher) and 3rd (for SD) BB of a total of 4 ResNet blocks. To give equal weight to the loss components of Eq. we chose ? = ?22 = 7/3 =1.0, for all the experiments. We performed all the experiments with two different seeds and report the average accuracy with std deviation (in bracket) in the tables.
CIFAR-10 and 200 epochs for CIFAR-100 and
be 0.9 and 0.7, respectively.
5.2 Data-available Distillation
To evaluate model performance, we conducted two types of distillation: distill to self (DtoS) [32]. where both teacher and student architectures are the same, and KD from a compute heavy teacher to reduced-parameter a reduced- parameter student (for example.0y" ResNet50, Øg: ResNet18, MobileNetV2). For DtoS, we also performed distillation with both the assumption of the model being heavy (Prys: ResNet50) and lite (Prys: ResNet18). 8). Table 2 shows the corresponding performance when distilled from nasty teacher. Skeptical students always outperform their normal counterparts providing better accuracy with improvements of to ~5..449 49%. These results clearly show the efficacy of skeptical students in mitigating the undistiilabilliy of a nasty teacher. We also measure the classification performance by ensembling the ACs and final classifier outputs and denote that 'Skeptical-E', However, the ensemble performance is always inferior to the final classifier, which is primarily due to inferior performance of the AC that distills knowledge from the nasty teacher.
Table BableBishow shows the performance of both skeptical and normal students when distilled from a normal teacher. As we can see, the ensemble output of skeptical students perform better than their normal counterparts. These results motivate the use of of skepticul students for distillation irrespective of whether the teacher is nasty or not. In both the tables Auce is the accuracy difference between skeptical and corresponding normal student when both are trained via distillation from a teacher, i.e. A Dace= {max(ace, acc..) a.c..).

---

Table 3: Performance of normal vs, skeptical student when distilled from normal teacher.
Dataset
T
Pr Acc. (%)
iss Base-
Student Acc. (%)
Aace
line Acc. (%)
Normal (acea.
Skeptical (ace,)
Skepticul-E (acc..)
8
95.15
ResNet18
95.15
95.38 (2)0.10)
95.45(.00110
95.42(110.09)
+0.07
MobileNetV2
90.12
91.36(20.17)
91.1(=0.155
92.00(=0.28)
+0.04
CIFAR -10
ResNet18
95.15
10.43:200111
95.31(=0.01)
95.27(=20.04)
40.12
ResNer50
94.9
ResNet50
94.9
15.15(20.13)
95.85(=0.05)
=0.09
+0.94
MohileNesV2
91.71C10.06)
91.71(20.18)
91.95(10016)
8
77:55
ResNec18
77.55
78,9 78.96(=0012)
78.79(=0:42)
79.68 #0.52)
+0.72
MobileNet
69.24
0..120100008
0.16920.19)
75.45
+0.33
CIFAR -100
78.04
ResNet18
77.55
79.21(10.24)
78.51(=0:44)
79.86
100.0)
+0.65
ResNet50
ResNet50
78.04
09.16(200133
80.66, =0.52)
81.96; 0.552)
+2.4
MobileNetV2
69.24
75.28(=0000)
7..76(=0.16)
76.321=#0.55
+1.04
Tiay- ImageNNei
ResNet18
63.07
ResNet18
63.07
67.35(10.188)
66.4% 0.300
67.43
+0.08
MobrieNee
57.01
64.99(10.03
0.001)
65.38: 10.01)
+0.39
Rormal Stucert
Stappial Studant
1.0
100
20.5
204
204
203
622
13
25.30
8M
4.5.6 6.6
0.0
Casses
Nerral Teacher
Stepiical
14 1
62
0.0
1.2
5
T
5
2
Figure 5: Logit response visualization after the softmax layer. Each row contains an example image from CIFAR-10 dataset and corresponding response for normal teacher, nasty teacher, normal student and skeptical student. We used ResNet50 and ResNet18 as teacher and student model, respectively.
Qualitative Analysis
We now present qualitative behavioral analysis of both normal and skeptical students upon distillation from both normal and nasty teachers. Fig. 5ig.5]ste shows that the nasty teacher has multiple nonnneglig peaks at its final softmax logit response, in contrast to the normal teacher having mainly one high valued peak. As mentioned in [18), this can create false sense of generalization a normal student causing the student to misclassify, as shown by its logit response. Our skeptical students, on the other hand, not only classify correctly, but also largely mitigate the issue of multi peak logit response of normal student.
We present visualizations of the t-distributed stochastic neighbor embedding (t-SNE) for output logits in Fig. 66 It shows that the inter class cluster distance is shifted and reduced for certain classes of the nasty teachers. A similar shift of class clusters is also observed for the normal students and even the AC of the skeptical student where teacher knowledge is transferred. However, the final classifier of the skeptical students has similar class clustering distribution normal teacher. This demonstrates that the remaining sections of the student model (4h 4'r) indeed remain free from the impact of the nasty teacher.
5.4 Ablation Studies
Ablation study with the Temperature т. To further evaluate the influence of the hyperparameter on the student distillation, we performed ablation with r E [2 10, 15, 20). As depicted in Fig. aa). when distilling from a nasty teacher, the skeptical students maintain their superiority compared to their normal counterparts at all different values r. While distilling from normal teacher, even reduced t, both the normal and skeptical student variants retain higher classification accuracy compared to their baseline (Fig. (c).
7

---

100
hermal betterr
195
T. N.Nim Norral
Nodss Banthe
Bacy S. Norm
300
autorobble
-50
-50
bird
100
GST
-100
-50
50
100
100
-90
110
-200
-10
50
10
-100
-50
50
160
300
Semel 5: Mertiaaaaca
100
Norm. S: Suppiorl
100
SteptcarAC
NoBa 3: Skepticaet
haree
Ship
Fog
Black
20
-30
-100
-200 100
-50
100
-200
-50
55
10c
Figure 6: Visualization of tSNE for normal and skeptical students (ResNet18) upon distillation from both normal and evasive teacher (ResNet50) on CIFAR-10. For the skeptical students we plot visualization both at the final classifier (C) and auxiliary classifier (AC).
Teacher
82
70
78
80
24
74
Mayd
70g
76.
Temperatrr DO
Ataha
IN
Tempenature
Aipha
16
Figure 7: Ablation study with a and F for normal and skeptical students (ResNet18) upon distillation from both normal and nasty teacher (ResNet50) on CIFAR-
Ablation study with the balancing term a. To determine the influence of the undistillable teacher on the performance of the presented models, we conducted distillation with a E [0.2, 0.4, 0.6, 0.8, 0.9]. As a reduces, the influence of the teacher is reduced and we see an obvious improvement in student performance. Interestingly, as we can see in Fig. bb), even at reduced a the skeptical students maintain improved performance compared to their normal counterparts. In distillation from a normal teacher, similar to the previous ablation, the skeptical students do not suffer from any significant performance drop compared to normal students (Fig. d)).
5.5 Limited Data and Data-Free Distillation
Instead of having full access to all training samples, KD with limited or no access
to training sample is considered more realistic scenario for model stealing. Fig. g shows the students' performance upon distillation from a teacher (both normal and
nasty) when only a fraction of the total training data is available. In particular, the figure shows under different % of training data availability the skeptical student performs consistently better than its normal variant upon distillation from nasty teacher. When distilled from normal teacher, the skeptical student perform similar to its normal counterpart.
Teacher: nesty ResNet50
Teacher normel Res
Taae Secunnn 196
68 8 800
90
5: Shothua Matinal
85
Shap Merrrl
20 40
60
80
100
60
20
40
60
80
100
Percantage of training data
Percentage of training data
Figure 8: ResNer18 on CIFAR-10 dataset under different percentage of limited training data.
To demonstrate skeptical student's performance under data-free scenario, we leverage the idea of zero shot knowledge transfer (20), a state-of-the-art data-free distillation technique. For this evaluation we used ResNet34 and ResNet50 as teacher models with ResNet18 8 as the student for both, on CIFAR-10 We used the same training hyperparameters as in 20] with the proposed loss introduced in Eq. and evaluated the performance when the teacher is both grey and white boxed. In particular, for the grey-box and black-box assumptions, we computed the final loss with and without the inrention-transfe (AT) loss from the teacher, respectively. Table 4 shows the skeptical students always yield higher classification accuracy compared to their normal counterparts upon distillation from both normal and nasty teachers. In particular, while distilling from nasty teacher the student's
8

---

Table 4: Performance of normal vs, skeptical student on data-free distillation 20 from teacher.
Dataset
4 Acc. (%)
Stadent Acc. (%)
Agge
type
Normal
Skeptical
loss (grey-box)
ResNet34
Nasty
94.81
ResNet18
1,270
91.76(:1:0300
+4.06
CIFAR -10
Normal
95.3
03.21(10.21)
93.52/ 1.0006
+0.11
ResNet50
Nasty
94.28
80.39(=1.19)
86.14(=00.01)
+5.80
Normal
94.9
90.54( 1.1)
91.93
+1.39
Without loss (black-box)
CIFAR
ResNet50
Nasty
94.28
ResNet18
0..95(
79.93 0.0.28)
+58.93
10
Normal
94.9
22.08( 0.556)
S0.76(
+58.63
performance can improve up to 5.8% and 58. 93,,, with grey-box and black-box teacher assumptions, respectively. These results clearly show that skeptical students can largely diminish the KD-immunity of nasty model under even the data-free scenario.
5.6 Transferability of Nastiness on Skeptical Students
Similar to the transferability test on normal students (see Table we also explored the transferability of nasty teacher to skeptical student. For this experiment, a nasty teacher (ResNet50) as teacher for a secondary student on CIFAR-100. Here, we used a normal ResNet18 the secondary student model. Interestingly, Table 5 shows the performance of the secondary student improves by 1.67% compared to the baseline ResNet18, 8, following the same trend student distilled from normal teacher. From these results, conclude that, skeptical student not only reduces the nastiness of a teacher on its own performance, but also breaks the chain of transferability of nastiness to a secondary student.
skeptical student trained from
Table 5: Performance of a skeptical student (ResNet18) under transferability test CIFAR-100.
Teacher
Teacher type
Teacher
Student %
Aboze
ReaNe150
Nasty
76.57
77.43
-0.12
ResNet18
Nasty distilled
77.43
79.22
+1.67
ResNei50
Normal
78.90
ReaNet18
Normal-distiled
78.90
79.92
+2.37
Conclusions
In this paper we presented skeptical student who leverages simple yet effective hybrid distillation strategy diminish the effect of a nasty teacher and largely retain its classification performance. In particular, our experimental results showed that, when distilling from nasty teacher, the performance of skeptical students up to ~59.5% higher than that of normal students. Our models also retain similar performance as the normal student when distilled from a normal teacher, showing the general efficacy of the proposed KD under both nasty and normal teacher scenarios.
7 Broader Impact
The diminishing immunity of nasty teacher for the proposed distillation technique highlights the growing concern of model IP protection in today's DNN-driven world. We believe this study will help the community understanding the limitations of such model IP protection techniques. Moreover, we also hope this will motivate the community further study the fundamental limits of maintaining model confidentiality given access to black-box teacher.
Acknowledgment
This work was supported in parts by NSF including grant number 1763747.

---

References
[1] Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3585-3592, 2020.
[2] Hanting Chen, Yunhe Wang. Chang Xu, Zhaohui Yang. Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3514-3522, 2019.
[3] Xinghao Chen, Yiman Zhang, Yunhe Wang, Han Shu, Chunjing Xu, and Chang Xu, Optical flow distillation: Towards efficient and stable video style transfer. In European Conference on Computer Vision, pages 61 4-630. Springer, 2020.
[4] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 172.:517.. 2017.
[5] Ting-Was Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1518-1528, 2020.
[6] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. arXiv preprint arXiv:1412.6577 2014.
[7] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and Huffman H coding, arXiv preprint arXiv:1510.00149, 2015.
[8] Lucas Hansen. Tiny ImageNet challenge submission. CS 231N, 2015.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1503. 2015.
2015.
[11] Sanjay Kariyappa and Moinuddin K Qureshi. Defending against model stealing attacks with adaptive misinformation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 770-778, 2020.
[12] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[13] Souvik Kundu, Mahdi Nazemi, Peter A Beerel, and Massoud Pedram. DNR: A tunable robust pruning framework through dynamic network rewiring of DNNs. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, pages 244-350, 2021.
[14] Souvik Kundu, Mahdi Nazemi, Massoud Pedram, Keith M Chugg, and Peter A Beerel. Pre- defined sparsity for low-complexity convolutional neural networks. IEEE Transactions on Computers, 69(7): 1045-1058, 2020.
(15) Souvik Kundu, Massoud Pedram, and Peter A. Beerel. Hire-snn: Harnessing the inherent robustness of mneryy-efficient deep spiking neural networks by training with crafted input noise. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 5209-5218, October 2021.
[16] Souvik Kundu, Saurav Prakash, Haleh Akrami, Peter Beerel, and Keith M Chugg. psconv: A pre-defined sparse kernel based convolution for deep cnns. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 100-107. IEEE, 2019.
[17] Souvik Kundu and Sairam Sundaresan. Attentionlite: Towards efficient self-attention models for vision. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2225-2229. IEEE, 2021.
10

---

[18] Haoyu Ma. Tianlong Chen, Ting-Kuet Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Undistillable: Making nasty teacher that cannot teach students. In International Conference on Learning Representations, 2021.
[19] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: (af:::170.000666 2017.
[20] Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching. arXiv preprint arXiv: 1905.09768, 2019.
[21] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5191-5198, 2020.
[22] Seyed-Mohsed Moosavi- -ezaaili,,,,,ii, Alhussein Fawzi, and Pascal Frossard. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2574-2582, 2016.
[23] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference Computer Vision (ECCV), pages 268 284,2844222 2018.
[24] Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Bit-flip attack: Crushing neural network with progressive bit search. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1211-1220, 2019.
[25] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. arXiv preprint arXiv: 2014.
[26] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Slang-Chich Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[27] Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. DeepFace: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1701-1708, 2014.
[28] Nima Tajbakhsh, Jae Y Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher Kendall, Michael B Gotway, and Jianming Liang. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions on medical imaging, 15(5):1299-1311) 2016.
[29] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933-4942, 2019.
[30] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 1.X.0:1609.08144 2016.
[31] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8724, 2020.
[32] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3903-3911, 2020.
[33] Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 13876-13885, 2020.

---

[34] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:16/2.033928 2016.
[35] Jie Zhang, Dongdong Chen, Jing Liao, Weiming Zhang, Gang Hua, and Nenghai Yu. Passport- aware normalization for deep model protection. Advances Neural Information Processing Systems, 33:22619-2222 226228, 2020.
[36] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3713-3722, 2019.
[37] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2018.
12


Reviews:
[{"review_id": "Y-cfC34Qr97", "reviewer": "Reviewer_mcZU", "summary": "The authors investigate the knowledge distillation immunity of Nasty teachers. A nasty teacher is a model that maintains good prediction performances but significantly degrades the performances of student models that distill knowledge from it. In this paper, the authors proposed a hybrid distillation that neutralizes the protection of nasty teachers, allowing the training of accurate student models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is addressing an important problem and the experiments conducted by the authors demonstrated that the proposed approach can effectively bypass the immunity of nasty teachers. \n\nMy main concern with the paper is that the ability to steal an IP from the nasty teacher is solely evaluated with the accuracy. However, accuracy can be misleading due to predictive multiplicity. That is, both teacher and student models can exhibit high accuracy while having different properties. So it would be interesting to report more details on the difference between students distilled from normal teachers and those distilled from nasty teachers. \n\nMinor remark:\nThe word \"privacy\" in the title can be misleading as it can give the impression that the paper is about the robustness of nasty teachers against privacy inference attacks, which is not the case. \n\nSection 2 has some readability issues and will benefit from a proofreading.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "SM4JbqX55I", "reviewer": "Reviewer_NwTU", "summary": "Knowledge distillation (KD) can be used by an adversary to obtain sensitive information from a teacher model presenting privacy risks. Nasty teachers were proposed to handle this problem where students were unable to learn or distill knowledge from nasty teachers. In this paper, the authors show that skeptical students using a combination of KD to intermediate shallow subsections of their model together with Self Distillation (SD) can extract information from a nasty teacher and perform well on classification tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides an attack against the nasty teacher-based defense proposed in prior work. \n\nSome of my concerns about the work are as follows:\na. The authors motivate the idea using knowledge leakage and then introduce nasty teacher as a proposed defense. However, the paper does not provide any explicit threat model corresponding to the defense. It is important to assess if the adversary is able to extract the information while adhering to the constraints of the threat model. \n\nb. Along similar lines, the metric used for showing the effectiveness of the attack is the classification accuracy. However, it is not clear what the exact leakage is due to the distillation. There is a need to decouple the two and show that indeed some sensitive information is being leaked. \n\nc. In Table 2, when a normal student is used, the drop in accuracy from the baseline is not very significant (other than when MobileNetV2 model is used for CIFAR-100 and Tiny-ImageNet). This indicates two problems: First, the baseline protection mechanism using a nasty teacher is itself not very strong. A normal student is able to still distill most of the information from the nasty teacher. Second, for models where we observe a drop in accuracy, I think there is a need to isolate the reason - is it due to the nasty teacher or the model architecture being used? The authors report improved performance with skeptical students, but from a privacy standpoint, I am not convinced this really means much. The system seems to be already leaking a lot of information. This further goes back to the point made in (b) above.\n\nd. I really like the use of self-distillation in the data-free distillation formulation. However, the improvement in test accuracy is not very significant as reported in Fig. 8. I agree that the gap is a little larger when only 20% of training data are available. Maybe, representing in tabular form will help bring out the improvement.\n\nWriting Issues:\nThe paper is well-written. Some minor issues are pointed below:\nLine 126-127: The sentence, \"One the other hand..\" is unclear. Moreover, this form of sentence structure is used to contrast with another observation. There is no such contrasting observation presented.\n\nLine 148: The notation used in Eqn (2) is confusing. How is \\Phi_{S}^{j} and  \\Phi_{S}^{'} related?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "RHOlhgbF2ma", "reviewer": "Reviewer_j2bi", "summary": "1) The main contribution of this paper is a new knowledge distillation (KD) framework called \"skeptical student\" that can circumvent the \"immunity (undistillability)\" provided by a \"nasty (defensive/secretive)\" teacher. \n\nThe concept of \"nasty\" teacher was proposed in [A] to prevent stealing of a teacher model by a malicious student. This paper proposes a countermeasure that borrows ideas from self-distillation, where knowledge is transferred to shallow subsections of the student model, each with its own auxiliary classifier (AC). The proposed skeptical student performs knowledge distillation both to the shallow sub-sections and to the full student model to overcome the impact of a nasty teacher.\n\n[A] Ma et al., \"Undistillable: Making a nasty teacher that cannot teach students\", ICLR 2021\n\n2) The paper also includes an algorithm for training skeptical students in a data-free KD scenario (the original data used to train the teacher model is not available). \n\n3) Experimental results on three datasets indicate that the proposed skeptical student can learn well both from the normal and nasty teachers. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The paper is moderately original. It uses existing concepts such as self-distillation and zero-shot knowledge transfer in a clever way to achieve the goal of training skeptical students that can learn even from defensive teachers.\n\n2) The main problem with the paper is its motivation. Under what real-world circumstances are defensive teachers necessary, especially when the training data is assumed to be known? Even in the data-free scenario, an attacker who wishes to steal IP information of the target teacher model (including data used to train the model) can carry out direct model extraction attacks, rather than relying on the knowledge distillation framework. A simple and straightforward approach to mitigate such attacks is to add random noise to the softmax values (if they ever need to be released), rather than train a defensive teacher. If the need for defensive teachers itself is not strong enough, the motivation for skeptical students is very weak.\n\n3) The key observation of the paper is that the proposed skeptical student can learn well from both normal and defensive teacher models. If that is the case, why not treat the skeptical student as a new knowledge distillation framework and analyze its properties more comprehensively purely from the knowledge distillation perspective (e.g., theoretical analysis on why this approach is better and/or experiments on more datasets to determine its strengths and limitations).\n\n4) As explained earlier, if there is likely to be any use case for defensive teachers and skeptical students, it will be only under the data-free KD scenario. However, this scenario is treated very casually both in terms of explanation and experimental results. For example, the paper claims to include attention transfer loss, which is not at all possible unless one has white/gray-box access to the target teacher model. If someone has white/gray-box access to the teacher model, why would he/she attempt to steal the IP that is already available to him/her? Even with the attention transfer loss and access to 20% of the original training data, the proposed skeptical student performs only marginally better than the regular/normal student under the data-free scenario. This further illustrates the futility of this whole approach.\n\n5) If my understanding of Figure 2 and Section 3.2 is correct, an auxiliary classifier trained with only level 1 features (shallow sub-section) of the student network works as well as the teacher model. In that case, why not simply go for a shallow student network with only level 1 features? In general, what is the impact of the depth of the student model on the proposed approach. In traditional knowledge distillation, one usually goes for shallower/simpler student architectures compared to the teacher. But if the goal is to steal the IP of the teacher, will going for shallower or deeper student networks help in any way?\n\n6) The overall organization of the paper is very confusing and does not flow well. For example, what is the need for Section 3.1 (transferability impact of nasty teachers) if it is not used anywhere else in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper can be enhanced by including strong motivating examples to justify the need for this work.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "P9Vfn2KY4Rz", "reviewer": "Reviewer_ELaj", "summary": "The paper introduces a skeptical student distillation framework that diminishes the effect of a nasty teacher. By placing auxiliary classifiers at different depths of the student model to form a shallow subsection and introducing a new self-distillation term, the resulting skeptical student clearly outperforms a vanilla student in all data-available, data-limited, and data-free settings when learning from a nasty teacher. The proposed KD method (especially the ensembled outcome) also provides similar performance as a vanilla student when learning from a normal teacher, thus being applicable in most cases. Quantitative analysis and visualizations are provided to show that the method works as expected. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: how to effectively use KD to learn from a protected nasty teacher model is an interesting new question. The self-distillation technique is not new but is applied in the new setting. Related works are well cited.\n\nQuality: The submission is technically sound and most claims are supported by empirical results. The work is complete and the comments for the evaluation experiments are fair.\n\nClarity: The paper is clearly written and should be detailed enough for reproducing the result. However, some major prior works (e.g. self-distillation network structure, details about using auxiliary classifier) are not clearly discussed and more details on those preliminaries can help readers to better understand the topic without jumping through references.\n\nSignificance: The paper discusses an interesting technique that to some extent compromises an existing model IP protection mechanism. There might be follow-up works on how to defend against such skeptical student model stealers. As also mentioned by the authors, the paper might motivate further investigations on model privacy protection.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the author clearly addressed them in the paper.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "Raise safety or security concerns: the proposed technique could be potentially used to extract model IP from black-box ML service. It diminishes the effect of existing protection approaches, which is clearly acknowledged by the author. \n"}}]

Author rebuttal:

Meta-review: Previous work defined a *nasty teacher* to be a specially trained network that yields nearly the same performance as a normal one; but if used as a teacher model, it will significantly degrade the performance of student models that try to imitate it (with the goal of protecting against model stealers). In this paper, the authors proposed an attack that neutralizes the protection of nasty teachers, allowing the training of accurate student models. The reviewers agree that this is an interesting paper, all of them supporting acceptance.
Decision: Accept (Poster)