What Knowledge Gets Distilled in Knowledge Distillation?
Authors: Utkarsh Ojha
Yuheng Li
Anirudh Sundara Rajan
Yingyu Liang
Yong Jae Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge distillation
TL;DR: This work studies different ways in which knowledge can get transferred from a teacher to a student.

Abstract: Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation?  In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. 
We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.

What Knowledge Gets Distilled in Knowledge Distillation?
Utkarsh Ojha'
Yuheng Li
Anirudh Sundara Rajan"
Yingyu Liang
Yong Jae Lee
University of Wisconsin- Maddiso
Abstract
Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's 's performance for the task hand. Over the years, there has been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.
Introduction
Knowledge distillation, first introduced in 22 12 which the 'knowledge" of teacher transferred to student. The thesis is that such a transfer (i) is possible and (ii) can help the student learn additional useful representations. 'hee seminal work by [12] demonstrated its effectivencss by making the student imitate the teacher's class probability outputs for image. This ushered in era of knowledge distillation algorithms, including those that try to mimic intermediate features of the teacher (26] 33 [3], or preserve the relationship between samples as modeled by the teacher 22
is procedure of training neural networks in
among others.
While thousands of papers have been published on different techniques and ways of using knowledge distillation, there appears to be a gap in our fundamental understanding of it. Yes, it is well-known that the student's performance on the task at hand can be improved with the help of a teacher. But what exactly is the so-called knowledge that gets distilled during the knowledge distillation process? For example, does distillation make the student look at similar regions as the teacher when classifying images? If one crafts an adversarial image fool the teacher, is the student also more prone to getting fooled by it? If the teacher invariant to a certain change in data, is that invariance also transferred to the student? Such questions have not been thoroughly answered in the existing literature.
This has become particularly relevant because there have been studies which present some surprising findings about the distillation process. (3] showed that performing knowledge distillation with
bigger teacher does necessarily improve the student's performance over that with smaller teacher,
and thus raised questions about the effectiveness of the distillation procedure in such cases. [29 showed that the agreement between the teacher and distilled student's predictions on test images
Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

class(r) classf))
Teacher
Studen:
Teacher
Student
Teacher
Student
f
f
(6/
all.ol
Figure 1: Methods used in this work. (1) KL [2] mimicking of class probabilities. (ii) Hint [26] mimicking of features at an intermediate layer. (iii) CRD [30]: features from the student and teacher for the same image constitute a positive pair, and those from different classes make up negative pair.
not that different to the agreement of those between the teacher and an independently trained student, raising further doubts about how knowledge distillation works, if works at all.
In this work, present comprehensive study tackling the above questions. We analyze three popular knowledge distillation methods [12] 26 30), Many of our findings are quite surprising. For example, by simply mimicking the teacher's output using the method of [12]. the student can inherit many implicit properties of the teacher. It can gain the adversarial vulnerability that the teacher has. If the teacher is invariant to color, the student also improves invariance to color. To understand why these properties get transferred without an explicit objective to do so, we study the distillation process through a geometric lens, where we think about the features from teacher as relative positions of an instance (i.e., distances) from its decision boundary. Mimicking those features, posit, can therefore help the student inherit the decision boundary and (consequently) the implicit properties of the teacher. We show that these findings have practical implications; e.g., otherwise fair student can inherit biases from an unfair teacher. Hence, by shedding some light on the 'dark knowledge" [ goal is to dissect the distillation process better.
Related work
Model compression (2) first introduced the idea of knowledge distillation by compressing an ensemble of models into a smaller network. [12 took the concept forward for modern deep learning by training the student to mimic the teacher's output probabilities. Some works train the student to be similar to the teacher in the intermediate feature spaces 26 33 Others train the student mimic the relationship between samples produced by the teacher 122
3 23,, that if two samples are close/far the teacher's representation, they remain close/far the student's representation. Contrastive learning has recently been shown to be an effective distillation objective in [30]. J. More recently, [I] present practical tips for performing knowledge distillation; e.g., providing the same view of the input to both the teacher and the student, and training the student long enough through distillation. Finally, the benefits of knowledge distillation have been observed even if the teacher and the student have the same architecture [S] 34). For more thorough survey of knowledge distillation, see [TO]. In this work, we choose to study three state-of-th oftte-of-thh-an art methods, each representative of the output-based, nastrrebbssed and contrastive- based families of distillation approaches.
There have been few papers that present some surprising results. [3] shows that a smaller less accurate teacher is often times better than bigger & more accurate teacher in increasing the distilled student's performance. More recent work shows that the agreement between the predictions of teacher and student is not necessarily much higher than that between the teacher and an independent student [29]. There has been work done which tries to explain why distillation improves student's performance by trying linking to the regularizing effect of soft labels [9 35) or what the key ingredients are which help in student's optimization process 24 [4] What we seek to understand in this work is different:
study different ways (beyond performance improvement) in which student becomes similar to the teacher by inheriting its implicit properties.
3 Distillation methods studied
To ensure that our findings are general and cover a range of distillation techniques, we select standard methods representative of three families of distillation techniques: output-based [12] feature-based [26], and contrastive-based [30]. The objectives of these methods, described below, combined with the cross entropy loss LOLS(Ze,
-i y, log o, (Za), where y is the

---

50.3
67.
52.9
55.0
50.2
52.9
67.4
Independent student
Distilled
1nd
KL
Hint
CRD
Ind
XL
Hint
CRD
Teucher
student
ResNet50
ResNet18
26 termed the VOG19 VGG11
Figure 2: Left: An example of how the distilled student can focus on similar regions as the teacher while classifying an image. Right: % where teacher's CAM is more similar the distilled student's CAM than to the independent student's CAM. The red line indicates chance performance (50%).
ground-trute one-hot label vector, Ze is the student's s logit output, o,(z) exp(z))/ exp(z:) is the softmax function, and e is the number of classes.
(1) KL: [2] proposed to the soft labels produced by the teacher additional target for the student to match, apart from the (hard) ground-trutie truth labels. This is done by minimizing the KL-divergence between the predictive distributions of the student and the teacher:
2x
LKL Rg 2e
jj log oj
where 2t the logit output of the teacher, and is scaling temperature. The overall loss function is LoLs + OLKL. where and a are balancing parameters. We refer to this method as KT.
(2) Hint:
FitNets [26] makes the student's ' intermediate features (fg) mimic those of the teacher's (f,) for an image 2, at some layer l. It first maps the student's features (with additional parameters r) to match the dimensions of the teacher's features, and then minimizes their mean-squared error:
Liime (f (f ((
1 1
r(
The overall loss is CCC.S + Himm where 7 and 3 are balancing parameters. teacher's intermediate representation as Hinl, and we adopt this name.
Contrastive representation distillation 30 proposed the following. Let s(x) and t(x) be the student's and teacher's penultimate feature representation for an image x. x and y are from different categories, then s(z) and t(xr should be similar (positive pair), and szx) and t(y) should be dissimilar (negative pair). key for better performance drawing a large number of negative samples N for each image, which is done using contantly updated memory bank.
(3) CRD:
N
LCRD = - log ((((x),((x))
Σ #=1
log(1 h(s()),
where Itaa b) (eab///)) and is the dot product. We use CRD to refer to this method. All other implementation details (e.g.. temperature for KL, layer index for Hint) can be found in appendix.
N) is the size
training data, is a scaling temperature,
4 Experiments
We now discuss our experimental design. To reach conclusions that are generalizable across different architectures and datasets, and robust across independent runs, we experiment with variety of maacher-studen architectures, and tune the hyperparameters so that the distillation objective improves the test performance of the student compared to independent training. For each setting, we average the results over two independent runs. We report the top-1 accuracy of all the models in the appendix. The notation Net; -> Net, indicates distilling the knowledge of Net, (teacher) into Net, (student).
4.1 Does localization knowledge get distilled?
We start by studying whether the localization properties teacher transfers student through knowledge distillation. For example, suppose that the teacher classifies image cat by focusing

---

100
15.0
56
76
76
31
Fooligg (N4
32
12
72
27
50.5
705
48
6
64
44
64
34
16.2
19
21.9
22.3
60
60
1nd
K1.
Ind
KT.
10st
CRn
1nd
K1.
10mm
CRD Ind
KL.
1811
CRN
1ed
CRD
ResNet50
ReaNe118
(b) VGG19
VGGG1
(c) VGG19 VGG11 (R13)
VGG19 RRsNett8
(0) VIT ResNet18
Figure 3: The images which fool the teacher fool the distilled student more than the independent student in (a), (b) and (d), but not in (e). If the adversarial attack is generated using a foreign network (ResNet18, (c)), it fails to convincingly fool the distilled student more than the independent one.
on its face, and an independent student classifies it as a cat by focusing on its fur. After knowledge distillation, will the student focus more on the face when classifying the image as a cat?
Experimental setup: We train three models for ImageNet classification: teacher, distilled and an independent student. For each network, we obtain the class activation map (CAM) of the ground truth class for each of random 5000 test images, using Grad-CAM [27], which visualizes the pixels that a classifier focuses on for a given class. We then compute how often (in terms of of images) the teacher's CAM is more similar (using cosine similarity) to the distilled student's CAM than to the independent student's CAM. A score of 550% means that, on average, the distilled student's CAMs become more similar to those of the teacher than without distillation. As sanity check, we also compute the same metric between the teacher's CAM and the CAMs of two independent students trained with different seeds (Ind), which should be equally distant from the teacher (score of ~50%).
Results: Fig. 2 (right) shows the results for two configurations of teacher-student architectures: (i) ResNet50 - ResNer18 and (ii) VGG19 - VGG11. For KL. and CRD, we observe a significant increase in similarity compared random chance (as achieved by the Ind baseline). Hint also shows a consistent increase, although the magnitude is not as large.
Discussion: This experiment shows that having access to the teacher's paas--robabiilitess, i.e. confidence for the ground-truth class, can give information on where the teacher is focusing on while making the classification decision. This corroborates,
some degree, the result obtained in the Grad-CAM paper (27). which showed that if the network is very confident of the presence of an object in an image, it focuses on a particular region (Fig. (cc) in (27). and when is much less confident, it focuses on some other region (Fig. 7(d) in (27)) Fig. 2 (left) shows a sample test image and the
corresponding CAMs produced by the independent student (left), teacher (middle), and distilled student (right) for the ground-truth class. The distilled student looks at similar regions as the teacher, and moves away from the regions it was looking at initially (independent student). So, regardless of the correctness of any network's 's CAM, our analysis shows that distilled student's CAM does become similar to those of the teacher for all three distillation techniques, albeit with varying degrees.
4.2 Does adversarial vulnerability get distilled?
Next, we study the transfer of a different kind of property. If we design an adversarial image to fool the teacher, then will that same image fool the distilled student more than the independent student?
Experimental setup: We train a teacher, distilled student, and an independent student for ImageNet classification. Given 5000 random test images, we convert each image / into an adversarial image Tode using iterative FGSM [9]
17 (see appendix for details), whose goal is to fool the teacher, so that teacher's 's prediction for addv changes from its original prediction for I. Fooling rate is then defined as the fraction of adversarial images which succeed at this task. In our experiments, fraction of adversarial images which fool the teacher, and apply them to different students.
only this
Results: We evaluate four configurations: (i) ResNet50 + ResNet18, 18, (ii) VGG19 + VGGI1, (iii) VGG19 > ResNet18, (iv) VIT (ViT-b-32) [4] -> ResNet18. The fooling rate is ~85% for all the teachers. Fig. 3 shows the fooling rate (y-axis) when applying these successful adversarial images to different types of students (x-axis). We see that for ResNet50 - ResNet18, the ability to fool the independent student drops to 44.2%, which is expected since the adversarial images aren't 't designed for that student. distilled students, we see an increase in the fooling rate relative to the independent one across all distillation methods (48%-52%). The trend holds for VGG19 + VGGII and VGG19

---

145
oolor strnggh
wiler slrmgg3
50
30
Agrecment (%)
Agreement
70
75
oclor streggt
(9) R30 R18
(b) R50 - R18
(c) Swin-TT R18
(4) R18 (Hint ablation)
Figure 4: (a) Agreement between two images with different color properties. * indicates distillation done by a teacher T not trained to be color invariant. (b, c) Agreement between two images having increasingly different color properties. (d) Effect of different layers used in iint distillation.
+ 88 Fig. (bb d). When distillation is done from transformer to CNN (ViT ResNet18) the fooling rates remain similar for the independent and distilled students; Fig. 3 (e). Here, we don't t the student becoming similar to the teacher to the extent observed for CNN CNN distillation.
Discussion: This result surprising. Iterative FGSM is a white box attack, which means that has full access to the target model (teacher), including its weights and gradients. Thus, the adversarial examples are specifically crafted to fool the teacher. In contrast, the student never has direct access to the weights and gradients of the teacher, regardless of the distillation method. Yet. by simply trying to mimic the teacher's ' soft probabilities (K L) or an intermediate feature layer (H intt CRD), the student network inherits, some extent, the particular way that teacher is fooled.
We conduct an additional study to ensure that the reason the distilled student is getting fooled more is because it is being attacked specifically by its teacher's adversarial images; i.e., if those images are designed for some other network, would the difference in fooling rates still be high? We test this in the VGG19 + VGG11 setting. This time we generate adversarial images (T + Todo) to fool an ImageNet pre-trained ResNet18 network, instead of VGG19 (the teacher). We then use those images to attack the same VGG11 students from the VGG19 + VGGII setting. In Fig. (c), we see that the fooling rates for the independent and distilled students remain similar. This indicates that distillation itself does not make the student more vulnerable to any adversarial attack, and instead, increase in fooling rate can be attributed to the student's ' inheritance of the teacher's 's adversarial vulnerability.
4.3 Does invariance to data transformations get distilled?
We have studied whether the response properties on single images get transferred from the teacher to the student. Now suppose that the teacher is invariant to certain changes in data, either learned explicitly through data augmentation or implicitly due to architectural choices. Can such properties about changes in images get transferred during distillation?
We study color invariance as a transferable property. We train three models for ImageNet classification: a teacher, a distilled and an independent student. While training the teacher, we add color jätter in addition to the standard augmentations (random crops, horizontal flips). Specifically, we alter an image's brightness, contrast, saturation, and hue, with magnitudes sampled uniformly in [0, 0.4] for the first three, and in [0, 0.2] for hue. This way, the teacher gets to see the same image with different color properties during training and can become color invariant. When training the student, we only use the standard augmentations without color jittering, and see whether such a distilled student can indirectly inherit the color invariance property through the teacher.
Experimental setup:
Results: We start with the ResNer50 - ResNet18 configuration. After training, we evaluate the models on 50k ImageNet validation images. For each image X, we construct its augmented version X' by altering the color properties in the same way as done while training the teacher. Fig. 4 (a) depicts the agreement scores between X and X' (y-axis) for different models (x-axis). The teacher (T), being trained to have that property, achieves high score of 92.5%. The independent student, which is not trained to be color invariant, has a much lower score of 71.3%. But, when the color- invariant teacher is used to perform distillation using KL or CRD, the agreement scores of the students jump up to 82. % and 79.5% respectively. To ensure that this increase is not due to some regularizing property of the distillation methods that has nothing to do with the teacher, we repeat the experiment, this time with a ResNet50 teacher (T*) that not trained with color augmentation. The new agreement scores for the distilled students (marked by #; e.g., KL*) drop considerably
5

---

find BKL. Mine nCRD
fnd BKL Bist nCRD
Teacher
Student
14
Consenvar (16)
KD: natural magos
Similar unseen
sbrrch
styllaed
000 wolfee
Sterkt
syRlzed
slBoosete
olge
0ce coffee
dornaine?
(4) VGG19 - ResNet18
(b) Swin-Hase
Swin-Tiny
Figure 5: Left: Does knowledge transferred about one domain give knowledge about other unseen domains? Right: Consensus scores between teacher and student for images from unseen domains.
compared to the previous case. This is strong indication that the student does inherit teacher-specific invariance properties during distillation.
In Fig. 4 (), we show that this trend in agreement scores (y-axis) holds even when the magnitude of change in brightness, contrast and saturation (x-axis) increased to create X' from X. We repeat this experiment for Swin-tiny (a transformer) 20 + ResNet18 in Fig. [(.. We again see some improvements in the agreement scores of the distilled students. This increase, however, not as significant as that observed in the previous CNN -> CNN setting. Throughout this work, we find that distilling the properties from a transformer teacher into a CNN student is difficult. The implicit biases introduced due to architectural differences between the teacher (transformer) and student (CNN) seem too big, as was studied in (25), to be overcome by current distillation methods.
We also note the ineffectiveness of Hint in distilling this knowledge. Our guess for this is the choice of for distillation, which is typically set be in the middle of the network as opposed deeper layers where KLL and CRD operate. So, we perform an ablation study for Hint, where the student mimics a different feature of the teacher each time; starting from a coarse feature of resolution 56 x 56 (F1) to the output scores (logits) of the teacher network (F7). We plot the agreement scores in Fig. Md), where we see that the score increases as we choose deeper features. Mimicking the deeper layers likely constrains the overall student more compared to middle layer, since in the latter case, the rest of the student (beyond middle layer) could still function differently compared to the teacher.
In sum, color invariance can be transferred during distillation. This is quite surprising since the teacher is not used in a way which would expose any of its invariance properties to the student. Remember that all the student has to do during distillation is to match the teacher's 's features for an image X; i.e. the student does not get to see its color augmented version, X'. during training. If so, then why should it get to know how the teacher would have responded to X"?
Discussion:
4.4 Does knowledge about unseen domains get distilled?
So far, our analysis has revolved around the original ImageNet dataset, something that was used to perform the distillation itself. So, does the teacher only transfer its knowledge pertaining to this domain, or also of domains has never seen (Fig. left)?
To test this, we first perform distillation using two settings (i) VGG19 -> ResNet18 and (ii) Swin-Base 200 Swin-Tiny, where the training of the teacher, as well as distillation is done on ImageNet. During test time, we take an image from an unseen domain and see how frequently the student's and teacher's class predictions match for it (regardless of whether the predicted label is correct or incorrect), which we call the consensus score. For the unseen domains, we consider the five datasets proposed in 77: sketch, stylized, edge, si silouutte, cue conflict. Images from these domains are originally from ImageNet, but have had their properties modified.
Experimental setup:
Fig. 5 (right) shows the consensus scores between the teacher and the student (y-axis). We see a nearly consistent increase in consensus brought about by the distillation methods in both settings. The extent of this increase, however, differs among the domains. There are cases, for example, in VGG19 >> ResNet18, where consensus over images from edge domain increases over 100% (12% + 33%% by some distillation methods. On the other hand, the increase can also be relatively modest in certain cases: e.g. Swin-Base - Swin-Tiny in stylized domain.
Results:
Discussion: 29 showed that the agreement in classification between the teacher and distilled student not much different than that to an independent student on CIFAR. We find this to be true in our ImageNet classification experiments well. That is why the increase in agreement observed for

---

100
100
Teaeber
Studeut
Top- sccuracy (%)
96
7
72
70
CRD
Ind Him CRD
CRD
Knowledge three
Distilled on
domains
domain
MNIST-Orig
MNIST-Color
MNIST-M
The teacher is trained on three domains: MWIST-Orig, MNIST-Color, and MNIST-M Distillation is done only on MNIST-Or -Org.. Right: Test accuracy of the students. Note the increase in performance on MNIST-Color & MNIST-M domains by the distilled students.
Figure Left:
unseen domains is surprising. After all, if the agreement is increasing when images are from the seen domain, why should it increase when they are from an unseen domain? It is possible that there more scope for increase in agreement in unseen domains seen domain. The consensus score between teacher and independent student for the seen domain is 775% (appendix), whereas for an unseen domain, e.g. sketch, <40%. So, it might not be that knowledge distillation does not work, as the authors wondered in [29]. but its effect could be more prominent in certain situations.
4.5
Other studies
In appendix, we study additional aspects of model such as shape/texture bias, invariance to random crops, and find that even these obscure properties can transfer from a teacher the student. We also explore the following questions: If (e.g. using crafted soft labels), will that student gain similar knowledge as a distilled student? distillation cannot increase the student's performance, is there knowledge transferred? Finally, also present results on some more datasets like CIFARIOS [16), VLCS and PACS (18), showing that the phenomena of the implicit transfer of properties dueing knowledge distillation extends even datasets beyond ImageNet and MNIST.
find alternative ways of increasing student's performance
Applications
Beyond the exploratory angle of studying whether certain properties get transferred during distillation, the idea of student becoming similar to teacher in a broad sense has practical implications. We discuss good and a bad example in this section.
5.1 The Good: the free ability of domain adaptation
Consider the following setup: the teacher trained for task by observing data from multiple domains (D, U D2). It is then used to distill knowledge into student on only D1. Apart from getting knowledge about D1, will the student also get knowledge about D2 indirectly?
Experimental setup: We use MNIST digit recognition, and train the teacher on three domains: (i) MNIST-orig: original gcay- images from MNIST, (ii) MWIST-Color: background of each image randomly colored, and (iii) MNIST-M M [6]: MNIST digits pasted on random natural image patches. The student models are trained only three domains. The network architecture is same for both the teacher and the student (see appendix).
WWIST-rrig and evaluated (top-1 accuracy) all
When the independent student is trained only on MWIST-orig, its performance drops on the unseen domains, which is expected due to domain shift. The distilled students (especially L & Hint), however, are able significantly improve their performance on both unseen domains; Fig.
Results:
This result shows distillation's 's practical benefits: once a teacher acquires an ability through computationally intensive training (e.g., training multiple datasets), that ability can be distilled into a student, to a decent extent, through much simpler process. The student sees the teacher's response to gray-scale images (MNIST-oria that lack any color information. But that information helps the student deal better with colored images (e.g., MNIST-Color), likely because the teacher has learned
Discussion:
representation (e.g., shape) which is distilled to the student.
7

---

5.2 The Bad: Students can inherit harmful biases from the teacher
Consider the problem of classifying gender from human faces. Imagine an independent student which performs the classification fairly across all races. The teacher, on the other hand, is biased against certain races, but is more accurate than the student on average. Will the student, which was originally fair, become unfair after mimicking the unfair teacher?
We consider a ResNet20 + ResNet20 setting, and use FairFace dataset [5 which contains images of human faces from 7 different races with their gender labeled. From its training split, we create two different subsets (D, and De)) with the following objectives - (i) D3 has a particular racial composition
Experimental setup:
that model trained on it will perform fairly across all races during test time; (ii) D23 is composition is intended to make the model perform unfairly for certain races. The exact composition of D. and Dj is given in the appendix. The teacher is trained on D. whereas the independent/distilledd students are trained on D,, We use K L for distillation.
Results: We observe in the figure the right that the independent student performs roughly fairly across all races, which is what we intended. The teacher is more accurate than the student, but performs relatively poorly on faces from Race 3 compared to others. After distillation, observe that the student's overall accuracy improves, but the gain in accuracy is less for images from Race 3. So, when is trained on D. by itself, it behaves fairly. But when it mimics the teacher on D., becomes unfair. This follows the observation made in (2D. where the authors found that the performance gains during distillation might not be spread uniformly across different sub-categories.
Fair student
Distilled student @ Unfair Teacher
Race
Race
Race 3
Race 4 Race
Race
Race 7
The practical takeaway from the experiment is that knowledge distillation can bring forth behaviour which is considered socially problematic if it is viewed simply as blackbox tool to increase a student's 's performance on test data, as it did in the previous example. Proper care must hence be taken regarding the transfer/amplification of unwanted biases in the student.
Discussion:
Why does knowledge distillation work in this way?
An Illustrative Example. Why should teacher's 's response an image contain such rich infor- mation about its implicit properties? We first intuitively explain this through a toy classification problem using K L as the distillation objective. Fig- (lg shows data points from two classes (red and blue). The teacher has access to the complete set, thereby learning to classify them appropriately (orange decision boundary). On the other hand, suppose that the training of the independent and distilled student is done on a subset of the points. The independent student can learn to classify them in different ways, where each decision boundary can be thought of looking for different features, hence giving rise to different properties. But the distilled student is trying to mimic the teacher's s class probabilities, which reflect the distance from the (orange) decision boundary of the teacher, visualized as circles around the points. Then the decision boundary of the distilled student should remain tangential these circles so that it preserves the teacher's 's distances. So, this very simple case depicts that mimicking the class probabilities can force the student's 's decision boundary to resemble that of the teacher, which is not true for the independent student.
mathematical formulation of the same case can be considered as follows. Let D be a linearly separable dataset: D e {(11 y1), (I2,, 92). ...... (In, Dn) æ E RL y [-1, 13). For an arbitrary linear classifier having weights W E 2 and biases E R, the binary classification objective for the teacher and the independent student can be described as V(z 31) E D, (W" x, + b)y; >
Since the dataset is linearly separable, both the teacher as well as the independent student can solve this system of inequalities in many ways. Let (W,, by} } be the parameters of one particular teacher.
Next, we look at the objective of the distilled student, who tries to map x; to 2; = W/ ar; + be, instead of yı. That is, the optimal solution satisfies: Viri, =x) E Daist, W x; + b, = z;. This system of linear equations will have a unique solution provided it has access to at least m + training instances
and that these instances are linearly independent. Since we know that the teacher's weights and
8

---

Decision boundaries
Teacher
Independent student
Distilled student
Teacher
Independent student Distilled student
Figure 7: Left. Teacher learns on the the entire dataset. Independent student without access to the full dataset might learn some spurious correlations. Distillation provides constraints to reconstruct the teacher's s decision boundary. Right. Sampled decision boundaries for points from MNIST-Color visualized using 28
The distilled student better reconstructing the teacher's decision boundary.
biases, (W, by ). can satisfy this equation, we can conclude that if there exists a unique solution, then W. W and b, bt So, the distilled student, when solving the distillation objective, will recover the weights and biases of the teacher, and consequently its decision boundary.
Complex neural networks: Given that the distances of data points, i.e., decision boundary, can be preserved by the distilled student in the very simple case presented above, we now study whether they can be preserved in more complex setting involving multi-layer neural networks as well.
There have been works 8 [28] which have studied this for the KL objective: if the teacher and the students are all trained on the same dataset D, the decision boundary of the teacher will be more similar to the distilled student than the independent student for points from D. And while this result useful, it is not clear whether (i) this can help explain why the student inherits teacher's knowledge about other domains, and (ii) whether this holds true for distillation objectives other than KT.
To study this, we revisit the setup of Sec. 5.1 where the teacher was trained on three domains and distillation was performed on one (Fig. left). We study what happens to the decision boundary similarity between the students and the teacher all three domains. To compute that similarity between two neural networks, we use the method introduced in [28]. We randomly sample a triplet of data points from given domain and create plane that passes over that triplet. The networks that are being compared receive the same dataset points in this plane, and we compute intersection over union between their predictions. We report the score averaged over 50 triplets.
Ind
KL
Hint
CRD
MNIST -
0.9107
0.9341
0.9317
0.9062
MNIST-Color
0.4156
0.6711
0.7980
0.4733
MNIST-M
0.4691
0.5872
0.6541
0.4862
Table 1: Decision boundary similarity calculated between students and the teacher.
Results: Table shows the similarity scores, where we see that the distilled students' decision boundary are almost always more similar to the teacher's than the independent student's. This is particularly evident when the scores are computed for domains not seen by the students; e.g., for MNIST-Color, the similarity score increases from 0.416 to 0.671 for the student distilled with KT.
Discussion: First, we can draw an analogy of this experimental setup with the toy example discussed before. The three MNIST related domains (for training the teacher) are similar to the overall set of blue+red points used train the toy teacher (Fig. leftt. The singular MNIST-orig domain (for training the students) is similar to the three points available to train the toy students. Now, what the results from Table show that the data points available for distillation can help determine the teacher's decision boundary even for points which the student did not have access to. So, similar to how the decision boundary estimated by the toy student can correctly estimate the distances of missing blue/red points, the neural network based student can also estimate the distances of points from unseen domains, e.g., MNIST-color, teacher's 's behavior these points. An example of this is given in Fig. 7 (right).
similar way as its teacher, thereby inheriting the

---

7 Limitations
While we have tried study the distillation process in many settings, there do exist many more which remain unexplored. This is because the analysis conducted in our work has many axes of generalization: whether these conclusions hold (i) for other datasets (e.g. SUN scene classification dataset), (ii) in other kinds of tasks (object detection, semantic segmentation), (iii) in other kinds of architectures (modern CNNs, like ResNext). (iv) or for more recent distillation objectives, etc. Moreover, in Sec. 3.5 in the supplementary, we explore the transferability of shape/texture bias from teacher to student, and find some discrepancy different distillation objective's s abilities. Therefore, the conclusions drawn from our work will be more helpful if one can study all the combinations of these factors. Due to limitations in terms of space and computational requirements, however, we have not been able to study all of those combinations.
Conclusion
Knowledge distillation is beautiful concept, but its success i.e. increase in student's accuracy. has often been explained by a transfer of dark knowledge from the teacher to the student. In this work, we have tried to shed some light on this dark knowledge. There are, however, additional open questions. which we did not tackle in this work: given the architectures of the teacher and student, is there a limit on how much knowledge can be transferred (e.g., issues with ViT - CNN)? If one wants actively avoid transferring certain property of the teacher into a student (Sec. (.2c. but wants to distill other useful properties, can we design an algorithm tailored for that? We hope this work also motivates other forms of investigation to have even better understanding of the distillation process.
Acknowledgement
This work was supported in part by NSF CAREER IIS2150012, and Institute of Information com- munications Technology Planning & Evaluation(IITP) grant funded by the Korea governmen((MSIT) AI Autonomy and Knowledge Enhancement for Al Agent Collaboration), Air Force Grant FA9550-184 1--00118----666 the National Science Foundation (NSF) Grants 2008559-IIS, 2023239-DMS, and CCF-2046710.
(No. 2022-0-00871, Development
References
[1] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In arXiv, 2021.
[2] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD, 2006.
[3] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv, 2020.
[5] Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anand- kumar. Born again neural networks. In JCML, 2018
[6] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversaria training of neural networks. JMLR, 2016.
[7] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In NewrIPS, 2021.
[8] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
10

---

[9] Ian Goodfellow, Jon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2014.
[10] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey, In arXiv, 2021.
[11] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Dark knowledge. TTIC Distinguished lecture series, 2014.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPS Deep Learning Workshop, 2014.
[13] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. In arXiv, 2017.
[14] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In JCML, 2021.
[15] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In WACV, 2021.
[16] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. In Technical report, Citeseer, 2009.
[17] Alexey Kurakin, Ian Goodfellow, and Sammy Bengio. Adversarial machine learning at scale. In arXiv, 2016.
[18] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
[19] Yuan Li, Francis T.H.Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020.
[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[21] Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases distillation. In arXiv, 2021.
[22] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
[23] Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang, Correlation congruence for knowledge distillation. In ICCV, 2019.
[24] Max Phuong and Christoph H. Lampert. Towards understanding knowledge distillation. In ICMI., 2019.
[25] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy, Do vision transformers see like convolutional neural networks? In NeurIPS, 2021.
[26] Adriana Romero, Nicholas Ballas, Samira Ebrahimi Kahau, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In JCLR, 2015.
[27] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IJCV, 2019.
[28] Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh- Yeh-Chiang, Yehuda Dar, Richard Bara- niuk. Micah Goldblum, and Tom Goldstein. Can neural learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. arXiv preprint arXiv:2203. 08/244..844.... 2022.
[29] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In NeurIPS, 2021.

---

[30] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[31] Frederick Tung and Greg Mori. Similarity-preserviin knowledge distillation. In ICCV, 2019.
[32]
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[33] Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. ICIR, 2017.
[34] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning, In CVPR, 2018.
[35] Zhilu Zhang and Mert Sabuncu. Self-distillation as instance-specific label smoothing. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2184-2195. Curran Associates, Inc., 2020.
12


Reviews:
[{"review_id": "NGtvBN08ui", "reviewer": "Reviewer_QC8K", "summary": "The paper designed a series of experiments to fundamentally dissect and understand knowledge distillation. The key questions include, Does localization knowledge get distilled? Does adversarial vulnerability get distilled? Does invariance to data transformations get distilled? Does knowledge about unseen domains get distilled? Can students inherit harmful biases from the teacher? and the main content of this paper is to design experiments to answer these questions empirically. At last, the paper provides some intuition about why knowledge distillation works in this way, and concludes with open questions on how to leverage such findings. \n", "questions": "The intuition presented in section 6 could be formalized lowering variance as in the following paper? \nMenon et al, “A Statistical Perspective on Distillation”, ICML 2021.\n", "limitations": "yes ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposed to comprehensively study the critical, interesting and fundamental problem about deep understanding of knowledge distillation, from an angle of what properties of the teacher are distilled into students, which is a novel angle.\n\nThe paper extracted the key questions in order to understand knowledge distillation, where those questions are interesting and essential to help the problem better. Also, for each question, the paper designed corresponding experiments on the representative models with standard datasets.\n\nThe paper is easy to follow.\n", "weaknesses": "Even though the paper proposed to study a critical fundamental problem, the paper focused more on designing the experiments to answer the potential questions that could help understand the questions. However, many answers towards the questions are not that surprising, which falls in the expectation. Purely relying on some empirical results to facilitate the understanding of the questions limits the contribution of this paper. If the paper could have some theoretical results to demonstrate why it works and when it works, it could solidify the contribution of this paper.\n\nFor each experiment, it’s unclear whether the conclusion could still hold for various architectures or datasets. So it would be great that the paper could include more representative architectures and broader datasets to make results more convincing. \n\nThe contributions of this paper would be more solid if there are some attempts to answer some of the open questions in the conclusion part, e.g. how to design algorithms to control which properties to distill or not distill. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. However, many answers towards the questions are not that surprising.**\n\nWhile we agree that an observation pertaining to knowledge distillation being surprising could be rather subjective, we want to reiterate the context in which one should look at our results. As explained in lines 30-36, there have been popular works which have casted doubt if knowledge distillation is really working at all, if it cannot improve the accuracy of the student or if it fails to improve the student and teacher's agreement on class predictions. And because of those suspicions, we believe that it *is* rather surprising that so much can happen when a student is being trained to mimic a teacher, even when the student's accuracy doesn't seem to improve (please see Sec. 3.1 in the appendix).  \n\n\n**2. For each experiment, it’s unclear whether the conclusion could still hold for various architectures or datasets. So it would be great that the paper could include more representative architectures and broader datasets to make results more convincing.**\n\nIn the main paper, we have presented results using various combinations of neural networks for teacher and student models. A list of architectures explored:\n1. Convolutional Neural Networks\n- ResNet50\n- ResNet18\n- ResNet20\n- VGG19\n- VGG11\n- We investigate simplistic CNN architectures as well (MNIST setting; Sec. 5.1)\n2. Vision Transformers\n- ViT (ViT-b-32)\n- Swin-Base and Swin-Tiny\n\nAdditionally, we have included experiments involving different types of CNNs in our rebuttal.\n- Wide-ResNets (WRN-40-2) \n- ShuffleNets (ShuffleNetV1)\n- MobileNets (MobileNetV2)\n\nWhile it is true that due to computational and page limit constraints, we could not study all the properties for every combination of architectures (please see our response to reviewer **r3A2**’s point about adding a weakness section), our goal in this work has simply been to show the possibility of the transferability of many properties from a teacher into a student network. \n\nFurthermore, we report results on some additional datasets as well (CIFAR-100, VLCS and PACS); please see our response to reviewer **B1Ad**.\n\n**3. The intuition presented in section 6 could be formalized lowering variance as in the following paper? Menon et al, “A Statistical Perspective on Distillation”, ICML 2021**\n\nWe thank the reviewer for bringing our attention to the referenced work. The paper effectively argues that using one-hot labels in the training objective significantly reduces granularity. Moreover, the paper provides a mathematical proof establishing that matching the true probabilities derived from the original data distribution results in a noteworthy reduction in the variance of empirical risk, thereby resulting in improved generalization. Additionally the paper shows that a good teacher must be capable of offering an accurate approximation of these true probabilities.\n\nNonetheless, we wish to underscore the primary objective behind the intuition presented in Section 6 of our manuscript. Rather than aiming to showcase the utility of knowledge distillation in bolstering generalization, this section primarily aims to elucidate a distinct phenomenon: the potential to reconstruct the precise decision boundary of a teacher even when operating with a limited subset of data points. We furthermore extend this to empirically show that the decision boundary is well approximated even in the cases of deep neural networks. This, we believe, will aid the reader in developing an intuitive understanding of why certain implicit properties are transferred.\n\nThat being said, we will include a discussion of this work in the camera ready into a section which deals with different ways in which people have tried to explain why knowledge distillation helps improve the student's performance.\n\n"}, {"review_id": "4lT9AUOJau", "reviewer": "Reviewer_B1Ad", "summary": "The paper explores different ways in which a student network can inherit implicit properties from a teacher network beyond just improving task performance. The authors study various distillation methods, including output-based, feature-based, and contrastive-based techniques, and show that these methods can indirectly distill properties. The paper also investigates why knowledge distillation works and discusses its practical implications. Overall, the paper aims to shed light on the \"dark knowledge\" of knowledge distillation and motivate further investigation into the distillation process.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper sheds light on the dark knowledge of knowledge distillation (KD) by investigating a series of questions in KD and gives interesting findings.\n2. Each sub-section in experiments is well-organized, and the question is answered with clearly designed experiments and supportive results, followed by insightful discussions. \n3. The paper is well-written and the presentation is decent. ", "weaknesses": "1. The major weakness of this paper is the setup appears to be more exploratory in nature. While exploratory experiments can certainly provide valuable insights and help to understand the findings, they do not substitute for a thorough evaluation that generalizes across diverse datasets. In this paper, the use of only a few datasets may not present a comprehensive picture and could yield misleading results.\n2. The selection of benchmark methods in this paper, while representative of some standard approaches, is notably restricted. The experimental comparison lacks several recent knowledge distillation (KD) methods, leaving the evaluation somewhat incomplete. The inclusion of these recent advances in KD techniques would not only enhance the comprehensiveness of the evaluation but also deepen our understanding of the KD process.\n(Ren et al., 2022) Better Supervisory Signals by Observing Learning Paths.\n(Chen et al., 2021) Distilling Knowledge via Knowledge Review.\n(Zhao et al., 2022) Decoupled Knowledge Distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments about the thoroughness of the experiments.\n\n\nThe reason we chose a single dataset, ImageNet, for most of our experiments is because it is a very large scale dataset of relatively high resolution images. Its ubiquity has led to many different kinds of architectures being developed to be trained on it: from CNN variants (ResNet/VGGs) to transformer variants (ViT/Swin). Because of this, the computer vision/machine learning community has often been interested in the properties of neural networks trained on ImageNet; e.g., whether the network is shape/texture biased [1], how robust is it to adversarial perturbations [2], or whether it generalizes to unseen domains, interpretability of those models [3]. Hence, if one is interested in studying many different aspects of “knowledge” being distilled into a student, it is more conceivable to do so in an ImageNet setting.\n\nThat being said, we do agree that having some results on more datasets will help strengthen the conclusions. Accordingly, we conducted the adversarial vulnerability experiment on CIFAR-100 (section 4.2 in the main paper). We report the results on three different teacher-student settings: (i) Wide ResNet 40-2 (WRN-40-2) -> ShuffleNetV1,  (ii) VGG13 -> VGG8, (iii) ResNet50 -> MobileNetV2. Both the teacher and the students are trained on the training split of CIFAR-100, and tested on 5000 random images from the test split. The results shown below depict the fooling rates (in %) of different kinds of students when using adversarial images crafted for the teacher. We see that the fooling rate increases for distilled students, following a similar pattern as in Sec. 4.2 - Fig. 3. So, adversarial vulnerability of the teacher does get distilled into the students trained with different distillation objectives.\n\n|  | WRN 40-2 &rarr; ShuffleNetV1| VGG-13 &rarr; VGG-8 | ResNet50 &rarr; MobileNetV2 |\n|-----------|:-----------:|:-----------:|:-----------:|\n| Independent | 31.23 | 42.42 | 36.57 |\n| KL | 48.62 | 51.87 | 43.32 |\n| Hint | 62.63| 49.79 | 43.91 |\n| CRD | 49.41| 54.68 | 46.07 |\n| DKD | 57.19| 60.01 | 47.03 |\n\nFurthermore, we also follow the reviewer’s suggestion to report the results on a more recent distillation objective. Specifically, of the works mentioned by the reviewer, we consider the most recent one - Decoupled Knowledge Distillation (DKD) (Zhao et al. 2022) - and present how well it transfers this knowledge, in the last row. We see that the conclusions hold even for this objective. In fact, in two out of the three settings, DKD transfers the adversarial vulnerability the most compared to the other distillation objectives. This gives a further sign that the various ways in which the community has tried to distill the knowledge from a teacher into a student does more than improve the student’s performance. In our camera ready, we will try to include the analysis with this new distillation objective for all the experiments in the paper.\n\nNext, similar to the MNIST domain adaptation experiment in Section 5.1, we conduct the experiment on a more real-world domain. We consider two datasets: VLCS and PACS [4]. VLCS\nconsists of images from four domains - VOC2007, LabelMe, Caltech-101, and SUN - where in each domain there are images belonging to five categories. PACS consists of images from four domains as well - sketch, photo, cartoon, art painting. Each domain consists of the same seven object categories. The teacher is trained on all the four domains, but the students (independent and distilled) are trained on three domains (images from one domain are never shown). The unseen domains are Caltech 101 and Photo when working with VLCS and PACS datasets respectively.\n\nThe goal is to see if mimicking the teacher on three domains also helps the student inherit teacher’s information on the fourth (hidden) domain; i.e., do we see an improvement in the accuracy on that hidden domain for the distilled students, compared to an independent student. The results, depicting the classification accuracy of different models, are shown below. The distilled students’ performance improves on the unseen domains in both the cases, simply by having access to teacher’s responses on the other three domains. This is particularly pronounced when distillation is done using KL. For example, in VLCS, the performance on the unseen domain (Caltech 101) improves from 54.31 by the independent student to 71.83 by the KL distilled student. \n\n| *VLCS* | Caltech 101 (unseen) | LabelMe (seen) | SUN09 (seen) | VOC 2007 (seen) |\n|-----------|:-----------:|:-----------:|:-----------:|:-----------:|\n| Independent | 54.31|\t61.07|\t60.38|\t51.94 |      \n| KL | 71.83|\t60.73|\t61.52|\t53.12 |\n| Hint | 67.95|\t60.84|\t60.34|\t52.76 |\n| CRD | 61.66|\t63.57|\t55.31|\t53.12 |\n\n| *PACS* | Photo (unseen) |\tSketch (seen)|\tCartoon (seen)|\tArt (seen) |\n|-----------|:-----------:|:-----------:|:-----------:|:-----------:|\n| Independent | 40.91|\t70.17|\t66.47|\t45.56 |      \n| KL |  49.93|\t72.82|\t69.69|\t46.64|\n| Hint | 48.34|\t71.13|\t68.08|\t45.89 |\n| CRD | 44.49|\t71.52|\t67.14|\t47.86 |\n\n\n\n\n**References**\n\n[1] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Geirhos et al. ICLR 2019. \n\n[2] Explaining and Harnessing Adversarial Examples. Goodfellow et al. arXiv 2015\n\n[3] Learning Deep Features for Discriminative Localization. Zhou et al. CVPR 2016.\n\n[4] Deeper, Broader and Artier Domain Generalization. Li et al. ICCV 2017.\n\n\n\n\n"}, {"review_id": "6sRF28tSlQ", "reviewer": "Reviewer_r3A2", "summary": "The paper presents an empirical study to analyze inner workings of knowledge distillation (KD) methods. The authors try to understand the nature of the knowledge \"distilled\" from a teacher to a student by studying how various properties get transferred in this learning. The properties include (a) localization information, (b) adversarial vulnerability, (c) invariance to data transformations and (d) domain adaptation. There is further analysis about some other properties in the supplement. The paper analyzes three broad distillation methods on ImageNet with multiple CNN and transformer architectures. Lastly, the authors present positive/negative applications of this understanding for specific tasks (domain adaptation, bias transfer).", "questions": "Questions: \n\n(1) **Other datasets/tasks**: How well do the results generalize over other datasets and different tasks (small/large scale images, fine-grained recognition, scene classification etc.). I understand that the number of possible experiments is too large and the paper can't cover everything. However, the same way the paper deserves credit for coverage in terms of distillation methods and architecture selection of teacher and student, it also needs to confirm/study if these results hold for other settings. At least a couple more settings should be included. This is specially important as there are some non-intuitive results and demand further analysis to be confirmed. Also, it would be great if you could illustrate domain adaptation application for some non-MNIST data too.\n\n(2) **Properties selected**: While I am assuming the current set of properties were selected as these are some popular topics of study in ML, I would like to know your rationale behind selecting them. For example did you also consider assessing if the uncertainty of teacher gets transferred to student? \n\n(3) **More questions about experiments**: Does a mixture of simultaneous KL + Hint or other mixtures provide stronger distillation of the properties? Is the localization transfer for transformer -> CNN poor? You could use some other saliency map as localization proxy if grad-CAM is not suitable. ", "limitations": "The authors mention \"We do not have a separate section for limitation, but since ours is an analysis paper, we are discussing the open questions all throughout the paper.\" While I understand their reasoning and thought process, I also believe they should summarize the limtiations of their experimental setup in the conclusion or a separate section. It is an experimental study and it can't possibly conduct all experiments as part of it. The open questions raised throughout the paper I believe or more related to depth of the analysis. The authors should certainly add the limitations regarding breadth of their analysis. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n\n(1) The paper is nicely organized and very well written. The motivations to study the topic are clear. To my knowledge, this analysis is quite original.\n\n(2) The authors investigate some relevant and interesting directions in a modular way to study KD.\n\n(3) Multiple different architectures and distillation techniques are carefully analyzed for various properties and generally demonstrate consistent results. \n", "weaknesses": "Weaknesses:\n\nThe weaknesses I see stem from the purely empirical nature of the study. \n\n(1) The generalizability of results to other types of datasets, tasks, domains remains unknown. (Major)\n\n(2) The selection of properties studied seems a little arbitrary. (Minor)\n\n(3) There is lack of any theoretical analysis to characterize the distilled knowledge. (Minor)\n\nOverall, I like the paper but can't favour stronger acceptance yet. I enjoyed reading through it. It is structured well, the experiments are quite thorough (except in one aspect!) and give some interesting insights about KD. Among the weaknesses I highlighted, I won't hold the last two weaknesses strongly against the paper as (3) is a difficult topic in itself and the studied set of properties could still be regarded as reasonably comprehensive in terms of popular ML topics today. However, considering the purely empirical nature, weakness (1) is major for me that should be tackled before I can more clearly favor acceptance. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Generalization over other datasets and different tasks**\n\nWhile we agree that results on other datasets can help bolster our conclusions (more on that later), we want to first clarify why we believe ImageNet is a comprehensive test bed for the kind of properties we have wanted to study in this work. Please take a look at our response to reviewer **B1Ad**, where we explain our reasoning, and also present the results on some of the new datasets (CIFAR, VLCS and PACS).\n\n**2. Rationale behind selecting the particular properties**\n\nOur goal was to study “any” property which is not explicitly encoded in the objective function. Conversely, for example, if a distillation objective was trying to make the class activation map of the student similar to the teacher, then studying CAM similarity (as in Sec. 4.1) will not be as interesting, since one would expect the CAM to become similar. \n\nAmong those potential properties (not encoded in the objective function), as the reviewer suggested, we did end up going with the ones that have attracted attention in the ML community. In particular, we wanted to study those properties which, based on our natural thinking, should have remained hidden from the student. For example, as we mention in lines 155-157, why should access to the teacher's class probability output for an image tell us how it gets fooled by an adversarial image?\n\nOn a practical end (Sec. 5), we wanted to study those properties which will likely have a real-life application. \n\nOverall, our goal is not to say that these are the properties which are important which should be  studied; but more to show that there can be much more going on in the backend of a distillation process than simply improving a student’s accuracy. And it is very much possible that there can be some even more interesting properties that we might have missed, but something that other people can study being motivated by our work.\n\n**3. Mixture of distillation objectives**\n\nDue to time constraints, we were only able to try one setting, where we combined CRD and KL objective in the case of WRN-40-2 -> ShuffleNetV1 distillation on CIFAR-100 dataset, and studied the transfer of adversarial vulnerability from a teacher into the student. The table below shows the results, where we see that using a combination of KL + CRD objectives for distillation results in better knowledge transfer, compared to using them individually. We will investigate this further for other experiments in our work in the camera ready version. But what this experiment tells us is that the knowledge extracted by one distillation objective (e.g., CRD) is not a superset of knowledge extracted by a different distillation objective; there is something new that each distillation objective adds to the inherited knowledge. We thank the reviewer for asking for this interesting experiment.\n\n|  | WRN 40-2 &rarr; ShuffleNetV1| \n|-----------|:-----------:|\n| Independent | 31.23 | \n| KL | 48.62 | \n| CRD | 49.41| \n| KL + CRD | 52.49| \n\n**4. Adding a section of limitation**\n\nWe thank the reviewer for pointing this out, and agree with the recommendation. The analysis conducted in our work has many different axes of generalization: generalization to other datasets (e.g. SUN scene classification dataset), generalization to other kinds of tasks (object detection, semantic segmentation), generalization to other kinds of architectures (modern CNNs, like ResNext), generalization to more recent distillation objectives. In theory, the conclusions will be more helpful if one can study all the combinations of these factors, and hopefully discover a common pattern between them. Due to limitations in terms of space and computational requirements, however, we have not been able to study all of those combinations. Our goal in this work has simply been to show the possibility of the transferability of many properties from a teacher into a student network.\n"}, {"review_id": "MgIrQA0KFZ", "reviewer": "Reviewer_8Y8m", "summary": "This paper argues that distillation implicitly transfers many different behaviors from the teacher to the student, even though these are not explicitly matched via the distillation objective. These include:\n1. The saliency map computed via Grad-CAM\n2. Specific adversarial examples of the teacher\n3. Invariance that the teacher has learned via data augmentations on color jitter, bridghtness, contrast etc.,\n4. Performance on Out-of-Distribution (OoD) data\n\nThe distillation approaches studied here are:\n- the standard K-divergence based loss, \n- the Hint loss which matches embeddings and \n- contrastive representation KD\n\nThe architectures covered are VGG, ResNet, ViT, and Swin (although not all these architectures are used in all experiments).\n\nAll main experiments are on ImageNet. A few additional experiments are on MNIST. \n\nThe paper also provides an intuitive argument for why such implicit knowledge is transferred: the distilled student learns the specific decision boundary of the teacher. This is verified for an MNIST setting where the network via some existing techniques for comparing decision boundaries. \n\nThe paper argues that these findings have practical implications in (a) allowing the student to generalizing to new domains, (b) but also in transferring harmful biases to the student. ", "questions": "1. Are there any types of knowledge that _do not_ get transferred? The main paper would benefit from a more balanced and scientific discussion of any such types knowledge.\n\n2. One of the implications laid out in the paper, namely that the teacher may transfer biases to the student, may have also been captured in  other work, which may have been prior to this paper [1]. \n\n[1] _Teacher's pet: understanding and mitigating biases in distillation_, Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv Kumar https://arxiv.org/abs/2106.10494", "limitations": "I couldn't find a limitation section in the paper. I'd appreciate explicitly listing out some limitations e.g., there is lack of clarity on\n1. how this generalizes to other datasets\n2. why these types of knowledge get transferred\n3. what types of knowledge do/do not get transferred", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The questions explored in the paper are novel and interesting. This is a new perspective on what type of knowledge may be inadvertently transferred from the teacher to the student.\n2. The paper covers various types of knowledge (namely, saliency maps, OOD, adversarial examples, invariances) and there is clear evidence across the board that the student typically imitates the teacher non-trivially.\n3. The paper covers three fundamentally different types of distillation losses, which strengthens their claim. \n4. The experiments are designed carefully. In all cases, an \"independent\" student is considered as a baseline so as to judge whether the distilled-student is indeed copying the teacher. ", "weaknesses": "1. My main concern is that all the main experiments appear to be limited to one (ImageNet) dataset. While this is certainly a non-trivial and sufficiently realistic dataset, one should be wary of drawing observations from a single dataset. Either a language dataset or another image dataset (say, CIFAR100) would have been ideal. But I understand that it would be hard to add these during the rebuttal. \n\n\n\n2.  The argument for \"why does KD transfer such implicit knowledge\" is not compelling and appears to be somewhat circular. The argument given is that \"KD transfers various implicit behaviors of the teacher because the student inherits the decision boundary\". The key puzzle however is why the KD objective is able transfer all of the decision boundary, and that too on OoD and adversarial regions the student isn't trained on. Overall, I find the claim that the paper's argument \"explains\" the phenomenon to be too strong.\n\n\n\n### Summary\n\nOverall, this paper presents is an interesting collection of observations, identifying an underlying pattern across various distillation algorithms. Although I have some concerns with (a) the specificity to ImageNet and (b) the paper's claim at explaining these phenomena, I believe future work would definitely find this useful. \n\n### Post-rebuttal update\n\nIncreasing the score by 1 to acknowledge the authors' new experiments on adversarial vulnerability and domain adaptation that go beyond ImageNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. All the main experiments appear to be limited to one (ImageNet) dataset. While this is certainly a non-trivial and sufficiently realistic dataset, one should be wary of drawing observations from a single dataset.**\n\nWe appreciate that the reviewer recognizes that ImageNet is a non-trivial dataset. But there is another point that we wish to discuss about ImageNet before we present results on some other datasets. Please refer to our response to reviewer **B1Ad**, where we explain our reasoning, and also present some results on new datasets (CIFAR, VLCS and PACS).\n\n**2. The argument for \"why does KD transfer such implicit knowledge\" is not compelling and appears to be somewhat circular.**\n\nWe thank the reviewer for raising an interesting point worth discussion. We will try to expand on our explanation given in Sec. 6 of the paper. The theory of decision boundary transfer (for unseen points as well) depends highly on the points that we are using to perform the distillation during training. For example, If we take the toy example of Fig. 7 (left), the reason the decision boundary could be replicated simply using three points is that those points uniquely define a decision boundary, which is the decision boundary used by the teacher. If we had used some other three points (e.g., collinear points), that precise recovery might not have been possible. In practice, this means that if you had performed distillation on random noise images, the student will not inherit the knowledge from the teacher. All of this is to say that the reason the decision boundary of the OOD points can get distilled is because of the certain points we have used during training, and it wouldn’t work with any arbitrary points. The MNIST data points used for distillation seem to be diverse/appropriate enough to work similarly to the three points in the toy example, and the other unseen domains can pictorially be considered to be the missing points in the figure. \n\nWe'll tone down on the 'explains' part a bit as suggested, and also incorporate the above explanation.\n\n**3. Any knowledge which does not get transferred?**\n\nYes, please refer to Sec. 3.5 in the appendix which shows that it is somewhat difficult to distill the shape bias of the teacher into a texture-biased student, at least in most of the distillation settings. Furthermore, as we mention in lines 197-198, distilling any kind of property from a transformer into a CNN is difficult (e.g., failure to transfer color invariance into a student).\n\n**4. Discussion of related work about bias transfer into a student, and adding a limitation section.**\n\nWe thank the reviewer for mentioning the work. The key result from Lukasik et al. is that after the distillation process, the resulting student can have an uneven improvement (or reduction) in performance, which can be linked to certain properties of the teacher. And while the results from Sec. 5.2 in our paper also are surrounding the same theme, we have focused on the fairness of the student; that is, with the same dataset, the student would have remained fair if it was trained by itself, but by mimicking the (unfair) teacher on the same dataset, it now becomes unfair (lines 288-290). We will include a discussion of this work in the camera ready version, and will also include a section detailing the limitations of our work, addressing the points mentioned by the reviewer.\n\n"}]

Author rebuttal:
We thank the reviewers for their time in giving us constructive feedback. It is encouraging to learn that the reviewers found our work to be an interesting investigation into a fundamental problem concerning knowledge distillation (QC8K, 8Y8m) using a novel angle (QC8K, r3A2, 8Y8m). The reviewers found the experiments to be well organized (B1Ad, r3A2, 8Y8m) studying the topic comprehensively (QC8K) by covering many kinds of distillation objectives and architectures (8Y8m, r3A2) and investigative different types of properties (8Y8m). Overall, the reviewers found the presentation easy to follow with insightful discussions (QC8K, B1Ad), having interesting findings about a consistent behavior of knowledge transfer from teacher to the student (r3A2, B1Ad, 8Y8m).

A common point raised by many of the reviewers is that the results will be more convincing if this topic is studied on datasets/settings other than the classification task on ImageNet. We have conducted some experiments accordingly, to show that the implicit transfer of properties from the teacher into a student is a rather universal phenomena spanning across a range of tasks/datasets. We address the specific points raised by the reviewers below.


Meta-review: The paper conducts an empirical study of behaviors that are being implicitly transferred through knowledge distillation (KD) from a teacher model to the student, even though these are not explicitly matched by the distillation objective. The paper includes a series of experiments for analyzing the nature of the knowledge "distilled", showing that various KD methods transfer implicit properties, such as localization information, adversarial vulnerabilities, invariance to data transformations and domain adaptation. 

The general concern of the reviewers was regarding whether the findings in this work would generalize to other datasets and models. The authors have addressed this in the rebuttal to an extent that the reviewers found reasonable. In addition, multiple reviewers raised concerns about the results not being entirely surprising or exciting, and the empirical nature of this study that does not provide any theoretical support for the findings.

All these points were thoroughly discussed with the reviewers and, overall, there was an agreement that the experiments in the paper are solid and sound, and the findings are novel. Moreover, the above concern seems to be not so critical given the empirical nature of this field, and is overshadowed by the potential of this work to spark interest in the community to better understand the inner workings of KD.
Decision: Accept (poster)