Title: What Knowledge Gets Distilled in Knowledge Distillation?
Venue: NeurIPS 2023
Accepted: yes
Keywords: knowledge distillation
Abstract: Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation?  In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. 
We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.

What Knowledge Gets Distilled in Knowledge Distillation?

Abstract

Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has a been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.

1 Introduction

Knowledge distillation, first introduced in [2, 12], is a procedure of training neural networks in which the 'knowledge' of a teacher is transferred to a student. The thesis is that such a transfer (i) is possible and (ii) can help the student learn additional useful representations. The seminal work by [12] demonstrated its effectiveness by making the student imitate the teacher's class probability outputs for an image. This ushered in an era of knowledge distillation algorithms, including those that try to mimic intermediate features of the teacher [26, 33, 13], or preserve the relationship between samples as modeled by the teacher [22, 32], among others.

While thousands of papers have been published on different techniques and ways of using knowledge distillation, there appears to be a gap in our fundamental understanding of it. Yes, it is well-known that the student's performance on the task at hand can be improved with the help of a teacher. But what exactly is the so-called knowledge that gets distilled during the knowledge distillation process? For example, does distillation make the student look at similar regions as the teacher when classifying images? If one crafts an adversarial image to fool the teacher, is the student also more prone to getting fooled by it? If the teacher is invariant to a certain change in data, is that invariance also transferred to the student? Such questions have not been thoroughly answered in the existing literature.

This has become particularly relevant because there have been studies which present some surprising findings about the distillation process. [3] showed that performing knowledge distillation with a bigger teacher does not necessarily improve the student's performance over that with a smaller teacher, and thus raised questions about the effectiveness of the distillation procedure in such cases. [29] showed that the agreement between the teacher and distilled student's predictions on test images is

x

x

x

class(x)≠ class(y)

Teacher

Student

Teacher

Student

Teacher

Student

flI

f

s(x)

≠

KL

Hint

CRD

not that different to the agreement of those between the teacher and an independently trained student, raising further doubts about how knowledge distillation works, if it works at all.

In this work, we present a comprehensive study tackling the above questions. We analyze three popular knowledge distillation methods [12, 26, 30]. Many of our findings are quite surprising. For example, by simply mimicking the teacher's output using the method of [12], the student can inherit many implicit properties of the teacher. It can gain the adversarial vulnerability that the teacher has. If the teacher is invariant to color, the student also improves its invariance to color. To understand why these properties get transferred without an explicit objective to do so, we study the distillation process through a geometric lens, where we think about the features from a teacher as relative positions of an instance (i.e., distances) from its decision boundary. Mimicking those features, we posit, can therefore help the student inherit the decision boundary and (consequently) the implicit properties of the teacher. We show that these findings have practical implications; e.g., an otherwise fair student can inherit biases from an unfair teacher. Hence, by shedding some light on the 'dark knowledge' [11], our goal is to dissect the distillation process better.

2 Related work

Model compression [2] first introduced the idea of knowledge distillation by compressing an ensemble of models into a smaller network. [12] took the concept forward for modern deep learning by training the student to mimic the teacher's output probabilities. Some works train the student to be similar to the teacher in the intermediate feature spaces [26, 33]. Others train the student to mimic the relationship between samples produced by the teacher [22, 31, 23], so that if two samples are close/far in the teacher's representation, they remain close/far in the student's representation. Contrastive learning has recently been shown to be an effective distillation objective in [30]. More recently, [1] present practical tips for performing knowledge distillation; e.g., providing the same view of the input to both the teacher and the student, and training the student long enough through distillation. Finally, the benefits of knowledge distillation have been observed even if the teacher and the student have the same architecture [5, 34]. For a more thorough survey of knowledge distillation, see [10]. In this work, we choose to study three state-of-the-art methods, each representative of the output-based, feature-based, and contrastive-based families of distillation approaches.

There have been a few papers that present some surprising results. [3] shows that a smaller & less accurate teacher is often times better than a bigger & more accurate teacher in increasing the distilled student's performance. More recent work shows that the agreement between the predictions of a teacher and student is not necessarily much higher than that between the teacher and an independent student [29]. There has been work done which tries to explain why distillation improves student's performance by trying linking it to the regularizing effect of soft labels [19, 35] or what the key ingredients are which help in student's optimization process [24, 14]. What we seek to understand in this work is different: we study different ways (beyond performance improvement) in which a student becomes similar to the teacher by inheriting its implicit properties.

3 Distillation methods studied

To ensure that our findings are general and cover a range of distillation techniques, we select standard methods representative of three families of distillation techniques: output-based [12], feature-based [26], and contrastive-based [30]. The objectives of these methods, described below, are combined with the cross entropy loss L CLS ( z s , y ) := -∑ c j =1 y j log σ j ( z s ) , where y is the

50.3

67.2

52.9

55.6

50.2

59.2

52.9

67.6

Independent

Distilled

Ind

KL

Hint

CRD

Ind

KL

Hint

CRD

Teacher

student

student

ResNet50→ResNet18

VGG19→VGG11

ground-truth one-hot label vector, z s is the student's logit output, σ j ( z ) = exp( z j ) / ∑ i exp( z i ) is the softmax function, and c is the number of classes.

(1) KL: [12] proposed to use the soft labels produced by the teacher as an additional target for the student to match, apart from the (hard) ground-truth labels. This is done by minimizing the KL-divergence between the predictive distributions of the student and the teacher:

where z t is the logit output of the teacher, and τ is a scaling temperature. The overall loss function is γ L CLS + α L KL , where γ and α are balancing parameters. We refer to this method as KL .

(2) Hint: FitNets [26] makes the student's intermediate features ( f s ) mimic those of the teacher's ( f t ) for an image x , at some layer l . It first maps the student's features (with additional parameters r ) to match the dimensions of the teacher's features, and then minimizes their mean-squared error:

The overall loss is γ L CLS + β L Hint , where γ and β are balancing parameters. [26] termed the teacher's intermediate representation as Hint , and we adopt this name.

(3) CRD: Contrastive representation distillation [30] proposed the following. Let s ( x ) and t ( x ) be the student's and teacher's penultimate feature representation for an image x . If x and y are from different categories, then s ( x ) and t ( x ) should be similar (positive pair), and s ( x ) and t ( y ) should be dissimilar (negative pair). A key for better performance is drawing a large number of negative samples N for each image, which is done using a contantly updated memory bank.

where h ( a, b ) = ( e a · b/τ ) / ( e a · b/τ + N M ) , M is the size of the training data, τ is a scaling temperature, and · is the dot product. We use CRD to refer to this method. All other implementation details (e.g., temperature for KL , layer index for Hint ) can be found in appendix.

4 Experiments

We now discuss our experimental design. To reach conclusions that are generalizable across different architectures and datasets, and robust across independent runs, we experiment with a variety of teacher-student architectures, and tune the hyperparameters so that the distillation objective improves the test performance of the student compared to independent training. For each setting, we average the results over two independent runs. We report the top-1 accuracy of all the models in the appendix. The notation Net 1 → Net 2 indicates distilling the knowledge of Net 1 (teacher) into Net 2 (student).

4.1 Does localization knowledge get distilled?

We start by studying whether the localization properties of a teacher transfers to a student through knowledge distillation. For example, suppose that the teacher classifies an image as a cat by focusing

60

80

80

50

35

79.8

49

56

76

76

46

47.7

31

rate

52

52

72

72

42

43

27

Fooling r

48

44

48.3

50.5

68

64

69.7

70.5

68

64

69

70.5.70.770.6

38

34

36.2

23

19

21.9

22.3

23.5

24

44.2

62.3

40

60

60

30

15

Ind

KLHint CRD

Ind

KL

Hint CRD

Ind

KLHint CRD

Ind

CRD

Ind

KLHint CRD

(a)ResNet50→ResNet18

(b)VGG19→VGG11

(c)VGG19→VGG11 (R18)

(d)VGG19→ResNet18

(e)ViT→ResNet18

on its face, and an independent student classifies it as a cat by focusing on its fur. After knowledge distillation, will the student focus more on the face when classifying the image as a cat?

Experimental setup: We train three models for ImageNet classification: a teacher, a distilled and an independent student. For each network, we obtain the class activation map (CAM) of the ground-truth class for each of random 5000 test images, using Grad-CAM [27], which visualizes the pixels that a classifier focuses on for a given class. We then compute how often (in terms of % of images) the teacher's CAM is more similar (using cosine similarity) to the distilled student's CAM than to the independent student's CAM. A score of > 50% means that, on average, the distilled student's CAMs become more similar to those of the teacher than without distillation. As a sanity check, we also compute the same metric between the teacher's CAM and the CAMs of two independent students trained with different seeds ( Ind ), which should be equally distant from the teacher (score of ∼ 50%).

Results: Fig. 2 (right) shows the results for two configurations of teacher-student architectures: (i) ResNet50 → ResNet18 and (ii) VGG19 → VGG11. For KL and CRD , we observe a significant increase in similarity compared to random chance (as achieved by the Ind baseline). Hint also shows a consistent increase, although the magnitude is not as large.

Discussion: This experiment shows that having access to the teacher's class-probabilities, i.e. confidence for the ground-truth class, can give information on where the teacher is focusing on while making the classification decision. This corroborates, to some degree, the result obtained in the Grad-CAM paper [27], which showed that if the network is very confident of the presence of an object in an image, it focuses on a particular region (Fig. 1(c) in [27]), and when it is much less confident, it focuses on some other region (Fig. 7(d) in [27]). Fig. 2 (left) shows a sample test image and the corresponding CAMs produced by the independent student (left), teacher (middle), and distilled student (right) for the ground-truth class. The distilled student looks at similar regions as the teacher, and moves away from the regions it was looking at initially (independent student). So, regardless of the correctness of any network's CAM, our analysis shows that a distilled student's CAM does become similar to those of the teacher for all three distillation techniques, albeit with varying degrees.

4.2 Does adversarial vulnerability get distilled?

Next, we study the transfer of a different kind of property. If we design an adversarial image to fool the teacher, then will that same image fool the distilled student more than the independent student?

Experimental setup: We train a teacher, a distilled student, and an independent student for ImageNet classification. Given 5000 random test images, we convert each image I into an adversarial image I adv using iterative FGSM [9, 17] (see appendix for details), whose goal is to fool the teacher, so that teacher's prediction for I adv changes from its original prediction for I . Fooling rate is then defined as the fraction of adversarial images which succeed at this task. In our experiments, we use only this fraction of adversarial images which fool the teacher, and apply them to different students.

Results: We evaluate four configurations: (i) ResNet50 → ResNet18, (ii) VGG19 → VGG11, (iii) VGG19 → ResNet18, (iv) ViT (ViT-b-32) [4] → ResNet18. The fooling rate is ∼ 85% for all the teachers. Fig. 3 shows the fooling rate (y-axis) when applying these successful adversarial images to different types of students (x-axis). We see that for ResNet50 → ResNet18, the ability to fool the independent student drops to 44.2%, which is expected since the adversarial images aren't designed for that student. In distilled students, we see an increase in the fooling rate relative to the independent one across all distillation methods (48%-52%). The trend holds for VGG19 → VGG11 and VGG19

95

color strength→→

color strength→→

Ind

Hint-F3

Hint-F2

Hint-F5

Hint-F6

90

92.5

90

90

90

Hint-F1

Hint-F4

Hint-F7

0

85

Agreement (%

80

80

80

Agreement

75

80

82.1

79.5

78

70

70

70

74

72.272.4

60

60

60

A

70

65

71.3

71.3

50

CRD

Hint

puI

50

CRD

Hint

Ind

50

60

40

KL

T

40

T

KL

40

color strength →

IndKLKL*Hint Hint*CRDCRD* TT*

0.3

0.4

0.5

0.6

0.3

0.4

0.5

0.6

0.3

0.4

0.5

0.6

(a)R50→R18

(b)R50→R18

(c)Swin-T→R18

(d)R50→R18(Hint ablation)

→ ResNet18; Fig. 3 (b, d). When distillation is done from a transformer to a CNN (ViT → ResNet18) the fooling rates remain similar for the independent and distilled students; Fig. 3 (e). Here, we don't see the student becoming similar to the teacher to the extent observed for a CNN → CNN distillation.

Discussion: This result is surprising. Iterative FGSM is a white box attack, which means that it has full access to the target model (teacher), including its weights and gradients. Thus, the adversarial examples are specifically crafted to fool the teacher. In contrast, the student never has direct access to the weights and gradients of the teacher, regardless of the distillation method. Yet, by simply trying to mimic the teacher's soft probabilities ( KL ) or an intermediate feature layer ( Hint , CRD ), the student network inherits, to some extent, the particular way that a teacher is fooled.

We conduct an additional study to ensure that the reason the distilled student is getting fooled more is because it is being attacked specifically by its teacher's adversarial images; i.e., if those images are designed for some other network, would the difference in fooling rates still be high? We test this in the VGG19 → VGG11 setting. This time we generate adversarial images ( I → I adv ) to fool an ImageNet pre-trained ResNet18 network, instead of VGG19 (the teacher). We then use those images to attack the same VGG11 students from the VGG19 → VGG11 setting. In Fig. 3 (c), we see that the fooling rates for the independent and distilled students remain similar. This indicates that distillation itself does not make the student more vulnerable to any adversarial attack, and instead, an increase in fooling rate can be attributed to the student's inheritance of the teacher's adversarial vulnerability.

4.3 Does invariance to data transformations get distilled?

We have studied whether the response properties on single images get transferred from the teacher to the student. Now suppose that the teacher is invariant to certain changes in data, either learned explicitly through data augmentation or implicitly due to architectural choices. Can such properties about changes in images get transferred during distillation?

Experimental setup: We study color invariance as a transferable property. We train three models for ImageNet classification: a teacher, a distilled and an independent student. While training the teacher, we add color jitter in addition to the standard augmentations (random crops, horizontal flips). Specifically, we alter an image's brightness, contrast, saturation, and hue, with magnitudes sampled uniformly in [0, 0.4] for the first three, and in [0, 0.2] for hue. This way, the teacher gets to see the same image with different color properties during training and can become color invariant. When training the student, we only use the standard augmentations without color jittering, and see whether such a distilled student can indirectly inherit the color invariance property through the teacher.

Results: We start with the ResNet50 → ResNet18 configuration. After training, we evaluate the models on 50k ImageNet validation images. For each image X , we construct its augmented version X ′ by altering the color properties in the same way as done while training the teacher. Fig. 4 (a) depicts the agreement scores between X and X ′ (y-axis) for different models (x-axis). The teacher (T), being trained to have that property, achieves a high score of 92.5%. The independent student, which is not trained to be color invariant, has a much lower score of 71.3%. But, when the colorinvariant teacher is used to perform distillation using KL or CRD , the agreement scores of the students jump up to 82.1% and 79.5% respectively. To ensure that this increase is not due to some regularizing property of the distillation methods that has nothing to do with the teacher, we repeat the experiment, this time with a ResNet50 teacher (T ∗ ) that is not trained with color augmentation. The new agreement scores for the distilled students (marked by *; e.g., KL ∗ ) drop considerably

IndKLHintCRD

IndKLHintCRD

Teacher

Student

50

09

Consensus

40

50

30

40

KD:naturalimages

20

30

10

20

Similaronunseen

0

sketch

stylized

silhouette

edge

cue conflict

10

sketch

stylized

silhouette

edge

cue conflict

domains?

(a)VGG19→ResNet18

compared to the previous case. This is a strong indication that the student does inherit teacher-specific invariance properties during distillation.

In Fig. 4(b), we show that this trend in agreement scores (y-axis) holds even when the magnitude of change in brightness, contrast and saturation (x-axis) is increased to create X ′ from X . We repeat this experiment for Swin-tiny (a transformer) [20] → ResNet18 in Fig. 4(c). We again see some improvements in the agreement scores of the distilled students. This increase, however, is not as significant as that observed in the previous CNN → CNN setting. Throughout this work, we find that distilling the properties from a transformer teacher into a CNN student is difficult. The implicit biases introduced due to architectural differences between the teacher (transformer) and student (CNN) seem too big, as was studied in [25], to be overcome by current distillation methods.

We also note the ineffectiveness of Hint in distilling this knowledge. Our guess for this is the choice of l for distillation, which is typically set to be in the middle of the network as opposed to deeper layers where KL and CRD operate. So, we perform an ablation study for Hint , where the student mimics a different feature of the teacher each time; starting from a coarse feature of resolution 56 × 56 (F1) to the output scores (logits) of the teacher network (F7). We plot the agreement scores in Fig. 4(d), where we see that the score increases as we choose deeper features. Mimicking the deeper layers likely constrains the overall student more compared to a middle layer, since in the latter case, the rest of the student (beyond middle layer) could still function differently compared to the teacher.

Discussion: In sum, color invariance can be transferred during distillation. This is quite surprising since the teacher is not used in a way which would expose any of its invariance properties to the student. Remember that all the student has to do during distillation is to match the teacher's features for an image X ; i.e., the student does not get to see its color augmented version, X ′ , during training. If so, then why should it get to know how the teacher would have responded to X ′ ?

4.4 Does knowledge about unseen domains get distilled?

So far, our analysis has revolved around the original ImageNet dataset, something that was used to perform the distillation itself. So, does the teacher only transfer its knowledge pertaining to this domain, or also of domains it has never seen (Fig. 5 left)?

Experimental setup: To test this, we first perform distillation using two settings (i) VGG19 → ResNet18 and (ii) Swin-Base [20] → Swin-Tiny, where the training of the teacher, as well as distillation is done on ImageNet. During test time, we take an image from an unseen domain and see how frequently the student's and teacher's class predictions match for it (regardless of whether the predicted label is correct or incorrect), which we call the consensus score. For the unseen domains, we consider the five datasets proposed in [7]: sketch , stylized , edge , silhouette , cue conflict . Images from these domains are originally from ImageNet, but have had their properties modified.

Results: Fig. 5 (right) shows the consensus scores between the teacher and the student (y-axis). We see a nearly consistent increase in consensus brought about by the distillation methods in both settings. The extent of this increase, however, differs among the domains. There are cases, for example, in VGG19 → ResNet18, where consensus over images from edge domain increases over 100% (12% →≥ 30%) by some distillation methods. On the other hand, the increase can also be relatively modest in certain cases: e.g. Swin-Base → Swin-Tiny in stylized domain.

Discussion: [29] showed that the agreement in classification between the teacher and distilled student is not much different than that to an independent student on CIFAR. We find this to be true in our ImageNet classification experiments as well. That is why the increase in agreement observed for

100

100

70

Teacher

Student

99.2

98.9

99.1

99.1

97.8

69.7

95

93

65

66.6

accuracy

90

86

89.4

60

85

79

55

53.5

53.2

80

72

50

3

-do

75

65

67.8

45

63.6

70

58

40

Ind KL Hint CRD

Ind

Ind KL Hint CRD

Knowledge on three

Distilled on one

domains

domain

MNIST-Orig

MNIST-Color

MNIST-M

unseen domains is surprising. After all, if the agreement is not increasing when images are from the seen domain, why should it increase when they are from an unseen domain? It is possible that there is more scope for increase in agreement in unseen domains vs seen domain. The consensus score between teacher and independent student for the seen domain is ≥ 75% (appendix), whereas for an unseen domain, e.g. sketch , it is ≤ 40%. So, it might not be that knowledge distillation does not work, as the authors wondered in [29], but its effect could be more prominent in certain situations.

4.5 Other studies

In appendix, we study additional aspects of a model such as shape/texture bias, invariance to random crops, and find that even these obscure properties can transfer from a teacher to the student. We also explore the following questions: If we can find alternative ways of increasing a student's performance (e.g. using crafted soft labels), will that student gain similar knowledge as a distilled student? If distillation cannot increase the student's performance, is there no knowledge transferred? Finally, we also present results on some more datasets like CIFAR100 [16], VLCS and PACS [18], showing that the phenomena of the implicit transfer of properties dueing knowledge distillation extends even to datasets beyond ImageNet and MNIST.

5 Applications

Beyond the exploratory angle of studying whether certain properties get transferred during distillation, the idea of a student becoming similar to a teacher in a broad sense has practical implications. We discuss a good and a bad example in this section.

5.1 The Good: the free ability of domain adaptation

Consider the following setup: the teacher is trained for a task by observing data from multiple domains ( D 1 ∪D 2 ). It is then used to distill knowledge into a student on only D 1 . Apart from getting knowledge about D 1 , will the student also get knowledge about D 2 indirectly?

Experimental setup: We use MNIST digit recognition, and train the teacher on three domains: (i) MNIST-orig : original gray-scale images from MNIST, (ii) MNIST-Color : background of each image randomly colored, and (iii) MNIST-M [6]: MNIST digits pasted on random natural image patches. The student models are trained only on MNIST-orig and evaluated (top-1 accuracy) on all three domains. The network architecture is same for both the teacher and the student (see appendix).

Results: When the independent student is trained only on MNIST-orig , its performance drops on the unseen domains, which is expected due to domain shift. The distilled students (especially KL & Hint ), however, are able to significantly improve their performance on both unseen domains; Fig. 6.

Discussion: This result shows distillation's practical benefits: once a teacher acquires an ability through computationally intensive training (e.g., training on multiple datasets), that ability can be distilled into a student, to a decent extent, through a much simpler process. The student sees the teacher's response to gray-scale images ( MNIST-orig ) that lack any color information. But that information helps the student to deal better with colored images (e.g., MNIST-Color ), likely because the teacher has learned a domain-invariant representation (e.g., shape) which is distilled to the student.

5.2 The Bad: Students can inherit harmful biases from the teacher

Consider the problem of classifying gender from human faces. Imagine an independent student which performs the classification fairly across all races. The teacher, on the other hand, is biased against certain races, but is more accurate than the student on average . Will the student, which was originally fair, become unfair after mimicking the unfair teacher?

Experimental setup: We consider a ResNet20 → ResNet20 setting, and use FairFace dataset [15], which contains images of human faces from 7 different races with their gender labeled. From its training split, we create two different subsets ( D s and D t ) with the following objectives - (i) D s has a particular racial composition so that a model trained on it will perform fairly across all races during test time; (ii) D t 's composition is intended to make the model perform unfairly for certain races. The exact composition of D s and D t is given in the appendix. The teacher is trained on D t whereas the independent/distilled students are trained on D s . We use KL for distillation.

Results: We observe in the figure on the right that the independent student performs roughly fairly across all races, which is what we intended. The teacher is more accurate than the student, but performs relatively poorly on faces from Race 3 compared to others. After distillation, we observe that the student's overall accuracy improves, but the gain in accuracy is less for images from Race 3. So, when it is trained on D s by itself, it behaves fairly. But when it mimics the teacher on D s , it becomes unfair.

Fair student Distilled student

UnfairTeacher

90

80

70

60

50

Race 1

Race2

Race3

Race 4Race5Race 6

Race7

This follows the observation made in [21], where the authors found that the performance gains during distillation might not be spread uniformly across different sub-categories.

Discussion: The practical takeaway from the experiment is that knowledge distillation can bring forth behaviour which is considered socially problematic if it is viewed simply as a blackbox tool to increase a student's performance on test data, as it did in the previous example. Proper care must hence be taken regarding the transfer/amplification of unwanted biases in the student.

6 Why does knowledge distillation work in this way?

An Illustrative Example. Why should a teacher's response to an image contain such rich information about its implicit properties? We first intuitively explain this through a toy classification problem using KL as the distillation objective. Fig. 7 (left) shows data points from two classes (red and blue). The teacher has access to the complete set, thereby learning to classify them appropriately (orange decision boundary). On the other hand, suppose that the training of the independent and distilled student is done on a subset of the points. The independent student can learn to classify them in different ways, where each decision boundary can be thought of as looking for different features, hence giving rise to different properties. But the distilled student is trying to mimic the teacher's class probabilities, which reflect the distance from the (orange) decision boundary of the teacher, visualized as circles around the points. Then the decision boundary of the distilled student should remain tangential to these circles so that it preserves the teacher's distances. So, this very simple case depicts that mimicking the class probabilities can force the student's decision boundary to resemble that of the teacher, which is not true for the independent student.

A mathematical formulation of the same case can be considered as follows. Let D be a linearly separable dataset: D ∈ { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ......, ( x n , y n ) | x i ∈ ℜ m , y i ∈ {-1 , 1 }} . For an arbitrary linear classifier having weights W ∈ ℜ m × 1 and biases b ∈ ℜ , the binary classification objective for the teacher and the independent student can be described as ∀ ( x i , y i ) ∈ D, ( W T x i + b ) y i > 0 .

Since the dataset is linearly separable, both the teacher as well as the independent student can solve this system of inequalities in many ways. Let { W t , b t } be the parameters of one particular teacher.

Next, we look at the objective of the distilled student, who tries to map x i to z i = W T t x i + b t , instead of y i . That is, the optimal solution satisfies: ∀ ( x i , z i ) ∈ D dist , W T s x i + b s = z i . This system of linear equations will have a unique solution provided it has access to at least m +1 training instances and that these instances are linearly independent. Since we know that the teacher's weights and

Decisionboundaries

Teacher

Independent student

Distilledstudent

Teacher

Independent student

Distilledstudent

biases, { W t , b t } , can satisfy this equation, we can conclude that if there exists a unique solution, then W s = W t and b s = b t . So, the distilled student, when solving the distillation objective, will recover the weights and biases of the teacher, and consequently its decision boundary.

Complex neural networks: Given that the distances of data points, i.e., decision boundary, can be preserved by the distilled student in the very simple case presented above, we now study whether they can be preserved in a more complex setting involving multi-layer neural networks as well.

There have been works [8, 28] which have studied this for the KL objective: if the teacher and the students are all trained on the same dataset D , the decision boundary of the teacher will be more similar to the distilled student than the independent student for points from D . And while this result is useful, it is not clear whether (i) this can help explain why the student inherits teacher's knowledge about other domains, and (ii) whether this holds true for distillation objectives other than KL .

To study this, we revisit the setup of Sec. 5.1, where the teacher was trained on three domains and distillation was performed on one (Fig. 6 left). We study what happens to the decision boundary similarity between the students and the teacher on all three domains. To compute that similarity between two neural networks, we use the method introduced in [28]. We randomly sample a triplet of data points from a given domain and create a plane that passes over that triplet. The networks that are being compared receive the same dataset of points in this plane, and we compute intersection over union between their predictions. We report the score averaged over 50 triplets.

Results: Table 1 shows the similarity scores, where we see that the distilled students' decision boundary are almost always more similar to the teacher's than the independent student's. This is particularly evident when the scores are computed for domains not seen by the students; e.g., for MNIST-Color , the similarity score increases from 0.416 to 0.671 for the student distilled with KL .

Discussion: First, we can draw an analogy of this experimental setup with the toy example discussed before. The three MNIST related domains (for training the teacher) are similar to the overall set of blue+red points used to train the toy teacher (Fig. 7 left). The singular MNIST-orig domain (for training the students) is similar to the three points available to train the toy students. Now, what the results from Table 1 show is that the data points available for distillation can help determine the teacher's decision boundary even for points which the student did not have access to. So, similar to how the decision boundary estimated by the toy student can correctly estimate the distances of missing blue/red points, the neural network based student can also estimate the distances of points from unseen domains, e.g., MNIST-color , in a similar way as its teacher, thereby inheriting the teacher's behavior on these points. An example of this is given in Fig. 7 (right).

7 Limitations

While we have tried to study the distillation process in many settings, there do exist many more which remain unexplored. This is because the analysis conducted in our work has many axes of generalization: whether these conclusions hold (i) for other datasets (e.g. SUN scene classification dataset), (ii) in other kinds of tasks (object detection, semantic segmentation), (iii) in other kinds of architectures (modern CNNs, like ResNext), (iv) or for more recent distillation objectives, etc. Moreover, in Sec. 3.5 in the supplementary, we explore the transferability of shape/texture bias from teacher to student, and find some discrepancy in different distillation objective's abilities. Therefore, the conclusions drawn from our work will be more helpful if one can study all the combinations of these factors. Due to limitations in terms of space and computational requirements, however, we have not been able to study all of those combinations.

8 Conclusion

Knowledge distillation is a beautiful concept, but its success i.e, increase in student's accuracy, has often been explained by a transfer of dark knowledge from the teacher to the student. In this work, we have tried to shed some light on this dark knowledge. There are, however, additional open questions, which we did not tackle in this work: given the architectures of the teacher and student, is there a limit on how much knowledge can be transferred (e.g., issues with ViT → CNN)? If one wants to actively avoid transferring a certain property of the teacher into a student (Sec. 5.2), but wants to distill other useful properties, can we design an algorithm tailored for that? We hope this work also motivates other forms of investigation to have an even better understanding of the distillation process.

References

Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In arXiv , 2021.

Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD , 2006.

Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv , 2020.

Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML , 2018.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR , 2016.

Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In NeurIPS , 2021.

Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence , 2020.

Ian Goodfellow, Jon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2014.

Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. In arXiv , 2021.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Dark knowledge. TTIC Distinguished lecture series , 2014.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS Deep Learning Workshop , 2014.

Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. In arXiv , 2017.

Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In ICML , 2021.

Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In WACV , 2021.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. In Technical report, Citeseer , 2009.

Alexey Kurakin, Ian Goodfellow, and Sammy Bengio. Adversarial machine learning at scale. In arXiv , 2016.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV , 2017.

Yuan Li, Francis E.H.Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR , 2020.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. In arXiv , 2021.

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR , 2019.

Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV , 2019.

Max Phuong and Christoph H. Lampert. Towards understanding knowledge distillation. In ICML , 2019.

Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurIPS , 2021.

Adriana Romero, Nicholas Ballas, Samira Ebrahimi Kahau, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR , 2015.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IJCV , 2019.

Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. arXiv preprint arXiv:2203.08124 , 2022.

Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In NeurIPS , 2021.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR , 2020.

Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV , 2019.

Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR , 2017.

Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR , 2017.

Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR , 2018.

Zhilu Zhang and Mert Sabuncu. Self-distillation as instance-specific label smoothing. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 2184-2195. Curran Associates, Inc., 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that distillation implicitly transfers many different behaviors from the teacher to the student, even though these are not explicitly matched via the distillation objective. These include:\n1. The saliency map computed via Grad-CAM\n2. Specific adversarial examples of the teacher\n3. Invariance that the teacher has learned via data augmentations on color jitter, bridghtness, contrast etc.,\n4. Performance on Out-of-Distribution (OoD) data\n\nThe distillation approaches studied here are:\n- the standard K-divergence based loss, \n- the Hint loss which matches embeddings and \n- contrastive representation KD\n\nThe architectures covered are VGG, ResNet, ViT, and Swin (although not all these architectures are used in all experiments).\n\nAll main experiments are on ImageNet. A few additional experiments are on MNIST. \n\nThe paper also provides an intuitive argument for why such implicit knowledge is transferred: the distilled student learns the specific decision boundary of the teacher. This is verified for an MNIST setting where the network via some existing techniques for comparing decision boundaries. \n\nThe paper argues that these findings have practical implications in (a) allowing the student to generalizing to new domains, (b) but also in transferring harmful biases to the student.\n\nSTRENGTHS:\n1. The questions explored in the paper are novel and interesting. This is a new perspective on what type of knowledge may be inadvertently transferred from the teacher to the student.\n2. The paper covers various types of knowledge (namely, saliency maps, OOD, adversarial examples, invariances) and there is clear evidence across the board that the student typically imitates the teacher non-trivially.\n3. The paper covers three fundamentally different types of distillation losses, which strengthens their claim. \n4. The experiments are designed carefully. In all cases, an \"independent\" student is considered as a baseline so as to judge whether the distilled-student is indeed copying the teacher.\n\nWEAKNESSES:\n1. My main concern is that all the main experiments appear to be limited to one (ImageNet) dataset. While this is certainly a non-trivial and sufficiently realistic dataset, one should be wary of drawing observations from a single dataset. Either a language dataset or another image dataset (say, CIFAR100) would have been ideal. But I understand that it would be hard to add these during the rebuttal. \n\n\n\n2.  The argument for \"why does KD transfer such implicit knowledge\" is not compelling and appears to be somewhat circular. The argument given is that \"KD transfers various implicit behaviors of the teacher because the student inherits the decision boundary\". The key puzzle however is why the KD objective is able transfer all of the decision boundary, and that too on OoD and adversarial regions the student isn't trained on. Overall, I find the claim that the paper's argument \"explains\" the phenomenon to be too strong.\n\n\n\n### Summary\n\nOverall, this paper presents is an interesting collection of observations, identifying an underlying pattern across various distillation algorithms. Although I have some concerns with (a) the specificity to ImageNet and (b) the paper's claim at explaining these phenomena, I believe future work would definitely find this useful. \n\n### Post-rebuttal update\n\nIncreasing the score by 1 to acknowledge the authors' new experiments on adversarial vulnerability and domain adaptation that go beyond ImageNet.\n\nQUESTIONS:\n1. Are there any types of knowledge that _do not_ get transferred? The main paper would benefit from a more balanced and scientific discussion of any such types knowledge.\n\n2. One of the implications laid out in the paper, namely that the teacher may transfer biases to the student, may have also been captured in  other work, which may have been prior to this paper [1]. \n\n[1] _Teacher's pet: understanding and mitigating biases in distillation_, Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv Kumar https://arxiv.org/abs/2106.10494\n\nLIMITATIONS:\nI couldn't find a limitation section in the paper. I'd appreciate explicitly listing out some limitations e.g., there is lack of clarity on\n1. how this generalizes to other datasets\n2. why these types of knowledge get transferred\n3. what types of knowledge do/do not get transferred", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an empirical study to analyze inner workings of knowledge distillation (KD) methods. The authors try to understand the nature of the knowledge \"distilled\" from a teacher to a student by studying how various properties get transferred in this learning. The properties include (a) localization information, (b) adversarial vulnerability, (c) invariance to data transformations and (d) domain adaptation. There is further analysis about some other properties in the supplement. The paper analyzes three broad distillation methods on ImageNet with multiple CNN and transformer architectures. Lastly, the authors present positive/negative applications of this understanding for specific tasks (domain adaptation, bias transfer).\n\nSTRENGTHS:\nStrengths:\n\n(1) The paper is nicely organized and very well written. The motivations to study the topic are clear. To my knowledge, this analysis is quite original.\n\n(2) The authors investigate some relevant and interesting directions in a modular way to study KD.\n\n(3) Multiple different architectures and distillation techniques are carefully analyzed for various properties and generally demonstrate consistent results.\n\nWEAKNESSES:\nWeaknesses:\n\nThe weaknesses I see stem from the purely empirical nature of the study. \n\n(1) The generalizability of results to other types of datasets, tasks, domains remains unknown. (Major)\n\n(2) The selection of properties studied seems a little arbitrary. (Minor)\n\n(3) There is lack of any theoretical analysis to characterize the distilled knowledge. (Minor)\n\nOverall, I like the paper but can't favour stronger acceptance yet. I enjoyed reading through it. It is structured well, the experiments are quite thorough (except in one aspect!) and give some interesting insights about KD. Among the weaknesses I highlighted, I won't hold the last two weaknesses strongly against the paper as (3) is a difficult topic in itself and the studied set of properties could still be regarded as reasonably comprehensive in terms of popular ML topics today. However, considering the purely empirical nature, weakness (1) is major for me that should be tackled before I can more clearly favor acceptance.\n\nQUESTIONS:\nQuestions: \n\n(1) **Other datasets/tasks**: How well do the results generalize over other datasets and different tasks (small/large scale images, fine-grained recognition, scene classification etc.). I understand that the number of possible experiments is too large and the paper can't cover everything. However, the same way the paper deserves credit for coverage in terms of distillation methods and architecture selection of teacher and student, it also needs to confirm/study if these results hold for other settings. At least a couple more settings should be included. This is specially important as there are some non-intuitive results and demand further analysis to be confirmed. Also, it would be great if you could illustrate domain adaptation application for some non-MNIST data too.\n\n(2) **Properties selected**: While I am assuming the current set of properties were selected as these are some popular topics of study in ML, I would like to know your rationale behind selecting them. For example did you also consider assessing if the uncertainty of teacher gets transferred to student? \n\n(3) **More questions about experiments**: Does a mixture of simultaneous KL + Hint or other mixtures provide stronger distillation of the properties? Is the localization transfer for transformer -> CNN poor? You could use some other saliency map as localization proxy if grad-CAM is not suitable.\n\nLIMITATIONS:\nThe authors mention \"We do not have a separate section for limitation, but since ours is an analysis paper, we are discussing the open questions all throughout the paper.\" While I understand their reasoning and thought process, I also believe they should summarize the limtiations of their experimental setup in the conclusion or a separate section. It is an experimental study and it can't possibly conduct all experiments as part of it. The open questions raised throughout the paper I believe or more related to depth of the analysis. The authors should certainly add the limitations regarding breadth of their analysis.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores different ways in which a student network can inherit implicit properties from a teacher network beyond just improving task performance. The authors study various distillation methods, including output-based, feature-based, and contrastive-based techniques, and show that these methods can indirectly distill properties. The paper also investigates why knowledge distillation works and discusses its practical implications. Overall, the paper aims to shed light on the \"dark knowledge\" of knowledge distillation and motivate further investigation into the distillation process.\n\nSTRENGTHS:\n1. This paper sheds light on the dark knowledge of knowledge distillation (KD) by investigating a series of questions in KD and gives interesting findings.\n2. Each sub-section in experiments is well-organized, and the question is answered with clearly designed experiments and supportive results, followed by insightful discussions. \n3. The paper is well-written and the presentation is decent.\n\nWEAKNESSES:\n1. The major weakness of this paper is the setup appears to be more exploratory in nature. While exploratory experiments can certainly provide valuable insights and help to understand the findings, they do not substitute for a thorough evaluation that generalizes across diverse datasets. In this paper, the use of only a few datasets may not present a comprehensive picture and could yield misleading results.\n2. The selection of benchmark methods in this paper, while representative of some standard approaches, is notably restricted. The experimental comparison lacks several recent knowledge distillation (KD) methods, leaving the evaluation somewhat incomplete. The inclusion of these recent advances in KD techniques would not only enhance the comprehensiveness of the evaluation but also deepen our understanding of the KD process.\n(Ren et al., 2022) Better Supervisory Signals by Observing Learning Paths.\n(Chen et al., 2021) Distilling Knowledge via Knowledge Review.\n(Zhao et al., 2022) Decoupled Knowledge Distillation.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper designed a series of experiments to fundamentally dissect and understand knowledge distillation. The key questions include, Does localization knowledge get distilled? Does adversarial vulnerability get distilled? Does invariance to data transformations get distilled? Does knowledge about unseen domains get distilled? Can students inherit harmful biases from the teacher? and the main content of this paper is to design experiments to answer these questions empirically. At last, the paper provides some intuition about why knowledge distillation works in this way, and concludes with open questions on how to leverage such findings.\n\nSTRENGTHS:\nThe paper proposed to comprehensively study the critical, interesting and fundamental problem about deep understanding of knowledge distillation, from an angle of what properties of the teacher are distilled into students, which is a novel angle.\n\nThe paper extracted the key questions in order to understand knowledge distillation, where those questions are interesting and essential to help the problem better. Also, for each question, the paper designed corresponding experiments on the representative models with standard datasets.\n\nThe paper is easy to follow.\n\nWEAKNESSES:\nEven though the paper proposed to study a critical fundamental problem, the paper focused more on designing the experiments to answer the potential questions that could help understand the questions. However, many answers towards the questions are not that surprising, which falls in the expectation. Purely relying on some empirical results to facilitate the understanding of the questions limits the contribution of this paper. If the paper could have some theoretical results to demonstrate why it works and when it works, it could solidify the contribution of this paper.\n\nFor each experiment, it’s unclear whether the conclusion could still hold for various architectures or datasets. So it would be great that the paper could include more representative architectures and broader datasets to make results more convincing. \n\nThe contributions of this paper would be more solid if there are some attempts to answer some of the open questions in the conclusion part, e.g. how to design algorithms to control which properties to distill or not distill.\n\nQUESTIONS:\nThe intuition presented in section 6 could be formalized lowering variance as in the following paper? \nMenon et al, “A Statistical Perspective on Distillation”, ICML 2021.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}