SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Utkarsh Ojha* Yuheng Li* Anirudh Sundara Rajan*
Yingyu Liang Yong Jae Lee
Knowledge distillation aims to transfer useful information from a teacher network to a student network, with the primary goal of improving the student's performance for the task at hand. Over the years, there has been a deluge of novel techniques and use cases of knowledge distillation. Yet, despite the various improvements, there seems to be a glaring gap in the community's fundamental understanding of the process. Specifically, what is the knowledge that gets distilled in knowledge distillation? In other words, in what ways does the student become similar to the teacher? Does it start to localize objects in the same way? Does it get fooled by the same adversarial samples? Does its data invariance properties become similar? Our work presents a comprehensive study to try to answer these questions. We show that existing methods can indeed indirectly distill these properties beyond improving task performance. We further study why knowledge distillation might work this way, and show that our findings have practical implications as well.
Knowledge distillation, first introduced in [2, 12], is a procedure of training neural networks in which the 'knowledge' of a teacher is transferred to a student. The thesis is that such a transfer (i) is possible and (ii) can help the student learn additional useful representations. The seminal work by [12] demonstrated its effectiveness by making the student imitate the teacher's class probability outputs for an image. This ushered in an era of knowledge distillation algorithms, including those that try to mimic intermediate features of the teacher [26, 33, 13], or preserve the relationship between samples as modeled by the teacher [22, 32], among others.
While thousands of papers have been published on different techniques and ways of using knowledge distillation, there appears to be a gap in our fundamental understanding of it. Yes, it is well-known that the student's performance on the task at hand can be improved with the help of a teacher. But what exactly is the so-called knowledge that gets distilled during the knowledge distillation process? For example, does distillation make the student look at similar regions as the teacher when classifying images? If one crafts an adversarial image to fool the teacher, is the student also more prone to getting fooled by it? If the teacher is invariant to a certain change in data, is that invariance also transferred to the student? Such questions have not been thoroughly answered in the existing literature.
This has become particularly relevant because there have been studies which present some surprising findings about the distillation process. [3] showed that performing knowledge distillation with a bigger teacher does not necessarily improve the student's performance over that with a smaller teacher, and thus raised questions about the effectiveness of the distillation procedure in such cases. [29] showed that the agreement between the teacher and distilled student's predictions on test images is
Figure 1: Methods used in this work. (i)  $KL$  [12]: mimicking of class probabilities. (ii) Hint [26]: mimicking of features at an intermediate layer. (iii) CRD [30]: features from the student and teacher for the same image constitute a positive pair, and those from different classes make up a negative pair.
not that different to the agreement of those between the teacher and an independently trained student, raising further doubts about how knowledge distillation works, if it works at all.
In this work, we present a comprehensive study tackling the above questions. We analyze three popular knowledge distillation methods [12, 26, 30]. Many of our findings are quite surprising. For example, by simply mimicking the teacher's output using the method of [12], the student can inherit many implicit properties of the teacher. It can gain the adversarial vulnerability that the teacher has. If the teacher is invariant to color, the student also improves its invariance to color. To understand why these properties get transferred without an explicit objective to do so, we study the distillation process through a geometric lens, where we think about the features from a teacher as relative positions of an instance (i.e., distances) from its decision boundary. Mimicking those features, we posit, can therefore help the student inherit the decision boundary and (consequently) the implicit properties of the teacher. We show that these findings have practical implications; e.g., an otherwise fair student can inherit biases from an unfair teacher. Hence, by shedding some light on the 'dark knowledge' [11], our goal is to dissect the distillation process better.
Model compression [2] first introduced the idea of knowledge distillation by compressing an ensemble of models into a smaller network. [12] took the concept forward for modern deep learning by training the student to mimic the teacher's output probabilities. Some works train the student to be similar to the teacher in the intermediate feature spaces [26, 33]. Others train the student to mimic the relationship between samples produced by the teacher [22, 31, 23], so that if two samples are close/far in the teacher's representation, they remain close/far in the student's representation. Contrastive learning has recently been shown to be an effective distillation objective in [30]. More recently, [1] present practical tips for performing knowledge distillation; e.g., providing the same view of the input to both the teacher and the student, and training the student long enough through distillation. Finally, the benefits of knowledge distillation have been observed even if the teacher and the student have the same architecture [5, 34]. For a more thorough survey of knowledge distillation, see [10]. In this work, we choose to study three state-of-the-art methods, each representative of the output-based, feature-based, and contrastive-based families of distillation approaches.
There have been a few papers that present some surprising results. [3] shows that a smaller & less accurate teacher is often times better than a bigger & more accurate teacher in increasing the distilled student's performance. More recent work shows that the agreement between the predictions of a teacher and student is not necessarily much higher than that between the teacher and an independent student [29]. There has been work done which tries to explain why distillation improves student's performance by trying linking it to the regularizing effect of soft labels [19, 35] or what the key ingredients are which help in student's optimization process [24, 14]. What we seek to understand in this work is different: we study different ways (beyond performance improvement) in which a student becomes similar to the teacher by inheriting its implicit properties.
3 Distillation methods studied
To ensure that our findings are general and cover a range of distillation techniques, we select standard methods representative of three families of distillation techniques: output-based [12], feature-based [26], and contrastive-based [30]. The objectives of these methods, described below, are combined with the cross entropy loss  $\mathcal{L}_{\mathrm{CLS}}(\mathbf{z_s},\mathbf{y})\coloneqq -\sum_{j = 1}^{c}y_j\log \sigma_j(\mathbf{z_s})$ , where  $\mathbf{y}$  is the
Independent student
Distilled student
ResNet50  $\rightarrow$  ResNet18
VGG19  $\rightarrow$  VGG11
Figure 2: Left: An example of how the distilled student can focus on similar regions as the teacher while classifying an image. Right:  $\%$  where teacher's CAM is more similar to the distilled student's CAM than to the independent student's CAM. The red line indicates chance performance (50%).
ground-truth one-hot label vector,  $\mathbf{z_s}$  is the student's logit output,  $\sigma_{j}(\mathbf{z}) = \exp (z_{j}) / \sum_{i}\exp (z_{i})$  is the softmax function, and  $c$  is the number of classes.
(1) KL: [12] proposed to use the soft labels produced by the teacher as an additional target for the student to match, apart from the (hard) ground-truth labels. This is done by minimizing the KL-divergence between the predictive distributions of the student and the teacher:
$$
\mathcal {L} _ {\mathrm {K L}} \left(\mathbf {z} _ {\mathbf {s}}, \mathbf {z} _ {\mathbf {t}}\right) := - \tau^ {2} \sum_ {j = 1} ^ {c} \sigma_ {j} \left(\frac {\mathbf {z} _ {\mathbf {t}}}{\tau}\right) \log \sigma_ {j} \left(\frac {\mathbf {z} _ {\mathbf {s}}}{\tau}\right), \tag {1}
$$
where  $\mathbf{z}_{\mathbf{t}}$  is the logit output of the teacher, and  $\tau$  is a scaling temperature. The overall loss function is  $\gamma \mathcal{L}_{\mathrm{CLS}} + \alpha \mathcal{L}_{\mathrm{KL}}$ , where  $\gamma$  and  $\alpha$  are balancing parameters. We refer to this method as  $KL$ .
(2) Hint: FitNets [26] makes the student's intermediate features  $(\mathbf{f_s})$  mimic those of the teacher's  $(\mathbf{f_t})$  for an image  $x$ , at some layer  $l$ . It first maps the student's features (with additional parameters  $r$ ) to match the dimensions of the teacher's features, and then minimizes their mean-squared error:
$$
\mathcal {L} _ {\text {H i n t}} \left(\mathbf {f} _ {\mathbf {s}} ^ {(1)}, \mathbf {f} _ {\mathbf {t}} ^ {(1)}\right) = \frac {1}{2} \left| \left| \mathbf {f} _ {\mathbf {t}} ^ {(1)} - r \left(\mathbf {f} _ {\mathbf {s}} ^ {(1)}\right) \right| \right| ^ {2} \tag {2}
$$
The overall loss is  $\gamma \mathcal{L}_{\mathrm{CLS}} + \beta \mathcal{L}_{\mathrm{Hint}}$ , where  $\gamma$  and  $\beta$  are balancing parameters. [26] termed the teacher's intermediate representation as Hint, and we adopt this name.
(3) CRD: Contrastive representation distillation [30] proposed the following. Let  $s(x)$  and  $t(x)$  be the student's and teacher's penultimate feature representation for an image  $x$ . If  $x$  and  $y$  are from different categories, then  $s(x)$  and  $t(x)$  should be similar (positive pair), and  $s(x)$  and  $t(y)$  should be dissimilar (negative pair). A key for better performance is drawing a large number of negative samples  $N$  for each image, which is done using a constantly updated memory bank.
$$
\mathcal {L} _ {\mathrm {C R D}} = - \log h (s (x), t (x)) - \sum_ {j = 1} ^ {N} \log \left(1 - h (s (x), t (y _ {j}))\right) \tag {3}
$$
where  $h(a,b) = (e^{a\cdot b / \tau}) / (e^{a\cdot b / \tau} + \frac{N}{M})$ ,  $M$  is the size of the training data,  $\tau$  is a scaling temperature, and  $\cdot$  is the dot product. We use CRD to refer to this method. All other implementation details (e.g., temperature for KL, layer index for Hint) can be found in appendix.
We now discuss our experimental design. To reach conclusions that are generalizable across different architectures and datasets, and robust across independent runs, we experiment with a variety of teacher-student architectures, and tune the hyperparameters so that the distillation objective improves the test performance of the student compared to independent training. For each setting, we average the results over two independent runs. We report the top-1 accuracy of all the models in the appendix. The notation  $\mathsf{Net}_1\to \mathsf{Net}_2$  indicates distilling the knowledge of  $\mathsf{Net}_1$  (teacher) into  $\mathsf{Net}_2$  (student).
4.1 Does localization knowledge get distilled?
We start by studying whether the localization properties of a teacher transfers to a student through knowledge distillation. For example, suppose that the teacher classifies an image as a cat by focusing
Figure 3: The images which fool the teacher fool the distilled student more than the independent student in (a), (b) and (d), but not in (e). If the adversarial attack is generated using a foreign network (ResNet18, (c)), it fails to convincingly fool the distilled student more than the independent one.
on its face, and an independent student classifies it as a cat by focusing on its fur. After knowledge distillation, will the student focus more on the face when classifying the image as a cat?
Experimental setup: We train three models for ImageNet classification: a teacher, a distilled and an independent student. For each network, we obtain the class activation map (CAM) of the ground-truth class for each of random 5000 test images, using Grad-CAM [27], which visualizes the pixels that a classifier focuses on for a given class. We then compute how often (in terms of % of images) the teacher's CAM is more similar (using cosine similarity) to the distilled student's CAM than to the independent student's CAM. A score of  $>50\%$  means that, on average, the distilled student's CAMs become more similar to those of the teacher than without distillation. As a sanity check, we also compute the same metric between the teacher's CAM and the CAMs of two independent students trained with different seeds (Ind), which should be equally distant from the teacher (score of  $\sim 50\%$ ).
Results: Fig. 2 (right) shows the results for two configurations of teacher-student architectures: (i) ResNet50  $\rightarrow$  ResNet18 and (ii) VGG19  $\rightarrow$  VGG11. For KL and CRD, we observe a significant increase in similarity compared to random chance (as achieved by the Ind baseline). Hint also shows a consistent increase, although the magnitude is not as large.
Discussion: This experiment shows that having access to the teacher's class-probabilities, i.e. confidence for the ground-truth class, can give information on where the teacher is focusing on while making the classification decision. This corroborates, to some degree, the result obtained in the Grad-CAM paper [27], which showed that if the network is very confident of the presence of an object in an image, it focuses on a particular region (Fig. 1(c) in [27]), and when it is much less confident, it focuses on some other region (Fig. 7(d) in [27]). Fig. 2 (left) shows a sample test image and the corresponding CAMs produced by the independent student (left), teacher (middle), and distilled student (right) for the ground-truth class. The distilled student looks at similar regions as the teacher, and moves away from the regions it was looking at initially (independent student). So, regardless of the correctness of any network's CAM, our analysis shows that a distilled student's CAM does become similar to those of the teacher for all three distillation techniques, albeit with varying degrees.
4.2 Does adversarial vulnerability get distilled?
Next, we study the transfer of a different kind of property. If we design an adversarial image to fool the teacher, then will that same image fool the distilled student more than the independent student?
Experimental setup: We train a teacher, a distilled student, and an independent student for ImageNet classification. Given 5000 random test images, we convert each image  $I$  into an adversarial image  $I^{adv}$  using iterative FGSM [9, 17] (see appendix for details), whose goal is to fool the teacher, so that teacher's prediction for  $I^{adv}$  changes from its original prediction for  $I$ . Fooling rate is then defined as the fraction of adversarial images which succeed at this task. In our experiments, we use only this fraction of adversarial images which fool the teacher, and apply them to different students.
Results: We evaluate four configurations: (i) ResNet50  $\rightarrow$  ResNet18, (ii) VGG19  $\rightarrow$  VGG11, (iii) VGG19  $\rightarrow$  ResNet18, (iv) ViT (ViT-b-32) [4]  $\rightarrow$  ResNet18. The fooling rate is  $\sim 85\%$  for all the teachers. Fig. 3 shows the fooling rate (y-axis) when applying these successful adversarial images to different types of students (x-axis). We see that for ResNet50  $\rightarrow$  ResNet18, the ability to fool the independent student drops to  $44.2\%$ , which is expected since the adversarial images aren't designed for that student. In distilled students, we see an increase in the fooling rate relative to the independent one across all distillation methods ( $48\% - 52\%$ ). The trend holds for VGG19  $\rightarrow$  VGG11 and VGG19
Figure 4: (a) Agreement between two images with different color properties. * indicates distillation done by a teacher  $\mathrm{T}^*$  not trained to be color invariant. (b, c) Agreement between two images having increasingly different color properties. (d) Effect of different layers used in Hint distillation.
$\rightarrow$  ResNet18; Fig. 3 (b, d). When distillation is done from a transformer to a CNN ( $\mathrm{ViT} \rightarrow \mathrm{ResNet18}$ ) the fooling rates remain similar for the independent and distilled students; Fig. 3 (e). Here, we don't see the student becoming similar to the teacher to the extent observed for a CNN  $\rightarrow$  CNN distillation.
Discussion: This result is surprising. Iterative FGSM is a white box attack, which means that it has full access to the target model (teacher), including its weights and gradients. Thus, the adversarial examples are specifically crafted to fool the teacher. In contrast, the student never has direct access to the weights and gradients of the teacher, regardless of the distillation method. Yet, by simply trying to mimic the teacher's soft probabilities  $(KL)$  or an intermediate feature layer (Hint, CRD), the student network inherits, to some extent, the particular way that a teacher is fooled.
We conduct an additional study to ensure that the reason the distilled student is getting fooled more is because it is being attacked specifically by its teacher's adversarial images; i.e., if those images are designed for some other network, would the difference in fooling rates still be high? We test this in the VGG19  $\rightarrow$  VGG11 setting. This time we generate adversarial images  $(I\to I^{adv})$  to fool an ImageNet pre-trained ResNet18 network, instead of VGG19 (the teacher). We then use those images to attack the same VGG11 students from the VGG19  $\rightarrow$  VGG11 setting. In Fig. 3 (c), we see that the fooling rates for the independent and distilled students remain similar. This indicates that distillation itself does not make the student more vulnerable to any adversarial attack, and instead, an increase in fooling rate can be attributed to the student's inheritance of the teacher's adversarial vulnerability.
4.3 Does invariance to data transformations get distilled?
We have studied whether the response properties on single images get transferred from the teacher to the student. Now suppose that the teacher is invariant to certain changes in data, either learned explicitly through data augmentation or implicitly due to architectural choices. Can such properties about changes in images get transferred during distillation?
Experimental setup: We study color invariance as a transferable property. We train three models for ImageNet classification: a teacher, a distilled and an independent student. While training the teacher, we add color jitter in addition to the standard augmentations (random crops, horizontal flips). Specifically, we alter an image's brightness, contrast, saturation, and hue, with magnitudes sampled uniformly in [0, 0.4] for the first three, and in [0, 0.2] for hue. This way, the teacher gets to see the same image with different color properties during training and can become color invariant. When training the student, we only use the standard augmentations without color jittering, and see whether such a distilled student can indirectly inherit the color invariance property through the teacher.
Results: We start with the ResNet50  $\rightarrow$  ResNet18 configuration. After training, we evaluate the models on 50k ImageNet validation images. For each image  $X$ , we construct its augmented version  $X'$  by altering the color properties in the same way as done while training the teacher. Fig. 4 (a) depicts the agreement scores between  $X$  and  $X'$  (y-axis) for different models (x-axis). The teacher (T), being trained to have that property, achieves a high score of  $92.5\%$ . The independent student, which is not trained to be color invariant, has a much lower score of  $71.3\%$ . But, when the color-invariant teacher is used to perform distillation using KL or CRD, the agreement scores of the students jump up to  $82.1\%$  and  $79.5\%$  respectively. To ensure that this increase is not due to some regularizing property of the distillation methods that has nothing to do with the teacher, we repeat the experiment, this time with a ResNet50 teacher ( $T^*$ ) that is not trained with color augmentation. The new agreement scores for the distilled students (marked by  $*$ ; e.g.,  $\mathrm{KL}^*$ ) drop considerably
Figure 5: Left: Does knowledge transferred about one domain give knowledge about other unseen domains? Right: Consensus scores between teacher and student for images from unseen domains.
compared to the previous case. This is a strong indication that the student does inherit teacher-specific invariance properties during distillation.
In Fig. 4(b), we show that this trend in agreement scores (y-axis) holds even when the magnitude of change in brightness, contrast and saturation (x-axis) is increased to create  $X'$  from  $X$ . We repeat this experiment for Swin-tiny (a transformer) [20] → ResNet18 in Fig. 4(c). We again see some improvements in the agreement scores of the distilled students. This increase, however, is not as significant as that observed in the previous CNN → CNN setting. Throughout this work, we find that distilling the properties from a transformer teacher into a CNN student is difficult. The implicit biases introduced due to architectural differences between the teacher (transformer) and student (CNN) seem too big, as was studied in [25], to be overcome by current distillation methods.
We also note the ineffectiveness of  $\text{Hint}$  in distilling this knowledge. Our guess for this is the choice of  $l$  for distillation, which is typically set to be in the middle of the network as opposed to deeper layers where  $KL$  and  $CRD$  operate. So, we perform an ablation study for  $\text{Hint}$ , where the student mimics a different feature of the teacher each time; starting from a coarse feature of resolution  $56 \times 56$  (F1) to the output scores (logits) of the teacher network (F7). We plot the agreement scores in Fig. 4(d), where we see that the score increases as we choose deeper features. Mimicking the deeper layers likely constrains the overall student more compared to a middle layer, since in the latter case, the rest of the student (beyond middle layer) could still function differently compared to the teacher.
Discussion: In sum, color invariance can be transferred during distillation. This is quite surprising since the teacher is not used in a way which would expose any of its invariance properties to the student. Remember that all the student has to do during distillation is to match the teacher's features for an image  $X$ ; i.e., the student does not get to see its color augmented version,  $X'$ , during training. If so, then why should it get to know how the teacher would have responded to  $X'$ ?
4.4 Does knowledge about unseen domains get distilled?
So far, our analysis has revolved around the original ImageNet dataset, something that was used to perform the distillation itself. So, does the teacher only transfer its knowledge pertaining to this domain, or also of domains it has never seen (Fig. 5 left)?
Experimental setup: To test this, we first perform distillation using two settings (i) VGG19  $\rightarrow$  ResNet18 and (ii) Swin-Base [20]  $\rightarrow$  Swin-Tiny, where the training of the teacher, as well as distillation is done on ImageNet. During test time, we take an image from an unseen domain and see how frequently the student's and teacher's class predictions match for it (regardless of whether the predicted label is correct or incorrect), which we call the consensus score. For the unseen domains, we consider the five datasets proposed in [7]: sketch, stylized, edge, silhouette, cue conflict. Images from these domains are originally from ImageNet, but have had their properties modified.