SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Results: Fig. 5 (right) shows the consensus scores between the teacher and the student (y-axis). We see a nearly consistent increase in consensus brought about by the distillation methods in both settings. The extent of this increase, however, differs among the domains. There are cases, for example, in VGG19  $\rightarrow$  ResNet18, where consensus over images from edge domain increases over  $100\%$  ( $12\% \rightarrow \geq 30\%$ ) by some distillation methods. On the other hand, the increase can also be relatively modest in certain cases: e.g. Swin-Base  $\rightarrow$  Swin-Tiny in stylized domain.
Discussion: [29] showed that the agreement in classification between the teacher and distilled student is not much different than that to an independent student on CIFAR. We find this to be true in our ImageNet classification experiments as well. That is why the increase in agreement observed for
Figure 6: Left: The teacher is trained on three domains: MNIST-Orig, MNIST-Color, and MNIST-M. Distillation is done only on MNIST-Orig. Right: Test accuracy of the students. Note the increase in performance on MNIST-Color & MNIST-M domains by the distilled students.
unseen domains is surprising. After all, if the agreement is not increasing when images are from the seen domain, why should it increase when they are from an unseen domain? It is possible that there is more scope for increase in agreement in unseen domains vs seen domain. The consensus score between teacher and independent student for the seen domain is  $\geq 75\%$  (appendix), whereas for an unseen domain, e.g. sketch, it is  $\leq 40\%$ . So, it might not be that knowledge distillation does not work, as the authors wondered in [29], but its effect could be more prominent in certain situations.
4.5 Other studies
In appendix, we study additional aspects of a model such as shape/texture bias, invariance to random crops, and find that even these obscure properties can transfer from a teacher to the student. We also explore the following questions: If we can find alternative ways of increasing a student's performance (e.g. using crafted soft labels), will that student gain similar knowledge as a distilled student? If distillation cannot increase the student's performance, is there no knowledge transferred? Finally, we also present results on some more datasets like CIFAR100 [16], VLCS and PACS [18], showing that the phenomena of the implicit transfer of properties dueing knowledge distillation extends even to datasets beyond ImageNet and MNIST.
Beyond the exploratory angle of studying whether certain properties get transferred during distillation, the idea of a student becoming similar to a teacher in a broad sense has practical implications. We discuss a good and a bad example in this section.
5.1 The Good: the free ability of domain adaptation
Consider the following setup: the teacher is trained for a task by observing data from multiple domains  $(\mathcal{D}_1 \cup \mathcal{D}_2)$ . It is then used to distill knowledge into a student on only  $\mathcal{D}_1$ . Apart from getting knowledge about  $\mathcal{D}_1$ , will the student also get knowledge about  $\mathcal{D}_2$  indirectly?
Experimental setup: We use MNIST digit recognition, and train the teacher on three domains: (i) MNIST-orig: original gray-scale images from MNIST, (ii) MNIST-Color: background of each image randomly colored, and (iii) MNIST-M [6]: MNIST digits pasted on random natural image patches. The student models are trained only on MNIST-orig and evaluated (top-1 accuracy) on all three domains. The network architecture is same for both the teacher and the student (see appendix).
Results: When the independent student is trained only on MNIST-orig, its performance drops on the unseen domains, which is expected due to domain shift. The distilled students (especially  $KL$  & Hint), however, are able to significantly improve their performance on both unseen domains; Fig. 6.
Discussion: This result shows distillation's practical benefits: once a teacher acquires an ability through computationally intensive training (e.g., training on multiple datasets), that ability can be distilled into a student, to a decent extent, through a much simpler process. The student sees the teacher's response to gray-scale images (MNIST-orig) that lack any color information. But that information helps the student to deal better with colored images (e.g., MNIST-Color), likely because the teacher has learned a domain-invariant representation (e.g., shape) which is distilled to the student.
5.2 The Bad: Students can inherit harmful biases from the teacher
Consider the problem of classifying gender from human faces. Imagine an independent student which performs the classification fairly across all races. The teacher, on the other hand, is biased against certain races, but is more accurate than the student on average. Will the student, which was originally fair, become unfair after mimicking the unfair teacher?
Experimental setup: We consider a ResNet20  $\rightarrow$  ResNet20 setting, and use FairFace dataset [15], which contains images of human faces from 7 different races with their gender labeled. From its training split, we create two different subsets  $(\mathcal{D}_s$  and  $\mathcal{D}_t)$  with the following objectives - (i)  $\mathcal{D}_s$  has a particular racial composition so that a model trained on it will perform fairly across all races during test time; (ii)  $\mathcal{D}_t$  's composition is intended to make the model perform unfairly for certain races. The exact composition of  $\mathcal{D}_s$  and  $\mathcal{D}_t$  is given in the appendix. The teacher is trained on  $\mathcal{D}_t$  whereas the independent/distilled students are trained on  $\mathcal{D}_s$ . We use  $KL$  for distillation.
Results: We observe in the figure on the right that the independent student performs roughly fairly across all races, which is what we intended. The teacher is more accurate than the student, but performs relatively poorly on faces from Race 3 compared to others. After distillation, we observe that the student's overall accuracy improves, but the gain in accuracy is less for images from Race 3. So, when it is trained on  $\mathcal{D}_s$  by itself, it behaves fairly. But when it mimics the teacher on  $\mathcal{D}_s$ , it becomes unfair.
This follows the observation made in [21], where the authors found that the performance gains during distillation might not be spread uniformly across different sub-categories.
Discussion: The practical takeaway from the experiment is that knowledge distillation can bring forth behaviour which is considered socially problematic if it is viewed simply as a blackbox tool to increase a student's performance on test data, as it did in the previous example. Proper care must hence be taken regarding the transfer/amplification of unwanted biases in the student.
6 Why does knowledge distillation work in this way?
An Illustrative Example. Why should a teacher's response to an image contain such rich information about its implicit properties? We first intuitively explain this through a toy classification problem using  $KL$  as the distillation objective. Fig. 7 (left) shows data points from two classes (red and blue). The teacher has access to the complete set, thereby learning to classify them appropriately (orange decision boundary). On the other hand, suppose that the training of the independent and distilled student is done on a subset of the points. The independent student can learn to classify them in different ways, where each decision boundary can be thought of as looking for different features, hence giving rise to different properties. But the distilled student is trying to mimic the teacher's class probabilities, which reflect the distance from the (orange) decision boundary of the teacher, visualized as circles around the points. Then the decision boundary of the distilled student should remain tangential to these circles so that it preserves the teacher's distances. So, this very simple case depicts that mimicking the class probabilities can force the student's decision boundary to resemble that of the teacher, which is not true for the independent student.
A mathematical formulation of the same case can be considered as follows. Let  $D$  be a linearly separable dataset:  $D \in \{(x_{1},y_{1}),(x_{2},y_{2}),\dots \dots,(x_{n},y_{n})|x_{i} \in \Re^{m},y_{i} \in \{-1,1\} \}$ . For an arbitrary linear classifier having weights  $W \in \Re^{m\times 1}$  and biases  $b \in \Re$ , the binary classification objective for the teacher and the independent student can be described as  $\forall (x_i,y_i) \in D, (W^T x_i + b)y_i > 0$ .
Since the dataset is linearly separable, both the teacher as well as the independent student can solve this system of inequalities in many ways. Let  $\{W_t, b_t\}$  be the parameters of one particular teacher.
Next, we look at the objective of the distilled student, who tries to map  $x_{i}$  to  $z_{i} = W_{t}^{T}x_{i} + b_{t}$ , instead of  $y_{i}$ . That is, the optimal solution satisfies:  $\forall (x_{i},z_{i})\in D_{dist},W_{s}^{T}x_{i} + b_{s} = z_{i}$ . This system of linear equations will have a unique solution provided it has access to at least  $m + 1$  training instances and that these instances are linearly independent. Since we know that the teacher's weights and
Decision boundaries
Teacher
Independent student
Figure 7: Left. Teacher learns on the the entire dataset. Independent student without access to the full dataset might learn some spurious correlations. Distillation provides constraints to reconstruct the teacher's decision boundary. Right. Sampled decision boundaries for points from MNIST-Color visualized using [28]. The distilled student is better at reconstructing the teacher's decision boundary.
biases,  $\{W_t, b_t\}$ , can satisfy this equation, we can conclude that if there exists a unique solution, then  $W_s = W_t$  and  $b_s = b_t$ . So, the distilled student, when solving the distillation objective, will recover the weights and biases of the teacher, and consequently its decision boundary.
Complex neural networks: Given that the distances of data points, i.e., decision boundary, can be preserved by the distilled student in the very simple case presented above, we now study whether they can be preserved in a more complex setting involving multi-layer neural networks as well.
There have been works [8, 28] which have studied this for the  $KL$  objective: if the teacher and the students are all trained on the same dataset  $\mathcal{D}$ , the decision boundary of the teacher will be more similar to the distilled student than the independent student for points from  $\mathcal{D}$ . And while this result is useful, it is not clear whether (i) this can help explain why the student inherits teacher's knowledge about other domains, and (ii) whether this holds true for distillation objectives other than  $KL$ .
To study this, we revisit the setup of Sec. 5.1, where the teacher was trained on three domains and distillation was performed on one (Fig. 6 left). We study what happens to the decision boundary similarity between the students and the teacher on all three domains. To compute that similarity between two neural networks, we use the method introduced in [28]. We randomly sample a triplet of data points from a given domain and create a plane that passes over that triplet. The networks that are being compared receive the same dataset of points in this plane, and we compute intersection over union between their predictions. We report the score averaged over 50 triplets.
Ind	KL	Hint	CRD
MNIST-Orig	0.9107	0.9341	0.9317	0.9062
MNIST-Color	0.4156	0.6711	0.7980	0.4733
MNIST-M	0.4691	0.5872	0.6541	0.4862
Table 1: Decision boundary similarity calculated between students and the teacher.
Results: Table 1 shows the similarity scores, where we see that the distilled students' decision boundary are almost always more similar to the teacher's than the independent student's. This is particularly evident when the scores are computed for domains not seen by the students; e.g., for MNIST-Color, the similarity score increases from 0.416 to 0.671 for the student distilled with  $KL$ .
Discussion: First, we can draw an analogy of this experimental setup with the toy example discussed before. The three MNIST related domains (for training the teacher) are similar to the overall set of blue+red points used to train the toy teacher (Fig. 7 left). The singular MNIST-orig domain (for training the students) is similar to the three points available to train the toy students. Now, what the results from Table 1 show is that the data points available for distillation can help determine the teacher's decision boundary even for points which the student did not have access to. So, similar to how the decision boundary estimated by the toy student can correctly estimate the distances of missing blue/red points, the neural network based student can also estimate the distances of points from unseen domains, e.g., MNIST-color, in a similar way as its teacher, thereby inheriting the teacher's behavior on these points. An example of this is given in Fig. 7 (right).
While we have tried to study the distillation process in many settings, there do exist many more which remain unexplored. This is because the analysis conducted in our work has many axes of generalization: whether these conclusions hold (i) for other datasets (e.g. SUN scene classification dataset), (ii) in other kinds of tasks (object detection, semantic segmentation), (iii) in other kinds of architectures (modern CNNs, like ResNext), (iv) or for more recent distillation objectives, etc. Moreover, in Sec. 3.5 in the supplementary, we explore the transferability of shape/texture bias from teacher to student, and find some discrepancy in different distillation objective's abilities. Therefore, the conclusions drawn from our work will be more helpful if one can study all the combinations of these factors. Due to limitations in terms of space and computational requirements, however, we have not been able to study all of those combinations.
Knowledge distillation is a beautiful concept, but its success i.e., increase in student's accuracy, has often been explained by a transfer of dark knowledge from the teacher to the student. In this work, we have tried to shed some light on this dark knowledge. There are, however, additional open questions, which we did not tackle in this work: given the architectures of the teacher and student, is there a limit on how much knowledge can be transferred (e.g., issues with ViT  $\rightarrow$  CNN)? If one wants to actively avoid transferring a certain property of the teacher into a student (Sec. 5.2), but wants to distill other useful properties, can we design an algorithm tailored for that? We hope this work also motivates other forms of investigation to have an even better understanding of the distillation process.
This work was supported in part by NSF CAREER IIS2150012, and Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration), Air Force Grant FA9550-18-1-0166, the National Science Foundation (NSF) Grants 2008559-IIS, 2023239-DMS, and CCF-2046710.
[1] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In arXiv, 2021.
[2] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In SIGKDD, 2006.
[3] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In arXiv, 2020.
[5] Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, 2018.
[6] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 2016.
[7] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In NeurIPS, 2021.
[8] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarily robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[9] Ian Goodfellow, Jon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2014.
[10] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. In arXiv, 2021.
[11] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Dark knowledge. TTIC Distinguished lecture series, 2014.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS Deep Learning Workshop, 2014.
[13] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. In arXiv, 2017.
[14] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In ICML, 2021.
[15] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In WACV, 2021.
[16] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. In Technical report, CiteSeer, 2009.
[17] Alexey Kurakin, Ian Goodfellow, and Sammy Bengio. Adversarial machine learning at scale. In arXiv, 2016.
[18] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
[19] Yuan Li, Francis E.H.Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In CVPR, 2020.
[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[21] Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. In arXiv, 2021.
[22] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
[23] Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, 2019.
[24] Max Phuong and Christoph H. Lampert. Towards understanding knowledge distillation. In ICML, 2019.
[25] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurIPS, 2021.
[26] Adriana Romero, Nicholas Ballas, Samira Ebrahimi Kahau, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[27] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IJCV, 2019.
[28] Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang, Yehuda Dar, Richard Baraniuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. arXiv preprint arXiv:2203.08124, 2022.
[29] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In NeurIPS, 2021.
[30] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[31] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV, 2019.
[32] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[33] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[34] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Hutchuan Lu. Deep mutual learning. In CVPR, 2018.
[35] Zhilu Zhang and Mert Sabuncu. Self-distillation as instance-specific label smoothing. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2184-2195. Curran Associates, Inc., 2020.