SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When used as the student model, the smallest variant of Swin-T demonstrates superior performance compared to several teachers when trained from scratch. For example, Swin-T outperforms Mixer-B/16 on the ImageNet-1K dataset. To facilitate comparisons between such teacher-student combinations, we introduce two modified versions of Swin-T: Swin-Nano and Swin-Pico, referred to as Swin-N and Swin-P respectively.
Datasets. We adopt the CIFAR-100 dataset [52] and the ImageNet-1K dataset [53] for evaluation. CIFAR-100 comprises 50K training samples and 10K testing samples of  $32 \times 32$  resolution, while the ImageNet-1K dataset is more extensive, containing 1.2 million training samples and 50,000 validation samples, all with a resolution of  $224 \times 224$ . Since ViTs and MLPs accept image patches as input, we upsample the images in CIFAR-100 to the resolution of  $224 \times 224$  for all subsequent experiments to facilitate the patch embedding process.
Baselines. We employ both logits-based and hint-based knowledge distillation (KD) approaches as our baselines. Specifically, the hint-based approaches include FitNet [6], CC [54], RKD [37], and CRD [40], while the logits-based baselines consist of KD [5], DKD [55], and DIST [9].
<sup>4</sup>We take the assumption of  $\gamma \in \mathbb{Z}_{+}$  above just for convenience, and there is no limitation of requiring  $\gamma$  to be an integer in practice.
Table 1: KD methods with heterogeneous architectures on ImageNet-1K. The best results are indicated in bold, while the second best results are underlined. †: results achieved by combining with FitNet.
Teacher	Student	From Scratch	hint-based	Logits-based
T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST	OFA
CNN-based students
DeiT-T	ResNet18	72.17	69.75	70.44	69.77	69.47	69.25	70.22	69.39	70.64	71.34
Swin-T	ResNet18	81.38	69.75	71.18	70.07	68.89	69.09	71.14	71.10	70.91	71.85
Mixer-B/16	ResNet18	76.62	69.75	70.78	70.05	69.46	68.40	70.89	69.89	70.66	71.38
DeiT-T	MobileNetV2	72.17	68.87	70.95	70.69	69.72	69.60	70.87	70.14	71.08	71.39
Swin-T	MobileNetV2	81.38	68.87	71.75	70.69	67.52	69.58	72.05	71.71	71.76	72.32
Mixer-B/16	MobileNetV2	76.62	68.87	71.59	70.79	69.86	68.89	71.92	70.93	71.74	72.12
ViT-based students
ResNet50	DeiT-T	80.38	72.17	75.84	72.56	72.06	68.53	75.10	75.60†	75.13†	76.55†
ConvNeXt-T	DeiT-T	82.05	72.17	70.45	73.12	71.47	69.18	74.00	73.95	74.07	74.41
Mixer-B/16	DeiT-T	76.62	72.17	74.38	72.82	72.24	68.23	74.16	72.82	74.22	74.46
ResNet50	Swin-N	80.38	75.53	78.33	76.05	75.90	73.90	77.58	78.23†	77.95†	78.64†
ConvNeXt-T	Swin-N	82.05	75.53	74.81	75.79	75.48	74.15	77.15	77.00	77.25	77.50
Mixer-B/16	Swin-N	76.62	75.53	76.17	75.81	75.52	73.38	76.26	75.03	76.54	76.63
MLP-based students
ResNet50	ResMLP-S12	80.38	76.65	78.13	76.21	75.45	73.23	77.41	78.23†	77.71†	78.53†
ConvNeXt-T	ResMLP-S12	82.05	76.65	74.69	75.79	75.28	73.57	76.84	77.23	77.24	77.53
Swin-T	ResMLP-S12	81.38	76.65	76.48	76.15	75.10	73.40	76.67	76.99	77.25	77.31
**Optimization.** In our implementation, we employ different optimizers for training the student models based on their architecture. Specifically, all CNN students are trained using the SGD optimizer, while those with a ViT or MLP architecture are trained using the AdamW optimizer. For the CIFAR-100 dataset, all models are trained for 300 epochs. When working with the ImageNet-1K dataset, CNNs are trained for 100 epochs, whereas ViTs and MLPs are trained for 300 epochs.
4.2 Distillation results on ImageNet-1K
We first conduct experiments on the ImageNet-1K dataset. To make a comprehensive comparison, we adopt five student models and five teacher models belonging to three different model architecture families, and evaluate KD approaches on fifteen heterogeneous combinations of teacher and student. We report the results in Table 1.
In comparison to previous KD methods, our distillation framework achieves remarkable performance improvements. Specifically, when training a student based on CNN architecture, our proposed method achieves performance improvements ranging from  $0.20\%$  to  $0.77\%$  over the second-best baselines. When the student model is based on a ViT or MLP architecture, which utilizes image patches as input, our OFA-KD also achieves remarkable performance improvement, with a maximum accuracy gain of  $0.71\%$ . Significantly, the hint-based baseline FitNet exhibits noteworthy performance when the teacher model is ResNet50 and the student model is a ViT or MLP. We speculate that designs based on traditional  $3\times 3$  convolution of ResNet50 have the capability to capture local information, such as texture details, better than ViT/MLP. In the process of "patchify", ViT/MLP might overlook these local details. Therefore, when ResNet50 serves as the teacher, FitNet can provide valuable intermediate representations. We thus report the results of logit-based methods obtained by combining FitNet and OFA-KD when adopting ResNet50 as the teacher.
Based on our experimental results, we observe that the second-best performance is not consistently achieved by the same baseline approach. FitNet, vanilla KD, and DIST take turns in attaining the second-best results, indicating that previous KD approaches face challenges in consistently improving performance when distilling knowledge between heterogeneous architectures. However, our proposed approach consistently outperforms all baselines in all cases, showcasing its effectiveness in cross-architecture distillation.
Table 2: KD methods with heterogeneous architectures on CIFAR-100. The best results are indicated in bold, while the second best results are underlined.
Teacher	Student	From Scratch	hint-based	Logits-based	OFA
T.	S.	FitNet	CC	RKD	CRD	KD	DKD	DIST
CNN-based students
Swin-T	ResNet18	89.26	74.01	78.87	74.19	74.11	77.63	78.74	80.26	77.75	80.54
ViT-S	ResNet18	92.04	74.01	77.71	74.26	73.72	76.60	77.26	78.10	76.49	80.15
Mixer-B/16	ResNet18	87.29	74.01	77.15	74.26	73.75	76.42	77.79	78.67	76.36	79.39
Swin-T	MobileNetV2	89.26	73.68	74.28	71.19	69.00	79.80	74.68	71.07	72.89	80.98
ViT-S	MobileNetV2	92.04	73.68	73.54	70.67	68.46	78.14	72.77	69.80	72.54	78.45
Mixer-B/16	MobileNetV2	87.29	73.68	73.78	70.73	68.95	78.15	73.33	70.20	73.26	78.78
ViT-based students
ConvNeXt-T	DeiT-T	88.41	68.00	60.78	68.01	69.79	65.94	72.99	74.60	73.55	75.76
Mixer-B/16	DeiT-T	87.29	68.00	71.05	68.13	69.89	65.35	71.36	73.44	71.67	73.90
ConvNeXt-T	Swin-P	88.41	72.63	24.06	72.63	71.73	67.09	76.44	76.80	76.41	78.32
Mixer-B/16	Swin-P	87.29	72.63	75.20	73.32	70.82	67.03	75.93	76.39	75.85	78.93
MLP-based students
ConvNeXt-T	ResMLP-S12	88.41	66.56	45.47	67.70	65.82	63.35	72.25	73.22	71.93	81.22
Swin-T	ResMLP-S12	89.26	66.56	63.12	68.37	64.66	61.72	71.89	72.82	11.05	80.63
4.3 Distillation results on CIFAR-100
In addition to the ImageNet-1K dataset, we also evaluate the proposed method on the CIFAR-100 dataset. We conduct experiments with twelve combinations of heterogeneous teacher and student models, and report the results in Table 2.
On this small-scale dataset, the hint-based approaches exhibit inferior performance, particularly when the student model is of ViT or MLP architecture, highlighting its limitations in distilling knowledge across heterogeneous architectures. For instance, FitNet only achieves a  $24.06\%$  accuracy when used in combination with a ConvNeXt-T teacher and a Swin-P student. By contrast, the OFA-KD method continues to achieve significant performance improvements, ranging from  $0.28\%$  to  $8.00\%$  over the second-best baselines.
Notably, when evaluating on the ImageNet-1K dataset, DIST achieves the second-best results at most times. while on the CIFAR-100 dataset, DKD tends to achieve the second-best results more frequently. This is because DIST relaxes the strict prediction imitation by correlation mimicry, which is more advantageous for learning from a stronger teacher trained on ImageNet-1K. On the other hand, DKD enhances the dark knowledge in teacher predictions, which proves beneficial when working with a smaller teacher trained on CIFAR-100. However, our OFA-KD method can enhance target information adaptively, enabling the student model to consistently achieve the best performance.
Number and position of exit branches. In our OFA-KD method, branches are integrated into the student to facilitate learning at intermediate layers. We conduct experiments to learn the impact of the number and position of branches. As the student models are divided into four stages, we use numbers 1 to 4 to indicate the insertion points of branches. As shown in Table 3, when comparing learning at the last layer with learning from intermediate layers, the ResNet18 student presents no preference to the position of the branch, as they all help improve the performance. However, DeiT-T only performs better when branches are added at the end of stage 1 or stage 4. In the evaluation of using multiple branches, The results demonstrate that learning at the end of all the four stages is the best configuration for the OFA-KD method.
Table 3: Impact of exit branch number and position on ImageNet-1K.  $\varnothing$  : only learning at the last layer.
Stage	T: DeiT-T S: ResNet18	T: ResNet50 S: DeiT-T
∅	70.62	75.18
{1}	70.75	75.35
{2}	70.70	75.21
{3}	70.78	75.17
{4}	70.78	75.28
{1,4}	71.04	75.52
{1,2,4}	71.08	75.61
{1,2,3,4}	71.34	75.73
Figure 3: Ablation study on different modulating parameter  $\gamma$  settings on ImageNet-1K.
Table 5: KD methods with homogeneous architectures on ImageNet-1K.  $T$  : ResNet34,  $S$  : ResNet18.
T.	S.	KD [5]	OFD [56]	Review [43]	CRD [37]	DKD [55]	DIST [9]	OFA
Accuracy	73.31	69.75	70.66	70.81	71.61	71.17	71.70	72.07	72.10
Table 6: Comparison of homogeneous and heterogeneous teacher on ImageNet-1K.
Teacher	T.	S.(ResNet50)	RKD [37]	Review [43]	CRD [37]	DKD [55]	DIST [9]	OFA
ResNet152	82.83	79.86	79.53	80.06	79.33	80.49	80.55	80.64
ViT-B	86.53	79.86	79.38	79.32	79.48	80.76	80.90	81.33
Scale of OFA loss. The modulation parameter is introduced at the exponential position of the term  $1 + \pmb{p}_{c}^{t}$ , which has a value greater than 1. This increases the overall loss scale. To ensure a stable training process, we compare two schemes: multiplying a scaling factor to the OFA loss and clipping gradient by setting a maximum norm value. As shown in Table 4, the best configuration is setting the scale factor to 1 and the maximum gradient norm to 5. Considering that searching the optimal scale factor and gradient clipping settings for each combination of teacher and student is expensive, we adopt the aforementioned best settings in all experiments.
Table 4: Comparison of scaling factors and clip grad settings on ImageNet-1K. The scaling factors are directly multiplied to the OFA loss. We adopt a ResNet50 teacher and a DeiT-T student with  $\gamma = 1.1$
Scaling factor	Acc.	Clip grad	Acc.
0.5	75.29	1	75.39
0.8	75.58	3	75.61
1.0	75.73	4	75.52
1.2	75.54	5	75.73
1.4	75.64	6	75.32
1.6	75.46	7	75.36
Modulating parameter. The modulation parameter  $\gamma$  controls the strength of the adaptive target information enhancement. To investigate its impact, we compare different configurations of the  $\gamma$  value setting on ImageNet-1K and present the results in Figure 3. Specifically, when the teacher model is DeiT-T and the student model is ResNet18, the best student performance is achieved when  $\gamma$  is set to 1.4. For the teacher-student pair of ResNet50 and DeiT-T, the optimal configuration for  $\gamma$  is 1.1. This difference in optimal  $\gamma$  values can be attributed to the varying strengths of the teacher models. As DeiT-T is a weaker teacher model compared to ResNet50, a larger  $\gamma$  value is required to mitigate the additional interference information. By increasing the  $\gamma$  value, the adaptive target information enhancement compensates for the limitations of the weaker teacher model, enabling the student model to better utilize the available information and improve its performance.
Distillation in homogeneous architectures. We adopt the commonly used combination of ResNet34 teacher and ResNet18 student on ImageNet-1K to further demonstrate the effectiveness of our method across homogeneous architectures. As the results shown in Table 5, the OFA method achieves performance on par with the best distillation baseline.
Homogeneous vs. heterogeneous teachers To assess the impact of utilizing a larger heterogeneous teacher model, we train a ResNet50 student with both a ResNet152 teacher (homogeneous architecture) and a ViT-B teacher (heterogeneous architecture). As indicated in Table 6, our OFA-KD approach achieves a notable  $0.41\%$  performance improvement when utilizing the ViT-B teacher compared to the ResNet152 teacher. This result underscores the significance of cross-architecture knowledge distillation (KD) in pursuing better performance improvement.
This paper studies how to distill heterogeneous architectures using hint-based knowledge. We demonstrate that the distinct representations learned by different architectures impede existing feature distillation approaches, and propose a one-for-all framework to bridge this gap in cross-architecture KD. Specifically, we introduce exit branches into the student to project features into the logits space and train the branches with the logits output of the teacher. The projection procedure discards architecture-specified information in feature, and thus hint-based distillation becomes feasible. Moreover, we propose a new loss function for cross-architecture distillation to adaptive target information enhancement, which is achieved by adding a modulating parameter to the reformulated KD loss. We conduct extensive experiments on CIFAR-100 and ImageNet-1K. Corresponding results demonstrate the effectiveness of the proposed cross-architecture KD method.
Limitation. While our approach achieves remarkable performance improvements in distilling ResNet50, it is worth noting that for some architectures, e.g., ResNet18, the performance distilled by a heterogeneous teacher is lower than that achieved by a homogeneous teacher. Furthermore, the requirement of finding an optimal setting for the modulating parameter adds a level of complexity. Improperly setting this parameter could potentially lead to inferior performance.
Border impact. Our work mainly studies hint-based distillation for heterogeneous architecture and proposes a generic framework. The performance improvement achieved by our OFA-KD demonstrates that there exists useful information between heterogeneous architectures. We hope that our research will bring new perspective to KD community for seeking better aligning approaches and inspire future works for cross-architecture distillation.
This work is supported by National Key Research and Development Program of China under No. SQ2021YFC3300128 National Natural Science Foundation of China under Grant 61971457. And Chang Xu was supported in part by the Australian Research Council under Projects DP210101859 and FT230100549.
[1] Zhiwei Hao, Yong Luo, Zhi Wang, Han Hu, and Jianping An. Cdfkd-mfs: Collaborative data-free knowledge distillation via multi-level feature sharing. IEEE Transactions on Multimedia, 2022. 1
[2] Zhiwei Hao, Yong Luo, Han Hu, Jianping An, and Yonggang Wen. Data-free ensemble knowledge distillation for privacy-conscious multimedia model compression. In Proceedings of the 29th ACM International Conference on Multimedia, 2021. 1
[3] Hanting Chen, Tianyu Guo, Chang Xu, Wenshuo Li, Chunjing Xu, Chao Xu, and Yunhe Wang. Learning student networks in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 1
[4] Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. 1
[5] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 1, 3, 6, 9
[6] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015. 1, 2, 3, 5, 6
[7] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017. 1, 3
[8] Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine-grained manifold distillation. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3
[46] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Advances in Neural Information Processing Systems, 2022. 3
[47] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems, 2021. 3
[48] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. J. Mach. Learn. Res., 13:795-828, 2012. 4
[49] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, volume 97, pages 3519-3529, 2019. 4
[50] Arthur Gretton, Kenji Fukumizu, Choon Hui Teo, Le Song, Bernhard Schölkopf, and Alexander J. Smola. A kernel statistical test of independence. In Neural Information Processing Systems, pages 585-592, 2007. 4
[51] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2018. 5
[52] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6
[53] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. 6
[54] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In IEEE/CVF International Conference on Computer Vision, 2019. 6
[55] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 6, 9
[56] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In International Conference on Computer Vision, pages 1921–1930, 2019. 9