SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and baseline at the beginning of the 3rd learning-rate decay stage. (2) Smaller teacher-student gap. As shown in Figure 7, features and logits of students via KD-Zero have stronger similarities with teachers than the original KD method.
In this section, we assess the efficacy of our proposed KD-Zero approach on classification, detection, and segmentation tasks. Additionally, we compare its performance with other KD methods, ensuring fair comparisons by utilizing the same training settings. We report mean results based on more than 3 repeated trials. More detailed experiment settings and results are available in the Appendix.
Table 3: Comparison of results on CIFAR-100. Most results of other methods refer to the original papers [8, 60]. Gain refers to the performance gain than baseline. We report Top-1 "mean (std)" accuracies (\%) for KD-Zero over 3 runs.
Model	Same architectural style	Different architectural style
Teacher	W-40-2	R110	R110	R32×4	VGG13	VGG13	R32×4	W-40-2	R56	R56	R56
Student	W-16-2	R20	R32	R8×4	VGG8	MNetV2	SNetV2	SNetV1	DeiT	T2T	PVT
Teacher	75.61	74.31	74.31	79.42	74.64	74.64	79.42	75.61	72.34	72.34	72.34
Student	73.26	69.06	71.14	72.50	70.36	64.60	71.82	70.50	65.08	69.37	69.22
FitNets [57]	73.58	68.99	71.06	73.50	71.02	64.14	73.54	73.73	70.82	71.96	73.42
AT [73]	74.08	70.22	72.31	73.44	71.43	59.40	72.73	73.32	73.51	74.01	73.60
SP [62]	73.83	70.04	72.69	72.94	72.68	66.30	74.56	74.52	67.36	72.26	70.48
RKD [48]	73.35	69.25	71.82	71.90	71.48	64.52	73.21	72.21	70.39	71.88	73.63
CRD [60]	75.48	71.46	73.48	75.51	73.94	69.73	75.65	76.05	NA	NA	NA
SRRL [29]	75.46	71.51	73.80	75.92	73.23	69.34	75.66	76.61	NA	NA	NA
KD [23]	74.92	70.67	73.08	73.33	72.98	67.37	74.45	74.83	73.25	74.15	73.68
DIST [27]	75.35	71.68	73.86	75.79	73.86	69.17	76.08	75.85	72.56	73.86	73.52
WSLD [79]	75.30	71.53	73.36	74.79	74.36	68.79	75.93	75.09	74.56	75.28	74.39
IPWD [47]	NA	71.32	73.91	76.03	NA	NA	NA	76.03	NA	NA	NA
KD-Zero	76.42	72.05	74.19	77.85	75.26	70.42	77.45	77.52	78.25	78.32	77.22
Gain±STD	3.16±0.16	2.99±0.21	3.05±0.12	5.35±0.22	4.90±0.11	5.82±0.18	5.63±0.25	7.02±0.14	13.17±0.36	8.95±0.29	8.00±0.32
4.1 Experiments on CIFAR-100
Implementation. We utilize the CIFAR-100 dataset [30] in knowledge distillation. During the distiller search phase, we apply  $5\%$  early-stopping training epochs with full training data for acceleration settings. Our evolutionary algorithm with 20 population sizes performs 100 iterations for each teacher-student pair. During the distillation phase, all teacher-student networks are trained using typical training settings, with a training epoch of 240. The multi-step learning rate commences at 0.1, which decays by 0.1 at 100 and 150 epochs. Recently, knowledge distillation enabled training Vision Transformers (ViT) from scratch with CNNs as teachers. To evaluate the effectiveness of KD-Zero, we conduct the evolutionary search for ViT-based distillation strategies with the same settings as the
Table 4: Comparison of results on ImageNet. The results of other methods quote the original papers report [8, 60, 71]. We report Top-1 "mean (std)" accuracies (\%) for KD-Zero.
Teacher	Student	Acc.	Teacher	Student	KD [23]	AT [73]	OFD [22]	SRRL [29]	CRD [60]	Review [50]	MGD [71]	KD-Zero
ResNet-34	ResNet-18	Top-1	73.40	69.75	70.66	70.69	70.81	71.73	71.17	71.61	71.58	72.17±0.15
Top-5	91.42	89.07	89.88	90.01	89.98	90.60	90.13	90.51	90.35	90.46±0.25
ResNet-50	MobileNet	Top-1	76.16	70.13	70.68	70.72	71.25	72.49	71.37	72.56	72.35	73.02±0.22
Top-5	92.86	89.49	90.30	90.03	90.34	90.92	90.41	91.00	90.71	91.05±0.26
CNN experiment. Subsequently, we train the ViT with the optimal distiller obtained and ResNet-56 as CNN teacher. The training is conducted on  $224 \times 224$  resolution images for 300 epochs, with an initial learning rate of 5e-4 and a weight decay 0.05 using the AdamW optimizer.
Comparison results on CNN models. Table 3 presents a comparative analysis of our KD-Zero with other state-of-the-art (SOTA) KD methods. We conduct multiple trials with randomly selected distillers in the same search space, called Rand-KD, to evaluate the efficacy of our EA search. For teacher-student pairs with the same architectural style, KD-Zero outperforms the baselines by margins ranging from  $3.16\% \sim 4.90\%$ . Compared with Rand-KD and other KDs, KD-Zero obtains consistent performance gains ( $1.0\% \sim 3.2\%$ ). Besides strengths in the same architecture pairs, KD-Zero exhibits even stronger performance when dealing with different architectural styles, while other KD methods suffer from noticeable accuracy reductions. Specifically, KD-Zero outperforms the baseline by margins of  $5.6\% \sim 7.3\%$  and the random search results by margins of  $1.9\% \sim 2.3\%$ , demonstrating the effectiveness of our design for different structures. Compared with other SOTA KD methods, our KD-Zero achieves  $1.2\% \sim 1.5\%$  gains. These results show that KD-Zero can improve each student model with simple settings under different teacher-student pairs.
Comparison results on vision transformer. Table 3 presents the results of the vanilla and distillation models employing different distillation methods. The results indicate that KD-Zero can significantly improve the performance of vision transformers with  $8.0\% \sim 13.1\%$  margins and consistently yields superior performance than other methods. In addition, our proposed method applies to various ViT architectures, thereby validating its effectiveness. Note that most ViT students possess larger model sizes (e.g., DeiT-Ti with 5 million parameters) and greater capabilities than the CNN teacher (e.g., ResNet-56 with only 0.86 million parameters). Some ViT students outperform the CNN model in the strong regularization setting on large-scale datasets. However, when it comes to ViT distillation on small datasets, employing CNN teachers helps address the issue of ViT models struggling to train effectively from scratch. In this context, using CNN teachers in distillation is akin to auxiliary training or providing additional regularization supervision. As a result, these ViTs demonstrate their original strong representation capability after distillation and consequently outperform the CNN teacher.
4.2 Experiments on ImageNet
Implementation. We additionally conduct experiments on the ImageNet[12]. Following CIFAR-100 trials, we employ similar EA settings on a subset of ImageNet for search acceleration. Then, we utilize the discovered distiller for the training of student models (e.g., ResNet-18 [21] and MobileNet [25]). The training settings are the same as the other KD methods and involve training for 100 epochs using a multi-step learning rate, which commences at 0.1 and decays by 0.1 at 30, 60, and 90 epochs.
Comparison results. As shown in Table 4, our proposed KD-Zero significantly improves the accuracy of baseline models, yielding gains of  $2.5\% \sim 2.9\%$  in Top-1 accuracy for ResNet-18 and MobileNet, respectively. In addition, KD-Zero surpasses other SOTA methods with clear gains, demonstrating its superiority in large-scale datasets. These findings substantiate the effectiveness of KD-Zero in distillation optimization with considerable benefits, establishing the versatility and potency of our framework. In summary, KD-Zero facilitates substantially improved predictive accuracy of student models on ImageNet, more complex domains while preserving superior performance.
Visualizations. The comparison between the Grad-CAM++ maps generated by the student model trained with KD-Zero and other methods is presented in Figure 8. The results indicate that the Grad-CAM++ map generated by the student model trained with KD-Zero is more similar to that of the teacher model compared to the student model trained independently. In contrast, independent training of the student model leads to incorrect focus areas. These findings suggest that the KD-Zero
Table 5: The results on the COCO val dataset for teacher (T) and student (S) models.
Method	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
Two-stage detectors
T: Cascade Mask RCNN-X101	45.6	64.1	49.7	26.2	49.6	60.0
S: Faster RCNN-R50	38.4	59.0	42.0	21.5	42.1	50.3
KD [24]	39.7	61.2	43.0	23.2	43.3	51.7
FKD [74]	41.5	62.2	45.1	23.5	45.0	55.3
CWD [59]	41.7	62.0	45.5	23.3	45.5	55.5
DIST [27]	40.4	61.7	43.8	23.9	44.6	52.6
KD-Zero	41.9	62.7	45.5	23.6	45.6	55.6
One-stage detectors
T: RetinaNet-X101	41.0	60.9	44.0	23.9	45.2	54.0
S: RetinaNet-R50	37.4	56.7	39.6	20.0	40.7	49.7
KD [24]*	37.2	56.5	39.3	20.4	40.4	49.5
FKD [74]	39.6	58.8	42.1	22.7	43.3	52.5
FGD [70]	40.4	59.9	43.3	23.4	44.7	54.1
DIST [27]	39.8	59.5	42.5	22.0	43.7	53.0
KD-Zero	40.9	60.4	43.5	23.2	45.2	54.8
Table 6: Results on Cityscapes val dataset with ImageNet Pretrain.
Method	mIoU (%)
T: DeepLabV3-R101	78.07
S: DeepLabV3-R18	74.21
SKD [44]	75.42
IFVD [65]	75.59
CWD [59]	75.55
CIRKD [68]	76.38
DIST [27]	77.10
KD-Zero	77.38
S: PSPNet-R18	72.55
SKD [44]	73.29
IFVD [65]	73.71
CWD [59]	74.36
CIRKD [68]	74.73
KD-Zero	76.25
Figure 9: Ablation study of search space (left) and organization (right) of ResNet-20 on C-100.
Figure 10: Comparison of search algorithms (left) and correlation visualization (right) of ResNet-20.
approach is more effective than traditional knowledge distillation methods in guiding the student model to learn from the teacher model, resulting in improved model interpretability and performance.
4.3 Experiments on Object Detection & Semantic Segmentation
Object detection. We conduct experiments on the MS-COCO dataset[41]. We use the optimal distiller on ImageNet to distill knowledge from teacher detectors to students. Based on the strong baseline [5], we apply KD-Zero to two-stage detector (e.g., Faster R-CNN [56]) and the one-stage detector (e.g., RetinaNet [40]), which are widely used object detection frameworks. Following common practice [40], all models are trained with a  $2 \times$  learning schedule (24 epochs). We train all the models with SGD optimizer, where the momentum is 0.9, and the weight decay is 0.0001. As shown in Table 5, our KD-Zero improves the AP by 3.5 on RetinaNet and 3.4 on Faster R-CNN, respectively, outperforms previous state-of-the-art techniques, including [59, 70, 74], for both object detectors. The results substantiate the potential of KD-Zero for scaling knowledge transfer to broader datasets and more complex computer vision problems while preserving improved accuracy.
Semantic segmentation. We evaluate KD-Zero on Cityscapes dataset[10]. Following the previous work, we adopt PSPNet-ResNet101 [78, 51] as the teacher and PSPNet and DeepLabV3[7] models with the ResNet18 backbone as the student. During distillation, the batch size is 8, and the models are trained for 40K iterations with the SGD optimizer, where the momentum is 0.9 and the weight decay is 0.0005. The results are reported with mean Intersection-over-Union (mIoU) under the single-scale evaluation setting. As shown in Table 6, the student PSPNet and DeepLabV3 get 3.17 and 3.7 mIoU improvement by adding our KD-Zero loss. These results indicate that our method surpasses the state-of-the-art distillation method for semantic segmentation, demonstrating that searched distillers facilitate student learning.
Table 7: Top-1 (%) accuracy of ResNet20 (baseline : 69.06%) with different teachers models on CIFAR-100. Note that the distiller is searched with ResNet110 as the teacher.
Teacher	R110	VGG13	W-28-2	W-40-2	W-16-4	W-28-4	R50	R32×4
74.31	74.64	75.45	75.61	77.51	78.60	79.34	79.42
KD	70.89±0.13	70.07±0.23	70.83±0.21	70.35±0.14	70.22±0.13	70.59±0.11	70.12±0.08	70.37±0.11
DIST	69.82±0.16	70.10±0.21	70.42±0.13	70.05±0.21	70.35±0.21	69.88±0.21	70.51±0.21	69.79±0.21
WSLD	70.89±0.17	70.12±0.18	71.19±0.23	70.85±0.26	70.86±0.22	70.39±0.16	70.62±0.22	69.48±0.24
KD-Zero	72.05±0.21	71.21±0.12	72.46±0.08	71.76±0.16	72.37±0.11	71.55±0.11	71.74±0.17	70.98±0.14
Table 8: Top-1 (%) accuracy of WRN-16-2 (baseline : 73.26%) with different teachers models on CIFAR-100. Note that the distiller is searched with WRN-40-2 as the teacher.
Teacher	R110
74.31	VGG13
74.64	W-28-2
75.45	W-40-2
75.61	W-16-4
77.51	W-28-4
78.60	R50
79.34	R32×4
79.42
KD	74.09±0.23	74.21±0.24	74.20±0.26	74.49±0.18	74.61±0.14	74.03±0.12	74.67±0.18	74.57±0.12
DIST	74.35±0.11	74.86±0.13	75.34±0.22	74.92±0.15	75.79±0.17	75.04±0.18	75.36±0.22	74.39±0.25
WSLD	74.93±0.28	74.88±0.19	75.44±0.13	75.30±0.16	76.29±0.26	75.58±0.18	75.42±0.21	74.86±0.26
KD-Zero	75.50±0.24	75.45±0.15	76.05±0.21	76.42±0.16	76.50±0.19	76.40±0.18	76.02±0.16	75.86±0.15
Search space organization. We individually explore various operation types and organized settings in Figure 9. Results indicate: (1) some operations like the normalized-based transforms, normalized-based losses, focal correction, and small-weighted values bring superior gains than others. (2) our design is optimal across different settings, and the removal of transform, weight factor, and temperature factor search leads to worse results, proving our organization's strength.
Search algorithm. We use the EA for distiller search, which is gradient-free and flexible for nonconvex optimization. As shown in Figure 10, the EA obtains faster convergence and better final search results than random search. In addition, for fitting objectives in EA, Shapeness and CKA metrics demonstrate good correlations with the final distillation accuracy and combine well with cross-entropy loss to improve evaluation quality.
Generalization of searched distillers for different Teachers. In Table 8 & 7, we present detailed results of the same searched distiller with various capacities teacher models on CIFAR-100. The results indicate that our KD-Zero can significantly improve the student model in different teacher models and surpass the KD and improved KD methods (e.g., DIST [27], WSLD [79]).
In addition, we provide combined trials in Table 2 and Figure 5, student training and teacher-student gap analysis in Figure 6 & 7. These ablation studies demonstrate that our KD-Zero can well alleviate distillation gaps and combine with other KD designs.
In this paper, we introduce the KD-Zero framework, an innovative framework for automatically designing distillers. We construct a flexible and unified distiller search space with advanced operators on knowledge transformation, distance function, and loss weight parts. Then, we employ evolutionary search with distillation-gap reduction as objectives and search acceleration strategies, enabling KD-Zero to efficiently find distillers and improve student model performance in distillation training. We employ the loss-rejection protocol, search space shrinkage, and proxy settings during the search process to enhance search efficiency. Furthermore, we present some practical guidelines based on the results analysis and propose various extensions of KD-Zero to cater to different KD designs and scenarios. Comprehensive experiments on three benchmarks demonstrate the effectiveness and universality of the KD-Zero framework for various CNNs, Vision Transformer models, object detection, and semantic segmentation. In future work, we will extend KD-Zero to more tasks and explore large language models as code encoders or searcher for KD-Zero. We hope this work will inspire future research on knowledge distillation designs.
[60] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020. 3, 7, 8
[61] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. pages 10347-10357, 2021. 2
[62] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV, 2019. 7
[63] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In CVPR, 2019. 6
[64] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV. 2
[65] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference on Computer Vision, pages 346-362. Springer, 2020. 9
[66] Zimian Wei, Hengyue Pan, Lujun Li Li, Menglong Lu, Xin Niu, Peijie Dong, and Dongsheng Li. Convformer: Closing the gap between cnn and vision transformers. arXiv preprint arXiv:2209.07738, 2022.1
[67] Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. 2022. 2, 6
[68] Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. arXiv preprint arXiv:2204.06986, 2022. 9
[69] Jie Yang et al. Automatically labeling video data using multi-class active learning. In Proceedings Ninth IEEE international conference on computer vision, pages 516-523. IEEE, 2003. 3
[70] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111.11837, 2021.9
[71] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. arXiv preprint arXiv:2205.01529, 2022. 2, 8
[72] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch onImagenet. In ICCV, 2021. 2
[73] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017. 7, 8
[74] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations, 2020. 9
[75] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019. 6
[76] Matthew Shunshi Zhang and Bradley Stadie. One-shot pruning of recurrent neural networks by jacobian spectrum evaluation. arXiv preprint arXiv:1912.00120, 2019. 6
[77] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018. 6
[78] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017. 9
[79] Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. 2021. 2, 4, 5, 6, 7, 10
[80] Chendi Zhu, Lujun Li, Yuli Wu, and Zhengxing Sun. Saswot: Real-time semantic segmentation architecture search without training. In AAAI, 2024. 3
[81] Zimian Zimian Wei, Lujun Li Li, Peijie Dong, Zheng Hui, Anggeng Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. In AAAI, 2024. 3