SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Distillation results of logits-based approaches with various teacher and student combinations. Number in the parenthesis reports the result of training corresponding model from scratch.
LR WD	ConvNeXt-XL (86.97)	- Res50 (79.86)	LR WD	BEiTv2-L (88.39)	DeiT-S (79.90)	ConvNeXt-XL (86.97)	DeiT-S (79.90)
DKD	DIST	KD	DKD	DIST	KD	DKD	DIST	KD
3e-3	0.02	80.50	80.50	80.71	3e-4	0.05	80.11	79.44	80.45	80.18	79.73	80.45
5e-3	0.02	81.05	80.89	80.94	5e-4	0.03	79.82	79.52	80.28	80.17	79.61	80.19
5e-3	0.03	81.02	80.85	80.98	5e-4	0.05	80.55	79.58	80.87	80.56	80.19	80.89
5e-3	0.04	80.90	80.84	81.10	5e-4	0.07	80.80	80.00	80.99	80.86	80.26	80.87
7e-3	0.02	80.81	80.86	81.07	7e-4	0.05	80.53	79.94	80.97	80.80	80.25	80.99
We compare various combinations of loss functions for vanilla KD and present the results in Table 4. Surprisingly, the binary cross-entropy (BCE) loss outperforms the multi-class cross-entropy loss in terms of accuracy. However, we find that the binary version of KL divergence, referred to as BKL loss, yields suboptimal performance. Moreover, the results indicate that training with or without supervision from hard labels has minimal impact on performance, even with a longer training duration. Consequently, we maintain hard label supervision [15, 19] throughout our experiments.
Ablation on training hyper-parameters. To ensure a thorough and fair comparison, we investigate various configurations of learning rate and weight decay for DKD, DIST, and vanilla KD using strategy "A2". The results, presented in Table 5, reveal an intriguing finding: vanilla KD outperforms the other methods across all configurations. In addition, it is noteworthy that vanilla KD consistently performs close to its best across different learning rates and weight decay configurations, demonstrating its robustness and ability to maintain competitive results under various settings.
3.5 Distillation results with different teacher-student pairs
In addition to the BEiTv2-L teacher and the Res50 student used in previous experiments, we introduce additional combinations of teacher-student pairs: ConvNeXt-XL-Res50, BEiTv2-L-DeiT-S, and ConvNeXt-XL-DeiT-S. We conduct experiments to compare the performance of vanilla KD with that of DKD and DIST on these combinations. Each student model is trained for 300 epochs using strategy "A2". We evaluate multiple hyper-parameter configurations, including learning rate and weight decay, and select the configuration with the highest performance for each method to ensure a fair comparison. Corresponding results are presented in Table 6. Consistent with the findings from previous experiments, vanilla KD achieves on par or slightly better performances compared to DKD and DIST on all combinations. These results demonstrate the potential of vanilla KD to achieve state-of-the-art performances across a diverse range of teacher-student combinations.
Marginal gains when teacher is sufficiently large. Although vanilla KD can achieve state-of-the-art results across various architectures in our experiments, it is not without limitations. For example, we observe that the Res50 student, trained with a BEiTv2-B teacher, achieves similar or even better result than that trained with a larger BEiTv2-L. This observation is consistent with the analysis presented in [47]. To gain deeper insights into the impact of teacher model when working with large-scale datasets, we conducted additional experiments in this section. We evaluate the impact of the BEiTv2-B teacher and the BEiTv2-L teacher with all the logits-based baselines, using three different epoch configurations. As shown in Table 9, the results demonstrate a trend of performance degradation as the capacity of the teacher model increases. This indicates that vanilla KD is not able to obtain benefit from a larger teacher model, even trained on a large-scale dataset.
Sustained benefits when enlarging the teacher's training set. we also evaluate the influence of the training set used for teacher models on student performance. We compare two BEiTv2-B teacher models: one pre-trained on ImageNet-1K and the other pre-trained on ImageNet-21K. Subsequently, we train two distinct students using strategy "A1" for 1200 epochs. The evaluation results are presented in Table 7. The results clearly demonstrate that when the teacher is trained on a larger-scale dataset, it positively affects the performance of the student. We hypothesize that
Table 7: Teacher trained with larger dataset can produce a better student. The teacher is BEiTv2-B.
Student	IN-1K T. (85.60)	IN-21K T. (86.53)
DeiT-T	76.12	77.19
Res50	81.97	82.35
Table 9: Impact of different teacher model sizes. The student model is Res50.
Method	Teacher	300 Ep.	600 Ep.	1200 Ep.
DKD	BEiTv2-B	80.76	81.68	82.31
BEiTv2-L	80.77	81.83	82.28
DIST	BEiTv2-B	80.94	81.88	82.33
BEiTv2-L	80.70	81.72	81.82
KD	BEiTv2-B	80.96	81.64	82.35
BEiTv2-L	80.89	81.68	82.27
Table 10: Vanilla KD vs. FCMIM on ConveNeXtV2-T. †: the GPU hours of fine-tuning stage is 568.
Method	EP.	Hours	Acc. (%)
FCMIM (IN-1K)	800	1334	-
+ Fine-tune on IN-1K	300	1902†	82.94
FCMIM (IN-1K)	800	1334	-
+ Fine-tune on IN-21K	90	3223	-
+ Fine-tune on IN-1K	90	3393	83.89
vanilla KD	300	653	84.42
vanilla KD	1200	2612	85.03
a teacher pre-trained on a larger dataset has a better understanding of the data distribution, which subsequently facilitates student learning and leads to improved performance.
3.6 Exploring the power of vanilla KD with longer training schedule
In Section 3.3, we notice a consistent improvement in the performance of vanilla KD as the training schedule is extended. To explore its ultimate performance, we further extended the training schedule to 4800 epochs. In this experiment, we utilized a BEiT-B teacher model, which achieved  $86.47\%$  top-1 accuracy on the ImageNet-1K validation set. As for students, we employed Res50 [2], ViT-T, ViT-S [3, 32], and ConveNextV2-T [34], and trained
them using strategy “A1”. The corresponding results are shown in Table 8. All three students achieve improved performance with longer training epochs. This trend is also consistent with the pattern observed in [35]. When trained with 4800 epochs, the students achieve new state-of-the-art performance, surpassing previous literature [35, 32]. Specifically, the Res50, ViT-S, ViT-T, and ConvNeXtV2-T achieve  $83.08\%$ ,  $84.33\%$ ,  $78.11\%$  and  $85.03\%^2$  top-1 accuracy, respectively.
Table 8: Results of extended training schedule. The teacher model is BEiTv2-B.
Model\Epoch	300	600	1200	4800
Res50	80.96	81.64	82.35	83.08
ViT-S	81.38	82.71	83.79	84.33
ViT-T	-	-	77.19	78.11
ConvNextV2-T	84.42	84.74	85.03	-
3.7 Comparison with masked image modeling
Masked image modeling (MIM) framework [48, 49, 50] has shown promising results in training models with excellent performance. However, it is worth noting that the pre-training and fine-tuning process of MIM can be time-consuming. In this section, we conduct a comparison between MIM and vanilla KD, taking into account both accuracy and time consumption. Specifically, we compare vanilla KD with the FCMIM framework proposed in ConvNeXtv2 [34]. We utilize a ConvNeXtv2-T student model and a BEiTv2-B teacher model. We train two student models using vanilla KD: one for 300 epochs and another for 1200 epochs.
The results are presented in Table 10, highlighting the efficiency of vanilla KD in training exceptional models. Specifically, even with a training duration of only 300 epochs, vanilla KD achieves an accuracy of  $84.42\%$ , surpassing MIM by  $0.53\%$ , while consuming only one-fifth of the training time. Moreover, when extending the training schedule to 1200 epochs, vanilla KD achieves a performance of  $85.03\%$ , surpassing the best-performing MIM-trained model by  $1.14\%$ . These results further demonstrate the effectiveness and time efficiency of vanilla KD compared to the MIM framework.
3.8 Transferring to downstream task
To assess the transfer learning performance of the student distilled on ImageNet, we conduct experiments on object detection and instance segmentation tasks using COCO benchmark [26]. We adopt Res50 and ConvNeXtV2-T as backbones, and initialize them with the distilled checkpoint. The detectors in our experiments are commonly used Mask RCNN [40] and Cascade Mask RCNN [41].
Here we find that the result of KD has already reached a satisfactory level, and the training cost is lower compared to the MIM approach. Hence, we did not pursue further scaling to 4800 epochs.
Table 11: Object detection and instance segmentation results on COCO. We compare checkpoints with different pre-trained accuracies on ImageNet when used as backbones in downstream tasks.
(a) Mask RCNN (Res50 [2]).	(b) Mask RCNN (ConvNeXtV2-T [34]).	(c) Cascade Mask RCNN (ConvNeXtV2-T [34]).
ckpt.	sche.	IN-1K	APb	APm	ckpt.	sche.	IN-1K	APb	APm	ckpt.	sche.	IN-1K	APb	APm
prev.	1×	77.1	38.2	34.7	prev.	1×	83.0	45.4	41.5	prev.	1×	83.0	49.8	43.5
prev.	1×	80.4	38.7	35.1	prev.	1×	83.9	45.6	41.6	prev.	1×	83.9	50.4	44.0
ours	1×	83.1	41.8	37.7	ours	1×	85.0	45.7	42.0	ours	1×	85.0	50.6	44.3
prev.	2×	77.1	39.2	35.4	prev.	3×	83.0	47.4	42.7	prev.	3×	83.0	51.1	44.5
prev.	2×	80.4	40.0	36.1	prev.	3×	83.9	47.7	42.9	prev.	3×	83.9	51.7	44.9
ours	2×	83.1	42.1	38.0	ours	3×	85.0	47.9	43.3	ours	3×	85.0	52.1	45.4
Table 11 reports the corresponding results. When utilizing our distilled Res50 model as the backbone, Mask RCNN outperforms the best counterpart using a backbone trained from scratch by a significant margin  $(+3.1\%)$  box AP using  $1\times$  schedule and  $+2.1\%$  box AP using  $2\times$  schedule). Similarly, when the backbone architecture is ConvNeXtV2-T, using the model trained by vanilla KD consistently leads to improved performance. These results demonstrate the efficient transferability of the performance improvements achieved by vanilla KD to downstream tasks.
Knowledge distillation (KD). Previous KD methods can be broadly categorized into two classes: logits-based methods and hint-based methods. Hinton et al. [15] first proposes to train the student to learn from logits of teacher. This method is referred as vanilla KD. Recently, researchers have introduced improvements to the vanilla KD method by enhancing the dark knowledge [19, 51], relaxing the constraints [18], using sample factors to replace the unified temperature [52]. Hint-based methods train the student to imitate learned representations of the teacher. These methods leverage various forms of knowledge, such as model activation [16, 53, 54], attention map [55], flow of solution procedure [56], or feature correlation among samples [22, 20]. To facilitate the alignment of features between the teacher and the student, techniques including design new projector [57, 58] and more complex alignment approaches [17, 59, 60] have been proposed. Besides directly mimicking the hint knowledge, contrastive learning [21] and masked modeling [61] have also been explored. Recently, the success of vision transformers [3] has prompted the development of specific distillation approaches tailored to this architecture, including introducing additional distillation tokens [32, 62] or perform distillation at the fine-grained patch level [63]. Additionally, vision transformer architectures have also been leveraged to facilitate KD between CNN models [64, 65]. Furthermore, some works attempt to address specific challenges in KD, such as distilling from ensembles of teachers [66, 67] and investigating the impact of different teachers on student performance [68, 47, 69].
Large-scale training. Recent developments in large-scale deep learning [31, 48] suggest that by scaling up data [3, 70], model size [6, 7, 8] and training iteration [35] to improve model performance, achieve state-of-the-art results [71] across diverse tasks, and enable better adaptability to real-world scenarios. For example, in computer vision, numerous large models have achieved outstanding results on tasks such as classification, detection, and segmentation. These models are usually trained on extensive labeled image datasets, including ImageNet-21K [30] and JFT [15, 72]. Similarly, in natural language processing, training models on massive text corpora [73, 74] has led to breakthroughs in language understanding, machine translation, and text generation tasks. Inspired by this scaling trend, it becomes crucial to evaluate KD approaches in more practical scenarios. After all, the goal of KD is to enhance the performance of student networks.
The closest work to ours is that of Beyer et al. [35]. They identify several design choices to make knowledge distillation work well in model compression. Taking it a step further, we point out the small data pitfall in current distillation literature: small-scale datasets limit the power of vanilla KD. We thoroughly study the crucial factors in deciding distillation performance. In addition, we have distilled several new state-of-the-art backbone architectures, further expanding the repertoire of high-performing models in vision arsenal.
Motivated by the growing emphasis on scaling up for remarkable performance gains in deep learning, this paper revisits several knowledge distillation (KD) approaches, spanning from small-scale to large-scale dataset settings. Our analysis has disclosed the small data pitfall present in previous KD methods, wherein the advantages offered by most KD variants over vanilla KD diminish when applied to large-scale datasets. Equipped with enhanced data augmentation techniques and larger datasets, vanilla KD emerges as a viable contender, capable of achieving comparable performance to the most advanced KD variants. Remarkably, our vanilla KD-trained ResNet-50, ViT-S, and ConvNeXtV2-T models attain new state-of-the-art performances without any additional modifications.
While the application of knowledge distillation on large-scale datasets yields significant performance improvements, it is important to acknowledge the associated longer training times, which inevitably contribute to increased carbon emissions. These heightened computational resource requirements may impede practitioners from thoroughly exploring and identifying optimal design choices. Future endeavors should focus on limitations posed by longer training times and carbon emissions, and exploring new avenues to further enhance the efficacy and efficiency of distillation methods.
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017. 1
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. 1, 2, 4, 8, 9
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 1, 2, 4, 8, 9
[4] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1
[5] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1, 4
[6] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015. 1, 9
[7] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the International Conference on Computer Vision, 2021. 1, 9
[8] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555, 2022. 1, 9
[9] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017. 1
[10] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016. 1
[11] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 2021. 1
[72] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017. 9
[73] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019. 9
[74] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020. 9