Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale
Authors: Zhiwei Hao
Jianyuan Guo
Kai Han
Han Hu
Chang Xu
Yunhe Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge distillation
small-data pitfall
vanilla kd
TL;DR: We point out that the vanilla KD method is underestimated in small-scale dataest, and it is able to achieve comparable performance with SOTA methods in large-scale dataset.

Abstract: The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rationality of designing knowledge distillation (KD) approaches for limited-capacity architectures solely based on small-scale datasets is now deemed imperative. In this paper, we identify the small data pitfall that presents in previous KD methods, which results in the underestimation of the power of vanilla KD framework on large-scale datasets such as ImageNet-1K. Specifically, we show that employing stronger data augmentation techniques and using larger datasets can directly decrease the gap between vanilla KD and other meticulously designed KD variants. This highlights the necessity of designing and evaluating KD approaches in the context of practical scenarios, casting off the limitations of small-scale datasets. Our investigation of the vanilla KD and its variants in more complex schemes, including stronger training strategies and different model capacities, demonstrates that vanilla KD is elegantly simple but astonishingly effective in large-scale scenarios. Without bells and whistles, we obtain state-of-the-art ResNet-50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83.1%, 84.3%, and 85.0% top-1 accuracy, respectively. PyTorch code and checkpoints can be found at https://github.com/Hao840/vanillaKD.

Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale
Zhiwei Hao 1,2, Jianyuan Guo Kai Han?, Han Hu Chang Xu², Yunhe WangSt
School of information and Electronics, Beijing Institute of Technology. Huawei Noah's Ark Lab.
School Computer Science, Faculty of Engineering, The University of Sydney. Chaozhw, hhu}@bit ceeeeiiiieuuuuu.eeeee en, jguo51720umi. [kai han, yunhe. wang) com, c. xu@sydney edu. au
edu. au,
Abstract
The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflec- tion on the rationality of designing knowledge distillation (KD) approaches for limited-capacity architectures solely based on small-scale datasets is now deemed imperative. this paper, we identify the small data pitfaal that presents in previous KD methods, which results in the underestimation of the power of vanilla KD framework on large-scale datasets such as ImageNet-1K Specifically, we show that employing stronger data augmentation techniques and using larger datasets directly decrease the gap between vanilla KD and other meticulously designed KD variants. This highlights the necessity of designing and evaluating KD approaches in the context of practical scenarios, casting off the limitations of small-scale datasets. Our investigation of the vanilla KD and its variants more complex schemes, including stronger training strategies and different model capacities, demonstrates that vanilla KD is elegantly simple but astonishingly effective in large-scale scenarios. Without bells and whistles, we obtain state-of-the-art ResNet- 50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83. 84.3%, %, and 85.0% top-1 accuracy, respectively. PyTorch code and checkpoints can be found https:/githbbb.om//aa44///////
Introduction
In recent years, deep learning has made remarkable progress computer vision, with models continually increasing in capability and capacity 2, 4, 5]. The philosophy behind prevailing approach achieving better performance has been larger is better, as evidenced by the success of increasingly deep [2, 6, 7, 8] and wide [9, 10, 11] models. Unfortunately, these cumbersome models with numerous parameters are difficult to be deployed on edge devices with limited computation resources [12, 13, 14], such as cell phones and autonomous vehicles. To overcome this challenge, knowledge distillation (KD) [15] and its variants [16, 17, 18, 19, 20, 21, 22, 23, 24] have been proposed improve the performance of compact student models by transferring knowledge from larger teacher models during training.
Thus far, most existing KD approaches in the literature are tailored for small-scale benchmarks (e.g., CIFAR [25]) and small teacher-student pairs (e.g., Res34-Res18 [2] and WVRN40-WRNI [10]). However, downstream vision tasks [26, 27, 28, 29] actually require the backbone models to be
"Equal contribution. 'Coreesponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

pre-trained on large-scale datasets (e.g., ImageNet [30] to achieve state-of- ofte-of-the-as art performances. Only exploring KD approaches on scall-scale datasets may fall short in providing a comprehensive understanding in practical scenarios. Given the availability of large-scale benchmarks and the capacity of large models [31, 3, 32, 33, 34], it remains uncertain whether previous approaches will remain effective in more complex schemes involving stronger training recipes, different model capacities, and larger data scales.
In this paper, we delve deep into this question and thor- oughly study the crucial factors in deciding distillation per-
Perfomaacce gap (%)
formances. We point out the small data pitfall in current knowledge distillation literature: once sufficient quantity of training data is reached, different conclusions emerge.
pap DIST
pap of va. DKD
1.5-
For example, when evaluated on CIFAR- 100 (50K train- ing images), KD methods meticulously designed on such datasets can easily surpass vanilla KD However, when evaluated on datasets with larger scale, i.e., ImageNet-IK IK (IM training images), vanilla KD achieves on par or even better results compared to other methods.
CIFARIOO (X0% IN-IK IN-IK (fall)
Figure 1: The performance gap between vanilla KD and other carefully designed ap- proaches gradually diminishes with the in- creasing scale of datasets.
To break down this problem, we start by compensating for the limited data via training for longer iterations [35] on scall-scale datasets. Albeit with longer schedule, meticulously designed KD methods still outperform vanilla KD by a large margin. It follows that large-scale datasets are necessary for vanilla KD) to reach its peak.
We further investigate the crucial factors in deciding KD performances and carefully study two key elements, i.e., training strategy and model capacity. In terms of different training strategies, we have made the following observations: (i) By evaluating the meticulously designed methods [18, 19] which perform well on small-scale benchmarks under different training recipes [16, 36, 37] on large- scale datasets such as ImageNet [30], it is evident that with the enhancement of data augmentation techniques and longer training iterations, the gap between vanilla KD [15] and other carefully designed KD methods gradually diminishes. (ii) Our experiments also demonstrate that logits- methods [15, 18, 19] outperform hint-hased methods [16, 17, 22, 21, 38] in terms of generalizability. With the growing magnitude of datasets and models, teacher and student models possess varying capacities to handle intricate distributions. Consequently, the utilization of the hint-based approach, wherein the student imitates the intermediate features of the teacher, becomes less conducive to achieving satisfactory results.
With regard to model capacity, we compare teacher-student pairs of different scales, e.g., using Res34 to teach Res18 vs. using Res152 to teach Res50. The outcome reveals that vanilla KD ends up achieving on par performance with the best carefully designed methods, indicating that the impact of model capacity on the effectiveness of knowledge distillation is in fact small.
Throughout our study, we emphasize that due to small data pitfall, the power of vanilla KD is significantly underestimated. In the meantime, meticulously designed distillation methods may instead become suboptimal when given stronger training strategies [36, 35] and larger datasets [39]. Furthermore, directly integrating vanilla KD to ResNet-50 [2]. ViT-Tiny, ViT-Small [3], and Con- vNeXtV2 [34] architectures leads to an accuracy of 83.1%, 1%, 78. 1 %, 84.3%, and 85.0% on ImageNet respectively, surpassing the best results reported in the literature [35] without additional bells and whistles. Our results provide valid evidence for the potential and power of vanilla KD and indicate its practical value. In addition, it bears further reflection on whether is appropriate to design and evaluate KD approaches on small-scale datasets. Finally, we demonstrate that improving the backbonn architecture with higher performance on ImageNet can also significantly enhance downstream tasks such as object detection and instance segmentation [26, 40, 41].
Although this work does not propose a novel distillation method, we believe identification of small data pitfall and series of analyzes based on this would provide valuable insights for the vision community in the field of knowledge distillation and the pursuit of new state-of-the-ard results under more practical scenarios. Moreover, we anticipate that the released checkpoints of commonly used architectures will facilitate further research on downstream tasks.
2

---

2 Small data pitfall: limited performance of vanilla KD on small-scale dataset
2.1 Review of knowledge distillation
Knowledge distillation (KD) techniques can be broadly categorized into two types based on the source of information used by the pre-trained teacher model: those that utilize the teacher's output probabilities (logits-based) [19, 18, 15] and those that utilize the teacher's intermediate representations (hint-based) [16, 17, 22, 21, 42]. Logits- based approaches leverage teacher outputs auxiliary signals to train a smaller model, known as the student:
Lu = aDa , + (1 ())
where p³ and p' are logits of the student and the teacher. y the one-hot ground truth label. Das and Dea are classification loss and distillation loss, such as cross-entropy and KL divergence, respectively. The syper-parameet a determines the balance between two loss terms. For convenience, we set a to 0.5 in all subsequent experiments.
Besides the logits, intermediate hints (features) [16] can also be used for KD. Considering student feature F and a teacher feature F`, hint-based distillation is achieved as follows:
Crim = Drim (F*), TF())))
where T; and T, are transformation modules for aligning two features. Drim is a measurement of feature divergence, e.g., 1 or 12-norm. In common practice, the hint-based loss is typically used in conjunction with the classification loss, and we adhere to this setting in our experiments.
2.2 Vanilla KD can not achieve satisfactory results on small-seale dataset
Recently, many studies have used simplified evaluation protocols that involve small models or datasets. However, there is a growing concern [35] about the limitations of evaluating KD methods solely in such small-scale settings, as many real-world applications involve larger models and datasets. In this section, we extensively investigate the impact of using capalccy models and small-scale datasets on different KD methods. To provide comprehensive analysis, specifically compare vanilla KD with two state-of-th of-thr-rrt logits-base based KD approaches, i.e., DKD [19] and DIST [18]. Different from [35] which compresses large models to Res50, our primary focus is ascertain whether the unsatisfactory performance of vanilla KD can be attributed to small student models or small-scale datasets.
Impact of limited model capacity. We conduct experiments using two commonly employed teacher- student pairs: Res34-Res and Res50-MobileNetv3 The corresponding results are shown in Table 1a. Previously, these models were trained using the SGD optimizer for 90 epochs ("Original" in table). However, to fully explore the power of KD approaches, strategy by employing the Adam optimizer with 300 training epochs ("Improve" in table), details can be found in appendix. The original results reported in DKD and DIST demonstrate clear performance advantage over vanilla KD. However, when adopting a stronger training strategy, all three approaches exhibit improved results. Notably, the performance gap between vanilla KD and the baselines surprisingly diminishes, indicating that the limited power of vanilla KD can be attributed to insufficient training, rather than to models with small capacity.
leverage a stronger training
Impact small dataset scale. investigate the impact of small dataset scale on the performance of vanilla KD, we conduct experiments using the widely used CIFAR- 100 dataset [25]. Similarly, we design a stronger training strategy by extending the training epochs from 240 to 2400 and introduce additional data augmentation schemes. As shown in Table lb, despite the improved strategy leads to better results for all three approaches again, there still remains a performance gap between vanilla KD and other baselines. Notably, the accuracy gap even increases from 1.3 1 % to 2.17% for the Res56-Res20 teacher-student pair. These observations clearly indicate that the underestimation of vanilla KD is not solely attributed to insufficient training, but also to the small-scale dataset.
Discussion. CIFAR-100 is widely adopted as standard benchmark for evaluating most distillation approaches. However, our experimental results show that the true potential of vanilla KD has been underestimated on such small-scale datasets. We refer this phenomenon as the small data pitfall, where performance improvements observed on small-scale datasets do not necessarily translate to
3

---

Table 1: Impact dataset seale on performance. When adopting a stronger training strategy on large- scale dataset, vanilla KD [15] can achieve on par result to the state-of-the-ar approaches, i.e. DKD [19] and DIST [18]. However, this phenomenon is not observed on small-scale dataset, revealing the small data pitfal! that underestimates the power of vanilla KD. The symbol A represents the accuracy gap between vanilla KD and the best result achieved by other approaches (marked with gray "previous recipe" refers to results reported
original papers, while 'stronger recipe' refers results obtained through our enhanced training strategy.
Teacher Student
Method
previous recipe
stronger recipe
Teacher Student
Method
previous recipe
stronger recipe
DKD
71.70
73.07
DKD DIST
71.97
73.10
Res34 Res18
DIST
72.07
73.52
Res56
Res20
71.78
74.51
(73.31)
(69.75)
vanilla KD
70.66
73,46
(72.34)
(69.09)
vanilla KD
70.66
72.34
A
-1.1
0.06
11.31
2.117
DKD DIST
72.05
73.71
DKD DIST
76.32
78.15
Res50 MBNetV2 (76.16) (68.87)
73.24
74.26
Res32x4- Res8
75.79
77.69
vanilla KD
68.58
74.23
(79.42)
(72.50)
vanilla KD
73.33
75.90
A
4.66
0.03
A
22.9
2225
(a) Vanilla KD achieves comparable results with the state-of-the-art approach on ImageNet-1IK [30].
(b) The performance gap persists CIFAR-100 [25],
even with stronger training strategy.
more complex real-world datasets. As shown in Figure 1, the performance gap between vanilla Kd and other approaches gradually diminishes with the increasing scale of benchmarks. Considering the elegantly simple of vanilla KD, more extensive experiments are required to explore its full potential.
3 Evaluate the power of vanilla KD on large-scale dataset
3.1 Experimental setup
In order to fully understand and tap into the potential of vanilla KD, we conduct experiments based on large-scale datasets and strong training strategies. The experimental setup is as following.
Datasets. We mainly evaluate KD baselines the larger and more complex ImageNet-Il dataset [30], which comprises 1.2 million training samples and 50,000 validation samples, spanning across 1000 different categories. Compared CIFAR- dataset [25], ImageNet- IK provides closer approximation to real-world data distribution, allowing for a more comprehensive evaluation.
Models. In our experiments, mainly utilize Res50 [2] as the student model and BEiTv2-L [43] as the teacher model, which pretrained on ImageNet- 1K and then finetuned on ImageNet-21K and ImageNet-11 1K successively. This choice is motivated by teacher's 's exceptional performance, as currently stands as the best available open-sourced model in terms of top-1 accuracy when using an input resolution of 224x224. In addition, also include VIT [3, 32] and ConvNeXtV2 [34] as student models, and ResNet [2] and ConvNeXt [5] as teacher models. Specifically, when training additional distillation token [32].
ViT student, we do not
Training strategy. We leverage two training strategies based on recent work [36] that trains high- performing models from scratch using sophisticated training schemes, such more complicated data augmentation and more advanced optimization methods [32]. Strategy "A1" " is slightly stronger than "A2" summarized in Table 2, and we use it with longer training schedule configurations.
Baseline distillation methods. To make comprehensive comparison, we adopt several recently proposed KD methods the baselines, such as logits-based vanilla KD [15], DKD [19], and DIST [18], hint-based CC [22], RKD [20], CRD [21], and ReviewKD [17].
3.2 Logits-based methods consistently outperform hint-based methods
In this section, we conduct comparative analysis between logits-based and hint-based distillation approaches. We utilize the widely adopted Res50 architecture [2] as the student model, with Res 152 and BEiTv2-L serving the teachers. The student models are distilled with two different epoch configurations: 300 and 600 epochs. The evaluation results, along with the total GPU training time for each student, presented in Table 3. To gain a deeper understanding of the generalization of student

---

Table 2: Training strategies used for distillation. "A1" and "A2" represent two strategies follow- ing [36], The upper table provides the shared settings, while the bottom presents their specific settings.
Setting
T. Res.
S. Res.
BS
Optimizer
LR
LR decay
Warmup ep.
AMP
EMA
Label Loss
Common
224
224
2048
LAMB
5e-3
cosine
5
BCE
Setting
Smoothing
Drop path
Repeated Aug.
H. Flip
PRC
Rand Aug.
Mixup
Cutmix
0.03
0.05
7/0.5
0.1
1.0
A1
0.01
0.1
0.05
7/0.5
0.2
1.0
Table
Comparison between hint-based and logits-based distillation methods. The student Res50, while teachers are Res and BEiTv2-L. with top-1 accuracy of 82.83% and 88. 39%,, respectively. "v" indicates that the corresponding method is hint-hased. GPU hours are evaluated on single machine with 8 V100 GPUs.
Scheme
Method
Hint
Epoch
GPU
Res152 teacher
BEiTv2-L teacher
hours
IN-1K
IN-Real
IN-V2
IN-IK
IN-Real
IN-V2
CC [22]
300
265
79.55
85,41
67.52
79.50
85.39
67.82
RKD [20]
300
282
79.53
85.18
67.42
79.23
85.14
67.60
CRD [21]
300
307
79.33
85.25
67.57
79.48
85.28
68.09
A2 (79.80)
ReviewKD [17]
300
439
80.06
85.73
68.85
79.11
85.41
67.36
DKD [19]
300
265
80.49
85.36
68.65
80.77
86.55
68.94
DIST [18]
300
265
80.61
86.26
69.22
80.70
86.06
69.35
vanilla KD [15]
300
264
80.55
86.23
69.03
80.89
86.32
69.65
CC [2]]
V
600
$29
80.33
85.72
68.82 88.82
RKD [20]
600
564
80.38
85.86
68.38
CRD [21]
600
80.18
85.75
68.32
ReviewKD [17]
600
877
80.76
86.41
69.31
A1 (80.38)
DKI) [19]
600
529
81.31
86.76
69.63
81.83
87.19
70.09
DIST [18]
600
529
81.23
86.72
70.09
81,72
86.94
70.04
vanilla KD [16]
600
529
81.33
86.77
69.59
81.68
86.92
69.80
DKD [19]
1200
1059
81.66
87.01
70.42
82.28
87.41
71.10
DIST [18]
1200
1058
81.65
86.92
70.43
81.82
87.10
70.35
vanilla KD [15]
1200
1057
81.61
86.86
70.47
82.27
87.27
70.88
models beyond the ImageNet-1K dataset, we extend the evaluation to include ImageNet-Real [44] and ImageNet- V2 matched frequency [45], which provide separate test sets.
After 300 epochs of training using strategy A2, all hint-based distillation methods exhibit inferior results compared the logits-based baselines. Despite extended training schedule with the stronger strategy A a notable performance gap remains between the two distillation categories. Moreover, it is worth noticing that hint-based approaches necessitate significantly more training time, highlighting their limitations in terms of both effectiveness and efficiency.
Discussion. Our experiments demonstrate that based methods consistently out- perform hint-based methods in terms of gen- eralizability. We speculate that this discrep- ancy can be attributed to the different ca- pabilities of the teacher and student mod- els when dealing with complex distributions. The hint-based approach mimicking the in- termediate features of the teacher model be- comes less suitable, hindering it from achiev- ing satisfactory results. Furthermore. hint- based methods may encounter difficulties when using heterogeneous teacher and stu- dent architectures due to distinct learned rep- resentations, which impede the feature align- ment process. To analyze this, we perform center kernel analysis (CKA) [46] to compare the features extracted by Res50 with those of Res152 and B-İ As depicted in Figure 2, there is noticeable dissimilarity between the intermediate
15-
15
ResNer50
15
30
45
0
15
30
ResNet152
BETTV2-L
Figure 2: Feature similarity heatmap measured by CKA. Left: similarity between homogeneous architec- tures; Right: similarity between heterogeneous archi- tectures. The coordinate axes represent layer indexes.
5

---

Table Ablation of different losses used learn from hard label and soft label in vanilla KD. The "CE", "BCE", and "KI." refers to cross-entropy loss, binary cross-entropy loss, and KI. divergence measurement, re- spectively. BKL is a binary counterpart of
Table Ablation about different configurations of learning rate and weight decay on ImageNet- IK. LR: learning rate; WD: weight decay.
LR
WD
DKD [19]
DIST [18]
KD [15]
2e-3
0.02
79.50
79.69
79.77
KL. Its definition provided in Equation
2e-3
0.03
79.52
79.58
79.78
3e-3
0.02
80.18
80.32
80.46
Epoch
Hard Label
Soft Label
Acc. (%)
3e-3
0.03
80.24
80.41
80.37
300
CE
KI.
80.49
5e-3
0.01
80.73
80.62
80.82
5e-3
0.02
80.76
80.71
80.82
300
BCE
KI.
80.89
5e-3
0.03
80.81
80.79
80.89
300
KI.
80.87
5e-3
0.04
80.72
80.39
80.77
300
CE
BKL
79.42
6e-3
0.02
80.58
80.42
80,89
300
BCE
BKL
79.36
7e-3
0.02
80.60
80.20
80.84
300
BKL
80.82
8c-3
0.02
80.51
80.32
80.61
600
BCE
KL
81.68
8c-3
0.03
80.52
80.46
80.66
600
KL
81.65
8c-3
0.04
80.55
80.18
80.61
features of BEiTv2-L and those of the Res50 student, while the Res152 features exhibit a closer resemblance to Res50. In addition to the suboptimal performance, the CKA figure further high- lights the incompatibility of hint-based approaches with heterogeneous scenarios, thereby limiting their practical applicability in real-world settings. Consequently, focus solely logits-based distillation approaches subsequent experiments.
3.3
Vanilla KD preserves strength with increasing teacher capacity
Table 3 also shows the comparison between vanilla KD and other two logits-based baselines using stronger heterogeneous BEiTv2-L teacher [43]. This teacher has achieved the stateooftthe-an top-1 accuracy on ImageNet-IK KK validation among open- open-sourced models.
In all the evaluated settings, the three regits-baseda approaches exhibit similar performance. After being distilled for 300 epochs with Res as the teacher, vanilla KD achieves a top-1 accuracy of 80.55% ImageNet-IK validation set, which is only 0.06% lower than the best student performance achieved by DIST. With an extended training duration of 600 epochs, the student trained using vanilla KD even achieves the top performance 81.33%. When taught by BEiT-L, vanilla KD obtains 80.89% top-1 accuracy, surpassing both DKD and DIST. Even in the presence of distribution shift, vanilla KD maintains comparable performance other approaches on ImageNet-Ress and ImageNet-V V2 matched frequency benchmarks. Additionally, further extending the training schedule 120 epochs improves vanilla KD and results in a performance on par with DKD and DIST, thus highlighting its effectiveness in delivering impressive results.
These results clearly indicate that vanilla KD has been underestimated in previous studies because they are designed and evaluated on small-scale datasets. When trained with both powerful homogeneous and heterogeneous teachers, vanilla KD achieves competitive performance that comparable state-of-the -hrt---rtt methods while still maintaining its simplicity.
3.4 Robustness of vanilla KD across different training components
In this section, we perform ablation studies to assess the robustness of vanilla KD under different training settings. Specifically, we compare various loss functions used in vanilla KD and explore different carlier.
configurations for the three logits-based distillation methods mentioned
Ablation on Loss functions. Since our strategies [36] focus on training the student using one-hot labels and the binary cross-entropy (BCE) loss function, also evaluate binary version of the KL divergence as an alternative for learning from soft labels:
CBKL Enyl-pi log(p;/pp) - (1 - pp')
(3)
where y denotes the label space.

---

Table 6: Distillation results of logits-based approaches with various teacher and student combinations. Number in the parenthesis reports the result of training corresponding model from scratch.
Conv-NeXI-XL
Res50
BEiTv2-L- DeiT-S
ConvNeXI-XI. DeiT-S
LR
WD
(86.97)
(79.86)
LR
WD
(88.39) (79.90)
(86.97)
(79.90)
DKD
DIST
KD
DKD
DIST
KD
DKD
DIST
KD
3e-3 0.02
80.50
80.50
80.71
3e-4 0.05
80.11
79.44
80.45
80.18
79.73
80.45
5e-3 0.02
81.05
80.89
80.94
5e-4 0.03
79.82
79.52
80.28
80.17
79.61
80.19
5e-3 0.03
81.02
80.85
80.98
5c-4 0.05
80.55
79.58
80.87
80.56
80.19
80.89
5e-3 0.04
80.90
80.84
81.10
5c-4 0.07
80.80
80.00
80.99
80.86
80.26
80.87
7e-3 0.02
80.81
80.86
81.07
7c-4 0.05
80.53
79.94
80.97
80.80
80.25
80.99
We compare various combinations of loss functions for vanilla KD and present the results in Table 4. Surprisingly, the binary cross-entropy (BCE) loss outperforms the multi-class cross-entropy loss in terms of accuracy. However, we find that the binary version of KL divergence, referred to as BKL loss, yields suboptimal performance. Moreover, the results indicate that training with or without supervision from hard labels has minimal impact on performance, even with a longer training duration. Consequently, we maintain hard label supervision [15, 19] throughout our experiments.
To ensure a thorough and fair comparison, we investigate various configurations of learning rate and weight decay for DKD, DIST, and vanilla KD using strategy "A2". The results, presented in Table 5, reveal an intriguing finding: vanilla KD outperforms the other methods across all configurations. In addition, it is noteworthy that vanilla KD consistently performs close to its best across different learning rates and weight decay configurations, demonstrating its robustness and ability to maintain competitive results under various settings.
Ablation on training hyper-parametern
3.5 Distillation results with different teacher-student pairs
In addition to the BEiTv2-L teacher and the Res50 student used in previous experiments, we introduce additional combinations of teacher-student pairs: ConvNeXt-XL-Res50, BEiTV2-L-DeiT-s, and ConvNeXt-XL-DeiiT.
We conduct experiments to compare the performance of vanilla KD with that of DKD and DIST on these combinations. Each student model trained for 300 epochs using strategy "A2". We evaluate multiple weight decay, and select the configuration with the highest performance for each method to ensure a fair comparison. Corresponding results are presented in Table 6. Consistent with the findings from previous experiments, vanilla KD achieves on par or slightly better performances compared to DKD and DIST on all combinations. These results demonstrate the potential of vanilla KD to achieve state-of-the-are performances across a diverse range of teacher-student combinations.
configurations, including learning rate and
Marginal gains when teacher is sufficiently large. Although vanilla KD can achieve state-of-the art results across various architectures in our experiments, is not without limitations. For example, we observe that the Res50 student, trained with a BEiTv2-B teacher, achieves similar or even better result than that trained with a larger BEiTv2-L. This observation
consistent with the analysis presented in [47]. To gain deeper insights into the impact of teacher model when working with aurgesccale datasets, we conducted additional experiments in this section. We evaluate the impact of the BEiT teacher and the BEÎT teacher with all the logits-based baselines, using three different epoch configurations. As shown Table 9, the results demonstrate a trend of performance degradation as the capacity of the teacher model increases. This indicates that vanilla KD is not able to obtain benefit from a larger teacher model, even trained on a large-scale dataset.
Sustained benefits when enlarging the teacher's training set. we also evaluate the influence of the training set used for teacher models on student performance. We compare two BEmTv2-B teacher models: one pre-trained on ImageNet- 1K and the other pre-trained on ImageNet-21K: 2. Subsequently, we train two distinct students using strategy "A1" for 1200 epochs. The evaluation results are presented in Table 7. The results clearly demonstrate that when the teacher trained on a larger-scale dataset, it posi- tively affects the performance of the student. We hypothesize that
Table 7: Teacher trained with larger dataset can produce a better student. The teacher is BEiTv2-B.
IN-IK T. (85.60)
IN-21K T.
Student
(86.53)
DeiT'T Res50
76.12
77.19
81.97
82.35
7

---

Table Impact different teacher model sizes. The
Table 10: Vanilla KD vs. FCMIM on ConveNeXIV2--T
student model is Res50.
the GPU hours of fine-tuning stage is 568.
Method Teacher
300 Ep.
600 Ep.
1200 Ep.
Method
EP.
Hours
Acc. (%)
BEiTv2-B
80.76 80.77
81.68
82.31
FCMIM (IN-IK)
800
1334
DKD
B-iTv2-L
81.83
82.28
Tine-tune on IN-IK
300
1902
82.94
DIST
BEiTv2-E B B-iTv2-L
80,94
81.88
82.33
FCMIM (IN-IK)
800
1334
80.70
81.72
81.82
Fine-tume tune on IN-21K
3223
BEiTv2-B BEiTv2-L
80,96 80.89
81.64
82.35
Fine-tune on IN-IK
90
3393
83.89
KD
81.68
82.27
vanilla KD
300
653
84.42
vanilla KD
1200
2612
85.03
teacher pre-trained on larger dataset has better understanding of the data distribution, which subsequently facilitates student learning and leads to improved performance.
3.6 Exploring the power of vanilla KD with longer training schedule
In Section 3.3, we notice a consistent improvement in the performance of vanilla KD as the training
Table 8: Results of extended training schedule.
The teacher model is BEÎTv2-B.
schedule is extended. To explore its ultimate perfor- mance, further extended the training schedule to 4800 epochs. In this experiment, we utilized a BETT-B teacher model, which achieved 86.47% top- I accuracy on the ImageNet- I validation set. As for students, employed Res50 [2], ViT-T, ViT- [3, 32], and ConveNextV2-I [34], and trained them using strategy "A1". The corresponding results shown in Table 8. All three students achieve improved performance with longer training epochs. This trend is also consistent with the pattern observed in [35]. When trained with 4800 epochs, the students achieve
Model Epoch
300
600
1200
4800
Res50 ViT-S
80.96
81.64
82.35
83.08
81.38
82.71
83.79
84.33
ViT-T
77.19
78.11
ConvNextV2-T
84.42
84.74
85.03
state-of-the-ar performance, surpassing previous literature [35, 32]. Specifically, the Res50, ViT-S, ViT-T, and ConvNeXtV2-T achieve 83.08%, 84.33%, 78.11% % and 85.03%2 top-1 accuracy, respectively.
3.7 Comparison with masked image modeling
Masked image modeling (MIM) framework [48, 49, 50] has shown promising results in training models with excellent performance. However, it worth noting that the pre-training and fine-tuning process of MIM can be time-consuming. In this section, we conduct a comparison between MIM and vanilla KD, taking into account both accuracy and time consumption. Specifically, we compare vanilla KD with the FCMIM framework proposed in ConyNeXtv2 [34]. We utilize a ConvNeXtvV-T student model and a BEiTv2-B teacher model. We train two student models using vanilla KD: one for 300 epochs and another for 1200 epochs.
The results are presented in Table 10, highlighting the efficiency of vanilla KD in training exceptional models. Specifically, even with training duration of only 300 epochs, vanilla KD achieves an accuracy of 84.42%, surpassing MIM by 0.53%, while consuming only one-fifth of the training time. Moreover, when extending the training schedule to 1200 epochs, vanilla KD achieves performance of 85.0 85.03%, surpassing the best- perfforrrnn MIM-trained model by 1.14%. These results further demonstrate the effectiveness and time efficiency of vanilla KD compared to the MIM framework.
3.8 Transferring to downstream task
To assess the transfer learning performance of the student distilled on ImageNet, we conduct ex- periments object detection and instance segmentation tasks using COCO benchmark [26]. We adopt Res50 and ConvNeXtV2-T as backbones, and initialize them with the distilled checkpoint. The detectors in experiments commonly used Mask RCNN [40] and Cascade Mask RCNN [41].
'Here we find that the result of KD has already reached satisfactory level, and the training cost is lower compared to the MIM approach. Hence, did not pursue further scaling to 4800 epochs.
8

---

Table 11: Object detection and instance segmentation results on COCO. We compare checkpoints with different pre- pre-trained accuracies on ImageNet when used as backbones in downstream tasks.
RCSN (Re50 E.).
(b) RCNN (CowNoXiV2-T(-I)
Cacade Mark RCNN (Com-NoiiV--
ckpt.
sche.
IN-IK
AP
Apm
ckpt.
sche.
IN-IK
APb
Apm
ckpt.
sche.
IN-IK
Ap
Apm
prev.
1x
77.1
38.2
34.7
prev.
1x
83,0
45,4
41.5
prev.
1x
83.0
49.8
43.5
prev.
1x
80.4
38.7
35.1
prev.
1x
83,9
45.6
41.6
prev.
1x
83.9
50.4
44.0
ours
1x
83.1
41.8
37.7
ours
1x
85.0
45.7
42.0
ours
1x
85.0
50.6
44.3
prev.
2x
77.1
39.2
35.4
prev.
3x
83.0
47.4
42.7
prev.
3x
83.0
51.1
44.5
prev.
2x
80.4
40.0
36.1
prev.
3x
83.9
47.7
42.9
prev.
3x
83.9
51.7
44.9
ours
2x
83.1
42.1
38.0
ours
3x
85.0
47.9
43.3
OuEs
3x
85.0
52.1
45.4
Table reports the corresponding results. When utilizing our distilled Res50 model as the backbone, Mask RCNN outperforms the best counterpart using backbone trained from scratch by significant margin (+3.1 box AP using "Ix" schedule and +2. 12.1% 1% box AP using "2x" schedule). Similarly, when the backbone architecture is Ne using the model trained by vanilla KD consis- tently leads to improved performance. These results demonstrate the efficient transferability of the performance improvements achieved by vanilla KD to downstream tasks.
4 Related work
Knowledge distillation (KD). Previous KD methods can be broadly categorized into two classes: logits-based methods and hint-based methods. Hinton el al. [15] first proposes train the student to learn from logits of teacher. This method is referred as vanilla KD. Recently, researchers have introduced improvements
the vanilla KD method by enhancing the dark knowledge [19, 51], relaxing the constraints [18]. using sample factors to replace the unified temperature [52]. Hint-based methods train the student to imitate learned representations of the teacher. These methods leverage various forms of knowledge, such as model activation [16, 53, 54], attention map [55]. flow of solution procedure [56]. or feature correlation among samples [22, 20]. To facilitate the alignment of features between the teacher and the student, techniques including design new projector [57, 58] and more complex alignment approaches [17, 59, 60] have been proposed. Besides directly mimicking the hint knowledge, contrastive learning [21] and masked modeling [61] have also been explored. Recently, the success of vision transformers [3] has prompted the development of specific distillation approaches tailored to this architecture, including introducing additional distillation tokens [32, 62) or perform distillation at the fine-grained patch level [63]. Additionally, vision transformer architectures have also been leveraged to facilitate KD between CNN models [64, 65]. Furthermore, some works attempt to address specific challenges in KD, such as distilling from ensembles of teachers [66, 67] and investigating the impact of different teachers student performance [68, 47, 69].
Large-scale training, Recent developments in large-scale deep learning [31, 48] suggest that by scaling up data [3, 70], model size [6, 7, 8] and training iteration [35] to improve model performance, achieve state-of of-the-art results [71] across diverse tasks, and enable better adaptability to real- world scenarios. For example, in computer vision, numerous large models have achieved outstanding results on tasks such as classification, detection, and segmentation. These models are usually trained on extensive labeled image datasets, including ImageNet-21K [30] and JFT [15, 72]. Similarly, in natural language processing, training models on massive text corpora [73, 74] has led to breakthroughs in language understanding, machine translation, and text generation tasks. Inspired by this scaling trend, it becomes crucial to evaluate KD approaches in more practical scenarios. After all, the goal of KD is to enhance the performance of student networks.
The closest work to ours is that of Beyer et al. [35]. They identify several design choices to make knowledge distillation work well in model compression. Taking it a step further, we point out the small data pitfall in current distillation literature: small-scale datasets limit the power of vanilla KD. We thoroughly study the crucial factors in deciding distillation performance. In addition, we have distilled several new state-of-the-ar backbone architectures, further expanding the repertoire of nigh-perffrriig models in vision arsenal.

---

Conclusion and discussion
Motivated by the growing emphasis on scaling up for remarkable performance gains in deep learning. this paper revisits several knowledge distillation (KD) approaches, spanning from small-scale to large-scale dataset settings. Our analysis has disclosed the small data pitfall present in previous KD methods, wherein the advantages offered by most KD variants over vanilla KD diminish when applied to large-scale datasets. Equipped with enhanced data augmentation techniques and larger datasets, vanilla KD emerges as viable contender, capable of achieving comparable performance to the most advanced KD variants. Remarkably, our vanilla KD-trained ResNet-50,, ViT-S, and ConvNeXtV2-TT models attain new state-of-the-ar performances without any additional modifications.
While the application of knowledge distillation on large-scale datasets yields significant performance improvements, it is important to acknowledge the associated longer training times, which inevitably contribute to increased carbon emissions. These heightened computational resource requirements may impede practitioners from thoroughly exploring and identifying optimal design choices. Future endeavors should focus on limitations posed by longer training times and carbon emissions, and exploring new avenues to further enhance the efficacy and efficiency of distillation methods.
Acknowledgement
This work is supported by National Key Research and Development Program of China under No. SQ2021 YFC33300C33011 National Natural Science Foundation of China under Grant 61971457. And Chang Xu was supported in part by the Australian Research Council under Projects DP210101859 and F1230100549.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 2017.
[2] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2016. 1, 2, 4, 8,
[3] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 1, 4, 8,9
[4] Jianyuan Guo. Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022. 1
[5] Zhuang Liu, Hanzi Mao. Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1.
[6] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015. 1.9
[7] Hugo Touvron, Mathieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the International Conference on Computer Vision, 2021. 1, 9
[8] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint 2203::2203000555. 2022. 1,9
[9] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In /EEE/CVF Conference on Computer Vision and Puttern Recognition, 2017. 1
[10] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1605 2016.
[11] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 2021. 1
10

---

[12] Yehui Tang, Kai Han, Yunhe Wang. Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[13] Zhiwei Hao, Yong Luo. Han Hu, Jianping An, and Yonggang Wen. Data-free ensemble knowledge distillation for privacy-conscione multimedia model compression. In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
[14] Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, and Wen Gao. Instance-aware dynamic neural network quantization. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1
[15] Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.02531, 2015. 1, 3, 4, 5,5,6 6, 7,9
[16] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, arXiv preprint arXiv:1412.6550 2014. 1, 2, 3, 9
[17] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021. 1, 2, 3, 5,9
[18] Tao Huang, Shan You, Fei Wang. Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3, 6,
[19] Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE/CVF Conference Computer Vision and Puttern Recognition, 2022. 1, 3, 4. 5, 6, 7.9
[20] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In /EEE/CVF Conference on Computer Vision and Pottern Recognition, 2019. 1, 5,9
[21] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In IEEF/CVF International Conference on Learning Representations, 2020. 2, 3, 5,
[22] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In International Conference on Computer Vision, 2019. 2, 4, 9
[23] Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. arXiv preprint vrXiv:2305.15711 2023.
[24] Hanting Chen, Yunhe Wang. Chang Xu, Zhaohui Yang. Chuanjian Liu. Boxin Shi, Chunjing Xu, Chao Xu. and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEE/CVF international conference computer vision, 2019.
[25] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009. 3,
[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In European Conference on Computer Vision, 2014. 1, 8
[27] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes datasct for semantic urban scene understanding. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2016.
[28] Eirikur Agustsson and Radu Timofte. Nure 2017 challenge on single image super-resolution: Dataset and study. In IEEE/CVF Conference on Computer Vision and Pattern Recognition workshops, pages 126-135, 2017.
[29] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-baset visual segmentation: survey. arXiv preprint 20Xv:2230.009854 2023.
[30] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009. 2, 4, 9
[31] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training training of deep bidirec- tional transformers for language understanding, arXiv preprint 1rx00:1810.04805, 2018. 2,
11

---

[32] Hugo Touvron. Matthieu Cord, Matthis Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 2021. 2, 4, 8,
[33] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2022. 2
[34] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining
Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. arXiv preprint arXiv:2301.00888 2023. 2, 4, 8.9
[35] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022. 2, 8,
[36] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint xxxx:2 2021. 2, 4, 5,6
[37] Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes a" good" data augmentation in knowledge distillation-a statistical perspective. Advances in Newrol Information Processing Systems, 2022.
[38] Zhiwei Hao, Yong Luo, Zhi Wang, Han Hu, and Jianping An. Cdfkd-mfs: Collaborative data-free knowledge distillation via multi-level feature sharing. IEEE Transactions on Multimedia, 2022.
[39] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270, 2021. 2
[40] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. International Conference on Computer Vision, 2017. 2,8
[41] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In IEEE/CVF Conference Computer Vision and Puttern Recognition, 2018. 2, 8
[42] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding, IEEE Transactions Neural Networks and Learning Systems, 2020.
[43] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-guantized visual tokenizers. arXiv preprint arXiv: 2208606366. 2022. 4, 6
[44] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are done with imagenet? arXiv preprint Xiix:20000001599, 2020.
[45] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 1389-5400, 2019.
[46] Simon Kornblith, Mohammad Norouzi, Honglak Lec, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, 2019. 5
[47] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In International Conference on Computer Vision, pages 4802, 2019. 7,9
[48] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International conference machine learning, 2015. 8,9
[49] Jianyuan Guo, Kai Han, Han Wa, Yehui Tang, Yunhe Wang, and Chang Xu. Fastmim: Expediting masked image modeling pre-training for vision. arXiv preprint urXiv22212.06593 2022. 8
[50] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2022. 8
[51] Zhendong Yang, Zhe Li, Yuan Gong, Tianke Zhang, Shanshan Lao, Chun Yuan, and Yu Li. Rethinking knowledge distillation via cross-entropy. arXiv preprint 2rXiv:2208. 2022. 9
[52] Kunran Xu, Lai Rui, Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image classifica- tion. In European Conference Computer Vision, 2020. 9
12

---

[53] Linfeng Zhang, Yukang Shi, Zuoqiang Shi, Kaisheng Ma. and Chenglong Bao. Task-oriented feature distillation. In Advances in Neural Information Procesxing Systems, 2020.
[54] Xianing Chen, Qiong Cao. Yujie Zhong. Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: data- efficient early knowledge distillation for vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12052-12062, 2022. 9
[55] Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the perfor- mance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017. 9
[56] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 4133-4141, 2017.
[57] Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. Advances in Neural Information Processing Systems, 2022.
[58] Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representa- tion matching. In International Conference on Learning Representations. 2023. 9
[59] Defang Chen, Jian Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng. and Chun Chen. Knowledge distillation with the reused teacher classifier. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11933-1 1942, 2022. 9
[60] Dongyang Liu, Meina Kan, Shiguang Shan, and CHEN Xilin. Function-consistent feature distillation. In International Conference on Learning Representations, 2023. 9
[61] Zhendong Yang. Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In European Conference on Computer Vision, pages 53-69, 2022. 9
[62] Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He. and Hang Zhao. Co-advise: Cross inductive bias distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16773-16782, 2022. 9
[63] Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine-grained manifold distillation. In Advances in Neural Information Processing Systems, 2022. 9
[64] Martin Zong. Zengyu Qiu, Xinzhu Ma. Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In International Conference on Learning Representations, 2023. 9
[65] Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10915-10924, 2022.
[66] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285-1294, 2017.9 9
[67] Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020. 9
[68] Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi, A comprehen- sive overhaul of feature distillation. In International Conference on Computer Vision, pages 1921-1930, 2019. 9
[69] Tianjin Huang, Lu Yin, Zhenyu Zhang, Li Shen, Meng Fang, Mykola Pechenizkiy, Zhangyang Wang, and Shiwei Liu. Are large kemels better teachers than transformers for convnets? arXiv preprint arXiv:2305. 19412, 2023. 9
[70] Qizhe Xie, Minh- Than-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. 9
[71] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019. 9
13

---

[72] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference compuier vision and paltern recognition, 2017. 9
[73] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary. Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data, arXiv preprint arXiv:1911.00359 2019.
[74] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-tex transformer. The Journal of Machine Learning Research, 2020. 9
14


Reviews:
[{"review_id": "ITbDY2NFyj", "reviewer": "Reviewer_oNYG", "summary": "This paper investigates the effectiveness of vanilla Knowledge Distillation (KD) in large-scale datasets. The authors identify a \"small data pitfall\"  which underestimates the power of vanilla KD on large-scale datasets and demonstrate that stronger data augmentation techniques and larger datasets can decrease the gap between vanilla KD and other KD variants.", "questions": "- Temperature is an important hyper-parameter for the KD method. what value is used for temperature in all experiments? How does the temperature affect the vanilla KD performance from small to large datasets? \n\n- Does the conclusion still hold in lightweight models such as MobileNetv3?", "limitations": " the authors adequately addressed the limitations", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Originality**: This is an empirical study. It has no technical novelty but sheds some insights into the knowledge distillation method.\n\n- **Quality**: Extensive experiments validate their conclusions.\n\n- **Clarity**: This paper is well-structured and easy to follow. \n\n- **Significance**: Knowledge Distillation is an important and interesting topic and exploring the impacts of the size of datasets for KD is beneficial for the community.\n\n- New state-of-the-art results for ResNet-50, ConvNeXt-T architectures.", "weaknesses": "-  Lightweight models, such as MobileNetv3, are deemed critical model architectures due to their efficiency and compactness, which make them ideal for deployment on devices with limited computational resources. However, it appears that there is a lack of dedicated experiments specifically for these lightweight models.\n\n- Two related papers are missing[1][2]\n\n[1] Yang, Zhendong, et al. \"Rethinking knowledge distillation via cross-entropy.\" arXiv preprint arXiv:2208.10139 (2022).\n\n[2] Huang, Tianjin, et al. \"Are Large Kernels Better Teachers than Transformers for ConvNets?.\" ICML. 2023.\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1:** Lightweight models, such as MobileNetv3, are deemed critical model architectures due to their efficiency and compactness, which make them ideal for deployment on devices with limited computational resources. However, it appears that there is a lack of dedicated experiments specifically for these lightweight models.\n> \n> **Question 2:** Does the conclusion still hold in lightweight models such as MobileNetv3?\n\n**Response to weakness 1 & question 2:** Thank you for your valuable suggestion. We have opted to employ MobileNetv3 as a representative lightweight model for our experimental investigations, as shown in the following table. The outcomes reveal that on the CIFAR-100 dataset, vanilla KD falls short of the leading baseline, DKD, by 0.51% in terms of top-1 accuracy. Nevertheless, on the ImageNet-1K dataset, vanilla KD emerges as the top performer. These results reaffirm the validity of the small data pitfall observed in our main paper.\n\n|   Dataset   |  Teacher   |      Student       | Method |   Top-1   |\n| :---------: | :--------: | :----------------: | :----: | :-------: |\n|  CIFAR-100  |     -      | MobileNet v3 Small |   -    |   64.76   |\n|  CIFAR-100  | ResNet32x4 | MobileNet v3 Small |  DKD   | **69.10** |\n|  CIFAR-100  | ResNet32x4 | MobileNet v3 Small |  DIST  |   67.72   |\n|  CIFAR-100  | ResNet32x4 | MobileNet v3 Small |   KD   |   68.59   |\n| ImageNet-1K |     -      | MobileNet v3 Small |   -    |   67.40   |\n| ImageNet-1K |  BeiTv2-B  | MobileNet v3 Small |  DKD   |   67.36   |\n| ImageNet-1K |  BeiTv2-B  | MobileNet v3 Small |  DIST  |   68.02   |\n| ImageNet-1K |  BeiTv2-B  | MobileNet v3 Small |   KD   | **68.05** |\n\n> **Weakness 2:** Two related papers are missing\n\n**Response to weakness 2:** Thank you for bringing the overlooked related papers to our attention. [1] decomposes the original KD loss into two components and proposes NKD loss to discard the challenging-to-optimize portion. [2] points out that large-kernel ConvNets are more effective teachers than Transformers for small-kernel ConvNets. We will incorporate references to these two papers in the related work section of our next version.\n\n> **Question 1:** Temperature is an important hyper-parameter for the KD method. what value is used for temperature in all experiments? How does the temperature affect the vanilla KD performance from small to large datasets?\n\n**Response to question 1:** In our experiments, we follow the implementation of DKD to employ the temperature setting of T=4 for all experiments on CIFAR-100 and T=1 for those on ImageNet-1K. \n\nTo assess the influence of temperature in knowledge distillation (KD), we conducted experiments on both the CIFAR-100 and ImageNet-1K datasets. As indicated in the table below, the effect of temperature on student performance is non-monotonic. However, a discernible trend emerges: larger temperature values (T > 1) tend to be more effective for small-scale datasets, while a smaller temperature value (T=1) is preferable for larger-scale datasets. We posit that smaller-scale datasets often lead to teacher models overfitting, resulting in predictions with low entropy. Consequently, softening these predictions with a higher temperature can yield more valuable information for the student. Conversely, in the context of larger-scale datasets, teachers inherently provide less certain predictions, and elevating uncertainty further by using a higher temperature setting could hinder student learning.\n\n|   Dataset   |  Teacher   |      Student       |  Method  | Top-1 |\n| :---------: | :--------: | :----------------: | :------: | :---: |\n|  CIFAR-100  | ResNet32x4 | MobileNet v3 Small | KD (T=1) | 67.07 |\n|  CIFAR-100  | ResNet32x4 | MobileNet v3 Small | KD (T=2) | 69.35 |\n|  CIFAR-100  | ResNet32x4 | MobileNet v3 Small | KD (T=4) | 68.59 |\n| ImageNet-1K |  BeiTv2-B  |      ResNet50      | KD (T=1) | 80.96 |\n| ImageNet-1K |  BeiTv2-B  |      ResNet50      | KD (T=2) | 80.39 |\n| ImageNet-1K |  BeiTv2-B  |      ResNet50      | KD (T=4) | 80.53 |\n\nWe have also added experiments concerning the temperature settings for DKD and DIST on ImageNet-1K. The specific results are provided below. We will incorporate these findings to Table 16 in our supplementary material.\n\n|   Dataset   | Teacher  | Student  |   Method   | Top-1 |\n| :---------: | :------: | :------: | :--------: | :---: |\n| ImageNet-1K | BeiTv2-B | ResNet50 | DIST (T=1) | 80.76 |\n| ImageNet-1K | BeiTv2-B | ResNet50 | DIST (T=2) | 80.46 |\n| ImageNet-1K | BeiTv2-B | ResNet50 | DIST (T=4) | 80.31 |\n| ImageNet-1K | BeiTv2-B | ResNet50 | DKD (T=1)  | 80.94 |\n| ImageNet-1K | BeiTv2-B | ResNet50 | DKD (T=2)  | 80.76 |\n| ImageNet-1K | BeiTv2-B | ResNet50 | DKD (T=4)  | 80.70 |"}, {"review_id": "q3J7aIKpmt", "reviewer": "Reviewer_mQmQ", "summary": "This paper revisits vanilla knowledge distillation and presents an empirical analysis of the impact of model size, dataset scale, and training strategy on student performance in knowledge distillation. It identifies: 1) the gap between vanilla KD and other carefully designed KD methods gradually diminishes when adopting stronger data augmentation techniques and longer training iterations on large-scale datasets such as ImageNet-1K; 2) on small scale datasets, vanilla KD consistently underperforms other KD approaches although stronger training strategy and longer training iterations are used; and 3) logits-based methods outperform hint-based methods in terms of generalizability in more challenging cases. Based on these observations, this paper trains four different student models achieving SOTA performance on ImageNet-1K solely using the vanilla KD method.", "questions": "Please refer to the weaknesses.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Personally, I agree that vanilla KD can serve as a formidable contender when employed with large-scale datasets. Hints serve as valuable priors during training **only** when the available data is severely limited. \n2. The authors diligently conducted extensive experiments to thoroughly examine the performance of vanilla KD across various factors, such as model capacity, dataset size, training epochs, learning rate, and regularization techniques, among others.\n3. Vanilla KD is indeed a practical approach with widespread applications, making this paper highly beneficial for a diverse range of readers.\n4. Notably, this paper presents state-of-the-art (SOTA) models trained using vanilla KD, which can significantly aid in further research.", "weaknesses": "- The evaluation of vanilla KD in this work is solid and self-contained. When dealing with a small training set, it is customary to incorporate additional priors, regularization techniques, augmentations, or hints like previous KD methods. Naturally, with a larger dataset, we can leverage vanilla KD with less regularisation to attain comparable performance levels. However, it would be better if the authors can provide a systematical analysis for the choice of algorithms under Small Scale & Large Scale settings.\n- The paper compares teacher models in two aspects: model size and dataset scale. Adopting a teacher model with more parameters is beneficial (ResNet152 vs. BEiTv2-L), but an extremely large teacher is harmful (BEiTv2-B vs. BEiTv2-L). As for the dataset scale, training the teacher on larger datasets is helpful. I wonder if we continue to increase the dataset scale, whether it will show a trend similar to increasing the model size, i.e., the student performance decreases. If not, what leads to this difference?\n- The conclusion of comparison KD with MIM is a bit weak as it assumes a stronger teacher model is available. In practice, this may not always hold. For example, if we want to train a BeITv2-L, perhaps BeITv2-L is not a good teacher, and we cannot find one achieving higher accuracy. So I think KD is only preferable when training small- or medium-size models.\n\nMinor typos:  \n1. \"reflection on whether\" in line 73\n2. \"results show that\" in line 127\n3. \"experimental setup is as follows\" in line 136\n4. \"other approaches\" in line 200\n5. \"that trained\" in line 243\n6. \"learn from\" in line 304\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to weakness 1:** Thanks for your valuable suggestions. The trend is that a small training set prefers a knowledge distillation (KD) method with stronger regularization or priors, as it can effectively bring in more informative knowledge from the teacher model to enhance the performance of the smaller student model. On the other hand, with a large training set, the need for additional prior diminishes. The sheer abundance of diverse data in a larger dataset already provides sufficient information for the student model to learn from. In such cases, a simpler KD approach may suffice, as the teacher model can effectively capture the complexities present in the data without requiring extensive regularization or priors.\n\nTo verify the above analysis, we present the average confidence on all samples and the entropy of predictions of pretrained ResNet50/ResNet152 teacher models on CIFAR-100/ImageNet-1K. We use the two following measurements: \n\n$\n\\max({p}(x))=\\mathbb{E}_{x\\in\\mathcal{X}}[\\max_i(p_i)],\n$\n\nand \n\n$\n\\text{entropy}({p}(x))=\\mathbb{E}_{x\\in\\mathcal{X}}[\\textstyle \\sum_i p_i\\log p_i],\n$\n\nwhere $p_i$ is predictive probability of sample $x$ belonging to class $i$.\n\nThe average confidence score provides an insight into how certain the teacher's predictions are across all samples. As shown in the table, model trained on CIFAR-100 exhibits higher average confidence and lower entropy value, suggesting that the model is more confident in its predictions, however, resulting in reduced mutual information or knowledge transfer among other classes.\n\n||Top-1|$\\max({p}(x))$|$\\text{entropy}({p}(x))$\n|:-:|:-:|:-:|:-:\n|CIFAR-100|79.33|89.78|0.3362\n|ImageNet-1K|75.81|79.63|0.7593\n\n> **Weakness 2:** increase the dataset scale\n\nTo demonstrate this, we first analyze the impact of model size by comparing the average confidence on all samples and the entropy of predictions of ResNet152/BEiTv2-B/BEiTv2-L teacher model on ImageNet-1K.\n\n||Top-1|$\\max({p}(x))$|$\\text{entopy}({p}(x))$|\n|:-:|:-:|:-:|:-:|\n|ResNet152|82.83|67.46|2.4418|\n|BEiTv2-B|86.39|80.71|1.2868|\n|BEiTv2-L|88.39|83.94|1.1172|\n\nAs depicted in the table, as the teacher model size increases, their predictions become less informative. Our hypothesis is that smaller teacher models may struggle to learn the dataset adequately, leading to noisy soft labels that are less beneficial for the student. On the other hand, larger teacher models tend to overfit the dataset, hindering the distillation process with their soft labels (more closely resembling one-hot ground truth and less transferable knowledge to other classes). Consequently, the model of medium size, BEiTv2-B, exhibits the best performance in our experiments\n\nBased on the analysis above, teacher models tend to overfit smaller datasets, leading to a lack of valuable information for distillation. Recent KD approaches have outperformed vanilla KD by introducing additional information on small-scale datasets. However, as the dataset scale increases, the overfitting issue diminishes, making it challenging to optimize the objectives of complex KD methods due to an abundance of information for the student to learn. On the contrary, the simplicity of vanilla KD, which solely relies on soft labels, becomes sufficient for distillation.\n\nConsequently, we expect an improvement in the student's performance owing to the ample information provided by the increasing dataset scale. And we believe that as the dataset scale further increases, vanilla KD will achieve comparable performance to other KD baselines. The reduction in overfitting with larger datasets mitigates the need for more intricate KD approaches, reaffirming the effectiveness of vanilla KD for knowledge transfer in such scenarios.\n\n**Response to weakness 3:** Indeed, there are instances where a more powerful teacher is not accessible. To assess the effectiveness of vanilla KD under such circumstances, we conduct experiments on ImageNet-1K, employing the identical architecture for both the teacher and student models. In this regard, we utilize the BEiTv2-B model as our chosen architecture and proceed to compare the performance of vanilla KD against the MIM pretraining method introduced by BEiTv2 [1], evaluating accuracy and training time consumption as key metrics. The GPU time measurements are conducted using a single NVIDIA Tesla V100 GPU.\n\n|Method|Epoch|Top-1|GPUtime(hours)\n|:-:|:-:|:-:|:-:\n|MIM|300(pretrain)+100(finetune)|85.0|853\n|MIM|1600(pretrain)+100(finetune)|85.5|3979\n|vanillaKD|300|84.6|575\n|vanillaKD|600|85.4|1151\n|vanillaKD|1600|85.7|3069\n\nBased on above results, it's evident that the performance of vanilla KD is on par with that of MIM. For example, the student model trained through vanilla KD (600 epochs) attains a top-1 accuracy of 85.4%, only 0.1% lower than the equivalent model trained with MIM (1600 epochs). However, the vanilla KD approach requires notably less training time. Furthermore, extending the training epochs to 1600 yields a top-1 accuracy of 85.7%, surpassing the MIM result by 0.2% under the same epoch settings. This underscores that vanilla KD remains competitive with MIM, even when a more robust teacher model is unavailable.\n\nSpecifically, we choose the BEiTv2-B with 85.5% top-1 accuracy (the 2nd line) as the teacher model in KD. In our reported GPU time, we omitted the training duration of the teacher model. Given that it is typically feasible to procure a pre-trained model with performance comparable to the student model, the necessity to train a teacher model from the ground up is minimized in most scenarios.\n\n[1] Beit v2: Masked image modeling with vector-quantized visual tokenizers\n\n**Response to weakness 4:** Thank you for thoroughly reviewing our submission and pointing out the mistakes. We will take great care to address these issues in our next version. Your feedback is invaluable in improving the quality of our work."}, {"review_id": "hbvAXNyQgc", "reviewer": "Reviewer_bACv", "summary": "The paper explores the effectiveness of knowledge distillation (KD) approaches for limited-capacity architectures based on small-scale datasets. The authors identify the \"small data pitfall\" in previous KD methods, which leads to underestimation of the power of the vanilla KD framework on large-scale datasets like ImageNet-1K.", "questions": "1. For classification tasks, if that's still the case, what about other tasks?\n2. Is there another possibility that this phenomenon is not solely due to the size of the data, but rather to the difficulty or performance bottleneck of the classification task？\n3. Can you provide a presentation of the classification results on a certain dataset, and compare the differences between DIST and vanilla KD in terms of whether the classification of certain key samples is correct or not? What about using a subset of imagenet-1k for training and distillation?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although this work does not propose a novel distillation method, the identification of small data pitfall and a series of analyzes based on this would provide valuable insights for the vision community in the field of knowledge distillation.\n2. The experiment is very complete and sufficient, with some persuasiveness.\n3. The findings of this article are very important and may correct the research direction of KD.", "weaknesses": "1. This article mainly focuses on experimental comparison, while neglecting theoretical analysis. Lack of in-depth analysis of the underlying causes of the observations.\n2. The author has overlooked a phenomenon where the KD improvement on large-scale datasets (i.e. ImageNet-1K) is inherently small. Why is this? What is the impact of this on the \"small data pitfall\" proposed in this paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer bACv part (1/2)\n\n**Response to weakness 1:** Thanks for your valuable comments. Our observation suggests that, in the context of knowledge distillation tasks, smaller training sets exhibit a preference for methods featuring stronger priors. These methods effectively impart more informative knowledge [1,2,3] to the student, compensating for the limited data available. Conversely, larger training sets negate the need for such measures. The substantial volume of diverse data inherent to larger datasets naturally imparts ample information for the student model's learning. In such scenarios, a simpler knowledge distillation approach might be adequate, as the teacher model is apt at capturing the inherent complexities within the data, obviating the requirement for extensive priors. In addition, when comparing subsets and the complete ImageNet-1K dataset, the discrepancy between vanilla KD and other methods narrows as the training set scale increases. This observation leads us to believe that informative 'dark' knowledge can naturally manifest within larger datasets. Similarly, Stanton et al. [4] evaluates the fidelity of KD and points out that the student fidelity will increase as the dataset grows: \"Enlarging the distillation dataset beyond the teacher training data makes it easier for the student to identify the correct solution, but also makes an already difficult optimization problem harder.\"\n\nWe have conducted additional experiments in our response to ***weakness 1 of mQmQ***, and we paste the results in the table below. To verify our analysis, we show the average confidence on all samples and the entropy of predictions of pretrained Res50/Res152 teacher models on CIFAR-100/ImageNet-1K. We use the two following measurements: \n\n$\n\\max({p}(x))=\\mathbb{E}_{x\\in\\mathcal{X}}[\\max_i(p_i)],\n$\n\nand \n\n$\n\\text{entropy}({p}(x))=\\mathbb{E}_{x\\in\\mathcal{X}}[\\textstyle \\sum_i p_i\\log p_i],\n$\n\nwhere $p_i$ is predictive probability of sample $x$ belonging to class $i$.\n\nThe average confidence score provides an insight into how certain the teacher's predictions are across all samples. As shown in the table, model trained on CIFAR-100 exhibits higher average confidence and lower entropy value, suggesting that the model is more confident in its predictions, however, resulting in reduced mutual information or knowledge transfer among other classes.\n\n||Top-1|$\\max({p}(x))$|$\\text{entropy}({p}(x))$|\n|:-:|:-:|:-:|:-:|\n|CIFAR-100|79.33|89.78|0.3362|\n|ImageNet-1K|75.81|79.63|0.7593|\n\nWe will add these analyses to our next version.\n\n[1] Huang, Tao, et al. \"Knowledge distillation from a stronger teacher.\" Advances in Neural Information Processing Systems 35 (2022): 33716-33727.\n\n[2] Zhao, Borui, et al. \"Decoupled knowledge distillation.\" Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.\n\n[3] Yang, Zhendong, et al. \"Rethinking knowledge distillation via cross-entropy.\" arXiv preprint arXiv:2208.10139 (2022).\n\n[4] Stanton, Samuel, et al. \"Does knowledge distillation really work?.\" Advances in Neural Information Processing Systems 34 (2021): 6906-6919.\n\n**Response to weakness 2:** If I understand correctly, one example of the phenomenon you've mentioned is that most KD approaches can only bring about 1% improvement on top-1 accuracy on ImageNet-1K, but can get about 3-5% improvement on CIFAR-100. Please correct me if I am mistaken. The reasons behind such phenomenon are two-fold. (1) CIFAR-100 is a small-scale dataset consisting of 50K/10K training/testing samples. It is hard for a model trained on such a small dataset to capture the real distribution well. Therefore, the improvement on the student's performance is significant by using the teacher's predictions as additional supervision. However, the ImageNet-1K dataset consists of 1.2M samples, where the diverse data itself is sufficient for the student model to learn the distribution well enough. In such case, the introduction of teacher model only help improve the student performance marginally. (2) Existing approaches usually use small networks such as Res18 and WRN-16 on CIFAR-100, while they use large networks such as Res50 on ImageNet. The difficulty of optimizing models of different sizes (the number of parameters) varies. Some studies [1] indicated that different optimization approaches also influence the results. The previously widely used 90-epoch protocol on ImageNet might now be considered insufficient. This is also aligned with the outcomes from our 'stronger recipe' in Table 1. It seems that we need to use a larger scale dataset to more comprehensively assess the performance of models and KD methods.\n\n[1] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in 420 timm. arXiv preprint arXiv:2110.00476, 2021.\n\n**Response to question 1:** Thanks for your valuable comments. We conduct experiments on COCO (118K training images) object detection task to directly compare DKD, DIST, and vanilla KD. Adhering to DKD's implementation, we applied distillation to the classification branch, with ResNet101 serving as the teacher backbone and ResNet18 as the student backbone. The outcomes indicate that vanilla KD and DIST achieve comparable performance, while DKD surpasses the other two methods. If we consider our assumption of the 'small data pitfall,' we conjecture that this result could be attributed to the limited scale of the COCO dataset. In the future, we plan to validate this speculation on a larger dataset, Objects365, comprising 1720K training images.\n\n|Iteration|Method|AP|AP50|AP75|\n|:-:|:-:|:-:|:-:|:-:|\n|180K|FromScratch|33.26|53.61|35.26|\n|180K|DKD|35.07|56.32|37.45|\n|180K|DIST|34.52|55.64|37.33|\n|180K|KD|34.75|55.95|37.31|\n|540K|DKD|37.44|58.75|40.37|\n|540K|DIST|37.01|58.00|40.07|\n|540K|KD|36.99|57.92|39.87|\n\n### ***The part (2/2) can be found in \"Author Rebuttal\" at the top of this page.***"}, {"review_id": "Cw65Mn4WCo", "reviewer": "Reviewer_aER7", "summary": "This paper explores the power of vanilla distillation on large datasets and strong training recipes. It shows the stronger data augmentation and using larger datasets can decrease the gap between vanilla KD and other meticulously designed methods. The extensive results show vanilla KD's power.", "questions": "see the weakness", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper reveals the enormous potential of vanilla KD. It inspires the community to take a broader look at knowledge distillation instead of small models or datasets.\n2. This paper takes an enormous number of experiments to explore vanilla KD and obtain many valuable results and SOTA models.", "weaknesses": "1. We usually train the models under an ordinary training recipe actually. And under this setting, vanilla KD performs worse. So it is still not clear which method we should apply for various settings. For example, in LKBT[1], this paper compares KD and another method NKD under various settings including stronger training recipes. But NKD performs much better than KD.\n\n    [1] Are Large Kernels Better Teachers than Transformers for ConvNets. ICML 2023\n\n2. The paper lacks the baseline performance under a stronger recipe. Generally, KD is introduced for a small model and ordinary training recipe. When the training recipe is stronger, the improvements that KD brings are fewer. In this paper, we cannot know how much KD helps the students under a stronger training recipe.\n3. The paper applies downstream tasks by replacing the backbones. However, DKD and Dist can be applied to the classification branch directly for detection distillation. They perform much better than KD. How about vanilla KD performs under a stronger training recipe for detection? Does the conclusion keep the same?\n4. The hyper-parameter $\\alpha$ and temperature for KD are also important. This paper lacks a discussion about this.\n5. Some typos, e.g. the performance of Res50 and MBNetV2 are reversed in Tab 1. Some details are missing, for example, the teacher BeiTV2 seems to be training with ImageNet-21K. The paper needs to clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to weakness 1:** Thanks for your valuable comments. We first answer for the method should be applied for different settings. Through our empirical sutdy, we observe a trend wherein a small training set prefers a knowledge distillation (KD) method with stronger regularization or priors, as it can effectively bring in more informative knowledge from the teacher model to enhance the performance of the smaller student model. On the other hand, with a large training set, the need for additional prior diminishes. The sheer abundance of diverse data in a larger dataset already provides sufficient information for the student model to learn from. In such cases, a simpler KD approach may suffice, as the teacher model can effectively capture the complexities present in the data without requiring extensive regularization or priors.\n\nTo verify the above analysis, we present the average confidence on all samples and the entropy of predictions of pretrained ResNet50/ResNet152 teacher models on CIFAR-100/ImageNet-1K. We use the two following measurements: \n\n$\n\\max({p}(x))=\\mathbb{E}_{x\\in\\mathcal{X}}[\\max_i(p_i)],\n$\n\nand \n\n$\n\\text{entropy}({p}(x))=\\mathbb{E}_{x\\in\\mathcal{X}}[\\textstyle \\sum_i p_i\\log p_i],\n$\n\nwhere $p_i$ is predictive probability of sample $x$ belonging to class $i$.\n\nThe average confidence score provides an insight into how certain the teacher's predictions are across all samples. As shown in the table below, model trained on CIFAR-100 exhibits higher average confidence and lower entropy value, suggesting that the model is more confident in its predictions, however, resulting in reduced mutual information or knowledge transfer among other classes.\n\n||Top-1|$\\max({p}(x))$|$\\text{entropy}({p}(x))$|\n|:-:|:-:|:-:|:-:|\n|CIFAR-100|79.33|89.78|0.3362|\n|ImageNet-1K|75.81|79.63|0.7593|\n\nNext, we address the question of why NKD [1] outperforms KD. Take the SLaK-T - Res50 teacher-student pari as an example, when distilling for 120 epochs on ImageNet, NKD achieves a top-1 accuracy of 78.57%, while KD achieves 77.05%. However, extending the distillation duration to 300 epochs leads to NKD achieving a top-1 accuracy of 80.24%, and in our implementation of this extended training (which was not reported in [1]), KD achieves a top-1 accuracy of 80.11%. Remarkably, the gap between KD and NKD narrows significantly with this increased training epoch, further corroborating our observation that aligns with the concept of small data pitfall.\n\n**Response to weakness 2:** In Table 1 of our main paper, we compared the performance of several baseline models using both ordinary and stronger training recipes. On small-scale datasets, the stronger training recipe improves the performance of vanilla KD, yet a significant performance gap remains between vanilla KD and recent KD baselines. On large-scale datasets, the stronger recipe also leads to improved performance for vanilla KD, resulting in a reduction of the performance gap. Building on the aforementioned observation, we introduce the concept of the \"small data pitfall.\" Furthermore, as a response to ***weakness 1 of oNYG***, we conducted experiments using the lightweight MobileNet V3 Small model, reaffirming the validity of the small data pitfall identified in our main paper. For your convenience, we present the results here:\n\n|Dataset|Teacher|Student|Method|Top-1|\n|:-:|:-:|:-:|:-:|:-:|\n|CIFAR-100|-|MobileNet v3 Small|-|64.76|\n|CIFAR-100|ResNet32x4|MobileNet v3 Small|DKD|**69.10**|\n|CIFAR-100|ResNet32x4|MobileNet v3 Small|DIST|67.72|\n|CIFAR-100|ResNet32x4|MobileNet v3 Small|KD|68.59|\n|ImageNet-1K|-|MobileNetv3Small|-|67.40|\n|ImageNet-1K|BeiTv2-B|MobileNet v3 Small|DKD|67.36|\n|ImageNet-1K|BeiTv2-B|MobileNet v3 Small|DIST|68.02|\n|ImageNet-1K|BeiTv2-B|MobileNet v3 Small|KD|**68.05**|\n\n**Response to weakness 3:** We followed the released DKD code to conduct object detection experiments, directly applying distillation to the classification branch. The results are presented in the following table. We used ResNet101 as the teacher backbone and ResNet18 as the student backbone. Among the results, vanilla KD and DIST achieved similar performance, while DKD outperformed the other two methods. Based on our small data pitfall assumption, we speculate that the performance gap is due to the limited scale of the COCO dataset. With a larger dataset, vanilla KD could potentially achieve more competitive performance. To further validate this, we plan to explore stronger training recipes (mosaic augmentation and large-scale jittering) and larger datasets (such as Objects365) for downstream tasks in our future work.\n\n|Iteration|Method|AP|AP50|AP75|\n|:-:|:-:|:-:|:-:|:-:|\n|180K|From Scratch|33.26|53.61|35.26|\n|180K|DKD|35.07|56.32|37.45|\n|180K|DIST|34.52|55.64|37.33|\n|180K|KD|34.75|55.95|37.31|\n|540K|DKD|37.44|58.75|40.37|\n|540K|DIST|37.01|58.00|40.07|\n|540K|KD|36.99|57.92|39.87|\n\n**Response to weakness 4:** Thanks for the valuable comments. We adopt a trivial setting of hyperparameter $\\alpha$ by setting its value to 1, as our primary focus is not on optimizing the configuration of vanilla KD, but rather on showcasing its latent potential. We believe that this straightforward setting is sufficient to support our conclusion. Furthermore, our ablation study, as presented in Table 4, has explored the scenario with $\\alpha=0$, which yielded performance similar to that of $\\alpha=1$. This suggests that the specific value of $\\alpha$ has minimal impact on our main findings.\n\n***Regarding the temperature parameter, please refer to our \"Author Rebuttal\" section.***\n\n**Response to weakness 5:** Thanks for your valuable suggestion. The BeiTV2 teacher is trained with ImageNet-21K. More specifically, it is pretrained on ImageNet-1K and then finetuned on ImageNet-21K and ImageNet-1K successively. We will correct the typos and added more details about the BeiTV2 teacher in our final version."}]

Author rebuttal:
# Response to all reviewers

We thank all four reviewers for their constructive feedbacks which greatly improved the quality of our paper.


### **Response to Reviewer bACv, part (2/2)**

**Response to question 2:** Thanks for the insightful question.  First, we analyzed the impact of task difficulty. In Figure 1 of our main paper, we compared the performance of vanilla KD and two baselines on the complete ImageNet-1K dataset as well as its two subsets. These subsets were obtained through stratified random sampling from the same 1000 classes, thus ensuring that the difficulty level (the number of categories) of tasks remains consistent across the datasets. The results illustrated that as the dataset scale increased, the performance gap between vanilla KD and the baselines diminished. This suggests that the observed phenomenon is driven by the dataset size rather than inherent task difficulty.

Next, we explored the performance bottleneck of the classification task. On the smaller-scale CIFAR-100 dataset, the baselines achieved superior results compared to vanilla KD. This implies that vanilla KD's performance has not yet reached a bottleneck (student can obtain higher accuracy). In the context of the larger-scale ImageNet-1K dataset, our experiments, which utilized an extended training schedule as detailed in Section 3.6, demonstrated that the student's performance did not reach a bottleneck even with training epochs extended to 4800. Since most of our experiments utilized 300/600/1200 epochs, we can confidently discount the performance bottleneck of the classification task as a factor influencing our conclusion.

**Response to question 3:** Thanks for the valuable comments. For the presentation, please refer to our "Author Rebuttal" section where we attached the PDF to show the cases.

As for using a subset of imagenet-1k, we show the results of using a subset of ImageNet-1K, i.e., 30% and 60%, to distill the student model. The results on validation set can be found in Table 14 in our supplementary material. When comparing the outcomes between subsets and the complete ImageNet-1K dataset, the discrepancy between vanilla KD and other methods narrows as the training set scale increases.

We choose the sutdent trained in Figure 1 of our main paper (ResNet50 trained on ImageNet-1K 30%/60%/full) to comapre the differences between DIST and vanilla KD. As the Table 1 shows, students trained by DIST and KD achieve similar performance. However, these two students give different predictions on more than 3000 samples. The result indicates that students trained via vanilla KD and DIST can be be very different at single sample level.

The marker "$\checkmark$" indicates the student trained by the corresponding method gives correct predictions, and marker "$\times$" indicates it gives wrong predictions. The adopted dataset is the validation set of ImageNet-1K, which consists of 50K samples. For example, the first row, DIST ($\checkmark$), indicates that when trained with a subset (30%) of ImageNet, there are 39872 samples correctly classified by DIST. The fourth row, DIST ($\checkmark$) & KD ($\times$), indicates that when trained with a subset (30%) of ImageNet, there are 1904 samples correctly classified by DIST but misclassified by KD.

||subset (30%)|subset (60%)|ImageNet (100%)|
|:-:|:-:|:-:|:-:|
|DIST ($\checkmark$)|39872|40665|40912|
|KD ($\checkmark$)|39781|40668|41137|
|DIST ($\checkmark$) & KD ($\checkmark$)|37968|38982|39270|
|DIST ($\checkmark$) & KD ($\times$)|1904|1683|1642|
|DIST ($\times$) & KD ($\checkmark$)|1813|1686|1867|
|DIST ($\times$) & KD ($\times$)|8315|7649|7221|

We have included visualizations of specific cases in the attached PDF. We selected four categories randomly and displayed the samples that are correctly classified by both DIST and KD, correctly classified by DIST but misclassified by KD, correctly classified by KD but misclassified by DIST, or misclassified by both DIST and KD. Upon analyzing the visualizations, we observed that vanilla KD tends to learn the distribution of the teacher, as seen in cases like the fourth row for the "Binoculars" category where the misclassified class by vanilla KD is the same as the teacher's class. We speculate that this behavior might be attributed to the KL divergence loss, which encourages the student to directly match the teacher's distribution. In contrast, DIST learns more implicit similarity relationships, which could result in a more explicit difference from the teacher's prediction.

### **Response to Reviewer aER7 weakness 4, part (2/2)**

Regarding the temperature parameter, we conducted supplementary experiments to investigate its impact on the performance of knowledge distillation (KD). We conduct experiments on both CIFAR-100 and ImageNet-1K dataset. From the results, the impact of temperature to the student performance is not monotonic, but there is a trend that a small temperature value is better for the small scale dataset while a larger temperature value is better for the larger scale dataset. We speculate that teacher models tend to overfit small scale dataset. Hence, predictions softened by a higher temperature can provide more useful information for the student. However, on large scale dataset, teachers already give uncertain predictions, and further increasing the uncertainty by using a higher temperature setting would impede learning of the student.

|Dataset|Teacher|Student|Method|Top-1|
|:-:|:-:|:-:|:-:|:-:|
|CIFAR-100|ResNet32x4|MobileNet v3 Small|KD(T=1)|67.07|
|CIFAR-100|ResNet32x4|MobileNet v3 Small|KD(T=2)|69.35|
|CIFAR-100|ResNet32x4|MobileNet v3 Small|KD(T=4)|68.59|
|IN-1K|BeiTv2-B|ResNet50|KD(T=1)|80.96|
|IN-1K|BeiTv2-B|ResNet50|KD(T=2)|80.39|
|IN-1K|BeiTv2-B|ResNet50|KD(T=4)|80.53|

Meta-review: This paper explores the potential of Vanilla Knowledge Distillation (KD) approaches for limited-capacity architectures, and identifies the small data pitfall that limits the effectiveness of previous KD methods on large-scale datasets. The main strengths of this paper include its thorough investigation of the crucial factors in deciding distillation performance. The authors demonstrate that using stronger data augmentation techniques and larger datasets can significantly improve the performance of vanilla KD, highlighting the importance of designing and evaluating KD approaches in practical scenarios. The study also compares the effectiveness of vanilla KD to other meticulously designed KD variants, and shows that vanilla KD can achieve state-of-the-art results on ImageNet without the need for complex training strategies or model capacities. 

The authors effectively addressed initial concerns raised by reviewers, including new experiments on compact NNs, an ablation study covering regularization, dataset size, and self-distillation, and clarifications regarding small versus large-scale datasets. While there was a debate with Reviewer aER7 about their training recipe (different from conventional ImageNet 90-epoch setting), AC concluded that it followed a reasonable experimental practice in KD literature and should not be penalized.

AC believes that the paper's main contribution is not necessarily in introducing a new SOTA KD method but in revealing the limitations and challenges of evaluating KD on small-scale datasets. The paper effectively conveys this message, making it sound and meaningful, leading AC to recommend its acceptance.
Decision: Accept (poster)