ScaleKD: Strong Vision Transformers Could Be Excellent Teachers
Authors: Jiawei Fan
Chao Li
Xiaolong Liu
Anbang Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: knowledge distillation
model compression
training acceleration
vision transformer
convolutional neural network
multi-layer perception
Primary area: optimization_for_deep_networks
TL;DR: In this paper, we present ScaleKD, showing that pre-trained ViT models could be used as teachers preserving scalable properties to advance cross-architecture knowledge distillation research.

Abstract: In this paper, we question if well pre-trained vision transformer (ViT) models could be used as teachers that exhibit scalable properties to advance cross architecture knowledge distillation research, in the context of adopting mainstream large-scale visual recognition datasets for evaluation. To make this possible, our analysis underlines the importance of seeking effective strategies to align (1) feature computing paradigm differences, (2) model scale differences, and (3) knowledge density differences. By combining three closely coupled components namely *cross attention projector*, *dual-view feature mimicking* and *teacher parameter perception* tailored to address the alignment problems stated above, we present a simple and effective knowledge distillation method, called *ScaleKD*. Our method can train student backbones that span across a variety of convolutional neural network (CNN), multi-layer perceptron (MLP), and ViT architectures on image classification datasets, achieving state-of-the-art knowledge distillation performance. For instance, taking a well pre-trained Swin-L as the teacher model, our method gets 75.15\%|82.03\%|84.16\%|78.63\%|81.96\%|83.93\%|83.80\%|85.53\%  top-1 accuracies for MobileNet-V1|ResNet-50|ConvNeXt-T|Mixer-S/16|Mixer-B/16|ViT-S/16|Swin-T|ViT-B/16 models trained on ImageNet-1K dataset from scratch, showing 3.05\%|3.39\%|2.02\%|4.61\%|5.52\%|4.03\%|2.62\%|3.73\% absolute gains to the individually trained counterparts. Intriguingly, when scaling up the size of teacher models or their pre-training datasets, our method showcases the desired scalable properties, bringing increasingly larger gains to student models. We also empirically show that the student backbones trained by our method transfer well on downstream MS-COCO and ADE20K datasets. More importantly, our method could be used as a more efficient alternative to the time-intensive pre-training paradigm for any target student model on large-scale datasets if a strong pre-trained ViT is available, reducing the amount of viewed training samples up to 195$\times$. The code is available at *https://github.com/deep-optimization/ScaleKD*.

ScaleKD: Strong Vision Transformers Could Be Excellent Teachers
Jiawei Fan"
Chao Li* Intel Labs China chao3 www.inneel...... com
Intel Labs China j jiawei. fan@intel.com com
Xiaolong Liu"
Anbang Yao* Intel Labs China anbang bobaangyaolle yaannngaa@@inte...mm com
iMotion Automotive Technology xiaolong- Liu@imotion.al
Abstract
In this paper, we question if well pre-trained vision transformer (ViT) models could be used as teachers that exhibit scalable properties to advance cross archi- tecture knowledge distillation research, in the context of adopting mainstream large-scale visual recognition datasets for evaluation. To make this possible, our analysis underlines the importance of seeking effective strategies align (1) feature computing paradigm differences, (2) model scale differences, and (3) knowledge density differences. By combining three closely coupled components namely cross attention projector, dual-view feature mimicking and teacher pa- rameter perception tailored to address the alignment problems stated above, we present simple and effective knowledge distillation method, called ScaleKD. Our method can train student backbones that span across variety convolutional neural network (CNN), multi-layer perceptron (MLP), and ViT architectures on image classification datasets, achieving statc-of-the-aee knowledge distillation per- formance. For instance, taking well pre-trained Swin-L the teacher model, our method gets 75.15% % 82.03 03 % 16% 78.63% 88.96% [83 93% 83.80% 885 53% top-1 accuracies for MobileNet-VII Res Nes V/T6 ViT- models trained on ImageNet- 1K dataset from scratch, showing 3.05% 33.9% ..2%|[4.6%%% 5.52%
Cww.eeeeeOOoCooNeeeeee Convoeet-TTTT Mixer-t66MM
22.62% (3.3%% absolute gains to the individually trained counterparts. Intriguingly, when scaling up the size of teacher models or their pre-training datasets, our method showcases the desired scalable properties, bringing increasingly larger gains to student models. We also empirically show that the student backbones trained by our method transfer well on downstream MS-COCO and ADE20K datasets. More importantly, our method could be used as more efficient alternative to the time-intensive pre-training paradigm for any target student model on large-scale datasets if a strong pre-trained ViT is available, reducing the amount of viewed training samples up to x95x. The code is available at phttps://et // hhtp//btttttttt///////// com/ wwwpppppt.iizzaiionnSee...ee
Introduction
Background. The great success of deep learning in computer vision (CV) has been driven by an explosion of neural network architectures among which convolutional neural networks (CNNs) 3 vision transformers (ViTs) 4 5 and multi-layer perceptrons (MLPs) 6 g are three major model categories. While CNNs were the de facto models for about decade, recent progress shows that large ViT models have attained state-of-the-art performance many visual recognition tasks such as image
Core authors contributed to method formulation, experimental design and analysis. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

classification, image segmentation, and object detection. In principle, ViTs extend the philosophy of predominant transformer architectures ( in natural language processing (NLP) to vision tasks. They convert an image into a sequence of equal-sized patches treated as tokens resembling words in NLP, then apply the dot-product self-attention mechanism over the sequence of image patches. ViTs designed in this way couple with a powerful data-hungry learning paradigm: models are first pre-trained on massive datasets (with supervised or self-supervised [10 or cross-modality learning 12 [ and then fine-tuned on target datasets (with supervised learning). As the size of ViT models or pre-training datasets increases, the pre-trained models tend to have improved generalization performance. Despite this notable model performance scalability, the pre-training process of ViTs leads to significantly huge expenses. Furthermore, large pre- vre-trained ViTs are nemory-hungry and computationally intensive, prohibiting their deployment in many eeoource-constrinee application scenarios. In contrast, CNNs and MI.ss are still widely used in industry, due to the wider availability of effective implementations and optimizations compared ViTs.
(KD) has proven to be promising model compression pathway and has attracted lots of research interests. It relies teacher-student framework that transfers the knowledge learned by a large teacher model to a compact student model, aiming to make the student model can have improved performance to substitute the teacher model in deployment. However, most existing KD methods [4 focus on CNN architectures, and usually perform evaluation on small datasets with non mainstream student models for industrial applications, lagging far behind the evolution of neural network architectures. Although there have been few recent efforts (36 on using ViT teachers, they explore narrow focuses that use small ViT teachers without pre-training on massive datasets, following the ways previously studied in CNN-based KD methods. In this paper, we attempt connect knowledge distillation research with well pre-trained ViT models that stand out for their remarkable scalability, via a new viewpoint. Specifically, we question whether well pre-trained ViT models could be used as teachers that effectively transfer their scalable properties to target student models having different typed architectures such as CNN and MLP or heterogeneous VIT structures (we refer "cross architecture KD' to such a more generalized formulation in this work), in the context of using mainstream large-scale visual recognition benchmarks.
Motivation of This Work. In parallel, knowledge distillation
Problem Analysis. To answer the question in our motivation, we think the knowledge transfer difficulties are rooted in the following three aspects of differences: (1) Dijferences in feature com- puting paradigm. In terms of semantic units, ViTs operate patches added with positional embeddings, whereas CNNs operate on regular grids of pixels. In terms of core operations, ViTs rely on self-attention operations to model global feature dependencies, whereas CNNs rely on convolution operations to model local features, Although MLPs also use a patchify stem as ViTs, they rely on fully connected operations instead of self-attention operations and do not use positional embeddings, showing inferior feature learning ability. These differences in feature computing paradigm pose the first knowledge transfer barrier to overeome. in model scale. On the micro scale, model scale differences among ViTs, CNNs, and MLPs lie in network width, network depth. building blocks, etc. On the macro scale, model scale differences come from the capability of scaling the model size for ViTs, CNNs and MLPs towards better per- formance and generalization ability. As a result, these differences in model scale make the capacity of different network architectures typically vary significantly, emerging as the second knowledge transfer barrier to address. (3) Differences knowledge density. Under the prevalent pre-training and fine-tuning paradigm, when scaling up pre-training datasets, large ViTs usually exhibit obviously superior performance scalability than top-performing CNNs and MLPs in terms of fine-tuning on both upstream image classification tasks and downstream dense prediction tasks 44 41 As for knowledge distillation in this work, we assume that pre-training datasets are no longer accessible and only well pre-trained ViT teacher models are available, avoiding the expensive pre-training process and making the setting well suited for real applications. Under this context, when training student models on upstream image classification datasets like ImageNet- IK, the knowledge density between teacher and student models is different, which appears as the third barrier to handle. From the above analysis, we can conclude that the design of effective schemes to align (1) feature computing paradigm differences, (2) model scale differences, and (3) knowledge density differences between the pre-trained ViT teacher and target student models, plays the key role attain our goal.
sequence of equal-sized image
(2) Differences
Design Insights and Contributions. Accordingly, we present Scalable Knowledge Distillation (ScaleKD), simple and effective cross architecture KD method, which addresses the above difficul- ties in a progressive manner. Fundamentally, to bridge the feature computing paradigm differences
2

---

Soalnn Foatire
Transtee Guerie
Teactar Aueuare aridee oOoOo
ASE
O
LoSs
Companting paraaigon Algamest
Altensttive Featuve Mirosinn
Paramcter-pprc
Alganser
Direr Componere
ALA Deun o
O
Projected Teatare
WF-la Faires
DCT
J0C7
ME
Teacher Stape
MST
Lose
(A) Cross Ulention Projector (CAP)
Dual-viex Feature Mimicking (DFM)
(c) ToTTcherr Parameter Perception (TPP)
Figure 1: Overview of three core components in our ScaleKD, which are (a) cross attention projector, (b) dual-view feature mimicking, and (c) teacher parameter perception. Note that the teacher model is frozen in the distillation process and there is no modification to the student's model at inference.
between ViT and the other heterogeneous architectures, we propose cross attention projector (CAP, shown in (igure[Taa). motivated by some previous works 44 that utilize cross attention mecha- nisms to align different modalities. For semantic unit differences, CAP utilizes positional embeddings and patchify stem to transform the semantic units of CNN and MLP into transformer-lik tokens. To further bridge core operation differences, CAP employs cross-attention operation and trainable queries that share the same attributes as the teacher's features to model global interdependencies on the student's features. In this way, CAP could align computing paradigm differences between the ViT teacher and the heterogeneous student in form, serving as the base component in our method.
Different from feature computing paradigm differences, model scale differences and knowledge density differences are not explicitly and separately modeled in the KD proces,, as they are intertwined under the prevailing pre-training and fine-tuning paradigm and are finally encoded in teacher and student models' feature space and parameter space. In light of this, we investigate both feature and parameter spaces of teacher and student models and observe two critical phenomena:
As shown in Figure 2 and Figure 5 the frequency distributions of the features for the pre-trained ViTs are extremely imbalanced, where the direct component (zero frequency) response is dominant among all frequencies. This indicates that conducting feature distillation under such an imbalanced distribution may neglect the features of all other alternative components.
Feature Space:
Parameter Space: As the parameters of the pre-trained ViTs in the fine-tuning stage are slightly changed, their pre-training knowledge remains in the parameter space. Although the pre-training datasets are not accessible in this work, the student still has the opportunity obtain the pre-training knowledge by aligning its parameter space to the teacher's.
Inspired by these two insightful observations, we formulate our method from two new perspectives. Based on the observation in feature space, we design dual-view feature mimicking (DFM, shown in Figure lb)). whose key insight is to complement the neglected alternative features in the KD process. Specifically, DFM employs CAP as the feature projector and incorporates two feature mimicking paths. In the first path, DFM conducts feature mimicking in the original space to learn the teacher's global features. In the second path, by removing the direct component in frequency space, DFM highlights the subtle alternative responses in feature mimicking, thus avoiding the neglect of these features. As a result, the two paths are complementary to each other, jointly promoting the feature space alignment. Based on the observation in parameter space, we propose teacher parameter perception (TPP, shown in Figure [)), whose target to transfer the pre-training knowledge by establishing connection between teacher's and student's parameter spaces. Thanks to the aligned feature computing paradigm by CAP, TPP could bridge the student's 's early stages to the teacher's 's later stages and form a proxy feature processing path, where their parameter spaces join hands for KD optimization. By applying feature distillation in this path, the student's ' parameter space tends to be gradually aligned with the teacher's, and the pre- pre-training knowledge would be transferred from the teacher to the student. Since the distillation learning processes in feature space and parameter space are the two sides of the same coin, DFM and TPP could naturally reinforce each other in essence.
Benefited from the progressive designs, CAP, DFM, and TPP can be seamlessly integrated into a neat and effective cross architecture knowledge distillation method, called ScaleKD, which addresses the above three problems as a whole. Although ScaleKD has multiple feature mimicking paths, they only exist in the training stage. That is, ScaleKD does not alter the student's 's structure and introduces no additional cost in the inference stage. By conducting systematic experiments on several mainstream large-scale vision benchmarks, we validate the effectiveness and generalization ability of our method.
3

---

2 Method
Given a pre-trained VIT teacher having m stages and a target student (CNN or MLP or ViT) having stages, let F and Ft; denote features from i-th stage of the student and j-th of the teacher, respectively. In what follows, we formulate all three com- ponents of ScaleKD in the form of performing feature distillation, for better clarifying their tightly coupled relationships.
2000
2.1 Three Core Components in ScaleKD
Cross Attention Projector. As shown Figure Ta), a), CAP adopts the structure of standard transformer decoder block, consisting of a transformer decoder layer and an MLP layer. but incorpo- rates three critical modifications. For brevity, taking CNN example, our modifications include: i) patchifying regular grids of pixels in CNN; ii) adding positional embeddings; iii) setting queries in the transformer decoder block as trainable variables that share the same resolution with the teacher's features. The first two modifications intend to narrow the discrepancy between different semantic units of the pre-trained ViT teacher and the CNN student, while the last modifica- tion endows the employed transformer decoder block with great flexibility to align feature semantics and spatial resolution. For MLP and VIT students, we adopt the same CAP structure to CNN students for simple implementation but they can adapt CAP with fewer modifications when necessary. Based on these modifications. the cross- crossaattention operation further models global dependencies on the projected student features. With CAP, the feature distillation loss is defined as:
16
Figure 2: Feature distribution of BFiT-L/11 [41] the frequency domain, where the direct compo- nent response dominant. De- tails drawing this figure shown in Figure
CeAP=aL(e oL(FF, = =O F fF(F":9)15
where fpr q, a( 0), and L(-) denote the CAP, the trainable queries, the loss weight, and the L2 L2-normed distance, respectively.
Dual-view Feature Mimicking. As shown in Figure [b), building upon CAP, DFM contains two feature mimicking paths. As we stated in Section I the first path aims to learn the teacher's global features and the second path aims excite and mimic the alternative features (neglected by existing KD methods). Specifically, in the first path, DFM conducts feature mimicking in the teacher's original feature space, which formulated as: Lort aL(F", fp. (F* 9q)), where fps and qı denote the CAP and its trainable queries in the first path, respectively. In the second path, the dominant direct component should be removed. To achieve this goal, we first employ discrete cosine transform (DCT), which maps the features from the spatial domain to the frequency domain: DCT X + Z. We then define an operator that removes direct component response from the features:
{
0, 0
d(x) = (C
s.t. o(2) =
(2)
2, +
Next, feature mimicking in the second path is formulated as: Lan = aL(((F") ((f 92))) where fp, and 4/ denote the CAP and its trainable queries in the second path, respectively. Now, the feature distillation loss of DFM is formulated
LDFM -Looi + (1 Battt
where B E [0, 1] denotes the balancing weight.
Teacher Parameter Perception. As we stated Section 1 TPP establishes a proxy feature processing path by connecting the student's early stages to the teacher's later stages through CAP. In our implementation, the proxy path consists of the student's first n-1 stages and the teacher's last stage, illustrated in Figure (c). By feature mimicking in this proxy path, the parameters of the

---

student part are gradually aligned with the parameters of the teacher part, thus enabling the transfer of the teacher's pre-training knowledge. Let Fst 9t... (fp' (F* ; q)) be the output features of the proxy path, where 9t... and fg denote the teacher's s last stage and the CAP in this path, respectively. The feature mimicking in the proxy path is formulated as: _ = aL F We further introduce Fst input-dependent queries for the CAP in the original path. This feature mimicking design aims to enhance the capability of CAP as such queries contain more teacher-related information, and its corresponding loss is formulated as L aT. FL, J (F5n; Ft()). With simple principle of equal treatment to the two feature mimicking paths, the feature distillation loss of TPP is defined as:
TPP = L Lst
(4)
2.2 Overall Formulation
From a general perspective, the progressive designs of our above three components are naturally coupled. As CAP serves as the basic component in DFM and TPP, we further introduce how to apply DFM in TPP and get neat formulation of our method, ScaleKD. Specifically, if treating DFM as an improved version of traditional feature mimicking, can substitute the original feature mimicking in each path of TPP. In this way, we formulate the overall design of ScaleKD, whose loss is defined as:
LSeateK D = Lrask Bd + (1 (1-B)dL
3) + B + (1 (1- (1-3) 3) + Lidd;
(5)
DFM for TPP Student Path
DFM for TPP Teacher Path
where E [0, 1] is the balancing weight, Leesk is the cross-entropy loss, and kd is the vanilla logits- based KD loss [14] widely used in previous KD research. As the features are standardized, we set a 1 for loss terms in DFM as the default. Hence, our method has only one hyper-pane arpeerprramtter
3 Main Experiments
We perform comprehensive experiments systematically validate the efficacy of our method and answer the question in our motivation. Specifically, our experimental verification contains six parts: validating the effectiveness of our method under basic settings; conducting main experiments on ImageNet- 1K (45) (IN-1K) dataset with various student backbones and showing the promising performance gains of our method against individually trained counterparts; iii) verifying whether our method could transfer the scalable properties of the teacher to the target student; iv) conducting transfer learning on downstream tasks with MS-COC AS-COCO (46) and ADE20K 47) datasets examine whether the performance gains from our method could be well preserved; v) comparing our method with recent top KD methods; vi) showing the potential impact of our method on model engineering.
Unless otherwise stated, in experiments, the student backbones are trained on IN-IK from scratch, without the pre-training other upstream datasets. Experimental details are in Appendix A and
3.1 Pilot Experiments under Basic Settings
As we mentioned in Section ScaleKD is tailored for: i) transferring the pre-trained ViT teacher's knowledge to the student having different model architectures; ii) making the student inherit the teacher's scalability. Therefore, we first perform the following two pilot experiments.
Cross Architecture Knowledge Distillation. To illustrate the difficulty of cross architecture feature distillation and validate the efficacy of ScaleKD under this setting, we compare ScaleKD with traditional feature distillation (FD) [15] on two different cross architecture teacher-student network pairs. From the results shown in Table I we can observe: i) due to architecture gaps between the teacher and the student, traditional FD shows limited performance gains; ii) comparatively, our ScaleKD achieves significantly better performance, bringing 2.75%|3.22%%% absolute top-1 gain for ResNNet-50MMieess With the above experiments, we preliminarily verify that ScaleKD could effectively handle cross architecture feature distillation, which difficult for traditional FD.
Large Pre-trained ViTs as Teachers. With ResNet-50 as the student, we examine the rationality of selecting large pre-trained ViTs as teachers ScaleKD. Specifically, we gradually scale up the teacher's 's model capability (first from Swin-S to Swin-B, and then to Swin-L) and perform experiments
5

---

Table 1: Pilot experiments on cross architec- ture distillation with ScaleKD and FD. de- notes the distillation is conducted on stage-i. To clearly show the performance gain, experiments this table are conducted without Lkd-
Table 2: Pilot experiments on scaling up the teacher size. The advanced training strategy sophisticated data augmentation and optimizer, and longer training epochs, as shown in Table I0
more
Teacher
Stalest
Raño TSS Param
Top-Ie9)
ATi--199
Teacher
Student
Method
(9p-
A ((pp)11(9)
Traiting Strangg
Re
Baseline
(94)
77.43
96.55
Swis B (85,
Swis 5 (53,02)
Marviler
RosNen
3.43 x
1.40
79.50
76.55
79.02
-3077
-3.25
(5g.54)
97.74
+1.19
Swis- (8624)
268 x
$0.10
-3.53
Swin
SealeKD
79.30
+2.75
SoaleRD rith Adbanced Training Stratey
Rosdine
54,00
Mixa-S
TD (64)
(sguk))
54.88
75.52
+0.85
Marcilee Swis ($3,02)
Swis (85,
50
3.43
194
78.64
$1.77
$1.43
-3.13
-2.79
ScaleKD
77.24
+3.22
Swind. (96.4)
768
$2.00
-3.90
Table 3: Main results of ScaleKD) on 11 teacher-student network pairs. t denotes the model pre-
trained on IN-22K [45] and I denotes the model pre-trained by EVA [41] which has the learned knowledge of LAION-2B 48
(M)
FLOPs
Accuracy (%)
Teacher
Student
Top-1
A Top-1
MohileNet.V V1 (72.10)
4.23
0.58
75.15
+3.05
ResNet-50 (78.64)
25.56
4.12
82.03
+3.39
Com:NeXi (82.14)
28.59
4.46
84.16
+2.02
Mixer-S/16 (74.02)
18.53
3.78
78.63
+4.61
Swin-T. (86.24)
196.53
Mixer B/16 (76.44)
59.88
12.61
81,95
+5.52
VIT-S/16 (79.90)
22.05
4.61
83,93
+4.03
Swin.T (81.18)
196.53
34,04
VIT-B/16 (81.80)
86.37
17.58
85.53
+3.73
ResNet-50 (78.64)
25.56
4.12
82.34
+3.70
BEIT-L/14 (88.58)
Mixer-B/14 (76.62)
304.14
59.88
81.06
16.45
82.89
+6.27
VIT-B/14 (82.02)
using two popular training strategies. From the results shown in Table we can conclude: i) ScaleKD can help the student inherit the scalability the teacher: it can seen that the performance gain consistently increased under both training strategies when scaling up the teacher's model size; ii) ScaleKD can adapt to teachers' training strategies: it can be seen that our ScaleKD always brings promising performance gains, although the baseline model under the advanced training strategy gets much more competitive performance than that under the traditional training strategy.
According to the above pilot experiments, our ScaleKD shows basic capabilitiess on handling cross architecture knowledge distillation from large pre-trained ViTs to CNN and MLP students.
3.2
Main Results
After verifying the effectiveness of our method under our basic settings, we move forward and perform extensive experiments on more teacher-student network pairs, in order to broadly examine the scalability of our method. Specifically, construct 11 teacher-student network pairs by choosing 2 large teachers and 10 popular models for students, covering the current mainstream architectures across VIT, MLP, and CNN.
From the results shown in Table 3 we can observe: in general, our ScaleKD shows great generaliza- tion ability matter for CNN, MLP and ViT students. Over 11 teacher-student pairs, the mean top-1 accuracy improvement reaches 3.94%, and the maximum is 6.27%; ii) considering the acceleration, with Swin-L as the teacher, ResNet-50 Mixer-S/16 ViT-S/16 trained by ScaleKD even outperforms individually trained ResNet- 152 M5leee--522MMii---Heee Mixer-B/16 Miixer-B/16//TT-B// VTT- by margin of 0.28% 22. 19% p2. 13%, achiev- ing over x.35x 3.23 /3883 compression in terms of model size; iii) the top-1 performance gain

---

Table Experiments on exploring scalable properties from the teacher's pre-training data. We use the best reported models with different pre- wre-traiing methods as our baselines to examine whether our student model has learned the teacher's 's pre-training knowledge. We use Swin-L. as the teacher for the first two experiments and BEIT-L/14 1 the teacher for the rest two experiments. = denotes transfer learning and * denotes the model is trained and tested with 384x 384 sample resolution.
Model
Method
Training Darases
Daraset Samples x Epochs (M)
Viewed Samples (M)
1(p-1(88)
Superrised pre training
IN-22K = IN-IK
13.7% 707X90-22 1.28x32
1274
83.97
Pre-training f
VIT-B/16
JFT-300M = IN-IK
300 + 1.2x32
2141
84.15
SealeKD
IN-IK
1,28 300
384
85.53
Self-supervvied pre-truining
HEIT
AU
IN-22K + IN-IK
13.7x100 + 1.28x
2183
83.20
VIT-R/16
iOT
IN/22K = IN-IK
13.7x320 + 1.28x100
4512
84.40
ScaleKD
IN-IK
1.28 300
384
85.64
Covs-moodal pre-troining
LAION-2B IN-IK
x320 + x.28xx50
74304
85.47
VIT-B/16
CLIP
[
LAION-28 IN-12K IN-IK
2320x + x2 x.28x50
75030
86.17
VIT-B/14
ScaleKD
IN-IK
1.28 300
384
86.43
Aybrid pre-training distillation crov--oodal pre-trained leacker)
EVA-02
IN-22K = IN-IK
13.7 x.28
3352
85,80
EVA02-S/14>
ScaleKD
IN-IK
1.28
300
384
86.22
Table 5: Transfer learning results (%) on MS-COCO
Framowork
Rockhone
Pro-training
Classification (IN-KK)
Object Delection (COCO)
Instunce Segmentation (COCO)
AP
APS
AP'S
AFM
KesNer-30
Boseline
78.64
(+3.39)
42.3
40.2
23.0
44.3
54.6
52.5
37.1
18.0
40.1
54.9
Misk R-CNN
Swin-T
Boseline
83.80
81,18
44.4
42.7
26.5
45.9
56.6
58.6
39.3
20.5
41.8
57.8
could be increased further, when choosing stronger teachers. For instance, ScaleKD brings 0.32% additional gain ResNet-50 when changing the pre-trained ViT teacher from Swin-L. to BEiT-L/14.
3.3 The Scalable Properties from Teacher's Pre-training Data
As we introduced in Section I. the ViT's performance scalability is related two factors: model scale and pre-training data scale. The experiments in Section 3.1 and 3.2 have validated that our method could help the student inherit the positive performance effect from increasing the teacher's model scale. In this subsection, we focus on exploring the second factor: whether or not our method could help the student learn the teacher's pre-training knowledge from massive pre-training datasets, mitigating the knowledge density gap. To examine this, we alter our baselines to models with pre-training and propose an evaluation principle: given that only IN-1K visible, if ScaleKD can help the student model achieve similar performance answer the above question Yes. With this principle, we design series experiments based on the selected teachers in Table Swin-I. having the pre-training knowledge of IN-22K and BEiT-L/14 having the pre-training knowledge of LAION-2B. We compare the performance of the student models trained by ScaleKD and the corresponding counterparts trained by prevailing pre- pre-training methods.
models with upstream pre-training, the
From Table we can observe that ScaleKD performs better than various pre-training methods across all four kinds. Note that the superior performance of ScaleKD is achieved conditioned on not viewing any pre-traiee pre-training data. In other words, ScaleKD merely views 5.58 11.75x, 195.39x, and 8.73x less samples than the counterpart methods based on supervised pre-training, self-supervised pre-training, cross-modal pre-training, and hybrid pre-training. Therefore, we can summarize two promising conclusions: i) our method could help the student learn the teacher's pre-training knowledge from massive datasets and mitigate the knowledge density gap; ii) if well pre-trained large ViT available, method can be a more efficient alternative to the iime-intensive pre-training.
7

---

Table Performance comparison with recent top- performing KD methods. Following the settings of them, the students are trained under the advanced training strategy. Best results are bolded.
Table 8: Performance comparison with model engineering methods. More comparisons are shown in Table 14 in the Appendix.
Model
Params (M)
FLOPs (G)
Top- (86)
CNV-based Archniecture
Model
Method
Teacher
Fjocis
Top-1 (58)
RorNet-50
22.56
4.12
78.64
From Seraich
300
$1.18
ResNet-50 ScaleKD
22.56
4.12
82,55
DIST 5X
Swin-L (86.30)
300
$2.30
Con-Nex-T
28,59
4.46
82.
Swin-T
DirKD
Swin-I.
300
Rep g
22,90
82.50
ScaleKD
MLP-bussd Architecture
Misar-BIP B
59.
12.61
76.44
From Sennch
300
78.60
Miser B/16 ScaleKD
59.88
12.61
81.96
DIST
Swin (86.30)
450
80.20
DiEKD
Swin (86.30)
450
80.50
ResMLP- P.
gMLF-B g
73,00
115.7
23.00
15.80
81.00
VIT-B (86.53)
300
$1.33
ResNet-50
W---hased Architerture
ScaleKD
300
WISS 4 VIES/16 ScaleKD
22,05
4.61
79.90
FunMaich
53
BiT-Res 52s2 (NIA)
1200
81.54
22,05
4.61
83.93
FanMatch
BiT-ResI5252 (N/A)
9600
82.31
Swin-T Swin
M
73,00
15
81.18
ScaleKT)
Swin-1, (86.24)
600
82.55
3.4 Transferring to Downstream Tasks
Table Transfer learning results (%) on ADE20K.
To further examine whether the performance gains from method could be well pre- served in transfer learning, we conduct com- parative experiments MS-COCO for ob- ject detection and instance segmentation, and on ADE20K for semantic segmentation.
Framework
Rockhone
Pre-tranning
(Top-1)
ADEZ0K (mf(0))
ResNer.50
Our
Baseline
78.64 $2.03 (+3.39)
(+2.13)))
42.37
UpenNeet
Swin-T
Boseline
(2.662)
(+1.92)
VIT-B/16
Boxeline
85.53 (+3.73)
81,30
50,84 (+4.09)
46.75
The results on MS-COCO and ADE20K are shown in Table and Table respectively, from which we can observe: i) overall, our pre-trained models outperform their baselines by significant margins across three downstream tasks and di different architectures; ii) for semantic segmentation on ADF20K, VIT-B/16 achieves the highest 4.09% absolute performance gain across three backbones, even higher than its gain on IN-IK; iii) for object detection and instance segmentation on MS-COCO, ResNet- S0|Swin-T pre-trained by ScaleKD outperforms its baseline by AP margin of 2. [% 1.7% and 2.0% 1.5%, respectively. The above observations illustrate that the performance gains from ScaleKD could be well transferred to various and challenging downstream tasks.
3.5 Comparison with Recent Top-Performing KD Methods
As we stated in Section D and 2 ScaleKD is unified design incorporating three novel focuses to align computing paradigm differences, model scale differences, and knowledge density differences, which are clearly different from existing KD methods. In order to validate the superiority of our method, we compare ScaleKD with recent top-toopoooiee sop-performing KD methods.
From the results shown in Table 7 we can see: i) compared to DIST, DifIKD and OFA, although our teacher not the best and number of training epochs is the smallest, our ScaleKD still outperforms the best of these methods by clear margins [0.70% on ResNet-50 Swin-T): ii) compared FunMatch, our method even shows superior performance, outperforming FunMatch by margin of 0.24% but only using less than 10% training epochs. As result, in the context of transferring the scalability the pre-trained ViT various student models, our systematic design and its focuses show obvious superiority to previous works, paving new path for future KD research.
3.6 Potential Impact on Model Engineering
we have noticed ScaleKD brings significant performance gains to target students, especially for the plain design each model category, such as ResNet, MLP-Mixer, and ViT. In parallel, model engineering is common solution to improve the model performance. Considering these two facts, we conjure that since our method could bring competitive performance gain compared to model engineering, larger flexibility would be provided when choosing models in practice.
In Section 3.2

---

Table Ablation studies. Experiments (b)-(d) are performed on -een----ReNNe--500 As DFM and TPP are designed based on CAP, CAP is added by default when choosing DFM and TPP in (a). Because of this, we treat CAP as another baseline method, when analyzing DFM and TPP in (c)-(d).
(a) Ablation on the overall design
(b) Ablation CAP
(c) Ablation DFM
Design
Projector
Top-1
Top-1
Festture Minicking
Top-1
ATop-
Tracher
Student
Top
Basrlow
76.55
Baseline
76.35
76.55
Linear
77.45
+0.88
CAP
77.87
+1.32
77.87
27.52
40.97
78.12
ResNet-30
78.51
CAP
77.87
+1.32
DFM
78.51
+1.96
78.62
79.30
(d) Ablation TPP
79.62
Swin-S
TPP Design
Accurccy (%)
74,02
Mothod
Path
Aduptive Queries
Top-1
ATop-1
75,03
MixesS
76.28
76.42
CAP
Boseline
76.55
77.87
+1.32
77.24
TPP
78,50
+1.95
77.59
TPP
78.62
+2.07
To study it, we apply ScaleKD to 3 standard designs of CNN, MLP and ViT, and compare the performance with recent advanced designs. From the results shown in Table we observe that our method could help these models reach better performance than advanced models. More interestingly. in Table 3] we can clearly see that the performance gap between plain designs (ResNet-50) ViT- S/16) and advanced designs (ConvNeXt-TV SwinnT)) no longer exists after applying ScaleKD.. These phenomena indicate ScaleKD could have potential impact on the model selection real applications.
Ablation Study
4.1
Tightly Coupled Design Properties of Three Core Components
Recall that our ScaleKD consists of three core components, CAP, DFM and TPP, which are progres- sively designed in a tightly coupled manner. In Table 9a| we perform ablation study to testify their complementarity via comparing different component combinations. We can notice: i) when gradually applying more of three component designs, the performance of ResNet-50 and Mixer-S shows similar increasing trends, showing that each component of ScaleKD is not designed for specific student architecture; ii) although CAP brings the two students promising performance gains, DFM and TPP further brings ResNet-50 Mixerr---SMSMSS extra performance gains, 0.64% 0.75% and 1.25% 1.39% respectively, verifying that DFM and TPP are complementary to CAP; iii) when using DFM and TPP together, both ResNet-50 and Mixer-S obtain additional performance boosts, which indicates that DFM and TPP are also complementary with each other.
4.2 Role of Each of Three Core Components
CAP vs. Popular Feature Projectors. We first compare CAP with two popular feature projectors, denoted Linear and Conv, to ver-
224
ify the superiority of CAP. The former projector consists of a linear layer and the latter projector consists of two 3x3 convolutional layers. From the results shown in Table 9b we can notice that CAP outperforms the other two projectors clearly, which validates the key role of CAP: aligning computing paradigm differences towards better KD performance.
396
72
Importance Alternative Feature Mimicking in DFM. The key insight of DFM is to complement the neglected alternative features the feature mimicking process. Table 9c we compare DFM with CAP and dual-path CAP to illustrate that the alternative feature mimicking is essential. We find that although the dual-path feature mimicking brings 0.25% extra performance gain to CAP, removing
Figure 3: Feature distances of alternative components in the spatial domain. Details on the figure drawing in Figure

---

the direct component in the second path can further bring 0.39% improvement. This verifies the rationality of the design. To better understand why the performance gain from the learning of teacher's is alternative features, we make comparisons between the methods with DFM and without DFM. Specifically, we measure the distances between the student's features and the teacher's features of alternative components in the spatial domain. In Figure [] we can clearly see that DFM can effectively reduce alternative feature distance between the teacher and the student.
Role of Proxy Path in TPP. Note that for transferring the teacher's pre-training knowledge in the parameter space to the student, TPP establishes proxy path that connects the student's former stages to the teacher's ' later stages. In Table (d], we study the design of TPP and verify whether the proxy path and its adaptive queries are effective. The results show that the feature mimicking in the proxy path can provide the student with performance improvement and providing input-dependent queries can further enhance the effectiveness of TPP, which indicates that these designs in TPP are essential for learning the knowledge in the teacher's parameter space.
More ablation studies on the hyper-parameter and the others are provided in Appendix D
5 Related Work
Knowledge Distillation. Traditional KD methods [5 student network pairs with small model scale gaps. Some recent works [54 55 50 further study how to conduct knowledge distillation with larger teachers. As vision transformers suffer from low convergence speeds, some recent works (56-58) explore leveraging CNNs to accelerate the training of vision transformers. Meanwhile, [36438 38 [5] discuss how to bridge the architecture gap when the teacher and the student are in different model categories.
generally focus on CNN-based teacher-
Frequency-based Knowledge Distillation. As traditional feature distillation only focuses on pixel- to-pixel differences. FAM [60] defines knowlwedge distillation in terms of frequency- -based attention maps. FreeKD [6]] explores how to climinate unfavorable information in the frequency domain for enhancing the distillation performance on dense prediction tasks. Different from our ScaleKD,, they consider feature distillation on CNN-based network pairs and have different formulations.
Teacher Parameter Reuse. Some previous KD methods also leverage the teacher's parameter for reusing a better classifier [32] or initializing the student's 's neck and head (62H64] or dismissing the
shortcuts in residual architectures (65] Unlike our ScaleKD, the motivation of these works focuses on parameter reuse or equivalent substitution, rather than aligning two parameter spaces for transferring the teacher's s pre-training knowledge the target student without the pre-training process.
6 Conclusion
In this paper, we present ScaleKD,, new cross architecture KD approach for transferring the scalable properties of pre-trained large ViT' to various CNNS, MIPs and heterogeneous ViTs. Our method consists of three tightly coupled components that rely on principled designs to align computing paradigm diffcrences, model scale differences, and knowledge density differences between the teacher and the student. By conducting systematic experiments vision benchmarks, we broadly validate the effectiveness and generalization ability of our method. Benefiting from its novel motivation and design insights, ScaleKD is the first work which successfully verified that KD can be a more efficient alternative to the time-intensive pre-training, to the best of our knowledge. This extends the application scope of KD from model compression to training acceleration. We hope our work would inspire feature KD research in this new direction.
several mainstream large-scale
Restricted by our computational resources, we do not conduct experiments on very large teachers, such as VIT-22B (66), or on large students, such as VIT-L the increasing model scale of teachers, the training cost of ScaleKD increases, which is common limitation to KD research. According to the analysis in Appendix [ the extra training cost of ScaleKD is acceptable to large extent. Actually, thanks to its promising performance, ScaleKD shows the great potential to replace the time-intensive pre-training of students on large-scale datasets.
Limitations.
Furthermore, with
10

---

References
[1] Alex Krizhevsky, nya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[3] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition scale. In ICLR, 2020.
[5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[6] Nya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. In NeurlPS, 2021.
[7] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthicu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard. Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. TPAMI, 2022.
[8] Hanxiao Liu, Zihang Dai, David So, and Quoe V Le.. Pay attention to mips. In NeurlPS, 2021.
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[10] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[11] Jinghao Zhou, Chen Wei. Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In /CLR, 2021.
[12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021.
[13] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini 150:::50300255 2015.
[15] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[16] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2016.
[17] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Varia- tional information distillation for knowledge transfer. In CVPR, 2019.
[18] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV, 2019.
[19] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, 2019.
[20] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.

---

[21] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
[22] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019.
[23] Jangho Kim, SeongUk Park. and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NeurlPS, 2018.
[24] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[25] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[26] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019.
[27] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV, 2020.
[28] Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. In NeurIPS, 2018.
[29] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In AAAI, 2021.
[30] Jing Yang, Brais Marinez, Bulat Adrian, and Tzimiropoulos Georgios. Knowledge distillation via softmax regression representation learning. In ICLR, 2021.
[31] Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. AAAI, 2021.
[32] Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In CVPR, 2022.
[33] Xueqing Deng, Dawei Sun, Shawn Newsam, and Peng Wang. Distpro: Searching fast knowledge distillation process via meta optimization. In ECCV,
[34] Xiaolong Liu, Lujun Li, Chao Li, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR, 2023.
[35] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022.
(36) Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation. In NeurlPS, 2023.
[37] Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingting Ding, and Liang Li. Cross-architecture knowledge distillation. In ACCV, 2022.
[38] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint arXiv:2205.I 14141, 2022.
[39] Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Feature-based knowledge distillation for vision transformers. In CVPRW, 2024.
[40] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. In ICLR, 2021.
[41] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
12

---

[42] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- rext-to-image diffusion models. In ICCV, 2023.
[43] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In CVPR, 2023.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[45] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[46] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[47] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. I/CV, 2019.
[48] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
[49] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: visual representation for neon genesis. arXiv preprint arXiv:2303. 1331, 2023.
[50] Tao Huang, Shan You, Fei Wang. Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. In NeurlPS, 2022.
[51] Tao Huang. Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. In NeurlPS, 2023.
[52] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. In CVPR, 2022.
[53] Ao Wang, Hui Chen, Zijia Lin, Hengjun Pu, and Guiguang Ding, Repvit: Revisiting mobile cnn from vit perspective. arXiv preprint arXiv:230700928. 2023.
[54] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
[55] Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In ICCV, 2021.
[56) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-effee efficient image transformers & distillation through attention. In International conference on machine learning. 2021.
[57] Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: Data-efficient early knowledge distillation for vision transformers. In CVPR. 2022.
[58] Borui Zhao, Renjie Song, and Jiajun Liang. Cumulative spatial knowledge distillation for vision transformers. In ICCV, 2023.
[59] Jinjing Zhu, Yunhao Luo, Xu Zheng, Hao Wang, and Lin Wang. A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation. In ICCV, 2023.
[60] Cuong Pham, Van-Anh Nguyen, Trung Le, Dinh Phung, Gustavo Carneiro, and Thanh-Toan Do. Frequency attention for knowledge distillation. In WACV, 2024.
[61] Yuan Zhang, Tao Huang, Jiaming Liu, Tao Jiang, Kuan Cheng, and Shanghang Zhang. Freekd: Knowledge distillation via semantic frequency prompt. In CVPR, 2024.
13

---

[62] Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. Instance- conditional knowledge distillation for object detection. In NeurIPS, 2021.
[63] Zhendong Yang. Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In CVPR, 2022.
[64] Jiabao Wang. Yuming Chen, Zhaohui Zheng, Xiang Li, Ming-Ming Cheng, and Qibin Hou. Crosskd: Cross-head knowledge distillation for object detection. In CVPR, 2024.
[65] Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin, Wei Zhang, Jiashi Feng, and Tong Zhang. Residual distillation: Towards portable deep neural networks without shortcuts. In NeurIPS, 2020.
[66] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICMI., 2023.
[67]
MMPreTrain Contributors, Openmmlab's pre-training toolbox and benchmark. https://
github. com/ www.//ppeemmama///nnooeeeeeeeeee mmw.a//mmreeerrinn.nmmmmmmmmmaaa 2023.
[68] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
[69] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021.
[70] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. ECCV, 2022.
[71] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 1906.0715, 2019.
[72] MMSegmentation Contributors, MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https: thub. com/ http://penmmmaa/mmmmssee www.lll//mnsementtti.imm 2020.
[73] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[74] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
[75] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Tech Report, 2009.
[76] Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao, Lin Song, Xiangyu Yue, and Ying Shan. Unireplknet: universal perception large-kernel convnet for audio, video, point cloud, time-series and image recognition. arXiv preprint PrXiv:2311.15599 2023.
[77] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML. 2019.
[78] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In ECCV, 2022.
[79] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-tor to-tosn vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
[80] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In ECCV, 2022.
[81] Jiahao Wang, Wenqi Shao, Mengzhao Chen, Chengyue WU, Yong Liu, Kaipeng Zhang, Songyang Zhang, Kai Chen, and Ping Luo. Adapting llama decoder to vision transformer. arXiv preprint 2400:24004 2024.
14

---

Appendix for "ScaleKD: Strong Vision Transformers Could Be Excellent Teachers"
Jiawei Fan" Intel Labs China j jiawei. fww@intee.ccommmmmmm com
Chao Li' Intel Labs China chao3 www.iiineee..... com
Xiaolong Liu" iMotion Automotive Technology xiaolong- liu@imotion.at
Anbang Yao*t Intel Labs China anbang -oanng-yan yww.gntelllnn.....mmo com
A Datasets
ImageNet-IK [45] a well-known large-scale classification dataset, comprising over 1.2 million training images and 50,000 validation images with 1,000 object categories.
MS-COCO [6 is a large-scale dataset for object detection and instance segmentation, which contains 118,000 training images and 5,000 validation images with 80 object categories.
ADE20K 47 is challenging semantic segmentation dataset, containing 20,210 training samples, 2,000 validation samples, and 3,352 testing samples with 150 categories.
B Experimental Setups
B.1 Experimental Setups on IN-1K
Training Strategy. We conduct experiments with two popular training strategies: traditional training strategy and advanced training strategy. The traditional training strategy is commonly used in previous KD approaches (shown in Table [0ap and the advanced training strategy adopted in training recently proposed CNNs, MLPs, and VITs (shown in Table TOb).
Compute Infrastructure. The experiments using the traditional training strategy conducted on x NVIDIA Tesla- V100 GPUs, while the experiments using the advanced training strategy are conducted on 32 NVIDIA Tesla-V100 GPUs.
Implementation Codebase. We implement our method based on MMClassification
67
Hyper- Syper-Parameter Settings. The overall loss for our ScaleKD is defined as Equation 5 Thanks to the simplicity of this formulation, have only one hyper-parameter in our ScaleKD. From the ablation study in the Appendix D we find that the best choice is B = 0.6 and we use it as the default setting throughout all experiments.
Selection of Teacher-Student Network Pairs. Overall, we construct teacher-student network pairs, which consist of pre-trained large ViTs, and 10 students covering mainstream architectures of VIT, MLP, and CNN. Specifically, for the teacher, we choose two different types of well pre-trained ViTs: supervised pre-trained Swin-L. [5] with the hierarchical architecture and hybrid pre-trained BEiT-L/14 [40] with the typical VIT architecture. Moreover, compared to Swin-L, BEiT-L/14 is much larger in terms of model size and stronger in terms of model performance. For the student
Core authors contributed to method formulation, experimental design analysis. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 10: Detailed settings of traditional training strategy and advanced training strategy on IN-IK.
(a) Traditional training strategy
(b) Advanced training strategy
Configuration
CNN
Configuration
CNN CNN/MLP/VE /VT
Size
2048 1536
Learning Rate
0.1
Learingg Rate
50-3 / 7o-4 Io-3
Learning Rate Schodule
Stepwise Decay'Cosine Decay
Leaming Rate Schedule
Cosine Decay
Optimizer
SGD
Optimizer
Lamb AdamW Adam
Optimizer Prper-Parameters
momentume 0.9
Optimizer Hypar- Srpar-Parameters
81 .a.. = 0.9. 0.009, 1e-8
Weight Decay
le-4
Weight Decay
0.02 / 00007 10.05
Training Epochs
100
Training Epoahs
300 / 20/ 20
Warmup Epochs
x
Warmup Epochs
Drop Path
x
Drop Path
0.05 0.1 /.11
Label Smoothing
Tabel Smoothing
0.1
Random Flip
Random Hlip
Random Resize Crup
(0.08.1)
Random Resize Crop
(0.08.1)
Random Augmentation
Random Augmentation
(7,0.5) (9..5)) (9.0.5)
Random Erasing
x
Random Erasing
0.25
Table 11; Detailed settings of transfer learning strategies on MS-COCO and ADE20K.
(a) MS-COCY
(b) ADE20K
Configuration
NesNet-50 / Swin-S
Configuration
ResNet-50/ Swin-5 WTTB
Weight Initialization
Pre-trained Checkpoint
Weight Initialization
Pre-trained Checkpoint
Batch Size
16
Batch Size
16 1e-4 / lc-4 2e-4
Learning Rate
le-4
Learnnng Rate Decay
Learning Rate Decay
Stage (0.7)
LR decay
Stage (0.9)/ Stage (0.9) Layer (0.6)
Learning Rate Schodule
Cosine Decay Adamm
Leamingg Rate Schedule
Cosine Decay
Optimizer
Optimizer
Adam
Optimizer Hyper. Purameters
$1. 0.9, 0.009, le.8
Optimizer Hyper Parameters
Pr. 82, = 0.9, 0.009, le
Weight Decay
0.05
Weight Decay
0.05
Training Epochs
Training lterations
160000
Crop Size
(1333, 800)
Crop Size
(512, 512)
Drop Path
0.2
Drop Path
0.0 0,0/033/0.2
architectures, we first choose the basic design in each architecture type, such as ResNet-50 ( Mixer-S/16 (6, and V [4). Then, also select some popular models, such as MobileNet-Vi (68. Con 3 and Swin-S. Next, we expand the basic designs to larger ones, like Mixer-B/16, 6, Mixer-B/14, ViT-B/16 and ViT-B/14, After separately selecting teachers and students,
finally organize them into 11 teacher-student network pairs for comprehensive experiments. Note that the performance for most individual trained baselines in Table are sourced from their original papers, except for ResNet-50, MobileNet-VI ViT-B/14, and Mixer-B/14 B/14, which we trained ourselves using the advanced training strategy due the absence of reference results in their original papers.
Counterpart Pre-training Methods. In the main paper, we select state-of-the-an methods in each choose the pioneering work [4] For self-supervised pre-training, we choose BEiT 40 and iBoT [11]. For cross-modal pre-training respectively.
pre- pre-training paradigm for comparison. For supervised pre- pre-training,
and hybrid pre-training, we choose CLIP 113 and EVA-02 49]
Counterpart Knowledge Distillation Methods. In the main paper, we make comparisons with many recent KD methods, such DIST (50), DifKK [51], OFA (3) and FuncMatch (52). In this Appendix, we further compare with CNN-based methods, such as KD 14) AT [66. OFD 26 RKD
(20]. CRD 25 DKD 35 SRRL (30 ReviewKD 169]. DistPro
33 and MGD (70).
Counterpart Model Engineering Methods. In the main paper, to better show the great potential of our ScaleKD, we apply it to the popular designs of each architecture type and make comparisons with various advanced counterparts. Driven by this target, we mainly select the so-called next-generation models. For ResNer-50, we select ConvNeXi-T 3 and RepViT-2.3M (53) for comparison. The former one the typical design of the new-era CNN and the latter one a popular model for deployment. For Mixer-B, we select gMLP-B B 8 and ResMLP-PP B24 (7). which are optimized to
16

---

Table 12: Performance comparison (%) of ScaleKD and CLIP on ViT-B for linear probing.
Model
Method
Pre-trining Dataset
IN-JK (Training)
1 shot
CIFAR-100 incar Prohing)
shot
10 shot
25 shot
Full
Frum-scratch
IN-K
81,80
33.86
60.30
66.77
72.65
81.76
77.21
WIB8/16
LAION-20 28, IN-IK
85,49
41.10
69.00
72.34
78.54
85.51
CLIP
LAION-28 28, IN-I2K, IN-IK
$6.17
44,90
70.19
76.77
31.45
58.88
OpenAl, IN-12K, IN-KK
85.99
47.40
70.85
71.37
81.52
WITR/14
Our
IN-IK
86.43
48.14
70.91
77.52
81,50
89.11
Teacher: DEIT-L/14
EVA
CLIP OpenAl, IN-22K, IN-I
88.58
63.74
85.20
87.39
89.27
93.36
Table 13: Performance comparison on IN-IK with more CNN-based KD methods. In the experiment, we adopt the same traditional training strategy as these methods.
Model
Teacher
Method
Top-1(98
From Scratch
69.63
AT [
70.72
OFD
71.25
RKD A0
71.23
CRD
71.40
MobileNessVV
ResNet- (66.16)
DKD 35
72.05
SRRL 30
72.49
Review KD
09
72.56
DIST 50
73.24
DisiPro 83
73.26
MGD
73.35
DittKD 51
73.62
Swin-1. (86.24) 24)
ScaleKD)
74.21
suppress the weaknesses of MLP-Mixer. For ViT-S, we choose Swin-S 5 and Swin-B as counterparts validate whether our ScaleKD could outperform larger advanced designs.
B.2 Experimental Setups MS-COCO and ADE20K
Training Strategy and Hyper- Parameter Settings. For the experiments on MS-COCO, we adopt the settings shown in Table Tla while for experiments on ADE20K, we adopt the settings shown in Table
Compute Infrastructure. All experiments on MS-COCO and ADE20K are conducted on 8 x NVIDIA 1100 GPUs.
Implementation Codebase. We conduct experiments based on MMDetection 71 and MMSegmen-
tation 72
Selection of Task Frameworks and Backbones. For different task frameworks, we choose Mask
R-CNN (73) for object detection and instance segmentation, and UperNet [74] for semantic segmen- tation. As for backbones, we select ResNet-50, Swin-T, and ViT-B/16.
C More Experiments
Linear Probing CIFAR-100. We conduct of linear probing experiments CIFAR- 100 [75), based on models in T From the results shown Table 12] we can observe: i) models pre-trained by CLIP greatly improve the backbone's generalization ability across different datasets; ii) our ScaleKD helps the student model reach mostly better performance than CLIP-based pre-training, without viewing pre-trale pre-training data.
Performance Comparison with More KD Methods. the main paper, we compare ScaleKD with mostly related cross architecture KD approaches in Table 7 as few previous works use medium-sized students, such as ResNet-50, for benchmarking. To make more comprehensive comparison with lots of CNN-based KD methods, conduct experiments traditional student network, MobileNet-Vi
17

---

Table 14: Performance comparison on IN-IK with lots of model engineering methods. We conduct ScaleKD on the simplest design of each architecture type and then make a performance comparison with various designs. Ours', Ours and Ours denote choosing ViT-B (training from seratch), Swin-I. (with IN-22K pre-training) and BETT-L/14 (with EVA pre-training) as the teacher, respectively.
Training
(M)
FLOPs
Top ((%)
CNW-based Architecture
ConyNext-T
IN-IK
224
ConoNext-T Ours
IN-IK
224
28.59
4.46
84.16
ConvNext-T 8
IN-22 = IN-IK
224
28.59
4.46
82,90
ConyNext-B 8
IN-IK
224
87.77
15.14
83.80
UniRepLKNetT
IN-IK
224
31,00
4,90
83,20
EficieniNet BS
2
IN.IK
456
30,00
9.90
83,60
RopVIT-M2.3 53
IN-IK
224
22.90
83,70
ML.-based Archizectore
Mixer B/16 (
IN-IK
224
59.88
12.61
76.44
Mixer-B/16+ Ours*
IN-1K
224
$9.88
12.61
81.62
Mixer-Bie Ours
224
Mixer-Bi Ours
224
16.45
Mixer B/16
2N = IN-IK
224
59.88
12.61
80.64
Mixer-L/16
IN-22 = IN-1K
224
208.2
44.57
82.89
ResMLP-B24
IN-IK
224
115.7
23.0
81.00
gMLP-B
IN-IK
224
73.00
15.80
81.60
Transormer- Architecture
VTT-S/16 S
IN-IK
224
22.05
4.61
79.90
VT--S/16 Ours!
IN-IK
224
22.08
4.61
83,93
IN:22K
224
224
4.36
T2T PIT-VI,--F
79
IN-IK
224
21.47
4.34
81.83
DaVIT-T 30
IN-IK
224
28.36
4.54
82.24
iLLaMA-S 31
IN-IK
224
21.90
79.90
EVA-02-S/14
IN-IK
3362
EVA-02-S/14 Ours
IN-IK
3362
22.13
15.51
86.22
VT-B/116 4
IN-IK
224
86.57
17.58
81.80
Swin-B
IN-IK
224
87.77
15.14
83.50
T2T-VIT,-22
A0
IN-IK
224
12.69
82.71
DaVIT-B
IN-IK
224
15.51
VIT-R/16 4
IN-22K IN-IK
2242
86.57
17.58
83.97
Swin-B
IN-22K = IN-IK
224
87.77
15.14
85.20
iLLaMA 81
IN 22K = IN-IK
224
86.30
85.00
using the same training strategy as them. From the results shown in Table 13 we can observe that by using Swin-L. as the teacher, our ScaleKD could help MobileNet-VI reach 74.21 % top-1 accuracy, outperforming previous methods which use ResNet-50 as the teacher by clear margins.
Performance Comparison with More Model Engineering Methods. In the main paper, we apply ScaleKD to the basic design of each architecture type and make comparisons with more recent variant architectures. As illustrated in Table 14 we choose more designs have more comprehensive comparison.
D More Ablation Studies
Ablation Study on Training Efficiency of ScaleKD. As TPP our ScaleKD leverages the teacher's last stage, it will introduce additional training costs compared to traditional FD. To clearly study its training efficiency, we conduct ablative experiments in this section: as shown in Table 15a we first compare the training efficiency of ScaleKD with traditional FD on three network pairs having increased teacher's model scale; ii) then, as shown in Table 15b we conduct the experiments on each component in ScaleKD. The experimental results show that: using large teachers would
18

---

Table 15: Experiments on the training efficiency of ScaleKD. The student model in all experiments is ResNet-50. In (a), we compare ScaleKD with traditional FD using three teachers with different model scales. (b), we conduct the experiments based on Srin-S--ReNNet--e
teacher-student network pair to illustrate the training costs (memory and time) introduced by each component of ScaleKD. Experiments are conducted on 8 x NVIDIA Tesla- V100 GPUs.
(a) Training costs comparison with FD
(b) Training costs of each component in ScaleKD
Tradher
Mathod
Top-1
CPU Manary
Trrosn
Mathed
CAP
DEM
Designs
GPU Memory
Treato
Swin.S
ScaleKD
ED
77.43
79.62
6.65
3,66
2.10
1.77
Swin-B
ScaleKD
FD
7030
77.76
3.36
3,66
2.53
183
75.51
27.87
3,77
4.03
1.71
1.50
Soivin
78.63
5:13
1.81
Swin
79.62
78:10
2.30
7.08
Table 16: Ablation study on pre- pre-training and distillation.
Table 17: Ablation study on the necessity of alternative components in the first path of DFM.
Model
Method
Top-1(58)
Training scratch IN-IK
79.90
Training from scrotch IN- IK w/ KD
81.42
Methad
Top-1 (58)
1opp- (9)
VIT-3/16
Pre training on IN-22K
80.05
Baseline
CAP
76.55
Pre Vve-training IN-22K KD
82.00
(1ir Alt
+1.68
Trinning from scratch on IN-IK ScaleKI)
DFM (FM Allo
78.5L
+1.96
induce more GPU memory occupation and longer training time: ii) comparatively, TPP is the most resource-comee consuming component, especially after combining with DFM. summary, our ScaleKD introduces additional training costs compared to traditional FD. However, if considering the significant performance gain it brings, these additional costs are acceptable.
Ablation Study on Hyper-paramette B. According to Equation in the main paper, our method only has one hyper-parameter B. which is the balancing weight of two feature mimicking paths in DFM. We conduct the ablation study on -Res-----RNNNt-5ii network pair to study the impact of different settings of B. Specifi- cally, we select the 3 uniformly from to only the first feature mimicking path exists, while @ = O indicates that only the second feature mimicking path exists. As shown in Figure 4 we can observe: i) in general, ScaleKD outperforms the baseline by significant margins at all settings, validating the stability of ScaleKD; ii) when 3 = 0.6, our ScaleKD achieves the best performance; iii) though the second feature mimicking path could be used individually, it is inferior to the first path, indicating that the direct component is essential in feature mimicking; iv) when the two paths work collaboratively, they perform better than two individual counterparts, which suggests that the two designs complementary with each other.
ScalenD
1.0 indicates that
77.5
77.0
76.5
02
0.4
0.0
0.3
-b 4m
Figure 4: Ablation study on the hyper-parameter B3.
Ablation Study on Pre-training and Distillation. In this study, we explore the originality of the distillation performance gain. We compare models trained by ScaleKD with upstream pre-trained models and upstream pre-training models with KD. From the results shown Table [6] we can notice: i) compared to individual pre-training, applying KD under this stage can significantly boost model performance; ii) ViT-S/16 trained by ScaleKD significantly outperforms the models trained with KD on IN-22K. These two observations indicate that: small students are difficult to capture the pre-training knowledge with traditional FD, even with the upstream pre-training dataset; ii) our ScaleKD could effectively help the student to learn useful pre-training knowledge from the teacher without viewing the pre-training dataset.
Ablation Study on the First Feature Mimicking Path of DFM. We explore the necessity of alternative components the first feature mimicking path of DFM. Specifically, we remove all alternative components in the first path of DFM and perform experiment under the same settings as Table 9b). For the results shown in Table [7] DFM(Dir + Alt) indicates the above new setting, while DFM(A Alt) is the original design. We can observe that removing all alternative components in the
19

---

5030
1000
2300
2000
800
2020
500
1000
500
500
0240011 9.00
a feir-l
10 14
(a) VIT-L/14
(b) Swin-L
(c) BEiT-L/14
Figure 5: More illustrative feature distributions large pre-trained ViTs the frequency domain. We first collect the output feature maps of 1600 samples from IN-IK, then conduct DCT on each channel, and finally take the average value across these samples after converting all responses into absolute values.
11
11x
Figure 6: Feature distance distributions of alternative components for the last stage features between teacher and student on IN-IK. We obtain 64,000 feature pairs on from 64,000 samples. After calculating the distance between teacher and student, we project the high-dimension distances into two-dimension space for illustration. Finally, we randomly select 6,400 data points for times to draw the seatters. Blue points denote the distances without DFM, while orange points denote the distances with DFM.
network pair
first path will slightly decrease the effectiveness of DFM, but its performance is still obviously higher than CAP. As we discussed in Section the first path DFM aims to capture the teacher's global features, where the subtle alternative components are also indispensable parts. Directly removing alternative components the first path will break the integrity of the original feature space (ScaleKI) is not conducted in the frequency space), thus lowering the efficacy of DFM.
E More Visualization Results
In this section, we provide more visualization results for a better understanding of our method. In Figure 5
provide the frequency distributions of three pre-trained large ViT models. We can observe that these pre-trained ViT's show a consistency in unbalanced frequency distributions: the direct responses are salient and significantly stronger than the alternative responses. And in Figure 6, we show more examples of feature distance distributions of alternative components, comparing scenarios with and without DFM, between the teacher and the student on IN-IK. The results validate DFM can effectively reduce the alternative feature distances between the teacher and the student.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction contain three parts: i) motivation, ii) methodol-
ogy, and iii) experimental results, which are explained in Section 1, Section 2, and Section 3. We discuss the related works in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Conclusion part our main paper
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all training setups and implementation details in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code has been made publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa////giieee/eeeSSbiiiiissss (Codeooomiissssseeeeeeee ssiosolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We detailed explain all experimental settings in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The main experiments are conducted on large-scale datasets, whose results stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We give the demand compute resources in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwrrppp.cpppppp////iiia
Answer: [Yes]
Justification:
We have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We clearly illustrate the dataset and codebase we use in the Appendix and
the Appendix B
respectively.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We obtained some high-performance models in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "H3AIx9Tx5y", "reviewer": "Reviewer_k9Ls", "summary": "This paper concentrates on the distillation of knowledge from a large-scale, pre-trained, ViT-based teacher model to heterogeneous architectures. It incorporates three distinct designs: a) a Cross Attention Projector (CAP), which serves as the fundamental design that bridges the structural disparity between a non-ViT model and the ViT teacher; b) a Dual-View Feature Mimicking and a Teacher Parameter Perception module, both of which are constructed on top of the CAP to facilitate the distillation process. The effectiveness of the proposed methodology is validated through extensive experimentation.", "questions": "please refer to the weakness part.", "limitations": "The limitations are properly discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed distillation technique is effective for heterogeneous architectures with a ViT-based teacher.\n- The conducted experiments are comprehensive, providing solid validation for the effectiveness of the method.\n- The proposed CAP structure is a good contribution which is feasible for various student architectures.", "weaknesses": "__Presentation__: The connection between DFM and TPP and their respective motivations is unclear. Specifically, it’s unclear how the modifications of DFM and TPP specifically address the issues of model scale and knowledge density.\n\n__Effects of CAP__:  While the introduced CAP appears to be a promising design, its effectiveness is only demonstrated within the context of the ScaleKD framework. It would be interesting to see if this architecture could be integrated with other heterogeneous architecture methods, such as OFA, to replace traditional projectors.\n\n__Effects of DFM__: The ablation study of DFM merely indicates that the filtering operation is beneficial to CAP-based distillation. It raises the question of whether this strategy is also compatible with other feature alignment architectures, such as linear head or convolutional head?\n\n__Effects of TPP__: The design of TPP seems to be at odds with its stated purpose. Given that different network architectures, such as CNNs or ViTs, are known to have different knowledge preferences, it’s questionable whether aligning the student’s parameters with the teacher’s under a heterogeneous architecture is an effective approach. Furthermore, it’s unclear why using the last layer of the teacher as an additional head function for distillation would aid in alignment in the parameter space. Empirical evidence (such as CKA visualization) or theoretical justification is needed to demonstrate whether TPP can encourage different network architectures to exhibit similar learning behavior.\n\n__Experiments__:  Some details of the experiments, such as the teacher information in Tables 6-8, are missing from the presented tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the recognition of our work and the constructive comments. \n\n**1. To your comments about the presentation of DFM and TPP,**\n> **Our responses: (1)** The basic motivation of our DFM and TPP components is to align model scale and knowledge density differences in a joint but not separate manner, **under the premise that**: our basic component CAP (a novel cross-attention projector containing positional embedding, patchify stem and trainable queries) has already aligned feature computing paradigm differences between the ViT teacher and the CNN/MLP/heterogeneous ViT student and made the student have the same tokens-based feature modeling in terms of semantic units and spatial resolution as the teacher to perform feature distillation; **(2)** According to our analysis (Line#68-81), model scale differences make teacher and student models have different learning capacities, and knowledge density differences are mainly due to the pre-training data which is supposed to be visible only for the teacher in our work. As a result, these differences are intertwined under the prevalent pre-training and fine-tuning paradigm and are finally encoded in both teacher and student models’ feature space and parameter space. In light of this, DFM and TPP address the issues of model scale and knowledge density differences via considering feature distillation in the perspectives of feature space and parameter space;  **(3)** In principle, i) DFM relies on an insightful observation that shows the frequency feature distributions of pre-trained ViTs are extremely imbalanced (dominated by the direct component). Inspired by this, DFM uses a novel dual-view feature mimicking formulation to promote feature alignment in the original and frequency-aware feature spaces; ii) TPP relies on another critical observation that the pre-training knowledge is only gained by the ViT teacher as the pre-training data is invisible to the student. Based on this, TPP forms a cross-network proxy parameter alignment path via bridging the early stages (projected by CAP) of the student to the later stages of the ViT teacher, exploiting the use of the pre-trained teacher's parameters to further reduce the knowledge density discrepancy; **(4)** Generally, CAP, DFM and TPP are progressively designed, and they are closely connected to each other as discussed in (Line#51-127, Section 2) and validated in Table 9/13, Fig 2/3/5/6.\n\n**2. To your comments about the effects of CAP to other heterogenous KD methods,**\n>**Our responses:** Following your suggestion, we added an ablation by applying CAP to OFA. According to the results shown below, our CAP brings 0.27% top-1 gain to OFA with traditional projectors, validating its superiority.\n|Teacher|Student|Method|Top-1(%)|\n|--|--|--|:--:|\n|DeiT-T(72.17)|ResNet-18(69.85)|OFA|71.33\n|DeiT-T(72.17)|ResNet-18(69.85)|OFA+CAP|71.60\n\n**3. To your comments about the effects of DFM on other feature projectors,**\n>**Our response: (1)** Yes, our DFM would be compatible with existing feature projectors such as Linear and Conv projectors, as it stems from the observation of the teacher's feature distribution; **(2)** To study this compatibility, we added an ablation, following your suggestion. From the results shown below, we can see: i) DFM brings 0.35%|0.45% top-1 gain to Linear|Conv projector, validating the compatibility of DFM with them; ii) Comparatively, the top-1 gain of DFM to CAP is much higher, indicating the coupled design property of DFM and CAP. \n|Method|Top-1(%)|$\\Delta$ Top-1(%)|\n|--|:--:|:--:|\nBaseline|76.55|-|\nLinear|77.43|+0.88\nLinear+DFM|77.78|+1.23\nConv|77.52|+0.97\nConv+DFM|77.97|+1.42\nCAP|77.87|+1.32\nCAP+DFM|78.51|+1.96\n\n**4. To your comments about the effects of TPP,**\n> **Our responses: (1)** We first hope this concern is alleviated after you read our responses to your first concern. Given heterogeneous teacher and student network architectures, it is indeed not reasonable to use TPP directly, as connecting the student’s early-stage parameters with the teacher’s later-stage parameters seems to be a conflict due to different feature computing paradigms (lead to different knowledge preferences); **(2)** However, in our method, TPP is built upon CAP which well aligns feature computing paradigm differences and DFM which jointly considers feature distributions in the original and frequency-aware feature spaces, paving a strong base to exploit the use of the pre-trained teacher's parameters to reduce the knowledge density discrepancy. For simplicity, in implementation, we use the last stage of the teacher to construct the proxy parameter alignment path, which already attains promising performance (Table 9) yet it may be not optimal (e.g., using the last two stages of the teacher is slightly better, bringing ~0.15% extra top-1 gain); **(3)** Following your suggestion, we add CKA visualizations to illustrate the effects of TPP.  As shown in Fig 1 of the PDF file in \"Author Rebuttal by Authors\", ScaleKD encourages the student to have similar behaviors as the teacher at the last stage where TPP is applied.\n\n**5. To your comments about teachers' information to experiments in Tables 6-8,**\n>**Our responses: (1)** There are no teachers for the experiments on downstream tasks in Table 7-8. As described in Line#187-189, the experiments were performed based on the student model trained by ScaleKD on IN-1K to verify whether the performance gain could be well preserved to downstream tasks in the standard transfer learning regime; **(2)** In Table 6, when comparing ScaleKD with supervised and self-supervised methods, our teacher is Swin-L pre-trained on IN-22K. When comparing ScaleKD with the other two pre-training paradigms, our teacher is BEiT-L pre-trained by EVA.\n\n**Finally, we will update the manuscript based on the above responses**. Regarding more experiments and discussions that we have made, you are referred to our responses to the other reviewers and in \"Author Rebuttal by Authors\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GDXjTsK7m0", "reviewer": "Reviewer_3qQD", "summary": "This paper focuses on whether the pre-trained vision transformer models could be used as teachers to distilling knowledges to heterogeneous neural network architectures. The proposed ScaleKD aims to solve three problems including 1) feature computing paradigm different, 2) model scale differences, and 3) knowledge density differences. Extensive experiments are conducted with different neural networks, including CNN, ViT, and MLP on image classification task. This paper also shows that when scaling up the size of teacher models or their pre-training datasets, ScaleKD showcases larger gains to the student models.", "questions": "Please refer to the weakness.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The difficulties of transferring knowledge between different architectures are well summarized, including the differences in feature computing paradigm, differences in model scale, and differences in knowledge density.\n\n2. All the figures, as well as the writing are clear and easy to follow.\n\n3. The extensive experiments show the effectiveness of the proposed ScaleKD.", "weaknesses": "1. The related work section should be improved to discuss the differences between the proposed ScaleKD and existing works. The CAP, DFM, and TPP should be compared with existing works to show the originality and novelty. The references include:\n\n[1] ViTKD: Feature-based Knowledge Distillation for Vision Transformers, CVPR 2024\n[2] Prefallkd: Pre-impact fall detection via cnn-vit knowledge distillation, ICASSP 2023\n[3] Distilling efficient vision transformers from cnns for semantic segmentation, Arxiv 2023\n[4] A good student is cooperative and reliable: CNN-transformer collaborative learning for semantic segmentation, ICCV 2023\n\n2. The experiments are conducted with ViT, SwinT, ResNet, MobileNet, ConvNeXt, MLP-Mixer. There are several types of CNN, however the claims in Abstract mentions that \"Our method can train student backbones that span across a variety of CNN, MLP and ViT.\" More backbones should be included, or the description in abstract is over-claimed.\n\n3. A question is the ScaleKD includes the MLP and ViT layers in CAP, does it mean the improved KD performance comes from such architecture update? Is the proposed method achieve KD with adding such layers to augment the student model? This is the main concern, please give more discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the recognition of our work and the constructive comments. \n\n**1. To your comments about discussing more related works [1-4],**\n>**Our responses: (1)** Thanks for pointing out these four existing works which address KD between two transformers [1] or between the transformer and CNN [2-4]; **(2) Generally, our ScaleKD differs from these works in application focus, motivation, and method formulation. Firstly**, the application focuses of [2-4] are obviously different from ours. Our paper explores cross-architecture KD in the context of training backbone, fitting both classification and various downstream tasks, while [3-4] are KD methods specialized for semantic segmentation, and [2] is a KD application for fall detection (a task that aims to detect people's falls for avoiding injuries). **Secondly**, our motivation moves further from [1-4]. Specifically, they follow the training regime of traditional KD settings: i) the teachers are always small-to-medium-sized without pre-training (sharing the same dataset as the student); ii) the model size difference between teacher and student is not large. In sharp contrast, our ScaleKD explores how to help heterogeneous students approximate medium-to-large-size ViTs through new motivations: **how to connect KD research with state-of-the-art ViTs, aiming to help students mimic teachers' behaviors from the higher model capacity and the massive pre-training**, which shows clear differences to [1-4]; **Thirdly**, the formulation of ScaleKD is different from [1-4]. Basically, [1] uses distinct feature distillation strategies on shallow and deep layers, and [2] only conducts vanilla logits distillation between ViT teacher and CNN student. Differently, [3-4] have deeper insights and design methods specialized for semantic segmentation. To avoid heterogeneous features, [3] utilizes an MHSA layer to model global interdependencies on CNN features, and a channel-attention layer to build linguistic features, while [4] moves from offline KD to online collaborative learning and conducted feature distillation in early layers based on the similar mechanism as [3]. For model capacity gaps, [3] decouples the pixel-wise distillation by categories, while [4] employs a selective mechanism to ensure reliable distillation areas. In contrast, although our method also discusses related problems, the design ideas and method formulation have clear differences from [3,4]. In the context of learning from large pre-trained ViTs, our method underlines differences affecting the efficacy of KD from three aspects, feature computing paradigm, model scale and knowledge density. We first design CAP to tactfully align different computing paradigms across different architectures by employing patchfy stem and learnable queries. Progressively, we notice the other two types of differences are intertwined under the pre-training and fine-tuning paradigm, and are finally encoded in both teacher and student models’ feature space and parameter space. In light of this, DFM and TPP address the differences in model scale and knowledge density via considering feature distillation in the perspectives of feature space and parameter space.\n\n>[1] Z Yang, et al. \"ViTKD...transformers\", CVPR-W 2024.   \n[2] TH Chi, et al. \"Prefallkd...distillation\", ICASSP 2023.   \n[3] X Zheng, et al. \"Distilling...segmentation\", arXiv 2023.  \n[4] J Zhu, et al. \"A good student...segmentation\", ICCV 2023.\n\n**2. To your comments on experiments with more ViT and MLP backbones,**\n>**Our response: (1)** Thank you for kind suggestion. Accordingly, we conducted extra experiments on more ViT and MLP students.  Due to the time limitation in the rebuttal phase, we performed experiments on two other typical models, ResMLP-S12 [5] and PVT-S [6], following the same setting as Table 3 in the manuscript.  From the results in the below table, we can see that both on ResMLP-S12 and PVT-S backbones, our method obtains promising top-1 gains, further verifying its good generalization ability to ViT and MLP backbones; **(2) Actually, the students in Table 3 are elaborately selected.** Specifically, we consider i) heterogeneous and homogeneous network pairs, ii) network pairs with different model capacity gaps, iii) the variety of the teacher and the student, and iv) the popularity of the networks. We initially selected 10 student models across 6 model types based on the above 4 principles. Now, we have 12 student models across 8 model types, which are more comprehensive. **The updated Table 3 is added to the PDF file attached in \"Author Rebuttal by Authors\"**.\n| Teacher|Student|Top-1(%)| $\\Delta$ Top-1(%)| \n|--|--|:--:|:--:|\n|Swin-L(86.24)|ResMLP-S12(76.51)|80.54|+4.03\n|Swin-L(86.24)|PVT-S(79.80)|83.72|+3.92  \n  \n>[5] H Touvron, et al. \"ResMLP...Training\", TPAMI 2023.  \n[6] W Wang, et al. \"Pyramid ...Without Convolutions\", ICCV 2021.\n\n**3. To your comments about whether the improvements come from the architecture updates,**\n>**Our response: (1)** Our ScaleKD does not alter the student network architecture as CAPs are connected to the student as auxiliary paths for feature distillation similar to conventional feature projectors. After the distillation training stage, all three components of ScaleKD, namely CAP, DFM and TPP will be removed, introducing no additional cost in the inference stage; **(2)** We guess it is Fig 1, especially Fig 1(c), that may result in this misunderstanding, where CAP seems to be involved in the student's network. For simplicity, in Fig 1, we only show the feature-mimicking process, not including the inference process; **(3)** Actually, we clarified this in Line#123-125 in the manuscript. We will revise Fig 1 and its caption to avoid this potential misunderstanding.\n\n**Finally, we will update the manuscript based on the above responses**. Regarding more experiments and discussions that we have made, you are referred to our responses to the other reviewers and in \"Author Rebuttal by Authors\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rQsxbMc378", "reviewer": "Reviewer_rjnT", "summary": "This paper presents a new knowledge distillation method, named ScaleKD. Previous works mostly use CNNs to distill vision transformers. However, how to use vision transformers to distill CNNs is less explored. This paper shows that pretrained vision transformers are good teachers for other types of student models. The cores of the proposed method are three main components, including the cross-attention projector, dual view feature mimicking, and teacher parameter perception.\n\nThe motivation of this paper is clear and the proposed method is also interesting. The authors propose to do model distillation from three aspects, which have been proven useful in the experiment section.", "questions": "I think the authors should elaborate more on why scaled teacher models help in the proposed method. I would like to see some analysis on this.", "limitations": "The authors have included limitations in the main paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of this paper is interesting and the novelty is significant. Unlike previous KD methods, this paper proposes a new way to do knowledge distillation.\n\n- The presentation of this paper is also good. It seems that the proposed method is easy to follow.\n\n- Experimental results are good. Compared to previous knowledge distillation methods, the results shown in this paper improve them. In addition, the authors also provide detailed analysis on the importance of each component.", "weaknesses": "- Reading this paper is too tedious. The paragraph is too long in the introduction section. It is difficult to capture the important content.\n\n- The proposed method consists of three parts, which make it look complicated.\n\n- From Table 2, it seems that when the teacher models' scale increases, there is improvement. However, according to my knowledge, previous KD methods mostly fail to do this. Can the authors explain why the proposed method can achieve this? I think this is important for the KD community to design better KD methods.\n\n- Though exploring how to use vision transformers as teachers is important, I am also curious about another thing. Have the authors used CNNs as teacher models? This is what most previous works did.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition of our work and the constructive comments. \n\n**1. To your concern regarding the presentation of the Introduction section,**\n>**Our response: (1)** Indeed, paragraphs in the Introduction section of our original manuscript are not short. Although the key messages are organized in a relatively dense manner, we think they are presented logically. In paragraph#1, we introduce the background on the evolution of network architectures and model learning paradigms and the problems. In paragraph#2, we first analyze the existing KD research and point out the weaknesses, then propose the motivation of our work, and finally elucidate three technical barriers for KD under our settings. In paragraph#3, we first describe the core ideas of our three components to address above barriers, then clarify how our method is formulated with the proposed components; **(2)** Following your suggestion, we will improve the presentation of the Introduction section, split long paragraph#2|#3 into two short paragraphs, making each paragraph be easy to capture the key content.\n\n**2. To your concern regarding the proposed method looks complicated,**\n>**Our response: (1)** Our method consists of three components, CAP, DFM and TPP, which are closely coupled to each other to address the differences in feature computing paradigm, model scale, and knowledge density. Our base component CAP is a neat cross-attention projector to transform CNN/MLP features into ViT-like features first, then DFM promotes feature alignment in both the original and frequency-aware feature spaces, inspired by two insightful observations (clarified in Line#98-105). Built upon CAP and DFM, TPP further establishes a proxy path to exploit the use of the pre-trained teacher's parameters to align the knowledge density discrepancy. In the ablation study, we validated the necessity of each component design (Table 9); **(2)** Thanks to the coupled design properties of CAP, DFM and TPP, the overall formulation of our method is simple, as discussed in Section 2.4.\n\n**3. To your comments regarding why our method could make pre-trained ViTs become good teachers,**\n>**Our response: (1)** Yes, many previous KD methods, such as [1-3], claim that large models are not always good teachers,  usually using CNNs as the teachers; **(2)** In the ViT era, ViT shows extraordinary scalability on model size and pre-training data size: i) larger model size provides ViT with stronger model learning capability; ii) the progress of pre-training makes large ViT learn better knowledge from more data. Thus, large ViTs are pre-trained on massive datasets, like IN-22K or LAION-2B. As a result, the capability gap between large-size models and small-size models in this era is larger than before, not only just suggested as the increasing model scale differences, but also as knowledge differences from pre-training data gaps; **(3)** Based on the above analysis, our work questions whether the student could inherit ViT teachers' scalable properties and underlines three important differences between teachers and students that lead to KD barriers. They are feature computing paradigm differences, model scale differences, and knowledge density differences. Accordingly, we design CAP, DFM and TPP to tackle the problems altogether. Fundamentally, we design CAP to directly align different computing paradigms across different architectures by employing patchfy stem and learnable queries. Progressively, we notice the other two types of differences are intertwined under the prevalent pre-training and fine-tuning paradigm, and are finally encoded in both teacher and student models’ feature space and parameter space. In light of this, DFM and TPP address the issues of model scale and knowledge density differences via considering feature distillation in the perspectives of feature space and parameter space. In feature space, DFM promotes feature alignment in both the original and frequency-aware feature spaces, giving more attention to subtle alternative components to promote feature space alignment. In parameter space, built upon CAP and DFM, TPP further establishes a proxy path to exploit the use of the pre-trained teacher's parameters to align the knowledge density discrepancy. As the three problems are well addressed, students can break down the barriers and inherit ViT teachers' capabilities. Thus, we can see the desired scalable performance trends in Table 2/3/6.\n\n>[1] S.I., Mirzadeh, et al. \"Improved...teacher assistant\", AAAI 2020.  \n[2]  W. Son, et al. “Densely...multiple teacher assistants”, CVPR 2021.  \n[3] T. Huang, et al. “Knowledge...stronger teacher”, NeurISP 2022.  \n\n**4. To your comments on using CNN teachers,** \n>**Our response: (1)** We did not use CNNs as teacher models in the original manuscript since our work focuses on exploring the scalable properties of pre-trained ViT teachers to advance cross architecture KD research, as clarified in the Abstract and Introduction section; **(2)** To your request, with ResNet-50 as the student, we additionally use CovNeXt-XL as the teacher (adopted in the recent work [4]) besides Swin-L and BEiT-L. From the results shown below, we can observe that: i) ViT teachers show superiority to CNN teachers, e.g., ScaleKD achieves a notable 0.62% top-1 gain when the teacher is changed from ConNeXt-XL to BEiT-L; ii) Given ConNeXt-XL as the teacher, ScaleKD also outperforms [4] with a clear margin.\n|Teacher|Student|Method|Top-1(%)|\n|--|--|--|--|\n|ConNeXt-XL(86.97)|ResNet-50(79.80)|VanillaKD|81.10\n|ConNeXt-XL(86.97)|ResNet-50(78.64)|ScaleKD| 81.72\n|Swin-L(86.24)|ResNet-50(78.64)|ScaleKD|82.02\n|BEiT-L(88.58)|ResNet-50(78.64)|ScaleKD|82.34\n\n>[4] Z Hao, et al. \"VanillaKD... to large scale\", arXiv 2023.   \n\n**Finally, we will update the manuscript based on the above responses**. Regarding more experiments and discussions that we have made, you are referred to our responses to the other reviewers and in \"Author Rebuttal by Authors\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EpCxjnpBh9", "reviewer": "Reviewer_mfZF", "summary": "This paper introduces a novel knowledge distillation method called ScaleKD. The method aims to leverage well pre-trained vision transformer models as teacher models for a variety of student model architectures.the authors first adopt a cross attention projector to align student features with the teacher's. Then, a dual-view feature mimicking module and a teacher parameter perception module are used to achieve better knowledge transfer. Extensive experiments demonstrate the effectiveness of ScaleKD across various tasks and model types.", "questions": "In the dual-view feature mimicking module, the direct component is omitted during the alternative feature mimicking process. However, the remaining non-direct features are duplicated in the normal matching process (as shown in the upper path in Figure 2b). How does the removal of the non-direct component in the upper path affect performance?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-structured.\n2. The authors provide extensive experimental results and analyses that demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The paper's motivation could be strengthened. As highlighted in previous research [1], a stronger teacher model does not always equate to a better teacher. The necessity of adapting a ViT teacher for training a CNN student needs further justification. Including a comparison between ViT teachers and CNN teachers would provide better support.\n2. The teacher parameter perception (TPP) module's cross-architecture KD paradigm is similar to techniques used in existing research [2,3]. While this does not diminish the novelty of other aspects of the paper, these related works should be properly discussed.\n3. The authors critique the focus on \"evaluation on small datasets with non-mainstream student models\" in existing works. Some relevant papers about these issues should also be discussed, such as [4,5].\n\n[1] Mirzadeh, Seyed Iman, et al. \"Improved knowledge distillation via teacher assistant.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 04. 2020.\n\n[2] Wang, Jiabao, et al. \"CrossKD: Cross-head knowledge distillation for object detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Bai, Haoli, et al. \"Few shot network compression via cross distillation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\n[4] Hao, Zhiwei, et al. \"Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale.\" arXiv preprint arXiv:2305.15781 (2023).\n\n[5] Stanton, Samuel, et al. \"Does knowledge distillation really work?.\" Advances in Neural Information Processing Systems 34 (2021): 6906-6919.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition of our work and the constructive comments. \n\n**1. To your comments regarding strengthening the motivation of our work,** \n>**Our responses: (1)** Yes, in previous KD works like [1] using CNNs as the teachers, a stronger model is not always a better teacher. Following your suggestion, with ResNet-50 as the student, we additionally use CovNeXt-XL (the current top-performing CNN) as the teacher (adopted in recent work [4]) besides Swin-L and BEiT-L. From the results shown below, we can observe that: i) ViT teachers are superior to CNN teachers for our method, e.g., a notable 0.3%|0.62 top-1 gain is achieved when the teacher is changed from ConNeXt-XL to Swin-L|BEiT-L; ii) Given ConNeXt-XL as the teacher, our method also outperforms [4] with a clear margin. These results validate the necessity of adapting large pre-trained ViTs as teachers for training a CNN student; **(2)** Besides, we would like to emphasize the motivation of our work. Along with the evolution of network architectures and model learning paradigms, ViTs show notably improved performance when scaling up the model size and the pre-training data size. The motivation of our work is to connect cross architecture KD research with well pre-trained ViT teacher models that stand out for their remarkable scalable properties, exploring an effective way (ScaleKD) to transfer teachers' scalable properties to student models (CNNs/MLPs/heterogeneous ViTs), given that the pre-training data is invisible to student models. \n|Teacher|Student|Method|Top-1(%)|\n|--|--|:--:|:--:|\n|ConNeXt-XL(86.97)|ResNet-50(79.80)|VanillaKD|81.10\n|ConNeXt-XL(86.97)|ResNet-50(78.64)|ScaleKD|81.72\n|Swin-L(86.24)|ResNet-50(78.64)|ScaleKD|82.02\n|BEiT-L(88.58)|ResNet-50(78.64) |ScaleKD|82.34\n\n**2. To your comments regarding the discussion of our TPP module with some existing research [2,3],**\n>**Our responses: (1)** Thank you for mentioning these two works [2,3] that are related to our TPP module. Actually, we discussed some similar related works [32,59,60,61] in Line#329-333 of the manuscript, which are in the same line of research as [2,3]. Generally, TPP differs from these works in motivation and designs; **(2)** The motivation of why previous works construct the proxy path is to avoid contradictory supervision signals from the annotations and the teacher’s predictions [2], or to avoid accumulated estimation errors [3]. In contrast, the key motivation of TPP is to align the teacher’s and student’s parameter spaces, which is beneficial for pursuing the pre-training knowledge implicitly contained in the teacher's parameter space; **(3)** As the motivations are not the same, our TPP has some special modifications. On the one hand, unlike previous works that reuse a proportion of detection head [2] or a single layer [3], TPP uses the last whole stage of the teacher to have an integrated perception of teachers' parameter space. On the other hand, to have a better feature alignment, the outputs of the proxy path are provided as input-dependent queries of our CAP module.\n\n**3. To your comments regarding the discussion of our method with [4,5],**\n>**Our responses: (1)** Thank you for mentioning two related works [4,5], which aim for a better understanding of vanilla KD based on logits ([14] in our manuscript). Specifically, the early work [5] studies how to improve fidelity (measuring the predication consistency between teacher and student models) instead of the prevailing generalization ability (model accuracy), and the later work [4] explores how advanced training recipes (e.g., larger datasets, stronger data augmentations/optimizers, significantly longer training epochs (4800 epochs)) affect the performance gap of vanilla KD to its variants. Generally, [4,5] do not present any new KD methods; **(2)** We already discussed a related work FuncMatch [49] in our manuscript, which has similar statements as [4]; **(3)** In contrast, our work presents a new cross architecture KD method, showing that well pre-trained ViT models could be used as teachers and their scalable properties could be transferred to CNN/MLP/heterogeneous ViT students;**(4)** Due to the orthogonal focuses, our method may get improved performance using advanced training recipes in [4]. We leave it for future work.\n\n**4. To your question about the effect of removing the non-direct components in the first path of DFM,**\n> **Our responses: (1)** Following your comments, we remove all non-direct components in the first path of DFM and perform an experiment under the same settings as Table 9(b) in our manuscript; **(2)** For the results shown below, DFM(Dir + Alt) indicates your mentioned setting, while DFM(All+ Alt) is the original design. We can observe that removing the non-direct components in the first path will slightly decrease the effectiveness of DFM, but its performance is still obviously higher than CAP. As we discussed in Line#109-113, the first path of DFM aims to capture the teacher's global features, where the subtle non-direct components are also indispensable parts. Directly removing non-direct components in the first path will break the integrity of the original feature space (ScaleKD is not conducted in the frequency space), thus lowering the efficacy of DFM.\n|Method|Top-1(%)|$\\Delta$Top-1(%)|\n|--|:--:|:--:|\nBaseline|76.55|-|\nCAP|77.87|+1.32\nDFM(Dir+Alt)|78.23|+1.68\nDFM(All+Alt)|78.51|+1.96\n\n>[1] S.I., Mirzadeh, et al. \"Improved...teacher assistant\", AAAI 2020.   \n[2] J Wang, et al. \"CrossKD...for object detection\", CVPR-W 2024.   \n[3] H Bai, et al. \"Few shot...cross distillation\", AAAI 2020.  \n[4] Z Hao, et al. \"VanillaKD... to large scale\", arXiv 2023.   \n[5] S Stanton, et al. \"Does...really work?\", NeurIPS 2021.   \n\n**Finally, we will update the manuscript based on the above responses**. Regarding more experiments and discussions that we have made, you are referred to our responses to the other reviewers and in \"Author Rebuttal by Authors\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Area Chairs, Senior Area Chairs and Program Chairs,

We sincerely thank all four reviewers for their thorough and constructive comments. We are glad that the novelty, method component designs, validation pipeline and performance of our work have been mostly recognized by all four reviewers.

In the past week, we carefully improved the experiments (using all computational resources we have), the clarifications and the discussions of our work to address the concerns, the questions and the requests by all four reviewers. **Summarily, we made the following improvements:**
> (1)  To have a better understanding of specific component designs in our ScaleKD framework, we follow the constructive comments/requests by Reviewer k9Ls and Reviewer mfZF, and add several ablation studies and analytical experiments, including: i) An ablation study on the compatibility of DFM with other types of feature alignment architectures; ii) An ablation study on verifying the necessity of non-zero frequency features in the first mimicking path of DFM; iii) An ablation study on the compatibility of CAP with other cross-architecture KD methods; iv) Analytical experiments on how TPP affects the student's learning behaviors in our ScaleKD framework.  
(2) To further strengthen the motivation of our work, we follow the constructive comments/requests by Reviewer 3qQD, Reviewer rjnT, and Reviewer mfZF, and provide: i) Experiments that compare ViT teachers and CNN teachers; ii) Experiments on more teacher-student network pairs; iii) Detailed analysis on why ScaleKD shows scalability in terms of the pre-trained ViT teacher's capability; iv) Comparisons with more related works.  
(3) We also provide detailed responses to the other concerns/questions/requests raised by each reviewer one by one.

Finally, in the attached one-page PDF file, all the aforementioned experimental results are summarized in different Tables. We will include the above experiments and discussions in our final paper. We hope our detailed responses are helpful to address the concerns, the questions and the requests of all four reviewers.

Meta-review: All reviewers agree on the quality of the paper that provide a technically solid distillation method with experiments to support is effectivity. I recommend acceptance for this paper.
Decision: Accept (poster)