One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
Authors: Zhiwei Hao
Jianyuan Guo
Kai Han
Yehui Tang
Han Hu
Yunhe Wang
Chang Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge distillation
feature distillation
heterogeneous architectures
TL;DR: We study the cross-architecture distillation and propose a OFA-KD method to get better performance on teacher-student belonging to different architectures, such as CNN, Transformer, and MLP.

Abstract: Knowledge distillation (KD) has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. However, most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family, particularly the hint-based approaches. By using centered kernel alignment (CKA) to compare the learned features between heterogeneous teacher and student models, we observe significant feature divergence. This divergence illustrates the ineffectiveness of previous hint-based methods in cross-architecture distillation. To tackle the challenge in distilling heterogeneous models, we propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.  Specifically, we project intermediate features into an aligned latent space such as the logits space, where architecture-specific information is discarded. Additionally, we introduce an adaptive target enhancement scheme to prevent the student from being disturbed by irrelevant information. Extensive experiments with various architectures, including CNN, Transformer, and MLP, demonstrate the superiority of our OFA-KD framework in enabling distillation between heterogeneous architectures. Specifically, when equipped with our OFA-KD, the student models achieve notable performance improvements, with a maximum gain of 8.0% on the CIFAR-100 dataset and 0.7% on the ImageNet-1K dataset. PyTorch code and checkpoints can be found at https://github.com/Hao840/OFAKD.

One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation
Zhiwei Hao 2.2, Jianyuan Guo", Kai Han", Yehui Tang",
Han Hu I Yunhe Wang?; Chang Xu
School of information and Electronics, Beijing Institute of Technology. Huuawei Noah's Ark Lab.
School of Computer Science, Faculty of Engineering, The University of Sydney.
(haozhw, hbul edu. cn, jguo5172@uni. www.i/ieeeii.ddeey..eee edu. au, [kai.ham yehui.
tehui,tang, tang, yunhe. (unhe.wang) wang) Chuawei. com, c.xu@sydney. edu
Abstract
Knowledge distillation (KD) has proven to be a highly effective approach for enhancing model performance through teacher-student training scheme. How- ever, most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family, particularly the hint-based approaches. By using centered kernel alignment (CKA) to compare the learned features between heterogeneous teacher and student models, we ob- serve significant feature divergence. This divergence illustrates the ineffectiveness previous hint-based methods in cross-architecture distillation. To tackle the challenge in distilling heterogeneous models, we propose simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distil- lation performance between heterogeneous architectures. Specifically, intermediate features into an aligned latent space such as the logits space, where architecture-spper speciffcc information is discarded. Additionally, we introduce an adap- tive target enhancement scheme to prevent the student from being disturbed by irrelevant information. Extensive experiments with various architectures, including CNN, Transformer, and MLP, demonstrate the superiority of our OFA-KD frame- work in enabling distillation between heterogeneous architectures. Specifically, when equipped with our OFA-KD, the student models achieve notable performance improvements, with maximum gain of 8.0% the CIFAR-100 dataset and 0.7% on the ImageNet- K dataset. Porch code and checkpoints can be found at http:///gitbb.coo////oo////0/01
project
Introduction
In recent years, knowledge distillation (KD) has emerged promising approach for training lightweight deep neural network models computer vision tasks 2, 3,4. 4]. The core idea behind KD to train a compact student model mimic the outputs, or soft labels, of pretrained cumbersome teacher model. This method was initially introduced by Hinton el al.. [5], and since then, researchers have been actively developing more effective KD approaches. One notable improvement in KD methods is the incorporation intermediate features as hint knowledge [6, 7, 8]. These approaches train the student to learn representations that closely resemble those generated by the teacher, and have achieved significant performance improvements.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Existing hint-based approaches have predominantly focused on distillation between teacher and student models with homogeneous architectures, e.g., ResNet-34 ws.ResNet-18 in [9], and Swin-S vx.Swin-T in [8], leaving cross-architecture distillation relatively unexplored. However, there are practical scenarios where cross-architecture distillation becomes necessary. For example, when the objective is to improve the performance of widely used architecture like ResNet50 [10], as our experiments will demonstrate, distilling knowledge from ViT-B [11] to ResNet50 can readily surpass the performance achieved by distilling knowledge from 15. Cross-architecture distillation thus presents a viable alternative, providing more feasible options for teacher models. It may not always be possible to find superior teacher model with a homogeneous architecture matching the student's ' architecture. Moreover, the emergence of new architectures [12, 13] further compounds this challenge, making it difficult to find readily available pre-trained teacher models of the same architecture online. This limitation necessitates the need for researchers to resort to pre-trained teachers of different architectures for knowledge distillation in order enhance their own models.
The few attempts that have been made in this direction mainly involve using heterogeneous CNN models for knowledge transfer [14, 15, 16]. In recent study, Touvron et al. [17] successfully trained a ViT student by using a CNN teacher and obtained promising results. These endeavors have sparked our interest in studying the question: whether it is possible to distill knowledge effectively across any architectures, such as CNN [10, 18, 19]. Transformer [20, 21, 11, 17, 22]. and MLP [13, 23, 24].
In previous studies where teachers and students share similar architectures, their learned represen- tations naturally exhibit similarity in the feature space. Simple similarity measurement functions such as mean square error (MSE) loss, are sufficient for information distillation [6]. However, in the case that the teacher and student architectures are heterogeneous, there is no guarantee of successful alignment of the learned features, due to the fact that features from heterogeneous models reside in different latent feature spaces, as shown in Figure I. Therefore, directly matching these irrelevant features is less meaningful and even impedes the performance of the student model, as shown in the third row of Table I, highlighting the difficulty in exploring cross-architecture distillation at intermediate layers.
To address above challenges, we propose a one-for-all KD (OFA-KD) framework for distillation between heterogeneous architectures, including CNNs, Transformers. and MLPs. Instead of di- rectly distilling in the feature space, we transfer the mismatched representations into the aligned logits space by incorporating additional exit branches into the student model. By matching the outputs of these branches with that from teacher's ' classifier layer in logits space, which contains less architectur--specffi information, cross-architecture distillation at intermediate layers becomes achievable. Furthermore, considering that heterogeneous models may learn distinct predictive dis- tributions due to their different inductive biases, we propose a modified KD loss formulation to mitigate the potential impact of irrelevant information in the logits. This includes the introduction of an additional modulating parameter into the reformulated loss of the vanilla KD approach. The modified OFA loss adaptively enhances the target information based on the predictive confidence of the teacher, effectively reducing the influence of irrelevant information. To evaluate our proposed framework, we conduct experiments using CNN, Transformer, and MLP architectures. We consider all six possible combinations of these architectures. Our OFA-KD framework improves student model performance with gains of up to 8.0% on CIFAR- and 0.7% on ImageNet- 1K datasets, respectively. Additionally, our ablation study demonstrates the effectiveness of the proposed framework.
2 Related works
Recently, significant advancements have been made in the design of model architectures for computer vision tasks. Here we briefly introduce two prominent architectures: Transformer and MLP.
Vision Transformer. Vaswani et al. [25] first proposed the transformer architecture for NLP tasks. Owing to the use of the attention mechanism, this architecture can capture long-term dependencies effectively and achieves remarkable performance. Inspired by its great success, researchers have made great efforts to design transformer-bes
models for CV tasks [26, 27, 28]. Dosovitskiy et al. [11] split an image into non-overlapped patches and projected the patches into embedding tokens. Then the tokens are processed by the transformer model like those in NI.. Their design achieves Mfte-of-the--a performance and sparks the design of a series of following architectures [21, 29].

---

MLP. MLP has performed inferior to CNN in the computer vision field for a long time. To explore the potential of the MLP. Tolstikhin et al. [13] proposed MLP-Mixer, which is based exclusively on the MIP structure. MIP-Mixer takes embedding tokens of an image patch as input and mixes channel and spatial information each layer alternately. This architecture performs comparably to the best CNN and ViT models. Touvron et al. [24] proposed another MLP architecture termed ResMLP.
The most advanced CNN, Transformer, and MLP achieve similar performance. However, these architectures hold distinct inductive biases, resulting different representation learning preferences. Generally, there are striking differences between features learned by heterogeneous architectures.
Knowledge distillation. KD is an effective approach for compressing cumbersome models [30, 31, 32, 33, 34, 35], where a lightweight student is trained to mimic the output logits of a pretrained teacher. Such idea was originally proposed by Bucila et al. [36] and improved by Hinton et al. [5]. Following works further improve logits-based KD via structural information 38). model ensemble [39], or contrastive learning [40]. Recently, Touvron et al. [17] proposed logits distillation method for training ViT students. Huang et al. [9] proposed to relax the KL divergence loss for distillation between teacher and student with large capacity gap. Besides logits, some KD approaches also use intermediate features as hint knowledge. Romero el al. [6] first proposed the hint-based distillation approach. They adopted a convolutional layer to project student features into the space of the teacher feature size, and then constrained the projected feature to be like the teacher feature via mean square loss. Zagoruyko et al. [7] proposed force the student to mimic attention maps. Yim er al. [41] adopted the flow of solution procedure matrix generated by the features as hint knowledge. There are also many other feature distillation approaches using various hint design [42, 43, 8]. Recently, applying KD to compress dataset has emerged as a popular topic [44, 45, 46].
Although existing hint-based distillation methods have achieved remarkable performance, they assume that student and teacher architectures are homogeneous. When the architectures are heterogeneous, existing approaches may fail because features of the student and the teacher are distinet. How to conduct hint-based distillation under this circumstance is still an open problem.
3 Method
3.1 Revisit feature distillation for heterogeneous architectures
Discrepancy among architectures. For a long time, CNN is the dominant architecture for CV tasks. Its inductive biases of locality and spatial invariance are strong priors for learning natural images, and restrict CNNs from learning global representations. Vision transformers [11,21, 47] inherited from the NLP field are attention-based models. This architecture lacks inductive biases inherent to CNNs and thus can learn global representations easily. Moreover, inspired by the success of ViTs, pure MlsP-based vision models have been proposed [13, 24], which are also low-priori models.
Knowledge distillation. KD trains a tiny student model to learn knowledge from a large pre- pre-trained teacher model. Logits and features are the most common used types of knowledge in KD.
Logits depict the predictive distribution of a model. Logits-based distillation trains the student to mimic output logits of the teacher, which can be formulated as:
(1) where ()) is the joint distribution of samples and class labels. p" and p² are predictions of the student model and the teacher model on sample æ, respectively. DeF denotes the cross-entropy function. DxI. denotes the Kullback-Leibbee divergence function. a hyperparameter controls the trade-off between one-hot label and soft label p'.
LKD AE (x) [Dee(p", .v) + (1 A)DKL p')],
hint-based distillation adopts more fine-grained teacher activations intermediate layers to train the student, which can be formulated as:
LFD = (( E F? (F
(2)
where F and F$ are features of the teacher and the student at layer i, respectively. ub is a feature projector that maps student features to match dimension of teacher features [6]. Knowledge from logits and features are not opposites. In most cases, feature distillation approaches also adopt logits as auxiliary supervision.
3

---

0
04
a/y-axis: CNNICNN
adg-axis: VIT/VIT
afy-axis: MLPMMP z/ty-axis: CNN/VIT
alg-axis: VIT/LP
2/y--aiss: CNN/MPP
Figure Similarity heatmap of intermediate features measured by CKA. We compare features from MobileNetV2 (CNN), ViT-Small (Transformer) and Mixer-B/16 (MIP model). The first three figures illustrate similarity between homogeneous models, and the last three illustrate feature similarity between heterogeneous models. Coordinate axes indicate the corresponding layer index.
Challenges in heterogeneous feature distillation. Existing feature distillation methods designed under the assumption of teacher and student belonging to the same model family. Simply adopt convolutional projector is able to align their features for distillation. However, heterogeneous models learn diverse features because of there distinct inductive biases. Under this circumstance, directly foreing features of the student mimic the teacher may impede the performance. Moreover, ViTs use embedded image patches as input, and some architectures even adopt an additional classification token [11, 17], which further aggregates the feature mismatch problem.
Centered kernel alignment analysis. To demonstrate the representation gap among heterogeneous architectures, adopt centered kernel alignment (CKA) [48, 49] compare features extracted by CNN, ViT, and MLP models. CKA is a feature similarity measurement allowing cross-architecture comparison achievable, as it can work with inputs having different dimensions.
CKA evaluates feature similarity over mini-batch. Suppose E Raxdd and Y Ruxd are features of n samples extracted by two different models, where d1 and d2 are their dimensions, respectively. CKA measures their similarity by:
DHSIC Dasic(K,L) L)
CKA(K,L)
(3)
DASIC (K K DHSIC (L, L)
where L XX" and K = YY are Gram matrices of the features, and Dhsic is the Hilbert-Schmice independence criterion [50], a non-parametrie independence measure. The empirical estimator of DHSIC can be formulated as:
DHSIc(K,L) (n - 1)2² tr KHLH),
(4)
where H is the centering matrix H, I, 11
Figure shows feature similarity heatmaps measured by CKA. From figures the first row, models of homogeneous architectures tend to learn similar features at layers of similar positions. On the country, heatmaps the second row demonstrate that features of heterogeneous architectures remarkably different. In particular, MobileNet v2 can only learn features similar to shallower layers of ViT-S or Mixer-B/16. Moreover, features of VIT-S and Mixer-B/16 are quite different as well.
From the CKA analysis, directly using hint-based distillation methods for cross-architecture KD unreasonable. Although only using logits for distillation is feasible, the absence of supervision at intermediate layers may lead to suboptimal results. New designs of distilling at intermediate layers needed to improve heterogeneous KD.
3.2 Generic heterogeneous knowledge distillation
To bridge the gap of heterogeneous architectures in KD, we propose one-for-all cross-architecture KD method, which can perform KD between models of any architectures, such as CNN, ViT, and MLP. Figure 2 provides overview of the proposed framework.
Learning in the logits space. Following the CKA analysis results, heterogeneous models learn distinct representations, which impedes the feature distillation process. Hence, the key for heteroge- neous feature distillation align the mismatched representations. However, existing approaches
Detailed settings of our CKA analysis provided in supplement.

---

Adipiive curfiicieni enfancement acconding teacher prediction
Stage
Stage
Stage
Orgizal
LXL
Stage
Stage
Stage
Torse predutiin pi
(a) Logits based distillation
Stage
Stage
Stage
Stage
Stage
Stage
Cor
Cora
Cora
CuesE
CusE
CAEE
CorA
LKL
clm,
ctm. oligner
protector
Stage
Stage
Stage
Stage
Stage
Stage
(b) Feaaurr-based distillation
(c) Our OFA distillation
Teacher module
Stage
Student module
Stage
Feature
Logits
Loss
COFA
student only learns from final predictions of the teacher; (b) hint-basee bast- distillation: besides final predictions, student also learns to mimic intermediate features of the teacher; and (c) our OFA-KD framework: student learns from final predictions of the teacher via multiple branches, where the OFA distillation loss adaptively adjusts the amount of target information to be learned based on the confidence teacher. Note: Only three stages shown for convenience. In our experiment, all models are split into four stages.
Figure 2: Comparison of different KD approaches. (a) Logits-based distillation:
directly align dimension of features via a convolutional projector [6], which is not generic solution for all heterogeneous architectures. Inspired by early-exit model architecture [5]], align features in the logits space by introducing additional exit branches into the student. In particular, we build each exit branch with feature projector and a classifier layer. On the teacher side, we direct adopt its final output as the knowledge. In the logits space, redundant architecture-specif information removed, and thus feature alignment across any architectures achievable. When training the student, the additional exit branches are optimized together with the student backbone. At test time, these branches are removed and render no extra inference overhead.
A Adaptive target information enhancement. Although different models learn the same targets in the logits space, their different inductive bias leads them variant destinations, i.e., their predictive distributions are not the same. For example, a CNN model may predict a sample with similar probabilities of belonging to cars and trucks, as CNNs tend to catch the shared local information of the two classes, e.g., round wheels. In contrast, ViTss may differentiate these samples better, as its attention mechanism apt learn the distinct global features, e.g., arc-shaped and rectangle-shaper bodies. This difference results in inconsistent understanding of the dark knowledge of the teacher and the student. To bridge this gap, we propose enhance the information of the target class as always relates to the real class of sample and brings no concern about information mismatch.
We take the first step by reformulating the original distillation loss decouple information about the target class:3
LKD log P Eawy Ip' log
(1 + pb) log p E Ee-y [p: p").
where we ignored the denominator term KL-divergence as it does not contribute to the gradient. is a predicted class and indicates the target class. To enhance information from the target class, we add modulating parameter y 2 to the p term:
LoFa -(1 pb) log Rowy pp: lo lo]
Without loss of generality, omit the hyperparameter
5

---

Discussion. To learn the behavior of the OFA loss with different y values, we take the case of E E+ as an example expand the binomial term:*
CoFa = -- (pb)* logpE - E E/wy (p' pol
= =- + pb) log - En-y/(A) (6) [p) log p.l - (2=) - p log
(7)
LKD + -E k- (p')* pb) log p
When 7 I, the OFA distillation loss is identical the logits-based KD loss. To study cases of
2 1, we take 2 as an example:
2 LKD - (p': + (p!)²) log
log P
(8)
Besides the KD loss term, the OFA distillation loss has an additional positive term that only relates to the target class. If the teacher is confident the target class, the high order term (p$)* decays slowly. Otherwise, the high order term decays faster, which prevents the student to learn from a less confident teacher and achieves adaptive enhancement target information. The top-right corner of Figure provides brief illustration about effect of the high order term in the OFA loss.
Heterogeneous models inherently possess differences in their learning capabilities and preferences. Our proposed adaptive target information enhancement method bridges this gap by mitigating the impact of soft labels when the teacher provides suboptimal predictions. Moreover, the efficacy of distillation between heterogeneous architectures can be further enhanced by combining the multi- branch learning paradigm the aligned logits space.
Experiment
4.1
Experimental setup
We conducted comprehensive experiments to evaluate the proposed OFA-KD framework. In this section, we offer a concise overview of our experimental settings. Additional details can be found in supplementary materials.
Models. We evaluated variety of models with heterogeneous architectures. Specifically, we employed ResNet [10], MobileNet v2 [18], and ConvNeXt [12] as CNN models, ViT DeiT [17], and Swin [21] as ViT models, and MLP-Mixer [13] and ResMLP [24] as MLP models.
In our OFA-KD framework, incorporating additional exit branches into the student model requires identifying appropriate branch insertion points. For models with pyramid structure, we consider the end of each stage as potential insertion points, resulting in a total of four points. In the case of other models like ViT, we evenly divide them into four parts and designate the end of each part as feasible insertion points for the branches. Exit branches in CNNs are constructed using wipth-width convolutional layers, while exit branches in ViTs and MLPs employ VIT blocks.
When used as the student model, the smallest variant of Swin-T demonstrates superior perfor- mance compared several teachers when trained from scratch. For example, Swin-T outperforms Mixer-B/16 on the ImageNet-I6 IK dataset. To facilitate comparisons between such teacher-student combinations, we introduce two modified versions of Swin-T: Swin-Nano and Swin-Pico, referred to as Swin-N and Swin-P respectively.
Datasets. We adopt the CIFAR- 100 dataset [52] and the ImageNet- IK dataset [53] for evaluation. CIFAR-100 comprises 50K training samples and 10K testing samples of 32x32 resolution, while the ImageNet-1K KK dataset is more extensive, containing 1.2 million training samples and 50,000 validation samples, all with resolution of 224. Since ViTs and MLPs accept image patches as input, we upsample the images in CIFAR- to the resolution of 224x224 for all subsequent experiments to facilitate the patch embedding process.
Baselines. We employ both logits-based and hint-based knowledge distillation (KD) approaches as our baselines. Specifically, the hint-based approaches include FitNet [6], CC [54], RKD [37]. and CRD [40], while the logits- based basclines consist of KD [5], DKD [55], and DIST [9].
We take the assumption of y Σ above just for convenience, and there is no limitation of requiring be integer in practice.

---

Table 1: KD methods with heterogeneous architectures on ImageNet- IK. The best results are indicated in bold, while the second best results are underlined. f: results achieved by combining with FitNet.
From Scratch
hint-based
Logits- based
Teacher
Student
T.
S.
FitNet
CC
RKD CRD
KD
DKD DIST
OFA
CNN-based students
DeiT-T
ResNet18
72.17
69.75
70.44
69.77
69.47
69.25
70.22
69.39
70.64
71.34
Swin-T
ResNet18
81.38
69.75
71.18
70.07
68.89
69.09
71.14
71.10
70.91
71.85
Mixer-B/16
ResNet18
76.62
69.75
70.78
70.05
69.46
68.40
70.89
69.89
70.66
71.38
DeïT-T
MobileNetV2
72.17
68.87
70.95
70.69
69.72
69.60
70.87
70.14
71.08
71.39
Swin-T
MobileNetV2
81.38
68.87
71.75
70.69
67.52
69.58
72.05
71.71
71.76
72.32
Mixer-B/16
MobileNetV2
76.62
68.87
71.59
70.79
69.86
68.89
71.92
70.93
71.74
72.12
ViT-based students
ResNe150
DeiT-T
80.38
72. 17
75.84
72.56
72.06
68.53
75.10
75.601
75.131
76.55
ConvNeXi-T
DeïT-T
82.05
72.17
70.45
73.12
71.47
69.18
74.00
73.95
74.07
74.41
Mixer-B/16
DeiT-T
76.62
72.17
74.38
72.82
72.24
68.23
74.16
72.82
74.22
74.46
ResNet50
Swin-N
80.38
75.53
78.33
76.05
75.90
73.90
77.58
78.231
77.951
78.64
Con-NcXt-T
Swin-N
82.05
75.53
74.81
75.79
75.48
74.15
77.15
77.00
77.25
77.50
Mixer-B/16
Swin-N
76.62
75.53
76.17
75.81
75.52
73.38
76.26
75.03
76.54
76.63
MLP- based students
ResNet50
ResMLP-S12
80.38
76.65
78,13
76.21
75.45
73.23
77.41
78.23
77.71
78.53
ConvNeXi-T
ResMLP-SIL
82.05
76.65
74.69
75.79
75.28
73.57
76.84
77.23
77.24
77.53
Swin-T
ResMLP-S12
81.38
76.65
76.48
76.15
75.10
73.40
76.67
76.99
77.25
77.31
Optimization. our implementation, we employ different optimizers for training the student models based on their architecture. Specifically, all CNN students trained using the SGD optimizer, while those with ViT or MI.P architecture are trained using the Adam optimizer. For the CIFAR-100 dataset, all models are trained for 300 epochs. When working with the ImageNet-IK K dataset, CNNs are trained for 100 epochs, whereas ViT's and MLPs are trained for 300 epochs.
4.2 Distillation results on ImageNet-1K
We first conduct experiments on the ImageNet- IK dataset. To make comprehensive comparison, we adopt five student models and five teacher models belonging to three different model architecture families, and evaluate KD approaches on fifteen heterogeneous combinations of teacher and student. We report the results in Table 1.
In comparison previous KD methods, our distillation framework achieves remarkable performance improvements. Specifically, when training a student based on CNN architecture, our proposed method achieves performance improvements ranging from 0.20% 0.77% the second-best baselines. When the student model is based on a ViT or MLP architecture, which utilizes image patches as input, our OFA-KD also achieves remarkable performance improvement, with a maximum accuracy gain of 0.71%. %. Significantly, the hint-based baseline FitNet exhibits noteworthy performance when the teacher model is ResNet50 and the student model ViT or MLP. We speculate that designs based on traditional 3 x convolution of ResNet50 have the capability to capture local information, such as texture details, better than ViT/MLP. In the process of "patchify", ViT/MLP might overlook these local details. Therefore, when ResNet50 serves as the teacher, FitNet can provide valuable intermediate representations. We thus report the results logit-based methods obtained by combining FitNet and OFA-KD when adopting ResNet50 as the teacher.
Based on our experimental results, we observe that the second-best performance is not consistently achieved by the same baseline approach. FitNet, vanilla KD, and DIST take turns in attaining the second-best results, indicating that previous KD approaches face challenges in consistently improving performance when distilling knowledge between heterogeneous architectures. However, our proposed approach consistently outperforms all baselines in all cases, showcasing its effectiveness in cross- aooss-rcciteectre distillation.

---

Table 2: KD methods with heterogeneous architectures on CIFAR-100. The best results are indicated in bold, while the second best results are underlined.
From Scratch
hint- bastbased
Logits- baste-
Teacher
Student
T.
S.
FitNet
CC
RKD
CRD
KD
DKD
DIST
OFA
CNN-based students
Swin-T
ResNet18
89.26
74.01
78.87
74.19
74.11
77.63
78.72 78.74
80.26
77.75
80.54
VIT-S
ResNet18
92.04
74.01
77.71
74.26
73.72
76.60
77.26
78.10
76.49
80.15
Mixer-B/16
ResNet18
87.29
74.01
77.15
74.26
73.75
76.42
77.79
78.67
76.36
79.39
Swin-T
Mob
89.26
73.68
74.28
71.19
69.00
79.80
74.68
71.07
72.89
80.98
ViT-S
MobileNetV2
92.04
73.68
73.54
70.67
68.46
78.14
72.77
69.80
72.54
78.45
Mixer-B/IK B/16
MobileNetV2
87.29
73.68
73.78
70.73
68.95
78.15
73.33
70.20
73.26
78.78
ViT-based students
ConvNeXt-T
DeïT-T
88.41
68.00
60.78
68.01
69.79
65.94
72.99
74.60
73.55
75.76
Mixer-B/16
DeïT-T
87.29
68.00
71.05
68.13
69.89
65.35
71.36
73.44
71.67
73.90
ConvNeXt-T
Swin-P
88.41
72.63
24.06
72.63
71.73
67.09
76.44
76.80
76.41
78.32
Mixer- B/16
Swin-P
87.29
72.63
75.20
73.32
70.82
67.03
75.93
76.39
75.85
78.93
MLP-based students
ConvNeXi-T
ResMLP-S12
88.41
66.56
45.47
67.70
65.82
63.35
72.25
73.22
71.93
81.22
Swin-T ResMLP-S12
89.26
66.56
63.12
68.37
64.66
61.72
71.89
72.82
11.05
80.63
4.3 Distillation results on CIFAR-100
In addition to the ImageNet-IK dataset, we also evaluate the proposed method on the CIFAR 100 dataset. We conduct experiments with twelve combinations of heterogeneous teacher and student models, and report the results Table
On this small-scale dataset, the hint-based approaches exhibit inferior performance, particularly when student model is ViT or MLP architecture, highlighting its limitations in distilling knowledge across heterogeneous architectures. For instance, FitNet only achieves a 24.06% accuracy when used combination with ConvNeXt-T teacher and a Swin-P student. By contrast, the OFA-KD method continues to achieve significant performance improvements, ranging from 0.28% to 8.00% over the second-ber best baselines.
Notably, when evaluating on the ImageNet-1 K dataset, DIST achieves the second-best results at most times. while on the CIFAR- dataset, DKD tends to achieve the second-best results more frequently. This is because DIST relaxes the strict prediction imitation by correlation mimicry, which more advantageous for learning from a stronger teacher trained on ImageNet- IK. the other hand, DKD enhances the dark knowledge in teacher predictions, which proves beneficial when working with a smaller teacher trained on CIFAR-100. However, our OFA-KD method can enhance target information adaptively, enabling the student model to consistently achieve the best performance.
4.4 Ablation study
Number and position exit branches. In our OFA-KD method, branches are integrated into the student to facilitate learning at intermediate layers. We conduct experiments to learn the impact of the number and position of branches. As the student models are divided into four stages, we use numbers 1 to to indicate the insertion points of branches. As shown in Table 3, when comparing learning at the last layer with learn- ing from intermediate layers, the ResNet18 student presents
Table 3: Impact of exit branch ber and position on ImageNet- Ø: only learning at the last layer.
T: DeiT-T
T: ResNer50
Stage
S: ResNet18
S: DeiT-T
Ø
70.62
75.18
(1)
70.75
75.35
(2]
70.70
75.21
no preference to the position of the branch, as they all help
[3]
70.78
75.17
improve the performance. However, DeiT-T only performs better when branches are added at the end of stage or stage the evaluation of using multiple branches. The results demonstrate that learning at the end of all the four stages is the best configuration for the OFA-KD method.
(4)
70.78
75.28
(1.4)
71.04
75.52
(1,2,4)
71.08
75.61
(1.2.3.4)
71.34
75.73

---

70.8
71.3
Accuracy
Accunacy
71.0
75.6
70.7
75.4
1.0
1.1
1.2
1.3
1.4
1.5
1.6
17
1.8
1.0
1.1
1.2
1.3
1.5
value (T: DeiT-T, St ResNet18)
vaauue (T: ResNet50, S: DeiT-T)
Figure 3: Ablation study on different modulating parameter ? settings on ImageNet-
Table 5: KD methods with homogeneous architectures ImageNet-1K, T: ResNet34, S: ResNet18.
T.
KD [5]
OFD [56]
Review [43]
CRD [37]
DKD [55]
DIST [9]
OFA
Accuracy
73.31
69.75
70.66
70.81
71.61
71.17
71.70
72.07
72.10
Table 6: Comparison of homogencous and heterogencous teacher on ImageNet- IK.
Teacher
T.
S.RRsNer550)
RKD [37]
Review [43]
CRD [37]
DKD [55]
DIST [9]
OFA
ResNet152
82.83
79.86
79.53
80.06
79.33
80.49
80.55
80.64
ViT-B
86.53
79.86
79.38
79.32
79.48
80.76
80.90
81.33
Table 4: Comparison of scaling factors and clip grad settings on ImageNet-1 The scal- ing factors are directly multiplied to the OFA loss. We adopt ResNet50 teacher and D student with 1.1.
Scale of OFA loss. The modulation parameter is intro- duced at the exponential position of the term + p which has value greater than 1. This increases the overall loss scale. To ensure stable training pro- cess, we compare two schemes: multiplying a scaling factor to the OFA loss and clipping gradient by set- ting maximum norm value. As shown in Table 4, the best configuration is setting the scale factor to and the maximum gradient that searching the optimal scale factor and gradient clipping settings for each combination of teacher and student is expensive, we adopt the aforementioned best settings in all experiments.
Scaling factor
Acc.
Clip grad
Acc.
0.5
75.29
75.39
0.8
75.58
75.61
1.0
75.73
75.52
to 5. Considering
1.2
75.54
5
75.73
1.4
75.64
6
75.32
1.6
75.46
7
75.36
Modulating parameter. The modulation parameter 7 controls the strength of the adaptive target information enhancement. To investigate its impact, compare different configurations of the value setting on ImageNet-IK and present the results in Figure 3. Specifically, when the teacher model is De and the student model is ResNet18, the best student performance is achieved when y is set to 1.4. For the reacher-student pair of ResNet50 and DeiT-T, the optimal configuration for y is 1.1. This difference in optimal values be attributed to the varying strengths of the teacher models. As DeiT-T is weaker teacher model compared to ResNet50, a larger y value required to mitigate the additional interference information. By increasing the y value, the adaptive target information enhancement compensates for the limitations of the weaker teacher model, enabling the student model to better utilize the available information and improve its performance.
Distillation homogeneous architectures. We adopt the commonly used combination of ResNet34 teacher and R student ImageNet-IK IK to further demonstrate the effectiveness of our method across homogeneous architectures. As the results shown in Table 5, the OFA method achieves performance on par with the best distillation baseline.
Homogeneous vs. heterogeneous teachers To assess the impact of utilizing a larger heterogeneous teacher model, we train ResNet50 student with both ResNet152 teacher (homogeneous archi- tecture) and a ViT-B teacher (heterogeneous architecture). As indicated in Table 6, our OFA-KD approach achieves notable 0.41% performance improvement when utilizing the ViT-B teacher compared the ResNet1 teacher. This result underscores the significance of cross-architecture knowledge distillation (KD) in pursuing better performance improvement.

---

5 Conclusion
This paper studies how to distill heterogeneous architectures using hint-based knowledge. We demonstrate that the distinct representations learned by different architectures impede existing feature distillation approaches, and propose one-for-all framework to bridge this gap in cross- architecture KD. Specifically, we introduce exit branches into the student to project features into the logits space and train the branches with the logits output of the teacher. The projection procedure discards architecture-specii specified information in feature, and thus hint-based distillation becomes feasible. Moreover, we propose a new loss function for arost-architectuue distillation to adaptive target information enhancement, which is achieved by adding modulating parameter to the reformulated KD loss. We conduct extensive experiments on CIFAR- 100 and ImageNet- IK. Corresponding results demonstrate the effectiveness of the proposed cross-architecture KD method.
Limitation. While our approach achieves remarkable performance improvements in distilling ResNet50, it is worth noting that for some architectures, e.g., ResNet18, 88, the performance distilled by a heterogeneous teacher is lower than that achieved by a homogeneous teacher. Furthermore, the requirement of finding an optimal setting for the modulating parameter adds level of complexity. Improperly setting this parameter could potentially lead to inferior performance.
Border impact. Our work mainly studies hint-based distillation for heterogeneous architecture and proposes a generic framework. The performance improvement achieved by our OFA-KD demonstrates that there exists useful information between heterogeneous architectures. We hope that our research will bring new perspective to KD community for seeking better aligning approaches and inspire future works for cross-acchitecture distillation.
Acknowledgement
This work is supported by National Key Research and Development Program of China under No. SQ2021 YFC330013 National Natural Science Foundation of China under Grant 61971457. And Chang Xu was supported in part by the Australian Research Council under Projects DP210101859 and FT230100549.
References
[1] Zhiwei Hao, Yong Luo, Zhi Wang, Han Hu, and Jianping An. Cdfkd-mfs: Collahorative data-free knowledge distillation via multi-level feature sharing. IEEE Transactions on Multimedia, 2022.
[2] Zhiwei Hao, Yong Luo, Han Hu, Jianping An, and Yonggang Wen. Data-free ensemble knowledge distillation for mrvacy-consccins multimedia model compression. In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
[3] Hanting Chen, Tianyu Guo, Chang Xu, Wenshuo Li, Chunjing Xu, Chao Xu, and Yunhe Wang. Learning student networks in the wild. In Proceedings the TEEF/CVF Conference Computer Vision and Pattern Recognition, 2021.
[4] Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. 1
[5] Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015. 3,
[6] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference Learning Representations, 2015. 2. 6,
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance convolutional neural networks via attention transfer. International Conference on Learning Representations, 2017. 1,3
[8] Zhiwei Hao. Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang. Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine grained manifold distillation. In Advances in Neural Information Processing Systems, 2022. 2,
10

---

[9] Tao Huang, Shan You, Fei Wang. Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. In Advances in Neural Information Processing Systems, 2022. 2, 3, 6,
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016. 2,6
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 2, 3, 6,
[12] Zhuang Liu, Hanzi Mao. Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition, pages 1196-119866 2022. 2, 6
[13] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkorcit, Mario Lucic, and Alexey Dosovitskiy, Mip-mixer: An all alp-mip architecture for vision. In Neural Information Processing Systems, pages 24261- 24272, 2021. 6,
[14] Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2339-2348, 2020. 2
[15] Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge amalgama- tion from heterogeneous networks by common feature learning. arXiv preprint 1906. 10546, 2019.
[16] Chengehan Shen, Menggi Xue, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation. In Proceedings of the JEEF/CVF International Conference on Computer Vision, 2019. 2
[17] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolies, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, 2021. 4,
[18] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and lincar bottlenecks. Computer Vision and Pottern Recognition, pages 4510-4520, 2018. 2, 6
[19] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. 2
[20] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang. Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision, pages 9992- 10002, 2021. 2, 3,
[22] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: survey. arXiv preprint arXiv:2304.000 0995444400044 2023. 2
[23] Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang- Hire-mlp: Vision mlp via hierarchical rearrangement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2
[24] Hugo Touvron, Piotr Bojanowski, Mathilde Caron. Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-ctficient training. arXiv preprint arXiv:2105.00 0334004 2021. 2, 3, 6
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, pages 5998 6008,6008,000 2017.
11

---

[26] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020. 2
[27] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu. Heewoo Jun, David Luan, et al. Generative pretraining from pixels. International Conference on Machine Learning, 2020.
[28] Chengcheng Wang. Wei He, Ying Nie, Jianyuan Guo, Chuanjian Liu, Kai Han, and Yunhe Wang. Gold- yolo: Efficient object detector via gather-and-distrine mechanism. arXiv preprint arXiv:2309.13331 2023. 2
[29] Wenhai Wang, Enze Xie. Xiang Li. Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In International Conference on Computer Vision, pages 548-558, 2021." 2
[30] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 3
[31] Linfeng Zhang, Jicbo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 3
[32] Mary Phuong and Christoph H Lampert. Distillation-based training for multi-exit architectures. In Proceedings of the IEEE/CVF international conference on computer vision, 2019.
[33] Yunteng Luan, Hanyu Zhao, Zhi Yang, and Yafei Dai. Msd: Mullt-self-distillatiinn learning via multi- classifiers within deep neural networks. arXiv preprint 18X1v:1911.09418, 2019. 3
[34] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, 2019. 3
[35] Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble, Advunces in neural information processing systems, 31, 2018.
[36] Cristian Bucila, Rich Caruana, and Alexandru Niculescu- Model compression. In ACM SIGKDD. pages 535-541, 2006. 3
[37] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 3, 6,
[38] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[39] Andrey Malinin, Bruno Mlodozenice, and Mark J. F. Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020. 3
[40] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. JEEE/CVF International Conference on Learning Representations, 2020. 3.6 6
[41] Junho Yim. Donggyu Joo, Ji-Hoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Computer Vision and Pattern Recognition, 2017. 3
[42] Linfeng Zhang, Yukang Shi, Zuoqiang Shi, Kaisheng Ma. and Chenglong Bao. Task-oriented feature distillation. In Neural Information Processing Systems, 2020. 3
[43] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008 5017,
2021. 3,9 9
[44] Tongzhou Wang, Jun-Yan Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 2018. 3
[45] George Cazenavette, Tongzhou Wang. Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4750-4755 2022.
12

---

[46] Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Advances in Neural Information Processing Systems, 2022.
[47] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems, 2021.
[48] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Mach. Learn. Res., 13:795-828, 2012.
[49] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey F. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, volume 97, pages 3519-3529, 2019. 4
[50] Arthur Gretton, Kenji Fukumizu, Choon Hui Tea, Le Song, Bernhard Schölkopf, and Alexander J. Smola. kernel statistical test independence. Neural Information Processing Systems, pages 592-592, 2007.
[51] Gao Huang. Danlu Chen, Tianhong Li, Felix Wu, Laurens van Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2018.
[52] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6
[53] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei. Imagenet: large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. 6
[54] Baoyun Peng. Xiao Jiaheng Liu. Dongsheng Li, Yichao Wu, Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In JEEESCVF International Conference on Computer Vision, 2019. 6
[55] Borui 7hao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE/CVF Conference Computer Vision and Puttern Recognition, 2022. 6,9
[56] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi, comprehen- sive overhaul of feature distillation. In International Conference on Computer Vision, pages 1921-1930, 2019. 9
13


Reviews:
[{"review_id": "VJOZzdyA9B", "reviewer": "Reviewer_sLfN", "summary": "This paper first demonstrates that there is significant feature divergence of the learned features between heterogeneous teacher and student models, which is a scenario rarely explored in previous knowledge distillation community. And the authors point out that the hint-based methods are ineffective in this cross-architecture distillation. They propose OFA-KD to improve the distillation performance between heterogeneous architectures. It first projects intermediate features into an aligned latent space (the logits space). In addition, they introduce an adaptive target enhancement scheme to prevent the student from being disturbed by irrelevant information. The extensive experiments with various architectures demonstrate the superiority of the OFA-KD framework.", "questions": "The OFA result (ViT-B teacher) in Table 6 of the main paper seems inconsistent with that in Figure 5 of the supplementary material. The accuracy gain of the res50 student is 1.19\\% in the main paper, while in the supplementary material, the accuracy gain is 1.47\\%. Please check it.\n\nAs there are additional branches introduced during the training procedure, it requires more computational resources to train the same student model than using traditional KD methods. Could the authors provide more details about the branches to illustrate the extra resources consumption? For example, I am interested in the number parameters and the FLOPs of the additional branches.\n\nWhy does the paper choose to use depth-width convolutional layers for branches in CNN models, while ViT blocks are used in ViTs and MLPs?\n\nThere are so many hyperparameters such as the scaling factor, the clip grad norm, and the \\gamma value. How to choose these hyperparameters in practice?", "limitations": "Yes. Limitations and border impact are discussed in the conclusion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper studies KD with different architectures. It is an interesting attempt to build a generic framework for distilling students with arbitrary mainstream model architectures, i.e., CNN, ViT, and MLP. Experiment results in both the main paper and the supplementary material demonstrate the necessity of doing cross-architecture distillation.\n\nThe performance improvement of OFA is remarkable. On ImageNet-1K, the maximum improvement is 0.8\\%, and on CIFAR-100, the maximum improvement is 5.0\\%, even compared with the most recent KD baselines, DIST and DKD. \n", "weaknesses": "The additional branches will increase the training cost. I think the authors should give more analyses on this.\n\nThe reported results of using res50 as the teacher on ImageNet-1K are obtained by using both FitNet and OFA-KD (Table 1). I think this is not a fair comparison. For example, what’s the result of FitNet + DKD/DIST?\n\nMissing some references also adopting multi-branch architecture:\n[A] Be your own teacher: Improve the performance of convolutional neural networks via self distillation, ICCV 2019\n[B] Distillation-based training for multi-exit architectures, ICCV 2019\n[C] MSD: Multi-Self-Distillation Learning via Multi-classifiers within Deep Neural Networks, arXiv 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1:** The additional branches will increase the training cost.\n> \n> **Question 2:** Could the authors provide more details about the branches\n\n**Response to the weakness 1 & question 2:** Thanks for your valuable comments. The inclusion of extra branches inevitably demands more computational resources during training. To mitigate this concern, we have employed a streamlined design for branch architecture. For example, in CNN models, we've reduced the number of channels in depth-wise convolution layers within the branches, and for ViT and MLP models, we've introduced patch merging blocks before vision transformer blocks within the branches to reduce the number of tokens that need to be processed.\n\nWe've chosen three distinct student architectures, i.e., ResNet18, DeiT-T, and ResMLP-S12, and conducted a comparison on the number of parameters and FLOPs between their backbones and exit branches. As depicted in the table below, considering the additional FLOPs introduced by exit branches, these branches demonstrate enhanced accuracy improvement in comparison to the backbone architectures. This improvement results in only a marginal increase in training cost, thereby contributing minimal augmentation to overall training expenses. Moreover, given that the primary objective of KD is to compress a pre-trained model for easier deployment, a slight elevation in training cost remains acceptable. Our OFA-KD approach enhances the performance of student models while incurring no additional inference cost.\n\n| Teacher  |  Student   | Student Params | Student FLOPs | Branch Params | Branch FLOPs |\n| :------: | :--------: | :------------: | :-----------: | :-----------: | :----------: |\n|  DeiT-T  |  ResNet18  |     11.69M     |     1.83G     |     3.04M     |    0.08G     |\n| ResNet50 |   DeiT-T   |     5.72M      |     1.26G     |     4.48M     |    0.19G     |\n| ResNet50 | ResMLP-S12 |     15.35M     |     3.01G     |    16.32M     |    0.74G     |\n\n> **Weakness 2:** result of FitNet + DKD/DIST?\n\n**Response to the weakness 2:** FitNet appears to be particularly effective as a knowledge distillation approach when utilizing ResNet50 as the teacher and ViT/MLP as the student. However, these promising outcomes are not observed with other model combinations. As a result, we exclusively present results obtained by combining OFA and FitNet for these specific teacher-student pairs. To ensure a fair comparison, we proceed to conduct experiments merging FitNet with DKD and DIST for these models, and subsequently report their respective performances in the table below.\n\n| Teacher  |  Student   | DKD+FitNet | DIST+FitNet | OFA+FitNet |\n| :------: | :--------: | :--------: | :---------: | :--------: |\n| ResNet50 |   DeiT-T   |   75.60    |    75.13    |   76.55    |\n| ResNet50 |   Swin-N   |   78.23    |    77.95    |   78.64    |\n| ResNet50 | ResMLP-S12 |   78.23    |    77.71    |   78.53    |\n\nAs the results demonstrate, even though DKD and DIST also yield improved performance when integrated with FitNet, our OFA-KD method consistently outperforms them. This underscores the efficacy of our approach. We will incorporate the new results obtained through FitNet + DKD/DIST into our next version.\"\n\n> **Weakness 3:** Missing some references\n\n**Response to the weakness 3:** Thank you for your valuable suggestion. We have appropriately integrated these references into the draft of our next version.\n\n> **Question 1:** the inconsistent result.\n\n**Response to the question 1:** Thank you for your thorough review of our paper. There was an error in our main paper regarding the accuracy gain, which should be 1.47%. We have rectified this mistake in the revised version.\n\n> **Question 3:** principle of choosing branches\n\n**Response to the question 3:** Our decision to adopt the CNN branch architecture for CNN models and the ViT branch architecture for ViT/MLP models is grounded in our belief that a heterogeneous pairing of branches and backbones could potentially hinder student performance. In order to substantiate this perspective, we conduct a series of experiments aimed at evaluating the outcomes of employing diverse branch and backbone configurations.\n\nThe results below show that a homogenous combination of branch and backbone consistently yielded superior performance. As a result, we embraced this unified configuration, aligning with our objective of improving student model performance. Furthermore, the incorporation of depth-wise convolutional layers within the CNN branch architecture serves to mitigate the additional training cost attributed to branches. These convolutional layers have also demonstrated their effectiveness in several related studies [1,2].\n\n|  Teacher   | Student  | Branch architecture | Top-1 |\n| :--------: | :------: | :-----------------: | :---: |\n|   DeiT-T   | ResNet18 |CNN| 71.34 |\n|   DeiT-T   | ResNet18 |ViT| 70.82 |\n| ConvNeXt-T |  DeiT-T  |CNN| 74.27 |\n| ConvNeXt-T |  DeiT-T  |ViT| 74.41 |\n\n[1] Be your own teacher: Improve the performance of convolutional neural networks via self distillation\n[2] Task-oriented feature distillation\n\n> **Question 4:** How to choose these hyperparameters\n\n**Response to the question 4:** In our ablation study, we evaluated the impact of these hyperparameters. Notably, we discovered that the scaling factor can be disregarded by simply setting it to 1, as this adjustment is sufficient to achieve satisfactory results. As for the clip grad norm, the best result is obtained when the value is set to 5, which is a common setting used by many other related works. So we can simply adopt the setting of clip_grad=5 for all combinations of teacher and student. While there isn't a universally applicable setting for the parameter $\\gamma$, we can determine its optimal value using a validation set, considering that the other hyperparameters remain fixed."}, {"review_id": "hOA9ydd6Nb", "reviewer": "Reviewer_d3Lq", "summary": "This paper tackles the problem of cross-architecture distillation, that is, the teacher and the student in KD are of different model architectures.\n\nBy using centered kernel alignment, the authors observe that features learned by models of different architectures shows significant feature divergence, indicating that previous hint-based methods are not suit for this task.\n\nTo bridge this gap, the authors propose a simple yet effective one-for-all KD framework called OFA-KD. Specifically, they project intermediate features into an aligned latent space to discard architecture-specific information.\n\nAnd an adaptive target enhancement scheme is proposed to prevent the student from being disturbed by irrelevant information.\n\nThe authors conduct experiments on CIFAR-100 and ImageNet-1k benchmarks with CNN, ViT and MLP architectures to demonstrate the effectiveness of the proposed method.", "questions": "Please refer to \"Weaknesses\"", "limitations": "Yes. Limitations and border impact are discussed in the conclusion.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Motivation: The motivation is clear. Cross-architecture distillation provides more feasible options for teacher models, as it may not always be possible to find a superior teacher model with a homogeneous architecture.\n\nOriginality: Learning in an aligned latent space is the first application in cross-architecture distillation, and the adaptive target information enhancement loss is novel.\n\nQuality: Written of the paper is good. Sufficient experiments and ablation studies with other methods and the proposed variants are implemented.\n\nClarity: The paper consists of text explanations and an illustration of the OFA-KD framework and the proposed loss.\n\nSignificance: The paper solves the problem of cross-architecture distillation, expanding feasible options for teacher models in practice, and improves the accuracy of distilled student models.", "weaknesses": "1. In the CKA analysis, it seems that when comparing features of models of the same architecture, the authors just using features of one model in both x-axis and y-axis, as the corresponding heatmaps are symmetry. If using two models, such as ResNet18 vs. ResNet34 or two ResNet18 trained with different initialization, would the results be different?\n\n2. The authors propose to using an aligned latent space for cross-architecture distillation, and adopting the logits space as a special instance. I wonder whether there are any other possible choices of the space, and what will happen when using them.\n\n3. Introduction of the adaptive target information enhancement loss is a bit complicated. Maybe it is possible to simplify the notations.\n\n4. What is the principle of designing the branches, such as its architecture and layer number? Have the authors tried other branch designs?\n\nAdditionally, there are some grammar mistakes and typos: \n\n- line 192, \"slow\" -> \"slowly\"\n- line 193, \"enhance\" -> \"enhancement\"\n- line 255, \"additional\" -> \"addition\"\n- line 257, \"reports\" -> \"report\"\n- line 299, \"strengthen\" -> \"strength\"\n- line 311, \"methods\" -> \"method\"\n- line 339, \"disitllation\" -> \"distillation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1:** In the CKA analysis, it seems that when comparing features of models of the same architecture, the authors just using features of one model in both x-axis and y-axis, as the corresponding heatmaps are symmetry. If using two models, such as ResNet18 vs. ResNet34 or two ResNet18 trained with different initialization, would the results be different?\n\n**Response to the weakness 1:** Thank you for your valuable suggestion. We have expanded our CKA analysis to encompass various model architectures, and presented these additional results in the PDF uploaded in \"Author Rebuttal\" section. We will also incorporate these results into our supplementary material. To conduct CKA analysis, specifically, we use ResNet18 and ResNet34 as CNN models, Swin-Small and Swin-Base as ViT models, Mixer-B/16 and ResMLP-S12 as MLP models. The results reveal that homogeneous models, such as a ResNet18 model and a ResNet34 model, exhibit similar feature learning at comparable positions within the model. Conversely, heterogeneous models continue to manifest distinct feature learning patterns. This corroborates the conclusions drawn in our main paper.\n\n> **Weakness 2:** The authors propose to using an aligned latent space for cross-architecture distillation, and adopting the logits space as a special instance. I wonder whether there are any other possible choices of the space, and what will happen when using them.\n\n**Response to the weakness 2:** The utilization of an aligned latent space aims to mitigate the adverse effects of disparate information present in the features learned by divergent teacher and student models. Therefore, the guiding principle in selecting an aligned latent space is to retain shared information while discarding irrelevant details. The logits space satisfies these prerequisites and is straightforward to implement, thus we have adopted it as the aligned latent space in our experiments.\n\nWhile we acknowledge that there might exist more efficient latent spaces for cross-architecture KD. For example, the use of manifold space as the latent space [1], which exclusively compares the relationships among learned features while remaining insensitive to the absolute feature values. However, crafting an optimal latent space remains a challenging task, given the absence of well-defined principles for measurement. Consequently, we consider this issue a subject for future research exploration.\n\n[1] Hao, Zhiwei, et al. \"Learning efficient vision transformers via fine-grained manifold distillation.\" Advances in Neural Information Processing Systems 35 (2022): 9164-9175.\n\n> **Weakness 3:** Introduction of the adaptive target information enhancement loss is a bit complicated. Maybe it is possible to simplify the notations.\n\n**Response to the weakness 3:** Thank you for your suggestion. We have simplified the notations and incorporated them into our next version.\n\n> **Weakness 4:** What is the principle of designing the branches, such as its architecture and layer number? Have the authors tried other branch designs?\n\n**Response to the weakness 4:** We initially outline our approach to determining the block number within each branch. In our experimental setup, we partition all models into four stages. For pyramid architectures, this division is straightforward (four down-sampling layers). However, for models like DeiT, we adopt the division scheme used in the Swin Transformer architecture. For instance, DeiT-T and Swin-T both consist of 12 blocks. Thus, we divide DeiT-T into four stages, each comprising 2, 2, 6, and 2 blocks, mirroring the structure of Swin-T.\n\nRegarding the branch architecture, we adopt depth-wise convolutional blocks for CNN models and vision transformer blocks for ViT and MLP models. This design is rooted in our belief that a mismatch between branch and backbone architectures could hinder student performance. To verify this assumption, we conducted experiments comparing various branch architectures. The results, as shown in the table below, underscore that homogeneous pairings of branch and backbone outperform heterogeneous pairings. Additionally, our supplementary material presents PyTorch-style pseudocode for constructing branches.\n\n|  Teacher   | Student  | Branch architecture | Top-1 |\n| :--------: | :------: | :-----------------: | :---: |\n|   DeiT-T   | ResNet18 |         CNN         | 71.34 |\n|   DeiT-T   | ResNet18 |         ViT         | 70.82 |\n| ConvNeXt-T |  DeiT-T  |         CNN         | 74.27 |\n| ConvNeXt-T |  DeiT-T  |         ViT         | 74.41 |\n\n> **Weakness 5:** Additionally, there are some grammar mistakes and typos:\n\n**Response to the weakness 5:** Thank you for thoroughly reviewing our submission and pointing out the mistakes. We have taken great care to address these issues in our next version."}, {"review_id": "Pf1O31aEct", "reviewer": "Reviewer_R7Hg", "summary": "This paper introduces a new method to distill knowledge between heterogeneous models named OFD-KD. This paper proposes to project the intermediate features into logits for distillation. A new loss function is also introduced in this paper to adaptively enhance the target information. Extensive experiments verify the effectiveness of this method.", "questions": "1. In Tables 1 and 2, the results of many distillation methods are not as high as the baseline results, please explain your implementation details and why this is the case.\n2. Is there any quantitative indicator to prove that the method proposed in this paper really bridges the gap of heterogeneous distillation?\n", "limitations": "See weaknesses and questions.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is easy to understand and clearly written.\n2. This paper uses CKA to visualize the differences between CNN, VIT, and MLP.\n3. Extensive experiments are conducted to verify the effectiveness of this method.", "weaknesses": "1. The improvement is relatively minor in ImageNet-1K. The most improvement is around 0.1%-0.3% in Tables 1, 5, and 6. This method does not exhibit a clear advantage over other techniques.\n2. The architecture employed in this work has been extensively explored. This form of intermediate logit supervision has been widely used in BYOT, DCM, DKS, and other methods. This study does not offer any significant novelty in the context of distillation.\n3. Table 1 fails to compare the latest feature-based distillation method, particularly SemCKD, which is a method dedicated to heterogeneous distillation. Thus, the experimental comparisons presented in this paper lack meaningfulness. The authors should compare with the recent state-of-the-art hint-based methods.\n4. The author's use of CKA comparison is unnecessary, given the obvious architectural differences between CNN, MLP, and VIT. The differences between these architectures have been discussed in many works.\n5. Please provide the results of OFA during distillation between heterogeneous networks such as VGG, ResNet, ShuffleNet, and MobileNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer R7Hg part (1/2)\n\n**Response to the weakness 1:** In Table 5, the improvement when compared to methods tailored for homogeneous architectures, such as ResNet34-ResNet18, is indeed marginal. However, when evaluating heterogeneous pairs, as shown in Table 1, our OFA-KD consistently outperforms the second-best baseline by a range of 0.1% to 0.7%. Notably, it's worth noting that the second-best result is not consistently obtained by the same baseline.\n\nGiven the intricate nature of cross-architecture KD, where architecture-specific nuances can hinder student learning, previous methods have struggled to consistently achieve satisfactory results. For instance, while DIST achieves the second-best performance across multiple teacher-student combinations, it slightly lags behind our proposed method. Nonetheless, on certain other teacher-student pairs, like the case of Swin-T teacher and ResNet18 student, DIST's performance significantly trails that of OFA. In contrast, OFA consistently attains the top performance, underscoring its applicability as a universal solution for both heterogeneous and homogeneous KD scenarios.\n\n**Response to the weakness 2:** Intermediate supervision is widely used in hint-based learning approaches like FitNet. BYOT and DCM's final structures, with the addition of auxiliary classifiers, bear similarities to our OFA. However, our focus is primarily on cross-architecture KD. Given the notable divergence in features learned by heterogeneous models, specialized information filtering method is essential to align features. We aim to emphasize the importance of noting that directly mimicking the feature space of a heterogeneous teacher can lead to suboptimal results. Instead, we find it more effective to transfer mismatched representations into the aligned logits space through the integration of additional exit branches within the student model. We leave the exploration of more efficient modules or alternative spaces for aligning features for future research.\n\nIn essence, intermediate logits supervision can be viewed as a specialized approach for aligning heterogeneous features. Furthermore, we introduce a novel distillation loss to enhance target information adaptively by introducing a modulating parameter into the original KD loss. Our ablation study validates the efficacy of this design. We will provide a summary of these architecture designs in the related work section, including BYOT/DCM/DKS.\n\n**Response to the weakness 3:** Thanks for your valuable comments. Firstly, we would like to clarify the scope of our usage of the term \"heterogeneous.\" In our paper, we specifically consider CNN, ViT, and MLP models as examples of heterogeneous architectures. However, in the context of SemCKD, the definition of heterogeneous models is broader. For instance, SemCKD categorizes models like VGG and ResNet as heterogeneous, whereas our paper treats them as homogeneous CNN models.\n\nWe have also carried out experiments using SemCKD. Given that SemCKD is tailored for CNN models, we transformed intermediate features of ViT models into the CNN format using an \"unpatchify\" operation. Our experiments involve two teacher-student pairs: DeiT-T - ResNet18 and ConvNeXt-T - DeiT-T. To facilitate comparison, we present the results of SemCKD alongside KD and OFA on ImageNet-1K.\n\n|Teacher|Student|KD|SemCKD|OFA\n|:-:|:-:|:-:|:-:|:-:\n|DeiT-T|ResNet18|70.22|70.12|71.34\n|ConvNeXt-T|DeiT-T|74.00|71.96|74.41\n\nThrough the above outcomes, within the more stringent \"heterogeneous\" context, SemCKD demonstrates lower performance compared to our OFA. We conjecture that the coarse \"unpatchify\" operation hinders effective information transfer. If we intend to extend the applicability of SemCKD to encompass generalized cross-architecture KD, a more meticulous design is imperative to achieve semantic calibration. We believe this as an important avenue for future research and plan to incorporate the discussion in our final revision.\n\n**Response to the weakness 4:** We employed CKA to illustrate that heterogeneous architectures learn distinct features, as indicated by their CKA similarity. This served as the foundation for devising feature alignment methods in the context of cross-architecture KD, enhancing the clarity and logical flow of the paper. While some existing works delve into different architectures, very few have simultaneously compared all three types. Hence, our analysis is more comprehensive. Furthermore, we haven't listed the CKA analysis as a specific contribution in our paper.\n\n**Response to the weakness 5:** Thanks for the valuable suggestions. Considering rebuttal time constraints, we restricted our experimentation to the ResNet50 teacher and MobileNet-v1/VGG-16 student pair. The particular ResNet-MobileNet teacher-student combination has been extensively utilized in numerous related studies, making it convenient to obtain baseline results. And we conducted the ResNet-VGG experiments ourselves to establish the baseline results. Following the recipe in DKD, we trained the student using our OFA and present the results in the following table.\n\n|Teacher|Student|KD|OFD|Review|CRD|DKD|DIST|OFA|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|ResNet50|MobileNet-v1|70.68|71.25|72.56|71.37|72.05|73.24|73.28|\n|ResNet50|VGG-16|73.96|73.31|74.08|74.02|74.69|74.75|74.88|\n\nFrom the results, the performance of OFA is comparable to the best baseline DIST with a slight accuracy gain of 0.04%. Notably, models mentioned above are considered homogeneous in our paper, as they all belong to the CNN architecture. Our OFA method is primarily tailored for cross-architecture KD and consistently surpasses previous baseline approaches. However, even when both the teacher and student belong to homogeneous architectures, OFA still manages to achieve competitive performance, underscoring its effectiveness.\n\n### ***The part (2/2) can be found in \"Author Rebuttal\" at the top of this page.***"}, {"review_id": "lfwO7mpCAK", "reviewer": "Reviewer_FwZi", "summary": "In this paper, the authors propose a new heterogeneous knowledge distillation approach. The core idea is to map the intermediate layer features of the network to a unified logit space to eliminate feature mismatches caused by different structures. The author conducts thorough experiments on distilling between CNN, ViT, and MLP networks. According to the experimental results in the paper, the proposed approach yields promising results.", "questions": "Please address the problem in Weakness", "limitations": "Potential negative societal is not applicable.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The experiments in this paper are comprehensive, considering distillation between various networks with different structures, and conducting experiments on both CIFAR-100 and ImageNet datasets.\n2. The proposed method is reasonable, as projecting network features onto a latent space to avoid the alignment issue of distillation between networks with different structures may indeed lead to better results.\n3. According to the experimental results, the proposed method achieves good performance in various experiments.", "weaknesses": "1. The newly proposed method is very similar to deep supervision in that both involve adding an auxiliary head to the intermediate layer to learn the final output. The only difference is that deep supervision previously learned hard labels, while the proposed method learns soft labels from the teacher. However, this paper does not discuss the differences between this method and deep supervision, including theoretical and experimental results.\n2. In some experimental settings, the improvement brought by the proposed method is very limited. And the comparison is not comprehensive. For example, in Table 1 and Table 2, OFD, Review, and CRD's results are missing.\n3. In addition, in Table 1, some experiments are combined with FitNet while others are not. Although the authors have provided an explanation, it still feels strange that we cannot conclude that ResNet50's features are more applicable just because it is the most commonly used network. And why ResNet50 is not adopted as the teacher in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1:** The newly proposed method is very similar to deep supervision\n\n**Response to the weakness 1:** Thanks for your suggestions in improving quality of our work. Deep supervision [1,2,3] introduces intermediate supervision during training to mitigate the gradient vanishing/exploding problem in early era of deep learning. Conversely, feature distillation involves transferring knowledge at intermediate layers, aiming to equip the student with a more comprehensive understanding of the teacher's knowledge. Our OFA method falls within this category of approaches. To conduct a comparison between deep supervision method an OFA, we trained a ResNet18 model using deep supervision by replacing the OFA loss with a hard-label cross-entropy loss. We evaluated its performance against that achieved by OFA with a DeiT-T teacher. Notably, the position and number of auxiliary branches used in deep supervision remained consistent with OFA.\n\n|Teacher|Student|Method|Top-1|\n|:-:|:-:|:-:|:-:|\n|DeiT-T|ResNet18|OFA|71.34|\n|-|ResNet18|Deepsupervision|70.06|\n\nBased on the results above, it is evident that the model trained using our OFA framework outperforms the model trained with deep supervision by a significant margin. This disparity in performance underscores the superiority of our OFA in comparison to deep supervision.\n\n[1] Wang, Liwei, et al. \"Training deeper convolutional networks with deep supervision.\"\n\n[2] Zhang, Linfeng, et al. \"Contrastive deep supervision.\"\n\n[3] Li, Renjie, et al. \"A comprehensive review on deep supervision: Theories and applications.\"\n\n> **Weakness 2:** In some experiment improvement is limited. Comparison is not comprehensive.\n\n**Response to the weakness 2:** Thanks for your valuable comments. Referring to the results in our main paper, our OFA method exhibits only a slight performance gain over the **second-best** baseline across certain teacher-student combinations. Notably, the second-best result isn't consistently obtained through the same baseline, signaling the challenges associated with generalizing existing methods to the context of heterogeneous KD. In contrast, our OFA method consistently outperforms all other baselines across all scenarios, yielding satisfactory results. This consistently strong performance underscores its generic applicability for cross-architecture KD.\n\nThe OFD method is tailored for CNN models, leveraging features between Batch Normalization (BN) and ReLU layers for distillation. However, ViT/MLP architectures generally lack such intermediary positions, posing a challenge to the application of OFD. Furthermore, previous works like DIST and DKD have demonstrated their superiority over OFD. Hence, we have opted not to include OFD in our analysis.\n\nTo assess the performance of Review, we selected two teacher-student pairs and conducted experiments on the ImageNet-1K dataset. The outcomes, as illustrated in the table below, reveal that Review falls short of our OFA methods by a noticeable margin. Notably, Review mandates features with dimensions of (N, C, H, W), where N, C, H, and W denote batch size, the number of channel, height, and width, respectively. Since ViT-generated features do not possess this structure, an \"unpatchify\" operation is required to transform them. We hypothesize that this transformation contributes to the less-than-ideal results obtained using Review. We intend to incorporate this discussion into our final revision.\n\n|Teacher|Student|KD|Review|OFA|\n|:-:|:-:|:-:|:-:|:-:|\n|DeiT-T|ResNet18|70.22|70.28|71.34|\n|ConvNeXt-T|DeiT-T|74.00|68.10|74.41|\n\nAs for CRD, we have conducted the missed experiments on CIFAR-100, as shown in the following table. And we will include these results to our final revision.\n\n|Teacher|Student|CRD|\n|:-:|:-:|:-:|\n|Swin-T|ResNet18|77.63|\n|ViT-S|ResNet18|76.60|\n|Mixer-B/16|ResNet18|76.42|\n|Swin-T|MobileNetV2|79.80|\n|ViT-S|MobileNetV2|78.14|\n|Mixer-B/16|MobileNetV2|78.15|\n|ConvNeXt-T|DeiT-T|65.94|\n|Mixer-B/16|DeiT-T|65.35|\n|ConvNeXt-T|Swin-P|67.09|\n|Mixer-B/16|Swin-P|67.03|\n|ConvNeXt-T|ResMLP-S12|63.35|\n|Swin-T|ResMLP-S12|61.72|\n\n> **Weakness 3:** In Table 1, some experiments are combined with FitNet while others are not. Why ResNet50 is not adopted in Table 2?\n\n**Response to the weakness 3:** Thanks for your professional concern. To ensure a more fair comparison, we have chosen the two most competitive baselines, i.e., DKD and DKD, and have integrated them with FitNet to train the student. The outcomes are illustrated in the table below.\n\n|Teacher|Student|DKD+FitNet|DIST+FitNet|OFA+FitNet|\n|:-:|:-:|:-:|:-:|:-:|\n|ResNet50|DeiT-T|75.60|75.13|76.55|\n|ResNet50|Swin-N|78.23|77.95|78.64|\n|ResNet50|ResMLP-S12|78.23|77.71|78.53|\n\nThe results indicate that DKD and DIST achieve satisfactory performance when integrated with FitNet. Nevertheless, our OFA method surpasses them, thereby highlighting the effectiveness of our approach.\n\nWe speculate that designs based on traditional 3x3 convolution like ResNet50 have the capability to capture local information, such as texture details, better than ViT/MLP. In the process of \"patchify\", ViT might overlook these local details. Therefore, when ResNet50 serves as the teacher, FitNet can provide valuable intermediate representations. And we will correct the original statements in Line 244-246 of the main paper. \n\nRegarding the selection of ResNet50 as the teacher in Table 2 (experiments conducted on CIFAR-100), we conduct experiments to compare the performances between FitNet and OFA on two heterogeneous teacher-student pairs, i.e., ResNet-DeiT and ResNet-ResMLP.\n\n|Teacher|Student|FitNet|OFA|OFA+FitNet|\n|:-:|:-:|:-:|:-:|:-:|\n|ResNet50|DeiT-T|75.62|75.88|76.14|\n|ResNet50|ResMLP-S12|73.67|74.38|74.50|\n\nAs the results shown in the above table, the integration of OFA and FitNet yields further enhancement in the performance of the student model. We will include the remaining omitted results, where ResNet50 serves as the teacher, and incorporate them into our final revision."}]

Author rebuttal:
# Response to all reviewers

We thank all the reviewers for their elaborate and constructive feedback. Their valuable suggestions help improve the quality of our paper greatly.

### **Response to Reviewer R7Hg part (2/2)**

**Response to the question 1:** In our experiments, we primarily utilize the training script from the timm library to train the student models, while also implementing the KD loss of the baselines following the DKD approach. However, due to the distinct nature of features learned by heterogeneous architectures, certain differences arise. For example, the feature shape of a CNN is of size (N, C, H, W), while that of a ViT is denoted as (N, L, D), where N indicates the batch size, C, H, and W refer to the channel, height, and width of the CNN model's feature map respectively, and L and D denote the patch number and embedding dimension of the ViT/MLP model's feature map. To apply previous feature distillation methods designed for CNN models, we need to transform the feature map of the ViT/MLP model into the CNN-style (shape) feature through an "unpatchify" operation. However, we speculate that this operation might be overly simplistic and overlooks intrinsic features specific to the ViT/MLP model's learned features. Consequently, some feature distillation methods yield suboptimal results compared to the baseline. Furthermore, we believe that more effective approaches can be developed to adapt existing feature distillation methods to cross-architecture KD, but we leave this question for future research.

Generally speaking, the performances from logits-based algorithms are superior to those from hint-based algorithms. This observation reinforces our conclusion that for heterogeneous architectures, directly learning the intermediate features of the teacher might lead to sub-optimal results. However, there are a few instances where logits-based methods fall short of the baseline (student trained from scratch) performance. This discrepancy could potentially be attributed to the distinct inductive biases of CNN and ViT architectures, which drive them toward diverse destinations and result in dissimilar distributions. As discussed in Lines 173-175 of our main text, this phenomenon motivates us to propose the adaptive target information enhancement KD loss.

**Response to the question 2:** We mainly adopt the top-1 accuracy as the measurement to evaluate different methods in our paper. Existing methods cannot achieve consistent top-1 accuracy improvement across all tested combinations of teacher and student. However, our OFA-KD outperforms those baselines consistently in terms of top-1 accuracy. This is an evidence that demonstrating OFA-KD is a generic method for cross-architecture distillation.

In this paper, we predominantly employ top-1 accuracy as the evaluation metric to assess various methods. The previous approaches do not uniformly achieve improvements in top-1 accuracy across all tested teacher-student combinations. Conversely, our OFA-KD consistently outperforms these baselines in terms of top-1 accuracy. This consistent superiority stands as evidence that underscores OFA-KD's effectiveness as a method for cross-architecture distillation. While we haven't identified another quantitative indicator at present, this area presents an intriguing avenue. Exploring more comprehensive ways to assess the performance of KD methods is a topic we view as a potential focus for future research.

Meta-review: This work first demonstrated the existence of feature divergence between heterogeneous teacher and student models, which is rarely discussed in previous knowledge distillation methods. Based on this observation, the authors propose a one-for-all KD framework (OFA-KD) where the intermediate features will be projected into an aligned latent space to discard architecture-specific information, and an adaptive target enhancement scheme is proposed to prevent the student from being disturbed by irrelevant information. Despite the additional training costs and the necessary need for hyper-parameter tuning, most reviewers agreed with the contribution of this paper to the community.
Decision: Accept (poster)