Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation
Authors: Martin Zong
Zengyu Qiu
Xinzhu Ma
Kunlin Yang
Chunya Liu
Jun Hou
Shuai Yi
Wanli Ouyang
Venue: ICLR 2023 poster (2023)
Keywords: Knowledge Distillation

Abstract: Knowledge distillation (KD) has shown very promising capabilities in transferring learning representations from large models (teachers) to small models (students). However, as the capacity gap between students and teachers becomes larger, existing KD methods fail to achieve better results. Our work shows that the 'prior knowledge' is vital to KD, especially when applying large teachers.  Particularly, we propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. This means that our method also takes the teacher's feature as `input', not just `target'. Besides, we dynamically adjust the ratio of the prior knowledge during the training phase according to the feature gap, thus guiding the student in an appropriate difficulty. To evaluate the proposed method, we conduct extensive experiments on two image classification benchmarks (i.e. CIFAR100 and ImageNet) and an object detection benchmark (\i.e. MS COCO). The results demonstrate the superiority of our method in performance under varying settings. Besides, our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers. More importantly, DPK provides a fast solution in teacher model selection for any given model. Our codes will be publicly available for reproducibility.

Published as a conference paper at ICLR 2023
BETTER TEACHER BETTER STUDENT: DYNAMIC PRIOR KNOWLEDGE FOR KNOWLEDGE DISTILLATION
Martin Zong"), Zengyu Q Xinzhu Ma* Kunlin Yang"),, Chunya Liu', Jun Hou', Shuai Yi and Wanli Ouyang
1 "SenseTimee Research, Shanghai AI Lab, The University of Sydney
Izangdaaming, giuzengye, yangkunlin, Licchurya, houjun, pishuoi (esensetima.com Ixinahu.ma, wanll.ouy Bul..ooyan.. ang @sydney. au
ABSTRACT
Knowledge distillation (KD) has shown very promising capabilities in transferring learning representations from large models (teachers) small models (students). However, as the capacity gap between students and teachers becomes larger, ex- isting KD methods fail to achieve better results. Our work shows that the 'prior knowledge' is vital to KD, especially when applying large teachers. Particularly, we propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. This means that our method also takes the teacher's s feature as 'input'. not just 'target'. Besides, we dynamically adjust the ratio of the prior knowledge during the training phase according to the feature gap, thus guiding the student in an appropriate difficulty. To evaluate the proposed method, classification benchmarks (i.e. CIFAR 100 and ImageNet) and an object detection benchmark (i.e. MS COCO). The results demonstrate the superiority of our method in performance under varying settings. Besides, our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers. More importantly, DPK provides a fast solution in teacher model selection for any given model. Our code will be released at https::: http//gttttttt com htt.c/cco//caiuce
conduct extensive experiments on two image
INTRODUCTION
Tremendous efforts have been made in crafting lightweight deep neural networks applicable to the real-world scenarios. Representative methods include network pruning (He et al. 2017). model
quantization (Habi et al. 2020), neural architecture search (NAS) (Wan et al. 2020). and knowledge distillation (KD) [Bucilua et al. 2006 Hinton et al. 2015), etc. Among them, KD has recently emerged one of the most flourishing topics due to effectiveness (Liu et al.
2021a 7hao et al.)
2022
Chen et al. 2021 1 Hco et al. (019a) and wide applications (Yang et al. 2022a Chong et al.
2022
Liu et al. 2019a] Yim et al. 2017b] Zhang & Ma 2020).
Particularly, the core idea of KD is to transfer the distilled knowledge from a well-performed but cumbersome teacher to a compact and lightweight student. Based on this, numerous methods have been proposed and achieved great success. However, with the deepening of research, some related issues are also discussed. In particular, several works (Cho & Hariharam] 2019 Mirzadeh et al.) 2020) Hinton et al.] 20155. Liu et al. 2021a) report that with the increase of teacher model in performance, the accuracy of student gets saturated (which might be unsurprising). To make matters worse, when playing the role of a teacher, the large teacher models lead to significantly worse performance than the relatively smaller ones. For example, as shown in Fig I ICKD (Liu et al.) 222 la), a strong baseline model which also points out this issue, performing better under the guidance of small teacher models, whereas applying a large model as the teacher would considerably degrade the student performance.
Same as (Cho & Hariharan 2019 Mirzadeh et al. 2020). attribute the cause of this issue to the capacity gap between the teachers and the students. More specifically, the small student is
"Equal contribution. CCoressponingg author.

---

Published as a conference paper at ICLR 2023
hard to 'understand' the high-order semanties extracted by the large model. This problem will be exacerbated when applying larger teachers, and it makes the student's i accuracy inversely correlated with the capacity of the teacher model"] Note that this problem also exists for humans, and human teachers often tell students some prior knowledge to facilitates their learning in this case. Moreover, the experienced teachers also adjust the amounts of provided prior knowledge accordingly for different students to fully develop their potentials.
Inspired by the above observations from hu- man teachers, we propose the dynamic prior knowledge (DPK) framework for feature distil- lation. Specifically, to provide the prior knowl- edge to the student, we replace student's s features at some random spatial positions with corre- sponding teacher features at the same positions. Besides, we further design a VIT (Dosovitskiy ct al. stylee module to fully integrate this prior knowledge' with student's 's features. Fur- thermore, our method also dynamically adjusts the amounts of the prior knowledge, reflected in the proportion of teacher features in the hybrid feature maps. Particularly, DPK dynamically computes the differences of features between the student and the teacher in the training phase, and updates the ratio of feature mixtures accord- ingly. this way, students always learn from the teacher at appropriate difficulty, thus al- leviating the performance degradation issue.
Distilied Student (ICKD)
ImageNet Top-1 Accuracy (96)
Distilled Student (Ours) )
74.0
73:29
72.99
73.0
72.86
Res-152
72.51
Res-100
72.5
Res-50
Res-34
72:31
72.0
72.19
72,17
71,86
73.31
76.13
77.37
78.31
Top-1 Accuracy of Teachers (96)
Figure 1: Top-1 accuracy of ResNet-18 w.r.f var- ious teachers on ImageNet. Different from the baseline model (ICKD (Liu et al., (021a)), method shows better performance and makes the performance of student positively correlated with that of the teacher.
We evaluate DPK on two image classification benchmarks (i.e. CIFAR- 100 (Krizhevsky et al., 2009) and ImageNet (Deng al.) 2009) and an object detection benchmark (i.e. MS COCO (Lin et al. 2014)). Experimental results indicate that DPK outperforms other baseline models under several settings. More importantly, our method can be further improved by applying larger teachers (see Fig. [or an example). We argue that this characteristic of DPK not only further boosts student performance, but also provides a quick solution in model selection for finding the best teacher for a given student. In addition, we conduct extensive ablations to validate each design of DPK in detail.
In summary, the main contributions of this work are:
We propose the prior knowledge mechanism for feature distillation, which can fully excavate the distillation potential of big models. To the best of our knowledge, our method is the first to take the features of teachers as 'input', not just 'target' in knowledge distillation.
Based on our first contribution, we further propose the dynamic prior knowledge (DPK). Our DPK provides solution to the 'larger models are not always better teachers' issue. Besides, also gives better (or comparable) results under several settings.
2 METHODOLOGGY
In this section, we first provide the background of KD, and then introduce the framework and details of the proposed DPK.
2.1 PRELIMINARY
The existing KD methods can be grouped into two groups In particular, the logits-based KD methods distill the dark knowledge from the teacher by aligning the soft targets between the student and
'ttueent's performance is positively correlated with the accuracy of the teacher if the capacity gap is fixed, Appendix A.6 for more details.
2

---

Published as a conference paper at ICLR 2023
teacher, which can be formulated as a loss item:
Llogits =
where 2" and z' are the logits from the student and the teacher. (()) is the softmax function that produces the category probabilities from the logits, and J is a non-negative temperature hyper- parameter to scale the smoothness of predictive distribution. Specifically, we have d+(2;7) softmax((xp)) (expaxx(xx((//T)) Diogita is a loss function which can capture the difference between two categorial distributions, e.g. Kullback-Leibler divergence. Similarly, the feature-based KD methods, whose main idea is to mimick the feature representations between students and teachers, can also be represented as an auxiliary loss item:
Djeat (eaa((T.(F"))
(2)
where F and F' denote the feature maps from the student and the teacher, respectively. T T denote the student and the teacher transformation module respectively. which align the dimensions of F' and F' (and transform the feature representations, such as (Tung & Mori 2019)). Deat denotes the function which can compute the distance between two feature maps, such as f or fo-norm. So the KD methods can be represented by a generic paradigm. The final loss is the weighted sum of the classification loss Lets (the original training loss), the logits distillation loss, and the feature distillation loss:
f Lats + aLLtogits + fB
where {a, 3} are hyper-parameters controlling the trade-off between these three losses.
2.2 DYNAMIC PRIOR KNOWLEDGE
An overview of the proposed DPK is presented in Fig. 2 As a feature distillation method, the main contributions of the DPK include the prior knowledge mechanism and dynamic mask generation. Here give the details of these two designs.
Prior knowledge. To introduce the prior knowledge, the teacher provides part of its features to the student, and the Eq. can be reformulated
L Deats (T, (F()),,
and then we introduce how to build the hybrid feature map, i.e. the student transformation module TG Specifically, given the paired feature maps F and F from the student and the teacher, we divide them into several non-overlapping patches using a k x k convolution with stride k, where k the pre-defined size of the patches. Meanwhile, we also align the dimension the features with this convolution layer. Then we randomly mask subset of feature patches of the student under an uniform distribution with the ratio n, and also generate the complementary feature patches from the teachers. We take these feature patches as token sequences and use two ViT (Dosovitskiy et al.) 2020)-based encoders to further process them. After that, we stitch these token sequences together to generate the hybrid features (tokens), add the positional embedding to these hybrid tokens, and further integrate them with another ViT-based decoder. Finally, we re-organize generated token sequences into original shape and apply the feature distillation loss on the hybrid features and original teacher features. The transformation module T, for teacher feature is an identical mapping.
Dynamic mechanism. In the above solution, we mix the features of teacher and student with the hyper-paue parameter т. Furthermore, we empirically find that: (i) the optimal π for different model combinations different (see Table (oor related experiments). and (ii) the feature gap different in the early and late stages of training phase. These facts inspire to adjust the value of the masking ratio flexibly and dynamically according to the teacher-student gap, e.g. when the teacher-student gap is large, students need more prior knowledge to guide them. Upholding this principle, given minibatch of feature sets for a teacher-student pair, we flatten them to yield F, E R xexxhx + X IR Bxp and F, E R Bxcxhxw + Y E R Bxp2, We then set the iterationsppecif masking ratio at the 7th minibatch as
CKA (X,, Yi),
: where i indexes the minibatch within training epoch. CKA Nguyen al. 2020 is the minibatch version of the Centered Kernel Aligument (CKA))Komblthh representation similarity measure that is widely used for quantitative understanding the representations
Korblith et al. (2019). which is
3

---

Published as a conference paper at ICLR 2023
ResBlock x
ResBlock 36
ResBlock x
Teacher (ResNet152)
Lyeatz
Lyeats
F
F
Diogits
Fl
F2
F2
Lts
Student (ResNet18)
ResBlock x
ResBlock x
ResBlock x
Teacher Encoder
Student Encoder
Decoder
Feature Distilation Loss Logit Distiletion
Task Loss
Figure 2: Illustaaioon of the proposed DPK. For each feature distillation stage, the student feature map and the teacher feature map are sent to corresponding encoders to generate the F and F'. Then, a subset student feature patches is replaced by that the teacher (the denotes the feature stitching operation). After that, DPK further integrates the hybrid feature F with decoder before applying feature distillation loss. Note that the proportion F and F' in F dynamically generated, which is omitted in this figure.
learned by neural networks. Specifically, CKA takes the two feature representations and Y as input and computes their normalized similarity in terms of the Hilbert-Schmidt Independence Criterion (HSIC). CKAministth instead uses an unbiased estimator of HSIC Song et al. (2012), called HISC,,
2 (n 1)(n 2) n n-2
1 1K
HSIC (K,L) n(n 3)
tr( (r(
(6)
where K and L are obtained by setting the diagonal entries of similarity matrices K and L to zero. Then CKAminibatch can be computed by averaging HISC,scores over minibatches:
E HSICI Y,Y!)
CKAminibatao
(7)
V E ISIII((XX)) x,X,) ELL HSIC (Y,C, Y,Y,)
where X, RBxpi and Y, E Bxp= now matrices containing activations of the 7th minibatch of B examples sampled without replacement.
Remarks. Eq. 7 allows us to efficiently and robustly estimate feature dissimilarity between the teacher-student pair using minibatches. A lower CKA indicates greater feature gap between students and teachers. And the higher the masking ratio, the larger the feature regions masked by the student and the larger the countparts provided by the teacher. Eq. (5) naturally establishes the connection between feature gap and masking ratio, with its effectiveness and design choices verfied in Sec. 3.3
Due to the space limitation, we only introduce the main designs of our DPK, and more details (e.g. how to apply DPK object detection) can be found Appendix A.2
EXPERIMENTS
We conduct extensive experiments image classification and object detection. Moreover, present various ablations and analysis for the proposed method. Besides, our codes as well as training recipes will be publicly available for reproducibility.
3.1 IMAGE CLASSTFICATION
We evaluate our method on CIFAR- 100 and ImageNet for image classification (see Appendix A. for the introduction of these datasets and related evaluation metrics). We compare DPK with wide
4

---

Published as a conference paper at ICLR 2023
range of baseline models, including KD (Hinton et al. 2015), FitNets (Romero et al. 2013). FT (Kim
et al. 2018). AB (Heo et al. 2019b). AT (Zagoruyko & Komodakis 2017). PKT (Passalis & Tefas
2018
SP (Tung & Moni 2019), SAD (Ji al. 2021). CC (eng et al. 22199, RKD (Park et al.
2019).
VID (Ahn et al. 2019). CRD (Tian et al. 2020), OFD (Heo et al. 2019a), Review (Chen
al.
2021
DKI) (Zhao et al. 2022). ICKD-C (Liu et.
2021a) and MGD (Yang et al. 2022b).
Results on CIFAR-100. We first evaluate DPK on the CIFAR-100 dataset and summarize the results on Table [ From these results, we can observe that the proposed DPK performs best for all six teacher-student pairs, which firmly demonstrates the effectiveness of our method.
Table Results on the CIFAR-100 validation set. We report the top-1 accuracy (%) of the methods for homogeneous teacher-student pairs. "." indicates results not available, and we highlight the best results in bold.
Teacher
WR.40-2
WRN40-2
ResNet56
ResNet1
ResNerl 10
VGG13
Acc.
75.61
75.61
72.34
74.31
74,31
74.64
Student
WRN16-2
WRN40-1
ResNer20
ResNer20
ResNet32
VGG8
Acc.
73.26
71.98
69.06
69.06
71.14
70.36
KD
74.92
73.54
70.66
70.67
73.08
72.98
FitNets
73.58
72.24
69.21
68.99
71.06
71.02
AT
74.08
72.77
70.55
70.22
72.31
71.43
PKT
74.54
73.45
70.34
70.25
72.61
72.88
SP 73.83
72.43
69.67
70.04
72.69
72.68
CC 73.56
72.21
69.63
69.48
71.48
70.71
RKD
73.35
72.22
69.61
69.25
71.22
71.48
VID 74.11
73.30
70.38
70.16
72.61
71.23
CRD
75.48
74.14
71.16
71.46
73.48
73.94
OFD
75.24
74.33
70.98
73.23
73.95
ReviewKD
76.12
75.09
71.89
73.89
74.84
DKD
76.24
74.81
71.97
74.11
74.68
ICKD-C 75.57
74.63
71.69
71.91
74.11
73.88
DPK
76.42
75.27
72.37
72.44
74.89
74.96
Results on ImageNet. We also conduct experiments on the large-scale ImageNet to evaluate our DPK. In particular, following the previous conventions (Tian et al.) 2020,Chhe Chen et al. 2021), we present the performance of ResNet-18 guided by ResNet-34 in Table The results show superiority of DPK in performance to other baselines.
KD for heterogeneous models. Table 1 and 2 show experiments for homogeneous models (e.g. 8 and ResNet34), and we show our method can also be applied to heterogeneous models (e.g. MobileNet and ResNet50) in this part. From the results shown in Table we can observe that DPK performs best among all listed methods for heterogeneous models (more experiments for this setting can be found Appendix A.4).
The above experiments show DPK performs well for the common teacher-student pairs. Here we show our method can be further improved with better teachers. As illustrated in Fig. the accuracy of the student model trained by our method is continuously improved by progressively replacing larger teacher models, while the model trained by other algorithms fluctu- ates performance. Meanwhile, our method surpasses counterparts each stage and progressively widen the performance gap. To show the generalization, we also present the performance change for other teacher-student combinations in Fig. 4 which confirms the same conclusion again, Note that is a reasonable phenomenon that the performance of the given student tends to be saturated gradually. but the performance fluctuation will cause many difficulties in practical application.
Better teacher, better student.
3.2 OBJECT DETECTION
DPK can also be applied other tasks, and evaluate it on popular one, i.e. object detection. Note that our method can be easily integrated into other KD methods, and we apply DPK into FGD YYng
et al.
2022a and evaluate the performance the most commonly used MS COCO dataset (Lin
et al.
2014) (see Appendix [A. A.2 for the details dataset/meerrcs and implementation).
Comparison with SOTA methods. As presented in Table detector (RetinaNet (Lin et al.] (2017b) and two-stage detector PFaster-RCN) (Ren et al.
evaluate model one-stage
2015)
with several strong baselines, including FGFI (Wang et al. 2019), GID DDii et al. 2021)
FGD
5

---

Published as a conference paper at ICLR 2023
Table 2: Results on ImageNet validation set. We show the top-1 I and top-5 accuracy (%) for ResNet18 guided by ResNet34. "-" indicates the results are not available.
Acc.
Student
Teacher
KD
RKD
CRD++D
SAD
CC
ICKD-C
ReviewKD>
D2K1)
MG1>
Ours
Top-1
69.55
73.31
70.68
71.34
71.38
71.38
70.74
72.19
71.61 90.51
71.70
71.80
72.51
Top-5
89.09
91.42
90.16
90.37
90.49
90.72
90.41
90.40
90.77
Table 3: Results for heterogeneous models. We show the top-1 and top-5 accuracy (%) of Mo- bileNetV2 guided by ResNet50 on ImageNet validation set.
Acc.
Student
Teacher
FT
AB
AT
OFD
CRD
ReviewKD
KD
DKD
MGD
Ours
Top-1
68.87
76.16
69.88
69.89
69.56
71.25
71.37
72.56
68.58
72.05
72.59
73.26
Top-5
88.76
92.86
89,50
88.71
89.33
90.34
90.41
91.00
88.98
91.05
90.94
91.17
CIFAR-100 Top-1 Accuracy (96
80
ImageNer Top-1 Accuracy (9))
Vin Teacher Studemt (W-16-2)
80
Vanile Teacher Student (R18)
ICKD
DKD
Ours
Ours
76
76:42
74
72
72
73.0
70
69.55
W-28-2 W-40-2
VGG19 Den-121
R50
Figure 3: Better teacher, better student. We show the top-1 accuracy of our method and some baselines on the CIFAR-100 (left) and ImageNet (right).
Student NonVNeXtEE (77.43)
Student ResNet18 (69.55)
Student MobileNetV2 (68.87)
Student ShufflNNtt2 (68,
ConvNaXt-T
Conviie
ReaNet34
RoaNet34
Teachers
CunNNatt-S
Teachers
Convñee Xt-S
Teachers
ResNetS0
Teachers
ResNet50
CaitNeXt-
Conufie Xt-I
72.36
73.29
RwsNet1
ConNNeXttL
ConviecXe XL-L
72.48
RcsNet152
KcsNet152
78
79
80
71
72
72
73
69.0
69.2
69.4
69.6
Figure 4: More teacher-student combinations, covering the lightweight ShuffleNetV2 Ma et al. 2018) the recently ConvNeXt (Liu et al. 2022). Top-1 accuracy (%) ImageNet val set is reported. Both homogeneous and heterogeneous settings are considered. By changing the teacher size, we see that, larger models are always better teachers holds true when applying DPK. Con- is a small model built by us, following the design principles of ConvNeXt (Liu et al. 2022) see
Appendix A.2 for details. Best viewed in color with zoom
(angg al.) 2022a), and DeFeat (Guo et al.) 2021). Following the previous conventions, first adopt the ResNNet011FPN as the teacher model and ResNet50-FPN as the student model, and our model achieves better (or matched) performance than other baselines, i.e. our model gets a matched performance with FGD, and significantly surpasses other baselines in accuracy.
Better teacher, better student. Similar to image classification, our method also benefits from better teacher model for object detection. In particular, we enlarge the capacity gap between the teacher and student models, and the results in Table suggest that effectiveness our method can further improved by replacing more powerful teacher models. For instance, replacing teacher model from ResNet101-FPN to ResNet152-FPN further improves the student's performance (ResNet50-FPN. Faster RCNN) by 0.6 mAP, while the number for FGD (Yang et al. 2022al is 0. for the same setting. This conclusion also stands for other teacher-student pairs and frameworks.
3.3 ABLATION STUDIES
In this section, provide extensive ablation studies to analyze the effects of each component of DPK. The experiments are conducted on ImageNet for classification task, ResNet34 and ResNet18 adopted teacher and student. Only last stage distillation is applied unless stated otherwise.

---

Published as a conference paper at ICLR 2023
Table Results on object detection. Experiments are evaluated on COCO validation set. 'T' and 'S' represent the teacher and the student, respectively. "_" indicates results are not available.
Methods
RetinaNet
Faster-RCNN
mAP
AP
AP
APs
APM
APt.
mAP
APso 30
AP
APs
APM
APr.
R101 (
38.9
58.0
41.5
21.0
42.8
52.4
39.9
60.1
43.3
23.5
44.2
51.5
R50-FPN(S)
37.4
56.7
39.6
20.6
40.7
49.7
38.4
59.0
42.0
21.5
42.1
50.3
FGFI
38.6
58.7
41.3
21.4
42.5
51.5
39.3
59.8
42.9
22.5
42.3
52.2
GID
39.1
59.0
42.3
22.8
43.1
52.3
40.2
60.7
43.8
22.7
44.0
53.2
FGD
39.6
22.9
43.7
53.6
40.4
22.8
44.5
53.5
Ours
39.7
58.6
42.5
22.8
43.6
53.6
40.6
60.7
44.4
22.8
44.7
54.0
R152-FPN(T T)
39.9
59.4
42.7
23.5
44.2
51.5
41.6
62.3
45.4
23.3
46.1
53.6
R50-FPN(S)
37.4
56.7
39.6
20.6
40.7
49.7
38.4
59.0
42.0
21.5
42.1
50.3
FGFI
38.9
21.9
42.5
52.2
39.9
22.9
43.6
52.8
DeFeat
39.7
23.4
43.6
$2.9
40.9
23.6
44.8
53.5
FGD
39.7
58.9
42.9
23.2
44.0
52.7
40.5
61.1
44.2
23.4
44.4
53.7
Ours
40.2
59.6
43.0
23.4
44.5
53.6
41.2
61.6
45.0
23.3
45.2
54.3
R101-FPN(O
38.9
58.0
41.5
21.0
42.8
52.4
39.9
60.1
43.3
23.5
44.2
$1.5
RI8-FPN(S)
33.1
51.4
34.9
17.4
35,8
43.4
33.8
53.2
36.8
19.2
36.6
43.6
FGD
34.7
52.5
37.1
17.8
38.0
48.5
36.0
55.8
39.2
18.3
39.4
48.7
Ours
35.8
53.9
38.2
19.1
39.4
49.1
36.7
56.2
40.1
19.2
40.3
49.2
Table 5: Ablations mask ratios. report top-1 accuracy on ImageNet with setting (a): ResNet18 as student, ResNet34 as teacher, and setting (b): ResNet18 as student, ResNet101 as teacher.
setting
teacher
student
15%
35%
55%
75%
95%
dynamic
(a)
73.31
69.55
72.33
72.34
72.38
72.36
72.34
72.46
(b)
77.37
69.55
72.46
72.55
72.57
72.60
72.53
72.87
Mask ratio. Feature masking (and stitching) is a key component of our method, and Table 5 reports the results of various DPK variants under different mask ratios. Surprisingly, broad range of masking ratios from 15% to 95% can offer considerable performance gains for students. This implies that the prior knowledge provided by teachers is very beneficial to students' network learning. Besides, note that the optimal mask ratio is inconsistent under different teacher-student pairs, e.g. 55% mask ratio performs best for ResNet-18 and ResNet34, while the optimal and ResNe101 is 75%. Table also shows students achieve the best accuracy using the proposed dynamic masking strategy, which suggests the necessity and effectiveness of the proposed dynamic masking strategy for automatic selection of mask ratio.
for ResNet-18
Mask strategy. Table summarizzs the effects of differ- ent mask strategies. ResNet34 and ResNet18 8 are adopted as teacher and student for all experiments in this part. For the fixed mask ratio, we take the simple random mask as baseline, and consider the bl lock-wkww wise mask strat- egy, introduced by BEIT (Bao et al., (022). We also consider grid didwiise mask, which regularly retains one of every four patches, similar MAE (He et al., (021). Besides fixed mask designs, several alternatives for re- alizing dynamic mask ratio are explored. Particularly, the cos ine iesimm indicates that we use the cosine sim- ilarity measure the teacher-student feature gap. The exponentis i 1. decay schedule divides the mask ratio by the same factor every epoch, which can be expressed T. mn (0.95)spoch, where To is the initial mask ratio and is set to 1.0. The i nearrr schedule decreases the mask ratio by the same decrement every epoch, which is defined as = (epoch; decrement), and we set decrement 0.95. The results reveal that: i) simple random sampling works best for our DPK when using fixed mask ratios. ii) 1 1-CKA outperforms its competitors, and we use it to generate dynamic mask ratios by default. and 1-CKA outperform the manually-set heuristic
Masking Strategy
Ratio
Top-1
Top-5
Vanilla
X
69.55 89.09
Random
75%
72.36
90.61
Block-wise Grid-wist
75%
72.36 90.57
75%
72.32 90.38
1-CKA
X
72.46 90.67
I-CosineSimi Exponential decay
X
72.38 90.58
X
72.35
90.51
Linear decay
X
72.36
90.58
Table 6: Ablations on mask strategies.
We also observe that both i-cosiness masking strategies, e.g., linear decay, demonstrating the advantage of dynamically adjusting the ratio of teacher prior knowledge based on the teacher-student feature gap. Furthermore, the CKA is superior to the cos nest mi, indicating that our CKA A can efficiently measure the similarity of
7

---

Published as a conference paper at ICLR 2023
Table 7: Ablations on prior knowledge.
Table 8: Ablations on transfor- mation functions.
Table 9: Ablations on loss cal- culation.
Prior Knowledge
Top-1 Top-5
Transform
Top-1 Top-5
Target
Top-1 Top-5
Zero-Padding
72.16 90.35
Baseline
71.08 90.00
Non- Non-Masked
72.39 90.63
Learnable
72.28 90.63
Conv
71.39 90.46
Full
72.46 90.67
Ours
72.46 90.67
MLP-Decoder
72.31 90.50
Encoder-Decoder
72.46 90.67
the hidden representations between teachers and students using minibatches, and provide robust way to automatically determine the masking ratio.
Prior knowledge. Table 7 ablates the significance of integrating teacher's knowledge in building hybrid student features (Eq. 4). Specifically, we use zero polo--padding and learnable mask token to play the role of teacher's 's feature in the masked position. The results show that no teacher-provided provided prior knowledge leads to worse performance, as it aggravates the burden feature mimicking from students to teachers. This strongly confirms the effectiveness of offering students the prior knowledge from teachers via feature masking and stitching.
Transformation module. For feature-based based distillation methods, the feature transformation modules T, and T, are required to convert the features into an easy- easy-to-transfer form. We take FitNets (Romero et al. 2015) baseline, which does not reduce the dimension of teacher's feature map and use Ix convolutional layer to transform the feature dimension of the student to that of the teacher. In our method, adopt the ViT-based encoder-decoder as the default transformation module. For further investigation, we also remove the encoder, and use a MLP layer to align the feature dimension (MLP-Decoder). Besides, the convolution based encoder-devoder structure also explored (Conv). More implementation details for these modules are deferred in the Appendix A.2 Table X shows the results, and we can observe that encoder- decoder reaches the best performance.
Loss. In the training phase, we apply the mean squared error (MSE) loss between the hybrid student features and teacher features in Eq. while the loss can be on only the non masked regions of student features, namely non-mas ked, or full feature maps, namely full. Table 9 shows the ablation results these two settings. As can seen, computing the loss on the full features performs better.
3.4 VISUALIZATIONS
In this part, we present some visualizations to show that our DPK does bridge the teacher-student gap the feature-level. In particular, we visualize the feature similarity between ResNet18 and ResNet34 in Fig.5) 5 We can find that our DPK significantly improves the feature similarity (measured by CKA) between the student and the teacher. ICKD gets a lower similarity than the baseline, probably due to the fact that it models the feature relationships, instead of the features themselves. More related visualizations can be found in Appendix A.8 including other teacher- stuudnt combinations, the CKA curve in the training phase, and similarity maps measured by other metrics.
CKA scores of Reshee11 10 Pes Neel 34
CKA scores of Res 18 (CKD) R4sRiet34
CGA scores of ResNetl8 (Ours) ResNet34
673
7.75
Sample Indox
67B
sample InDex
3.70
3.0
Sampr
655
46
10
Sample
Sample Index
Figure 5: CKA similarity between ResNet18 and ResNet34. We visualize the CKA similarity between the original models (leff), the models trained by ICKD (middle), and the models trained by our DPK (right). The experiments are conducted on the sampled ImageNet validation set (12,800 samples). We compute CKA with batch size of 32 for the last stage, these are 400 CKA values for each experiment. For better presentation, we rank these values and organize them as the heatmap representation. The larger the value, the more similar the features are.
8

---

Published as a conference paper at ICLR 2023
4 RELATED WORK
Knowledge distillation. Existing studies on knowledge distillation (KD) can be roughly categorized into two groups: logits-based distillation and rature-based distillation. The former, pioneered by Hinton et al. (2015), is known as the classic KD, aiming to learn compact student by mimicking the softmax outputs (logits) of an wver-parameterized teacher. This line of work focuses on proposing effective regularization and optimization techniques (Zhang et al.] 2018)) Recently, DKD (Zhao et al., 2022) proposes to decouple the classical KD loss into two parts, i.e. target class KD and non-target KD. Besides, several works (Phuong & Lampert 2019 Cheng et al. 2020) also attempt to interpret the classical KD. The latter, represented by FitNet (Romero et al. 2015), encourages students features from the hidden layers of teacher models. Since features
to mimic the intermediate-leve are more informative than logits, feature-based distillation methods usually perform better than logits-based ones in the tasks that involve the localization information, such as object detection (Li et al. 2017; Wang et al., 2019: Dai et al., 2021: Guo et al. 2021 Yang et al. 2022a). This line of work mainly investigates what kinds of intermediate representations of features should be. These representations include singular value decomposition (Lee et al. 2018), attention maps (Zagoruyko ko & Komodakis 2017), Gramian matrices (Yim et al., 2017a). gradients (Srinivas & Fleuret, 2018). pre-activations (Heo et al.] (019b), similarities and dissimilarities (Tung & Mori, Mori//2019). instance relationships (Liu et al., 2019b, Park et al. 2019), inter-channel correlations (Liu et al., 2021a). A noteworthy work similar to DPK is AGNL (Zhang & Ma 2020). In particular, AGNL has two attractive properties: i) attention attention-guided distillation, letting students' learning focus on the foreground objects and suppresses students' learning the background pixels; and ti) non- non-local distillation, transferring the relation between different pixels from teachers to students. At a high level, both DPK and AGNL apply the masking strategy (random mask and attention-guided mask) and non-local relation modeling module (transformer and desi-nes non-local module). The key difference is that DPK integrates features of students and teachers with dynamic mechanism.
Performance degradation. Prior works (Cho & Hariharan 2019; Mirzadeh et al. 2020 Hinton al., 2015; Liu al. 2021a) also report that the performance of distilled student degrades when the gap between students and teachers becomes large. To solve this issue, ESKD (Cho & Hariha
ran 2019) stops the teacher training early to make it under convergence and yield more softened
logits. TAKD (Mirzadeh et al. (2200) introduces an extra intermediate-sized network termed teacher assistant to bridge the gap between teachers and students (more discussion for this method and our (A.3). Different from the above logits-based methods, our method directly reduces the gap between teachers and students in the feature space, and does not need extra intermediate models.
method can be found in Appendix (A.3).
Masked image modeling. Emerged with the masked language modeling in NLP community, such as BERT (Devlin et al.]|201 (018) and GPT (Radford et al., 2019: Brown et al. (020), masked image modeling (MIM) (Pathak et al., 2016; Henaff] 2020) has gained increasing attention and shows promising potentials in representation learning. Particularly, MIM-based approaches generally i) divide an image or vedio into several non-overlapping patches or discrete visual tokens, ii) mask random subsets of these patches/tokens, and iii) predict the patches masked visual tokens (Bao et al.) 2022), the feature of the masked regions such as HOG (Wei et al. (22TT). or reconstruct the masked pixels et al. [021 Xie et al. 202T]. Most recently, MGD (Yang et al., 20226) attempts to generate the entire teacher's feature map by student's 's masked feature map. In contrast to these approaches, our method operates on feature level and aims to narrow the feature gap between students and teachers.
Besides, our masking ratio is dynamic and knowledge-aware.
CONCLUSION
In this paper we demonstrate the potential of masked feature prediction in mining richer knowledge from teacher networks. Particularly, we design a prior knowledge based feature distillation method, named DPK, and use a dynamic mask ratio scheme, achieved by capturing the feature gap between the teacher-student pairs, to dynamically regulate the training process. The extensive experiments show that our knowledge distillation method achieves state-of-the-ar performances on the commonly used benchmarks (i.e., CIFAR100. ImageNet, and MS COCO) under various settings. More importantly, our DPK can make the accuracy of students positively correlated with that of teachers. This feature further improves the performance of method, and provides 'shortcut" in teacher model selection.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was supported in part by the National Natual Science Foundation of China (NSFC) under Grants No. 10961000000, 61976038, U1908210 and 61772108. Wanli Ouyang was supported by the Australian Research Council Grant DP200103223, FT210100228, and Australian Medical Research Future Fund MRFA1000085.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In CVPR, 2019.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurlPS, 2020.
Cristian Bucilua, Rich Caruana, and Alexandru Viculescu-Mizil Model compression. KDD, 2006.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv, 2019.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021.
Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowledge. In CVPR, 2020.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang. Monodistill: Learning spatial features for monocular 3d object detection. In ICLR, 2022.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223 223. JMLR Workshop and Conference Proceedings, 2011.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for object detection. In CVPR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, abs/1810.04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In CVPR, 2021.
Hai Vietor Habi, Roy H Jennings, and Amon Netzer. Hmq: Hardware friendly mixed precision quantization block for cnns. In ECCV, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2021.
10

---

Published as a conference paper at ICLR 2023
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019a.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI, 2019b.
Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. Distilling the knowledge in a neural network. ArXiv, 2015.
Mingi Ji, Byeongho Heo, and S. Park. Show, attend and distill: Knowledge distillation via attention- based feature matching. In AAAI, 2021.
Ding Jia, Kai Han, Yunhe Wang, Yehui Tang. Jianyuan Guo, Chao Zhang, and Dacheng Tao. Efficient vision transformers via fine-grained manifold distillation. arXiv preprint urXiv:2107.01377 2021.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurlPS, 2017.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NewrlPS, 2018.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In ECCV, 2018
Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In CVPR, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017b.
Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for diversity-preserved knowledge distillation. In /CCV, 2021a.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In CVPR, 2019a.
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In CVPR, 2019b.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 202
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018

---

Published as a conference paper at ICLR 2023
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In ICLR, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. CVPR. 2019.
N. Passalis and A. Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Baoyun Peng. Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV,2019
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In ICML, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
A. Romero, Nicolas Ballas, S. Kahou, Antoine Chassang, C. Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. In JMLR, 2012.
Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. In ICMI., 2018.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICMLL, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML. PMLR, 2021.
F. Tung and G. Mori. Similarity-presert knowledge distillation. In ICCV, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnety2: Differentiable neural architecture search for spatial and channel dimensions. In CVPR, 2020.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In CVPR, 2019.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervise visual pre-training. arXiv preprint arXiv:21 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2021.
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In CVPR, 2022a.
12

---

Published as a conference paper at ICLR 2023
Zhendong Yang. Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. arXiv preprint arXiv:2205.01559, 2022b.
Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017a.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017b.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In ICLR, 2020.
Linfeng Zhang, Yukang Shi, Zuoqiang Shi, Kaisheng Ma, and Chenglong Bao. Task-oriented feature distillation. In NeurlPS, 2020.
Y. Zhang, T. Xiang, Timothy M. Hospedales, and H. Lu. Deep mutual learning. In CVPR, 2018.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distllation. In CVPR, 2022.
B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba, Learning deep features for discrimina- tive localization. CVPR, 2016.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASETS AND METRICS
CIFAR- 100. CIFAR-100 (Krizhevsky et al. 2009 contains 50K images for training and 10K images for testing, labeled into 100 fine-grained categories. The size of each image is x2 We evaluate the proposed DPK on this dataset with image recognition and report the top-1 accuracy.
ImageNet. ImageNet (Deng al. 2009) consists of 1.2M images for training and 50K images for validation, covering 1,000 categories. All images are resized to 224 x 224 during training and testing. We report the top-1 and t accuracy on this dataset for image recognition.
MS COCO. MS COCO (Lin et al., 2014) is the most commonly used object detection benchmark, which contains 80 categories. We also conduct experiments for object detection to further evaluate our DPK. In particular, we use train2017 (118K images) for training, and test on va va12017 7 (5K images). We adopt the standard evaluation protocol introduced by MS COCO, e.g. mAP, APso. AP75, etc.
A.2 IMPLEMENTATIONS
Training details. On CIFAR- 100, the batch size and initial learning rate are set to 64 and 0.05. We train the models for 240 epochs in total with SGD optimizer, and decay the learning rate by 0.1 at 150, 180, and 210 epochs. The weight decay and the momentum are set to 5e-4 and 0.9. On ImageNet, we adopt the SGD optimizer (with 0.9 momentum) to train the student networks for 100 epochs with batch size of 256. The learning rate is set to 0.1, and we decay it by 0.5 every 25 epochs. We set the weight decay to 0.0001. We also apply the vanilla logits distillation loss (Hinton et al., 2015) in our method. For the loss weights in Eqn. equation 3 we set o 0.8 and , = 0.2 for all experiments. The temperature J used on the ImageNet dataset is set to 1.0, and the same parameter on the CIFAR-I00 dataset is set to 4.0. The loss weights each stage is set to 1.0 in the multi-stage feature distillation setting. Our implementation on MS-COCO for object detection follows the same setting used in (Yang et al., (022a)). We adopt mean squared error (MSE) as the feature distillation loss D feat- All experiments are conducted on 8 Tesla V100 GPUs, and our implementation is based on mmdetection framework (Chen et al., 2019).
Transformation modules. To perform feature mimicking, we reshape the original student feature map F E && into a sequence of flattened 2D patches, where (H; W) the resolution of the original student feature map, C' denotes the number of channels, (P; P) is the resolution of each feature patch, and N = HW //2 is the resulted number of patches, which also serves as the effective input sequence length for the Transformer (Vaswani al. 2017).
The Transformer uses constant latent vector size d through all of its layers, so we flatten the patches and map to d dimensions with trainable linear projection. Particularly, name the output through this projection the patch embeddings, similar to ViT (Dosovitskiy et al. 2020). We also add position embeddings the patch embeddings retain positional information. We use standard learnable ID position embeddings. since no significant performance gain has been observed from using more advanced 2D-aware position embeddings. The resulted sequence of embedding vectors serves as input to the Transformer encoder. The Transformer encoder (Vaswani et al. 2017) consists of alternating layers of multi-head self attention and MLP blocks. Layernorm (LN) is applied before every block, and residual connections after every block. Specifically, the number of encoder layer/block is set to 6 in all experiments.
Same design for the original teacher feature map Ft E RC is adopted. The output of the student encoder referred to as the student tokens F E Nxd, and the output of the teacher encoder is referred to as the teacher tokens F' e Nxd, Notice that F and F' are dimensionally aligned. Then we can perform masking on them and eventually construct the so-called hybrid tokens F".
The shared decoder is designed to perform the teacher feature prediction task. We send the hybrid tokens to the shared decoder, which finally produces the output tokens of the same dimension as the original teacher feature map Ft & R XHXW Feature distillation losses are thus computed between the output tokens and the original teacher feature map. During inference, all transformation modules dropped. Therefore, there is additional computational cost over the original student network.
14

---

Published as a conference paper at ICLR 2023
Conv: consisting of several convolutional layers. a pooling layer and fully connected layer.
2020e-based based encoder/decoder in our default transformation module. The encoder Vaswani et al. (2017) consists 6 blocks, and decoder consists of blocks for the single-stage feature distillation and blocks for the multi-stage feature distillation. F and F' are encoded by their encoders to form hybrid tokens, and then used as the input the shared decoder.
Encoder-Decoder. We apply ViT Dosovitskiy et (2020)
Decoder: Different from necode,, F* and F' do need to go through their respective encoder networks, but align their feature dimensions through a layer of simple MLP, then add position encoding to form hybrid tokens the imput of the decoder network.
Details of ConvNeXt-E. To evaluate the proposed DPK, we conduct experiments (Fig. on the recently published ConvNeXt (Liu et al. (022). We take four ConvNeXt variants, ConvNeXt- T/S/B/L, teachers, and build a build smaller ConvNeXt-E serve as the student by reducing the blocks/channels in each stage. The other details, such as training strategies, are same as the official models. The configurations are summarized in Table I0 FLOPS, and accuracy for reference.
We also report the numbers of parameters.
Table 10: Detailed settings of ConyNeXt variants. We also report the parameters, FLOPS and the performance.
Architecture
Channels
Blocks
#Param.
#FLOPS Top-1 Acc. Top-5 Acc.
Student
ConyNeXt-E
(96.192.384.788)
(2,2,4,2
17.43M
2.6G
77.43%
93.30%
ConvNeXt-T
(96, 19,,3422,847769
(3,3,9,3)
28.59M
4.5G
82.06%
95.85%
CovnNeXt-S (96, 192,3392884776
(3,3,27.3)
50.22M
8.7G
83.15%
96.43%
Teacher
ConvNeXt-B (128,256.512, 1024)
(3,3,27,3)
88.59M
15.4G
86.59%
19%
ConyNeXt-L (192,384.768,155)) 1536) (3,32273)
197.77M
34.4G
87.40%
98.37%
Details for object detection. We implement Faster RCNN (Ren et al. 2015) and RetinaNet (Lin
et al. 20176) with different backbones. To integrate the multi-scale features, FPN (Lin et al. 2017a)
is adopted for all experiments, We take FGD (Yang et al. 2022a) (recently published on CVPR 22)
as the baseline model, and build our method on it. In particular, the feature distillation loss in FGD can be formulated as follows:
L wy A CF ((")) +w. M)A f(F*))²
(8)
where Fs, F' denotes the feature map from student detector and teacher detector, respectively. the adaption layer to reshape the F" to the same dimension as F'. M is the binary mask, indicating the foreground and background regions, derived from the ground truth. A A denote the learnable spatial attention map and channel attention map used in FGD. Besides, wy and W/ are the hyper- parameters balance the losses for foreground and background regions. To combine model with FGD, we replace the feature mimicking part with Eq. 4, and we have:
L Wy MA A frar (F*, F*) + wss M)A* A Dear (F" F*),
(9)
where hhh denotes the hybrid features, whose construction process is stated in the main paper. As for the hyper-parameters, such as wy and wa in Eq. g we follow the settings in FGD and fine-tune them on each training fold. Specifically, we adopt the hyper-parameters {w,- 5E-5, w, 2.5E-5) for all the two-stage detectors, and f 2E-3 3, wy 5E-4) for all the one-stage detectors. We train all the detectors for 24 epochs with SGD optimizer, with the momentum 0.9 and the weight decay
as 0.0001.
A.3 COMPARISON WITH TAKD
TAKD (Mirzadeh et al. 2020) introduces intermediate models as teacher assistants (TAs) to bridge the capacity gap between the teacher models and the student models. This work shares similar motivation with DPK, and we provide additional experiments to compare these two works. In particular, we report the performance of TA TAKD with one TA (official setting), here we compare our method and
15

---

Published as a conference paper at ICLR 2023
TAKD with more TAs. The experiments are based the ResNet, and we use ResNet- 101 as the teacher and ResNet-18 18 as the student. For TAKD, we adopt ResNet-50 and ResNet-34 as TAs, and then train the TAs and students one by one. For DPK, directly train the student under the teacher's guidance. The results are summarized in Table T1 We can find that our DPK surpasses TAKD in performance (e.g. 73.00 v.s. 71,41). Also, note DPK does not need multiple training.
Table 11: Comparison DPK and TAKD on ImageNet. We use ResNet-18 as the student model. For our DPK, we train the student model under the guidance of ResNet- 101. For TAKD, we use ResNet-100 101 as the teacher model, and use ResNet-50 and ResNet-35 as TAs.
Acc.
Baseline (R18)
-R34--R344-- (TAKD) R101 -RI (DPK)
Top-1 69.55
71.41
73.00
Top-5
89.09
90.22
90.95
A.4 MORE EXPERIMENTS FOR HETEROGENEOUS MODELS
REshveSOTT Mablenehaass
Figure 6: Heterogeneous experiments on CIFAR-100. Top-1 accuracy is reported. Best viewed in color with zoom in. "T" denotes the teacher, and "S" denotes the student. Statistically, DPK ranks 1st for two pairs and 2nd for four pairs.
We report the heterogeneous experiments for some common settings in the main paper. In this section, we give the experiments on 100, and more cases on ImageNet.
Experiments on CIFAR-100. Fig. prresents the experimental results CIFAR-100 According to these results, we can find our model outperforms all baseline methods across two model pairs, i.e. ResNet50(T)-VGGG8S) and TRRN40-2(T-ShuffleNee (S), and ranks second for the remaining four model pairs, which demonstrates the effectiveness and robustness of DPK. Besides, we also observe that CRD (Tian et al. 2020) achieves promising performance on this dataset for heterogeneous settings. Note that our method significantly performs better than CRD for homogeneous settings on CIFAR-100 (see Table and ImageNet (see Table and heterogeneous settings on ImageNet (see Table 3)
Experiments on ImageNet. On ImageNet, we adopt EfficientNet (Tan & Le 2019) as the teacher and R 8 as the student to conduct heterogeneous experiments. We only distill the features in the last stage for this experiment. The results shown Table 12 suggest that DPK also works for other teacher-student pairs, and can be further improved by applying larger models.
A.5 MORE EXPERIMENTS ON TRANSFORMER-STEE TEACHER-STUDENT DISTILLATION PAIRS
Performance on ImageNet. We first validate the effectiveness of the proposed method on different transformer architectural distillation pairs on ImageNet-1K. K. The results are shown in Table 13 From Table [3] we can see that DPK outperforms all the competitors. To be specificl, DPK obtains 6.78% gain over Hard (Touvron al., 202 l) on DeiT-Tiny (77.1 % v.s. 74.5%), which introduces hard decision token and a distillation token for distilling the inductive bias from a large pretrained CNN teacher. When applying to the prevalent Swin-tiny tiny (Liu et al.] (021b), hierarchical vision transformer using shifted windows, DPK can still bring a further 1.7 % improvement. The results reval that DPK is not limited to CNN architectures, but also is effective in transformer architectures.
16

---

Published as a conference paper at ICLR 2023
Table 12: Results for heterogeneous models. We ResNet18 the student and networks from EfficientNet Tan & Le 2019) series as teachers.
Acc.
Student
Efficien(Ne--BO
EfficientNetBII
Efficiem(NetB22
Efficien(Net-B3
Efficien[Net-B4
Top-1
69.55
72.46
73.39
73.81 91.56
73.89
74.05 91.58
Top-5
89.09
90.56
91.10
91.63
Table 13: Comparisons of different transformer-basel distillation pairs ImageNet-1K. KD:
the vanilla KD algorithm popularized by Hinton et al. (Hinton et al. 2015). Hard: similar to
DeiT [Tourron al. 2021), introduce hard decision and distillation token. Manifold: training
a tiny student model to match pre-trained teacher model in the patch-level manifold space (Jia et al.
2021.
Distillation Method
Teacher
Top-1 Acc. (%)
Student
Top-1 Acc. (%)
Vanilla Student
72.2
KD
(Hinton et al. 2015)
73.0
Hard
Touvron et al.
2021
74.5
Manitold (Ja al.
2021
CaiT-S24
83.4
DeiT-Tiny
76.5
Ours
77.1
Vanilla Student
79.9
KD
(Hinton et al. 2015
80.0
Hard
Touvron
202
81.3
Manitold (Jia et al.
2021
CaiT-S24
83.4
DeiT-Small
82.2
Ours
82.5
Vanilla Student
81.2
KD (Hinton et al. 2013)
81.7
Hard
Touvron et a.
2021
81.7
Manitold UJa al.
Swin-Small
83.2
Swin-Tiny
82.2
Ours
82.6
Table 14: Distillation results of gradually increasing teacher capacity ImageNet-IK.
Teacher Acc.
Student Acc. Top-1 (%)
Teacher Models
#Param.
Top-1 (%)
Student
ViT-Tiny
5.01M
72.2
ViT-small
22.05M
79.9
73.8
Teacher
ViT-base
86.57M
81.8
74.0
CaiT-S24
46.92M
83.4
76.1
Student
Swin-tiny
29M
81.2
Swin-small
49.61M
83.2
82.6
Teacher
Swin-base
87.78M
85.2
82.7
Swin-large
196.53M
86.3
82.9
Bigger Model, Better Teacher. Recall that DPK performs well for varying-sized CNN- teacher- student transfer pairs. We thus conlude that students can consistently benefit from teachers with higher capacity using DPK. Here, by progressively using larger teachers. Perhaps not surprisingly, as listed in Table 14 the accuracy of the student distilled by method is continuously improved by progressively replacing larger teacher models, while the students guilded by other algorithms fluctuates in performance, reaffirming conclusion.
whether this conclusion holds for transformer-style transfer pairs
17

---

Published as a conference paper at ICLR 2023
Table 15: ResNet-18 trained with varying teacher models. We report the top-1 and top-5 accuracy on ImageNet. All teacher models are re-trained by us to adjust their final performances.
Teacher Acc. (%)
Student Acc. (%)
Models
#Param.
Top-1
Top-5
Top-1
Top-5
Student
ResNet18
11.69M
69.55
89.09
ResNer34_ v1
21.80M
73.31
91.42
70.68
90.16
ResNet34
21.80M
74.37
91.90
71.36
90.08
ResNet34 v3
21.80M
75.81
92.70
71.60
90.19
ResNet34_ v-4
21.80M
76.27
92.96
71.41
90.22
Teacher
ResNet34_ye .v3
21.80M
75.81
92.70
71.60
90.19
ResNet50._y v1
25.56M
75.86
92.88
71.63
90.16
ResNer101_ vl
44.55M
75.97
92.73
71.32
89.95
ResNet152__l
60.19M
76.24
92.80
71.21
89.89
A.6 FACTORS AFFECTING THE EFFECTIVENESS OF KD
In the main paper, we suppose there are two main factors affecting students' performance: (i) the capacity of the teacher model, and (ii) the performance of the teacher model. We conduct toy experiment to support these two assumptions in this part.
Performance the teacher model. First, we train ResNet-34 with different teacher models using the vanilla KD (Hinton et al. 2015) to get several ResNet-34 with different performance. Then ResNet- as student model, ResNet-34 (with different accuracy) as teachers to explore the relation of student's s accuracy and teacher's ' accuracy. As shown in Table 15] can find better teachers generally lead to better students when the teachers share the same CNN model.
Capacity of the teacher model. Similarly, also select some different distilled ResNet models and keep their performance at a similar level to explore the impact capacity difference on student's final accuracy. The results in Table 15 suggest that larger teachers generally degrade the students when the teachers have similar performance.
The above two factors make the choice of the teacher models become special 'trade-off' between accuracy and capacity, and our DPK alleviates this issue by reducing the capacity gap in feature-level.
A.7 REPRESENTATION TRANSFERABILITY.
Following previous works in (Zhao et al. 2022 Tian et al. 2020), we evaluate the generalization ability of learned representations by transferring them to unseen datasets. Specifically, we adopt WRN-16-2 student distilled from a WRN-400 before logits) trained CIFAR 100. We then train single linear layer classifier on top of the frozen representations perform 10-way (for STL-10 Coates et al. (2011) and 200-way (for Tiny ImageNe(3) classification. To better quantify the transferability of the representations, fixed and only update the linear probing heads. Table compares several baseline methods. From Table 16 we note that, when transferring the representations learned from CIFAR- 100 to STL-10 and TinyImageNe our method outperforms all basclines, confirming its superiority in improving the transferability of representations.
teacher as a frozen representation extractor (layers
keep the representations
Table 16: Comparison of Top-1 accuracy of different distillation methods on transferring representations learned from CIFAR- 100 to STL-10 or TinylmageNet (TinyIN), where WRN- 16-2 taken as the student and WRN-40-2 as the teacher: 'Baseline' means that no distillation method is applied and the student WRN-16-2 is trained from scratch CIFAR-100.
Baseline
KD
AT
FitNet CRD
CRD+KD ReviewKD DKD Ours
STL-10
69.7
70,9
70.7
70.3
71.6
72.2
72.4
72,9
73.2
TinyIN
33.7
33.9
34.2
33.5
35.6
35.5
36.6
37.1
37.8
htttss://ww.ke/tt com http:///w....o///o//ooioiiii/e/e
18

---

Published as a conference paper at ICLR 2023
A.8 VISUALIZATIONS
CKA curve in training. To qualitatively analyze the proposed DPK, we take ResNet-18 18 as the student model, and visualize the CKA similarities trained with four teacher models in the training phase. As shown in Fig. 7 we can find the following two observations: (i) the CKA values increase with training, which demonstrates that our DPK does narrow the gap of teacher-student models at feature level, and (ii) the CKA for the lager teacher is significantly lower than that for small teacher, which suggests the necessity of our dynamic design (observation (?) also support this conclusion). We also visualize the dynamic mask ratios in Fig. for reference.
Imsgeter-- -1K Miibbaich CXA Valu
0.705
ImageNer-1K MaaNNer-1KMMa Minibathh CKA Value
0.70
0.68
0.685
0.66
0.675
(esNet34(1) (R
0.64
ResNet50011
100
150
200
50
100
150
200
Training Epochs
Training Epochs
ImageNer MaageNer-1KMMMii Minbbatcch Value
0.70
ImageNer-1K Minibatch CKA Value
0.44
0.68
0.43
0.42
0.66
0.41
0.64
Resseet152(1-RReseetto
100
150
200
50
100
150
200
Training Epochs
Training Epochs
Figure 7: CKA curves in the training phase. We visualize the CKA similarities in the training for four teacher-student pairs. The CKA values computed at the batch-level, and we average them at the epoch-level for better presentation. The corresponding mask ratios to these CKA values are visualized in Fig. 8
Feature similarity before/after distillation. Here we give more visualizations to show the feature similarity between students and teachers before/after distillation. The feature similarities measured by CKA are shown in Fig. andd the features similarities measured by Cosine are shown in Fig. I0 These results qualitatively show the effectiveness of our DPK.
A.9 COMPLEXITY ANALYSIS.
Here provide the complexity analysis in Table 17 which may be helpful the potential of the proposed model. Particularly, we report the parameters, FLOPS, and corresponding performance (top-1 accuracy on ImageNet) for some baseline models and several variants of DPK. The parameters and FLOPS are only counted for the transformation modules, and the corresponding numbers for ResNet-34 are 21.8M and 3.6G respectively. As shown in T7 we can see that is optional use several convolution layers or lightweight decoder to realize teacher/student feature transformation. Furthermore, compared with some previous feature- KD methods such as OFD (Heo et al.) 2019a). TOFD (Zhang et al. 2020), VID (Ahn et al. 2019) and ReviewKD (Chen et al. [202]] I), DPK has comparable computational overhead but always shows superior distillation results when using the same feature transformation. From these results, style encoder-decoder (1-1) can achieve a top-1 accuracy of 72.43, which surpasses all counterparts (including those with heavier transformation modules such ReviewKD). These results also suggest
also find that lightweight ViT-
19

---

Published as a conference paper at ICLR 2023
0.36
(essNet5(T)Reeseeitba
Dynamic Masking Ratio (%)
0.36
0.34
0.34
Dynamic Masking
0.32
0.30
0.30
50
100
150
200
50
100
150
200
Training Epochs
Training Epochs
ResNet101 (e)
0.61
ReaNee152(T-ResNeLLB5)
Dynamic Masking Ratio (%)
0.36
0.60
Dynamic Masking
0.59
0.34
0.32
0.57
0.56
0.30
50
100
150
200
50
100
150
200
Training Epochs
Training Epochs
Figure 8: Dynamic mask ratios. We visualize the mask ratios dynamically adjusted according to CKA. The mask ratios are adjusted at the batch-level, and we average them at the epoch-level for better presentation. The corresponding CKA curves are visualized Fig.
the performance can be further improved with stronger transformation modules, and the potential users can choose different modules accordingly.
Methods
Teacher Transform Student Transform Top-1 Acc. #params FLOPs
FitNet (Romero al.
2015)
None
1xl
70.81
5M
0.9G
AT
Zagoruyko & Komodakis
2017
Attention
Attention
70.63
9.3M
1.7G
AB
Heo al.
2019b)
Binarization
1xl
70.76
6.1M
1.4G
OFD
Heo et al.
2019a
Margin ReLU
Ixl conv
71.34
13.3M
2.5G
TOFD Zhang et al.
2020
Ixl conv
Ixl conv
70.29
22.9M
3.2G
ReviewKD (Chen et al.
2021
Ixl conv
Ixl conv
71.61
24.2M
2.8G
VID Ahn et al.
2019)
4x4 transposed conv
Ixl conv
70.43
18.8M
2.2G
DPK
Ixl conv
Ixl conv
71.39
17.2M
2.4G
DPK
None
M..P-Decoder
72.37
18.3M
2.5G
DPK
None
Enc,-Dec. (1-1)
72.43
10.5M
1.6G
DPK
None
Ene.-Dec. (2-2)
72.31
20.0M
2.7G
DPK
None
Enc.-Dec. (4-2)
72.46
32.6M
4.5G
DPK
None
Enc.-Dec. (4-4)
72.48
38.9M
4.8G
DPK
None
Enc.-Dec. (6-4)
72.46
51.5M
5.9G
Table 17: Settings: ResNet34 as student and ResNet18 student. All experiments are conducted on eight Tesla V100 GPUs using the same image augmentations, batch size. The symbols (i - j) indicates that the encoder contains layers and the decoder contains j layers.
20

---

Published as a conference paper at ICLR 2023
Resher:3
CKA (ICKD) ResNer34
CCA ResNetll ResNe:101
378
Sample Index
Sample
330
C0
0.54
1.50
Sample
Sample Iriees
Sample Incle
CKA scores ResNet18 ResNerl01
CKA scores of ResNer11 (IXD)) ResNie
CKA scores of ResNec18 (Ours) vs. ResNoc34
375
075
Sample FnSox
5amp Indox
348
3.5
155
30
Sample
CKC Resher152
CKA scores of ResNer18 HCKD) ResNet152 5P
CKA scores of ResNett (Ours) vs. BesNerl
0.75
Semple XXXX
6.80
0.00
22-
6372
6388
6.00
Sample
Index
Semple
Figure 9: Feature similarity measured by CKA. We visualize the CKA similarities for teacher- student pairs before/after distillation. We adopt the same setting used Fig. for better presentation.
21

---

Published as a conference paper at ICLR 2023
ResNet1B Restvor34
Cosine scures of ResNe(1B0CKK) vs,
Cosine scones ResNet18(Ours) Restooc24
Sample Inder
5.03
082
9.33
1.70
6783
10
Sample
Sample
Sample
Casine scores of RasNet18 vs. ResNet101
Cosine scores of ResNet1BBICKD) vs. Reshigt:
Casine scares of BiOsun)) vs. Rosh
15-
635
Samppe Index
Semple Inder
663
035
Sample Index
Sample Indox
Sample Indox
ResNet18 ResNet152
scores BrssNetBBIICD) ResNet152
scores Resteet
ResNet152
Sample Index
004
004
631
30
34
16
Samplc
Sample
Index
Figure 10: Feature similarity measured by Cosine distance. We visualize the cosine similarities for teacher-student pairs before/after distillation. We adopt the same setting used in Fig. f better presentation.
A.10 LIMITATIONS AND DISCUSSIONS
Limitations. The Fig. I 3 and Table 12 report that the proposed DPK can continue to benefit outliers. In particular, when the teachers
from larger teacher models. However, we also observe of different size belong to different model architectures, the student trained with best teacher may not perform best. For example, the ResNet- 18 trained with ConvNeXt-T achieves 71.80 top-1 accuracy on ImageNet (see Figure 44), and then it can achieve 72.51 top-1 accuracy under the guidance of ResNet-34 (see Table 2}. Although Con-NeXt-T performs better than ResNet-34, but provides less guidance to the students. Note that this does not always happen, and we argue that the better teachers still give better results when they belong the same network family.
Future work. This paper investigates a new paradigm for feature distillation, i.e. introducing the teacher's feature to the student as prior knowledge before conducting feature distillation. We show the effectiveness of this idea, meanwhile, this idea can be further explored in future work. For example, we randomly mask the student's features and fill them with the teacher's feature. A possible solution is to actively choose the prior knowledge according to some rules, such as the discriminability (Zhou) al., 2016) or uncertainty (Kendall & Gal 2017). Furthermore, different tasks may require different kinds of prior knowledge, e.g. object detection may focus more on the foreground features than background features. We hope the idea of this work and the relevant discussions can provide insights to the community.
A.11 POTENTIAL IMPACTS.
DPK aims to learn more powerful representations for the student model, and theoretically, it can be applied most CNN-based models and tasks, including these may have negative impacts on the society (e.g. face recognition). Besides, same as other data-driven methods, DPK may also give biased results if the models are trained from biased data.
22


Reviews:
[{"review_id": "qb6tCyfKdcd", "reviewer": "Reviewer_cqrC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the prior knowledge mechanism for feature distillation, which can fully excavate the distillation potential of big models. Furthermore, the authors propose the dynamic prior knowledge (DPK) to solve the ‘larger models are not always better teachers’ issue, which makes the performance of the student model positively correlated with that of the teacher model.\nFinally, experiments are conducted to verify the effectiveness of the proposed algorithm.\n", "summary_of_the_review": "It lacks evidence and reasons to support its assumption on prior knowledge, and also this assumption may be unfair to the general teaching. Besides, the authors avoided some related work from machine teaching. I don't know why.", "strength_and_weaknesses": "Main Review:\n\nStrength: \n\n1)This paper presents a dynamic prior knowledge mechanism for feature distillation to provide a solution to the ‘larger models are not always better teachers’ issue. The proposed method is simple and technically sound.  \n\n2)The experimental results of the proposed method outperform that of the other comparable distillation knowledge models.\n\n3)This paper is well-written and easy to follow.\n\nWeaknesses:\n\n1)The technical novelty of the proposed method is somewhat limited, and the authors did not provide theoretical analysis to support the proposed algorithm. This article lacks a strong reason to explain why the proposed prior knowledge mechanism can perform well. What is the relationship between the proposed prior knowledge mechanism and the feature gap?\n\n2)There is no detailed description of the progress of knowledge distillation research in the introduction. Are there any other articles that attempt to address the ‘larger models are not always better teachers’ issue? I think this article lacks the most relevant baseline, e.g. [1].\n\n3)Is the prior knowledge mechanism that incorporates the teacher's feature into the student model fair in comparison to other baselines?\n\n4)This paper lacks complexity analysis, which is important.\n\n[1] Beyer L, Zhai X, Royer A, et al. Knowledge distillation: A good teacher is patient and consistent[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10925-10934.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lC3gjj4lW1a", "reviewer": "Reviewer_ERUi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new method on feature distillation. They utilize a transformer-like module for feature alignment between each stage of the teacher and student networks. Considering the gaps between teacher and student network, they propose the mask ratios to dynamically guide the feature distillation training. The experiments obtain good performance on image classification (CIFAR-10 and ImageNet) and object detection (COCO). ", "summary_of_the_review": "See strength and weakness. \n\nThis is an incremental work for community. The authors introduce some existing techniques into feature distillation. They do lots of exmperiments to demonstrate the superiorities of the proposed DPK method, but lack detailed analyses of the essential problems for feature distillation. \n\nI am glad to improve the final rating if the authors address my concerns or point out that I misunderstand  some parts of this work. ", "strength_and_weaknesses": "Strength:\n\t\n1. The authors utilize a transformer-like module to integrate the student and teacher features for feature alignment.\n2. Considering the capacity gaps between teacher and student networks, the authors propose a dynamic mechanism for training.\n3. The authors do lots of experiments and the proposed method obtain good performance on image classification and objection detection tasks.\n4. The paper is well-organized and easy to follow. \n\t\nWeakness:\n\t\n1. The technical novelty is marginally novel for the community. Although lots of ablation studies and comparisons demonstrate the good performance of the proposed method, this paper is an incremental work. The authors introduce widely used transformer module and mask ratio for feature distillations.  \n2. The authors should give more analysis of the Receptive Field (RF) for feature distillation. For object detection, how to facilitate the different RF between teacher and student features is important for feature distillation. \n3. In Appendix 4, I notice that CRD outperforms the proposed DPK on four settings. Is the performance also influenced by the RF?\n4. I am also curious about the values of dynamic mask ratios for each stage in training. According to the difference between receptive filed, the mask ratio for the shallower stage should be larger than the deeper stage. \n5. The similar work [1] should be discussed and compared in the paper. In [1], they use nonlocal module, which is similar to transformer, for feature distillation of object detection. Although the teacher backbone is a little different, they obtain better performance (e.g. Faster RCNN R50(S), 41.5[1] vs 40.6). \n\n\n[1] Zhang, Linfeng, and Kaisheng Ma. \"Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors.\" In International Conference on Learning Representations. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "igOa_1sPgJ", "reviewer": "Reviewer_aN37", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use dynamic prior knowledge (DPK) that uses the teacher’s feature as inputs for the feature distillation by mixing the student features and using an encoder-decoder framework that learns the teacher’s feature as the target. Using DPK has an advantage in that the performance of the student network has a positive correlation with that of the teacher.\nThe performance gain in image classification datasets (CIFAR-100, ImageNet) and object detection (COCO) is remarkable, achieving SOTA performance.\n", "summary_of_the_review": "I recommend this paper to be accepted since it tackles a very important issue in knowledge distillation and solved the issue with a novel idea. It has done extensive experiments to support the proposed idea and the results.  However, its performance gain shown in the ablation study is somewhat marginal.\n", "strength_and_weaknesses": "Strength\n- To the author’s best of knowledge, this work is the first work to take the features of teachers as ‘input’, not just ‘target’ in knowledge distillation. If this is true, this paper can take credit for novelty. Also, this work showed the ‘large models are not always better teachers’ issue and tried to solve the issue with the proposed idea. This work did a good job of analyzing the introduced problem with an experiment. The proposed method shows the SOTA performance in various benchmarks regarding the classification and the object detection task. \n\nWeakness\n- It seems that the transformation module is too heavy, for example, it uses the ViT encoder and decoder module as the feature transformation module which requires heavy computation and process. Also, its dynamic mechanism process is a little hard to follow. \nIn table 8, the Encoder-Decoder shows only marginal performance gain compared to MLP-Decoder, considering the heavy computation required by the ViT encoder.\n\n- Since the encoder-decoder is ViT, it would have been more informative to address the training computational cost, even though it does not affect the test phase.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster