Function-Consistent Feature Distillation
Authors: Dongyang Liu
Meina Kan
Shiguang Shan
Xilin CHEN
Venue: ICLR 2023 poster (2023)
Keywords: knowledge distillation
feature distillation
function consistency

Abstract: Feature distillation makes the student mimic the intermediate features of the teacher. Nearly all existing feature-distillation methods use L2 distance or its slight variants as the distance metric between teacher and student features. However, while L2 distance is isotropic w.r.t. all dimensions, the neural network’s operation on different dimensions is usually anisotropic, i.e., perturbations with the same 2-norm but in different dimensions of intermediate features lead to changes in the final output with largely different magnitude. Considering this, we argue that the similarity between teacher and student features should \textit{not} be measured merely based on their appearance (i.e., L2 distance), but should, more importantly, be measured by their difference in function, namely how later layers of the network will read, decode, and process them. Therefore, we propose Function-Consistent Feature Distillation (FCFD), which explicitly optimizes the functional similarity between teacher and student features. The core idea of FCFD is to make teacher and student features not only numerically similar, but more importantly produce similar outputs when fed to the later part of the same network. With FCFD, the student mimics the teacher more faithfully and learns more from the teacher. Extensive experiments on image classification and object detection demonstrate the superiority of FCFD to existing methods. Furthermore, we can combine FCFD with many existing methods to obtain even higher accuracy. Our codes are available at https://github.com/LiuDongyang6/FCFD.

Published as a conference paper at ICLR 2023
FUNCTION-CONSISTENY FEATURE DISTILLATION
1.2.3, Dongyang Liu!:2, Meina Kan Shiguang Shan Xilin CHEN1,2
1 Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS
2 University of Chinese Academy of Sciences Peng Cheng Laboratory Ludongyang21s, kanmeina, sgshan, (icchnneicctdee en
ABSTRACT
Feature distillation makes the student mimic the intermediate features of the teacher. Nearly all existing feature-distillation methods use 1.2 distance or its slight variants as the distance metric between teacher and student features. How- ever, while L2 distance is isotropic w.rt. all dimensions, the neural network's op- eration on different dimensions usually anisotropic, i.e,, perturbations with the same 2-norm but in different dimensions of intermediate features lead to changes in the final output with largely different magnitude. Considering this, we argue that the similarity between teacher and student features should nol be measured merely based on their appearance (i.e., 1.2 distance), but should, more impor- tantly, be measured by their difference in function, namely how later layers of the network will read, decode, and process them. Therefore, we propose Function- Consistent Feature Distillation (FCFD), which explicitly optimizes the functional similarity between teacher and student features. The core idea of FCFD is to make teacher and student features not only numerically similar, but more impor- tantly produce similar outputs when fed to the later part of the same network. With FCFD, the student mimics the teacher more faithfully and learns more from the teacher. Extensive experiments demonstrate the superiority of FCFD to existing methods. Furthermore, we can combine FCFD with many existing methods obtain higher accuracy. Our codes are available at
image classification and object detection
INTRODUCTION
Deep neural networks (DNNs) have demonstrated great power on variety of tasks. However, the high performance DNN models often accompanied by large storage and computational costs, which barricade their deployment on edge devices. A common solution to this problem is Knowledge Distillation (KD), whose central idea to transfer the knowledge from a strong teacher model to compact student model, in hopes that the additional guidance could raise the performance of the student (Gou et al., 2021; Wang & Yoon, 2021). According to the type of carrier for knowledge transfer, existing KD methods can be roughly divided into the following two categories:
1.1 KNOWLEDGE DISTILLATION BASED ON FINAL OUTPUT
Hinton et al. (2015) first clarifies the concept of knowledge distillation. In their method, softened probability distribution output by the teacher is used the guidance the student. Following works (Ding et al., 2019; Wen et al., 2019) step further to explore the trade-off between soft logits and hard task label. Zhao et al. (2022) propose decouple target-class and ton-ttrget--lass knowl- edge transfer, and Chen et al. (2022) make the student to reuse the teacher's 's classifier. Park et al. (2019) claims that the relationship among representations of different samples implies important formation, and they used this relationship as the knowledge carrier. Recently, some works introduce contrastive learning to knowledge distillation. Specifically, Tian et al. (2020) propose to identify congruent (same input provided to teacher and
if pair of teacher and student representations student) or not, and SSKD (Xu et al., 2020) introduce an additional self-superrviion task to the training process. Whereas all the aforementioned methods employ fixed teacher model during distillation, Zhang et al. (2018b) 8b) propose online knowledge distillation, where both the large and the small models randomly initialized and learn mutually from each other during training.

---

Published as a conference paper at ICLR 2023
1.2 KNOWLEDGE DISTILLATION BASED ON INTERMEDIATE FEATURES
Comparing with final output, intermediate features contain richer information, and extensive meth- ods try to use them for distillation. FitNet (Romero et al., 2015) pioneers this line of works, where certain positions, intermediate features of the student are transformed to mimic corresponding teacher features w.r.t. L2 distance. FSP (Yim et al., 2017) and ICKD (Liu et al., 2021) utilize the Gramian matrix to transfer knowledge, and AT (Zagoruyko & Komodakis, 2017) uses the attention map. Generally, one student feature only learns from one teacher feature. However, some works have extended this paradigm: Review (Chen et al., 2021b) makes a student layer not only learn from the corresponding teacher layer, but also from teacher layers prior to that; AFD (Ji et al., 2021) and SemCKD (Chen et al., 2021a) make a student layer to learn from all candidate teacher layers, with attention mechanism proposed to allocate weight each pair of features. Considering that negative values are filtered out by ReLU activation, Heo et al. (2019) propose the marginal ReLU activation and the partial L.2 loss function to suppress the transfer of useless information.
1.3 OUR MOTIVATION
Extensive methods have been proposed to enhance feature distillation, from perspectives like distil- lation position (Chen et al., 2021b; Ji et al., 2021; Chen et al., 2021a), feature transformation (Yue et al., 2020; Zhang et al., 2020; Lin et al., 2022), etc. However, all these methods use L2 distance or its slight variants (Heo al., 2019) as the distance function optimize. In this paper, we argue that the simple L2 distance (as well as LI, smooth LI, etc.) has significant weakness that impedes itself from being an ideal feature distance metric: it measures features' similarity independently to the context. In other words, L2 distance merely measures the numerical distance between two features, which can be considered as the appearance of the feature. This kind of appearance- treats all dimensions in an isotropic manner, without considering feature similarity in function. The function of intermediate features is to serve as the input to the network's later part. Features playing similar functions express similar semanties and lead to similar results. Since neural network's 's op- cration on different dimensions are usually anisotropic, changing the feature by the same distance along different dimensions can lead changes in the final output with completely different mag- nitude. As a result, with only L2-based feature matching, student features with a large variety of functional differences to teacher feature are considered equally-good imitations, which inevitably impedes knowledge transfer. On the other hand, suppose that among these imitations, an additional function- based supervision could be provided to identify the better ones that lead to less change in the final output, by enforcing the student to prioritize them, the semantic consistency between teacher and student features can be better guaranteed.
distance
As a toy example, considering network that computes (5.² 1)a + 5(2r 2) Assume that the network can be divided into cascaded modules: the first calcu- lates (m1, m2) = (5x² - 1,2x 2x + 2), and the second calculates out = m1 + 5m22 (Fig. I(a)). With x as input, we have (m1, m2) = (4, 4) and out = 336. Now let a student to mimic the intermediate fea- ture, ie.. (m,, m2). Considering two pos- sible solutions: (3, 4) and (4,3), while L2 distance tells they are equally good, the function perspective points out that (4.3) is clearly better because it leads to less de- viation in final output. Comparing with m2, changes on my has stronger impact on the final output, which means task-relevant informa- tion changes more rapidly along my. The student should thus pay more attention to for learning more task-relevant information. Unfortunately, with only L2 b.2-bsed feature distillation, no such im- petus is available to push the student to prioritize (4,3) to (3, 4). As Fig. 1(b) shows, all points on the red ring have the same L2 distance to the mimicking target, while their function varies greatly. We reasonably speculate that if an extra matching mechanism, which drives the student to the solu- tion with highest function consistency, could be provided, knowledge transfer will be more effective
out = m, 5m
(mg + 5mg
836)
1000
2 2x100
(4,4333)
(3,3311)
4.0%
(3.41611
5
306255
(4,3,1255)
4
m2
12
3
3
(a)
(b)
Figure 1: A toy example illustrating the relationship between appearance and function. Points on the red ring in (b) have the same L.2 distance to target (4,4), but they cause largely different deviations in final output.
2

---

Published as a conference paper at ICLR 2023
and student could benefit more from distillation. The aforementioned phenomenon also exists in practical tasks and networks: with a ResNet32x4 model pre-trained on CIFARI00, after randomly selecting 512 directions and move intermediate features along these direction by certain L2 distance, the KI.-Divergence between the final output before and after feature manipulation averagely ranges from 0.59 1.29. When always selecting directions causing the greatest difference in output, the accuracy on the whole validation set drops by 24.42%, while it only drops by 10. 69% when al- ways selecting the directions causing the slightest difference. The results inspire us that even when the student has well mimicked the teacher's features w.r.t. L2 distance, there is still large room for further improvement if such guidance that differentiates functional differences could be provided.
In this paper, we propose mizes the functional similarity between intermediate features from teacher and student. We clarify that the similarity between features is not only determined by the features themselves, but more importantly defined by how the later layers will read, decode, and process them. Therefore, we introduce novel function- -baied feature-matching mechanism, which takes the information about later network layers into consideration. The proposed mechanism could tell the good and bad from features that are considered equally good by L2 distance, thus overcoming the weakness of existing feature distillation methods. In this way, FCFD enables the student to mimic the teacher more faith- fully and learn more from the teacher. Overall, our main contributions are summarized below:
Feature Distillation (FCFD), which explicitly opti-
1. We propose a novel knowledge distillation mechanism named FCFD, which emphasizes the matching between teacher and student intermediate features w.r.t. function.
2. Extensive experiments on image classification and object detection are conducted to investigate the effectiveness and generalization of FCFD. The results show that FCFD significantly outper- forms Ltate-of-the-art knowledge distillation methods.
3. FCFD is compatible with many advanced knowledge distillation techniques with orthogonal con- tributions. When combined with these methods, FCFD can achieve even better performance.
2
METHOD
In this section, we start with brief review of existing knowledge distillation methods in Sec. 2.1 for easier understanding of FCFD. Sec. 2.2, we specify the proposed modules that align teacher and student features w.r.t function. Finally, we introduce the complete training process in See. 2.3.
2.1 PRELIMINARIES
Original KD: Hinton et al. (2015) first clarified the concept of knowledge distillation (KD). In their work, probability distribution softened by temperature parameter is used to transfer the knowledge from teacher to student:
(I) where x is input image, T represents temperature used to soften the output distributions, denotes the softmax function, and z is logit scores output by the penultimate layer of a neural network. KL-Divergence is then used to make the student learn from the teacher:
p(z) (z))=((<<)//))
Lad KL (L((()) Il px()),,
where and denote teacher and student, respectively.
Feature Distillation: Instead of using final probability distributions, feature distillation utilizes the knowledge in the intermediate features to guide the student. Compared with probability distribution, the intermediate feature has much larger volume, and theoretically it could provide more supervi- sion. Considering a pair of teacher and student models, each of which is divided into N sequential modules (stages) and a final linear classifier. The forward process of the networks is as follows:
Pt (x) = C. M.N o...o Mt Mf (x()) = C. M.
M o Mz)
(3)
M and - denote the modules in teacher and student networks, C, and C, are their respective classifiers. Now consider the intermediate features:
F M² M² M,(x): F = M¹ ....o M2 M. (x). (x).
(4)
3

---

Published as a conference paper at ICLR 2023
With pre-defined list of distillation positions K C {1, ... .N), each position & K. the teacher feature F (hint layer) and the student feature F (guided layer) are matched for distil- lation. With bridge module B which is simple combination of one convolution and one BatchNorm (loffe & Szegedy, 2015) layer, as student transformation to match the shape of F the simplest form of feature distillation can be achieved with the following appearance loss:
Capp = L2(FF, B (1+)).
(5)
2.2 FUNCTION-CONSISTENT FEATURE DISTILLATION
We propose FCFD, where both the numerical value of features and the information about later layers are combined together for faithful feature mimicking. An illustration of FCFD is shown in Fig. 2. Due to limited space, here we focus on the methodology on image classification. However, FCFD is also applicable to object detection, and relevant introduction is deferred to Appendix D.
As analyzed before, the appearance perspective (represented by Lapp) only not satisfactory for feature matching. FCFD's solution to this problem is to investigate the function perspective through the lens of the network's later part: we feed both the teacher and the student features to the later part of the teacher/student network, and explicitly calculate the differences between consequent results as the function distance between the features. As the later part of the network defines the semantics of intermediate features, features making similar results after later network processing express similar semantics and are by definition functionally consistent. Depending on whether using the later part of the teacher the student network the lens, the proposed function-conssisenncy matching mechanism could be divided into two parts:
Using teacher's later part as the lens: given matching position k; hope that the transformed student feature B3,, (Fi) not only looks like Ft (which is guaranteed by Eq. 5), but also plays similar function E. Therefore, we feed both B (FF)) and F the later part of the teacher network, and use the similarity in results to reflect feature's similarity in function:
M MM ... Σ dra (M) 1=
N+1
yun
....0 ........tt (FF), M MM M1+ (12))
(6)
M, N+1 refers to Ct and MN++1 refers to C.. dya is feature distance metric, which set to KL- Divergence when applied to probability distributions (1 = N + 1), and set to L2 distance otherwise. If teacher and student features are really semantically consistent, we would expect that after feeding them both to the teacher's 's later part, the output will be similar. Though L2 distance is used here, does not match P and B% (Ff) directly, but match them after processed by several later teacher modules. As a result, while M/ ..... ..... o M (FF) and Mt o ..... M, k++ (B% (FP)) are matched w.r.t. appearance in Eq. 6, their underlying effect is to match F and B (F. w.r.t. function.
Using student's later part as the lens: Now that we have used the later part of the teacher model as the lens, is natural to consider the later part of student model can play similar role. We define Bi, which makes B (F.)) of equal size to Fk. and consider to pass B (F.) through the student's s later modules. However, unlike the case for Eyne where the teacher features are well-trained and informative enough to guide the student, now the transformed teacher features pass through ran- domly initialized bridge and student modules, so directly matching student features towards them not reasonable. We thus make a different choice: dent's 's later modules completely (akin to the = N + case in Eq. 6) and obtain the corresponding final outputs ps and ph.: We then match both of them towards pt, which is achieved through Lkd for ps, and f june' for pi.-
pass both F and B (FF) through the stu-
Clunc = KL. C C o...o .... MR+1 Cx MN o o Bi,(Ft)
D
p!a
In this way, we encourage FR and B (F.) to be functionally consistent w.r.t. student's later part.
Synergy between Appearance and Function Perspectives: L and L consider the func- tion perspective for feature matching, which is complementary to the appearance perspective consid- ered by Tapp: assuming appearance difference is small and constant, minimizing function distance
4

---

Published as a conference paper at ICLR 2023
Capp
Lapp
Lapp
Teacher
Pt
M
F
M
F
M
P
Ce
Lkd
M3
M3
M
Pg
||1000
Student
Ps
L Tun
Random Sampler
Tune
Pr
Pott 1,- 2-1
C Func
Parh 2 dodto
Pst
<k-1,
L f
Path <k=2, 8=0=
Pis
An overview of FCFD. Top: illustration of the traditional KD loss (Lid) and appearance- based feature matching loss (Lapp). Bottom: illustration of our proposed function matching losses fie and funce. Note that the three { terms in the figure sum up complete Cu In each iteration, randomly sample two paths (shown with yellow and purple arrows). Together with the pure teacher path (blue arrow) and pure student path (green arrow), data flow through totally four paths, and the losses appended whenever applicable. Bridge modules omitted for simplicity.
Figure 2:
makes the student to pay more attention to sensitive directions for more task-related information. The mimicking then becomes more faithful, and the student becomes more likely to interpret rather than blindly memorize. On the other hand, the mapping from intermediate features to final out- put could be many-to-one, minimizing appearance distance encourages the student features to be located in the same local area as the teacher features. In this way, more information can be exploited as guidance to the student. Ablation studies Sec. 3.3 validates the aforementioned synergy.
2.3 FULL PIPELINE
Here we introduce the full pipeline of FCFD. See Fig. 2 for illustration. Before training, a list of candidate matching positions K. is defined. According to Sec. 2.2, the complete loss function is:
L= + Least + (Ch Clynne + Jume).
(8)
kek
However, directly matching all these features causes great resource cost. Therefore, in practice, a random sampling strategy is adopted for efficiency. Specifically, for any matching position k, each of C func and L involves an extra forward path. Suppose there are K elements in K, there are thus 2K total paths. We use pair < k, > to denote a specific path, where = the path starts from the student and turns to the teacher for Lu and 5 0 when the path starts from the teacher and turns to the student for June". In each iteration, we sample a list of paths Only these chosen paths will be propagated through, and the relevant losses will be computed:
= Ltd + Liask + Σ + Σ (DC)une + - (L-
(9)
KEK
<koō>ES
Each loss term is scaled by corresponding weight hyper-parameter (which is omitted in Eq. 9) before summing up. our default implementation, there are always paths S. Combining with the pure teacher and student paths, in each iteration there are totally 4 paths involved. Note that some paths would share common data flows, and such cases we only need to calculate them once.

---

Published as a conference paper at ICLR 2023
Note that some BatchNorm layers may re- ceive inputs calculated through different paths. For example, the BatchNorm lay- ers in M may receive inputs processed by M2 Mt. M2 B2, M1. or B3, M² o M,, respectively. We empirically find that since they have different distri- butions, using the default BatchNorm be- havior to accumulate their rumning statis- tics together spoils the inference process. Therefore, for BatchNorm layers, we ac- cumulate running statistics separately for input processed by different sets of mod- ules. Mcanwhile, the affine parameters are shared among all paths (Fig. 3).
Running Statistics
Affinc
My (O)
Parameters
From Bir MGMMAMM
From Mi M
Mi-Mi Mean
Shared
Mean
Mean
Scale
Var.
Var.
Var.
Bias
M (x))
Normalize
Affine
Figure 3: An illustration of the operations within BatchNorm layer in M,. Running Statistics are ac- cumulated separately w.r.t different input paths, while affine parameters are shared among all paths.
3 EXPERIMENT
In this section, empirically validate the effectiveness of FCFD. In Sec. 3.1, we compare FCFD with state-of-the-art methods on image classification and object detection. Sec. 3.2, we com- bine FCFD with advanced KD methods with orthogonal contributions to ours, and we show that in this way, FCFD can achieve further improvement. Ablative experiments are conducted in Sec. 3.3 to dissect the effect of cach role in method. Finally, in Sec. 3.4, we experimentally demonstrate that FCFD does improve teacher-student feature similarity from the function perspective. Details about implementation and experiment settings are provided in Appendix.
3.1 COMPARATIVE EXPERIMENTS
Image Classification CIFAR100: We first compare FCFD with six existing methods CI- FAR100 (Krizhevsky et al., 2009). For methods distilling last layer's output, we select KD (Hinton al., 2015), CRD (Tian et al., 2020), DKD (Zhao al., 2022) for comparison; for intermediate feature distillation, we compare with FitNet (Romero et al., 2015), VID (Ahn et al., 2019), and Re- view (Chen al., 2021b). Following CRD (Tian al., 2020), we choose CIFAR-style ResNet (He et al., 201 Wide-ResNet (Zagoruyko & Komodakis, 2016), VGG (Simonyan & 7isserman, 2014), MobileNetV2 (Sandler al., 2018), ShuffleNetVi (Zhang et al., 2018a), and ShuffleNetV2 (Ma et al., 2018) as model architecture. Experiments are divided into two parts: in the first part, teacher and student have architectures of the same style; in the second part, they have different architectures.
Results are shown in Tab. 1. For teacher and student with similar architecture, FCFD averagely raises the student's performance by 3.70% over non-distillation baseline, and performs significantly better than vanilla KD, with average improvement by 2.05%. Furthermore, FCFD outperforms state-of-the-art methods on 4 out of teacher-student pairs, demonstrating the effectiveness of the proposed method with steady improvement. For teacher and student with different architecture, the improvement is more obvious. Comparing with non-distillation baseline, the student shows im- provement by 6.78% on average. FCFD also averagely outperforms original KD by 3.57%. More- over, FCFD is the best-performing method with all teacher-student pairs. Comparing with the second best method, Review, FCFD enjoys an average improvement by 0.66%, with the largest improve- ment by 1.11% (ResNet50-MobileNettt Mobile
Image Classification on ImageNet: To validate the efficacy of our method on large-scale datasets, we also compare FCFD with other methods on ImageNet (Deng et al., 2009). We compare with (Zagoruyko Komodakis, 2017), OFD (Heo al., 2019), SRRL (Yang et al., 2021), and four methods already considered in CIFAR 100 experiments. ResNet (He et al., 2016) and Mo- bileNet (Howard et al., 2017) are used as model architecture. The results are shown in Tab. 2. With the BesNet34-RReNette
pair, comparing with non-distillation baseline and KD, FCFD achieves im- provement by 2.5% and 1.6% top-1 accuracy, respectively. Furthermore, FCFD outperforms the second best method by 0.55% (v.s. DKD) over top-1 accuracy and by 0.2% (v.s. Review) over top-5 accuracy. With the DosNet50-Mobilldd curacy reaching 73.26%, which is 0.7% above the best existing method (Review). The results show
pair, the advantage FCFD is even larger, with top- ac-

---

Published as a conference paper at ICLR 2023
Table 1: Top-1 accuracy (%) CIFARI00. Bold and underline denote the best and the second best results, respectively. Results of DKD and Review are quoted from their paper, and others are quoted from the CRD paper. * denotes the method includes Ckd. FCFD results are averaged over three runs.
Teacher and Student of Similar Architectures
Teacher WRN-40-2
WRN-4044 WRN-40-1
ResNet56
ResNet32x4
VGG13
Student
WRN-16-2
ResNet20
ResNet8x4
VGGS
Teacher
75.61
75.61
72.34
79.42
74.64
Student
73.26
71.98
69.06
72.50
70.36
KD
74.92
73.54
70.66
73.33
72.98
CRD
75.64
74.38
71.63
75.46
74.29
DKD
76.24
74.81
71.97
76.32
74.68
FitNet
73.58
72.24
69.21
73.50
71.02
VID
74.11
73.30
70.38
73.09
71.23
Review
76.12
75.09
71.89
75.63
74.84
Ours (w/o Lud)
76.34
75.43
71.68
76.80
74.86
Ours
76.43
75.46
71.96
76.62
75.22
Teacher and Student Different Architectures
Teacher Student
ResNet32x4
WRN-40-2
VGG13
ResNec50
ResNer32x4
ShuffleV1
Shuffle
MobileNetV2
MobileNetV2
Shuffle
Teacher
79.42
75.61
74.64
79.34
79,42
Student
70.50
70.50
64.60
64.60
71.82
K
74.07
74.83
67.37
67.35
74,45
CRD
75.11
76.05
69.73
69.11
75.65
DKD
76.45
76.70
69.71
70.35
77.07
FitNet
73,59
73.73
64.14
63.16
73.54
VID
73.38
73.61
65.56
67.57
73.40
Review
77.45
77.14
70.37
69.89
77.78
Ours (w/o Cad)
78.12
77.81
70.67
71.07
78.20
Ours
78.12
77.99
70.65
71.00
78.18
Table 2: Accuracy (%) on ImageNet validation set. Setting (a): ResNet-34 as teacher and ResNet18 as student. Setting (b): ResNet-50 as teacher and MobileNet as student. Bold and underline denote the best and the second best results, respectively. denotes the method includes Lkd- For both settings, the first and the second row show the top-1 and the top-5 accuracy, respectively.
setting
Teacher
Student
KD*
AT
OFD
CRD* Review
DKD SRRL
Ours Ours*
73.31
69.75
70.66
70.69
70.81
71.17
71.61
71.70
71.73
72.24
72.25
(a)
91.42
89.07
89.88
90.01
89.98
90.13
90.51
90.41
90.60
90.74
90.71
76.16
68.87
68.58
69.56
71.25
71.37
72.56
72.05
72.49
73.37
73.26
(b)
92.86
88.76
88.98
89.33
90.34
90.41
91.00
91.05
90.92
91.35
91.24
that the proposed distillation mechanism could steadily produce high-quality models regardless of dataset volume and task complexity.
Object Detection on MS-COCO: We further validate the effectiveness FCFD on the MS- COCO (Lin et al., 2014) object detection task, using -CNNN-- with different backbone models. FGFI (Wang et al., 2019), ICD (Kang et al., 2021), and Review (Chen et al., 2021b) are adopted for comparison. Results are shown in Tab. 3. FCFD enjoys average improvement non-distillation baseline by 4.03% mAP, and consistently outperforms existing methods.
Considering both image classification and object detection, find that FCFD has greater advan- tage when teacher and student models are of different architectures, which may indicate that function consistency between teacher and student features are harder to be guaranteed without explicit opti- mization when the two models have different inductive bias.
3.2 COMPATIBILITY WITH EXISTING METHODS
In this section, we combine the proposed FCFD with existing knowledge distillation techniques with orthogonal contributions. Two recent works considered: DKD (Zhao al., 2022) and

---

Teacher Published as a conference paper at ICLR 2023
Table Compare FCFD with other knowledge distillation methods object detection, Models trained on MS- MS-COCO train2011 and tested on MS-COCO v va12017.
Method
mAP
AP50
AP75
API
APm
Aps
Teacher Student
Faster R-CNN w/ RI01-FPNN
42,04
62.48
45.88
54.60
45.55
25,22
Faster R-CNN w/ R18-FPN
33.26
53.61
35.26
43.16
35.68
18.96
w/ FGFI
35.44 (+2.18)
55.51
38.17
47.34
38.29
19.04
w/ ICD
35.90 (+2.64)
56.02
38.75
46.83
38.78
20.22
w/ Review
36.75 (+3.49)
56.72
34,00
49.58
39.51
19.42
w/ Our Method
37.37 (+4.11)
57.60
40.34
50.33
40.23
19.84
Teacher Student
Faster R-CNN w/ R101-FPN
42,04
62.48
45.88
54.60
45.55
25.22
Faster R-CNN w/ R50-FPN
37.93
58.84
41.05
49.10
41.14
22.44
w/ FGFT
39.44 (+1.51)
60.27
43.04
$1.97
42.51
22.89
w/ ICD
40.39 (+2.46)
61.14
43.97
52.22
44.16
23.69
w/ Review
40.36 (+2.43)
60.97
44.08
52.87
43.81
23.60
w/ Our Method
40.42 (+2.49)
61.01
43.66
53.77
43.61
24.63
Teacher Student
Faster R-CNN w/ R50-FPN
40.22
61.02
43.81
51.98
43.53
24.16
Faster R-CNN w/ MV2-FPN
29.47
48.87
30.90
38.86
30.77
16.33
w/ FGFI
31.16 (+1.69)
50.68
32.92
42.12
32.63
16.73
w/ ICD
32.88 (+3.41)
52.56
34.93
42.65
34.73
18.19
w/ Review
33.71 (+4.24)
53.15
36.13
46.47
35.81
16.77
w/ Our Method
34.97 (+5.50)
55.04
37.51
47.60
37.09
18.82
Table 4: Combine FCFD with DKD (Zhao et al., 2022) and SimKD (Chen et al., 2022).
Top-1
accuracy (%) on CIFAR- 100 is provided. Result format of the DKD and SimKD: result reported by original authors (our reimplemented result). means not reported.
WRN-40-2
WRN-40-2
ResNer32x4 ResNet8x4
ResNet32x4 ShuffleV1
ResNe132x4 Shuflle2
Student
WRN-16-2
WRN-40-1
Teacher
75.61
75.61
79.42
79.42
79.42
Student
73.26
71.98
72.50
70.50
71.82
FCFD
76.43
75.46
76.62
78.12
78.18
DKD
76.24 16)
74.81 (74.99)
76.32 (76.23)
76.45 (76.60)
77.07 (76.88)
FCED4DKD
76.37
75.4
76.95
78.20
78.79
SimKD
(75.73)
75.56
(75.32)
78.08 (77.98)
77.18 (77.41)
78.39 (77.74) 79.33
FCFD4SimKD
76.32
76.08
78.82
78.29
SimKD (Chen et al., 2022). We show that when combined with these methods, FCFD could achieve even better performance than its original version, demonstrating the great potential of FCFD, All experiments are conducted on CIFARI00. The introduction of DKD and SimKD, well as the technical details on how to combine these two methods with FCFD are provided in Appendix C.
Tab. shows the results. When combined with DKD, which be considered as enhanced ver- sion of traditional KL-Divergence loss function, FCFD can at most obtain an improvement by 0.61% % (ResNet32x4-Shufte Shuff--ueeeee2 V2). One significant advantage of DKD loss is that it induces additional cost during both training and inference. Therefore, this combination very friendly to practical appli- cation. When combined with SimKD, the improvement is greater, with the magnitude most 2.2% CeesNet33x4-RRsseetBB
Note that for SimKD, the last bridge module is still required during in- ference, and thus it involves extra inference cost, However, considering the introduced performance improvement, it is still worthwhile in many cases.
The improvement is much larger when the teacher is ResNet32x4 rather than WRN-40-2, one possi- ble explanation is that the performance of WRN-40-2 itself is relatively low and has barricaded the progress of the student. performance of the student has approached or even surpassed the teacher.
shown in Tab. 4, with either WRN-16-2 or WRN-40-1 as student, the
3.3 ABLATION STUDY
In this section, we conduct ablative experiments to analyze the role played by each component adopted in FCFD. Experiment results shown in Tab. 5.
8

---

Published as a conference paper at ICLR 2023
Table 5: Ablative experiments investigate the effect of each component in FCFD. Experiments are conducted on CIFAR-100, with ResNet32x4 as teacher. Top- accuracy (%) is provided.
#
Ablation
Cid + Lrask
Capp
L
Fune"
ResNet8x4
ShuffleVI
ShuffleV2
1 Baseline
74.77
74.19
75.42
Appeurance only
75.89
76.77
76.49
Function only
V
75.93
76.73
77.35
w/o Liune!
76.32
77.92
78.02
w/o fiunc
76.62
77.19
77.43
FCFD
76.62
78.12
78.18
Table 6: Further analysis to investigate student-teacher intermediate feature similarity in terms of function. Intermediate features extracted by teacher and student models are fed to exit branches, and the top-1 accuracy (%) of the outputs are reported. Dataset: CIFAR100. Teacher: ResNet32x4
Position
After the second module
After the third module
Student
ResNet8x4
V
S2
ResNet8x4
Shuffle
S2
Teacher
78.68
78.68
78.68
78.37
78.37
78.37
Lkd + Liask + FCFD
Lupp
78.
77.21
77.69
76.14
73.61
73.80
78.33
78.20
78.61
77.31
75.45
76.96
Synergistic appearance and function perspectives: Considering D2@ in Tab. 5, the results show that matching intermediate representations from appearance and function perspectives are both ef- fective approaches to improve the student's performance. After taking @ into consideration, we can find that these two perspectives can be combined together for further improvement. The results supports our expectation that the two perspectives are synergistic in intermediate feature distillation.
Complementary Fune and C func' considering C66 in Tab. 5, we can draw the conclusion that both fune and fune' are indispensable in FCFD. Specifically, in some cases (e.g.. Shuffle and S2) L brings greater improvement, while the opposite is true in other cases (e.g.. ResNet8x4). Nevertheless, when both of them are adopted, the result is always better than using either only. Note that since we always sample 2 paths per iteration (i.e., there are two elements in $), considering both funne and fiueee brings about extra training cost except the negligible static storage load to store additional bridges.
3.4 FURTHER ANALYSIS
In this section, empirically show that FCFD does help enhance teacher-student intermediate feature similarity in terms of semantics. Specifically, we append exit branches on top of intermediate positions of the pre-trained teacher model, and train the branches with the original teacher model fixed. The training task is the simple hard label classification task. In this way, we create mapping that extracts knowledge from teacher features and estimates the hard label. Note that we make the exit branches strong enough (even larger than the later part of teacher after the given position), so that the information implied teacher features could fully exploited. We then directly feed student features (after passing the bridge module) to these exit branches, and record the accuracy of outputs. The results are shown in Tab. 6. Without any exposure of student features during the training of exit branches, comparing with simple appearance-based features distillation, using FCFD raises the accuracy the output. It means that by matching features w.r.t. function, the student better comprehends the semantics under the teacher's 's features, and shares more information that task relevant. In this way, the student features become closer to teacher features w.r.t. the semantics they express. Details about the experiment settings provided in Appendix A.2.
4 CONCLUSION
In this paper, we propose novel knowledge distillation method called FCFD. FCFD aims to match the functional similarity between teacher and student features, in hopes that the student can benefit more from the distillation process. Extensive experiments demonstrate that FCFD steadily outper- forms state-of-thee the-aat methods, and can also be combined with some existing methods for further performance improvement. Furthermore, ablation studies provided to dissect the inner workings.

---

Published as a conference paper at ICLR 2023
5 ACKNOWLEDGEMENT
This work was partially supported by the National Key R&D Program of China (2021ZD01 the Natural Science Foundation of China (No. 62122074).
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In IEEE/CVE Conference Computer Vision 2019.
and Pattern Recognition (CVPR), pp.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. In AAAI Conference on Artificial Intelligence, pp- 7028-7036, 2021a. 202la.
Defang Chen, Jian-Ping Mei, Hailin Zhang. Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1933-11942, 2022.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5008-5017 2021,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Shu-Tao Xia. Adaptive regularization of labels. arXiv preprint arXiv: 05474, 2019.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision (ICC)), 12996);179--1899, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. Ad- vances in Neural Information Processing Systems (NIPS), 2015.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704007700..46.,,, 2017.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International conference machine learning (ICML.) pp- 448-456, 456, 2015.
Mingi Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via attention-based feature matching. AAAI Conference on Artificial Intelligence, pp. 7945-7952, 2021.
Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. coniancecooniiioonll knowledge distillation for object detection. Advances in Neural Information Processing Systems (NeurIPS), pp. 16468-16480. 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2023
Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10915-10924, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision (ECCV), pp. 740-755, 2014.
Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for unversity-preservee knowledge distillation. In IEEE/CVF International Conference Computer Vision (ICCV), pp. 8271-8280 2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In European Conference on Computer Vision (ECCV), pp- 116-131, 2018.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3967-3976, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations (ICLR), 2015.
Liang-Chich Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations (ICLR), 2020.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual in- telligence: A review and new outlooks. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2021.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
pp.
Tiancheng Wen, Shenqi Lai, and Xueming Qian. Preparing lessons: Improve knowledge distillation with better supervision. arXiv preprint 1rX1v:1911.07471, 2019.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-
supervision. In European Conference on Computer Vision (ECCV), pp. 588-604, 2020.
Jing Yang, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos, et al. Knowledge distillation via softmax regression representation learning. International Conference on Learning Represen- tations (ICLR), 2021.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4133-4141, 2017.
Kaiyu Yue, Jiangfan Deng, and Feng Zhou. Matching guided distillation. In European Conference on Computer Vision (ECCV), pp. 312-328, 2020.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
160iv:1605.07144 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. International Conference on Learning Representations (ICLR), 2017.
11

---

Published as a conference paper at ICLR 2023
Linfeng Zhang, Yukang Shi, Zuoqiang Shi, Kaisheng Ma, and Chenglong Bao. Task-oriented feature distillation. Advances in Neural Information Processing Systems (NeurIPS), 33:14759-14771, 2020.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6848-6856, 2018a.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In IEEE/CVF conference computer vision and pattern recognition (CVPR), pp. 4320-4328, 2018b.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distilla- tion. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11953- 11962, 2022.
12

---

Published as a conference paper at ICLR 2023
A EXPERIMENT DETAILS
A.1 BASIC SETTINGS
Here we introduce the basic settings of our experiments. These settings are used for comparative experiments in Sec. 3.1, and are also used in other experiments unless otherwise specified.
CIFARI00 The CIFARIO0 (Krizhevsky et al., 2009) dataset consists of 60K images from 100 cat- egories with size of 32 x 32. In the standard protocol, 50k images are used for training and 10k for testing. Following CRD (Tian et al., 2020), we train all the models for 240 epochs, and the learning rate is decayed by a factor of 10 at 150., 180, and 210 epochs. respectively. The initial learning rate is 0.01 for MobileNetV2, ShuffleNet, ShuffleNerV2, and is 0.05 for other student models. The batch size is 64, and SGD optimizer with a 0.0005 weight decay and 0.9 momentum is adopted. For all experiments, the weights for Ltask and Lkd are set to 1. The KL-Divergence term in & fune and fune share the same weight that is either or 0.2. Lapp and L2 terms in L share the same weight, which is either 0.02 5.0. For Lkd. the KL term L func: and the KL term in & Lunc', the same temperature (Hinton et al., 2015) is used and is either 4 or 8.
The ImageNet (Deng et al., 2009) dataset consists of 1.28 million training images and 50k validation images from 1000 categories. Following the mainstream settings, all methods are trained on the entire training set and evaluated on the single-crop validation set. The input image resolution is 224 x 224 for both training and evaluation. Following mainstream settings (Chen et al., 2021; Tian et al., 2020), we train all the models for 100 epochs. The initial learning rate is 0.1 and is decayed by a factor of 10 at 30, 60, and 90 epochs, respectively. We run experiments on one Tesla- V100 GPU with batch size 256 and initial learning rate 0.1. Automatic Mixed Precision (AMP) provided by PyTorch is used for acceleration. An SGD optimizer with 0.0001 weight decay and 0.9 momentum is adopted. The weights for Lrask, Lkd, finnee' and the KL--Divereence term in C fnes are set to 1, and the weights for Lapp and the L2 term in fane are set to 0.02 for R8sNet18 and 5 for MobileNet. The temperature for Lkd, the KL term in L funce and f is set to 1.
ImageNet
The design of the bridge module (-)) follows CRD (Tian et al., 2020). Specifically, The bridge module is composed of exactly one convolution layer and one BatchNorm Layer When the mim- icking target is Rest- feature, and additional leaky ReLU activation is appended. The kernel size is 3x3, and is of stride 2 when the target feature 2 times smaller, and is of stride 1 when the target is as large the source feature. When the target is 2 times larger than the source, transposed convolution with kernel size 4 and stride 2 is used.
A.2 SETTINGS FOR FURTHER ANALYSIS
Here we introduce the details of the experiments conducted in Sec. 3.4, We first select two interme- diate positions. Specifically, considering that ResNet32x4, ResNet8x4, Shuffle and Shuffle are all composed of four cascaded modules, select the end of the second and the third module. For all the experiments, we first take a ResNet32x4 model pre-trained on CIFAR 100 (which is the teacher model for distillation), and add extra exit branches the aforementioned positions. For each position, the structure of the exit branch is the same as the later part of a ResNet50x4 model after this position. More specifically, the structure of the exit branch at the end of the second module is the same as the combination of the last two modules of ResNet50x4, and the structure of the exit branch at the end of the third module is the same as the last module of ResNet50x4. In this way, we guarantee that the exit branch is strong enough and could well capture the task relevant information implied in teacher features.
The exit branches are then optimized, with teacher features as input, to estimate the hard label. Cross entropy loss is used as loss function. Similar to settings in Sec. A, 1, the training process lasts for 240 epochs. The initial learning rate is 0.05 and is decayed by a factor of 10 at 150, 180, and 210 epochs, respectively. Batch size is 64. SGD optimizer with a 0.0005 weight decay and 0.9 momentum is adopted. Note that the ResNet32x4 backbone model is frozen in this process.
After training, we directly feed student features (transformed by the bridge modules) to these exit branches, and test the performance on CIFAR100 test set. Since the running statistics in the Batch- Norm layers in the exit branches are accumulated with teacher features as input, we re-calibrate the running statistics on CIFAR 100 training set before testing.
13

---

Published as a conference paper at ICLR 2023
From Student From Teacher Overlap
FCFD
Leg Leask Eapp
Visualization of the semantic similarity between teacher and student intermediate features. We images belonging to the first 10 classes in the CIFAR- -F0 test set. Teacher: ResNet32x4. Student: S2uffleV2. Student and Teacher features originally show light blue and light red @. When the student feature overlaps with the teacher feature, the overlapping region will become dark brown o. With FCFD, fewer light blue and light red points can be observed, signifying that more student features similar their corresponding teacher features. Best viewed color with zoom in.
Figure 4:
B VISUALIZATION FOR FEATURE SIMILARITY
We have empirically shown in Sec. 3.4 that FCFD improves the semantic similarity between teacher and student features. In this section, we provide visualization to further demonstrate this effect. We feed the teacher and student intermediate features into the exit branches mentioned in Sec. 3.4 and Sec. A.2. We then collect the corresponding penultimate features and use t-SNE to project these features into points in IR The results shown in Fig. 4. With only aopearance-based feature matching, there are a lot of light-blue (o From Student) and light-red (o From Teacher) points. These points show the light color because they do not overlap with the nearest points showing the other light color, i.e., the student features relatively far from the nearest teacher features and vice versa. In contrast, with FCFD there are fewer light-color points and more dark-brown (0 Overlap) points, which means more student features are similar to their corresponding teacher features. The results support our claim in Sec. 3.4.
C DETAILS ON COMBINING FCFD WITH EXITING METHODS
In this section, we briefly introduce how we combine DKD (Zhao et al., 2022) and SimKD (Chen al., 2022) with FCFD.
C.1 COMBINE DKD
The key idea of DKD (Zhao al., 2022) to decompose traditional knowledge distillation into two parts: Target Class Knowledge DistillationTCCKD),, which matches the teacher's 's and student's binary probabilities the target class, and Non-Targe -on-Target Class Knowledge DistillationNNCKD),, which optimizes the similarity between the teacher's and student's probabilities among non-target classes. In traditional KD, the weights of TKCD and NCKD terms are coupled with teacher's confidence on the target class. In contrast, DKD propose assign constant weights to these terms.
The DKD loss can be considered as an enhanced version of KL-Divergence. Therefore, to combine DKD with FCFD, fume, In our experiments, we set weight for TCKD to 1, and set weight for NCKD to 2. Other training settings are aligned with those in Sec. A.1.
simply replace with DKD loss the KL-Divergence loss in Lkds Cfune- and
14

---

Published as a conference paper at ICLR 2023
C.2 COMBINE WITH SIMKD
The key idea of SimKD is to make the student reuse the teacher's classifier layer. In this way, the forward process of the student is:
M² o.... Pu(simka) (z) = C2 B M, ....o (x)
(10)
Note that unlike other methods, for SimKD, the last bridge module, By is still required during inference, and thus involves extra inference cost.
To combine SimKD with FCFD, whenever we originally feed feature to C.,, now as the substitute, feed feature to C₁ o BA Specifically, the computation of ps and pi (ar) now ends up with C B rather than C.e All other settings, including target function, training strategy, are not changed. During inference, the forward path of the student is the same as Eq. 10.
As suggested in Chen et al. (2022), the capacity of the bridge module B has significant impact on the final accuracy, and there exists an accuracy-latency trade-off. To make experiment results more informative, when combined with SimKD, we change B to exactly match the architecture used in the original SimKD paper, and the squeeze factor set Note that bridges modules other than B are not changed for simplicity.
D DETAILS ABOUT OBJECT DETECTION
In Sec. D. we introduce the methodology of migrating FCFD to object detection. In Sec. D.2, we show the implementation details.
D.1 METHOD
On object detection, do match features by feed the transformed student (teacher) backbone features to the later part of the teacher (student) backbone model. The reason are twofold: 1) whereas the propagation of image classification models is unidirectional, intermediate features calculated by detector's 's backbone model may be revisited long after its first usage due to the existence of FPN (Lin al., 2017), region proposal networks (RPN), and ROI heads. Therefore, directly matching inter- mediate features calculated by detectors' backbone models w.r.t. function will make the subsequent propagation process complicated, and could involve great extra overhead; 2) existing works, the common practice for conducting feature distillation
detectors is to match the features output by FPN (Chen et al., 2021; Wang et al., 2019; Yang et al., 2022), rather than by the backbone. When considering how to functionally align FPN features, the situation becomes clearer and easier. There- fore, the detection version of FCFD is designed to match FPN features.
Denote the list teacher features output by FPN as F, = M and the list of student features output by FPN F. { number of FPN features. Given an input image X and a region proposal r, the ROI head (denoted as ROIH) selects the corresponding part of feature from F, and outputs the probability p(x, r) and positional offset 6(x, r) corresponding to all candidate classes:
Jk=1
Here with slight abuse of notation, use K to denote the
< pe(æ, &x,,) > RO.H,
(11)
(2. >= (r,
(12)
subscript and denote teacher and student, respectively. Eq. 11 and Eq. 12 show that the function of F is serve as the input to the ROI head module. Following the same idea as our method on classification, we match F, and F,, w.r.t. both appearance and function simultaneously. Appearance matching is achieved by:
C. (2 B (F
(13)
Where By is the student--o-teacher bridge module. For function matching, we first use the teacher's ROI head (ROTH,) as the lens to measure the functional similarity between teacher and student features: we feed transformed student features to ROIH;
Pss (x. r), > RCTH,(r.
eF.))
(14)
15

---

Published as a conference paper at ICLR 2023
and then align the results with those given by the teacher:
fiune
Σ PER
(K) + L2(0, (x,
(15)
I denotes the set of region proposals fed to ROI heads. We then use the ROI head of the student (ROIH,) as the lens for function matching: we feed both student features and transformed teacher features to ROIHa, and optimize both results towards that given by the teacher. For the student features, the optimization is achieved by Lkd:
Lkd
(16)
PER
and for the transformed teacher features, the optimization achieved by Liune
(2, >=
E
(17)
Σ
(KLL(p( L2( ((,(x)))
(18)
PER
Bis denotes the teacher-to-studen bridge module.
In conclusion, the complete loss function for our detection-version FCFD is:
== Lrask E + ffunc + Li + Lidd
(19)
Crask here denotes the original loss terms used in object detection. Since the ROI heads generally contain few parameters and the propagation is fast and efficient, the induced extra overhead for directly optimizing Eq. 19 is limited and acceptable. Therefore, random sampling mechanism like that proposed Sec. 2.3 is applied for object detection.
D.2 IMPLEMENTATION DETAILS
Our implementation based on Detectron2 (Wu et al., 2019). The weights for all of our proposed loss terms are set to 1. Temperature is set to I when calculating the KL-Divergence between two distributions. Automatic Mixed Precision (AMP) provided by PyTorch used for acceleration. Batch size is set to 8 and initial learning rate is set to 0.01. For fair comparison with existing methods, we adopt the 1x training procedure. All experiments are conducted on one Tesla- V100 GPU. Other settings are inherited from Detectron2 and left unchanged.
E
MORE EXPERIMENTS ON OBJECT DETECTION
Inheriting strategy, which initializes the student's non-backbone modules with the teacher's pre- trained parameters, was proposed in ICD (Kang et al., 2021) and was shown to boost the student's performance clearly. We show in Tab. that FCFD is also compatible with this strategy. Note that the inheriting strategy is applicable only when the output feature from the teacher's 's and the student's 's pair.
backbones have the same shape, so we report the results of the ResNet101-ResNNer
Following Review (Chen et al., 2021) and ICD (Kang et al., 2021), for object detection, we con- duct experiments on Detectron2 (Wu al., 2019). Therefore, in Tab. we compare with methods implemented on the same framework for fairness. Nevertheless, we also notice that some other works (Yang et al., 2022: Dai et al., 2021) report their performance on MMDetection (Chen et al., 2019). Here re-implement our FCFD MMDetection and compare it with these works to provide readers with a comprehensive view FCFD's 's performance, see Tab. for results.
F ONLINE KD
We also validate the proposed method in the context online knowledge distillation (Zhang et al., 201 8). Specifically, both the student and the teacher are randomly initialized before training. During
16

---

Published as a conference paper at ICLR 2023
Table 7: Results of distillation with inheriting strategy. f denotes the inheriting strategy.
Method
mAP
AP50
AP75
API
APm
Aps
Teacher Student
Faster R-CNN w/ RI01-FPN
42.04
62.48
45.88
54.60
45.55
25.22
Faster R- CNN w/ R50-FPN
37.93
58.84
41.05
49.10
41.14
22.44
w/ Our Method
40.42 (+2.49)
61.01
43.66
53.77
43.61
24.63
w/ Our Method!
40.96 (+3.03)
61.78
44.79
54.55
44.40
24.19
Table 8: Comparison with existing methods on MMDetcction. Results other than ours are quoted from Yang et al. (2022). f denotes the inheriting strategy.
Method
mAP
API
APm
Aps
Teacher Student
Faster R-CNN R101-FPN
39.8
52.8
43.6
22.5
Faster R-CNN w/ R50-FPN
38.4
50.3
42.1
21.5
w/ FGFI (Wang et al., 2019)
39.3
52.2
42.3
22.5
w/ GID (Dai et al., 2021)
40.2
53.2
44.0
22.7
FGD (Yang al., 2022)
40.4
53.5
44.5
22.8
w/ Our Method
40.8 (+2.4)
53.4
44.9
23.0
w/ FGD+
40.5
53.2
44.7
22.6
w/ Our Method
40.9 (+2.5)
53.9
44.8
23.4
training, the teacher and the student learn mutually from each other. Apart from the loss function Eq. 9, the task loss w.r.t. teacher output and hard label, well KD loss that the teacher learns from the student, are appended.
The training settings are the same as those in Sec. A.I. DML (Zhang et al., 2018) and CRD (Tian et al., 2020) are used for comparison. For DML, we re-implement the method for results. For CRD, we directly quote the results provided the appendix of their paper. As shown in Tab. 9, FCFD not only performs well on offline knowledge distillation, but also outperforms existing methods on online knowledge distillation. This shows that the idea of matching intermediate features w.r.t. function is effective in wide range of scenarios.
G ABLATION OVER THE DESIGN OF Lune AND L fiune
In this section, we delve deep into the design choices of Lfume and Liune'-
Considering Eq. 6, fune be divided into two parts: the KL-Divergence loss defined the final outputs (1 = + 1), and L2 losses defined on earlier layers (1 < + 1). We denoted the aforementioned KL-Divergence part Lface-K' K.ne-KK, and the L2 part as Cfanc-12-
Considering Eq. 7, for reasons stated in Sec. 2.2, it only involves a KL term. To investigate if feature matching loss similar to Liune- 1. is useful in the case of Liune', define the following loss:
N
c Cjund'L2
Σ 1=
L2
M o..... M ( (Bt) o... M (r+t)
(20)
We conduct experiments to analyze these terms. Results are shown in Tab. 10. First, we find that C) fune 1.2 is generally not useful. Moreover, when independently used, Lyune-L2 brings more improvement than f KL. though it is better to combine them both.
H COMPLEXITY ANALYSIS AND EFFICIENT DESIGNS
In this section, we start with the complexity analysis of FCFD. The analysis suggests some directions to lower the training of FCFD, and thus experimentally explore these directions in the rest of this section. We hope that the contents in this section could guide engineers to customize their implementation of FCFD if the training cost is concern.
17

---

Published as a conference paper at ICLR 2023
Table 9: Validate the effectiveness of FCFD in the context of online knowledge distillation. Both teacher and student models are trained from scratch and learn mutually from each other. Top-1 test accuracy (%) is reported. We 't' and 's' denote teacher and student models, respectively, Bold denotes the best result.
Teacher
WRN-40-2
WRN-40- WRN-40-1
resnet56
resnet 10
resnet32x4
vgg
Student
WRN-16-2
resner20
ResNet32
resnet8x4
vgg8
Vanilla 75.61
75.61
72.34
74.31
79.42
74.64
DML
77,96
77.88
74.68
75,47
79.68
75.83
CRD
78.01
77.39
73.86
75.53
79.36
77.23
FCFD
78.95
78.65
75.13
77.4
80.21
76.62
Vanilla
73.26
71.98
69.06
71.14
72.5
70.36
DML
75.24
73.69
70.99
72.4
74.23
72.55
CRD
75.89
74.12
70.9
73.07
75.34
74.08
FCFD
76.44
75.01
72.01
74.22
76.7
74.65
Table 10: Detailed ablation over the design of L and Liune'. Teacher: ResNet32x4. Dataset:
CIFAR-100 100. Top-1 accuracy (%) reported.
Diat Liask Luppl CSane-KL Lsane-L2 L2 Dyune' Lyanc - L2
ResNet8x4
Shuffle S2
76.
77.46
77.85
76.43
77.94
77.84
76.32
77.92
78.02
76.62
77.19
77.43
75.58
76.56
76.30
76.62
78.12
78.18
V
76.57
77.82
78.14
H.1 COMPLEXITY
Suppose that both teacher and student have N stages. Given a path used for feature matching k2 >. the triggered additional training cost is:
cost BB:) + k+1 (os)(M)) + cost(C.), 5= 0 cost B + ++++1 cost(M)) cost(Ci), 8 (21)
Additional Training Cost
Comparing with existing methods, our FCFD involves specially cost terms Σe cost (M!) and E 1 c)s)(M)). However, FCFD only needs very simple bridge modules and does not in- corporate complex mechanisms like contrastive learning (Tian et al., 2020; Xu et al., 2020), which lowers the overall complexity of FCFD.
There are mainly two factors that determine the practical cost of FCFD; the number of paths sampled every iteration, |3], and from which stage we start do functional matching, namely the least value of & in < k, >, and we denote this as kmin- Given a path < k ô >, the smaller & is, the more extra blocks will be passed through, so larger kinin lowers the overall extra cost.
After dividing the networks into 4 stages, there are 4 candidate positions for features matching: F', F², F3, and Fr, In practice, we set kmin 2. and we also do not functionally match the last feature, FT, since the focus of FCFD is to investigate the effectiveness of functional matching over intermediate features. There are thus four candidate paths: {< k: = 2, = >,< k = 3, 8 = >,< >}. In each iteration, mentioned in Sec. 2.3, we by default
k 2,6 8 1 >,< & 3, sample two paths from the candidates.
Now we empirically compare the training speed and memory cost of FCFD with existing meth- ods. We choose the M.ANeetxx-ReeNet322 pair on CIFARI00. Experiments are conducted on the same machine with irrelevant variables aligned. Results shown in Tab. 11. DKD (Zhao al., 2022) is the least complex method and its complexity is by theory exactly identical to the original KD (Hinton al., 2015). Meanwhile, FCFD is faster than SOTA feature distillation method
18

---

Published as a conference paper at ICLR 2023
Table 11: Training complexity. Teacher: ResNet32x4. Student: ResNer8x4. Dataset: CIFAR-100.
Method
DKD
Review
CRD
FCFD
Seconds / Epoch
33.12
48.40
54.02
43,98
Peak GPU Memory Usage (MB)
459
1231
790
1132
Top-1 Acc (%)
76.32
75.63
75.56
76.62
Table 12: Effect of different number of sampled paths ([S)) for functional feature matching. Teacher: ResNet32x4. Dataset: CIFAR-100 Top-1 accuracy (%) is reported for all student models, and training cost is reported for ResNet8x4 for reference.
Shuffle V Top-1
ResNet8x4
S
Top-1
Top-1
Seconds / Epoch
Peak Memory (MB)
1
77.50
77.69
75.92
36.76
868
78.12
78.18
76.62
43.98
1132
4
78.20
78.98
77.07
61.27
1268
Review (Chen et al., 2021) and CRD (Tian et al., 2020). The results show that training complexity not significant shortage of FCFD.
H.2 EFFECT OF NUMBER SAMPLED PATHS [S]
We show in Tab. 12 the effect of using different number of sample paths per iteration. A clear trend that more sampled paths improve the accuracy while inducing longer training time and larger memory costs. We find that with all 4 candidate paths selected in one iteration, the final accuracy of the student model can be very high. Therefore, for scenarios where model accuracy is relatively ample, recommend directly setting
of great importance while computing resources larger [S]. When resources are limited, [S] could be lowered accordingly.
H.3 EFFECT OF DIFFERENT MATCHING POSITIONS
As mentioned before, it brings more training costs to functionally match shallow features. As result, starting function matching since a relatively deep position (i.e., increasing kmin) can lower the cost of FCFD. We conduct experiments to explore how the performance of the student changes select different positions for functional feature matching, the results shown in Tab. 13.
The conclusion is that, while functional matching is effective over both shallow and deep features, brings larger improvement when applied only to deep features than only to shallow features. Meanwhile, involving both shallow and deep features with random sampling is of medium speed, and performs the best. Taking the ||] = 4 case in Tab. 12 into consideration, the advantage from functionally matching shallow and deep features can be accumulated.
conclude that
H.4 EFFECT OF USING PARTIAL TERMS IN Li
As shown in Eq. 6, for calculating I-funes make B&, (P²) pass through all the later teacher modules (1 E 1, N+1.. Another possible approach to lower the training cost of FCFD is to only propagate 3% (FF)) through the first several teacher modules. Since in scenario transformed student feature at most passes through teacher modules (classifier C, excluded), we conduct an experiment to see how the student performs when Lfune only involves one-module propagation, namely:
unnc-partial L2 M (F²), Mi+1 Bh (F!)
(22)
The result is shown in Tab. 14. First, replacing Cl with C) fune-partial as most of the improvement is preserved after the replacement. However, compared with the func- tionally match F$ only strategy in Tab. 13, the partial-propagation strategy achieves lower accuracy
partial is an effective strategy
19

---

Published as a conference paper at ICLR 2023
Table 13: Effect of conducting functional matching different positions. [8] 2 for all experiments. Teacher: ResNet32x4. Dataset: CIFAR-I00. Top-1 accuracy (%) is reported for all student models, and training cost is reported for ResNet8x4 for reference.
ShufifleVI Top-1
ShuffleV2
ResNet8x4
Setting
Top-1
Top-1 Seconds Epoch Peak Memory (MB)
Functionally match F only
77.03
77.78
76.46
49.38
1132
Functionally match F only
77.75
77.90
76,60
38.73
891
Functionally match F2 and F3
78.12
78.18
76.62
43.98
1132
Table 14: Replacing C fune with C func-ponn fancepartial- Teacher: ResNet32x4. Dataset: CIFAR-100. Top-1
accuracy (%) is reported for all student models, and training cost is reported for ResNet8x4.
ShufffleV1 Topl
S2
ResNet32x4
Setting
Top1
Topl
Seconds/Epoch Peak Memory (MB)
with C with C) "fune-partin
78.12
78.18
76.62
43.98
1132
fune
77.69
77.72
76.52
37.13
980
while the training cost is almost similar. Therefore, in practice, if the computing resources really scarce, it is suggested to try raising kimin first before trying L partial-
I DIFFERENCES BETWEEN FCFD AND EXISTING WORKS
Some existing works (Bai et al., 2020; Li et al., 2020a;b; Yang et al., 2021) also feed representa- tions from one network to another network to distill knowledge. In this section, we analyze their differences from FCFD.
From the perspective of problems to solve, Bai al. (2020) and Li et al. (2020a) are designed to solve problems that only exist in special topics and do not generalize to common KD scenarios. Specifically, Bai et al. (2020) aims to alleviate overfitting in few-shot distillation, they propose a block-wise supervision strategy. Li et al. (2020a) focuses on a more special scenario, where a residual network helps non-residual network to overcome gradient vanishing. In contrast, FCFD is motivated by the function consistency problem that widely exists in feature distillation scenarios, and is applicable and effective in general knowledge distillation cases. From the methodology per- spective, there are still clear differences: Bai et al. (2020) directly matches the behavior of paired teacher and student modules, so they feed the same input feature the two blocks and hope the output could be similar. In contrast, FCFD takes a matchig perspective and hopes paired features could lead to similar outputs after process by the same module. Our method can deal with situations where teacher and student features have different shapes, while Bai et al. (2020) cannot. For Li et al. (2020a), though they do put student features into the teacher network, their aim is to obtain the residual gradient which is originally inaccessible for non-residual network. Further- more, they do not feed teacher features into the student modules, which we show to be effective with ablation. While SRRL (Yang et al., 2021) does target the general KD scenario, it focuses narrower problem of penultimate feature matching and can be considered as a special case of FCFD, where only penultimate features are functionally aligned with Lfunc- The comparative experiment in Tab. also shows that FCFD clearly outperforms SRRL.
REFERENCES
Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation. In AAAI Conference Artificial Intelligence, volume 34, pp. 3203-322 3210, 2020.
Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11933-11942. 2022.
20

---

Published as a conference paper at ICLR 2023
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:
2019.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp- 5008-5017, 2021.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for object detection. In IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 7842-7851, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. Ad- vances in Neural Information Processing Systems (NIPS), 2015.
Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. instance-conditional knowledge distillation for object detection. Advances in Neural Information Processing Systems (NeurIPS). pp. 16468-16480, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Guilin Li, Junlei Zhang, Yunhe Wang, Chuanjian Liu, Matthias Tan, Yunfeng Lin. Wei Zhang. Jiashi Feng, and Tong Zhang. Residual distillation: Towards portable deep neural networks without shortcuts. Advances in Neural Information Processing Systems (NeurIPS), pp. 8935-898899 1946-894, 2020a.
Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, and Ke He. Explicit connection distillation. 2020b.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE conference on computer vision and pattern recognition (CVPR). pp. 2117-2125, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations (ICLR), 2020.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4933-4942, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https: / github.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference on Computer Vision (ECCV), pp. 588-604, 2020.
Jing Yang, Brais Martinez, Adrian Bulat, Georgios Tzimiropoulos, et al. Knowledge distillation via softmax regression representation learning. International Conference on Learning Represen- tations (ICLR), 2021.
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4643-4652, 2022.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In 2018.
IEEE/CVF conference on computer vision and pattern recognition (CVPR). pp.
Borui Zhao. Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distilla- tion. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11953- 11962, 2022.
21


Reviews:
[{"review_id": "tCYR8F5R9Q", "reviewer": "Reviewer_8x8h", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new knowledge distillation (KD) method to better align student features with teacher features. The authors claim that traditional L2 loss for feature-level KD is not appropriate due to the deviation of functional mapping of similar features. In order to alleviate this problem, a function-consistent feature-level KD is utilized to align the cross-propagated features of both the student and the teacher. Various designs, such as separate batch statistics and path sampling, are incorporated to help the training and inference process. Experiments on both classification and object detection datasets demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper could be accepted if the authors can answer the aforementioned concerns to some extent.", "strength_and_weaknesses": "1. The asymmetric loss design for the student and teacher network requires more ablation studies. Specifically, when “using student’s lateral part as the lens”, the authors think that “randomly initialized bridge” may bring unstable training status, and thus feature-level supervision is not available in \"$L_{func’}^{k}$\". When considering “teacher’s lateral part”, the bridge module still exists, however, the feature-level supervision is activated in \"$L_{func}^{k}$\". There had better be ablations like:\n    1. Only KL supervision in \"$L_{func}^{k}$\"\n    2. KL plus L2 supervision in \"$L_{func’}^{k}$\"\n\n    Moreover, the following ablations are also expected:\n    1. Only L2 supervision in \"$L_{func}^{k}$\"\n    2. Only L2 supervision in \"$L_{func’}^{k}$\"\n    3. L2 supervision on partial layers (Currently, all layers after the \"k-th\" one seem to be considered in the loss. Is it the best practice?)\n\n2. For the path sampling scheme used in this paper, the authors simply sample two paths during training. I wonder the effects of different path sampling schemes, such as the number of sampled paths, the sampling probability (uniformly or imposing some priors to different layers) and the relative number of teacher-to-student and student-to-teacher paths if there are more than two paths. By the way, the sampling seems to be canceled for object detection, it might also work for classification to freeze the sampled path during the whole training process. In short, I think more ablations about the sampling scheme should be included in this paper.\n\n3. Although the authors have repeatedly emphasized the extra training cost is acceptable, I think some numerical indicator, such as the training time and GPU memory peak, should be included in the experimental results.\n\n4. Can the proposed method work for segmentation? Intuitively, I still think the propagation of sampled paths would introduce a lot of cost, which might hinder the training of dense prediction tasks, such as semantic segmentation.\n\n5. For the separate batch statistics, which one will be applied during inference, or is there some fusion scheme to mix statistics of different paths during inference?\n\n6. I observe that “FCFD+DKD” is inferior to “FCFD” under the settings of “WRN-40-2/WRN-40-1” and “WRN-40-2/WRN-16-2” in Table 4. Still in Table 4, the result of “SimKD” under “WRN-40-2/WRN-16-2” is missing, and “FCFD+SimKD” is also inferior to “FCFD” under this setting. Is there an explanation for these results?\n\n7. Can the authors provide some feature visualizations to explain the results in Table 6? For example, the PCA of student and teacher features w.r.t. “FCFD” and \"$L_{kd}+L_{task}+L_{app}$\".\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "7kIxy2acWiN", "reviewer": "Reviewer_ScN2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a function-consistent feature distillation method in the computer vision scenario. The motivation of the proposed method is based on the assumption that the widely used l2 loss between features does not take function-consistent into consideration, bearing the consequences that the student model may suffer from the negative transfer. Then the author introduces Function-Consistent Feature Distillation(FCFD), which measures the similarity between student and teacher's outputs generated by the lateral part of the same network. Extensive experiments show the improvement of the proposed method.", "summary_of_the_review": "The paper proposes a feature distillation method targeting the computer vision scenario.\nAlthough the author claims that the conventional l2 loss between features contains some unfixed flaws, the motivation and insight are not clear. A toy example of shallow learning is far from supporting the claim. \nOn the other hand, the proposed solution is complicated. Considering the improvement in performance is not significant (needless to say, some important baselines are missing), the whole cost is too high to be practical in the real industry environment.\nOverall, I think the paper is below the borderline of ICLR. ", "strength_and_weaknesses": "Strength:\n- The paper proposes a novel function-consistent feature distillation method targeting the computer vision scenario.\n- The discussion on negative transfer from L2 loss between features is interesting. \n- The experimental evaluation is adequate, and the results somehow support the main claims.\n\nWeakness:\n- The motivation is not clear to me. The author claimed in the paper that there are some intermediate Conv and BN modules to map the student features to teacher features into the same size. Some methods, such as Review[1] and MGD[2], all consider multiple projection layers. It seems that the negative transfer from L2 loss is curtailed by these projection modules. I wonder if there is any theoretical analysis that supports the author's claim. A toy example of shallow learning can not provide adequate support for author's claim from a deep learning perspective. \n- The method is complicated. The proposed method considers both the student and teacher's lateral parts of the network, and then the training cost is relatively high. To me, the minor performance improvement can not make up for the additional complexity of the system.\n- Some important experiments are missing: 1) The details of the projection module B() is missing. 2) In table 1 and table 2, the results w/o KD are omitted. \n\n[1] Yang, Zhendong, et al. \"Masked Generative Distillation.\" arXiv preprint arXiv:2205.01529 (2022).\n[2] Chen, Pengguang, et al. \"Distilling knowledge via knowledge review.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nI hope the authors can reply to me on the following:\n- More mathematical insights on negative aspects of l2 loss.\n- The complexity analysis of the proposed distillation method.\n- The details on projection module B() and the missing experimental results on table 1 and table2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Ei3byzmbPy", "reviewer": "Reviewer_co3V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the new way to measure the distance between the teacher and student features, for better feature distillation (FD) performance. Specifically, the authors first point out the commonly used L2-norm feature distance is not good in semantic function, and then present a new FD method called function-consistent feature distillation (FCFD). Besides logits based KD loss at the heads and common FD loss at some intermediate teacher-student layer pairs, FCFD introduces two extra loss terms based on two kinds of lateral network parameter reuses: feeding dimension-aligned student features at a certain layer to the paired teacher layer, and feeding dimension-aligned teacher features at a certain layer to the paired student layer. The effectiveness of the proposed method is validated on both image classification (CIFAR100 and ImageNet) and object detection (MS-COCO) tasks.", "summary_of_the_review": "The novelty of this paper is very limited.\n\nPlease refer to my comments in 'Strength And Weaknesses' for details.", "strength_and_weaknesses": "Strengths.\n\n+ Improving feature matching is a critical research topic in FD research.\n\n+ The basic ideas of the proposed method are easy to understand.\n\n+ Experimental comparisons are performed on both image classification and object detection tasks with diverse teacher-student network pairs.\n\nWeaknesses.\n\n- The method.\n\nThe novelty of the proposed method, FCFD, is very limited. Note that the core designs of FCFD are two kinds of lateral network parameter reuses: feeding dimension-aligned student features at a certain layer to the paired teacher layer, and feeding dimension-aligned teacher features at a certain layer to the paired student layer. However, these two ideas have already been well explored in many existing knowledge distillation works such as cross distillation [1], residual distillation [2], explicit connection distillation [3],  softmax regression representation learning [4]. Unfortunately, these works are completely missed by the authors. Given the existence of these works, FCFD has no new technical contribution, to the best of my knowledge.\n\n[1] Few Shot Network Compression via Cross Distillation, AAAI 2020.\n\n[2] Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts, NeurIPS 2020.\n\n[3] Explicit Connection Distillation, ICLR 2020 submission.\n\n[4] Knowledge distillation via softmax regression representation learning, ICLR 2021.\n\n- The experiments.\n\nExperimental comparisons are not convincing enough.  \n\nFor experimental comparisons on image classification datasets (CIFAR100 and ImageNet), in Table 4, etc., it seems that the authors directly use the results reported in the papers of Review KD and DKD as the baselines. This is not fair, as for ablation, the comparison of FCFD+DKD/FCFD+SimKD vs. DKD/SimKD needs to run on the same training machines and code settings. \n\nFor experimental comparisons on object detection dataset MS-COCO, experimental settings are also not optimal: 1) for object detection task, mainstream feature distillation methods, such as [1-3], already reuse/share feature pyramid network (FPN, the neck) of the teacher backbone as the FPN for the student. In this context, is it necessary to apply the proposed FCFD? 2) For experimental comparisons, FCFD should be compared to mainstream feature distillation methods for object detection, but not FD methods for image classification as they usually perform much worse.\n\nIn the final formulation of FCFD, there are five loss terms, how to set proper weights to them? \n\nHow about the training cost of FCFD? As it will lead to heavy extra memory cost, it is necessary to compare training cost of FCFD with other FD methods. \n\n[1] Distilling Object Detectors with Feature Richness, NeurIPS 2021.\n\n[2] Instance-Conditional Knowledge Distillation for Object Detection, NeurIPS 2021.\n\n[3] Focal and global knowledge distillation for detectors, CVPR 2022.\n\n**----Update----**\n\nBalancing both positive and negative aspects of the rebuttal from the authors, although I raised the score from 3 to 5 (I would more like to give a score of 4 if there is such a choice), I still think this paper is not good enough to reach the acceptance bar of ICLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1pF46VFr_a", "reviewer": "Reviewer_pBzA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel feature distillation method. Instead of directly restricting features between teacher and student using explicit loss functions like L2, the proposed method, FCFD, utilizes lateral networks to define the feature loss,  and improves over existing feature distillation methods. ", "summary_of_the_review": "The idea of distilling using a part of the network seems novel to me. Extensive experiments have been conducted to verify the effectiveness of the proposed method in both image classification and object detection. Moreover, the authors illustrate that the proposed method can be effectively combined with other existing methods, showing a strong potential in application. In general, I recommend acceptance of this paper. ", "strength_and_weaknesses": "strength:\n+ A novel distillation method, FCFD, is proposed emphasizing the matching between teacher and student intermediate features not w.r.t pre-defined losses like L2 but w.r.t. function (aka the lateral network).\n+ Extensive experiments on image classification and object detection are conducted to investigate the effectiveness and generalization of FCFD. The results show that the proposed FCFD significantly outperforms state-of-the-art knowledge distillation methods.\n+ The proposed FCFD is compatible with many advanced knowledge distillation techniques with orthogonal contributions. Experiments verify its effectiveness when combined with other methods.\n+ I like the toy example given in Figure 1, it illustrate some limitation of the L2 loss. (I suggest also mentioning other losses like smooth L1).\n\nweakness:\n- Even though the random sampler is introduced in Section 2.3, I could not find how it is implemented in experiments. Especially how the \\delta values be sampled?\n- In Table 4, why SimKD result under WRN-40-2/WRN-16-2 is not given? The values in this row seem to be directly borrowed from the SimKD paper, but without a confidence interval. If the authors reproduced SimKD results, then why not filling all the cells?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster