PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
Authors: Weihan Cao
Yifan Zhang
Jianfei Gao
Anda Cheng
Ke Cheng
Jian Cheng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Knowledge Distillation
Object Detection

Abstract: Knowledge distillation(KD) is a widely-used technique to train compact models in object detection. However, there is still a lack of study on how to distill between heterogeneous detectors. In this paper, we empirically find that better FPN features from a heterogeneous teacher detector can help the student although their detection heads and label assignments are different. However, directly aligning the feature maps to distill detectors suffers from two problems. First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints on the student. Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise. To address the above issues, we propose to imitate features with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features. Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teacher pairs. Furthermore, it converges faster. With a powerful MaskRCNN-Swin detector as the teacher, ResNet-50 based RetinaNet and FCOS achieve 41.5% and 43.9% $mAP$ on COCO2017, which are 4.1% and 4.8% higher than the baseline, respectively.

PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
Weihan Cao
Yifan Zhang's
Jianfei Gao
Anda Cheng
Ke Cheng" 1.3
Jian Cheng!
NLPR & AIRIA, Institute of Automation, Chinese Academy of Sciences'
Shanghai Al Laboratory²
School of Artificial Intelligence. University of Chinese Academy of Sciences?
caoreihan20200ia.co ac. cn
fyfzhang, (ashen))@nlprisse ai. gaoj jenfjaine@@ppp jaab org. cn
{chenganda, chengke)201700a.cc ac. cn
Abstract
Knowledge distillation(KD)) a widely-used technique to train compact models object detection. However, there is still lack of study how to distill between heterogeneous detectors. this paper, we empirically find that better FPN fea- tures from heterogeneous teacher detector can help the student although their detection heads and label assignments are different. However, directly aligning the feature maps distill detectors suffers from two problems. First, the difference in feature magnitude between the teacher and the student could enforce overly strict constraints the student. Second, the FPN stages and channels with large feature magnitude from the teacher model could dominate the gradient of distillation loss, which will overwhelm the effects of other features in KD and introduce much noise. To address the above issues, we propose to imitate features with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features. Our method consistently outperforms the existing detection KD methods and works for both homogeneous and heterogeneous student-teachts pairs, Furthermore, it converges faster. With a powerful MaskRCNN-Swin detector the teacher, ResNet-50 based RetinaNet and FCOS achieve 41 41.5% and 43.9% m.AP on COCO2017, which are 4. % and 4.8% higher than the baseline, respectively. Our implementation is available https://gihubb.com/ppe mmlab/mmrazor..
Introduction
Knowledge distillation(KD)) a widely-used technique train compact models object detection. However, there is still a lack of study on how to distill between heterogeneous detectors, Most previous works [40, 35, 10, 20, 13] rely on detector-specific designs and can only be applied homogeneous detectors. [44, 50) conduct experiments on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted. Object detection is developing rapidly and algorithms with better performance are proposed continuously. Nevertheless, it not easy to change detectors frequently in terms of stability in practical applications. Furthermore, in some scenarios, only detectors with specific architecture can be deployed due to hardware limitations (e.g., two-stage detectors are hard deploy), while most powerful teachers belong to different categories. Thus, it is promising if knowledge distillation be conducted
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

P4
Talur
An example to illustrate the problems suffered by directly aligning the feature maps. The first and second lines correspond to the teacher and the student. Left: Feature magnitude of the teacher and the student. Middle: Dominant channels in FPN stage 'P3'. Let S1,u,0 RC denote the feature vector located in pixel (u. v) from /-th FPN stage and omit / for clarity. Then number, Eu. [arg max, su,e
Figure I:
(c) il where is the channel index. We define channels with
larger neumer dominant channels. Right: Visualization of the activation patterns from the teacher and the student. More visualization is provided in the Appendix.
between heterogeneous detector pairs. In addition, current distillation methods, such as [44, 47], usually introduce several complementary loss functions to further improve their performance, so several hyper- syper-parameters are used to adjust the contribution of each loss function which heavily affects their abilities to transfer to other datasets.
In this paper, we first empirically verify that FPN feature imitation can distill knowledge successfully even though the student-teacher detector pairs are heterogeneous. However, directly minimizing the Mean Square Error (MSE) between features of the teacher and the student leads to sub-optimal results, with results shown in Table 5. Similar conclusions are drawn in [6, 13, 50, 44). In order to explore the limitations of MSE, we elaborately visualize the FPN feature responses of the teacher and student detectors, as shown in Figure 1. Specifically, for an output feature St E R Cx XW from 1-th FPN stage, we select the maximum value in the dimension C each pixel and obtain 2-D matrix. Then we normalize the values to 0-255 according to the maximum and minimum values of / 2-D matrices. Through these comparisons, we obtain the following observations:
(1) The feature value magnitudes of the teacher and the student are different, especially for heterogeneous detectors, as shown in Figure 1 (left). So directly aligning the feature maps between the teacher and the student may enforce overly strict constraints and do harm to the student.
(2) The values of several FPN stages larger than the others, as shown in Figure 1 (right). obvious that FPN stage 'P6' of the teacher is less activated than FPN stage 'P3'. However, for detectors such as RetinaNet [24] and FCOS [37], all FPN stages share the same detection head. Hence, FPN stages with larger values could dominate the gradient of the distillation loss, which will overwhelm the effects of other features in KD and lead to sub-optimal results.
(3) The values of some channels are significantly larger than the others, as shown in Figure 1 (middle). However, it is mentioned in [35, 48] that the less activated features are still practical for distillation. they are not correctly balanced, the small gradients produced by these less activated channels can be drowned in the large gradients produced by the dominant ones, thus limiting further refinement. Furthermore, from the first column of Figure (right), we observe that there is much noise in the object-ireelevan area, because the values of the pixels in certain channels are significantly larger than those in other channels, thus being visualized in the figure. However, these pixels may be noise. Hence, directly imitating the feature maps may introduce much noise.
According to these observations, we propose Knowledge Distillation via Pearson Correlation Co- efficient (PKD) shown in Figure 2, which focuses on the linear correlation between features of the teacher and the student. To remove the negative influences of magnitude difference between the teacher-student detector pair and within the detector among different FPN stages and channels, we first normalize the feature maps to have zero mean and unit variances and minimize the MSE loss between the normalized features. Mathematically, it is equivalent to firstly calculating the Pearson

---

P6
vanilla
Backbone
FPN
Tectrrogencous Teacher
Lavel 0
Level
Level
Level
nomalized
Pearson Correlation Coefficient
noralized
Lavel o
Level
Level 2
Level
Student
Backbone
FPN
vanilla
Figure Overview of the proposed distillation via Pearson Correlation Coefficient (PKD). To demonstrate the effectiveness of PKD, we visualize the discord activation patterns of pre-normalized and post-normalized FPN features before distillation. The normalization mechanism bridges the gap between the activation patterns of the student and the teacher.
Correlation Coefficient (r) [2] between two original feature vectors, and then using 1 - r as the feature imitation loss.
Compared with the previous methods, our method offers the following advantages. First, as the distillation loss is calculated just FPN features, it can be easily applied to heterogeneous detector pairs, including models with heterogeneous backbones, heterogeneous detection heads and different training strategies such label assignment. Second, there is need to forward the teacher's detection head, the training time can be reduced significantly, especially for those models with cascaded heads. Besides, PKD converges faster than previous methods. Last but not least, it has only one syper-parameter- distillation loss weight and is not sensitive to it. So it can be easily applied to other datasets. We conduct extensive experiments verify the significant performance boosts brought by our approach on COCO dataset [25]. Using the same two-stage detector as the teacher, ResNet50 based RetinaNet [24] and FCOS [24] achieve 41.5% mAP and 43.9% mAP. which surpasses the baseline by 4.1% and 4.8% respectively, and it also outperforms the previous sate-of-the--eee methods by a large margin.
In summary, the contributions of this paper are threefold:
We argue that FPN feature imitation can distill knowledge successfully even though the student-teacher detector pairs are heterogeneous.
We propose to imitate FPN features with PCC to focus on relational information and relax the distribution constraints of the student feature's 's magnitude. It is capable distilling knowledge for both homogeneous and heterogeneous detector pairs.
We verify the effectiveness of our method on various detectors via extensive experiments on the COCO [25], and achieve otate-of-theeart performance without bells or whistles. Moreover, our method converges faster and is not sensitive to the only one hyper-parameter distillation loss weight, which simple yet effective.
Related Works
2.1 Object Detection
Object detection is considered as one of the most challenging vision tasks which aims at detecting semantic objects of a certain class images. Modern detectors are roughly divided into two-stage
detectors [32, 4, 14] and one-stage detectors [24, 37, 22, 12]. In two-stage detectors, Region

---

Proposal (ePN is usually adopted to generate initial rough predictions refined by a task- specific detection head, A typical example is Faster R-CNN [32]. In contrast, One-stage detectors, such as RetinaNet [24] and FCOS [37], can directly and densely predict bounding boxes on the output feature maps. Among these works, multi-scale features are usually adopted to handle objects of various scales, e.g. FPN [23], which is considered as typical case for our study. The proposed PKD only distills the intermediate features and does not rely on detector-specific designs so that can be used directly various detectors.
2.2 Knowledge Distillation
Knowledge Distillation (KD) is a kind of model compression and acceleration approach aiming at transferring knowledge from teacher model student model. It is popularized by [16] and then its effectiveness in image classification has been explored by subsequent works [33, 17, 15, 1, 46, 29, 27, 36, 38]. However, adapting KD to object detectors is nontrivial since minimizing the Beulbacck-eiblee (KL) divergence between the classification head outputs fails to transfer the spatial information from the teacher and only brings limited performance gain to the student. The following three strategies are usually adopted by previous methods in detection to handle the above challenge. First, the distillation is usually conducted among multi-scale intermediate features [21], which provides rich spatial information for detection. Second, different feature selection methods are proposed overcome the foreground- breground-backgrounte backgrounddd imbalance. Most of these works can be divided into three kinds according to the feature selection approach [19]: proposal based [6. 21, 10, 45], rule-based [40, 13] and attention- contains valuable information, many previous works try to improve the performance of detectors by enabling detectors to capture and make use of these relations, such as non-local modules [47], global distillation [44] and structured instance graph [8].
[47, 50, 20J. Third, as the relation between different objects
Unlike the previous works, we consider the magnitude difference, dominant FPN stages and channels as the key problem. We hope our method could serve as a solid baseline and help ease future research in knowledge distillation for object detectors.
3 Method
3.1
Preliminaries
In this part, we briefly recap the traditional knowledge distillation for object detection. Recently, fcature-based distillation multi-scale features is adopted to deal with rich spatial information for detection. Different imitation masks M are proposed to form an attention mechanism for foreground features and filter away noises in the background. The objective can be formulated as:
L
W H
LPPN Σ No
ΣΣΣ
Micis (Fles Oudapt (Fics))²
(I)
where L is the number of FPN layers, represents the 1-th FPN layer, i, j are the location of the corresponding feature map with width W and height H. .N. Σ Σ." are the (-th layer of feature of student and teacher detectors, respectively. Function Ondapt is a 1x1 convolution layer to upsample the number of channels for the student network the number of channels mismatches between the teacher and the student.
W Σ" Meay Ff and F7
The definition of M is different in these methods. For example, FRS [50] uses the aggregated classification score map from the corresponding FPN layer, and FGD [44] considers spatial attention, channel attention, object size and foreground-backgrrumm information simultancously.
3.2 Is FPN feature imitation applicable for heterogeneous detector pairs?
Most of the previous works perform distillation on FPN, as FPN integrates multiple layers of the backbone and provides rich spatial information of multi-scale objects. It is reasonable to force the student to imitate FPN features from its homogeneous teacher as they have the same detection head and label assignment, and better features could lead to better performance. However, there still a lack of study on how to distill between heterogeneous detectors. [44, 50] conduct experiments

---

on detectors with heterogeneous backbones, but detectors with heterogeneous detection heads and different label assignments are always omitted. Thus, we are motivated investigate whether FPN feature imitation still makes sense for these heterogeneous detector pairs.
We conduct backbone and neck replacement experiments on three popular detectors: GFL [22], FCOS [37] and RetinaNet [24]. First, we replace the backbone and neck of FCOS with the well-trained (by 12 epochs) ones of GFL. Since the main idea of feature-base based distillation methods is to directly align the feature activations of the teacher and the student, it can be considered as the extreme case of FPN feature imitation between FCOS and GFL. Then the FCOS head is finetuned with the frozen replaced GFL. backbone and neck. In Table 1, it can be seen that by replacing with the backbone and neck of GFL,, the detector achieves higher performance (from 36.5 to 37.6). It is verified in some extent that FPN feature imitation is applicable between heterogeneous detectors. contrast, we replace the backbone and neck of RetinaNet with the well-trained (by 12 epochs) ones of FCOS. Due to feature value magnitude difference between the two models caused by group normalization in FCOS head, a significant mAP drop (from 36.3 to 35.2) can be observed. This means the feature value magnitude difference could interfere with the knowledge distillation between two heterogenous detectors.
Table Results of the backbone and neck replacement on COCO.
Backbone & Neck
Head
mAP
FCOS-Res50 GFI,-ess50
FCOS FCOS
36.5
37.6
Retina-Res50 FCOS-Res5h
Retina 36.3
Retina 35.2
3.3 Feature Imitation with Pearson Correlation Coefficient
As discussed in Section 3.2, a promising feature distillation approach needs to consider the magnitude difference when constructing them into pairs for imitation. Moreover, by comparing the activation patterns shown in Figure I, we find the dominant FPN stages and channels can negatively interfere with the training phase of the student and lead to sub-optimal results, which ignored by previous works.
To address the above issues, we propose first normalizing the features of the teacher and the student to have zero mean and unit variances and minimizing the MSE between normalized features. Addition- ally, we want the normalization obey the convolutional property so that different elements of the same feature map, at different locations, are normalized in the same way. Let IB be the set of all values feature map across both the elements of mini-batch and spatial locations. So for a mini- batch of size and feature maps of size h x w, we use the effective mini-batcr of size m [B] = b hww. Let (()) I be the e-th channel of batch of FPN outputs and omit for clarity. Then we get the normalized values m and f1...m from the student and the teacher, respectively. Instead of delicately designing imitation masks M in Eq. choose the important features, our PKD operates on the full feature map. That is, the imitation mask M is filled with the scalar value 1. Hence we can formulate our distillation loss the following:
m
CFPN = 2m
[8
E
Moving forward, minimizing the loss function above is equivalent to maximizing the PCC between the pre-normalized features of the student and the teacher. PCC can be computed as:
Em Ms)(te 14)
r(s,t)
(3)
VE (st -M2)² VE Ei=i(t 41
Since 8 1), we get m-1 E m-1 Σ, 82 Then, reformulate Eq. 2 as:
1
LEPN 1
m
(2m- -2) - > 8
2m
2m 2
(1 - r(s, t)) = (01-r(stt) r(s,t).
2m
The Pearson correlation coefficient is essentially a normalized measurement of the covariance, such that the result always has value between -1 and 1. Hence, LEPN = 1 - always has value
5

---

between O and 2. It focuses on the linear correlation between the features of the teacher and the student, and relaxes constraints on the magnitude of the features. Actually, the feature maps 8, t R 0 implies that all data points lie on a line for which s increases as increases. Hence the student is well-trained. And vice versa for LePN A value of implies that there is no linear dependency between the features of the student and the teacher.
can be regarded as data points (&t, 44). TPPN =
During training, the gradient of PCC, 0L/0s,, with respect to each FPN output s;, is given by:
LFPN
1 (8: r(8, t) - (:),
0s:
mas
where r(s, t) is PCC in Eq. and o, is the sample standard deviation of the student's feature 8.
In conclusion, PCC focuses on the relational information from the teacher and relaxes the distribution constraints on the student feature's 's magnitude. Moreover, eliminates the negative impacts of the dominant FPN stages and channels, leading to better performance. As result, the normalization mechanism bridge the gap between the activation patterns of the student and the teacher (see Figure 2). Hence, feature imitation with PCC works for heterogeneous detector pairs. We train the student detector with the total loss as follows:
L LeT + aLrPN,
where Ler is the detection training loss, a is the hyper-parameter to balance the detection training loss and distillation loss.
3.4 Connection of PCC and KL. divergence
As discussed in Section 3.3, the normalization mechanism is the key addressing the three issues mentioned above. In previous works [16, 41, 34, 49), KI. divergence has been widely used in distillation. They first convert activations into probability distribution with softmax function and then minimize the asymmetry KI. divergence of the normalized activation maps:
((ti)
LKI = T
Σ ((4)) logl
(s,))
where T a hyper- to control the degree of softness of the targets, and ø is the softmax function:
exp(t:/TT
(1)) ΣT= exp(t,/T)
(8)
Here, we show that minimizing KI. divergence between normalized features in the high-temperature limit is equivalent to minimizing MSE between normalized ones, and hence equivalent maximizing PCC between original ones.
Let p, = off)) and = (($)) denote the probabilities from the teacher and the student, respectively. The KL divergence gradient, ÖLkL/0š, with respect to each normalized activation s; of the student given by:
i LKL = (g: pp)) = 0s,
ea/TT
(9)
E/ei
If T is large compared with the magnitude of the normalized activations, approximate:
0LKL T
S//T m + E/T
1+4/T
(10)
08;
8/
+ Σ,
As &; and & have been zero-meaned, Eq. 10 simplifies to:
dLKL 05; m
(s+ -
(11)
Employing the relation between MSE and PCC shown in Eq. gives the desired result. Experimental results are provided in Section A.3.

---

Table Results the proposed method with different detection frameworks on the COCO dataset.
T and S mean the teacher and student detector, respectively. indicates the results reproduced by us.
Method
schedule
mAP
50
APrs
APs
APM
APt.
Retina-ResX100 (T)
2x
40.8
60.5
43.7
22.9
44.5
54.6
Retina-Res50 (S)
2x
37.4
56.7
39.6
20.0
40.7
49.7
FKD [47]
2x
39.6 (+2.2)
58.8
42.1
22.7
43.3
52.5
FRS [50]
2x
40.1 (+2.7)
59.5
42.5
21.9
43.7
54.3
FGD [44]
2x
40.4 (+3.0)
59.9
43.3
23.4
44.7
54.1
PKD (Ours)
2x
40.8 (+3.4)
60.3
43.4
23.0
45.1
54.7
FasterRCNN-Rest (T)
2x
39.8
60.1
43.3
22.5
43.6
52.8
FasterRCNN-Ren 50 (S)
2x
38.4
59.0
42.0
21.5
42.1
50.3
GID [10]
2x
40.2 (+1.8)
60.7
43.8
22.7
44.0
53.2
FRS [50]
2x
40..
60.8
44.0
23.2
44.4
53.1
FGD [44]
2x
40.4 (+2.0)
60.7
44.3
22.8
44.5
53.5
PKD (Ours)
2x
40.5 (+2.1)
60.9
44.4
22.6
44.8
53.1
FCOS-Res 101 (T)
2x+ms
41.2
60.4
44.2
24.7
45.3
52.7
FCOS-Res50 (S)
2x+ms
39.1
58.4
41.6
24.0
42.7
48.7
FRS [50] *
2x+ms
42.2 (+3.1)
60.6
45.6
27.1
46.5
53.0
FGD [44]
2x+ms
42.3 (+3.2)
60.8
45.8
26.1
46.7
53.3
PKD (Ours)
2x+ms
42.8 (+3.7)
61.4
46.2
25.9
47.2
54.6
RepPoints ResNeXt101 (T)
2x
44.2
65.5
47.8
26.2
48.4
58.5
RepPoints Res50 (S)
2x
38.6
59.6
41.6
22.5
42.2
50.4
FGD [44]
2x
41.3 (+2.7)
24.5
45.2
54.0
PKD (Ours)
2x
42.3 (+3.7)
63.1
45.4
23.9
46.6
56.5
TOOD-ResXI01 (T)
2x+ms
47.6
68.5
51.6
30.6
51.4
59.7
TOOD-Res50 (S)
1x
42.4
59.7
46.2
25.4
45.5
55.7
PKD (Ours)
45.5 (+3.1)
62.8
49.3
27.4
49.8
58.4
4
Experiments
order to verify the effectiveness and robustness of our method, we conduct experiments on different detection frameworks on the COCO [25] dataset. We choose the default 120k images split for training and 5k images split for the test. use the standard training settings following [44] and report mean Average Precision (AP)
evaluation metrie, together with AP under different thresholds and scales, i.e., APso. AP7s. APs. APM and APL-
For distillation, the hyper- parameear is set to 6 when using two-stage detector as the teacher and 10 when using a one-stage one. Feature levels of heterogeneous detector pairs may not be strictly aligned, e.g., FastenRCNN constructs the feature pyramid from P2 to P6, while RetinaNet uses P3 to P7. To address the above issue, we upsample the low-resolution feature maps to have the same spatial size as the high-resolution ones. In addition, the adaptive layer (dndapt in Eq. 1) is unnecessary for our PKD.
All experiments are performed on 8 Tesla A100 GPUs with two images in each. Our implementation based on mmdetection [7] and mmrazor [9] with Pytorch framework [30]. "Ix' (namely 12epochs), '2x' (namely 24 epochs) and '2x+mss (namely 24epochs with multi-scale training) training schedules used. More details are given the Appendix.
4.1 Main Results
Our method can be applied to different detection frameworks easily, so we first conduct experiments on five popular detectors, including a two-stage detector (Faster RCNN[32]), two anchor-based one-stage detector (RetinaNet [24], RepPoints [43] and TOOD [12]) and an anchor-free detector (FCOS [37]. Table shows the comparison the results of state-of-the-arr distillation methods on the COCO. Our distillation method surpasses other state-of-the- art methods. All the student detectors

---

gain significant improvements in AP with the knowledge transferred from teacher detectors, e.g.. FCOS with ResNet-50 gets 3.7% mAP improvement the COCO dataset. These results indicate the effectiveness and generality of our method in both one-stage and two-stage detectors.
4.2 Distilling More Student Detectors with Stronger Heterogeneous Teachers
Most of the current methods are designed for homogeneous detector pairs. PKD is general enough to distill knowledge between both homogeneous and heterogeneous detector pairs. Here we conduct experiments on more detectors and use stronger heterogeneous teacher detectors, as shown in Table 3. Comparing with Table 2, we find that student detectors perform better with stronger teacher detectors, egg. Retina-Res50 model achieves 41.5% and 39.6% mP with Mask RCNN-Swin [28] Retina- Res101, respectively. Results show that mimicking the feature maps of stronger heterogeneous teacher detectors can further boost the student's performance when applicd with our PKD.
Table 3: Results of more detectors with stronger heterogeneous teacher detectors on the COCO dataset.
Method
schedule
mAP
AP3o
AP15
APs
APM
APr.
Mask RCNN-Swin (T)
3x+ms
48.2
69.8
52.8
32.1
51.8
62.7
Retina-Reshh (S)
2x
37.4
56.7
39.6
20.0
40.7
49.7
PKD (Ours)
2x
41.5 (+4.1)
60.6
44.1
22.9
45.2
56.4
Mask RCNN-Swin (T)
3x+ms
48.2
69.8
52.8
32.1
51.8
62.7
FCOS-ResST (S)
2x+ms
39.1
58.4
41.6
24.0
42.7
48.7
PKD (Ours)
2x+ms
43.9 (+4.8)
62.3
47.5
27.2
48.0
57.1
GFL-Res 101 (T)
2x+ms
44.9
63.1
49.0
28.0
49.1
57.2
FCOS-Res50 (S)
2x+ms
39.1
58.4
41.6
24.0
42.7
48.7
PKD (Ours)
2x+ms
43.5 (+4.4)
61.9
47.1
26.5
47.7
55.7
Table 4: Results of other feature-based distillation methods with the normalized feature maps.
GFL101
(44.7)
FitNet GT Mask
FRS feature imitation
FGD Global
PKD
GFL550 (40.2)
41.3
43.3
41.4
43.0
43,2
43.4
40.2
43.3
4.3 Other feature-based distillation methods with the normalized features
In object detection, some feature-bases distillation methods [50, 44] transfer the knowledge within some pre-defined imitation regions, and achieved competitive results. In contrast, believe full map feature imitation with our PKD can already outperform them. In order to show the generality of our method, take FitNet [33], GT Mask, FRS [50] and FGD [44] as baselines and build PKD them.
Recaps. Full feature map distillation with MSE loss is used in FitNet. While the GT Mask only imitates features that overlap with ground truth bounding boxes. Feature imitation with FRS uses the aggregated classification score, takes maximum operation in channel direction, the classification head output weighted score mask, guide the distillation of the FPN. And global distillation in FGD rebuilds the relation between different pixels and transfers it from teachers to students through GcBlock [5].

---

Results. For a fair comparison, all the hyper-paramede parameters are copied from the original paper. Results in Table show that combining PKD with these feature imitation methods can further improve their performance.
Table 5; Comparisons between feature imitation with MSE and our PKD. We address the above three issues and achieve better performance.
25.0
COCO mAP%
Detector Pairs
Methods mAP APs APL
55.0
FCOS- 1es
MSE
36.3
20.0
47.1
Retina-Res50
Ours
41.3
24.2
55.4
27.
GFL-Res101 FCOS-Res50
MSE
43.0
27.1
54.3
Ours
43.5
26.5
55.7
76.1
93.6
56.2
16.2
96.7
17.7
96.1
10.0
01.8
11.0
61.9
422
31.
58.0
57.9
10.1
41.8
11.9
421
425
Retina-ResX 101
MSE
40.4
22.2
54.4
Retina-Res50
Ours
40.8
23.0
54.7
epech
Figure 3: Comparison the convergence speed of FCOS-Res5K among other state-of-the-ar distilla- tion methods and ours.
4.4 Analysis
4.4.1 Effectiveness of Pearson Correlation Coefficient
Among most previous works, feature-based distillation over multi-scale features is adopted for distillation. In this study, we argue that the magnitude difference, dominant FPN stages and channels negatively interfere with the training phase of the student and lead to sub-optimal results. To study this empirically, we conduct the following three experiments to explore the effects of MSE loss which is widely-used loss function in distillation and suffers from the above three issues simultaneously. For a fair comparison, we tune the loss weight MSE for all experiments. More details are listed in Section A.4.
For the first pair, different from that of the student Retina-Res50 (refer to Figure 4). In this case, what constitutes the knowledge is better presented by relational information from the teacher's 's features than the absolute values. For the second pair, GFL-Res101 is the teacher and FCOS-Ress teacher and student, features in FPN stages 'P5' and 'P6' are less activated than those stages 'P3' and 'P4' (refer to Figure 5). Hence, features in stages 'P3' and 'P4' could dominate the gradient of the distillation loss, which will overwhelm the effects of other features. Since larger objects are usually assigned to higher feature levels, the student's ' performance for large objects is significantly lower than that of ours when MSE is used as the distillation loss. For the last pair, Retina-ResX 101 is the teacher and Res5n is the student. There are always few channels with greater values (refer to Figure and Figure 7). so directly imitating the feature maps may introduce much noise in dominant channels.
is used as the teacher whose feature magnitude is significantly
is the student. For both
By comparing the results in Table 5, we find that our proposed PKD addresses the above three issues and achieves better performance. Hence, effective distillation loss function should have the ability to handle the above three problems. We hope PKD could serve as a solid baseline and help ease future research in knowledge distillation community.
4.4.2 Convergence Speed
In this subsection, we conduct experiments with FCOS to compare the convergence speed of our method with other state-of-the-art distillation methods on the COCO benchmark. As shown in Figure 3, the training convergence can significantly speed up in the early training stage with our PKD. Meanwhile, the final performance of the student detector FCOS-Res50 is about 0.5%-0.6% mAP higher than that of FGD and FRS. In addition, there is no need to forward the detection head of the

---

teacher, which reduces the training time significantly, especially for those models with cascaded heads.
4.4.3 Sensitivity study of loss weight a
In Eq. 6, we use the loss weight hyper-parameter a to bal- ance the detection training loss and distillation loss. Here, conduct several experiments to investigate the influence of a. As shown in Table 6, the worst result is just 0.3 mAP drop compared with the best result, indicating our method is not sen- sitive to the only hyper-parameter a. In addition, thanks the normalization mechanism in PCC, relatively uniform value of loss weight a can be found among different teacher-student detector pairs to keep the balance of the detection loss and the distillation loss unchanged.
Table 6: Ablation study of loss weight hyper-parameter on FCOS ResX101 RetinaNet Res50.
3
5
8
10
13
m.P 41.0 41.1 41.1
41.3 41.1
Conclusion
This paper empirically finds that FPN feature imitation is applicable for heterogeneous detector pairs although their detection heads and label assignments are different. Then we propose feature imitation with Pearson Correlation Coefficient focus on the relational information from the teacher and relax the distribution constraints of the student's feature value magnitude. Furthermore, a general KD framework is proposed, capable of distilling knowledge for both homogeneous and heterogeneous detector pairs. It converges faster and only introduces one applied to other datasets. However, our understanding of whether our proposed PKD is capable of other tasks such as text recognition is preliminary and left as future works.
which can easily be
Acknowledgments
This work is supported in part by the Shanghai Committee of Science and Technology, China (Grant No. 20DZ1 100800), NSFC 62273347, the National Key Research and Development Program of China (2020AAA0103402), Jiangsu Key Research and Development Plan (No. (.0011220211)22 and NSFC 61876182.
References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9163-9171, 2019.
[2] Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen, Pearson correlation cocfficient. In Noise reduction in speech processing, pages 1-4. Springer, 2009,
[3] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: Delving into high quality object detection. In Proceedingy of the IEEE conference computer vision and pattern recognition, pages 6154-6162, 2018.
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnm: high quality object detection and instance segmenta- tion. IEEE transactions on pattern analysis and machine intelligence, 43(5))1483-144 2019.
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze- excitation networks and beyond. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0-0, 2019.
[6] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. leamiing efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30. 2017.
[7] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: moxiv:/9066 /otp:/90000.555 2019.
[8] Yixin Chen, Pengguang Chen. Shu Liu, Liwei Wang. and Jiaya Jia. Deep structured instance graph for distilling object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4359 4368, 2021.
10

---

[9] MMRazor Contributors. Openmmlab model compression toolbox and benchmark. https://github. com/ open-mmlab/mmrazeee 2021.
[10] Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang. Si Liu, and Erjin Zhou. General instance distillation for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7842-7851, 2021.
[11] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi. Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6569-66 6578, 2019.
[12] Chengjian Feng, Yujie 7hong, Yu Gao, Matthew R Scott, and Weilin Huang. Tood: Task-aligned one- stage object detection. In 2021 IEFF/CVF International Conference on Computer Vision (ICCV), pages 3490-3499. IEEE Computer Society, 2021.
[13] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors decoupled features. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21 2154-2164, 2021.
[14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[15] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3779-3787, 2019.
[16] Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint www.i::/0..0053350000 2015.
2015.
[17] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv: (70::17770000 012.0,11199.. 2017.
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[19] Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. Instance-conditional knowledge distillation for object detection. Advances in Neural Information Processing Systems, 34, 2021.
[20] Gang Li, Xiang Li, Yujie Wang. Shanshan Zhang, Yichao Wu, and Ding Liang. Knowledge distilla- tion for object detection via rank mimicking and prediction predictionguideed guided feature imitation. arXiv preprint arXiv:21 04840, 2021.
[21] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference computer vision and pattern recognition, pages 6356-6364, 2017.
[22] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li. Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems, 33:21002 21012, 2020.
[23] Tsung-Y Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21 17-2125, 2017.
[24] Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEFE international conference on computer vision, pages 2980-2988, 2017.
[25] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-7 755. Springer, 2014.
[26] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yann Fu, and Alexander Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.
[27] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7096-7104, 2019.

---

[28] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 1000: 2021.
[29] Wonpyo Park, Dongju Kim. Yan Lu, and Minsu Cho. Relational knowledge distillation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[30] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Leter. Automatic differentiation in pytorch. 2017.
[31] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: arXiv: 1804.02767 2018.
[32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[33] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: (eXiv:/412.66 1412.6550, 2014.
[34] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 531 1-5320, 2021.
[35] Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Distilling object detectors with task adaptive regularization, arXiv preprint 2tdi::2006.11000 2020.
[36] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 1910. 10699, 2019.
[37] 7hi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9627-9636, 2019.
[38] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1365-1374, 2019.
[39] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arviv: 6607.0802, 2016.
[40] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with finc- fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4943-4942, 2019.
[41] Yukang Wang. Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference on Computer Vision, pages 346-362. Springer. 2020.
[42] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pages 19, 2018
[43] 7e Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. Proceedings the 9657-9666, 2019.
the IEEE/CVE International Conference on Computer Vision, pages
[44] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao and Chun Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv:2111 2021.
[45] Lewei Yao, Renjie Pi Hang Xu, Wei Zhang. Zhenguo Li, and Tong Zhang. G-detkd: Towards general distillation framework for object detectors via contrastive and semantic-guided feature imitation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3591-3600, 2021.
[46] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133-4141, 2017.
[47] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations, 2020.
[48] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. arXiv preprint arXiv 086799 2022.
12

---

[49] Zhaohui Zheng, Rongguang Ye, Ping Wang, Dongwei Ren, Wangmeng Zuo, Qibin Hou, and Ming-Ming Cheng. Localization distillation for dense object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9407-9416, 2022.
[50] Du Zhixing, Rui Zhang. Ming Chang, Shaoli Liu, Tiansii Chen, Yunji Chen, al. Distilling object detectors with feature richness, Advances in Neural Information Processing Systems, 34. 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 3.3 and Section 3.4
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A] To the best of our knowledge, there is no potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
3.3 and Section 3.4
(b) Did you include complete proofs all theoretical results? [Yes] See Section 3.3 and Section 3.4
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL)? [Yes] Codes and instructions are provided in the Appendix. All datasets used are public datasets.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section More details are given the Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] We follow the common practice used in baseline methods, where no error bars are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details are given in the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL.? [No] using/curating? [Yes] All datasets used public datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "aaiE6x90aY_", "reviewer": "Reviewer_aChz", "summary": "In this submission, the authors begin with a clear motivation that the feature value magnitude of the teacher and student is different in pixels, channels and stages for different detector pipelines with different backbones. Motivated by this, the authors propose the Pearson correlation coefficient based approach to tackle the magnitude difference of the distilled features between both homogeneous and heterogeneous detectors. Experiments and ablation studies on the COCO dataset evidence the effectiveness of the proposed method with part of the experiment settings, however, there are some results that should be clarified. Please refer to the weaknesses part.", "questions": "I am open to changing my mind if the authors addressed my concerns above.", "limitations": "No. They claim there is no potential negative social impact of this work to their knowledge.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The submission is well-written and easy to follow.\n+ The motivation is clearly explained and evidenced by comprehensive experiments with promising results under some teacher-student pairs settings.\n\n**Weaknesses**\n\nHowever, there are still some concerns to be addressed:\n\n1. The normalization process proposed in lines 166-175 is somehow vague. Please clarify it in more details, like $\\hat{s} = ?$.\n\n2. To my understanding, the proposed PCC is able to normalize the magnitude of the feature map so that it can compute the loss on the pre-normalized feature maps. Here comes a question, how will other feature-based distillation methods work with the normalized feature maps? \n\n3. As for the 'MSE' in Table 4, could you please clarify whether the MSE works on the pre-normalized feature or the post-normalized feature?\n\n4. Table 2 shows the results of different distillations in the homogeneous setting. The SOTA result is the proposed PKD with inheriting strategy. But, the inheriting strategy which initializes the student with the teacher’s neck and head parameters and gets better results can be applied to all other methods as well in Table 2. To this end, it is unfair to claim PKD is the SOTA.\n\n5. Table 3 also misses some other feature-based distillation methods between heterogeneous detectors, like FKD[37] to further evidence the effectiveness of the PKD. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zgb_p6rheF8", "reviewer": "Reviewer_96XY", "summary": "The paper proposes a new KD method for object detection model. They distill the features of FPN by first normalize the output of the feature and then minimize the MSE loss which is equivalant to maximizing the pearson correlation coefficient. The experimental results demonstrate the effectiveness of the proposed method.", "questions": "- Is teacher the first line in Fig.1? Besides, the font size is too small in Fig.1.\n\n- Lack of ablation studies of using KL divergence with different T.\n\n- Other questions see cons above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- The paper is well written and easy to understand.\n\n- The experimental results well demonstrate the effectiveness of the proposed method.\n\nCons:\n- It seems that conducting MSE loss between features after normalizing the feature is not novel. Many KD papers for image classification add conv-bn layers after the output of the feature map before applying MSE loss, which seems quite similar to the proposed method.\n\n- I am confused about Sec.3.4, which shows that the proposed method equals to using KL divergence with a quite large T. However, when T is large, the KL divergence flatten the two feature maps, and makes the original distribution less important. If this is so, why is the difference of the FPN feature maps between student and teacher still matter? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FF39nVLh2iS", "reviewer": "Reviewer_JKs3", "summary": "Feature imitation is one of the widely used knowledge distillation techniques, where MSE loss is widely used to enforce the consistency of feature representation between the teacher-student pair. This paper proposes a new loss function for feature imitation in object detection. It first conducts feature standardization and then calculates the MSE loss. Methematically, it is equivalent to firstly calculate the Pearson Correlation Coefficient $r$ between two feature vectors, and then use 1-$r$ as the new feature imitation loss. Expreiments show that it can be better than traditional MSE loss in distilling both heterogeneous detector pairs and homogeneous detector pairs.", "questions": "see weakness", "limitations": "I like the novelty, the effectiveness, and the simplicity of this paper. However, my pre-rebuttal rating score is 4 because I have some major concerns about the arguments and experiments.\n\nThe most important concerns are Weakness 1, 2, 3. If they are sovled correctly, I will raise my rating score. If there still exists one of the Weakness 1, 2, 3, I will not change the rating score. I'm happy to discuss with the authors and other reviewers and exchange the ideas about this paper.\n\nThe story needs to be reorganized. The paper needs a major revision.\n\nThe best modifications are:\n\n1. See Weakness 1, remove the fine-tuning Table 1. Combining Fig. 1 and Table B in the Appendix \"FCOS-ResX101 $\\rightarrow$ Retina-Res50 1x 33.9 (-2.6)\" \"GFL-Res101 $\\rightarrow$ FCOS-Res50 1x 39.2 (+2.6)\". Then you make the argument that the feature misalignment, or something like the feature magnitude difference, severely affects the performance of feature imitation. According to these observations, we propose ... We find that feature standardization is the key to effectively mitigate ... As shown in Fig 2, the feature standardization can ...\n\n2. See Weakness 2, 5 and 7.\n\n3. See Weakness 3.\n\nThe authors should also describe the property of the proposed loss function, about its boundness, optimization goal (linear correlation of two features), more relaxed than MSE loss, and the limit cases (if $\\mathcal{L}_{FPN}\\rightarrow0,1,2$, what are the meanings of these situations.)\n\nAfter making all the above modification, it can meet the acceptance bar.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. I highly like the contribution of this paper, because I've also observed before that this idea, the correlation imitation loss (replace MSE with 1-PCC) did work. And it did speed up the training process, in other words, it did reduce the training difficulty in the early training stage. However, from my observation, the performance improvement is limited. The AP can be further improved by only +0.2 based on LD [r2] +KD+GI imitation [9] (42.4$\\rightarrow$42.6, GFLv1 Res101 2x $\\rightarrow$ GFLv1 Res50 1x). I just simply replaced the MSE loss of GI imitation with correlation imitation loss (1-PCC), and searched a best loss weight of course.\n\n2. This paper also incorporate their method to distill heterogeneous detector and demonstrate that PKD works better than traditional MSE loss. They find that feature standardization is the key to conduct feature imitation for distilling the heterogeneous detectors. This is really a good idea.\n\n3. Due to its novelty, effectiveness and simplicity, it has a potential to serve as a strong baseline for future works on knowledge distillation for object detection.\n\nWeaknesses:\n\n1. In Sec. 3.2, **the fine-tuning experiments cannot support your argument** “Hence, FPN feature imitation is applicable for heterogeneous detectors after removing the negative impacts of magnitude difference.” (line 158-159). If you want to make that argument, you must list the results of Table 4 and need further improvement. For example, FCOS-ResX101 $\\rightarrow$ Retina-Res50 MSE is worse than baseline (use Table B in the Appendix). GFL-Res101 $\\rightarrow$ FCOS-Res50 MSE and Retina-ResX101 $\\rightarrow$ Retina-Res50 MSE works well. This phenomenon combined with the feature magnitude difference, **not fine-tuning**, shows the importance of feature standardization which aligns the features between the heterogeneous detectors so as to make the feature imitation more efficient. As for the homogeneous detectors, the feature magnitude difference is relatively small. And your PKD can still produce a slight improvement. (This is very important to the whole story and it affects my rating score by at least 1 point.)\n\n2. From Table 4, it can be seen that PKD can only improve the performance by +0.4~0.5 AP over MSE loss and even no improvement (RetinaNet Res101 $\\rightarrow$ Res50) if homogeneous teachers are used. In Table 2, you compare PKD with various KD methods such as FRS and FGD. It is confusing to me that Retina-ResX101 $\\rightarrow$ Retina-Res50 FGD gets 40.4 AP and your PKD gets 40.8 AP, while GFL-Res101 $\\rightarrow$ GFL-Res50 FGD gets 41.3 AP and PKD gets 43.3 AP. **Why is there such a big difference?** I notice that you implement your method by setting the imitation mask to 1, which means that your PKD is based on FitNet, i.e., full map feature imitation, right? If it is true, your results in Table 2 is weird to me, e.g., GFL-Res101 $\\rightarrow$ GFL-Res50 1x PKD is 2 point higher than FGD, but 0.3 lower than FRS. It seems that PKD+FitNet can already outperform these feature imitation methods, like FRS and FGD. However, I believe that PKD can be combined with these feature imitation methods since only the MSE loss needs to be replaced. And this is exactly what you show in Table 4. Therefore, you must check the context in Sec. 3.3 and the experiments, and **express the implementation details correctly**. You must emphasize which feature imitation method the PKD results in Table 2 are based on.\n\n3. In 4.3.3 Convergence Speed, the comparison of Table 5 is **unfair**. You choose FCOS-ResX101 as the teacher, while all the others are Retina-Res101. It must be the same Retina-Res101 to compare your PKD with the others. Notice that, in Table 6, FCOS ResX101 $\\rightarrow$ RetinaNet Res50 2x gets 41.3 AP, i.e., the 2x training schedule can get higher AP than 1x, which is about +1.0. And in Table 4, Retina-ResX101 $\\rightarrow$ Retina-Res50 PKD 2x gets 40.8 AP. Therefore, the performance of Retina-Res101 $\\rightarrow$ Retina-Res50 PKD 1x seems to be 39.3 around. BTW, from my observation, the training convergence can be significantly speed up in the early training stage. Here are my previous results: GFLv1 Res101 2x $\\rightarrow$ GFLv1 Res50 1x.\n\n|Method / epoch | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |\n|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\nLD+KD+GI imitation (MSE) | 20.7 | 27.1 | 31.4 | 32.9 | 35.2 | 36.3 | 36.7 | 37.7 | 41.8 | 42.1 | 42.3 | 42.4 |\nLD+KD+GI imitation (CI loss) | 25.5 | 30.4 | 33.8 | 35.3 | 36.7 | 37.1 | 37.8 | 37.8 | 42.2 | 42.3 | 42.4 | 42.6 |\n\nTherefore, if you want to show the convergence speed, you'd better draw some AP curves to exhibit.\n\n4. From Sec. 3.4 \"Connection of PCC and KL divergence\", if the temperature $T$ is high, minimizing KL divergence between post-normalized features is equivalent to minimizing MSE between post-normalized ones,  and equivalent to PKD. However, the experiment results in the Appendix only show the performance of $T=50$. What would be the performance if lower temperatures were used?\n\n5. In object detection, some feature imitation methods transfer the knowledge within some pre-defined imitation regions, and be validated to be better than distilling on the full feature map. While your method operates on full map. How does PKD compare to these approaches, e.g., Fine-Grained feature imitation [33], GI imitation [9]? More specically, conducting PKD within some pre-defined imitation regions, **not just comparing to the results from the original papers**. I know it will work, but just want to see these relevant experiments in the paper, cuz it's about the universality of this method.\n\n6. Part of the experiments are redundant. You spend lots of space to show the effectiveness of PKD on various detectors, such as RetinaNet with different backbones, FCOS, GFL and TOOD. These one-stage detectors share almost the same framework. It would be better if we could choose the results and leave the space for the more important experiments, see above. BTW, Sec. 4.3.2 \" Sensitivity study of loss weight $\\alpha$\" is not important and can be moved to the Appendix.\n\n7. The inheriting strategy of [16] is unnecessary at all. It is irrelevant to your contribution. What readers really want to see is to what extent your PKD helps improve the distillation performance, not anything else. This problem is mentioned in Weakness 2 either. As I know, some previous works (like FRS) use both feature imitation and logit mimicking (classification KD). One solution is to specify which method your PKD is based on. Since PKD is made for replacing MSE loss, it is not difficult to implement that. For example:\n\n|Method | schedule | AP | AP50 | AP75 | APs | APm | APl |\n|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\nGFL-Res101 (T) | 1x | 44.9 | \nGFL-Res50 (S) | 1x | 40.2 | \nFitNet | 1x | xxx\nGI imitation | 1x | xxx |\nFGFI | 1x | xxx |\nFRS | 1x | 43.6 |\nFGD | 1x | 41.3 |\n||\nFitNet + PKD | 1x | xxx\nGI imitation + PKD | 1x | xxx |\nFGFI + PKD | 1x | xxx |\nFRS + PKD | 1x | xxx |\nFGD + PKD | 1x | xxx |\n\nAnd the same goes for the heterogeneous detector pair (choose two typical pairs are enough. One is a detector pair that MSE fails. The other is MSE works). The above two tables are Sec. 4.1 \"Main Results\", the rest detector pairs can be reported in Sec. 4.2, namely \"More Teacher-Student Pairs\", and picked the best results of your PKD. (among FitNet+PKD, GI imitation+PKD, FGFI+PKD, FRS+PKD, FGD+PKD ect.)\n\n8. Some important relavant works are missing.\n\n[r1] Chen Y, Chen P, Liu S, et al. Deep structured instance graph for distilling object detectors[C] Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4359-4368.\n\n[r2] Zheng Z, Ye R, Hou Q, et al. Localization distillation for object detection[J]. arXiv preprint arXiv:2204.05957, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept