Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors
Authors: Linfeng Zhang
Kaisheng Ma
Venue: ICLR 2021 Poster (2021)
Keywords: Knowledge Distillation
Object Detection
Teacher-Student Learning
Non-Local Modules
Attention Modules

Abstract: Knowledge distillation, in which a student model is trained to mimic a teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, designed for image classification, have failed on more challenging tasks, such as object detection. In this paper, we suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and background and (2) lack of distillation on the relation between different pixels. Observing the above reasons, we propose attention-guided distillation and non-local distillation to address the two problems, respectively.  Attention-guided distillation is proposed to find the crucial pixels of foreground objects with attention mechanism and then make the students take more effort to learn their features. Non-local distillation is proposed to enable students to learn not only the feature of an individual pixel but also the relation between different pixels captured by non-local modules. Experiments show that our methods achieve excellent AP improvements on both one-stage and two-stage, both anchor-based and anchor-free detectors. For example, Faster RCNN (ResNet101 backbone) with our distillation achieves 43.9 AP on COCO2017, which is 4.1 higher than the baseline. Codes have been released on Github.

Published as a conference paper at ICLR 2021
IMPROVE OBJECT DETECTION WITH FEATURE-BASED KNOWLEDGE DISTILLATION: TOWARDS ACCURATE AND EFFICIENT DETECTORS
Linfeng Zhang
Kaisheng Ma
Institute for Interdisciplinary Information Sciences, Tsinghua University
Institute for Interdisciplinary Information Sciences, Tsinghua University
ABSTRACT
Knowledge distillation, in which student model is trained to mimic teacher model, has been proved as an effective technique for model compression and model accuracy boosting. However, most knowledge distillation methods, de- signed for image classification, have failed on more challenging tasks, such as ob- ject detection. In this paper, suggest that the failure of knowledge distillation on object detection is mainly caused by two reasons: (1) the imbalance between pixels of foreground and background and (2) lack of distillation on the relation be- tween different pixels. Observing the above reasons, we propose attention-guiie distillation and non-local distillation to address the two problems, respectively. Attention-guider distillation is proposed to find the crucial pixels of foreground objects with attention mechanism and then make the students take more effort to learn their features. Non-local distillation is proposed to enable students to learn not only the feature of an individual pixel but also the relation between dif- ferent pixels captured by non-local modules. Experiments show that our meth- ods achieve excellent AP improvements on both one-stage and two-stage, both anchor-based and anchor-free detectors. For example, Faster RCNN (ResNet101 backbone) with our distillation achieves 43.9 AP on COCO2017, which is 4.1 higher than the bascline. Codes have been released on Github
INTRODUCTION
Recently, excellent breakthrough in various domains has been achieved with the success of deep learning (Ron- neberger et al. 2015); Devlin et al.) 2018: Ren et al.]
Distilled Model
2015). However, the most advanced deep neural net- works always consume a large amount of computa- tion and memory, which has limited their deployment in edge devices such as self-driving cars and mobile phones. To address this problem, abundant techniques are proposed, including pruning (Han et al.; 2016
Daseline Model
(MS COCO2017)
42.5
41.5
40.5
30.5
Zhang et al., 2018: Liu et al., 2018: Frankle & Carbin 2018), quantization (Nagel et al. [019 Zhou al. 2017). compact model design (Sandler et al., 2018 Howard et al. 2019 Ma et al. 2018; Landola al. 2016) and knowledge distillation (Hinton et al.) 2014 Buciluà et al. 2006). Knowledge distillation, which is also known as teacher-student learning, aims to trans-
38.5
Zhou
37.5
36,5
Castado Mlute pessiii)
Casl REWwio
Drsumit wore Cold PC
Coccall wereee ('ucte
tdak RCNISU Dysaenic SCr Rechiets
aual Kestash cenernatt Eoniaaiest
staal
Figure 1: Results overview.
fer the knowledge of an wee-perraeteri teacher lightweight student. Since the student is trained to mimic the logits or features of the teacher, the student can inherit the dark knowledge from the teacher, and thus often achieves much higher accu- racy. Due to its simplicity and effectiveness, knowledge distillation has become a popular technique for both model compression and model accuracy boosting.
"Corresponding author Archiplrrhchllaae
I http:///wtttttttmoo//./////////e http://hhs../ooooooo/oooooooooot

---

Published as a conference paper at ICLR 2021
As one of the most crucial challenges in computer vision, object detection has an urgent requirement of both accurate and efficient models. Unfortunately, most of the existing knowledge distillation methods in computer vision are designed for image classification and usually leads to trivial im- provements on object detection (Li et al. 2017). In this paper, we impute the failure of knowledge distillation object detection the following two issues, which will be solved later, respectively.
Imbalance between foreground and background. In an image to be detected, the background pixels are often more overwhelming than the pixels of the foreground objects. However, in previous knowledge distillation, the student is always trained to mimic the features of all pixels with the same priority. As result, students have paid most of their attention to learning background pixels features, which suppresses student's learning on features of the foreground objects. Since foreground pixels are more crucial in detection, the imbalance hurts the performance of knowledge distillation severely. To overcome this obstacle, we propose the attention- guideed distillation which distills only the crucial foreground pixels. Since the attention map can reflect the position of the important pixels (Zhou et al. 2016), we adopt the attention map the mask for knowledge distillation. Concretely, the pixel with a higher attention value is regarded as a pixel of a foreground object and then is learned by the student model with higher priority. Compared with the previous binary mask method (Wang al.) 2019). the mask generated by attention maps in our methods is more fine- ane-ggrined and requires no additional supervision. Compared with the previous attention- & Komodakis 2017), the attention map in our methods is not only utilized as the information to be distilled but also utilized as the mask signal for feature distillation.
distillation methods (Zagoruyko
Lack of distillation on relation information. is generally acknowledged that the relation between different objects contains valuable information in object detection. Recently, lots of researchers suc- cessfully improve the performance of detectors by enabling detectors to capture and make use of these relations, such as non-local modules (Wang et al. 2018) and relation networks (Hu et al., 2018). However, the existing object detection knowledge distillation methods only distill the infor- mation of individual pixels but ignore the relation of different pixels. To solve this issue, we propose the non-local distillation, which aims to capture the relation information of students and teachers with non-local modules and then distill them from teachers to students
Since the non-local modules and attention mechanism in our methods are only required in the train- ing period, our methods don't introduce additional computation and parameters in the inference period. Besides, our methods are feature-base distillation methods which do not depend on specific detection algorithm so they can be directly utilized in all kinds of detectors without any modification. On MS COCO2017, 2.9, 2.9 and 2.2 AP improvements can be observed one-stage, and anchor-free models on average, respectively. Experiments on Mask RCNN show that our methods can also improve the performance of instance segmentation by 2.0 AP, on average. We have conducted a detailed ablation study and sensitivity study to show the effectiveness and stability of each distillation loss. Moreover, we study the relation between teachers and students on object detection and find that knowledge distillation on object detection requires a high AP teacher, which different from the conclusion in image classification where a high AP teacher may harm the per- formunce of students (Mirzadeh et al. 2019: Cho & Hariharan) 2019). We hope that these results are worth more contemplation knowledge distillation on tasks except for image classification. To sum up, the contribution of this paper can be summarized as follows.
two-stage,
We propose the attention- guided distillation, which emphasizes students' learning on the foreground objects and suppresses students' learning the background pixels.
We propose the non-local distillation, which enables the students to learn not only the infor- mation of the individual pixel but also the relation between different pixels from teachers.
We show that teacher with higher AP is usually better teacher in knowledge distillation on object detection, which is different from the conclusion in image classification.
2 RELATED WORK
As an effective method for model compression and model accuracy boosting, knowledge distillation has been widely utilized in various domains and tasks, including image classification (Hinton al. 2014 Romero et al., 2015 Zagoruyko & Komodakis 2017). object detection (Chen et al. 20.1[117]]
al. 2017 Wang al. 2019 Bajestani & Yang 2020), semantic segmentation (Liu et al. 2019
2

---

Published as a conference paper at ICLR 2021
Stndent Detector
Student Detector
Baddoee Fecuse of
Auchhow Foanes
g
Rock & fead
Layers
Mack that
Jo Denaliu Neall
Fuectios Reaat
Nos Gocoll Midhlle
Masia Foodact
Clonneet AsgPoof
Spactal Anghost
Spurtal
Soukest Chaveel Aticsiion
Studiee
FIX
Q
Relationn
Finstore
in
(c) Non-local Module
AB Soon
Mari 1.2
+
Lote
F.S Lose
Stadeot Dotrctor
Shortal Mont R
Chosseel Muct
Toncher Spatial
Chaveed Atevsiin
Achation
ditertion
Aon'
Chnmoll Aughlao
Sparial Proe
Loce Mochiic
Distitation
Cony Aaper
Noct & Hood D Disvctionn Rowi
Yoic Head
Devchhe Bersl
a stot
Bacthowe Poance
Botioes Foster- I..
Boak
Teacher Detector
Teacher Detector
Trarher Detivaar
(a) Attention-Guidel Distillation
(b) Non-Local Distillation
(d) Overall Framewrok
Figure 2: Details of our methods: (a) Attention-guided distillation generates the spatial and channel attention with average pooling in the channel and spatial dimension, respectively. Then, students are encouraged to mimic the attention of teachers. Besides, students are also trained mimic the feature of teachers, which is masked by the attention of both students and teachers. (b) Non- local distillation captures the relation of pixels in an image with non-local modules. The relation information of teachers is learned by students with L2 norm loss. (c) The architecture of non-local modules. 'Ix1' is convolution layer with Ixl kernel. (d) Distillation loss is applied to backbone features with different resolutions. The detection head and neck are not involved in our methods.
face recognition (Ge et al. 2018), pretrained language model (Sanh et al. 2019 Xu et al. 2020). multi-exit networks training (Zhang et al.. 2019b a). model robustness (Zhang et al.. 2020b) and so on. Hinton et al. (2014) first propose the concept of knowledge distillation where the students trained mimic the results after softmax layers of teachers. Then, abundant methods are pro- posed to transfer the knowledge in teacher's features (Romero et al. (015)) or the variants, such as 2019), FSP (Yim et al. (017), mutual infor- 2019). relation of samples in a batch (Park) mation (Ahn et al. 2019). positive features (Heo et al.
attention (Zagoruyko & Komodakis 2017 Hou et al.
et al.
2019 Tung & Mori 2019).
Improving the performance of object detection becomes hot topic knowledge distillation recently. Chen et al.] (2017) design the first knowledge distillation method on object de- tection, which includes distillation loss on the backbone, the classification head and the re- gression head. Then, many researchers find that the imbalance between the foreground objects and background is crucial problem in detec- tion distillation. Instead of distilling the whole features of backbone networks, Li al. (2017) only apply L2 distillation loss to the features sampled by RPN. Bajestaai & Yang) (2020) propose the temporal knowledge distillation, which introduces hyper-parameter to balance the distillation loss between the pixels of the foreground and background. WWan et al. (2019) propose the fine-grained feature imitation, which only distills the feature near object anchor locations. However, although these works have tried to distill only the pixels of foreground objects, they always reply on the annotation in groundtruth, anchors, and bounding boxes and thus can not be transferred to different kinds of detectors and tasks. In contrast, in our method, the pixels of foreground objects are found with attention mechanism, which can be easily generated from features. As result, it can be directly utilized in all kinds of detectors without any modification. As shown in Figure 3 the difference between the previous mask-based detection distillation method (Wang al. 2019) and our attention-guided distillation can be summarized as
Image
Our Method
Wang al.'s Method
Comparison between the proposed attention attention-guided distillation and other methods.
Figure 3:
3

---

Published as a conference paper at ICLR 2021
follows (i) Our methods generate the mask with attention mechanism while they generate the mask with ground truth bounding boxes and anchor priors. (ii) The mask in our methods is a pixel-wise and fine-grained mask while the mask in their method is an object-wise and binary mask. (iii) The masks in our methods are composed of a spatial mask and a channel mask while they only have a spatial mask. More detailed comparison with related work be found in Appendix
3 METHODOL.OGY
3.1 ATTENTION-GUIDED DISTILLATION
We use E RC.H.W to denote the feature of the backbone in an object detection model, where C, H. W denotes its channel number, height and width, respectively. Then, the generation of the spatial attention map and channel attention map is equivalent to finding the mapping function g* R.C.HW
R.H.W and G R.H.WW TRC respectively. Note that the superscripts and here are utilized to discriminate 'spatial' and 'channel'. Since the absolute value of each element in the feature implies its importance, we construct G by summing the absolute values across the chan- nel dimension and construct 9 by summing the absolute values across the width and height dimen- sion, which can be formulated as (A) H E' Eir |A,1al and (A)) ? Ee Am |AR,..I where J. & denotes the ieh. Jeh. ken slice of in the height, width, and channel dimension, respec- tively. Then, the spatial attention mask M and the channel attention mask M used in attention- guided distillation can be obtained by summing the attention maps from the teacher and the stu- dent detector, which can be formulated M HW (
(*(A))) I)/T), M C'- softmax((G*(AA) ) + ()T))) Note that the superscripts S and T here are used to discrimi- nate students and teachers. T is a hyper-paraaa parameter in softmax introduced by Hinton et al. adjust the distribution of elements in attention masks (see Figure 4). The attention-guide distillation loss LAGD is composed of two components attention transfer loss LAT and attention-maske masked loss LAM- LAT is utilized to encourage the student model to mimic the spatial and channel attention of the teacher model, which can be formulated as
AAT = (2( (A'), - (A9
(1)
AM is utilized to encourage the student to mimic the features of teacher models by a L2 norm loss masked by M and M which be formulated
c
H
W
LAM
ΣΣΣ
(AT
A MM.. Mg
(2)
3.2 NON-LOCAL DISTILLATION
Non-local module (Wang et al. 2018 ) is an effective method to improve the performance of neu- ral networks by capturing the global relation information. In this paper, we apply non-local modules to capture the relation between pixels in an image, which can be formulated as rı,j =
WH EI=' J(A...., A, 5') (' where denotes the obtained relation information. j are the spatial indexes of an output position whose response is to be computed. أ are the spa- tial indexes that enumerates all possible positions. f a pairwise function for computing the relation of two pixels and is an unary function for computing the representation of an individual pixel. Now, we can introduce the proposed non-local distillation loss LNLD as the L2 loss between the relation information of the students and teachers, which can be formulated as LXLD (r ;T))
1
3.3 OVERALL LOSS FUNCTION
We introduce three hyper-parameters a, 3, to balance different distillation loss in our methods. The overall distillation loss can be formulated as
L Distau (AT A,,
LAT + B. LAM+ + J I.LNDD,
(3)
Artention-guided distillation
Non-local distillation
4

---

Published as a conference paper at ICLR 2021
Image
Attriou
Spatial-Atetiion
Spotial (T (T-1.00)
Ditribation
Viluo
Figure 4: Visualization and distribution of the spatial attention with different T. With smaller T, the pixels of high attention values are emphasized more in knowledge distillation.
The overall distillation loss is model-agnostic loss, which be added to the original training loss of any detection model directly. The sensitivity study of each hyper-parameter and the ablation study of each loss are shown in Figure 3 and Table 4 respectively.
4 EXPERIMENT
4.1 EXPERIMENTS SETTINGS
The proposed knowledge distillation method is evaluated on MS COCO2017, which is large-scale dataset that contains over 120k images spanning 80 categories (Lin et al. 2014). The benchmark detection networks are composed of both two-stage detection models, including Faster RCNN (Ren et al. 2015). Cascade RCNN (Cai & Vasconcelos) 2019). Dynamic RCNN (Zhang et al.) (220a)). Grid RCNN (Lu et al.
2019) and one-stage detection models, including the RetinaNet (Lin al. 2017). Fsaf RetinaNet (Zhu et al.) 2019). Besides, we also evaluate our methods on the Mask RC et al.] 2017). Cascade Mask RCNN (Cai & Vasconcelos 2019), and anchor-free models RepPoints (Yang et al.) 2019). We adopt the ResNet50 and ResNer101 (He et al. 2016) as the backbone network each detection model. We pre-train the backbone model on ImageNet (Deng ett al. 2009) and then finetune it MS COCO2017. We have compared our methods with three kinds of object detection knowledge distillation methods (Chen et al., 2017 Wang et al. 2019 Heo al. (2019). All the experiments in this paper implemented with py Torch (Paszke et al. 2019) with mmdetection2 framework (Chen et al. 2019). The reported fps is measured on one RTX 2080TI
GPU. We adopt the same hyper-parameters settings {a - - 10 3 4 10 3,T 0.1} for all the two-stage models and {a stage models. Cascade Mask RCNN with ResNeXt101 backbone is utilized as the teacher for all the two-stage students and RetinaNet with ResNeXt101 backbone is utilized as the teacher for all the one-stage students. Please refer to the codes in Github for more details.
= 4 x 10 10
x 10 20-2,7
0.5) for all the one-
4.2 EXPERIMENT RESULTS
In this section, show the experiment results of the baseline detectors and models in Table and Table and compare our methods with other three knowledge distillation methods in Table observed that: (i) Consistent and significant AP boost can be observed on all the kinds of detectors. On average, there are 2.9, 2.9, and 2.2 AP improvements on the two-stage, one-stage, and anchor-free detectors, respectively. (ii) With the proposed method, student model with ResNet50 backbone can outperform the same model with ResNet101 backbone by 1.2 AP on average. (iii) On Mask RCNN related models, there are 2.3 improvements on bounding box AP and 2.0 improvements on mask AP average respectively, indicating the proposed method can be utilized in not only object detection but also instance segmentation. (iv) Our methods achieve 2.2 higher AP than the second-Be best distillation method, on average. (v) There are 2.7 and 2.9 AP improvements models with ResNet50 and ResNet101 backbones, respectively, indicating that deeper detectors benefit more from knowledge distillation.
5

---

Published as a conference paper at ICLR 2021
Table I: Experiments MS
COC02017 with the proposed distillation method.
Model
Backbone
AP
AP33 50
AP75
APs
APM
A
FPS
Params
Faster RCNN
ResNet50
38.4
59.0
42.0
21.5
42.1
50.3
18.1
43.57
Our Faster RCNN
ResNet50
41.5
62.2
45.1
23.5
45.0
55.3
18.1
43.57
Faster RCNN
ResNet101
39.8
60.1
43.3
22.5
43.6
52.8
14.2
62.57
Our Faster RCNN
ResNet101
43.9
64.2
48.1
25.3
48.0
58.7
14.2
62.57
Cascade RCNN
ResNet50
41.0
59.4
44.4
22.7
44.4
54.3
15.4
71.22
Our Cascade RCNN
ResNet50
44.4
62.7
48.3
24.8
48.0
59.3
15.4
71.22
Cascade RCNN
ResNet101
42.5
60.7
46.4
23.5
46.5
56.4
11.7
90.21
Our Cascade RCNN
ResNet101
45.2
63.5
49.4
26.2
48.7
60.8
11.7
90.21
Dynamic RCNN
ResNer50
39.8
58.3
43.2
23.0
42.8
52.4
18.1
43.57
Our Dynamic RCNN
ResNet50
42.8
61.2
47.0
23.9
46.2
57.7
18.1
43.57
Dynamic RCNN
ResNet101
41.2
59.7
45.3
24.0
44.9
54.3
14.2
62.57
Our Dynamic RCNN
ResNet101
44.8
63.0
48.9
25.0
48.9
60.4
14.2
62.57
Grid RCNN
ResNet50
40.4
58.4
43.6
22.8
43.9
53.3
14.0
66.37
Our Grid RCNN
ResNet50
42.6
61.1
46.1
24.2
46.6
55.8
14.0
66.37
Grid RCNN
ResNet101
41.6
59.8
45.0
23.7
45.7
54.7
11.0
85.36
Our Grid RCNN
ResNet101
44.8
63.6
48.9
26.5
48.9
59.6
11.0
85.36
RetinaNet
ResNet50
37.4
56.7
39.6
20.0
40.7
49.7
17.7
37.74
Our RetinaNet
ResNet50
39.6
58.8
42.1
22.7
43.3
52.5
17.7
37.74
RetinaNet
ResNet101
38.9
58.0
41.5
21.0
42.8
52.4
13.5
56.74
Our RetinaNet
ResNet101
41.3
60.8
44.3
22.7
46.0
55.2
13.5
56.74
Fsaf RetinaNet
ResNet50
37.8
56.8
39.8
20.4
41.1
48.8
20.0
36.19
Our Fsaf RetinaNet
ResNet50
41.4
61.0
44.2
23.1
45.2
55.2
20.0
36.19
Fsaf RetinaNet
ResNet101
39.3
58.6
42.1
22.1
43.4
51.2
15.0
55.19
Our Fsaf RetinaNet
ResNet101
42.6
62.0
45.5
24.5
47.0
56.2
15.0
55.19
RepPoints
ResNet50
38.6
59.6
41.6
22.5
42.2
50.4
18.2
36.62
Our RepPoints
ResNet50
40.6
61.7
43.8
23.4
44.6
53.0
18.2
36.62
RepPoints
ResNet101
40.5
61.3
43.5
23.4
44.7
53.2
13.2
55.62
Our RepPoints
ResNet101
42.7
63.7
46.4
24.9
47.2
56.4
13.2
55.62
Table 2: Experiments on MS COC02017 with the proposed distillation method on Mask RCNN.
Bounding box AP
Mask AP
Model
Backbone
AP
APs
APM
APr
AP
APs
APM
APı
Mask RCNN
ResNet50
39.2
22.9
42.6
51.2
35.4
19.1
38.6
48.4
Our mask RCNN
ResNet50
41.7
23.4
45.3
55.8
37.4
19.7
40.5
52.1
Mask RCNN
ResNet101
40.8
23.0
45.0
54.1
36.6
19.2
40.2
50.5
Our Mask RCNN
ResNet101
43.0
24.7
47.2
57.1
38.7
20.7
42.3
53.3
Cascade Mask RCNN
ResNet50
41.9
23.2
44.9
55.9
36.5
18.9
39.2
50.7
Our Cascade Mask RCNN
ResNet50
43.8
24.9
47.2
58.4
38.0
20.2
40.9
52.8
Cascade Mask RCNN
ResNet101
42.9
24.4
46.5
57.0
37.3
19.7
40.6
51.5
Our Cascade Mask RCNN
ResNet101
45.4
26.3
49.0
60.9
39.6
21.3
42.8
55.0
4.3 ABLATION STUDY AND SENSITIVITY STUDY
Ablation study. Table shows the ablation study of the proposed attention-guided distillation AT and LAM) and non-local distillation (CxLD). It is observed that: (i) Attention-guide distillation and non-local distillation lead to 2.8 and 1.4 AP improvements, respectively. (i) LAT and LAM lead to 1.2 and 2.4 AP improvements respectively, indicating that most the benefits of attention- guided distillation obtained from the feature loss masked by the attention maps (LAM) (iii)

---

Published as a conference paper at ICLR 2021
There are 3.1 AP improvements with the combination of attention aitention-guided distillation and non-local distillation. These observations indicate that each distillation loss our methods has their individual effectiveness and they be utilized together to achieve better performance. We also give ablation study to the spatial and channel attention in Appendix
Sensitivity study byper-parameters. Four nyper-parameter 3, and y are utilized to balance the magnitude of different distillation loss and T is utilized to adjust the distribution attention masks. The hyper-parameter sensitivity study on MS COCO2017 with Faster RCNN (ResNet50 backbone) is introduced in Figure [ It is observed that the worst Hyper-parametesn only lead to 0.3 AP drop compared with the highest AP, which still 2.9 higher compared with the baseline model, indicating that our methods are not sensitive to the choice of hyper-parameters
introduced in this paper. a,
Sensitivity study on the types of non-local modules. There are four kinds of non-local modules, including Gaussian, embedded Gaussian, dot production, and concatenation. Table [ shows the performance of our methods with different types of non-local modules. It is observed that the worst non-local type (Gaussian) is only 0.2 AP lower than the best non-local type (Embedded Gaussian and Concatenation), indicating our methods are not sensitive to the choice of non-local modules.
Table 3: Comparison between our methods and other distillation methods. Note that don't
compare our methods with Chen's 'ss and Wang's methods on RetinaNet because their methods can not be utilized in one-stage models. ResNet50 is utilized backbone in these models.
Model
AP
AP50 50
AP75
APs
APM
APL
Faster RCNN
38.4
59.0
42.0
21.5
42.1
50.3
Chen et al.
Method
38.7
59.0
42.1
22.0
41.9
51.0
Wang
Method
39.1
59.8
42.8
22.2
42.9
51.1
Heo al.
Method
38.9
60.1
42.6
21.8
42.7
50.7
Our Methods
41.5
62.2
45.1
23.5
45.0
55.3
Cascade RCNN
41.0
59.4
44.4
22.7
44.4
54.3
Chen al.
Method
41.6
62.1
45.7
23.9
45.8
54.7
Wang et al.
Method
42.1
62.7
46.1
24.4
46.4
55.6
Heo
Method
41.7
62.3
45.7
23.9
45.7
55.3
Our Methods
44.4
62.7
48.3
24.8
48.0
59.3
RetinaNet
37.4
56.7
39.6
20.0
40.7
49.7
Heo al. Method
37.8
58.3
41.1
21.6
41.2
48.3
Our Methods
39.6
58.8
42.1
22.7
43.3
52.5
Table 5: Results of different types of non-local modules on Faster RCNN (ResNet50 backbone).
Table 4: Ablation study of the three distillation loss.
CAT
X
X
X
Loss
CAM
/
LNLD
X
X
X
X
Non-Local Type
AP
AP
38.4
39.6
40.8
39.8
41.2
41.5
Embedded Gaussian
41.5
AP3
21.5
22.7
22.8
22.7
23.0
23.5
Result
Dot Production
41.4
APM
42.1
42.9
44.3
43.1
44.6
45.0
Concatenation
41.5
AP,
50.3
52.5
54.3
52.3
55.3
55.3
Gaussian
41.3
DISCUSSION
5.1 ANALYSIS ON THE BENEFITS KNOWLEDGE DISTILLATION
Qualitative analysis. Figure shows the comparison of detection results between baseline and distilled detector, is observed that: (i) Our methods improve the detection ability on small-objects. In the first three figures, the distilled model correctly detect cars, the handbag, and the person in

---

Published as a conference paper at ICLR 2021
the car, respectively. (ii) Our methods prevent models from generating multiple bounding boxes for the same object. In the last two figures, the baseline model generates multiple bounding boxes for the boat and the train while the distilled model avoids these errors.
Analysis on the types of detection error. We have analyzed the different types of detection errors in the baseline and distilled models in Figure The number in the legend indicates AUC (area under the curve). It is observed that our distillation method leads to error reduction on all kinds of error. In brief, our methods can improve the ability of both localization and classification.
Avenuge Precision
41.7
41.7
41.7
416
xxx 30
F 10
y 10
91.6
41.6
41.2
41.2
61.2
41.0
o
value
Figure 5: Hyper-parameter sensitivity study of a, B, ,, T with Faster RCNN MS COC02017.
Baseline Model
listiin
Distilled Midel
Furnii
Figure 6: Qualitative analysis on MS COC02017 with distilled and baseline Faster RCNN. We mark the undetected and wrongly detected objects of the baseline detector with orange boxes.
5.2 RELATION BETWEEN STUDENT DETECTORS AND TEACHER DETECTORS.
There is sufficient research focusing on the relation between students and teachers. Mirzadeh et al. (2019) and Cho & Hariharan 2019) show that a teacher with higher accuracy may not be the better teacher for knowledge distillation and sometimes teacher with too high accuracy may harm the performance of students. Besides, Mobahi et al.] (2020) and Yuan et al. (219)) show that the same model and even model with lower accuracy than the student model can be utilized as the teacher model for knowledge distillation. However, all their experiments are conducted on image classifica- tion. In this section, we study whether these observations still hold in the task of object detection. As shown in Figure 8 we conduct experiments on Faster RCNN (ResNet50 backbone) and Cas- cade RCNN (ResNet50 backbone) students with teacher models of different AP. is observed that: (i) In all of our experiments, the student with higher AP teacher always achieves higher AP. (ii) When the teacher has lower or the same AP as the student, there are very limited and even negative improvements with knowledge distillation. This observation indicates that the relation between stu- dents and teachers on object detection opposite to that on image classification. Our experiment results suggest that there is strong positive correlation between the AP of students and teachers. A high AP teacher tends to improve the performance of students significantly.
We think that the reason why a high AP teacher model is crucial in object detection but not very necessary in image classification is that object detection a more challenging task. As a result, a weaker teacher model may introduce more negative influence students, which prevents students from achieving higher AP. In contrast, on image classification, most of teacher models can achieve very high training accuracy they don't 't introduce much error.
8

---

Published as a conference paper at ICLR 2021
Dieiiled Faster RCNN50
RCNN30 Beseline
1.0
0.8
0.5
43-
procsion
0.-
prodisio
0..6
Stulent
42
(8,53IC19
0.4
[6A221558
12 34C2226
Faster RCNNEO
0.2
[077410m
(1
(60055m
154,205
39-
Base
RCHNSD
Baseine
8.
0.6
01000/m
012
0.4
0.b
0..5
1.0
04
163
014
06
16.5
1.0
30
46
recall
recall
Teacher AP
Figure 7: Distribution of error types on distilled and baseline Faster RCNN. Loc Localization error; Sim & Oth Classifica- tion error on similar & not similar classes; BG False positive prediction fired on background. FN False Negative prediction.
Figure 8: Relation between stu- dents and teachers Faster RCNN and Cascade RCNN in MS COC02017.
6 CONCLUSION
In this paper. we have proposed two knowledge distillation methods, including attention -etenttion-uuieedd guided distillation and the non-local distillation to improve the performance of object detection models. iitention-guiden distillation manages to find the crucial pixels and channels from the whole feature map with attention mechanism and then enables the student to focus more on these crucial pixels and channels instead of the whole feature map. Non-local distillation enables students to learn not only the information of an individual pixel but also the relation between different pixels captured by the non-local modules. Experiments 9 kinds of models including two-stage, one-stage, anchor-free and anchor-based models have been provided to evaluate our methods.
Besides. we have also given study the relation between students and teachers in object detection. Our experiments show that there is a strong positive correlation between the AP of teachers and stu- dents. A high AP teacher detector plays essential role in knowledge distillation. This observation much different from the previous conclusion in image classification, where a teacher model with very high accuracy may harm the performance of knowledge distillation. We hope that our result may call for more rethinking works on knowledge distillation in tasks except image classification.
7 ACKNOWLEDGEMENT
This work was partially supported by Institute for interdisciplinary Information Core Technology.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference Com- puter Vision and Pattern Recognition, pp. 9163-9171, 2019.
Mohammad Farhadi Bajestani and Yezhou Yang. Tkd: Temporal knowledge distillation for active perception. In The IEEE Winter Conference Applications of Computer Vision, pp. 953-962, 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 531-541. ACM, 2006.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance seg- mentation. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2019. ISSN 1939-3539.
doi: 10. http://pmmm.....29555.2
do wwwp://ww.o.oo...oooooo/oooooo URL http:://.........rg//0000.010000
tpami 201 2956516
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances Neural Information Process- ing Systems, pp. 742-751, 2017.

---

Published as a conference paper at ICLR 2021
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint 1r066v1906.011
2019.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1803::1803.03660 2018.
Shiming Ge, Shengwei Zhao, Chenyu Li. and Jia Li. Low-resolution face recognition in the wild via selective knowledge distillation. IEEE Transactions on Image Processing, 28(4)2205-2 2018.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. 2017 IEEE Interna- tional Conference on Computer Vision (ICCV), Oct 2017.
Byeongho Heo, Jecsoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1921-1930, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NewrlPS, 2014.
Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detec- tion cnns by self attention distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1013-1021, 2019.
Andrew Howard. Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint arXiv: 022444 2019.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588-3597 2018
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. In ICLR, 2016.
Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference computer vision and patlern recognition, pp. 6356-6364, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, 2017.
10

---

Published as a conference paper at ICLR 2021
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2604-2613, 2019.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprini arXiv: 1810.55270 2018.
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid r-cnn. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowl- edge distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint arXiv: 902...393........ 2019.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-distillation amplifies regularization in hilbert space. arXiv preprint arXiv:2002.05715, 2020.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In The IEEE International Conference on Com- puter Vision (ICCV), October 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, pp. 8024-8035. 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn; Towards real-time object detection with region proposal networks, In Advances in neural information processing systems, pp. 91-99, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chios Chiih Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01100 2019.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision pp. 1365-1374, 2019.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 4933-4942, 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
11

---

Published as a conference paper at ICLR 2021
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseuss Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
Ze Yang. Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. In The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: teacher-free framework. arXiv preprint arXiv: 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Hongkai Zhang, Hong Chang, Bingpeng Ma, Naiyan Wang, and Xilin Chen. Dynamic R-CNN: Towards high quality object detection via dynamic training. arXiv preprint arXiv:2004.0600 2020a.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.
Linfeng Zhang, Zhanhong Tan, Jiebo Song, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Scan: A scalable neural networks framework towards compact and efficient models. In Advances in Neural Information Processing Systems 32, pp. 1027-4036. Curran Associates, Inc., 2019b.
Linfeng Zhang, Muzhou Yu, Tong Chen, Zuoqiang Shi, Chenglong Bao, and Kaisheng Ma. Auxil- iary training: Towards accurate and robust models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 372-381, 2020b.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In The European Conference on Computer Vision (ECCV), September 2018.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03000 707000000044, 2017.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Chenchen Zhu, Yihui He, and Marios Savvides. Feature selective anchor-free module for single- shot object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 840-849, 2019.
A ABLATION STUDY ON THE SPATIAL AND CHANNEL ATTENTION
Different from previous attention-based knowledge distillation methods, the attention-guider distil- lation in our methods uses not only spatial attention but also the channel attention. In this appendix, we have conducted an ablation study on the two kinds of attention with Faster RCNN (ResNet50) on MS COC02017 to show their individual effectiveness.
is observed that spatial attention and channel attention lead to 2.6 and 2.3 AP improvements, re- spectively. In contrast, the combination of the two kinds of attention leads to 2.8 AP improvements. These results indicate that both spatial and channel attention have their individual effectiveness and they can be utilized together to achieve better performance.
12

---

Published as a conference paper at ICLR 2021
Table 6: Ablation study the spatial attention and channel attention.
Spatial Channel
Attention Type
X
V
AP
38.4
41.0
40.7
41.2
APs
21.5
22.7
22.9
23.0
Result
AP M
42.1
44.7
44.1
44.6
APL
50.3
54.2
54.1
55.3
B ADAPTATION LAYERS IN KNOWLEDGE DISTILLATION
The adaptation layers in knowledge distillation are first proposed by Romero et al. (2015) to adjust the feature size of students and teachers. Then, recent research finds that the adaptation layers play an important role in improving the performance students (Chen et al. 2017). this paper, we adopt different kinds of adaptation layers for different distillation loss. Concretely, We adopt 1x1 convolutional layers for LAM and LNLD. 3x3 convolutional layers for Lapetaal and fully connected layers for AT Cohanmel Note that the adaptation layers only utilized the training period and they don't introduce additional computation and parameters.
C EXPERIMENTS ON SMALLER BACKBONES
According to the insightful comments of the reviewers, we conduct a series of experiments on mod- els with small backbones including ResNet18 and RegNet-80OM, and compact detectors including Yolo v3 and SSD. As shown in Table 7 our methods also achieve significant AP improvements on these compact models. Note that more experiments with small backbones will be added in the camera ready version.
Table 7: Experiments on MS
COCO2017 with our method on small backbones.
Model
Backbone
AP
APsu
AP
APs
AP
APL
FPS
Params
Faster RCNN
ResNet18
34.6
55.0
37.1
19.3
36.9
45.9
28.1
30.57
Our Faster RCNN
ResNet18
37.0
57.2
39.7
19.9
39.7
50.3
28.1
30.57
Grid RCNN
RcsNet18
36.6
54.2
39.7
20.1
39.8
48.2
26.7
66.37
Our Grid RCNN
ResNet18
38.8
56.7
41.5
21.1
41.6
52.7
26.7
66.37
RetinaNet
ResNet18
33.4
51.8
35.1
16.9
35.6
44.9
25.8
23.30
Our RetinaNet
ResNet18
35.9
54.4
38.0
17.9
39.1
49.4
25.8
23.30
RetinaNet
RegNet-800M
35.6
54.7
37.7
19.7
39.0
47.8
22.4
19.27
Our RetinaNet
RegNet-800M
38.4
57.4
40.7
21.4
42.0
52.3
22.4
19.27
Yolo v3
DarkNet53
33.4
56.3
35.2
19.5
36.4
43.6
42.2
61.95
Yolo
DarkNet53
35.8
58.2
38.1
21.2
39.0
45.6
42.2
61.95
SSD
VGG16
29.4
49.3
31.0
11.7
34.1
44.9
26.1
38.08
Our SSD
VGG16
31.2
52.
32.8
12.6
37.4
46.2
26.1
38.08
D EXPERIMENTS ON CITYSCAPES
According to the insightful comments of the reviewers, shown in Table 8 conduct a series of experiments to show the effectiveness of our method on Cityscapes. Note that more experiments on Cityscapes will be added in the camera ready version.
13

---

Published as a conference paper at ICLR 2021
Table 8: Experiments Cityscapes with method.
Model
Backbone
Box AP
Mask AP
Faster RCNN
ResNer50
40.3
N/A
Our Faster RCNN
ResNer50
43.5
N/A
Mask RCNN
ResNet50 ResNet50
41.0
35,8
Our Mask RCNN
43.0
37.5
E COMPARISION WITH RELATED WORK
Comparison Methodology and Application. Feature distillation is utilized in all the five meth- ods. However, Chen et al.] distill not only the feature, but also the classification logits and bound- ing box regression results, which has limited their application scenes in stage and anchor-fres models. [ et al. and Wang et al.] distill the features in the regions of proposals and near object anchor locations, respectively. As result, their methods reply the supervision of anchors and groundtruths and can't t be utilized in one stage and anchor free models. Bajestani & Yang is utilized for active perception on video, which can not be utilized in image-based detection. In contrast, in our method, the attention mask and relation information can be easily generated from the backbone features, which has no requirements on groundtruths, anchors and proposals. As result, it can be easily used in different kinds of models and tasks without any modification.
Table 9: Comparision on methodology and application.
Method
Feature Classify Regress Relation Stage Two-Stage Anchor based Anchor-fiet free Segment.
Our Method
X
x
et
AXXXX
EXX
X
Li ct al
X
X
X
X
Wang el al.
X
X
Bajestuni &
V
X
X
X
Comparison on Motivation. Chen et al. method is direct application of knowledge distillation on object detection. The other three methods and our method are motivated by the imbalance be- tween foreground and background piexles and these methods try address this issue by reweighting the distillation loss. Besides, our method also motivated by the effect of the relation among pixels in image, which is ignored by the other methods.
14


Reviews:
[{"review_id": "ob0v3MR2Dj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the knowledge distillation problem in object detection. It claims that the failure of knowledge distillation in object detection is mainly caused by the imbalance between pixels of foreground and background, and the relation distillation between different pixels. The authors then propose non-local distillation to tackle the problem. Extensive experiments are conducted on MS COCO and verify the effectiveness of the proposed method. \n\n1. Strengthens\n\na. This paper explores the knowledge distillation problem in object detection and gives promising results and conclusions. \n\nb. The results with different detectors greatly prove the effectiveness of the method. \n\nc. It is good that the distillation method does not bring any extra costs during the inference time.\n\n2. Weaknesses\n\na. I recommend the authors do experiments on much smaller models. For example, in real cases, mimicking ResNet50 with larger models is meaningless. It would be great if we can observe large improvements when mimicking a small backbone with ResNet-50 or even larger models. These conclusions and experiments are crucial for the applications of the method.\n\nb. The English writing of this paper can be improved. \n\nc. The names of the models in this paper are not proper. It is weird for me to call Faster RCNN101. Generally, Faster R-CNN is Faster R-CNN, and ResNet is ResNet. You can use any backbones with Faster R-CNN. Please define it as Faster R-CNN with ResNet101. \n\nAs stated comments, I think this paper is good for knowledge distillation in object detection and publishable, but encourages the authors improve the draft according to the weaknesses.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kbedPNPDFwa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n- The different attention techniques seem to consistently improve object detectors across different models. \n- The ablation studies are important in showing the advantage and impact of each proposed module. \n- Please clarify if the student models start from random weights or are initialized after the teacher training. In this regard to what extend is the approach is transferable to students with random weights?\n\nCons:\n\n- The authors only show results when training a network of the same structure with the added modules, in an attempt to boost accuracy. However, an important use of knowledge of knowledge distillation is to train smaller models, and such results are missing from the paper. Overall, the results in the paper somewhat justify the \"accurate\" part of the title but not the \"efficient\".\n\n- The importance of high AP is a bit overstated. It is important to demonstrate the different behavior between classification and detection networks, however, the same improvement is observed irrespective of the teacher AP performance. \n\n- The structure of the paper can be improved. The related work section for example can be moved earlier in the paper to give the bigger picture and the position of this work with respect to the literature. \n\n- For a more complete comparison YOLO and SSD networks should be included.\n\n- In the qualitative analysis observation (ii) states that the proposed methods leads to single box per object compared to the baselines. In this case do the baselines use non-maximum suppression? This is a standard post processing technique that alleviates this problem. No explanation is given on why the proposed method leads to this behavior.\n\nThe authors have made clear some of my concerns and made revisions accordingly. Thus I am in a position now to recommend this paper, thus I update my initial recommendation from 5 to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_B-C27rmvO1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a knowledge distillation method for object detection. In particular, the technical contribution is mainly two-fold: attention-guided distillation module and non-local distillation module, as shown in (a) and (b) of Figure 2. The proposed modules provide consistent improvements in detection MAP across different architectures.\n\nOn the positive side, I believe the paper has the following merits:\n- The proposed modules are well-motivated, and seem to provide a consistent boost in different architectures.\n- The authors provide very detailed information to reproduce the method. Besides, I also appreciate the authors have provided the code in the supplementary material.\n- The observation that a high-AP teacher is important for distillation is quite intriguing.\n\nOverall I believe the paper has made a decent contribution, but the following aspects can be improved:\n- As mentioned by the authors, distillation is discussed for object detection in several works (n (Chen et al., 2017; Li et al., 2017; Wang et al., 2019; Bajestani & Yang, 2020). I believe it would be helpful to summarize the key difference/similarity wrt to the previous work, thus to provide a better understanding of the relation in a larger context.\n- it's not fully clear to me what the attention in attention-guided distillation captures, would the authors provide further experimental analysis? and what are the failure cases of attention-guided distillation? \n- it would be helpful to provide results on another dataset(e.g. cityscapes) to confirm the effectiveness in a different detection setting.\n\n--- \nAfter reading the authors' response and other reviews. I still believe the paper has made a good contribution thus I would stick with my original rating.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal stage, all reviewers lean positive (in final scores and/or in comments during the discussion phase). The AC found no reason to disagree. The benefit of the proposed method is demonstrated in many diverse settings, and the authors argue novelty in that no prior work addresses both fg/bg imbalance and relation distillation. 
Decision: Accept (Poster)