Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors

1 INTRODUCTION Recently , excellent breakthrough in various domains has been achieved with the success of deep learning ( Ronneberger et al. , 2015 ; Devlin et al. , 2018 ; Ren et al. , 2015 ) . However , the most advanced deep neural networks always consume a large amount of computation and memory , which has limited their deployment in edge devices such as self-driving cars and mobile phones . To address this problem , abundant techniques are proposed , including pruning ( Han et al. , 2016 ; Zhang et al. , 2018 ; Liu et al. , 2018 ; Frankle & Carbin , 2018 ) , quantization ( Nagel et al. , 2019 ; Zhou et al. , 2017 ) , compact model design ( Sandler et al. , 2018 ; Howard et al. , 2019 ; Ma et al. , 2018 ; Iandola et al. , 2016 ) and knowledge distillation ( Hinton et al. , 2014 ; Buciluǎ et al. , 2006 ) . Knowledge distillation , which is also known as teacher-student learning , aims to transfer the knowledge of an over-parameterized teacher to a lightweight student . Since the student is trained to mimic the logits or features of the teacher , the student can inherit the dark knowledge from the teacher , and thus often achieves much higher accuracy . Due to its simplicity and effectiveness , knowledge distillation has become a popular technique for both model compression and model accuracy boosting . ∗Corresponding author †https : //github.com/ArchipLab-LinfengZhang/Object-Detection-Knowledge-Distillation-ICLR2021 As one of the most crucial challenges in computer vision , object detection has an urgent requirement of both accurate and efficient models . Unfortunately , most of the existing knowledge distillation methods in computer vision are designed for image classification and usually leads to trivial improvements on object detection ( Li et al. , 2017 ) . In this paper , we impute the failure of knowledge distillation on object detection to the following two issues , which will be solved later , respectively . Imbalance between foreground and background . In an image to be detected , the background pixels are often more overwhelming than the pixels of the foreground objects . However , in previous knowledge distillation , the student is always trained to mimic the features of all pixels with the same priority . As a result , students have paid most of their attention to learning background pixels features , which suppresses student ’ s learning on features of the foreground objects . Since foreground pixels are more crucial in detection , the imbalance hurts the performance of knowledge distillation severely . To overcome this obstacle , we propose the attention-guided distillation which distills only the crucial foreground pixels . Since the attention map can reflect the position of the important pixels ( Zhou et al. , 2016 ) , we adopt the attention map as the mask for knowledge distillation . Concretely , the pixel with a higher attention value is regarded as a pixel of a foreground object and then is learned by the student model with a higher priority . Compared with the previous binary mask method ( Wang et al. , 2019 ) , the mask generated by attention maps in our methods is more fine-grained and requires no additional supervision . Compared with the previous attention-based distillation methods ( Zagoruyko & Komodakis , 2017 ) , the attention map in our methods is not only utilized as the information to be distilled but also utilized as the mask signal for feature distillation . Lack of distillation on relation information . It is generally acknowledged that the relation between different objects contains valuable information in object detection . Recently , lots of researchers successfully improve the performance of detectors by enabling detectors to capture and make use of these relations , such as non-local modules ( Wang et al. , 2018 ) and relation networks ( Hu et al. , 2018 ) . However , the existing object detection knowledge distillation methods only distill the information of individual pixels but ignore the relation of different pixels . To solve this issue , we propose the non-local distillation , which aims to capture the relation information of students and teachers with non-local modules and then distill them from teachers to students . Since the non-local modules and attention mechanism in our methods are only required in the training period , our methods don ’ t introduce additional computation and parameters in the inference period . Besides , our methods are feature-based distillation methods which do not depend on a specific detection algorithm so they can be directly utilized in all kinds of detectors without any modification . On MS COCO2017 , 2.9 , 2.9 and 2.2 AP improvements can be observed on two-stage , one-stage , and anchor-free models on average , respectively . Experiments on Mask RCNN show that our methods can also improve the performance of instance segmentation by 2.0 AP , on average . We have conducted a detailed ablation study and sensitivity study to show the effectiveness and stability of each distillation loss . Moreover , we study the relation between teachers and students on object detection and find that knowledge distillation on object detection requires a high AP teacher , which is different from the conclusion in image classification where a high AP teacher may harm the performance of students ( Mirzadeh et al. , 2019 ; Cho & Hariharan , 2019 ) . We hope that these results are worth more contemplation of knowledge distillation on tasks except for image classification . To sum up , the contribution of this paper can be summarized as follows . • We propose the attention-guided distillation , which emphasizes students ’ learning on the foreground objects and suppresses students ’ learning on the background pixels . • We propose the non-local distillation , which enables the students to learn not only the information of the individual pixel but also the relation between different pixels from teachers . • We show that a teacher with higher AP is usually a better teacher in knowledge distillation on object detection , which is different from the conclusion in image classification . 2 RELATED WORK . As an effective method for model compression and model accuracy boosting , knowledge distillation has been widely utilized in various domains and tasks , including image classification ( Hinton et al. , 2014 ; Romero et al. , 2015 ; Zagoruyko & Komodakis , 2017 ) , object detection ( Chen et al. , 2017 ; Li et al. , 2017 ; Wang et al. , 2019 ; Bajestani & Yang , 2020 ) , semantic segmentation ( Liu et al. , 2019 ) , face recognition ( Ge et al. , 2018 ) , pretrained language model ( Sanh et al. , 2019 ; Xu et al. , 2020 ) , multi-exit networks training ( Zhang et al. , 2019b ; a ) , model robustness ( Zhang et al. , 2020b ) and so on . Hinton et al . ( 2014 ) first propose the concept of knowledge distillation where the students are trained to mimic the results after softmax layers of teachers . Then , abundant methods are proposed to transfer the knowledge in teacher ’ s features ( Romero et al. , 2015 ) or the variants , such as attention ( Zagoruyko & Komodakis , 2017 ; Hou et al. , 2019 ) , FSP ( Yim et al. , 2017 ) , mutual information ( Ahn et al. , 2019 ) , positive features ( Heo et al. , 2019 ) , relation of samples in a batch ( Park et al. , 2019 ; Tung & Mori , 2019 ) . Improving the performance of object detection becomes a hot topic in knowledge distillation recently . Chen et al . ( 2017 ) design the first knowledge distillation method on object detection , which includes distillation loss on the backbone , the classification head and the regression head . Then , many researchers find that the imbalance between the foreground objects and background is a crucial problem in detection distillation . Instead of distilling the whole features of backbone networks , Li et al . ( 2017 ) only apply L2 distillation loss to the features sampled by RPN . Bajestani & Yang ( 2020 ) propose the temporal knowledge distillation , which introduces a hyper-parameter to balance the distillation loss between the pixels of the foreground and background . Wang et al . ( 2019 ) propose the fine-grained feature imitation , which only distills the feature near object anchor locations . However , although these works have tried to distill only the pixels of foreground objects , they always reply on the annotation in groundtruth , anchors , and bounding boxes and thus can not be transferred to different kinds of detectors and tasks . In contrast , in our method , the pixels of foreground objects are found with attention mechanism , which can be easily generated from features . As a result , it can be directly utilized in all kinds of detectors without any modification . As shown in Figure 3 , the difference between the previous mask-based detection distillation method ( Wang et al. , 2019 ) and our attention-guided distillation can be summarized as follows ( i ) Our methods generate the mask with attention mechanism while they generate the mask with ground truth bounding boxes and anchor priors . ( ii ) The mask in our methods is a pixel-wise and fine-grained mask while the mask in their method is an object-wise and binary mask . ( iii ) The masks in our methods are composed of a spatial mask and a channel mask while they only have a spatial mask . More detailed comparison with related work can be found in Appendix.E . 3 METHODOLOGY . 3.1 ATTENTION-GUIDED DISTILLATION . We use A ∈ RC , H , W to denote the feature of the backbone in an object detection model , where C , H , W denotes its channel number , height and width , respectively . Then , the generation of the spatial attention map and channel attention map is equivalent to finding the mapping function Gs : RC , H , W −→ RH , W and Gc : RC , H , W −→ RC , respectively . Note that the superscripts s and c here are utilized to discriminate ‘ spatial ’ and ‘ channel ’ . Since the absolute value of each element in the feature implies its importance , we construct Gs by summing the absolute values across the channel dimension and construct Gc by summing the absolute values across the width and height dimension , which can be formulated as Gc ( A ) = 1HW ∑i=1 H ∑j=1 W |A· , i , j | and Gs ( A ) = 1 C ∑k=1 C |Ak , · , ·| , where i , j , k denotes the ith , jth , kth slice of A in the height , width , and channel dimension , respectively . Then , the spatial attention mask Ms and the channel attention mask M c used in attentionguided distillation can be obtained by summing the attention maps from the teacher and the student detector , which can be formulated as Ms = HW · softmax ( ( Gs ( AS ) + Gs ( AT ) ) /T ) , M c = C · softmax ( ( Gc ( AS ) + Gc ( AT ) ) /T ) . Note that the superscripts S and T here are used to discriminate students and teachers . T is a hyper-parameter in softmax introduced by Hinton et al . to adjust the distribution of elements in attention masks ( see Figure 4 ) . The attention-guided distillation loss LAGD is composed of two components – attention transfer loss LAT and attention-masked loss LAM . LAT is utilized to encourage the student model to mimic the spatial and channel attention of the teacher model , which can be formulated as LAT = L2 ( Gs ( AS ) , Gs ( AT ) ) + L2 ( Gc ( AS ) , Gc ( AT ) ) . ( 1 ) LAM is utilized to encourage the student to mimic the features of teacher models by a L2 norm loss masked by Ms and M c , which can be formulated as LAM = C∑ k=1 H∑ i=1 W∑ j=1 ( ATk , i , j −ASk , i , j ) 2 ·Msi , j ·M ck 12 . ( 2 )

Summary: This paper explores the knowledge distillation problem in object detection. It claims that the failure of knowledge distillation in object detection is mainly caused by the imbalance between pixels of foreground and background, and the relation distillation between different pixels. The authors then propose non-local distillation to tackle the problem. Extensive experiments are conducted on MS COCO and verify the effectiveness of the proposed method.