Distilling Object Detectors with Feature Richness

1 Introduction . Owe to the widespread use of deep learning , object detection method have developed relatively rapidly . Large-scale deep models have achieved overwhelming success , but the huge computational complexity and storage requirements limit their deployment in real-time applications , such as video surveillance , autonomous vehicles . Therefore , how to find a better balance between accuracy and efficiency has become a key issue . Knowledge Distillation [ 12 ] is a promising solution for the above problem . It is a model compression and acceleration method that can effectively improve the performance of small models under the guidance of the teacher model . For object detection , distilling detectors through imitating all features is inefficient , because the objectirrelevant area would unavoidably introduce a lot of noises . Therefore , how to choose the important features beneficial to distillation remains an unsolved issue . Most of the previous distillation-based detection methods mainly imitate features that overlap with bounding boxes ( i.e . ground truth objects ) , because they believe the features of foreground which can be selected from bounding boxes are ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . important . However , these methods suffer from two limitations . First , foreground features selected from bounding boxes only contain categories in the dataset , but ignore the categories of objects outside the dataset , which leads to the omission of some important features , as shown in Figure 1 ( b ) -bottom . For example , the mannequin category is not included in the COCO dataset , but the person category is included . Since mannequins are visually similar to persons , the features of mannequins contain many useful characteristics of persons which are beneficial for improving the detectability of person for detectors in distillation . Second , only using the prior knowledge of bounding boxes to select features for distillation ignores the defects of the teacher detector . Imitating features that are mistakenly regarded as the background by the teacher detector will mislead and damage the distillation , as shown in Figure 1 ( b ) -top . To handle the above problem , we propose a novel Feature-Richness Score ( FRS ) method to choose important features that are beneficial to distillation . Feature richness refers to the amount of object information contained in the features , meanwhile it can be represented by the probability that these features are objects . Distilling features that have high feature richness instead of features in bounding boxes can effectively solve the above two limitations . First , features of objects whose categories are not included in the dataset have high feature richness . Thus , using feature richness can retrieve the important features outside the bounding boxes , which can guide the student detector to learn the generalized detectability of the teacher detector . For example , features of mannequins that have high feature richness can promote student detector to improve its generalized detectability of persons , as shown in Figure 1 ( c ) -bottom . Second , features in the bounding boxes but are misclassified with teacher detector have low feature richness . Thus , using feature richness can remove the misleading features of the teacher detector in the bounding boxes , as shown in Figure 1 ( c ) -top . Consequently , the importance of features is strongly correlated with the feature richness , namely feature richness is appropriate to choose important features for distillation . Since the classification score aggregating of all categories is an approximation of probability that the features are objects , we use the aggregated classification score as the criterion for feature richness . In practice , we utilize the aggregated classification score corresponding to each FPN level in teacher detector as the feature mask which is used as feature richness map to guide student detector , in both the FPN feature and the subsequent classification head . Compared with the previous methods which use prior knowledge of bounding box information , our method uses aggregated classification score of the feature map as the mask of feature richness , which is related to the objects and teacher detector . Our method offers the following advantages . First of all , the mask in our method is pixel-wise and fine-grained , so we can distill the student detector in a more refined approach to promote effective features and suppress the influence of ineffective features of teacher detector . Besides , our method is more suitable for detector with the FPN module , because our method can generate corresponding feature richness masks for each FPN layer of student detector based on the features extracted from each FPN layer of teacher detector . Finally , our method is a plug-and-play block to any architecture . We implement our approach in multiple popular object detection frameworks , including one-stage , two-stage methods and anchor-free methods . To demonstrate the advantages of the proposed FRS , we evaluate it on the COCO dataset on various framework : Faster-RCNN [ 24 ] , Retinanet [ 18 ] , GFL [ 17 ] and FCOS [ 27 ] . With FRS , we have outperformed state-of-the-art methods on all distillation-based detection frameworks . This achievement shows that the proposed FRS effectively chooses the important features extracted from the teacher . 2 Related Works . 2.1 Object Detection . In recent years , noticeable improvements in accuracy have been made in object detection . The current mainstream object detection algorithms are mainly divided into two-stage detectors [ 24 , 2 , 9 ] and one-stage detectors [ 22 , 17 , 27 , 18 , 23 ] . Two-stage methods such as FasterRCNN [ 24 ] region proposal network and refinement procedure of classification and location . Although detectors fitted with very deep backbone have better detection accuracy , they are expensive in terms of computation cost and hard to deploy to real-time applications . While one-stage methods , such as Retinanet [ 18 ] and FCOS [ 27 ] , maintain an advantage in speed that can directly generate the category probability and position coordinate value of the object . In object detection , the most anchor-based models [ 18 , 24 ] have achieved good results by using predefined anchors , but the predefined anchors will bring a large amount of output and adjustments to the positions and sizes corresponding to different anchors . The anchor-free methods [ 27 , 14 , 6 ] is lighter , getting rid of redundant hyper-parameters by detecting an object bounding box as a pair of key points such as center and distance to boundaries . Although large-scale deep neural networks have achieved great success , the huge computational complexity and storage requirements make it a great challenge to deploy in real-time applications , especially on video surveillance and other resource-limited devices . The model compression and acceleration techniques , such as quantized [ 34 ] and knowledge distillation [ 12 ] , become an important way to solve this problem . 2.2 Knowledge Distillation . Knowledge distillation is an effective method of model compression , which can improve the performance of small models . Hinton et al . [ 12 ] proposed to transfer the dark knowledge from teacher network to student network through the soft outputs which opens the door to the rapid development of knowledge distillation . There are three categories of knowledge for knowledge distillation [ 7 ] . Response-Based methods use the logits of a large deep model as teacher knowledge [ 12 , 13 , 21 , 1 , 33 ] . Feature-Based methods think that the features of intermediate layers also play an important role in guiding the learning of the student detector [ 25 , 31 , 11 ] . And Relation-Based methods believe that the relationship between different activations , neurons or sample pairs contains rich information learned by the teacher model [ 29 , 30 , 15 , 20 ] . Recently , there are several methods which are used to compress object detectors by knowledge distillation . Chen et al . [ 3 ] proposed to distills the student detector through all components that ignore the imbalance in foreground and background , leading to poor results . Wang et al . [ 28 ] pay more attention to the near ground truth area . Li et al . [ 16 ] address the problem by distilling the positive and negative instances in a certain proportion sampled by RPN . Ruoyu et al . [ 26 ] propose to use a two-dimensional Gaussian mask to suppress irrelevant redundant background while emphasizing the target information . Dai et al . [ 5 ] propose a distillation method for detection tasks based on discriminative instances without considering the positive or negative distinguished by ground truth . Guo et al . [ 8 ] point out that the information of features derived from regions excluding objects is also essential for distilling the student detector , which did not explore the background area in more detail . However , we find the feature-rich areas outside the bounding box regions also play an important role in knowledge distillation . It is unreasonable to extract features by simply distinguishing positive and negative samples . 3 Distilling Object Detectors with Feature Richness Score . In previous works , most knowledge distillation methods pay more attention to the features in bounding box areas , while ignoring the important feature outside the bounding box and detrimental feature within the bounding box . To address this issue , we propose the Feature Richness Score as the criterion for selecting distillation areas . As shown in Figure 2 , we use the aggregated classification score to represent the feature richness and then perform distillation on both FPN layers and the classification head . 3.1 Feature Richness Score Module . Given feature f , the probability that these features are objects can be described as : yc = P ( c|f ) , ( 1 ) where c represents an object category whether included in the dataset or not , so yc is the probability that feature f is an object of category c. This conditional probability can be modeled with a deep neural network , which can be expressed as a parameter model θ , described as : yc = P ( c|f , θ ) . ( 2 ) The proposed feature richness is defined as the probability that these features are objects , whether the categories of objects are included in the dataset or not . To make a summary of all categories , we can get the feature richness by aggregating the probability of all categories . Here we use maximum to perform the aggregating : S = max c yc = max c P ( c|f , θ ) . ( 3 ) In the framework of distillation , we use the trained classification head of teacher detector θt as an instantiation of θ . At the same time , since there are objects of various categories , we use the categories c′ ∈ [ 1 , C ] in the dataset as the samples to represent all the objects c , where C is the number of categories in the dataset . Therefore , the feature richness can be approximated as a classification probability of teacher detector : S = max 1≤c′≤C ytc′ = max 1≤c′≤C P ( c′|f t , θt ) , ( 4 ) where yt is the classification probability of teacher detector , namely the classification score map which is the output of classification head from teacher detector , f t is the features fed into the classification head of teacher detector . In practice , we can get the feature richness S by integrating the classification score map output of the teacher detector in the direction of the channel .

Summary: This paper introduces a novel knowledge distillation method for object detection, named Feature-Richness Score(FRS), which makes good use of aggregated classification score of the feature map as the mask of FRS. This pixel-wise FRS mask would be used as the weighted matrix to guide the distillation from teacher to student within selected layers, such as FPN and head. Plenty of well-designed experiments could show the effectiveness and availability of their method on serval mainstream detection framework, also prove it surpasses other recent methods.