Distilling Object Detectors with Feature Richness

1 Introduction . Owe to the widespread use of deep learning , object detection method have developed relatively rapidly . Large-scale deep models have achieved overwhelming success , but the huge computational complexity and storage requirements limit their deployment in real-time applications , such as video surveillance , autonomous vehicles . Therefore , how to find a better balance between accuracy and efficiency has become a key issue . Knowledge Distillation [ 12 ] is a promising solution for the above problem . It is a model compression and acceleration method that can effectively improve the performance of small models under the guidance of the teacher model . For object detection , distilling detectors through imitating all features is inefficient , because the objectirrelevant area would unavoidably introduce a lot of noises . Therefore , how to choose the important features beneficial to distillation remains an unsolved issue . Most of the previous distillation-based detection methods mainly imitate features that overlap with bounding boxes ( i.e . ground truth objects ) , because they believe the features of foreground which can be selected from bounding boxes are ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . important . However , these methods suffer from two limitations . First , foreground features selected from bounding boxes only contain categories in the dataset , but ignore the categories of objects outside the dataset , which leads to the omission of some important features , as shown in Figure 1 ( b ) -bottom . For example , the mannequin category is not included in the COCO dataset , but the person category is included . Since mannequins are visually similar to persons , the features of mannequins contain many useful characteristics of persons which are beneficial for improving the detectability of person for detectors in distillation . Second , only using the prior knowledge of bounding boxes to select features for distillation ignores the defects of the teacher detector . Imitating features that are mistakenly regarded as the background by the teacher detector will mislead and damage the distillation , as shown in Figure 1 ( b ) -top . To handle the above problem , we propose a novel Feature-Richness Score ( FRS ) method to choose important features that are beneficial to distillation . Feature richness refers to the amount of object information contained in the features , meanwhile it can be represented by the probability that these features are objects . Distilling features that have high feature richness instead of features in bounding boxes can effectively solve the above two limitations . First , features of objects whose categories are not included in the dataset have high feature richness . Thus , using feature richness can retrieve the important features outside the bounding boxes , which can guide the student detector to learn the generalized detectability of the teacher detector . For example , features of mannequins that have high feature richness can promote student detector to improve its generalized detectability of persons , as shown in Figure 1 ( c ) -bottom . Second , features in the bounding boxes but are misclassified with teacher detector have low feature richness . Thus , using feature richness can remove the misleading features of the teacher detector in the bounding boxes , as shown in Figure 1 ( c ) -top . Consequently , the importance of features is strongly correlated with the feature richness , namely feature richness is appropriate to choose important features for distillation . Since the classification score aggregating of all categories is an approximation of probability that the features are objects , we use the aggregated classification score as the criterion for feature richness . In practice , we utilize the aggregated classification score corresponding to each FPN level in teacher detector as the feature mask which is used as feature richness map to guide student detector , in both the FPN feature and the subsequent classification head . Compared with the previous methods which use prior knowledge of bounding box information , our method uses aggregated classification score of the feature map as the mask of feature richness , which is related to the objects and teacher detector . Our method offers the following advantages . First of all , the mask in our method is pixel-wise and fine-grained , so we can distill the student detector in a more refined approach to promote effective features and suppress the influence of ineffective features of teacher detector . Besides , our method is more suitable for detector with the FPN module , because our method can generate corresponding feature richness masks for each FPN layer of student detector based on the features extracted from each FPN layer of teacher detector . Finally , our method is a plug-and-play block to any architecture . We implement our approach in multiple popular object detection frameworks , including one-stage , two-stage methods and anchor-free methods . To demonstrate the advantages of the proposed FRS , we evaluate it on the COCO dataset on various framework : Faster-RCNN [ 24 ] , Retinanet [ 18 ] , GFL [ 17 ] and FCOS [ 27 ] . With FRS , we have outperformed state-of-the-art methods on all distillation-based detection frameworks . This achievement shows that the proposed FRS effectively chooses the important features extracted from the teacher . 2 Related Works . 2.1 Object Detection . In recent years , noticeable improvements in accuracy have been made in object detection . The current mainstream object detection algorithms are mainly divided into two-stage detectors [ 24 , 2 , 9 ] and one-stage detectors [ 22 , 17 , 27 , 18 , 23 ] . Two-stage methods such as FasterRCNN [ 24 ] region proposal network and refinement procedure of classification and location . Although detectors fitted with very deep backbone have better detection accuracy , they are expensive in terms of computation cost and hard to deploy to real-time applications . While one-stage methods , such as Retinanet [ 18 ] and FCOS [ 27 ] , maintain an advantage in speed that can directly generate the category probability and position coordinate value of the object . In object detection , the most anchor-based models [ 18 , 24 ] have achieved good results by using predefined anchors , but the predefined anchors will bring a large amount of output and adjustments to the positions and sizes corresponding to different anchors . The anchor-free methods [ 27 , 14 , 6 ] is lighter , getting rid of redundant hyper-parameters by detecting an object bounding box as a pair of key points such as center and distance to boundaries . Although large-scale deep neural networks have achieved great success , the huge computational complexity and storage requirements make it a great challenge to deploy in real-time applications , especially on video surveillance and other resource-limited devices . The model compression and acceleration techniques , such as quantized [ 34 ] and knowledge distillation [ 12 ] , become an important way to solve this problem . 2.2 Knowledge Distillation . Knowledge distillation is an effective method of model compression , which can improve the performance of small models . Hinton et al . [ 12 ] proposed to transfer the dark knowledge from teacher network to student network through the soft outputs which opens the door to the rapid development of knowledge distillation . There are three categories of knowledge for knowledge distillation [ 7 ] . Response-Based methods use the logits of a large deep model as teacher knowledge [ 12 , 13 , 21 , 1 , 33 ] . Feature-Based methods think that the features of intermediate layers also play an important role in guiding the learning of the student detector [ 25 , 31 , 11 ] . And Relation-Based methods believe that the relationship between different activations , neurons or sample pairs contains rich information learned by the teacher model [ 29 , 30 , 15 , 20 ] . Recently , there are several methods which are used to compress object detectors by knowledge distillation . Chen et al . [ 3 ] proposed to distills the student detector through all components that ignore the imbalance in foreground and background , leading to poor results . Wang et al . [ 28 ] pay more attention to the near ground truth area . Li et al . [ 16 ] address the problem by distilling the positive and negative instances in a certain proportion sampled by RPN . Ruoyu et al . [ 26 ] propose to use a two-dimensional Gaussian mask to suppress irrelevant redundant background while emphasizing the target information . Dai et al . [ 5 ] propose a distillation method for detection tasks based on discriminative instances without considering the positive or negative distinguished by ground truth . Guo et al . [ 8 ] point out that the information of features derived from regions excluding objects is also essential for distilling the student detector , which did not explore the background area in more detail . However , we find the feature-rich areas outside the bounding box regions also play an important role in knowledge distillation . It is unreasonable to extract features by simply distinguishing positive and negative samples . 3 Distilling Object Detectors with Feature Richness Score . In previous works , most knowledge distillation methods pay more attention to the features in bounding box areas , while ignoring the important feature outside the bounding box and detrimental feature within the bounding box . To address this issue , we propose the Feature Richness Score as the criterion for selecting distillation areas . As shown in Figure 2 , we use the aggregated classification score to represent the feature richness and then perform distillation on both FPN layers and the classification head . 3.1 Feature Richness Score Module . Given feature f , the probability that these features are objects can be described as : yc = P ( c|f ) , ( 1 ) where c represents an object category whether included in the dataset or not , so yc is the probability that feature f is an object of category c. This conditional probability can be modeled with a deep neural network , which can be expressed as a parameter model θ , described as : yc = P ( c|f , θ ) . ( 2 ) The proposed feature richness is defined as the probability that these features are objects , whether the categories of objects are included in the dataset or not . To make a summary of all categories , we can get the feature richness by aggregating the probability of all categories . Here we use maximum to perform the aggregating : S = max c yc = max c P ( c|f , θ ) . ( 3 ) In the framework of distillation , we use the trained classification head of teacher detector θt as an instantiation of θ . At the same time , since there are objects of various categories , we use the categories c′ ∈ [ 1 , C ] in the dataset as the samples to represent all the objects c , where C is the number of categories in the dataset . Therefore , the feature richness can be approximated as a classification probability of teacher detector : S = max 1≤c′≤C ytc′ = max 1≤c′≤C P ( c′|f t , θt ) , ( 4 ) where yt is the classification probability of teacher detector , namely the classification score map which is the output of classification head from teacher detector , f t is the features fed into the classification head of teacher detector . In practice , we can get the feature richness S by integrating the classification score map output of the teacher detector in the direction of the channel .

Summary: One of the challenges in object detection is to train models with reduced computational and memory complexity for deployment on the resource-limited devices. One of the central methods is knowledge distillation which transfers knowledge from large teacher detector into small one with almost preserving the quality of the teacher model. However, existing methods have several issues: i) they ignore feature representation outside bounding boxes which can be helpful (e.g. similar but absent from the class categories objects which detection can improve target class detection)  ii) they use some detrimental representations within bounding boxes (ignore teacher model mistakes). To overcome these issues the paper proposes a novel method (FRS: Feature-Richness Score) to choose important features which can improve detection during distillation process. With experiments on anchor-based/anchor-free and one-stage/two-stage detectors authors demonstrate significant improvement in detection with FRS: FRS outperformes state-of-the-art methods on all distillation-based detection frameworks, Faster-RCNN, Retinanet, GFL and FCOS.