Distilling Image Classifiers in Object Detectors

1 Introduction . Object detection plays a critical role in many real-world applications , such as autonomous driving and video surveillance . While deep learning has achieved tremendous success in this task [ 25 , 26 , 31 , 32 , 40 ] , the speed-accuracy trade-off of the resulting models remains a challenge . This is particularly important for real-time prediction on embedded platforms , whose limited memory and computation power impose strict constraints on the deep network architecture . To address this , much progress has recently been made to obtain compact deep networks . Existing methods include pruning [ 1 , 2 , 13 , 22 , 38 ] and quantization [ 7 , 30 , 44 ] , both of which aim to reduce the size of an initial deep architecture , as well as knowledge distillation , whose goal is to exploit a deep teacher network to improve the training of a given compact student one . In this paper , we introduce a knowledge distillation approach for object detection . While early knowledge distillation techniques [ 18 , 33 , 36 ] focused on the task of image classification , several attempts have nonetheless been made for object detection . To this end , existing techniques [ 5 , 12 , 39 ] typically leverage the fact that object detection frameworks consist of three main stages depicted by Figure 1 ( a ) : A backbone to extract features ; a neck to fuse the extracted features ; and heads to predict classes and bounding boxes . Knowledge distillation is then achieved using a teacher with the same architecture as the student but a deeper and wider backbone , such as a Faster RCNN [ 32 ] with ResNet152 [ 14 ] teacher for a Faster RCNN with ResNet50 student , thus facilitating knowledge transfer at all three stages of the frameworks . To the best of our knowledge , [ 43 ] constitutes the only exception to this strategy , demonstrating distillation across different detection frameworks , such as from a RetinaNet [ 25 ] teacher to a RepPoints [ 40 ] student . This method , however , requires the teacher and the student to rely on a similar detection strategy , i.e. , both must be either one-stage detectors or two-stage ones , and , more importantly , still follows a detector-to-detector approach to distillation . ∗The work is done during an internship at NVIDIA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In other words , the study of knowledge distillation remains limited to transfer across architectures tackling the same task . Our classification teacher tackles a different task from the detection student and is trained in a different manner but on the same dataset . Therefore , the classification teacher is capable of providing a different knowledge to the student , for both classification and localization , than that extracted by a detection teacher . In this paper , we investigate the problem of transferring knowledge not only across architectures but also across tasks . In particular , we observed that the classification head of state-of-the-art object detectors still typically yields inferior performance compared to what can be expected from an image classifier . Thus , as depicted by Figure 1 ( b ) , we focus on the scenario where the teacher is an image classifier while the student is an object detector . We then develop distillation strategies to improve both the recognition accuracy and the localization ability of the student . Our contributions can thus be summarized as follows : • We introduce the idea of classifier-to-detector knowledge distillation to improve the performance of a student detector using a classification teacher . • We propose a distillation method to improve the student ’ s classification accuracy , applicable when the student uses either a categorical cross-entropy loss or a binary cross-entropy one . • We develop a distillation strategy to improve the localization performance of the student be exploiting the feature maps from the classification teacher . We demonstrate the effectiveness of our approach on the COCO2017 benchmark [ 23 ] using diverse detectors , including the relatively large two-stage Faster RCNN and single-stage RetinaNet used in previous knowledge distillation works , as well as more compact detectors , such as SSD300 , SSD512 [ 26 ] and Faster RCNNs [ 32 ] with lightweight backbones . Our classifier-to-detector distillation approach outperforms the detector-to-detector distillation ones in the presence of compact students , and helps to further boost the performance of detector-to-detector distillation techniques for larger ones , such as Faster RCNN and RetinaNet with a ResNet50 backbone . Our code is avlaible at : https : //github.com/NVlabs/DICOD . 2 Related work . Object detection is one of the fundamental tasks in computer vision , aiming to localize the objects observed in an image and classify them . Recently , much progress has been made via the development of both one-stage [ 9 , 21 , 26 , 31 , 37 ] and two-stage [ 4 , 15 , 24 , 32 ] deep object detection frameworks , significantly improving the mean average precision ( mAP ) on standard benchmarks [ 10 , 11 , 23 ] . However , the performance of these models typically increases with their size , and so does their inference runtime . This conflicts with their deployment on embedded platforms , such as mobile phones , drones , and autonomous vehicles , which involve computation and memory constraints . While some efforts have been made to design smaller detectors , such as SSD [ 26 ] , YOLO [ 31 ] and detectors with lightweight backbones [ 19 , 35 ] , the performance of these methods does not match that of deeper ones . Knowledge distillation offers the promise to boost the performance of such compact networks by exploiting deeper teacher architectures . Early work in this space focused on the task of image classification . In particular , Hinton et al . [ 18 ] proposed to distill the teacher ’ s class probability distribution into the student , and Romero et al . [ 33 ] encouraged the student ’ s intermediate feature maps to mimic the teacher ’ s ones . These initial works were followed by a rapid growth in the number of knowledge distillation strategies , including methods based on attention maps [ 42 ] , on transferring feature flows defined by the inner product of features [ 41 ] , and on contrastive learning to structure the knowledge distilled from teacher to the student [ 36 ] . Heo et al . [ 17 ] proposed a synergistic distillation strategy aiming to jointly leverage a teacher feature transform , a student feature transform , the distillation feature position and a better distance function . Compared to image classification , object detection poses the challenge of involving both recognition and localization . As such , several works have introduced knowledge distillation methods specifically tailored to this task . This trend was initiated by Chen et al . [ 5 ] , which proposed to distill knowledge from a teacher detector to a student detector in both the backbone and head stages . Then , Wang et al . [ 39 ] proposed to restrict the teacher-student feature imitation to regions around positive anchor boxes ; Dai et al . [ 8 ] produced general instances based on both the teacher ’ s and student ’ s outputs , and distilled feature-based , relation-based and response-based knowledge in these general instances ; Guo et al . [ 12 ] proposed to decouple the intermediate features and classification predictions of the positive and negative regions during knowledge distillation . All the aforementioned knowledge distillation methods require the student and the teacher to follow the same kind of detection framework , and thus typically transfer knowledge between models that only differ in terms of backbone , such as from a RetinaNet-ResNet152 to a RetinaNet-ResNet50 . In [ 43 ] , such a constraint was relaxed via a method able to transfer knowledge across the feature maps of different frameworks . This allowed the authors to leverage the best one-stage , resp . two-stage , teacher model to perform distillation to any one-stage , resp . two-stage , student . This method , however , still assumes that the teacher is a detector . In short , existing knowledge distillation methods for object detection all follow a detector-to-detector transfer strategy . In fact , to the best of our knowledge , distillation has only been studied across two architectures that tackle the same task , may it be image classification , object detection , or even semantic segmentation [ 16 , 27 ] . In this paper , by contrast , we investigate the use of knowledge distillation across tasks and develop strategies to distill the knowledge of an image classification teacher to an object detection student . 3 Our Approach . Our goal is to investigate the transfer of knowledge from an image classifier to an object detector . As illustrated in Figure 1 , this contrasts with existing knowledge distillation techniques for object detection , which typically assume that the teacher and the student both follow a similar three-stage detection pipeline . For our classifier-to-detector knowledge distillation to be effective , we nonetheless need the student and teacher to process the same data and use the same loss for classification . To this end , given a detection dataset Ddet depicting C foreground object categories , we construct a classification dataset Dcls by extracting all objects from Ddet according to their ground-truth bounding boxes and labels . We then train our classification teacher Ft , with parameters θt , on Dcls in a standard classification manner . In the remainder of this section , we introduce our strategies to exploit the resulting teacher to improve both the classification and localization accuracy of the student detector Fs , with parameters θs . 3.1 KDcls : Knowledge Distillation for Classification Our first approach to classifier-to-detector distillation focuses on the classification accuracy of the student network . To this end , we make use of the class-wise probability distributions obtained by the teacher and the student , softened by making use of a temperature parameter T . Below , we first derive our general formulation for distillation for classification , and then discuss in more detail how we obtain the teacher and student class distributions for the two types of classification losses commonly used by object detection frameworks . Formally , given K positive anchor boxes or object proposals , which are assigned with one of the ground-truth labels and bounding boxes during training , let ps , Tk denote the vector of softened class probabilities for box k from the student network , obtained at temperature T , and let pt , Tk denote the corresponding softened probability vector from the teacher network . We express knowledge distillation for classification as a loss function measuring the Kullback-Leibler ( KL ) divergence between the teacher and student softened distributions . This can be written as Lkd−cls = 1 K K∑ k=1 KL ( pt , Tk ∥ p s , T k ) . ( 1 ) The specific way we define the probability vectors ps , Tk and p t , T k then depends on the loss function that the student detector uses for classification . Indeed , existing detectors follow two main trends : some , such as Faster RCNN and SSD , exploit the categorical cross-entropy loss with a softmax , accouting for the C foreground classes and 1 background one ; others , such as RetinaNet , employ a form of binary cross-entropy loss with a sigmoid2 , focusing only on the C foreground classes . Let us now discuss these two cases in more detail . Categorical cross-entropy . In this case , for each positive object bounding box k , the student detector outputs logits zsk ∈ ( C + 1 ) . We then compute the corresponding softened probability for class c with temperature T as ps , Tk ( c|θ s ) = ez s k , c/T∑C+1 j=1 e zsk , j/T , ( 2 ) where zsk , c denote the logit corresponding to class c. By contrast , as our teacher is a C-way classifier , it produces logits ztk ∈ C. We thus compute its softened probability for class c as p̃t , Tk ( c|θ t ) = ez t k , c/T∑C j=1 e ztk , j/T , ( 3 ) and , assuming that all true objects should be classified as background with 0 probability , augment the resulting distribution to account for the background class as pt , T = [ p̃t , T , 0 ] . The KL-divergence between the teacher and student softened distributions for object k can then be written as KL ( pt , Tk ∥ p s , T k ) = T 2 C+1∑ c=1 pt , Tk , c log p t , T k , c − p t , T k , c log p s , T k , c . ( 4 ) Binary cross-entropy . The detectors that rely on the binary cross-entropy output a score between 0 and 1 for each of the C foreground classes , but , together , these scores do not form a valid distribution over the C classes as they do not sum to 1 . To nonetheless use them in a KL-divergence measure between the teacher and student , we rely on the following strategy . Given the student and teacher C-dimensional logit vectors for an object k , we compute softened probabilities as p̃s , Tk ( c|θ s ) = ( 1 + e−z s k , c/T ) −1 , p̃t , Tk ( c|θ t ) = ( 1 + e−z t k , c/T ) −1 . ( 5 ) We then build a 2-class ( False-True ) probability distribution for each category according to the ground-truth label l of object k. Specifically , for each category c , we write ps , Tk , c = [ 1− p̃ s , T k , c , p̃ s , T k , c ] , ( 6 ) 2In essence , the RetinaNet focal loss follows a binary cross-entropy formulation . for the student , and similarly for the teacher . This lets us express the KL-divergence for object k as KL ( pt , Tk ∥ p s , T k ) = T 2 C C∑ c=1 1∑ i=0 pt , Tk , c ( i ) log p t , T k , c ( i ) − p t , T k , c ( i ) log p s , T k , c ( i ) , ( 7 ) where pt , Tk , c ( i ) indicates the i-th element of the 2-class distribution p t , T k , c . 3.2 KDloc : Knowledge Distillation for Localization While , as will be shown by our experiments , knowledge distillation for classification already helps the student detector , it does not aim to improve its localization performance . Nevertheless , localization , or bounding box regression , is critical for the success of a detector and is typically addressed by existing detector-to-detector distillation frameworks [ 5 , 8 ] . To also tackle this in our classifier-to-detector approach , we develop a feature-level distillation strategy , exploiting the intuition that the intermediate features extracted by the classification teacher from a bounding box produced by the student should match those of the ground-truth bounding box . Formally , given an input image I of size w × h , let us denote by Bk = ( x1 , y1 , x2 , y2 ) the top-left and bottom-right corners of the k-th bounding box produced by the student network . Typically , this is achieved by regressing the offset of an anchor box or object proposal . We then make use of a Spatial Transformer [ 20 ] unit to extract the image region corresponding to Bk . It is a non-parametric differentiable module that links the regressed bounding boxes with the classification teacher to yield an end-to-end model during training . Specifically , we compute the transformer matrix Ak = [ ( x2 − x1 ) /w 0 −1 + ( x1 + x2 ) /w 0 ( y2 − y1 ) /h −1 + ( y1 + y2 ) /h ] , ( 8 ) which allows us to extract the predicted object region Opk with a grid sampling size s as Opk = fST ( Ak , I , s ) , ( 9 ) where fST denotes the spatial transformer function . As illustrated in the right portion of Figure 1 ( b ) , we then perform distillation by comparing the teacher ’ s intermediate features within the predicted object region Opk to those within its assigned ground-truth one O gt k . Specifically , for a given layer ℓ , we seek to compare the features Fℓt ( O p k ) and Fℓt ( O gt k ) of the positive box k. To relax the pixel-wise difference between the features , we make use of the adaptive pooling strategy of [ 28 ] , which produces a feature map AP ( Fℓt ( O ) ) of a fixed size M ×W ×H from the features extracted within region O . We therefore write our localization distillation loss as LLkd−loc = 1 KLMHW K∑ k=1 L∑ ℓ=1 1ℓ∥AP ( Fℓt ( O p k ) ) −AP ( F ℓ t ( O gt k ) ) ∥1 , ( 10 ) where K is the number of positive anchor boxes or proposals , L is the number of layers at which we perform distillation , 1l is the indicator function to denote whether the layer ℓ is used or not to distill knowledge , and ∥ · ∥1 denotes the L1 norm . As both the spatial transformer and the adaptive pooling operation are differentiable , this loss can be backpropagated through the student detector . Note that , as a special case , our localization distillation strategy can be employed not only on intermediate feature maps but on the object region itself ( ℓ0 ) , encouraging the student to produce bounding boxes whose underlying image pixels match those of the ground-truth box . This translates to a loss function that does not exploit the teacher and can be expressed as L0kd−loc ( Op , Ogt ) = 1 KMHW K∑ k=1 ∥AP ( Opk ) −AP ( O gt k ) ∥1 . ( 11 ) Depending on the output size of the adaptive pooling operation , this loss function encodes a moreor-less relaxed localization error . As will be shown by our experiments , it can serve as an attractive complement to the standard bounding box regression loss of existing object detectors , whether using distillation or not .

Summary: This paper designs a new framework for distillation in object detection tasks. It claimed itself as a new method for opening the cross-task distillation directions. However, it is a bit overstated.