Distilling Image Classifiers in Object Detectors

1 Introduction . Object detection plays a critical role in many real-world applications , such as autonomous driving and video surveillance . While deep learning has achieved tremendous success in this task [ 25 , 26 , 31 , 32 , 40 ] , the speed-accuracy trade-off of the resulting models remains a challenge . This is particularly important for real-time prediction on embedded platforms , whose limited memory and computation power impose strict constraints on the deep network architecture . To address this , much progress has recently been made to obtain compact deep networks . Existing methods include pruning [ 1 , 2 , 13 , 22 , 38 ] and quantization [ 7 , 30 , 44 ] , both of which aim to reduce the size of an initial deep architecture , as well as knowledge distillation , whose goal is to exploit a deep teacher network to improve the training of a given compact student one . In this paper , we introduce a knowledge distillation approach for object detection . While early knowledge distillation techniques [ 18 , 33 , 36 ] focused on the task of image classification , several attempts have nonetheless been made for object detection . To this end , existing techniques [ 5 , 12 , 39 ] typically leverage the fact that object detection frameworks consist of three main stages depicted by Figure 1 ( a ) : A backbone to extract features ; a neck to fuse the extracted features ; and heads to predict classes and bounding boxes . Knowledge distillation is then achieved using a teacher with the same architecture as the student but a deeper and wider backbone , such as a Faster RCNN [ 32 ] with ResNet152 [ 14 ] teacher for a Faster RCNN with ResNet50 student , thus facilitating knowledge transfer at all three stages of the frameworks . To the best of our knowledge , [ 43 ] constitutes the only exception to this strategy , demonstrating distillation across different detection frameworks , such as from a RetinaNet [ 25 ] teacher to a RepPoints [ 40 ] student . This method , however , requires the teacher and the student to rely on a similar detection strategy , i.e. , both must be either one-stage detectors or two-stage ones , and , more importantly , still follows a detector-to-detector approach to distillation . ∗The work is done during an internship at NVIDIA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In other words , the study of knowledge distillation remains limited to transfer across architectures tackling the same task . Our classification teacher tackles a different task from the detection student and is trained in a different manner but on the same dataset . Therefore , the classification teacher is capable of providing a different knowledge to the student , for both classification and localization , than that extracted by a detection teacher . In this paper , we investigate the problem of transferring knowledge not only across architectures but also across tasks . In particular , we observed that the classification head of state-of-the-art object detectors still typically yields inferior performance compared to what can be expected from an image classifier . Thus , as depicted by Figure 1 ( b ) , we focus on the scenario where the teacher is an image classifier while the student is an object detector . We then develop distillation strategies to improve both the recognition accuracy and the localization ability of the student . Our contributions can thus be summarized as follows : • We introduce the idea of classifier-to-detector knowledge distillation to improve the performance of a student detector using a classification teacher . • We propose a distillation method to improve the student ’ s classification accuracy , applicable when the student uses either a categorical cross-entropy loss or a binary cross-entropy one . • We develop a distillation strategy to improve the localization performance of the student be exploiting the feature maps from the classification teacher . We demonstrate the effectiveness of our approach on the COCO2017 benchmark [ 23 ] using diverse detectors , including the relatively large two-stage Faster RCNN and single-stage RetinaNet used in previous knowledge distillation works , as well as more compact detectors , such as SSD300 , SSD512 [ 26 ] and Faster RCNNs [ 32 ] with lightweight backbones . Our classifier-to-detector distillation approach outperforms the detector-to-detector distillation ones in the presence of compact students , and helps to further boost the performance of detector-to-detector distillation techniques for larger ones , such as Faster RCNN and RetinaNet with a ResNet50 backbone . Our code is avlaible at : https : //github.com/NVlabs/DICOD . 2 Related work . Object detection is one of the fundamental tasks in computer vision , aiming to localize the objects observed in an image and classify them . Recently , much progress has been made via the development of both one-stage [ 9 , 21 , 26 , 31 , 37 ] and two-stage [ 4 , 15 , 24 , 32 ] deep object detection frameworks , significantly improving the mean average precision ( mAP ) on standard benchmarks [ 10 , 11 , 23 ] . However , the performance of these models typically increases with their size , and so does their inference runtime . This conflicts with their deployment on embedded platforms , such as mobile phones , drones , and autonomous vehicles , which involve computation and memory constraints . While some efforts have been made to design smaller detectors , such as SSD [ 26 ] , YOLO [ 31 ] and detectors with lightweight backbones [ 19 , 35 ] , the performance of these methods does not match that of deeper ones . Knowledge distillation offers the promise to boost the performance of such compact networks by exploiting deeper teacher architectures . Early work in this space focused on the task of image classification . In particular , Hinton et al . [ 18 ] proposed to distill the teacher ’ s class probability distribution into the student , and Romero et al . [ 33 ] encouraged the student ’ s intermediate feature maps to mimic the teacher ’ s ones . These initial works were followed by a rapid growth in the number of knowledge distillation strategies , including methods based on attention maps [ 42 ] , on transferring feature flows defined by the inner product of features [ 41 ] , and on contrastive learning to structure the knowledge distilled from teacher to the student [ 36 ] . Heo et al . [ 17 ] proposed a synergistic distillation strategy aiming to jointly leverage a teacher feature transform , a student feature transform , the distillation feature position and a better distance function . Compared to image classification , object detection poses the challenge of involving both recognition and localization . As such , several works have introduced knowledge distillation methods specifically tailored to this task . This trend was initiated by Chen et al . [ 5 ] , which proposed to distill knowledge from a teacher detector to a student detector in both the backbone and head stages . Then , Wang et al . [ 39 ] proposed to restrict the teacher-student feature imitation to regions around positive anchor boxes ; Dai et al . [ 8 ] produced general instances based on both the teacher ’ s and student ’ s outputs , and distilled feature-based , relation-based and response-based knowledge in these general instances ; Guo et al . [ 12 ] proposed to decouple the intermediate features and classification predictions of the positive and negative regions during knowledge distillation . All the aforementioned knowledge distillation methods require the student and the teacher to follow the same kind of detection framework , and thus typically transfer knowledge between models that only differ in terms of backbone , such as from a RetinaNet-ResNet152 to a RetinaNet-ResNet50 . In [ 43 ] , such a constraint was relaxed via a method able to transfer knowledge across the feature maps of different frameworks . This allowed the authors to leverage the best one-stage , resp . two-stage , teacher model to perform distillation to any one-stage , resp . two-stage , student . This method , however , still assumes that the teacher is a detector . In short , existing knowledge distillation methods for object detection all follow a detector-to-detector transfer strategy . In fact , to the best of our knowledge , distillation has only been studied across two architectures that tackle the same task , may it be image classification , object detection , or even semantic segmentation [ 16 , 27 ] . In this paper , by contrast , we investigate the use of knowledge distillation across tasks and develop strategies to distill the knowledge of an image classification teacher to an object detection student . 3 Our Approach . Our goal is to investigate the transfer of knowledge from an image classifier to an object detector . As illustrated in Figure 1 , this contrasts with existing knowledge distillation techniques for object detection , which typically assume that the teacher and the student both follow a similar three-stage detection pipeline . For our classifier-to-detector knowledge distillation to be effective , we nonetheless need the student and teacher to process the same data and use the same loss for classification . To this end , given a detection dataset Ddet depicting C foreground object categories , we construct a classification dataset Dcls by extracting all objects from Ddet according to their ground-truth bounding boxes and labels . We then train our classification teacher Ft , with parameters θt , on Dcls in a standard classification manner . In the remainder of this section , we introduce our strategies to exploit the resulting teacher to improve both the classification and localization accuracy of the student detector Fs , with parameters θs . 3.1 KDcls : Knowledge Distillation for Classification Our first approach to classifier-to-detector distillation focuses on the classification accuracy of the student network . To this end , we make use of the class-wise probability distributions obtained by the teacher and the student , softened by making use of a temperature parameter T . Below , we first derive our general formulation for distillation for classification , and then discuss in more detail how we obtain the teacher and student class distributions for the two types of classification losses commonly used by object detection frameworks . Formally , given K positive anchor boxes or object proposals , which are assigned with one of the ground-truth labels and bounding boxes during training , let ps , Tk denote the vector of softened class probabilities for box k from the student network , obtained at temperature T , and let pt , Tk denote the corresponding softened probability vector from the teacher network . We express knowledge distillation for classification as a loss function measuring the Kullback-Leibler ( KL ) divergence between the teacher and student softened distributions . This can be written as Lkd−cls = 1 K K∑ k=1 KL ( pt , Tk ∥ p s , T k ) . ( 1 ) The specific way we define the probability vectors ps , Tk and p t , T k then depends on the loss function that the student detector uses for classification . Indeed , existing detectors follow two main trends : some , such as Faster RCNN and SSD , exploit the categorical cross-entropy loss with a softmax , accouting for the C foreground classes and 1 background one ; others , such as RetinaNet , employ a form of binary cross-entropy loss with a sigmoid2 , focusing only on the C foreground classes . Let us now discuss these two cases in more detail . Categorical cross-entropy . In this case , for each positive object bounding box k , the student detector outputs logits zsk ∈ ( C + 1 ) . We then compute the corresponding softened probability for class c with temperature T as ps , Tk ( c|θ s ) = ez s k , c/T∑C+1 j=1 e zsk , j/T , ( 2 ) where zsk , c denote the logit corresponding to class c. By contrast , as our teacher is a C-way classifier , it produces logits ztk ∈ C. We thus compute its softened probability for class c as p̃t , Tk ( c|θ t ) = ez t k , c/T∑C j=1 e ztk , j/T , ( 3 ) and , assuming that all true objects should be classified as background with 0 probability , augment the resulting distribution to account for the background class as pt , T = [ p̃t , T , 0 ] . The KL-divergence between the teacher and student softened distributions for object k can then be written as KL ( pt , Tk ∥ p s , T k ) = T 2 C+1∑ c=1 pt , Tk , c log p t , T k , c − p t , T k , c log p s , T k , c . ( 4 ) Binary cross-entropy . The detectors that rely on the binary cross-entropy output a score between 0 and 1 for each of the C foreground classes , but , together , these scores do not form a valid distribution over the C classes as they do not sum to 1 . To nonetheless use them in a KL-divergence measure between the teacher and student , we rely on the following strategy . Given the student and teacher C-dimensional logit vectors for an object k , we compute softened probabilities as p̃s , Tk ( c|θ s ) = ( 1 + e−z s k , c/T ) −1 , p̃t , Tk ( c|θ t ) = ( 1 + e−z t k , c/T ) −1 . ( 5 ) We then build a 2-class ( False-True ) probability distribution for each category according to the ground-truth label l of object k. Specifically , for each category c , we write ps , Tk , c = [ 1− p̃ s , T k , c , p̃ s , T k , c ] , ( 6 ) 2In essence , the RetinaNet focal loss follows a binary cross-entropy formulation . for the student , and similarly for the teacher . This lets us express the KL-divergence for object k as KL ( pt , Tk ∥ p s , T k ) = T 2 C C∑ c=1 1∑ i=0 pt , Tk , c ( i ) log p t , T k , c ( i ) − p t , T k , c ( i ) log p s , T k , c ( i ) , ( 7 ) where pt , Tk , c ( i ) indicates the i-th element of the 2-class distribution p t , T k , c . 3.2 KDloc : Knowledge Distillation for Localization While , as will be shown by our experiments , knowledge distillation for classification already helps the student detector , it does not aim to improve its localization performance . Nevertheless , localization , or bounding box regression , is critical for the success of a detector and is typically addressed by existing detector-to-detector distillation frameworks [ 5 , 8 ] . To also tackle this in our classifier-to-detector approach , we develop a feature-level distillation strategy , exploiting the intuition that the intermediate features extracted by the classification teacher from a bounding box produced by the student should match those of the ground-truth bounding box . Formally , given an input image I of size w × h , let us denote by Bk = ( x1 , y1 , x2 , y2 ) the top-left and bottom-right corners of the k-th bounding box produced by the student network . Typically , this is achieved by regressing the offset of an anchor box or object proposal . We then make use of a Spatial Transformer [ 20 ] unit to extract the image region corresponding to Bk . It is a non-parametric differentiable module that links the regressed bounding boxes with the classification teacher to yield an end-to-end model during training . Specifically , we compute the transformer matrix Ak = [ ( x2 − x1 ) /w 0 −1 + ( x1 + x2 ) /w 0 ( y2 − y1 ) /h −1 + ( y1 + y2 ) /h ] , ( 8 ) which allows us to extract the predicted object region Opk with a grid sampling size s as Opk = fST ( Ak , I , s ) , ( 9 ) where fST denotes the spatial transformer function . As illustrated in the right portion of Figure 1 ( b ) , we then perform distillation by comparing the teacher ’ s intermediate features within the predicted object region Opk to those within its assigned ground-truth one O gt k . Specifically , for a given layer ℓ , we seek to compare the features Fℓt ( O p k ) and Fℓt ( O gt k ) of the positive box k. To relax the pixel-wise difference between the features , we make use of the adaptive pooling strategy of [ 28 ] , which produces a feature map AP ( Fℓt ( O ) ) of a fixed size M ×W ×H from the features extracted within region O . We therefore write our localization distillation loss as LLkd−loc = 1 KLMHW K∑ k=1 L∑ ℓ=1 1ℓ∥AP ( Fℓt ( O p k ) ) −AP ( F ℓ t ( O gt k ) ) ∥1 , ( 10 ) where K is the number of positive anchor boxes or proposals , L is the number of layers at which we perform distillation , 1l is the indicator function to denote whether the layer ℓ is used or not to distill knowledge , and ∥ · ∥1 denotes the L1 norm . As both the spatial transformer and the adaptive pooling operation are differentiable , this loss can be backpropagated through the student detector . Note that , as a special case , our localization distillation strategy can be employed not only on intermediate feature maps but on the object region itself ( ℓ0 ) , encouraging the student to produce bounding boxes whose underlying image pixels match those of the ground-truth box . This translates to a loss function that does not exploit the teacher and can be expressed as L0kd−loc ( Op , Ogt ) = 1 KMHW K∑ k=1 ∥AP ( Opk ) −AP ( O gt k ) ∥1 . ( 11 ) Depending on the output size of the adaptive pooling operation , this loss function encodes a moreor-less relaxed localization error . As will be shown by our experiments , it can serve as an attractive complement to the standard bounding box regression loss of existing object detectors , whether using distillation or not .

Summary: The paper aims to transfer knowledge from image classifiers to object detectors using distillation. The teacher and student models are trained on the same dataset, and distillation is performed from teacher to student via classification and localization components. The said model improves on the non-distilled baseline student detectors, and also compete with prior related detector-to-detector distillation works. The ablation study demonstrates the effectiveness of the loss terms and also shows the complementary nature of the classification and localization distillation loss terms.