Distilling Image Classifiers in Object Detectors
Year: 2022

Abstract:
Knowledge distillation constitutes a simple yet effective way to improve the performance of a compact student network by exploiting the knowledge of a more powerful teacher. Nevertheless, the knowledge distillation literature remains limited to the scenario where the student and the teacher tackle the same task. Here, we investigate the problem of transferring knowledge not only across architectures but also across tasks. To this end, we study the case of object detection and, instead of following the standard detector-to-detector distillation approach, introduce a classifier-to-detector knowledge transfer framework. In particular, we propose strategies to exploit the classification teacher to improve both the detector’s recognition accuracy and localization performance. Our experiments on several detectors with different backbones demonstrate the effectiveness of our approach, allowing us to outperform the state-of-the-art detector-to-detector distillation methods.

heading: 1 Introduction
text: Object detection plays a critical role in many real-world applications, such as autonomous driving and video surveillance. While deep learning has achieved tremendous success in this task [25, 26, 31, 32, 40], the speed-accuracy trade-off of the resulting models remains a challenge. This is particularly important for real-time prediction on embedded platforms, whose limited memory and computation power impose strict constraints on the deep network architecture.
To address this, much progress has recently been made to obtain compact deep networks. Existing methods include pruning [1, 2, 13, 22, 38] and quantization [7, 30, 44], both of which aim to reduce the size of an initial deep architecture, as well as knowledge distillation, whose goal is to exploit a deep teacher network to improve the training of a given compact student one. In this paper, we introduce a knowledge distillation approach for object detection.
While early knowledge distillation techniques [18, 33, 36] focused on the task of image classification, several attempts have nonetheless been made for object detection. To this end, existing techniques [5, 12, 39] typically leverage the fact that object detection frameworks consist of three main stages depicted by Figure 1(a): A backbone to extract features; a neck to fuse the extracted features; and heads to predict classes and bounding boxes. Knowledge distillation is then achieved using a teacher with the same architecture as the student but a deeper and wider backbone, such as a Faster RCNN [32] with ResNet152 [14] teacher for a Faster RCNN with ResNet50 student, thus facilitating knowledge transfer at all three stages of the frameworks. To the best of our knowledge, [43] constitutes the only exception to this strategy, demonstrating distillation across different detection frameworks, such as from a RetinaNet [25] teacher to a RepPoints [40] student. This method, however, requires the teacher and the student to rely on a similar detection strategy, i.e., both must be either one-stage detectors or two-stage ones, and, more importantly, still follows a detector-to-detector approach to distillation.
∗The work is done during an internship at NVIDIA.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In other words, the study of knowledge distillation remains limited to transfer across architectures tackling the same task. Our classification teacher tackles a different task from the detection student and is trained in a different manner but on the same dataset. Therefore, the classification teacher is capable of providing a different knowledge to the student, for both classification and localization, than that extracted by a detection teacher.
In this paper, we investigate the problem of transferring knowledge not only across architectures but also across tasks. In particular, we observed that the classification head of state-of-the-art object detectors still typically yields inferior performance compared to what can be expected from an image classifier. Thus, as depicted by Figure 1(b), we focus on the scenario where the teacher is an image classifier while the student is an object detector. We then develop distillation strategies to improve both the recognition accuracy and the localization ability of the student.
Our contributions can thus be summarized as follows:
• We introduce the idea of classifier-to-detector knowledge distillation to improve the performance of a student detector using a classification teacher.
• We propose a distillation method to improve the student’s classification accuracy, applicable when the student uses either a categorical cross-entropy loss or a binary cross-entropy one.
• We develop a distillation strategy to improve the localization performance of the student be exploiting the feature maps from the classification teacher.
We demonstrate the effectiveness of our approach on the COCO2017 benchmark [23] using diverse detectors, including the relatively large two-stage Faster RCNN and single-stage RetinaNet used in previous knowledge distillation works, as well as more compact detectors, such as SSD300, SSD512 [26] and Faster RCNNs[32] with lightweight backbones. Our classifier-to-detector distillation approach outperforms the detector-to-detector distillation ones in the presence of compact students, and helps to further boost the performance of detector-to-detector distillation techniques for larger ones, such as Faster RCNN and RetinaNet with a ResNet50 backbone. Our code is avlaible at: https://github.com/NVlabs/DICOD.

heading: 2 Related work
text: Object detection is one of the fundamental tasks in computer vision, aiming to localize the objects observed in an image and classify them. Recently, much progress has been made via the development of both one-stage [9, 21, 26, 31, 37] and two-stage [4, 15, 24, 32] deep object detection frameworks, significantly improving the mean average precision (mAP) on standard benchmarks [10, 11, 23]. However, the performance of these models typically increases with their size, and so does their inference runtime. This conflicts with their deployment on embedded platforms, such as mobile phones, drones, and autonomous vehicles, which involve computation and memory constraints. While some efforts have been made to design smaller detectors, such as SSD [26], YOLO [31] and detectors with lightweight backbones [19, 35], the performance of these methods does not match that of deeper ones.
Knowledge distillation offers the promise to boost the performance of such compact networks by exploiting deeper teacher architectures. Early work in this space focused on the task of image classification. In particular, Hinton et al. [18] proposed to distill the teacher’s class probability distribution into the student, and Romero et al. [33] encouraged the student’s intermediate feature maps to mimic the teacher’s ones. These initial works were followed by a rapid growth in the number of knowledge distillation strategies, including methods based on attention maps [42], on transferring feature flows defined by the inner product of features [41], and on contrastive learning to structure the knowledge distilled from teacher to the student [36]. Heo et al. [17] proposed a synergistic distillation strategy aiming to jointly leverage a teacher feature transform, a student feature transform, the distillation feature position and a better distance function.
Compared to image classification, object detection poses the challenge of involving both recognition and localization. As such, several works have introduced knowledge distillation methods specifically tailored to this task. This trend was initiated by Chen et al. [5], which proposed to distill knowledge from a teacher detector to a student detector in both the backbone and head stages. Then, Wang et al. [39] proposed to restrict the teacher-student feature imitation to regions around positive anchor
boxes; Dai et al. [8] produced general instances based on both the teacher’s and student’s outputs, and distilled feature-based, relation-based and response-based knowledge in these general instances; Guo et al. [12] proposed to decouple the intermediate features and classification predictions of the positive and negative regions during knowledge distillation. All the aforementioned knowledge distillation methods require the student and the teacher to follow the same kind of detection framework, and thus typically transfer knowledge between models that only differ in terms of backbone, such as from a RetinaNet-ResNet152 to a RetinaNet-ResNet50. In [43], such a constraint was relaxed via a method able to transfer knowledge across the feature maps of different frameworks. This allowed the authors to leverage the best one-stage, resp. two-stage, teacher model to perform distillation to any one-stage, resp. two-stage, student. This method, however, still assumes that the teacher is a detector.
In short, existing knowledge distillation methods for object detection all follow a detector-to-detector transfer strategy. In fact, to the best of our knowledge, distillation has only been studied across two architectures that tackle the same task, may it be image classification, object detection, or even semantic segmentation [16, 27]. In this paper, by contrast, we investigate the use of knowledge distillation across tasks and develop strategies to distill the knowledge of an image classification teacher to an object detection student.

heading: 3 Our Approach
text: Our goal is to investigate the transfer of knowledge from an image classifier to an object detector. As illustrated in Figure 1, this contrasts with existing knowledge distillation techniques for object detection, which typically assume that the teacher and the student both follow a similar three-stage detection pipeline. For our classifier-to-detector knowledge distillation to be effective, we nonetheless need the student and teacher to process the same data and use the same loss for classification. To this end, given a detection dataset Ddet depicting C foreground object categories, we construct a classification dataset Dcls by extracting all objects from Ddet according to their ground-truth bounding boxes and labels. We then train our classification teacher Ft, with parameters θt, on Dcls in a standard classification manner. In the remainder of this section, we introduce our strategies to exploit the resulting teacher to improve both the classification and localization accuracy of the student detector Fs, with parameters θs.
3.1 KDcls: Knowledge Distillation for Classification
Our first approach to classifier-to-detector distillation focuses on the classification accuracy of the student network. To this end, we make use of the class-wise probability distributions obtained by the teacher and the student, softened by making use of a temperature parameter T . Below, we first derive our general formulation for distillation for classification, and then discuss in more detail how we obtain the teacher and student class distributions for the two types of classification losses commonly used by object detection frameworks.
Formally, given K positive anchor boxes or object proposals, which are assigned with one of the ground-truth labels and bounding boxes during training, let ps,Tk denote the vector of softened class probabilities for box k from the student network, obtained at temperature T , and let pt,Tk denote the corresponding softened probability vector from the teacher network. We express knowledge distillation for classification as a loss function measuring the Kullback-Leibler (KL) divergence between the teacher and student softened distributions. This can be written as
Lkd−cls = 1
K K∑ k=1 KL(pt,Tk ∥ p s,T k ) . (1)
The specific way we define the probability vectors ps,Tk and p t,T k then depends on the loss function that the student detector uses for classification. Indeed, existing detectors follow two main trends: some, such as Faster RCNN and SSD, exploit the categorical cross-entropy loss with a softmax, accouting for the C foreground classes and 1 background one; others, such as RetinaNet, employ a form of binary cross-entropy loss with a sigmoid2, focusing only on the C foreground classes. Let us now discuss these two cases in more detail.
Categorical cross-entropy. In this case, for each positive object bounding box k, the student detector outputs logits zsk ∈ (C + 1). We then compute the corresponding softened probability for class c with temperature T as
ps,Tk (c|θ s) = ez s k,c/T∑C+1
j=1 e zsk,j/T
, (2)
where zsk,c denote the logit corresponding to class c. By contrast, as our teacher is a C-way classifier, it produces logits ztk ∈ C. We thus compute its softened probability for class c as
p̃t,Tk (c|θ t) = ez t k,c/T∑C
j=1 e ztk,j/T
, (3)
and, assuming that all true objects should be classified as background with 0 probability, augment the resulting distribution to account for the background class as pt,T = [p̃t,T , 0].
The KL-divergence between the teacher and student softened distributions for object k can then be written as
KL(pt,Tk ∥ p s,T k ) = T 2 C+1∑ c=1 pt,Tk,c log p t,T k,c − p t,T k,c log p s,T k,c . (4)
Binary cross-entropy. The detectors that rely on the binary cross-entropy output a score between 0 and 1 for each of the C foreground classes, but, together, these scores do not form a valid distribution over the C classes as they do not sum to 1. To nonetheless use them in a KL-divergence measure between the teacher and student, we rely on the following strategy. Given the student and teacher C-dimensional logit vectors for an object k, we compute softened probabilities as
p̃s,Tk (c|θ s) = (1 + e−z s k,c/T )−1 ,
p̃t,Tk (c|θ t) = (1 + e−z t k,c/T )−1 .
(5)
We then build a 2-class (False-True) probability distribution for each category according to the ground-truth label l of object k. Specifically, for each category c, we write
ps,Tk,c = [1− p̃ s,T k,c , p̃ s,T k,c ], (6)
2In essence, the RetinaNet focal loss follows a binary cross-entropy formulation.
for the student, and similarly for the teacher. This lets us express the KL-divergence for object k as
KL(pt,Tk ∥ p s,T k ) =
T 2
C C∑ c=1 1∑ i=0 pt,Tk,c (i) log p t,T k,c (i)− p t,T k,c (i) log p s,T k,c (i) , (7)
where pt,Tk,c (i) indicates the i-th element of the 2-class distribution p t,T k,c .
3.2 KDloc: Knowledge Distillation for Localization
While, as will be shown by our experiments, knowledge distillation for classification already helps the student detector, it does not aim to improve its localization performance. Nevertheless, localization, or bounding box regression, is critical for the success of a detector and is typically addressed by existing detector-to-detector distillation frameworks [5, 8]. To also tackle this in our classifier-to-detector approach, we develop a feature-level distillation strategy, exploiting the intuition that the intermediate features extracted by the classification teacher from a bounding box produced by the student should match those of the ground-truth bounding box.
Formally, given an input image I of size w × h, let us denote by Bk = (x1, y1, x2, y2) the top-left and bottom-right corners of the k-th bounding box produced by the student network. Typically, this is achieved by regressing the offset of an anchor box or object proposal. We then make use of a Spatial Transformer [20] unit to extract the image region corresponding to Bk. It is a non-parametric differentiable module that links the regressed bounding boxes with the classification teacher to yield an end-to-end model during training. Specifically, we compute the transformer matrix
Ak =
[ (x2 − x1)/w 0 −1 + (x1 + x2)/w
0 (y2 − y1)/h −1 + (y1 + y2)/h
] , (8)
which allows us to extract the predicted object region Opk with a grid sampling size s as
Opk = fST (Ak, I, s) , (9)
where fST denotes the spatial transformer function. As illustrated in the right portion of Figure 1(b), we then perform distillation by comparing the teacher’s intermediate features within the predicted object region Opk to those within its assigned ground-truth one O gt k .
Specifically, for a given layer ℓ, we seek to compare the features Fℓt (O p k) and Fℓt (O gt k ) of the positive box k. To relax the pixel-wise difference between the features, we make use of the adaptive pooling strategy of [28], which produces a feature map AP (Fℓt (O)) of a fixed size M ×W ×H from the features extracted within region O. We therefore write our localization distillation loss as
LLkd−loc = 1
KLMHW K∑ k=1 L∑ ℓ=1 1ℓ∥AP (Fℓt (O p k))−AP (F ℓ t (O gt k ))∥1 , (10)
where K is the number of positive anchor boxes or proposals, L is the number of layers at which we perform distillation, 1l is the indicator function to denote whether the layer ℓ is used or not to distill knowledge, and ∥ · ∥1 denotes the L1 norm. As both the spatial transformer and the adaptive pooling operation are differentiable, this loss can be backpropagated through the student detector.
Note that, as a special case, our localization distillation strategy can be employed not only on intermediate feature maps but on the object region itself (ℓ0), encouraging the student to produce bounding boxes whose underlying image pixels match those of the ground-truth box. This translates to a loss function that does not exploit the teacher and can be expressed as
L0kd−loc(Op, Ogt) = 1
KMHW K∑ k=1 ∥AP (Opk)−AP (O gt k )∥1 . (11)
Depending on the output size of the adaptive pooling operation, this loss function encodes a moreor-less relaxed localization error. As will be shown by our experiments, it can serve as an attractive complement to the standard bounding box regression loss of existing object detectors, whether using distillation or not.

heading: 3.3 Overall Training Loss
text: To train the student detector given the image classification teacher, we then seek to minimize the overall loss L = Ldet + λkcLkd−cls + λklLkd−loc , (12) where Ldet encompasses the standard classification and localization losses used to train the student detector of interest. λkc and λkl are hyper-parameters setting the influence of each loss.

heading: 4 Experiments
text: In this section, we first conduct a full study of our classification and localization distillation methods on several compact detectors, and then compare our classifier-to-detector approach to the state-ofthe-art detector-to-detector ones. Finally, we perform an extensive ablation study of our method and analyze how it improves the class recognition and localization in object detection. All models are trained and evaluated on MS COCO2017 [23], which contains over 118k images for training and 5k images for validation (minival) depicting 80 foreground object classes. Our implementation is based on MMDetection [6] with Pytorch [29]. Otherwise specified, we take the ResNet50 as the classification teacher. We will use the same teacher for all two-stage Faster RCNNs and one-stage RetinaNets in our classifier-to-detector distillation method. We consider this to be an advantage of our method, since it lets us use the same teacher for multiple detectors. To train this classification teacher, we use the losses from Faster RCNN and RetinaNet frameworks jointly. Since SSDs use different data augmentation, we train another ResNet50 classification teacher for them. Additional experimental details on how to train our classification teachers are provided in the supplementary material.

heading: 4.1 Classifier-to-Detector Distillation on Compact Students
text: We first demonstrate the effectiveness of our classifier-to-detector distillation method on compact detectors, namely, SSD300, SSD512 [26] and the two-stage Faster RCNN [32] detector with lightweight backbones, i.e., MobileNetV2 [35] and Quartered-ResNet50 (QR50), obtained by dividing the number of channels by 4 in every layer of ResNet50, reaching a 66.33% top-1 accuracy on ImageNet [34].
Experimental setting. All object detectors are trained in their default settings on Tesla V100 GPUs. The SSDs follows the basic training recipe in MMDetection [6]. The lightweight Faster RCNNs are
trained with a 1× training schedule for 12 epochs. The details for the training settings of each model are provided in the supplementary material. We use a ResNet50 with input resolution 112 × 112 as classification teacher for all student detectors. We report the mean average precision (mAP) and mean average recall (mAR) for intersection over unions (IoUs) in [0.5:0.95], the APs at IoU=0.5 and 0.75, and the APs and ARs for small, medium and large objects.
Results. The results are shown in Table 1. Our classification distillation yields improvements of at least 0.7 mAP for all student detectors. It reaches a 2.6 mAP improvement for Faster RCNN-QR50, which indicates that the classification in this model is much weaker. The classification distillation improves AP50 more than AP75, while the localization distillation improves AP75 more than AP50. As increasing AP75 requires more precise localization, these results indicate that each of our distillation losses plays its expected role. Note that the SSDs benefit more from the localization method than the Faster RCNNs. We conjecture this to be due to the denser, more accurate proposals of the Faster RCNNs compared to the generic anchors of the SSDs. Note also that a Faster RCNNs with a smaller backbone benefits more from our distillation than a larger one.

heading: 4.2 Comparison with Detector-to-detector Distillation
text: We then compare our classifier-to-detector distillation approach with the state-of-the-art detector-todetector ones, such as KD [5], FGFI [39], GID [8] and FKD [43]. Here, in addition to the compact students used in Section 4.1, we also report results on the larger students that are commonly used in the literature, i.e., Faster RCNN and RetinaNet with deeper ResNet50 (R50) backbones.
Experimental setting. Following [43], the Faster RCNN-R50 and RetinaNet-R50 are trained with a 2× schedule for 24 epochs. To illustrate the generality of our approach, we also report the results of our distillation strategy used in conjunction with FKD [43], one of the current best detector-todetector distillation methods. Note that, while preparing this work, we also noticed the concurrent work of [12], whose DeFeat method also follows a detector-to-detector distillation approach, and thus could also be complemented with out strategy.
Table 2: Comparison to detector-to-detector distillation methods on the COCO2017 validation set.
Method mAP APs APm APl Faster RCNN-QR50 23.3 13.1 25.0 30.7
+ FKD [43] 26.1 14.6 27.3 35.0 + Ours 27.2 15.2 29.3 36.2 + Ours + FKD 28.0 15.4 29.8 38.5
SSD512-VGG16 29.4 11.7 34.1 44.9 + FKD [43] 31.2 12.6 37.4 46.2 + Ours 32.1 13.3 36.6 47.9 + Ours + FKD 32.6 13.5 37.6 48.3
Faster RCNN-MV2 31.9 18.5 34.4 41.0 + FKD [43] 33.9 18.3 36.3 45.4 + Ours 32.7 19.0 35.0 42.9 + Ours + FKD 34.2 18.5 36.3 45.9
Faster RCNN-R50 38.4 21.5 42.1 50.3 + KD [5] 38.7 22.0 41.9 51.0 + FGFI [39] 39.1 22.2 42.9 51.1 + GID [8] 40.2 22.7 44.0 53.2 + FKD [43] 41.5 23.5 45.0 55.3 + Ours 38.8 22.5 42.5 50.8 + Ours + FKD 41.9 23.8 45.2 56.0
RetinaNet-R50 37.4 20.0 40.7 49.7 + FGFI [39] 38.6 21.4 42.5 51.5 + GID [8] 39.1 22.8 43.1 52.3 + FKD [43] 39.6 22.7 43.3 52.5 + Ours 37.9 20.5 41.3 50.5 + Ours +FKD 40.7 23.1 44.7 53.8
Results. We report the results in Table 2. For compact student detectors, such as Faster RCNN-QR50 and SSD512, our classifier-todetector distillation surpasses the best detectorto-detector one by 1.1 and 0.9 mAP points, respectively. For student detectors with deeper backbones, our method improves the baseline by 0.8, 0.4 and 0.5 points. Furthermore, using it in conjunction with the FKD detector-todetector distillation method boosts the performance to the state-of-the-art of 28.0, 32.6, 34.2, 41.9 and 40.7 mAP. Overall, these results evidence that our approach is orthogonal to the detector-to-detector distillation methods, allowing us to achieve state-of-the-art performance by itself or by combining it with a detector-todetector distillation strategy.

heading: 4.3 Ablation Study
text: In this section, we investigate the influence of the hyper-parameters and of different classification teachers in our approach. To this end, we use the SSD300 student detector.
Ablation study of KDcls. We first study the effect of the loss weight λkc and the temperature T for classification distillation. As shown in Table 3a, these two hyper-parameters have a mild impact on the results, and we obtain the best results with λkc = 0.4 and T = 2, which were used for all other experiments with SSDs.
We then investigate the impact of different classification teacher networks. To this end, we trained three teacher networks ranging from shallow to deep: ResNet18, ResNet50 and ResNext101-32×8d. We further study the impact of the input size to these teachers on classification distillation, using the three sizes [56× 56, 112× 112, 224× 224]. As shown in Table 3b, even the shallow ResNet18 classification teacher can improve the performance of the student detector by 0.3 points, and the improvement increases by another 0.4 points with the deeper ResNet50 teacher. However, the performance drops with the ResNeXt101 teacher, which is the teacher with the highest top-1 accuracy. This indicates that a deeper teacher is not always helpful, as it might be overconfident to bring much additional information compared to the ground-truth labels. As for the input size, we observe only small variations across the different sizes, and thus use a size of 112 in all other experiments.
Ablation study of KDloc. We then evaluate the influence of the two main hyper-parameters of localization distillation, i.e., the grid sampling size of the spatial transformer and the adaptive pooling size of the feature maps. To this end, we vary the sampling size in [14, 28, 56, 112, 224] and the pooling size in [2× 2, 4× 4, 8× 8, 16× 16]. As shown in Table 4a, our localization distillation method benefits from a larger sampling size, although the improvement saturates after a size of 112. This lets us use the same classification teacher, with input size 112, for both classification and localization distillation. The adaptive pooling size has a milder effect on the performance, as shown in Table 4b, with a size of 8 yielding the best mAP. In our experiments, we adopt either 4 or 8, according to the best performance on the validation set.
We further study the layers to be distilled in our localization distillation. To this end, we extract features from the first convolutional layer ℓ1, and from the following bottleneck block ℓ2 of the ResNet50 teacher. As shown in Table 4c, distilling the knowledge of only the object regions (ℓ0) yields a better mAP than using the ℓ1 features. However, combining the object regions (ℓ0) with the feature maps from ℓ1 improves the results. Adding more layers does not help, which we conjecture to be due to the fact that these layers extract higher-level features that are thus less localized.

heading: 4.4 Analysis
text: To further understand how our classifier-to-detector distillation method affects the quality of the classification and localization, in Table 5, we report the APs obtained with IoUs in [0.5, 0.95] with a step of 0.05. These results highlight that our classification and localization distillation strategies behave differently for different IoU thresholds. Specifically, KDcls yields larger improvements
for smaller IoUs, whereas KDloc is more effective with IoUs larger than 0.75. This indicates that KDloc indeed focuses on precise localization, while KDcls distills category information. The complementarity of both terms is further evidenced by the fact that all APs increase when using both of them jointly.
Detection error analysis. We analyze the different types of detection errors using the tool proposed by Bolya et al. [3] for the baseline SSD300 and the distilled models with our KDcls and KDloc. We focus on the classification and localization errors, which are the main errors in object detection. The details of all error types are provided in the supplementary material. As shown in Figure 2a, KDcls decreases the classification error especially for IoUs smaller than 0.65. By contrast, as shown in Figure 2b, the
effect of KDloc increases with the IoU. This again shows the complementary nature of these terms.
Qualitative analysis. Figure 3 compares the detection results of the baseline model and of our distilled model on a few images. We observe that (i) the bounding box predictions of the distilled model are more precise than those of the baseline; (ii) the distilled model generates higher confidences for the correct predictions and is thus able to detect objects that were missed by the baseline, such as the boat in Figure 3c and the giraffe in Figure 3d.

heading: 5 Conclusion
text: We have introduced a novel approach to knowledge distillation for object detection, replacing the standard detector-to-detector strategy with a classifier-to-detector one. To this end, we have developed a classification distillation loss function and a localization distillation one, allowing us to exploit the classification teacher in two complementary manners. Our approach outperforms the state-of-theart detector-to-detector ones on compact student detectors. While the improvement decreases for larger student networks, our approach can nonetheless boost the performance of detector-to-detector distillation. We have further shown that the same classification teacher could be used for all student detectors if they employ the same data augmentation strategy, thus reducing the burden of training a separate teacher for every student detector. Ultimately, we believe that our work opens the door to
a new approach to distillation beyond object detection: Knowledge should be transferred not only across architectures, but also across tasks.
Broader impact
Knowledge distillation is a simple yet effective method to improve the performance of a compact neural network by exploiting the knowledge of a more powerful teacher model. Our work introduces a general approach to knowledge distillation for object detection to transfer knowledge across architectures and tasks. Our approach enables distilling knowledge from a single classification teacher into different student detectors. As such, our work reduces the need for a separate deep teacher detector for each student networks; therefore, we reduce training resources and memory footprint. As we focus on compact networks, our work could significantly impact applications in resource-constrained environments, such as mobile phones, drones, or autonomous vehicles. We do not foresee any obvious undesirable ethical/social impact at this moment.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Swiss National Science Foundation and by NVIDIA during an internship. We would like to thank Maying Shen at NVIDIA for the valuable discussions and for running some of the experiments. We also thank NVIDIA for providing their excellent experiment platform and computing resources.

Review summaries:
The paper proposes a method for distilling a low-compute object detector. Contrary to prior work, which used a more standard object detector-to-object detector distillation approach (where object detectors were most often restricted to be of the same category), the authors propose to use a classification backbone as teacher. To do this, the classifier is trained on the ground truth bounding boxes, and then used to impose both a classification head distillation, and a localization head distillation. The classification distillation is rather straightforward (using a logit matching approach), while the localization distillation is more involved. A spatial transformer is used on top of the bounding box prediction, and a feature matching loss is imposed so that the predicted bbox -> transfomer -> adaptive pooling branch, and the ground truth bbox -> classifier teacher -> adaptive pooling branch are similar to each other.

This paper designs a new framework for distillation in object detection tasks. It claimed itself as a new method for opening the cross-task distillation directions. However, it is a bit overstated.

The paper aims to transfer knowledge from image classifiers to object detectors using distillation. The teacher and student models are trained on the same dataset, and distillation is performed from teacher to student via classification and localization components. The said model improves on the non-distilled baseline student detectors, and also compete with prior related detector-to-detector distillation works. The ablation study demonstrates the effectiveness of the loss terms and also shows the complementary nature of the classification and localization distillation loss terms.