Structural Knowledge Distillation for Object Detection
Authors: Philip De Rijk
Lukas Schneider
Marius Cordts
Dariu Gavrila
Venue: NeurIPS 2022 Accept (2022)
Keywords: computer vision
CNNs
knowledge distillation
object detection
deep learning

Abstract: Knowledge Distillation (KD) is a well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student.
KD has proven to be an effective technique to significantly improve the student's performance for various tasks including object detection. 
As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an $\ell_{p}$-norm distance between teacher and student activations during training. 
In this paper, we propose a replacement for the pixel-wise independent $\ell_{p}$-norm based on the structural similarity (SSIM).
By taking into account additional contrast and structural cues, more information within intermediate feature maps can be preserved. 
Extensive experiments on MSCOCO demonstrate the effectiveness of our method across different training schemes and architectures. 
Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard $\ell_p$-norms.
Moreover, more complex state-of-the-art KD methods using attention-based sampling mechanisms are outperformed, including a +3.5 AP gain using a Faster R-CNN R-50 compared to a vanilla model. 

Structural Knowledge Distillation for Object Detection
Philip de Rijk!. Lukas Schneider
Marius Cordts
Dariu M. Gavrila
'TU Delft
Mercedes- AG
Abstract
Knowledge Distillation (KD) is well-known training paradigm in deep neural networks where knowledge acquired by a large teacher model is transferred to a small student. KD has proven to be effective technique to significantly improve the student's performance for various tasks including object detection. As such, KD techniques mostly rely on guidance at the intermediate feature level, which is typically implemented by minimizing an f--norm distance between teacher and student activations during training. In this paper, we propose replacement for the pixel-wise independent 2,-norm based on the structural similarity (SSIM) (28). By taking into account additional contrast and structural cues, feature importance, correlation and spatial dependence in the feature space are considered in the loss formulation. Extensive experiments on MSCOCO [6] demonstrate the effec- tiveness of our method across different training schemes and architectures. Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard f--norms. Moreover, more complex state-of-the-arr KD methods [13 33 using attention-basee sampling mechanisms are outperformed, including +3.5 AP gain using Faster R-CNN R-50 [2I] compared to a vanilla model.
Introduction
Over the last decade, Convolutional Neural Networks (CNNs), have shown to be very effective tool in solving fundamental computer vision tasks [I4]. One major application of CNNs includes real-time perception systems found in e.g. autonomous vehicles, where object detection is often task of major importance. Deployment of CNNs into real-time applications, however, introduces strict limitations memory and latency. On the other hand, increased performance of state-of-the-an detectors typically comes with an increase in memory requirements and inference time [12]. Thus, the choice of network model and its according detection performance is strictly limited. Several techniques have been proposed to tackle this problem, e.g. pruning [8], weight quantization (9).
[6] and Knowledge Distillation (KD) [11]. In this work interested in the latter, as it provides an intuitive way of performance improvement without the need for architectural modifications existing networks.
parameter prediction
particularly
With KD, the knowledge acquired by computationally expensive teacher model is transferred to smaller student model during training. KD has proven to be very effective in tasks such as classification [1], segmentation [19], and in particular has seen considerable progress detection very recently 5 13
33 335, Due to the complexity of the output space of typical detection model, it is necessary to apply KD at the intermediate feature level, as solely relying on output-based based KD has proven ineffective 3 5
7 3 15 25 35) In feature-hased KD), in addition existing objectives, a training objective is introduced which minimizes the error hetween teacher and student activations and is de-facto standard defined by the fy-norm distance between individual feature activations 5 7 13 25 33 35). as shown in fig. Ta The fn-norm however ignores three important pieces of information present in the feature maps: (1) spatial relationships between features, (ii) the correlation
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Rotla Keop
Shades!
0
propeptiio
4TT,
leckmurt
4
Ao
1
/
ST
Tusker
Teacher
UT
OT
Meatrre m98
T'
Nuatur
(a) Previous methods
5
25 33
(b) Our proposed method. We distill relational knowledge in the form of local mean fa, variance o² and furthermore cross-correlation Ss between feature spaces.
35
After sampling features an E-- norm is applied between selected feature activations.
Figure I: Feature-based Knowledge Distillation (KD).
between the teacher and student features and (iii) importance of individual features. We notice recent work has (implicitly) focused on bypassing the latter point through mechanisms that sample feature activations by assuming that object regions are more "knowledge-dense" [3 225 However, as demonstrated by Guo et al. 7 (2021), even distilling exclusively background feature activations can lead to significant performance improvement, therefore it cannot be assumed that solely object regions contain useful knowledge. Sampling mechanisms furthermore introduce additional drawbacks which may limit their broader implementation into real-world applications, e.g. the need for labeled data 7 25
In this work, we propose Structural Knowledge Distillation, which aims to improve the downsides associated with the nn-mrmm central driver for KD methods, rather than designing an ever more sophisticated sampling mechanism. Our key insight is illustrated in fig. Tb The feature space of CNN can be locally decomposed into luminance (mean), contrast (variance) and structure (cross- correlation) components, strategy that has seen successful application in the image domain the form of SSIM (28) The new training objective becomes minimize local differences in mean and variance, and maximize local zero-normalized coossccorrelation between the teacher and student activations. Doing so allows us capture additional knowledge contained in spatial relations and correlations between feature activations of the teacher and additionally the student, rather than directly minimizing the difference in individual activations.
In order to demonstrate the effectiveness of our method we perform extensive experiments using various detection architectures and training schemes. Overall our contribution is as follows:
We propose Structural Knowledge Distillation, which introduces fSSI and variations as replacement of the f--norm for bastree-based KD object detection models. This enables the capture of additional knowledge manifested in local mean, variance and cross-correlation relationships in the feature space of student and teacher networks.
We illustrate through an analysis of the feature space that our method focuses on different areas than fn-norms, and that therefore solely sampling from object regions is suboptimal the entire feature space can contain useful knowledge depending on the activation pattern.
We demonstrate consistent quantitative improvement in detection accuracy for various training settings and model architectures by performing extensive experiments on MSCOCO (16). Our method even performs on par or outperforms carefully tuned state-of-the-ar object sampling mechanisms [13 33 and fundamentally achieves this by only introducing one line of code.

---

2 Related Work
Knowledge Distillation KD aims to transfer knowledge acquired by a cumbersome teacher model to smaller student model. Bucilă et al. ( (2006) demonstrate that the knowledge acquired by large ensemble of models can be transferred to single small model. Hinton et al. [1] (2015) provide a more general solution applied in DNN in which they raise the temperature of the final softmax until the large model produces suitably soft set of targets. Most KD research in the computer vision domain focuses on the classification task [24]] However, as our main interest lies the real-time domain we focus on the more relevant object detection task.
Object Detection Object detection is one of the fundamental computer vision tasks, where speed and accuracy are often two key requirements. Object detectors can be classified into one-stage and two-stage methods, in this work we investigate our approach for both variations. Generally, two-stage detectors allow for higher accuracy at a higher computational cost, and one-stage detectors allow for lower inference times and complexity in exchange for penalty on accuracy [12]. This notion is however highly dependent on the choice of feature extractor and hyperparameter configuration, which is not straightforward procedure. The main archttcccie within the one-stage domain is RetinaNet [7., with extensions including anchor-free modules and Reppoints [37 32]. In the two-stage domain Faster R-CNN 21 is regarded as the most widely used meta-architecture, where iterations include Cascade R- R-CNN 2 Furthermore, regardless of architecture, ResNet [10] back- bones are very commonly used to extract features, which are furthermore fused at multiple scales using e.g. a FPN 17].
Knowledge Distillation for Object Detection Several methods have been proposed that use KD for object detectors, where it has been found that typically guidance the intermediate feature level rather than the output is critical due to the complex nature of the output space in detection As the detection task requires the identification of multiple objects at different locations, a major complexity introduced is the imbalance between foreground and background, which manifests itself in the intermediate features. Typically, the assumption is made that object regions are "knowledge-dense" and background regions less so. As result, recent work has implicitly focused on designing mechanisms which sample object-relevant features to distill
models 3 5
15
25 33
35
knowledge from [5] 13 15 25 355.
Li et al. [15 (2017) mimic the features sampled from the region proposals in a two stage detector. Wang et al. [25] (2019) propose imitation masks which locate knowledge dense feature locations based on the annotated boxes. Dai et al. [5] (202I) propose module which distills based on distance between classification scores. Similarly, Zhixing et al. (35) (202T) use output class probability to determine feature object probability. Recently Kang et al. 13 (202T) proposed method in which
they encode instance annotations in an attention mechanism 23 to locate "knowledge-dens" regions.
Contrary aforementioned methods, Zhang and Ma method in which they aim to both mimic the attention maps (36 as a sampling mechanism, and furthermore distill through non-local modules (2 Regardless of the sampling technique, it has been demonstrated by Guo et al. 7 (202T) that it is not necessarily the case that background features are less important for distillation.
33
1021 propose purely feature-based
Objective Functions The objective in feature-based KD is minimize the error between teacher and student feature spaces during training, typically in addition to existing objectives. The most widely used objective function in feature-based KD is the e--norm with p
5 7 15
25
33 35).
and less commonly p- 1 [3]. The f--norm however, ignores the spatial re relationships between features, the correlation between teacher and student and the importance of individual features, of which the latter has been the main focus of previous work. To take into account spatial dependencies, we need to furthermore compare features locally rather than pointwise. SSIM provides an elegant way to take into account spatial dependencies by making local comparisons of intensity and contrast, rather than just pointwise. It is further able to take into account the relationship between the teacher and the student by integrating nero-normalized cross-correlation. Contrary to alternative image signal quality metrics such as VIF 27,, GMSD [31] and FSIM [34], SSIM is less complex to formulate mathematically and is differentiable, making it suitable an objective function.
3

---

3 Method
3.1 Overview
We start off by defining the general form of feature-based distillation loss. For the purposes of this work, we divide detector into three components: (i) the backbone, used for extracting features, (ii) the neck, for fusing features at different scales (typically a FPN [7]), and (iii) the head, for generating regression and classification scores. For feature- based KD, we select intermediate representations W and S E H.C.W.WW from the teacher and student respectively at the output of the neck. The feature-based distillation loss between and S can subsequently be formulated as:
T E &.CHWW
R
H
W C
feeat Σ
LE Le (v(( v (Tr,b,,,e)
(1)
h=1 w=
where H, W. C, are the height, width, number of channels and number of neck outputs respectively, N, HWC the total number of elements for the r-th output scale. Additionally we define v(-)
as a normalization function which maps the values of T and S to [0, 1], in our case a min-max rescaling layer, and oi-) an optional adaptation layer [3 which matches the dimensionality of T and S, in our case a x 1 convolutional layer. We introduce the shortened notation Le which represents the choice of difference measurement function at a single feature position r, r.h, w, normalized features and including the adaptation layer, i.e. L. = Le (v (ø (Sr.,.w.e)) v we)).. S and 'T to denote normalized and adapted student and normalized teacher activations respectively, egg. S = v (Se,h,a))))
Accordingly,
3.2 Measuring Difference
As ascertained, the de-facto standard choice for L.g is the Ep norm. p 2 penalizes large errors, but is more tolerable to smaller errors. On the other hand, p = 1 does not over porr-penllze large errors, but smaller errors are penalized more harshly. The lp norm in its general form is given by:
lp C = (IS - TPP)
(2)
Clearly such a function is not able to capture spatial relationships between features. order to capture second-order information we need to involve at least two feature positions, therefore change the problem statement from a point-wise wise comparison to local patch-wise comparison. For each such patch, extract three fundamental properties: the mean μr, the variance o², and the cross-correlation aST which captures the relationship between S and T. We follow (28) and compute these quantities using aauusian--weighee patch Fop of size 11 11 and or 1.5. The proposed SSIM framework (88 compares each of the properties, and is therefore composed of three components: luminance 1, contrast and structure s, which are defined follows:
7 2/sHT + C1 113 + 1
20g0 + C2 c=
OST + C3.
(3a)
(3b)
(3c)
oi + o7 +2
SSo Cs
where uss. MoT refer to the mean, ds. oy refer the variance and asT refers the covariance within the patch. Furthermore, to prevent instability C, = (K, L)² C2 = (K,L), C2 = C2/2, where is the dynamic range of the feature map and Ki 0.01, K2 0.03. An important property of eq. (3) is that it assigns more importance to relative changes in / and e due to the quadratic terms in the denominator. Furthermore, 8 direct measurement of the zero-normalized correlation coefficient between S and T. and hence is formulated as the ratio between their covariance and product of standard deviations. As the range of eq. (3) [-1, 1]. combining the three components I, c, results in the following objective:
(4) where the prevalence of each function can be tuned, with a = 3 = 7 1.0 as a default. As our method is purely beaddre-based and therefore independent of the type of head or bounding box labels, we simply add feet to the existing detection objective function Lder (typically Cets and Lreg) using weighting factor X, which results in the following overall training objective:
ESSIM -1 (1-
c -2)))/
L = AL + Laes
(5)

---

Experiments
4.1 Experiment Settings
we assess the performance of ESSIM on the MSCOCO [16 validation dataset. We report mean Average Precision (AP) as the main evaluation metric, and additionally report AP at specified IoU thresholds APso, A7r and object sizes APs, APM, APr. Our central points of comparison are the two most widely used one- and two-stage meta-architectures. RetinaNet (RN) [8] and Faster-RCNI (FRCNN) [21] We use ResNedRes backbones [10 30 for the teachers and R-50 [T] backbones for students, with = 4, 2 respectively. We conduct Nvidia RTX8000 GPU with 48GB of memory. Each model is trained using SGD optimization with momentum 0.9, weight decay le-4 and batch size 8. The learning rate is set at 0.01 (RN) / 0.02 (FRCNN) and decreased tenfold at step 8 and 11, for total of 12 epochs. We additionally implement batch normalization layers after each convolutional layer, and use focal loss [88 with 7 2.0 and ft 0.25. The input images are resized to minimum spatial dimensions 800 while retaining the original ratios, and add padding to both fulfill the stride requirements and retain equal dimensionality across each batch. Finally the images are randomly flipped with 0.5 and normalized.
Following literature 7 25
33 35
experiments in Pytorch 20 using the MMDetection2 4 framework
4.2 Comparison with lp nonomm
In this first set of experiments compare the performance of fn-norms to ESSIM- Table T shows the results of the main experiments comparing the best performance of each Le. It can be observed that: (i) &SSIM outperforms b, norms by significant margin, boosting performance with up +3.7AP. (ii) Adopting any form of feature-basse based distillation results in an improvement over the vanilla network, except in Faster R-CNN [21] (iii) Even though previous work uses lz, & outperforms & with AP improvements of 2.3 vs. 0.4 and 1.2 vs. 0.0 respectively.
Table I: Comparison of objective functions MSCOCO [16].
Backbone
Le
AP
AP30
AP7s
APs
APM
APL
RetinaNet [18]
Teacher R101
41.0
60.3
44.0
24.1
45.3
53.8
Vanilla R50
36.4
55.6
38.7
21.1
40.3
46.6
R50
lz
36.8 (+0.4)
55.7
39.1
20.6
40.5
47.3
R50
e
38.7 (+2.3)
57.6
41.6
22.7
42.7
50.5
R50
ESSIM
40.1 (+3.7)
59.2
43.1
23.1
44.6
53.2
Faster R-CNN R50
21
Teacher X-101
45.6
64.1
49.7
26.2
49.6
60.0
Vanilla R50
37.4
58.
40.4
21.2
41.0
48.
R50
l
37.4 (+0.0)
57.6
40.9
21.2
41.3
48.1
R50
38.6 (+1.2)
58.8
42.1
21.8
42.1
49.9
R-50
ESSIM
40.9 (+3.5)
61.0
44.9
23.7
44.5
53.5
To investigate what this improvement performance can be attributed to, we analyze the distribution of the training stimulus in the feature space. Figure illustrates the comparison between Essim and lp (p = 2) for the magnitude of the loss, averaged over all channels and 12 training epochs in neck r= This tells us something about which regions in the feature space are focused more. It can be observed that with fg. high loss is assigned to object regions in particular, and furthermore regions with high brightness, such as the sky in fig. 2a the window in fig. 2b Exom however assigns the loss differently, where not only object regions are focused, but additionally more diverse background regions targeted, while little importance is given to low-contrast background regions.
One of issues highlighted by regions. As can be seen in fig. 2 the loss applied by fxim is much more distributed over the feature space than l,, which as a direct result causes more distributed application of the gradient in the feature space. As result, is can be observed that the feature map of a l fesi distilled model is much
that losses are higher object regions than in background
5

---

(b)
(c)
(d)
Figure 2: Distribution of the magnitude of the loss in the feature space at output scale r = 1 for various images, averaged over all channels and 12 epochs of training MSCOCO [T6). From left to right: Image, student trained with lg, student trained with ssim A darker color indicates a higher loss, object regions have been highlighted with bounding boxes, and feature maps have been normalized.
similar the teacher than ep distilled model, shown in fig. which directly translates the increase in performance.
(a) Vanilla
(b) Teacher
(c) 82
(d) &
Figure 3: Qualitative comparison a channel sampled randomly from RetinaNet neck output scale 1. Lighter colors indicate higher activation values.
18 intermediate
4.3 Influence Luminance, Contrast and Structure
(a) Luminance (a)
(b) Contrast (3)
Structure (7)
(d) ssim (a,, 3,
Figure 4: Distribution of the channel averaged magnitude of the loss for each individual component
luminance, contrast and stucture, averaged over 12 epochs of training on MSCOCO 16
Next we compare the influence of the luminance, contrast and structure components by tuning o, B and y respectively (refer to cq.44), for this experiment we use the RetinaNet [18] teacher-student pair. The results are shown in 2 The main observation that can be made here is that the influence of structure (7)) is more substantial than the other components, and even its own provides an increase of +3.2 AP. Furthermore, the comparisons of luminance a and contrast 3 alone result in performance comparable or better than nn-nrss (compare to table IT.. fig. 4 we furthermore compare the average magnitude of the loss during training. The luminance provides similar training stimulus to the fp-norm (ref. to fig. 2c). but more "smooth" due to the Gaussian kernel. Contrarily, contrast mostly targets background areas, as it is more sensitive to differences where base contrast is already low. Finally, it can noticed that structure has the most influence over the total loss, both in magnitude and spatial distribution.

---

Table 2: Comparison of objective functions for RetinaNet 3 tunes contrast and y tunes structure.
18 on MSCOCO 16 tunes luminance,
Backbone
a
B
7
AP
APs0
AP73
APs
APM
APL
Teacher ResNet-101
41.0
60.3
44.0
24.1
45.3
53.8
Vanilla ResNer-50
36.4
55.6
38.7
21.1
40.3
46.6
ResNet-50
0
38.7 (+2.3)
57.9
41.6
21.8
42.8
50.8
ResNet-50
0
1
0
38.9 (+2.5)
57.7
41.6
21.7
42.6
51.3
ResNet-50
0
0
39.6 (+3.2)
58.6
42.7
22.5
44.0
52.5
ResNet-50
0
1
1
40.0 (+3.6)
59.0
42.8
22.4
44.4
53.3
ResNet-550
1 40.1 (+3.7)
59.2
43.1
23.1
44.6
53.2
Additionally, fig. 5 we illustrate the differences between student and teacher activations for the differently trained models. It can be observed that the structure objective results in feature space that has converged to a very similar local optimum as the teacher, with few noisy or large differences. Luminance contains more noisy differences, and especially in the last layer demonstrates high differences. Although the contrast objective performs relatively well on its own, it seems to provide different activations the teacher, particularly in the object area, the student network converged to different local minimum.
(1)
(b) r 2
(c) r=)=3
(d) r 4
r
18 Teacher. Subsequent rows illustrate channel averaged differences in activations between teacher and student, distilled with: 2nd row: luminance (a). 3rd row: contrast (3), 4th row: structure (t), represents the output scales
Figure 5: Top row: Channel averaged activations in the RetinaNet R101
of the feature map (eq. 1. Differences have been normalized, where darker color indicates higher value.
4.4 Comparison to State-of-the-Art Methods
Next, compare recent work, for which the following methods serve baselines: Zhang and Ma (33] (2021), a purely fcature-based approach leveraging attention masks (36) and non-local
modules 26 and (ii) Kang et [13] (20TT) who encode labeled instance annotations in an attention
mechanism 2 and report state-or of-te-of-thearr art for distillation methods for RetinaNet [18] and Faster
R-CNN
2 on MSCOCO [16] the time of writing. To further demonstrate the simplicity and
versatility of our method, we use the original code and teachers as the authors to compare to our proposed method in the same experimental setup.
33 use same MMDetection2 [4] framework, while [3 use Detectron2 29 For the comparison with [3 we furthermore adopt inheritance, a practice proposed by the authors which the FPN 17 and head of the student are initialized with teacher parameters. This leads to faster training convergence, but may not be applicable when

---

architectures differ between teacher and student. As the teachers and exact configurations slightly vary, we split up the comparison into two parts, shown in table
Table 3: Comparison to state-of-the-ad methods on MSCOCO 16
denotes inheritance.
RetinaNet
18
Faster R- R-CNN
21
Method
AP
APs
APM
APL
AP
APs
APM
AP2
Teacher
41.0
24.1
45.3
53.8
45.6
26.2
49.6
60.0
Vanilla
36.4
21.1
40.3
46.6
37.4
21.2
41.0
48.1
Zhang and Ma 33
38.5 (+2.1)
21.7
42.6
51.5
38.9 (+1.5)
21.9
42.1
51.5
Ours
40.1 (+3.7)
23.1
44.6
53.2
40.9 (+3.5)
23.7
44.5
53.5
Teacher
40.4
24.0
44.3
52.2
42.0
25.2
45.6
54.6
Vanilla
37.4
23.1
41.6
48.3
37.9
22.4
41.1
49.1
Kang et al. 13
40.7 (+3.3)
24.6
44.9
52.4
40.9 (+3.0)
24.5
44.2
53.3
Ours
1
40.7 (+3.3)
24.0
45.0
53.1
41.0 (+3.1)
23.8
44.5
53.7
can be observed that: (i) the adoption of our &SSIM as the distillation function results in an improvement of +3.7 AP, and outperforms [33] for all box sizes and IoU thresholds. (ii) Both our (SSIM and [13] result in an improvement +3.3 AP over the vanilla network. In particular, our method scores high for APr. while [33 mainly show better performance in the small object APs category. Additionally can be observed that the student is able to outperform the teacher with RetinaNet [18].
4.5
Ablation Studies
Generalizability Detection Architectures and Schedules We perform additional studies on several different detection architectures to demonstrate the generalizability of our method. We evaluate our distillation method on the smaller ResNet- 18 backbone and two alternative one-stage architectures,
(37), which extends RetinaNet [18) with an anchor-free module, and Reppoints [32], which replaces the regular bounding box representation objects by a set of sample points. The results of our experiments are shown in table It can be observed that: (i) For each detection architecture our method significantly improves performance, with +3.5AP for the ResNet18 backbone, +2.3 AP for
(37, +3.3 AP for Reppoints [32]. (ii) In general, our method modular and can significantly improve performance regardless of the detection framework used.
Table 4: Investigation of several popular detection architectures on MSCOCO 16
Model
AP
AP50
AP75
APs
APA A
AP, A
RetinaNet-R100 R10 (Teacher)
41.0
60.3
44.0
24.1
45.3
53.8
RetinaNet-R118 (Vanilla)
32.6
50.6
34.6
17.8
35.2
43.5
RetinaNet-R18 (Ours)
36.1 (+3.5)
54.3
38.6
18.9
39.7
49.2
RetinaNet-R100 (Teacher)
41.0
60.3
44.0
24.1
45.3
53,8
RetinaNet-R50 (Vanilla, 2x)
37.4
56.7
39.6
20.0
40.7
49.7
RetinaNet-R50 (Ours, 2x)
40.6 (+3.2)
59.7
43.7
23.6
44.8
53.9
Seaf-RetinaNeet-X10011
37 (Teacher)
42.4
62.5
45.5
24.6
46.1
55.5
Fsaf- Saff-RetinaNet-R550
37
(Vanilla)
37.4
56.8
39.8
20.4
41.1
48.8
Fsaf-RetinaNet-RS5I
37
(Ours)
39.7 (+2.3)
59.3
42.4
22.0
43.3
52.0
Reppoints X-101
32
(Teacher)
44.2
65.5
47.8
26.2
48.4
58.5
Reppoints-R50
32
(Vanilla)
37.0
56.7
39.7
20.4
41.0
49.0
Reppoints
(Ours)
40.3 (+3.3)
60.3
43.5
22.6
44.4
53.9
Effects an Adaptation Layer Adaptation layers can implemented when channel or spatial dimensions between teacher and student do not match, and have shown to generally improve per- formance in previous methods 25 33). We implement the commonly used convolution to investigate the influence performance with method. We adopt the RetinaNet R101-50

---

seacher-student pair and the CRCNN X101 FRCNN R50 teacher-student pair. The results are shown in 5 We notice that table Sa there no additional benefit of adopting an adaptation layer. while in table thethe the difference is significant, and implementing the adaptation layer is critical. Our method can therefore be used both with and without adaptation. However, when architectures and backbones differ between teacher and student the adaptation layer is highly beneficial.
Table 5: Investigation of the effect of adaptation layers
(a) RetinaNet R101
R50
18
(b) Cascade RCNN X101
2
FRCNN R50 21
Adap. layer
AP
APs
5
AРM M
AP r.
Adap. layer
AP
APs s
AP M
AP,. E.
none
40.1
23.1
44.6
53,2
none 1 1
39.8
22.6
43.4
52.1
1 X
1
40.1
23.1
44.4
53.4
40.9
23.7
44.5
53.5
Varying Patch Size F and Loss Prevalence X We additionally investigate the two remaining main nyperparameters that we introduce in this work, for which we use the RetinaNet R101-50 [ teacher-student pair. The influence of the prevalence of fent tuned by À is shown in fig. 6 where it can be noticed that the choice of À can cause difference of up +0.5 AP, with À 2, providing
the best performance. Additionally the influence of the local patch size F over which we calculate each component of ssim is investigated, the results are shown in fig. 6 can be noticed that the choice of kernel size does not have significant influence over performance.
Error Types Finally we are interested in the type of improvements made by our distillation method, as illustrated in fig. 7 We can particularly observe improvements in the AP,, category (rightmost set of columns), which can also be noticed in table 3 where we perform better in this category see that the structural part ensures convergence towards the teacher, particularly in the deeper layers which responsible for detecting large objects. We furthermore observe that the main performance improvements take place in localization, as seen in the AP7s. APso and APIo (Loc) categories in [7 The remaining categories refer to performance evaluation after removal of class supercategories (Sim), all classifications (Oth) and all background FP's, s, where across the board improvement remains more limited than in the localization categories.
than previous methods. One explanation is illustrated in fig. [d [cc] where
40.2
300
40.0
c
4.3 &
39.0
Small
Meakum
Lange
Figure 6: Performance differ- ence when varying hyperpa- rameters. Top: KD loss preva- Ience A. Bottom: kernel size
Figure 7: RetinaNet R-50 18 AP score for varying box sizes. Hatched areas represent the vanilla model, solid areas repre- sent the performance increase obtained through our distillation method.
F.

---

Conclusion
This paper proposed Sii replacement for the conventional ly-norm a building block for feature-based KD in object detection. By taking into account additional contrast and structural cues, feature importance, correlation and spatial dependence are considered in the loss formulation. Lssim outperforms fp-norms by great margin and is able to reach performance par even surpass stateoof-th--an without the need for carefully designed and complex sampling mechanisms. Our method is simple and can be implemented by replacing one line of code. We propose three main directions for future work: First, using fssm as a building block for future KD methods within the object-centric vision domain. Second, integrating and modifying Éssim to work in other types of (vision) tasks. Finally, an investigation from theoretical point-of-view on the impact convergence trajectories and optimization performance of loss functions, not limited to those presented in this paper, as applied in the feature space of a DNN.
Reproducibility Statement
All experiments are based on publicly available frameworks 4 29) and datasets [16] An example implementation of KD loss between teacher and student features is shown below. Omitting the import and using library such as Kornia [22), change from E2 εSSIM only requires change in line of code.
12 implementation
from torch, nn functional import mse_loss
def kd_loss (studenttt_teee teacher_feats)
# inputs have shape [B, C, H, W]
= aase_lose Selludettffeatta teacher_feats
return kd_ rd_feat_lose,
ssim implementation
from kornia. losses import ssim_los def kd_loss kd_losssstuddntt (estudet_feettee teacher_ teacher_feats);
# inputs have shape [B, C, H, W]
kd_feat_loss = ssim_loss astudent_feate, teacher_feats window_size=11 return kd_feat_loss
Acknowledgements
The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Climate Action within the project "KI Delta Learning" (Förderkenmzzichen 19A19013A). The authors would like to thank the consortium for the successful cooperation.
10

---

References
[1] C. Bucilá, R. Caruana, and A. Niculescu-Mizil. Model compression. SIGKDD, 2006.
[2] 7. Cai and N. Vasconcelos. Cascade R-CNN: high quality object detection and instance segmentation. CVPR, 2018.
[3] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker. Learning efficient object detection models with knowledge distillation, NeurIPS, 2017.
[4] K. Chen, J. Wang. J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, Shi,
W. Ouyang, C. C. Loy, and D. Lin. MMDetection: Open MMLab detection toolbox and benchmark, arXiv preprint 1rXiv:1906.000
2019.
[5] X. Dai, Z. Jiang, Z. Wu, Y. Bao, Z. Wang. S. Liu, and E. Zhou. General Instance Distillation for Object Detection. CVPR, 2021.
[6] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. De Freitas. Predicting parameters in deep learning. NeurIPS, 2013.
[7] J. Guo, K. Han, Y. Wang, H. Wu, X. Chen, C. Xu, and C. Xu. Distilling object detectors via decoupled features. CVPR, 2021.
[8] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural networks. NeurIPS, 2015.
[9] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. ICLR, 2016.
[10] K. He, Zhang, Ren, and Sun, Deep residual learning for image recognition. CVPR, 2016.
[11] G. Hinton, O. Vinyals, and J. Dean. Distilling the Knowledge in Neural Network. NeurlPS Deep Learning and Representation Learning Workshop, 2015.
[12] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, Guadarrama, and others. Speed/accuracy trade-offs for modern convolutional object detectors. CVPR, 2017.
[13] Z. Kang, P. Zhang, X. Zhang, Sun, and N. Zheng. for Object Detection. NeurIPS, 2021.
Knowledge Distillation
[14] A. Krizhevsky, L. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
[15] Q. Li, S. Jin, and J. Yan. Mimicking very efficient network for object detection. CVPR, 2017. 2017.
[16] T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: Common objects in context. ECCV, 2014.
[17] T. Y. Lin, P. Dollár, R. Girshick. K. Hc, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. CVPR, 2017.
[18] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. CVPR, 2017.
[19] Y. Liu, K. Chen, C. Liu, Z. Qin. Z. Luo, and J. Wang. Structured knowledge distillation for semantic segmentation. CVPR, 2019.
[20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vetoto, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I., Fang, J. Bai, and S. Chintala. PyTorch: Imperative Style, Riihh-eerfrrmace Deep Learning Library. NeurIPS, 2019.

---

[21] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards Real-Time Time Object Detection with Region Proposal Networks. NeurIPS, 2015.
[22] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Kornia: an Open Source Differen- tiable Computer Vision Library for PyTorch. WACV, 2020.
[23] A.
Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, Kaiser, and I. Polo- sukhin. Attention is all you need. NeurlPS, 2017.
[24] L.. Wang and K. J. Yoon. Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[25] T. Wang. L. Yuan, X. Zhang, and J. Feng. Distilling object detectors with fine-grained feature imitation. In CVPR, 2019.
[26] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. CVPR, 2018.
[27] Z. Wang and Q. Li. Information content weighting for perceptual image quality assessment. IEEE Transactions on image processing, 2010.
(28) Z. Wang, A. C. Bovik, H. R. Sheikh, and E. Simoncelli. Image quality assessment: from
error visibility to structural similarity. IEEE transactions on image processing, 2004.
[29] Y. Wu, A. Kinillov, F. Massa, Y. Lo, and R. Girshick. Detectron2, 2019. URL https
7/gi //githu..com wwt.////ccbbborrrrrrrreeeeeeeeee
[30] Xie, R. Girshick, Dollár, Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
[31] W. Xue, L. Zhang, X. Mou, and A. C. Bovik. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE transactions on image processing, 2013.
[32] Z. Yang, S. Liu, H. Hu, L.. Wang, and S. Lin. Reppoints: Point set representation for object detection. CVPR, 2019.
[33] L. Zhang and Ma. Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors. ICLR, 2021.
[34] L. Zhang, L. Zhang, X. Mou, and D. Zhang. FSIM: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 2011.
[35] D. Zhixing, R. Zhang, M. Chang, S. Liu, T. Chen, Y. Chen, and others. Distilling Object Detectors with Feature Richness. NeurlPS, 2021.
[36] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. CVPR, 2016.
[37] C. Zhu, Y. He, and M. Savvides. Feature selective anchor-free module for single-shot object detection, CVPR, 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Because of resource limitations/training hardware, and it furthermore not commonplace this domain.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we only use
publicly available data and code: MSCOCO [16), PyTorch (0)] MMDetection2 4
Detectron2 [29]. Kornia [22].
(b) Did you mention the license of the assets? [No] Citing is sufficient according to the authors.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "zWs1jfbOt4", "reviewer": "Reviewer_cG4E", "summary": "Instead of minimizing the feature distance directly, the paper distills student models by minimizing three components: luminance, contrast, and feature structure, which are expressed by the mean, variance, and covariance between feature patches. It is verified on MS-COCO with two detection frameworks, e.g., RetinaNet and Faster R-CNN.  \n", "questions": "- In the feature space, what's the meaning of luminance? \n- If the distillation method doesn't depend on the heads, is it possible to extend it to a more general method that can be used for classification, instance segmentation (Mask R-CNN)?\n- Please update Figure 1 using a high-resolution version.", "limitations": "None.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Writing is pretty well-formulated and well-organized. It’s easy to follow. \n- Interesting idea that proposes three components in feature space.\n- The proposed method doesn't depend on the detection head or other parts. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "voh6Ki47YK1", "reviewer": "Reviewer_VXDw", "summary": "1. In this paper the author proposed SSIM loss for object detection distillation and verified it's improvement for various training settings and model architectures. In this paper, feature map was devided into small patched with gaussian smooth, loss was computed per patch. In each patch, mean, variance and convariance were computed and SSIM loss composed of there weighted product.\n2. This paper showed SSIM loss can give more siganl(especially from background part) in model training and lead to a better optimization. Also, SSIM loss focuses on differentareas than lp -norms", "questions": "1. Give more detailed analysis to help readers understand how SSIM loss help  train better object detection model.\n2. Mention all concepts in related work part.", "limitations": "Yes the author adequately addressed the limitations and potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Propose a simple and effective way to boost the performance of  knowledge distillation, especially on object detection.\n2. Did enough experiments to illustrate that SSIM loss works for different object detection models and teacher-student pair.\n3. Provide Insight why SSIM loss works better than l1/l2 loss and showed which component is the most effective.\n Weaknesses\n1. The paper is not easy to follow for ones who is not familiar with object detection or distillation. For example, the author mentioned \"r = 1\" in line 180 and fig 2,3,5. However author  does not tell it refers to different layer in FPN. \n2. There are some interesting phenomenon, such as SSIM provide higher loss in learning, distillation contribute more on bounding box location accuracy and adaption layer works in some setting but does not work in others. However, the authors failed to provide insights, certainly doesn't provide theory to explain this. Thus, this paper looks more like a tech report rather than a conference paper.\n3. The SSIM loss works directly on feature maps, does not change object detection heads or losses. The methods proposed in the paper should works in all vision tasks such as classification, segmentation or recognition. However this is not mentioned in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tgmF-w1O83u", "reviewer": "Reviewer_XGaw", "summary": "This paper focuses on knowledge distillation for object detection. Authors conclude the limitations of conventional feature based knowledge distrillation from three aspects, and propose novel distillation strategy to solve these challenges. Particularly, it performs knowledge distillation on patch-level features instead of existing point-level features. It estimates mean, variance, and covariance on student and teacher feature maps, and compute three terms to formulate the knowledge distillation learning objective. The experimental results support the analysis in the paper and demonstrate the effectiveness of proposed method.", "questions": "My questions have been written in the weakness part.", "limitations": "No obvious negative social impact.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The analysis of conventional feature based knowledge distillation makes sense, and the modified patch-level knowledge distillation is well motivated and can reflect the internal spatial and structural information.\n2. The proposed new formulation is simple and easy to follow, it has potential to boost general knowledge distillation in computer vision tasks with little extra computation consumption.\n3. The performance improvement is significant. The ablation studies well support that the structural information is crucial in konwledge distillation, and it can enlight new research directions in this area.\n\nWeaknesses:\n1. How will the performance change if replace original knowledge distillation objective in [14] with $l_{ssim}$?\n\nOverall, from my perspective, this paper is well written and the method is simple and generalizable. I did not see significant weakness in this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l1Ejglegn9_", "reviewer": "Reviewer_6uNd", "summary": "This paper proposes Structural Knowledge Distillation (SKD) as a replacement for the pixel-wise independent Lp norm. The proposed method is motivated with SSIM to take into account additional contrast and structural cues to preserve more information. Extensive experiments on MSCOCO demonstrate the effectiveness of this method across different training schemes and architectures. ", "questions": "question:\n1.In line 124, can you explame what normalization function you use to limit the value of feature maps to [0, 1]?\n2.In the field of image classification, most KD methods based on feature can do a better performance with the combination of logits-based KD methods. I notice that your methods totally based on feature map, did or will you do some experiments on combining your distillation method with other distillation method?", "limitations": "1. The explanation and theory analysis of the proposed method is limited.\n2. The compared SOTA methods is not enough.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strength:\n1.First use ssim as the function to decribe the difference between teacher and student feature maps.\n2.Simple method design with good result improvement over the vanilla model.\n3.Nice ablation experiment to show the effectiveness of Structural Knowledge Distillation.\n\nweakness:\n1.There is not enough theory in this article to explain the effectiveness of Structural Knowledge Distillation.\n2.In sec4.4, only two SOTA KD methods in detections used for comparing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept