Instance-Conditional Knowledge Distillation for Object Detection
Authors: Zijian Kang
Peizhen Zhang
Xiangyu Zhang
Jian Sun
Nanning Zheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Knowledge Distillation
Object Detection
Instance Segmentation

Abstract: Knowledge distillation has shown great success in classification, however, it is still challenging for detection. In a typical image for detection, representations from different locations may have different contributions to detection targets, making the distillation hard to balance. 
In this paper, we propose a conditional distillation framework to distill the desired knowledge, namely knowledge that is beneficial in terms of both classification and localization for every instance. 
The framework introduces a learnable conditional decoding module, which retrieves information given each target instance as query. Specifically, we encode the condition information as query and use the teacher's representations as key. The attention between query and key is used to measure the contribution of different features, guided by a localization-recognition-sensitive auxiliary task. 
Extensive experiments demonstrate the efficacy of our method: we observe impressive improvements under various settings. Notably, we boost RetinaNet with ResNet-50 backbone from $37.4$ to $40.7$ mAP ($+3.3$) under $1\times$ schedule, that even surpasses the teacher ($40.4$ mAP) with ResNet-101 backbone under $3\times$ schedule. Code has been released on https://github.com/megvii-research/ICD.

Instance-Conddiiional Knowledge Distillation for Object Detection
Zijian Kang" Xi'an Jiaotong University kzj 1230stu. xjtu. edu.
Peizhen Zhang" MEGVII Technology
Xiangyu Zhang MEGVII Technology
zhangpe1zhenn@mggvii. com
zhangxiangyuumeevii.
Jian Sun
Nanning Zheng
MEGVII Technology sunjian@megvii. com
Xi an Jiaotong University
nnzhengGmail xjtu. edu. cn
Abstract
Knowledge distillation has shown great success in classification, however, is still challenging for detection. In typical image for detection, representations from different locations may have different contributions to detection targets, making the distillation hard to balance. In this paper, propose conditional distillation framework to distill the desired knowledge, namely knowledge that is beneficial in terms of both classification and localization for every instance. The framework introduces a learnable conditional decoding module, which retrieves information given each target instance query. Specifically, encode the condition informa- tion as query and use the teacher's representations as key. The attention between query and key is used to measure the contribution of different features, guided by a ietirattoo-rcoognitton--eee
auxiliary task. Extensive experiments demon- strate the efficacy of our method: we observe impressive improvements under various settings. Notably, we boost RetinaNet with ResNet-50 backbone from 37.4 40.7 mAP (+3333) under 1x schedule, that even surpasses the teacher (40.4 4 mAP) with ResNet-101 backbone under x schedule. Code has been released on http://ggttubbcommm//viirreeeeet
Introduction
Deep learning applications blossom in recent years with the breakthrough of Deep Neural Networks (DNNs) 217124 In pursuit of high performance, advanced DNNs usually stack tons of blocks with millions of parameters, which are computation and memory consuming. The heavy design hinders the deployment of many practical downstream applications like object detection in limtted devices. Plenty of techniques have been proposed to address this issue, like network pruning [15] 27
18
quantization 222 23 35), mobile architecture design [38 39 and knowledge distillation (KD)
19 43) without introducing extra inference-time burden or modifications.
431 Among them, KD is one of the most popular choices, since it can boost target network
KD is popularized by Hinton et al. [9 where knowledge of strong pretrained teacher network is transferred to small target student network in the classification scenario. Many good works emerge following the classification track [50] 37 32) However, most methods for classification perform badly in the detection: only slight improvements are observed [28] 51 This can be attributed to two reasons: (1) Other than category classification, another challenging goal to localize the object is seldomly considered. (2) Multiple target objects presented in image for detection, where objects
Equual contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Teachor
Logits
Teacher
Festure Mag
Twncher
Fostue Saczling &c KD: (1 baned
Quallars Dacoding Makido
Lakaer Haag)
KL Davngeee
Festure bosed
131
Stali
Distillation
Gratirre Knowlety Retrosal
i--
Candaooe Knowlety Distiation
Dessstasiod
Studert
Logits
Stadent
Feature Map
Strdent
(a) KD 19
(b) Conventional detection KD.
(c) Ours: Instance-condition KD.
Figure I: Compare with different methods for knowledge distillation. (a) KD [9] for classification transfers logits. (b) Recent methods for detection KD distill intermediate features, different region- based sampling methods are proposed. (c) Our method explicitly distill the desired knowledge.
can distribute in different locations. Due to these reasons, the knowledge becomes rather ambiguous and imbalance in detection: representations from different positions like foreground or background, borders or centers, could have different contributions, which makes distillation challenging.
To handle the above challenge, two strategies are usually adopted by previous methods in detection. First, the distillation is usually conducted among intermediate representations, which cover all necessary features for both classification and localization. Second, different feature selection methods are proposed to overcome the imbalance issue. Existent works could be divided into three types according to the feature selection paradigm: proposal-based, rule-based and attention -baee.. In proposal-based methods [28 11 proposal regions predicted by the RPN [56] or detector are sampled for distillation. In rule-based methods [14
regions selected by predesigned rules like foreground or label-assigned regions are sampled. Despite their improvements, limitations still exist due to the hand-crafted designs, e.g., many methods neglect the informative context regions or involve meticulous decisions. Recently, Zhang el al. (51) propose to use attention 43 type of intermediate activation the network, to guide the distillation. Although attention provides inherent hints for discriminative areas, the relation between activation and knowledge for detection is still unclear. To further improve KD quality, we hope to provide an explicit solution to connect the desired knowledge with feature selection.
45
Towards this goal, we present Instance-Conditiona knowledge Distillation (ICD), which introduces a new KD framework based-on conditional knowledge retrieval. In ICD, we propose to use decoding network to find and distill knowledge associated with different instances, we deem such knowledge as iestaace-conditionn
knowledge. Fig. [1 shows the framework and compares it with former ones, ICD learns to find desired knowledge, which is much more flexible than previous methods, and more consistent with detection targets. In detail, we design conditional decoding module to locate knowledge, the correlation between knowledge and each instance is measured by the instance-awane attention via the transformer decoder [5] 443. In which human observed instances are projected to query and the correlation is measured by scaled-produs attention between query and teacher's representations. Following this formulation, the distillation is conducted over features decomposed by the decoder and weighted by the instance-aware attention. Last but not least, to optimize the decoding module, we also introduce auxiliary task, which teaches the decoder to find useful information for identification and localization. The task defines the goal for knowledge retrieval, facilitates the decoder instead of the student. Overall, our contribution is summarized in three-fold:
We introduce novel framework to locate useful knowledge in detection KD, we formulate the knowledge retrieval explicitly by a decoding network and optimize it via an auxiliary task.
We adopt the conditional modeling paradigm facilitate instance-wise knowledge transferring. We encode human observed instances as query and decompose teacher's 's representations to key and value locate fine- graing knowledge. To our knowledge, is the first trial to explore instance-oriented knowledge for detection.
We perform comprehensive experiments on challenging benchmarks. Results demonstrate impres- sive improvements over various detectors with up to 4 AP gain in MS-COCO, including recent detectors for instance segmentation [41 46 16] some cases, students with x schedule are even able to outperform their teachers with larger backbones trained 3x longer.

---

2 Related Works
2.1 Knowledge Distillation
Knowledge distillation aims to transfer knowledge from a strong teacher to a weaker student network to facilitate supervised learning. The teacher is usually a large pretrained network, who provides smoother supervision and more hints on visual concepts, that improves the training quality and convergence speed 49
[9]. KD for image classification has been studied for years, they are usually categorized into three types 13: response-based [19, feature-based 37 and relation-based [32
Among them, Seature-based distillation over multi-scale features is adopted from most of detection KD works, to deal with knowledge among multiple instances in different regions. Most of these works can be formulated as region selection for distillation, where foreground-bace ba---rrokkddd unbalancing is considered as key problem in some studies 45 [4]. Under this paradigm, we divide them into three kinds: (1) proposal-based. (2) rule-based and (3) attention-base based..
(I) Proposal-based methods rely the prediction of the RPN detection network to find foreground regions, e.g., Chen et al. [6] and Li et al. (28) propose to distilling regions predicted by RPN (36). Dai et al. 11 proposes GI scores to locate controversial predictions for distillation.
(2) Rule-based methods rely on designed rules that can be inflexible and hyper-paramene
inefficient, egg. Wang et al. [45] distill assigned regions where anchor and ground-trute -ruth have a large IoU, Guo et al. [4] distill foreground and background regions separately with different factors.
(3) Attention- based methods rely on activations to locate discriminative areas, yet they do not direct to knowledge that the student needs. Only recent work from Zhang et al. [51] considers attention, they build the spatial-channel-wiss attention to weigh the distillation.
To overcome the above limitations, we explore the instance-condittonaa knowledge retrieval formu- lated by a decoder to explicitly search for useful knowledge. Like other methods, ICD does not have extra cost during inference or use extra data (besides existing labels and pretrained teacher).
2.2 Conditional Computation
Conditional computation is widely adopted to infer contents on a given condition. Our study mostly focuses on how to identify visual contents given an instance condition. This is usually formulated as query an instance on the image, e.g., visual question answer [1] [ and image-text matching [26] queries information and regions specified by natural language. Besides query by language, other types of query are proposed in recent years. For example, DETR (5] queries on fixed proposal embeddings, suueelss learning. These works adopt
Chen et al. [8] encodes points as queries to facilitate transformer decoder to infer upon global receptive fields that cover all visual contents, yet they usually rely on cascaded decoders that are costly for training. From another perspective, CondInst [4] and SOLOv2 [46] generate queries based on network predictions and achieves great performance on instance segmentation. Different from them, this work adopts the query-baser approach retrieve knowledge and build query based on annotated instances.
2.3 Object Detection
Object detection has been developed rapidly. Modern detectors are roughly divided into two-stage or one-stage detectors. Two-stage detectors usually adopt Region Proposal Network (RPN) to generate initial rough predictions and refine them with detection heads, the typical example is Faster R-CNN 36) On the contrary, one-stage detectors directly predict on the feature map, which are usually faster, they include RetinaNet [30], FCOS 42.. Besides of this rough division, many extensions are introduced in recent years, e.g., extension to instance segmentation [1] 46 [6]. anchor-free free models
42 25 and end-to-end detection 5 44 20 A Among these works, multi-scale features are usually
adopted to enhance performance, e.g., FPN [29], which is considered typical case for our study.
To generalize to various methods, the proposed method distills the intermediate features and does not rely on detector-specific designs.
3

---

3 Method
3.1 Overview
As discussed in former studies [4, 5T, useful knowledge for detection distributes unevenly in intermediate features. To facilitate KD, we propose to transfer cantaice-oonditionall knowledge between student and teacher network, termed iif and k?? corresponding to the in. instance:
Caistsul
Cole R
(1)
The knowledge towards teacher's representations and condition yi is formulated as K? G(T, yi) (xef similarly), where the ristance-conditioall decoding module, optimized by an auxiliary loss illustrated in Sec. 3.3 The overall framework is shown in Fig. 2
In the following sections, will introduce the instance-condittionl knowledge (Sec. describe the auxiliary task design (Sec. 3.3 and discuss the knowledge transferring (Sec. 3.4
3.2 Instance-conditional Knowledge
Conditional
Dacuding Module Lo
In this section, elaborate the instance- conditional decoding module G, which com- putes instance-condittonal knowledge x. from (I) unconditional knowledge given (2) instance conditions.
A
VEF
Iestincee ownditinns Quercy
(1) The unconditional knowledge T, symbolizes all available information from the teacher detec- tor. Since modern detectors commonly involve feature pyramid network (FPN) 29 to
Instaace coocitiooal Knowloige
Aucilury Tisk for Optimiizztioon
MSE Loss
tract rich multi-scale representations, we present multi-scale representations as T (X, E DDHH,xW,, IEP,, where P signifies the spatial resolutions while is the channel dimension. different
Distilation Mudule
Figure 2: We propose decoding module to re- trieve knowledge via query-based attention, where instance annotations auxiliary task is proposed to optimize the decod- ing module and the feature-based distillation loss weighted by the attention is used to update student.
By concatenating representations scales along the spatial dimension, we obtain AT E L where T, Epep HpW, is the sum of total pixels number across scales.
encoded query. An
(2) The instance condition, originally describing human-observed object, is denoted by y (y: J where N is the object number and y; = (c,, b:) the annotation for the i-th instance, including category e, and box location b, = (x4, Di, wi, h.) which specifies the localization and size information.
To produce learnable embeddings for each instance, the annotation is mapped to a query feature vector qi in the hidden space, which specifies condition to collect desired knowledge:
q = (E(y,))) q
(2)
where E(-) is a instance encoding function (detailed in Sec. and Fg a Multi- Perception network (MLP).
We retrieve knowledge from T given qi by measuring responses of correlation. This is formulated by dot-product attention 43 with M concurrent heads in a query-key attention manner. In which each w.n. the key, query and value construction.
head j corresponds three linear layers
The key feature KT is computed by projecting teacher's representations A with the positional embeddings [43 5 P E IR Lxd as in Eq. 3 where Fpe denotes a linear projection over position
embeddings. The value feature VJ and query qı is projected by linear mappings to a sub-space with d = D/M channels, Fy on AT and F3 over qi respectively, shown in Eq. At last, an

---

Ercoding Cencration:
Ar there
or
Given Given quar & predict (L. (,,
Arnciased) Consreled
Raw infommaiion
Enoodinng Vector
Jtrroon
Larse
Fak Car
paztimm
Bepraston
E(
Cinggey Fulei]
= concat
Puskie Fui6,a))
Hurenn
(uCh Scat PuiCtegitnd)
y
Fe: Onachet Fanciion Pyo wwstreeeeiinggeeeeee
Encoding Vector
Query Cenaration:
e(
(a) Encoding for instance conditions.
(b) Identification task.
(c) Localization task.
The illustration of the auxiliary task. (a) The instance encoding function encodes instance condition vector, it is then projected as query features. (b) The identification task learns to identify the existence the queried instance. (c) The localization task learns to predict the boundary given an uncertain position provided by the query.
Figure 3:
instance-aware attention mask maj of the i-th instance by the j-th head is obtained by normalized dot-product between K3 and qty
KJ- = FATAT + (P)), Kÿ ER
(3)
vī Fj(AT), vī RR
(4)
O6 = (q)), 4s E R
(5)
KJ qt Vd
m:j = softmar
), mij E R
Intuitively, the querying along the key features and value features describes the correlation be- tween representations and instances. We collect n.? {(mij. V] )= the instance-conditional
knowledge from T, which encodes knowledge corresponds to the äth instance.
3.3 Auxiliary Task
In this section, we introduce the auxiliary task to optimize the decoding module G. First, we aggregate instance-level information to identify and localize objects. This could be obtained by aggregating the instance-conditionn knowledge by the function Fagg. which includes cum-product aggregation over attention mi; and V/. concatenate features from each head, add residuals and project with feed-forward network as proposed |3.5k: 5]:
g ( qd) g. R
To let the instance-level aggregated information g. retain sufficient instance cues, one could design an instance-sensitive task to optimize it as below:
Laux (g. H(y:))
8
where encodes the instance information as targets. However, directly adopt Eq. might lead to trivial solution, since is accessible from both g! (through q: see Eq. 2) and H(y:). It is possible that parameters will learn shortcut, that ignore the teacher representations T. To resolve this issue, we propose to drop information on encoding function E(.), to force the aggregation function Fagy to excavate hints from? T.
The information dropping is adopted by replacing the accurate annotation for instance conditions uncertain ones. For bounding box annotations, we relieve them to rough box centers with rough scales indicators. The rough box center (a%, s/4 is obtained by random jittering as shown below:
{
x = σ + DyWi //
0yh4,
where (w;, h4) is the width and height of the bounding box and Φr, by are sampled from uniform distribution & U[-a, a), where set a=0.3 empirically. The scales indicators are obtained by
5

---

rounding the box sizes in the logarithmic space, i.e., [log2 (w;),, [ogg(h,)) In addition, to let the decoder learn identify instances and be aware of the uncertainty. we generate fake instances for the identification task according to dataset distributions, this is detailed in Appendix A. As a result, it collect coarse information and obtain instance encoding through E() as depicted in Fig. Bal where O; is the category, For is the one hot vectorization, concat is the concatenation operator and pe is the position embedding function.
Finally, the aggregated representation g are optimized by the auxiliary task. We introduces two predictors denoted by Febs and Freg respectively to predict identification and localization results. We adopt binary cross entropy loss (BCE) to optimize the real-fake identification and L1 loss to optimize the regression.
Laux LBCE(Fobj(B) Soh)(y))) + L1 (Freg(g.) (reg(yi))
(10)
where 6-) is an indicator, yields if ys is real and 0 otherwise. Following common practice, the localization loss for fake examples is ignored. öreg is the preparing function for regression targets, Appendix provides more implementation details.
following 42
3.4 Conditoa- Distillation
Lastly, we present the formulation for conditional knowledge distillation, We obtain the projected value features V E IR. Lxd of the student representations analogous to Eq. 4 in Sec. 3.2 By adopting
the instance-aware attention mask as measurement of correlations between Teature and each instance, we formulate the distillation loss as value features mimicking guided by the attention:
M
N
Edistill
dor (y.)- < maj, CMSE(V vJ) >
(1I)
MN,
where N, Eim dat (y+). (N, S N) the real instances number, LMSE VJ) R2 is the pixel-wise mean-square error along the hidden dimensiorf and < <, >> is the Dirac notation for inner product. For stability, the learnable variable m,j and V are detached during distillation. Combine with the supervised learning loss Lett, the overall loss with a coefficient À is summarized below:
Lioral Ldset + Louz + ALdisttill
(12)
is worth noticing, only the gradients w.r.t Edistill and Latt back-propagate to the student network (from representations S). The gradients of Laux only update the instance-conditional decoding function G and auxiliary task related modules.
4
Experiments
4.1 Experiment Settings
We conduct experiments Pytorch (34 with the widely used Detectron2 library [47 and AdelaiDet library [40]. All experiments are rumning on eight 2080ti GPUs with 2 images in each. We adopt the 1x scheduler, which denotes 9k iterations of training. following the standard protocols in Detectron2 unless otherwise specified. Scale jittering with random flip is adopted as data augmentation.
For distillation, the respectively. To optimize the transformer decoder, we adopt AdamW optimizer (33] for the decoder and MLPs following common settings for transformer [3 5.
À is set to 8 for one-stage detectors and 3 for two-stage detectors
Corresponding hyper-parameters follows [5), where initial learning rate and weight decay are set to le-4. We adopt the 256 hidden dimension for our decoder and all MLPs, the decoder has 8 heads in parallel. The projection layer Fy is a 3 layer MLP, Freg and Faty share another layer MLP. In addition, we notice some newly initialized modules of the student share the same size of the teacher, e.g., the detection head. FPN.
2 (MME(() conducted over normalized features with parameter-free LayerNorm 3 for stabilization.
Alll libraries are open-sources and public available, please refer to citations for more details.

---

Table 1: Comparison with previous methods challenging benchmark MS-COCO. The method proposed by et al. 28 does not apply to RetinaNet. denotes the inheriting strategy.
Faster R-CNN
[36]
RetinaNet
30
Method
AP
APs
APM
AP1.
AP
APs
APM
APt.
Teacher w. ResNet-101 101 (3x)
42.0
25.2
45.6
54.6
40.4
24.0
44.3
52.2
Student ResNet-50 (1x)
37.9
22.4
41.1
49.1
37.4
23.1
41.6
48.3
FitNet [37]
(+1.4)
22.7
42.3
51.7
38.2 (+0.8)
21.8
42.6
48.8
Li et (28
(+1.5)
23.3
43.0
51.4
Wang al. 45
(+1.3)
23.2
42.8
50.4
38.4 (+1.0)
23.3
42.6
49.1
Zhang et al. [1
(42.1)
23.2
43.3
52.5
39.3 (+1.9)
23.4
43.6
50.6
Ours
40.4
23.4
44.0
52.0
39.9 (+2.5)
25.0
43.9
51.0
Ours f
40.9 (+3.0)
24.5
44.2
53.5
40.7 (+3.3)
24.2
45.0
52.7
Table 2: Experiments on more detectors with ICD, Type denotes the AP score is evaluated on bounding box (BBox) or instance mask (Mask). denotes using the inheriting strategy.
Detector
Setting
Type
AP
AP50
AP75
APs
APM
APt.
Teacher (3x)
42.6
61.6
45.8
26.2
46.3
53.8
FCOS 42
Student (1x)
39.4
58.2
42.4
24.2
43.4
Teacher: 18.8 JPS 51.2M
Ours
BBox
41.7(+2.3)
60.3
45.4
26.9
45.9
52.6
Studen: FPS 32
Ours
42.9(+3.5)
61.6
46.6
27.8
46.8
54.6
Teacher (3x)
42.9
63.3
46.8
26.4
46.6
56.1
Student (1x)
38.6
59.5
42.1
22.5
42.0
BBox
Ours
40.4 (+1.8)
60.9
44.2
24.4
43.7
52.0
Mask R-CNN 16
Ours
41.2 (+2.6)
62.0
45.0
25.1
44.5
53.6
Teacher: 17.5 FPS 63.3
Teacher (3x)
38.6
60.4
41.3
19.5
41.3
55.3
32.9
Student (1x)
35.2
56.3
37.5
17.2
37.7
50.3
Mask
Ours
36.7 (+1.5)
58.0
39.2
18.4
38.9
52.5
Ours t
37.4 (+2.2)
58.7
40.1
19.1
39.8
53.7
Teacher (3x)
39.0
59.4
41.9
16.2
43.1
58.2
SOLOv2 46
Student
34.6
54.7
36.9
13.2
37.9
53.3
Teacher: 16.6 TPS 65.5M
Ours
Mask
37.2 (+2.6)
57.6
39.8
14.8
40.7
57.0
Student: 21 FPS/46.85
Ours t
38.5 (+3.9)
59.0
41.2
15.9
42.3
58.9
Teacher (3x)
44.6
63.7
48.4
27.5
47.8
58.4
Student (1x)
39.7
58.8
43.1
23.9
43.3
BBox
Ours
42.4 (+2.7)
61.5
46.1
25.3
46.0
54.3
CondInst
Ours t
43.7 (+4.0)
62.9
47.2
27.1
47.3
56.6
Texcher: 16.8 FPS 53.5M
Teacher (3x)
39.8
61.4
42.6
19.4
43.5
58.3
Student: 21.3 FPS 34.1M
Student (1x)
35.7
56.7
37.7
16.8
39.1
50.3
Mask
Ours
37.8 (+2.1)
59.1
40.4
17.5
41.4
Ours 1
39.1 (+3.4)
60.5
42.0
19.1
42.6
57.0
We find initialize these modules with teacher's parameters will lead to faster convergence, we call the inheriting strategy in experiments.
Most experiments are conducted on large scale object detection benchmark MS-COCO 3 31 with 80 classes. We train models on MS-COCO 2017 trainvall alr11k subset and validate on minival subset. Following common protocols, we report mean Average Precision (AP) an evaluation metric, together with AP under different thresholds and scales, i.e., APso. AP75. APs. APM. APL. Other experiments are listed in Appendix B, e.g., on VOC [2 and Cityscapes [10] more ablations.
4.2 Main Results
Compare with state-of-the-art methods. We compare method (ICD) with previous state- of-the-arts (SOTAs), including classic distillation method FitNet [37], two typical detection KD methods 28 45) and very recent work with strong performance from Zhang et al. [5I]. The
comparison is conducted on two classic detectors: Faster R-CNN (36 and RetinaNet 30
We adopt
AIS-COCO is publicly available, the annotations are licensed under Creative Commons Attribution 4.0 License and the use of the images follows Flickr Terms of Use. Refer to [] for more details.
7

---

Table 3: Experiments mobile backbones. f denotes using the inheriting strategy.
Detector
Setting
Backbone
AP
AP
AP
APs
APM
APL
Teacher (3x)
ResNet-
40.4
60.3
43.2
24.0
44.3
52.2
Student (1x) x)
26.4
42.0
27.8
13.8
28.8
RetinaNet 30
+ Ours
MBV2 38
29.5 (+3.1)
45.5
31.2
16.2
32.2
38.3
Ours t
31.6 (+5.2)
48.5
33.4
17.6
34.7
41.3
Teacher (3x)
ResNet-101
40.4
60.3
43.2
24.0
44.3
52.2
Student x)
34.9
54.8
37.0
20.9
38.9
44.8
RetinaNet 80
+ Ours
Eff-BO 39
36.7 (+1.8)
56.0
38.7
21.1
40.6
48.1
Ours t
(+3.))
57.5
40.2
22.4
41.6
50.3
Teacher (3x)
ResNet-101
42.0
62.5
45.9
25.2
45.6
54.6
Student (1x)
27.2
44.7
28.8
14.6
29.6
35.6
FRCNN 36
+ Ours
MBV2 38
30,2 (+3.0)
48.0
32.5
17.0
32.2
39.1
Ours t
31.4 (+4.2)
49.4
33.6
17.6
33.5
41.3
Teacher (3x)
ResNet-101
42.0
62.5
45.9
25.2
45.6
54.6
Student (1x)
35.3
56.8
37.8
20.8
38.2
45.1
FRCNN 36
+ Ours
Eff-BO 39
(+1.7)
58.0
39.6
21.1
40.0
48.3
Ours i
37.9 (+2.6)
58.7
40.8
21.4
40.9
49.5
detectron2 official released models with ResNet- 101 backbone trained on 3x scheduler teacher networks, the student is trained on with ResNet-50 backbone following the above settings.
As shown in Table 1 ICD brings about 2.5 AP and 3.0 AP improvement for plain training and training with the inheriting strategy respectively. Especially for RetinaNet, the student with distillation even outperforms strong teacher trained on x scheduler. Compare with previous SOTAs, the proposed method leads to considerable margin for about 0.5 AP without the inheriting strategy.
Results other settings. We further evaluate ICD under various detectors, e.g., commonly used anchor-free detector FCOS (42. and three detectors that have been extended to instance segmentation: Mask R-CNN [66,, SOLOv2 [46] and CondInst [41). We adopt networks with ResNet-100 101 on 3x scheduler as teachers and networks with ResNet-50 on x scheduler as students following the above settings. As shown in Table 2 we observe consistent improvements for both detection and instance segmentation. There are at most around AP improvement on CondInst [41 on object detection and SOLOv2 [46] instance segmentation. Moreover, students with weaker backbone (ResNer-50 v.s. ResNet- 101) and less training images (1 /3) even outperform (FCOS) or perform on par (SOLOv2, CondInst) with teachers. Note that ICD does not introduce extra burden during inference, our method improves about 25% of FPS and reduces 40% of parameters compared with teachers.
Mobile backbones. Aside from main experiments on commonly used ResNet [7] we also conduct experiments on mobile backbones, which are frequently used in low-power devices. We evaluate our method on two prevalent architectures: MobileNet V2 (MBV2) (38] and EfficientNet-BO (Eff-BO) (39. The latter one adopts the MobileNet V2 as basis, and further extends it with advanced designs like stronger data augmentation and better activations.
Experiments are conducted on Faster R-CNN (abbr., FRCNN) 36 and RetinaNet 30 following the above settings. As shown in Table our method also significantly improves the performance on smaller backbones. For instance, we improve the RetinaNet with MobileNet V2 backbone with 5.2 AP gain and 3.1 AP gain with and without inheriting strategy respectively. and up to 3.1 AP gain for EfficientNet-B BO. We also observe consistent improvements over Faster R-CNN, with up to 4.2 AP gain for MobileNet- V2 and 2.6 AP gain for EfficientNet-BO.
Ablation Studies
To verify the design options and the effectiveness of each component, we conduct ablation studies with the classic RetinaNet detector on MS-COCO following the above settings.
Design of the auxiliary task. To better understand the role of auxiliary task. we conduct experiments to evaluate the contribution of each sub-task. Specifically, our auxiliary task is composed
SFPS is evaluated on Nvidia Tesla V100.

---

of an identification task with binary cross eross-yntropy loss and localization task with regression loss, the localization task is further augmented with hint on bounding box scales. As shown in Table 4 the identification task itself leads to 2.2 AP gain compare with the baseline, this high-light the importance of knowledge on object perception. The regression task itself leads 1.8 AP gain, and the scale information boosts it for extra 0.2 AP gain. Combine two of them, we achieve 2.5 AP overall improvement, which indicates the fusion of two knowledge brings extra benefits, Note the auxiliary task only update the decoder and does not introduce extra data, which is very different from multitask learning, e.g, Mask R-CNN[66.
Table 4: Comparison with different auxiliary task designs.
Identification
Localization
Scale
AP
AP
AP75
75
APs
APM
APr.
37.4
56.7
40.3
23.1
41.6
48.3
39.6
59.2
42.8
23.4
44.0
50.4
39.2
58.6
42.4
23.1
43.5
50.3
39.4
58.8
42.4
23.2
43.8
50.5
39.9
59.4
43.1
25.0
43.9
51.0
Impact of the instance-aware attention. To verify the effectiveness of instance-awarr attention learned by conditional computation, we directly replace it with different variations: the fine-gre ine-grained mask in [45], pixel-wise attention activation [3) foreground mask analog [14] and no attention mask. The result Fig. [ shows our instance-aware mask leads to about 0.9 AP gain over the baseline and 0.4 AP gain compare with the best replacement.
Table 5: Comparison with different types of attention.
Attention Type
AP
AP50
AP75
APs
APM
APi
No Attention
39.0
58.4
42.1
23.5
43.2
49.9
Foreground Mask
39.4
58.9
42.4
23.5
43.6
50.0
Fine-grained Mask
(45]
39.5
59.0
42.4
23.4
43.8
50.2
Attention Activation
43
39.3
58.6
42.3
22.6
43.6
50.1
Instance-conditionn Attention
39.9
59.4
43.1
25.0
43.9
51.0
Design of the decoder. We mainly verify two properties the decoder design, as shown in Table 6 First, we find proper number heads is important for the best performance. The head number balances the dimension for each sub-space and the number of spaces. The best number lies around which is consistent with former studies [5]. Second, we evaluate the effectiveness of cascaded decoders, as it is widely adopted in former studies 43 5 g However, we do not find significant differences between different cascade levels, it might because we use a fixed teacher to provide knowledge, that limited the learning ability.
Table 6: Ablation the number of heads and cascade levels.
(a) Number of heads.
(b) Number of cascade levels.
Heads
AP
APs
APM
APt.
Levels
AP
APs
APM
APr.
39.4
23.7
43.8
50.5
I
39.9
25.0
43.9
51.0
39.7
24.0
43.9
51.2
2
39.9
24.1
44.0
51.4
8
39.9
25.0
43.9
51.0
39.8
24.3
44.0
50.9
39.7
23.6
44.2
50.5
5
Discussion
Qualitative analysis. We present a visualization our learned instance-aware attention in Fig. We find different heads attend to different components related to each instance, e.g., salient parts, boundaries. This might suggest that salient parts are important and knowledge for regression mostly lies around boundaries, instead of averaged on the foreground as assumed in former studies. The head (vi) and (vii) are smoother around key regions, which may relate to context. The head (iv) mainly attends on image corner,, which might be degenerated case relate some implicit descriptors.

---

Images
Instances
(3)
(n)
(9)
(91)
(91)
Figure 4: Visualization of our learned instance-aware attention over each head. Red denotes weak areas and pink denotes strong areas.
Resource consumption. We benchmark the train- ing speed for our distillation method. As shown in Fig. [] training the decoder introduces negligible cost. Speeifiially.,
22h
benchmark 1x scheduler on RetinaNet (30) with eight 2080ti, following the same configuration in Section 4.2 The major time consumption is spent on training the teacher (3x), which takes about 33 hours. Training (forward and backward) the student takes about 8 hours, while training decoder and distillation only take 1.3 hours. Besides the training time, memory consumption of our method also limited. One can update the de- coder part and the student part in consecutive itera- tions, leaving only intermediate features for distillation memory across iterations.
Tiain Toncher
8b
Tain Sucken
Fowvrr Tember
Distilation
Figure 5: Time consumption.
Real-world impact. The proposed method provides a convenient approach to enhance detector under certain setting, resulting in that small model can perform on par with a larger one. On the positive side, allows users replace a big model with a smaller one, which reduces energy consumption. On the potentially negative side, the teacher could be costly in training, also the student might inherit biases from the teacher, which is hard for tracing.
Conclusion
We introduce a novel framework for knowledge distillation. The proposed Instance-Condie Conditional knowledge Distillation (ICD) method utilizes instance-feature cross attention select and locate knowledge that correlates with human observed instances, which provides new framework for KD. Our formulation encodes instance as query and teacher's 's representation as key. To teach the decoder how to find knowledge, we design an auxiliary task that relies on knowledge for recognition and localization. The proposed method consistently improves various detectors, leading to impressive performance gain, some student networks even surpass their teachers. In addition to our design of the auxiliary task, we believe there are other alternations that can cover different situations or provide theoretical formulation of knowledge, which will be a potential direction for further researches.
10

---

Acknowledgments and Disclosure of Funding
This paper is supported by the National Key R&D Plan of the Ministry of Science and Technology (Project No. 2020AAA01044000 and Beijing Academy of Artificial Intelligence (BAAI).
References
[1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitiick, and Devi Parikh. VQA: visual question answering. In (CCV,2015
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. ArXiv preprint, 2016.
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[6] Guobin Chen, Wongun Choi, Xiang Yu, Tony X. Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In NeuriPS, 2017.
[7] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
2019.
[8] Liangyu Chen, Tong Yang. Xiangyu Zhang, Wei Zhang, and Jian Sun. Points as queries: Weakly semi-supervised object detection by points. In CVPR, 2021.
[9] Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowledge. In CVPR, 2020.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[11] Xing Dai, Zeren Jiang. Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for object detection. CVPR, 2021.
[12] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: retrospective. IJCV, 2015.
[13] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, IJCV, 2021.
[14] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In CVPR, 2021.
[15] Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In NeurlPS, 2015.
[16] Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.

---

[18] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. ICCV. 2017.
[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurIPS, 2014.
[20] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In CVPR, 2018.
[21] Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[22] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In NeurIPS, 2016.
[23] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Vaniv,, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. JMLR, 2017.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[25] Hei Law and Jia Deng. Cornemet: Detecting objects as paired keypoints. In ECCV, 2018.
[26] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, 2018.
[27] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
[28] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. CVPR, 2017.
[29] Tsung-Yo Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[30] Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[31] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[32] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yungiang Duan. Knowledge distillation via instance relationship graph. In CVPR, 2019.
[33] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- nigh-performance deep learning library. In NeurlPS, 2019.
[35] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
[36] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[37] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[38] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
12

---

[39] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, Proceedings of Machine Learning Research, 2019.
[40] Zhi Tian, Hao Chen, Xinlong Wang, Yuliang Liu, and Chunhua Shen. AdelaiDet: A toolbox for instance-level recognition tasks. hattpa://gitt adaadaddeeee 2019.
[41] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation.
In Proc. Eur. Conf. Computer Vision (ECCV), 2020.
[42] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: fully convolutional one-stage object detection. In ICCV, 2019.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[44] Jianfeng Wang, Lin Song, Zeming Li, Hongbin Sun, Jian Sun, and Nanning Zheng, End-to-end object detection with fully convolutional network. In CVPR, 2021.
[45] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In CVPR, 2019.
[46] Xinlong Wang, Rufeng Zhang, Tao Kong. Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. In NeurIPS, 2020.
[47] Yuxin Wu, Alexander Kirillov, Francisco Massa, Yen Lo, and Ross Girshick. Detectron2. 2019.
ptttps//ggtn htt.://wwwoooooeooooooeeeeeeeenn
[48] Jianwei Yang, Jiasen Lu, Dhruv Batra, and Devi Parikh. faster pytorch implementation of faster r-cnn. https: / github..h.... cow wwwm://wyaangaasteeerrmmm pythrcc 2017.
[49] Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang. and Jiashi Feng. Revisiting knowledge
distillation via label smoothing regularization. In CVPR, 2020.
[50] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[51] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In ICLR, 2021.
13


Reviews:
[{"review_id": "xxYb9Oo9-zB", "reviewer": "Reviewer_zCpx", "summary": "This paper propose a novel method for knowledge distillation on object detection by measuring the correlation between each object and human observed instances. They have introduced a conditional modeling paradigm to find the crucial regions for distillation and apply the auxiliary tasks to supervise its training. Good experiment results have been achieved on COCO.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. The ideas that applying KD on object detection and find the important regions for KD are not new. But the ideas that applying transformer decoder and conditional computation to find the KD regions are very interesting and novel.\n\nQuality. This paper has sufficient experiments and ablations to show the effectiveness of their methods. Although there is no theoretical proof about their methods, the visualization results provided in the paper show that they can find better KD regions in object detection. I think it's a new technical sound in this domain.\n\nClarity. This paper is easy to follow but some details need to be improved.\n\nSignificance. They have outperform the existing object detection KD methods by a clear margin in Table.1. And good results have been achieved on the state-of-the-art detectors such as SOLOv2 and CondInst and efficient detectors such as EfficientNet-B0.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. All the experiments are conducted on COCO, it will be better with experiments on more datasets such as Cityscpaes.\n2. In table 2, it will be better to report the inference time and parameters of each model.\n3. Figure 1 \"Previous studies for object detection distill discriminate regions by sampling proposals or center areas\". This is not correct. There have been some methods which find the regions without proposals such as [47].\n4. Figure 2 needs to be explained better.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "p4FzFmik9CX", "reviewer": "Reviewer_LAyY", "summary": "This submission proposes a novel knowledge distillation approach for object detection. The authors attempt to use an auxiliary task with the key-query (transformer-liked) method to determine where and how the feature map can be transferred from a teacher to a student. In the paper, the target instances are encoded as queries to determine the knowledge distilled from the teacher. Extensive experiments show the effectiveness of the proposed approach.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n+ Novel idea. The authors propose to make use of the transformer-liked method to generate the attention mask based on real instances such that the approach can locate the informative knowledge to guide the student network's training.\n+ Promising results and convincing ablation study.\n\nWeakness:\n\n- The English writing of this paper should be improved. Several typos in the paper, such as 'wildly' in line 99, 'Direc' in line 201. Please check the paper carefully.\n- Figures 1, 2, and 3 are hard to follow. Please add related notations along with the text and some sub captions to refine them.\n- I believe that generating fake instances is crucial in the method, how and why 5? It would be better to discuss more on this part.\n- The inheriting parameters strategy, which improves the performance significantly, can only be used in the scenario where the student and teacher share the same architecture in detection head and FPN. It is more or less a big limitation in the knowledge distillation domain where the student and teacher may vary a lot in the network architecture.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed distillation method only explores the situation where teacher and student have the same framework but different backbones, such that from Resnet101-based teachers to ResNet50-based students. The inheriting parameters strategy is not possible to use if the teacher and student have different detection heads and FPNs.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "dJpBchUxR0", "reviewer": "Reviewer_Ccfq", "summary": "The paper proposes a target-oriented solution towards knowledge distillation for object detection. Unlike prior works which focus on distilling the global representations or foreground-region representations, the paper aims to find relevant regions for distillation based on the instance-conditional information. The auxiliary task optimization helps in finding the related knowledge. The paper performs extensive experiments on MS-COCO to show the efficacy of their method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The idea of instance-aware knowledge distillation is novel.\n\n- The paper achieves respectable performance improvement over the previous approaches even beats the teacher in\nsome cases.\n\n- In Table 1, the backbone of the teacher network used is Resnet-101. For a fair comparison with [47],  the backbone\nthat should be used is Cascade Mask RCNN with ResNetXt101. [47] obtains 41.5% mAP on Resnet-50 Faster-RCNN student in this setting. Similar setting for RetinaNet would be appreciated. The method beats Wang et al. [42] with a good margin though.\n\n- Although the paper argues that the cost associated with the additional training is negligible, the concatenation of the feature maps across different scales (L) seems to add more parameters for training.\n\n- The method seems to depend on FPN. Is that the case ? This will limit the usage of the method significantly.\n\n- In Figure 4, the learned attention over the heads is degenerate for some heads (v). Better techniques for learning\nattention over each head should be explored.\n\n- Correction: Line no. 229, I think it should be “a very recent work with strong performance from Zhang et al. [47]”.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations for their method in the discussion section. \n\n1. Fail cases in the form of learning some degenerate heads.\n\n2. Learning from a teacher can be costly.\n\n3. Student might inherit some bias from the teacher.\n\nHowever, Points 2 and 3 are the limitations of the Knowledge Distillation setup in general, not particular to this method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "SplIBFwaSOr", "reviewer": "Reviewer_WeoR", "summary": "This paper proposes a new method named Instance-Conditional knowledge Distillation (ICD) to deal with the knowledge distillation problem in image object detection task. Motivated by the fact the traditional classification-level distillation methods cannot focus on the locations of objects, they insert additional instance information to the distillation training together with auxiliary tasks, i.e. recognition and localization. Their proposed distillation framework improves the performance of the student network, and even outperforms the teacher network under some specific settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: This method is original in this area, although it combines several well-known techniques. Related works are also well-summarized and discussed. \n\n**Quality**: With my experience, this method is technically correct. The motivation of is clear and straightforward. Their experiment supports their statements in their methodology. There are two things that might improve the quality of this paper:\n\n(1)\tThe analysis of why the student network outperforms the teacher network. Is it possible that the teacher network is not well-trained, and the new modules and tasks added in the KD framework make up for the previous training deficiencies?\n\n(2)\tThe analysis of the hyper-parameter $\\lambda$ in equation (12).\n\n**Clarity**: In general, the paper is well-organized and easy to follow. However, I have some suggestions about the figures:\n\n(1)\tIn Figure 1, the last column uses ”object-oriented“ as its title, but this term does not appear in the introduction part. I only find ”task-oriented“ described in the context. I am not sure if the last column represents their proposed method. \n\n(2)\tIn Figure 1, there is no description of the blue and gray blocks. I assume the blue one is the student network and the gray one is the teacher network.\n\n(3)\tIn Figure 2, there are a lot of different kinds of arrows and rectangles. Without any descriptions, it is really hard to understand the details of the proposed method.\n\n**Significance**: The significance and novelty are the main reason for my current rating. Although their experiment indeed shows improvement compared with the 4 previous methods, the contribution of the proposed method is still limited. If I understand the method correctly, their improvement mainly comes from the auxiliary tasks of instance information, which is not a fundamental modification.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the authors mention in Section 5, one limitation of their method is that the student might inherit biases from the teacher which could be hard to identify. Another limitation I can imagine is the access to instance information. What if we cannot access the training label? Where can we get the instance information?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["Discrimination / Bias / Fairness Concerns"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the paper originally received slightly mixed ratings, with one reviewer recommending rejection and 3 acceptance, the authors' feedback eventually convinced the most negative reviewer to update their score. Altogether, the reviewers acknowledge the novelty of the proposed method and that the empirical results are convincing. The authors' feedback also provided additional results and clarifications, which we strongly encourage the authors to incorporate in the final version.
Decision: Accept (Poster)