Instance-Conditional Knowledge Distillation for Object Detection

1 Introduction . Deep learning applications blossom in recent years with the breakthrough of Deep Neural Networks ( DNNs ) [ 17 , 24 , 21 ] . In pursuit of high performance , advanced DNNs usually stack tons of blocks with millions of parameters , which are computation and memory consuming . The heavy design hinders the deployment of many practical downstream applications like object detection in resource-limited devices . Plenty of techniques have been proposed to address this issue , like network pruning [ 15 , 27 , 18 ] , quantization [ 22 , 23 , 35 ] , mobile architecture design [ 38 , 39 ] and knowledge distillation ( KD ) [ 19 , 37 , 43 ] . Among them , KD is one of the most popular choices , since it can boost a target network without introducing extra inference-time burden or modifications . KD is popularized by Hinton et al . [ 19 ] , where knowledge of a strong pretrained teacher network is transferred to a small target student network in the classification scenario . Many good works emerge following the classification track [ 50 , 37 , 32 ] . However , most methods for classification perform badly in the detection : only slight improvements are observed [ 28 , 51 ] . This can be attributed to two reasons : ( 1 ) Other than category classification , another challenging goal to localize the object is seldomly considered . ( 2 ) Multiple target objects are presented in an image for detection , where objects ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can distribute in different locations . Due to these reasons , the knowledge becomes rather ambiguous and imbalance in detection : representations from different positions like foreground or background , borders or centers , could have different contributions , which makes distillation challenging . To handle the above challenge , two strategies are usually adopted by previous methods in detection . First , the distillation is usually conducted among intermediate representations , which cover all necessary features for both classification and localization . Second , different feature selection methods are proposed to overcome the imbalance issue . Existent works could be divided into three types according to the feature selection paradigm : proposal-based , rule-based and attention-based . In proposal-based methods [ 28 , 11 , 6 ] , proposal regions predicted by the RPN [ 36 ] or detector are sampled for distillation . In rule-based methods [ 14 , 45 ] , regions selected by predesigned rules like foreground or label-assigned regions are sampled . Despite their improvements , limitations still exist due to the hand-crafted designs , e.g. , many methods neglect the informative context regions or involve meticulous decisions . Recently , Zhang et al . [ 51 ] propose to use attention [ 43 ] , a type of intermediate activation of the network , to guide the distillation . Although attention provides inherent hints for discriminative areas , the relation between activation and knowledge for detection is still unclear . To further improve KD quality , we hope to provide an explicit solution to connect the desired knowledge with feature selection . Towards this goal , we present Instance-Conditional knowledge Distillation ( ICD ) , which introduces a new KD framework based-on conditional knowledge retrieval . In ICD , we propose to use a decoding network to find and distill knowledge associated with different instances , we deem such knowledge as instance-conditional knowledge . Fig . 1 shows the framework and compares it with former ones , ICD learns to find desired knowledge , which is much more flexible than previous methods , and is more consistent with detection targets . In detail , we design a conditional decoding module to locate knowledge , the correlation between knowledge and each instance is measured by the instance-aware attention via the transformer decoder [ 5 , 43 ] . In which human observed instances are projected to query and the correlation is measured by scaled-product attention between query and teacher ’ s representations . Following this formulation , the distillation is conducted over features decomposed by the decoder and weighted by the instance-aware attention . Last but not least , to optimize the decoding module , we also introduce an auxiliary task , which teaches the decoder to find useful information for identification and localization . The task defines the goal for knowledge retrieval , it facilitates the decoder instead of the student . Overall , our contribution is summarized in three-fold : • We introduce a novel framework to locate useful knowledge in detection KD , we formulate the knowledge retrieval explicitly by a decoding network and optimize it via an auxiliary task . • We adopt the conditional modeling paradigm to facilitate instance-wise knowledge transferring . We encode human observed instances as query and decompose teacher ’ s representations to key and value to locate fine-grained knowledge . To our knowledge , it is the first trial to explore instance-oriented knowledge for detection . • We perform comprehensive experiments on challenging benchmarks . Results demonstrate impressive improvements over various detectors with up to 4 AP gain in MS-COCO , including recent detectors for instance segmentation [ 41 , 46 , 16 ] . In some cases , students with 1× schedule are even able to outperform their teachers with larger backbones trained 3× longer . 2 Related Works . 2.1 Knowledge Distillation . Knowledge distillation aims to transfer knowledge from a strong teacher to a weaker student network to facilitate supervised learning . The teacher is usually a large pretrained network , who provides smoother supervision and more hints on visual concepts , that improves the training quality and convergence speed [ 49 , 9 ] . KD for image classification has been studied for years , they are usually categorized into three types [ 13 ] : response-based [ 19 ] , feature-based [ 37 , 43 ] and relation-based [ 32 ] . Among them , feature-based distillation over multi-scale features is adopted from most of detection KD works , to deal with knowledge among multiple instances in different regions . Most of these works can be formulated as region selection for distillation , where foreground-background unbalancing is considered as a key problem in some studies [ 45 , 51 , 14 ] . Under this paradigm , we divide them into three kinds : ( 1 ) proposal-based , ( 2 ) rule-based and ( 3 ) attention-based . ( 1 ) Proposal-based methods rely on the prediction of the RPN or detection network to find foreground regions , e.g. , Chen et al . [ 6 ] and Li et al . [ 28 ] propose to distilling regions predicted by RPN [ 36 ] , Dai et al . [ 11 ] proposes GI scores to locate controversial predictions for distillation . ( 2 ) Rule-based methods rely on designed rules that can be inflexible and hyper-parameters inefficient , e.g. , Wang et al . [ 45 ] distill assigned regions where anchor and ground-truth have a large IoU , Guo et al . [ 14 ] distill foreground and background regions separately with different factors . ( 3 ) Attention-based methods rely on activations to locate discriminative areas , yet they do not direct to knowledge that the student needs . Only a recent work from Zhang et al . [ 51 ] considers attention , they build the spatial-channel-wise attention to weigh the distillation . To overcome the above limitations , we explore the instance-conditional knowledge retrieval formulated by a decoder to explicitly search for useful knowledge . Like other methods , ICD does not have extra cost during inference or use extra data ( besides existing labels and a pretrained teacher ) . 2.2 Conditional Computation . Conditional computation is widely adopted to infer contents on a given condition . Our study mostly focuses on how to identify visual contents given an instance as a condition . This is usually formulated as query an instance on the image , e.g. , visual question answer [ 1 , 2 ] and image-text matching [ 26 ] queries information and regions specified by natural language . Besides query by language , other types of query are proposed in recent years . For example , DETR [ 5 ] queries on fixed proposal embeddings , Chen et al . [ 8 ] encodes points as queries to facilitate weakly-supervised learning . These works adopt transformer decoder to infer upon global receptive fields that cover all visual contents , yet they usually rely on cascaded decoders that are costly for training . From another perspective , CondInst [ 41 ] and SOLOv2 [ 46 ] generate queries based on network predictions and achieves great performance on instance segmentation . Different from them , this work adopts the query-based approach to retrieve knowledge and build query based on annotated instances . 2.3 Object Detection . Object detection has been developed rapidly . Modern detectors are roughly divided into two-stage or one-stage detectors . Two-stage detectors usually adopt Region Proposal Network ( RPN ) to generate initial rough predictions and refine them with detection heads , the typical example is Faster R-CNN [ 36 ] . On the contrary , one-stage detectors directly predict on the feature map , which are usually faster , they include RetinaNet [ 30 ] , FCOS [ 42 ] . Besides of this rough division , many extensions are introduced in recent years , e.g. , extension to instance segmentation [ 41 , 46 , 16 ] , anchor-free models [ 42 , 25 ] and end-to-end detection [ 5 , 44 , 20 ] . Among these works , multi-scale features are usually adopted to enhance performance , e.g. , FPN [ 29 ] , which is considered as a typical case for our study . To generalize to various methods , the proposed method distills the intermediate features and does not rely on detector-specific designs . 3 Method . 3.1 Overview . As discussed in former studies [ 14 , 51 ] , useful knowledge for detection distributes unevenly in intermediate features . To facilitate KD , we propose to transfer instance-conditional knowledge between student and teacher network , termed κSi and κ T i corresponding to the ith instance : Ldistill = N∑ i=1 Ld ( κSi , κTi ) ( 1 ) The knowledge towards teacher ’ s representations T and condition yi is formulated as κTi = G ( T , yi ) ( κSi similarly ) , where G is the instance-conditional decoding module , optimized by an auxiliary loss illustrated in Sec . 3.3 . The overall framework is shown in Fig . 2 . In the following sections , we will introduce the instance-conditional knowledge ( Sec . 3.2 ) , describe the auxiliary task design ( Sec . 3.3 ) , and discuss the knowledge transferring ( Sec . 3.4 ) . 3.2 Instance-conditional Knowledge In this section , we elaborate the instanceconditional decoding module G , which computes instance-conditional knowledge κTi from ( 1 ) unconditional knowledge given ( 2 ) instance conditions . ( 1 ) The unconditional knowledge T , symbolizes all available information from the teacher detector . Since modern detectors commonly involve a feature pyramid network ( FPN ) [ 29 ] to extract rich multi-scale representations , we present multi-scale representations as T = { Xp ∈ RD×Hp×Wp } p∈P , where P signifies the spatial resolutions while D is the channel dimension . By concatenating representations at different scales along the spatial dimension , we obtain AT ∈ RL×D , where L = ∑ p∈P HpWp is the sum of total pixels number across scales . ( 2 ) The instance condition , originally describing a human-observed object , is denoted by Y = { yi } Ni=1 , where N is the object number and yi = ( ci , bi ) is the annotation for the i-th instance , including category ci and box location bi = ( xi , yi , wi , hi ) which specifies the localization and size information . To produce learnable embeddings for each instance , the annotation is mapped to a query feature vector qi in the hidden space , which specifies a condition to collect desired knowledge : qi = Fq ( E ( yi ) ) , qi ∈ RD ( 2 ) where E ( · ) is a instance encoding function ( detailed in in Sec . 3 ) and Fq is a Multi-Layer Perception network ( MLP ) . We retrieve knowledge from T given qi by measuring responses of correlation . This is formulated by dot-product attention [ 43 ] with M concurrent heads in a query-key attention manner . In which each head j corresponds to three linear layers ( Fkj , F q j , Fvj ) w.r.t . the key , query and value construction . The key feature KTj is computed by projecting teacher ’ s representations A T with the positional embeddings [ 43 , 5 ] P ∈ RL×d as in Eq . 3 , where Fpe denotes a linear projection over position embeddings . The value feature VTj and query qij is projected by linear mappings to a sub-space with d = D/M channels , Fvj on AT and F q j over qi respectively , as shown in Eq . 4 . At last , an instance-aware attention mask mij of the i-th instance by the j-th head is obtained by normalized dot-product between KTj and qij : KTj = Fkj ( AT + Fpe ( P ) ) , KTj ∈ RL×d ( 3 ) VTj = Fvj ( AT ) , VTj ∈ RL×d ( 4 ) qij = Fqj ( qi ) , qij ∈ R d ( 5 ) mij = softmax ( KTj qij√ d ) , mij ∈ RL ( 6 ) Intuitively , the querying along the key features and value features describes the correlation between representations and instances . We collect κTi = { ( mij , VTj ) } Mj=1 as the instance-conditional knowledge from T , which encodes knowledge corresponds to the ith instance .

Summary: This submission proposes a novel knowledge distillation approach for object detection. The authors attempt to use an auxiliary task with the key-query (transformer-liked) method to determine where and how the feature map can be transferred from a teacher to a student. In the paper, the target instances are encoded as queries to determine the knowledge distilled from the teacher. Extensive experiments show the effectiveness of the proposed approach.