Instance-Conditional Knowledge Distillation for Object Detection

1 Introduction . Deep learning applications blossom in recent years with the breakthrough of Deep Neural Networks ( DNNs ) [ 17 , 24 , 21 ] . In pursuit of high performance , advanced DNNs usually stack tons of blocks with millions of parameters , which are computation and memory consuming . The heavy design hinders the deployment of many practical downstream applications like object detection in resource-limited devices . Plenty of techniques have been proposed to address this issue , like network pruning [ 15 , 27 , 18 ] , quantization [ 22 , 23 , 35 ] , mobile architecture design [ 38 , 39 ] and knowledge distillation ( KD ) [ 19 , 37 , 43 ] . Among them , KD is one of the most popular choices , since it can boost a target network without introducing extra inference-time burden or modifications . KD is popularized by Hinton et al . [ 19 ] , where knowledge of a strong pretrained teacher network is transferred to a small target student network in the classification scenario . Many good works emerge following the classification track [ 50 , 37 , 32 ] . However , most methods for classification perform badly in the detection : only slight improvements are observed [ 28 , 51 ] . This can be attributed to two reasons : ( 1 ) Other than category classification , another challenging goal to localize the object is seldomly considered . ( 2 ) Multiple target objects are presented in an image for detection , where objects ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . can distribute in different locations . Due to these reasons , the knowledge becomes rather ambiguous and imbalance in detection : representations from different positions like foreground or background , borders or centers , could have different contributions , which makes distillation challenging . To handle the above challenge , two strategies are usually adopted by previous methods in detection . First , the distillation is usually conducted among intermediate representations , which cover all necessary features for both classification and localization . Second , different feature selection methods are proposed to overcome the imbalance issue . Existent works could be divided into three types according to the feature selection paradigm : proposal-based , rule-based and attention-based . In proposal-based methods [ 28 , 11 , 6 ] , proposal regions predicted by the RPN [ 36 ] or detector are sampled for distillation . In rule-based methods [ 14 , 45 ] , regions selected by predesigned rules like foreground or label-assigned regions are sampled . Despite their improvements , limitations still exist due to the hand-crafted designs , e.g. , many methods neglect the informative context regions or involve meticulous decisions . Recently , Zhang et al . [ 51 ] propose to use attention [ 43 ] , a type of intermediate activation of the network , to guide the distillation . Although attention provides inherent hints for discriminative areas , the relation between activation and knowledge for detection is still unclear . To further improve KD quality , we hope to provide an explicit solution to connect the desired knowledge with feature selection . Towards this goal , we present Instance-Conditional knowledge Distillation ( ICD ) , which introduces a new KD framework based-on conditional knowledge retrieval . In ICD , we propose to use a decoding network to find and distill knowledge associated with different instances , we deem such knowledge as instance-conditional knowledge . Fig . 1 shows the framework and compares it with former ones , ICD learns to find desired knowledge , which is much more flexible than previous methods , and is more consistent with detection targets . In detail , we design a conditional decoding module to locate knowledge , the correlation between knowledge and each instance is measured by the instance-aware attention via the transformer decoder [ 5 , 43 ] . In which human observed instances are projected to query and the correlation is measured by scaled-product attention between query and teacher ’ s representations . Following this formulation , the distillation is conducted over features decomposed by the decoder and weighted by the instance-aware attention . Last but not least , to optimize the decoding module , we also introduce an auxiliary task , which teaches the decoder to find useful information for identification and localization . The task defines the goal for knowledge retrieval , it facilitates the decoder instead of the student . Overall , our contribution is summarized in three-fold : • We introduce a novel framework to locate useful knowledge in detection KD , we formulate the knowledge retrieval explicitly by a decoding network and optimize it via an auxiliary task . • We adopt the conditional modeling paradigm to facilitate instance-wise knowledge transferring . We encode human observed instances as query and decompose teacher ’ s representations to key and value to locate fine-grained knowledge . To our knowledge , it is the first trial to explore instance-oriented knowledge for detection . • We perform comprehensive experiments on challenging benchmarks . Results demonstrate impressive improvements over various detectors with up to 4 AP gain in MS-COCO , including recent detectors for instance segmentation [ 41 , 46 , 16 ] . In some cases , students with 1× schedule are even able to outperform their teachers with larger backbones trained 3× longer . 2 Related Works . 2.1 Knowledge Distillation . Knowledge distillation aims to transfer knowledge from a strong teacher to a weaker student network to facilitate supervised learning . The teacher is usually a large pretrained network , who provides smoother supervision and more hints on visual concepts , that improves the training quality and convergence speed [ 49 , 9 ] . KD for image classification has been studied for years , they are usually categorized into three types [ 13 ] : response-based [ 19 ] , feature-based [ 37 , 43 ] and relation-based [ 32 ] . Among them , feature-based distillation over multi-scale features is adopted from most of detection KD works , to deal with knowledge among multiple instances in different regions . Most of these works can be formulated as region selection for distillation , where foreground-background unbalancing is considered as a key problem in some studies [ 45 , 51 , 14 ] . Under this paradigm , we divide them into three kinds : ( 1 ) proposal-based , ( 2 ) rule-based and ( 3 ) attention-based . ( 1 ) Proposal-based methods rely on the prediction of the RPN or detection network to find foreground regions , e.g. , Chen et al . [ 6 ] and Li et al . [ 28 ] propose to distilling regions predicted by RPN [ 36 ] , Dai et al . [ 11 ] proposes GI scores to locate controversial predictions for distillation . ( 2 ) Rule-based methods rely on designed rules that can be inflexible and hyper-parameters inefficient , e.g. , Wang et al . [ 45 ] distill assigned regions where anchor and ground-truth have a large IoU , Guo et al . [ 14 ] distill foreground and background regions separately with different factors . ( 3 ) Attention-based methods rely on activations to locate discriminative areas , yet they do not direct to knowledge that the student needs . Only a recent work from Zhang et al . [ 51 ] considers attention , they build the spatial-channel-wise attention to weigh the distillation . To overcome the above limitations , we explore the instance-conditional knowledge retrieval formulated by a decoder to explicitly search for useful knowledge . Like other methods , ICD does not have extra cost during inference or use extra data ( besides existing labels and a pretrained teacher ) . 2.2 Conditional Computation . Conditional computation is widely adopted to infer contents on a given condition . Our study mostly focuses on how to identify visual contents given an instance as a condition . This is usually formulated as query an instance on the image , e.g. , visual question answer [ 1 , 2 ] and image-text matching [ 26 ] queries information and regions specified by natural language . Besides query by language , other types of query are proposed in recent years . For example , DETR [ 5 ] queries on fixed proposal embeddings , Chen et al . [ 8 ] encodes points as queries to facilitate weakly-supervised learning . These works adopt transformer decoder to infer upon global receptive fields that cover all visual contents , yet they usually rely on cascaded decoders that are costly for training . From another perspective , CondInst [ 41 ] and SOLOv2 [ 46 ] generate queries based on network predictions and achieves great performance on instance segmentation . Different from them , this work adopts the query-based approach to retrieve knowledge and build query based on annotated instances . 2.3 Object Detection . Object detection has been developed rapidly . Modern detectors are roughly divided into two-stage or one-stage detectors . Two-stage detectors usually adopt Region Proposal Network ( RPN ) to generate initial rough predictions and refine them with detection heads , the typical example is Faster R-CNN [ 36 ] . On the contrary , one-stage detectors directly predict on the feature map , which are usually faster , they include RetinaNet [ 30 ] , FCOS [ 42 ] . Besides of this rough division , many extensions are introduced in recent years , e.g. , extension to instance segmentation [ 41 , 46 , 16 ] , anchor-free models [ 42 , 25 ] and end-to-end detection [ 5 , 44 , 20 ] . Among these works , multi-scale features are usually adopted to enhance performance , e.g. , FPN [ 29 ] , which is considered as a typical case for our study . To generalize to various methods , the proposed method distills the intermediate features and does not rely on detector-specific designs . 3 Method . 3.1 Overview . As discussed in former studies [ 14 , 51 ] , useful knowledge for detection distributes unevenly in intermediate features . To facilitate KD , we propose to transfer instance-conditional knowledge between student and teacher network , termed κSi and κ T i corresponding to the ith instance : Ldistill = N∑ i=1 Ld ( κSi , κTi ) ( 1 ) The knowledge towards teacher ’ s representations T and condition yi is formulated as κTi = G ( T , yi ) ( κSi similarly ) , where G is the instance-conditional decoding module , optimized by an auxiliary loss illustrated in Sec . 3.3 . The overall framework is shown in Fig . 2 . In the following sections , we will introduce the instance-conditional knowledge ( Sec . 3.2 ) , describe the auxiliary task design ( Sec . 3.3 ) , and discuss the knowledge transferring ( Sec . 3.4 ) . 3.2 Instance-conditional Knowledge In this section , we elaborate the instanceconditional decoding module G , which computes instance-conditional knowledge κTi from ( 1 ) unconditional knowledge given ( 2 ) instance conditions . ( 1 ) The unconditional knowledge T , symbolizes all available information from the teacher detector . Since modern detectors commonly involve a feature pyramid network ( FPN ) [ 29 ] to extract rich multi-scale representations , we present multi-scale representations as T = { Xp ∈ RD×Hp×Wp } p∈P , where P signifies the spatial resolutions while D is the channel dimension . By concatenating representations at different scales along the spatial dimension , we obtain AT ∈ RL×D , where L = ∑ p∈P HpWp is the sum of total pixels number across scales . ( 2 ) The instance condition , originally describing a human-observed object , is denoted by Y = { yi } Ni=1 , where N is the object number and yi = ( ci , bi ) is the annotation for the i-th instance , including category ci and box location bi = ( xi , yi , wi , hi ) which specifies the localization and size information . To produce learnable embeddings for each instance , the annotation is mapped to a query feature vector qi in the hidden space , which specifies a condition to collect desired knowledge : qi = Fq ( E ( yi ) ) , qi ∈ RD ( 2 ) where E ( · ) is a instance encoding function ( detailed in in Sec . 3 ) and Fq is a Multi-Layer Perception network ( MLP ) . We retrieve knowledge from T given qi by measuring responses of correlation . This is formulated by dot-product attention [ 43 ] with M concurrent heads in a query-key attention manner . In which each head j corresponds to three linear layers ( Fkj , F q j , Fvj ) w.r.t . the key , query and value construction . The key feature KTj is computed by projecting teacher ’ s representations A T with the positional embeddings [ 43 , 5 ] P ∈ RL×d as in Eq . 3 , where Fpe denotes a linear projection over position embeddings . The value feature VTj and query qij is projected by linear mappings to a sub-space with d = D/M channels , Fvj on AT and F q j over qi respectively , as shown in Eq . 4 . At last , an instance-aware attention mask mij of the i-th instance by the j-th head is obtained by normalized dot-product between KTj and qij : KTj = Fkj ( AT + Fpe ( P ) ) , KTj ∈ RL×d ( 3 ) VTj = Fvj ( AT ) , VTj ∈ RL×d ( 4 ) qij = Fqj ( qi ) , qij ∈ R d ( 5 ) mij = softmax ( KTj qij√ d ) , mij ∈ RL ( 6 ) Intuitively , the querying along the key features and value features describes the correlation between representations and instances . We collect κTi = { ( mij , VTj ) } Mj=1 as the instance-conditional knowledge from T , which encodes knowledge corresponds to the ith instance .

Summary: This paper proposes a new method named Instance-Conditional knowledge Distillation (ICD) to deal with the knowledge distillation problem in image object detection task. Motivated by the fact the traditional classification-level distillation methods cannot focus on the locations of objects, they insert additional instance information to the distillation training together with auxiliary tasks, i.e. recognition and localization. Their proposed distillation framework improves the performance of the student network, and even outperforms the teacher network under some specific settings.