Instance-Conditional Knowledge Distillation for Object Detection
Year: 2022

Abstract:
Knowledge distillation has shown great success in classification, however, it is still challenging for detection. In a typical image for detection, representations from different locations may have different contributions to detection targets, making the distillation hard to balance. In this paper, we propose a conditional distillation framework to distill the desired knowledge, namely knowledge that is beneficial in terms of both classification and localization for every instance. The framework introduces a learnable conditional decoding module, which retrieves information given each target instance as query. Specifically, we encode the condition information as query and use the teacher’s representations as key. The attention between query and key is used to measure the contribution of different features, guided by a localization-recognition-sensitive auxiliary task. Extensive experiments demonstrate the efficacy of our method: we observe impressive improvements under various settings. Notably, we boost RetinaNet with ResNet-50 backbone from 37.4 to 40.7 mAP (+3.3) under 1× schedule, that even surpasses the teacher (40.4 mAP) with ResNet-101 backbone under 3× schedule. Code has been released on https://github.com/megvii-research/ICD.

heading: 1 Introduction
text: Deep learning applications blossom in recent years with the breakthrough of Deep Neural Networks (DNNs) [17, 24, 21]. In pursuit of high performance, advanced DNNs usually stack tons of blocks with millions of parameters, which are computation and memory consuming. The heavy design hinders the deployment of many practical downstream applications like object detection in resource-limited devices. Plenty of techniques have been proposed to address this issue, like network pruning [15, 27, 18], quantization [22, 23, 35], mobile architecture design [38, 39] and knowledge distillation (KD) [19, 37, 43]. Among them, KD is one of the most popular choices, since it can boost a target network without introducing extra inference-time burden or modifications.
KD is popularized by Hinton et al. [19], where knowledge of a strong pretrained teacher network is transferred to a small target student network in the classification scenario. Many good works emerge following the classification track [50, 37, 32]. However, most methods for classification perform badly in the detection: only slight improvements are observed [28, 51]. This can be attributed to two reasons: (1) Other than category classification, another challenging goal to localize the object is seldomly considered. (2) Multiple target objects are presented in an image for detection, where objects
∗Equal contribution.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
can distribute in different locations. Due to these reasons, the knowledge becomes rather ambiguous and imbalance in detection: representations from different positions like foreground or background, borders or centers, could have different contributions, which makes distillation challenging.
To handle the above challenge, two strategies are usually adopted by previous methods in detection. First, the distillation is usually conducted among intermediate representations, which cover all necessary features for both classification and localization. Second, different feature selection methods are proposed to overcome the imbalance issue. Existent works could be divided into three types according to the feature selection paradigm: proposal-based, rule-based and attention-based. In proposal-based methods [28, 11, 6], proposal regions predicted by the RPN [36] or detector are sampled for distillation. In rule-based methods [14, 45], regions selected by predesigned rules like foreground or label-assigned regions are sampled. Despite their improvements, limitations still exist due to the hand-crafted designs, e.g., many methods neglect the informative context regions or involve meticulous decisions. Recently, Zhang et al. [51] propose to use attention [43], a type of intermediate activation of the network, to guide the distillation. Although attention provides inherent hints for discriminative areas, the relation between activation and knowledge for detection is still unclear. To further improve KD quality, we hope to provide an explicit solution to connect the desired knowledge with feature selection.
Towards this goal, we present Instance-Conditional knowledge Distillation (ICD), which introduces a new KD framework based-on conditional knowledge retrieval. In ICD, we propose to use a decoding network to find and distill knowledge associated with different instances, we deem such knowledge as instance-conditional knowledge. Fig. 1 shows the framework and compares it with former ones, ICD learns to find desired knowledge, which is much more flexible than previous methods, and is more consistent with detection targets. In detail, we design a conditional decoding module to locate knowledge, the correlation between knowledge and each instance is measured by the instance-aware attention via the transformer decoder [5, 43]. In which human observed instances are projected to query and the correlation is measured by scaled-product attention between query and teacher’s representations. Following this formulation, the distillation is conducted over features decomposed by the decoder and weighted by the instance-aware attention. Last but not least, to optimize the decoding module, we also introduce an auxiliary task, which teaches the decoder to find useful information for identification and localization. The task defines the goal for knowledge retrieval, it facilitates the decoder instead of the student. Overall, our contribution is summarized in three-fold:
• We introduce a novel framework to locate useful knowledge in detection KD, we formulate the knowledge retrieval explicitly by a decoding network and optimize it via an auxiliary task.
• We adopt the conditional modeling paradigm to facilitate instance-wise knowledge transferring. We encode human observed instances as query and decompose teacher’s representations to key and value to locate fine-grained knowledge. To our knowledge, it is the first trial to explore instance-oriented knowledge for detection.
• We perform comprehensive experiments on challenging benchmarks. Results demonstrate impressive improvements over various detectors with up to 4 AP gain in MS-COCO, including recent detectors for instance segmentation [41, 46, 16]. In some cases, students with 1× schedule are even able to outperform their teachers with larger backbones trained 3× longer.

heading: 2 Related Works

heading: 2.1 Knowledge Distillation
text: Knowledge distillation aims to transfer knowledge from a strong teacher to a weaker student network to facilitate supervised learning. The teacher is usually a large pretrained network, who provides smoother supervision and more hints on visual concepts, that improves the training quality and convergence speed [49, 9]. KD for image classification has been studied for years, they are usually categorized into three types [13]: response-based [19], feature-based [37, 43] and relation-based [32].
Among them, feature-based distillation over multi-scale features is adopted from most of detection KD works, to deal with knowledge among multiple instances in different regions. Most of these works can be formulated as region selection for distillation, where foreground-background unbalancing is considered as a key problem in some studies [45, 51, 14]. Under this paradigm, we divide them into three kinds: (1) proposal-based, (2) rule-based and (3) attention-based.
(1) Proposal-based methods rely on the prediction of the RPN or detection network to find foreground regions, e.g., Chen et al. [6] and Li et al. [28] propose to distilling regions predicted by RPN [36], Dai et al. [11] proposes GI scores to locate controversial predictions for distillation.
(2) Rule-based methods rely on designed rules that can be inflexible and hyper-parameters inefficient, e.g., Wang et al. [45] distill assigned regions where anchor and ground-truth have a large IoU, Guo et al. [14] distill foreground and background regions separately with different factors.
(3) Attention-based methods rely on activations to locate discriminative areas, yet they do not direct to knowledge that the student needs. Only a recent work from Zhang et al. [51] considers attention, they build the spatial-channel-wise attention to weigh the distillation.
To overcome the above limitations, we explore the instance-conditional knowledge retrieval formulated by a decoder to explicitly search for useful knowledge. Like other methods, ICD does not have extra cost during inference or use extra data (besides existing labels and a pretrained teacher).

heading: 2.2 Conditional Computation
text: Conditional computation is widely adopted to infer contents on a given condition. Our study mostly focuses on how to identify visual contents given an instance as a condition. This is usually formulated as query an instance on the image, e.g., visual question answer [1, 2] and image-text matching [26] queries information and regions specified by natural language. Besides query by language, other types of query are proposed in recent years. For example, DETR [5] queries on fixed proposal embeddings, Chen et al. [8] encodes points as queries to facilitate weakly-supervised learning. These works adopt transformer decoder to infer upon global receptive fields that cover all visual contents, yet they usually rely on cascaded decoders that are costly for training. From another perspective, CondInst [41] and SOLOv2 [46] generate queries based on network predictions and achieves great performance on instance segmentation. Different from them, this work adopts the query-based approach to retrieve knowledge and build query based on annotated instances.

heading: 2.3 Object Detection
text: Object detection has been developed rapidly. Modern detectors are roughly divided into two-stage or one-stage detectors. Two-stage detectors usually adopt Region Proposal Network (RPN) to generate initial rough predictions and refine them with detection heads, the typical example is Faster R-CNN [36]. On the contrary, one-stage detectors directly predict on the feature map, which are usually faster, they include RetinaNet [30], FCOS [42]. Besides of this rough division, many extensions are introduced in recent years, e.g., extension to instance segmentation [41, 46, 16], anchor-free models [42, 25] and end-to-end detection [5, 44, 20]. Among these works, multi-scale features are usually adopted to enhance performance, e.g., FPN [29], which is considered as a typical case for our study. To generalize to various methods, the proposed method distills the intermediate features and does not rely on detector-specific designs.

heading: 3 Method

heading: 3.1 Overview
text: As discussed in former studies [14, 51], useful knowledge for detection distributes unevenly in intermediate features. To facilitate KD, we propose to transfer instance-conditional knowledge between student and teacher network, termed κSi and κ T i corresponding to the ith instance:
Ldistill = N∑ i=1 Ld(κSi , κTi ) (1)
The knowledge towards teacher’s representations T and condition yi is formulated as κTi = G(T , yi) (κSi similarly), where G is the instance-conditional decoding module, optimized by an auxiliary loss illustrated in Sec. 3.3. The overall framework is shown in Fig. 2.
In the following sections, we will introduce the instance-conditional knowledge (Sec. 3.2), describe the auxiliary task design (Sec. 3.3), and discuss the knowledge transferring (Sec. 3.4).
3.2 Instance-conditional Knowledge
In this section, we elaborate the instanceconditional decoding module G, which computes instance-conditional knowledge κTi from (1) unconditional knowledge given (2) instance conditions.
(1) The unconditional knowledge T , symbolizes all available information from the teacher detector. Since modern detectors commonly involve a feature pyramid network (FPN) [29] to extract rich multi-scale representations, we present multi-scale representations as T = {Xp ∈ RD×Hp×Wp}p∈P , where P signifies the spatial resolutions while D is the channel dimension. By concatenating representations at different scales along the spatial dimension, we obtain AT ∈ RL×D, where L = ∑ p∈P HpWp is the sum of total pixels number across scales.
(2) The instance condition, originally describing a human-observed object, is denoted by Y = {yi}Ni=1, where N is the object number and yi = (ci,bi) is the annotation for the i-th instance, including category ci and box location bi = (xi, yi, wi, hi) which specifies the localization and size information.
To produce learnable embeddings for each instance, the annotation is mapped to a query feature vector qi in the hidden space, which specifies a condition to collect desired knowledge:
qi = Fq(E(yi)), qi ∈ RD (2)
where E(·) is a instance encoding function (detailed in in Sec. 3) and Fq is a Multi-Layer Perception network (MLP).
We retrieve knowledge from T given qi by measuring responses of correlation. This is formulated by dot-product attention [43] with M concurrent heads in a query-key attention manner. In which each head j corresponds to three linear layers (Fkj , F q j , Fvj ) w.r.t. the key, query and value construction.
The key feature KTj is computed by projecting teacher’s representations A T with the positional embeddings [43, 5] P ∈ RL×d as in Eq. 3, where Fpe denotes a linear projection over position embeddings. The value feature VTj and query qij is projected by linear mappings to a sub-space with d = D/M channels, Fvj on AT and F q j over qi respectively, as shown in Eq. 4. At last, an
instance-aware attention mask mij of the i-th instance by the j-th head is obtained by normalized dot-product between KTj and qij :
KTj = Fkj (AT + Fpe(P)), KTj ∈ RL×d (3) VTj = Fvj (AT ), VTj ∈ RL×d (4) qij = Fqj (qi), qij ∈ R d (5) mij = softmax( KTj qij√
d ), mij ∈ RL (6)
Intuitively, the querying along the key features and value features describes the correlation between representations and instances. We collect κTi = {(mij ,VTj )}Mj=1 as the instance-conditional knowledge from T , which encodes knowledge corresponds to the ith instance.

heading: 3.3 Auxiliary Task
text: In this section, we introduce the auxiliary task to optimize the decoding module G. First, we aggregate instance-level information to identify and localize objects. This could be obtained by aggregating the instance-conditional knowledge by the function Fagg, which includes sum-product aggregation over attention mij and VTj , concatenate features from each head, add residuals and project with a feed-forward network as proposed in [43, 5]:
gTi = Fagg(κTi ,qi), gTi ∈ RD (7)
To let the instance-level aggregated information gTi retain sufficient instance cues, one could design an instance-sensitive task to optimize it as below:
Laux = Lins(gTi ,H(yi)) (8)
where H encodes the instance information as targets. However, directly adopt Eq. 8 might lead to trivial solution, since yi is accessible from both gTi (through qi, see Eq. 2) andH(yi). It is possible that parameters will learn a shortcut, that ignore the teacher representations T . To resolve this issue, we propose to drop information on encoding function E(·), to force the aggregation function Fagg to excavate hints from T . The information dropping is adopted by replacing the accurate annotation for instance conditions to uncertain ones. For bounding box annotations, we relieve them to rough box centers with rough scales indicators. The rough box center (x′i, y
′ i) is obtained by random jittering as shown below:{
x′i = xi + φxwi, y′i = yi + φyhi,
(9)
where (wi, hi) is the width and height of the bounding box and φx, φy are sampled from a uniform distribution Φ ∼ U [−a, a], where we set a=0.3 empirically. The scales indicators are obtained by
rounding the box sizes in the logarithmic space, i.e., blog2(wi)c, blog2(hi)c. In addition, to let the decoder learn to identify instances and be aware of the uncertainty, we generate fake instances for the identification task according to dataset distributions, this is detailed in Appendix A. As a result, it collect coarse information and obtain instance encoding through E(·) as depicted in Fig. 3a. where ci is the category, Foh is the one hot vectorization, concat is the concatenation operator and Fpe is the position embedding function.
Finally, the aggregated representation gTi are optimized by the auxiliary task. We introduces two predictors denoted by Fobj and Freg respectively to predict identification and localization results. We adopt binary cross entropy loss (BCE) to optimize the real-fake identification and L1 loss to optimize the regression.
Laux = LBCE(Fobj(gTi ), δobj(yi)) + L1(Freg(gTi ), δreg(yi)) (10)
where δobj(·) is an indicator, it yields 1 if yi is real and 0 otherwise. Following common practice, the localization loss for fake examples is ignored. δreg is the preparing function for regression targets, following [42]. Appendix A provides more implementation details.

heading: 3.4 Instance-Conditional Distillation
text: Lastly, we present the formulation for conditional knowledge distillation. We obtain the projected value features VSj ∈ RL×d of the student representations analogous to Eq. 4 in Sec. 3.2. By adopting the instance-aware attention mask as a measurement of correlations between feature and each instance, we formulate the distillation loss as value features mimicking guided by the attention:
Ldistill = 1
MNr M∑ j=1 N∑ i=1 δobj(yi)· <mij ,LMSE(VSj ,VTj ) > (11)
where Nr = ∑N
i=1 δobj(yi), (Nr ≤ N) is the real instances number, LMSE(VSj ,VTj ) ∈ RL is the pixel-wise mean-square error along the hidden dimension2 and < ·, · > is the Dirac notation for inner product. For stability, the learnable variable mij and VTj are detached during distillation. Combine with the supervised learning loss Ldet, the overall loss with a coefficient λ is summarized below:
Ltotal = Ldet + Laux + λLdistill (12)
It is worth noticing, only the gradients w.r.t. Ldistill and Ldet back-propagate to the student network (from representations S). The gradients of Laux only update the instance-conditional decoding function G and auxiliary task related modules.

heading: 4 Experiments

heading: 4.1 Experiment Settings
text: We conduct experiments on Pytorch [34] with the widely used Detectron2 library [47] and AdelaiDet library 3 [40]. All experiments are running on eight 2080ti GPUs with 2 images in each. We adopt the 1× scheduler, which denotes 9k iterations of training, following the standard protocols in Detectron2 unless otherwise specified. Scale jittering with random flip is adopted as data augmentation.
For distillation, the hyper-parameter λ is set to 8 for one-stage detectors and 3 for two-stage detectors respectively. To optimize the transformer decoder, we adopt AdamW optimizer [33] for the decoder and MLPs following common settings for transformer [43, 5]. Corresponding hyper-parameters follows [5], where the initial learning rate and weight decay are set to 1e-4. We adopt the 256 hidden dimension for our decoder and all MLPs, the decoder has 8 heads in parallel. The projection layer Fq is a 3 layer MLP, Freg and Fobj share another 3 layer MLP. In addition, we notice some newly initialized modules of the student share the same size of the teacher, e.g., the detection head, FPN.
2LMSE(.) is conducted over normalized features with parameter-free LayerNorm [3] for stabilization. 3All libraries are open-sourced and public available, please refer to citations for more details.
We find initialize these modules with teacher’s parameters will lead to faster convergence, we call it the inheriting strategy in experiments.
Most experiments are conducted on a large scale object detection benchmark MS-COCO 4[31] with 80 classes. We train models on MS-COCO 2017 trainval115k subset and validate on minival subset. Following common protocols, we report mean Average Precision (AP) as an evaluation metric, together with AP under different thresholds and scales, i.e., AP50, AP75, APS, APM, APL. Other experiments are listed in Appendix B, e.g., on VOC [12] and Cityscapes [10], more ablations.

heading: 4.2 Main Results
text: Compare with state-of-the-art methods. We compare our method (ICD) with previous stateof-the-arts (SOTAs), including a classic distillation method FitNet [37], two typical detection KD methods [28, 45], and a very recent work with strong performance from Zhang et al. [51]. The comparison is conducted on two classic detectors: Faster R-CNN [36] and RetinaNet [30]. We adopt
4MS-COCO is publicly available, the annotations are licensed under a Creative Commons Attribution 4.0 License and the use of the images follows Flickr Terms of Use. Refer to [31] for more details.
detectron2 official released models with ResNet-101 backbone trained on 3× scheduler as teacher networks, the student is trained on 1× with ResNet-50 backbone following the above settings. As shown in Table 1, ICD brings about 2.5 AP and 3.0 AP improvement for plain training and training with the inheriting strategy respectively. Especially for RetinaNet, the student with distillation even outperforms a strong teacher trained on 3× scheduler. Compare with previous SOTAs, the proposed method leads to a considerable margin for about 0.5 AP without the inheriting strategy.
Results on other settings. We further evaluate ICD under various detectors, e.g., a commonly used anchor-free detector FCOS [42], and three detectors that have been extended to instance segmentation: Mask R-CNN [16], SOLOv2 [46] and CondInst [41]. We adopt networks with ResNet-101 on 3× scheduler as teachers and networks with ResNet-50 on 1× scheduler as students following the above settings. As shown in Table 2, we observe consistent improvements for both detection and instance segmentation. There are at most around 4 AP improvement on CondInst [41] on object detection and SOLOv2 [46] on instance segmentation. Moreover, students with weaker backbone (ResNet-50 v.s. ResNet-101) and less training images (1/3) even outperform (FCOS) or perform on par (SOLOv2, CondInst) with teachers. Note that ICD does not introduce extra burden during inference, our method improves about 25% of FPS5 and reduces 40% of parameters compared with teachers.
Mobile backbones. Aside from main experiments on commonly used ResNet [17], we also conduct experiments on mobile backbones, which are frequently used in low-power devices. We evaluate our method on two prevalent architectures: MobileNet V2 (MBV2) [38] and EfficientNet-B0 (Eff-B0) [39]. The latter one adopts the MobileNet V2 as basis, and further extends it with advanced designs like stronger data augmentation and better activations.
Experiments are conducted on Faster R-CNN (abbr., FRCNN) [36] and RetinaNet [30] following the above settings. As shown in Table 3, our method also significantly improves the performance on smaller backbones. For instance, we improve the RetinaNet with MobileNet V2 backbone with 5.2 AP gain and 3.1 AP gain with and without inheriting strategy respectively, and up to 3.1 AP gain for EfficientNet-B0. We also observe consistent improvements over Faster R-CNN, with up to 4.2 AP gain for MobileNet-V2 and 2.6 AP gain for EfficientNet-B0.

heading: 4.3 Ablation Studies
text: To verify the design options and the effectiveness of each component, we conduct ablation studies with the classic RetinaNet detector on MS-COCO following the above settings.
Design of the auxiliary task. To better understand the role of our auxiliary task, we conduct experiments to evaluate the contribution of each sub-task. Specifically, our auxiliary task is composed
5FPS is evaluated on Nvidia Tesla V100.
of an identification task with binary cross-entropy loss and a localization task with regression loss, the localization task is further augmented with a hint on bounding box scales. As shown in Table 4, the identification task itself leads to 2.2 AP gain compare with the baseline, this high-light the importance of knowledge on object perception. The regression task itself leads to 1.8 AP gain, and the scale information boosts it for extra 0.2 AP gain. Combine two of them, we achieve 2.5 AP overall improvement, which indicates the fusion of two knowledge brings extra benefits. Note the auxiliary task only update the decoder and does not introduce extra data, which is very different from multitask learning, e.g., Mask R-CNN[16].
Impact of the instance-aware attention. To verify the effectiveness of instance-aware attention learned by conditional computation, we directly replace it with different variations: the fine-grained mask in [45], pixel-wise attention activation [51, 43], foreground mask analog to [14] and no attention mask. The result in Fig. 5 shows our instance-aware mask leads to about 0.9 AP gain over the baseline and 0.4 AP gain compare with the best replacement.
Design of the decoder. We mainly verify two properties of the decoder design, as shown in Table 6. First, we find a proper number of heads is important for the best performance. The head number balances the dimension for each sub-space and the number of spaces. The best number lies around 8, which is consistent with former studies [5]. Second, we evaluate the effectiveness of cascaded decoders, as it is widely adopted in former studies [43, 5, 8]. However, we do not find significant differences between different cascade levels, it might because we use a fixed teacher to provide knowledge, that limited the learning ability.

heading: 5 Discussion
text: Qualitative analysis. We present a visualization of our learned instance-aware attention in Fig. 4. We find different heads attend to different components related to each instance, e.g., salient parts, boundaries. This might suggest that salient parts are important and knowledge for regression mostly lies around boundaries, instead of averaged on the foreground as assumed in former studies. The head (vi) and (vii) are smoother around key regions, which may relate to context. The head (iv) mainly attends on image corners, which might be a degenerated case or relate to some implicit descriptors.
Resource consumption. We benchmark the training speed for our distillation method. As shown in Fig. 5, training the decoder introduces negligible cost. Specifically, we benchmark on 1× scheduler on RetinaNet [30] with eight 2080ti, following the same configuration in Section 4.2. The major time consumption is spent on training the teacher (3×), which takes about 33 hours. Training (forward and backward) the student takes about 8 hours, while training decoder and distillation only take 1.3 hours. Besides the training time, memory consumption of our method is also limited. One can update the decoder part and the student part in consecutive itera-
tions, leaving only intermediate features for distillation in memory across iterations.
Real-world impact. The proposed method provides a convenient approach to enhance a detector under a certain setting, resulting in that a small model can perform on par with a larger one. On the positive side, it allows users to replace a big model with a smaller one, which reduces energy consumption. On the potentially negative side, the teacher could be costly in training, also the student might inherit biases from the teacher, which is hard for tracing.

heading: 6 Conclusion
text: We introduce a novel framework for knowledge distillation. The proposed Instance-Conditional knowledge Distillation (ICD) method utilizes instance-feature cross attention to select and locate knowledge that correlates with human observed instances, which provides a new framework for KD. Our formulation encodes instance as query and teacher’s representation as key. To teach the decoder how to find knowledge, we design an auxiliary task that relies on knowledge for recognition and localization. The proposed method consistently improves various detectors, leading to impressive performance gain, some student networks even surpass their teachers. In addition to our design of the auxiliary task, we believe there are other alternations that can cover different situations or provide a theoretical formulation of knowledge, which will be a potential direction for further researches.
Acknowledgments and Disclosure of Funding
This paper is supported by the National Key R&D Plan of the Ministry of Science and Technology (Project No. 2020AAA0104400) and Beijing Academy of Artificial Intelligence (BAAI).

Review summaries:
The paper proposes a target-oriented solution towards knowledge distillation for object detection. Unlike prior works which focus on distilling the global representations or foreground-region representations, the paper aims to find relevant regions for distillation based on the instance-conditional information. The auxiliary task optimization helps in finding the related knowledge. The paper performs extensive experiments on MS-COCO to show the efficacy of their method.

This paper proposes a new method named Instance-Conditional knowledge Distillation (ICD) to deal with the knowledge distillation problem in image object detection task. Motivated by the fact the traditional classification-level distillation methods cannot focus on the locations of objects, they insert additional instance information to the distillation training together with auxiliary tasks, i.e. recognition and localization. Their proposed distillation framework improves the performance of the student network, and even outperforms the teacher network under some specific settings.

This paper propose a novel method for knowledge distillation on object detection by measuring the correlation between each object and human observed instances. They have introduced a conditional modeling paradigm to find the crucial regions for distillation and apply the auxiliary tasks to supervise its training. Good experiment results have been achieved on COCO.

This submission proposes a novel knowledge distillation approach for object detection. The authors attempt to use an auxiliary task with the key-query (transformer-liked) method to determine where and how the feature map can be transferred from a teacher to a student. In the paper, the target instances are encoded as queries to determine the knowledge distilled from the teacher. Extensive experiments show the effectiveness of the proposed approach.