Learning Lightweight Object Detectors via Progressive Knowledge Distillation
Authors: Shengcao Cao
Mengtian Li
James Hays
Deva Ramanan
Yu-Xiong Wang
Liangyan Gui
Venue: Submitted to ICLR 2023 (2023)
Keywords: object detection
knowledge distillation

Abstract: Resource-constrained perception systems such as edge computing and vision-for-robotics require vision models to be both accurate and lightweight in computation and memory usage. Knowledge distillation is one effective strategy to improve the performance of lightweight classification models, but it is less well-explored for structured outputs such as object detection and instance segmentation, where the variable number of outputs and complex internal network modules complicate the distillation. In this paper, we propose a simple yet surprisingly effective sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teachers to a given lightweight student. Our approach is inspired by curriculum learning: To distill knowledge from a highly accurate but complex teacher model, we construct a sequence of teachers to help the student gradually adapt. Our progressive distillation strategy can be easily combined with existing distillation mechanisms to consistently maximize student performance in various settings. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students, and unprecedentedly boost the performance of ResNet-50 based RetinaNet from 36.5% to 42.0% AP and Mask R-CNN from 38.2% to 42.5% AP on the MS COCO benchmark.

Under review as a conference paper at ICLR 2023
LEARNING LIGHTWEIGHT OBJECT DETECTORS VIA PROGRESSIVE KNOWLEDGE DISTILLATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Resource-constrainee perception systems such as edge computing and vision-for- robotics require vision models to be both accurate and lightweight in computation and memory usage. Knowledge distillation is one effective strategy to improve the performance of lightweight classification models, but less well-explored for structured outputs such as object detection and instance segmentation, where the variable number of outputs and complex internal network modules complicate the distillation. In this paper, we propose simple yet surprisingly effective sequen- tial approach to knowledge distillation that progressively transfers the knowledge of set of teachers curriculum learning: To distill knowledge from a highly accurate but complex teacher model, we construct sequence of teachers to help the student gradually adapt. Our progressive distillation strategy can be easily combined with existing distillation mechanisms settings. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-base dents, and unprecedentedly boost the performance of ResNet-50 based RetinaNet from 36.5% 42.0% AP and Mask R-CNN from 38. 2% to 42.5 % AP on the MS COCO benchmark.
given lightweight student. Our approach is inspired by
consistently maximize student performance in various
teacher detectors to convolution-baal
stu-
INTRODUCTION
The success of recent deep neural network models generally depends on an elaborate design of architectures with tens or hundreds of millions of model parameters. However, their huge computa- tional complexity and massive memory/storage requirements make them challenging to deployed in safety-critical real-time applications, especially on devices with limited resources, such as self- driving cars or virtual/augmenntdd reality models. Such concerns have spawned wide body of literature compression and acceleration techniques. Many approaches focus on reducing com- putation demands by sparsifying/prunina quantization (Rastegari et al., 2016; Wu et al., 2016), neural architecture search (Zoph & Le, 2017; Liu et al., 2019), but reduced computation does not always translate into lower latency because of subtle issues with memory access and caching GPUs (Tan et al., 2019; Ding al., 2021).
networks (Lebedev Lempitsky, 2016: Han al., 2016),
Rather than searching over new architectures, we seek to better train existing lightweight architec- tures that have already been carefully engineered for efficient memory access. Instead of relying on additional data or human supervision, we follow the large body of work on knowledge distilla- tion (Buciluă al., 2006; Hinton al., 2014) for compressing the information from large model into a small model. While most recent efforts in knowledge distillation focus on image classifica- tion. relatively less work exists for distilling object detectors. The extension from classification to object detection and instance segmentation is nontrivial due to the complicated outputs of the tasks. Most detectors operate with multi-task heads (for classification, and box/mask regression) that can generate variable-length outputs. In the literature of detector distillation, recent work (Zhang & Ma, 2021; Shu et al., 2021; Yang et al., 2022b) mainly focuses on designing advanced distillation loss functions for transferring features from teachers to students. However, there are two unsolved chal- lenges: 1) The capacity gap (Cho & Hariharan, 2019; Mirzadeh et al., 2020) between models can result in sub-optimal distilled student even if the strongest teacher has been employed. which is undesired when optimizing the accuracy-efficiency trade-off of the student. Moreover, when trying to distill knowledge from Transformer-based teachers (Dosovitskiy et al., 2020; Liu al., 2021)

---

Under review as a conference paper at ICLR 2023
ginell convelutional
44
Directly
Figure 1: Our proposed progressive distilla- tion leads state-of-the-arl student perfor- mance. When switching the teacher model from a convolution- based detector to Transformer bassoreerr---ddddddd based with stronger detection performance, the stu- dent does not become accurate, due the chitectural difference hetween the teacher-student pair. Progressively distilling knowledge from mul- tiple teachers can mitigate the capacity gap and re- sult in the best student performance.
Directly distiled Tharsformer teacher
Progressteely disilled by Trandormer teachers
StudartAP HS
60
36-
RetinaNecRRsseett55 S0
Mask R. CNNRRaNNet
classical convolution- based students, this architectural difference can enlarge the teacher-studed student gap. 2) Current methods assume that one target teacher has been selected. However, this meta-level optimization of teacher selection is neglected in the existing literature detector distillation. In fact, finding pool of strong teacher candidates is easy, but trial-and-error may be necessary before determining one most compatible teacher for a specific student.
To address these challenges, we propose framework to learn lightweight detectors through progres- sive knowledge distillation: 1) We find sequential distillation from multiple teachers arranged into curriculum significantly improves knowledge distillation and bridges the teacher-student capac- ity gap. As shown Figure 1, even with huge architectural difference, our progressive distillation effectively transfer knowledge from Transformer based teachers convolution-basse students, while previous methods cannot. 2) For the teacher selection problem, we design a heuristic algo- rithm for given student and a pool of teacher candidates, automatically determine the order of teachers to use. This algorithm is based on the analysis of the representation similarity between models, which does not require knowledge of the specific distillation mechanism be used.
Overall, our progressive distillation is a general meta-level strategy that consistently improves both simple maathrg distillation and more sophisticated ones. With the help of modern distil- lation mechanisms and teacher detectors, our progressive distillation learns lightweight RetinaNet and Mask R-CNN students with state-of-the-art accuracy. Furthermore, by analyzing the training loss dynamics of the student model, we find the improvement is not due to minimizing the training loss better. Rather, the knowledge transferred from multiple teachers can lead the student flat minimum, and thus help the student generalize better. To summarize, transfer knowledge from multiple teachers to progressively distill student. Our main contributions include:
We propose framework for learning lightweight detectors through progressive knowledge dis- tillation, which is simple, general, yet effective. We develop a principled way to automatically design sequence of teachers appropriate for given student and progressively distill it.
Our progressive distillation is a meta-level strategy that can be easily combined with previous ef- forts in detection distillation. We perform comprehensive empirical evaluation the challenging
MS COCO dataset and observe consistent gains.
For the first time, investigate distillation from Transformer-based teacher detectors to convolution- based student, and find progressive distillation is the key to bridge their gap. We show the performance gain comes from better generalization rather than better optimization.
RELATED WORK
Knowledge distillation or transfer, an idea of training shallow student network with supervision from a deep teacher, was originally proposed by Buciluă al. (2006), and later formally popularized by Hinton et al. (2014). Different knowledge can be used, such as response-based knowledge (Hinton al., 2014), and feature-based knowledge (Romero et al., 2015; Heo et al., 2019). Several multi-teacher knowledge distillation methods have been pro- posed (Vongkulbhisal et al., 2019; Sau & Balasubramanian, 2016), which usually use the average of logits and feature representations as the knowledge (You et al., 2017; Fukuda et al., 2017). Mirzadeh et al. (2020) find that an intermediate teacher assistant, decided by architectural similarities, can bridge the gap between the student and the teacher. We find it more effective to sequence of teachers instead of their ensemble, and extend Mirzadeh et al. (2020) to a more general case where teacher models have diverse architectures and their relative ordering is unknown.
Knowledge Distillation:
2

---

Under review as a conference paper at ICLR 2023
Object Detection and Instance Segmentation: A variety of convolutional neural network (CNN) based object detection frameworks have been proposed, and can be generally divided into single- stage and two-stage detectors. Typical single-stage methods include YOLO (Redmon et al., 2016; Redmon & Farhadi, 2018) and RetinaNet (Lin et al., 2017h), and two-stage methods include Faster R-CNN (Ren et al., 2014), and Mask R-CNN (He al., 2017). Recently, several multi-stage models are proposed, such as HTC (Chen et al., 2019a) and DetectoRS (Qiao et al., 2021). These detection frameworks achieve better detection accuracy with better backbone networks feature extractors and with more complicated heads, which are more computationally expensive.
Knowledge Distillation for detection and segmentation: To reduce the computational cost, knowl- edge distillation has been used to develop efficient detectors. Chen et al. (2017) first use knowledge distillation enforce the student detector to mimic the teacher's predictions. More recent efforts usually focus on learning from the teacher's features, rather than final predictions. Various distil- lation mechanisms have been proposed to leverage the impact of foreground and background ob- jects (Wang et al., 2019; Guo et al., 2021), relation between individual objects (Zhang & Ma, 2021; Dai et al., 1021), or relation between local and global information (Yang et al., 2022a;b), Different from these methods that distill from single teacher, study distillation from multiple teachers, where a proper sequence of teachers is required, and we find a very simple feature- Teature-matching loss is adequate to significantly boost student performance.
APPROACH
We propose to progressively distill student model S with a pool of N teachers P = [T: Typical object detectors are composed of four modules: (1) backbone, which extracts visual features, such ResNet (He et al., 2016) and ResNeXt (Xie al., 2017); (2) neck, which extracts multi-level feature maps from various stages of the backbone, such as FPN (Lin et al., 2017a) and Bi-FPN (Tan et al., 2020); (3) optional region proposal network (RPN). which is used in two-stage detectors; and (4) head, which generates final predictions for object detection and segmentation. We denote the output feature maps of the neck FNNtt where Net can be either the student model S or one of the teachers 7, E P. With neck modules like FPN, the feature maps can be multi-level.
We propose meta-strategy for detector distillation that progressively learns student using a quence of teachers. To examine this meta-strategy without involving sophisticated distillation mech- anisms, we introduce a simple feature-matching distillation for single teacher T3 in Section 3.1. Then we discuss progressive distillation with multiple teachers from P in Section 3.2.
3.1 SINGLE TEACHER DISTILLATION VIA SIMPLE FEATURE MATCHING
In order to learn a efficient student detector S through distillation, we encourage the feature repre- sentation of student be similar to that of the teacher (Chen et al., 2017; Yang al., 2021). To this end, we minimize the discrepancy between the feature representations of the teacher and the student. Without bells and whistles, we simply minimize the 1.2 distance between Fl, and Fs:
Laistll = =| r ||
(1)
where r(.) is function used to match the feature map dimensions of the teacher and the student.
We define r(-) as follows:
(Homogeneous case) If the numbers of channels and the spatial resolutions are both the same between T. and S, r--) an identity function.
(Heterogeneous case) If the numbers of channels are different but the spatial resolutions are the same, we use x convolutional filters as r(-). the spatial resolutions are different but the numbers of channels are the same, we use upsampling layer r(-).
Note that the mapping r(-) is only required at training time and thus not adding any overhead to the inference. Overall, our loss function can be written as:
XLdisill + Lietects
where À is balancing hyper-parameter and Tatetea is the detection loss based the ground truth labels. Compared to state-of-the-ar detection distillation approaches (Zhang & Ma, 2021; Shu al., 2021; Yang et al., 2022a;b), which introduce more complex designs of the distillation loss, this feature-matching distillation is simpler and does not require running the heads of the teacher model. Our distillation loss is illustrated in Figure LeLeft.
3

---

Under review as a conference paper at ICLR 2023
Distillation #1
Distillation #2
Toucher
Ticklono
Neck
Coood Touth
Teacher
Teacher
Studeat
Nock
Head
Prodictions
Stodent
Student 1
Distiled
Studeut 2
Ditilled
For each teacher-student pair, the training target is composed of two parts: Laisn minimizes the discrepancy between the neck feature maps of the student and the current teacher, and Liene is the original detection loss based on the ground truth. Right: We sequence of teacher models distill the lightweight student detector. The sequence of teachers forms curriculum. Using a proper sequence teachers can significantly boost the student model's performance. The example performance curve illustrates our method improves the COCO validation AP of ResNet-50 backboned RetinaNet student first from 36.5% to 37.9% using HTC (Teacher 1), and then from 37.9% to 39.9% using DetectoRS (Teacher 2).
Figure 2: Progressive knowledge distillation for object detectors. Left:
3.2 PROGRESSIVE DISTILLATION WITH MULTIPLE TEACHERS
The overall aim of knowledge distillation is to make a student mimic a teacher's output, so that the student able obtain similar performance to teacher's. However, the capacity of the student model is limited, making it hard for the student to learn from a highly complex teacher (Cho & Hariharan, 2019). To address this issue, multiple teacher networks are used to provide more supervision to a student (Sau & Balasubramanian, 2016; You et al., 2017). Unlike previous methods which distill knowledge from the ensemble of logits or features simultaneously, we propose to distill feature- based knowledge from multiple teachers sequentially. Our key insight is that instead of mimicking the ensemble of all feature information together, the student can be distilled more effectively by the knowledge provided by one proper teacher each time. This progressive knowledge distillation approach can be considered as designing a curriculum (Bengio et al., 2009) offered by a sequence of teachers, illustrated in Figure 2-Right.
The crucial question is: What is the proper order O of the teachers when distilling the student? A frute-force approach might search over all orders and pick the best (that produces distilled student with the highest validation accuracy). However, the space of permutation orders grows exponentially with the number of teachers, making this impractical to scale. Therefore, we propose principled and efficient approach based on a correlation analysis of each model's 's learned feature representation.
First, we quantify the dissimilarity between each pair of models' representations, as a proxy for the capacity gap between them. Representation (dis) similaity (Raghu et al., 2017: Wang et al., 2018; Kornblith et al., 2019) has been studied to understand the learning capacity of neural models. In our setting, we find linear regression model is adequate for measuring the representation dissimilarity. Given two pre-trained detectors A and B, we can freeze their parameters, and thus fixing the feature representations. Then we can learn a linear mapping r()), implemented by a 1 1x convolutional layer at each feature level, as specified in the heterogeneous case in Section 3. r(-) is trained to minimize Ldisin. so it can transform A's features to approximate B's features. After training r(-), we evaluate it by Lisstill on the validation set, and denote the validation loss as the adaptation cost C(A, B). This metric can be proxy of the capacity gap between pair of models: When C(A, B) is zero, linear mapping can transform A's features to B's, and there is no additional knowledge from B. When C(A, B) large, it is more difficult adapt A's representation to B's. Note that the adaptation cost is non-symmetric - it is relatively easier to adapt a high-capacity model's representations to a low-capacity model's representations, than the other way around.
We design a heuristic algorithm to acquire a proper distillation order O automatically (details are shown in Algorithm in the appendix). Suppose the maximum number of teachers to be selected is limited by & (which can be arbitrarily decided according to desired training time), and we aim to find a teacher index sequence a no longer than k. We construct the teacher order backwards: The best performing teacher is set as the final target Ta,; before the final teacher, we use another teacher, which has the smallest adaptation cost C(,,Ta.) to that final teacher, as the penultimate teacher To, 1- We repeat this procedure to find preceding teachers, until: (1) when trying to select Tay, we find the transfer costs from remaining teachers to the next teacher C(., .) are all larger than the transfer cost from the student to the next teacher C(S, Tay+ ); or (2) we reach the given maximum step limit k. Intuitively, the resulting sequence of teachers bridges the gap between the student model and the teacher, with increasingly difficult curriculum.
4

---

Under review as a conference paper at ICLR 2023
Tj
0:963
Moxid
Taput Res.
Bacihone
Nack
Head
Box
AP
Mask
Rustine (ms))
0.930
0,934
Teachers
Redtandek Ty
1x
R50
EPN
Mask CNN
38.2
34.7
51
1.181
0.070
0.80
T
RTC
42.3
DesecioS
R50-SAC
HTC (Detectoo..)
49.1
42.6
0.060
1.94
R50-NAC
R-UNNN
45.1
142
T 2t
L-201
Studears
1x
R:50
FPN
RetinaNes
36.5
41
1x
R.50
FPN
Mask R-CNN
38.2
54.7
51
1x
R18
I9N
Mask I CNN
33.3
50.5
29
Figure 3: Adaptation costs among models. The number on each directed edge the adaptation cost metric described Scction 3.2. Some edges are not shown for visual clarity. The red path is suggested by our proposed Algorithm 1 when & = 3 teachers are selected: (1) use the best performing Teacher IV the final teacher in the sequence, (2) use the teacher closest to Teacher IV, which is Teacher III, as the sec- ond teacher, and (3) use the teacher closest to Teacher III. which is Teacher
IV
0.20
K30
19N
Mask CNN
25.8
23.0
17
Table 1: Configuration and COCO performance of the student and teacher defectors. We inves- tigate a variety of models with heterogeneous in- put resolutions, backbones, necks, and head struc- tures. 'Ix' input resolution refers to the standard 1333 x 800 resolution, and "0.25x" means 333 x 200 resolution. 'R-" backbones are ResNets with different number of layers.
the first teacher.
Our algorithm for designing teacher orders is lightweight. In fact, the main computation overhead of our algorithm is to optimize set of tiny linear mappings (R2266 + 2256 for -ased detectors). takes about GPU hours for each student model, which negligible compared to the distillation process that takes hundreds of GPU hours.
Since our progressive knowledge distillation a meta-level strategy, it can be combined with previ- ous designs of distillation mechanisms, without much efforts. Starting with student detector and pool of candidate teachers, we can first select a subset of teachers and design their distillation order. In place of the simple feature matching loss, we then apply more advanced distillation mechanism with each teacher sequentially to train the student detector.
EXPERIMENTS
We study the efficacy of our proposed strategy, progressive knowledge distillation, from multiple perspectives. First of all in Section 4.1, we use a controlled experiment to demonstrate that our heuristic Algorithm 1 consistently produces teacher orders that are near-optimal compared to all possibilities. Then in Section 4.2 and 4.3, we apply the progressive distillation strategy along with the simple feature-matching loss (Section 3.1) 1) to show this strategy alone brings significant gains to knowledge distillation. Since our contribution of progressive distillation is orthogonal to previous efforts in designing distillation mechanisms, in Section 4.4 we then combine it with state-of-the- art distillation mechanisms distillation is the key to the success of distillation from Transformer- teachers to convolution- based students. Finally in Section 4.5, try understand the performance gain of progressive distillation by analyzing the training loss dynamics.
maximize the student performance, and we show our progressive
Student and teacher models: nations, as shown in Table 1, we construct a variety of teacher-studens pairs from a set of widely- object detection and instance segmentation networks, including RetinaNet (Lin et al., 2017b). Mask R-CNN (He et al., 2017), FCOS (Tian et al., 2019), HTC (Chen et al., 2019a), and DetectoRS (Qiao al., 2021). They have wide range of runtime and detection performance. We select ResNet-50 backboned RetinaNet and Mask R-CNN as the student models (Student I & II), due to their low latency, simple structure, and wide application, for single-stage and two-stage object detection re- spectively. More advanced models such as DetectoRS have better detection performance, but require much more training/inference time, so we use them as teachers. We mainly consider the RetinaNet and Mask R-C R-CNN as the student models, and lightweight variants with smaller backbone (Student III), or reduced input resolution (Student IV).
To investigate the impact of different teacher models and their combi-
Datasets and evaluation metries: We mainly evaluate on the challenging object detection dataset MS COCO 2017 (Lin et al., 2014), which contains bounding boxes and instance segmentations for 80 common object categories. We train our models on the split of train2017 (118k images) and report results on va12017 7 (5k images). We report the standard COCO-style Average Precision (AP) metric and end-to-end latency (from images to predictions) as the runtime. We also evaluate
5

---

Under review as a conference paper at ICLR 2023
Our teacher onders
Suggested teacher order
Student AP
student AP range
Ranking all orders
37.6
IV
36.7
[36.2. 36.8]
2/4
III-IV
20.6
I-III-IV
37.9
[36.2, 38.01
2/40
36.4
1 III -V
37.9
[36.2. 38.21
7/64
36,0
Table 2: Comparison of teacher order suggested by Algorithm 1 with all other orders under limited training budgets (Li et al., 2020b). & denotes the maximum number of used teachers. Left: We show some statistics of possible student AP performance and the ranking of the student using our distillation order. Right: We visualize the comparative advantage of our teacher orders (red dois) over all other orders (black dots). Some black scatter points overlap due the same student AP. Our proposed Algorithm highly competitive distillation orders of teachers.
consistently produce
on another object detection dataset Argoverse- (Chang et al., 2019), and a more challenging evaluation protocol streaming perception (Li et al., 2020a). These results are in Appendix D.
Baselines: Our main contribution is orthogonal to previous methods: We leverage a sequence of teachers to distill the student, instead of designing a sophisticated distillation loss to better transfer knowledge from onc single teacher. Since we are studying a new setting where multiple teachers are available, which is missing in previous literature, we mainly focus on the absolute improvements the performance our distilled student models compared with the original student models and with the performance upper-bound of the teacher models. We find using a sequence of teachers, instead of their ensemble, is more effective. Due to limited space, we leave this comparison in Appendix B&C.
4.1 SEARCHING FOR THE NEAR-OPTIMAL TEACHER ORDER
As we have discussed in Section 3.2, finding the optimal order of teachers for the progressive knowl- edge distillation takes factorial time complexity. To acquire near-optimal teacher order, we propose the heuristic Algorithm 1. In this section, we will validate that this algorithm is near-optimal. To achieve this comprehensive comparison, we distill Student with all orders of teachers from the pool Teacher I-IV. We use a reduced training budget: For each teacher, we only train the student for epochs on MS COCO. We use the linear learning rate schedule, which has been shown comparably effective in limited budget setting by Li et al. (2020b).
We first measure the adaptation costs among the student and teacher models. A visualization of the cost graph is shown in Figure 3. Following Algorithm 1, we can construct a sequence of teachers. We compare the teacher orders given by our proposed algorithm against all other orders, via the performance of the distilled student's performance. As shown in Table 2, teacher orders suggested by Algorithm are consistently near-optimal in this setting. In the following sections, we will use order provided by Algorithm 1, without brute-force iterating over all possible orders. One may question that the greedy path selection shown in Figure 3 is be inferior to a global optimization algorithm. However, we need to greedily select teachers from the sequence tail. More details and comparison with other heuristics provided in Appendix A.
find the later teachers impact the student performance more profoundly, so
We start by distilling RetinaNet and Mask R-CNN with a ResNet-50 backbone (Student I & II). Here we consider homogeneous teachers where the numbers of channels and the spatial resolutions of feature maps are consistent between the student and teacher. For the RetinaNet student, we still consider the pool of Teacher I-IV, the same as Section 1. For the Mask R-CNN student, we should no longer use Teacher (the student itself) or Teacher II (the single-stage teacher does not outperform the student by a large margin). To compensate for that, we include Teacher V, which can be considered as a hybrid model of DetectoRS backbone/neck and Mask R-CNN head. Thus, the teacher pool for Mask R-CNN includes Teacher III-V. To control the total training time, we limit the number of teachers to be 2. Thus, initialize from an off-the-shelf ('OTS") student, and sequentially distill with 2 teachers, in total 24 epochs (equivalent to a 2x training schedule). Besides the OTS students, we also compare with two other baselines: 1) students trained with longer 3x training schedule, and 2) students directly distilled by the final target teacher, using a 2x training schedule. More architectural details are listed in Table 1.
Following Section 4.1, we use Algorithm to determine the sequence of teachers to use for each student. For RetinaNet student, our algorithm suggests teacher sequence III-+IV. For Mask R-

---

Under review as a conference paper at ICLR 2023
Model
APts
APIs
APs
APM
APr
APw
APIs
APS
APM
APL
OTS
36.5
55.4
39.1
20.4
40.3
48.1
RetinaNerr (Studeet D)
Longer ds: training schedule
59.5
58.8
42.1
23.8
43.2
501.
Diroelly disiilldd by Teacher TV
59.5
58.6
41.9
21.0
42.8
$4.0
Progressively distilled by Teachers III -1V
39.9
59.2
42.7
21.7
43.3
$4.1
OTS
36.2
58.8
41.4
31.9
40.9
49.5
54.7
55.7
57.3
18.3
37.
47.2
Mask R CNN
Longer training schedule
64.8
24.4
52.3
58.3
39.9
18.4
39.8
51.9
Direcily distilled Teacher
45.0
23.5
30.8
Progressively distilled by Teachers V->1V
45.1
23.3
39.8
19.4
Table 3: Homogeneous distillation of COCO detectors, where students with ResNet-50 backbones are dis- tilled with teachers with ResNet-50 backbones. We report the detection ('Box") and segmentation ('Mask") APs, and we compare our distilled student with off tff-thh-sheff ('OTS") student, longer trained student, and the state-of-the-ar distillation baselines. Our distilled student significantly improves the detection AP over the 'OTS' student by 3.4% for RetinaNet and 3.2% for Mask R-CNN, and outperforms the baselines.
CNN student, our algorithm suggests teacher sequence V-IV. Table 3 shows the results on COCO. Additional results, analysis, and ablation studies of Mask R-CNN distillation are in Appendix B.
4.2 DISTILLATION WITH HOMOGENEOUS TEACHERS
Overall performance: Our distilled student models (row 4&8) significantly improves over the 'OTS' students (row 1&5). The box AP of RetinaNet is improved from 36.5% to 39.9% (+3.4%). The box AP of Mask R-CNN is improved from 38. 2% to 41.4% (+3.2% and the mask AP of Mask R-CNN is improved from 34.7% to 37.3% (+2.6%). After progressive distillation, our resulting Mask R-CNN detector has comparable performance with HTC teacher, but much less runtime (51
vs. 181ms).
Comparison with baselines: First, the performance gain is not merely from a longer training sched- ule. Our distilled student models (row 4&8) consistently outperform original students trained with 3x schedule (row 2&6). Second, progressive distillation using a curriculum of teachers (row 4&8) better than direct distillation from a strong teacher (row 3&7), even if the total training time is the same. It is worth noting that our detection performance for large objects receives the most gain (about 6% APL improvement for both models). The reason why we emphasize APL is that, in an efficiency-centric real-world application (e.g. autonomous driving, robot navigation), detecting nearby larger objects is more crucial than others. From realistic perspective, better AP,, shows better applicability of our approach.
4.3 DISTILLATION WITH HETEROGENEOUS TEACHERS
To validate our progressive distillation approach is general, we now consider a more challenging heterogeneous scenario, where students and teachers have different backbones or input resolutions. Specifically, Student III, ResNet-18 18 Mask R-CNN, is distilled with ResNet-50 teachers; Student IV, a model with reduced input resolution, is distilled with teachers trained with larger input resolu- tions. The results summarized in Table 4, and additional results included in Appendix C.
Heterogeneous backbones: Student III has a ResNet- 18 backbone and about half runtime as its ResNet-50 counterpart (Teacher 1). We find the proper distillation scheme for Student III is to the sequence of Teacher I->V V-IV, which significantly improves Student III over the 'OTS' model. The box AP of Student III is improved from 33.3% to 37.0% (+3.7%), and especially for large objects, APt is improved from 43.6% to 50.0% (+6.4%).
Heterogeneous input resolutions: Although inputs with varying resolutions can be fed into most object detectors without changing the architecture, the performance often degenerates when there resolution mismatch between training and evaluation (Tan al., 2020; Li al., 2020a). If ul- timately we want to apply detector to low-resolution inputs for fast inference, it is better to use low-resolution inputs during training. On the other hand, we conjecture that teachers with high- resolution inputs may provide finer details that can assist the student. With our progressive distilla- tion approach, we investigate the improvement of a low-resolution student distilled by a sequence of teachers with high-resolution inputs. We denote the standard input resolution 1333 x 800 as 1x, and a reduced resolution 333 x 200 as 0.25x. We distill Student IV (with 0.25x resolution) by sequence of Teacher I variants (0.5x 0.75x 1x). From Table 4, we can substantial improvement brought by progressive knowledge distillation: the box AP is improved from 25.8% to 31.5% (+5.7%) and the mask AP is improved from 23.0% to 28. 2% (+5.2%).

---

Under review as a conference paper at ICLR 2023
Model
Backbone
Resolution
Box
AP
Mask
Model
Distilation
Box
Mask
1
Snudent III, OTS
R18
1x
53.3
30.5
RetinaNet
Direct RetinaNet//Sin-S
41.0
2
Suudent III, Our distilled
R18
1x
37.0
33.7
(Student D)
Progresaive RedinaNetiSwia-T
-8
42,0
3
Student IV, OTS
R50
0,25%
25.8
23.0
Mask R-CNN
Direct MRCKNSSwin-
42.0
37.7
Student Our distilled
R50
0.25x
28.2
(Student
Progresáve MRCNNSSia-T--
Heterogeneous distillation of COCO detectors, where students with smaller backbones (ResNet-18 vs. ResNet-50) or input resolutions (333 x 200 vs. 1333 x 800) are distilled with hetero- gencous teachers, requiring additional transfer logic (Sec. 3.1). We report the detection ("Box'), segmen- tation ("Mask') ) APs and runtime, and compare our distilled student with its teachers (see Table 1) and off-the-shelf ( ('OTS') student. Our progressive distil- lation significantly improves the 'OTS' students by over 3% AP.
Table 4:
Table 5: Distillation from Transformer-based teachers (Liu al., 2021) to convolution- -baed students. Due to the architectural difference and ca- pacity gap. directly distilling from stronger teacher with Swin-S backbone does not yield better students than convolution based teachers Figure 4. in- termediate Swin-T teacher and progressive distilla- fion solve this issue without increasing training time, Compared to off-the-shelf models, our RetinaNet and Mask R-CNN students improve by 5.5% AP and 4. 4.3% box AP, respectively.
4.4 COMBINATION WITH OFATE-OF-THE-AR DISTILLATION MECHANISMS
Our meta-leve strategy of using sequence of teachers to progressively distill student is indepen- dent of choice of the distillation mechanism for each teacher. We have shown progressive distilla- tion can boost the simple distillation based on feature matching above, and in this section, we will combine progressive distillation with sfate-of-the-art distillation mechanisms for object detection to further improve student accuracy.
We evaluate our progressive distillation with three most recent works on de- tector distillation: CWD (Shu et al., 2021). FGD (Yang et al., 2022a), and MGD (Yang et al., 2022b). To ensure fair comparison, we use the same teacher-student pairs as them: RetinaNet/ResNet-550 and RetinaNee/ResNeettt 101 are the single-stage student and final teacher. Mask R- Nett and Cascade Mask R- CeCNNNeesNett--I CNNRRsNeex--OO-DDCC 1O1- are the two-stage student and final teacher. Between them, insert one medium-capacit, teacher progressively distill the student: RetinaNet/RRsNet- 101 for single-stage and Cascade Mask R-CNN/ResNet5O-DDCC for two-stage. Also for fairness, we keep the total training epochs the same. We set "1x" training schedule for each teacher, that the total training time is equivalent to "2x", the same as in previous works.
Distillation protocol:
Figure 4 shows our progressive distillation strategy consistently improves students' final accu- racy. For example, the performance of FGD-distilled RetinaNed/ResNet5500 improves from 40.7% to 41.5% AP (+0.8%), and this gain is larger than mechanism advance from FGD MGD (+0.3%). We bring performance gains to state-of- ttate-of-the-ass thethhe-ttt detection distillation almost for free.
Next, we investigate how further maximize the student performance. Swin Transformer (Liu et al,, 2021) can act as an even stronger teacher than the convolution-based teachers used in previous works. However compared to convolution-bass improve the student performance, even if we use the -fth--artt method MGD. For example, RetinaNet/Swin-Small (47.1% AP) is much stronger than direct distillation from both yields the same student performance (41.0% AP). To bridge the archi- tectural difference and capacity gap between the ResNet-50 student and Swin-Small teacher, we can utilize an intermediate Swin-Ting teacher. As shown in Table 5, this progressive approach brings the best students: the performance of ResNet-50 based RetinaNet increases to 42.0% AP and Mask R-CNN increases to 42.5% AP.
teachers, directly distill from such a teacher cannot
AP), but
4.5 UNPACKING THE PERFORMANCE GAIN: GENERALIZATION OR OPTIMIZATION?
We have shown that our distilled student significantly improves the accuracy on the validation data over the off-the-shelf student. As further demonstrated in Figure 5a, the validation accuracy of the distilled student gradually increases during distillation, and achieves higher value compared with the student trained without teachers. A natural question then arises - why is distillation helping? There are two possible hypotheses: (1) improved optimization: distillation facilitates the optimiza- tion procedure, leading to a better local minimum, and (2) improved generalization: the distillation process helps the student generalize unseen data.
Improved optimization is typically manifested through a better model, a lower training loss and higher validation accuracy, which is exactly the case for Mask R-CNN, HTC and DetectoRS.
8

---

Under review as a conference paper at ICLR 2023
PANC S-OAARP Shuteer Mark
63.0
62.0
28.0
00.5
61.5
37.5
Bass Iis Direct dal aillucoo
Buse INc. rust darl at on Outs: Poggessrr d64 8-Bin
Bisel ne: Drickt dld Rabion OUSS: Prognsshh dol Jallon
Outsi Pagrosdee goole did 68
29.5
MGD BOOV 2022
40.5
CVD ICV 2521
RGD CXPA
NGD ECOW 2072
36.5
CAR
EGD CWPR 2072
MGD ECOV 2022
2021
CUPA
ICDV 2023
Figure 4: Our progressive distillation strategy consistently benefits state-of-the-art distillation mech- anisms. Using an intermediate RetinaNe//ReNNe-1001 teacher between RetinaNet/ResNet-300 student and RetinaNadResNeXi-DDe CNN/ResNet-500 and Cascade Mask ACCNN/esNeex-101-D2 AP), improve the direct distillation baselines by 0.2% to 0.8% AP, without increasing training time,
teacher (Left), or Cascade Mask R-CNN/ResNetSO-DCN between Mask R-
(Middle for Box AP and Right for Mask
2:00
G3-2 No taacher la
seacher
0.5
V2l00b 40
99 Clidi teacher
125
609
669
(A) No laadiar 3x
seadher by
250-
Dutliut Desudatfi teaderr
325
1dk
llaratona
200c
+1.0
-15
-05
0. +100
-15
-23
(a)
(b)
(c)
(d)
Figure 5: Comparisons of student models trained with and without teachers. We train ResNet-50 back- honed RetinaNet (Student 1) with: (A) prolonged 3x training schedule (curves in blue); (B) progressive knowledge distillation from HTC (Teacher III) and then DetectoRS (Teacher IV) (curves in orange-green-red) We compare the validation AP (Figure 5a) and the training detection loss Tamaa (Figure Sh) of the two students during the training process. Despite worse training loss, distilled student can generalize better on validation set. We also compare the loss landscapes (Li et al., 2018) of the original student (Figure Sc) and the distilled student (Figure 5d). Distillation can guide the student to converge to flatter local minimum. These observations suggest distillation helps generalization rather than optimization.
Consequently, one might think that distillation works in the same way. However, our investigation suggests the opposite our progressive distillation increases both the validation accuracy and the training loss, and therefore effectively reduces the generalization gap. In Figure 5, we compare the original RetinaNet model and the distilled student, which have the same architecture, the same latency and are trained on the same data, but with different supervision (only ground -trnt labels vs. additional knowledge distillation). To eliminate the influence of learning rate changes, we train the original student with a 3x schedule and restart the learning rate at the same time with the distilled student. Interestingly, although distillation can improve the student's validation performance, the training detection loss of the distilled student is higher than the original student. This suggests that distillation does not help the optimization process find a local minimum with a lower training loss, but rather strengthen the generalizability of the student model.
To further support this observation, we also visualize the local loss landscape (Li et al., 2018). The distilled student has flatter loss landscape (Figure 5d) compared to the original one (Figure 5c). As widely believed the machine learning literature, flat minima lead to better generalization (Hochre- iter & Schmidhuber, 1997; Keskar et al., 2017). The observation shown in Figure is illustrated for RetinaNet, but we also have similar observation in other students. As conclusion, knowledge dis- tillation, which enforces the student to mimic the teachers' features, can be considered an implicit regularization, and helps the student combat overfitting and achieve better generalization.
CONCLUSION
We present a simple yet effective approach to knowledge distillation, which progressively transfers the knowledge of sequence of teachers to lear a lightweight object detector. Our approach auto- matically arranges multiple teachers into a curriculum, and thus effectively mitigating the capacity gap between the teacher and student. We successfully distill knowledge from Transformer-basee teachers to convolution based students, and achieve state-of-the- performance on the challenging COCO dataset. Our analysis also finds distillation improves generalization rather than optimization.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. CVPR, 2019. 21
Khalid Ashraf, Bichen Wu, Forrest landola, Mattthew W Moskewicz, and Kurt Keutzer. Shallow networks for high-accuracy road object-detection. arXiv preprint arXiv: 10666666666111 015,1, 2016. 18
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In ICML, 2009.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006. 2
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018. 20
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hart- nett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019. 6, 19
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In NeurIPS, 2017. 3. 20
Kai Chen, Jiangmiao Pang. Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, 2019a. 5,5
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 2019b. 23
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for object detection. In CVPR, 2021. 3, 20
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg:
Making vgg-style convnets great again. In ECCV, 2021. 1
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ram- abhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, 2017.
Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Dis- tilling object detectors via decoupled features. In CVPR, 2021. 3, 20
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo.
Online knowledge distillation via collaborative learning. In CVPR, 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 3,
10

---

Under review as a conference paper at ICLR 2023
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeuriPS Workshop, 2014. 1,2
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 1997. 9
Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng, Instance-conditional knowledge distillation for object detection. 2021. 23
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICMI,, 2019.
Xu Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In NeurlPS, 2018. 21
Vadim Lebedev and Victor Lempitsky. Fast convnets using group-wise brain damage. In CVPR, 2016.
Chengcheng Li, Zi Wang, and Hairong Qi. Online knowledge distillation by temporal-spatial boost- ing. In WACV, 2022. 21
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. NeurIPS, 2018.
Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. Towards streaming perception. In ECCV, 2020a. 6, 7, 19, 20
Mengtian Li, Ersin Yumer, and Deva Ramanan. Budgeted training: Rethinking deep neural network
training under resource constraints. In ICLR, 2020b. 6
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.
Feature pyramid networks for object detection. In CVPR, 2017a. 3
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object
detection. ICCV, 2017b. 3,5
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 1, 8, 23
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020. 1, 20, 21
Siyuan Qiao, Liang-Chias Chich Chen, and Alan Yuille. DetectoRS: Detecting objects with recursive feature pyramid and switchable atrous convolution. In CVPR, 2021. 5, 20
Maithra Raghu. Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NeurIPS, 2017.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-NET; Imagenet classification using binary convolutional neural networks. In ECCV, 2016.

---

Under review as a conference paper at ICLR 2023
Joseph Redmon and Ali Farhadi. arXiv:1804.02267, 2018. 3
Yolov3: An incremental improvement. arXiv preprint
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: real-time object detection. In CVPR, 2016. 3
Unified,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurIPS, 2014. 3, 20
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. ICLR, 2015. 2, 21
Bharat Bhusan Sau and Vineeth N Balasubramanian. Deep model compression: Distilling knowl- edge from noisy teachers. arXiv preprint arXiv:1610.09650, 2016. 2,4
Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise knowledge distillation for dense prediction. In ICCV, 2021. 3, 8, 21, 22,
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In CVPR, 2020. 3,
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In ICCV, 2019. 5
Jayakorn Vongkulbhisal, Phongtharin Vinayavekhin, and Marco Visentini-Scarzaeella. Unifying heterogeneous classifiers with distillation. In CVPR, 2019. 2
Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In NeurlPS, 2018.
Tao Wang. Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In CVPR, 2019. 3, 20
Jiaxiang Wu, Cong Leng. Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, 2016.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Chenglin Yang, Lingxi Xie, Chi Su, and Alan I. Yuille. Snapshot distillation: Teacher-student optimization in one generation. CVPR, 2019a.
Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In ICLRR 2021.
Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. In ICCV, 2019b. 22
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao and Chun Yuan. Focal and global knowledge distillation for detectors. In CVPR, 2022a. 8. 20, 21, 22, 23
Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked gener- ative distillation. In ECCV, 2022b. 1, 8, 20, 21, 22, 23
Anhang Yao and Dawei Sun. Knowledge transfer via dense cross-layer mutual-distillation. In ECCV, 2020. 21
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In KDD, 2017. 2, 2,4,2 21
12

---

Under review as a conference paper at ICLR 2023
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017. 21
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature- based knowledge distilla- tion: Towards accurate and efficient detectors. In ICLR, 2021. 1, 20
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. ICLR, 2017.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
We summarize the content of this supplementary document as follows. Section A includes additional results and analysis of our proposed algorithm for teacher order selection. Sections and C provide additional ablation study on distillation with homogeneous teachers and heterogeneous teachers, respectively. Section D shows the generalizability of our approach, which demonstrate the exper- imental results of the Argoverse-HD dataset with streaming accuracy metric. Section E compares our work with prior knowledge distillation methods in detail. Section F provides some additional experiments on combining our method with state-of-th of-the-art distillation mechanism. Section G lists our implementation details.
A MORE RESULTS ON SEARCHING FOR THE NEAR-OPTIMAL TEACHER ORDER
In this section, we show more detailed results about searching a proper teacher order for progressive knowledge distillation, and validate the approach we propose the main paper. As described in Section 3.2, we first quantify the adaptation cost C(---) between every pair of models in our pool, and then use a heuristic method (Algorithm 1) construct a sequence teachers. We have shown that the teacher order suggested by our algorithm is highly competitive in Table 2. One might think there should be better choices than a greedy algorithm on a directed graph, such as a shortest-path algorithm. To validate our algorithm design, we compare our Algorithm against several other algorithms.
Algorithm 1: Determining the Teacher Order
Input: Student model S, pool of teacher models P = [T] (/:), teacher models' performance
(Q(T,) I maximum number of selected teachers k
Output: Sequence of teachers O, len(O) S k
Pick the best performing teacher: Tay + arg max EP Q(T.). [Taz]
Exclude from pool: p P (Tac)
3 for j k --
to do
Get candidate sub-pool: Pj (Tu Tu E P,,((TT, Tast1) < C(S, Tasti)}
if P, * then Pick the teacher closest to Tagzi Tay & argminy,es C(Ta, Tayz1) Prepend Ta to
Exclude from pool: P < P\ (Tay)
else Break
return O
To begin with, include the detailed adaptation costs C(.,.) among RetinaNet (Student I) and its teachers (Teacher I-IV) Table 6. As described in Section 4.1, 1, we have distilled Student with all possible teacher orders in the pool, using reduced training budget of 3 epochs for each teacher. The results of these mini-budget distillation are summarized in Table 7.
Table Adaptation costs among Student (RetinaNet) Teacher I-IV (Mask R-CNN. FCOS, HTC, De- (ectoRS). The adaptation cost is computed pair-wise as described in Section 3.2 of the main paper. Using this metric we can construct directed graph, as illustrated in Figure
To
Student
Teacher I
Teacher
Teacher III
Teacher IV
From
Student
0.939
0.060
1.568
1.254
Teacher
0.183
0.070
0.934
0.963
Teacher II
0.339
1.181
1,940
1.401
Teacher III
0.191
0.484
0.082
0.890
Teacher IV
0.232
0.767
0.077
1.248
14

---

Under review as a conference paper at ICLR 2023
7:blee: Performance of Student (RetinaNet) distilled with different teacher sequences, under reduced training budgets. For each teacher sequence, student trained for epochs on COCO. After progressive knowledge distillation, the student evaluated on the COCO validation set. The teacher orders suggested by
Algorithm are marked bold.
Teacher Sequence
Student AP
Teacher Sequence
Student AP
Teacher Sequence
Student AP
Length
Length
Length
III
36.8
III->II> -V
38.0
III-III|
-V
38.2
36.7
I-IIII-IV
37.9
I1I---I1-
38.1
1
36.4
III-IV-Il
37.9
1-II--II-II
38.1
II
36.2
I[-III-IV
37.9
I[-III-I-IVV
38.0
III-I--IV
37.8
I->II--IV-
38.0
II--IV
37.6
I-II-=I
37.7
I-I--:II--II
37.9
V--II
37.3
I-IV-II
37.6
III---
37.9
II++I
37.3
IV-II-III
37.5
--II-II--IV
37.9
I-I
37.3
IVIIIIIIII
37.5
IV=I-->II-II
37.7
IV-IIII
37.2
I-I-II
37.5
III-+IV-
37.7
I-ll
37.1
I-1II-11
37.5
1-II-MV--I
37.7
2
IV-
37.0
IV-I-III
37.4
---II--III
37.6
3
4
I
37.0
I-IV-III
37.4
IV-I-II-III
37.6
II-I
37.0
-I-I--I
37.4
-I-VV-II- 37.6
1-1
36,9
III---II1
37.4
--V--IIIII 37.6
I-III 36.8
I-IV--III
37.4
I
37.6
I-I
36.8
IV-II-I
37.3
--V--II-III
37.6
IV-II--
37.3
|--I--I-III
37.5
IV-I-II
37.3
IV=III-:IIII
37.5
[-II--III
37.3
II-IV-IIIIII
37.5
--MV-I
37.2
-1-III-I--1
37.5
|---III
37.2
--V--IIIII
37.5
I---
37.2
II-I--II--
37.4
---I----
37.1
IV-II--II--
37.3
Table 8: Comparison of four algorithms for teacher order selection, in the mini-budget distillation setting. Our Algorithm I can consistently produce better teacher order than other algorithms.
Suggested teacher order
Studkent AP
Ranking in all
Suggested teacher order
Student AP
Ranking in all
Algorithm
Algoritlam
Shortest-palt (sum)
IV
36.7
2/4
Shertest-pati (sum)
1-1- I-II
37.5
9/
Forward construction
Shortest-path (max)
π
36.2
36.7
4/4
2/4
3
Sbortest-path (max)
Forward constraction
I1 ---IIII
I-III-IV
37.2
37.9
25 /0
/440
Our Algorithm
36.7
2/4
Our Algorithm
I-III-IV
37.9
2/40
Shortest-path (sum)
-V
37.0
7/
Shortest-path (sum)
11-1-111-11
37.9
7/
2
Shortest-pali (max]
I-VV
37.3
2/16
Sbertest-pati patth (max)
III--II-IV
37.9
7/
Forward construction
I-I
36.9
10/ 16
Forward construction
-II--II-
37.9
7/64
Algorithm
Algorithm
I-II-IV
37.9
7/64
Given the adaptation costs Table 6, we can construct directed graph, part which has been illustrated in Figure On the directed graph, our Algorithm 1, one may also propose these algorithms:
several algorithms to select path. Besides
Shortest-pa -aatessttpattttee (sum): Set the student as the source node, and the best performing teacher as the target node Tx,- Find path S + Th, + ... + Tx, that minimizes the sum of adaptation costs along the path:
miny, ..., C(S, (XS, プ C(TY, Tyyt.)
Shortest-path (max): Set the student as the source node, and set the best performing teacher as the
target node Tx,, Find a path S 5 Tú, +
Tx, that minimizes the maximum of adaptation
costs along the path: minT,, ....TA, max Tx),, C(T),, Txz)...., C(T.,,, T
Tx, }}
Forward construction: Contrary to Algorithm 1, we may start from the student and choose the nearest teacher from the current one, to construct the sequence:
Ty, arg minr.ep EP C(S,T.) T Ta
arg minr, C(T,,, T.).
15

---

Under review as a conference paper at ICLR 2023
The output teacher sequences and corresponding student performance of these three algorithms summarized in Table 8. In this setting, our Algorithm 1 can consistently produce competitive teacher order that leads to good performance of the distilled student. Compared to our Algorithm 1, shortest-path (max) can achieve a similar performance, and it is only worse than ours when = 2. Forward construction performs worst among the four algorithms.
In summary, a greedy backward construction like Algorithm works the best in our setting, rather than globally optimized shortest-path algorithms. The final target teacher has the most profound impact on the distilled student's performance. In order to fully assist the final teacher, we need to use another teacher with the minimal adaptation cost to the final teacher before it, which is exactly the behavior of Algorithm 1.
B ABLATION STUDY ON DISTILLATION WITH HOMOGENEOUS TEACHERS
In this section, we provide more details about distillation with homogeneous teachers (Section 4.2). We investigate (1) the impact of each individual teacher; and (2) distillation with teachers simulta- neously vs. sequentially.
Table 9: Homogeneous distillation of COCO detectors, where students with ResNet-50 backbones are distilled with teachers with ResNet-50 50 backbones. We report the detection ("Box') and segmentation (' ('Mask') APs and runtime, and compare distilled student with its teachers, off -he-thh-sheff ('OTS') student. Our distilled student significantly improves the APs the 'OTS' student by around 3%.
Box
Mask
Runtime (ms)
ID
Model
AP
AP5a
AP7s
APs
APM
AP1
AP
AP50
AP7a
APs
AP
AP2
Teacher III
42.3
61.1
45.8
23.7
45.6
56.3
37.4
58.4
40,2
19.6
40.4
51.7
181
Teacher IV
49.1
67.7
53.4
29.9
53.0
65.2
42.6
65.1
46.0
24.1
46,4
58.6
223
Teucher V
45.1
66.3
49.3
27.8
49.0
59.3
40.1
63.1
42.8
22.9
43.8
54.8
142
Student II (OTS)
38.2
58.8
41.4
21.9
40.9
49.5
34.7
55.7
37.2
18.3
37.4
47.2
51
Student (distilled)
41.4
45.1
23.3
45.0
55.4
58.8
39.8
19.4
52.1
Table 10: Ablation study of homogeneous distillation of COCO detectors (models in Table 9). Our distillation strategy is consistently effective irrespective of teacher rype. Morcover, sequential distillation with two teachers outperforms both distillation with single teacher and simultancous distillation with two teachers. Our best distilled student is obtained by progressive distillation, where Student II is first distilled with Teacher V (a weaker, similar teacher with the same head Student II) and then distilled with Teacher IV (a stronger teacher whose architecture is completely different from Student II).
ID
Student II
Box
Mask
AP
APso
APs
APM
APp
APs0
APIs
APs
AP
APr.
OTS
38.2
58.8
41.4
21.9
40.9
49.5
34.7
55.7
37.2
18.3
37.4
47.2
Distilled by Teacher III
40.2
60.7
43.8
22.5
43.8
53.4
36.3
57.3
38.7
18.9
39.3
50.3
3
Distilled by Teacher IV
40.8
61.5
44.6
23.0
44.3
54.2
36.8
58.3
39.4
19.2
39.9
51.0
Distilled by Teacher V
58.1
39.1
39.6
51.0
Distilled by Teachers IV-V
39.8
60.3
43.4
22.1
43.3
52.9
35.9
57.1
38.1
18.3
39.0
49.8
Distilled by Teachers IV
41.0
61.7
44.
23.0
44.3
54.9
36.8
58.3
39.2
19.5
39.9
51.3
Distilled by Teachers V-IV
41.4
61.9
45.1
23.3
45.0
55.4
37.3
58.8
39.8
19.4
40.4
52.1
We first distill Student II with each of the three teachers individu- ally: Teacher III has the same backbone and neck but more advanced head; Teacher IV has more advanced backbone, neck, and head; Teacher V has the same head but more advanced backbone and neck. Table provides the performance of the three teachers, where Teacher IV achieves the best performance (row 1-3). From Table 10, we can see that our distilled students (row 2-7) significantly and consistently outperform the off-the-shelf student (row 1), demonstrating the effectiveness of our distillation strategy irrespective of the types of teachers. Moreover, the improvement of the student distilled with Teacher (row 2) over that with Teacher (row 3) shows that a more powerful teacher generally leads to better distilled student. Interestingly, although Teacher IV is more pow- erful than Teacher V, Table 10 shows that their distilled students achieve quite similar AP (row 2 vs. powerful teacher does not necessarily further improve the
Impact of individual teachers:
row 4). This indicates that an
16

---

Under review as a conference paper at ICLR 2023
distilled student; too large capacity and structure gap between the teacher and student will limit the effectiveness of distillation. Also, it is easier to distill from teachers with the same head.
Simultaneous vs. progressive distillation: We now distill Student II with the combined teachers, and we choose the top-performing Teacher IV and Teacher V. We investigate two types of combi- nation- simultaneous distillation with feature matching loss between each teacher and the student (row 5), and sequential distillation with teachers one by one (row 6-7). First, we find that using both teachers simultaneously (row 5) even worse than our method using a single teacher (row 2-4). This shows that integrating different types of knowledge from multiple teachers is not a trivial task simultancously using the features from multiple teachers might provide conflicting supervisions to the student model and thus hinder its distillation process. By contrast, our sequential distillation overcomes this issue and improves the performance irrespective of the order of the teachers (row 6-7 vs. row 1-4). Second, the sequential order of the teachers makes a difference. A curriculum-like progression (row 7), where the teacher with a smaller adaptation cost used first and that with a larger adaptation cost & a higher performance is used later, leads to the best performance.
Overall performance: Our best distillation performance is achieved when we first distill Student II with a curriculum of teachers (Teacher VVIV). Overall, the box AP is improved from 38.2% to 41.4% and the mask AP is improved from 34.7% to 37 3%. Our resulting Mask R-CNN detector has comparable performance with HTC, but much smaller runtime.
C ABLATION STUDY ON DISTILLATION WITH HETEROGENEOUS TEACHERS
In this section, we provide more details about distillation with heterogeneous teachers (Section 1 11). We investigate the heterogeneous cases where the backbones input resolutions are different be- tween the teachers and student.
Heterogenous distillation COCO detectors, where students with ResNet-18 backbones are distilled with teachers with ResNet-50 backbone, requiring additional transfer logic. We report the detection ('Box') and segmentation ('Mask") APs and runtime, and we compare our distilled student with its teachers, and off- she-shelf ('OTS") student. Our distilled student significantly improves the APs over the "OTS' student by over 3%.
Table 11:
Box
Mask
Runtime (ms)
ID
Model
AP
AP5a
AP
APs
APM
AP1
AP
AP5o
APr
APs
A
AP1
Teacher 1
38.2
58.8
41.4
21.9
40,9
49.5
34.7
55.7
37.2
18.3
37.4
47.2
51
Teacher III
42.3
61.1
45.8
23.7
45.6
56.3
37.4
58.4
40.2
19.6
40.4
51.7
181
Teacher IV
49.1
67.7
53.4
29.9
53.0
65.2
42.6
65.1
46.0
24.1
46.4
58.6
223
Teacher
45.1
66.3
49.3
27.8
49.0
59.3
40.1
63.1
42.8
22.9
43.8
142
Student III (OTS)
33.3
52.9
35.9
18.2
35.9
43.6
30.5
50.0
32.1
15.5
32.9
41.8
29
6
Student III (Distilled)
37.0
56.8
39.9
20.2
39.8
50.0
33.7
53.6
36.0
17.2
36.0
47.3
29
Overall performance: Again, Tables 11 and 12 show that our distillation strategy is consistently effective with respect to all the teachers and their combinations, e.g., the box AP improves from 33.3% to 37.0% and the mask AP improves from 30.5% to 33.7%.
Two signature findings in heterogeneous distillation: Compared to the homogeneous case, we find the capacity gap between models is a more important factor, and bridge this gap a proper teacher order plays more critical role. Details are explained as follows.
The student-teacher capacity gap is more pronounced in heterogeneous distillation. Among the four teachers, Teacher shares exactly the same neck and head structure with the student, and has a similar but larger backbone; Teacher V has the same head with the student as well, but has a different backbone and neck; Teacher III has similar backbone and neck, but has a different head; and Teacher IV is the most powerful one with completely different architecture. Table 12 (rows 3-6) summarizes the distillation results with single teachers. First, directly distilling from the strongest teacher (Teacher IV) does not yield the largest improvement. Second, a relatively less powerful but more similar teacher (Teacher I) leads to the best distillation performance, improving the APs by 2%, although teachers V. III, and IV are all stronger than Teacher I. One possible reason is that Teacher I has the same neck and head Student III as well as similar but deeper backbone. the capacity gap between Student III and Teacher is the smallest. Finally, we find that Teacher III is a strong but not particularly helpful teacher, achieving the worst distillation results. One possible reason is that
17

---

Under review as a conference paper at ICLR 2023
Table 12: Ablation study for heterogeneous COCO detector distillation (models in Table 11). Student III (Mask R-CNN with a ResNet-18 backbone) distilled with teachers with different and larger ResNet-50 backbones. Training Student III for more epochs improves its performance, but not as much as progressive distillation with teachers. Note that for each distillation we train 12 epochs. Our distillation strategy is consistenrly effective irrespective of the rypes of teachers. Moreover, simultancous distillation with multiple teachers. Our best distilled student is obtained by progressive distilla- tion, where Student is first distilled with Teacher (a most similar teacher with the same head and neck as Student III and a deeper backbone), then distilled with Teacher V (a stronger teacher with the same head as Student III), and finally distilled with Teacher IV (a strongest teacher whose architecture is completely different from Student III).
sequential distillation with multiple teachers outperforms
Bux
Mask
Model
AP
AP50
AP75
APş
AP2.
AP
APsc
AP7s
APs
APM
APL
Student (OTS)
32.9
+12 epochss
34.6
54.5
37.2
18.8
36.9
46.1
31.6
51.5
33.6
15.8
33.7
44.0
+24 epochs
34.5
54.2
37.2
18.8
36.5
45.8
31.5
51.2
33.8
16.0
33.4
43.7
+36 epochs
34.6
54.2
37.4
36.9
46.7
31.6
51.1
33.8
15.7
33.6
44.3
3 Distilled by Teacher I
35.8
55.8
38.8
19.3
38.8
47.9
32.6
52.7
34,8
16.0
35.3
45.5
Distilled by Teacher III
35.2
55.2
37.8
19.1
37.8
47.4
32.1
16.1
34.5
45.2
Distilled by Teacher IV
35.5
55.2
38.2
19.0
37.9
48.0
32.4
51.9
34.5
15.9
34.8
45.6
Distilled by Teacher
35.4
55.2
38.3
19.4
37.9
48.4
32.2
52.2
34.3
15.4
34.4
45.8
Distilled by Teachers IV
34.8
54.9
37.2
37.2
47.0
31.6
51.7
33.9
15.7
33.8
44.2
8
Distilled by Teachers I+IV+V
36.0
55.4
39.1
18.2
38.1
48.3
32.1
53.0
34.7
15.8
34.7
46.1
Distilled by Teachers +V+++
38.2
10 Distilled by Teachers 1-+V
36.5
56.3
39.3
19.5
38.8
49.4
33.2
53.2
35.3
16.4
35.4
46.8
Distilled by Teachers V->IV
35.2
55.2
37.8
19.1
37.8
47.4
32.1
52.0
34.0
16.1
34.5
45.2
Distilled by Teachers V-V
37.0
56.8
39.9
20.2
39.8
50.0
33.7
53.6
36.0
17.2
36.0
47.3
Teacher III has very different head from Student III, while not as stand-alone accurate as Teacher IV, making it unable to provide enough guidance Student III. These observations suggest that smaller capacity gap between the student and the teacher may facilities knowledge transfer.
The sequential order of the teachers plays more critical role in the heterogeneous setting. Table 12 (row 7-12) presents representative results with different orders or combinations of the teachers. Again, a proper progressive distillation (row 12) outperforms simultaneous distillation (row 7-9)- Notably, it is necessary start with Teacher I, since the capacity gap between Student III and Teacher minimal, with difference only on the depth their ResNet backbones. These results confirm the importance of our curriculum-like progression to best benefit from multiple teachers.
Training a student longer vs. distilling a student: As another sanity check, Table 12 includes results of training Student III with more epochs without distillation (row 2-4). We first 12 additional epochs improve APs by 1%, but there are no significant improvements even if we train for longer period. This shows the effectiveness of detector distillation.
that the
Distillation with different model resolutions: Table 12, we have performed distillation where the student and teacher models operated on the same input image resolution (e.g.. the standard reso- lution 1,333 x 800 on MS COCO). In practice, one way to further reduce the latency/runtime of the student is to operate on lower-resolution images. However, this poses additional challenges with teacher of high input resolution and a student of low input resolution, they become even more het- erogeneous. Moreover, image resolution substantially affects object detection performance (Ashraf et al., 2016). Here, we are interested in performing distillation with models trained with images of different resolutions to further investigate the generalizability our approach. More specifically, we use high-resolution models teachers and low-resolution models students, shown in Table 13 (row 1-4).
In these experiments, the teacher and student feature maps have different spatial resolution. To tackle this, we simply upsample the spatial maps of the student and supervise the student with the teachers' features. Again, Table 13 shows that our approach is effective in this more challenging scenario. Our best performance achieved by progressively distilling the student with its Teacher I-3, 1-2, and [-1.
18

---

Under review as a conference paper at ICLR 2023
Table 13: Detectors trained with different input resolutions on the COCO dataset. We use series of Teacher I variants: Teacher is trained with the standard input resolution of 1, 333 x 800; Teacher is trained with 1. 000 x 600 input: Teacher 1-3 is trained with 666 x 400 imput; and the student is trained with 333 x 200 input. We report the detection ("Box') and segmentation ("Mask') APs and runtime. We compare our distilled student with its teachers, and off-the uff-the-shelf ('OTS') student. Our approach is effective with even more heterogeneous teacher and student models of different input resolutions.
ID Model
Input Resolution
Box
Musk
Runtime
AP
AP
APs
APM
AP1
AP
APtn
AP
APg
APM
APr
(ms)
Teacher 1-1
1313 x 800
38.2
58.8
41.4
21.9
40.9
49.5
34.7
55.7
37.2
18.3
37.4
47.2
31.5
Teacher
57.7
33.6
54.3
35.9
37.0
47.7
3
Teacher 1-3
666 1001
34.7
54.0
37.2
15.6
38.1
50.4
31.2
50.5
33.2
12.2
34.4
47.0
19.7
Student (OTS)
41.9
23.0
38.7
23.7
5.0
23.7
41.3
16.9
Student (distilled)
333 200
31.5
49.8
33.3
12.3
34.3
48.9
28.2
46.5
29.0
9.3
30.3
45.4
16.9
D GENERALIZABILITY TO OTHER DATASETS AND EVALUATION PROTOCOLS
In this section, we study the generalizability of our approach. As an extension from the gold- standard COCO benchmark, we evaluate our distilled student (trained on COCO) on another dataset, Argoverse-HD, and with another metric, streaming accuracy, and perform distillation on Argoverse- HD directly.
Table 14: Generalizability on Argoverse- HD.. On the left, we report standard detection accuracy. 'OTS' and distilled students are trained on COCO. We observe 2% AP gains through distillation, even on novel testsets. On the right, we report streaming detection accuracy as defined in Li et al. (2020a), in the detection-only setting a Tesla V 1000 GPU. The second column denotes the optimal input resolution (that maximizes streaming accuracy). First, we discover that a lighter model and full- full-resolution input much more helpful than having an accurate but complex model that needs to downsize input resolution. Second, our proposed distillation approach further improves over the lightweight model.
Model
box AP
APso
APt:
APs
AP
AP,
Stud. II
OTS Distilled
34.4
32.7
54,2
52
34.5
35.9
15.0
14.7
35.8
57.7
52.8
Desecier
Cas MRCNNSO (Li al., 2000a)
Expui
0.5%
14.0
APs.
26.8
APt.
12.2
AP:
1.0
AP
9,9
AP.
28.
MRCNNI8 (Oues)
1.0x
23.7
44.8
22.6
104
23.1
37.8
Stud. III
OTS
Distilled
30.6
28.9
49.7
48.8
30.0
31.8
12.9
12.8
31.3
32.6
51.9
49.2
MICCNNIS Dosil)
100x
25.0
85.8
24.2
10.5
24.1
39.3
Table 15: Heterogenous distillation of Argoverse- HD detectors, where a student with ResNet-18 18 backbone is distilled with teachers with ResNet-50 backbones. We report the detection ("Box') APs and runtime. We com- pare our distilled student with its teachers, and off-the-shelf ('OTS') student. Our distilled student significantly improves APs over the 'OTS' student by over 2%. Notably, our distilled student achieves detection accuracy that comparable with Teacher A but with only around third of the runtime.
Box
Runtime
ID
Model
Backbone
Neck
Method (Head)
APs0
AP1s
APs
AP
APı
(ms)
Teacher Teacher B
ResNet-50
FPN
Faster R-CNN
29.6
48.2
30.5
16.4
33.1
45.1
79.2
ResNet-50
FPN
Cascude
32.3
50.4
35.0
16.4
37.1
47.7
89.0
Teacher C
ResNet-50 + SAC
RHP
Faster R-CNN
32.9
51.0
35.5
17.6
33.7
52.9
23018
Teacher
ResNet-50 SAC
RFP
Cascade
34.5
52.0
37.7
17.9
37.0
52.8
241.2
Student (OTS)
ResNet-18
FPN
Faster R-CNN
27.1
48.1
27.5
14.4
31.2
40.0
29.3
Student (distilled)
ResNet-18
FPN
Faster R-CNN
29.2
49
30.9
15
31.7
45.6
29.5
Argoverse-HD is more challenging dataset than COCO due higher resolution images and signif- icantly more small objects. Constructed from the autonomous driving dataset Argoverse 1.1 (Chang al., 2019), Argoverse-HD) contains RGB video sequences and dense bounding box annotations (1,260k boxes in total). It consists of 8 object categories, which are subset of 80 COCO classes and are directly relevant to autonomous driving: person, bicycle, car, motorcycle, bus, truck, traffic light, and stop sign. There are 38k training images and 15k validation images. We report results on the validation images. We test the distilled models trained on COCO on Argoverse-Hil without re-training. Table 14-left shows the generalizability of our approach.
Streaming accuracy is a recently proposed metric that simultaneously evaluates both the accuracy and latency of algorithms in online real-time setting (Li et al., 2020a). The evaluator queries
19

---

Under review as a conference paper at ICLR 2023
the state of the world at all time instants, forcing algorithms to consider the amount of streaming data that must be ignored while processing the last frame. Following the setup proposed in Li et al. (2020a), we evaluate streaming AP in the context of real-time object detection for autonomous vehicles. Table 14-right shows our approach outperforming the prior results from Li et al. (2020a) by dramatic margin. We find significant wins by using an exceedingly lightweight network (ResNet- 18 based Mask R-CNN) that can process full-resolution images without sacrificing latency. Due to much higher quantities of small objects, high-reslution processing is more effective than deeper network structures. In addition, progressive distillation further improves performance.
Direct distillation on Argoverse-HD: After testing the distilled model which is trained on COCO, on the Argoverse- Hrroverse-HDoooo dataset (Li et al., 2020a) without re- re-training. we have shown the generaliz- ability of the already-distilles models. Here we directly distill the student model on Hrgoverse-HD. using Faster R-CNN with 18 backbone as the student model. As shown in Table 15, we use four teachers with ResNet-50 backbones (row 1-4,, including Faster R-CNN (Ren et al., 2014) (Teacher A), Cascade R-CNN (Cai & Vasconcelos, 2018) (Teacher B), and DetectoRS (Qiao et al., 2021) (Teachers C & D).
The results are summarized in Table 15. Our best distillation performance is achieved when we first distill the student with a similar teacher (Teacher A), and then progressively distill with more powerful teachers (Teachers B, then C, and finally D). Overall, the bbox mAP is improved from 27.1% 1 to 29.2%.
In addition, comparing with Table 14-left, the absolute performance of the teachers and students in Table 15 is lower. This is because here we use weaker teachers and student models (Faster R- CNN for fast experiments) than the models used in Table 14-left (Mask R-CNN). However, the relative improvement (between the distilled and OTS students) of box AP (2.1%) is larger than that in Table 14-1eft (1.7%), indicating that learning distillation directly on Argoverse-Hl further improves the performance.
E MORE COMPARISON WITH PRIOR KNOWLEDGE DISTILLATION METHODS
The most profound difference between this work and most of the prior work on knowledge distilla- tion is that prior work mainly focuses on the image classification task, while we address the object detection task. The detection task (and the associated model architectures) much more compli- cated than the classification task. This makes the distillation methods developed in the context of classification often not directly applicable to detection. That why dedicated distillation meth- ods (Chen et al., 2017; Wang et al., 2019; Guo et al., 2021; Zhang & Ma, 2021; Dai et al., 2021; Guo et al., 2021; Yang et al., 2022a;b) need to be developed for the detection task in the literature. Here, we discuss the difference between our method and prior method on knowledge distillation in detail:
Progressive distillation: Mirzadeh et al. (2020) is related to our method, in the sense that this work progressively distills a student from multiple teachers (one teacher and several additional teacher assistants (TAs)). With one TA, the distillation process in Mirzadeh et al. (2020) contains three steps: 1) The TA is first distilled from the teacher; 2) The student is distilled from the TA; and 3) The student trained by the TA is further distilled from the teacher. When there are multiple TAs, the shallower TAs are distilled from deeper ones, so that they form distillation path. However, our work is different from Mirzadeh et al. (2020) in three important ways:
As mentioned above, Mirzadeh al. (2020) focuses on image classification, while we study pro- gressive distillation in the context of object detection. In our case, the transferred knowledge is no longer classification logits but intermediate features or structured predictions. This would require additional consideration and algorithmic designs to extend progressive distillation from image classification to object detection.
Our strategy to construct the teacher sequence is a novel contribution and is fundamentally dif- ferent from Mirzadeh et al. (2020). In our work, we propose a heuristic algorithm (Algorithm 1) based on the representation similarities between different models (Section 3.2), which automat- ically generates the teacher order. In Mirzadeh et al. (2020), a series of deep networks with increasing depths act the student, the TA(s), and the teacher. One can intuitively determine the distillation sequence of TA(s) according to their increasing depths (which imply learning capaci- ties). However in our case, there is pool of teachers with diverse architectures and their relative
20

---

Under review as a conference paper at ICLR 2023
ordering is unknown. This challenge motivates us to design algorithm to automatically decide the teacher order based on their representation similarities; importantly, the strategy in Mirzadeh et al. (2020) is not applicable in our task.
Mirzadeh et al. (2020) is more cumbersome and time-consuming. An intermediate TA in Mirzadeh al. (2020) needs to be first distilled from the teacher deeper TA, the TÀs have to be trained one by one. By contrast, all of our teachers (including the intermediate teachers and the final teacher) are trained independently. This makes the generation of our teachers parallelizable
Multi-feacher distillation: The key difference lies in: These methods (You et al., 2017; Lan et al., 2018: Guo et al., 2020) use an ensemble of multiple teachers simultaneously guide the student learning, while our work distills from multiple teachers sequentially, and we proposed novel method to construct the appropriate teacher order. Empirically, we compare these two strategies, and demonstrate that our sequential progressive strategy outperforms the simultancous strategy (via teacher ensemble by taking the average of teacher features) for object detection.
This comparison is provided in Appendix B (Table 10) and Appendix C (Table 12). For example in Table 12, if we compare experiments with ID 7-9 (simultaneous distillation from teacher en- sembles) vs. experiments with ID 10-12 (progressive distillation from teacher sequences), we find that progressive distillation is a better choice.
Our performance superiority is because in the object detection task, the teacher's knowledge is transferred from intermediate features, rather than from final classification predictions. Thus, the ensemble of multiple teachers might provide conflicting supervision signals for the student, leading to performance interior to our progressive distillation.
Online distillation, deep mutual learning: Although the teacher model is also changing during
online distillation (Yang et al., 2019a; Guo et al., 2020; Yao & Sun, 2020; Li et al., 2022). the principle of our sequential teachers is significantly different from online distillation for the following reasons:
Strictly speaking, in online distillation, there is only one teacher - This teacher's architecture is fixed, and its weights keep updating in online manner. By contrast, we have multiple teachers These teachers have different architectures, and their weights are first trained independently, and then frozen in the progressive distillation process; in our progressive distillation, we switch the whole teacher model.
The type of discrepancy between the student and the teacher is different for ours and online distil- lation. Online distillation often uses similar or even the same architecture for both the teacher and student models. Consequently, their capacities are the same level, and they can evolve together. Our study is quite different: The key question we want to address is the capacity gap between the student and the teacher (the capacity gap is due to the architectural difference between the student and the teacher); and our solution is to progressively distill using other teachers with intermediate capacities.
Other general distillation mechanisms: These methods (Romero et al., 2015; Zagoruyko & Ko- modakis, 2017; Ahn et al., 2019) introduce other types of distillation mechanisms, but still consider the setting where only one single fixed teacher is involved. Different from these methods, use multiple teachers to progressively transfer knowledge from them to the student. Meanwhile, we share some similarities with Romero et al. (2015); Zagoruyko & Komodakis (2017); Ahn et al. (2019) in that they are distilling knowledge from the activations of intermediate layers. Our simple neatur-mmtching loss (Section 3.1) and other recent work in detector distillation (e.g.. CWD (Shu et al., 2021), FGD (Yang et al., 2022a), and MGD (Yang et al., 2022b)) are based on the "hint" distillation (learning from intermediate layers' outputs) from Romero et al. (2015).
F MORE RESULTS ON COMBINATION WITH STATE-OF-THE-ART DISTILLATION MECHANISMS
In this section, we include additional experimental results requested by the reviewers.
Other object detectors: In the main paper, we mainly use RetinaNet and Mask R-CNN as student detectors for a fair comparison with existing methods, because most of the prior work on detector distillation has primarily focused on RetinaNet and Mask R-CNN. Being commonly-used in real-
21

---

Under review as a conference paper at ICLR 2023
world applications, RetinaNet and Mask R-CNN represent the two important families of object detectors (single-stage and two-stage).
In principle, work is general and applicable to different types of detection models, our pro- gressive distillation strategy is designed without assumption on particular types of detector archi- additional experiment on another detection
tectures. To demonstrate this, we have performed model, RepPoints (Yang et al., 2019b). RepPoints is an anchor-fres detector, with high efficiency- accuracy trade-off. Meanwhile, RepPoints has been experimented in state-of-the-art detector distil- lation methods such as MGD (Yang et al., 2022b), so we can provide an informative comparison. Therefore, we choose RepPoints as the detection model in this additional experiment. The student model is RepPoints/ResNet-S (38.6 AP on COCO), and following state-of-the- method MGD, we use RepPoints/ResNeXt-100 (44.2 AP) as the final teacher model. The intermediate teacher in our progressive distillation is RepPoints/ResNet- 101 (40.5 AP). The following Table 16 shows the results obtained.
Table 16: Distillation of RepPoints (Yang al., 2019b) detector. The student detector is RepPoints/ResNet-500, and the final teacher detector RopPoints/ResNext 101. By using an intermediate teacher Ropsoints/ResNet- 101 for progressive distillation, we improve the student performance to 42.9 % AP.
ID
Student
Distillation
Teacher(s)
AP
APs
MPM
APr.
Repeo:nts/ResNet:50
None
None
38.6
22.5
42.2
50.4
CWD (Shu al., 2021)
RepPoints/ResNeX-1000
42.0
24.1
46.1
55.0
RepPointsReesNet550
FGD (Yang et al., 2022a)
RepPoints/ReNNett-IDI
42.0
24.0
45.7
55.6
MGD (Yang et al., 20226)
RepPoink/ResNeX--10
42.3
24.4
46.2
55.9
MGD (Yang et al., 2022b)
RepPoint//ReNNe-101
RepPoints/ReSNet:50
Progressive distillation (Ours)
-RepPointsReeNee---00
42.9
25.6
46.9
56.3
Our progressive distillation strategy improves the performance of the student detector by 4.3% AP, which also 0.6% AP better than the previous state-of-the-art MGD. This result achieved by only introducing intermediate teacher and without increasing the training cost. This experiment further demonstrates that our progressive distillation strategy is general and can be applied to various detectors.
Keep using the intermediate teacher: In the main paper, we have discussed the capacity gap between the teacher and student and how to mitigate this gap via progressive distillation. One may question that, since the capacity gap between the intermediate teacher and the student is smaller, keeping using this intermediate teacher throughout the distillation procedure might also lead good student. Here we provide exemplary experiment result (in addition to Figure 4-left) as an answer to this question: Only using the intermediate teacher is still suboptimal compared with our proposed progressive distillation.
In this experiment, we use state-of-the-art distillation method MGD as the base method. We use RetinaNNe/RRssNets (37.4 AP on COCO) as the student model, RetinaNet/ResNet- (38.9 AP) as the intermediate teacher model, and RetinaNe//ResNeXt- 101 (40.8 AP) as the final teacher model. The capacity gap between the intermediate teacher and the student is smaller than that between the final teacher and the student. The following Table 17 shows the results.
Distillation of RetinaNet detector. The capacity gap between the student NRetinaNee/Reseet and the intermediate teacher (RetinaNet/ResNNtt 101) is smaller. Our progressive distillation is better than both distillation schemes that keep using the intermediate teacher (ID 3) or the final teacher (ID 2).
Table 17:
Student
Distillation
Teacher(s)
Training Schedule
AP
RetinaNet/ReeNeet550
None
None
2x
37.4
RetinaNeUResNeet-I1 RetinaNer/ReSNe-1011
2x
41.0
RetinaNet/ResNet550
MGD (Yang et al., 2022b)
2x
40.7
RetinaNeeURcsNee-100
1x
40.2
MGD (Yang al., 2022b) Progressive distillation (Ours)
RetinaNNeRReNNe--01
RetinaNe//eeNet-500
+1x
41.4
- RetinaNNedReeNeXt-1100
Keeping using the intermediate teacher (RetinaNeedReNNet-1O0) for a longer 2x training schedule indeed improves the performance from 40.2% AP 40.7%, but it is still not better than directly
22

---

Under review as a conference paper at ICLR 2023
using the final teacher (RetinaNee/Reeseett-00)) Our progressive distillation, which first uses the intermediate teacher and then the final teacher for distillation, outperforms both direct distillation schemes and achieves 41.4% AP performance. As always. we use the same total training time (2x training schedule) as direct distillation for a fair comparison. This experiment supports the conclusions that 1) employing
intermediate teacher throughout the distillation process is not good option; and 2) our progressive distillation, which uses both the intermediate teacher and the best performing teacher sequentially, leads the best student performance.
G IMPLEMENTATION DETAILS
We implement detectors and their distillation using the MMDetection codebase (Chen et al., 2019b). We train on 8 GPUs for 12 epochs for each distillation. For MS COCO, we use the standard input resolution of 1, 333 x 800, with each GPU hosting images. For Argoverse- HD. we use its much higher native resolution as the input at 1, 1,920 x 1, 200, with each GPU hosting image. We use an initial learning rate of 0.01 (for RetinaNet students) or 0.02 (for Mask R-CNN students). We use stochastic gradient descent and momentum of 0.9. For the simple feature-matching loss (see Section 3.1), we perform a grid search over the hyper-parameter dependent the architectures of the teacher and student models, we find the performance is not very sensitive to À between 0.3 and 0.8. We set = 0.5 for RetinaNet students and = 0.8 for Mask R- R-CNN students.
While the optimal values are
When we combine our progressive distillation with state-of- of-the-art distillation mechanisms includ- ing CWD (Shu al., 2021), FGD (Yang et al., 2022a), and MGD (Yang al., 2022b) (Section 4.4), we strictly follow the publicly available implementation from their authors, and use an intermediate teacher (RetinaNet/ResNet--Il 101 or Cascade Mask R-CNN/ResNet5O-DCCN for progressive distilla- tion. In the original implementation of FGD and MGD, an inheriting strategy (Kang et al., 2021) is utilized, which initializes the student with the teacher's neck and head parameters train the stu- dent when they have the same head structure. In our progressive distillation, we adopt this inheriting strategy only once for the first teacher.
For the Transformer- baasdedrerr-bsed teachers, we use Swin Transformer backbone, which has hierarchical architecture and shares the "same feature map resolutions as those of typical convolutional networks (e.g., ResNet-50)" (Liu et al., 2021). Following the original implementation of the Swin Trans- former, the backbone is equipped with an FPN neck, so the number of neck feature channels the same the student. As a result, Swin Transformer based teachers can be used like some other convolution- based teachers without the feature map matching function (r(-) in Section 3.1).
23


Reviews:
[{"review_id": "khUtoAnCJ8", "reviewer": "Reviewer_1tUC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple progressive knowledge distillation framework with a sequence of teachers for detectors. ", "summary_of_the_review": "The work is somewhat simple yet effective, my major concern is the novelty and experiments. \nIf the authors can make the experimental part simple and clear, and highlight the competitiveness of the results with more SOTAs, I will consider improve the score. ", "strength_and_weaknesses": "\nStrength:\n1. The idea of progressively transferring knowledge from a sequence of teachers to a lightweight detector is somewhat novel. \n2. It represents the first effort to distill knowledge from Transformer-based teacher detectors to convolution-based students.\n3. It shows the performance gain comes from better generalization rather than better optimization. \n\nWeaknesses:\n1. Why not compare with other general[4,5,6] / multi-teacher[9,10] / progressive[1-3,7,8] KD methods? \n2. Please clarify the difference between the proposed method and online KD[] algorithms. And it's better to give more experiments and analysis to highlight the strength of your idea. I think that the principle of sequence teacher is similar to online distillation. \n3. The ranking of teachers seems to require empirical design. It's better to give more theoretical analysis. \n4. Another serious concern is that the pre-training of multiple teachers can be cumbersome and time-consuming, making the pipeline complicated and unpractical. \n5. Since the author propose to transfer knowledge between Transformer and CNN, why not adopt Transformer-based student detectors?\n6. There is lack of some important baselines [11-16].\n\n## References\n\n[1] Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher. Mirzadeh et al. arXiv:1902.03393\n\n[2] Snapshot Distillation: Teacher-Student Optimization in One Generation. Yang, Chenglin et al. CVPR 2019\n\n[3] Online Knowledge Distillation by Temporal-Spatial Boosting. Li, Chengcheng et al. WACV 2022\n\n[4] Fitnets: Hints for thin deep nets. Romero, Adriana et al. arXiv:1412.6550\n\n[5] Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. Zagoruyko et al. ICLR 2017\n\n[6] Variational Information Distillation for Knowledge Transfer. Ahn, Sungsoo et al. CVPR 2019\n\n[7] Online Knowledge Distillation via Collaborative Learning. Guo, Qiushan et al. CVPR 2020\n\n[8] Knowledge Transfer via Dense Cross-layer Mutual-distillation. ECCV 2020\n\n[9] Learning from Multiple Teacher Networks. You, Shan et al. KDD 2017\n\n[10] Knowledge distillation by on-the-fly native ensemble. Lan, Xu et al. NeurIPS 2018\n\n[11] Learning efficient object detection models with knowledge distillation. Chen, Guobin et al. NeurIPS 2017\n\n[12] Distilling Object Detectors with Fine-grained Feature Imitation. Wang, Tao et al. CVPR 2019\n\n[13] Enabling Incremental Knowledge Transfer for Object Detection at the Edge. arXiv:2004.05746\n\n[14] General Instance Distillation for Object Detection. Dai, Xing et al. CVPR 2021\n\n[15] Distilling Image Classifiers in Object Detectors. Guo, Shuxuan et al. NeurIPS 2021\n\n[16] Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors. ICLR 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f7d1YyXKUQ", "reviewer": "Reviewer_R4rk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the progressive knowledge distillation method for object detection by distilling knowledge from multiple teachers. The main contribution of this work is designing a heuristic algorithm based on the correlation of feature representations to generate the proper sequence of several teachers given a student model. Comprehensive experiments and ablation studies on the COCO dataset evidence the effectiveness of the proposed method.", "summary_of_the_review": "The paper is in good shape with extensive experiments, which show promising results. However, there are still some concerns. Please clarify them. ", "strength_and_weaknesses": "**Strengths**\n+ Novel idea. The authors propose a heuristic algorithm to produce a near-optimal sequence of multiple teachers, even not the best, still outperforms multiple KD approaches distilled from a single teacher.\n+ Promising results and convincing ablation studies with multiple settings for object detection.\n\n**Weaknesses**\n\nHowever, there are still some concerns to be addressed:\n\n- Even though the student can get better results with the selected order of the teachers by Algorithm 1, sometimes it is not the best, especially with more teachers. Could the author please provide some potential explanation on this? In other world,  how reliable the proposed Algorithm 1 is?\n\n- As stated in Section 4.1, the student is initialized by an off-the-shell student for the proposed KD method. Does the student use the same strategy of initialization for exp 3 and 7 (Directly distilled by Teacher IV) and blue bars in Figure 4? Moreover, as for Figure 4, it is not quite fair to compare the performance of 2x training with 2 1x training. Please clarify it.\n\n- The capacity gap between the student and the best performance teacher may a problem for knowledge distillation. Maybe the intermediate teacher is better to be distilled for the student. Thus, it will be interesting to compare with the distilled results from the intermediate teachers in Tables 3, 4 and 5, and Figure 4.\n\n**Minor comments**\n\nIt is worth discussing a related work [A], which also makes use of several teachers. It is interesting to the research line of KD in object detection.\n\n**Reference**\n\n[A] Guo. et al. Distilling Image Classifiers in Object Detectors. NeurIPS2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WCtjD2-TJDy", "reviewer": "Reviewer_zfUg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a progressive distillation strategy for object detection and instance segmentation models, i.e., multiple teacher models transfer knowledge to a student model in a sequential manner. A heuristic method is also given to choose the order of teacher models and its efficacy is demonstrated by experiments. A method for distilling knowledge from transformer-based teachers to convolutional-based students is simply introduced. Many experiments are done to verify the capability of the proposed strategy and ablation studies are provided.", "summary_of_the_review": "The reviewer generally thinks the paper is written in sound quality as specified by the strengths above. It proposes a sequential manner for knowledge distillation. However, it would be recommended that authors address the weakness mentioned above and also the following comments:\n\n1. In table 7, for length 4, there is no bold sequence marked. Is this a result of algorithm 1? Please add some explanations.\n2. Maybe give a concrete example of $r$ in (1) for two concrete models in a heterogeneous case. It also would be great to see how the transformer-based model is addressed here.\n\n-------------------\nUpdate: I believe the authors well-addressed my comments.", "strength_and_weaknesses": "Strengths:\n1. The sequential strategy is inspiring and of interest for knowledge distillation for object detection and segmentation.\n2. The performance of the proposed method is great, as shown in the results.\n3. Comparison between the sequential way and simultaneous way is of interest. \n\nWeakness:\n1. Compared to other studies in the paper, the introduction to the distillation from the transformer-based model to the convolutional-based model is simple. For example, it seems that the transformer-based model is not combined with other models as teacher models. It would be better to provide a possible way or give some discussions in the paper.\n2. How to address a large k, i.e., the maximum number of selected teachers, or how to pick a good k is not given or well-discussed. In practice, it may happen when users have a large pool of teachers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JFlK8QzC1pF", "reviewer": "Reviewer_XUDr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel knowledge distillation methods which decide the progressive distillation process by the cost of linear regression on validation set, which achieves good performance on multiple student-teacher settings.", "summary_of_the_review": "Please refer to the strength and weaknesses. In summary, I like the core idea of this paper, while some more experiments are still necessary.\n\nUpdate-2022-11-14\nThe response from the authors basically addressed my concern. So I increase my score to 8.", "strength_and_weaknesses": "Strength: \n1. This paper proposes to choose the policy of student-teacher in knowledge distillation by using the cost of linear regression on the validation set, which is novel in this domain,\n2. Good experimental results have been achieved.\n3. Sufficient ablation studies have been conducted to demonstrate their performance.\n\nWeakness.\n1. Most of the experiments are conducted on RetinaNet and Mask RCNN with different backbones. It will be better if results on more SOTA detection models can be provided, such as Deformable Detr, Yolov4, CenterNet and so on.\n2. This paper applies the naive feature-based knowledge distillation as their knowledge distillation method. It is ok since this paper aims to focus on the order of teachers instead of the specific KD method. But It will be better if better more SOTA KD methods can be utilized to evaluate the effectiveness of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DhSxs9BlD1", "reviewer": "Reviewer_df3Q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a simple yet effective sequential approach to distill knowledge from Transformer-based teacher detectors to convolution-based student detectors. Instead of learning from a single teacher, the proposed framework develops a principled way to automatically design a sequence of teachers appropriate for the student and progressively distill it. Notably, the authors claimed that they were the first to investigate distillation from Transformer-based teacher detectors to a convolution-based student. Extensive experiments and ablation study is given to show the effectiveness of the proposed method.", "summary_of_the_review": "This paper is somewhat interesting but lacks rationality of clear motivation, technical novelty, and critical ablation studies. Details can be referred to the strength and weakness", "strength_and_weaknesses": "(1) This paper is well-written and organized.\n\n(2) The idea of learning lightweight object detectors via progressive knowledge distillation and investigating heterogeneous knowledge distillation is interesting.\n\n(3) The experimental results significantly outperform the existing SOTA knowledge distillation methods.\n\n\nWeakness:\n\n(1) The authors failed to prove that the observation ‘teacher-student capacity gap can be solved by the sequential distillation from multiple teachers arranged into a curriculum’, which is claimed in the Introduction.\n\n(2) Why the heterogeneous knowledge distillation should be performed from the Transformer-based teachers to convolution-based students? Why not from the convolution-based teachers to Transformer-based students?\n\n(3) Is the multiple teachers necessary? How about utilizing a single-teacher network with multiple forward progress? This operation also gives more supervision to the selected student and meanwhile saves the effort of searching for the near-optimal teacher order. Authors need to show the rationality of multiple teachers rather than multiple forward passes. \n\n(4) More SOTA and widely-adopted detection models should be involved, eg., YoloV4.\n\n(5) This paper repeatedly mentioned about \"proper\" either in selecting the teacher network or 'arranging teacher sequence'. Indeed, this reflected that this paper proposed a method that is not adaptable enough but based on \"proper\" (manual) selection and usage to \nachieve good performance. It is actually not easy to be extended by the followers if published.\n\n(6) As mentioned, selecting multiple teachers is actually not very easy in reality, especially for Transformers, which can not be easily trained. But it is more approachable to transfer from CNNs to the transformer as CNNs are already matured.  Therefore, the rationality of this method is less clear.\n\n(7) I do agree that teacher selection is one important aspect, however, currently, the method seems to lack enough novelty and complexity to meet the acceptance bar of ICLR.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject