Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides a simple yet effective sequential approach to distill knowledge from Transformer-based teacher detectors to convolution-based student detectors. Instead of learning from a single teacher, the proposed framework develops a principled way to automatically design a sequence of teachers appropriate for the student and progressively distill it. Notably, the authors claimed that they were the first to investigate distillation from Transformer-based teacher detectors to a convolution-based student. Extensive experiments and ablation study is given to show the effectiveness of the proposed method.
strength_and_weaknesses: (1) This paper is well-written and organized.

(2) The idea of learning lightweight object detectors via progressive knowledge distillation and investigating heterogeneous knowledge distillation is interesting.

(3) The experimental results significantly outperform the existing SOTA knowledge distillation methods.


Weakness:

(1) The authors failed to prove that the observation ‘teacher-student capacity gap can be solved by the sequential distillation from multiple teachers arranged into a curriculum’, which is claimed in the Introduction.

(2) Why the heterogeneous knowledge distillation should be performed from the Transformer-based teachers to convolution-based students? Why not from the convolution-based teachers to Transformer-based students?

(3) Is the multiple teachers necessary? How about utilizing a single-teacher network with multiple forward progress? This operation also gives more supervision to the selected student and meanwhile saves the effort of searching for the near-optimal teacher order. Authors need to show the rationality of multiple teachers rather than multiple forward passes. 

(4) More SOTA and widely-adopted detection models should be involved, eg., YoloV4.

(5) This paper repeatedly mentioned about "proper" either in selecting the teacher network or 'arranging teacher sequence'. Indeed, this reflected that this paper proposed a method that is not adaptable enough but based on "proper" (manual) selection and usage to 
achieve good performance. It is actually not easy to be extended by the followers if published.

(6) As mentioned, selecting multiple teachers is actually not very easy in reality, especially for Transformers, which can not be easily trained. But it is more approachable to transfer from CNNs to the transformer as CNNs are already matured.  Therefore, the rationality of this method is less clear.

(7) I do agree that teacher selection is one important aspect, however, currently, the method seems to lack enough novelty and complexity to meet the acceptance bar of ICLR.  

clarity,_quality,_novelty_and_reproducibility: This paper addresses somehow an interesting problem of teacher selection. However, it does don't make enough sense of why selecting multiple transformer models and distill them to learn a CNN student. The selection strategy of teacher models lacks flexibility as it should be based on a "proper" selection and measure for successful distillation. From this, we can see that the method itself is not very intuitive and persuasive. What if bad teachers can still be used for learning a good student? References, such as [1,2,3], can be good examples.

[1] Knowledge Distillation: Bad Models Can Be Good Role Models" 
[2] Student Customized Knowledge Distillation: Bridging the Gap Between Student and Teacher
[3] Distilling the Undistillable: Learning from a Nasty Teacher
summary_of_the_review: This paper is somewhat interesting but lacks rationality of clear motivation, technical novelty, and critical ablation studies. Details can be referred to the strength and weakness
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel knowledge distillation methods which decide the progressive distillation process by the cost of linear regression on validation set, which achieves good performance on multiple student-teacher settings.
strength_and_weaknesses: Strength: 
1. This paper proposes to choose the policy of student-teacher in knowledge distillation by using the cost of linear regression on the validation set, which is novel in this domain,
2. Good experimental results have been achieved.
3. Sufficient ablation studies have been conducted to demonstrate their performance.

Weakness.
1. Most of the experiments are conducted on RetinaNet and Mask RCNN with different backbones. It will be better if results on more SOTA detection models can be provided, such as Deformable Detr, Yolov4, CenterNet and so on.
2. This paper applies the naive feature-based knowledge distillation as their knowledge distillation method. It is ok since this paper aims to focus on the order of teachers instead of the specific KD method. But It will be better if better more SOTA KD methods can be utilized to evaluate the effectiveness of the proposed method.
clarity,_quality,_novelty_and_reproducibility: Good Clarity, Good Quality, Good Novelty, Poor Reproducibility. No codes are provided.
summary_of_the_review: Please refer to the strength and weaknesses. In summary, I like the core idea of this paper, while some more experiments are still necessary.

Update-2022-11-14
The response from the authors basically addressed my concern. So I increase my score to 8.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple progressive knowledge distillation framework with a sequence of teachers for detectors. 
strength_and_weaknesses: 
Strength:
1. The idea of progressively transferring knowledge from a sequence of teachers to a lightweight detector is somewhat novel. 
2. It represents the first effort to distill knowledge from Transformer-based teacher detectors to convolution-based students.
3. It shows the performance gain comes from better generalization rather than better optimization. 

Weaknesses:
1. Why not compare with other general[4,5,6] / multi-teacher[9,10] / progressive[1-3,7,8] KD methods? 
2. Please clarify the difference between the proposed method and online KD[] algorithms. And it's better to give more experiments and analysis to highlight the strength of your idea. I think that the principle of sequence teacher is similar to online distillation. 
3. The ranking of teachers seems to require empirical design. It's better to give more theoretical analysis. 
4. Another serious concern is that the pre-training of multiple teachers can be cumbersome and time-consuming, making the pipeline complicated and unpractical. 
5. Since the author propose to transfer knowledge between Transformer and CNN, why not adopt Transformer-based student detectors?
6. There is lack of some important baselines [11-16].

## References

[1] Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher. Mirzadeh et al. arXiv:1902.03393

[2] Snapshot Distillation: Teacher-Student Optimization in One Generation. Yang, Chenglin et al. CVPR 2019

[3] Online Knowledge Distillation by Temporal-Spatial Boosting. Li, Chengcheng et al. WACV 2022

[4] Fitnets: Hints for thin deep nets. Romero, Adriana et al. arXiv:1412.6550

[5] Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. Zagoruyko et al. ICLR 2017

[6] Variational Information Distillation for Knowledge Transfer. Ahn, Sungsoo et al. CVPR 2019

[7] Online Knowledge Distillation via Collaborative Learning. Guo, Qiushan et al. CVPR 2020

[8] Knowledge Transfer via Dense Cross-layer Mutual-distillation. ECCV 2020

[9] Learning from Multiple Teacher Networks. You, Shan et al. KDD 2017

[10] Knowledge distillation by on-the-fly native ensemble. Lan, Xu et al. NeurIPS 2018

[11] Learning efficient object detection models with knowledge distillation. Chen, Guobin et al. NeurIPS 2017

[12] Distilling Object Detectors with Fine-grained Feature Imitation. Wang, Tao et al. CVPR 2019

[13] Enabling Incremental Knowledge Transfer for Object Detection at the Edge. arXiv:2004.05746

[14] General Instance Distillation for Object Detection. Dai, Xing et al. CVPR 2021

[15] Distilling Image Classifiers in Object Detectors. Guo, Shuxuan et al. NeurIPS 2021

[16] Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors. ICLR 2021

clarity,_quality,_novelty_and_reproducibility: Clarity: good.
Quality: The organization and architecture of the paper is not very clear, especially the experiment part.
Novelty: somewhat, but a little incremental. The expression can be improved.
Reproducibility: I think it is easy to follow. But the code in appendix is not complete and not conducted by the common MMDetection framework like most works, e.g. FGD, MGD. 

summary_of_the_review: The work is somewhat simple yet effective, my major concern is the novelty and experiments. 
If the authors can make the experimental part simple and clear, and highlight the competitiveness of the results with more SOTAs, I will consider improve the score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the progressive knowledge distillation method for object detection by distilling knowledge from multiple teachers. The main contribution of this work is designing a heuristic algorithm based on the correlation of feature representations to generate the proper sequence of several teachers given a student model. Comprehensive experiments and ablation studies on the COCO dataset evidence the effectiveness of the proposed method.
strength_and_weaknesses: **Strengths**
+ Novel idea. The authors propose a heuristic algorithm to produce a near-optimal sequence of multiple teachers, even not the best, still outperforms multiple KD approaches distilled from a single teacher.
+ Promising results and convincing ablation studies with multiple settings for object detection.

**Weaknesses**

However, there are still some concerns to be addressed:

- Even though the student can get better results with the selected order of the teachers by Algorithm 1, sometimes it is not the best, especially with more teachers. Could the author please provide some potential explanation on this? In other world,  how reliable the proposed Algorithm 1 is?

- As stated in Section 4.1, the student is initialized by an off-the-shell student for the proposed KD method. Does the student use the same strategy of initialization for exp 3 and 7 (Directly distilled by Teacher IV) and blue bars in Figure 4? Moreover, as for Figure 4, it is not quite fair to compare the performance of 2x training with 2 1x training. Please clarify it.

- The capacity gap between the student and the best performance teacher may a problem for knowledge distillation. Maybe the intermediate teacher is better to be distilled for the student. Thus, it will be interesting to compare with the distilled results from the intermediate teachers in Tables 3, 4 and 5, and Figure 4.

**Minor comments**

It is worth discussing a related work [A], which also makes use of several teachers. It is interesting to the research line of KD in object detection.

**Reference**

[A] Guo. et al. Distilling Image Classifiers in Object Detectors. NeurIPS2021

clarity,_quality,_novelty_and_reproducibility: Clarity, Quality: Good. The paper is well-written and easy to follow.  

Novelty: Good. The proposed method improves the KD performance in object detection by using the capacity of different teacher models sequentially.

Reproducibility: Good. The supplementary code is provided. 
summary_of_the_review: The paper is in good shape with extensive experiments, which show promising results. However, there are still some concerns. Please clarify them.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a progressive distillation strategy for object detection and instance segmentation models, i.e., multiple teacher models transfer knowledge to a student model in a sequential manner. A heuristic method is also given to choose the order of teacher models and its efficacy is demonstrated by experiments. A method for distilling knowledge from transformer-based teachers to convolutional-based students is simply introduced. Many experiments are done to verify the capability of the proposed strategy and ablation studies are provided.
strength_and_weaknesses: Strengths:
1. The sequential strategy is inspiring and of interest for knowledge distillation for object detection and segmentation.
2. The performance of the proposed method is great, as shown in the results.
3. Comparison between the sequential way and simultaneous way is of interest. 

Weakness:
1. Compared to other studies in the paper, the introduction to the distillation from the transformer-based model to the convolutional-based model is simple. For example, it seems that the transformer-based model is not combined with other models as teacher models. It would be better to provide a possible way or give some discussions in the paper.
2. How to address a large k, i.e., the maximum number of selected teachers, or how to pick a good k is not given or well-discussed. In practice, it may happen when users have a large pool of teachers.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
In general, the clarity and quality of this paper are good.

Novelty:
To the best knowledge, the paper provides some novelty ideas for this area.

summary_of_the_review: The reviewer generally thinks the paper is written in sound quality as specified by the strengths above. It proposes a sequential manner for knowledge distillation. However, it would be recommended that authors address the weakness mentioned above and also the following comments:

1. In table 7, for length 4, there is no bold sequence marked. Is this a result of algorithm 1? Please add some explanations.
2. Maybe give a concrete example of $r$ in (1) for two concrete models in a heterogeneous case. It also would be great to see how the transformer-based model is addressed here.

-------------------
Update: I believe the authors well-addressed my comments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission introduces a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teachers to a lightweight student. Reviewers overall like the good results but are concerned about the technical novelty of the work. Post rebuttal, there is a lot of discussion about this submission.  AC and reviewers discussed the paper on Open Review (which are not visible to authors or public), and the AC invites both positive and negative reviewer to comment and exchange opinions. Reviewers have reached the consensus that the technical novelty of this paper is limited, given the vast amount of closely relevant work listed by multiple reviewers.  Reviewer XUDr was the most positive about the submission; unfortunately, the AC was not able to get the reviewer to participate in any discussion or respond to the novelty concern.  However, XUDr's updated comments and the authors' note to AC were all taken into consideration. 

Considering all the pros and cons, the AC recommends rejection so that the authors will be able to systematically address the concerns on novelty. The authors are encouraged to clearly state and demonstrate how this submission adds to the extensive literature on knowledge distillation.
justification_for_why_not_higher_score: Reviewers have reached the consensus that the technical novelty of this paper is limited, given the vast amount of closely relevant work listed by multiple reviewers.
Decision: Reject