Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new method on feature distillation. They utilize a transformer-like module for feature alignment between each stage of the teacher and student networks. Considering the gaps between teacher and student network, they propose the mask ratios to dynamically guide the feature distillation training. The experiments obtain good performance on image classification (CIFAR-10 and ImageNet) and object detection (COCO). 
strength_and_weaknesses: Strength:
	
1. The authors utilize a transformer-like module to integrate the student and teacher features for feature alignment.
2. Considering the capacity gaps between teacher and student networks, the authors propose a dynamic mechanism for training.
3. The authors do lots of experiments and the proposed method obtain good performance on image classification and objection detection tasks.
4. The paper is well-organized and easy to follow. 
	
Weakness:
	
1. The technical novelty is marginally novel for the community. Although lots of ablation studies and comparisons demonstrate the good performance of the proposed method, this paper is an incremental work. The authors introduce widely used transformer module and mask ratio for feature distillations.  
2. The authors should give more analysis of the Receptive Field (RF) for feature distillation. For object detection, how to facilitate the different RF between teacher and student features is important for feature distillation. 
3. In Appendix 4, I notice that CRD outperforms the proposed DPK on four settings. Is the performance also influenced by the RF?
4. I am also curious about the values of dynamic mask ratios for each stage in training. According to the difference between receptive filed, the mask ratio for the shallower stage should be larger than the deeper stage. 
5. The similar work [1] should be discussed and compared in the paper. In [1], they use nonlocal module, which is similar to transformer, for feature distillation of object detection. Although the teacher backbone is a little different, they obtain better performance (e.g. Faster RCNN R50(S), 41.5[1] vs 40.6). 


[1] Zhang, Linfeng, and Kaisheng Ma. "Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors." In International Conference on Learning Representations. 2021.
clarity,_quality,_novelty_and_reproducibility: See strength and weakness.
summary_of_the_review: See strength and weakness. 

This is an incremental work for community. The authors introduce some existing techniques into feature distillation. They do lots of exmperiments to demonstrate the superiorities of the proposed DPK method, but lack detailed analyses of the essential problems for feature distillation. 

I am glad to improve the final rating if the authors address my concerns or point out that I misunderstand  some parts of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the prior knowledge mechanism for feature distillation, which can fully excavate the distillation potential of big models. Furthermore, the authors propose the dynamic prior knowledge (DPK) to solve the ‘larger models are not always better teachers’ issue, which makes the performance of the student model positively correlated with that of the teacher model.
Finally, experiments are conducted to verify the effectiveness of the proposed algorithm.

strength_and_weaknesses: Main Review:

Strength: 

1)This paper presents a dynamic prior knowledge mechanism for feature distillation to provide a solution to the ‘larger models are not always better teachers’ issue. The proposed method is simple and technically sound.  

2)The experimental results of the proposed method outperform that of the other comparable distillation knowledge models.

3)This paper is well-written and easy to follow.

Weaknesses:

1)The technical novelty of the proposed method is somewhat limited, and the authors did not provide theoretical analysis to support the proposed algorithm. This article lacks a strong reason to explain why the proposed prior knowledge mechanism can perform well. What is the relationship between the proposed prior knowledge mechanism and the feature gap?

2)There is no detailed description of the progress of knowledge distillation research in the introduction. Are there any other articles that attempt to address the ‘larger models are not always better teachers’ issue? I think this article lacks the most relevant baseline, e.g. [1].

3)Is the prior knowledge mechanism that incorporates the teacher's feature into the student model fair in comparison to other baselines?

4)This paper lacks complexity analysis, which is important.

[1] Beyer L, Zhai X, Royer A, et al. Knowledge distillation: A good teacher is patient and consistent[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10925-10934.

clarity,_quality,_novelty_and_reproducibility: This paper presents a dynamic prior knowledge mechanism for feature distillation to provide a solution to the ‘larger models are not always better teachers’ issue. The proposed method is simple and technically sound. However, the motivation behind "why prior knowledge can relieve the feature gap" is unclear.
summary_of_the_review: It lacks evidence and reasons to support its assumption on prior knowledge, and also this assumption may be unfair to the general teaching. Besides, the authors avoided some related work from machine teaching. I don't know why.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to use dynamic prior knowledge (DPK) that uses the teacher’s feature as inputs for the feature distillation by mixing the student features and using an encoder-decoder framework that learns the teacher’s feature as the target. Using DPK has an advantage in that the performance of the student network has a positive correlation with that of the teacher.
The performance gain in image classification datasets (CIFAR-100, ImageNet) and object detection (COCO) is remarkable, achieving SOTA performance.

strength_and_weaknesses: Strength
- To the author’s best of knowledge, this work is the first work to take the features of teachers as ‘input’, not just ‘target’ in knowledge distillation. If this is true, this paper can take credit for novelty. Also, this work showed the ‘large models are not always better teachers’ issue and tried to solve the issue with the proposed idea. This work did a good job of analyzing the introduced problem with an experiment. The proposed method shows the SOTA performance in various benchmarks regarding the classification and the object detection task. 

Weakness
- It seems that the transformation module is too heavy, for example, it uses the ViT encoder and decoder module as the feature transformation module which requires heavy computation and process. Also, its dynamic mechanism process is a little hard to follow. 
In table 8, the Encoder-Decoder shows only marginal performance gain compared to MLP-Decoder, considering the heavy computation required by the ViT encoder.

- Since the encoder-decoder is ViT, it would have been more informative to address the training computational cost, even though it does not affect the test phase.



clarity,_quality,_novelty_and_reproducibility: The paper is well-written and shows justifying ablation study results that support the proposed idea of this work. Also, it shows SOTA performance compared to existing methods. Its overall clarity, quality, and novelty are good enough, and it provides detailed information to reproduce the reported results. 

summary_of_the_review: I recommend this paper to be accepted since it tackles a very important issue in knowledge distillation and solved the issue with a novel idea. It has done extensive experiments to support the proposed idea and the results.  However, its performance gain shown in the ablation study is somewhat marginal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary 

This paper studies the KD problem when the teacher model is much stronger (i.e., having larger capacity) than the student model and aims to address the model capacity gap issue. To this end, it proposes a dynamic prior knowledge (DPK) method to address this issue. Specifically, the DPK method proposes to replace some features at certain spatial location by the features output from the teachers, in order to relieve the KD-based learning difficulty for the student model. 

Strength:

- This paper studies an important problem in KD field, i.e. how to address the capacity gap issue between the teacher and student models.
- The proposed method, DPK, is new in terms to using the teacher model's features as "prior knowledge".

Weakness:
- The overall pipeline is complicated. 
- The ViT encoder-decoder seems to contribute a lot to the performance improvement, making it unclear that how much the teacher prior knowledge indeed contributes.
- The novelty is not significant. Using masked teacher features as target for training a student model has been explored by existing works, like BEIT-V3. Although the concrete setting is different, the idea is similar. 
justification_for_why_not_higher_score: The reviewers raise the concerns on the limited novelty, comparison with related works and the marginal performance improvement in the experiments. AC reads these comments, the author's response and the paper, and agree with these comments. The performance improvement seems to be more from the ViT encoder-decoder. The idea of masking features in KD has been explored by prior works. 
justification_for_why_not_lower_score: This paper investigates an important problem when applying KD in practice. The proposed method is reasonable and inspiring for the community. Most of the reviewers also agree with accepting this paper.
Decision: Accept: poster