Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this submission, the authors begin with a clear motivation that the feature value magnitude of the teacher and student is different in pixels, channels and stages for different detector pipelines with different backbones. Motivated by this, the authors propose the Pearson correlation coefficient based approach to tackle the magnitude difference of the distilled features between both homogeneous and heterogeneous detectors. Experiments and ablation studies on the COCO dataset evidence the effectiveness of the proposed method with part of the experiment settings, however, there are some results that should be clarified. Please refer to the weaknesses part.
strengths_and_weaknesses: **Strengths**
+ The submission is well-written and easy to follow.
+ The motivation is clearly explained and evidenced by comprehensive experiments with promising results under some teacher-student pairs settings.

**Weaknesses**

However, there are still some concerns to be addressed:

1. The normalization process proposed in lines 166-175 is somehow vague. Please clarify it in more details, like $\hat{s} = ?$.

2. To my understanding, the proposed PCC is able to normalize the magnitude of the feature map so that it can compute the loss on the pre-normalized feature maps. Here comes a question, how will other feature-based distillation methods work with the normalized feature maps? 

3. As for the 'MSE' in Table 4, could you please clarify whether the MSE works on the pre-normalized feature or the post-normalized feature?

4. Table 2 shows the results of different distillations in the homogeneous setting. The SOTA result is the proposed PKD with inheriting strategy. But, the inheriting strategy which initializes the student with the teacher’s neck and head parameters and gets better results can be applied to all other methods as well in Table 2. To this end, it is unfair to claim PKD is the SOTA.

5. Table 3 also misses some other feature-based distillation methods between heterogeneous detectors, like FKD[37] to further evidence the effectiveness of the PKD. 

questions: I am open to changing my mind if the authors addressed my concerns above.
limitations: No. They claim there is no potential negative social impact of this work to their knowledge.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new KD method for object detection model. They distill the features of FPN by first normalize the output of the feature and then minimize the MSE loss which is equivalant to maximizing the pearson correlation coefficient. The experimental results demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Pros
- The paper is well written and easy to understand.

- The experimental results well demonstrate the effectiveness of the proposed method.

Cons:
- It seems that conducting MSE loss between features after normalizing the feature is not novel. Many KD papers for image classification add conv-bn layers after the output of the feature map before applying MSE loss, which seems quite similar to the proposed method.

- I am confused about Sec.3.4, which shows that the proposed method equals to using KL divergence with a quite large T. However, when T is large, the KL divergence flatten the two feature maps, and makes the original distribution less important. If this is so, why is the difference of the FPN feature maps between student and teacher still matter? 
questions: - Is teacher the first line in Fig.1? Besides, the font size is too small in Fig.1.

- Lack of ablation studies of using KL divergence with different T.

- Other questions see cons above.
limitations: See above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Feature imitation is one of the widely used knowledge distillation techniques, where MSE loss is widely used to enforce the consistency of feature representation between the teacher-student pair. This paper proposes a new loss function for feature imitation in object detection. It first conducts feature standardization and then calculates the MSE loss. Methematically, it is equivalent to firstly calculate the Pearson Correlation Coefficient $r$ between two feature vectors, and then use 1-$r$ as the new feature imitation loss. Expreiments show that it can be better than traditional MSE loss in distilling both heterogeneous detector pairs and homogeneous detector pairs.
strengths_and_weaknesses: Strengths:

1. I highly like the contribution of this paper, because I've also observed before that this idea, the correlation imitation loss (replace MSE with 1-PCC) did work. And it did speed up the training process, in other words, it did reduce the training difficulty in the early training stage. However, from my observation, the performance improvement is limited. The AP can be further improved by only +0.2 based on LD [r2] +KD+GI imitation [9] (42.4$\rightarrow$42.6, GFLv1 Res101 2x $\rightarrow$ GFLv1 Res50 1x). I just simply replaced the MSE loss of GI imitation with correlation imitation loss (1-PCC), and searched a best loss weight of course.

2. This paper also incorporate their method to distill heterogeneous detector and demonstrate that PKD works better than traditional MSE loss. They find that feature standardization is the key to conduct feature imitation for distilling the heterogeneous detectors. This is really a good idea.

3. Due to its novelty, effectiveness and simplicity, it has a potential to serve as a strong baseline for future works on knowledge distillation for object detection.

Weaknesses:

1. In Sec. 3.2, **the fine-tuning experiments cannot support your argument** “Hence, FPN feature imitation is applicable for heterogeneous detectors after removing the negative impacts of magnitude difference.” (line 158-159). If you want to make that argument, you must list the results of Table 4 and need further improvement. For example, FCOS-ResX101 $\rightarrow$ Retina-Res50 MSE is worse than baseline (use Table B in the Appendix). GFL-Res101 $\rightarrow$ FCOS-Res50 MSE and Retina-ResX101 $\rightarrow$ Retina-Res50 MSE works well. This phenomenon combined with the feature magnitude difference, **not fine-tuning**, shows the importance of feature standardization which aligns the features between the heterogeneous detectors so as to make the feature imitation more efficient. As for the homogeneous detectors, the feature magnitude difference is relatively small. And your PKD can still produce a slight improvement. (This is very important to the whole story and it affects my rating score by at least 1 point.)

2. From Table 4, it can be seen that PKD can only improve the performance by +0.4~0.5 AP over MSE loss and even no improvement (RetinaNet Res101 $\rightarrow$ Res50) if homogeneous teachers are used. In Table 2, you compare PKD with various KD methods such as FRS and FGD. It is confusing to me that Retina-ResX101 $\rightarrow$ Retina-Res50 FGD gets 40.4 AP and your PKD gets 40.8 AP, while GFL-Res101 $\rightarrow$ GFL-Res50 FGD gets 41.3 AP and PKD gets 43.3 AP. **Why is there such a big difference?** I notice that you implement your method by setting the imitation mask to 1, which means that your PKD is based on FitNet, i.e., full map feature imitation, right? If it is true, your results in Table 2 is weird to me, e.g., GFL-Res101 $\rightarrow$ GFL-Res50 1x PKD is 2 point higher than FGD, but 0.3 lower than FRS. It seems that PKD+FitNet can already outperform these feature imitation methods, like FRS and FGD. However, I believe that PKD can be combined with these feature imitation methods since only the MSE loss needs to be replaced. And this is exactly what you show in Table 4. Therefore, you must check the context in Sec. 3.3 and the experiments, and **express the implementation details correctly**. You must emphasize which feature imitation method the PKD results in Table 2 are based on.

3. In 4.3.3 Convergence Speed, the comparison of Table 5 is **unfair**. You choose FCOS-ResX101 as the teacher, while all the others are Retina-Res101. It must be the same Retina-Res101 to compare your PKD with the others. Notice that, in Table 6, FCOS ResX101 $\rightarrow$ RetinaNet Res50 2x gets 41.3 AP, i.e., the 2x training schedule can get higher AP than 1x, which is about +1.0. And in Table 4, Retina-ResX101 $\rightarrow$ Retina-Res50 PKD 2x gets 40.8 AP. Therefore, the performance of Retina-Res101 $\rightarrow$ Retina-Res50 PKD 1x seems to be 39.3 around. BTW, from my observation, the training convergence can be significantly speed up in the early training stage. Here are my previous results: GFLv1 Res101 2x $\rightarrow$ GFLv1 Res50 1x.

|Method / epoch | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
LD+KD+GI imitation (MSE) | 20.7 | 27.1 | 31.4 | 32.9 | 35.2 | 36.3 | 36.7 | 37.7 | 41.8 | 42.1 | 42.3 | 42.4 |
LD+KD+GI imitation (CI loss) | 25.5 | 30.4 | 33.8 | 35.3 | 36.7 | 37.1 | 37.8 | 37.8 | 42.2 | 42.3 | 42.4 | 42.6 |

Therefore, if you want to show the convergence speed, you'd better draw some AP curves to exhibit.

4. From Sec. 3.4 "Connection of PCC and KL divergence", if the temperature $T$ is high, minimizing KL divergence between post-normalized features is equivalent to minimizing MSE between post-normalized ones,  and equivalent to PKD. However, the experiment results in the Appendix only show the performance of $T=50$. What would be the performance if lower temperatures were used?

5. In object detection, some feature imitation methods transfer the knowledge within some pre-defined imitation regions, and be validated to be better than distilling on the full feature map. While your method operates on full map. How does PKD compare to these approaches, e.g., Fine-Grained feature imitation [33], GI imitation [9]? More specically, conducting PKD within some pre-defined imitation regions, **not just comparing to the results from the original papers**. I know it will work, but just want to see these relevant experiments in the paper, cuz it's about the universality of this method.

6. Part of the experiments are redundant. You spend lots of space to show the effectiveness of PKD on various detectors, such as RetinaNet with different backbones, FCOS, GFL and TOOD. These one-stage detectors share almost the same framework. It would be better if we could choose the results and leave the space for the more important experiments, see above. BTW, Sec. 4.3.2 " Sensitivity study of loss weight $\alpha$" is not important and can be moved to the Appendix.

7. The inheriting strategy of [16] is unnecessary at all. It is irrelevant to your contribution. What readers really want to see is to what extent your PKD helps improve the distillation performance, not anything else. This problem is mentioned in Weakness 2 either. As I know, some previous works (like FRS) use both feature imitation and logit mimicking (classification KD). One solution is to specify which method your PKD is based on. Since PKD is made for replacing MSE loss, it is not difficult to implement that. For example:

|Method | schedule | AP | AP50 | AP75 | APs | APm | APl |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
GFL-Res101 (T) | 1x | 44.9 | 
GFL-Res50 (S) | 1x | 40.2 | 
FitNet | 1x | xxx
GI imitation | 1x | xxx |
FGFI | 1x | xxx |
FRS | 1x | 43.6 |
FGD | 1x | 41.3 |
||
FitNet + PKD | 1x | xxx
GI imitation + PKD | 1x | xxx |
FGFI + PKD | 1x | xxx |
FRS + PKD | 1x | xxx |
FGD + PKD | 1x | xxx |

And the same goes for the heterogeneous detector pair (choose two typical pairs are enough. One is a detector pair that MSE fails. The other is MSE works). The above two tables are Sec. 4.1 "Main Results", the rest detector pairs can be reported in Sec. 4.2, namely "More Teacher-Student Pairs", and picked the best results of your PKD. (among FitNet+PKD, GI imitation+PKD, FGFI+PKD, FRS+PKD, FGD+PKD ect.)

8. Some important relavant works are missing.

[r1] Chen Y, Chen P, Liu S, et al. Deep structured instance graph for distilling object detectors[C] Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 4359-4368.

[r2] Zheng Z, Ye R, Hou Q, et al. Localization distillation for object detection[J]. arXiv preprint arXiv:2204.05957, 2022.
questions: see weakness
limitations: I like the novelty, the effectiveness, and the simplicity of this paper. However, my pre-rebuttal rating score is 4 because I have some major concerns about the arguments and experiments.

The most important concerns are Weakness 1, 2, 3. If they are sovled correctly, I will raise my rating score. If there still exists one of the Weakness 1, 2, 3, I will not change the rating score. I'm happy to discuss with the authors and other reviewers and exchange the ideas about this paper.

The story needs to be reorganized. The paper needs a major revision.

The best modifications are:

1. See Weakness 1, remove the fine-tuning Table 1. Combining Fig. 1 and Table B in the Appendix "FCOS-ResX101 $\rightarrow$ Retina-Res50 1x 33.9 (-2.6)" "GFL-Res101 $\rightarrow$ FCOS-Res50 1x 39.2 (+2.6)". Then you make the argument that the feature misalignment, or something like the feature magnitude difference, severely affects the performance of feature imitation. According to these observations, we propose ... We find that feature standardization is the key to effectively mitigate ... As shown in Fig 2, the feature standardization can ...

2. See Weakness 2, 5 and 7.

3. See Weakness 3.

The authors should also describe the property of the proposed loss function, about its boundness, optimization goal (linear correlation of two features), more relaxed than MSE loss, and the limit cases (if $\mathcal{L}_{FPN}\rightarrow0,1,2$, what are the meanings of these situations.)

After making all the above modification, it can meet the acceptance bar.
review_title: null

Meta-review: metareview: This paper proposes a novel knowledge distillation method for object detection. As the feature value magnitude of the teacher and student is different, a new loss function for feature imitation is introduced by conducting feature standardization and calculating the MSE loss, which  is equivalent to calculate the Pearson Correlation Coefficient between two features. Experimental results demonstrate the effectiveness of the proposed method. After an in-depth discussion between the authors and reviewers, the concerns have been well addressed. All the reviewers recommend acceptance. Considering that the overall quality is clearly above the bar, the paper should be accepted for publication. The AC strongly urges the authors to consider all the comments in preparing the final version.
confidence: Certain
Decision: Accept