Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection

Authors: Junfei YI, Jianxu Mao, Tengfei Liu, Mingjie Li, Hanyu Gu, Hui Zhang, Xiaojun Chang, Yaonan Wang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Object Detection, Knowledge Distillation, Uncertainty Estimation

Abstract:
Knowledge distillation (KD) has become a fundamental technique for model compression in object detection tasks. The data noise and training randomness may cause the knowledge of the teacher model to be unreliable, referred to as knowledge uncertainty. Existing methods only transfer this knowledge and could limit the student's ability to capture and understand the potential ``dark knowledge''. In this work, we introduce a new strategy that explicitly incorporates knowledge uncertainty, named Uncertainty-Driven Knowledge Extraction and Transfer (UET). Given that the knowledge distribution is unknown and high-dimensional in practice, we introduce a simple yet effective sampling method with Monte Carlo dropout (MC dropout)  to estimate the teacher’s knowledge uncertainty. Leveraging information theory, we integrate knowledge uncertainty into the conventional KD process, allowing the student model to benefit from knowledge diversity. UET is a plug-and-play method that integrates seamlessly with existing distillation techniques. We validate our approach through comprehensive experiments across various distillation strategies, detectors, and backbones. Specifically, UET achieves state-of-the-art results, with a ResNet50-based GFL detector obtaining 44.1\% mAP on the COCO dataset—surpassing baseline performance by 3.9\%.

Decision: reject
Reviewer scores: 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730263795578

thread_type:
official_review

<thread_object>
creator: reviewer_mlfv

<content>
summary:
In this work, we introduce a new strategy that explicitly incorporates knowledge uncertainty, named Uncertainty-Driven Knowledge Extraction and Transfer (UET). The author introduces a simple yet effective sampling method with Monte Carlo dropout (MC dropout) to estimate the teacher’s knowledge uncertainty, integrating knowledge uncertainty into the conventional KD process. The author estimates the proposed method on COCO dataset and achieves 44.1 mAP performance, surpassing baseline by 3.9.

strengths:
- The paper introduces uncertainty estimation into object detection KD, which is kind novel in my view.
- The proposed UET achieves a SOTA performace on COCO dataset.

weaknesses:
- A minor weaknesses I think is that the Monte Carlo dropout needs to sample several times, which may slow down the training process.

questions:
- As shown in Tab 7, LD+UET only increase performance by 0.2 mAP from the LD only model. I am curious about why logit-based KD methods seem to less benefit from UET.
- The UET method adds teacher uncertainty knowledge into the original teacher features. Can the student also generates a uncertainty features and direct imitate the feature uncertainty, which may be more proper in my view.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377636000

thread_type:
official_review

<thread_object>
creator: reviewer_kb7d

<content>
summary:
On this paper, the authors develop a new knowledge-distillation method targeted at the task of object detection. The method works by measuring the uncertainty of the teacher (using MC-dropout) before passing the knowledge to the student, with the idea that this should reduce the amount of noise passed to the student. The method is quite general (it works in both feature and logit distillation, and is trivially extended to classification and segmentation tasks) and is measured in MS-COCO using several convolutional-based backbones, showing that it outperforms other baselines.

strengths:
I think the paper has the following strengths:

1) The idea is extremely simple, and seems to work okay in practice. In general, I tend to like papers that find gaps in literature, and are obvious in hindsight. So, while it can look A (knowledge distillation) + B (MC-dropout) = C (better results), I actually think that this is a strength rather than weakness of the paper, and are quite happy to see that at times simple ideas can outperform more complex ones.

2) The paper is relatively well written. There are some concerns about the writing (see the weaknesses section), but in general after reading the paper, I think that I understood it quite well, and would be able to reproduce it from scratch without many issues.

3) The paper for most part outperforms the other methods in several detectors, using generalized focal loss, Faster-RCNN, Retina-Net (focal loss) and FCOS, with a ResNet101 teacher and ResNet-50 student, showing a relative completeness of results in both one-stage and two-stage detectors. Furthermore, the authors show results with smaller backbones (Res18/Res-34) in Appendix, and results in classification and segmentation at the end of the paper.

weaknesses:
I think the paper has the following weaknesses:

1) While the paper is relatively well-written, it still has some writing issues:

a) I feel like the writing in the Method part is slightly obfuscated and gives the feeling that the method is more complex than it is. For example, I find the equations 2 and 3 quite superflous and the writer could have directly gone to equation 4 with minimal changes. I think that the authors should be proud of their simple idea that works well, instead of making it deliberately look more complex.

b) The big claim in the first page 'Why rely on a single teacher's deterministic knowledge when both teachers or even additional ones, may offer diverse insights?' while completely true, is something that is well known in the distillation literature and has been extensively studied in the past. The authors should cite at this sentence previous works:

[A] Son et al., Densely Guided Knowledge Distillation using Multiple Teacher Assistants, ICCV 2021.

[B] Wu et al., One Teacher is Enough?Pre-trained Language Model Distillation from Multiple Teachers, ACL 2021

[C] Pham et al., Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks, WACV 2023

[D] Liu et al., Adaptive Multi-Teacher Multi-level Knowledge Distillation, Neurocomputing 2020

The papers are sufficiently different to this one (ensembles instead of MC-dropout) so putting them in context of this work would strengthen, instead of weakening the paper.

2) The results while at the first look might look strong, there are a few inconsistencies there.

a) In Table 1, the results of CrossKD are not those from the paper, but results reproduced by the authors. It is unclear why the authors have done so, and when you look in the paper, the results are higher than those provided by the authors.

Method mAP AP50 AP75 APS APM APL

CrossKD_authors 43.6 61.9 47.4 26.1 47.9 56.4

CrossKD_paper 43.9 62.0 47.7 26.4 48.5 56.9

UET 44.1 62.3 47.8 26.6 48.2 56.9

As can be seen, the performance improvement from CrossKD (results reported in the paper) is marginal, only 0.2 percentage points in mAP score. Which makes the results of this paper look less impressive than what is reported.

b) Similarly, the results of FGD method reported in Table 2 under FCOS setting are not consistent with those in the paper.

Method mAP APS APM APL

FGD_authors 42.1 27.0 46.0 54.6

FGD_paper 42.7 27.2 46.5 55.5

UET 42.9 27.1 46.8 54.7

In fact, as can be seen, there is next to no improvement in this method when compared to official results of the FGD.

3) Lack of comparisons with Transformer models. While I appreciate the comprehensive evaluation done by the authors, I must say that I was disappointed to not see any Transformer-based method in this paper. Considering that Transformers are the de-facto state-of-the-art methods in virtually any machine learning task, it would have been interesting to see how this method works in Transformer setting. Some recent papers that do KD in Transformer settings:

[E] Miles et al., VkD : Improving Knowledge Distillation using Orthogonal Projections, CVPR 2024

[F] Song et al., Vidt: An efficient and effective fully transformer-based object detector (under Token matching setting), ICLR 2022

4) Lack of reporting in calibration. The entire idea of doing distillation adjusted for noise is to also ensure that the results of the detector are more reliable, not only better. To do so, the authors should have provided calibration results, where they show that their method is more precise than the others. Such an analysis would strengthen the paper and potentially offer new insights.

5) Timing performance. While discussed in Appendix A.5, it is completely unclear to me. If the authors are doing 5, 10 and 15 forward passes, how is it possible that the training time almost does not change at all. I know that backprop of the student takes more time than the forward pass of the teacher, but if you are doing k forward steps that should accumulate and increase the timing (the student will be waiting for the forward passes before the 'teacher knowledge' is passed to it).

Furthermore, it was unclear to me if the authors are doing MC-dropout in the student during inference, and if so, how much does that effect the timing.

questions:
I have the following questions which the authors should ideally give a response on:

1) Please justify why the results provided in this paper are different to those reported in the original paper (weakness 2)

2) Would it be possible to have some comparison in Transformer models? (weakness 3)

3) Please provide calibration plots/results (weakness 4) and timing performance (weakness 5).

4) Please adjust the citation discussion with regards to weakness 1.

I like the idea of the paper, but I feel that the paper needs to address these issues before it is ready for acceptance. I am initially scoring the paper as (5) but will improve the score providing that the authors give satisfactory responses in my points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537491099

thread_type:
official_review

<thread_object>
creator: reviewer_phpx

<content>
summary:
This paper presents a method (UET) for improving knowledge distillation on object detection. UET leverages the MC dropout to estimate the teacher’s knowledge uncertainty, which allows the student to distill diverse knowledge from the teacher. Comprehensive experiments demonstrate the effectiveness of UET.

strengths:
1.The motivation is convincing, i.e., multiple teachers can provide more diverse and informative supervision to the student.
2.UET can be seamlessly integrated with the existing KD methods and achieve a new state-of-the-art result with FGD [1] on COCO. 
[1]. Focal and Global Knowledge Distillation for Detectors, 2022, CVPR.
3.	UET is successfully extended into classification and semantic segmentation tasks.

weaknesses:
1.The idea is similar to [1], weakening the overall novelty of this paper, and the major difference with [1] is that the UET also included the original teacher in the distillation process. However, an ablative study on w/ and w/o the original teacher is not given, and the direct comparison with [1] is limited (only Table 3).
[1] Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty, 2023, MM.
2.	The superior results in Table 1 and Table 2 are obtained with FGD, which is a strong approach in distilling detectors. Therefore, the comparison with the SOTA KD methods seems to be unfair.
3.	The generalization of the proposed method is not fully evaluated. It is expected to embed UET to multiple detectors across different KD methods.
4.	 In Table 6, what is the student’s knowledge uncertainty (denoted by S). I cannot find relevant statements in the Methods section. Would the authors clarify this.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732200559256

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the AC and all reviewers for their valuable time and constructive comments. We are encouraged by the reviewers' acknowledgments regarding the novelty of our UET paradigm and its effectiveness in enhancing knowledge distillation (Reviewer kB7D and mLFV). We appreciate the reviewers' praise for the paper being well-organized, well-written, and easy to follow (Reviewer kB7D). We also thank the reviewers for recognizing the proposed method's effectiveness and improvements (Reviewer pHpX, kB7D, and mLFV). 
Below, we provide detailed responses to the main points raised:

1. **Comparisons with Transformer-Based Models**

We have extended our experiments to include Transformer-to-Transformer configurations. Specifically, we used Swin Transformer-tiny as the teacher backbone and Swin Transformer-nano as the student backbone. The results are shown in follows table. When employing PKD for distillation, the student network achieves an mAP of 32.2. By integrating the proposed UET module, the mAP of the student model is further improved, reaching 32.5. These findings further confirm the generalization capability of UET in Transformer-based settings.

| Methods                 | Schedule | mAP          | $AP_{50}$ | $AP_{75}$ | $AP_S$ | $AP_M$ | $AP_L$ |
|-------------------------|----------|--------------|-----------|-----------|--------|--------|--------|
| SwinT-tiny (T)          | 1$\times$ | 37.3         | 57.5      | 39.9      | 22.7   | 41.0   | 49.6   |
| SwinT-nano (S)          | 1$\times$ | 31.4         | 50.5      | 32.9      | 18.0   | 34.0   | 41.5   |
| PKD                     | 1$\times$ | 32.2 (+0.8)  | 50.5      | 34.2      | 17.2   | 34.9   | 45.3   |
| **PKD+Ours**            | 1$\times$ | **32.5 (+1.1)** | 50.8      | 34.4      | 18.6   | 35.2   | 45.5   |

2. **Generalization to different KDs on RetinaNet detector**

We added experiments embedding UET into multiple KD methods (MGD and PKD) on the RetinaNet detector to evaluate the generalization ability further. The results are presented in the following Table. When using MGD for deterministic knowledge distillation, the student model achieves a performance of 39.3 mAP, representing a 1.9 improvement over the original student model. By adapting MGD to follow the UET paradigm, the student model's performance further increases to 39.6 mAP, achieving a total improvement of 2.2 points over the original student.
Similarly, with PKD, the student model achieves 39.6 mAP, which is a 2.2 improvement over the original student. When PKD is adapted to the UET paradigm, the student model's performance further improves to 39.8 mAP, resulting in a total improvement of 2.4 points over the original student.   

These extensive experiments demonstrate that UET is a versatile and robust framework, capable of improving various KD methods across multiple detectors and configurations. 

| Method                 | Schedule      | mAP           | $AP_{50}$ | $AP_{75}$ | $AP_S$ | $AP_M$ | $AP_L$ |
|------------------------|---------------|---------------|-----------|-----------|--------|--------|--------|
| RetinaNet-Res101 (T)   | 2$\times$, ms | 38.9          | 58.0      | 41.5      | 21.0   | 42.8   | 52.4   |
| RetinaNet-Res50 (S)    | 1$\times$     | 37.4          | 56.7      | 39.6      | 20.6   | 40.7   | 49.7   |
| MGD*                   | 1$\times$     | 39.3 (+1.9)   | 58.6      | 41.9      | 22.3   | 43.2   | 52.3   |
| **MGD+Ours**           | 1$\times$     | **39.6 (+2.2)** | 58.6      | 42.5      | 22.3   | 43.8   | 52.6   |
| PKD*                   | 1$\times$     | 39.6 (+2.2)   | 58.8      | 42.7      | 22.3   | 43.8   | 54.1   |
| **PKD+Ours**           | 1$\times$     | **39.8 (+2.4)** | 58.8      | 42.6      | 22.2   | 43.9   | 53.9   |
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) This paper proposes UET, a method for improving knowledge distillation in object detection by leveraging MC dropout to estimate teacher uncertainty, enabling diverse knowledge transfer, and demonstrates its effectiveness through comprehensive experiments.

(b) Strengths: The paper presents a convincing motivation by highlighting the benefits of multiple teachers providing diverse and informative supervision to students. UET is seamlessly integrated with existing knowledge distillation (KD) methods and achieves state-of-the-art performance. It extends successfully to classification and semantic segmentation tasks, demonstrating versatility across domains. The method shows strong empirical results, outperforming other methods across various detectors, including one-stage and two-stage models, with results on smaller backbones and additional tasks provided for completeness. The writing is clear and allows easy reproducibility.

(c) The weaknesses of the initial submission highlight concerns about novelty, as UET's idea is similar to prior work (AKD), though an ablation study on this aspect is missing. Generalization remains underexplored, as UET is not extensively evaluated across different detectors and KD methods, and comparisons with Transformer-based models are absent. Some results show inconsistencies, and the paper lacks clarity on specific details. Additionally, the use of Monte Carlo dropout for sampling may slow down the training process.

(d) The most important reasons for the AC's decision to reject are: the proposed UET indeed has not significant differences over AKD. There are two reviewers (pHpX, mLFV) engaged in the discussions about the issue. First of all, the AC appreciates that the authors clarify clearly the key differences between UET and AKD: 1) dropout formulation; 2) complementary knowledge integration vs. uncertainty weighting. The AC carefully check the discussions and related papers: From the perspective of the algorithm pipeline, there is indeed not much difference between the two works. The process consists of two steps: the first is feature sampling, and the second is uncertainty-driven distillation. The proposed UET and the related AKD share the same pipeline but just differ in the design details in each step, e.g., MC Dropout vs. perturbation Dropout. That's also the reason why Reviewer pHpX believes that UET has no more significant differences over AKD. Further, from Tabel 3 and 5, the AC notices that the performance of AKD is close to that of UET (43.9 vs. 44.1), and the hyper-parameter could also lead to the performance gap > 0.2 (e.g., M = 15, 43.8; M = 10, 44.1). The optimal performance of UET (44.1) appears to result from careful tuning, particularly with respect to M. However, it is unclear whether the performance of AKD has been similarly optimized through its hyperparameters (the AC fails to find corresponding descriptions about this). It could be unfair as the authors highlight the performance gain of 0.2 over AKD in the revised manuscript.