Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection

## Abstract

Abstract Knowledge distillation (KD) is a widely adopted and effective method for compressing models in object detection tasks. Particularly, feature-based distillation methods have shown remarkable performance. Existing approaches often ignore the uncertainty in the teacher model’s knowledge, which stems from data noise and imperfect training. This limits the student model’s ability to learn latent knowledge, as it may overly rely on the teacher’s imperfect guidance. In this paper, we propose a novel feature-based distillation paradigm with knowledge uncertainty for object detection, termed " U ncertainty Estimation-Discriminative Knowledge E xtraction-Knowledge T ransfer ( UET )", which can seamlessly integrate with existing distillation methods. By leveraging the Monte Carlo dropout technique, we introduce knowledge uncertainty into the training process of the student model, facilitating deeper exploration of latent knowledge. Our method performs effectively during the KD process without requiring intricate structures or extensive computational resources. Extensive experiments validate the effectiveness of our proposed approach across various distillation strategies, detectors, and backbone architectures. Specifically, following our proposed paradigm, the existing FGD method achieves state-of-the-art (SoTA) performance, with ResNet50-based GFL achieving 44.1% mAP on the COCO dataset, surpassing the baselines by 3.9%.

## 1 Introduction

Deep learning (DL) has demonstrated remarkable dominion in various fundamental visual tasks [], such as object detection []. However, the scale and parameterization of models restrict the practical application of DL-based detectors in real-world tasks. To address this, several techniques have been proposed, including pruning [], weight quantization [], and knowledge distillation []. Knowledge distillation (KD), as a means of model compression, aims to utilize well-trained teacher models to guide student model learning for knowledge transfer.

Figure: Figure 1: We chose the popular ResNet-101 GFL framework which trained with 24 epochs and employed multi-scale training on the COCO dataset as the teacher detector. Subsequently, we assess its performance on the training set of the COCO dataset. The heatmap on the left showcases the multi-scale features of the GFL detector, while the middle segment illustrates the prediction results of GFL. On the right, we present the evaluation performance of the GFL.
Refer to caption: x1.png

In object detection tasks, aligning the intermediate representations (feature-level) of the student and teacher models to achieve knowledge transfer is a common practice [, , , ]. In feature-based KD, a paradigm of “Discriminative Knowledge Extraction-Knowledge Transfer (ET)” is typically followed. Object-related knowledge extraction involves designing specific modules to extract effective knowledge from the intermediate features of the teacher model, such as structural knowledge [], and instance-level fine-grained knowledge []. Knowledge transfer involves a loss function that constrains the distance between the extracted knowledge of the student and teacher models, like the $l_{p}$-norm distance. For a specific example, FGD [] uses a GC-block module [] to extract the global relationship knowledge of pixel-to-pixel from the intermediate features of the teacher model and then transfers this knowledge to the student model.

For teacher detectors, the knowledge they acquire inherently exhibits uncertainty. This arises primarily because the teacher detector’s knowledge is derived from learning through training data and parameter optimization. Despite meticulous annotation, the training data for detection can still introduce noise due to subjective judgments by labeling staff, which is referred to as aleatoric uncertainty [], especially in complex scenarios such as occlusions. Moreover, the design of the detector and the inherent randomness in the training process contribute to epistemic uncertainty [], further complicating the knowledge acquired. Essentially, the knowledge possessed by teacher detectors stems from their previous learning experiences, which are inherently imperfect. Therefore, epistemic and aleatoric uncertainties collectively contribute to the knowledge uncertainties.

In reality, uncertainty is an intrinsic characteristic of knowledge, significantly impacting the process of knowledge transfer []. To further explore the uncertainty of teachers’ knowledge, we evaluate the popular teacher detector (GFL []) on the train set of the MS COCO []. Surprisingly, the mean Average Precision (mAP) of the teacher detector only reached 54.4%, as shown in Figure 1 (right). We notice inconsistencies in the model’s multi-scale feature heatmaps in complex scenarios, such as occlusions. For instance, in the first row of the heatmap (Figure 1, left), the teacher detector fails to focus on the red-highlighted area, whereas in the second row, it does. This inconsistency of knowledge may confuse the teacher detector, potentially leading to missed detections (Figure 1 (middle)).
Ignoring knowledge uncertainty and directly proceeding with knowledge extraction and transfer can be unwise, as it may cause the student model to be misled by the teacher’s unreliable knowledge. Explicitly quantifying uncertainty has shown benefits in various areas, such as segmentation [] and localization [].
Existing approaches typically focus on using elegant formulas to quantify uncertainty. Consequently, the success of these methods is highly dependent on the accuracy of the quantification techniques. However, representing quantified uncertainty during the KD process poses challenges, and these imperfect approximation methods may restrict the student’s learning potential.

In this work, we argue that considering the uncertainty of the teacher’s knowledge seriously is crucial in the process of KD for object detection. To address this, we propose a novel paradigm with knowledge uncertainty for feature-based distillation detection, termed Uncertainty Estimation-Discriminative Knowledge Extraction-Knowledge Transfer (UET). Unlike previous approaches, we model the KD process with knowledge uncertainty. And, we propose an uncertainty knowledge estimate method for the KD process. We estimate the uncertainty knowledge of the teacher models using Monte Carlo dropout (MC dropout) [] before knowledge extraction, and then incorporate this knowledge into the training process of the student model. This strategy mitigates the misleading risk of the student model caused by the overconfidence of the teacher model, thereby enhancing the training effectiveness of the student model. Notably, our proposed paradigm can be seamlessly integrated with other distillation methods, offering substantial improvements with minimal computational overhead. We conduct extensive experiments on the MS COCO to verify the effectiveness of our proposed method across various distillation strategies, detectors, and backbone architectures. When adopting FGD as the baseline in our paradigm, ResNet50-based [] detectors GFL [], Faster R-CNN [], RetinaNet [], and FCOS [] can achieve 44.1% mAP, 40.8% mAP, 39.9% mAP, and 42.9% mAP respectively on the COCO dataset, which surpasses the previous advanced KD methods.
In summary, the contributions of this paper can be summarized:

- •
We propose a new paradigm of "Uncertainty Estimation-Discriminative Knowledge Extraction-Knowledge Transfer", for the feature-based distillation methods. This approach introduces knowledge uncertainty into the learning process of the student model, reducing the risk of being misled by the teacher model. UET seamlessly integrates with existing distillation methods, delivering improvements without any additional complexity.

- •
We propose a straightforward yet effective method for uncertainty estimation by combining Monte Carlo dropout, enabling the capture of the teacher model’s uncertainty during the knowledge distillation process.

- •
We conduct extensive experiments on the COCO dataset to verify the effectiveness of the proposed paradigm across various distillation strategies, detectors, and backbone architectures, achieving SoTA performance.

## 2 Related works

### 2.1 Object detection with knowledge distillation

Object detection is one of the most fundamental tasks in image processing. Due to the advancements in deep learning, detectors based on convolutional neural networks (CNNs) have achieved remarkable results [, , , ]. Currently, CNN-based detectors can be categorized into two-stage detectors and one-stage detectors, depending on whether they are equipped with a Region Proposal Network (RPN). Two-stage detectors, such as Faster R-CNN [], utilize RPN for foreground and background object identification, enabling finer detection. On the other hand, one-stage models directly predict targets densely using multi-scale features extracted by FPN [], as seen in RetinaNet [] and GFL []. Additionally, recent developments have introduced anchor-free methods (like FCOS []) to reduce the detector’s reliance on anchors. In this work, we explore the effectiveness of the proposed method across different types of detectors.

Knowledge distillation aims to transfer knowledge from a well-trained but cumbersome teacher detector to a lightweight student detector. Currently, KD in object detection can be categorized into feature-based distillation [, , , , , , , ] and logit-based distillation [, , , ]. Due to its simpler and more uniform form, feature-based knowledge distillation is popular in object detection. Presently, feature-based knowledge distillation follows the paradigm of "ET". Most works focus on the design of discriminative knowledge extraction modules (such as scale-aware knowledge [], relationship-related knowledge [], etc.) and transfer methods (Pearson Correlation Coefficient [], SSIM [], etc.). In contrast to other works, we consider the uncertainty of knowledge in the teacher model and propose a novel UET paradigm for feature-based distillation. We aim to improve the existing feature distillation procedure by following UET to help the student model explore potential knowledge.

### 2.2 Uncertainty modeling

Uncertainty often arises from insufficient knowledge and data during model training, prompting the need for robust uncertainty estimation methods to quantify prediction reliability [, , , , , , , , ]. Recent research has witnessed a surge in exploring uncertainty in the knowledge distillation task. For instance, the UNIX [] proposed to reduce computation costs by combining uncertainty sampling and adaptive mixup to prioritize informative samples. Uncertainty Distillation method [], on the other hand, aimed to quantify prediction uncertainty by training a compact model to mimic the output distribution of a large ensemble of models, enabling efficient and reliable uncertainty estimation. Similarly, the Uncertainty-aware Contrastive Distillation (UCD) [] method attempts to alleviate catastrophic forgetting in incremental semantic segmentation by contrasting features between new and frozen models. Avatar Knowledge Distillation (AKD) [] strived to generate inference ensemble models (Avatars) from a teacher model and adaptively adjust their contribution to knowledge transfer using an uncertainty-aware factor, refining the previous distillation methods for dense prediction tasks. Despite these advances, existing uncertainty estimation methods often require fine-grained modeling, and there has been limited exploration in the object detection domain. To address this limitation, we introduce knowledge uncertainty into the KD framework for object detection, guiding students to learn more potential knowledge to enhance its detection performance.

## 3 Methods

In this section, we first present the conventional ET paradigm of feature distillation. Next, we formulate our proposed UET paradigm for feature distillation, which aims to alleviate the problem of misleading due to excessive reliance on teacher knowledge. Additionally, we provide the pseudocode of the UET paradigm in the Appendix 1. Finally, we introduce an uncertainty estimation method based on MC dropout, which allows for the simple yet effective incorporation of knowledge uncertainty during the KD process. The overview of UET is represented in Figure 2.

Figure: Figure 2: The overview of the proposed UET paradigm.
Refer to caption: x2.png

### 3.1 KD model in ET paradigm

In object detection, we typically construct a multi-scale feature using the FPN network [] to enhance the detector’s perception of features at different scales. In feature-based knowledge distillation, knowledge transfer usually occurs on the multi-scale features $F=[F_{1},\ldots,F_{M}]$, where $F_{i}$ is the $i$-th scale feature after passing an image through the FPN network. The typical value of $M$ in the literature is 5. Conventional feature-based distillation methods follow the ET paradigm. Specifically, knowledge extraction methods $f_{\textbf{E}}(\cdot)$ are first used to extract the discriminative knowledge that the student model is expected to learn from the multi-scale features of the teacher model, such as the GC module [] used in the FGD method [] for extracting pixel-wise relationships. Then, a designed constraint function $d_{\textbf{T}}(\cdot)$ is used to minimize the difference between the discriminative knowledge of the teacher and student models, such as the SSIM [], to complete knowledge transfer. This process can be formalized as:

$$ $\arg\min_{\theta}\mathcal{L}_{\text{KD}}(\theta)=\frac{1}{N}\sum_{i}d_{\textbf {T}}(f_{\textbf{E}}(F^{(T)}(x_{i})),f_{\textbf{E}}(g(F^{(S)}(x_{i},\theta)))),$ (1) $$

where $x_{i}$ is the input image from training data, $\theta$ is the parameters of the student detector, $F^{(T)}(x)$ and $F^{(S)}(x)$ represent the FPN network’s output features of the teacher and student models, respectively. Additionally, $g(x)$ denotes the adaptive function typically used to align the multi-scale features of the teacher and student models. It is worth mentioning that during the training process, only the parameters of the student model are updated through back-propagation.

### 3.2 Formulation of KD with knowledge uncertainty

As mentioned earlier, the inevitable uncertainty in the knowledge of the teacher model arises from data noise and the randomness of training. However, the ET paradigm overlooks the inherent uncertainty in the knowledge of the teacher model, which may cause the student model to be misled by the overconfidence in the teacher’s knowledge.
Assuming that the knowledge of the teacher model, $(\textbf{x},\textbf{F}^{(T)})$, follows a distribution $\mathcal{U}$ on $\mathcal{X}\times\mathcal{Y}$, where $\mathcal{X}$ is the instance space of input images and $\mathcal{Y}$ is the knowledge space. Based on this, the KD process of UET paradigm can be modeled as follows:

$$ $\arg\min_{\theta}\mathcal{L}_{\text{KD}}(\theta)=\mathbb{E}_{\mathcal{U}}\{d_{ \textbf{T}}(f_{\textbf{E}}(\textbf{F}^{(T)}),f_{\textbf{E}}(g(F^{(S)}(\textbf{ x},\theta))))\}.$ (2) $$

However, due to the intractability of computing the expectation for high dimensional distribution, we consider the approximation

$$ $\arg\min_{\theta}\overline{\mathcal{L}}_{\text{KD}}(\theta)=\mathbb{E}_{ \mathcal{U}_{x}}\{d_{\textbf{T}}(f_{\textbf{E}}(\mathbb{E}_{\mathcal{U}_{|x}} \{\textbf{F}^{(T)}\},f_{\textbf{E}}(g(F^{(S)}(\textbf{x},\theta))))\},$ (3) $$

where $\mathcal{U}_{x}$ is the marginal distribution of $x$, and $\mathcal{U}_{|x}$ is the conditional distribution of the knowledge on $x$. Since
$\mathcal{U}_{|x}$ is unknown in practice, it can be approximated if a sample $\{F^{(T)}_{i}(x)|i=1,\ldots,N\}$ is available. Then, we have

$$ $\arg\min_{\theta}\widehat{\mathcal{L}}_{\text{KD}}(\theta)=\mathbb{E}_{ \mathcal{U}_{x}}\{d_{\textbf{T}}(f_{\textbf{E}}(\frac{1}{N}\sum_{i}^{N}F^{(T)} _{i}(x)),f_{\textbf{E}}(g(F^{(S)}(\textbf{x},\theta))))\}.$ (4) $$

From Equation 4, we formulate the KD process with knowledge uncertainty. By minimizing the objective function in Equation 4, we can incorporate the uncertainty of teacher model knowledge into the optimization process of the student model.

### 3.3 Uncertainty estimation

After formulating the feature distillation with knowledge uncertainty, we can make use of the teacher’s knowledge uncertainty in the training process of the student model.
However, the conditional distribution of the teacher’s knowledge is unknown, which results in sampling from this distribution is also challenging. Modeling the uncertainty of the deep learning model has been a topic of considerable attention [, ]. Especially, [] establishes a theoretical connection between dropout training in deep neural networks and Bayesian inference in deep Gaussian processes. More importantly, they introduced a Monte Carlo estimate method based on the dropout, referred to as MC dropout, which can realize the estimating of uncertainty. There has been a lot of work [, , ] estimating uncertainty through MC dropout and achieving outstanding performance in specific fields.
Inspired by this, we utilize MC dropout to estimate the uncertain knowledge of the teacher models. More specifically, we perform $N$-times the teacher’s forward process with dropout and denote the teacher’s knowledge of the $i$-th run as $Dropout_{i}^{(T)}(x)$. The set $\{Dropout_{i}^{(T)}(x)|i=1,\ldots,N\}$ is used as a sample of the uncertainty knowledge $\textbf{F}^{(T)}$ as in Equation 4. Then, we have

$$ $U_{K}=\frac{1}{N}\sum_{i}^{N}F^{(T)}_{i}(x)\approx\frac{1}{N}\sum_{i=1}^{N} Dropout_{i}^{(T)}(x).$ (5) $$

It is impractical and resource-intensive in real-world scenarios with a large sample size of $N$.
Due to the limitations in the number of samples during the actual implementation process, the estimated uncertainty knowledge $U_{k}$ may have a large variance which could potentially limit the effectiveness of our proposed method in knowledge transfer. To mitigate this, we combine the original teacher knowledge with the estimated uncertainty knowledge $U_{K}$ (similar to residual learning), enhancing the student’s training process. Our proposed UET paradigm for feature distillation in object detection can be summarized as:

$$ $\arg\min_{\theta}\mathcal{L}_{\text{KD}}(\theta)=\frac{1}{N}\sum_{i}d_{\textbf {T}}(f_{\textbf{E}}(U_{K}+F^{(T)}(x_{i})),f_{\textbf{E}}(g(F^{(S)}(x_{i}, \theta)))).$ (6) $$

Compared to the traditional ET paradigm, this approach incorporates knowledge uncertainty into the student’s learning process in a straightforward manner. Unlike the complex uncertainty quantification methods typically used, our approach does not require explicit quantification of knowledge uncertainty. Instead, we introduce a simple yet effective method that integrates knowledge uncertainty into the student model’s training with minimal computational cost, which is almost negligible. Additionally, our method functions as a plug-and-play solution for other distillation models, enhancing the performance of the student model without additional complexity.

## 4 Experiments

**Table 1: Results in different type framework on MS COCO.**
| Method | Schedule | mAP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Two-stage detectors |  |  |  |  |  |  |  |
| Faster R-CNN-Res101 (T) | 2$\times$ | 39.8 | 60.1 | 43.3 | 22.5 | 43.6 | 52.8 |
| Faster R-CNN-Res50 (S) | 2$\times$ | 38.4 | 59.0 | 42.0 | 21.5 | 42.1 | 50.3 |
| FitNet [] | 2$\times$ | 38.9 | 59.5 | 42.4 | 21.9 | 42.2 | 51.6 |
| FRS [] | 2$\times$ | 39.5 | 60.1 | 43.3 | 22.3 | 43.6 | 51.7 |
| FGD [] | 2$\times$ | 40.5 | - | - | 22.6 | 44.7 | 53.2 |
| DiffKD [] | 2$\times$ | 40.6 | 60.9 | 43.9 | 23.0 | 44.5 | 54.0 |
| Ours+FGD | 2$\times$ | 40.8 (+2.4) | 61.0 | 44.5 | 23.5 | 44.9 | 53.7 |
| One-stage detectors |  |  |  |  |  |  |  |
| RetinaNet-Res101 (T) | 2$\times$ | 38.9 | 58.0 | 41.5 | 21.0 | 42.8 | 52.4 |
| RetinaNet-Res50 (S) | 2$\times$ | 37.4 | 56.7 | 39.6 | 20.6 | 40.7 | 49.7 |
| FitNet [] | 2$\times$ | 37.4 | 57.1 | 40.0 | 20.8 | 40.8 | 50.9 |
| FRS [] | 2$\times$ | 39.3 | 58.8 | 42.0 | 21.5 | 43.3 | 52.6 |
| CrossKD [] | 2$\times$ | 39.7 | 58.9 | 42.5 | 22.4 | 43.6 | 52.8 |
| FGD [] | 2$\times$ | 39.7 | - | - | 22.0 | 43.7 | 53.6 |
| DiffKD [] | 2$\times$ | 39.7 | 58.6 | 42.1 | 21.6 | 43.8 | 53.3 |
| Ours+FGD | 2$\times$ | 39.9 (+2.5) | 59.0 | 42.7 | 22.1 | 43.9 | 53.4 |
| Anchor-free detectors |  |  |  |  |  |  |  |
| FCOS-Res101 (T) | 2$\times$, ms | 40.8 | 60.0 | 44.0 | 24.2 | 44.3 | 52.4 |
| FCOS-Res50 (S) | 2$\times$, ms | 38.5 | 57.7 | 41.0 | 21.9 | 42.8 | 48.6 |
| FRS [] | 2$\times$ | 40.9 | 60.3 | 43.6 | 25.7 | 45.2 | 51.2 |
| CrossKD [] | 2$\times$ | 41.3 | 60.6 | 44.2 | 25.1 | 45.5 | 52.4 |
| DiffKD [] | 2$\times$ | 42.4 | 61.0 | 45.8 | 26.6 | 45.9 | 54.8 |
| FGD [] | 2$\times$ | 42.7 | - | - | 27.2 | 46.5 | 55.5 |
| Ours+FGD | 2$\times$ | 42.9 (+4.4) | 61.6 | 46.3 | 27.1 | 46.8 | 54.7 |

### 4.1 Settings

To validate the effectiveness of our methods, we conduct extensive experiments on the MS-COCO dataset across various detectors, KDs, and different backbones. We use the 120k images of datasets to train the models and 5k val images for evaluating the experiments. We report mean Average Precision (mAP) as the evaluation metric along with AP at different scales and thresholds, including $AP_{50}$, $AP_{75}$, $AP_{S}$, $AP_{M}$, and $AP_{L}$.
All experiments are conducted on a machine equipped with 2 NVIDIA GeForce RTX 3090 GPUs. Additionally, we select FGD [] as our baseline, completing the "ET" process in our proposed "UET" paradigm. All experiments were implemented using the mmdetection [] and PyTorch framework. Except for setting the batch size to 8, we follow the training settings of FGD []. In this paper, 1$\times$, 2$\times$, and ms respectively denote 12 epochs, 24 epochs, and multi-scale training. Unless otherwise specified, we set $N$=5 in our experiments. The first group employs a ratio strategy with an initial ratio of 0.05 and a common difference of 0.05. In these experiments, we introduce knowledge uncertainty solely on the teacher’s side.

### 4.2 Main results

Comparison experiments with other detectors.
To explore the possibility of our proposed method across different types of detectors, we conduct comparative experiments on Faster R-CNN (two-stage detector), RetinaNet (anchor-based detector), and FCOS (anchor-free detector). The experimental results, as presented in Table 1, demonstrate that our method, when incorporated on top of FGD, achieves SoTA performance across three different types of detectors. Specifically, within the Faster R-CNN framework, we attain a 40.8% mAP, marking a 2.4% improvement over the student model. Similarly, in the RetinaNet detector, we achieve a 39.9% mAP, representing a 2.5% improvement over the student model. Furthermore, within the FCOS detector, we achieve a 42.9% mAP, reflecting a 4.4% improvement over the student model.

Comparison experiments with GFL framework. To further demonstrate the superiority of our approach, we conduct comparisons with the previous SoTA KD methods on the popular GFL framework. In this experiment, we utilize a ResNet101 backbone for the GFL, trained for 24 epochs with multi-scale training, as our teacher network. As for the student model, we employ a ResNet50 backbone for the GFL, trained for 12 epochs. Without resorting to any elaborate techniques, our method achieves new SoTA performance, as depicted in Table 4. Specifically, our approach achieves an mAP of 44.1%, surpassing the previous SoTA distillation methods CrossKD [] (43.7% mAP) and BCKD []
(43.2% mAP), as well as the baseline FGD [] (43.4% mAP). Moreover, our method exhibited a notable improvement of 3.9% over the original student model (40.2% mAP).

These experiments validate that following our proposed UET paradigm effectively enhances the learning potential of student detectors during the KD process. Furthermore, we demonstrate that our paradigm is adaptable to various styles of detectors.

**Table 2: Comparison results in GFL framework on MS COCO. * denotes our reproduced results. We set $N$ to 10, choose the b) as ratio strategy, and only introduce knowledge uncertainty into the teacher side in this experiment.**
| Method | mAP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ |
| --- | --- | --- | --- | --- | --- | --- |
| GFL-Res101 (T) | 44.9 | 63.1 | 49.0 | 28.0 | 49.1 | 57.2 |
| GFL-Res50 (S) | 40.2 | 58.4 | 43.3 | 23.3 | 44.0 | 52.2 |
| FitNets [] | 40.7 | 58.6 | 44.0 | 23.7 | 44.4 | 53.2 |
| Inside GT Box | 40.7 | 58.6 | 44.2 | 23.1 | 44.5 | 53.5 |
| Defeat [] | 40.8 | 58.6 | 44.2 | 24.3 | 44.6 | 53.7 |
| Main Region [] | 41.1 | 58.7 | 44.4 | 24.1 | 44.6 | 53.6 |
| Fine-Grained [] | 41.1 | 58.8 | 44.8 | 23.3 | 45.4 | 53.1 |
| GID [] | 41.5 | 59.6 | 45.2 | 24.3 | 45.7 | 53.6 |
| SKD [] | 42.3 | 60.2 | 45.9 | 24.4 | 46.7 | 55.6 |
| ScaleKD [] | 42.5 | - | - | 25.9 | 46.2 | 54.6 |
| BCKD [] | 43.2 | 61.6 | 46.9 | 25.7 | 47.3 | 55.9 |
| FGD* [] | 43.4 | 61.7 | 47.0 | 26.2 | 47.4 | 56.4 |
| CrossKD [] | 43.7 | 62.1 | 47.4 | 26.9 | 48.0 | 56.2 |
| Ours+FGD | 44.1 | 62.3 | 47.8 | 26.6 | 48.2 | 56.9 |

### 4.3 Ablation analysis

In our proposed method for estimating knowledge uncertainty, dropout technology plays a crucial role. Therefore, in this part, we primarily investigate the impact of dropout on student learning, such as dropout ratio and frequency. Here we adopt the GFL framework, and the experimental settings are consistent with Section 4.2. Next, we will delve into these aspects in detail.

Sensitivity analysis of the ratios strategy. We design three different strategies to investigate the impact of dropout ratio on the model: a) where $N$ groups of ratios are all fixed at 0.15; b) where the first group has a ratio of 0.05 with a common difference of 0.05; c) which builds upon b) by increasing with epochs, at a growth rate of 0.025 per epoch. These results, as shown in Table 4, indicate that our network is not sensitive to different ratio strategies. Particularly, when adopting strategy c), after 10 epochs of training, with a dropout ratio ranging from 0.3 to 0.5, our method still maintains an mAP of 44.0%.

Sensitivity study of the $N$. To explore the influence of the sampling count $N$ in MC dropout on the model, we conduct comparative experiments using different $N$ values. The experimental results, as presented in Table 4, demonstrate the beneficial effect of introducing uncertainty from the teacher model to the student model. Even with just one dropout, the model’s performance reaches 43.9%, a 0.5% improvement over the pure FGD method. When increasing $N$ to 10, we observe the model’s mAP reaching 44.1%, a 0.7% enhancement compared to the pure FGD method. However, with a further increase in $N$ to 15, the model’s mAP reaches 43.8%, a slight decrease of 0.3% compared to $N=10$, yet still outperforming the FGD method by 0.4%.

Effect of knowledge sources.
We conduct comparative experiments on introducing uncertainty from different knowledge sources. As shown in Table 5, when uncertainty is exclusively integrated into the teacher’s knowledge, the mAP reaches 44.0%. Similarly, when it is introduced solely into the student knowledge, the mAP is 43.6%. In both cases, these values represent enhancements of 0.6% and 0.2% respectively compared to the pure FGD method. This suggests that incorporating knowledge uncertainty into the KD process benefits the learning process of the student model. However, when knowledge uncertainty is simultaneously introduced to the teacher and student, the model’s mAP remains at 44.0%. In this case, simultaneously handling both types of uncertain knowledge may have increased learning complexity, making the optimization process more challenging.

Effect of Residual structure. We also explore the impact of residual structures, and the results are shown in Table 5. When we do not introduce the residual structure, the student detector achieves an mAP of 43.5%, which is 0.5% lower than after introducing it. In the presence of a constrained sample size, our methodology may not fully manifest its efficacy. This also indicates that incorporating knowledge uncertainty estimation with residual structures is more beneficial to the learning process of the student detector.

**Table 5: Effect of knowledge sources. Teacher: the uncertainty is exclusively integrated into the teacher’s knowledge, Student: the uncertainty is exclusively integrated into the student’s knowledge, Residual: introduce the residual learning.**
| Teacher | Student | Residual | mAP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | 43.4 | 61.7 | 47.0 | 26.2 | 47.4 | 56.4 |
|  | $\checkmark$ | $\checkmark$ | 43.6 | 61.8 | 47.3 | 26.1 | 47.8 | 56.8 |
| $\checkmark$ |  | $\checkmark$ | 44.0 | 62.2 | 47.7 | 26.8 | 48.4 | 57.0 |
| $\checkmark$ |  |  | 43.5 | 61.7 | 46.9 | 26.0 | 47.7 | 56.4 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | 44.0 | 62.3 | 47.5 | 26.7 | 48.2 | 57.0 |

### 4.4 Generalization to different KDs

To further explore the universality of the proposed "UET" paradigm, we conduct a series of experiments by introducing the knowledge uncertainty into different KD methods. Similarly, we employ the GFL framework for this part, following the configurations outlined in Section 4.2. Moreover, $N$ was set to 5, with a ratio strategy of b) for our uncertainty estimate. Specifically, we conduct the generalization experiments on three advanced feature distillation methods, (MGD [], PKD [], and FGD []), as shown in Table 6. Upon introducing knowledge uncertainty, MGD, PKD, and FGD yielded improvements of 2.8%, 2.4%, and 3.8%, respectively, for the student model, surpassing their corresponding pure versions of KDs. These results confirm the applicability of the proposed paradigm across different feature-based KD situations and also suggest that incorporating knowledge uncertainty can optimize the KD process to a certain extent.

**Table 6: Generalization to different KDs. * denotes our reproduced results.**
| Method | Schedule | mAP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GFL-Res101 (T) | 2$\times$, ms | 44.9 | 63.1 | 49.0 | 28.0 | 49.1 | 57.2 |
| GFL-Res50 (S) | 1$\times$ | 40.2 | 58.4 | 43.3 | 23.3 | 44.0 | 52.2 |
| Logits-based KDs |  |  |  |  |  |  |  |
| LD* [] | 1$\times$ | 41.0 (+0.8) | 58.6 | 44.2 | 23.4 | 45.0 | 53.1 |
| LD+ours | 1$\times$ | 41.2 (+1.0) | 58.7 | 44.5 | 24.4 | 45.1 | 52.9 |
| Feature-based KDs |  |  |  |  |  |  |  |
| MGD* [] | 1$\times$ | 42.1 (+1.9) | 60.3 | 45.8 | 24.4 | 46.2 | 54.7 |
| MGD+Ours | 1$\times$ | 43.0 (+2.8) | 61.3 | 46.4 | 26.1 | 47.1 | 55.7 |
| PKD* [] | 1$\times$ | 42.5 (+2.3) | 60.9 | 46.0 | 24.2 | 46.7 | 55.9 |
| PKD+Ours | 1$\times$ | 42.6 (+2.4) | 60.6 | 46.1 | 23.9 | 46.7 | 55.8 |
| FGD* [] | 1$\times$ | 43.4 (+3.2) | 61.7 | 47.0 | 26.2 | 47.4 | 56.4 |
| FGD+Ours | 1$\times$ | 44.0 (+3.8) | 62.2 | 47.7 | 26.8 | 48.4 | 57.0 |

Extended to logits-based distillation. In Section 4.4, we reveal that introducing knowledge uncertainty into the feature-based distillation process would contribute to student learning. A pertinent question arises: Could the knowledge uncertainty also be applied to logits-based distillation? Compared to feature-based distillation, logit-based distillation directly transfers the output results of the teacher detector, such as classification scores and detection positions. LD [] is one of the representative logits-based distillation methods, which designs a valuable localization region to distill the student detector in a separate distillation region manner. Following the uncertainty estimation manner in feature-based distillation, we introduce it in the output features of the teacher detector’s FPN network and then feed them into the head to obtain predicted logits, as shown in Table 6. As expected, after introducing our method, the detector’s mAP reached 41.2%, compared to the original LD’s mAP of 41.0%. This result further illustrates the importance of introducing knowledge uncertainty for knowledge distillation in object detection.

### 4.5 Discussions and analysis

Extended to lightweight detectors.
Expanding to lightweight detectors presents additional challenges in KD, as the knowledge gap between teachers and students tends to be larger. Furthermore, due to the limitations in learning capacity, student models may not fully comprehend the knowledge of the teacher model. This implies that knowledge incomprehensible to these student models may affect their learning process as noise. Therefore, introducing knowledge uncertainty in the teacher model may be more crucial in the KD process for lightweight detectors. To further explore this, we conduct experiments using the GFL framework, and the results are presented in Table 7. Compared with the setting of Section 4.1, we only alternate the ResNet50 with a lightweight backbone in the student detector. As hypothesized, introducing knowledge uncertainty is crucial in lightweight detectors, leading to significant improvements. When the student model with ResNet34 as the backbone, introducing knowledge uncertainty results in a detector achieving 41.9% mAP, which is 2.2% higher than the 39.7% obtained with pure FGD. We analyze the impact of introducing knowledge uncertainty on the model’s convergence in lightweight detectors in Section A.2.

**Table 7: Quantitative results for lightweight detectors.**
| Student | FGD | Ours | mAP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Res18 |  |  | 35.8 | 53.1 | 38.2 | 18.9 | 38.9 | 47.9 |
| $\checkmark$ |  | 33.3 (-2.5) | 49.2 | 36.0 | 20.3 | 36.1 | 42.7 |  |
| $\checkmark$ | $\checkmark$ | 37.9 (+2.1) | 54.9 | 41.0 | 21.9 | 41.5 | 49.3 |  |
| Res34 |  |  | 38.9 | 56.6 | 42.2 | 21.5 | 42.8 | 51.4 |
| $\checkmark$ |  | 39.7 (+0.8) | 57.1 | 43.0 | 23.0 | 43.7 | 51.2 |  |
| $\checkmark$ | $\checkmark$ | 41.9 (+3.0) | 59.6 | 45.3 | 24.3 | 46.0 | 54.4 |  |

Extended to the backbone of the different types.
We conduct experiments on the RetinaNet detector with heterogeneous and homogeneous backbones, and more details are listed in Section A.3.

Visualization of detection results.
We validate the effectiveness of our proposed method through visualization evidence, and more details are listed in Section A.5.

## 5 Conclusions

In this paper, we investigate the significance of introducing knowledge uncertainty from teacher models in object detection distillation and its impact on the learning process of student detectors. Building upon this, we propose a novel UET general paradigm for feature distillation, aimed at facilitating the acquisition of latent knowledge by student models, while easily being adaptable to other distillation methods. Additionally, we present a simple yet effective uncertainty estimation approach by integrating MC dropout, which seamlessly introduces uncertainty knowledge at minimal computational cost. Extensive experiments validate the effectiveness of following the UET paradigm across various types of KDs, detectors, and backbones. In summary, we demonstrate that introducing teacher knowledge uncertainty enhances the learning capabilities of student models in the KD process.

## Appendix A Appendix / supplemental material

### A.1 Limitations.

In this paper, we utilize MC dropout to estimate the uncertainty in the teacher’s knowledge. In reality, more advanced uncertainty estimation methods could potentially further enhance the impact of the UET paradigm in knowledge distillation. Additionally, we incorporate estimated uncertainty into the original teacher knowledge using the simple structure of residual learning. Employing more effective integration techniques may yield even better performance. However, in this work, the main insight is that we should seriously consider the uncertainty of teachers’ knowledge, and we also provide a simple solution for the KD process with knowledge uncertainty.

### A.2 Convergence Analysis.

We also analyze the impact of introducing knowledge uncertainty on the model’s convergence in lightweight detectors. As illustrated in Figure 3, when we introduce knowledge uncertainty during training, the model’s convergence speed significantly improved. Particularly, with ResNet34 as the backbone, the student model following our paradigm even outperformed the baseline’s performance in the third epoch. This indicates that introducing knowledge uncertainty in KD detection not only enhances student learning ability but also accelerates the convergence speed.

Figure: (a) The lightweight detector with ResNet18.
Refer to caption: x3.png

### A.3 Experiments for detectors with heterogeneous and homogeneous backbone.

We conduct experiments on the RetinaNet detector with both heterogeneous and homogeneous backbones, and the results are presented in Table 8. Unsurprisingly, regardless of the backbone configuration being heterogeneous or homogeneous, the FGD method demonstrates stronger knowledge transfer capabilities after following the UET paradigm. For instance, under the heterogeneous backbone setting of SwinT [], the student detector achieves 37.7% mAP after introducing knowledge uncertainty, surpassing the 37.4% obtained with pure FGD.

**Table 8: Experiments for detectors with Heterogeneous and homogeneous backbone.**
| Methods | Schedule | mAP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SwinT [] (T) | 1$\times$ | 37.3 | 57.5 | 39.9 | 22.7 | 41.0 | 49.6 |
| Res50 [] (S) | 1$\times$ | 36.5 | 55.4 | 39.1 | 20.4 | 40.3 | 48.1 |
| FGD [] | 1$\times$ | 37.4 (+0.9) | 56.8 | 39.9 | 22.6 | 41.3 | 48.7 |
| FGD+Ours | 1$\times$ | 37.7 (+1.2) | 57.2 | 40.2 | 21.9 | 41.7 | 59.0 |
| Res50 [] (T) | 1$\times$ | 36.5 | 55.4 | 39.1 | 20.4 | 40.3 | 48.1 |
| Res50 [] (S) | 1$\times$ | 36.5 | 55.4 | 39.1 | 20.4 | 40.3 | 48.1 |
| FGD [] | 1$\times$ | 37.4 (+0.9) | 56.7 | 39.7 | 20.6 | 40.9 | 49.0 |
| FGD+Ours | 1$\times$ | 37.9 (+1.4) | 57.3 | 40.4 | 21.2 | 41.6 | 50.1 |

### A.4 The pseudo-code of our UET paradigm

In comparison to the ET paradigm, we introduce the UET paradigm by incorporating knowledge uncertainty. Building upon the ET paradigm, we estimate the uncertainty in the teacher detector’s knowledge using Eq. 5 and then perform knowledge transfer according to Eq. 6. Existing feature distillation methods can transform from the ET to the UET paradigm with minimal additional computational overhead (only requires a change in the high light parts of code).

Figure: Algorithm 1 UET Paradigm

### A.5 Visualization of detection results

We provide visual evidence to validate the effectiveness of our proposed method by presenting detection results from the val2017 set of MS COCO []. Figure 4 shows that following our UET paradigm allows FGD to outperform the original FGD in detecting more high-quality bounding boxes. This suggests that our method effectively enhances the learning potential of student detectors during the KD process.

Figure: Figure 4: Visualization of detection results of Our method.
Refer to caption: x5.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weakness.
soundness: 2
strengths: 1.The motivation is convincing, i.e., multiple teachers can provide more diverse and informative supervision to the student.
2.UET can be seamlessly integrated with the existing KD methods and achieve a new state-of-the-art result with FGD [1] on COCO. 
[1]. Focal and Global Knowledge Distillation for Detectors, 2022, CVPR.
3.	UET is successfully extended into classification and semantic segmentation tasks.
summary: This paper presents a method (UET) for improving knowledge distillation on object detection. UET leverages the MC dropout to estimate the teacher’s knowledge uncertainty, which allows the student to distill diverse knowledge from the teacher. Comprehensive experiments demonstrate the effectiveness of UET.
weaknesses: 1.The idea is similar to [1], weakening the overall novelty of this paper, and the major difference with [1] is that the UET also included the original teacher in the distillation process. However, an ablative study on w/ and w/o the original teacher is not given, and the direct comparison with [1] is limited (only Table 3).
[1] Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty, 2023, MM.
2.	The superior results in Table 1 and Table 2 are obtained with FGD, which is a strong approach in distilling detectors. Therefore, the comparison with the SOTA KD methods seems to be unfair.
3.	The generalization of the proposed method is not fully evaluated. It is expected to embed UET to multiple detectors across different KD methods.
4.	 In Table 6, what is the student’s knowledge uncertainty (denoted by S). I cannot find relevant statements in the Methods section. Would the authors clarify this.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: I have the following questions which the authors should ideally give a response on:

1) Please justify why the results provided in this paper are different to those reported in the original paper (weakness 2)

2) Would it be possible to have some comparison in Transformer models? (weakness 3)

3) Please provide calibration plots/results (weakness 4) and timing performance (weakness 5).

4) Please adjust the citation discussion with regards to weakness 1.

I like the idea of the paper, but I feel that the paper needs to address these issues before it is ready for acceptance. I am initially scoring the paper as (5) but will improve the score providing that the authors give satisfactory responses in my points.
soundness: 3
strengths: I think the paper has the following strengths:

1) The idea is extremely simple, and seems to work okay in practice. In general, I tend to like papers that find gaps in literature, and are obvious in hindsight. So, while it can look A (knowledge distillation) + B (MC-dropout) = C (better results), I actually think that this is a strength rather than weakness of the paper, and are quite happy to see that at times simple ideas can outperform more complex ones.

2) The paper is relatively well written. There are some concerns about the writing (see the weaknesses section), but in general after reading the paper, I think that I understood it quite well, and would be able to reproduce it from scratch without many issues.

3) The paper for most part outperforms the other methods in several detectors, using generalized focal loss, Faster-RCNN, Retina-Net (focal loss) and FCOS, with a ResNet101 teacher and ResNet-50 student, showing a relative completeness of results in both one-stage and two-stage detectors. Furthermore, the authors show results with smaller backbones (Res18/Res-34) in Appendix, and results in classification and segmentation at the end of the paper.
summary: On this paper, the authors develop a new knowledge-distillation method targeted at the task of object detection. The method works by measuring the uncertainty of the teacher (using MC-dropout) before passing the knowledge to the student, with the idea that this should reduce the amount of noise passed to the student. The method is quite general (it works in both feature and logit distillation, and is trivially extended to classification and segmentation tasks) and is measured in MS-COCO using several convolutional-based backbones, showing that it outperforms other baselines.
weaknesses: I think the paper has the following weaknesses:

1) While the paper is relatively well-written, it still has some writing issues:

a) I feel like the writing in the Method part is slightly obfuscated and gives the feeling that the method is more complex than it is. For example, I find the equations 2 and 3 quite superflous and the writer could have directly gone to equation 4 with minimal changes. I think that the authors should be proud of their simple idea that works well, instead of making it deliberately look more complex.

b) The big claim in the first page 'Why rely on a single teacher's deterministic knowledge when both teachers or even additional ones, may offer diverse insights?' while completely true, is something that is well known in the distillation literature and has been extensively studied in the past. The authors should cite at this sentence previous works:

[A] Son et al., Densely Guided Knowledge Distillation using Multiple Teacher Assistants, ICCV 2021.

[B] Wu et al., One Teacher is Enough?Pre-trained Language Model Distillation from Multiple Teachers, ACL 2021

[C] Pham et al., Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks, WACV 2023

[D] Liu et al., Adaptive Multi-Teacher Multi-level Knowledge Distillation, Neurocomputing 2020

The papers are sufficiently different to this one (ensembles instead of MC-dropout) so putting them in context of this work would strengthen, instead of weakening the paper.

2) The results while at the first look might look strong, there are a few inconsistencies there.

a) In Table 1, the results of CrossKD are not those from the paper, but results reproduced by the authors. It is unclear why the authors have done so, and when you look in the paper, the results are higher than those provided by the authors.

Method mAP AP50 AP75 APS APM APL

CrossKD_authors 43.6 61.9 47.4 26.1 47.9 56.4

CrossKD_paper 43.9 62.0 47.7 26.4 48.5 56.9

UET 44.1 62.3 47.8 26.6 48.2 56.9

As can be seen, the performance improvement from CrossKD (results reported in the paper) is marginal, only 0.2 percentage points in mAP score. Which makes the results of this paper look less impressive than what is reported.

b) Similarly, the results of FGD method reported in Table 2 under FCOS setting are not consistent with those in the paper.

Method mAP APS APM APL

FGD_authors 42.1 27.0 46.0 54.6

FGD_paper 42.7 27.2 46.5 55.5

UET 42.9 27.1 46.8 54.7

In fact, as can be seen, there is next to no improvement in this method when compared to official results of the FGD.

3) Lack of comparisons with Transformer models. While I appreciate the comprehensive evaluation done by the authors, I must say that I was disappointed to not see any Transformer-based method in this paper. Considering that Transformers are the de-facto state-of-the-art methods in virtually any machine learning task, it would have been interesting to see how this method works in Transformer setting. Some recent papers that do KD in Transformer settings:

[E] Miles et al., VkD : Improving Knowledge Distillation using Orthogonal Projections, CVPR 2024

[F] Song et al., Vidt: An efficient and effective fully transformer-based object detector (under Token matching setting), ICLR 2022

4) Lack of reporting in calibration. The entire idea of doing distillation adjusted for noise is to also ensure that the results of the detector are more reliable, not only better. To do so, the authors should have provided calibration results, where they show that their method is more precise than the others. Such an analysis would strengthen the paper and potentially offer new insights.

5) Timing performance. While discussed in Appendix A.5, it is completely unclear to me. If the authors are doing 5, 10 and 15 forward passes, how is it possible that the training time almost does not change at all. I know that backprop of the student takes more time than the forward pass of the teacher, but if you are doing k forward steps that should accumulate and increase the timing (the student will be waiting for the forward passes before the 'teacher knowledge' is passed to it).

Furthermore, it was unclear to me if the authors are doing MC-dropout in the student during inference, and if so, how much does that effect the timing.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: - As shown in Tab 7, LD+UET only increase performance by 0.2 mAP from the LD only model. I am curious about why logit-based KD methods seem to less benefit from UET.
- The UET method adds teacher uncertainty knowledge into the original teacher features. Can the student also generates a uncertainty features and direct imitate the feature uncertainty, which may be more proper in my view.
soundness: 2
strengths: - The paper introduces uncertainty estimation into object detection KD, which is kind novel in my view.
- The proposed UET achieves a SOTA performace on COCO dataset.
summary: In this work, we introduce a new strategy that explicitly incorporates knowledge uncertainty, named Uncertainty-Driven Knowledge Extraction and Transfer (UET). The author introduces a simple yet effective sampling method with Monte Carlo dropout (MC dropout) to estimate the teacher’s knowledge uncertainty, integrating knowledge uncertainty into the conventional KD process. The author estimates the proposed method on COCO dataset and achieves 44.1 mAP performance, surpassing baseline by 3.9.
weaknesses: - A minor weaknesses I think is that the Monte Carlo dropout needs to sample several times, which may slow down the training process.

Meta-review: (a) This paper proposes UET, a method for improving knowledge distillation in object detection by leveraging MC dropout to estimate teacher uncertainty, enabling diverse knowledge transfer, and demonstrates its effectiveness through comprehensive experiments.

(b) Strengths: The paper presents a convincing motivation by highlighting the benefits of multiple teachers providing diverse and informative supervision to students. UET is seamlessly integrated with existing knowledge distillation (KD) methods and achieves state-of-the-art performance. It extends successfully to classification and semantic segmentation tasks, demonstrating versatility across domains. The method shows strong empirical results, outperforming other methods across various detectors, including one-stage and two-stage models, with results on smaller backbones and additional tasks provided for completeness. The writing is clear and allows easy reproducibility.

(c) The weaknesses of the initial submission highlight concerns about novelty, as UET's idea is similar to prior work (AKD), though an ablation study on this aspect is missing. Generalization remains underexplored, as UET is not extensively evaluated across different detectors and KD methods, and comparisons with Transformer-based models are absent. Some results show inconsistencies, and the paper lacks clarity on specific details. Additionally, the use of Monte Carlo dropout for sampling may slow down the training process.

(d) The most important reasons for the AC's decision to reject are: the proposed UET indeed has not significant differences over AKD. There are two reviewers (pHpX, mLFV) engaged in the discussions about the issue. First of all, the AC appreciates that the authors clarify clearly the key differences between UET and AKD: 1) dropout formulation; 2) complementary knowledge integration vs. uncertainty weighting. The AC carefully check the discussions and related papers: From the perspective of the algorithm pipeline, there is indeed not much difference between the two works. The process consists of two steps: the first is feature sampling, and the second is uncertainty-driven distillation. The proposed UET and the related AKD share the same pipeline but just differ in the design details in each step, e.g., MC Dropout vs. perturbation Dropout. That's also the reason why Reviewer pHpX believes that UET has no more significant differences over AKD. Further, from Tabel 3 and 5, the AC notices that the performance of AKD is close to that of UET (43.9 vs. 44.1), and the hyper-parameter could also lead to the performance gap > 0.2 (e.g., M = 15, 43.8; M = 10, 44.1). The optimal performance of UET (44.1) appears to result from careful tuning, particularly with respect to M. However, it is unclear whether the performance of AKD has been similarly optimized through its hyperparameters (the AC fails to find corresponding descriptions about this). It could be unfair as the authors highlight the performance gain of 0.2 over AKD in the revised manuscript.

Decision: Reject