Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a model evidence head (MEH) and a hierarchical uncertainty aggregation (HUA) for active selecting informative images under the evidential deep learning (EDL) framework. The experiments conducted on PASCAL VOC and MS-COCO seem to validate the efficacy of the proposed method.

strength_and_weaknesses: ## Strengths
- Overall the paper is well written and easy to follow.
- The idea of MEH and HUA is interesting.
- The experimental results are solid and look promising.

## Weaknesses
- Why both ReLU and ReLU + MEH perform worse than the random baseline in Table 1?
- Is the proposed method able to handle the label noise? Usually we cannot guarantee that the human annotations are 100% accurate and therefore a solid active learning method should also be able to handle the label noise to some extent.
- The baselines of active learning are not enough. The authors should compare with the latest methods not limited to:

[1] Yoo, Donggeun, and In So Kweon. "Learning loss for active learning." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.

[2] Kim, K., Park, D., Kim, K.I. and Chun, S.Y., 2021. Task-aware variational adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8166-8175).

[3] Yu, Weiping, Sijie Zhu, Taojiannan Yang, and Chen Chen. "Consistency-based active learning for object detection." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3951-3960. 2022.

clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is well-written and the idea is very clear.
- Quality: the paper is in a good quality but the experimental validation is still insufficient.
- Novelty: intermediate novelty.
- Reproducibility: the source code is provided in the supplementary.
summary_of_the_review: I initially rated the paper as "5: marginally below the acceptance threshold". After carefully reading the authors' response, I am glad to see that they have addressed most of my concerns and therefore I would like to upgrade my rating to "6: marginally above the acceptance threshold".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work fist considers epistemic uncertainty for capturing the usefulness of the sample by model evidence head (MEH) which is a kind of evidential deep learning (EDL), and then proposes hierarchical uncertainty aggregation (HUA) for obtaining the informativeness of an image. Experimental results show the effectiveness of proposed method.
strength_and_weaknesses: Strengths:

1. This paper aims to label a set of samples based on the uncertainty by adopting Evidential Deep Learning (EDL) to effectively compute epistemic uncertainty, and proposes the Model Evidence Head (MEH) that makes EDL highly compatible with object detection. 
2. The proposed Hierarchical Uncertainty Aggregation (HUA) makes use of attributes in bounding boxes, such as nearest object and box size, to capture the context within the image and improves the quality of the expected informativeness of images, instead of simply depending on the maximum/mean of all bounding boxes.

Weaknesses:

1. The motivation of MEH module is not clear, eg., why applying softmax function instead of ReLU operation can produce sufficiently confident prediction? Would replacing ReLU with sigmoid function can help MEH module?
2. The lack of the denotation of symbol \beta_{c} in equation 4.
3. How to compute the large error in equation 6?
4. Can authors provide the performance of variants of HUA where the order of `scale and class’ is switched. 
5. Why the concentration parameter cannot be measured by Reg.Head but for Cls. Head? 



clarity,_quality,_novelty_and_reproducibility: The quality of the paper is good: it studies the fundamental task, object detection in computer vision, and has achieved the new state-of-the-art results.

The clarity of the paper is good: introduction of the method, experiments and claims are clear enough to read and understand.


Source code is provided in the supplementary material for reproducibility.
summary_of_the_review: I would recommend a marginally above acceptance. The experimental results and reported performances are the state-of-the-arts and may have a good impact to the field of classical objection detection, but i would like to know the influences in the DETR-like models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines evidential deep learning (EDL) and active learning in the object detection task and achieve good results. The authors propose a new module model evidence head (MEH) and a new loss function to ensure that EDL can be effectively applied to object detection. For active learning, a hierarchical uncertainty aggregation (HUA) method is proposed to obtain the informativeness of an image. Experiments show that the proposed method brings a very good performance improvement.

strength_and_weaknesses: Strength:

- EDL is applied to predict the epistemic uncertainty of samples. 
- The authors propose a HUA method to aggregate object-level uncertainty into image-level uncertainty.
- Mathematical proofs make this method more rigorous and convincing. 
The proposed method shows good performance.

Weaknesses: 

- There are some points unclear on the uncertainty score. Is the epistemic uncertainty (Eq. 1) directly used as the uncertainty score of the box? If so, does this bring better results? Does this mean that object-level uncertainty does not need to account for aleatoric uncertainty. 
- Algorithm 1 seems to be just an overview of an active learning framework. The core part of active learning seems not present. The authors may need to describe Step 2 in detail: how to calculate the box uncertainty and how to aggregate the box uncertainty, which can make the algorithm clearer.
- There seems no experiments to tell how about combining traditional methods and HUA.
- The results of "Examples of easy and hard samples" are interesting. The uncertainty scores of complex images are significantly higher than those of simple images. However, does this lead to selecting only complex images and ignoring simple images each time? The images that are ultimately selected tend to be those that contain more objects. It seems that the active selection strategy is unnecessary. We may not need the epistemic uncertainty, but only need to roughly estimate which images contain more objects. Are there any further experiments to have a discussion?
- Do the results obained using only MEH (without HUA) have a clear advantage over others? It would be better to have experiments on this point. Otherwise, it is difficult to prove the validity of epistemic uncertainty.

clarity,_quality,_novelty_and_reproducibility: The work of this paper is of high quality, and the authors point out the shortcomings of traditional methods and gives better solutions. Experiments show that the proposed method brings a high performance improvement, which demonstrates the effectiveness of the method. The paper is well presented; however, there is still some ambiguity as mentioned in the weaknesses.

summary_of_the_review: The paper cleverly combines EDL and active learning in the object detection task. It would be better to address the weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper employs evidential deep learning to assign uncertainty to unlabeled samples. Then, based on the assigned uncertainty, active learning is leveraged to ask humans to label more samples from the dataset. The authors have modified the original EDL. Specifically, the concentration parameters $\alpha$ >1 have been changed to $\alpha$>0.  Doing so would make Dirichlet distribution concentrate on sparse samples. The authors also introduced hierarchy uncertainty aggregation to decide the informativeness of an image. 
strength_and_weaknesses: + The paper is well-motivated and well-written. Overall the idea is very clear. I can easily understand what the authors want to do.
+ The experiments support the claims. In particular, the proposed method achieves better performance than the baselines.

- The novelty of introducing EDL into active learning seems very straightforward. EDL is able to find some out-of-distribution samples during the labeling process, whether the unlabeled samples are out-of-distribution? This has not been explained clearly by the authors.

- The authors have changed the concentration parameters from greater than 1 to greater than 0, this would affect the Dirichlet distribution. The authors should conduct experiments to show the differences, or at least some toy examples. Moreover, I cannot tell the significance of employing $\lambda$. How do the differences between these two heat maps affect the final labeling? Will the differences impact significantly?

- The authors only employ single-stage detectors to conduct the experiments. Whether the proposed method still works for two-stage detectors?

- As this work focuses on active learning, in each round there would be some randomness. Therefore, the authors should report the variations of each experiment. In other words, by using different labeled samples at the beginning what the variations would be?

- Last but not least, the authors use a certain ratio / percentage of the dataset samples. As different datasets would have different samples and this would make the contributions unclear as to whether the benefits come from the EDL active learning or more labeled samples at the beginning. 

- The baselines seems out of date. MI-AOD is published in CVPR 2021, which is the newest one in the comparisons. 
Sequential Voting with Relational Box Fields for Active Object Detection, CVPR2022. Note that the github of this work is available.

- The equations in the appendix is not new, which are very similar to EDL paper. Therefore, I do not think adding those equations make this paper valuable as those are not orignal deduction.

clarity,_quality,_novelty_and_reproducibility: The work has its originality. However, the experiments are not comprehensive enough.
summary_of_the_review: This work is well presented. My two concerns are about the novelty and the experiemnts.
1. The novelty is not clear as I cannot really tell why the difference would make such performance improvements in Fig. 5.
2. The experiments seem not strong: missing two-stage detector, the experiments do not provide variations, the state-of-the-art methods are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of model uncertainty estimation as the acquisition function in Active Learning for object
detection. To this end, they first adapt Evidential Deep Learning (EDL) to predict the epistemic uncertainty estimate 
for each bounding box separately. They modify EDL in three ways: (a) replace the last layer's ReLU by softmax, 
(b) introduce a separate Model Evidence Head (MEH) that predicts model evidence, effectively re-scaling the
concentration parameter and (c) change the loss function to encourage the model evidence to be inversely proportional to
the classification loss. Once uncertainty is estimated for each bounding box, the authors introduce a three-level
hierarchical scheme to aggregate uncertainty across bounding boxes by iteratively merging them based on their predicted
classes, scales and spatial overlap (objects).

Experiments carried out on Pascal VOC and MS-COCO show superiority to competing state-of-the-art approaches in active
learning (final mAP when reaching a fixed proportion of labelled samples in a pool-based active learning simulation).
Ablation studies are also provided showing the relative improvement of each design decision and computational
complexity.
strength_and_weaknesses: * Strengths:

  * High potential impact, given the relatively little work on Active Learning for object detection and the high
  practical need for such methods in the industry (ubiquity of object detection in practice, high annotation costs)
  * Low inference time makes this method quite attractive compared to other epistemic uncertainty evaluation approaches  
  * Extensive evaluation, providing a fair comparison against relevant alternative methods, while also providing
  ablation studies and runtime comparisons. 
  * The paper is well-structured and easy to follow.

* Weaknesses:

  * While the authors provide an intuition for their loss function for MEH (decrease model evidence when the bounding
  box classification error is large and vice-versa), no probabilistic interpretation is given to this loss within the
  EDL framework. The lack of a formal derivation makes it look somewhat empirical and disconnected to the rest of the
  EDL framework.
  * The MEH is attached to the classification head. However, epistemic uncertainty is not modelled for the regression
  head (bounding box localization), and the method relies on the presence of multiple detections as a proxy for
  localization uncertainty. While such improvements may be out of scope, I think the possibility of modelling epistemic
  uncertainty jointly in classification/localization space should be discussed.
clarity,_quality,_novelty_and_reproducibility: * Clarity:

  * Clearly written and well-structured paper overall.
  * Minor observations:
    * Section 3.3, subsection on re-aligning bounding boxes: "each bounding box contains much more information: 
    <b>object</b>, scale and category". As output by the detector, bounding boxes at each scale do not come with an
    assigned object identifier, nor are ground truth object bounding box annotations available during active learning
    to perform an assignment. Rather, if my understanding is correct, detections are first clustered based on an IoU 
    criterion, and each cluster becomes an object hypothesis, with several detections assigned to it. I think this
    should be clarified.
    * The middle quantity
    $\frac{\Gamma(\alpha_j + 1)\Gamma(\sum_j \alpha_j)}{\Gamma(\alpha_j)\Gamma(\sum_j \alpha_j + 1)}$ in equation (3) 
    seems to be wrong (it does not depend on $i$, while the left hand side $p(y_i \mid \alpha)$ and 
    right hand side both do).
    * Typos and grammar:
      * Section 3.2: "applying EDL to objection to describe" => "applying EDL to object detection and describe" 
      * Section 4.1: "at the every cycle" => "at every cycle"
      * Section 4.3: "focal coss" => "focal loss", "as din in" => "as done in"

* Novelty:

  * Up to my knowledge, this is the first work to apply EDL to object detection. It also brings several technical
  innovations to the method, as discussed in the paper summary. A formal motivation of the MEH loss and a discussion
  of modelling epistemic uncertainty for bounding box localization would add further novelty.

* Quality:

  * Relevant baselines and benchmarks are considered, comparisons are fair, and ablation studies are provided. Overall,
  the scientific quality of this work seems adequate.

* Reproducibility:

 * The authors mention the relevant details needed to replicate their work.
 * The authors provide full source code in the supplementary material.
summary_of_the_review: Overall, I think this is good quality work, with sufficient technical contribution and good empirical support.
The technical contribution could further be strengthened by providing a formal motivation for the MEH loss and a
discussion of how one could model  epistemic uncertainty for the bounding box localization predictor head. Minor
improvements to clarity/corrections should be considered.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Five reviewers have evaluated this paper and after rebuttal they all scored paper 6 (which his an improvement over initial scores). On that basis, AC advocates an acceptance as a poster and asks authors to include all key aspects of rebuttal into the final paper. 

Regarding uncertainty, AC encourages authors to also extend a bit related works as this has been a highly popular topic in recent years. Some ideas include:
A. What uncertainties do we need in bayesian deep learning for computer vision? Kendall et al., NeurIPS'17
B. Towards a Robust Differentiable Architecture Search under Label Noise, Simon et al., WACV'22.
C. Uncertainty-Guided Probabilistic Transformer for Complex Action Recognition, Guo et al, CVPR'22
D. Uncertainty-DTW for Time Series and Sequences, Wang et al, ECCV'22.

For example, [A] contains a very good discussion on what uncertainties are needed in computer vision, [B] uses reparametrization trick/uncertainty modeling for the label noise, [C] uses uncertainty modeling in action recognition while [D] uses reparametrization trick and  epistemic uncertainty in transformers, etc. 

AC encourages authors to have a thorough look around to connect to more works on uncertainty and include them into related works.

justification_for_why_not_higher_score: The paper is still on the borderline even after reviewers increased their scores. Reviewers still have various minor concerns about this work, motivation why it works well with the label noise etc.
Decision: Accept: poster