Active Learning for Object Detection with Evidential Deep Learning and Hierarchical Uncertainty Aggregation
Authors: Younghyun Park
Wonjeong Choi
Soyeong Kim
Dong-Jun Han
Jaekyun Moon
Venue: ICLR 2023 poster (2023)
Keywords: Active Learning
Object Detection
Uncertainty Estimation
Bayesian Learning

Abstract: Despite the huge success of object detection, the training process still requires an immense amount of labeled data. Although various active learning solutions for object detection have been proposed, most existing works do not take advantage of epistemic uncertainty, which is an important metric for capturing the usefulness of the sample. Also, previous works pay little attention to the attributes of each bounding box (e.g., nearest object, box size) when computing the informativeness of an image. In this paper, we propose a new active learning strategy for object detection that overcomes the shortcomings of prior works. To make use of epistemic uncertainty, we adopt evidential deep learning (EDL) and propose a new module termed model evidence head (MEH), that makes EDL highly compatible with object detection. Based on the computed epistemic uncertainty of each bounding box, we propose hierarchical uncertainty aggregation (HUA) for obtaining the informativeness of an image. HUA realigns all bounding boxes into multiple levels based on the attributes and aggregates uncertainties in a bottom-up order, to effectively capture the context within the image. Experimental results show that our method outperforms existing state-of-the-art methods by a considerable margin.

Published as a conference paper at ICLR 2023
ACTIVE LEARNING FOR OBJECT DETECTION WITH EVIDENTIAL DEEP LEARNING AND HIERARCHICAL UNCERTAINTY AGGREGATION
Younghyun Park', Wonjeong Choi', Soyeong Kim', Dong-Jun Han", Jackyun Moon
1 Korea Advanced Institute of Science and Technology (KAIST), 2 Purdue University
(dnEfk53E9, dnwswnjjd57 besto0d besp.//.eeeett//aaaatttcccc aaastt.....t. han762Ppurdue. edu, jmocnAkeist.edd
ABSTRACT
Despite the huge success of object detection, the training process still requires an immense amount of labeled data. Although various active learning solutions for object detection have been proposed, most existing works do not take advantage of epistemic uncertainty. which is an important metric for capturing the useful- ness of sample. Also, previous works pay little attention to the attributes of each bounding box (e.g., nearest object. box size) when computing the informativeness of an image. In this paper, we propose a new active learning strategy for object de- tection that overcomes the shortcomings of prior works. To make use of epistemic uncertainty, we adopt evidential deep learning (EDL) and propose a new module termed model evidence head (MEH), that makes EDL highly compatible with ob- ject detection. Based on the computed epistemic uncertainty of each bounding box, we propose hierarchical uncertainty aggregation (HUA) for obtaining the in- formativeness of an image. HUA realigns all bounding boxes into multiple levels based on the attributes and aggregates uncertainties in a bottom-up order, to ef- fectively capture the context within the image. Experimental results show that our solution outperforms existing state-of-the-an methods by a considerable margin.
INTRODUCTION
Deep learning contributes to huge success in computer vision problems such as semantic segmenta- tion (Long et al., 2015; Ronneberger et al., 2015; Chen et al., 2018) and object detection (Liu et al., 2016; Lin et al., 2017; Redmon al., 2016). However, training deep neural network typically comes with a cost of large labeled datasets. Labeling data for complex vision problems requires in- tensive labor of human experts, which makes preparing for practical application challenging. Active learning, which gradually labels a set of samples based on the informativeness (e.g., uncertainty), is a promising solution for this problem due to simplicity and high performance.
Although active learning has been extensively studied on image classification, only a few prior works focused on object detection (Yuan al., 2021; Su al., 2020; Haussmann et al., 2020; Yu al., 2021) despite its practical importance. Furthermore, existing works on active learning for object de- tection have two limitations. First, when computing the informativeness of an image, most previous works only use the aleatoric uncertainty. not taking the epistemic uncertainty into account. Epis- temic uncertainty, also known as knowledge uncertainty, captures the lack of knowledge of model (caused by a lack of data) and can be reduced when large amounts of data are available. Aleatoric uncertainty, on the other hand, captures the noise inherent in the observed data and irreducible. As stated in (Nguyen et al., 2022; Hafner et al., 2018; Hüllermeier & Wacgeman, 2021), epistemic uncertainty can reflect the usefulness of samples and support active learning better than alcatoric uncertainty. Secondly, previous works active learning for object detection generally ignore the attributes of bounding boxes (e.g., nearest object, box size) when computing the informativeness of an image: informativeness is often defined as the maximum or mean of the uncertainty values of all bounding boxes in the image. This can be a problem because a cluttered image with many objects belonging various categories can be enforced to have similar uncertainty value relative just simple image with only few objects belonging to a single category.
"oorrsponningg author

---

Published as a conference paper at ICLR 2023
Goal and challenge. The general goal of this paper is propose active learning strategy for object detection, that handles the above limitations of existing works. First, we aim to build an algorithm that can compute epistemic uncertainty quickly yet correctly in object detection. Some existing works on object detection (Haussmann et al., 2020; Feng et al., 2019) calculate epistemic uncertainty using multi-model based methods (e.g.. model ensemble (Beluch al., 2018), Monte Carlo (MC) dropout (Gal & Ghahramani, 2016)). However, these methods require multiple models or repetitive forward propagations for MC integration, consequently making practical applications difficult. Secondly, we aim to design an uncertainty aggregation scheme which can consider at- tributes of bounding boxes and understand the context within images. This goal meaningful since aggregation schemes of previous works (Yuan et al., 2021; Roy et al., 201 8; Choi et al., 2021), which simply rely on the maximum/mean of all bounding boxes, are hard to reflect the context images.
Main contributions. Our first key idea is to adopt Evidential Deep Learning (EDL. to effectively compute epistemic uncertainty, and to propose new module that makes EDL highly compatible with object detection. Introduced by (Sensoy et al., 2018; Amini et al., 2020), EDL is useful tool to compute epistemic uncertainty for detecting unfamiliar data (e.g., unseen unlabeled data) since it samples model ensembles almost instantly. However, previous works on EDL have mainly focused on image classification and induce unconfident prediction and unstable training when simply applied to object detection. To this end, we propose new module named Model Evidence Head (MEH) to enable confident prediction and stable training of EDL. Specifically, MEH predicts the expected difficulty, or model evidence, and is optimized independently of the object detector. To our knowledge, this is the first research to make EDL compatible with object detection on 2D images.
Another key ingredient for our solution is Hierarchical Uncertainty Aggregation (HUA), which makes use of attributes in bounding boxes for computing the informativeness of an image. Ac- cording to the attributes, HUA realigns boxes into multiple levels and aggregates uncertainties in a bottom-up order. This helps to capture the context within the image and improves the quality of the expected informativeness of images. Overall, our main contributions are summarized as follows:
We make use of EDL to effectively compute epistemic uncertainty in object detection, and design a new module termed Model Evidence Head (MEH) which solely predicts the model evidence independently of the class confidence make EDL adaptable to object detection.
We propose Hierarchical Uncertainty Aggregation (HUA), which reorganizes all bounding boxes into several levels and aggregate uncertainties of each level in a bottom-up manner, to better capture the context within the image.
We validate the efficacy of proposed methods using RetinaNet and SSD as base models on well- known datasets: PASCAL VOC, MS-COCO. Extensive experiments demonstrate that the proposed methods significantly improve performance achieving new tate-offthe-art results.
2 RELATED WORKS
Active learning for object detection. Active learning (Sinha et al., 2019; Yoo & Kweon, 2019; Sener & Savarese, 2017; Gal et al., 2017; Wang et al., 2016) aims to select a small subset of infor- mative unlabeled samples which is expected to be most effective. Although active learning has been extensively studied for the classification problem, only a few works focus on object detection (Yuan et al., 2021; Su et al., 2020; Haussmann al., 2020; Yu al., 2021), despite its practical importance. (Yuan et al., 2021: Su et al., 2020) train discriminators using unlabeled data to predict whether an image is from the labeled set or the unlabeled set. LL4AL (Yoo & Kweon, 2019) trains an auxiliary module to predict loss of data samples, where a sample with high predicted loss is considered as the one with high informativeness. The concurrent work CDAL (Agarwal et al., 2020) introduced distance measure to select diverse samples in semantic and spatial context. Recently, (Choi et al., 2021) predict the parameters of Gaussian mixture models and computes epistemic uncertainty as the variance of Gaussian modes. However, there is no guarantee that model uncertainty leads to variance in GMM, and the size of the model ensemble is fixed to be small since the number of Gaus- sian modes is not allowed to change after training. Overall, existing approaches do not fully take advantage of epistemic uncertainty and pay little attention to attributes of bounding boxes during uncertainty aggregation. Our work resolves both limitations using novel and effective methods.
Bayesian deep learning. The goal of Bayesian deep learning is to build a credible machine that measures uncertainty its decision well. In contrast to the frequentist approach where model
2

---

Published as a conference paper at ICLR 2023
parameters not random variables but fixed quantities, the Bayesian approach assumes prior dis- tribution on model parameters and estimates posterior distributions based on given data. Recently, many computer vision works have proposed the use of uncertainty through Bayesian deep learning. (Kendall & Gal, 2017) investigates the benefits of modeling epistemic/aleatoric uncertainty in vision tasks, while (Simon et al., 2022) and (Guo al., 2022) use the reparameterizatioo trick estimate uncertainty for neural architecture search and action recognition, respectively. However, previous Bayesian deep learning methods such as Bayes by Backprop (Gal & Ghahramani, 2015; Blundell et al., 2015), MC dropout (Gal & Ghahramani, 2016) and Variational Inference (Blei et al., 2017; Hoffman et al., 2013; Kingma & Welling, 2013) often require costly sampling- MC approx- imation to estimate the posterior distribution. Instead, our work adopts Evidential Deep Learning (EDL) (Sensoy et al., 2018; Amini et al., 2020) to compute epistemic uncertainty through a single forward propagation. Different from typical Bayesian approaches, EDL tries to predict prior dis- tributions of posterior models without repetitive model sampling. However, EDL has hardly been applied to complicated computer vision problems such object detection yet, and it usually quires adversarial learning using a regularization loss which makes training unstable. In this work, propose MEH to overcome aforementioned limitations of previous works.
3 PROPOSED METHOD
3.1 PROBLEM SETUP AND ORGANIZATION
Active learning consists of multiple learning cycles. At the first cycle, large unlabeled dataset U and a small labeled dataset L are given. Once an object detector is trained using L the network selects the most informative images 1 from U7) based on some measures. Human oracles then manually label 1 and construct new labeled/unlabeled sets for the next cycle as L' L U 7 and U² -0 1 The same process is repeated until the annotation budget is exhausted.
In this section, we describe our method for computing the informativeness of an image for active learning in object detection. Section 3.2 describes how we estimate epistemic uncertainty of bounding box using EDL and the proposed MEH. Based on the results in Section 3.2, we describe how we compute the informativeness of an image using HUA, in Section 3.3.
3.2 EVIDENTIAL. DEEP LEARNING FOR EPISTEMIC UNCERTAINTY
Overview of our approach. Fig. shows the high-level descriptions of our approach under the EDL framework for computing epistemic uncertainty of bounding box. MEH in Fig. ia our new mod- ule that makes EDL adabtable to object detection. Under the Dirichlet Birichlet-Cateoorica Categorical Bayesian frame- work, we propose to predict class confidences 3 = |= and model evidence X, to obtain the concentration parameter a [a and construct the prior Dirichlet distribution Dēr(Ola) for each bounding box. Epistemic uncertainty is then computed using model ensembles 0 ~ Dfr(f) as the mutual information between predictions and model posterior as
Ily 0
PPp(fl)) [H[p[u]0)]]
Epistemic Unc. (Uap.)
(Utos)
Aleatoric Unc. (Uate)
where H denotes Shannon entropy and 0 parameterizes categorical likelihood Cat(O); p(y(0) and p(0)a) are probability functions of categorical and Dirichlet distributions. Fig. 1b shows an example on how the 3 types of uncertainty in (1) change according to À predicted from our MEH module.
Prior works on active learning without model sampling fail to take advantage of epistemic uncer- tainty Uept since p(0)a) becomes a point estimate, enforcing Ulot Uate and Ueps 0. Our two-stage Bayesian framework based on EDL enables us to sample 0 almost instantly, which is significantly faster than multi-model- mani, 2016) and model ensemble (Haussmann et al., 2020). In the following, we start by applying EDL to object detection to describe our algorithm in details.
Bayesian methods such as MC-dropout (Gal & Ghahra-
Applying EDL to object detection. Inspired by EDL (Sensoy et al., 2018; Amini et al., 2020; Zhao et al., 2020), unlike typical object detectors where the classification head predicts parameters 0 of the categorical distribution, conjugate prior of the lower-order categorical likelihood (Q). To fit our evidential model to data, maximize the marginal likelihood, also known as maximum likelihood Type II. The marginal
predict high-order Dirichlet distribution Dir(0la) which is
3

---

Published as a conference paper at ICLR 2023
Dir(Ole)
Backbone
Cls. Head
Irobal
Urot 11004
Mase 10.07
MEH
Mapi 00.97
Disimilar models [9] High Episteric Uncertainty
Dtr(0 (a)
Unlabeled Image
Reg. Head
0 Dir (o)
High some 2
Uhat 1.04
(8) (x)
E (122,... K)
Vate : 0.83
Mege: :021
Similar models Low Epistemic Uncertaintyy
(a) Obtaining concentration parameter
(b) Calculation of epistemic uncertainty with sampling
Figure 1: Proposed EDL-based uncertainty computation of a bounding box. (a) First, for an unlabeled image, the classification head produces class confidences 3 = {B:) while our model evidence head (MEH)) pro- duces model evidence 8 and À are used to compute parameter
of Dirichlet distribution ()) (b) Based on a, parameters of categorical distribution Cat(@) are sampled from Dir(0la). Epistemic uncer- tainty is then computed as a dissonance between sampled Note that a larger À indicates a larger a, making Dir(Ola) a)) sharper; sharp Dir (o) produces similar Cot( decreasing the epistemic uncertainty.
likelihood for bounding box y: can be obtained by marginalizing over the likelihood parameter
pluila)) f pr00)=-f((0000001 P(u:10)((0la)dd
/
/II
K
1 Lm,
r 9g)
II
K
Pj dp.
j=1
The first term in the integral comes from the categorical distribution, while the second term is from the Dirichlet distribution; the integration can be interpreted as marginalization over every possible categorical model 0 ~ Dir(Ola). Thanks to the Dirichlet-Cann Caachltt-aategorical conjugacy, the integral can be written closed-form:
plulaa) = re ag) 11 [(a;) T +1)
11 I(a;) +1)
02 Σ,
The network is then optimized to minimize negative log marginal likelihood Lets ->x Un (pgg = kla)), where i is an one-hot label vector. At inference, the expected proba- bility for the k-th category is computed as p&- S5 where Ex x aą is the Dirichlet strength.
Issues. When the EDL method used for image classification directly applied to object detection, several problems arise: (i) training becomes unstable due the adversarial regularization loss, and (ii) the prediction becomes unconfident due to the use of ReLU as hypothesis function. Given class confidences 3 { K previous works (Sensoy al., 2018; Zhao al., 2020; Hemmer et al., 2022) adopt ReI.U to compute the concentration parameter as a'. ReL.U( + Although ReLU performs well image classification with simple datasets, noticeably decreases the mAP score in object detection where the number of classes is large and confident prediction is important. For example, the case 80-way classification, 3& should be least 7820 to achieve Pa. 0.99 even when 3: = 0 for all i * k, according to a ReLU( + and (3).
MEH module. To overcome the aforementioned issues, we propose MEH and introduce several exp(
techniques. As a first step, we apply softmax as a = Σ. exp(8.) instead of ReLU to produce sufficiently confident prediction. However, although softmax enables confident prediction, it forces the Dirichlet strength changeable and deprives the model of the ability to predict model evidence. To tackle this issue, propose a new module termed MEH, which solely predicts model evidence A. When is obtained from MEH and class confidences {.k 1 k=1 obtained from the classification head, we propose to re-scale the concentration parameter a as
Σn ag to be This makes the entropy of Dirichlet distribution un-
exp(Bk)
0% > E. exp(33)
where 3g is the class confidence for class c. The above re-scaling process with À transforms Dir(fla) to be either concentrated or flat. Epistemic uncertainty is then computed based the re-scaled Dir(Ala) via (1). Fig. lb illustrates the effect of À on epistemic uncertainty. The effects of softmax and À are validated via experiments in Section 4.
4

---

Published as a conference paper at ICLR 2023
MEH loss function. We separate the training of MEH from the primary object detector with novel loss function for stability. For a Bayesian model to work properly, it must be able to recognize the "I don't know" state, ie., the model should generate low model evidence for difficult samples. To this end, previous works on EDL (Hemmer et al., 2022; Sensoy et al., 2018) shrink the model evidence for singletons which do not correspond to target category, via the following regularization loss
Lreg( (a. y) = KL [I[Dir(0(a]] Dir(@) < <1...,, }>},
(5)
where à i 3) is disregarded. However, this loss enforces the non-target elements to be one, no matter how large an error is detected, since the uniform Dirichlet distribution Dir(O) 1,...: 1 >) is considered as constant supervisory signal. Furthermore, this adversarial regularization loss suppresses the overall model evidence and makes training unstable. To overcome these issues, we propose a simple yet effective solution: we utilize the continuous loss score from the classification head as a target value. Specifically, when classification loss 1s,5 is given for bounding box i at scale s, the proposed loss function for MEH is defined as
is the concentration parameter where the element for target category
L LME E (':
(6)
This loss encourages MEH decrease model evidence Au.i when the classfication loss l,,i is large; similarly, MEH is guided to increase X for small 4,,1. LMEH has the same effect as the previous regularization loss (5) in that it prevents overconfidence and helps the model to predict uncertainty better; however, by using the target loss as a changeable supervisory signal, LMEH is able to dy- namically penalize the model to better predict the model evidence.
Stability. Note that we deliberately supervise only model evidence À independently of Dirichlet parameters {ar LK to train MEH. À is utilized only for re-scaling the Dirichlet strength and has pt. since À will naturally vanish when division ax/S occurs. Hence, the MEH network PASE EH and remaining network & ΦME can be updated in a disjoint manner. We in turn optimize & PEE and & \ bMEH, thus training of ΦMEн never affects the performance of the primary object detector; this resolves the stability issue of previous EDL approaches.
no effect on p(z[0)
Applications to recent models. Our method is not limited to softmax-based object detectors like SSD (Liu et al., 2016), but also applicable to sigmoid-based detectors like RetinaNet (Lin et al., 2017). Given sigmoid-based prediction p E [0, 1], focal loss can be defined as FL((p.) -(1 log(() where pt = p if y 1. Pt = 1 - otherwise. To enable evidential learning, we replace sigmoid-basse binary prediction p by Sircchet--Categricce multiclass prediction p(x|aa) of equation (3). We train RetinaNet with the modified evidential focal loss and report its performance in Fig. 3.
3.3 HIERARCHICAL UNCERTAINTY AGGREGATION
Based on the epistemic uncertainty of a bounding box, we propose HUA: a new method for comput- ing the final informativeness score of an image. Recent active learning methods for object detection (Yuan et al., 2021; Choi et al., 2021; Yu et al., 2021) typically compute the informativeness of an image as the mean or maximum of all bounding boxes. To this end, propose generalized framework that can apply different kinds of aggregation functions based on the level of information (predicted category, box size, nearby object) of each bounding box, as described in Fig. 2.
Filtering bounding boxes. Single-stage object detectors such as RetinaNet and SSD generate bounding boxes at every scale, pixel and anchor. Since most boxes correspond to background, we first filter out background boxes whose max;((e) is lower than threshold Yscore.
Realigning bounding boxes. Besides the uncertainty score, each bounding box contains much more information: predicted object, scale and category it belongs to. Based on these information, we propose to realign the boxes into hierarchical structure. First, boxes are matched the nearest object depending but the green box is ignored since IoU is lower than the threshold ToUU. Secondly, boxes are further grouped based on the scale to which they belong. Lastly, boxes divided based on the category (argmax, P&) to which they belong. See Fig. 2b for the resulting hierarchy of bounding boxes.
the loU score. For example, in Fig. 2a, the blue box is matched to "object",
Uncertainty aggregation. Once the bounding boxes are fully realigned, individual uncertainty scores are unified in a hierarchical order. As shown in Fig. 2b, uncertainties of the boxes a lower level (e.g., class) aggregated into single value through predefined aggregation function;
5

---

Published as a conference paper at ICLR 2023
object
scale
class
hox c object
ocale
Rerzan
boz
olypcL
object
Informationnns
scale
herrer
dox
the image
dox
scale
object
barso
Box
bunse
dox
scale
object
object
dox
(a) Matching of bounding boxes
(b) Uncertainty aggregation based on attributes
An overview of the proposed hicrarchical uncertainty aggregation (HUA). (a) Bounding boxes are grouped or ignored based on IoU with the near objects. (b) Bounding boxes in an image are hierarchically realigned based on object, scale, category to which the boxes belong. Uncertainties of bounding boxes in the same level are aggregated and then passed to the higher level. For the above example, set of functions (sum, max, sum) is chosen for each information level (object, scale, class).
Figure 2:
the aggregated value is then passed to an upper level (e.g, scale). This aggregation repeats from the "class level" to "object level". Note that HUA can be viewed as a generalized scheme where different types of aggregation functions can be adopted at different levels of information; it can cover aggregation methods of previous works (Choi et al., 2021; Yuan al., 2021) where identical functions (max mean) applied to the every level. As discussed in Appendix, we specifically propose to adopt sum operation when aggregating the uncertainties at the object level, since this reflects the number of objects in final informativeness of the image. We also propose to apply sum operation to the class level. Given that ambiguous bounding boxes are often classified into more than one class, considering these classes altogether helps to increase the uncertainty score of the difficult object. By considering the attributes of each bounding box, HUA effectively captures the context within the image in the informativeness score.
3.4 SELECTION OF INFORMATIVE IMAGE
Now the epistemic uncertainties of all unlabeled images can be computed at each active learning cycle. While previous works typically select the top-k uncertain images, we propose to select from the filtered-out images as well. These images are also valuable since the model was incapable of sensing any objects due to lack of knowledge. Composing a certain fraction of selections with these images further improves the performance, where the detailed analysis is provided in Appendix.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset. Our work is validated on PASCAL VOC (Everingham et al., 2010) and MS-COCO (Lin et al., 2014). As to PASCAL VOC, VOC07+12 trainval is used for training and VOC07 test is used for evaluation. While mAP50 is used for evaluation metric in PASCAL VOC, mean of mAP at IOU=.50:05:90 is used for MS-COCO. In the first cycle for PASCAL VOC, 5% from 16,551 samples are randomly selected. Then 2.5% of samples are labeled at each cycle until reaching 20%. As for MS-COCO with 117,267 samples, labeled sets increased from 2% to 10% in steps of 2%.
Implementation details. For fair comparisons with previous works (Yuan al., 2021; Agarwal et al., 2020; Yoo & Kweon, 2019), we adopt RetinaNer (Lin et al., 2017) and SSD (Liu et al., 2016) which use ResNet50 and VGG16 as backbones, respectively. The structure of MEH the same as the regression head except that the dimension of the output layer is one. As for RetinaNet, we follow the setting of (Yuan al., 2021) and set epoch to be 26, batch size to be 2; also, we follow the setting of (Yoo & Kweon, 2019; Agarwal et al., 2020) for SSD and set epoch to be 300, batch size to be 32. In both experiments on PASCAL VOC and MS-COCO, the SGD optimizer used with learning rate of 0.001; E2 weight decay rate of 0.0001; momentum of 0.9; and warm-up strategy for the early 500 steps. As for the backbones, ResNet50 and VGG16 are pretrained on ImageNet. Note that at every cycle, the object detector is trained from scratch with the pretrained backbone. We implement our framework using Pytorch 1.5.0 and mmdetection 2.13.0. In every experiment, single NVIDIA GeForce RTX 2080Ti is used. We set the confidence threshold for box

---

Published as a conference paper at ICLR 2023
Uieiinaledd un PASCAL vOc
530 PASCAL VBC
Retnoliet on MS COCO
16
RandonSurg
Faney Singlig
Rander Semsing
(EECV20
(SCCV/21
30
tmages
Used BaiiehS of Luteled images
(a) RetinaNet on PASCAL VOC
(b) SSD on PASCAL VOC
(c) RetinaNet on MS-COCO
Figure 3: Performance comparison with state-of-the-nn methods.
Relzudiei on PASCAL VOC
550 on POS
Rebirhat on MS LOCO
12
23
10
23
22
01211
22
Runden Semping
Rznder $20raing MC Inpeir
22
Barzondampiiy Pedeirrandia
55
HC Draact
Chcteral SCCV
8c
Raboch) Laaasse
Used Ratiot) Labcted
(a) RetinaNet on PASCAL VOC
(h) SSD on PASCAL. VOC
(c) RetinaNet on MS-COCO
Figure 4: Performance comparison with multi-mode model-based Bayesian methods.
filtering ?score to 0.3, and IoU threshold for object grouping Yoot to 0.5. 500 ensemble members are sampled for uncertainty estimation. For our scheme, (sum, max, sum) operations are adopted for each information level (object, scale, class) when applying HUA. Ablation studies with different operations are provided in Appendix. We compose 15% of selections with filtered images, where the results with various selection ratios reported in Appendix.
Baselines. We compare our work with state-of-the the- works: MI-AOD (Yuan et al., 2021), CDAL (Agarwal et al., 2020), LL4AL (Yoo & Kweon, 2019), Core-set (Sener & Savarese, 2017), CALD (Yu et al., 2022). Also, as basic baselines, Entropy Sampling (Roy et al., 2018), Random Sam- pling with vanilla RetinaNet/SSDD are considered well. For Core-set, features from backbone (ResNet50/VGG16 are used following the setting of (Yoo & Kweon, 2019; Choi et al., 2021). While Random Sampling randomly selects unlabeled samples to be labeled for the next cycle, En- tropy Sampling selects the samples based on Shannon entropy; the average value of Shannon entropy among all of bounding boxes is regarded as the informativeness as done (Yuan et al., 2021: Choi et al., 2021; Roy et al., 2018). Five independent networks are trained with different seeds and the averaged performance is reported with 95% confidence intervals.
4.2 MAIN EXPERIMENTAL RESULTS
Comparison with sfate-of-the-arrs. Fig. shows mAP scores under various settings. Our proposed method exhibits clear edge over other baselines, giving the best performance. In Figs. 3a and 3c, the proposed method shows superiority in early cycles, proving that the proposed evidential focal loss is effective when labeled data is extremely limited. It is noteworthy to consider our method trained only on a labeled set, while MI-AOD benefits from semi-supervised learning using an unlabeled set as well. In Fig. 3b, the reason SSD has lower initial performance than RetinaNet comes from the difference of the loss function. Unlike RetinaNet trained with the focal loss, SSD is trained with the CE loss. As in equation (3), since calculated a using softmax, the loss expression does not change at all in the case SSD. Still, it is remarkable that our methods (MEH, HUA) gradually increase the performance by selecting informative images
with the same loss.
Comparison with multi-model based Bayesian methods. In recent active learning works, multi- model based methods used to compute epistemic uncertainty. Here, compare existing meth-

---

Published as a conference paper at ICLR 2023
ods and our work. As for baselines, we first consider MC-dropout; following (Beluch et al., 2018; Choi et al., 2021), dropout layers whose rate is 0.1 are added to the convolutional layers and 25 forward passes are sampled to compute epistemic uncertainty using equation (1). Secondly, model ensemble is considered, where we follow (Haussmann et al., 2020; Choi et al., 2021) and train three independent models construct an ensemble. Lastly, we consider the work of (Choi et al., 2021); it predicts the parameter of Gaussian mixture model (GMM) and regards the variance of Gaussian modes as the epistemic uncertainty. In every baseline, we average uncertainties from all bounding boxes for the computation of the final image-level uncertainty. as done in (Yuan et al., 2021; Choi et al.. 2021; Roy et al., 2018). In Fig. 4, we first observe that multi-model based methods yield higher scores than random sampling, underlying the importance of epistemic uncertainty. Still, the large performance gap between the baselines and our work confirms the benefits of the proposed techniques: MEH and HUA. It is noteworthy that baselines ignore the hierarchy between bounding boxes, which implics that there possibly exist important yet underestimated boxes.
4.3 ABLATION STUDIES AND DISCUSSIONS
Effects of MEH and HUA. Table 7 shows ablation studies on the proposed methods. While Random and Entropy use vanilla RetinaNet with focal loss, proposed methods use the proposed evidential focal loss. While ReLU computes with ReLU as done in (Sensoy et al., 2018; Zhao et al., 2020; Hemmer et al., 2022), Softmax uses softmax. All methods except HUA compute uncertainty of an image as the mean of all boxes, as in (Yuan et al., 2021; Choi et al., 2021). At every cycle, the proposed methods (Softmax, MEH, HUA) increase performances by large margin. The results indicates that informativeness can be captured much better via our proposed methods. Interestingly, it can be seen that the ReLU method performs even worse than Random. This result confirms that previous works of EDL for image classification cannot be simply applied to object detection.
Ratio (%) of Laheled Samples
Method
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Random
29.13
50.35
58.45
61.27
62.31
64.67
66.72
Entropy (Roy et al., 2018)
29.13
49.41
56.02
59.83
64.03
65.96
67.08
ReLU (Sensoy et al., 2018)
22.46
27.83
31.39
33.18
35.61
37.03
38.95
ReLU (Sensoy et al., 2018) MEH
22.46
29.10
32.71
34.95
37.16
38.47
39.84
Softmax
52.53
58.26
61.13
65.05
66.44
68.41
69.55
Softmax + MEH
52.53
59.68
64.78
67.80
68.28
70.85
71.84
Softmax + MEH + HUA (Ours)
52.53
62.02
66.84
69.82
71.31
72.86
74.08
Table 1: mAP (%) of RetinaNet on PASCAL VOC.
Comparison of computing costs. In Table 2, we compare computing costs of the proposed method with multi-model based baselines (MC-dropout, Ensemble) and Entropy baseline. Specifically, the first column indicates the latency time for uncertainty calculation of single image with 95% con- fidence interval and the second column shows the size of all parameters in the models. It can be seen that the proposed method has similar model size to that of MC-dropout and Ensemble baselines; a slight increase is due to the MEH module. With respect to the latency time, multi-model based methods require significant computing cost because of multiple forward propagations for MC inte- gration. Specifically, MC-dropout and Ensemble take 25 and 3 forward passes, respectively. In com- parison, our method requires much less latency time, although it samples 500 ensemble members (o)( This is because our method requires only single forward propagation and each ensemble member is sampled virtually (i.e.. f Dir(fla)). Note that Entropy has the lowest latency, since this method dose not use epistemic uncertainty and there is no need to sample ensemble members.
Visualization analysis of model evidence X. To clearly understand the role of model evidence X, we illustrate the effect of X in Fig. 5. In the 2nd column, when model evidence X is not considered, uncertain regions are observed over a wide area inside the object. On the other hand, in the 3rd column, MEH estimates high À only around the center of objects. This is because the difficulty of classification increases as the center of the bounding box moves away from the center of objects due to ambiguity. The MEH output, X, is used to readjust the prior Dirichlet distribution Dir(@)a) a)). As noted in Section 3.2, a high À increases Dirichlet strength and make D(fla) concentrated. makes similar 0 ~ Dir(Ola)) be sampled and decreases the epistemic uncertainty according to equation (1).
8

---

Published as a conference paper at ICLR 2023
RetinaNet
SSD
Method
Une. calculation (img/seec
Model size (MB)
Unc. calculation
Model size
Ensemble
0.1387 0.0036
420.882
0.0837 = 0.0023
300.813
MC-dropout
1.1058 + 0.0048
140.294
0.6184 + 0.0032
100.271
Entropy
0.0445 1 0.0013
140.294
0.0256 1 0.0013
100.271
Proposed
0.0629 +: 0.0022
149.377
0.0310 = 0.0010
100.781
Table 2: Comparison of computing costs on PASCAL VOC.
For example, in the 4th column, it can be seen that the uncertainty of the central part of the object is reduced due to high X; conversely, the uncertainty of the edge part of the object is preserved high.
Readjust prior Dir(Bla)
exp(At dg Σ (x)(A
Unlabeled Image
without considering
with considering
model
Figure 5: Effect of model evidence when calculating epistemic uncertainty. Warm color indicates high value. In the area with high model evidence, epistemic uncertainty decreases.
Examples of easy and hard samples. In Fig. for intuitive understanding. we compare uncertain- ties of easy and hard samples with Entropy Sampling (Roy et al., 2018), and MI-AOD (Yuan et al., 2021) which represent SoTA. In methods, easy examples tend to have just one unoceluded object; however, hard examples tend to have numerous objects, which are unclear heavily occluded. One interesting thing to note that difference the hard samples. In previous methods, the range of scores between images is not wide, since these methods average the uncertainties of all bounding boxes to get single score. On the other hand, in more
informativeness scores hetween the easy samples and
the proposed method, it can be seen that the informativeness scores of the hard examples than 50 times higher than those of the easy examples. This is because the proposed uncertainty aggregation scheme (HUA) adds up the informativeness scores of meaningful bonding boxes with consideration for additional information (e.g., the number of objects, categories in images).
Easy Sample
Proposed Enteupy
0.1177
02331
22078
0.257
0L3411
0.561
1:4186
1.3825
15ID
1,2951
113355
14-A0a
54104e-6
31992:
32153 :
4.0025
3.04
30540
Hard Sampla
Proposed
148747 1.508
15.6655 19724
15,778
190559
17,4991
160
Fatreyy
6691
1.207
1,211
Figure Examples of easy and hard samples. Informativeness scores are reported below images.
5 CONCLUSION
We proposed a new active learning strategy for object detection. With the evidential deep learning framework and model evidence head, our scheme effectively estimates the epistemic uncertainty of bounding box. Also, our hierarchical uncertainty aggregation provides ing informativeness of an image. Our scheme presents an up-and-come and-acminggggggg direction for active object detection, where estimating epistemic uncertainty accurately yet quickly is of crucial importance.
guideline for comput-

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide theoretical proof and implementation details in Appendix. Specifically, our experimental setup including hyperparameter setting is described in Appendix, and our source codes are included in Supplementary Material. The code is available at https: //github. www.ccom/ooooo.oc Mot//oooonL----eoo YH/AOD MEN HUA
5.1 ACKNOWLEDGEMENT
This work was conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UDI90031RD), Also, this work was supported by Institute of Informa- tion & Communications Technology Planning & Evaluation(IITP) funds from MSIT of Korea (No. 2020-0-00626).
REFERENCES
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In European Conference on Computer Vision, pp. 137-153. Springer, 2020.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9368-93677 2018.
Mélanie Bernhardt, Daniel Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C'Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew P Lungren, Aditya Nori, Ben Glocker, et al. Active label cleaning for improved dataset quality under resource constraints. Nature communications, 13(1): 1-11, 2022.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisti- cians. Journal of the American statistical Association, 112(518):859-877 2017.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Siang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Jiwoong Choi, Ismail Elezi, Hyuk-Jae Lee, Clement Farabet, and Jose M Alvarez. Active learning for deep object detection via probabilistic modeling. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 10264-10273. 2021.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Di Feng, Xiao Wei, Lars Rosenbaum, Atsuto Maki, and Klaus Dietmayer. Deep active learning for efficient training of a lidar 3d object detector. In 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 667-674. IEEE, 2019.
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approx- imate variational inference. arXiy preprint / 158,, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192, PMLR, 2017.
10

---

Published as a conference paper at ICLR 2023
Hongji Guo, Hanjing Wang, and Qiang Ji. Uncertainty-guidee probabilistic transformer for complex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2022.
Recognition, pp.
Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, and James Davidson. Reliable uncer- tainty estimates in deep neural networks using noise contrastive priors. 2018.
Elmar Haussmann, Michele Fenzi, Kashyap Chitta, Jan Ivanecky, Hanson Xu, Donna Roy, Akshita Mittel, Nicolas Koumchatzky, Clement Farabet, and Jose M Alvarez. Scalable active learning for object detection. 2020 IEEE intelligent vehicles symposium (iv). pp. 1430-1435. IEEE, 2020.
Patrick Hemmer, Niklas Kühl, and Jakob Schöffer. Deal: deep evidential active learning for image classification. In Deep Learning Applications, Volume 3, pp. 171-192. Springer, 2022.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. Journal of Machine Learning Research, 2013.
Eyke Hüllermeier and Willem Waegeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110())457-500 2021.
Chich-Chi Kao, Teng- Yok Lee, Pradeep Sen, and Ming- Yu Liu. Localization active learning for object detection. In Asian Conference on Computer Vision, pp. 506-522. Springer, 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Diederik P Kingma and Max Welling,
Auto-encoding variational bayes. arXiv preprint
1rXiv:1312.6114, 2013.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision. pp- 2980-2988, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference computer vision, pp. 21-37. Springer, 2016.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 303333331400 2015.
Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 11 111(1):89-22, 2022.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Soumya Roy, Asim Unmesh, and Vinay P Namboodiri. Deep active learning for object detection. In BMYC, pp. 91, 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: arXiv:/708.00 0ete:778.00000 00489, 2017.

---

Published as a conference paper at ICLR 2023
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in Neural Information Processing Systems, 31, 2018.
Christian Simon, Piotr Koniusz, Lars Petersson, Yan Han, and Mehrtash Harandi. Towards a robust differentiable architecture search under label noise. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 3256-3266, 2022.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5972-5981, 2019.
Jong-Chyi Su, Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Subhransu Maji, and Manmohan Chan- draker. Active adversarial domain adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 739-748, 2020.
Keze Wang. Dongyu Zhang, Ya Li. Ruimao Zhang, and Liang Lin. Cost-effective active learning for deep image classification. IEEE Transactions on Circuits and Systems for Video Technology, 27(12):2591-266 -2000, 2016.
Keze Wang, Xiaopeng Yan, Dongyu Zhang, Lei Zhang, and Liang Lin. Towards cooperation: Self-supervised sample mining for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1605-1613, 2018.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 93-102, 2019.
Taraneh Younesian, Zilong Zhao, Amirmasoud Ghiassi, Robert Birke, and Lydia Chen. Qactor: Active learning on noisy labels. In Asian Conference on Machine Learning, pp. 548-563. PMLR, 2021.
Weiping Yu, Sijie Zhu, Taojiannan Yang, Chen Chen, and Mengyuan Liu. Consistency- active learning for object detection. arXiv preprint arXiv:2103. 10374, 2021.
Weiping Yu, Sijie Zhu, Taojiannan Yang, and Chen Chen. Consistency-basee active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3951-3960, 2022.
Tianning Yuan, Fang Wan, Mengying Fu, Jianzhuang Liu, Songcen Xu, Xiangyang Ji, and Qixiang Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5330-5339, 2021.
Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning graph data. Advances in Neural Information Processing Systems, 13:12827-12836, 2020.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
B ADDITIONAL EXPERIMENTAL SETUP
In this section, we provide more detailed information our experimental setting. Following previ- ous works, two well-known object detectors are chosen as base models: RetinaNet (Lin et al., 2017) and SSD (Liu et al., 2016). First, RetinaNet adopts ResNet-50 which is pretrained on ImageNet as backbone. ResNet-50 has four stages and output of each stage is passed to the neck structure. The output dimension of each stage 256, 512, 1024, 2048, respectively. The parameters of the backbone in the first stages are frozen during training. As for the neck structure, Feature Pyramid Network (FPN) with five stages is adopted. FPN takes the output of ResNet as input and returns 256-dimensional features at the every stage. The output of FPN then goes through a classification head and a regression head, independently. The structure of the classification head and the regression head are the same as that of vanilla RetinaNet (Lin et al., 2017) which consists of three convolutional modules and one last convolutional layer for classification and regression, respectively. Following (Lin et al., 2017; Yuan et al., 2021), utilize bounding box generator whose octave base scale is 4; scales per octave is 3; ratios of bounding boxes are (0.5, 1.0, 2.0). The base sizes of boxes are (8, 16, 32, 64, 128) from the lowest scale the highest scale. The coordinates of bounding boxes and ground truth are encoded to the delta and considered as the target value of the regres- sion head. Specifically, given central coordinates (z, y), width (w) and height (h) are given for the bounding box (px, Py. pw, ph) and the ground truth (gz, gy, gw, gh)), values of delta is computed as: dx = gz-pz dd = 2M-P% dw = -oog( (pw) ) dh = log( ph Regression heads are supervised to ph
paw predict the encoded value (gx, gy. gw. gh) and these values are decoded to the coordinates of the up- permost left corner and the lowermost right corner to make the final prediction. Regarding the loss, as stated in the main paper, focal loss (Lin et al., 2017) with marginal likelihood is adopted. Hyper- parameters for focal loss (y, a) are set to 2.0 and 0.25, respectively. As to assigning positive/negative boxes, bounding boxes whose IoU scores with the ground truth are higher than 0.5 are assigned to the positive box; boxes with iou scores greater than 0.4 and less than 0.5 are assigned as negative boxes and all remaining boxes are ignored. As for evaluation, 1000 bounding boxes with the highest confidence are used for Non-Maximum Suppression (NMS). The IoU threshold for NMS is set to 0.5 and the number of boxes after NMS is most 100. Secondly, SSD adopts VGG-16 which is pretrained on ImageNet as backbone. Following (Liu et al., 2016), the backbone of SSD has extra layers generate multi-scale feature maps. After these layers, six levels of features are created with dimensions of (512, 1024, 512, 256, 256, 256) and these features are taken as inputs to the classification head and the regression head. The same encoding/decoding scheme as RetinaNet was used. Different from RetinaNet, hard negative mining is applied to assign positive/negative boxes; specifically, three times more negative boxes than positive boxes are assigned. As for evaluation, the same NMS scheme is adopted as RetinaNet except the number of bounding boxes after NMS is set to 200. For both models, the SGD optimizer is used with learning rate of 0.001; a momentum of 0.9 and weight decay of 0.0001.
C ALGORITHM DESCRIPTION
The detailed procedure of the proposed active object detection method is given in Algorithm 1.
D PROOF OF MARGINAL LIKELIHOOD Loss
We consider the Crrichlet-Catgoriccl Bayesian framework where the low-order predictive distri- bution parameters 0 are sampled from the high-order prior distribution (i.e., 9 ~ Dir(fla) and 0 parameterizes categorical likelihood Cat(O).) To train our evidential model, we maximize the marginal likelihood which is given as below:
r K
1=1 03) I(aj)
K
p(ylaa= pluil0)((01)a00
-
I
j=1
13

---

Published as a conference paper at ICLR 2023
Algorithm Proposed Active Learning Method
Input:
Initialized model 4
[.b. [MER where to denotes all model parameters excluding MEH,
labeled set L unlabeled set U
Output: Model o D & ME) after I active learning cycles
for each active learning cycle f 1,.,, T do Step 1: Train model (including MEH module) using I,"
I: for each labeled data 3; L' do Predict class confidence 33 = .2, 3x3 and model evidence Aezpl8k)
Compute concentration parameter a {ar= E, cxp(d;) [k: 2, K. K}
Compute predictive prohability p(y = k(aa) = E.a.
du
5: Compute classification loss Lote (v. ply k(a))
6:
Update object detector D using Leta
7: Compute MEH loss LMER (A, Lats)
Update MEH 4 2
MEE using LACEH with bb fixed
9:
for
Step 2: Compute informativeness unlabeled samples via EDL and HUA
1: for each unlabeled data Iu U do
2: for each bounding box do
3:
Predict class confidence 3 = {B1, .2,..., 3x and model evidence À Aopp(ee) (9g)
Compute concentration parameter a {as Σ. (xp( 1, ..., K}
5: Sample categorical distributions as Dir(Ola)
6: Compute epistemic uncertainty Ijy, 0] (( [)]] Ex [H([((]))]]
7:
end for
8:
9: Aggregate Tly, 0) of all bounding boxes via HUA to get informativeness I(xr.)
Step 3: Select informative data and update L', U
I: Select the most informative data I' C U' based on I(U') (I(x.)la, E U*)
2: Human oracles make labels of I'
- L' I
Update labeled set // Update unlabeled set
4: U = U'nI
end
which is equivalent to
rE 03) IT I(a;) 1=1
/II
#
K
do
o3) IT I(aa)) 5=1
DE 1 /
do
(8)
The above integral can be simplified using a useful trick. Given Dir(Ola), integrating probability density function over 0 results in one:
EE aj) I(a,)
K 1 d0
ME ai) I I(ay))
/ 1 do =
K
I
1=1
j=1
Dividing the above equation by a constant part, which not involved in the integration, gives the following result:
[I _ do T(az) T(
(10)
Now we can regard equation (8) as the integration over "shifted parameter" (i.e., aj + L(y; = i)). This shift of parameter changes the equation (10) to:
K II 1=1
00 +11++ do = T + L(y; = =i))
(11)
N( + 1(y, = =i]))
Considering the case when yj = separately, the above equation (11) can be written as:
j M(a + (1 = i}) = 1 [(a;) +1) r< (a, 1 = i)))
(12)
r( +1)
14

---

Published as a conference paper at ICLR 2023
Now the fact that [( 1) rI(x) to simplify the above equation as:
K 1
I l'ay)L' +1) -y= I(aj)) (a))=(ay)) (ay)
(13)
(()==()a))) [a=(a,)
By substituting the result of the above equation in the integral of equation (8), we obtain:
p(y.la)= p(ux10)((0la)dd = - K 9j)
I ((a))
ai
ag
(14)
I r(ay) r()=((a))) Ej=(ay)
E)=((a))
E ANALYSIS OF THRESHOLD VALUES
To understand HUA better, we investigate the effect of each threshold value: Tscore: HoU. We vary the thresholds and report the results in Table 3. Note that aggregate functions (Sum, Max, Sum) are applied to information levels (Object, Scale, Class) all tries for fair comparison. can be seen that higher "IoU increases the performance. This is because excessively low thresholds cause too many unnecessary bounding boxes to be taken into account, thus deteriorating the quality of estimated informativeness. Otherwise, too high or too low Yscore impairs the quality of filtering and degrades the final performance.
Thresholds
Ratio (%) of Labeled Samples
Tscove
Yto
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0.2
0.7
52.53
60.40
66.35
69.19
70.82
72.11
73.32
0.2
0.8
52.53
60.62
66.44
69.22
70.59
72.52
73.63
0.2
0.9
52.53
60.95
66.68
69.71
70.90
72.69
73.71
0.3
0.7
52.53
61.16
66.09
69.29
70.98
72.57
73.80
0.3
0.8
52.53
61.54
66.62
69.57
71.28
72.70
74.02
0.3
0.9
52.53
62.02
66.84
69.82
71.31
72.86
74.08
0.4
0.7
52.53
59,43
65.84
67.92
70.07
71.90
73.15
0.4
0.8
52.53
61.30
65.35
67.82
70.66
71.69
73.35
0.4
0.9
52.53
60.75
65.68
68.66
71.09
71.81
73.78
Table Performance (mAP%) of RetinaNet PASCAL VOC.
F ANALYSIS OF HUA
Fig. 7 visualizes how HUA works in detail. Note that aggregation functions (Sum, Max, Sum) are chosen for (Object, Scale, Class). First, can be seen that unclear occluded objects have high uncertainty. For example in the left image, occluded boats in the left side show higher uncertainty than the one in the right side. can also be seen that uncertainties of numerous objects within the left image summed and contribute to the higher informativeness score. Note that since the aggregation function for "Class" is "Sum", bounding box which classified into multiple classes simultaneously is supposed to have high uncertainty. For example in the right image, the head of horse in the right image is classified into the horse and the cow at the same time; bounding boxes which predict either category are separated into independent groups, and uncertainty for each category group is computed independently and summed together.
G COMPARISON WITH OTHER LOSS TYPE FOR MEH
For the training of MEH, the mean Square Error (MSE) loss is adopted for regression of the target loss. In this section, we further investigate performances of other typical losses which are well- known for the regression problem. Table 4 compares the MAP50 score on PASCAL VOC with RetinaNet as the base model. The method MAE indicates the mean absolute error; given prediction ÿ and ground truth y, Loss(y.3 ÿ) ly ÿl. The MSLE method refers to the mean squared logarith- mie error, (i.e., Loss (y, 5) = (loy(y + 1) loy(ū + 1))²). Experimental results show that general losses for regression are all effective, yet MSE loss shows the best result.
15

---

Published as a conference paper at ICLR 2023
bost
boxt 0.3851
onL 35167 S466
KGD
Informativeness: [,(Unc) = 1.8663
Informativeness : Ej(Unc) 1.2331
Figure 7: Epistemic uncertainty of bounding boxes and eventual informativeness score of images.
Ratio (%) of Labeled Samples
Loss Type
5.0
7.5
10.0
12.5
15.0
17.5
20.0
MAE
52.53
59.55
65.65
68.78
70.54
71.80
73.50
MSLE
52.53
61.50
66.76
69.43
70.76
72.20
72.95
MSE
52.53
62.02
66.84
69.82
71.31
72.86
74.08
Table 4: Performance (mAP%) of RetinaNet on PASCAL VOC.
H ANALYSIS ABOUT THE SELECTION RATIO OF FILTERED SAMPLES
In this section, compare performance of the proposed method varying the selection ratio of filtered samples. Note that the bounding boxes whose confidence value are lower than "scorr are filtered
ignore predictions for background. This scheme naturally makes the informativeness score of any image to zero, if all bounding boxes in the image are unconfident. We stress that these images with zero informativeness are important since the object detector is unable find any object due to its lack of knowledge. In Table 5, it can be seen that taking some portion of these images of zero informativeness into account helps to improve the performance. This is because training with filtered images provides the model to experience difficult objects from unfamiliar categories. However, including too much filtered images into the labeled set of the next stage rather impair the performance. This is because some objcets with low confidence are too small/large or heavily occluded and interfere with training of the detector.
Ratio (%) of Labeled Samples
Selection Ratio
5.0
7.5
10.0
12.5
15.0
17.5
20.0
5%
52.53
61.07
66.15
68.09
70.26
72.24
72.80
10%
52.53
62.25
66.23
69.55
70.96
71.57
73.12
%
52.53
62.02
66.84
69.82
71.31
72.86
74.08
20%
52.53
62.28
66.86
69.42
71.27
72.36
73.56
30%
52.53
60.54
65.73
69.02
71.13
72.38
73.50
50%
52.53
59.40
65.04
68.79
70.01
70.26
72.46
Table Performance (mAP%) RetinaNet on PASCAL VOC.
16

---

Published as a conference paper at ICLR 2023
I COMPARISON OF AGGREGATION FUNCTION
An obvious question in HUA is: which combination of aggregation functions yields the best perfor- mance? Table we compare three types of functions: Avg. Max. Sum. In all tries, Tacore: "oo is set to 0.3, 0.9, respectively. We try various combinations applying different aggregation functions in each level information. It can seen (Sum, Max, Sum) gives the best performance exhibiting a large margin naive aggregation schemes of (Choi et al., 2021; Yuan al., 2021). Besides, operation observed to perform the best for "Object", since it reflects the number of objects inside an image. Likewise, sum operation shows the best result for "Class". It confirms the importance of considering uncertainties from ambiguous classes together. Otherwise, max operation prevails for "Scale". Since predictions can appear at various scales depending on the shape of the object regardless of the amount of information, it seems that the quality of uncertainty is rather impaired if all scales are considered.
Function
Ratio (%) of Labeled Samples
(Ohj, Scale, Class)
5.0
7.5
10.0
12.5
15.0
17.5
20.0
(Avg, Avg, Avg) (Yuan et al., 2021)
52.53
60.04
63.89
67.30
69.25
70.03
71.12
(Avg, Avg, Sum)
52.53
60.71
62.64
66.01
68.58
69.80
69.89
(Avg, Sum, Sum)
52.53
59.74
62.17
64.79
67.50
68.92
70.82
(Avg, Max, Sum)
52.53
59.78
63.50
66.52
67.85
69.38
70.03
(Max, Avg. Sum)
52.53
61.09
65.60
68.98
70.25
71.70
72.21
(Max, Max. Max) (Choi al., 2021)
52.53
60.29
64.52
68.
69.06
71.53
72.60
(Sum, Sum, Sum)
52.53
61.35
66.09
69.06
70.17
71.94
73.09
(Sum, Max, Avg)
52.53
61.12
65.19
67.89
70.45
72.28
73.10
(Sum, Max, Sum)
52.53
62.02
66.84
69.82
71.31
72.86
74.08
Table 6: Performance (mAP %) of RetinaNet PASCAL VOC.
J DETECTION OF NOISY LABELS
label!
Wrong Herse 4.327
label
Label (y) (RG.9)
Heese
Cow 4,605
Sheep 0.516
Cat 0.294
Dog
0,408
3.817
Figure 8: Noisy label detection via cross cross-entropy.
In active learning work, generally, it is assumed that perfect oracle generates 100% accurate labels unlabeled samples, that the main focus of existing active learning works (including our work) becomes effectively seleting the informative samples under the assumption with no label noise. In handle the label noise issue when
this section, we deal with an interesting question: how should human annotations are not 100% accurate? Orthogonal to the existing active learning methods, simple way utilize new module that identifies the noisy labels and re-label them or exclude them during training. As illustrated in works of (Bernhardt et al., 2022; Younesian et al., 2021), cross-entropy is commonly adopted to detect the mislabeled samples in noise-robust learning liter- model for detecting noisy labels, present simple toy
atures. To demonstrate the capability examples in Figure 8. To reproduce the noisy label, we change labels (y) of some bounding boxes to other similiar categories (e.g., horse - cow), and compute the cross entropy using the probability distribution (ý) Equation for comparison. As can be seen Figure 8, bounding boxes with wrong labels show much higher entropy, ensuring opportunities cope with noisy labels.
17

---

Published as a conference paper at ICLR 2023
Human 011296
Hurman: 2915
0.1255 BIDE
Hunman O 2.846
Human: 011183
avom
Human: 0.2111 Materhier 0,3600 0.6881
Human 0.1611
Informativeness L:(Epistemic Unc.) 1.5090
Informativeness (Epistemic Unc.) 0.7589
Figure Epistemic uncertainty of bounding boxes and resulting informativeness scores of images.
K ANALYSIS ON HUA WITH RESPECT TO THE NUMBER OF OBJECTS
As can be seen in Figure 6, HUA calculates image informativeness considering various information inside the image (number of objects, difficulty of each object), which makes the uncertainty scores of hard examples significantly higher than those of casy examples. However, this characteristic of HUA raises the following question: Does HUA favor only complex images with many objects? Wouldn't 't it be possible replace HUA by simply selecting images in which many objects appear? To answer these question, we investigated the relationship between the number of objects in the image and the uncertainty score, and found that a large number of objects does not always lead to high uncertainty score, as shown in Figure 9. This is because the resulting uncertainty of an image depends only on the number objects but also on the uncertainty value of each object. For example, in the image on the left of Figure 9, there exists two objects, but the total uncertainty higher than that of the right image, because the bounding boxes corresponding classified as people and motorbikes at the same time. On the other hand, since the objects in the right image are easy objects that are clearly classified only as people, the uncertainty of all five objects is not too large.
the objects are
L ANALYSIS ON THE ORDER OF HUA
In this section, we investigate the effect of the aggregation order in HUA to provide additional insight. Under the same conditions as the experiment conducted in the main manuscript, we only changed the order scale and class, i.e., the aggregation hierarchy in HUA is changed from "object (sum) scale (max) + class (sum)" to "object (sum) class (sum) +- scale (max)". As shown in the table below, the performance is degraded compared to the proposed scheme when we switch the order.
Ratio (%) of Labeled Samples
Method
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Reversed Order
52.53
62.42
66.42
68.69
70.58
72.06
72.98
Proposed Order
52.53
62.02
66.84
69.82
71.31
72.86
74.08
Table 7: mAP (%) of RetinaNet on PASCAL VOC.
We interpret this result as follows: 1) Since most objects do not span multiple scales (they mostly span 1 ~ scales), there is little difference even if the order of class and scale have changed. 2) If the order is changed, uncertainty values from multiple scales may be mixed into a single informativeness score of object. Rather, it seems compute the score of the image. Overall, both our intuition and experimental results indicate that the original order is more powerful and makes more sense.
appropriate to focus
single informative scale to
18

---

Published as a conference paper at ICLR 2023
M ANALYSIS ON GENERALITY OF HUA
To demonstrate the generality the proposed HUA, we conducted additional experiments. We combined traditional methods (Roy et al., 2018; Yuan et al., 2021) with HUA and reported the performance gain in Table 8.
Ratio (%)
Laheled Samples
Method
7.5
10.0
12.5
15.0
17.5
20.0
Entropy (Roy et al., 2018)
29.13
49.41
56.02
59.83
64.03
65.96
67.08
Entropy (Roy et al., 2018) + HUA
29.13
50.24
56.91
60.5
65.17
66.49
68.11
MIAOD (Yuan al., 2021)
47.18
58.11
63.72
67.32
69.39
70.83
71.91
MIAOD (Yuan al., 2021) HUA
47.18
59.07
64.61
68.76
70.04
71.95
72.63
Table 8: mAP (%) of RetinaNet on PASCAL VOC.
As shown in the table, HUA generally improves the performance in both cases. This is because HUA helps to better understand the context within images by considering the information hierarchy (e.g.. class, object, scale), than just taking the maximum value of the uncertainties of all bounding boxes as done in (Roy al., 2018; Yuan et al., 2021). This result confirms the intuition that existing works can be enhanced by adopting HUA for uncertainty aggregation.
N APPLICATION TO TWO-STAGE DETECTOR
FasterRCCN on PASCAL VOC
Backbone
features
RPN
70
Image
Proposals
65
Pooling
60
mAP(%)
pooled features
Cls. Head
pooled features
55
Random Sampling
MEH
SSM
50
LL4AL
pooled features
VAAL
pooled features
Reg. Head
45
LT/C
LS4C
Pmppoeed
40
1000
1500
2000
2500
3000
3500
Number of Labeled Images
(a) Obtaining concentration parameter a using two- stage detector
(b) Performance of FasterRCNN on PASCAL VOC
Figure 10: Proposed EDL-based uncertainty computation of bounding box using two-stage detector. (a) First, for an unlabeled image, backbone network and RPN produce features and proposals, respectively. Taken ROI pooled features as input, classification head generates class confidences 8 (&, [ while our
model evidence head (MEH) produces model evidence X. B and À are used to compute parameter set a of Dirichlet distribution Dir(O)a), (b) Performance (mAP %)) of FasterRCNN on PASCAI. VOC
Although we have verified our algorithms (MEH, HUA) only for single-stage detectors (RetinaNet, SSD) in the main paper. these algorithms can be casily extended to two-stage detectors as well. two- Fig. 10a shows the high-level descriptions of our approach under the EDL framework stage detector. For each proposal, the ROI pooled features are taken input by the classification head and MEH. Then class confidence 3 = {3x |K and model evidence À are generated, and concentration parameter of the prior Dirichlet distribution Dir (o)a) can be computed via equation (4). Given ~ Dir(fla), epistemic uncertainty of every proposal is computed via equation (1). Afterwards, epistemic uncertainty of all proposals are aggregated considering information hierarchy (e.g.. predicted class, scale, number of detected object) through HUA to get total uncertainty of the image. The only difference from the single-stage detector is that it computes uncertainty only for
19

---

Published as a conference paper at ICLR 2023
proposed bounding boxes by RPN rather than all bounding boxes. Fig. 10b shows the performance in PASCAL VOC 2012 when the proposed methods are applied to FasterRCNN (Ren et al., 2015). It can be seen that the proposed method performs better than various baselines (SSM(Wang et al., 2018), 1), LS+C and LT/C(Kao et al., 2018), VAAL(Sinha et al., 2019), LL4AL((o0 & Kweon, 2019)), which confirms that our scheme give general performance gain even in the two-stage detector.
20


Reviews:
[{"review_id": "m8_EhrdGhQ", "reviewer": "Reviewer_4FYw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines evidential deep learning (EDL) and active learning in the object detection task and achieve good results. The authors propose a new module model evidence head (MEH) and a new loss function to ensure that EDL can be effectively applied to object detection. For active learning, a hierarchical uncertainty aggregation (HUA) method is proposed to obtain the informativeness of an image. Experiments show that the proposed method brings a very good performance improvement.\n", "summary_of_the_review": "The paper cleverly combines EDL and active learning in the object detection task. It would be better to address the weaknesses.", "strength_and_weaknesses": "Strength:\n\n- EDL is applied to predict the epistemic uncertainty of samples. \n- The authors propose a HUA method to aggregate object-level uncertainty into image-level uncertainty.\n- Mathematical proofs make this method more rigorous and convincing. \nThe proposed method shows good performance.\n\nWeaknesses: \n\n- There are some points unclear on the uncertainty score. Is the epistemic uncertainty (Eq. 1) directly used as the uncertainty score of the box? If so, does this bring better results? Does this mean that object-level uncertainty does not need to account for aleatoric uncertainty. \n- Algorithm 1 seems to be just an overview of an active learning framework. The core part of active learning seems not present. The authors may need to describe Step 2 in detail: how to calculate the box uncertainty and how to aggregate the box uncertainty, which can make the algorithm clearer.\n- There seems no experiments to tell how about combining traditional methods and HUA.\n- The results of \"Examples of easy and hard samples\" are interesting. The uncertainty scores of complex images are significantly higher than those of simple images. However, does this lead to selecting only complex images and ignoring simple images each time? The images that are ultimately selected tend to be those that contain more objects. It seems that the active selection strategy is unnecessary. We may not need the epistemic uncertainty, but only need to roughly estimate which images contain more objects. Are there any further experiments to have a discussion?\n- Do the results obained using only MEH (without HUA) have a clear advantage over others? It would be better to have experiments on this point. Otherwise, it is difficult to prove the validity of epistemic uncertainty.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZdUfYAuPTM", "reviewer": "Reviewer_P2kB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors tackle the problem of model uncertainty estimation as the acquisition function in Active Learning for object\ndetection. To this end, they first adapt Evidential Deep Learning (EDL) to predict the epistemic uncertainty estimate \nfor each bounding box separately. They modify EDL in three ways: (a) replace the last layer's ReLU by softmax, \n(b) introduce a separate Model Evidence Head (MEH) that predicts model evidence, effectively re-scaling the\nconcentration parameter and (c) change the loss function to encourage the model evidence to be inversely proportional to\nthe classification loss. Once uncertainty is estimated for each bounding box, the authors introduce a three-level\nhierarchical scheme to aggregate uncertainty across bounding boxes by iteratively merging them based on their predicted\nclasses, scales and spatial overlap (objects).\n\nExperiments carried out on Pascal VOC and MS-COCO show superiority to competing state-of-the-art approaches in active\nlearning (final mAP when reaching a fixed proportion of labelled samples in a pool-based active learning simulation).\nAblation studies are also provided showing the relative improvement of each design decision and computational\ncomplexity.", "summary_of_the_review": "Overall, I think this is good quality work, with sufficient technical contribution and good empirical support.\nThe technical contribution could further be strengthened by providing a formal motivation for the MEH loss and a\ndiscussion of how one could model  epistemic uncertainty for the bounding box localization predictor head. Minor\nimprovements to clarity/corrections should be considered.\n", "strength_and_weaknesses": "* Strengths:\n\n  * High potential impact, given the relatively little work on Active Learning for object detection and the high\n  practical need for such methods in the industry (ubiquity of object detection in practice, high annotation costs)\n  * Low inference time makes this method quite attractive compared to other epistemic uncertainty evaluation approaches  \n  * Extensive evaluation, providing a fair comparison against relevant alternative methods, while also providing\n  ablation studies and runtime comparisons. \n  * The paper is well-structured and easy to follow.\n\n* Weaknesses:\n\n  * While the authors provide an intuition for their loss function for MEH (decrease model evidence when the bounding\n  box classification error is large and vice-versa), no probabilistic interpretation is given to this loss within the\n  EDL framework. The lack of a formal derivation makes it look somewhat empirical and disconnected to the rest of the\n  EDL framework.\n  * The MEH is attached to the classification head. However, epistemic uncertainty is not modelled for the regression\n  head (bounding box localization), and the method relies on the presence of multiple detections as a proxy for\n  localization uncertainty. While such improvements may be out of scope, I think the possibility of modelling epistemic\n  uncertainty jointly in classification/localization space should be discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OdbvNJsNoT7", "reviewer": "Reviewer_boP4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a model evidence head (MEH) and a hierarchical uncertainty aggregation (HUA) for active selecting informative images under the evidential deep learning (EDL) framework. The experiments conducted on PASCAL VOC and MS-COCO seem to validate the efficacy of the proposed method.\n", "summary_of_the_review": "I initially rated the paper as \"5: marginally below the acceptance threshold\". After carefully reading the authors' response, I am glad to see that they have addressed most of my concerns and therefore I would like to upgrade my rating to \"6: marginally above the acceptance threshold\".", "strength_and_weaknesses": "## Strengths\n- Overall the paper is well written and easy to follow.\n- The idea of MEH and HUA is interesting.\n- The experimental results are solid and look promising.\n\n## Weaknesses\n- Why both ReLU and ReLU + MEH perform worse than the random baseline in Table 1?\n- Is the proposed method able to handle the label noise? Usually we cannot guarantee that the human annotations are 100% accurate and therefore a solid active learning method should also be able to handle the label noise to some extent.\n- The baselines of active learning are not enough. The authors should compare with the latest methods not limited to:\n\n[1] Yoo, Donggeun, and In So Kweon. \"Learning loss for active learning.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.\n\n[2] Kim, K., Park, D., Kim, K.I. and Chun, S.Y., 2021. Task-aware variational adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8166-8175).\n\n[3] Yu, Weiping, Sijie Zhu, Taojiannan Yang, and Chen Chen. \"Consistency-based active learning for object detection.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3951-3960. 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G4N94PMpzXZ", "reviewer": "Reviewer_HUy5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work fist considers epistemic uncertainty for capturing the usefulness of the sample by model evidence head (MEH) which is a kind of evidential deep learning (EDL), and then proposes hierarchical uncertainty aggregation (HUA) for obtaining the informativeness of an image. Experimental results show the effectiveness of proposed method.", "summary_of_the_review": "I would recommend a marginally above acceptance. The experimental results and reported performances are the state-of-the-arts and may have a good impact to the field of classical objection detection, but i would like to know the influences in the DETR-like models. ", "strength_and_weaknesses": "Strengths:\n\n1. This paper aims to label a set of samples based on the uncertainty by adopting Evidential Deep Learning (EDL) to effectively compute epistemic uncertainty, and proposes the Model Evidence Head (MEH) that makes EDL highly compatible with object detection. \n2. The proposed Hierarchical Uncertainty Aggregation (HUA) makes use of attributes in bounding boxes, such as nearest object and box size, to capture the context within the image and improves the quality of the expected informativeness of images, instead of simply depending on the maximum/mean of all bounding boxes.\n\nWeaknesses:\n\n1. The motivation of MEH module is not clear, eg., why applying softmax function instead of ReLU operation can produce sufficiently confident prediction? Would replacing ReLU with sigmoid function can help MEH module?\n2. The lack of the denotation of symbol \\beta_{c} in equation 4.\n3. How to compute the large error in equation 6?\n4. Can authors provide the performance of variants of HUA where the order of `scale and class’ is switched. \n5. Why the concentration parameter cannot be measured by Reg.Head but for Cls. Head? \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4rzRTO-WzT", "reviewer": "Reviewer_tNAG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper employs evidential deep learning to assign uncertainty to unlabeled samples. Then, based on the assigned uncertainty, active learning is leveraged to ask humans to label more samples from the dataset. The authors have modified the original EDL. Specifically, the concentration parameters $\\alpha$ >1 have been changed to $\\alpha$>0.  Doing so would make Dirichlet distribution concentrate on sparse samples. The authors also introduced hierarchy uncertainty aggregation to decide the informativeness of an image. ", "summary_of_the_review": "This work is well presented. My two concerns are about the novelty and the experiemnts.\n1. The novelty is not clear as I cannot really tell why the difference would make such performance improvements in Fig. 5.\n2. The experiments seem not strong: missing two-stage detector, the experiments do not provide variations, the state-of-the-art methods are missing.", "strength_and_weaknesses": "+ The paper is well-motivated and well-written. Overall the idea is very clear. I can easily understand what the authors want to do.\n+ The experiments support the claims. In particular, the proposed method achieves better performance than the baselines.\n\n- The novelty of introducing EDL into active learning seems very straightforward. EDL is able to find some out-of-distribution samples during the labeling process, whether the unlabeled samples are out-of-distribution? This has not been explained clearly by the authors.\n\n- The authors have changed the concentration parameters from greater than 1 to greater than 0, this would affect the Dirichlet distribution. The authors should conduct experiments to show the differences, or at least some toy examples. Moreover, I cannot tell the significance of employing $\\lambda$. How do the differences between these two heat maps affect the final labeling? Will the differences impact significantly?\n\n- The authors only employ single-stage detectors to conduct the experiments. Whether the proposed method still works for two-stage detectors?\n\n- As this work focuses on active learning, in each round there would be some randomness. Therefore, the authors should report the variations of each experiment. In other words, by using different labeled samples at the beginning what the variations would be?\n\n- Last but not least, the authors use a certain ratio / percentage of the dataset samples. As different datasets would have different samples and this would make the contributions unclear as to whether the benefits come from the EDL active learning or more labeled samples at the beginning. \n\n- The baselines seems out of date. MI-AOD is published in CVPR 2021, which is the newest one in the comparisons. \nSequential Voting with Relational Box Fields for Active Object Detection, CVPR2022. Note that the github of this work is available.\n\n- The equations in the appendix is not new, which are very similar to EDL paper. Therefore, I do not think adding those equations make this paper valuable as those are not orignal deduction.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster