Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors
Authors: Ali Harakeh
Steven L. Waslander
Venue: ICLR 2021 Poster (2021)
Keywords: Object Detection
Predictive Uncertainty Estimation
Proper Scoring Rules
Variance Networks
Energy Score
Computer Vision

Abstract: Predictive uncertainty estimation is an essential next step for the reliable deployment of deep object detectors in safety-critical tasks. In this work, we focus on estimating predictive distributions for bounding box regression output with variance networks. We show that in the context of object detection, training variance networks with negative log likelihood (NLL) can lead to high entropy predictive distributions regardless of the correctness of the output mean. We propose to use the energy score as a non-local proper scoring rule and find that when used for training, the energy score leads to better calibrated and lower entropy predictive distributions than NLL. We also address the widespread use of non-proper scoring metrics for evaluating predictive distributions from deep object detectors by proposing an alternate evaluation approach founded on proper scoring rules. Using the proposed evaluation tools, we show that although variance networks can be used to produce high quality predictive distributions, ad-hoc approaches used by seminal object detectors for choosing regression targets during training do not provide wide enough data support for reliable variance learning. We hope that our work helps shift evaluation in probabilistic object detection to better align with predictive uncertainty evaluation in other machine learning domains. Code for all models, evaluation, and datasets is available at: https://github.com/asharakeh/probdet.git.

Published as a conference paper at ICLR 2021
ESTIMATING AND EVALUATING REGRESSION PREDIC- TIVE UNCERTAINTY IN DEEP OBJECT DETECTORS
Ali Harakeh
Steven L. Waslander
Institute for Acrospace Studies University of Toronto ali, www.aaaeuuttoooooooooo.o
Institute for Acrospace Studies
University of Toronto
stevenw@utias wtw.:aasstorooooooooom
A ABSTRACT
Predictive uncertainty estimation is an essential next step for the reliable deploy- ment of deep object detectors in safety-critical tasks. In this work, we foeus on estimating predictive distributions for bounding box regression output with vari- ance networks. We show that the context of object detection, training vari- ance networks with negative log likelihood (NLL) can lead to high entropy pre- dictive distributions regardless the correctness of the output mean. We pro- pose the energy score when used for training, the energy score leads to better calibrated and lower en- tropy predictive distributions than NLL. We also address the widespread use of non-proper scoring metrics for evaluating predictive distributions from deep ob- ject detectors by proposing an alternate evaluation approach founded on proper scoring rules. Using the proposed evaluation tools, we show that although vari- ance networks can be used to produce high quality predictive distributions, ad- hoc approaches used by seminal object detectors for choosing regression targets during training do not provide wide enough data support for reliable variance learning. We hope that our work helps shift evaluation in probabilistic object detection to better align with predictive uncertainty evaluation in other machine learning domains. Code for all models, evaluation, and datasets is available at:
non-local proper scoring rule and find that
INTRODUCTION
Deep object detectors are being increasingly deployed as perception components safety critical robotics and automation applications. For reliable and safe operation, subsequent tasks using detec- tors as sensors require meaningful predictive uncertainty estimates correlated with their outputs. As example, overconfident incorrect predictions can lead to non-optimal decision making in plan- ning tasks, while underconfident correct predictions can lead to under-utilizing information in sensor fusion. This paper investigates probabilistic object detectors, extensions of standard object detectors that estimate predictive distributions for output categories and bounding boxes simultancously.
This paper aims identify the shortcomings recent trends followed by state-of-the-ar probabilis- tic object detectors, and provides theoretically founded solutions for identified issues. Specifically, observe that the majority of state-of-the-ar probabilistic object detectors methods (Feng al.
2018a Le et al. 2018 Feng et al. 20186 He et al. 2019 Kraus & Dietmayer, 2019 Meyer et al.
2019 Choi et al. 2019 Feng et 2020 He Wang (020))aarakel
al. 2020 al. 2020) build on deterministie object detection backends to estimate bounding box predictive distributions by
modifying such backends with variance networks (Detlefsen et al., 2019). The mean and variance of bounding box predictive distributions estimated using variance networks are then learnt using negative log likelihood (NLL). also common for these methods to use non non-proper scoring rules such the mean Average Precision (mAP) when evaluating the quality of their output predictive distributions.
Pitfalls of NLL We show that under standard training procedures used by common object detectors, using NLL as a minimization objective results in variance networks that output high entropy dis- tributions regardless of the correctness of output bounding box. We address this issue by using

---

Published as a conference paper at ICLR 2021
the Energy Score (Gneiting& & Raftery 2007). pestance-sensitivee proper scoring rule based on engy statistics (Székely & Rizzo 2013), as an alternative for training variance networks. We show that predictive distributions learnt with the energy score are lower entropy, better calibrated, and of higher quality when evaluated using proper scoring rules.
Pitfalls of Evaluation We address the widespread use of non-proper scoring rules for evaluating probabilistic object detectors by providing evaluation tools based on well established proper scoring rules (Gneiting & Raftery (007) that are only minimized if the estimated predictive distribution is equal to the true target distribution, for both classification and regression. Using the proposed tools, we benchmark probabilistic extensions of three common object detection architectures on in-distribution, shifted, and out-tf-distributiin data. Our results show that variance networks can differentiate between in-distribution, shifted, and out-of-distribute tropy. We find that ad-hoc approaches used by seminal object detectors for choosing their regression targets during training do not provide a wide enough data support for reliable learning in variance networks. Finally, we provide clear recommendations in Sec. to avoid the pitfalls described above.
data using their predictive en-
RELATED WORK
Estimating predictive distributions with deep neural networks has long been a topic of interest for the research community. Bayesian Neural Networks (BNNs) (MacKay 1992) quantify predic- tive uncertainty by approximating a posterior distribution over a set of network parameters given a predefined prior distribution. Variance networks (Nix & Weigend 1994) capture predictive uncer- tainty (Kendall & Gal 2017) by estimating the mean and variance of every output through separate neural network branches, and are usually trained using maximum likelihood estimation (Detlefsen et al. 2019). Deep ensembles [Lakshminarayana, et al. (017) train multiple copies of the vari- ance networks from different network initializations to estimate predictive distributions from output sample sets. Monte Carlo (MC) Dropout (Gal & Ghahramani] 2016) provides predictive uncertainty estimates based on output samples generated by activating dropout layers at test time. We refer the reader to the work of Detlefsen et al. (2019) for an in depth comparison of the performance of vari- ance networks, BNNs, Ensembles, and MC dropout on regression tasks. We find variance networks to be the most scalable of these methods for the object detection task. Finally, we do not distinguish between aleatoric and epistemic uncertainty as is done in Kendall & Gal (2017). instead focusing on predictive uncertainty (Detlefsen et al. 2019) which reflects both types.
State-of-the-ae probabilistic object detectors model predictive uncertainty by adapting the work Kendall & Gal (017) to state-of-the-ard object detectors. Standard detectors are extended with variance network, usually referred to as the variance regression head, alongside the mean bounding box regression head and the resulting network is trained using NLL (Feng et al., 2018a Le al.) 2018 He et al. 2219; Lee et al. 2020; Feng et al., 2020 He & Wang] 2020). Some approaches combine the variance networks with dropout (Feng et al. 201 1 Kraus Dietmayer 2019) and use Monte Carlo sampling at test time. Others (Meyer et al. 2019 Choi et al. 2019 Harakeh et al.) 2020 make use of the output predicted variance by modifying the non-maximum suppression post- processing stage. Such modifications are orthogonal to the scope of the paper. It is important to note that a substantial portion of existing probabilistic object detectors focus on non-proper scoring metries such the mAP and calibration errors to evaluate the quality of their predictive distributions. More recent methods (Harakch et al. 2020 He & Wang 2020) use the probability- -ased detection quality (PDQ) proposed by Hall et al. (2020) for evaluating probabilistic object detectors, which can also be shown to be non-proper (See appendix (D3). Instead, we combine the error decomposition proposed by Hoiem al. (2012) with well established proper scoring rules (Gneiting & Raftery 2007) to evaluate probabilistic object detectors.
LEARNING BOUNDING Box DISTRIBUTIONS WITH PROPER SCORING RULES
3.1 NOTATION AND PROBLEM FORMULATION
Let X E R. be a set of m- mi-dimnnsional features, y E {I...., be classification labels for K-way classification, and z E Rdd be bounding box regression targets associated with object instances in the
2

---

Published as a conference paper at ICLR 2021
scene. Given a training dataset D = {I Vn. zn) 2n) of N ii. samples from true joint condi- tional probability distribution p" (y, 2 x = p² (zly, x)p" (ylx), we use neural networks with parame- ter vector 0 to model po (zly,x) x) and po (y(x). Po (z is fixed to be multivariate Gaussian distri- bution N (j((x, O), Exx, 0)), and po (y())) as a categorical distribution Cat(p1 (x, , o),...,PPP. P Unless mentioned otherwise,
< (x, 9))- R. and is represented as (umin: Umin; Mmax, Umax) where the (Umin, Umin), (mmax, Umax) are the pixel coordinates of the top-left and bottom-right bounding box corners respectively. Throughout this work, we denote random variables with bold characters, and the associated ground truth instances of random variables that are realized in the dataset 'D are itali- cized.
3.2 PROPER SCORING RULES
A scoring rule is function S(pe, (a, (a,z)) that assigns numerical value to the quality of the predictive distribution po(a(x) given the actual event that materialized a x ~ p p"(alar), (a]ar,, where a lower value indicates better quality. With slight abuse of notation, let S(po. p²) also refer to the expected value of S(po..), then a scoring rule is said to be proper if S(p", .p") I S(pe. .. with equality if and only if po = =p", meaning that the actual data generating distribution is assigned the lowest possible score value (Giniiting & Raftery 2007). We provide a more formal definition of proper scoring rules in Appendix D.2 Beyond the notion of proper, scoring rules can be further divided into local and non-local rules. Local scoring rules evaluate a predictive distribution based on its value only at the true target, whereas non-local rules take into account other characteristics of the predictive distribution. As an example, distance-sensitiy non-local proper scoring rules reward predictive distributions that assign probability mass to the vicinity of the true target, even if not exactly at that target. Lakshminarayanan et al. (2017) noted the utility of using proper scoring rules as neural network loss functions for learning predictive distributions.
Let be either
Predictive uncertainty for classification tasks has been extensively studied in recent literature Ova-- dia et al. 2019; AAhukka et al., 2020). We find commonly used proper scoring rules such as NLL and the Brier score (Brier 1950) to be satisfactory to learn and evaluate categorical predictive distri- butions for probabilistic object detectors. On the other hand, regression tasks have overwhelmingly relied single proper scoring rule, the negative log likelihood (Kendall & Gal 2017 Lakshmi- narayanan et al.) 2017 Detlefsen et al. 2019). NLL is a local scoring rule and should be satisfactory if used evaluate pure inference problems (Bernardo 1979). In addition, the choice of proper scoring rule should not matter as asymptotically, the true parameters of pe should be recovered by minimizing any proper scoring rule (Gneiting & Raftery 2007) during training. Unfortunately, object detection does not conform to these idcalized assumptions, we show in the next section the pitfalls of using NLL for learning and evaluating bounding box predictive distributions. We also explore the Energy Score (Gneiting & Raftery| 2007), proper and non-local scoring rule as an alternative for learning and evaluating multivariate Gaussian predictive distributions.
3.3 NEGATIVE LOG LIKELIHOOD AS A SCORING RULE
For multivariate Gaussian, the NLL can be written as:
N Zn u(xn,, O)), 0) 0) (Zn (1 0)) - log det E(on, O),
1
NLL- = 2N
(1)
where N is the size of the dataset D. NLL the only proper scoring rule that is also local (Bernardo) 1979]. with higher values implying a worse predictive density quality at the true target value. When used as loss to minimize, the first term of NLL encourages increasing the entropy of the predictive distribution as the mean estimate diverges from the true target value. The log determinant regular- ization term has a contrasting effect, penalizing high entropy distributions and preventing a zero loss from infinitely high uncertainty predictions at all data points (Kendall & Gal 2017).
has been shown by Machete (2013) that NLL prefers predictive densities that are less informative, likely outcome. We
penalizing overconfidence even when the probability mass is concentrated demonstrate the relevance of this property to object detection with a simple toy example. Bounding box results from the state-of-the-at object detector DETR (Carion et al.) 2020), trained to achieve a competitive mAP of 42% on the COCO validation split, are assigned a single mock multivariate Gaussian probability distribution N (H( I), where I is 4 x 4 isotropic covariance matrix
3

---

Published as a conference paper at ICLR 2021
Likelhoodd
10a
0.5
Direct Moment Matching
10a
0.5
10a
IOU x 0.5
IOU 0.5
10a
I0U
0.5
10a
10
10
Energy
10
10a
Negative
20
10
10a
10a
10
10-3
10
10a
10a
10:3
10a
10a
10
3
Figure 1: A toy example showing values of NLL (Blue), ES (Green) and DMM (Orange) plotted against the parameter a of an isotropic covariance matrix oI, when assigned to low error (Solid Lines) and high error (Dashed Lines) detection outputs from DETR.
with a as a variable parameter. We split the detection results into high error set with an IOU <= 0.5, and low enror set with IOU 0.5, where the IOU is determined the maximum IOU with any ground truth bounding box in the scene. We plot the value of the NLL loss of both high error and low error detection instances in Figure [ where NLL is estimated values of between [10-2, As expected, the variance value that minimizes NLL for low error detections is two orders of magnitude lower than for high error detections. What is more interesting the behavior of NLL away from its minimum for both low error and high error detections. NLL is seen to penalize lower entropy distributions (smaller values of o) more severely than higher entropy distributions, property that is shown to be detrimental for training variance networks in Section
3.4 THE ENERGY SCORE (ES)
The energy score is a strictly proper and non-local (Gneiting et al. 2008) scoring rule used to assess probabilistic forecasts of multivariate quantities. The energy score and can be written
N
M
M
M
ES
Zn.i
znl1
Znni
(2)
N
M
2M
n=1
where 2n is the ground truth bounding (H(x, 0), E(x,,, 0)).
box,
and
Zn,i the ith i.idd sample from
The energy score is derived from the energy distance Riizoo & Székely 2016 a maximum mean discrepancy metric (Sejdinovic al., 2013) that measures distance between distributions of random vectors. Beyond its theoretical appeal, the energy score has an efficient Monte-Carlo approxima- tion (Gneiting al. 2008) for multivariate Gaussian distributions, written as:
1 ES
Znns 2ns-111
Mnni z,
N M
2
which requires only single set of M samples to be drawn from N (M 0). E(x,, 0)) for every object instance in the minibatch. For training our object detectors, we find that setting . to be equal to 1000 allows to compute the approximation in equation 3 with very little computational overhead. We also use the value of 1000 for M when using the energy score as an evaluation metric.
The energy score has been previously used successfully as an optimization objective in DISCO Nets (Bouchacourt et al. (016) to train neural networks that output samples from a posterior prob- ability distribution, and to train generative adversarial networks (Bellemare et al. 2017). Unlike DISCO nets, we use the energy score learn parametric distributions with differentiable sampling.
Since the energy distance is non-local, it favors distributions that place probability mass near the ground truth target value, even if not exactly at that target. Going back to the toy example in Figure ES is shown to be minimized at similar values to the NLL, not surprising obscrvation given that both are proper scoring rules. Unlike NLL, ES penalizes high entropy distributions more severely than low entropy
In the next section, observe the effects of this property when using
See Appendix
4

---

Published as a conference paper at ICLR 2021
the energy score as a loss, leading to better calibrated, lower entropy predictive distributions when compared to NLL.
3.5 DIRECT MOMENT MATCHING (DMM)
final scoring rule we consider in our experiments for learning predictive distributions is the Direct Moment Matching (DMM) (Feng et al. 2020):
1 N Σ 2nn (a O)I + (Zn 0) (Zn pa ()) =1(22,, 0)) Ip- (4)
DMM
where 11-1|p is a p norm. DMM was previously proposed by Feng et al. (0220) as an auxiliary loss to calibrate 3D object detectors using multi-stage training procedure. DMM matches the mean and covariance matrix of the predictive distribution to sample statistics obtained using the predicted and true target values. DMM is not a proper scoring rule, we show this property with an example. If (z., f( 0)) a vector of zeros, DMM is minimized only if all entries of E( 0) are O regardless of the actual covariance of the data generating distribution. We use results from variance networks trained with DMM to discuss the pitfalls of only relying on distance-sensitive proper scoring rules for evaluation.
EXPERIMENTS AND RESULTS
For experiments. extend three common object detection methods, DETR (Carion et al. 2020), RetinaNet (Lin et al., 2017) and FasterRCNN (Ren et al.) (015)) with variance networks to estimate the parameters of multivariate Gaussian bounding box predictive distribution. We use cross-entropy to learn categorical predictive distributions and NLL, DMM, or ES to learn bounding box predictive distributions, resulting in 9 architecturefregression loss combinations. All networks have open source code and well established hyperparameters, which are used as is whenever possi- ble. Our probabilistic extensions closely match the mAP reported in the original implementation of their deterministic counterparts (See Fig. F.5). The value of used for computing DMM in equa- tion equation 4 is determined according to original norm loss used for bounding box regression in each deterministic backend. For FaNterRCNN and RetinaNet, we use the smooth L1 loss, while for DETR, we use the LI norm. Additional details on model training and inference implementations can be found in Appendix A
All probabilistic object detectors are trained on the COCO LLn et al. 2014) training data split. For testing, the COCO validation dataset is used as in-distribution data. Following recent recommen- dations for evaluating the quality of predictive uncertainty estimates (Ovadia et al. 2019). we also test our probabiliitic object detectors on shifted data distributions. We construct 3 distorted versions of the COCO validation dataset (C1, C3, and C5) by applying 18 different image corruptions intro- duced by Hendrycks & Dietterich (2019) increasing intensity levels (1, 3, 5). To test on natural dataset shift, we use Openlmages data (Kuznetsova et al. 2020) to create a shifted dataset with the same categories as COCO and -iist--dittibuttio dataset that contain none of the 80 categories found in COCO. More details on these datasets and their construction can be found in Appendix B
The three deterministic backends are chosen to represent one-stage (RetinaNet), two-stage (Faster- RCNN), and the recently proposed set-based (DETR) object detectors. In addition, the implementa- tion DETR RetinaNet, and FasterRCNN models is publicly available under the Detectron2 (Wu) et al. 2019) object detection framework, with hyperparameters optimized to produce the best detec- tion results for the COCO dataset.
4.1 EVALUATING PREDICTIVE DISTRIBUTIONS WITH PROPER SCORING RULES
Following the work of Hoiem al. 20112) we partition output object instances into four categories: true positives, duplicates, localization errors, and false positives based on their IOU with ground truth instances in a scene. False positives are defned as detection instances with an IOU < 0.1 with any ground truth object instance in the image frame, whereas localization errors are detection
5

---

Published as a conference paper at ICLR 2021
Negpttve Lieethood
10000
DETA DEIA DAW
DETA ES
50000
Scone
150
B5000
920
2000
100
13
100
Test Dotacet
Opentn
Test Datasat
Openim
Test Bataoat
Opentm
Test Datasct
Openkm
Test Datasot
Openm
600
Aarisnter MI
Negerttee Log Unditoed
600
Sarinatiet 3HM
NelnnoNLt :5
2250
300
Log
Engrgs
500
200
2000
530
100
C0C0
Operin
0000
Oserim
COCO
Openkm
Ocaco
Openim
COCo
220
Hegettve Log Lbsthood
Sxpuavee Enot
600
Energe 1
220
100
300
200
150
2000
Test Datacet
Openin
Test Datace
Opentn
Text Batscef
Openth
0000
Test
Datazet
Opertm
0000
Test Dataset
Openin
(a) Truc Positives
(b) Localization Errors
Figure 2: Average over 80 classification categories of NLL, ES, and MSE for bounding box pre- dictive distributions estimates from probabilistic detectors with DETR, RetinaNet, and RaN backends on in in-distribution (COCO), artificially shifted (CI-C5), and naturally shifted (OpenIm) datasets. Error bars represent the 95% confidence intervals around the mean.
True Positives
Loc, Errors
False Positives
36:00
2.00
1.50
34.00
F4.2
0.00
2.03
5.07c
(a) NLL
Brier Score
Figure 3: Average over 80 classification categories of NLL and Brier score for classification predic- tive distributions generated using DETR. Error bars represent the 95% confidence intervals around the mean. Similar trends are seen for RetinaNet and FasterRCNN backends Figures F2 F3
instances with 0.1 IOU 0.5. Detections with 0.5 IOU are considered true positives. If multiple detections have 0.5 < IOU with the same ground truth object, the one with the lower classification score is considered a duplicate. In practice, report the average of all scores for true positives and duplicates at multiple IOU thresholds between 0.5 and 0.95, similar to how mAP is evaluated for the COCO dataset. Table [ shows that the number of output objects in each of the four partitions is very similar for probabilistic detectors sharing the same backend, reinforcing the fairness of our evaluation. Duplicates seen to comprise small fraction of the output detections, and analyzing them is not found to provide any additional insight over evaluation of the other three partitions (See Figure F.I) Partitioning details and IOU thresholds can be found in Appendix C
True positives and localization eтrors have corresponding ground truth targets, their predictive distributions can be evaluated using proper scoring rules. As non-local rules, we use the Brier score (Brier 1950) for evaluating categorical predictive distributions and the energy score for eval- uating bounding box predictive distributions. As local rule, we use the negative log likelihood for evaluating both. False positives are not assigned ground truth target; argue that these output instances should be classified as background and assign them the background category as their clas- sification ground truth target. The quality of false positive categorical predictive distributions then evaluated using NLL or the Brier score. Bounding box targets cannot be assigned to false positives and as such only look their differential predictive entropy. In addition using proper

---

Published as a conference paper at ICLR 2021
Table I: Left: Results of mAP and calibration errors of probabilistic extensions of DETR, Reti- naNet, and FasterRCNN. Right: The number of output detection instances classified as true posi- tives (TP), duplicates, localization errors, and false positives (FP).
mAP(%)
MCE (Cls)
CE (Reg) 4
Counts Per Partition (in-dist)
Detector
Loss
COCO
OpenIm
COCO
OpenIm
COCO
OpenIm
TP
Dup.
Loc. Errors
FP
NLL
37.10
35.16
0.0735
0.0973
0.0512
0.0331
24452
1103
11228
12604
DETR
DMM
39.82
38.88
0.0737
0.0996
0.0091
0.0171
24929
1197
10374
10406
ES
39,96
39.31
0,0728
0.0993
0,0068
0.0167
24914
1216
10236
9761
NLL
35.11
34,60
0.0260
0.0385
0.0242
0.0176
22733
2835
8992
6654
RetinaNet
DMM
36,28
34.84
0.0244
0.0372
0.0102
0.0189
23096
2783
8902
6411
ES
36.02
34.97
0.0245
0.0370
0.0059
0.0138
23096
2783
6411
NLL
37.14
34.30
0.0490
0.0713
0.0049
0.0133
23145
4841
7323
6989
FasterRCNN
DMM
37.24
34.42
0.0496
0,0854
0.0112
0.0229
23187
4628
7165
6905
ES
37.32
34.75
0.0495
0.0849
0,0051
0.0121
23123
4612
7208
6760
scoring rules, we provide mAP, classification marginal calibration error (MCE) (Kumar et al.
2019)
and regression calibration error (Kulehhov et al. 2018) results for all methods in Table
4.2 RESULTS ANALYSIS
Figures 2 and 3 show the results of evaluating the classification and regression predictive distribu- tions for true positives, localization errors, and false positives under dataset shift and using proper scoring rules. figure@alse shows that the bounding box mean squared errors (MSE) for probabilis- tic extensions of the same backend very similar, meaning that differences between regression proper scoring rules among these methods arise from different predictive covariance estimates and not predictive mean estimates. Similar to what has been reported by Ovadia et al. (2019) for pure classification tasks, we observe that the quality of both category and bounding box predictive dis- tributions for all output partitions degrades under dataset shift. Probabilistic detectors sharing the same detection backend are shown in FigureBJte to have similar classification scores, which we expect given that we do not modify the classification loss function. However, when comparing regression proper scoring rules in Figure 2 the rank of methods can vary based on which proper scoring rule one looks at, a phenomenon we explore later in this section.
Advantages of Our Evaluation: Independently evaluating the quality of predictive distributions for various types of errors leads to a more insightful analysis when compared standard evaluation using mAp PDQ HHall et al. 2220 Fl As an example, Figure 3 shows probabilistic extensions of DETR to have a lower classification NLL, but a higher Brier score for their false positives when compared to their localization errors. We conclude that for its false positives, DETR assigns high probability mass to the correct target category while simultaneously assigning high probability mass to small number of other erroneous categories, leading to lower NLL but higher Brier score when compared to localization errors for which the probability mass is distributed across many categories. Our observation highlights the importance of using non-local proper scoring rules alongside local scoring rules for evaluation.
Pitfalls of Training and Evaluation Using NLL: Figure Soauee4hhooo shows the differential entropy of bound- ing box predictive distributions plotted against the error, measured as the IOU of their means with ground truth boxes. When using DETR or RetinaNet as backend, variance networks trained with NLL are shown to predict higher entropy values when compared to those trained with ES regard- less of the error. This observation does not extend the FasterRCNN backend, where variance networks trained with NLL and ES are seen to have similar predictive entropy values. Figure E shows the distribution of errors, measured as IOU with ground truth, of targets used to compute regression loss functions during training of all three detection backends. At all stages of training, DETR and RetinaNet backends compute regression losses on targets with a much lower IOU than FasterRCNN, which are seen in Figure 4 as low IOU tails in their estimated histograms. We ob- serve a direct correlation between the number of low IOU regression targets used during training and the overall magnitude of the entropy of predictive distributions learnt using NLL. DETR, using
For more details see Appendix D.3

---

Published as a conference paper at ICLR 2021
DETR
RetinaNeet
RCMN
DETA
RetinaNet RCNN
ONf Engropy (Nts)
DMM
Dift Eneropy (aat)
1.0
Rg
0.8
Entrapy
Targee
0.4
peg.1
0.25
0.25
0.50 0.75
0.25 0.50 0.75
0.00
100 with GT
Training Progress
Figure 4: Left: Differential Entropy vs IOU with ground truth plots for bounding box predictive distribution estimates on in-distribution data. Right: Histograms of the IOU of ground truth boxes with boxes assigned as regression targets during network training, plotted at 0%. 50%, and a 100% of the training process. The red dashed line signifies the 0.5 IOU level both plots.
RCNN NLL
RONK DMH
2.20
RCNN ES
Robimafett NLL
RotinaNet DMN
Retinatie
9.15
0.28
0.15
Inf
0.26
8:36
2.36
Sh
0:16
0.15
Stillz
20
50117
17
000
22 12
20.10 23-12
22013
00D
20.0
0.08
29.08
g0.01
20.00
5.06
3.56
0.05
0.26
0.05
0.52
0.02
3.02
0.02
0.00
0.02
20.5
30.0
20
01
20.5
30.0
0.0
70.0
3.5
2 230
Exrrpy
Enirpyy
Encropy
Endropy (hats)
Encropy
(a) FasterRCNN
(b) RetinaNet
Figure 5: Histogram of differential entropy for false positives bounding box predictive distributions produced by probabilistic detectors with FasterRCNN and RetinaNet as a backend. Results for DETR exhibit similar trends (Figure F.4.
the largest number of low IOU regression targets is seen in Figure 4 learn the highest entropy predictive distributions, while FasterRCNN using the fewest number of low IOU regression targets learns the lowest entropy predictive distributions. Variance networks trained with ES do not exhibit similar behavior producing a consistent magnitude for entropy regardless of their backend. Table [] shows NLL have a 2. 2.7%, 0.9%, and 0.14% reduction in mAP compared ES and DMM when used to train DETR, RetinaNet, and FasterRCNN, respectively. Figure 4 shows that the drop in mAP when using NLL for training is directly correlated the number of low IOU regression targets chosen during training by the three deterministic backends. Table I distributions of DETR-ES and RetinaNet-FS to achieve a much lower regression calibration error than high entropy distributions of DETR-NLL and RetinaNet-NLLL
also shows low entropy
Columns 2 and 3 of Figure 2 show DETR-ES and RetinaNet-ES to have lower negative log likeli- hood and energy score for bounding box predictive distributions of true positives when compared to DETR-NLL and RetinaNet-NLL. For localization errors (Columns and 5), the deviation from the mean is large, and as such the high predictive entropy provided by DETR-NLL and RetinaNet-NLL leads to lower values on proper scoring rules when compared to DETR-ES and RetinaNet-ES. We notice that if one uses only NLL for evaluation, networks estimating high entropy predictive distributions regardless of the deviation from the mean, as seen for DETR-NLL on true positives in Figure 2 We can mitigate this issue by evaluating with the en- ergy score, which is seen to distinguish between correct and incorrect high entropy distributions. As an example, DETR-NLL shown to have lower negative log likelihood but much higher Energy score when compared to DETR-ES for true positives on shifted data. On the other hand, DETR-NLL shows a slightly lower energy score when compared to DETR-ES on localization errors, meaning the energy score can indicate that the high entropy values provided by DETR-NLL are better esti- mate for localization errors than the low entropy ones provided by DETR-ES. Considering that true positives outnumber localization errors by least two in Table we argue that it is more beneficial to train with the energy score for higher quality true positives predictive distributions over training with the negative log likelihood and predicting high entropy distributions regardless of the true error.
achieve constant value of NLL by
Evaluating only with the energy score also has disadvantages, show that it does not sufficiently discriminate between the quality of low entropy distributions. Figure 2 shows that CMMMtrained networks predicting the lowest entropy (Figure 4 achieve similar values of the energy score, but
8

---

Published as a conference paper at ICLR 2021
much higher values of negative log likelihood when compared networks trained with ES. The higher NLL score seen in Figure 2 for all networks trained with DMM a lower quality distributions when compared to networks trained with ES, specifically at the correct ground truth target value.
Pitfalls of Common Approaches For Regression Target Assignment: Figure shows the entropy for all methods with the DETR backend to steadily decrease function of decreasing error. On the other hand, the entropy of methods using RetinaNet and FasterRCNN backend is seen to have two inflection points, one at an IOU of 0.5 and another higher IOU of around 0.9. As a func- tion of decreasing error, the entropy increases before the first inflection point, decreases between the two, and then increases again after the second inflection point. We hypothesize that this phe- nomenon is caused by the way backends choose their regression targets during training. DETR uses optimal assignment to choose regression targets that span the whole range of possible IOU with GT, even during final stages of training, as is visible in Figure 4 On the other hand, RetinaNet and FasterRCNN use ad-hoc assignment with IOU thresholds Ren et al. (2015), a method that seen to provide regression targets concentrated in the 0.5 to 0.9 IOU range throughout the training process, resulting in a much narrower data support when compared to DETR. We conclude that out- side the data support, variance networks with RetinaNet and FasterRCNN backends fail to provide uncertainty estimates that capture the quality of mean predictions. Our conclusion is not unprece- dented, Detlefsen al.] (2019) has previously shown variance networks perform poorly out of the training data support for multiple regression tasks. However, our analysis pinpoints the reason of such behavior in probabilistic object detectors, showing that well established training approaches based on choosing high IOU regression targets work well for predictive mean estimation, but are not necessarily optimal for estimating predictive uncertainty.
Performance on OOD Data: Finally, 3 histograms of regression predictive entropy of false positives from probabilitic detectors with DETR and FasterRCNN backend on in diniistribution, naturally shifted, and out-of- oi-diitrinttiin data. Variance networks trained with any of the three loss functions considered achieve the highest predictive differential entropy on muttrf-distribution data, followed by the skewed data and achieve the lowest entropy on the in-distribution data, showing that variance networks are capable of reliably capturing dataset shift when used to predict uncertainty.
5 TAKEAWAYS
We propose to use the energy score, a proper and non-local scoring rule to train probabilistic de- tectors. Answering the call to aim for more reliable benchmarks in numerous setups of uncertainty estimation (Ashukha et al., 2020) we also present tools to evaluate probabilistic object detectors using well-established proper scoring rules. We summarize main findings below:
No single proper scoring rule can capture all the desirable properties of category classifica- tion and bounding box regression predictive distributions in probabilistic object detection. We recommend using both local and non-local proper scoring rules on multiple output partitions for a more expressive evaluation of probabilistic object detectors.
Using a proper scoring rule as a minimization objective does not guarantee good predictive uncertainty estimates for probabilistic object detectors. Non-local rules, like the energy score, learn better calibrated, lower entropy, and higher quality predictive distributions when compared to local scoring rules like NI.I.
1OU-based assignment approaches used by established object detectors for choosing re- gression targets during training overly restrict data support to high IOU candidates, leading to unreliable bounding box predictive uncertainty estimates over the full range of possible errors.
Variance networks are capable of reliably outputting increased predictive differential en- tropy when presented with out-of-distribution data for the bounding box regression task.
In this paper, we provide better tools for estimation and assessment of predictive distributions, by using variance networks with existing object detection architectures as a backend. Designing novel architectures that aim for accurate bounding box predictive distributions, rather than just accurate predictive means, remains important open question.

---

Published as a conference paper at ICLR 2021
REFERENCES
Phil Ammirato and Alexander C Berg. A mask-renn baseline for probabilistic object detection. In CVPR workshop the Robotic Vision Probabilistic Object Detection Challenge, 2019.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain
uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations (ICIR), 2020.
Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as a solution to biased wasserstein gradi- ents. arXiv preprint arXiv: 2017.
José M Bernardo. Expected information expected utility, the Annals of Statistics, pp. 686-690,
1979.
Diane Bouchacourt, Pawan Mudigonda, and Sebastian Nowozin. Disco nets Dissim- ilarity coefficients networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 352-360. Curran Associates, Inc., 2016. URL http::/paperss nips. cc/paper/ 6143-di www--diico--es------------------
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78,1):1-3, 1950.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. 2020.
Jiwoong Choi, Dayoung Chun, Hyun Kim, and Hyuk-Jae Lee. Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 502-511, 2019.
Nicki S. Detlefsen, Martin Jorgensen, and Soren Hauberg. Reliable training and estimation of vari- ance networks. In Advances in Neural Information Processing Systems 32, pp. 6326 6336. 2019.
Di Feng, Lars Rosenbaum, and Klaus Dietmayer. Leveraging heteroscedastic aleatoric uncertainties for robust real-time lidar 3d object detection. In 2018 2 International Conference on Intelligent Transportation Systems (ITSC), 2018a.
Di Feng, Lars Rosenbaum, and Klaus Dietmayer. Towards safe autonomous driving: Capture un- certainty in the deep neural network for lidar 3d vehicle detection. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018b.
Di Feng, Lars Rosenbaum, Claudius Glaeser, Fabian Timm, and Klaus Dietmayer. Can we trust you? on calibration of a probabilistic object detector for autonomous driving. The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. pp. 1050-1059, 2016.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation.
Journal of the American statistical Association, 102(477):359-37, 2007.
Tilmann Gneiting, Larissa I Stanberry, Eric P Grimit, Leonhard Held, and Nicholas A Johnson. Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble pre- dictions of surface winds. Test, 17(2)2211, 2008
Alexey A Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snock, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. arXiv preprint arXiv:2008.01 160, 2020.
David Hall, Feras Dayoub, John Skinner, Haoyang Zhang, Dimity Miller, Peter Corke, Gustavo Cameiro, Anelia Angelova, and Niko Sünderhauf. Probabilistic object detection: Definition and evaluation. In The IEEE Winter Conference on Applications of Computer Vision, pp. 1031-1040, 2020.
10

---

Published as a conference paper at ICLR 2021
Ali Harakeh, Michael Smart, and Steven L Waslander. Bayesod: A bayesian approach for un- certainty estimation in deep object detectors. 2020 International Conference on Robotics and Automation (ICRA), 2020.
Yihui He and Jianren Wang. Deep mixture density network for probabilisticc object detection. The JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang. Bounding box regression with uncertainty for accurate object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2888-2897, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. International Conference on Learning Representations (ICLR), 2019.
Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. Diagnosing error in object detectors. In European conference computer vision, pp. 340-353. Springer, 2012.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In I. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5574-5584. Curran Associates, Inc., 2017.
Florian Kraus and Klaus Dietmayer. Uncertainty estimation in one-stage object detection. 2019 2 International Conference Intelligent Transportation Systems (ITSC), 2019.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2796-2804, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL bttp://proceciings. mlr. /tt.e/eee.........o...eeeeeeeee. html
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems 32, pp. 3792-3803. 2019.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Kamali, Stefan Popov, Matteo Malloei, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision pp. 1-26, 2020.
Shahab
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6402-6413. Curran Associates, Inc., 2017.
Michael Truong Le, Frederik Diehl, Thomas Brunner, and Alois Knol. Uncertainty estimation for deep neural object detectors in safety-critical applications. In 2018 21 International Conference on Intelligent Transportation Systems (ITSC). IEFF, 2018.
Youngwan Lee, Joong-won Hwang, Hyung-ll Kim, Kimin Yun, and Joungyoul Park. Localization uncertainty estimation for anchor-free object detection. arXiv preprint arXiv:2006. 15607, 2020.
Chun-Liang Li, Wei-Cheng Chang. Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pp. 2203 2203,221,,200 2017.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference Machine Learning, pp. 1718-1727, 2015.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
11

---

Published as a conference paper at ICLR 2021
Reason L Machete. Contrasting probabilistic scoring rules. Journal of Statistical Planning and
Inference, 143(10):1711-1790 2013.
David JC MacKay, A practical bayesian framework for backpropagation networks. Neural compu- tation, 4()):488472, 1992.
Gregory Meyer, Ankit Laddha, Eric Kee, Carlos Vallesp--Gonzales and Carl Wellington. Laser- net: An efficient probabilistic 3d object detector for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12677-12686, 2019.
David A Nix and Andreas Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings /994 ieee international conference on neural networks (CNN"94), volume 1, pp. 55-60. IEEE, 1994.
Yaniv Ovadia, Jasper Snoek, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Scul- ley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32, pp. 13991-14002. 2019.
Shaoqing Ren, Kaiming He, Ross Girshiek, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances Neural Information Processing Systems 28, pp. 91-99. Curran Associates, Inc., 2015.
Maria I. Rizzo and Gábor J Székely. Energy distance. wiley intendixciplinary reviews: Computa- tional statistics, 8(1):27-38, 2016.
Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics, pp. 2263- 2013,
Gábor J Székely and Maria L Rizzo. Energy statistics: A class of statistics based on distances. Journal of statistical planning and inference, 1449 1272, 2013.
Chuan-Wad Wang, Chin-An Cheng, Ching-Ju Cheng, Hou-Ning Hu, Hung-Kuo Chu, and Min Sun. Augpod: Augmentationn-orienee probabilistic object detection. In CVPR workshop on the Robotic Vision Probabilistic Object Detection Challenge, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. http:p//ttthnoooooceeceeeeeeeccc 2019.
12

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
A.1 MODEL IMPLEMENTATION
The implementation of DETR RetinaNet, and FaSterRCNN models is based on original PyTorch implementations available under the Detectron2 (Wu et al. 2019) object detection framework. Reti- naNet, FasterRCNN, and DETR do not directly estimate bounding box corners, but work on trans- formed bounding box representation b T(z) where T T + Rd is an invertible transformation with inverse T ()). As an example, FasterRCNN estimates the difference between proposals gen- erated from region proposal network and ground truth bounding boxes. We train all probabilistic object detectors to estimate the covariance matrix Σg of the transformed bounding box representa- tion b. To predict positive semi- covariance matrices, each of these three object detection architectures are extended with a covariance regression head that outputs the 10 parameters of the lower triangular matrix L of the Cholesky decomposition Σ, LLT. The diagonal parameters of the matrix L are passed through the exponential function to guarantee that the output covariance Σ, positive semi-definite. We train two models for each of the 9 auchitecture/regression loss combina- tion, one using a full covariance assumption and the other using a diagonal covariance assumption. We report the results of the top performing variant chosen according to its performance on mAP as well as regression proper scoring rules. Considering the covariance structure as hyperparam- eter was necessary for fair evaluation as we could not get RetinaNet to converge with NLL and a full covariance assumption (See Figure [.5). In case of a diagonal covariance matrix assumption, off-diagonal elements of Σ, are set to O. The covariance prediction head for each object detection architecture is an exact copy of the bounding box regression head used by that architecture, taking the same feature maps as input. Variance networks are initialized to produce identity covariance matrices, which find to be key for stable training, especially for NLL loss.
A.2 MODEL INFERENCE
When evaluating our probabilistic detectors one needs to be agnostic to the source of probabilistic bounding box predictions. As such, all considered methods are required to provide a consistent output bounding box representation and a corresponding covariance matrix Σ as presented in Sec- tion 3 We approximate z ~ N(,, Σ) by drawing 1000 samples from b ~ N Σb), passing those through (-), and then estimate i and Σ as the sample mean and covariance matrix. Note that because of T(-) a diagonal Eg does not in general lead to diagonal Σ. Other than the spe- cial consideration required to estimate the final bounding box probability distribution, the inference process is fixed to be the one provided in the original implementation for all detectors.
A.3 MODEL. TRAINING
For training all probabilistic extensions of the three object detectors, We stick the hyperparameters provided by the original implementation whenever possible. Complex architectures such as DETR need couple of weeks of training on our hardware setup and as such searching for the optimal values of hyperparameters for each tested configuration is outside the scope of this paper. All models are trained using a fixed random seed, which shared across all APIs (numpy, torch, detectron2, etc...). This insures that empirical results are not determined based on lucky convergence, we train using 5 random seeds per configuration and find the results to be consistent with very small variance in terms of mAp and probabilistic metrics.
RetinaNet and FaNterRCNN both use ResNet-50 followed by a feature pyramid network (FPN) for feature extraction. Since both models are trained with SGD and momentum in the original implementation, we use the linear scaling rule to scale down from batch size of 16 to a batch size of 4 due to hardware limitations. We train our probabilistic extensions of those models using 2 GPUS with a learning rate of 0.0025 for RetinaNet and 0.005 for FasterRCNN, Both models are trained for 270000 iterations, and the learning rate is dropped by factor of 10 at 210000 and then again at 250000 iterations. All additional hyperparameters are left intact. Both RetinaNet and Faster-RCNi was trained using a soft mean warmup stage (Detlefsen et al. 2019). This is achieved through loss
13

---

Published as a conference paper at ICLR 2021
annealing, where the bounding box regression loss is defined as:
Lreg = (1 X)
Loriginal + Alerrobebilisties
min (1, 250000
(5)
100"
À = 100 -
where Loriginal is the regression loss in the original non-probabilistic implementation the respec- tive object detector, and is the current training step. The value of 100 base for the exponent was chosen using hyperparameter tuning. This loss formulation ensures that the network starts by emphasizing learning of the bounding box, slowly shifting learning the probabilistic regression loss as training proceeds. For the last 20000 steps, only the probabilistic regression loss is used for training. We found this loss formulation to be essential for convergence of models trained using the NLL. Each FasterRCNN model takes ~ 3 days to train using 2 P-100 GPUs. On the same setup, RetinaNet models take- 4 days to finish training.
DETR also uses ResNet-50 as base feature extractor. DETR's original implementation requires very long training schedule for convergence, leading us to use hard mean warmup for all probabilis- tic DETR models. We use the model parameters provided by DETR's 's authors after training for 500 epochs to reach 42% mAP on the COCO validation dataset. Weights from this deterministic model used as initial weights of all probabilistic extension of DETR, which are trained for additional 50 epochs using losses presented Section 3 and the same hyperparameters of the original deter- ministic implementation. We reduce the batch size from 64 to 16, but use the same initial learning rate as DETR is trained with ADAM. The learning rate is then dropped by a factor of 10 after at 30 epochs. Training for 50 epochs takes 4 days using 4 T-4 GPUs.
B SHIFTED AND OOD DATASETS
B. SHIFTING COCO DATASET WITH IMAGENET-C CORRUPTIONS
We corrupt the COCO validation dataset using 18 corruption types proposed in ImageNet- Hendryycks & Dietterich] 2019) at 5 increasing levels of intensity. The frames of COCO vali- dation dataset skewed using every corruption type in repeating sequential order, such that the first corruption type is applied to frame 1, 19, 37, 19,37..... the second to frames 2, 20, 38.... and so on. By increasing the corruption intensity from level level 5, we create 5 shifted versions of the 5000 frames of the COCO validation dataset such that every frame is skewed with the same corruption type, but an increasing intensity.
20"
Openim
104
of instarces
107
101
mataronh
triilic feth
Categories
Figure B.1: Histogram comparing the number of instances of every category present in the COCO validation dataset with those present in our shifted dataset constructed from OpenImages-V V4 frames.
14

---

Published as a conference paper at ICLR 2021
B.2 SHIFTED AND OUT OF DISTRIBUTION DATASETS FROM OPENIMAGES- V4
Shifted Dataset: To test methods beyond artificially shifted datasets, we create new dataset com- prising of 9, 3,3 frames from OpenImages- (Kuznetsova et al. 2020) 2D detection data, contain- ing instances belonging to the 80 object categories found in the COCO dataset. This testing data is naturally shifted due to differences in image quality, and different labeling mechanisms employed to generate the ground truth object boxes. Figure shows the number of instances belonging to each one of the 80 categories in the COCO validation dataset when compared to generated OpenIm- ages dataset. We tried to maintain the balance categories found in the COCO validation dataset, aim to highlight performance differences originating from distribution shift, rather than the number of instances per category.
To test methods out-of-distribution data, collect 1, 852 frames from Openlmages-Ve V4, containing no instances from any of the 80 categories found in the COCO dataset. All frames were manually checked minimize the existence of unlabeled in-distribution category instances.
tut-Of-Diitribution Dataset:
C PARTITIONING ERRORS IN OBJECT DETECTION
To maximize mAP, all our detection architectures are designed to produce fixed number of detec- tions per frame, usually 100 (Carion al., 2020), regardless of their classification performing our probabilistic evaluation on objects that can be trivially eliminated by score thresh- old, we filter the output of our probabilistic detectors based on a classification score that maximizes the F-1 score on the COCO in-distribution dataset.
To avoid
To analyze the predictive distributions provided by object detection models, we partition their fil- tered results into mutually exclusive subsets by adapting the object detection eтror decomposition presented by Hoiem et al. (2012). For every detection instance, the intersection-ove over-union(IOU) is calculated with every ground truth instance in the scene to determine the largest IOU, IOU Based on IOUmazi detection instances are partitioned into false positives, true positives, localization errors, duplicates. False positives are determined as detection instances for which IOU S 0.1 whereas localization errors are defined as ones with 0.1 < IOU (aUmaz < 0.5. Multiple localization er- rors be assigned the same ground truth detection; failed post-processing stages usually found in modern object detectors (non-maxine) maximum suppression for instance) and should neither be ignored nor lumped with false positives. To avoid discussion the definition of true positives, we define an IOU threshold Thp E {0.5, 0.55, 0.95} and con- sider the detection with YOUman 2 Thtp a true positive detection. For finer error decomposition, two detections are assigned the same ground truth instance, the one with lower classification score considered duplicate. The choice of Mtp is inspired by how mean average precision is computed on the COCO (Lin et al. 2014). By computing averages of evaluation metrics on true positives and
argue that such duplication is artifact of
Figure C.I: Detection instances from deterministic Faster- RCNN partitioned into true positives
(blue), duplicates (magenta), localization errors (teal), and false positives (red) according to Mtp 0.5.
15

---

Published as a conference paper at ICLR 2021
duplicates determined using multiple IOU thresholds, we aim for a fair evaluation for models that are skewed for better performance at either a low or high IOU.
Unlike the enror decomposition presented in Hoiem et al.) (2012) do not combine duplicates with localization errors as they are expected to have different predictive uncertainty qualities. Further- more, we do not put any constraint the category of ground truth instances assigned to predicted objects. We argue that well localized but miss- ciisssllassffied object instances should not be counted as false positives, but should instead have their predictive categorical distribution evaluated through classification proper scoring rules (Ovadia et al. 2019). Finally. we do not consider false negatives in our evaluation as analyzing this type of error from probabílistic detection prospective does not provide additional insights beyond what was provided by Hoiem et al. (2012).
D EVALUATING PREDICTIVE DISTRIBUTIONS
With the increase in interest surrounding the estimation of predictive distributions in recent deep learning literature, correctly evaluating such distributions given members of a test dataset is of ut- most importance. This section aims explain what characteristics we would like see in high quality predictive distributions. It also provides a mathematical explanation of proper scoring rules.
D.1 CALIBRATION AND SHARPNESS
Given a predictive distribution p(z) x, D; 0) learnt using a training dataset D, and a testing dataset D' / 2/1 n fI....,'') we need a scoring rule S that assigns a numerical score to the predictive distribution p(z|ar', D; 0) given the actual event z/, that materialized in the testing dataset. question naturally arises on the nature of the qualities of the predictive distributions than need to be captured by S. /Gneitig & Raftery) (2007) contended that the goal of a predictive distribution is to maximize the sharpness around the materialized event z,, subject to calibration. Calibration (Kuleshov et al., 2018 Kumar et al. 2019) is joint property of the estimated pre- dictive distribution as well as the events or values that materialized that reflects their statistical consistency (Gneiting & Raftery 2007). In simple words, if well-calibrated predictive distribution assigns a 0.8 probability to an event, the event should occur around 80% of the time. From its defini- tion, one can see that calibration by itself is enough to guarantee useful predictive distributions. As an example, a predictive distribution
1 E(z] otherwise
p(z|x, D) = -)=
perfectly calibrated, but very useful (unless one wants always predict E(z)). Sharpness on the other hand quantifies the concentration of the predictive distribution around the true materialized event z and property of the predictive distribution only. Good predictions need be sharp, but ideal predictions should be both sharp and well-calibrated We would like our scoring rules S to capture both the sharpness and the calibration predictive distribution.
D.2 PROPER SCORING RULES
Let Ω be a general sample space, A be a r-algebra of Ω and P be a convex class of probabil- ity measure on (2, A) such that {p.p" P). A scoring rule S P x Ω + R is written as S(p(z/æ', D; D) 2 and maps a predictive distribution and a materialized event to scalar value. Throughout the rest of this dissertation, scoring rules will be negatively oriented, that is the lower the score the better the predictive distribution.
With some abuse of notation, given the real data generating distribution p'(z write the expected score under p* as:
S(p(z|a', D;O), (xla()) = S(p(z|a' D;0), 2%) dd))d"(2',))) (2/,)=)))
(6)
A scoring rule is said to be proper relative to P if:
S (p" (z(z²)) , p² (z)xx,)) & S (p(z|ar, D; (211))((2)),) Vp.p" EP,
16

---

Published as a conference paper at ICLR 2021
(a) Case
(b) Case
(c) Case
Figure D.I: A toy example showing the three bounding box distributions used to show the spatial quality 2s to be non-proper scoring rule. The green bounding box is the ground truth sample, whereas the red bounding box is the predicted mean of the probability distribution. The 95% confi- dence ellipse of bounding box corner predictive distributions is also plotted in red.
Table NLL, ES, and Spatial Quality (Qs) results of the three predictive distributions from our toy example.
Case #
NLL 4
ES
Qs (%)
20.49
23.10
51.88
2
20.49
23.10
45.98
3
19.33
21.21
50.02
and strictly proper relative to P Equation equation holds with equality only if p = p*. In simple words, strictly proper scoring rule is only minimized if the predictive distribution is exactly equal to the true data generating distribution. In both cases, the a lower score signifies predictive distributions that closer the data generating distribution. It is clear that scoring rules to rank predictive distributions based on theoretically founded quantities.
proper
D.3 PDQ IS NOT PROPER SCORING RULE
PDQ can written as:
PDQ(G,D) [g] NFP 6.5
EPPDQIG(,DY)
where G 7th ground truth object instance the fth frame a dataset, DJ is the jth matched detection from the same frame, |G] the number of ground truth instances in the dataset, and NFP the total number of false positives. The pPDQ be written as:
PPDQ(G! . / Os( e.). D/),
where 2s is spatial quality quantifying the quality of the bounding box predictive distribution, and Q1 is the label quality quantifying the quality of the categorical predictive distribution. Let us assume for the sake of our argument that both Qs and 2z are proper scoring rules. Different combinations of Qs and Qr can lead to the same value of pPDQ. An extreme example would be if 2s O, pPDQ will be regardless Qr. This would result in erroneous predictive distributions having the same of pPDQ correct ones, meaning that PDQ cannot correctly rank probabilistic object detectors.
A more substantial problem with PDQ relates to the spatial quality Cs. which empirically show to not a proper scoring rule. To do so, we will generate a toy example consisting of one ground truth bounding box and three corresponding predictive probability distributions. The ground truth bounding box is defined using the top left and bottom right corners: (umin: Umin). (Umaxx Umax). ) We will refer the first predictive distribution case 1, and will assign it the following mean
17

---

Published as a conference paper at ICLR 2021
M41 (Mmin + 15, Umin + 15), (umax 15, Umax 15), which is the ground truth bounding box shrunken by 15 pixels on the two image axes. The second predictive distribution, case 2 will have
M2 (Umin 15, Umin 15), (ummax + 15, Umax + 15), which is the ground truth bounding box expanded by 15 pixels on the two image axes. The final predictive distribution, case 3, will also be
assigned the ground truth bounding box expanded with 14 pixels as: μ33 (Umiin 14, Umin 14),
(Umax + 14, Umax + 14). All three predictive distributions will be assigned an identical 4 x 4 isotropic covariance matrix o² I with o² 50. The three distributions, along with the ground truth bounding box, can be visually seen in Figure
Our argument is simple: if 2s proper scoring rule, it should rank the three predictive distri- butions in an identical manner to NLL and ES. Table 2 shows the NLL, ES, and PDQ results of evaluating the three predictive distributions using the ground truth box sample. The first thing to note is that the NLL. and ES values for cases and 2 are identical. This is because both distributions share an identical covariance matrix, as well as an identical error with the ground truth bounding box (15 pixels in each image axis). However, the value of Qs for case 1 is around 6% higher than that of case 2. This phenomenon is due to the ad-hoc design choices used to define 23 in Hall et al. 20200) particularly the way Qs is built from "foreground" and "background" components. A more definitive proof that 2s is not proper stems from a comparison between the results of cases and in Table to the ground truth, 1pizel less than the error of case 1. The lower error translates to a lower NLL., as well as ES values for case 3 when compared case 1, implying that the predictive distribution of case 3 is of higher quality than that of case 1. However, when using Qs for comparison between the two cases, case has a 2% better value when compared case 3. In short, 2s can provide higher rank to lower quality predictive distributions when compared to higher quality ones (as measured by proper scoring rules such as NLL and ES), and as such is not a proper scoring rule.
and 3). Case has error of 14 pixels in every image axis when compared
In addition to not being proper scoring rule, pPDQ is computed on a single output partition, con- structed by using optimal assignment to match each ground truth object to a single detection output. This prevents in-depth analysis on the sources of errors such as the one presented in Section 4 Fi- nally, PDQ has been previously shown to be gameable, where ad-hoc modifications of predictive distributions lead to performance gains, regardless of the theoretical soundness of such modifica- tions. Contestants in a recent probabilistic object detection challenge (CVPR 2019) that relies on PDQ for evaluation showed that replacing the categorical output probability distribution with one- hot vector representation led to strictly higher PDQ scores than any method attempting to accurately represent the data generating distribution (see Sections 4.3 in (Ammirato & Berg 2019) and 3.2 in (Wang et af.) 2019)). The gameability of PDQ highlights the importance of our suggestion to use proper scoring rules for evaluation, as practitioners are seen to be susceptible to letting go of theoretical soundness in favor of performance gains on evaluation metrics.
E MAXIMUM MEAN DISCREPANCY AND THE ENERGY DISTANCE
Maximum Mean Discrepancy (MMD) has been previously used to train generative models (Li et al. 2013 2017). where minimizing MDD can be interpreted as matching the moments of the predicted model distribution to the empirical data distribution. In this work we are concerned with the Energy Distance (Rizzo& & Székely 2016), a maximum mean discrepancy (see Seejinniio et al. (013)) that simple and efficient to estimate from distribution samples.
Given two independent random vectors f, g.g E RR with cumulative distribution function F, G re- spectively, the squared Energy Distance (ED) can be written as:
D2 D2 G) = 2 gll Ellf- I'll E g'll,
(10)
where f, f' are i.i.d samples from F, and g. g' i.i.d samples from G. Rizzo & Székely (2016) show that the energy distance satisfies all axioms of a metric, providing a measure of equality of distribu- tions and insuring that (F, G) = 0 if and only if F G. In context of deep learning, the energy distance has been previously used to parallelize texe-to-speech generative models (Gritsenko et al.) 2020), and to train generative adversarial networks (Bellemare al. 2017). It can be shown (Gneit- ing & Raftery] 2007) that Equation can be written as:
ES = El f
gll
Ell - f'll
(11)
2
18

---

Published as a conference paper at ICLR 2021
It is easy to that the energy presented in equation 11 iss the energy distance when only one sample, g. available from G.
F ADDITIONAL RESULTS
DETR ALL
Nogativ Log LiNellooo
6.00
go
2200
Moin
0.60
LO02
1.00
Operim
mp
Spenm
3pen
Test Datasot
Test Data5ct
Tost Datazet
Tast Datazet
Tist D3052
Log Ukebbood
1.00
1500
Rettabbe DNM
Redinaboe NL
Hegpetter Leg Libelhoodd
1.50
Rhoi-uhi t5
1005
1.10
20
1.00
C000
Dabaset
C3 C3 Operim
COCO
Text Datssat
C3 C5 Sporm
COCO
C3 Betoset
c5 Openim
CDCO
C3 Datsset
Osonin
COCO C3
Dataser
C3 C5
Openn
Negatiee Lop Usetbood
0.60
1502
ACMN
Fhe only
0.55
ACWN DAM
RCWN ES
Roos Spout
792
2.00
0000
C3 C5 Openim
0:35
COCO
C3
CS Openm
COCO
C3
CS Openin
COCO CI C3 <5 Ozenin
18
0OCD () C3 CS Openm
Figure F.l: Point plots showing the results of the mean squared errors, NLL, and Energy score when used to evaluate regression predictive distributions from from probabilistic detectors with DETR (Top), RetinaNet (Middle), and FasterRCNN (Bottom) backends on in-distribution (COCO), arti- ficially shifted (C1-C5), and naturally shifted (OpenIm) datasets for duplicate detections. Results of regression metrics seem have the same trend as those of true positives, while results of clas- sification metrics follow those of localization errors. This is no surprise given true positives and duplicates have the same IOU with ground truth, the only difference being duplicates having lower class probability.
19

---

Published as a conference paper at ICLR 2021
1.50
1.50
1,50
Redrustee
Retrolie
1.25
1.25
1.25
RetinaMe
Rettn
Brier
Brier Score
Brier Score
0.75
0.75
0.75
0.50
0.50
5..50
ArthoMit
NathoNit
0.00 0000
Cl
C3
C5
Openlm
0.00 COCO
C1
C3 Test Datasel
Openlm
0.00
COCO
C1
C3
C5
Openim
Test Datasel
Test Datasel
3.00
3.00
3.00
Likelihedd
Likellhod
ActioMM
Negative Log Likellhodd
RetroWe
2.50
Retivale
2.50
Aetioolle new
2.50
Recro Mee DVM
RetraMe ES
Rernolee
2.00
1.50
1.50
1.50
Negative
Neg
0.50
0.50
0.00 CoCo
C1
Test Openlm
0.00 COCO
C1
Openim
0.00 COCO
C1 C3
Openim
Test
(a) Truc Positives
(b) Localization Errors
(c) False Positives
The same trends are seen as in Figure with the expection of false positives having lower Brier score than localization emross and duplicates. We suspect that this is due to usage of multilable classification with sigmoid instead of multiclass classification with softmax in the original RetinaNet implementation.
Figure F.2: Point plots of the Brier Score and NLL for results from RetinaNet.
1.20
1.20
RCW
1.20
1.00
RCZN DVW
1.00
ACNN
1.00
HCNN
NOWN
Bner Score
Bmerr Scote
Brier Scote
0.80
0.60
0.60
0.60
0.20
0.20
0.20
nOW DOM
ROW NU.
coCo
Opentm
0.00
Opentm
0.00
C3 Test Dataset
Openlm
Test Dataset
Test Dataset
Negative 19 Likelhood
5,00
RCNN
RCHR NL.
RONN
Negative (33 Likellhood
5.00
ACWN ALL
Negative Log licellhood
5,00
PORN NLL
RCNN
4.00
4.00
4.00
2.00
2.00
2.00
1.00
1.00
1.00
coco
C1
Openim
COCO
Opentm
COCO
C3
Test Dataset
Test Dataset
Test Dataset
(a) True Positives
(b) Localization Errrs
(c) False Positives
Figure F.3: Point plots of the Brier Score and NLL for results from FasterRCNN, The same trends
are seen as in Figure
20

---

Published as a conference paper at ICLR 2021
0.40
DETR NUL
0.40
DETR DMM
0.40
DETR ES
0.36
0.36
0.36
InDi
0.32
0.32
0.32
Shifted
0.28
0.28
20.24
20.24
00.24
E..00
20.20
0.20
2016
0.16
0016
0.12
0.12
0.12
0.08
0.08
0.08
0.04
0.04
0.04
15.0
20.0
25.0 30.0
15.0 20.0
20.0
Diff. Entropy (Nats)
Diff. Entropy (Nats)
Diff. Entropy
Figure F:4: Histogram of differential entropy for false positives bounding box predictive distribu- tions on DETR. For networks trained using NLL and ES, the same trend observed as Figure The network trained with DMM is shown to be able differentiate between in and out-of- distribution false positives.
Mean Average Precision (96
DETH KLL DETH NIL CIAN))
DETR DM Crwad
DETK 1F-RCwan
DET4 1S bnservbs
30
.20
10
C3
Openlm
Test Dataset
Mean Average Precision (96)
Ratknahic Retiaaiic DMM (hull Covert
RutioatetEs Rethoated Covar
Fatralie Ratraaktt ES Eraurbka
Aec natrat MLL
nallee (Tal
20
10
Val
C3
C5
Openlm
Test Dataset
Mean Average Precision (%6)
-40
ACKM
KCWR CMM
RCHM E5
RCW ES+ Bropout RCW Ensemble
Cover)
Couari
30
Val
C3 Test Dataset
C5
Openim
Figure F.5: Bar plots showing that our probabilistic detectors implementations achieve the same level of performance their deterministic counterparts (in Red) on mean AP. On RetinaNet using NLL and full covariance matrix assumption results model with much lower mAP than original RetinaNet implementation.
21

---

Published as a conference paper at ICLR 2021
G QUALITATIVE RESULTS
(a) DETR-NLL
(b) DETR-ES
(c) DETR-DMM
Figure G.I: Quantitative results from probabilistic detectors with DETR backend. The color of bounding boxes signifies the entropy of their categorical distributions, where yellow means low en- tropy, orange means moderate entropy, and red means high entropy. We also plot the 95% confidence ellipse of bounding box corner predictive distributions. The bounding box predictive distributions for DETR-NLL detections are of much higher uncertainty than the predictive distributions of DETR- ES and DETR-DMA detections. Ground truth bounding boxes shown in green.
22

---

Published as a conference paper at ICLR 2021
(a) RCNN-NLL
(b) RCNN-ES
(c) RCNN-DMM
Figure G.2: Quantitative results from probabilistic detectors with a RCNN backend. The color of bounding boxes signifies the entropy of their categorical distributions, where yellow means low entropy, orange means moderate entropy, and red means high entropy. We also plot the 95% confidence ellipse of bounding box corner predictive distributions. It can be clearly seen that the bounding box predictive distributions for RCNN-NLL, RCNN-ES and RCNN-DMM have very sim- ilar uncertainty values. Ground truth bounding boxes are shown in green.
23


Reviews:
[{"review_id": "pqlF9cDasl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces several ways to assess probabilistic object detectors that output predictive distributions. Moreover, it presents a proper and non-local scoring rule for training such probabilistic object detectors.  The model backbone consists of standard detectors (DETR, RetinaNet, FasterRCNN) with a variance regression head and a mean bounding box regression head.  The loss functions explored in the paper are the negative log likelihood (NLL, which is local and proper), energy score (ES, which is non-local and proper), and direct moment matching (DMM, which is non-local and non-proper), so this means that nine different combinations of detectors and losses can be formed.   The assessment uses NLL, MSE, ES and Brier Score on four partitions of the detection results (based on their IOU with ground truth): true positives, duplicates, localization errors, and false positives.  The training set is COCO, where there are three testing sets: the in-distribution test set is the COCO validation set, the artificially shifted data is the COCO validation dataset modified with 18 different image corruptions (Hendrycks & Dietterich, 2019), and the naturally shifted data is the OpenImages dataset (Kuznetsova et al., 2020).\n\nThe main results found by the paper are: 1) none of the proper score functions alone enables a complete analysis of the probabilistic object detectors -- for a thorough analysis, one should use both local and non-local proper scoring functions on multiple testing partitions; 2) proper scoring functions not necessarily provide good predictive uncertainty estimates for probabilistic object detectors, and non-local scoring functions allow better calibrated and lower entropy predictive distributions than local functions; 3) training based on IOU restricts results to high IOU candidates, which can lead to unreliable bounding box predictive uncertainty estimates; and 4) variance regression outputs large predictive differential entropy when presented with out-of-distribution data.\n\nThis paper has potential, but I found it not particularly well written.  In fact I found it hard to get its main point since it touches in many aspects of probabilistic object detectors, but none of the points are really clear. There are some other issues that need to be clarified -- please see below.\n1- It is not clear from the formulation how the paper handles multiple detections\n2- In Section 3.1, there are inconsistencies in the nomenclature -- x is written as boldface and then in italics, but they mean the same thing.\n3- The experimental setup is not clear -- can you defined better \"with any corresponding ground truth bounding box\" (page 4)?\n4- From the definitions of ES and DMM, it is not clear what is done with multiple instances of objects to be detected in an image.\n5- Given the ES formulation, can we say that ES is prone to be overconfident?\n6- The proofs for the scoring functions being proper or not should be more formal.\n7- In Fig. 2, in each column the graphs should have the same range.\n8- Letters in Table 1 are too small.\n9- The definition of true positives, duplicates, localization errors, and false positives should be present.\n10- On page 6, the \"Advantages of our evaluation\" sub-section only shows an anecdotal example of the advantages of the proposed measure.  It is expected a more direct comparison with competing approaches, such as PDQ.\n\nAfter rebuttal, I raised my score from 4 to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "algJiWi46Ca", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper gives an empirical analysis of the predictive uncertainty for object detection and proposes a tool to establish and evaluate the uncertainty. The main claim of this paper is that, for the bounding box regression, non-local based algorithms have superiority in the consistency with the predictive distributions than local-based methods. The experiments are conducted on some state-of-the-art detectors and fairly demonstrate them.\n\nStrength:\n\nThe ablation studies are sufficient and insightful, especially three kinds of state-of-the-art detectors and two important datasets are explored.\nThe analysis is technical sound and considerably well written.\nThe performance gain on the transformer based detector (DETR) looks satisfactory.  \n\nWeakness:\nSection3.3 presents a toy example to show the properties of NLL and the authors claim the weakness of local-based algorithms comes from penalizing lower entropy distributions more severely than higher entropy distributions. But I think this claim is not convincing enough.\nThe effectiveness of conventional detectors (Faster RCNN and RetinanNet) is relatively limited. Specifically ES has only 0.16 mAP absolute gains over NLL.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QaZcLxgiPJs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThis paper examines the question of how to best estimate the bounding box distribution in probabilistic object detectors, specially regarding the use of non proper scoring rules and issues with the standard negative log-likelihood (NLL, where there is no ground truth for predictive variance).\n\nI believe this paper has many interesting findings that will be useful for the design and training of future probabilistic object detectors. Two main takeaways are that the NLL is not appropriate for training bounding box regressors since it produces high entropy distributions, penalizing overconfident predictions too much. The second message is that the way object detectors select bounding box regression targets with an IoU > 0.5 threshold affects uncertainty in the bounding boxes, since this prevents variations across all IoU ranges to be presented and learned by the detector.\n\n#### Reasons for Score\nThis paper provides a much needed evaluation of probabilistic object detection from a statistical point of view. Many Computer Vision researchers focus into chasing state of the art metrics (mAP, PDQ, etc) and do not put enough attention into statistical issues such as using non-proper scoring rules, which can hide details and hinder progress.\nI believe that this paper will bring a breath of fresh air to the probabilistic object detection field, from its takeaways there are clear directions to improve this kind of detectors and advance the state of the art into other than just chasing metrics. The quality of predictive uncertainty is very important for safe applications of object detection, for example in autonomous driving or robot perception.\n\n#### Pros\n- Very good evaluation, with multiple datasets (COCO, OpenImages, and corrupted variations of COCO), producing a robust results and conclusions.\n- Good selection of object detectors (DETR, Faster R-CNN, RetinaNet), including one stage and two stage detectors and the recent DETR that proposes a set-based approach to object detection.\n- Clear recommendations for future research: use the energy score for training, evaluate with multiple scoring rules, and put attention in the way how bounding box regression targets are selected, with a varied selection of targets being the best, instead of being limited to higher IoU values. We should also rethink the way probabilistic object detectors are evaluated, using proper scoring rules and multiple metrics, since the standard mAP and PDQ can hide details and differences between detectors.\n- Using the energy score even for training has a ~2% improvement in mAP over NLL which should also be noted.\n- This paper provides a more statistical view on object detection, focused on the bounding box regression problem of a probabilistic object detector, which is not that common in the literature, specially in Computer Vision.\n- The paper is well written and a pleasure to read.\n\n#### Cons\n- I believe that the energy score might be a bit problematic for object detection, since it requires sampling from a Gaussian, the loss then becomes stochastic (adding noise to the process) and sampling bounding boxes might become a bottleneck during training. This is a minor issue.\n- The paper focuses into predictive uncertainty, instead of separating epistemic and aleatoric uncertainty. This is explicitly mentioned in the paper and only a minor issue, since separating both kinds of uncertainty is difficult.\n\n#### Questions for Rebuttal Period\nCan you motivate the selection of object detectors? There are many probabilistic object detectors, including Gaussian YOLO for example, so a motivation for these specific detectors would be an improvement.\nCan you also motivate the selection of the scoring rules? Why is were the energy score and DMM selected, were other alternatives considered?\n\n#### Minor Issues\n- In Section 3.3 and Figure 1, please specify the values of p for DMM and M for the energy score (I assume it is 1000 as mentioned in the Appendix) that are used for evaluation.\n- In Figure 3, please add that these results are produced using DETR to the caption, it will be easier to read and interpret.\n- This paper has a very statistical view on the object detection problem, maybe it is worth to also take a look at per-class metrics and visualize the produced bounding boxes, some of this analysis could be added to the appendix. Aggregated metrics could also hide details in some classes.\n- I think the title could be more informative, since this work focuses into bounding regression uncertainty/variance, this could be part of the title.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IaB3dspX9Q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the predictive uncertainty estimation problem in object detection. They observe that the commonly used NLL loss leads to high entropy predictive distributions but regardless of the correctness of the output mean. Instead, they use energy score as a non-local proper scoring rule. They also propose an alternative evaluation method. \n\nStrengthens:\n1. The paper is well formulated. \n2. The authors give a thorough analysis of the experimental results. \n\nWeaknesses and Questions:\nI am confused by the main contribution of this paper. Although the authors summary their observations at the end of this paper. I didn't clearly get the differences in the analysis part. \n1. No single proper scoring rule can capture all the desirable properties of object detection seems common sense for me. As object detection is a complicated problem that includes both region classification and localization. \n2. In Table-1, what does it mean the results of DMM and ES for DETR are much better than NLL , but incremental for RetinaNet and Faster R-CNN?\n3. I cannot figure out enough differences between NLL, DMM, and ES from the analysis figures in the experimental part.\n\nI hope the authors can highlight their technical contributions during the rebuttal. I am glad to hear more high-level and straightforward statements for their contributions.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews were mixed (2 positive, 2 negative).  The main concerns were about presentation issues: unclear contribution or main point; unclear analysis of figures; missing some motivation of selecting object detectors; etc.).  On the other hand, reviewers appreciated the well-formulated paper, analysis and recommendations from the experiments; 

The author response addressed the presentation issues and added additional motivations and clarifications. All reviewers in the end recommended accept. 
Decision: Accept (Poster)