Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors

1 INTRODUCTION . Deep object detectors are being increasingly deployed as perception components in safety critical robotics and automation applications . For reliable and safe operation , subsequent tasks using detectors as sensors require meaningful predictive uncertainty estimates correlated with their outputs . As an example , overconfident incorrect predictions can lead to non-optimal decision making in planning tasks , while underconfident correct predictions can lead to under-utilizing information in sensor fusion . This paper investigates probabilistic object detectors , extensions of standard object detectors that estimate predictive distributions for output categories and bounding boxes simultaneously . This paper aims to identify the shortcomings of recent trends followed by state-of-the-art probabilistic object detectors , and provides theoretically founded solutions for identified issues . Specifically , we observe that the majority of state-of-the-art probabilistic object detectors methods ( Feng et al. , 2018a ; Le et al. , 2018 ; Feng et al. , 2018b ; He et al. , 2019 ; Kraus & Dietmayer , 2019 ; Meyer et al. , 2019 ; Choi et al. , 2019 ; Feng et al. , 2020 ; He & Wang , 2020 ; Harakeh et al. , 2020 ; Lee et al. , 2020 ) build on deterministic object detection backends to estimate bounding box predictive distributions by modifying such backends with variance networks ( Detlefsen et al. , 2019 ) . The mean and variance of bounding box predictive distributions estimated using variance networks are then learnt using negative log likelihood ( NLL ) . It is also common for these methods to use non-proper scoring rules such as the mean Average Precision ( mAP ) when evaluating the quality of their output predictive distributions . Pitfalls of NLL We show that under standard training procedures used by common object detectors , using NLL as a minimization objective results in variance networks that output high entropy distributions regardless of the correctness of an output bounding box . We address this issue by using the Energy Score ( Gneiting & Raftery , 2007 ) , a distance-sensitive proper scoring rule based on energy statistics ( Székely & Rizzo , 2013 ) , as an alternative for training variance networks . We show that predictive distributions learnt with the energy score are lower entropy , better calibrated , and of higher quality when evaluated using proper scoring rules . Pitfalls of Evaluation We address the widespread use of non-proper scoring rules for evaluating probabilistic object detectors by providing evaluation tools based on well established proper scoring rules ( Gneiting & Raftery , 2007 ) that are only minimized if the estimated predictive distribution is equal to the true target distribution , for both classification and regression . Using the proposed tools , we benchmark probabilistic extensions of three common object detection architectures on in-distribution , shifted , and out-of-distribution data . Our results show that variance networks can differentiate between in-distribution , shifted , and out-of-distribution data using their predictive entropy . We find that ad-hoc approaches used by seminal object detectors for choosing their regression targets during training do not provide a wide enough data support for reliable learning in variance networks . Finally , we provide clear recommendations in Sec . 5 to avoid the pitfalls described above . 2 RELATED WORK . Estimating predictive distributions with deep neural networks has long been a topic of interest for the research community . Bayesian Neural Networks ( BNNs ) ( MacKay , 1992 ) quantify predictive uncertainty by approximating a posterior distribution over a set of network parameters given a predefined prior distribution . Variance networks ( Nix & Weigend , 1994 ) capture predictive uncertainty ( Kendall & Gal , 2017 ) by estimating the mean and variance of every output through separate neural network branches , and are usually trained using maximum likelihood estimation ( Detlefsen et al. , 2019 ) . Deep ensembles ( Lakshminarayanan et al. , 2017 ) train multiple copies of the variance networks from different network initializations to estimate predictive distributions from output sample sets . Monte Carlo ( MC ) Dropout ( Gal & Ghahramani , 2016 ) provides predictive uncertainty estimates based on output samples generated by activating dropout layers at test time . We refer the reader to the work of Detlefsen et al . ( 2019 ) for an in depth comparison of the performance of variance networks , BNNs , Ensembles , and MC dropout on regression tasks . We find variance networks to be the most scalable of these methods for the object detection task . Finally , we do not distinguish between aleatoric and epistemic uncertainty as is done in Kendall & Gal ( 2017 ) , instead focusing on predictive uncertainty ( Detlefsen et al. , 2019 ) which reflects both types . State-of-the-art probabilistic object detectors model predictive uncertainty by adapting the work of Kendall & Gal ( 2017 ) to state-of-the-art object detectors . Standard detectors are extended with a variance network , usually referred to as the variance regression head , alongside the mean bounding box regression head and the resulting network is trained using NLL ( Feng et al. , 2018a ; Le et al. , 2018 ; He et al. , 2019 ; Lee et al. , 2020 ; Feng et al. , 2020 ; He & Wang , 2020 ) . Some approaches combine the variance networks with dropout ( Feng et al. , 2018b ; Kraus & Dietmayer , 2019 ) and use Monte Carlo sampling at test time . Others ( Meyer et al. , 2019 ; Choi et al. , 2019 ; Harakeh et al. , 2020 ) make use of the output predicted variance by modifying the non-maximum suppression postprocessing stage . Such modifications are orthogonal to the scope of the paper . It is important to note that a substantial portion of existing probabilistic object detectors focus on non-proper scoring metrics such as the mAP and calibration errors to evaluate the quality of their predictive distributions . More recent methods ( Harakeh et al. , 2020 ; He & Wang , 2020 ) use the probability-based detection quality ( PDQ ) proposed by Hall et al . ( 2020 ) for evaluating probabilistic object detectors , which can also be shown to be non-proper ( See appendix D.3 ) . Instead , we combine the error decomposition proposed by Hoiem et al . ( 2012 ) with well established proper scoring rules ( Gneiting & Raftery , 2007 ) to evaluate probabilistic object detectors . 3 LEARNING BOUNDING BOX DISTRIBUTIONS WITH PROPER SCORING RULES . 3.1 NOTATION AND PROBLEM FORMULATION . Let x ∈ Rm be a set of m-dimensional features , y ∈ { 1 , . . . , K } be classification labels for K-way classification , and z ∈ Rd be bounding box regression targets associated with object instances in the scene . Given a training dataset D = { ( xn , yn , zn ) } Nn=1 of N i.i.d samples from a true joint conditional probability distribution p∗ ( y , z|x ) = p∗ ( z|y , x ) p∗ ( y|x ) , we use neural networks with parameter vector θ to model pθ ( z|y , x ) and pθ ( y|x ) . pθ ( z|y , x ) is fixed to be a multivariate Gaussian distributionN ( µ ( x , θ ) , Σ ( x , θ ) ) , and pθ ( y|x ) as a categorical distribution Cat ( p1 ( x , θ ) , . . . , pK ( x , θ ) ) . Unless mentioned otherwise , z ∈ R4 and is represented as ( umin , vmin , umax , vmax ) where the ( umin , vmin ) , ( umax , vmax ) are the pixel coordinates of the top-left and bottom-right bounding box corners respectively . Throughout this work , we denote random variables with bold characters , and the associated ground truth instances of random variables that are realized in the dataset D are italicized . 3.2 PROPER SCORING RULES . Let a be either y or z . A scoring rule is a function S ( pθ , ( a , x ) ) that assigns a numerical value to the quality of the predictive distribution pθ ( a|x ) given the actual event that materialized a|x ∼ p∗ ( a|x ) , where a lower value indicates better quality . With slight abuse of notation , let S ( pθ , p∗ ) also refer to the expected value of S ( pθ , . ) , then a scoring rule is said to be proper if S ( p∗ , p∗ ) ≤ S ( pθ , p∗ ) , with equality if and only if pθ = p∗ , meaning that the actual data generating distribution is assigned the lowest possible score value ( Gneiting & Raftery , 2007 ) . We provide a more formal definition of proper scoring rules in Appendix D.2 . Beyond the notion of proper , scoring rules can be further divided into local and non-local rules . Local scoring rules evaluate a predictive distribution based on its value only at the true target , whereas non-local rules take into account other characteristics of the predictive distribution . As an example , distance-sensitive non-local proper scoring rules reward predictive distributions that assign probability mass to the vicinity of the true target , even if not exactly at that target . Lakshminarayanan et al . ( 2017 ) noted the utility of using proper scoring rules as neural network loss functions for learning predictive distributions . Predictive uncertainty for classification tasks has been extensively studied in recent literature ( Ovadia et al. , 2019 ; Ashukha et al. , 2020 ) . We find commonly used proper scoring rules such as NLL and the Brier score ( Brier , 1950 ) to be satisfactory to learn and evaluate categorical predictive distributions for probabilistic object detectors . On the other hand , regression tasks have overwhelmingly relied on a single proper scoring rule , the negative log likelihood ( Kendall & Gal , 2017 ; Lakshminarayanan et al. , 2017 ; Detlefsen et al. , 2019 ) . NLL is a local scoring rule and should be satisfactory if used to evaluate pure inference problems ( Bernardo , 1979 ) . In addition , the choice of a proper scoring rule should not matter as asymptotically , the true parameters of pθ should be recovered by minimizing any proper scoring rule ( Gneiting & Raftery , 2007 ) during training . Unfortunately , object detection does not conform to these idealized assumptions , we show in the next section the pitfalls of using NLL for learning and evaluating bounding box predictive distributions . We also explore the Energy Score ( Gneiting & Raftery , 2007 ) , a proper and non-local scoring rule as an alternative for learning and evaluating multivariate Gaussian predictive distributions . 3.3 NEGATIVE LOG LIKELIHOOD AS A SCORING RULE . For a multivariate Gaussian , the NLL can be written as : NLL = 1 2N N∑ n=1 ( zn − µ ( xn , θ ) ) ᵀΣ ( xn , θ ) −1 ( zn − µ ( xn , θ ) ) + log det Σ ( xn , θ ) , ( 1 ) whereN is the size of the datasetD . NLL is the only proper scoring rule that is also local ( Bernardo , 1979 ) , with higher values implying a worse predictive density quality at the true target value . When used as a loss to minimize , the first term of NLL encourages increasing the entropy of the predictive distribution as the mean estimate diverges from the true target value . The log determinant regularization term has a contrasting effect , penalizing high entropy distributions and preventing a zero loss from infinitely high uncertainty predictions at all data points ( Kendall & Gal , 2017 ) . It has been shown by Machete ( 2013 ) that NLL prefers predictive densities that are less informative , penalizing overconfidence even when the probability mass is concentrated on a likely outcome . We demonstrate the relevance of this property to object detection with a simple toy example . Bounding box results from the state-of-the-art object detector DETR ( Carion et al. , 2020 ) , trained to achieve a competitive mAP of 42 % on the COCO validation split , are assigned a single mock multivariate Gaussian probability distributionN ( µ ( xn , θ ) , σI ) , where σI is a 4×4 isotropic covariance matrix with σ as a variable parameter . We split the detection results into a high error set with an IOU < = 0.5 , and a low error set with an IOU > 0.5 , where the IOU is determined as the maximum IOU with any ground truth bounding box in the scene . We plot the value of the NLL loss of both high error and low error detection instances in Figure 1 , where NLL is estimated at values of σ between [ 10−2 , 105 ] . As expected , the variance value that minimizes NLL for low error detections is two orders of magnitude lower than for high error detections . What is more interesting is the behavior of NLL away from its minimum for both low error and high error detections . NLL is seen to penalize lower entropy distributions ( smaller values of σ ) more severely than higher entropy distributions , a property that is shown to be detrimental for training variance networks in Section 4 .

Summary: This paper gives an empirical analysis of the predictive uncertainty for object detection and proposes a tool to establish and evaluate the uncertainty. The main claim of this paper is that, for the bounding box regression, non-local based algorithms have superiority in the consistency with the predictive distributions than local-based methods. The experiments are conducted on some state-of-the-art detectors and fairly demonstrate them.