Estimating and Evaluating Regression Predictive Uncertainty in Deep Object Detectors

1 INTRODUCTION . Deep object detectors are being increasingly deployed as perception components in safety critical robotics and automation applications . For reliable and safe operation , subsequent tasks using detectors as sensors require meaningful predictive uncertainty estimates correlated with their outputs . As an example , overconfident incorrect predictions can lead to non-optimal decision making in planning tasks , while underconfident correct predictions can lead to under-utilizing information in sensor fusion . This paper investigates probabilistic object detectors , extensions of standard object detectors that estimate predictive distributions for output categories and bounding boxes simultaneously . This paper aims to identify the shortcomings of recent trends followed by state-of-the-art probabilistic object detectors , and provides theoretically founded solutions for identified issues . Specifically , we observe that the majority of state-of-the-art probabilistic object detectors methods ( Feng et al. , 2018a ; Le et al. , 2018 ; Feng et al. , 2018b ; He et al. , 2019 ; Kraus & Dietmayer , 2019 ; Meyer et al. , 2019 ; Choi et al. , 2019 ; Feng et al. , 2020 ; He & Wang , 2020 ; Harakeh et al. , 2020 ; Lee et al. , 2020 ) build on deterministic object detection backends to estimate bounding box predictive distributions by modifying such backends with variance networks ( Detlefsen et al. , 2019 ) . The mean and variance of bounding box predictive distributions estimated using variance networks are then learnt using negative log likelihood ( NLL ) . It is also common for these methods to use non-proper scoring rules such as the mean Average Precision ( mAP ) when evaluating the quality of their output predictive distributions . Pitfalls of NLL We show that under standard training procedures used by common object detectors , using NLL as a minimization objective results in variance networks that output high entropy distributions regardless of the correctness of an output bounding box . We address this issue by using the Energy Score ( Gneiting & Raftery , 2007 ) , a distance-sensitive proper scoring rule based on energy statistics ( Székely & Rizzo , 2013 ) , as an alternative for training variance networks . We show that predictive distributions learnt with the energy score are lower entropy , better calibrated , and of higher quality when evaluated using proper scoring rules . Pitfalls of Evaluation We address the widespread use of non-proper scoring rules for evaluating probabilistic object detectors by providing evaluation tools based on well established proper scoring rules ( Gneiting & Raftery , 2007 ) that are only minimized if the estimated predictive distribution is equal to the true target distribution , for both classification and regression . Using the proposed tools , we benchmark probabilistic extensions of three common object detection architectures on in-distribution , shifted , and out-of-distribution data . Our results show that variance networks can differentiate between in-distribution , shifted , and out-of-distribution data using their predictive entropy . We find that ad-hoc approaches used by seminal object detectors for choosing their regression targets during training do not provide a wide enough data support for reliable learning in variance networks . Finally , we provide clear recommendations in Sec . 5 to avoid the pitfalls described above . 2 RELATED WORK . Estimating predictive distributions with deep neural networks has long been a topic of interest for the research community . Bayesian Neural Networks ( BNNs ) ( MacKay , 1992 ) quantify predictive uncertainty by approximating a posterior distribution over a set of network parameters given a predefined prior distribution . Variance networks ( Nix & Weigend , 1994 ) capture predictive uncertainty ( Kendall & Gal , 2017 ) by estimating the mean and variance of every output through separate neural network branches , and are usually trained using maximum likelihood estimation ( Detlefsen et al. , 2019 ) . Deep ensembles ( Lakshminarayanan et al. , 2017 ) train multiple copies of the variance networks from different network initializations to estimate predictive distributions from output sample sets . Monte Carlo ( MC ) Dropout ( Gal & Ghahramani , 2016 ) provides predictive uncertainty estimates based on output samples generated by activating dropout layers at test time . We refer the reader to the work of Detlefsen et al . ( 2019 ) for an in depth comparison of the performance of variance networks , BNNs , Ensembles , and MC dropout on regression tasks . We find variance networks to be the most scalable of these methods for the object detection task . Finally , we do not distinguish between aleatoric and epistemic uncertainty as is done in Kendall & Gal ( 2017 ) , instead focusing on predictive uncertainty ( Detlefsen et al. , 2019 ) which reflects both types . State-of-the-art probabilistic object detectors model predictive uncertainty by adapting the work of Kendall & Gal ( 2017 ) to state-of-the-art object detectors . Standard detectors are extended with a variance network , usually referred to as the variance regression head , alongside the mean bounding box regression head and the resulting network is trained using NLL ( Feng et al. , 2018a ; Le et al. , 2018 ; He et al. , 2019 ; Lee et al. , 2020 ; Feng et al. , 2020 ; He & Wang , 2020 ) . Some approaches combine the variance networks with dropout ( Feng et al. , 2018b ; Kraus & Dietmayer , 2019 ) and use Monte Carlo sampling at test time . Others ( Meyer et al. , 2019 ; Choi et al. , 2019 ; Harakeh et al. , 2020 ) make use of the output predicted variance by modifying the non-maximum suppression postprocessing stage . Such modifications are orthogonal to the scope of the paper . It is important to note that a substantial portion of existing probabilistic object detectors focus on non-proper scoring metrics such as the mAP and calibration errors to evaluate the quality of their predictive distributions . More recent methods ( Harakeh et al. , 2020 ; He & Wang , 2020 ) use the probability-based detection quality ( PDQ ) proposed by Hall et al . ( 2020 ) for evaluating probabilistic object detectors , which can also be shown to be non-proper ( See appendix D.3 ) . Instead , we combine the error decomposition proposed by Hoiem et al . ( 2012 ) with well established proper scoring rules ( Gneiting & Raftery , 2007 ) to evaluate probabilistic object detectors . 3 LEARNING BOUNDING BOX DISTRIBUTIONS WITH PROPER SCORING RULES . 3.1 NOTATION AND PROBLEM FORMULATION . Let x ∈ Rm be a set of m-dimensional features , y ∈ { 1 , . . . , K } be classification labels for K-way classification , and z ∈ Rd be bounding box regression targets associated with object instances in the scene . Given a training dataset D = { ( xn , yn , zn ) } Nn=1 of N i.i.d samples from a true joint conditional probability distribution p∗ ( y , z|x ) = p∗ ( z|y , x ) p∗ ( y|x ) , we use neural networks with parameter vector θ to model pθ ( z|y , x ) and pθ ( y|x ) . pθ ( z|y , x ) is fixed to be a multivariate Gaussian distributionN ( µ ( x , θ ) , Σ ( x , θ ) ) , and pθ ( y|x ) as a categorical distribution Cat ( p1 ( x , θ ) , . . . , pK ( x , θ ) ) . Unless mentioned otherwise , z ∈ R4 and is represented as ( umin , vmin , umax , vmax ) where the ( umin , vmin ) , ( umax , vmax ) are the pixel coordinates of the top-left and bottom-right bounding box corners respectively . Throughout this work , we denote random variables with bold characters , and the associated ground truth instances of random variables that are realized in the dataset D are italicized . 3.2 PROPER SCORING RULES . Let a be either y or z . A scoring rule is a function S ( pθ , ( a , x ) ) that assigns a numerical value to the quality of the predictive distribution pθ ( a|x ) given the actual event that materialized a|x ∼ p∗ ( a|x ) , where a lower value indicates better quality . With slight abuse of notation , let S ( pθ , p∗ ) also refer to the expected value of S ( pθ , . ) , then a scoring rule is said to be proper if S ( p∗ , p∗ ) ≤ S ( pθ , p∗ ) , with equality if and only if pθ = p∗ , meaning that the actual data generating distribution is assigned the lowest possible score value ( Gneiting & Raftery , 2007 ) . We provide a more formal definition of proper scoring rules in Appendix D.2 . Beyond the notion of proper , scoring rules can be further divided into local and non-local rules . Local scoring rules evaluate a predictive distribution based on its value only at the true target , whereas non-local rules take into account other characteristics of the predictive distribution . As an example , distance-sensitive non-local proper scoring rules reward predictive distributions that assign probability mass to the vicinity of the true target , even if not exactly at that target . Lakshminarayanan et al . ( 2017 ) noted the utility of using proper scoring rules as neural network loss functions for learning predictive distributions . Predictive uncertainty for classification tasks has been extensively studied in recent literature ( Ovadia et al. , 2019 ; Ashukha et al. , 2020 ) . We find commonly used proper scoring rules such as NLL and the Brier score ( Brier , 1950 ) to be satisfactory to learn and evaluate categorical predictive distributions for probabilistic object detectors . On the other hand , regression tasks have overwhelmingly relied on a single proper scoring rule , the negative log likelihood ( Kendall & Gal , 2017 ; Lakshminarayanan et al. , 2017 ; Detlefsen et al. , 2019 ) . NLL is a local scoring rule and should be satisfactory if used to evaluate pure inference problems ( Bernardo , 1979 ) . In addition , the choice of a proper scoring rule should not matter as asymptotically , the true parameters of pθ should be recovered by minimizing any proper scoring rule ( Gneiting & Raftery , 2007 ) during training . Unfortunately , object detection does not conform to these idealized assumptions , we show in the next section the pitfalls of using NLL for learning and evaluating bounding box predictive distributions . We also explore the Energy Score ( Gneiting & Raftery , 2007 ) , a proper and non-local scoring rule as an alternative for learning and evaluating multivariate Gaussian predictive distributions . 3.3 NEGATIVE LOG LIKELIHOOD AS A SCORING RULE . For a multivariate Gaussian , the NLL can be written as : NLL = 1 2N N∑ n=1 ( zn − µ ( xn , θ ) ) ᵀΣ ( xn , θ ) −1 ( zn − µ ( xn , θ ) ) + log det Σ ( xn , θ ) , ( 1 ) whereN is the size of the datasetD . NLL is the only proper scoring rule that is also local ( Bernardo , 1979 ) , with higher values implying a worse predictive density quality at the true target value . When used as a loss to minimize , the first term of NLL encourages increasing the entropy of the predictive distribution as the mean estimate diverges from the true target value . The log determinant regularization term has a contrasting effect , penalizing high entropy distributions and preventing a zero loss from infinitely high uncertainty predictions at all data points ( Kendall & Gal , 2017 ) . It has been shown by Machete ( 2013 ) that NLL prefers predictive densities that are less informative , penalizing overconfidence even when the probability mass is concentrated on a likely outcome . We demonstrate the relevance of this property to object detection with a simple toy example . Bounding box results from the state-of-the-art object detector DETR ( Carion et al. , 2020 ) , trained to achieve a competitive mAP of 42 % on the COCO validation split , are assigned a single mock multivariate Gaussian probability distributionN ( µ ( xn , θ ) , σI ) , where σI is a 4×4 isotropic covariance matrix with σ as a variable parameter . We split the detection results into a high error set with an IOU < = 0.5 , and a low error set with an IOU > 0.5 , where the IOU is determined as the maximum IOU with any ground truth bounding box in the scene . We plot the value of the NLL loss of both high error and low error detection instances in Figure 1 , where NLL is estimated at values of σ between [ 10−2 , 105 ] . As expected , the variance value that minimizes NLL for low error detections is two orders of magnitude lower than for high error detections . What is more interesting is the behavior of NLL away from its minimum for both low error and high error detections . NLL is seen to penalize lower entropy distributions ( smaller values of σ ) more severely than higher entropy distributions , a property that is shown to be detrimental for training variance networks in Section 4 .

Summary: This paper introduces several ways to assess probabilistic object detectors that output predictive distributions. Moreover, it presents a proper and non-local scoring rule for training such probabilistic object detectors.  The model backbone consists of standard detectors (DETR, RetinaNet, FasterRCNN) with a variance regression head and a mean bounding box regression head.  The loss functions explored in the paper are the negative log likelihood (NLL, which is local and proper), energy score (ES, which is non-local and proper), and direct moment matching (DMM, which is non-local and non-proper), so this means that nine different combinations of detectors and losses can be formed.   The assessment uses NLL, MSE, ES and Brier Score on four partitions of the detection results (based on their IOU with ground truth): true positives, duplicates, localization errors, and false positives.  The training set is COCO, where there are three testing sets: the in-distribution test set is the COCO validation set, the artificially shifted data is the COCO validation dataset modified with 18 different image corruptions (Hendrycks & Dietterich, 2019), and the naturally shifted data is the OpenImages dataset (Kuznetsova et al., 2020).