MERCI: A NEW METRIC TO EVALUATE THE CORRELATION BETWEEN PREDICTIVE UNCERTAINTY AND TRUE ERROR

1 INTRODUCTION . Nowadays , we are able to teach machines a wide and impressive variety of tasks . With the outcome of deep learning these past few years , we are also capable of reaching - and even sometimes surpass - human performances , e.g . for vision tasks ( Kokkinos , 2015 ; Lu & Tang , 2015 ) . However , although deep learning allows us to attain great performance , it remains imperfect and we still lack the means to assign a confidence to the predictions of a trained network . While the robustness of deep learning algorithms to adversarial attacks has been very actively addressed in the recent literature ( see e.g . ( Papernot et al. , 2016 ) ) , the question of predicting the uncertainty of predictions made by deep neural networks has received much less attention . Yet , for countless applications it is a matter of critical importance to know whether the estimate output of the algorithm is reliable or not . In this paper we take the example of depth estimation from single monocular images ( see e.g . ( Liu et al. , 2016 ) ) . For such a task , estimation algorithms are trained to regress a depth map from an input RGB image . If we want to use such algorithms as a measurement tool , it is required to have , in addition to the predicted depth , an uncertainty map providing an estimate of the accuracy for each pixel . Uncertainty prediction is indeed a very valuable information , but it depends on the content of the images . It is likely that depth estimation on textureless regions will be less accurate , or , that dark images lead to less accurate estimations than well-exposed images . In the literature of deep learning , some recent techniques have emerged to estimate uncertainties along with the network ’ s predictions ( Gal & Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ; Kendall & Gal , 2017 ) . However , these predictive uncertainties are evaluated either by their influence on the predictions performance , or qualitatively by visually checking the uncertainty maps . This situation is in contrast with other domains such as forecasting that have established standard tools to evaluate predictive uncertainty ( Uusitalo et al. , 2015 ; Gneiting et al. , 2007 ; Gneiting & Raftery , 2007 ) . We believe that these evaluation metrics should benefit the computer vision literature to evaluate the quality of the uncertainty estimate itself . One of the contributions of this paper is a comprehensive review of the existing metrics that allow to evaluate the quality of predictive uncertainty . Another contribution is an in-depth analysis of the behavior of these metrics on regression problems , when deep convolutional networks are involved and for several current predictive uncertainty approaches . Finally , this paper proposes a novel evaluation metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to the current practice in deep learning . This metric is evaluated and compared to the reviewed ones on a toy dataset as well as in the context of monocular depth estimation from images . 2 RELATED WORK . Making uncertainty estimates along with tasks predictions is becoming more and more crucial to the field of deep learning . To this end , a common path is to transform deterministic neural networks , in which each parameter is a scalar , into probabilistic Bayesian models , where parameters are replaced by probabilty distributions . Because the inference becomes intractable , several papers rely on approximate solutions . Among the most popular techniques , we count Probabilistic BackPropagation ( PBP ) ( Hernández-Lobato & Adams , 2015 ) , Monte Carlo Dropout ( Gal & Ghahramani , 2016 ) , but also some non-Bayesian ones such as Deep ensembles ( Lakshminarayanan et al. , 2017 ) or the learned attenuation loss proposed by Kendall & Gal ( 2017 ) to capture the aleatoric uncertainty in the data . However , this literature still lacks proper metrics to evaluate the quality of the uncertainties . Indeed , systematic assessment of the uncertainty estimate itself is largely underestimated in the deep learning community . It is usually the quality of the prediction which is measured , and not the quality of the estimated uncertainty . Concerning the qualitative results , it is common to see maps of the estimated uncertainty displayed as images and visually evaluated . Yet , the literature on deep neural networks still lacks specific methods for the quantitative evaluation of the uncertainty estimates . In contrast , evaluating predictive uncertainty has been a concern for decades in other fields , leading to the development of well established metrics . The evaluating predictive uncertainty challenge ( PASCAL Challenge ) , Quinonero-Candela et al . ( 2006 ) proposed two metrics adapted to regression tasks , that are still commonly used . The normalized MSE aims at evaluating the quality of a prediction and uses the empirical variance of the test estimates to normalize the scores . This is therefore not a direct way to evaluate the predictive uncertainty . The second one , the average negative log predictive density ( NLPD ) , directly takes into account the estimated uncertainty and penalizes both over and under-confident predictions . A more in-depth treatment can be found in the forecasting literature . Uusitalo et al . ( 2015 ) draw up an overview of methods to estimate uncertainty with deterministic models such as : expert assessment , where an expert in the area rates the confidence of a method using his knowledge ; model sensitivity analysis , determining how the output response of the model change w.r.t . the inputs and model ’ s parameters ; model emulation , consisting of approximating a complex model by a lowerorder function , to then apply more easily a sensitivity analysis ; spatiotemporal variability , using the spatiotemporal variability as an estimate of the variance ; multiple models , where an ensemble of models are used to describe the same domain ; and finally data-based approaches , where enough data are available to directly assess the statistical uncertainty related to the model outputs . Gneiting et al . ( 2007 ) propose to evaluate a predictive uncertainty forecast in terms of calibration and sharpness . Calibration refers to the statistical consistency between the predictive distributions and the observations ; sharpness refers to the concentration of the predictive distributions . Several graphical tools are proposed to visually evaluate if the probabilistic forecasts are calibrated and sharp . More recently , Kuleshov et al . ( 2018 ) propose a method to recalibrate regression algorithms so that the uncertainty estimates lie in the expected confidence interval . They also propose metrics to assess both calibration and sharpness of the predictive distribution . However , they consider them separately while we were driven by the will of assessing both these properties jointly following the tracks of Gneiting et al . , using scoring rules ( Gneiting & Raftery , 2007 ) . We are therefore interested in the scoring rules , that assess the quality of a forecast , maximizing sharpness subject to calibration . However , all these metrics imply that the prediction is a probability distribution , which is seldom true in practice . Instead , existing deep learning methods usually predict uncertainty as being a scalar representing the variance of some predictions ( e.g . with Monte Carlo sampling ) . We can easily convert prediction/uncertainty pairs to a probability distribution but we therefore have to choose a prior on the distribution . We assess that within reasonable variations , this prior is not of critical importance since it does not change drastically the ranking between different uncertainty predictions , as we will see later . For these reasons , we propose a novel metric to evaluate prediction/uncertainty couples . We call our metric the Mean Rescaled Confidence Interval ( MeRCI ) . It is based on the calculation of a scaling coefficient , for the interval centered on the prediction to encompass the ground-truth observation . It needs no prior on the distribution , allows to effectively rank different methods of estimations and is parametrized to be robust to outliers . 3 COMMON PERFORMANCE METRICS FOR UNCERTAINTY PREDICTION . We assume a regression settings with a training set of N samples denoted as T = { ( xi , y∗i ) } where xi ∈ Rd is the input vector of dimension d and y∗i ∈ R the scalar value to be predicted . In a standard deterministic regression framework , the regressor is a function defined as fθ : Rd → R , where fθ ( x ) is an estimator of the true function f ( x ) , and ŷ = fθ ( x ) is the prediction made by the model . Learning such a regressor consists in finding the parameters θ minimizing an empirical loss defined by 1N ∑N i=1 L ( fθ ( xi ) , y ∗ i ) where L ( · , · ) accounts for the proximity of its two arguments ( e.g . a squared difference ) . In a probabilistic framework , we predict a whole distribution Py instead of a deterministic prediction . This distribution has a probability density function py ( x ) and a cumulative distribution Fy . In practice , Py is often characterized by its mean value , which is usually used as the prediction ŷ , and by its standard deviation σ . Different approaches to perform such kind of predictions shall be reviewed in section 5 . In the literature of forecasting , we can find several metrics for assessing the quality of a predictive distribution , or forecast . We present here the five most popular ones . All of them are computed as an empirical expectation : 1 N N∑ i=1 S ( Pyi , y ∗ i ) ( 1 ) where the function S relates to the harmony between the prediction Py and the actual value y∗ . Depending on the variant under consideration , it may account for sharpness , calibration or both . In the following paragraphs we will refer to S as the adequacy function . 3.1 COVERAGE . The coverage is defined for a fixed confidence level α ( e.g . 95 % ) as the proportion of observations that actually belong to the associated confidence interval . Denoting CIα ( Pyi ) the confidence interval of level α for the forecast , then the adequacy function is just the indicator function of the confidence interval : Scov ( Pyi , y ∗ i ) = 1 { y∗i ∈CIα ( Pyi ) } ( 2 ) The coverage is minimum ( 0 % ) if none of the values are in the corresponding confidence interval , and can reach 100 % if all the observations are in the interval . As a result , the coverage only accounts for the calibration , and can be arbitrarily improved by making non-sharp predictions ( in other words by being pessimistic on purpose ) . 3.2 STRICTLY PROPER SCORING RULES . In the forecast literature , adequacy functions that are positively oriented are known as scoring rules ( Gneiting & Raftery , 2007 ) . A scoring rule is said strictly proper if in average , it singles out the true distribution as the optimal forecast in the following sense : EY∼P∗y [ S ( P ∗ y , Y ) ] < EY∼P∗y [ S ( Py , Y ) ] ( 3 ) for any forecast Py different from the ground-truth distribution P ∗y . Three common and popular strictly proper scoring rules are : SQuadratic ( Py , y ∗ ) = 2py ( y ∗ ) − ∫ +∞ −∞ p2y ( x ) dx ( 4 ) SLogarithmic ( Py , y ∗ ) = log ( py ( y ∗ ) ) ( 5 ) SSpherical ( Py , y ∗ ) = py ( y ∗ ) √∫ +∞ −∞ p 2 y ( x ) dx ( 6 )

Summary: The authors propose mean rescaled confidence interval (MERCI) as a way to measure the quality of predictive uncertainty for regression problems. The main idea is to rescale confidence intervals, and use average width of the confidence intervals as a measure for calibration. Due to the rescaling, the MERCI score is insensitive to the absolute scale; while this could be a feature in some cases, it can also be problematic in applications where the absolute scale of uncertainty matters.