Deep Evidential Uncertainty
Authors: Alexander Amini
Wilko Schwarting
Ava Soleimany
Daniela Rus
Venue:  (2020)
Keywords: Evidential deep learning
Uncertainty estimation
Epistemic uncertainty

Abstract: Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust and efficient measures of uncertainty are crucial. While it is possible to train regression networks to output the parameters of a probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying confidence of its predictions. In this paper, we propose a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target. We accomplish this by  placing evidential priors over our original Gaussian likelihood function and training our NN to infer the hyperparameters of our evidential distribution. We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output. Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters. We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.


Under review as a conference paper at ICLR 2020
DEEP EVIDENTIAL REGRESSION
Anonymous authors
Paper under double-blind review
ABSTRACT
Deterministic neural networks (NNs) increasingly being deployed in safety critical domains, where calibrated, robust and efficient measures of uncertainty are crucial. While is possible to train regression networks output the parameters of probability distribution by maximizing a Gaussian likelihood function, the resulting model remains oblivious to the underlying confidence of its predictions. In this paper, we propose novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target. We accomplish this by placing evidential priors over our original Gaussian likelihood function and training our NN to infer the hyperparameters evidential distribution. We impose priors during training such that the model is penalized when its predicted evidence is not aligned with the correct output. Thus the model estimates not only the probabilistic mean and variance of our target but also the underlying uncertainty associated with each of those parameters. We observe that our evidential regression method learns well-calibrated measures of uncertainty on various benchmarks, scales to complex computer vision tasks, and is robust to adversarial input perturbations.
INTRODUCTION
Recent advances in deep supervised learning have yielded super human level performance and precision. While these models empirically generalize well when placed into new test enviormeents, they are often easily fooled by adver- sarial perturbations (Goodfellow et al. 2014), and have difficulty understanding when their predictions should not be trusted. Today, regression based neural networks (NNs) are being deployed in safety critical domains of computer vision (Godard et al., 2017) well as in robotics and control (Bojarski et al. 2016) where the ability to infer model uncertainty crucial for eventual wide-scale adop- tion. Furthermore, precise uncertainty estimates are useful both for human interpretation of confidence and anomaly detection, and also for propagating these estimates to other autonomous components of larger, connected system.
Likellhood Function
Evidential Distribution
R Ma
y N~~N(,
Popppii (or,8)
Figure 1: Evidential distributions. Maxi- mum likelihood optimization learns likeli- hood distribution given data, while evidential distributions model higher-order probability distribution over the likelihood parameters.
Existing approaches to uncertainty estimation are roughly split into two categories: (1) learning aleatoric uncertainty (uncertainty in the data) and (2) epistemic uncertainty (uncertainty in the prediction). While representations for aleatoric uncertainty can be learned directly from data, approaches for estimating epistemic uncertainty focus on placing probabilistic priors over the weights and sampling to obtain measure of variance. In practice, many challenges arise with this approach, such as the computational expense sampling during inference, how to pick appropriate weight prior, or even how learn such a representation given your prior.
Instead, we formulate learning as an evidence acquisition process, where the model can acquire evidence during training in support of its prediction (Sensoy al. 2018 Malinin & Gales 2018). Every training example adds support to a learned higher-order, evidential distribution. Sampling from this distribution yields instances of loder-order, likelihood functions from which the data was drawn (cf. Fig. D. We demonstrate that, by placing priors over our likelihood function, we can learn grounded representation of epistemic and aleatoric uncertainty without sampling during inference.

---

Under review as a conference paper at ICLR 2020
In summary, this work makes the following contributions:
1. A novel and scalable method for learning representations of epistemic and aleatoric uncer- tainty, specifically on regression problems, by placing evidential priors over the likelihood;
2. Formulation of novel evidential regularizer for continuous regression problems, which we show is necessary for expressing lack of evidence on out-of-distriben
examples; 3. Evaluation of learned epistemic uncertainty on benchmark regression tasks and comparison against other state-of-the-an uncertainty estimation techniques for neural networks; and
4. Robustness evaluation against out of distribution and adversarially perturbed test data.
MODELLING UNCERTAINTIES FROM DATA
2.1 PRELIMINARIES
Consider the following supervised optimization problem: given a dataset, D, of N paired training examples, (æ1, .1) ........ yN), aim to learn function ʃ. parameterized by set of weights, w, which approximately solves the following optimization problem:
N
min J(w); J(w) = N 40
Σ (w)),
(1)
where (+(+) describes loss function. In this work, consider deterministic regression problems, which commonly optimize the sum of squared errors, Li(w) llys [x,;; w²)l In doing so, the model is encouraged to learn the average correct answer for a given input. but does not explicitly model any underlying noise or uncertainty in the data when making its estimation.
2.2
MAXIMUM LIKELIHOOD ESTIMATION
We can also approach our optimization problem from maximum likelihood perspective, where learn model parameters that maximize the likelihood of observing a particular set of training data. In the context of deterministic regression, distribution with mean and variance parameters 0 = (Jr, o²). In maximum likelihood estimation, we aim learn model infer 0 (j4, o²) that maximize the likelihood of observing our targets, y. given by p(y (0). In practice, we minimize the negative log likelihood by setting:
assume targets, yı, were drawn i.idd from Gaussian
log + (y/ - -²)²
(.(w) = - log p(y H. o
202
In learning the parameters 0, this likelihood function allows successfully model the uncertainty of our data, also known as the aleatoric uncertainty. However, our model remains oblivious to the predictive model or epistemic uncertainty (Kendall & Gall 2017).
In this paper, we present novel approach for estimating the evidence in support of network predic- tions by directly learning both the inferred aleatoric uncertainty well the underlying epistemic uncertainty over its predictions. We achieve this by placing higher-order prior distributions over the learned parameters governing the distribution from which our observations drawn.
3 EVIDENTIAL UNCERTAINTY FOR REGRESSION
3.1 PROBLEM SETUP
We consider the problem where our observed targets, yi. are drawn i.i.d. from a Gaussian distribution now with wnknown mean and variance (p, o²), which we seek to probabilistically estimate. We model this by placing a conjugate prior distribution on (n, o²). we assume our observations are drawn from Gaussian, this leads to placing Gaussian prior our unknown mean and an Inverse-Gamma prior on our unknown variance:
(V...., ,9N) ~N(,003
/ x~)
o² (a,
2

---

Under review as a conference paper at ICLR 2020
Higher Order (Evidential): p(ys 1 oy,,,, 0) = p(0(m)
Lower Order (Likelihood): Kiylu,a") p(y(0)
Bayesian Plate Notation
B
Incressing Evvieecce
KPM
Normal Inverse-Gamma distribution. Different realizations of evidential distribution (A) correspond to different levels confidences in the parameters (e.g. μ,, ,2, Sampling from a single realization of a higher-orde order evidential distribution (B). yields lower-order likelihoods (C) over the data (e.g. plylu, o o²)). Darker shading indicates higher probability mass. We aim to learn model (D) that predicts the target, y, from an input, with evidential prior imposed on our likelihond enable uncertainty estimation.
Figure 2:
where T(.) is the gamma function, m = 0, and 7 E R 0, 0.
Our aim is to estimate posterior distribution ( o²) p(u, 1y11y,...... ..... UNN) To obtain approx- imation for the true posterior, we assume that the estimated distribution can be factorized (Parisi
1988) such that q(u, o²) q(u) q(a²). Thus, our approximation takes the form of the Gaussian conjugate prior, the Normal Inverse-Gamme (N.I.G.) distribution:
8 V
1
a+1
{
23 + X(T 11)-2)² 1x2
p(u, 71 2.
exp
(3)
F(a)V2mm 2ma2
202
popular interpretation of the parameters of the conjugate prior distribution is in terms of "virtual- observations" in support of given property (Jordan 2009). For example, the mean of N.I.G. distribution can be interpreted as being estimated from À virtuallobservations with sample mean y while its variance was estimated from 2a vibtual-obserrattions with sample mean y and sum of squared deviations 283. Following from this interpretation, we define the total evidence, 4 of our evidential distributions as the sum of all inferred virtual-observations counts: (# x 2a).
Drawing a sample 0j from the N.I.G. distribution yields single instance of our likelihood function, namely ((,,, dz). Thus, the N.L.G. hyperparameters, (7, A, at 3), determine not only the location but also the dispersion concentrations, uncertainty, associated with inferred likelihood function. Therefore, we can interpret the N.I.G. distribution as higher-onder, evidential, distribution on top of the unknown lower-order likelihood distribution from which observations drawn.
For example, in Fig. A we visualize different evidential N.I.G. distributions with varying model parameters. We illustrate that by increasing the evidential parameters (i.e. X, o)) of this distribution, the p.d.f. becomes tightly concentrated about its inferred likelihood function. Considering a single parameter realization of this higher-order distribution, cf. Fig. 2B, we can subsequently sample many lower-order realizations of our likelihood function, as shown in Fig. ZC.
In this work, we use neural networks infer the hyperparameters of this higher-order, evidential distribution, given an input. This approach presents several distinct advantages compared to prior work. First, our method enables simultaneous learning of the desired regression task, along with aleatoric and epistemic uncertainty estimation, built in, by enforcing evidential priors. Second, since the evidential prior is a higher-order N.I.G. distribution, the maximum likelihood Gaussian can be computed analytically from the expected values of the (n, a²) parameters, without the need for sampling. Third, we can effectively estimate the epistemic or model uncertainty associated with the network's prediction by simply evaluating the variance of our inferred evidential distribution.
3.2 LEARNING THE EVIDENTIAL DISTRIBUTION
Having formalized the use of an evidential distribution to capture both aleatoric and epistemic uncer- tainty, we next describe approach for learning model (c.f. Fig. 2)) to output the hyperparameters
3

---

Under review as a conference paper at ICLR 2020
of this distribution. For clarity, we will structure the learning objective into two distinct parts: (1) acquiring or maximizing model evidence in support of our observations and (2) minimizing evidence or inflating uncertainty when the prediction is wrong. At high level, we can think of (1) way of fitring our data to the evidential model while (2) enforces a prior to inflate our uncertainty estimates.
(1) Maximizing the model fit.
From Bayesian probability theory, the "model evidence", or marginal likelihood, is defined as the likelihood of observation, Vs. given the evidential distribution parameters m and is computed by marginalizing over the likelihood parameters 0:
p(u (m) p(u,0,, m) m))(0mm))
J. plus m)) do.
(4)
p(0 yi: m)
The model evidence is not, in general, straightforward to evaluate since computing it involves integrating out the dependence on latent model parameters:
[ plus(m)) =2==
pluilu, p') pm) du
(S)
However, by placing a N.I.G. evidential prior on our Gaussian likelihood function an analytical solution for the model evidence does exist. For computational reasons, minimize the negative logarithm of the model evidence (CNLL((W) For a complete derivation please refer to Sec. 7.1
X(7 y:) 23 +
CNLL(W)) pogp((lln = log 2
ažtaga ga
(6)
2 + X)
1
Instead of modeling this loss using empirical Bayes, where the objective is to maximize model evidence, we alternatively can minimize the sum-of-squared (SOS) errors, between the evidential prior and the data that would be sampled from the associated likelihood. Thus, we define 2:50s (w) as
CSOS (w) Egg mp(0mm)) Ey' [Ile' vill_]] [
Byinpiyluo [lVs' vvil2] p(u, mm) du
(8)
02=0 Nc-0X I'a- 1) (lT(a)>b
(23(1 () (2xx 1)A(1i - -))
A step-by-step derivation is given in Sec. 7.1 In our experiments, using (500(w)) resulted in greater training stability and increased performance, compared to the (N)) loss. Therefore, CS0S (w) is used in all presented results.
(2) Minimizing evidence on errors.
In the first term of objective above, outlined loss function for training NN output parameters of a N.I.G. distribution to fit our observations, either by maximizing the model evidence or minimizing the sum-of-squared errors. Now, we describe how to regularize training by applying lack of evidence prior (i.e., maximum uncertainty). Therefore, during training we aim to minimize our evidence (or maximize our uncertainty) everywhere except where we have training data.
This can be done by minimizing the KL-divergence between the inferred posterior, q(0), and prior, p(0). This has been demonstrated with success in the categorical setting where the uncertainty prior can be set to a uniform Dirichlet (Malinin & Gales] 2018 Sensoy et al. 2018). In the regression setting, the -i-diiergee between our posterior and N.I.G. zero evidence prior (i.e., {a, A) 0) is not well defined (Soch & Allefeld 2016), please refer to Sec. 7.2 for a derivation. Furthermore, this prior needs to be enforced specifically where there is no support from the data. Past works in classification accomplish this by using the ground truth likelihoood classification (i.e., the one-hot encoded labels) to remove the nonmiisleading evidence. However, in regression, labels are provided point targets (not ground truth Gaussian likelihoods). Unlike classification, it is not possible to penalize evidence everywhere except our single point estimate, as this space is infinite and unbounded. Thus, these previously explored approaches for evidential optimization not directly applicable.
4

---

Under review as a conference paper at ICLR 2020
To address both of these shortcomings of past works, now in the regression setting, formulate novel evidence regularizer, L based on the error of the 2-th prediction,
C (w) 1 E| + |1 >11, (2a + A),
(10)
where |a||p represents the L-p norm of z. The value of p impacts the penalty imposed the evidence when a wrong prediction is made. For example, p 2, heavily over-penalizes the evidence on larger errors, whereas p and p 0.5 saturate the evidence penalty for larger errors. We found that p = provided the optimal stability during training and use this value all presented results.
This regularization loss imposes penalty whenever there is error in the prediction that scales with the total evidence of our inferred posterior. Conversely, large amounts of predicted evidence will not be penalized as long as the prediction is close the target observation. We provide ablation analysis to quantitatively demonstrate the added value of this evidential regularizer in See 7.3.2
The combined loss function employed during training consists of the two loss terms for maximizing model evidence and regularizing evidence,
Li(w)= C. (w)).
(11)
3.3 EVALUATING ALEATORIC AND EPISTEMIC UNCERTAINTY
The aleatoric uncertainty, also referred to statistical or data uncertainty, is representative of unknowns that differ each time we run the same experiment. We evaluate the aleatoric uncertainty from E[o²] a-I- The epistemic, also known as the model uncertainty, describes the estimated uncertainty in the learned model and defined as Varlu] و Note that Var[u] = B[a²)/A, which is expected as is one of our two evidential virtual-observation counts.
4
EXPERIMENTS
4.1 PREDICTIVE ACCURACY AND UNCERTAINTY BENCHMARKING
We first qualitatively compare the performance of our approach against set of benchmarks on a one-dimensional toy regression dataset (Fig. 33. For training and dataset details please refer to Sec. 1 We compare deterministic regression, as well as techniques using empiri- cal variance of the networks' predictions such as MC-dropout, ensellens,,bble, and Bayes-by Backprop which underestimate the uncertainty outside the training distribution. In contrast, ev- idential regression estimates uncertainty appro- priately and grows the uncertainty estimate with increasing distance from the training data.
Regression
Regression (Oun)
Legand
Duta
Grundd Truth
Prediction
Epoitmicc Uncertatrcy
Bayes: by Backprop
Dropout Sampling ling
Model
Figure 3: Epistemic uncertainty estimation. Model- ing the supportive evidence during lcarning enables pre- cise prediction within the training regime and conserva- tive uncertainty estimates where there data. Comparisons other epistemic uncertainty esti- mation methods are illustrated (bottom).
Additionally, we compare our approach to state- of-the-art methods for predictive uncertainty estimation using NNs on common real world datasets used in (Hernández-Lobaae
training
Adams, 2015 Lakshminarayanan et al. [017]] Gal & Ghahramani 2016). We evaluate our proposed evidential regression method against model-ensembles and BBB based on root mean squared error (RMSE), and negative log- likelihood (NLL). We do not provide results for MC-dropout since it consistently performed inferior to the other baselines. The results in Table T indicate that although the loss function for evidential regression is more complex than competing approaches, it is the top performer in RMSE and NLL in out datasets.
Furthermore, demonstrate that, synthetic dataset with priori known noise, evidential models can additionally estimate and recover the underlying aleatoric uncertainty. For more information please refer Sec. 7.3.3 for results and experiment details.
5

---

Under review as a conference paper at ICLR 2020
RMSE
NLL
Dataset
Ensembles
BBB
Evidential
Ensembles
BBB
Evidential
Boston
0.09 1: 4.3e.4
0.09 +: 3.7e-4
0.09
1.0c-6
-0.89 1: 6.5e-2
40.67 1: 1.5e-2
+0.87 :: 2.2e-2
Concrete
0.07 4,4c-3
0.06 +: 3,3e-6
0.06
7.0c-7
-1.29 1 4.1e-2
-1.32 4.3e-3
-1.31 :: 1.9c-2
Energy
0.10 2.3e-4
0.10 + 1.6e-S
0.10 = 9.0k-7
-0.61 +: 8.9c-2
-0.60
-0.75
: 1.4e22 144e2
Kin8nm
0.07 3.5e.4
0.17 +: 3.5e-4
0.08 3.8e-3
-0.78 +: 1
6.3e-
-1.17 1: 2.6e-2
Naval
0.01 1.0c-7
0.04 +: 1.2c-2
0.01 3.4e-4
-2.55 3.36-2
-1.83
-3.17 1: 2.1e-3
Power
0.06 4.0e-7
0,06 + 2.3e-6
0.06 5.3e-6
-1.29 1: 6.9e-2
-
-1.40 : 6,2e-3
Protein
0.17 = 1.0e-6
0.17 = 8.0c-4
0.17 = 1.6e-6
-0.27 - 6.7e-2
0.32
-0.29 + 1.le2
Wine
0.10 3.0e-4
0.10 + 2.9e-4
0.10 3.8e-5
-0.46
+: 2.5e-1
-0.89 244-3
-0.85
+ 6.9c-3
Yacht
0.07 1.3c-3
0.07 3.4e-3
0.06 6.2e-5
-1.16 6.30-2
4.774
-1.28 : 9.4e3
Table I: Benchmark regression tests. We evaluate RMSE and negative log-likelihood (NLL) for model ensembling (Lakshminarayanan et al. 2017). Bryees-y-Backpree (BBB) (Blundell et al. 2013) and evidential regression. Evidential achieves top scores (bolded, within statistical significance) of the 9 datasets.
4.2 DEPTH ESTIMATION
After establishing benchmark comparison results, in this subsection we demonstrate the scalability of our evidential learning by extending to the complex, high-dimensional task of depth estimation. Monocular end-to-end depth estimation a central problem in computer vision which aims to learn a representation of depth directly from RGB image of the scene. This is a challenging learning task since the output target y is very high-dimensional. For every pixel in the image, we regress over the desired depth and simultaneously estimate the uncertainty associated to that individual pixel.
Our training data consists of over 27k RGB-to -eetth-depth pairs of indoor scenes (e.g. kitchen, bedroom, etc.) from the NYU Depth v2 dataset (Nathan Silberman & Fergus 2012). We train U-Net style NN (Ronneberger et al. 2015) for inference. The final layer of our model outputs a single H x W activation map in the case of deterministic regression, dropout, ensembling and BBB. Evidential models output four final activation maps, corresponding to (7, X, a).
Table 2 summarizes the size and speed of all models. Evidential models contain significantly fewer trainable parameters than ensembles (where the number of parameters scales linearly with the size of the ensemble). BBB maintains trainable mean and variance for every weight in the network, so its size is roughly 2 larger as well. Since evidential regression models do not require sampling in order to estimate their uncertainty. their forward- forward-pass pass inference times are also significantly more efficient. Finally, we demonstrate comparable predictive accuracy (through RMSE and NIL.) to the other models. For more detailed breakdown of how the number of samples effects the baselines please refer to Tab. 3 Note that the output size of the depth estimation problem presented significant learning challenges for the BBB baseline, and it was unable to converge during training. As a result, for the remainder of this analysis we compare against only spatial dropout and ensembles.
We evaluate these models in terms of their accuracy and their predictive uncertainty on unseen test data. Fig. 44-C visualizes the predicted depth, absolute error from ground truth, and predictive uncertainty across three randomly picked test images. Ideally, a strong predictive uncertainty would capture any errors in the prediction (i.e., roughly correspond to where the model is making errors). Compared to dropout and ensembling, evidential uncertainty modeling captures the depth errors while providing clear and localized predictions of confidence. In general, dropout drastically underestimates the amount of uncertainty present, while ensembling occasionally overestimates the uncertainty.
To evaluate uncertainty calibration the ground-truth errors, we fit receiver operating character- istic (ROC) curves to normalized estimates of eтror and uncertainty. Thus, we test the network's 's ability to detect how likely it is make an error at a given pixel using its predictive uncertainty.
# Parameters
Inference Speed
RMSE
NLL
Ahsolute
Relative
Seconds
Relative
Evidential (Ours)
7,846,776
1.00
0.003
1.00
0.024 + 0.032
-1.128 0.290
Spatial Dropout
7,846.637
0.031
11.48
0.031 1 0.033
-1.227 0.374
Ensembles
39,233,285
5.00
0.010
3.72
0.023 4 0.027
-1.077 : 0.298
BBB
11,772,929
1.50
0.070
25.55
Table 2: Benchmark performance comparison on depth prediction. For fair performance comparison, sampling methods were all parallelized and sampled 5 times RMSE and NLL did not significantly improve with greater samples. For an extended analysis with larger number samples please refer Table

---

Under review as a conference paper at ICLR 2020
Depti
RGill
Depth Label
Doph
Estimating error from uncertainty
Spatial
Sundialty (TPR)
Dropout
Edidndial
Freavrie
63
24
Depth
Absolute
Predicted
Dooth
Assolute
Uncertainty
Predicive
Predktted
Absoluto
Predictive Uncertainty
Spectriity (FPR)
Figure 4: Modeling uncertainty depth estimation. Three methods for estimating epistemic (model) uncer- tainty are evaluated in the context of monocular depth cstimation. For cach model, we visualize the prediction, error to ground-truth, and estimated uncertainty for three randomly chosen examples (A-C). Ideally, the model should predict high uncertainty whenever it does know the answer (i.e., large error). We evaluate the sensitivity and specificity of predictive uncertainty in identifying likely errors with ROC curves (D).
ROC curves take into account sensitivity and specificity of the uncertainties towards error predictions and are stronger they contain greater area under their curve (AUC). Fig. 4D demonstrates that our evidential model provides uncertainty estimates concentrate to where the model making the errors.
Small Minoor
Goound
Truth
In addition to epistemic uncertainty, we also evaluate the aleatoric uncertainty estimates that are learned from our idential models as well. Fig. compares the evidential alcatoric uncertainty to those obtained by Gaussian likelihood optimiza- tion in several domains with high data uncertainty (mirror reflec- tions and poor illumination). The results between both methods are in strong agreement, identifying mirror reflections and dark regions without visible geometry as sources of high uncertainty.
Aleatoric
Evdeenid
(Ouns)
Aleatonic
Gauss on
NLL
Figure Alcatoric uncertainty in depth. Visualizing predicted aleatoric uncertainty in challenging reflection and illumination scenes. Comparison between evidential (Kendall & Gal 2017) show strong semantic agreement.
4.3 OUT-OF DISTRIBUTION TESTING
key use of uncertainty estimation is to understand when model is faced with test samples that fall out-of out-of-distribution (OOD) or when the model's output cannot trusted. the previous subsection, we showed that our evidential uncertainties were well calibrated with the model's errors. In this subsection, we investigate the performance on out-of- distribution samples. Fig. 6 illustrates predicted depth on various test input images (left) and outside (right) of the original distribution. All images have not been seen by the model during training. We qualitatively and quantitatively demonstrate that the epistemic uncertainty predicted by our evidential model consistently increases the OOD samples.
4.3. ROBUSTNESS TO ADVERSARIAL SAMPLES
Next, we consider the extreme case of OOD detection where the inputs adversarially perturbed to inflict maximum error on the model. We compute adversarial perturbations to our test set using the fast gradient sign method (Goodfellow al. 2014).. with increasing scales, of noise. Fig. 7
In-distribution Test Samples
Out-of-distribution Test Samples
Input
Image
Prodicted Depth
Far
Episternic
High
Uncertianty
Low
(Absolute)
(Relative)
3.08
13.95
Figure 6: Out-of-distributioo (OOD) data samples. Evidential models estimate and inflate epistemic uncer- tainty on data, where the prediction should not trusted. All samples were not seen during training.
7

---

Under review as a conference paper at ICLR 2020
A
D
RMSE
Adverearial
Input
Depth
Nolie
Label
B
Uncirtarty
350 spe
Predicted
Depth
Absolute
Noiso
Error
Senstidyy
Pradictive
Uncertainty
Specialtyy
Increasing Pertubation
Figure 7: Evidential robustness under adversarial noise. Increasing levels of adversarial noise (A) corrupt the predicted depth, and our model hegins to incur greater amounts of error. As adversarial noise increases, inferred epistemic uncertainty increases (localized to where the error occurs). Adversarially perturhed accuracy (B), epistemic uncertainty (C), well the noise evidential error estimation (D) also provided.
confirms that the absolute error of all methods increasing as adversarial noise added. We also observe positive effect noise our predictive uncertainty estimates in Fig. 73. An additional desirable property of evidential uncertainty modeling is that it presents higher overall uncertainty when presented with adversarial inputs compared to dropout and ensembling methods. Furthermore, we observe this strong overall uncertainty estimation despite the model losing calibration accuracy from the adversarial examples (Fig. (C).
The robustness evidential uncertainty against adversarial perturbations is visualized in greater detail in Fig. P. which illustrates the predicted depth, error, and estimated pixel-wise uncertainty as we perturb the input image with greater amounts of noise (left-to-right). Note that the predictive uncertainty not only steadily increases
increase the noise, but the spatial concentrations of uncertainty throughout the image maintain tight correspondence with the error.
DISCUSSION AND RELATED WORK
Uncertainty estimation has a long history in neural networks, from modeling probability distribution parameters over outputs (Bishop) 1994) to Bayesian deep learning (Kendall & Gal 2017). Our work builds on this foundation and presents a scalable representation for inferring the parameters of an evidential uncertainty distribution while simultaneously learning regression tasks via MLE.
In Bayesian deep learning, priors are placed over network weights and estimated using variational inference (Kingma et al., 2015). Dropout (Gal & Ghahramani 2016 Molchanov al. 2017 and BBB (Blundell et al. (2015) rely on multiple samples to estimate predictive variance. Ensembles (Lak) shminarayanan al.) 2017) provide tangential approach where sampling occurs over multiple trained instances. In contrast, we place uncertainty priors over the likelihood function and thus only need a single forward pass to evaluate both prediction and uncertainty. Additionally, our approach of uncertainty estimation proved to be better calibrated and capable of predicting where the model fails.
large topic of research in Bayesian inference focuses on placing prior distributions over hierarchical models to estimate uncertainty (Gelman et al.) 2006 2008). Our methodology falls under the class of evidential deep learning which models higher-order distribution priors over neural network predictions to interpret uncertainty. Prior works in this field (Sensoy et al. 2018 Malinin & Gales) 2018) have focused exclusively on modeling uncertainty in the classification domain with Dirichlet prior distributions. Our work extends this field into the broad range of regression learning tasks (e.g. depth estimation, forecasting, robotic control learning, etc.) and demonstrates generalizability to out-of-distribution test samples and complex learning problems.
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
In this paper, we develop a novel method for training deterministic NNs that both estimates desired target and evaluates the evidence in support of the target to generate robust metrics of model uncertainty. We formalize this in terms of learning evidential distributions, and achieve stable training by penalizing our model for prediction errors that scale with the available evidence. Our approach for evidential regression is validated method robustly scales to a key task in computer vision, depth estimation, and that the predictive uncertainty increases with increasing out-of-distribution adversarial perturbation. This framework for evidential representation learning provides a means to achieve the precise uncertainty metrics required for robust neural network deployment in safety-critical domains.
benchmark regression task. We further demonstrate that this
REFERENCES
Christopher M Bishop. Mixture density networks. In Tech. Rep. NCRG/94/004, Neural Computing Research Group. Aston University, 1994.
Charles Blundell, Julien Coreeiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv: 2015.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint 1rXiv:1604.07311 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Andrew Gelman, Aleks Jakulin, Maria Grazia Pittau, Yu-Sung Su, et al. A weakly informative default prior distribution for logistic and other regression models. The Annals of Applied Statisties, 2(4): 1360-1383, 2008.
Andrew Gelman et al. Prior distributions for variance parameters in hierarchical models (comment article by browne and draper). Bayesian analysis, 13)::115-334 2006.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 270-279, 2017.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.66 1412.657, 2014.
José Miguel Hernández-Looann and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. International Conference on Machine Learning, pp. 1861-1869, 2015.
Michael Jordan. The exponential family: Conjugate priors, 2009.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Advances in neural information processing systems, pp. 5574 55844, 2017.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems, pp.
2015.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 6402-6413 2017.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.

---

Under review as a conference paper at ICLR 2020
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. JMLR. org, 2017.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, 2012.
Giorgio Parisi. Statistical field theory. Addison-Wesles 1988.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems, pp. 3179-3189, 2018.
Joram Soch and Carsten Allefeld. Kullback-leible divergence for the normal -aamaa distribution. arXiv preprint arXiv:/611.01437, 2016.
7 APPENDIX
7.1 MODEL EVIDENCE DERIVATIONS
For convenience, define 1/o² be the precision of a Gaussian distribution. The change of variables transforms the Normal Inverse-Gamme distribution p(u, o² A, a, 3) to the equivalent Normal Gamma distribution p(u, T|7, X, ax B), parameterized by precision 7 (0, oc) instead of variance o².
)) = 8a / a- le BT Ar(u-n)?
(12)
T(a)22T
7.1.1 TYPE II MAXIMUM LIKELIHOOD Loss
Marginalizing out μ and 7 gives the result of equation
p(y m) f. plyelu, 7) p(u, =/,, 3) du
(13)
00
Ba 7 - Br
A
dudr
(14)
2n
[(a) 2n
(3r)a
X
-)(x-m)²2
-
2(1-x) dr
(15)
T(a)
|
2TT(1 2)
r=0
22 go
A
2B
A(7 - yi)²
A(7-1i13
(16)
2m
A)
1 1 X
For computational reasons it is common to instead minimize the negative logarithm of the model evidence.
-
X(> 14))2
CNL (w) - log p()(mm = loog 2 20
23 +
(17)
2m + X)
1 X
10

---

Under review as a conference paper at ICLR 2020
7.1.2 SUM OF SQUARES Loss
Similarly, we can marignalize out μ and o² to receive the result of equation g
J // E /By-polloo [lvv - vll p(u, 02 h,>, X, a, 8) du
2,00(10)
(18)
ar 11ye - y vllp(ulu, ,lu, 2,,,,,,,,,, ar 8) dy dy
(19)
0² p
(mmw)
119 - 2112
2T
20
22
8 VA
1
e
20
du
(20)
((a)
0
000
gavi
a+
Ma-s)?
[1yi --1)= 00]
e 02
2
dud
[(a)
0²
P=
(21)
0==0 Al(a) exp (-B/o²) 2(a-1) (o² + --yx-(y)))
-7) do?
do?
(22)
I(a- - }) AT(a)>VB
(23(1 + (2a 1)A(11 - 7)
(23)
7.2 KLEDIVERGENCE OF THE NORMAL DIVERSE-GAMMA
The KL-divergence between two Normal Inverse-Gamma. functions is given by (Soch & Allefeld 2016):
KIL (I(,,, 0² 22 ( 31)11p((,, o 12 X2, 012, (22,33)
(24)
la1 (M1 -22)² ) 22 + 1 A2
81
(((11) l((2))
+ az log
-log
(25)
2 31
2 A
2
82
+ (a1- -x2 - (B1 - 32) a1
(26)
31
r(-) the Gamma function and (-) is the Digamma function. The evidence defined by (2a + A).
For zero evidence, both = 0 and X 0. To compute the KL divergence between one N.I.G
distribution and another with zero evidence we can set either (u2, A2) = (i.e., forward-KL) in which case, [(0) is not well defined, {ai, = 0 (i.e. reverse-KL. KL) which causes divide-by-zer error of λ1. In either approach, the KL-divergence between an arbitrary N.I.G and one with zero evidence can not be evaluated.
7.3 BENCHMARK REGRESSION TASK EVALUATIONS
7.3.1 EPISTEMIC UNCERTAINTY ESTIMATION
The training set consists of training examples drawn from y sin(3r)/(3)) + e, where (0, 0.02) in the region -3 < x < 3, whereas the test data is unbounded. All models consisted of 100 neurons with 3 hidden layers and were trained to convergence. The data presented in Fig. 3 iillustrates the estimated epistemic uncertainty and predicted mean accross the entire test set, x 3.
7.3.2 IMPACT OF THE EVIDENTIAL REGULARIZER
In the following experiment, we demonstrate the importance of augmenting the training objective with evidential regularizer L as introduced in Sec. 3.2 Fig. 8 provides quantitative results training the same regression problem presented in 7.3. with and without this evidential regularization term. This term introduces an "uncertain" prior into learning process out-of-distributio (OOD)
11

---

Under review as a conference paper at ICLR 2020
samples exhibit high epistemic uncertainty. Without the of this novel loss term, the learned epistemic uncertainty is unreliable on OOD data.
Without Evidence Regularizer
With Evidencce Regularizer
Prediction
Unceriaiity
Figure 8: Evidential regularizer. The use of our novel L loss during training helps minimize evidence (maximize uncertainty) on out-of-distribution data, thus enabling OOD uncertainty robustness for regression prediction problems.
7.3.3 ALEATORIC UNCERTAINTY ESTIMATION
The training set consists of training examples drawn from y sin(3r)/(3z)) + c(x), where c(x) ~ (O, s(x),, and s(x) 20 cos(3.3x) 0.1. We evaluate against (Kendall & Gal 2017) which
presents algorithm for heteroscedastic aleatoric uncertainty estimation by inferring the mean and variance of a Gaussian likelihood function. As presented in the paper, training is done by minimizing the negative log-like likellooooodd of the data given the inferred likelihood parameters. Both network and the baseline Gaussian NLL network consisted of 100 neurons with 3 hidden layers and were trained convergence.
Ground Truth Aleatoric Uncertainty
Gaussian NLL
Evidential
Data Sample
Mean Extimate
Aleaboric Uncettainty
Figure 9: Alcatoric uncertainty estimation. Comparing the ability to learn the heteroscedastic aleatoric uncertainty in synthetic dataset. Evidential modelling is able to match the performance of Gaussian likelihood optimization (Knddlll & Gal 2017).
7.4 ADDITIONAL DEPTH ESTIMATION PERFORMANCE RESULTS
Parameters Absolute
Inference Speed Relative
RMSE
NLL
Relative
Seconds
Evidential (Ours)
7,846,776
1.00
0.003
1.00
0.024
0.032
-1128 1 0.290
Spatial Dropout
7,846,657
1.00
0.028
0.231
Spatial Dropout
7,846,657
1.00
0.031
11.48
0.031 0.033
-1.227 1 0.374
Spatial Dropout
7,846,657
1.00
0.037
Spatial Dropout
25
7,846,657
1.00
0.065
23.99
0.022 - 0.035
-1.137 1 0.327
Spatial Dropout
7,846.657
1.00
0.388
Ensembles
15, 15,6933314
2.00
0.005
1.94
0.026 0.032
-1080 1 3.334
Ensembles
39 29,2233285
Ensembles
10
78. 466.570
10.00
0.019
6.82
0.025 0.038
-0.900 + 0.298
Ensembles
425
Ensembles
392, 3500
50.00
0.112
41.26
0.022 0.031
-0.9% + 0.275
BBB
11.772.929
1.50
0.064
BBB
11,772.929
1.50
0.070
25.55
BBB
11.772.929
BBB
25
11,772,929
1.50
0.144
53.09
BBB
11,772.929
0.284
Table Depth estimation performance. Comparison of different epistemic uncertainty estimation algorithms and predictive performance on an unseen test set. Dropout, ensembles, and basos--yBBackrropp were sampled times on parallel threads. The evidential method outperforms all other algorithms in terms of space (#Parameters) and inference speed while maintaining competetive RMSE and NLL.
12


Reviews:
[{"review_id": "ryxl-cgTYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed deep evidential regression, a method for training neural networks to not only estimate the output but also the associated evidence in support of that output. The main idea follows the evidential deep learning work proposed in (Sensoy et al., 2018) extending it from the classification regime to the regression regime, by placing evidential priors over the Gaussian likelihood function and performing the type-II maximum likelihood estimation similar to the empirical Bayes method [1,2]. The authors demonstrated that the both the epistemic and aleatoric uncertainties could be estimated in one forward pass under the proposed framework without resorting to multiple passes and showed favorable uncertainty comparing to existing methods. Robustness against out of distribution and adversarially perturbed data is illustrated as well.\n\nOn the technical side, the novelty is incremental. The extension from the classification regime to the regression regime, from the conjugate Dirichet prior to the conjugate Normal-Inverse-Gamma prior, is quite straightforward. Besides, the presentation of the paper could be largely improved. It is not easy to follow the derivation in Section 3. The discussion of concepts and problem definitions look fragmented and incoherent. Even though the presentation largely follows (Sensoy et al., 2018) and uses terms from theory of evidence, the derivation actually is more aligned with the prior network [3] under the Bayesian framework which is missing from the references. It is really confusing that the authors talked about the variational inference when conjugate prior is used, and it is unclear how the variational distributions are used in Section 3.2 or how the \"I don't know\" loss term relates to the KL-divergence between the variational distribution and the prior in Section 3.3. This term was manually added as additional regularization to \"prefer the evidence to shrink to zero for a sample if it cannot be correctly classified\" in (Sensoy et al., 2018), and a different regularization was used to encourage distributional uncertainty in [3]. I hope that the authors could spend more efforts clarifying their ideas, especially the derivations in Section 3.2 and 3.3.\n\nOn the other hand, there is no referring to the input x in the entire derivation and problem formulation in Section 3. It took me a while to realize that the formulation in (4) actually defines the generation for a particular input, not for all the inputs. That is, the model is trying to model heteroscedastic uncertainty, not the homoscedastic counterpart. It could be better to call out the dependence on the input explicitly. \n\nOn the quantitative side, the baseline models considered in Section 4 are mainly concerned with epistemic uncertainty estimation. So it would be good to explicitly discuss which uncertainty estimation was compared with. This work estimates both aleatoric and epistemic uncertainties, so a better comparison is to models that estimate both quantities (Kendall & Gal, 2017)[4] which has been shown to give better output estimation comparing to epistemic uncertainty estimation only (Kendall & Gal, 2017).\n\nOther comments:\n- What is the \\pi in equation (8)?\n- The \"I don't know\" loss introduced in Section 3.3 used L-p norm. What is the originality of the L-p norm here? In practice, which p value should be used? In the experiments, which p value was used?\n- The RMSE results of the depth estimation presented in Table 2 are orders of magnitude smaller than those from existing work, for example Table 2(b) in (Kendall & Gal, 2017). Was a different RMSE computation used in this work?\n- From the caption in Table 2, it seems that only 5 samples were used in MC-dropout, which is considerably smaller than those used in existing work (Kendall & Gal, 2017).\n\n[1] D.J.C. MacKay. Hyperparameters: optimize, or integrate out? Maximum Entropy and Bayesian Methods, Springer 1996.\n[2] B. Efron. Two modeling strategies for empirical Bayes estimation. Statistical Science, 2014.\n[3] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. NeurIPS 2018.\n[4] Y. Kwon, J.-H. Won, B.J. Kim, and M.C. Paik. Uncertainty quantification using Bayesian neural networks in classification: application to ischemic stroke lesion segmentation. MIDL 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HklhT8o3KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel approach to estimate the confidence of predictions in a regression setting. The approach starts from the standard modelling assuming iid samples from a Gaussian distribution with unknown mean and variances and places evidential priors (relying on the Dempster-Shafer Theory of Evidence [1] /subjective logic [2]) on those quantities to model uncertainty in a deterministic fashion, i.e. without relying on sampling as most previous approaches. This opens the door to online applications with fully integrated uncertainty estimates. \nThis is a very relevant topic in deep learning, as deep learning methods are increasingly deployed in safety-critical domains, and I think that this works deserves its place at ICLR.\n\nPros:\n1.\tNovel approach to regression (a similar work has been published at NeurIPS last year for classification [3]), but the extension of the work to regression is important.\n2.\tThe experimental results show consistent improvement in performance over a wide base of benchmarks, scales to large vision problems and behaves robustly against adversarial examples.\n3.\tThe presentation of the paper is overall nice, and the Figures are very useful to the general comprehension of the article.\nCons:\n1.\tThe theory of evidence, which is not widely known in the ML community, is not clearly introduced. \nI think that the authors should consider adding a section similar to Section 3 of Sensoy et al. [3] should be considered. Currently, the only step explaining the evidential approach that I found was in section 3.1, in a very small paragraph (between “the mean of […] to \\lambda + 2\\alpha.”). I believe that the article would greatly benefit from a more thorough introduction of concepts linked to the theory of evidence.\n2.\tThe authors briefly mention that KL is not well defined between some NIG distributions (p.5) and propose a custom evidence regularizer, but there’s very little insight given on how this connects to/departs from the ELBO approach. \n\nOther comments/questions:\n1.\t(p.1)  I’m not sure to fully understand what’s meant by higher-order/lower-order distributions, could you clarify?\n2.\t(p.3) In section 3.1, the term in the total evidence \\phi_j is not defined.\n3.\t(p.3) Could you comment on the implications of assuming that the estimated distribution can be factorized? \n4.\t(p.4) Could you comment on the difference that there is between NLL_ML and NLL_SOS from a modelling perspective?\n5.\t(p.4) The ELBO loss (6) is unclearly defined, and not connected to the direct context. I would suggest moving this to the section 3.3, where the prior p(\\theta) used in eq. (6) is actually defined.\n6.\t(p.4) In equation (6), p_m(y|\\theta) isn’t defined, and q(\\theta|y) is already parameterized on y if I understand that q(\\theta)=p(t\\heta|y1,…,yN). Making the conditioning explicit in equation (6) might make the connection to the ELBO clearer. \n7.\t(p.7) I’m not sure to understand how the calibration of the predictive uncertainty can be tested by the ROC curves if both the uncertainty and estimates error are normalized. Could you also define more clearly what you mean by an “error at a given pixel”? \n8.\tSpelling & typos:\n-\t(p.4) There are several typos in equation (8), where tau should be replaced with 1/\\sigma^2. \n-\t(p.8) In the last sentence, there is “ntwork” instead of network.\n-\t(p.9) There is a typo in the name of Jøsang in the references. \n-\t(p.10) In equation (13), due to the change of variable, there should be a \n-(1/\\tau^2) added;  \n-\t(p.10) In equation (14), the \\exp(-\\lambda*\\pi*(…)) should be replaced with \\exp(-\\lambda*\\tau*(…)). \n\n[1] Bahador Khaleghi, Alaa Khamis, Fakhreddine O Karray, and Saiedeh N Razavi. Multisensor data fusion: A review of the state-of-the-art. Information fusion, 14(1):28–44, 2013. \n[2] Audun Jøsang. Subjective Logic: A formalism for reasoning under uncertainty. Springer Publishing Company, Incorporated, 2018. \n[3] Sensoy, Murat, Lance Kaplan, and Melih Kandemir. \"Evidential deep learning to quantify classification uncertainty.\" Advances in Neural Information Processing Systems. 2018.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlBIefWcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the aleatoric uncertainty and epistemic uncertainty in machine learning. The evaluation was performed on benchmark regression tasks. The comparison with other state-of-the-art methods was provided. The evaluation of the robustness against out of distribution and adversarially perturbed test data was performed.\n\nStrength:\n1. Experiments were complete. Analyses were provided with useful information.\n2. A model with smaller number of parameters was proposed.\n3. Computation efficiency was improved.\n\nWeakness:\n1. Total evidence and model evidence were defined. The derivation of these evidences should be clarified.\n2. Theoretical justification for related methods could be improved.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for providing uncertainty for deep learning regressors through assigning a notion of evidence to the predictions.  This is done by putting priors on the parameters of the Gaussian outputs of the model and estimating these via an empirical Bayes-like optimization.  The reviewers in general found the methodology sensible although incremental in light of Sensoy et al. and Malinin & Gales but found the experiments thorough.  A comment on the paper pointed out that the approach was very similar to something presented in the thesis of Malinin (it seems unfair to expect the authors to have been aware of this, but the thesis should be cited and not just the paper which is a different contribution).  In discussion, one reviewer raised their score from weak reject to weak accept but the highest scoring reviewer explicitly was not willing to champion the paper and raise their score to accept.  Thus the recommendation here is to reject.  Taking the reviewer feedback into account, incorporating the proposed changes and adding more careful treatment of related work would make this a much stronger submission to a future conference.
Decision: Reject