Second-Moment Loss: A Novel Regression Objective for Improved Uncertainties
Authors: Joachim Sicking
Maram Akila
Maximilian Alexander Pintz
Tim Wirtz
Asja Fischer
Stefan Wrobel
Venue:  (2021)
Keywords: regression
uncertainty quantification
uncertainty evaluation
dropout

Abstract: Quantification of uncertainty is one of the most promising  approaches to establish safe machine learning. Despite its importance, it is far from being generally solved, especially for neural networks. One of the most commonly used approaches so far is Monte Carlo dropout, which is computationally cheap and easy to apply in practice. However, it can underestimate the uncertainty. We propose a new objective, referred to as second-moment loss (SML), to address this issue. While the full network is encouraged to model the mean, the dropout networks are explicitly used to optimize the model variance. We analyze the performance of the new objective on various toy and UCI regression datasets. Comparing to the state-of-the-art of deep ensembles, SML leads to comparable prediction accuracies and uncertainty estimates while only requiring a single model. Under distribution shift, we observe moderate improvements. From a safety perspective also the study of worst-case uncertainties is crucial. In this regard we improve considerably. Finally, we show that SML can be successfully applied to SqueezeDet, a modern object detection network. We improve on its uncertainty-related scores while not deteriorating regression quality. As a side result, we introduce an intuitive Wasserstein distance-based uncertainty measure that is non-saturating and thus allows to resolve quality differences between any two uncertainty estimates.

Under review as a conference paper at ICLR 2021
SECOND-MOMENT Loss: A NOVEL REGRESSION OBJECTIVE FOR IMPROVED UNCERTAINTIES
Anonymous authors Paper under double-blind review
ABSTRACT
Quantification of uncertainty is one of the most promising approaches to estab- lish safe machine learning. Despite its importance, is far from being gener- ally solved, especially for neural networks. One of the most commonly used ap- proaches so far Monte Carlo dropout, which is computationally cheap and easy to apply in practice. However, it can underestimate the uncertainty. We propose a new objective, referred as second-moment loss (SML), to address this issue. While the full network is encouraged model the mean, the dropout networks are explicitly used to optimize the model variance. We analyze the performance various toy and UCI regression datasets. Comparing
of the new objective the state-of-th tite-oof-thearr art of deep ensembles, SML leads to comparable prediction ac- curacies and uncertainty estimates while only requiring single model. Under distribution shift, we observe moderate improvements. From a safety perspective also the study of worst-case uncertainties is crucial. In this regard we improve con- siderably. Finally, we show that SML can be successfully applied to SqueezeDet, modern object detection network. We improve on its uncertainty-relate scores while not deteriorating regression quality. As a side result, we introduce an in- tuitive Wasserstein ristance-based uncertainty measure that is non saturatin and thus allows to resolve quality differences between any two uncertainty estimates.
INTRODUCTION
Having attracted great attention in both academia and digital economy, deep neural networks (DNNs, Goodfellow et al. 2016)) are about to become vital components of safety- ceiiticll applications. Ex-
amples are autonomous driving (Pomerlcau 1989 Bojarski et al. 2016) or medical diagnostics (Liu
et al. 2014), where prediction errors potentially put humans risk. These systems require methods that are robust not only under lab conditions (i.i.d. data sampling). but also under continuous domain shifts, think e.g. of adults on e-scooters growing numbers of mobile health sensors. Besides shifts in the data, the data distribution itself poses further challenges. Critical situations are (fortunately) rare and thus strongly rnder-represented in datasets. Despite their rareness, these critical situations have a significant impact on the safety of operations. This calls for comprehensive aelssassesmment capabilities of DNNs and recent uncertainty mechanisms can be seen as step in that direction.
While a variety of uncertainty approaches has been established, stable quantification of uncertainty still an open problem. Many recent machine learning applications are e.g. equipped with Monte Carlo (MC) dropout (Gal & Ghahramani 2016) that offers conceptual simplicity and scalability. However, is tends to underestimate uncertainties thus bearing disadvantages compared to more re- cent approaches such deep ensembles (Lakshminarayanan et al. (017)). We propose alternative uncertainty mechanism. It builds on dropout neb-netwrrks and explicitly optimizes variances (see Fig. for illustrative example). Technically, this is realized by simple additive loss term, the second-moment loss. To address the above outlined requirements for safety-critical systems, we evaluate our approach systematically w.r.t. continuous data shifts and worst-case performances.

---

Under review as a conference paper at ICLR 2021
7.
7.
Figure I: Sampling-basse based uncertainty mechanisms on toy datasets. contrast MC dropout (left), the second- loss (right) induces uncertainties that capture aleatoric uncertainty. The ground truth data shown in red. Each grey line represents the outputs of one of 200 random sub-networks that are obtained by applying dropout-hasel sampling to the trained full network. For details on the data sets ( ('toy-hf", 'toy-noise', ) the neural architecture and the uncertainty methods please refer to section X and references therein.
In detail, our contribution is follows:
we introduce novel regression loss for better calibrated uncertainties applicable dropout networks,
we reach -ate-offtthe-art performance in an empirical study and improve on it when con- sidering data shift and worst-case performances, and
we demonstrate applicability to real-world applications by example of 2D bounding box regression.
2 RELATED WORK
Approaches estimate predictive uncertainties can be broadly categorized into three groups: Bayesian approximations, ensemble approaches and parametric models.
Monte Carlo dropout (Gal & Ghahramani 2016) is a prominent representative of the first group- offers a Bayesian motivation, conceptual simplicity and scalability to application-size neural net- works (NNs). This combination distinguishes MC dropout from other Bayesian neural network (BNN) approximations like Blundell et al. (2015) and Ritter et al. (2018). A computationally more efficient version of MC dropout is one- one-rayer last-layer dropout (see e.g. Kendall & Gal (2017). Alternatively, analytical moment propagation allows sampling-free MC-dropout inference at the price of additional approximations (e.g. Postels et al. (2019)) Further extensions of MC dropout tar- get tuned performance by learning layer-specific drop rates using Concrete distributions (Gal et al., 2017) and the integration of alcatoric uncertainty (Kendall & Gal 2017). Note that dropout train- ing used- inod--indeeeneet from an uncertainty context for better model generalization (Srivastaval
al. 2014).
Ensembles of neural networks, so-called deep ensembles [Lakshmnnaryyanan et al. 2017), pose another popular approach to uncertainty modelling. Comparative studies of uncertainty mechanisms (Snoek al. 2019 Gustafsson et al. 2020) highlight their advantageous uncertainty quality, making deep ensembles a of method. Fort et al. (2019) argue that deep ensembles capture multi- modality of loss landscapes thus yiclding potentially more diverse sets of solutions.
The third group are parametric modelling approaches that extend point estimations by adding a model output that is interpreted as variance or covariance (Nix & Weigend 1994; Heskes 1997). Typically, these approaches optimize (Gaussian) negative log-likelihood (NLL, Nix & Weigend (1994). A more recent representative of this group e.g., Kendall & Gal (017), for review see Khosravi et al. (201T). A closely related model class is deep kernel learning. It approaches uncertainty modelling by combining NNs and Gaussian processes (GPs) in various ways, e.g. via
2

---

Under review as a conference paper at ICLR 2021
an additional layer (Wilson et al.) 2016: Iwata & Ghahramani 2017). by using networks GP kernels (Garnelo et al. 2018) or by matching NN residuals with a GP (Qiu et al. 2019).
In the context of object detection, various uncertainty approaches can be encountered, e.g. MC dropout in Bhattacharyya et (2118) and et al. (2018), or parametric approaches Hn et al. (219
Hall et al. (2020) advocate to account for uncertainty in bounding box detection.
The quality of uncertainties is typically evaluated using negative log-likelihodd (Blei et al. 2006
Walker et al.] [2166] Gal & Ghahramani] 2016), expected calibration error (ECE) (Naeini et al.
SSoek et al. 2019) and its variants and by considering correlations between uncertainty esti-
mates and model errors, e.g. area under the sparsification error curve (AUSE, [l et al. (20T8) for image tasks. Moreover, it is common to study how useful uncertainty estimates are for solving aux- iliary tasks like out-of-distribution classification [Lakshminrrayanan et al. 2017)) or robustness w.r.t. adversarial attacks. An alternative approach is the investigation of qualitative uncertainty behaviors: Kendall & Gal (2017) check if the epistemic uncertainty decreases when increasing the training set or Wirges et al. (2019) studies how the level of uncertainty depends on the distance of the object to a car for some 3D environment regression task.
3 SECOND-MOMENT LOSS
Monte Carlo (MC) dropout was proposed as a computationally cheap approximation of perform- ing Bayesian inference in neural networks (Gil & Ghahramani 2016). Given neural network fo : Rdd -> Th with parameters 0, MC dropout samples sub-networks Sg by randomly dropping nodes from the main model fe. During MC dropout inference the prediction given by the mean estimate over the predictions of given sample of sub-networks, while the uncertainty associated with this prediction can be estimated, e.g. terms of the sample variance. During MC dropout training the objective function, e.g. (in our case) the mean squared error (MSE), is applied to the sub-networks separately. Due to this training procedure, all sub-network predictions are shifted towards the same training targets. which can result in overconfident predictions, i.e. in an underesti- mation of prediction uncertainty]
Based on this observation, we propose to use the sub-networks fa in different way: they are explicitly not encouraged to fit the data mean directly. This is the task of the full network fe. The sub-networks fā instead model aleatoric uncertainty and prediction residuals if the prediction of the full network fo is incorrect. Thus, we deliberately assign different 'jobs' to the main network fe on the one hand and its sub-networks on the other hand. Formalizing this idea into optimization objective yields
M
L = Lrege + Lygul
regression Σ M
1
(fo(z:) - vi)2+ - +B B (1fa(z)) - fo(x:)l-110(( -b:)));)>)))>)))) (1)
second moment losss
where the sum runs over a mini-batch of size M- < N taken from the set of observed samples D [2,, W) I; Rd denotes the input, y/s RT the ground-truth label, and 8 0 is hyper-parameter that weights both terms. The first term, Tregr: is the MSE w.r.t. the full network fo. The second term, Luml, seeks to optimize the networksss fğ- It aims at finding sub-networks such that the distance (z.) fo(x;)) matches the prediction residual, quantified by (fo( 10---44 ysl. which also serves as proxy to the aleatoric uncertainty (compare Fig. 2 top). As our choice of Lami removes all directional information of the residual, possible (optimal) solutions for the fg are not uniquely determined
3 This leads to a significant increase in the variance the sub-networks, i.e. the second moment of fa- compared to standard MC dropout, which is why name Lami the second- moment loss (SML) The standard deviations Ototal of the predictions of the sub-networks w.r.t
intuitive explanation is as follows: Let fo be NN with one-dimensional output. For MC dropout with the MSE loss we get (()(2z) - w)²) = ((f(2)) =)² + o (Jz(z)). Therefore, it simultaneously minimizes the squared error between sub-network mean and target and the variance o² (fa (x)) = (z)()) - (fa(z))²
over the sub-networks
"To avoid unintended optimization of full fe direction of fz. we only back-proppgate through f3 in Lont- FFor an analytical study of the loss landscape induced hy Lsmil see appendix A.
*For brevity, also refer the entire loss objective T. second-moment loss during evaluation.
3

---

Under review as a conference paper at ICLR 2021
the prediction of the mean network induced by the SML have two components: the spread drop of the sub-networks and an offset fa (ff) between the full network and the sub-network mean that our loss might cause, concretely, Tooal dhop + Ifo {fa)ll While Iso- - (fa)l is reminiscent of residual matching, drop seems to be more closely related to modelling uncertainties. We show in appendix A.2 that drop accounts on average for more than 80% of Crotal in our experiments.
Note that while we investigate the proposed objective in terms of dropout sub-networks in this paper, our arguments as well as the actual approach are generally applicable to other models that allow to formulate sub-networks given some kind of mean model. Besides the regression tasks considered here our approach could be useful for other objectives which use or benefit from an underlying distribution, e.g. uncertainty quantification in classification.
4 EXPERIMENTS
We begin this section with an illustrative and visualizable toy dataset (section 4.1 and continue with benchmarks on various UCI datasets (Dua & Graff, 2017) in section 4.2 To conclude in 4.3 the second-moment loss is applied to more complex task: object detection in the form of 2D
bounding box regression using the compact SqueezeDet architecture (Wu et al. 2017). Technical details for this section are relegated to appendix
For the first two parts we use an identical set-up of hidden layers with 50 neurons each and ReLu activations. As benchmark methods we consider: MC dropout (abbreviated as MC), last-layer MC dropout (MC-LL), parametric uncertainty (PU), deep ensembles with (PU-DE) and without (DE) explicit PU and PU combined with MC (PU-MC). All considered types of networks provide estimates (14; o:), where a; is obtained directly as model output (PU), by sampling (MC, MC-LL, SML) as ensemble aggregate (DE, PU-DE). For PU-MC, combination of parametric output and sampling is employed.
While the toy model has a stronger focus on visual inspection, the UCI evaluation relies on a variety of measures: root-mean-square error (RMSE), negative log-likelihood (NLL), expected calibration error (ECE), and novel usage of the Wasserstein distance (WS). In using a least squares regression, we make the standard assumption that errors follow a Gaussian distribution. This assumption is re- flected in the (standard) definitions of above named measures, i.c. all uncertainty measures quantify the set of outputs {(14, (J)) relative to Gaussian distribution Details on the definition of those measures, as well as on the network and training procedure and the implementation of methods can be found in appendix B.1
4.1 TOY DATASETS
To illustrate qualitative behaviors of the different uncertainty techniques, we consider two RR IR toy datasets. This benchmark puts a special focus on the handling of data-inherent uncertainty. The first dataset is Gaussian white noise with an z-dependent (non-linear) amplitude, first row of The second dataset is a polynomial overlayed with a high-frequency, amplitude-mduulated
Fig. 2 sine, fourth row of Fig. 2 The explicit equations for the toy datasets used here be found in appendix B.2 While the uncertainty in the first dataset ("toy-noise" ") is clearly visible, it is less obvi- ous for the fully deterministic second dataset ("toy-hf"). There is an effective uncertainty though, as the shallow networks employed are empirically not able to fit (all) fluctuations of 'toy-hf' (see fifth row of Fig. 2) One might (rightfully) argue that this is sign of insufficient model capacity. But, in more realistic, c.g., higher dimensional and sparser datasets the distinction between true noise and complex information becomes exceedingly difficult to make. As the Nyquist-Shanne sampling theorem states, with limited data deterministic fluctuations above a cut-off frequency can no longer be resolved (Landau 1967). They therefore become virtually indistinguishable from random noise.
The mean estimates of all uncertainty methods (second and fifth row in Fig. 2 look alike on both datasets. They approximate the noise mean and the polynomial, respectively. In the latter case, all methods rudimentarily fit some individual fluctuations. The variance estimation (third and sixth row in Fig. in contrast reveals significant differences between the methods: While PU, PU-
While different distributions, e.g. exponentially decaying or mixtures, could be used, restrict the scope here the standard Gaussian case.
4

---

Under review as a conference paper at ICLR 2021
DE, and the network trained with SML capable of capturing aleatoric uncertainty. MC dropout variants and non-parametric ensembles are not. This behavior of MC dropout is expectable as was introduced to account for model uncertainty not data-inherent uncertainty. The non-parametric ensemble is effectively optimized in a similar fashion. contrast, NLL-optiimzed PU networks have home-turf advantage these datasets since the parametric variance is explicitly optimized to account for the present alcatoric uncertainty. The SML provides comparably good uncertainty estimates. They are evoked by the Lami-term that incentivizes sub-networks fa to keep adequate distance from fo. The results on the "toy-hf" dataset exemplify that the SML can provide good uncertainty estimates even for networks with insufficient expressiveness. While the outcomes of both PU (PU-DE) and SML-trained network look similar, the mechanics of the two approaches are fundamentally different. Accompanying quantitative evaluations can be found in appendix B.2
investigate the drivers behind the adjustments the sub-networks in
appendices A and A.3
In the following, we substantiate the corroborative results of the SML on toy data by an empirical study on UCI datasets and application to modern object detection network.
25
meean
22
0
D3
Comparison of uncertainty approaches (columns) on two 1D toy functions: a noisy one (top) and high-frequency one (bottom). Test data ground truth (respective first row) shown with mean estimates (resp. second row) and standard deviations (resp. third row).
Figure 2:
4.2 UCI REGRESSION DATASETS
Next, we study UCI regression datasets, extending the dataset selection in Gal & Ghahramani (016)) by adding three further datasets: 'diabetes', 'california', and 'superconduct" Apart from train- and test-data results, we study regression performance and uncertainty quality under data shift. Such distributional changes and uncertainty quantification are closely linked since the latter ones are rudi- mentary selffaassesmment mechanisms that help to judge model reliability. These judgements gain importance for model inputs that are structurally different from train data. Appendix B.3 elaborates on different ways of splitting the data, namely pca- pca-based splits in input space (using the first princi- pal component) and label-based splits. For both consider interpolation testing and extrapolation testing, where the training omits data within or at the edges the full data range, respectively. For example, for labels running from 0 to 1, (label-based) extrapolation testing would consider only data
5

---

Under review as a conference paper at ICLR 2021
with label larger 0.9, while training would be performed on the smaller label values. More general information on training and dataset-dependens modifications to the experimental setup are relegated to the technical appendix B.1 For brevity of exposition, we limit our discussion here largely to the ECE and the worst-case uncertainty performance. An evaluation of the remaining measures, includ- ing the Wasserstein measure, is given in appendix [B. All presented results are 5- or 10-fold cross validated.
Fig. provides average ECE values of the outlined uncertainty methods under i.i.d. conditions (first and second panel), under label-based data shifts (third and fourth panel) and under pea-based data shifts (fifth and sixth panel). For the shifts we either use interpolation or extrapolation where we The visualized mean (median) values and quantile intervals are obtained by averaging over ECE values on 13 UCT datasets. On training data, ECEs are smallest for PU, followed by PU-MC, PU-DE and the SML-trained network. On test data, however, PU-MC, PU-DE and the SMI--triine network share the first place. Looking at the stability w.r.t. data shift, i.e. extra- and interpolation based on label-split pca-split, PU loses in performance while PU-DE, PU-MC and SML reach the smallest calibration errors. Regarding the 75% quantiles, SML consistently provides one of the best result on the standard test set and on all oft-of-data test sets.
For NLL and Wasserstein measure, PU-DE, PU-MC and the SML-trained network reach comparably small average values with advantages for the SML-trained network under data shift, see Fig. 2 and Fig. 10 in appendix [B33] for detailed evaluations. Especially for NLL a strong difference can be seen with respect to the consistency of performance under label-based data shifts, which suggests that SML is more "reliable" compared to PU based approaches. In contrast to uncertainty quality, regression performances are almost identical for all uncertainty methods (see Fig. 8 and Table 5 in appendix B.3).
Summarizing these evaluations on UCI datasets, we find SML to be as strong as the sfate-of-the- art methods of PU-DE and PU-MC. In comparison to PU-DE, PU-MC and SML use only a single network compared an ensemble of 5 networks. We moreover observe advantages for SML under PCA- and label-based data shifts. Three datasets lead to overestimated uncertainties for the SML,, see discussion appendix [33 visual tool to further inspect uncertainty quality are residual- uncertainty scatter plots as shown in appendix B.4] For a reflection on NLL and comparisons of the different uncertainty measures on UCI data see again appendix B.3
From a safety perspective the study of worst-case uncertainties is crucial. A better understanding of these least appropriate uncertainties might allow to determine lower bounds on operation qual- ity of safety-critical systems. For this we define normalized residuals T2 = (μ4 w))aa, based on the prediction estimates (14,0) for a given data point (It, yt). We restrict our analysis to un- certainty estimates that under-estimate prediction residuals, i.e. [rs] > 1. These cases might be more harmful than overly large uncertainties, |r: < 1, that likely trigger a conservative system behavior. We quantify worst-case uncertainty performance as follows: for a given (test) dataset, the absolute normalized residuals {|ri]]; are calculated. We determine the 99% quantile go.99 of this set and calculate the mean value over all Irgl 90.99. the so-called expected tail loss at quantile 99% (ETLo.99) (Rockafellar & Uryasev 2002). The ETLo.99 measures the average performance of the worst performing 1%
For both toy datasets and 12 UCI datasets, the test data ETL..99 's of all trained network are cal- culated, yielding a total of 105 ETLg,99 values per uncertainty method. Table T reports the mean value and the maximal value of these ETLo.99's for PU-MC, PU-DE and SML-trained networks as these three methods show the strongest performances throughout this work. While none of these methods gets close to the ideal ETL..99 's of a (0, 1). SML-trained networks exhibit significantly less pronounced tails and therefore higher stability compared to PU-MC and PU-DE. This holds true over all considered test sets. Deviations from standard normal grow from i.i.d. test over PCA-based train-test split to label-based train-test split. We attribute the lower stability of PU-DE to the nature of the PU networks composing the ensemble. The inherent instability of parametric uncertainty timation (see Table 5 in appendix B.3) is largely suppressed by ensembling. Considering the tail of the rrl-distribution however reveals that regularization of PU by ensembling works not in every single case. It is unlikely that larger ensemble are able to fully cure this instability issue. Regulariz- ing PU by applying dropout (PU-MC) leads comparably weak results. SML-trained networks in contrast encode uncertainty into the structure of the entire network thus yielding preferable stability compared to parametric approaches.

---

Under review as a conference paper at ICLR 2021
1.I.d. data split
label-based data shift (extrap.)
pca-based data shift
(interp.)
(extrap.)
test (interp.)
2.0
1.5
Xxxx
*
1.0
100
X
FX
0.5
0.0
me me coo no me pa de Aor
as mo
mc
nu po
mo
me
cuio
nu
mo me con
nu mu
me. com
mi po
met
me coes nu me Ae iu
Figure 3: Expected calibration errors (ECEs) of different uncertainty methods under i..d. conditions (first and second panel) and under various kinds of data shift (third to sixth panel, see text for details). SML ('ours') is compared to benchmark approaches. Each blue cross is the mean over ECE values from 13 UCI regression datasets. Orange line markers indicate median values. The gray vertical bars reach from the 25% quantile (bottom horizontal line) the 75% quantile (top horizontal line).
Table 1: Worst-case uncertainty quality for different uncertainty methods: SML-induced uncertain- ties ('Ours'), PU-DE and PU-MC are compared the ideal Gaussian case for i.i.d. and non-i.i.d. data splits. Worst-case uncertainty quality is quantified by the expected tail loss at the 99% quantile (ETL..99). Each mean and max value is taken over the ETLs of 105 models trained on 14 different datasets.
measure
data split
N(O01) 1)
Ours
PU-DE
PU-MC
mean .0.99
ETL ETLg.99
i.i.d.
2.89
3.80
5.03
5.55
max
ETL 0.99
i.i.d.
3.01
9.71
19.69
30.35
mean
ETLo.99
pea
2.89
4.62
6.71
6.12
max
ETLg.99
0.99
pea
3.01
13.0
39.34
21.58
ETLg.99
label
2.89
5.18
38.65
49.82
max ETL,.99
label
3.01
35.96
799.78
631,84
4.3 APPLICATION TO OBJECT REGRESSION
After studying toy and UCI datasets, we turn towards the challenging real-world task of object detection, namely the SqueezeDet model (Wu al. 2017). fully convolutional neural network. trained and evaluated on KITTI (Geiger et al.) 2012). For details on the Squeeze architecture and the KITTI data split, see B.5 We compare standard SqueezeDet with SML-SqueezeDet that uses the second-moment loss instead of the original MSE regression loss (see appendix B.5 for more details). In both settings the model is trained for 150,000 mini-batches of size 20, i.e. for 815 epochs. After training. we keep dropout active and compute 50 forward passes for each test image. For standard SqueezeDet, all forward passes are individually matched with ground truth. We exclude predictions from the evaluation if their loU with ground truth is < 0.1. While standard SqueezeDet (with activated dropout at inference) uses the mean of the dropout samples for prediction, SML- SqueezeDet uses the full network instead (see section 3. These predictions and their corresponding dropout samples are matched based on the respective anchor. The dropout samples are summarized by their means and variances.
To assess model performance, we report the mean intersection over union (mloU) and RMSE (in pixel space) between predicted bounding boxes and matched ground truths. The quality of the un- certainty estimates measured by (coordinate-wise NLL, ECE and Wasserstein distance. Table 2 shows summary of results train and test data. The results for NLI., ECE and WS have been

---

Under review as a conference paper at ICLR 2021
averaged across the 4 regression coordinates. SeueezeDet and AMML-SquezzeDer show comparable regression results, with slight advantages for SML-SqueezeDet on test data. Considering uncer- tainties quality, find substantial advantages for AMLL-qquezeDee across all evaluation measures. These findings resemble those on the UCI regression datasets and indicate that the second-moment loss works well modern application-scale network.
Regression performance and uncertainty quality of SqueezeDet-type networks KITTI test data. SML-trained SqueezeDet (ours) is compared with the default SqueezeDet that uses one- layer dropout to estimate uncertainties. The measures of NLL, ECE and WS are aggregated along their respective four dimensions, for details see appendix
Table 2:
B. and Table 6 therein.
measure
SqueezeDet
SML-SqueezeDet
SqueezeDet
SML- AML--qquezeDett
train
test
mloU (1)
0.816
0.812
0.738
0.744
RMSE (4)
6.418
6,862
18.225
17.492
NLL (1)
20.746
3,916
98,807
17.875
ECE (4)
0.996
0,554
1.198
0.834
WS (4)
2.487
0.874
4.587
1.734
CONCLUSION
We approach dropout-based uncertainty quantification from new direction: sub-networks ex- plicitly not encouraged to model the data mean, they capture data-inherent uncertainties and po- tential fitting residuals of the full network instead. Technically, this is realized by an additional loss term that accompanies the standard regression objective: the second-moment loss. Our loss enables stable training. Training complexity and runtime behavior inference are comparable to MC dropout. Task performances and uncertainty qualities of these models are on par with (para- metric) deep ensembles, the widely used state-of-the-art for uncertainty quantification. However, unlike deep ensembles, use single networks. In practice, this might allow to reduce training ef- fort significantly compared to deep ensembles, especially for application-scat single network requires only fraction of the storage of deep ensemble, making models with competitive uncertainties more accessible for mobile or embedded applications.
networks. Moreover,
An extensive study of uncertainties under data shift revealed advantages of SML-trained models compared to deep ensembles: while both methods on average provide comparable results, we find a higher stability across variety of datasets and data shifts. With respect worst-case uncertainties SML-trained networks are by large margin better than deep ensembles. A quite relevant finding for safety-critical applications like automated driving or medical diagnosis where (even rarely occur- ring) inadequate uncertainty estimates might lead to injuries and damage. Technically, we attribute this gain in stability to our network-basse approach: like MC dropout, we integrate uncertainty estimates into the very structure of the network, rendering it more robust towards unseen inputs than a parameter estimate.
Moreover, the second-momen moment loss can serve general drop-in replacement for MC dropout on regression tasks. For already trained MC dropout models, post-training with the aecond-moment loss might suffice to improve on uncertainty quality. As outlook, our first such post-training experiments on UCI datasets are encouraging. Another interesting variant is the combination of SML with last-layer dropout as it enables sampling-fie free inference (Postels et al., 2019). Preliminary experiments on UCI datasets show clearly improved uncertainties qualities compared to standard MC-LL.. A potentially interesting avenue for near-real time applications.
The simple additive structure of the second-moment loss makes applicable to a variety of opti- mization objectives. For classification, might be able to construct non-maramttric counterpart to prior networks (Malinin & Gales) 2018). Taking a step back, we demonstrated an easily feasible approach influence and train sub-network distributions. This could be promising avenue, for distribution matching but also for theoretical investigations.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07877 077775 2017.
Wasserstein GAN. arXiv preprint
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Long-term on-board prediction of people in traffic scenes under uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4194-42000 2018.
Christopher M Bishop. Puttern recognition and machine learning. Springer, 2006.
David Blei, Michael Jordan, al. Variational inference for Dirichlet process mixtures. Bayesian analysis, 1(1):121-143, 2006.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05442: 2015.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint
2016.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htLhht:pp//achhie
ics.uci. mici.edu/ml
Andrew YK Foong, Yingzhen Li, José Miguel Hermández-Lobato and Richard E Turner. 'in- between uncertainty in Bayesian neural networks. arXiv preprint 19Xiv:1906.1133: 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per-
spective. arXiv preprint arXiv:
2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050-1059, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall, Concrete dropout. In Advances Neural Information Processing Systems, pp. 3581-3590, 2017.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In Conference Computer Vision and Pattern Recognition (CVPR), 2012.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. vol- ume 15 of Proceedings of Machine Learning Research, pp. 315-323, Fort Lauderdale, FL., USA, 11-13 Apr 2011. JMLR Workshop and Conference Proceedings.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation.
Journal of the American Statistical Association, 102(477):359-377 2007.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Fredrik K Gustafsson, Martin Danelljan, and Thomas Schon. Evaluating scalable Bayesian deep learning methods for robust computer vision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 318-319, 2020.
David Hall, Feras Dayoub, John Skinner, Haoyang Zhang, Dimity Miller, Peter Corke, Gustavo Carneiro, Anelia Angelova, and Niko Sünderhauf. Probabilistic object detection: Definition and evaluation. In The IEEE Winter Conference on Applications of Computer Vision, pp. 1031-1040, 2020.
Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang. Bounding box regression with uncertainty for accurate object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2888-2897, 2019.

---

Under review as a conference paper at ICLR 2021
Tom Heskes. Practical confidence and prediction intervals. In Advances in Neural Information Processing Systems, pp. 176-182, 1997.
Eddy 1lg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. Uncertainty estimates and multi-hypotheses networks for optical flow. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 652-667, 2018.
Tomoharu Twata and Zoubin Ghahramani. Improving output uncertainty estimation and generaliza- tion in deep learning via neural network Gaussian processes. arXiv preprint arXiv: /007::777705000 05922, 2017.
Alex Kendall and Yarin Gal. What uncertainties do need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, pp. 5574-5584, 2017.
Abhas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F Atiya. Comprehensive review of neural baaed prediction intervals and new advances. IEEE Transactions Neural Networks, 22(9): 1341-135661 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In 35th International Conference on Machine Learning, ICML 2018, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
HJ Landau. Sampling, data transmission, and the Nyquist rate. Proceedings of the IEEE, 55(10): 1701-1706, 1967.
Siqi Liu, Sidong Liu, Weidong Cai, Sonia Pujol, Ron Kikinis, and Dagan Feng. Early diagno- sis of alzheimer's 's disease with deep learning. In 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), pp. 1015-1018. IEEE, 2014.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhaaff. Dropout sampling for bust object detection in open-set conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-7. IEEE, 2018.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, volume 2015, pp. 2901, 2015.
David A Nix and Andreas S Weigend. Estimating the mean and variance of the target probability dis- tribution. In Proceedings of IEEE International Conference Neural Networks 1994, volume 1, pp. 55-60. 60. IEEE, 1994.
Dean A Pomerleau. Alvinn: An autonomous land vehicle a neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2931-2940, 2019.
Xin Qiu, Elliot Meyerson, and Risto Miikkulainen. Quantifying pridt-prediction uncertainty in
neural networks via residual estimation with i/o kemell. arXiv preprint arXiv:1906.000 00588, 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. ICLR, 2018.
10

---

Under review as a conference paper at ICLR 2021
R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at at-risk for general loss distributions. Journal of Banking Finance, 26(7):1443-144 1471, 2002.
Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. A metric for distributions with applications to image databases. In Proceedings of the Sixth International Conference on Computer Vision, ICCV '98, pp. 59, USA, 1998.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Scul- ley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 13969-13980, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Jourmal of Machine Learning Research, 15())1122-1198, 2014.
Michael A Stephens. EDF statistics for goodness of fit and some comparisons. Journal of the American Statistical Association, 69(347):70-7377, 1974.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, pp. 835-851. Springer, 2016.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pp. 370-378, 2016.
Sascha Wirges, Marcel Reith-Braun, Martin Lauer, and Christoph Stiller. Capturing object detection uncertainty in multi-layer grid maps. arXiv preprint
1284, 2019.
Bichen Wu, Forrest landola, Peter H Jin, and Kurt Keutzer. SqueezeDet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 129-137, 2017.
Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3d voxel patterns for object category recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1903-1911, 2015.

---

Under review as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIAL
This part accompanies our paper Second-Moment Loss: A Novel Regression Objective for Improved Uncertainties" and provides further in-depth information. In Section A we provide both theoretical and numerical insight into the resulting uncertainties of our loss modification. Large parts of the empirical evaluation on UCI can be found in section B including details on the setup, data splits as well as further uncertainty measures. Details for use and modification of SqueezeDet are located in sub-section B.5 As the second-moment loss couples to the usual MSE regression loss via a hyper- syper-parameter 3 test various values in section C
finding strong correlation between result and parameter. We close with a discussion on the relation between uncertainty measures and their respective sensitivity in section D
A MECHANICS OF THE SECOND-MOMENT LOSS
We analytically study the optimization landscape evoked by the second-moment loss in A.1 This analysis provides building blocks to better understand the composition of the SML-uncertainties as detailed in the remainder of this section.
A.1 ANALYTICAL PROPERTIES OF THE SECOND-MOMENT LOSS
In the following, we look closer at the behaviour of the second moment loss with respect to aleatoric uncertainty. For this, we assume that the residuals, compare eq. (I) are given by Gaussian distri- bution with, for simplicity, JARes. = 0 and Res.. 1. We want to determine the resulting loss for the Luml term in eq. [ governing the uncertainty estimation of the model. It depends on the underling distribution of the effective MC dropout distribution, which me model as N (Harop- dropp) such that:
Limll dy1dyy (1a1) - lual)² P1 (91) P2 (22)),
(2)
where and P2 are the Gaussian distributions discussed above. After some calculation this yields:
1 Parop
Mdrrop
Tm drop exp
Mirrop Frf
+ drop + Perrop + 1 (3)
drop
which is visualized in Fig. 4 The two global minima can be found for drrup 0 and Murop model randomized residual y1 these minima do not reach zero. We
+ 1//2/m 2//T. However, find that it is favourable to move /dmp away from the network prediction of fARe. 0, the mean of the underlying data distribution. But, this is only the case as long the inherent uncertainty in the dropout distribution can be brought below ddrp < 2/T,, which is still smaller than the uncertainty of TRas. assumed within the training data distribution. Otherwise, is more favourable have Mutro = fese.. = O. In the following sections we investigate the practical implications of this finding. For instance, as detailed in section B.2 the highly oscillating or noisy toy model experiments clearly exhibited the type of separation discussed here. Decomposing the uncertainty for the UCI datasets in section A.2] on the other hand, showed mixed behaviour with indications bi-modal shifts in ddro as well as improved values of drop-
We already showed the effect of this bi-modality in Fig. [ at the beginning of the paper, where various sub-networks where sampled. Clearly visible is stronger variation between the networks compared to MC, but also a concentration around the two possible minima. While this Fig. provides good visual estimate of drop the total uncertainty Twat would additionally contain the systematic shift fe - (- Given the roughly symmetric distribution of the sub-networks we can expect it to be comparatively small.
A.2 COMPOSITION OF THE UNCERTAINTY ESTIMATE
The uncertainty estimate of the second-momen loss is comprised of two parts: Frotal drrop + Iff - (fa)l Fig. revells that drop contributes to more than 80% of Fental for the three presented datasets and for all applied data splits. A highly similar behavior can be observed for all other
12

---

Under review as a conference paper at ICLR 2021
Laand
16
14
ddrop
Parop
Figure 4: Shown is the value of the loss component L2 as given by eq. (3) over Mdrop and dropp describing the implicit dropout ensemble. The blue line shows the position of the minima of L2 for fixed values of drop- Clearly visible are the global minima at drop 0 and the bifurcation at drop 2/T.
datasets. The analytical consideration in appendix A.1 suggests that for cases without data-inherent uncertainty the SML provides no incentive for [ffe (fa) > O. The same holds true in the presence of aleatoric uncertainty as long drop is comparably large. For aleatoric uncertainty and small drep larger life - (fg) are favorable. However, as our loss is radial symmetric, all directions are equivalent and initialization and randomness determine the direction of the spread [foe (fa)l for each individual sub- netwokk. This symmetry leads again to a small averaged ff (fa)l- drop on the contrary describes the width of bi-modal set sub-networks in these cases.
A.3 DETAILED ANALYSIS OF THE TWO LOSS COMPONENTS
deeper look into the structure of the second-moment loss possible we investigate behaviour component-wise, To clarify the results presented in Fig. 6 we recall the loss structure as
L L=
L2
E
a + ((b,) |a,1)²1
with a, fo(z) y/ and b, = fa(x) - (xx)) Histograms of the a, (Fig. 6 first column) enable detailed view network performance. The uncertainty quality of the networks by studying the L2 loss term more closely, namely by visualizing histograms of [6:] - |asl (fourth column). The second and third column zoom into L2 and show histograms of the b, and scatter plots of (b,,a,), respectively. Only test datasets are visualized and as applied 90 10 train-test splits, this explains the low resolution of some histograms in the first column. All quantities involving b; require the sampling of sub-networks. We draw 200 sub-networks. This sampling procedure explains the higher plot resolutions columns two four.
be judged
Qualitatively, we observe that both the 's and by's are centered around zero which hints at suc- cessful optimization of regression performance and of uncertainty quality. Details on how the op- timization is technically realistic, can be gained from the scatter plots. They show three qualitative shapes: a 'cross' (first row), a 'line" (second row) and 'blob' (third and fourth row). The 'cross' occurs for toy-hf and reflects the bi-modal sub-network structure found in Fig. T For in-detail discussion of the uni- and bi-modality of the second-moment loss landscape see A. A 'line' shape reflects that all sub-networks occupy the same minimum given bi-modal case. Following appendix A.I a 'blob' indicates a uni-modal case that might be evoked by large standard deviations Odrop-
B EXTENSION TO THE EMPIRICAL STUDY
Accompanying to the evaluation sketched in the body of the paper, section|4 we provide more details on the setup, used benchmarks and measures in the following sub-section. Further information on the toy dataset experiments can be found in section B.2 The same holds for the UCI experiments in section B.3 which we extend by the measures skipped in the main text, and include description on the used label splits. A close look the predicted UCI uncertainties (per method) is given via
13

---

Under review as a conference paper at ICLR 2021
1000
1000
s0/ppe
600
600
400
400
0
fraction_de
fraction
2un
Iran
800
800
600
600
400
400
200
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
fraction_dre dirppouttsts
fraction_ spread
Ite temilace
1500
1500
500
500
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.6
1.0
pocctionddrapout
fraction_spread
Figure 5: The second-moment loss induces uncertainties dioua = drop fo (fa)l. The relative contribution of both components ("fraction ("fraction_dropuut.tee prrppoutssdd" ssdd", "fraction_spreat is shown for three ex- emplary datasets (top: toy-noise, middle: superconduct, bottom: protein) and i.i.d. (train: blue, test: orange) well as non-i.i.d. data splits (test-label: red, test-pca: yellow).
scatter plots in section B.4 Details the SML version of the SqueezedDet are found in the last sub-section.
B. EXPERIMENTAL SETUP
The experimental setup used for the toy data and UCI experiments is presented in three parts: the benchmark approaches we compare with, the evaluation measures we apply to quantify uncertainty, and description of the neural networks and training procedures we employ.
Benchmark approaches We compare dropout networks trained with the SML to archetypes of uncertainty modelling, namely approximate Bayesian techniques, parametric uncertainty, and sembling approaches. From the first group, we pick MC dropout (abbreviated as MC) and its variant last-layer MC dropout (MC-LL). While these dropout approaches integrate uncertainty estimation into the very structure of the network, parametric approaches model the variance directly as the out- put of the neural network (Nix Weigend 1994). Such networks typically output mean and vari- ance of Gaussian distribution (p, and are trained by likelihood maximization. This approach denoted as PU for parametric uncertainty. Ensembles of PU-networks (Lakshminarayanee et al.) 2017), referred to deep ensembles, pose widely used state-of-the-art method for uncertainty estimation (Snock et al., 2019). (Kendall Gal 2017) consider drawing multiple dropout samples from parametric uncertainty model and aggregating multiple predictions for f and o. We denote this approach PU-MC. Moreover, we consider ensembles non con-paramttric standard networks. We refer the latter ones DEs while call those using PU PU-DEs. All considered types of net-
14

---

Under review as a conference paper at ICLR 2021
20
800
600
15
ITC P0. mr.
20
200
200
ress_residull
mm_no_mc_aaf
805 07 000_000,MC.0000 ____m_______m_m_ mo_mc atw.cit.sttno.m. nnn.mm
50
TTC PP me dM
30
1000
20
500
10
residual
12500
mi.F m. 3P
10000
60
7500
5000
20
-2
-2
-2
0
-2
0
dIff
50000
60000
130
50000
PO. mc dP
20000
20000
-2
0
prod residual
mc no mc dilf
08. no mc dif
abs dff mc no mc abs ciff at no mc
Figure 6: Visualisation of the components (columns) of the second-moment loss for selected test datasets (rows). The prediction residual f0( - y; (first column), model spread fg(2;) - fa( (second column), scatter plot of both quantities (third column) and f (z:) fe(xx)]-- (x,) (fourth column) are shown. The chosen datasets from top to bottom are: toy-hf, wine-red, power, california.
works provide estimates (mi oz) where is obtained either analytically (PU), by sampling (MC, MC-LL, SML) or as an ensemble aggregate (DE, PU-DE).
Evaluation measures In all experiments evaluate both regression performance and uncer- tainty quality. Regression performance is quantified by the root-mean-square error (RMSE),
((INN 1,(4-14 (Bishop 2006). Another established metric in the uncertainty community
the (Gaussian) negative log-likelihood (NIL), 1/N (log a, + (14 y1 22)2/(20f) + c). a hy- brid between performance and uncertainty measure (Gneiting & Raitery 2007) p. see appendix D.2 for discussion ()) The expected calibration error (ECE, Kuleshov et al. (2018) in contrast is not bi- ased towards well- rell-performing models and that sense a pure uncertainty measure. reads ECE Σe 1/B] for B equally spaced bins in quantile space and βy = (ra lg, S q(r.) < 9/1)l/N empirical frequency of data points falling into such bin. The normalized prediction residuals ry are defined r; (μ, y:) y:)/or, Further, is the edi of the standard normal // 1) and [g) (i+1) are equally spaced intervals on [0, 1]. i.e. qi = (j 1)/ Additionally, we propose to consider the Wasserstein distance of normalized prediction residuals (WS). The Wasserstein distance (Villani 2008), also known as earth mover's distance (Rubner et al. 1998), is transport based measure de- noted by (dws) between probability densities, with Wasserstein GANs (Arjovsky et al. 2017) as its most prominent application in MI. For ideally calibrated uncertainties, we expect y: ~ N(4,, 7;) and therefore r; (0, 1). Thus we use dws( (rr |,,, 1)) behavior. As ECE, this is pure uncertainty measure. However, it does not use binning and can
measure deviations from this ideal
"Throughout the paper, ignore the constant log 2 of the NL.I..
15

---

Under review as a conference paper at ICLR 2021
therefore resolves deviations all scales. For example, two strongly ill-calibrated uncertainties (f1, 12 > 1, r1 < r2) would result (almost) identical ECE values while WS would resolve this difference in magnitude.
Technical details All investigated neural networks have the same architecture, 2 hidden layers of width 50, and ReLu activations (Glorot et al.) 2011). I). For all dropout-based methods (MC, MC-LL, SML) we set the drop rate p 0.1. Like MC, SML-trained networks apply Bernoulli dropout to all hidden activations. In the case of MC-LL the dropout is only applied to the last hidden layer. For employ networks. For PE networks,
normalize the value using softplus (Glorot et al. 2011) and optimzie the NL. instead of the MSE. For the optimization of all NNs we use the ADAM-opting Mootimiiee (Kingma & Ba 2014) with learning rate of 0.001. For "california', , the learning rate is reduced to 0.0001 as training of PU and PU-DE is unstable using the standard setup. Additionally, we apply standard normalization the input and output features of all datasets to enable better comparability.
ensemble methods (DE, DE-PU)
Number of epochs trained and amount of cross validation differs by the training-set size. We cate- gorize the Toy and UCI datasets as follows: small datasets {toy-hf, yacht, diabetes, boston, energy, concrete, wine-red| }. large datasets (toy-noise, abalone, power, naval, california, superconduct, pro- tein and very large datasets {year) }. For small datasets, NNs are trained for 1, 1,000 epochs using mini-batches of size 100. All results are 10-fold cross validated. For large datasets, we train for 150 epochs and apply 5-fold cross validation. We keep this large-dataset setting for the very large "year' dataset but increase mini-batch size to 500.
All experiments are conducted on Core Intel (R) Xeon (R) Gold 6126 CPUs and NVidia Tesla V100 GPUs. Conducting the described experiments with cross validation on one CPU takes 6 h for toy data, 80 / for UCI regression. And 8 for object regression the GPU.
For SML turns out that as long as on the optimization result, appendix for details. Larger 8-values can however favour uncer- tainty optimization at an expanse of task performance. Throughout the body of the paper we use a conservative value of 0.5.
< the actual value of 3 has only a limited influence
B.2 TOY DATASETS: SYSTEMATIC EVALUATION
sampled from fnoise(1) NO exp(-0.02220 for & E [-15, 15] and (=r(2) = 0.25 x2 00.01 xه x5 exp(-(x+1)2/ 220) sin x) for æ E 15, 20). respectively. Standard normalization is applied input and output values. The aggregated measures of the seperate uncertainty methods achieved on these datasets are given in table 3
The toy-noise and toy-hf datasets
Table 3: Regression performance and uncertainty quality of networks with different uncertainty mechanisms. All scores are calculated on the test of toy-hf and toy-noise, respectively.
measure
dataset
MC
MC-LL
Ours
PU
PU-DE
DE
RMSE (L)
toy-hf
0.69
0.69
0.69
0.71
0.70
0.66
NLL (L)
toy-hf
51.22
65.34
-0.06
-0.08
-0.08
46.31
ECE (L)
toy-hf
1.48
1.56
0.50
0.60
0.61
1.49
WS (L)
toy-hf
6.13
8.17
0.26
0.26
0.26
6.14
RMSE (L)
toy-noise
1.00
1.00
1.01
1.00
1.00
1.00
NLL (L)
toy-noise
9.9 x 1
2.3 x 10
0.23
0.39
-0.39
7942.37
ECE (L)
toy-noise
1.76
1.75
0.19
0.10
0.09
1.65
WS (L)
toy-noise
2.6 x 10
1.7 x 10P
0.16
0.04
0.04
72.85
B.3
UCI DATASETS: RMSES. NLLS AND SYSTEMATIC EVALUATION
This sub-section provides further details UCI experiments covering: overview the datasets and splits used for the data-shift studies, further uncertainty measure evaluations (RMSE, NL.., WS), and close with discussion the weaker SML results.
16

---

Under review as a conference paper at ICLR 2021
Datasets and data splits For the UCI regression data, Table 4 provides details dataset refer- ences, preprocessing and basic statistics. Extrapolation and interpolation data-shifts are, technically, introduced by applying non-ii.d. (independent and identically distributed) data splits. Natural can- didates for such non-i.i.d. splits are splits along the main directions of data in input and output space, respectively. Here, consider 1D) regression tasks. Therefore, output- based splits simply done on scalar label variable (see Fig. split, see, e.g., Foong al. (2019)) In input space, the first component of a principal component analysis (PCA) provides natural direction (see Fig. 7 left), The actual PCA-split is then based on projections of the data points onto this first PCA-component" Splittin data along such an direction in input or output space in e.g. 10 equally large chunks, creates 2 outer data chunks and 8 inner data chunks. Training a model on of these chunks such that the remaining chunk for evaluation an inner chunk is called data interpolation. If the remaining test chunk is an outer chunk, it is data extrapolation. We introduce this distinction as extrapolation is expected to be considerably more difficult than 'brigging' between feature combinations that were seen during training.
7 right), We call such a split label-based (for comparable
Table Details on UCI regression datasets. Ground truth (gt) partially pre-processed match the ID regression setup.
dataset
# features
# datapoints
reference
remarks
yacht
6
308
UCI
diabetes
7
442
sklearn
boston
13
506
skleam
energy
8
768
UCI
two gt labels: only "cooling load" gt is used
concrete
8
1030
UCI
wine-red
11
1599
UCI
abalone
4176
UCI
1st feature ignored, also called "kin8nm"
power
4
9568
UCI
naval
11934
UCI
two gt labels: only "turbine" gt is used
california
8
20640
skleam
superconduct
81
21263
UCI
protein
9
45730
UCI
year
90
515345
UCI
NN
input space
output space
input space
output space
Figure 7: Scheme of two non-i.i.d. splits: PCA-based split in input space (left) and label-based split in output space (right). While datasets appear to be convex here, they are (most likely) not in reality.
Regression quality First, we consider regression performance (see the first and second panel of Fig. 8 Averaging the RMSE values over the considered 13 datasets ('mean' column) yields almost identical results for all uncertainty methods. The only exceptions pose PU, PU-MC and PU-DE with
Note that these projections only considered for data splitting, they are not used for model training.
17

---

Under review as a conference paper at ICLR 2021
larger train data RMSEs which could be due to NLL optimization favoring to adapt variance rather than mean. However, this regularizing NLL-training comes along with smaller generalization gap, leading to competitive test RMSEs. Next, we investigate model performance under data shift, visualized in the third to sixth panel of Fig. 8 Again, regression quality is comparable between all methods. As expected, performances under data shift worse compared to those i.i.d. test sets.
Negative log-likelihoods For NLL. results are less balanced compared to RMSE (see Fig. PU-DE, PU-MC and the SML-trained network reach comparably small average values on the test set, followed by MC and DE. The average NLL values of MC-LL and PU are above the upper plot limit indicating a rather weak stability of these methods. On PCA-interpolate and PCA-extrapolate test sets, again PU-DE, PU-MC and SML-trained networks perform best. On label-interpolate and label-extrapolate test sets, however, SML-trained networks take the first place with a large margin. The mean NLL values of most other approaches are above the upper plot limit. Note that median results are not as widely spread and PU-DE and SMI. perform comparably well. These qualitative differences between mean and median behavior indicate that most methods perform poorly "once in a while'. A noteworthy observation as stability across a variety of data shifts and datasets can be seen crucial requirement for an uncertainty method. SML-based models yield the highest stability in that sense w.r.t. NLL.
Wasserstein distances Studying Wasserstein distances, we again observe equally strong results for PU-DE, PU-MC and SML on train and test data (see first two columns of Fig. L0 PU in contrast possesses a large generalization gap thus yielding weak test set performances. MC, MC-LL, and DE behave consistently weak on train and test sets with MC-LL even falling out of plot range. Under data shift (bottom panel of Fig. [0), the picture remains similar. PU-DE, PU-MC and SML are in the lead and comparably strong for pca-bacc pca-based data shift. On label-based data shifts SML outperforms all other methods by significant margin. As for NLL., we find these mean values of PU-DE and PU-MC to be significantly above the respective median values indicating again weaknesses in the stability of parametric ensembles.
Slight overestimation of small uncertainties for SML. The second-moment loss yields weak results on 'yacht', 'energy" and 'naval', the three easiest datasets judging by test set RMSE, com- pare [5 On these datasets neither aleatoric uncertainty nor modelling residuals play mayor role. In such cases, the second-moment loss seems to cause slightly overshooting uncertainty esti- mates (compare edges Fig. M for a visual clue), likely due to its sub-network 'repulsion'. Back- propagating not only through Já but also through the full network fe in Lami might mitigate this effect. In practice, slight overestimation of small uncertainties might be acceptable. In contrast, our method performs consistently strong on all more challenging datasets ('california', 'superconduct', 'protein', 'year'). A beneficial characteristic for virtually any real-world task.
B.4 RESIDUAL-UNCERTAAIN SCATTER PLOTS
Visual inspection of uncertainties can be helpful to understand their qualitative behaviour. We scatter model residuals 14, (respective x-axis in Fig. 12 against model uncertainties as (resp. y-axis in Fig. 12 For a hypothetical ideal uncertainty mechanism, we expect (yi- /4,) (O0,, dx), i.e. model residuals following the predictive uncertainty distribution. More concretely, 68.3% of all (y:: p1)) would lie within the respective interval [-o;, and 99.7% of all (y - p4) within 3,3 o.). Fig. IT visualizes this hypothetical ideal. Geometrically, the described Gaussian properties imply that 99.7% of all scatter points, e.g., in Fig. [2] should lie above the blue 30 lines and 68.3% of them above the yellow la lines. For 'abalone' test data (third row of Fig. [2]. PU and SML qualitatively fulfil this requirement while MC and DE tend to underestimate uncertainties. This finding is in accordance with our systematic evaluation. For toy-noise, abalone and superconduct, we qualitatively find PU, PU-DE and SML-trained networks to provide more realistic uncertainties compared to MC, MC-LL and DE (see Fig. 12 The naval dataset poses an exception in this regard as all uncertainty methods lead to comparably convincing uncertainty estimates. The small test RMSEs of all methods naval (see appendix 8) indicate relatively small aleatoric uncertainties and model residuals. Epistemic uncertainty might thus be a key driving factor and coherently MC, MC-LL and DE perform well.
18

---

Under review as a conference paper at ICLR 2021
1.i.d. data split
label-based data shift test (extrap.)
pca-based data shift
(Interp.)
(extrap.)
test (interp.)
1.5
xx xx xx
XxxX
0.5
x
XXX
0.0
1000
Figure 8: Root-mean-squarc errors (RMSEs) of different uncertainty methods under i.i.d. conditions (first and second panel) and under various kinds of data shift (third to sixth panel, see text for details). SML ('ours') ) compared
benchmark approaches. Each blue cross is the mean over RMSE values from 13 UCI regression datasets. Orange line markers indicate median values. The gray vertical bars reach from the 25% quantile (bottom horizontal line) to the 75% quantile (top horizontal line).
i.i.d. data split
label-based data shift
pca-based data shift
train
test
test (extrap.)
test (interp.)
test (extrap.)
test (interp.)
40
30
10
x
x
x
1
X x
Xx
Figure 9: Negative logllikelshoods (NLLs) different uncertainty methods under i.i.d. conditions (first and second panel) and under various kinds of data shift (third to sixth panel, see text for details). SML ('ours') ) is compared from 13 UCI regression datasets. Orange line markers indicate median values. The gray vertical bars reach from the 25% quantile (bottom horizontal line) to the 75% quantile (top horizontal line).
benchmark approaches. Each blue cross is the mean over NLL values
19

---

Under review as a conference paper at ICLR 2021
Table 5: Regression performance and uncertainty quality of networks with different uncertainty mechanisms. The scores are calculated on the test sets 13 UCI datasets.
measure
dataset
DE
MC
MC-LL
Ours
PU-MC
PU-DE
PU
RMSE (L)
yacht
0.04
0.07
0.06
0.08
0.08
0.05
0.06
(L)
yacht
2.76
2.95
2.54
3.27
0.39
ECE (L)
yacht
0.66
0.91
0.8
1.09
1.06
1.0
0.96
WS (
yacht
0.4
0.43
0.36
0.53
0.52
0.48
1.16
RMSE (L)
diabetes
0.92
0.84
0.94
0.87
0.79
0.81
0.82
NLL (U)
diabetes
4.5
7.34
47.46
3.19
2.43
7.55
4268.61
ECE (L)
diabetes
0.91
1.1
1.48
0.9
0.91
1.03
1.27
(
diabctes
1.6
2.47
6.9
1.49
1.29
2.09
21.42
RMSE (L)
hoston
0.33
0.33
0.34
0.32
0.31
0.34
0.36
NLL(U)
boston
1.89
12.41
0.08
0.27
2.88
123.59
ECE (L)
0.66
1.38
WS (L)
boston
1.53
1.4
3.29
0.62
0.72
1.43
7.93
RMSE (L)
energy
0.08
0.12
NLL
energy
-0.7
2.14
1.45
2.03
4.94
ECE (L)
energy
0.53
0.49
0.52
0.63
0.69
0.51
1.02
WS (L)
energy
0.71
0.25
0.48
0.32
0.35
0.43
2.23
RMSE (L)
concrete
0.25
0.25
0.26
0.25
0.24
0.26
0.27
NL)
concrete
4.67
-0.05
5.32
0.89
-0.6
0.32
80.28
ECE (
concrete
0.66
0.52
0.73
0.39
0.4
0.54
0.79
WS
L.43
0.63
1.78
0.24
0.22
0.74
3.4
RMSE (L)
wine-red
0.77
0.85
0.78
0.78
0.76
0.8
NLL
wine-red
3.56
20.49
6.35
4.11
1968347.99
ECE (L)
wine-red
0.51
0.74
0.99
0.58
0.53
0.45
0.77
WS (L)
wine-red
0.73
1.43
3.74
0.58
0.78
0.82
86.81
RMSE
abalone
0.64
0.65
0.65
0.65
0.64
0.64
0.64
NLL (L)
abalone
29.61
19.36
67.22
0.25
-0.1
0.07
0.02
ECE (L)
abalone
1.31
1.31
1.51
0.43
0.3
0.27
0.26
())
abalone
4.05
7.53
0.47
0.17
0.17
0.18
RMSE (L)
power
0.22
0.23
0.23
0.22
0.23
0.22
0.22
NLL
power
13.17
3.32
6.48
0.84
-1.0
-1.0
0.98
ECE
power
1.06
0.93
1.03
0.21
0.15
0.12
(L)
power
2.91
1.75
2.35
0.25
0.15
0.09
0.08
RMSE
0.13
0.08
0.2
0.18
0.18
NL)
naval
-281
1.65
0.71
1.68
1.36
1.82
2.28
ECE (L)
naval
0.97
0.54
0.66
0.9
0.64
1.03
0.75
(
naval
0.35
0.49
1.04
0.5
0.4
0.53
0.46
RMSE (L)
california
0.45
0.46
0.46
0.46
0.47
0.53
0.53
NL)
california
37.4
5.09
21.72
0.18
0.46
0.52
0.47
ECE (
california
0.92
1.22
0.23
0.25
0.22
WS (
california
5.07
1.85
3.94
0.27
0.2
0.15
0.15
RMSE (L)
superconduct
0.29
0.31
0.3
0.3
0.34
0.32
0.34
NLL(U)
superconduct
2.43
1.67
8.44
1.02
1124
0.56
ECE (L)
superconduct
0.54
0.61
0.78
0.2
0.26
0.21
0.17
())
superconduct
1.16
1.14
2.1
0.16
0.16
0.14
0.24
RMSE (L)
protein
0.59
0.61
0.61
0.62
0.67
0.67
0.69
NL)
protein
4.38
3.7
13.14
0.07
0.07
0.12
0.14
ECR (
protein
0.68
0.8
1.0
0.25
0.31
0.3
0.2
protein
1.56
0.21
0.19
0.15
RMSE (L)
year
0.77
0.79
0.81
0.81
0.79
0.78
0.79
NLL (L)
year
11.63
15.42
1240419.1
0.15
00.02
0.17
ECE (L)
year
1.0
1.26
1.42
0.25
0.26
0.25
WS (L)
year
2.49
3.45
852087.22
2.
0.24
0.18
0.16
0.19
20

---

Under review as a conference paper at ICLR 2021
i.idd data split
label-based data shift
pca-based data shift
train
test
test (extrap.)
test (interp.)
test (extrap.)
test (interp.)
10
Wasserstein distance
x
X
* Ix
2-
X
H+0
xxxx
po nu
Figure 10: Wasserstein distances of different uncertainty methods under i.idd conditions (first and second panel) and under various kinds of data shift (third to sixth panel, see text for details). SML ('ours') is compared to benchmark approaches. Each blue cross is mean over WS values from 13 UCI regression datasets. Orange line markers indicate median values. The gray vertical bars reach from the 25% quantile (bottom horizontal line) to the 75% quantile (top horizontal line).
2.00
1.75
prod 3t
10
-4
-2
prod_ resiual
Figure 11: Prediction residuals (x-axis) and predictive uncertainty (y-axis) for a hypothetical idcal uncertainty mechanism. The Gaussian errors matched by Gaussian uncertainty predictions the exact same scale. 68.3% of all uncertainty estimates (plot points) lie above the orange la-lines and 99.7% of them above the blue -onlines.
The hypothetical ideal residual-uncertainne scatter plot we use Fig. [1 is generated as follows: We draw 3000 standard deviations a, (0, 2) and sample residuals ri from the respective normal distributions, ~ (O, az). The pairs (ri, ai) are visualized. By construction, uncertainty esti- mates now ideally match residuals distributional sense. But even in this perfect case, Pearson correlation between uncertainty estimates and absolute residuals is only approximately 55%.
B.5 DETAILS ON SML- SML-SQUEEZEDET
Architecture SqueezeDet (Wu et al. 2017) takes an input image and predicts three quantities: (i) 2D bounding boxes for detected objects (formalized as 4D regression task), (ii) confidence score for each predicted bounding box and (iii) the class of each detection. Its architecture as follows: First, sequence of convolutional layers extracts features from the input image. Next, dropout with drop rate of p = 0.5 is applied to the final feature representations. Another convolu- tional layer, the ConvDet layer, finally estimates prediction candidates. In more detail, SqueezeDet
21

---

Under review as a conference paper at ICLR 2021
53
Prediction residuals (respective x-axis) and predictive uncertainty (respective y-axis) for different uncertainty mechanisms (columns) and datasets (rows). Fach light blue dot in each plot corresponds to one test data point. Realistic uncertainty estimates should lie mostly above the blue 3o-lines. The datasets toy-noise, naval, abalone and superconduct are shown, from top to bottom.
Figure 12:
predictions are based on so-called anchors, initial bounding boxes with prototypical shapes. The ConvDet layer computes for each such anchor confidence score, class scores and offsets to the initial position and shape. The final prediction outputs are obtained by applying suppression (NMS) procedure to the prediction candidates. The original loss of SqueezeDet is the sum of three terms. It reads LisquezzeDet = Lregres + Leonf + Lelass with the bounding box regres- sion loss Lregres. confidence-score loss Loonf and the object-classfficatin loss Lelass Our modifi- cation the learning objective is restricted to the L2 regression loss:
mon-maximum-
W ΣΣ
Lregres
Abbox Nobi
Iig [ Sx (ōytHk 58l52)2
(5) with (ōzzzk, Öyzzke Swigks 8hezk) and (6x igk Sy d 8hG (x being estimate and ground truth 13hs
(Sulijk 5w (ōhizt 5h.
expressed in coordinates relative to the k-th anchor at grid point (i. Tilde denotes that variable
is subject to dropout-induced randomness. See Wu et al. (2017) for descriptions of all other loss parameters. Applying the second-moment loss component-wen wise to this 4D regression problem yields
Abbox 2
Lregres,SMI.
Tijk
Nobj
k=1
(6xzzh (x2 (10zisk 8zigjK
2
(ōysşk )2(5))2 + (1506sH 8yijkl
18yt5k )1
(ōwizk 52 + (lowGjt SWIJA
18W35K Sw)
(ōhizk - G) + (1ōhesk - 5hegkl
18hish - 5n,
(6)
with deterministic network outputs (Saszae, öyisk, Suigge Shijk)-
22

---

Under review as a conference paper at ICLR 2021
Details data split and evaluation Our evaluation is based the KITTI dataset (Geiger al, 2012), which contains image sequences of driving scenes. We split the original KITTI training set into 3682 train and 3799 test images such that all images of given sequence or test set (see Xiang et al. (2015). Predicted bounding boxes are matched with ground truth using minimum-wett weight maximum bipartite matching. The edge weights of this bipartite graph given by the intersection over union (IoU) between ground truth and predicted bounding boxes.
either in the train
Table 6: Regression performance and uncertainty quality of SqueezeDet-type networks KITTI train/test data. SML-trained SqueezeDet (ours) is compared with the default SqueezeDet that uses one-layer dropout estimate uncertainties.
measure
SqueezeDet
SML-SqueezeDet
SqueezeDet
SML-SqueezeDet
train
test
mloU (1)
0.816
0.812
0.738
0.744
RMSE (4)
6.418
6.862
18.225
17.492
NLL, (4)
20.070
3,488
95.621
14.252
NLL,
6,064
1,453
15.376
3.709
NLL,
33,715
6,593
219.778
39,870
NLL
22.594
4.128
64.451
13.667
ECE, (4)
0.919
0.431
1.145
0.783
ECE,
0,955
0,520
1.140
0.775
ECE
1.025
0,512
1.296
0.915
ECE,
1.085
0.751
1.209
0.864
WS, (1)
2.616
0.675
5.342
1.942
WS, 9
2.201
0.822
3.555
1.373
WS
2.422
0.785
5,813
1.975
WSA
2.710
1.212
4.037
1.647
C STABILITY W.R.T. -APEE--AR B
Here, we analyze the impact of the SML parammeeet 3 on the uncertainty quality of accordingly trained models. For 3 0.1, 0.25, 0.5, 0.75, 0.9, we observe only relatively small differences in both ECE (see Fig. 13) and Wasserstein distance (see Fig. 14). B = 0.5 provides (by small margin) the best average test set performance both scores. However, the best-performing 3-value for individual dataset can vary.
Experiments with 3 > (not shown here) cause non-convergent training in many cases as primarily uncertainty quality is optimized at the expense of task performance. The opposite extreme case is = 0, i.e. network optimization without any dropout mechanism. Applying dropout at inference will therefore cause uncontrolled random fluctuations around the network prediction.
D IN-DEPTH INVESTIGATION OF UNCERTAINTY MEASURES
D.1 DEPENDENCIES BETWERN UNCERTAINTY MEASURES
All uncertainty-related measures (NLL,, ECE, Wasserstein distance) relate predicted uncertainties to actually occurring model residuals. Each of them putting emphasize on different aspects of the considered samples: NLL biased towards well-performing models, ECE measures deviations within quantile ranges, Wasserstein distance resolves distances between normalized residuals. The empirically observed dependencies between these uncertainty measures are visualized in Fig. [5] Additionally to Wasserstein distances, we consider Kolmogorov-Smimat (KS) distances (Stephens, 1974) on normalized residuals there. estimates a distance between the sample normalized residuals and standard Gaussian. Different from the Wasserstein distance, the KS-distance is
23

---

Under review as a conference paper at ICLR 2021
1.i.d. data split
label-based data shift
pca-based data shift
train
test
test (extrap.)
test (interp.)
test (extrap.)
test (interp.)
2.0
FX
IX
IXI
x
1.0
xx
1x0
122
11
XI
DX
|X
XX-1
FX1
X-X11
0.5
*
XX
1.88
128
1120
0.0
fano
(ro of ano
(gt ana
(Gc d0 ano
caaa tmo
Grosg tne
Grong wne
sar.e ano
cood omo
Gr000 uno
(CC00 tmo
coo. ana
de ono 10 Con
c2o-aa time
(t'
(co
de on
tno
Figure 13: Expected calibration emrors (ECEs) for SML-trained networks with hyper-parameters 3 = 0.1, 0.25, 0.5, 0.75, 0.9 under i.i.d. conditions (first and second panel) and under various kinds of data shift (third to sixth panel, see text for details). Each blue cross the mean over ECE values from 13 UCI regression datasets. Orange line markers indicate median values. The gray vertical bars reach from the 25% quantile (bottom horizontal line) to the 75% quantile (top horizontal line).
i.I.d. data split
label-based data shift
pca-based data shift test (interp.)
train
test
test (extrap.))
test (interp.)
test (extrap.)
Wasserstein distance
3-
2-
1-
x
K
DX
120
EX
10
IK
86
10X
13
1X
FX
13
1-X
11
0-
15tte cooo 15t (027
16 rosa seg
1920 - sem (' too
(Eeo somo
Paoo uso
lzze tno
Tice emo
0.25)
aom ip 0.75)
man 0.9)
0.1) (.2)
cose de 0225
0.25)
mon d0 9.75)
uon 18
sou 19
one o8
ane
Figure 14: Wasserstein distances for SML-trained networks with syper-parammters 3 0.1, 0.25, 0.5, 0.75, 0.9 under i.i.d. conditions (first and second panel) and under various kinds of data shift (third to sixth panel, see text for details). Each blue cross is the mean over WS values from 13 UCI regression datasets. Orange line markers indicate median values. The gray vertical bars reach from the 25% quantile (bottom horizontal line) to the 75% quantile (top horizontal line).
24

---

Under review as a conference paper at ICLR 2021
not rransport-based but determined by the largest distance between the empirical CDFs of the two samples. is therefore bounded to [0, 1] and unable to resolve differences between samples that strongly deviate from standard Gaussian one.
While all these scores are expectably correlated, noteworthy deviations from ideal correlation occur. Therefore, advocate for uncertainty evaluations based various measures to avoid overfitting to specific formalization of uncertainty.
The data splits in Fig. 15 color-coded as follows: train is green, test is blue, pea-interpolate is green-yellow, pea-extrapolate is orange-yellow, label-interpolate is red and label-extrapolate is light red. The mapping between uncertainty methods and plot markers reads: MC is 'diamond', MC-LL is 'thin diamond', DE is 'cross', PU is 'point', PU-DE is 'pentagon' and mooonttmmment loss is "square'. The data base of this visualization is toy-noise, toy-hf and 13 UCI regression datasets. Some Wasserstein distances lie above the x-axis cut-off and are thus not visualized.
D.2 DISCUSSION OF NLL AS A MEASURE OF UNCERTAINTY
Typically, DNNs using uncertainty are often evaluated in terms of their negative log- ligeliidddood (NLL). This property is affected not only by the uncertainty, but also by the DNNs performance. Ad- ditionally, it is difficult to interpret, sometimes leading to contraintuitive results, which we want to elaborate here. As first example, take the likelihood of two datasets x; {(0} and 2 {005), each consisting of single point, with respect to a normal distribution W00, 1). Naturally, we find 3i to be located at the maximum of the considered normal distribution and deem it the more likely candidate. But, if we extend these datasets to more than single points, i.e. æ1 {0, 0.1, O. -0.1, 0} and ह2 {0.5, -..4. 0, 1.9, -.77, becomes obvious that i2 much more likely to follow the intended Gaussian distribution. Nonetheless, NLL((T2 R 1.4 > 0.9 R (L..(E1), where
NLL(g) log v 2
1
Σ
N
4)²
N
20
This may be consider the distribution of the elements in ø4. From this observation also follows that a model with high prediction accuracy will have lower NLL score predicted in the same way. Independent of whether those reflected the "true" uncertainty in either second example. Consider two other datasets
direct consequence of the point-wise definition of NLL, which does not
performing one if uncertainties
case. This issue be further substantiated 22 drawn i.i.d. from Gaussian distributions (O, o;) with two differing values σi < a2. If we determine the NLL of each with respect its own distribution the offset term in equation 7 leads to NLL(22) NLL(a1) (=1) + log (02/01) with log (02/01) >> O. Although both accurately reflect their own distributions, or uncertainties so speak. the narrower 21 is more "likely". This offset makes difficult to assess reported NI.I. values for systems with heteroscedastic uncertainty. While smaller typically "better", is highly data- (and prediction-) dependent which value is good the sense of reasonable correlation between performance and uncertainty.
25

---

Under review as a conference paper at ICLR 2021
2.0
1.5
ece
1.0
0.5
0.0
10-2
10-1
10
10a
ws_dist
1.0
0.8
ks diii
0.6
0.4
0.2
0.0
10
10-1
10
10a
ws_ Ws_dist
2.0
1.5
ece
1.0
0.5
0.0
0.0
0.2
0.4
0.6
0.8
1.0
ks dist
Dependencies between the three uncertainty measures ECE, Wasserstein distance and Kolmogorov- Smiirov distance. Uncertainty methods are encoded via plot markers, data splits via color. Datasets are not encoded and cannot distinguished (see text for more details). Each plot point corresponds to cross-validated trained network. The clearly visible deviations from ideal correlations point at the potential of these uncertainty measures to complement one another.
Figure 15:
26


Reviews:
[{"review_id": "YOX1M15QQ0V", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes a new objective function to prevent underestimating uncertainties. The authors claimed that the new form leads to state-of-the-art performance in several numerical experiments.\n\n##########################################################################\n\nPros: \n\n- A new objective function is easy to understand.\n- The motivation for the work is sensible.\n\n##########################################################################\n\nCons: \n\n- Although the motivation of the SML loss in Section 3 is sensible, but the theoretical grounds for the SML loss look somewhat weak. MC dropout (Gal & Ghahramani, 2016) is to maximize the evidence lower bound and learns Bayesian neural networks. What are the properties of the optimal solution of the proposed objective function?\n- Following the question, is it restricted to the dropout networks? How can this new loss function be applied other than the dropout networks?\n- The authors use the (gaussian) negative log-likelihood with the mean and the variance of the sub-network outputs as one of the evaluation measures. In case of MC dropout, the authors use posterior mean estimates $E(E(Y |\\theta, X))$ for $\\mu$, but a conditional posterior variance estimates $Var(E( Y | \\theta, X))$ for $\\sigma^2$. Thus, it makes sense the MC dropout might underestimate uncertainties because $Var(E( Y | \\theta, X)) \\leq Var( Y | X)$. However, Kendall & Gal (2017), which the author cited in the manuscript, addressed this issue and provided a better uncertainty quantification method. But this method is not considered in the experiments.\n- The paper's writing makes it very hard to understand the results. For instance, implementation details for Figure 1 or its pointer are not provided. Also, Figure 3 should be improved. The current presentation is confusing and it's hard to recognize which is better.\n\n##########################################################################\n\nI vote for rejection. I may well have missed some points in my reading, so clarification is welcome.\n\n##########################################################################\n\nAfter the author response\n\nAs the authors address the reviewer's concerns, I changed the rating.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LKHKJF3F2Zu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an objective function i.e., second-moment loss (SML) to better evaluate the uncertainty based on MC dropout. A full network is used to model the mean, while sub-networks are explicitly used to optimize the model variance.\n\n\nAccording to the claim, it seems the main novelty is introducing a new network for variance. First, it is not new that using one network for accuracy (e.g., mean) and another for uncertainty (e.g., variance). There are existing models that should be compared and discussed. \n\nThe authors claim that the model can be to adaptive to domain shift, but there is no explanation why the model can do this?\n\nIt is unclear or at least not well motivated why the authors proposed the second-term-moment and why it is advantageous than others. The authors should discuss more recent proposed methods in introduction to show the neccessarity or clear advantages for the proposed one.\n\nIt is unclear what the gray lines (sub-networks) mean in Fig. 1? And how to obtain the gray lines? It is better to explain in the introduction or caption of fig. 1.\n\nThe authors used “prediction uncertainty” and “data-independent uncertainty”. What is the relationship between predictive uncertainty and epistemic uncertainty? What is data-inherent uncertainty? what the relationships between it and aleatoric uncertainty?  Why the authors call the distance $|f_\\tilde{\\theta} – f_ {\\theta} |$ aleatoric uncertainty?\n\nAlthough the objective is simple, in the current state, the model is still not clear enough to follow. \n\n----------------------------------------------------------------------\nUpdate after rebuttal\n----------------------------------------------------------------------\nThanks for the feedback from the authors.\n\nUnfortunately, although some parts are clarified, the main issues still exist. Overall, the novelty is limited and the motivation is still not clear enough.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HsArUP46oEC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\nThe paper proposes a variant of Monte-Carlo Dropout that aims to obtain\nbetter uncertainty estimates by a better adjustment of the output variance. In\nconcrete, the paper proposes a novel objective function consisting of two\nterms. The first term is a simple regression loss between the model expected\noutput and the data label. The second term, named \"second-moment loss\" penalizes\nthe differences in the gap between the expected and sample output and the gap\nbetween the expected output and the data label. In other words, the output\nshould match the data label in expectation, and the output variance produced by\nDropout \"sub-networks\" should follow the variance of the prediction residuals,\nas a proxy for aleatoric uncertainty.\n\n\n##### Pros\n- I think the idea overall makes sense, it is simple to implement and could be\n  an interesting addition to the MC Dropout toolkit.\n- Experimental results on toy datasets show the models trained with the proposed\n  loss are able to effectively capture the aleatoric uncertainty.\n- On real datasets, the proposed method is compared to many uncertainty\n  estimation methods and shows competitive performance.\n\n##### Cons\n- One potential pitfall could happen when the expected output is far from\n  correct due to model limitations and the random \"sub-network\" predictions\n  become heavily biased. I think further discussion on this issue would be\n  welcome.\n- The paper is in general correctly structured but leaves some important details\n  to the appendix. Maybe it is because my lack of familiarity with these\n  evaluation metrics, but in my opinion it may be good to clarify them in the\n  main paper. For example:\n    - The \"normalized residuals\" $r_i$ in page 6 are not properly introduced in\n      the main text.\n    - Isn't the ECE, as defined in the appendix, a measure of fit of the\n      residuals to a uniform distribution? If yes, why is this a reasonable\n      assumption and why would this indicate a correctly characterized\n      uncertainty?\n    - Are the $\\mu_i$ and $\\sigma_i$ in the evaluation metrics computed from the\n      empirical predictions or from the data? If computed from the predictions,\n      then the proposed Wasserstein metric looks like a measure of\n      Gaussianity. This would be in contradiction with the ECE metric, and the\n      same questions as in the previous point would apply: why is Gaussianity a\n      reasonable model for the residuals and a desiderata for the output\n      distribution? what is the intuition for why this test indicates good\n      uncertainty estimates?\n- For a fair comparison with ensemble methods, the total number of parameters\n  and operations should be taken into account.\n\n\n**************\nAfter Rebuttal:\nI thank the authors for their extensive answers and clarifications.\nOverall, I maintain my positive outlook on this work. Although theoretical justification could be improved, I think the experiments do signal that there is something interesting and valuable in this simple approach for characterizing uncertainty.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dear Authors,

Thank you very much for your very detailed feedback and also updating the manuscript in the rebuttal phase. Your effort has highly contributed to clarifying some of the concerns raised by the reviewers and improving our understanding of your work. 

On the other hand, we still think that the current work has rather limited novelty, and motivation and theoretical justification need to be further enhanced to be accepted for ICLR.

For these reasons, I suggest rejection of this paper, in comparison with many other strong submissions. The reviewers added further comments after receiving your feedback. I hope their comments are useful for improving your work for future publication.
Decision: Reject