Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions
Authors: Bertrand Charpentier
Oliver Borchert
Daniel Zügner
Simon Geisler
Stephan Günnemann
Venue: ICLR 2022 Spotlight (2022)
Keywords: Uncertainty
Exponential Family
Bayesian Update
Conjugate Prior

Abstract: Uncertainty awareness is crucial to develop reliable machine learning models. In this work, we propose the Natural Posterior Network (NatPN) for fast and high-quality uncertainty estimation for any task where the target distribution belongs to the exponential family. Thus, NatPN finds application for both classification and general regression settings. Unlike many previous approaches, NatPN does not require out-of-distribution (OOD) data at training time. Instead, it leverages Normalizing Flows to fit a single density on a learned low-dimensional and task-dependent latent space. For any input sample, NatPN uses the predicted likelihood to perform a Bayesian update over the target distribution. Theoretically, NatPN assigns high uncertainty far away from training data. Empirically, our extensive experiments on calibration and OOD detection show that NatPN delivers highly competitive performance for classification, regression and count prediction tasks.

Published as a conference paper at ICLR 2022
NATURAL POSTERIOR NETWORK: DEEP BAYESIAN UNCERTAINTY FOR EXPONENTIAL FAMILY DISTRIBU-
TIONS
Bertrand Charpentier Oliver Borchert Daniel Zügner, Simon Geisler, Stephan Günnemann Department of Informatics & Munich Data Science Institute
Technical University of Munich, Germany (charpent, borchero, zuegnerd, geisler, guennemann ) @in. tum. de
ABSTRACT
Uncertainty awareness is crucial to develop reliable machine learning models. In this work, we propose the Natural Posterior Network (NatPN) for fast and high- quality uncertainty estimation for any task where the target distribution belongs to the exponential family. Thus, NatPN finds application for both classification and general regression settings. Unlike many previous approaches, NatPN does not require out-o--distributiin (OOD) data at training time. Instead, it leverages Normalizing Flows to fit a single density on a learned cow-dimensional and task- dependent latent space. For any input sample, NatPN uses the predicted likelihood to perform a Bayesian update over the target distribution. Theoretically, NatPN assigns high uncertainty far away from training data. Empirically, experiments on calibration and OOD detection show that NatPN delivers highly competitive performance for classification, regression and count prediction tasks.
extensive
INTRODUCTION
Model Predictions
Accurate and rigorous uncertainty estimation is key for reliable machine learning models in safety-critical domains. It quantifies the confidence of machine learning models, thus allowing them to validate knowledgeable predictions corresponding to cor- rect/wrong predictions, flag predictions unknown input domains corresponding to anomaly or Out-of- Distribution detection, or detect natural shifts of the data facilitating real-time model maintenance (Filos
Men
Tee Data
inditik Camalyy
al. 2019; Malinin et al. 2021 Ovadia al. 2019). Specifically, a reliable model can handle all these fail- modes with high-quality estimates of aleatoric and epistemic uncertainty (Gall 2016). These two levels of uncertainty allow model to account for both irreducible data uncertainty (e.g. fair dice's chance of 1/6 for each face) and uncertainty due the lack of knowledge about unseen data (e.g. input features differing significantly from training data or a covariate shift) respectively. Aleatoric and epistemic uncertainty levels can eventually be combined into
Toy Regression Task
Osnw)
Toy Classification Task
overall predictive uncertainty Gall 2016)
Figure 1: Visualization of the alcatoric and predictive uncertainty estimates NatPN on two toy regressions and classification tasks. NatPN correctly assigns higher uncertainty to regions far from the training data.
Traditional neural networks are not readily applicable in critical domains they show overconfident prediction, in particular on data that is different from training data (Guo et al. 2017 Lakshminarayanan
"quual contribution

---

Published as a conference paper at ICLR 2022
et al., 2017). To mitigate this problem, important model family for uncertainty estimation directly predicts the parameters of a conjugate prior distribution on the predicted target probability distribu- tion, thus accounting for the different levels of uncertainty. These models efficient as they only require a single forward pass for target and uncertainty prediction. Most of those models focus on classification and thus predict parameters of a Dirichlet distribution (Biloš et al. 2019 Charpentier
ict al.
2020; Gast & Roth 20118] Malinin & Gales 2018 2019 Nandy et al. 20200] Sensoy et al. Shi et al. [220]] Stadler et al.) 2021; Zhao et al. 2020b). However, only two works (Amini et al. Malinin et al. 2020a) have focused on regression by learning parameters of a Normal (NIG) distribution as conjugate prior. Hence, all these models are limited
2020:
single task (e.g. either classification or regression). Some approaches even require out- out-of-distribution (OOD) data at training time (Malinin & Gales, 2018 2019) which is an unrealistic assumption in many real-world applications where anomalies are a priori diverse, rare or unknown.
Our contribution. We propose Natural Posterior Network (NatPN) as a new approach parametrizing conjugate prior distributions for versatile uncertainty estimation. NatPN is motivated from both the theoretical and practical perspective. (1) NatPN can estimate predictive uncertainty for any task described by the general group of exponential family distributions contrary existing approaches from this family of models. Notably, this encompasses very common tasks such as classification, regression and count prediction which can be described with Categorical, Normal and Poisson distri- butions, respectively. (2) In theory, NatPN is based on a new unified exponential family framework which performs an imput-dependent Bayesian update. For every input, it predicts the parameters of the posterior over the target exponential family distribution. We show that this Bayesian update is guaranteed to predict high uncertainty far from training data. (3) In practice, NatPN requires no OOD data for training, only adds a single normalizing flow density to the last predictor layer and provides fast uncertainty estimation in a single forward pass. Our extensive experiments showcase the high performances of NatPN for various criteria (accuracy, calibration, OOD and shift detection) and tasks (classification, regression and count prediction). We illustrate the accurate aleatoric and predictive uncertainty predictions of NatPN on two toy examples for classification and regression in Fig. I None of the conjugate prior related works have similar thcoretical and practical properties.
RELATED WORK
In this section, we describe other work related uncertainty estimation for supervised learning. We refer to Gawlikowski et al. 1021 for a detailed survey on uncertainty estimation in deep learning.
Sampling-based methods. A first family of models estimates uncertainty by aggregating statistics (e.g. mean and variance) from different samples of an implicit predictive distribution. Examples are ensemble (Kim & Ghahramani] 2012: Lakshmmnarayanan et al.] [017]] [Simpson et al. [2012] [Wen let al. 2020; Wenzel et al.) (020) and dropout (Gal & Ghahramani (016)) models which provide high-quality uncertainty estimates (Ovadia et al.) 2019) at the cost of an expensive sampling phase at inference time. Moreover, ensembles usually require training multiple models. Further, Bayesian neural networks (BNN) (Blundell et al., 2015); Maddox et al., 2019; Ritter et al., 2018) model the uncertainty on the weights and also require multiple samples estimate the uncertainty
the final
prediction. While recent BNNs have shown reasonably good performance (Dusenberry et al.. 2020 Farquhar et al., 2020 Osawa al. 2019). modelling the distribution on the weights suffers from
pathological behavior thus limiting these approaches in practice (Foong et al.] 2020 Graves 2011 Izmailov et al. 202T). In particular, [zmailov et al. (202T) uses an enormous computation budget
by parallelizing the computation over 512 TPUv3 devices and running tens of thousands of training epochs to achieve more exact Bayesian inference which is not suitable for practical applications. In
contrast, NatPN predicts uncertainty in single forward pass with a closed-form posterior distribution over the target variable. NatPN does not model uncertainty on the weights.
Sampling-free methods. A second family of models is capable of estimating uncertainty in single forward pass. The family of models parametrizing conjugate prior distributions is the main focus of this paper (Gast& & Roth 2018 Kopetzki et al. 202 I Nandy al. 2020 Sensoy et al. 2020 Shi et al.) 2020, Stadler et al.] |021 Umerr] (022)). Beyond this family of models, we differentiate between four other families of sampling- free models for uncertainty estimation. A first family aims at learning deep
Gaussian processes with random features projections or learned inducing points (Bilos et al.) 2019
Lakshminarayanan al. 2020 van Amersfoort et al. 2020 202T). 1). A second family aims at learning
2

---

Published as a conference paper at ICLR 2022
deep energy -baee models (Elflein et al. 2021 Grathwohl al. 2020). Another family of models
aims at propagating uncertainty across layers Gast & Roth 2018 Meemandez-Lobatte Adams 2013 Postels et al. 2019] Shekhovtsov & Flach 2019 Wang et al. 2016). They model uncertainty at the weight and/or activation levels and are generally constrained to specific transformations, In contrast, NatPN only models the uncertainty on the predicted target variable and does not enforce any constraint on the encoder architecture. Further, some of the models propagating uncertainty already used the exponential faailly framework (Ranganath al. 2013 Wang al. 2016). However, while they parametrize exponential family distributions, NatPN parametrizes the conjugate prior of the targer exponential family distributions which accounts for the epistemic uncertainty. Finally, while
the family of calibration models aims at calibrating predictions (Kuleshov et al.) 2018, Moon et al.)
2020, Rahimii et al., 2020; Song et al. 2019; Zhao et al. [020a], NatPN aims at accurately modelling both aleatoric and epistemic uncertainty on in- and out-of nut-of-distributio data.
3 NATURAL POSTERIOR NETWORK
At the very core of NatPN stands the Bayesian update rule: Q(0 x P(D x ( where P(D| () is the target distribution of the target data D given its parameter 0, and ())) and Q()) D) are the prior and posterior distributions, respectively, over the target distribution parameters. The target distribution ((D) 0) could be any likelihood describing the observed target labels. The Bayesian update has three main advantages (1) it introduces prior belief which represents the safe default prediction if no data observed, (2) it updates the prior prediction based on observed target labels, and (3) it assigns a confidence for the new target prediction given the aggregated evidence count of observed target labels. While NatPN is capable to perform Bayesian update for every possible imput given the observed training data, we first recall the Bayesian background for a single exponential family distribution.
Likelihood
Conjugate Prior
Parametrication Mapping
Bayesian (Bg
gi Catip)
Dir(ar)
a/2
(alll) ((
log Blac)) 10 Cyotob") ) Extad ( Dyllo!")
(-Rt mo02 v(0)) log 1x)
( lox ((2-) (1 m()) b A o (o Dola)
Pol(A)
A ((.
0/
@ (ote) fyg 9 El- lock
log lag al4(0))
Table I: Examples of Exponential Family Distributions where (xx) and B(x) denote Digamma and Beta function, respectively.
3.1 EXPONENTIAL FAMILY DISTRIBUTION
Distributions from the exponential family are very widely used and have favorable analytical prop- erties. Indeed, (1) they cover a wide range of target variables like discrete, continuous, counts or spherical coordinates, and (2) they benefit from intuitive and generic formulae for their parameters, density functions and statistics which can often be evaluated closed-form. Important examples of exponential family distributions are Normal, Categorical and Poisson distributions (see Tab. []. Formally, an exponential family distribution on a target variable E IR with natural parameters e RR can be denoted as
Ply h(y) exp (8 u()) A()))
where h IR + H is the carrier or base measure, A R² R. the log-no nog-oommllizer and u IR + g
the sufficient statistics (Bishop) 2006 Nielsen & Nock 2010). The entropy an exponential family distribution always be written as HIP = A()) 0' VoA(0) Ellog h(v)] (Nielsen & Nock 2010). An exponential family distribution always admits a conjugate prior, which often also is a member of the exponential family:
2(0),n) 7(,, nz) exp (n OT 11000xnAAll n())))
where 71(,,) n)) is normalization coefficient, E IR_L are prior parameters and E IR+ is the evidence. Given a set of N target observations ( it is easy to compute closed-form Bayesian
update ((0 post, pOR) x ( (0) x Q(0 xxrior nndtor):
((1 x post poay xx exp (n (oss 0 post n post A()))
(3)
3

---

Published as a conference paper at ICLR 2022
Fola!!!y
Austelll x!I Sale
Folu
Lo (arliny
xc
oi HuPtelllua v.. mapte"ll
Input Space
Labent Space
Exponential Family Parameter Space
Figure 2: Overview of Natural Posterior Network. Inputs (+)) are first mapped to low-dimensional latent representation 20) by the encoder fa. From 2 the decoder ge derives the parameter update () while normalizing flow P. yields the evidence update () Posterior parameters are obtained from weighted combination of prior and update parameters according to (t))
where x ni mire + Ey u(yl))) and no = nprior + N. We see that x (resp. X m0t) can be yapere +N
viewed as the average sufficient statistics of re (resp. ne fictitious samples (Bishop. 2006).
Further, the average sufficient statistic of fictitious samples is equal to the expected sufficient statistic of the conjugate distribution, i.e. X = ( [0] (Brown. 1986 Diaconis & Ylvisaker 1979). Thus, the parameter x carries the inherent aleatoric uncertainty on the target distribution with natural parameters O, while the evidence po aligns well with the epistemic uncertainty (i.e. a low evidence means few prior target observations). We stress that the natural conjugate prior parametrization X. n is often different from the -nownn" parametrization k used by standard coding libraries. By definition, bijective mapping m(k) (x, n) from the natural parametrization to the commonly used parametrization always exists (see examples in Tab. [ Finally, exponential family distributions always admit a closed-form posterior predictive distribution (Gelman et al., 2004).
3.2 INPUT-DEPENDENT BAYESIAN UPDATE FOR EXPONENTIAL FAMILY DISTRIBUTIONS
We propose to leverage the power of exponential family distributions for the more complex task when the prediction 7) depends on the input (f)) Hence, NatPN extends the Bayesian treatment of a single exponential family distribution prediction by predicting an individual posterior update per input. We distinguish between the chosen prior parameters x niorr shared among samples, and the additional predicted parameters x (+) dependent on the input (+) leading to the updated posterior parameters:
(1) nerrorr prior + (0
(i) pricr (.)
pypiior + nz())
Equivalently, NatPN may be interpreted as predicting set of (1) pseudo observations (1( such that their aggregated sufficient statistics satisfy Σ yاأر = n (()), and perform the respective Bayesian update. This Bayesian update works for any choice of exponential family distributions as long as parameters are mapped to their standard form (see Tab. D.. According to the principle of maximum entropy (Rosenkrantz) 1989), practical choice for the prior is to enforce high entropy for the prior distribution which is usually considered less informative. It is typically achieved when the prior pseudo-count prior is small and the prior parameter x prior shows a high aleatoric uncertainty.
Hence, NatPN proposes generic way to perform the input-dependent Bayesian update (1, ()) for any exponential family distribution in three steps (see Fig.
2 (1) An encoder S maps the input (+))) onto low-dimensional latent vector ( = (z) E R representing useful features for the prediction task (see left Fig D). Note that the architecture the encoder can be arbitrarily
complex. Then, (2) the latent representation x0 is used two different ways to predict the parameter
update (9) and the evidence update ((()) (see center Fig D. On the one hand, linear decoder
Iy is trained to output the parameter update (8) G ( _ accounting for the alcatoric uncertainty. On the other hand, a single normalized density is trained to output the evidence update x N₁ (()(0) w) accounting for the epistemic uncertainty. The intuition is that increasing the evidence training data during training forces the evidence everywhere else (incl. far from
4

---

Published as a conference paper at ICLR 2022
training data) to decrease thanks to the density normalization constraint. The constant Ne is certainty budget distributed by the normalized density P(z(1) () over the latent representations 2(1) i.e. Nu = Nu P(z(() w)dz(+) f dz()) In practice, we observed that scaling the certainty budget w.r.t. the latent dimension H helped the density to cover larger volumes in higher dimension (see app.). Finally, (3) NatPN computes the posterior parameters x and n poss,()) which can be viewed respectively as the mean and concentration of the posterior distribution (see right Fig Note that the posterior parameter (8)s.(() is a simple weighted average of the prior parameter x and the update parameter (*) as shown by Eq.
NatPN extends PostNet (Charpentier al. 2020) which also performs an input-dependent Bayesian update with density estimation. Yet, it has three crucial differences which lead to major practical improvements. First, the new exponential family framework is signifieantly more flexible and is not restricted to classification. Second, the Dirichlet a parameter computation is different: NatPN computes the X parameters which can be viewed as standard softmax output and the n evidence separately (i.e. a nx) while PostNet computes one evidence pseudo-count per class, Third, NatPN computationally more efficient. requires a single density while PostNet requires C' densities.
3.3
ID AND OOD UNCERTAINTY ESTIMATES
NatPN intuitively leads to reasonable uncertainty estimation for the two limit cases of strong in- distribution (ID) and out- -fiiiiiibiittoionnnn (OOD) inputs (see red and purple samples in Fig. [ For
very likely in-distribution data (i.e. (P(2(() w) + oo), the posterior parameter overrules the prior (i.e. (t) + x ()) Conversely, for very unlikely out-of-distributiin data (i.e. P(z()) () - 0), the prior parameter takes over in the posterior update (i.e. poat(() ptior, Hence, the choice of the prior parameter should reflect the default prediction when the model lacks knowledge. We formally show under mild assumptions on the encoder that NatPN predicts very low additional evidence
(1) x 0) for (almost) any input ax() far away from the training data (i.e. ||a:10|| +o0), thus recovering prior predictions (i.e. x (is = miorr) (see proof in app.).
Let a NatPN model be parametrized with a (deep) encoder fo with ReLU activations, a decoder g6 and the density TP(z w.) Let fa(x)) = V x + a be the piecewise affine representation of the ReLU network Jo on the finite number of affine regions Q(1) (Arora et al. 2018). Suppose that (i) have independent rows and the density function P(z w) has bounded derivatives, then for 0. i.e the evidence becomes small far from training data.
Theorem 1.
almost any x we have P(fa(ō xx) w)
5-000
This theorem only requires that the density avoids very unlikely pathological behavior with unbounded derivatives (Dix 2013). A slightly weaker conclusion holds using the notion of limit in density if the density function does not have bounded derivatives (Niculescu & Popovici 2011). Finally, the independent rows condition is realistic for trained networks with no constant output (Hein al. 2019). It advantageously leads NatPN to consistent uncertainty estimation contrary to standard ReL.U networks which are overconfident far from training data (Hein al. 2019).
3.4 BAYESIAN NATPN ENSEMBLE
Interestingly, it is natural to extend the Bayesian treatment of a single NatPN to ensemble of NatPN models (NatPE). An ensemble of m NatPN models intuitively equivalent to performing m successive Bayesian updates using each NatPN member separately. More formally, given an input (+) and an ensemble of m jointly trained NatPN models, the Bayesian update for the posterior
distribution becomes (s) witke x price +E n. (1)
(i) Note that and (+) prior Σ5 m n
nomor +2 n
the standard Bayesian averaging which is used in many ensembling methods (Chipman al.
2007: Lakshminarayanan et al., 2017): Wen et al. 2020 Wenzel et al.) (020)) is different from this Bayesian combination. While Bayesian averaging assume that only one model is correct, the Bayesian combination of NatPN allows more or none of the models to be "expert" for some input (Monteith)
(6) xs 0) would recover et al. 2011). For example, an input (+) unfamiliar every model m (i.e. nm
the prior default prediction prior y Existing models already had similar properties for Bayesian combination of classifiers (Kim & Ghahramani 2012
Simpson et al. 2012).
5

---

Published as a conference paper at ICLR 2022
3.5 OPTIMIZATION
The choice of the optimization procedure is of primary importance in order to obtain both high-quality target predictions and uncertainty estimates regardless of the task.
Bayesian Loss. We follow Charpentier et al. (020) and aim at minimizing the Bayesian formulation:
((0 =
Egin 00 log ((y
(())
(5)
where [[] denotes the entropy of the predicted posterior distribution @PPat -Similally to
the BL loss, this loss is guaranteed to be optimal when the predicted posterior distribution is
close to the true posterior distribution (0
x (0)
i.e. Queat(()) Rs Q*(0 (0)) (Bissiri et al. 2016
Shawe-Tayler Taylor & Williamson 1997| Zellner 1988). However, this loss is generally nor equal the
ELBO loss especially for real valued targets Le. y E R (see app.). The term (i) is the expected likelihood under the predicted posterior distribution. It can be viewed as the Uncertain Cross Entropy (UCE) loss (Bilo% et al., 2019) which is known to reduce uncertainty on observed data. The term (ii) is an entropy regularizer acting as a prior which favors uninformative distributions (pon. with high entropy. In our case, we assume the likelihood (f(y(() ((+) and the posterior (Pos)(()) to be members of the exponential family. We take advantage of the convenient computations for such distributions and derive a more explicit formula for the Bayesian formulation 5
(see derivation in the appendix):
2 O (00 [[A(O)] - A [[[(Poa.(0))
(6)
where A is an additional regularization weight tuned with a grid search. Note that the term ((O)Tu(y()) favors a good alignment of the expected sufficient statistic E[Ø] = X with the ob- served sufficient statistic ))) In practice, all terms can be computed efficiently in closed form for most exponential family distributions (see examples in Tab. T.b. In particular, simplifications are possible when the conjugate prior distribution is also in an exponential family which is often the case. Ultimately Eq. (6) applies to any exponential family distribution unlike Charpentier et al. (2020).
Optimization Scheme. NatPN is fully differentiable using the closed-form Bayesian loss. Thus, we train the encoder fo. the parameter decoder By and the normalizing flow P(z (+) () w.r.t parameters 6, ,b, w jointly. Further, we observed that "warm-up training" (Ash & Adams 2020) and "fine-tuning" (Käding et al. 2016) of the density helped to improve uncertainty estimation for more complex flows and datasets. Thus, we train the normalizing flow density to maximize the likelihood of the latent representations before and after the joint optimization while keeping all other parameters fixed.
3.6 MODEL LIMITATIONS
Task-Specific OOD, Previous works show that density estimation is unsuitable for acting on the raw image input (Choi et al. 2019 Nalisnick et al., 2019 2020) or non-carefully transformed space (Lan & Dinh 2020). To circumvent this issue, NatPN does not perform OOD detection directly on the input but rather fits normalizing flow a learned space. In particular, the latent space is (1) low-dimensional, (2) task-specific and (3) encodes meaningful semantic features. Similarly, Charpentier al. (2020); Kirichenko al. (2020); Morningstar al. (2020); Winkens et al. (2020) methods by leveraging a task-induced bias or low-dimensional statistics. In the case of NatPN, the low-dimensional latent space has contain relevant features to linearly predict the sufficient statistics required for the task. For example, NatPN aims at linearly separable latent space for classification. The downside is that NatPN is capable of detecting OOD samples only with respect to the considered task and requires labeled examples during training. As an example, NatPN likely fails to detect a change of image color if the task aims at classifying object shapes and the latent space has notion of color. Hence, we underline that NatPN comes with a task-dependent OOD definition, which is a reasonable choice in practice.
already improved OOD detection of density-
Model-Task Mismatch. Second, we emphasize that the uncertainty estimation quality of NatPN for (close to) ID data depends on the convergence of the model, the encoder architecture (e.g. MLP, Conv., DenseDepth (Eigen et al., 2014)) and the target distribution (e.g. Poisson, Normal distributions) choice which should match the task needs. However, we show empirically that NatPN provides high quality uncertainty estimates in practice wide range of tasks. Further, we show theoretically that

---

Published as a conference paper at ICLR 2022
NatPN leads to uncertain prediction far away from training data for any exponential family target distributions. In comparison, Meinke & Hein (2020) showed akin guarantees for classification only.
4 EXPERIMENTS
In this section, we compare NatPN to existing methods on extensive experiments including three different tasks; classification, regression and count prediction. For each task type, we evaluate the prediction quality based on target error and uncertainty metrics. These various set-ups aim to highlight the versatility of NatPN. In particular, NatPN is the only model that adapts to all tasks and achieves high performances for all metrics without requiring multiple forward passes.
Table 2: Classification results on Sensorless Drive with Categorical target distribution. Best scores among all single-pass models are in bold. Best scores among all models are starred.
Accuracy
Brier
9/0 Alea.
9/10 Epist.
OODom Alea.
OODom Epist.
Dropout
98.62 =: 0.11
3.79 0.29
30.20 +: 0.85
32.57 2: 1.45
27.03 1 0.51
95.30 #: 1.66
Ensemble
98.83 0.17
3,00
30:79 + 0.74
32.61 1.06
27.16
NatPE
*99.66 1 0.03
40.68 1: 0.05
77.05 + 1.93
83.73
1.89
99.99 0.00
*100.00 1: 0.00
R-PriorNet
98.85 = 0.25
2.01 4 0.47
40.13 +: 2.99
30.07 1: 0.81
*100.00 1: 0.00
23,59 :: 0.00
En1>
93.95 2.35
28.09 6.40
26
0.60
24.85
0.43
+ 15:21
23.58
PostNet
99.64 0.02
0.75
0.08
80.60 1: 1.68
*92.57
2: 1.41
*100.00 0,00
*100.00 0.00
NutPN
99.61 0.05
1.04 +: 0.29
*81.43
1,89
79.54
2.62
99.98
0.00
*100.00 0.00
Table Classification results on CIFAR-10 with Categorical target distribution. Best scores among all single-pass models are in bold. Best scores among all models are starred. Gray numbers indicate that R- R-PrrorNet has seen samples from the SVHN dataset during training.
Accuracy
Brier
SVHN Alea.
SVIN Eyist.
Celebi Alea.
Colebik Epist.
000om Aleo.
0ODum Epeat
Drepout
88.55
19.50 0.41
50.63
1.50
71884
71.04 3.02
18422 1:31
Ensensble
17.33
55.26
76,20
74.25 0.78
25:30 4.00
89.21
17.41 0.112
55.66
983.1 0.67
77.955
*82.0
87:37
1.79
N-PriorNet
88.94 0.23
+15.99
00.57 0:00
67.74
59.55 7:00
42:31 8.77
38.25
EaD
40.84
*86.47
75.94 1.52
42.19
8.77
PostNet
20.19
62:00
76.87 3.38
Table 4: Results on the Bike Sharing Dataset with Normal N and Poison Poi target distributions. Best scores among all single-pass models are in bold. Best scores among all models are starred.
RMSE
Calibration
Winter Epist.
Spring Epist.
Autumm Epist.
00Dom Epist.
Dropont-W
70.20 + 1,30
6.05 + 0.77
15.26 + 0.51
13.66 0.16
15.11 + 0.46
99.99 - 0.01
Ensemble.V
+48.02 + 2.78
5.88 + 1.00
42,46 + 2.29
21.28 0.38
21.97 + 0.58
1100.00 + 0.00
EvReg-N
49.58 1.51
3.77 0.81
17.19 0.76
15.54
0.65
14.75 1:
34,99 17.02
NadPN.N
49.85 4: 1,38
$1,95 :: 0.34
*55.04 J: 6.81
*23.25 4 1,20
+27.78
2.47
+100.00 1 0.00
Dropout-Poi
66.57 1: 4.61
55.00 1: 0.22
16:02 0.48
13.48 0.38
18.09 1: 0.82
$100.00 1 0,00
Ensemble-Poi
*48.22 2,06
55.31 J: 0.21
83.88
1.22
34.21
1.81
41.29 1. 3.23
$100.00 1 0.00
NatPN-Poi
51.79 0.78
*31.04 1.81
*85.15 3.61
*37.03 + 2.35
+42.73 4.38
*100.00 + 0.00
4.1 SETUP
In our experiments, we change the encoder ar- chitecture of NatPN to match the dataset needs. We perform grid search over normalizing flows types (i.e. radial flows (Rezende & Mohamed 2015) and MAF (Germain et al.] 22115] Papa- makarios et al. 2017)) and latent dimensions. We show further experiments on architecture, latent dimension. normalizing flow choices and certainty budget choice in the appendix. Further- more, we use approximations of the log-Gamma log [(x) and the Digamma (x(x) functions for large input values to avoid unstable floating com- putations (see app.). As prior parameters, we set prior = / CC prior = C' for classification, anar (0, 100) T prior = 1 for regression and prior = 1, prior = 1 for count prediction en- forcing high entropy for prior distributions.
RePrior Nat
EnD
Postive
NePPN
RelltAve Gonfidonos
Accornty
TE%
60%
12%
Severlyy of battar
Figure 3: Averaged accuracy and confidence under 15 dataset shifts on CIFAR-I0 (Hendrycks & Di- etterich 2019). On more severe perturbations (i.e. data further away from data distribution), NatPN maintains competitive accuracy while assigning higher epistemic uncertainty as desired. Basclines provide a slower relative confidence decay.
7

---

Published as a conference paper at ICLR 2022
Table 5: Regression results on models trained on different UCI datasets with Normal target distribution. The upper half displays models trained on Kin8nm, the lower half shows models trained on Concrete Compressive Strength.
RMSK
Calibration
Emergy Alea.
Energy Epist.
Concrete Alea.
Concrete Epist.
Kinšim Alea.
Kintiun Egist.
Dropout
0.09 =: 0.00
3.13 = 0.43
90.18 = 6.00
99.94 =: 0.06
*100.00 = 0.00
*100.000 = 0.00
Ensemble
*0.07 1: 0.00
2.69 a: 0.49
+100.00 1: 0.00
+100.00 1: 0.00
*100.00 2: 0.00
*100.00 +: 0.00
in disttribution
NatPE
0.08 0.00
5:19 0.30
1100.00 0.00
1000.0
0.00
1000.0 0.00
*100.00 1: 0.00
ExReg
0.09 0.00
3.74 0.53
88.06
11.91
11.91
1000.00
0.00
96.81
13.16
in in-distrbbution
NalPN
0.08 1: 0.00
*2.04 1: 0.45
$100.00 0.00
0.00 $100.00
+100.00
0.00
$100.00 :: 0.00
Dropout
5.67 0:07
*3.03 0.40
V.33
43.53
1,39 0.13
64.30 +: 7.14
Easemble
5.69
3.81
0.67
54.19
18.93
*100:00
0.00
in-distributton
72.57 19,32
1100.00
NatP
*4.78 + 0.20
5.58 + 1.27
100.00 + 0.00
1100.00
0.00
100.00 0.00
1100.00 + 0.00
NatPN
EvReg
5.83 1: 0.23
0.18
5.41
7,36 1: 1.01
1,33
$100.00 0.00
8,93 1: 0.02
$100.00
51.39 1: 18.56
0.00
in-distribation
100,00 0.00
0.93 1 0.00
$100,00 0.00
34,44 1. 20.95
Table 6: Regression results on NYU Depth v2 with Normal target distribution. RMSE is in cm. OOD
scores on LSUN are reported on the held-out classes 'classrooms' (left) and 'churches' (right).
RMSE
Calibration
LSUN Alea.
LSUN Epist.
KITTI Alea.
KITTI Epist.
0ODom Alea.
OODom Epist.
Dropout
46.95
4.03
*95.29 97.74
83.89/83 83.22
98.07
84,90
74.40
*100.00
EyReg
228.88
*1.05
58.70
70.19 64.002
56.60
62,67
75.43
56.39
NatPN
29.72
1.14
94.13/ *98.67
*89.08 / 990.56
*98.93
*93.15
*100.00
*100.00
Baselines. We focus on recent models parametrizing prior distributions over the target distribution. For classification, we compare NatPN to Reverse KL divergence Prior Networks (R (R-PriorNet) (Malinin
& Gales 2019), Ensemble Distribution Distillation (EnD²) (Malinin et al., 2020b) and Posterior Networks (PostNet) (Charpentier et al., 2020). Note that Prior Networks require OOD training data- we use an auxiliary dataset when available and Gaussian noise otherwise. For regression, we compare to Evidential Regression (EvReg) (Amini et al. (2020). Beyond baselines parametrizing conjugate prior distributions, we also compare to dropout (Dropout) (Gal & Ghahramani 2016) and ensemble (Ensemble) (Lakshminarayanan et al. 2017) models for all tasks. These sampling baselines require multiple forward passes for uncertainty estimation. Further details are given in the appendix.
Datasets. We split all datasets into train, validation and test sets. For classification, we one tabular dataset (Sensorless Drive (Dua & Graff (017)) and three image datasets (MNIST (LeCun et al. 2010 FMNIST (Xiao et 2017) and CIFAR-10 (Krizhevsky al. 2009)) ). For count prediction, we use the Bike Sharing dataset (Fanace-T T & Gama 2014) to predict the number of bike rentals within an hour. For regression, we also use the Bike Sharing dataset where the target viewed as continuous, real-world UCI datasets used in Amiii et al. (2200; Hernandez-Lobate & Adams (2015) and the image NYU Depth v2 dataset (Nathan Silberman & Fergus (012) where the goal to predict the image depth per pixel. All inputs are rescaled with zero mean and unit variance. We also scale the output target for regression. Further details are given in the appendix.
Metrics. Beyond the target prediction error, we evaluate model uncertainty estimation using cal- ibration and OOD detection scores. Furthermore, we report the inference speed. Further results including histograms with uncertainty estimates or latent space visualization are presented in ap- pendix. Target error: We use the accuracy (Accuracy) for classification and the Root Mean Squared Error (RMSE) for regression and count prediction. Calibration: For classification, we use the Brier score (Brier) (Gneiting & Raftery| 2007). For regression and count prediction, we use the quantile calibration score (Calibration) (Kuleshov et al. 2018). OOD detection: We evaluate how the uncer- tainty scores enable to detect OOD data using the area under the precision-recall curve (AUC-PR) the area under the receiver operating characteristic curve (AUC-ROC) in the appendix. We use two different uncertainty measures: the negative entropy of the predicted target distribution accounting for the alcatoric uncertainty (Alea. OOD) and the predicted evidence or variance of the predicted mean (Epist. OOD). Similarly to (Charpentier et al. 2020): Ovadia et al. 2019). we use four different types of clear OOD samples: Unseen datasets (KMNIST TEMNIST/Clanuwat et al. (2018), Fashion-MNIST (Xiao et al. 2017), SVHN (Netzer et al., 2011). LSUN (Yu et al. 2015). CelebA (Liu et al. 2015). KITTI (Geiger et al. 2013)), left-out data (classes 9/10 for Sensorless Drive, winter/spring/autumn seasons for Bike Sharing), oft-oo-domain data not normalized in [0. 1] (OODom) and dataset shifts (corrupted CIFAR-10 (Hendrycks & Dietterich 2019)). Further details are given in the appendix.
4.2 RESULTS
Classification. We show results for the tabular dataset Sensorless Drive with unbounded input domain in Tab. 2 and the image datasets MNIST, FMNIST and CIFAR-10 with bounded input domain in
8

---

Published as a conference paper at ICLR 2022
Tab. and appendix. Overall, for classification NatPN performs on par with best single-pass baselines (i.e. 12/30 top-1 scores, 25/30 top-2 scores) and NatPE performs the best among multiple-pat models (i.e. 28/30 top-1 scores). A single NatPN achieves accuracy and calibration performance on par with the most calibrated baselines, namely PostNet and R-PriorNet which requires one normalizing flow per class or training OOD data. Further, NatPE consistently improves accuracy and calibration performance of single NatPN which underlines the benefit of aggregating multiple models predictions for accuracy and calibration (Lakshminarayanan al. 2017). Without requiring OOD data during training, both NatPN and NatPE achieve excellent OOD detection scores w.r.t all OOD types. This strongly suggests that NatPN does not suffer from the flaws in Choi et al. (2019); Nalisnick et al. (2019) 2020). In particular, NatPN and NatPE achieve almost perfect OODom scores contrary to all other baselines except PostNet. This observation aligns well with the theoretical guarantee of NatPN far from training data (see Thm. I which also applies to each NatPE member. The similar performance of NatPN and PostNet for classification is intuitively explained by their akin design: both models perform density estimation on a low-dimensional latent space. Similarly to Charpentier et al. (220)), we compute the average confidence avg-conf = N E. n = N Σ %g and then compare the average confidence change. The average confidence change is computed by taking the ratio of the average confidence of N corrupted data at severity s and the average confidence of N clean data (i.e. the corrupted data at severity 0) i.e. avg-coml, for E [1, 5]. NatPN maintains competitive accuracy (Fig. 3 left) while assigning higher epistémii uncertainty as desired (Fig. 3 right). Baselines provide a slower relative confidence decay.
avg cynf,
Regression & Count Prediction. We show the results for the Bike Sharing, the tabular UCI datasets and the image NYU Depth v2 datasets in Tab. 5 For the large NYU dataset, we compare against all baselines which require only single model to be trained. Overall, NatPN outperforms other single-pas pagsee-ass models for 23/26 scores for regression, thus significantly improving calibration and OOD detection scores. Further, NatPN shows strong improvement for calibration and OOD detection for count prediction with Poisson distributions among all models. Interestingly, all the models are less calibrated the Bike Sharing dataset using target Poisson distribution rather than target Normal distribution. This suggests a mismatch of the Poisson distribution for this particular task. The almost perfect OODom scores of NatPN validate again Thm. I which also holds for regression.
Inference Speed. We show the average inference time per batch for all models CIFAR-10 for clas- sification and the NYU Depth v2 dataset for regres- sion in Tab. 7 NatPN shows a significant improve- ment over Dropout and Ensemble which are both ap- proximately five times slower since they require five forward passes for prediction. Notably, the NatPN speedup does not deteriorate target error and un- certainty scores. NatPN is slightly slower than R- PriorNet, EnD² and EvReg as they do not evaluate an additional normalizing flow. However, NatPN which uses a single normalizing flow - is faster than PostNet which scales linearly w.r.t. the number of classes since it evaluates one normalizing flow per class. Lastly, NatPN is the only single-pass model that can be used for both tasks.
Table 7: Batched Inference Time (in ms), NVIDIA GTX 1080 Ti
CIFAR-10
NYU Depth v2 (batch size 4)
(batch size 4,0%6)
Dropout
407.91 +: 5.65
650.96 = 0.22 649.78 = 0.18
Ensemble
361.61 5.41
R-PriorNet
61.83 + 2.57
EnD
61.83 + 2.57
PostNet
88.56 0.06
EvReg
129.88 - 0.75
NatPN
75.64 = 0.04
137. - 0.18
NatPE
370.17 0.09
676.74 0.38
5 CONCLUSION
We introduce Natural Posterior Network which belongs to the family of models parametrizing conjugate prior distributions. NatPN is capable of efficient uncertainty estimation for any task where the target distribution is in the exponential family (incl. classification, regression and count prediction). NatPN relies on the Bayes formula and the general form of exponential family distributions perform a closed-form input-dependent posterior update over the target distribution. Further, an ensemble of NatPNs has a principled Bayesian combination interpretation. Theoretically, NatPN guarantees high uncertainty far from training data. Experimentally, NatPN achieves fast, versatile and high-quality uncertainty predictions with strong performance calibration and OOD detection.

---

Published as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Accurate uncertainty estimation aims at improving trust in safety-critical domains subject to automa- tion, and in a maintenance context where the underlying data distribution might slowly shift over time, In this regard, NatPN significantly improves the applicability of uncertainty estimation across wide range of prediction domains (e.g. classification, regression, count prediction, etc) while maintaining fast inference time. This could be particularly beneficial in industrial applications with time pressure and potential critical consequences (e.g. finance, medicine, policy decision making, etc).
Nonetheless, while NatPN achieves high-quality uncertainty estimation, there is always risk that NatPN does not fully capture the real-world complexity e.g. for OOD data close to ID data. Further- more, raise awareness about two other risks of excessive trust related to the Dunning-Kruger effect (Kruger& Dunning, 2000p: human excessive trust in Machine Learning model capacity, and human excessive trust in its interpretation capacity. Therefore, encourage practitioners proactively confront the model design and its uncertainty estimates to desired behaviors in real-world use cases.
7 REPRODUCIBILITY STATEMENT
We provide all datasets and the model code at the following project page In App. F we give de- tailed description for each dataset used in this paper. This description includes the task description, the dataset size, the input/output dimensions, the input/output preprossessing and the train/ caiin/vaidation/tess splits used in the experiments. In App- G we give detailed description for the architecture and grid search performed for each model used in this paper. Specifically, we provide an syyeer-parameer study for NatPN in App. [.6 App. we give a detailed description of the metrics used in the experiments. Finally, we provide a complete proof of Thm. detailed description of the Bayesian loss and relevant formulae for exponential family distributions in App. App. and App.
REFERENCES
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. NeurIPS, 2020.
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. ICLR. 2018.
Jordan Ash and Ryan P Adams. warm-starting neural network training. NeurIPS, 2020.
Marin Biloš, Bertrand Charpentier, and Stephan Günnemann. Uncertainty on asynchronous time event prediction. NeurIPS, 2019.
Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
P. G. Bissiri, C. C. Holmes, and S. G. Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2016.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. ICMI., 2015.
L. D. Brown. Fundamentals of Statistical Exponential Families: With Applications in Statistical Decision Theory. Institute of Mathematical Statistics, 1986.
David M Chan, Roshan Rao, Forrest Huang, and John F Canny. Gpu accelerated t-distributed stochastic neighbor embedding. JDPC, 2019.
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. NewrIPS, 2020.
Hugh Chipman, Edward George, and Robert Mcculloch. Bayesian ensemble learning, NewrIPS, 2007.
https: http:://wwww.... dam1 tum. de/natnn
10

---

Published as a conference paper at ICLR 2022
Hyunsun Choi, Eric Jang, and Alexander A Alemi Generative ensembles for robust anomaly detection. ICLR, 2019.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.
Persi. Diaconis and Donald Ylvisaker. Conjugate priors for exponential families, The Annals of Statistics, 1979.
James Dix. Existence of the limit at infinity for function that is integrable on the half line, Ros- Hulman Undergraduate Mathematics Journal, 2013.
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017.
Michael W. Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-An Ma. Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. JCML, 2020.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using multi-scale deep network. NeurIPS, 2014.
Sven Elflein, Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. On out-of-distribution detection with energy-based models. CoRR, 2021.
Hadi Fanaee-T and João Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 2014.
Sebastian Farquhar, Lewis Smith, and Yarin Gal. Liberty or depth: Deep bayesian neural nets do not need complex weight posterior approximations, NeurlPS, 2020.
Andrej Karpathy Fei-Fei Li and Justin Johnson. Tiny imagenet. herokuapp.com/ 2017.
imageneet
Angelos Filos, Sebastian Farquhar, Aidan N. Gomez, Tim G. J. Rudner, Zachary Kenton, Lewis
Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. A systematic comparison of bayesian deep learning robustness diabetic retinopathy tasks. arXiv preprint 1rXiv:19/2.1048/ 2019. 2019.
Andrew Y. K. Foong, David R. Burt, Yingzhen Li, and Richard E. Turner. On the expressiveness of approximate inference bayesian neural networks. NeurlPS, 2020.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model
uncertainty in deep learning. ICML., 2016.
Jochen Gast and Stefan Roth. Lightweight probabilistic deep networks. CVPR, 2018.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad. Wen Yang. Richard Bamler, and Xiao Xiang Zhu. A survey of uncertainty in deep neural networks. arXiv preprint arXiv: 2/07.03342. 2021.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013.
Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman and Hal//CRC, 2004.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. ICMI,,001.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007.

---

Published as a conference paper at ICLR 2022
Will Grathwohl, Kuan- Chan- Wang, Joern- Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2020.
Alex Graves. Practical variational inference for neural networks. NeurIPS, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ICML, 2017.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how mitigate the problem. CVPR, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. ICLR, 2019.
Jose Miguel Hermandez- and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. JCML, 2015.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Wilson. What are bayesian neural network posteriors really like? ICML, 2021.
Christoph Käding, Erik Rodner, Alexander Freytag, and Joachim Denzler. Fine-tuning deep neural networks in continuous learning scenarios. In ACCV Workshops, 2016.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? NeurIPS, 2017.
Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. AISTATS, 2012.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect out-of-distribution data. NeurlPS, 2020.
Anna-Kathrin Kopetzki, Bertrand Charpentier, Daniel Zügner, Sandhya Giri, and Stephan Günnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-basee models reliable? In ICML, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Justin Kruger and David Dunning. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessmeets. JPSP, 2000.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. ICML., 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. NeurIPS, 2017.
Balaji Lakshminarayanan. Dustin Tran, Jeremiah Liu, Shreyas Padhy, Tania Bedrax-Weed and Zi Lin. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. NeurIPS, 2020.
Charline Le Lan and Laurent Dinh. Perfect density models cannot guarantee anomaly detection. arXiv preprint arXi::0012.03800 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. 2010.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. ICCV, 2015.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. NeurlPS, 2019.
12

---

Published as a conference paper at ICLR 2022
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. NeurIPS, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. NeurIPS, 2019.
Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, and Mark Gales. Regression prior networks. arXiv preprini arXiv:2006. 11590, 2020a.
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. JCLR, 2020b.
Andrey Malinin, Neil Band, Ganshin, Alexander, German Chesnokov, Yarin Gal, Mark Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Roginskiy, Denis, Mariya Shmatova, Panos Tigas, and Boris Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv: 207.0:2077 2021.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. ICLR, 2020.
K. Monteith, J. L. Carroll, K. Seppi, and T. Martinez. Turning bayesian model averaging into bayesian model combination. IJCNN, 2011.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. ICML., 2020.
Warren R. Morningstar, Cusuh Ham, Andrew G. Gallagher, Balaji Lakshminarayanan, Alexander A. Alemi, and Joshua Dillon. Density of states estimation for out-of-distribution detection, arXiv preprint arXiv:2006.0922 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? ICLR, 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXiv preprint arXiv: (tti:::900029999 2020.
Jay Nandy, Wynne Hsu, and Mong-Li Lee. Towards maximizing the representation gap between in-domain & out-of-distribution examples. NeurlPS, 2020.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Constantin P. Niculescu and Florin Popovici. A note on the behavior of integrable functions at infinity. Journal of Mathematical Analysis and Applications, 2011.
Frank Nielsen and Richard Nock. Entropies and cross-entropies of exponential families. 2010.
2010.
Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with bayesian principles. arXiv preprint arXiv:I906 02506. 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. NeurlPS, 2019.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. NeurIPS, 2017.
13

---

Published as a conference paper at ICLR 2022
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. arXiv preprint arXiv:1908.0055 005988 2019.
Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. Intra order- preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems, 2020.
Rajesh Ranganath, Linpeng Tang, Laurent Charlin, and David Blei. Deep exponential families. AISTATS, 2015.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. ICML, 2015.
Hippolyt Ritter, Aleksandar Botev, and David Barber. scalable laplace approximation for neural networks. ICLR, 2018.
Otto Rudolf Rocktäschel. Methoden zur Berechnung der Gammafunktion für Komplexes Argumens. PhD thesis, Dresden University of Technology, 1922.
Roger Rosenkrantz. Prior Probabilities (1968). Springer Netherlands, 1989.
Murat Sensoy, Lance Kaplan, Federico Cerutti, and Maryam Saleki. Uncertainty-aware deep classi- fiers using generative models. AAAI, 2020.
John Shawe-Taylor and Robert C. Williamson. A pac analysis of a bayesian estimator. COLT, 1997.
Alexander Shekhovtsov and Boris Flach. Feed-forward propagation in probabilistic neural networks with categorical and max layers. ICLR, 2019.
Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for label- efficient deep learning. NeurlPS, 2020.
Edwin Simpson, Stephen Roberts, Ioannis Psorakis, and Arfon Smith. Dynamic bayesian combination of multiple imperfect classifiers. arXiv preprint arXiv: /206. 1831, 2012.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression, arXiv preprint 1rXiv:1905.0600) 2019.
Maximilian Stadler, Bertrand Charpentier, Simon Geisler, Daniel Zügner, and Stephan Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. In NeurlPS, 2021.
Dennis Ulmer. A survey on evidential deep learning for single-pass uncertainty estimation. CoRR, 2021.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. ICML, 2020.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409 2021.
Hao Wang, Xingjian SHI, and Dit-Yan Yeung. Natural-parameet networks: A class of probabilistic neural networks. NeurIPS, 2016.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. ICLR, 2020.
Florian Wenzel, Jasper Snock, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. NewrIPS, 2020.
Edmund Whittaker and George Watson. Course of Modern Analysis. Cambridge University Press, 1927.
14

---

Published as a conference paper at ICLR 2022
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, Taylan Cemgil, S. M. Ali Eslami, and Olaf Ronneberger. Contrastive training for improved out-of-distribution detection, arXiv preprint arXiv:2007.05560 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXii: 2017.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint 1566:::300603355 2015.
Arnold Zellner. Optimal information processing and bayes's theorem. The American Statistician, 1988.
Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. TPAMI, 2018.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. ICMI., 2020a.
Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. Uncertainty aware semi-supervised learning on graph data. NeurlPS, 2020b.
15

---

Published as a conference paper at ICLR 2022
A THEOREM 1
We prove theorem based on Lemmas 2 and 3 Lemma
2 states that the input space can be divided
in finite number of linear regions (Arora et al. 2018). Lemma 3 states that a probability density
with bounded derivatives has to converge to 0 at infinity Dix, 2013 B. We additionally recall Lemma 4
which provide a similar convergence guarantee without the bounded derivative constraint (Niculescu & Popovici 2011). Finally, Lemma [S] particularly shows that the guarantee of theorem I can be obtained with Gaussian Mixtures which are commonly used for density estimation or radial flows which are used in the experiments.
Lemma 2. (Arora et al. 2018) Let {Qi)" be the set of linear regions associated to the piecewise ReLU network fo(x). For any E R2, there exists 8 E Rt and I* E L....,, R such that òæ Q for all ó > 8".
This convergence is stronger than Lem. as the limit is not in density but with standard limit notation. Lemma 4. Niculescu & Popovici 201 Let I,- (0, 00), then p(5) 6-000
Lemma 3. Dix 2013) Let p E L' (0, oo) with bounded first derivative p', then p(o)
5-200
O in density. This means
that the sets where p(t) is far from its 0 limit (i.e. {t 2 0: lp(t) 2 é} with e 0) has zero density. Lemma 5. Let IP(z w) be parametrized with Gaussian Mixture Model (GMM) radial flow, then (( w)
0.
||=||-000
Proof. We prove now lem. for GMM and radial flow. The proof is straightforward for the GMM parametrization since every Gaussian component of the mixture has 0 limit when ||=|| + co.
Let denote now pl (z) = [P( w) be parametrized with a radial flow transformation g(z) and a base
unit Gaussian distribution pa i.e.:
(z) = (=() x |det 0g(z)
8z
Further, we can express the transformation g(z) and its determinant det 0o(x) 8z follows:
g(z) = + 8h((a r) z0)
det 0g(2) 1+ 1+,h(a,,)) + Bh'
Oz
(1 ()),))))-11
where h(a,r) 1 and r -= || zoll. On one hand, we have [[g())] too when ||ж|| + oo atr since (1 (2)2 <
Thus, the base Gaussian density (g( - 0 when ||||| - oc. On the other hand, we have |det 0g(z)) + 1 since 3h(a, r) + 0 and 8h (a, r)r - 0 when ||||| + co.
Therefore, the transformed density (o()))) x |det (g(2)] -> when ||||| + oo which ends the proof. Note that this proof can be extended to stacked radial flows by induction.
dz
Theorem. Let a NatPN model parametrized with (deep) encoder fo with piecewise ReLU acti- vations, decoder gø and the density (p w). Let fa(ar) = V n + (! be the piecewise affine representation of the ReLU network fe on the finite number of affine regions Q1) Arora et al. 2018). Suppose that V have independent rows and the density function P( w) has bounded derivatives, then for almost any x we have (P(f((6 (5 x:) (w) $-000
O. ie the evidence becomes small far from
training data.
I Let æ e R2 be a non-zero input and fo be ReLU network. Lem. implies that there exists 8* E IR and / E {l,..., R} such that x E Q(h) for all 8 > 8*. Thus, 2x = fo x) = 6- (V + ( for all 8 > 8*. Note that for 5 E +oo). zs follows an affine
Proof. We prove now thm.
half line Sg = {z z = 8. [) (1)),
8*3 in the latent space. Further, note that (10) + too since x + 0 and y()) has independent rows.
and || || 00
We now define the function p(0) = P(xs () which is the density function P(x w) restricted on the affine half line Sz. Since (p(z w)) is normalized probability density, then the function 8 + p(ō - 8*)
16

---

Published as a conference paper at ICLR 2022
is integrable [0, +o0 Indeed we have:
+00
p 5")
p(0)d8
IP (V()x) + a w))d
P(z wlld +00
Further since the function P(z (w) has bounded derivatives, we can apply Lem. 3 to the function 5 + p(8 8*) to get the expected result i.e.
P(fo(ò x)) w) = p(d) p((0 + - -0) 60200
0
which ends the proof.
Alternatively a slightly weaker conclusion also holds if the density function does not have bounded derivatives using lem. 4 (instead of lem. with the notion of limit in density. The stronger conclusion valid we parametrize (p( w) with a Gaussian Mixture Model or a radial flow density according to lem. since ||=s|| 6-100 +00.
Further, we provide additional comments the assumption that trained network converges to linear transformation with exactly two or more dependent rows in Thm. [ Under this realistic condition Hein et al. (2019), the null space is reduced to O according to the rank-nullity theorem meaning that there should be no dead input feature/pixel. If this condition does not hold, this would mean that this specific input feature/pixel is not informative for the prediction task. Thus it could be desired in practice that it does not affect the uncertainty on the prediction. This latter aspect is discussed in the "Task-Spechi OOD" paragraph in Sec. 3.6
B BAYESIAN LOSS
NatPN minimizes the following Bayesian formulation:
c -ggr gere. [log ((y (5) [(0
H
(i)
where [0) denotes the entropy of the predicted posterior distribution (1) This loss is generally not equal to the ELBO loss. While the term (i) can be viewed KL regularization, the term (ii) is not necessarily equal to the prior KI. regularization term in the ELBO loss since proper uniform prior might not exist (e.g. the target y real number). Indeed, if the target y is a real number, there exists no uniform prior on O and the Bayesian loss and ELBO loss are different i.c. KL @ (() log(
ELBO loss without
(o) dd * Q()) log Q(0)d0
Q(0)
H(Q).
Nonetheless, when uniform prior Q!iif exists (e.g. the target y is class), the loss optimiza- tion can be seen as an amortized variational optimization of an ELBO loss (Zhang et al., 2018)
i.e. (() - Equaa.c [log () (()) + (L) Qumif]]] where the predicted distribution (1)st,(() is the variational distribution - which approximates the true posterior distribution. In- deed, the KL regularization term equal to the entropy regularization term i.e. KL(Q) || "
[Q()) log( qei d0) x [())) log ((0)d0) = HQ).. Hence, the loss name "Bayesian loss" CChar
C(8)
pentier al. 2020) is motivated by its difference with the ELBO loss and its Bayesian property at optimum.
C FORMULAE FOR EXPONENTIAL FAMILY DISTRIBUTIONS
C.I GENERAL CASE
Target Distribution. An exponential family distribution on a target variable E RR with natural parameters E TR can be denoted as
Ply 0) h((y) exp (0T ull)) A(0))
(8)
17

---

Published as a conference paper at ICLR 2022
where h R - R is the carrier measure, A A: IR the log normalizzer and u R the sufficient statistics.
Conjugate Prior Distribution. An exponential family distribution IP always admits conjugate prior:
Q x. n) - 7(x, n) exp (n 0" - -))(0)))
where 7 RR x R + R a normalization factor and A the nog-normalizer of the distribution as in Eq. 8).
Posterior Predictive Distribution. The posterior predictive distribution is given as
[ [P((())) ()) Q(0)x , (P dd where the parameter 0 marginalized out (Gal 2016). This distribution always be computed in closed form for exponential family distributions:
n7(,nn
Pl X n) h(y)
(10)
17
ux+ n+1
n .n+1
where is the carrier measure defined Eq. rj is the normalization factor defined in Eq. In par- ticular, the posterior predictive distributions for Categorical, Normal and Poisson target distributions are Categorical, Student and Negative Binomial distributions, respectively.
Likelihood. The log-likelihood of exponential family distribution can be written as follows: log P(
() log h)) 0" u ()) A(0)
Expected Log-I Ligg-iiellhhod. Given the log-likelihood of exponential family distribution, its expec- tation under the conjugate prior distribution (0)x, n.) can be written as
_ ) [lo (p) 0)] log (y (3) + IE Q(x)Q(x))) 00' u(y nx, (12)
where Eq( xm) [0] x Brown 1986
Diaconis Ylvisaker 1979
Entropy. The entropy of random variable y 1P()(0) for exponential family distribution P can be written as follows (Niclsen & Nock, 2010):
HT 0)] = A(O) ((0)) E log h(y)
(13)
C.2 CATEGORICAL & DIRICHLET DISTRIBUTIONS
The Dirichlet distribution p Dir(a:) is the conjugate prior of the categorical distributions y Cat(p).
Target Distribution. The density and the entropy of the categorical distribution are:
K
Cat(y p)
Iyy 1]p,
(14)
H[Cat(p)) =
log
(15)
c=1
Conjugate Prior Distribution. The density and the entropy of the Dirichlet distribution are:
r' (EE a.)
Dir(pla) I ((ac))
-
(16)
H[Dir(a)] = log B(a) + (ao C)}(an) - - 1)bb(a,)
(17)
where (aa and B(x) denote Digamma and Beta functions, respectively, and cxo = Ee Ole.
Expected Likelikel The expected likelihood categorical distribution Cal(p) (p) under the Dirichlet distribution Dir(ce is
Bp-Dit(a) [log Cal(y) (p)) = v(a,)) v(()))
(18)
where (a) denotes Digamma function.
18

---

Published as a conference paper at ICLR 2022
C.3 NORMAL & NORMAL-INVERSS--AANE DISTRIBUTIONS
The Normal-Inverse-Gaaee Inverse-Gamma (NIG) distribution O ~ NT X, at 3) is the conjugate prior of the normal distribution y ~ ((,,,,, Note that both parameters À and a can be viewed as pseudo- counts. However, the natural prior parametrization enforces a single pseudo-count n corresponding to 2a.
Target Distribution. The density and the entropy of the Normal distribution are:
(- (xr (x- 11)
N(ylu, 2v
exp
(19)
20²
H[W(H,)) = 2 log(2To²)
(20)
Conjugate Prior Distribution. The density and the entropy of the NIG distribution are:
8aVA
a+1
23 + M1x {p-Ma))
N po, A, a)
exp
(21)
I(a)v 2TO2
02
202
HWTT X, a. B)] 2 + log (2m))
1
log ta
(at 3)
2) v(aa)
(22)
2
where ((x:) denotes the Gamma function.
Expected Log-Likelihood. The expected likelihood of the Normal distribution W(,, σ) under the NIG distribution .T (mo, A, o, () is:
((A))) (p.A, [log.V]] N(y 14, o))
(23)
(y u2²
E
log(o
(24)
2x
1
(y
12)2
E
- E
[log 2²] - log 2nT
(25)
2
20
1
1 o
EE
2y IE
E
+ E
log
log 2m
(26)
2
6
02
1
po)2
((a)
logg B log 2TT
(27)
2
where (a)) denotes the Digamma function. We used here the moments of the NIG distribution
E [ 3 [ 3
X and the moment of the inverse Gamma distribution
E log
g2 (a)) log 3.
C.4 POISSON & GAMMA DISTRIBUTIONS
The Gamma distribution À ~ [(, .3) is the conjugate prior of the Poisson distributions y ~ Poi(A).
Target Distribution. The density and the entropy of the Poisson distribution are:
Poi(y A) A exx((())
(28)
y!
log(k!)
Poi()) = A(1 - log()))) + exp(-
(29)
k!
k=0
Conjugate Prior Distribution. The density and the entropy of the Gamma distribution are:
g
r(A aa,3) 3) a xa-1expl exp(-BA)
(30)
r(a)
[ 8)] log ((a) log 3 (1 a)p((a
(31)
19

---

Published as a conference paper at ICLR 2022
where r(a) denotes the Gamma function.
Expected Log-Likedn Lig-Liiel, The expected likelihood the Poisson distribution Poi(A) under the Gamma distribution I( 3) is
E (a log Poi(y/ A)] [Bllog Aly [E[A
Σ
log
(32)
k=1
(vb(a)) - log B)y
log k
(33)
8
where eb(a) denotes Digamma function. We used here the moments the Gamma distributions Ellog Aj (b)))= - log 8 and B[X] Note that Σ k=1 log & is constant w.r.t. parameters a, B.
D APPROXIMATION OF ENTROPIES
The computation of a distribution's 's entropy often requires subtracting huge numbers from each other. While these numbers tend to be very close together, this introduces numerical challenges. For large parameter values, we therefore approximate the entropy by substituting numerically unstable terms and simplifying the resulting formula. For this procedure, we make use of the following equivalences (taken from Rocktäschel (1922) and Whittaker & Watson (927). respectively):
1 log [(x) log 2T
1
log
(34)
2
2
1
1
(xxx) log x
0
(35)
2x
We note that Eq. 35 especially implies (x) z log x and x (x) = x: ælog for large x.
D.1 DIRICHLET DISTRIBUTION
We consider a Dirichlet distribution Dir(x)) of order K with og = Eim liol Cr. For ag 2 104, we use the following approximation:
K
K
K-
H [ir(a)) 2J
(1 + log 2n)) +
log ai-
K -K-
log
(36)
2
2
1=1
D.2 NORMAL-INTERSF-GAMMS DISTRIBUTION
We consider Normal-Inverse- Inverse-Gamme distribution .T (H, X, a, 9). For 104, we use the following approximation:
1
H [WT (n, ~ + log 2x 22lo a + log 3 log A
(37)
2 2
D.3 GAMMA DISTRIBUTION
We consider Gamma distribution [(a, B). For 2 104, we use the following approximation:
H (a, B)] log 2T log a log 8
(38)
E FORMULAE FOR UNCERTAINTY ESTIMATES
Aleatoric Uncertainty. The entropy of the target distribution PPy was used to estimate the aleatoric uncertainty i.e. H Hv))))
Epistemic Uncertainty.
The evidence parameter (1) used to estimate the epistemic uncer- tainty. Due to its interpretation as a pseudo-count of observed labels, the posterior evidence parameter indeed natural indicator for the epistemic uncertainty.
20

---

Published as a conference paper at ICLR 2022
Predictive Uncertainty. The entropy of the posterior distribution @ (())) (o0s,.))) m)aa(()) estimate the predictive uncertainty.
was used to
F DATASET DETAILS
We use rrant/valiatiion/test split in all experiments. For datasets with dedicated test split, split the rest of the data into training and validation sets of size 80%/20%. For all other datasets, we used 70%/15% for the train/validationntess sets. All inputs are rescaled with zero mean and unit variance, Similarly, we also scale the output target for regression. We provide the datasets at the project page
Drive (Dua & Graff, (017) This is tabular dataset where the goal is to classify extracted motor current measurements into 11 different classes. We remove the last two classes (9 and 10) from training and use them as the OOD dataset for OOD detection experiments. Each input is composed of 48 attributes describing motor behavior. The dataset contains 58, 509 samples in total.
Sensorless
MNIST (LeCun et al., 2010) & Fashion-MNIST (Xiao et al., 2017) These are image dataset where the goal is classify pictures of hand-drawn digits into 10 classes (from digit digit 9) or classify pictures of clothers. Each input is composed of 1 x 28 x 28 tensor. The dataset contains 70, 000 samples. For OOD detection experiments againt MNIST data, we use KMNIST (Clanuwat et al., 2018) and Fashion-MNIST (Xiao et al. 2017) containing images of Japanese characters and images of clothes, respectively. For OOD detection experiments againt Fasgion-MNIST" data, we use KMNIST (Clanuwat et al. 2018) and MNIST (LeCun et al. 2010) containing images of Japanese characters and images of digits, respectively. It uses the MIT License (MIT).
CIFAR-10 Krizhevsky et al. 2009) This is image dataset where the goal is to classify a picture of objects into 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). Each input is 3 32 32 tensor. The dataset contains 60, 000 samples. For OOD detection experiments, we use street view house numbers (SVHN) (Netzer et al., (011]) containing images of numbers and CelebA (Liu et al. 2015) containing images of celebrity faces. We do not use CIFAR 100 (Krizhevsky jet al. 2009) or Tiny Imyaageeeet (Fei-Fei Li & Johnson 2017) as OOD as they also contain images of vehicles and animals similar to CIFARIÓ. This rightly questions to what extent are these datasets really OOD for CIFARIO. Furthermore, we generate the corrupted CIFAR- 10 dataset (Hendrycks & Dietterich 2019) with 15 corruption types per image, each with different severities. It uses the MIT License (MIT).
Bike Sharing (Fanaee-T Gama 2014) This is tabular dataset where the goal is to predict the total number of rentals within an hour. Each input is composed of 15 attributes. We removed features related to the year period (i.e. record index, date, season, months) which would make OOD detection trivial, leading to 11 attributes. The dataset contains 17, 389 samples in total. For OOD detection, we removed the attribute season from the input data and only trained on the summer season. The samples related to winter, spring and autumn were used as OOD datasets.
Conerete (Dua& & Graff, 2017) This is tabular dataset where the goal is to predict the compressive strength of high-performance concrete. Each input is composed of 8 attributes. The dataset contains 1, 030 samples in total. For OOD detection, we use the Energy and Kin8nm datasets which have the same input size.
Kin8nm (Dua & Graff, 2017) This is a tabular dataset where the goal is to predict the forward kinematics of an 8-link robot arm. Each input is composed of 8 attributes. The dataset contains 8, 192 samples in total. For OOD detection, we the Concrete and Energy datasets which have the same input
size.
NYU Depth v2 (Nathan Silberman & Fergus 2012) This is an image dataset where the goal is to predict the depth of room images each pixel position. All inputs of shape 3x640x480 tensors while we rescale outputs to be 320x240 tensors at both training and test time. This setting is slightly different from Kendall & Gal (2017) and Nathan Silberman & Fergus (0122). Indeed. (Kendall & Gall 2017) up-scales the model output to 640x480 at training and test time while (Nathan Silberman & Fergus 2012) up-scales the model output to 640x480 at test time only. The dataset contains 50, 000
httpp::///// http:://wwwwddr www.tuum.eeeeeeee de/natpn
21

---

Published as a conference paper at ICLR 2022
samples in total available the DenseDepth GitHub For OOD detection, we use the KITTI (Geiger et al. 2013) dataset containing images of driving cars and two out of the 20 categories from the
LSUN (Yu et al. 2015) dataset.
G MODEL DETAILS
We train all models using 5 seeds except for the large NYU dataset where we use a single randomly selected seed. All models are optimized with the Adam optimizer without further learning rate scheduling. We perform early stopping by checking loss improvement every epoch and patience
selected per dataset (Sensorless Drive: p 15, MNIST: p 15, CIFARIO: p 20, Bike Sharing:
P 50, Concrete: = 50, Kin8nm: p 30, NYU Depth v2: p 2). We train all models on single GPU (NVIDIA GTX 1080 Ti or NVIDIA GTX 2080 Ti, 11 GB memory). All models are trained after grid search for the learning rate in [1e-2,50 55 "j. The backbone architecture is shared across models and selected per dataset to match the task needs (Sensorless Drive: lin. layers with 64 hidden dim, MNIST: 6 conv. layers with 32/32/32/64/641/4 filters + lin. layers with hidden dim 1024/128/64, CIFARIO: conv. layers with 32/32/32/64/64/128/ 128/122 filters + 3 lin. layers with hidden dim 1024/128/64, Bike Sharing: 3 lin. layers with 16/16/16 hidden dim, Concrete: 2 lin. layers with 16/16 hidden dim, Kin8nm: 2 lin. layers with 16/16 hidden dim, NYU Depth v2: DenseDepth + 4 upsampling layers with convolutions and skip connections). For the NYU Depth v2 dataset, a pretrained DenseNet for initialization of the backbone architecture which was fine-tuned during training. The remaining layers are trained from scratch. All architectures use Leaky R activations. For further details, we provide the code at the project page
Baselines. For the dropout models, we use the best drop out rate Pdrop per dataset after a grid search in f0.1,0.2. 0.25, 0.4] and sample 5 times for uncertainty estimation. Similarly, we use m = 5 for the ensemble bascline and the distribution distillation. Note Ohaddvaadia et al. (2019) found that a relative small ensemble size (e.g. m 5) may indeed be sufficient in practice. We also train Prior Networks where we set P.m le as suggested in the original papers (Malinin & Gales, 2018 2019). Prior Networks use MaSSion-MNIST and SVHN as training OOD datasets for MNIST and CIFAR-10. respectively. As there is no available OOD dataset for the Sensorless Drive dataset, we use Gaussian noise training OOD data.
Natural Posterior Network. We perform a grid search for the entropy regularizer À in the range [Le-5, 0), for the latent dimension H in {4, 8, 16, 32). for the certainty budget NH in fet# et clog(V(tHHH) ). and for normalizing flow type between radial flows (Rezende & Mohamed 2015) with 8, 16 layers and Masked Autoregressive flows (Germain et al. |0 Papamakarios et al. 2017) with 4, 8, 16 layers. Further results on latent dimensions, density types, number of normalizing flow layers and certainty budget presented in Sec. [66] We use "warm-up" training for the nor- malizing flows for all datasets except for the simple Concrete and Kin8nm datasets, and the NYU Depth v2 dataset which starts from pretrained encoder. We use "fine-tuning" for the normalizing flows for all datasets except for the simple Concrete and Kin8nm datasets. As prior parameters, we set X prior 1c/C, prior C for classification, X prior = (0, 100)² n for regression and X = 1, prior = 1 for count prediction. Note that the mean of these prior distributions correspond to an equiprobable Categorical distribution Cat (1c/C), Normal distribution with large variance (O 10) and a Poisson distribution with unitary mean Poi(1). Those prior target distributions represent the safe default prediction when no evidence predicted.
H EXPERIMENT DETAILS
Target Error Metric, For classification, we use the standard accuracy 1// Σ, T (00 y ()) where y (0) is the one-hot true label and y ()) is the one-hot predicted label. For regression, we use the
standard Root Mean Square Error V +/2 Σ (1) y2)
Calibration Metric. For classification, we use the Brier score which is computed as J/
20 2/2 where ( is the predicted sofumax probability and (+) is the one-hot encoded ground-truth
https://qithub.. com www..///o/aaaaaaaiiitteeteeeeeee https: hwtp///ww.w0
tum. tww.de//atmmmnm
22

---

Published as a conference paper at ICLR 2022
label. For regression and count prediction, p and the percentage of target lying in the confidence interval 1 = [0. b
the absolute difference between the percentile 2. under the predicted target distribution. Formally, we compute Ppred N Σ. For (v)) E Ipl where is the cumulative function of the predicted target distribution evaluated at the true target. For example, the percentile = 0.1 would be compared to Ppred = 1
1
Fyrs ()) (p(y S y²((0) (
N Ex Fecc (u-.()) E [0, 0.05] U [0.95, 1]] which should be close to 0.10 for calibrated predictions. We compute a single calibration score by summing the square difference for p E (0.1....,0.. 0.9) i.e.
V Ex Ppred)2 (Kuleshov et al. 2018).
OOD Metric. The OOD detection task can be evaluated as binary classification. Hence, we assign class I to ID data and class to OOD data task and use the aleatoric and epistemic uncertainty estimates as scores for OOD data. It enables to compute final scores using the area under the precision-recall curve (AUC-PR) and the area under the receiver operating characteristic curve (AUC-ROC). Both metries have been scaled by 100. We obtain numbers in [0, 100] for all scores instead of [0. 1]. Results for AUC-ROC are reported in Sec. 1.7 For the aleatoric uncertainty, we use the negative entropy of the predicted target distribution. For the epistemic uncertainty, the predicted evidence for models parametrizing conjugate-prios or the variance of the predicted winning probability class for classification and the variance of the mean for regression and class count prediction for ensemble or dropout.
Inference Time Metric. We measure inference time of models in and used NVIDIA GTX 1080 Ti GPUs. We evaluate the inference for one classification dataset (CIFAR-10) 10) and one regression dataset (NYU Depth v2). For evaluation, we use randomly initialized model and simply push random data through the model with batch size of 4.096 CIFAR10 and batch size of 4 for NYU Depth v2. The final numbers are averaged over 100 batches excluding the first batch due to GPU initialization. Compared models shared the same backbone architecture.
I ADDITIONAL EXPERIMENTS
1.1 MNIST, FASHION MNIST, CIFARIO AND BIKE SHARING RESULTS
Table 8: Results on MNIST (classification with Categorical target distribution). Best scores among all single-pass models in bold. Best scores among all models are starred. Gray numbers indicate that R-PriorNet has seen samples from the FMNIST dataset during training.
Accuracy
Brier
K. Alea.
K. Epist.
Aleaa
F. Fgist.
000om Alea.
0ODom Fpist.
Dropont NatPE
1.07 0.05
0.05
97.2 0.08
*09
98.01 0.14
1.60
0.90
0.05
0:05
98.73
1,50
0.01
10.44
96.39 0.73
199.61
97.49 0.85
199.70
1100000 = 0.001
1100000 +
StandardNet
96.91 0.06
1.51 + 0.14
95.81 = 0.44
96.29 1.04
47.53 + 3.44
SNGP
99.34 2: 0.03
2,60 2: 0.04
98.85
0.11
98.04
0.34
+100,00 2: 0.00
R Prlooreet
0.0c
0,97 1: 0.03
99.28 0.25
100.00
0.00
100000 0.00
97.48 1: 0.66
31.05 4 0.13
6.39 +
0.15
98.76
0:13
99,25
0.16
99.35
+ 1.38
31.601 + 11.39
PostNet
09.36
0.02
1:33 1: 0.04
98.85 0.05
08.79
0:07
89
0.23
06. 85
0.23
1100.00 0.00
*10000 :: 0.00
NadPN
99.47 1: 0.02
1.09 1: 0.03
99.20 = 0.20
99.39 1: 0.08
99.16
0.28
99.54 0.09
99.99 +: (101
*100,00 :: 0.00
Table 9: Results on FMNIST (classification with Categorical target distribution). Best scores among all single-pass models are in bold. Best scores among all models are starred. Gray numbers indicate that R- PriorNet has seen samples from the KMNIST dataset during training.
Accurucy
Alea.
Epist.
Alea.
Egist.
OODom Epist.
Dropout
92.46 1. 0.17
13.89 1 0.31
60.75 1 1.41
75.85 1 1.73
76.57
1:30
92.48
0.16
39,97 0.69
50,90 1 1.74
Eusembble
92.64 0.10
13.63 0.25
77.14 # 1.49
98.78 0.75
W6.20 +: 0.76
95.16 2 0.35
37.30 = 0.43
82.93 # 1.96
NaPP::
992.9 : 0.06
14:44 0:06
0.38 82.56
96.38 0.29
92.12 02122:0.11 0.17
+98.7022 0.09
100000 2: 0.00
+10000 ::
StandardNet
17.12
71811
52.82 0.73
SNGP
1673
89.40
95.31
100.00
R: R-PriorNet
91.53 0.10
*12.21 0.20
#98.3 1. 0.09
#99.54 0.18
99.96 0.02
99.99 0.00
72.23
6.32
48.51 1. 6.09
EaD"
91.34
0,83
29:23 0.79
79.32 +: 1.39
91.61 1.04
91. 0u06
98.36 0.20
43.70 =: 3.37
26.73 4: 3.74
PodNNt
91.04 2 0.10
16 0.30
90.56 2: 1:25
92.10 2: 1.77
*96.65 :: 0.33
07.06-2 0.42
1100.0 :: 0.00
*100U0 :: (.00)
NarPN
91.65.1 0.14
81.12 2.77
95.31 0.1
93.05
98.35
99,99 0.01
1100.0 1. 0.00
23

---

Published as a conference paper at ICLR 2022
Table 10: Classification results on CIFAR-10 with Categorical target distribution. Best scores among all single-pass models in bold. Best scores among all models are starred. Gray numbers indicate that R-PriorNet has seen samples from the SVHN dataset during training.
Accuracy
Brier
SVHN Alea.
SVIN Egiet
CelebA Aleo.
Celeb. Epist.
0000m Aleo.
00Dum Eieet
Drepout
88.15
71.04
1842-
Ensembble
969.05 0.1.1
17.33 0.17
$5.26 0.84
82.51
76.20 0.87
76.25 0.78
25.30 400
89.21
NatPE
89.21
0.09
17.41 0.12
$5.06 0.34
98316 0.67
78.95 1 1.15
*8206
87:27 1.79
+98.88 0.26
RePriorNNtt
7.00
42:31 8.77
Emit"
40.84
186447
0.66
14.84-1
75.4 1.8
42.19 8.77
PontNet
72.96
75.44
1.61
87.90
0.46
$2.20
76.01
76.87
3.38
99.67 300
Table 11: Results on the Bike Sharing Dataset with Normal A and Poison Poi target distributions. Best scores among all single-pass models are in bold. Best scores among all models are starred.
RMSE
Calibration
Winter Epist.
Spring Epist.
Epist.
OODom Epist.
Dropout-N
70.20
1.30
6.05 1. 0.77
15.26 0.51
13.66 1 0.16
15.11 1 0.46
99.99 1 0.01 *100.00
Ensemble-W
42.46
StandardNet.W
58.49 4.37
2.32 + 0.88
EyReg-N
49.58
1.51
3.77 0.81
17.19 0.76
15.54 0.65
14.75 0.29
34.99 17.02
NatPN.V
49.85
1.38
*1.8 0.34
*55.04 6.81
223.5 1.20
+27.78 2.47
*100.00 + 0.00
1.2 UNCERTAINTY VISUALIZATION ON ToY DATASETS
We visualize the aleatoric and the epistemic uncertainty for two toy datasets with three classes with the same number of training examples for the three classes (see Fig. 4 and different number of training examples for the three classes (see Fig. 5). The predictions close to training samples. The preidctions are more epistemically uncertain close to fewer training examples and very epistemically uncertain for region far from all training data.
more aleatorically certain
Figure 4: Visualization of the alcatoric and epistemic uncertainty 2D toy dataset with 3 classes with 900 training samples for each class.
Model Predictions
Epistemic Certainty
Class
Class
Class
Class #3
Class #3
Certainty
Epistamic Cortainty
Figure 5: Visualization of the aleatoric and epistemic uncertainty on 2D toy dataset with 3 classes with 900 training samples for class (green), 600 training samples for class 2 (red) and 300 training samples for class 2 (blue).
Model Predictions
Epistemic Certainty
Class #2
Class
Class #t
Class
Alestoric Certainty
Epistemic Certainty
24

---

Published as a conference paper at ICLR 2022
1.3 LATENT SPACE VISUALIZATIONS
We propose additional visualizations of the latent space for MNIST with t-SNE (Chan al. 2019) with different perplexities (see Fig. 66. For all perplexities. we clearly observe ten green clusters corresponding to the ten classes for MNIST. The KNMIST (OOD) samples red can easily be separated from the MNIST (ID) samples in green. As desired, NatPN assigns higher log-proe ong-robabbilities used evidence computation to ID samples from MNIST.
Embedding
Predicsed Lialltod
Figure 6: t-SNE visualization of the latent space of NatPN on MNIST (ID) vs KMNIST (OOD). On the left, The ID data (MNIST in green) can easily be distinguished from the OOD data (KMNIST in red). On the right, NatPN correctly assigns higher likelihood to ID data.
1.4 HISTOGRAM OF UNCERTAINTY ESTIMATES
We visualize the histogram distribution of the entropy of the posterior distribution accounting for predictive uncertainty for ID (MNIST/NYU) and OOD (KNMIST and Pashion-MNIST/LSUUN classroom and LSUN church KITTI) (see Fig. 7). We clearly observe lower predictive entropy for ID data than for OOD data for both MNIST and NYU datasets. On one hand, the entropy clearly differentiates between ID data (MNIST) and any other OOD datasets (KMNIST, Fashion MNIST, OODom) for classification. We intuitively explain this clear distinction since the samples from the OOD datasets are irrelevant for the digit classification task. On the other hand, the entropy still good indicator of ID (NYU) and OOD datasets (LSUN classroom and LSUN church + KITTI) for regression although the distinction between ID and OOD datasets is less strong compared to MNIST. We intuitively explain this behavior since the task of depth estimation is still relevant to LSUN classroom and LSUN church KITTI.
Figure 7: Histogram of the entropy of the posterior distribution accounting for the predictive uncer- tainty of NatPN on MNIST (ID) vs KMNIST, Fashion-MNIST Out-Of-Domain (OOD) and NYU (ID) vs LSUN classroom and LSUN church KITTI (OOD). In both cases, low entropy good indicator of in-distribution data.
MMST (eDrbbbbudtoe) (Dut Butorr Fashion of distirtiionn
0.8
Clsssoom (Dut LSUN Church 1 (abiiiii
Density
0.66
Out Domaann
0.6
0.1
140
-20 Entropy
-40
-10
Entresy
25

---

Published as a conference paper at ICLR 2022
1.5 UNCERTAINTY VISUALIZATION ON NYU DEPTH V2 DATASET
We visualize the prediction and the predictive uncertainty per pixel for the NYU Depth v2 dataset (see Fig.(8.. [8. We observe accurate target predictions compared to the ground truth depth of the images. Further NatPN assigns higher uncertainty for pixels close to object edges, which is reasonable since the depth abruptly change at these locations.
Figure 8: Visualization of the predicted depth and predictive uncertainty estimates of NatPN per pixel on the NYU Depth v2 dataset. NatPN predicts accurate depth uncertainty and reasonably assigns higher uncertainty to object edges.
Ground trah beyth
Ground Trahh Septh
1.6 HYPER-PARAMETEE STUDY
As ablation study, we also report the results of the grid search on the latent dimension, normalizing flow types and number of normalizing flow layers for MNIST, CIFAR- 10 and Bike Sharing datasets in Tab. 12 13
14 15 While most models converge to fairly good uncertainty estimates, notice that 16 layers of simple radial flows on latent spaces of 16 dimensions were achieving very good results in practice.
Changing the flow type or the number normalizing flow layers does not lead to strong variations of the results except for Bike sharing with Poisson target distributions. In this case, more complex MAF normalizing flows improve NatPN performance.
The latent dimension appears to be more important choice for the model convergence. As example, a higher latent dimension of 16 or 32 leads to significantly better performances than a latent dimension of 4 on MNIST, CIFARIO and Bike Sharing datasets. We hypothesize that too low latent dimensions are less able to encode the necessary information for the prediction task, leading to worse target errors.
Further, we compare three different variants of the certainty budget NH used in the evidence com- putation () Nu (() w): a constant unit budget (i.e. Nu = (I)) corresponding to a fixed budget regardless of the number of training data and the latent dimension, a budget equals to the number of training data (i.e. N₂ = N (II) similarly to Charrpenneeer et al. (2020), or budget which scales exponentially with the number of latent dimensions (c.g. NH equal to e H (III), e (IV) or (oo((Vá)) (V)). We observe that scaling the budget w.r.t. the latent space dimension (H- -udget) is more stable in practice than constant budget (1-budget) and budget related to the number of training data (N-budget) (see. Tab. 13 15). In aprticular, the H Huddetsss achieve more better results
26

---

Published as a conference paper at ICLR 2022
higher latent dimensions and performance on par with the other certainty budget scheme otherwise. The intuition is that due to the curse of dimensionality, the expected value of a probability density function E2 [P)()) tends to decrease exponentially fast. For example, we have [=]] (vin)
when x ~ (0, 1) in H-dimensional space. Increasing the certainty budget N1 exponentially w.r.t. to the dimension H avoids numerical issues by allocating close to 0 evidence to latent representations. In our experiments, we use a grid search in different exponential scaling NH equal to e (III), e! (IV), e Jogl ( Hf (V).
1.7 OOD DETECTION WITH AUC-ROC SCORES
In addition to the AUC-APR scores, we report the OOD detection results for MNIST, CIFARIO and Bike Sharing datasets in Tab. 16 17 18 with AUC-ROC scores. Similarly with AUC-APR scores, NatPN and NatPE achieve very competitive performances compare to the baselines. It particular, they outperform all baselines detect challenging OODom data.
Table 12: MNIST comparison (<latent dim> <certainty budget> <radial layers>/<MAF layyerss
Bold and starred number indicate best score among all models.
Accuracc
Ala.
Epia.
E.Ala..
Epis.
000am Alea.
001oon Epiet
H1 1--,, Mar =1, Flolo - BD
64.33 =: 3.98 77:46 1: 6.36
61,92 +: 3.79
89.70 + 1.65
87.00- 2: 2.47
50007 2:343
89:52 #: 2.25
*100.00 = 0.00
1100.00 0,00
4. No. =1. =
91-48
93.45
93.08 1: 1.00
59.01 2: 5.04
+106 (.00
Nar -lloww Niy L. D8
75.04 10.10
3.34
92.64
91:50 :
1100.01 0.00
44, Nir II.
99:33
0.05
67.54
71.57 : 4.00
1100.00 =
0.00
160
=4, Nu HI. Flow
98.04
+100.00 0,00
N Nwr HI, Flor
0.33
1.57
98
0.25
ST.57
98,35
0.44
900
1100.11 0,80
If 44, Nur
Flow
78.14
48.46 4.78
88.79 1
87.28 1.56
90.55 1.10
89.50 1.58
110000
99,59
0.01
H a4 Nor
Flow 1600
75.56
41.44
M1
1:25
$3.19 0.39
0154
55.64
0.5%
59.92
0.07
TH0UI11 (,80
H 44, Nar I1.. Flow 004
76.06 + 7.67
58.55 11.50
03.72
3.51
94.58 +: 2.56
94.34
2
95.50 :: 1.36
99.56 0.05
1100.00 0,80
1 44 Nor I11, Flow 108
19.01 8: 10
50.74 10:90
94.91 2.7.3
96 64 3: 1.49
95.77 2:1 1:47
48.15 1: 0.54
49.78 5: 0.0
1100111 0.00
Hf 14,4, Nar IV. Flow - 500
85.31 +: 3.89
34.01 +: 5.20
89,55 1.24
36.51 +: 1.36
92.75 0.40
35335 :: 2.47
$100.00 : 0.00
99:59
0,00
M =. Nir IV, Flau = 160
88 16, 0.0.
78.68 2: 7.58
91.56
63.48
2: 0.6%
2: 1.01
94.74 6: 1.76
99.96 5: 0:0
*100LI1 1.00
Hf 1444.31 4,3 -V, Flow 004
91.64 2: 6.70
22.61 +: 13.67
96.63
1.02
93.07
+ 1.49
98.43
0.0.5
94.55 2: 2.01
99.81 0.0
1100000
0,00
1f 44, IV. Flow - 088
98.14 +: 0.46
12:73 = 5.11
08.65 0.25
97.61
+ 0.64
00.93 4
97.73 6: 0.63
99.76 = 0.05
$100.00 0,00
#1 244,Ni 41. V. F.oow 200
03.59 +: 2.32
20.40 = 3.68
92.06 1.39
87.15 + 2:25
90.55
1.00
87.11 +: 2.97
$100.00 = 0.00
99,59 +: 0,00
1 Nur .Flo 1660
99:09 1: 1.99
14.50 1.
1.45
92.83
1:
91.00
09.37
1: 0.70
*100M (UM)
*100L11 1.00
Hf Nu. Fllw 04
95.60 1.56
18 6:11
97.7%
1.0m1
94.58
3: 1.06
9605
1.76
92:04
1.34
99.00
0.88
*100.01 B.00
May Flw
08.34
1.54
08.85
98.10
0.49
0.00
98.41 0.34
90.00 0004
H #16, NNw Flow 89
45.45
1,49 U.11
47.75
0.33
89 16 0.14
M8.13 1.27
89.16 0.11
110UM (UB)
H 116 Ny. *1. Flow 1s
99.45 4: 0.01
L41 2: 0.07
+99.32 0.11
99.34
2: 0.06
+99.2 = 0.07
999.59
:: 0,65
$100,0 : 0.00
$100.00
0,80
H 1==16 Nw =1. Vlow 14
vs.18 2: 0.05
1.83 2 008
98.76
0.11
S7.90 2: 0.15
91 2.
98.12 1 0.11
99991 2: 000
*100L11 0J00
N #116. Nw =1. Vlow wes
08.1% 0.00
1.97 0.009
98.91 1.044
5K.55 2: 0.07
+W053 :: 1.64
99.78 1: 0.09
99.88 2: 0000
*100I11 (J80
N W=16. Nw HI, Plow Nm
MB-47 d: 0.07
1 1:10
W663 0:37
$2.95 2: 1.46
v7 34 0.
$7.65 1: 1.37
"1I00 :: IUR
*100L11
H1 11-16. Nir II. Flaw 1600
99.39 2: 0.02
1.31 0:03
98.19 0.35
95.92
= 1.30
98.45 0.3.3
95.20
:: 1.30
110000 0000
1100000
0,80
H1 11.16 Nir II. Flow- tia
99.24 +: 0.04
1.74 0.05
08.63 2.12
97.40
+ 0.20
99 0.009
93.62
+ 0.20
$9.94
0.02
1100.00
0J00
31 10=1. 16, No. II, Flow
99:36 3: 0.044
1.73 0.00
0.05 58.80
96.09
1: 0.12
1.011
98:13
1: 0.21
99.95
6.01
*100.00 (.00
16, Ny. III, Flow See
9944 1: 0.07
1:90 0.43
99.08
33.20
1: 0.4%
9601 1:33
33:95
6:10
"1000 (UM)
*100I11
(J00
16, Ni. III. Flow 1660
99.4 1: 0.00
11.09
0.20
999.99 0.08
95.16 0.25
09.54
0.09
99.99
0.01
*100.01
0J00
Nyr
99.25
0.07
0.07
97.05
99 0:04
93.56
0.17
99.95
1100.00
Nyr
99.33 0.04
07.05 0.35
77.40 =:
0.00
77.46 1.82
"10000
Flow
05.23
1.74 0.10
98
0105
S7.00
98.24
0.35 98.28 0.34
99/93 900
7100L01
0,80
H Nw
Flow
M5.17
1 0.94
08.54:1
0.10
S7.63
0.15
9.1
1100011 0.80
H #6, NN
95.36 d:
1 0112
57.00 0.63
72.45
033
72.50
3.58
1100M
0,80
H 16, NW Flow
95.35 3.
1.54 0.1
16.17
0.1
50.44 1.50
S1.12
1.57
1100 (U))
H w 16, Ny, V. 64
99.15 2 0.04
189 0.05
98.57
0.13
96.06 2: 0.34
99.05 0.1.L
97.79
2 042
98.1 2 0.1
$100.00
0,80
M = 16, Nw V. Flaw oas
06.31 1: 0.04
1.04 2 0.0
08.53
0.1
47.00 2: 0.19
98
21011
47.63
1: 0.70
99 2: 0.01
1100I11 (J00
Ny. =1, Flow 20
190.49 0003
0.80
1:25
26. 3.10
91:0 1:90
18.38 2.11
"1000 ::
*100.01
0J00
NNr -1,
99.47
0.25
0.25
95.55
0.77
95,97
1,34
110000
7100.01 0.00
0.60
32,
Bloe
73.60
33.96
*10MM
N 233, 11, How 1601
99.46
1994 0.79
16.57
51.13 1.47
51,0 1.39
1100M
1100L11 0,80
N 233 NW. 11, Hlw
95.26 11 0.01
1 0/0
96.90
S7.96 d:
99.39 0.0
98.44 0.99
001
17 33, Ny, II, Blow 08
99.30 4: 0.06
1.39 0.09
08.89
0.07
98.14 6: 0.14
99.45 0.006
93.54 5: 0.13
99.98
0.01
$100.00 0.80
H -3, Nv. HHI Flow 89
999.49 0.01
1.20 0.10
97.91 2: 0.70
13.77 +: 3.06
98.73 1
73.79 4: 2.97
*10000 :: 0.00
*180.00 0
H7 +3,, Ny. IHI. Flow 1670
99.44 2: 0.02
L 2: 0.69
96.15 1.53
90.95 2: 1.20
97 1.12
91,46 2: 1.24
$100,0 = 0.00
1100.00
0,80
N 23, Nw. III. Plow 444
16.11 di 0.00
1:34 0.0
08.80
0.10
47.7
2: 0.06
09.70-1
98.41
1: 0.06
99.9
2L 001
*100HI1
1f -2, Nir III. Flow - as
99.25 2: 0.04
1.33 2 0.05
08.30 2: 0.09
98.16 2 0.05
99
2 0.08
95.52 4: 0.13
$9.96 5 0.01
1000000 0,00
N 33, Nw. LV, Vlow = 200
W6 0.07
1.47 2 0.05
97.07 0.930
31.87 3: 1.31
9000 1.
13.38 1: 1.30
"IRUM 22 FUR
*100LI1
RJHH
11 -3., Nir IV, Flow - 1s6
99.35 1: 0.00
2 0.66
05.45 1.73
83.06 = 1.62
96.93 1.24
83.48 4: 1.55
"10000
0U0
1100000
0J50
8 33, Ny. IN, Flom 204
98:33 3:
1.50 0.0
98.71
0.08
47:49 1: 0.11
0.005
98.15
0.09
09.02
5.01
*106.01
(U00
8 33, .N. IN, Flow 0n
99.25 1: 0.04
1.34 0.07
0.00 95
97.20 2:
99.24 0.06
98.09
3: 0.34
99.09 0.00
*100.00 (U00
33, Ny.
99.15 0.07
1.43 0.04
03.41 0.20
65.15 2:
99.21
64.37 1: 1.84
*10000 (UN)
*100L01
32, Ny. V.Flau
09.33 1:
1.06 0.31
97.41 0.91
14.15 2:
0.60
$5.01 3: 3,01
*10000 0.00
*100.00
0
NNr V.
09.15
0.09
96.71
97.34 0.17
99.19-3:0.03
27

---

Published as a conference paper at ICLR 2022
Table 13: CIFARIC 10 comparison (<latent dim> - <certainty budget> - <radial layers>/<MAF layyers)) layers>). Bold and starred number indicate best score among all models.
Acroracy
Hitker
SVHN Alea.
SVHN Fiilt.
ColchA Alra
CatebA Rgist.
O0lban Alea
OODloo ligitt
H7 44, Nar 11, Flow 80
42:73
5:40
$1.56 d: 2.72
67.03 3:155
49.84 6.00
56.96 2: 2.13
30 7.65
95.33 2: 4.45
97.68 :: 2.32
M u-d Nar Flom 1660
66:30
6.
77:35 2: 3.57
66-13 2: 7.39
16663 2 5.92
06:33 2: 3.62
41 0.53
98.05 1: 0.55
99.1 2: 0:3:9
1 =44 Nar =1, Flom 144
55.70 2:41
80.55 2: 4.54
50.71 1 1.83
15.96 2.49
64.95 2: 1.00
08:21 1 4.85
07:66 2: 1:50
95 0.07
11 1--4. (11. 1. Flow 07
51.11 = 3.67
79:13 + 2:10
58.22 4
13.22 2 8.00
63.97 2 3.12
40.75 2: 9.47
$4.02 = 10.85
$5.24 = 11.76
N 44 Nir II, Flow 80
68:07 1:
22.11 2: 0.18
75,00 2:83
40 1.95
62:15 2: 6:12
15 1.66
82:17 1.49
09-66 7.37
11 11,,. II. Flo 169
$7.80
0.00
22:43
+: 0.35
78,00 1.65
60.02 2.96
63.55 = 2.08
46.15
4:4.0
88.55
3.72
89.82 2.85
11 11.4. 4. IL Flou -8
57.00 = 10:0
22.42 +: 0.16
x3.35 0.0.2
67.89
4.11
75.76 +: 1.35
63.04 4-4.44
89:20 3.60
91.43 3.41
31 1/.4. No.
Flov 68
16 0.74
24.15 2:
0.48
2: 5:22
75:30 2: 1.66
5:255
8:90
83.04
7.45
# 2244, 4, Hl, Flow
58.54
4.00
64.35 +: 3,30
7161 + 2.05
$7.05 5.00
20.14 = 2.66
51.87 4:3.5
93.39 6.00
U5.61 2: 3:322
Ni. Flow 1660
49.1
4.00
5996 2:
74.52 1.14
6.96
73.48
$1.30 7.00
3.42
Nay Flow
4.50
63.53
06.83
10.01
0:08
0.03
Now
3:35
Nay
1.71 17.43
71.08
52
5:61
7.76
Nu
2.90
45.57
71.77
0.53
0.09
H Nw Flow us
1.64
68:02
18:48
69:00
51.73 4.489
90.15 0.44
05.11
0.068
44, Flow H. Nor V. How 1601 H 4.4 Nw. V. Flow
56.32 0.64
32.24
4.29
50.79 3.67
66.37
7:335
7.30
3.99
1618
11
27:92
0.16
7467 7.46
16.96
400
74,68
7.69
05.79
77.59
10:21
19.75
8.14
00.00
0.94
17.15 d 3.58
77.11 2.252
11.99
74:14 1.57
$1.91 1:155
93.63 5.16
91.35 1.61
Hf Nar
Flow 88
17.84 0.71
42.01 + 1.08
75.27 8 2:23
6.42 40.57
73.46 2 1.51
33.15 5:009
95.59 2: 0.72
99/08
0.61
H W--1h 16, Ny =1, Flav s0 16, Ni. =I.
$2.93 1154
30,59 4: 1:71
81.35 0.000
05.51 2.05
75.48 + 1.06
61,89 2:38
99:73 0.24
99322 2: 0.18
0.60
25:34
82.39 1.11
2.52
75:30
m9.12
0.06
0.04
16, Ni. =I. Ne
83.71 1.37
27:35
8L36 0.94
56.29
75.88
0.65
99.92
16, N2. 1I, Flow
0.46
22.45 +
1.25
56,00 3.15
76:96 +
B5:41 = 11.05
87.50
16, Flow
19.53
12.46 1888
74.19
0.59
0,07
Flow 00
19:97
0.70
1.90
74.14
0.01
0.00
Ny:
$6.54 0.55
23.44
1.33
73.31
3.35
2.43
16,
0:20
22.07
1.29
75.52
2.51
H =16 Nw= III. Flow 844
18.16 1n
19:54 2: 0:12
82.50 1.29
62.54
74.99 3: 0.96
$7.06 1:
90:59
0.01
1100.00 2: 0.80
H 11-16, Nip III, Flow 68
$7.07 0.17
20.33 :: 0.38
9403 2 0.93
70.91
73.43
61.20 1.55
99:50 0:00::0:00 0:08
99.99 0.00
# W=16 Nw. =IN Flow 3/0
$7.92
0.07
19:88 a: 0.17
*$442 0.8
62
0.56
71.35 1.55
65.11 4.
92:66 4.50
81 9.18
Hf 116, Nyp IV, Flo 1600
$7.90
0.16
19:99 2: 0.46
92.29
1.11
77.83 1.32
76.01
= 1.18
176077 = 3.34
93.67 #: 3.08
14.50 2 3109
H 16, Nir IIV, Flo ars
57.92
11:22
19:53 2: 0.1%
92.25
1.23
67.00 2.75
73.28
=: 1.25
63 +
99:91 d: 9:08
T100.I 2: 0.00
1f 16, Ni. TV. The 65
$7.00
= 0.10
19:50 2: 0.28
81.2 1 0.44
64.76 4.18
74.50 5: 0.94
06.46 1: 3.04
02.52 2: 0.12
99.99 0.01
If 16, Ner V, Flow K0
17 1.17
19:78 e: 0.29
X367 2 0.65
56.34 :: 035
74.32 # 2.32
62.47 :: 1.08
U5.06 1.51
52.09 0.85
8 m 16, Ni V. Flow 169
18.17
0.16
19:51 5: 0.34
81.7 1 1.19
68.55
1.00
72:66
2: 199
71.06
1:1.1
83.74 2:2:5-5555 6.25
86.55 2: 3.37
N 16, Ni V. Plow
M8
12
19:30 2: 0.25
835T 10077
16 3212
76.16
2: 1.94
630 7.70
99:10 0.54
MEST &E 0013
N 16, Nw Plolo
10 0:20
19:33 3: 0.40
81.23
1
18 2.05
75.42 3: 0.96
6606 1.00
90.74 1: 0.18
9189 0.00
=33 Ny H 333 Nu =1. Flaw Bie H7 333, Nu =I. Flaw
95.00
20.35
13.46 1.45
35.00
0.64
8,80
$3.94
0.02
14.55
75.62
4:15
0.02
55.30=0
0.88
34.77 2: 1.56
82.00 1.54
256 66.88
1615 138
62n1 1.096
9263 2: 0.02
1100.00 0.80
55 0.32
24.04 à: 0.73
82.77 1 1.10
57.03
75.48 1.5
$6.44 1.77
99.54 2: 0.11
1100.00 0.60
H =3, Nw. 11, Flow 800
16.1
0.41
23.51 1: 0.52
83.36 0.71
52.99 2 2.55
16.14 1.43
60.20 1.20
95.50 2: 2.55
95.44 2: 2.70
H 333, Ni. 1B, Flow 1670
66.28
0.31
23.11 4: 0.54
83.00
1.533
63.22 2 3.35
73.67
1.19
72.00
0:90-::3.10 3.19
91.57 2.04
99.92 2: 0.65
M 33, Nir 11, klow 111
$7.9
12
1963 d: 0.13
32.72 0.77
54.33 2 4.37
74.45:
2: 1.6
77.66.2-.33
99:53 0.04
1190.0 2: 0.00
17 =3, Ni. 11, Flom 08
$7.41
= 0.45
20.73 + 0.76
81.11
4 1:58
63.56 : 2.1
75.30
5: 0.55
3.26 $9.86
0.01 99:97
1000000 0.00
N 33, Ner 11I. Flow am
M0.05 110
23.55 a: 0.18
30.35 0.45
11004 2 1:13
74.93 2: 0:97
62.13 4: 1.93
V5.30 20:30-42 2.24
M3.17 5:43
87 33, Ni. III. Flow 1600
55.71
0.47
24.29
+: 0.56
82 1.0.9
60.3
3.54
76.55
2: 1:22
73.25
6: 1.09
95.27
1.24
95.75
1.14
11 33, Ni. IBI. Flaw
58.05
0.12
19:54
+: 0.34
81.55 1.034
63.19 2.75
7621
+ 0.64
61.44 4.430
99:75
0.21
1100000 0.00
11 33, No
Plaw
10:46 3:
8400
1: 1:42
18 2 5.11
#78.26
1: 1.21
6408 1: 1:55
1: 0J03
1100.00 0.00
81 33, Ny: Flaw -
58.02
0.10
19:78 +: 0.32
83.24 0.655
50.51 4.65
77.52 4: 1,20
60.05 2.16
05.24 2.30
9LA1 3.61
N 31. No =IN,
1600
M8.70
14
19:16 2:
g765 1
5.41
73.49 3:
5.33
1.00
99.17 0.39
=31 Nw. Flow 884
14
1962
1: 0.34
8121 1.7.1
57.29 5.
36.99 3:
65 3.16
90/05-1: 0.04
*100.0 0.00
N/ 33, Ni: IN,
0123
119.25 0.44
2.96
73.58 =: 2.12
0.43
Nw F.olo
0:11
10:43 ::
0.60
48.8 3.13
73.55 à:
$4.44
2.56
8184
Nir Y.
0117
19:41
64.63
75,59 =
0.75
33.
3.95
0.00
19:28
0.669
73.06
0.26
0.60
28

---

Published as a conference paper at ICLR 2022
Table 14: Bike Sharing (Normal M) comparison (<latent dim> <certainty budget> <radial layers>/<MAF layers>). Bold and starred number indicate best score among all models.
RMSK
Calibration
Winter Epist.
Spring Epist.
Autumen Epist.
00Dom Epist.
H[ 44 Nn =I, Flow -
00.12::41
21.15 + 2.06
21.11 + 2.59
14.97
+: 0:48
17.61 + 1.02
*100.00 :: 0.00
Nu Flow 160
21.92
1665
17.54
+ 0.87
*100.00 + 0.00
H m4, Nu =1. Flow 0/4
83.51 2 8.56
23.94
1.94
90.25
+ 6.17
17.49
+ 1.50
18.79 + 1.30
1100.00 :: 0.00
H #4, Nu Flow 088
75.16 8.9777777 8.97
25.93
LO1
29.86
+ 4.07
17.71 + 0.78
17.98 + 0.43
"100.00 + 0.00
H 4,, Nir 11, Flo 800
53.90 + 3.72
22.76 0.95
24.59
2.43
16.69 1: 0.76
18,30
1.10
*100.00 1: 0.00
Flow
1.32
17.11
1100.00
H =4, Nir = 1I, How 0/4
65.03 8.07
23.10
1.68
27.99
3.46 31.66
17.04 + 1.06
22.55
+ 2,60
"100.00 : 0.00
H N, N1=== I1, Flow 0/8
74,25 + 7.89
26.51
1.09
+: 3.21
18.31 +: 0.90
20.35
+: 2.26
*100.00 1: 0.00
H 44, Nи
Flow 8/0
51.68 +:
19.04 1.75
25.06 2.21
17.05 +: 0.47
17.44 + 1,50
"100.00 + 0.00
Nu
1.13
H 44, NH III, Flow 004
79.25 +: 11.47
25.54
0.94
26.66
2.88
16.73 +: 0.92
19.28 + 0.68
"100.00 : 0.00
H1 Nur
III, Flow 0/8
92.99 3.68
0.41 26.48
28.84 1: 1.87
17,26 4: 0.43
22.35
2.26
1100.00 0.00
H m 4, NH
1V, Flow 800
51.71 1,50
18.36
23.72
2.95
16.19 +: 0.75
16.77 + 0.38
"100.00 :: 0.00
NH
1.35
H =4, N3 1V, Flow 044
57.74 4: 4.29
22.78
1.022
26.75
1.99 34.19
17.57 + 0.36
18.85 + 1,22
"100.00 : 0.00
I1 NHr
Flow 608
78.64 8.29
23.29
1.91
2.72
19.14 0.86
20.46
*100.00 0.00
H =4, N1
V, Flow 800
53.74 = 2,95
23.27 1.51
30.83 + 4.89
17.34 + 0.25
20,26 + 1.24
"100.00 + 0.00
N
1680
2.32
23.35
0.94
27.03
"100.00
H 24, N1
N, Flow = 044
83.87 = 8.67
24.20
2.03
27.64
3.59
16.48 + 0.78
21.24 + 2.11
"100.00 + 0.00
NH
Flow
17,46
"100.00 0.00
HI = 16. Ni
Flow 8/0
58.52
5,08
9,86 d: 3.10
22.85 4: 3.51
15:57 d: 1.36
19,51 :: 3.12
1100.00 :: 0.00
H NH
Flow 16/0
58.33
10.29
4.02
38.20
3.14
19.10 +: 0.91
"100.00 : 0.00
II 16, Nur
Flow 0/4
52.02 1.93
7.98
2.43
48.51
d: 5.78
22.11
1: 1.76
31,46 d: 3.21
1100.00 :: 0.00
H NH
Flow 0/8
59,43 5.49
9,20
1000::00 3.32
55.93
+ 8.21
25.67 +: 2.12
32,76
+ 5.61
"100.00 1 0.00
II 16, Nur
11. Flow 8/0
58.02 =:: 2,35
4.25 -: 1.32
23.71
+ 1,27
16.69 +: 0.69
17.95 + 0.53
1100.00 :: 0.00
4.69
"100.00 +
If 16, Nir
HI, Flow 0/4
$1.87 2:45
8.01
+ 1.89
48.39
10.55
23.12 1: 3.23
25.23
2.61
"100.00 1: 0.00
H N1===
Flow 8/8
58.00 5.68
4.57
1.15
$1.38
8.78
24.60
+ 2.22
26.31
+ 3.59
"100.00 + 0,00
H = 16, Nir
I1I, Flow M/
60.73 =:04 4.01
4,45 + 0.89
30.21 2.21
17.74 0.74
20,59 + 2,19
"100.00 :: 0.00
H = 16, Nir
III, Flow 0/4
58,90 4,49
7:34 2.08
56.72
3.51
25.77 +: 1.72
27.21
+: 3.06
"100.00 :: 0.00
H N
III, Flow 0/8
54,08 2::0:::2330 2.95
8.56 +: 2.70
55.37 + 8.22
25.82
2.32
31,20
+ 4.73
*100.00 1: 0,00
H = 16, N
Flow 8/0
$2.49 =
5,54
33.26 + 4.84
16.93
1,23
22.32 + 1.87
"100.00 + 0.00
16/0
H = 16, NH IV, Flow = 0/4
56.11 3:47
5.48 1: 1.63
$4.10
10.31
23.80
3.44
25.87
+ 2.60
"100.00 + 0.00
H Nu
Flow 688
56.03 2.51
3.74 1.59
55.41 1: 7.83
23.78
2.41
30.82
2.93
*100.00 1: 0.00
H m 16, Ner
V. Flow 800
55.11 3,53
6,72 2: 1.17
39.26 7.71
19.03 + 1.41
23.03
+: 3.74
"100.00 :: 0.00
H = 16, NH
V, Flow
14.33-22.35
4:31 + 1.05
40.28
7:97 57.11 9.02
1.47 26.22
18.99
24.85
+: 3.95
"100.00 : 0.00
11 NH
Flow
55.97 4.00
2.09
0.55
3.44
25.91
4.23
1100.00 1: 0.00
H = 32, NH
11, Flow 800
55.82 -: 1.19
3.19 + 0.78
33.14 1: 4.82
18.14 L.42
19.36 + 0.80
*100.00 :: 0.00
I1 .N
Flow- 1600
60.20 4.97
3.11
d: 0.78
40.89
d: 5.92
19.51
4: 1,50
19.08
1.31
*100.00 1: 0.00
H 10=33 32.2. NH =1, Flow
59.49 2.94
2,90
0.54
40.57
:: 3.66
19.14
4: 2.25
26.81
+: 2.90
*100.00 :: 0.00
11 32, Nt
Flow 0/8
61.86 d: 4.98
1.94
3: 0.33
*72.0
8.34
228.46
1.68
32,84
3.41
*100.00 1: 0.00
H1 33. Ny
HI. Flow 8/0
57.46 a2:46.t:333 3.24
5.08
d: 1.96
30.09
d: 2.78
17.55 4: 0.67
18.25
4: 0.78
*100.00 d: 0.00
H1 32, NH
Flow 16/0
55.34
2,15 0.44
38.14
19.24 +: 1.00
20.53
"100.00 1: 0.00
II 33,, Nur
II. Flow 0/4
58.50
7.43
2.23
4: 0.29
$7.94
d: 5.29
24.43
1: 0.60
29.14
3.00
*100.00 1: 0.00
H 32, Ng
Flow 0/8
54.68 =: 4.01
2,26
+: 0.60
45.24
5.14
19,82 +: 1.48
29,23
4: 2,28
"100.00 1: 0.00
I1 32, Nur
III, Flow 8/0
$3.35 =:: 3.39
2,88 d: 0.81
32,52 d: 1.58
18.28 4: 1.29
20.08
:: 1.05
*100.00 :: 0.00
NH
Flow
2.39
0.99
"100.00 1: 0.00
11 32, Nir
III, Flow 0/4
58.84 1.61
*1.49 0.20
$0.90
11.50 55.22
21.83
2.72
33.23
5.33
1100.00 1: 0.00
H 32, Nu
III, Flow 078
*51.12 1,93
2.17 + 0.35
5.21
2.43 24.55
28.35 + 2.18
"100.00 1 0.00
H 32, Nrr
IV, Flow 8m
$2.58 0.:82:20 2.37
6.09 +: 1.30
28
5.31
16.77 + 1.24
20.28 +: 2.63
*100.00 :: 0.00
.N
16/0
"100.00
H = 32, Nir IV, Flow 0/4
$1.88 1.57
2.72 0.74
55
7.42
24.11
2.10
#35.59
1: 3.99
"100.00 :: 0.00
H = 32, Nu
Flow 0/8
53.84 2.82
1.85 4: 0.33
19.44
3.65
2.11 21.06
30.14
2.12
*100.00 1: 0.00
I1 = 32, Nr
N. Flow 8/0
53.52 1.64
8.24 0.64
30.51 2.22
18.50 4: 0.67
20.98 2.00
*100.00 1: 0.00
.N Flow
H 32, NH =N, Flow 004
56.22
4,42
1,98 0.45
40.97 5.06
20.21 +: 1.98
23.40
1.69
"100.00 : 0.00
# = Nu V, homm=
54.06 3.18
2.09 0.32
35.75 2:
18.86 2.13
22.27
1.83
*100.00 1: 0.00
29

---

Published as a conference paper at ICLR 2022
Table 15: Bike Sharing (Poisson Poi) comparison (<latent dim> <certainty budget> <radial layers>/<MAF layers>). Bold and starred number indicate best score among all models.
RMSE
Winter Epist.
Spring Epist.
Autumn Epist.
OODom Epist.
11 4,, N1 =1 Flow 8/0
937.56 238.40
33,50
1 4,83
18.04
1: 0.58
20.44 1 1.53
+100.00
0.00
H1
Flow 16/0
777.32 328.71
22.53
+ 2.16
17:00
0.74
18.30 1: 1.25
+100.00
0.00
H Ni Flow 0/4
33780.69 + 19607.73
45.22 = 7.96
21.35 = 2,09
37.89
4.13
*100.00
= 0.00
H1
Flow 0/8
22686.67 5815.18
53.51
5.83
21.80
1.14
39.25
2.59
100.00
0.00
H 4,
II, Flow 800
60383.07 21389.07
15.20
0.57
13.50 0.22
15.28
0.77
*100.00
0.00
H 44, 21
11, Flow 16/0
329W2.6, 63 + 17819.36
33.09
= 8.79
17.55 = 2.25
19.17 + 1.19
$100.00 = 0.00
11
Flow
21467.31 17019.59
44.60
0.00
H = 4, Nr
11, Flow 0/8
11231.30 = 4784.62
42.64
7,58
21,06 = 2.63
23.93 + 2.05
1100,00 = 0.00
I1 =4,4, Nr
11I, Flow 8/9
2048,78 J. 538. 12
22. 84
1. 1,90
16.59 0.73
17.21 1 0.50
100.000 1. 0.00
11
Flow
5181.92: 3581
25.4
0.53
17.00
100.00
0.00
H 4, NH
III, Flow 0/4
35092.52 52-522222 + 10813.54
47.42
5.98
22.49 1,88
28.50 + 1.52
1000000
0.00
N
Flow IV8
42792.69
2.17
33.78
0.00
H = =1 Flow
10255.94 + 6207.90 900
19.89
1.82
14.93 0.30
16.38 0.57
*100.00
0.00
H= N
Flow 16/0
6665.70 3160.53
29.95
6.24
17.66 1.06
17.33 + 0.46
$100,00
0,00
11 NW = IV, Flow 0/4
119600.14 1. 85229.53
35.15
5,20
19.58
1. 2.37
27.15 1. 2.87
100.000
0.00
H N IV, Flow 0/8
132950.39 + 98199.93
56.85
8.13
24.17 1,99
40.23 # 5.69
1000000
0,00
II
V. Flow 800
131051.70 124947.53
23.89
1. 2.55
15.68 0.71
17.94 1. 1.34
1000.00
0.00
H 4, N2 V. Flow 16/0
16481 96 + 7339.53
26.55
+ 1.59
16.40
0.25
26.36
: 5.15
100.00
: 0.00
H
V. Flow= 0/4
28238.41 10202.38
39.68
19.04 2.13
29.09
3:57
$100.00
0.00
H 4, Næ
V. Flow 0/8
27167.10 10 + 9698. 30
46.59
:: 6.57
23.03 = 1,22
27.77 + 5.03
100.00
0.00
H = 16, 1 Flow
633.14 237.64
35,20 + 5.92
18.16 0.91
22.16
2.43
1000000
0.00
11 Nи
Flow 16/0
408.28 246.12
45.32
4.53
19,96 1,66
34,91
6.35
100.000
0.00
H 16, Nu
Flow
276.23 + 151.72
64.86 + 8.92
3,85 26.40
37,05
5.43
100.00
0.00
11
Flow 078
262.91 126.12
80.30
1. 4.85
32.12 2,86
38.83
3.14
100,00
0.00
H 16,
Flow 8/0
1325.94 79.27
30.06 = 3.95
18.63 1,33
21
1.82
100.00
0.00
H #1, N
I1, Flow 16/0
1042.48 + 413.69
45.96 = 5.13
20.63 = 1.63
24.25
+ 1.57
$100.00 = 0.00
H NH
Flow 0/4
129,87 + 79.26
71.03
5.39
34.02 4.65
36.16
5.78
100.00
0.00
H 16, N
11, Flow 0/8
182.97 + 129.05
81.19 6.60
36.17 5.02
*43.97 + 7.12
1000000 = 0.00
H 16, 1 11
111, Flow 8/0
1233.19 19 + 76.60
34,22 + 4.42
19.09 :: 1.47
25.34 + 1.63
100.000 ::: 0.00
Flow
1.47
22.85
0.00
H 16, 11 111, Flow 0/4
89.12 + 37.11
84.06 = 3.99
37.00 1 3.23
36.77
1: 5.20
1000,00 ::: 0.00
16, R
III, Flow
76.50
37,96
H 16, N
Flow
1893.36 + 970.62
40.06 -
20.01 = 1.96
25.71 + 4.91
1000000 0.00
H N IV, Flow 16/0
2212.13 + 13
41.60 = 4.41
17.93 0.92
29.77 = 3.60
1100.00
0.00
H 16, N Flow
56.60 + 2.25
72.66 =
32.09
4,74
34.56
5.52
1000000
0.00
H N IV, Flow
52.01 + 2.10
79.58 = 5.81
2.84 33,47
36.19 4.91
1100,00
0,00
H N Flow
4434.32 1: 3059.38
31.63 1: 4.31
17.82
1.02
20.77 1: 2.04
$000.00
0.00
11 16, Nu V, Flow 16/0
4115.66 1: 1891.56
47.58
1: 4.69
22.82
2.46
26.51
1: 5.27
100.00
1: 0.00
11 N V. Flow 0/4
50.47 +: 1.54
83.71
:: 5.23
*37.46
5.13
42.63
1: 4.37
100.000
0.00
11 =/=10 16, NH V, Flow 0/8
$1.79 1: 0.78
*85.15 3.61
37.03
1: 2.35
42,73 1. 4.38
100.00
0.00
H 32, n Flow
351.49 + 157.14
38.59 = 5.39
21.90 = 2.62
25,23 2.66
$100.00
0.00
H 32, Nu
Flow 16/0
167.67 = 116.18
40 = 6.51
21.90 2,76
24.84 + 3.40
1100.00
0.00
= n
Flow
50.10 1.55
73.09 = 9.70
27.
2.42
40.78 + 8.19
$100.00
0.00
11 32,
Flow 078
51.97 1: 2.57
58.80 10.68
2.46 23.64
5.51 30.77
100.000
0.00
H1 32, Ny
II, Flow 8/0
580.40 =:: 250.82
38.80 1: 5.56
20.62 1.89
25,80 1.92
100.00
0.00
11 32, N
11, Flow 16/0
49.96 1: 1.60
46.52 1: 6.52
22.78
1.69
27.16
5.09
100.000
0.00
H1 32,
Flow 064
+48.85 0.92
60.12 10.37
22.57
2.44
37.51 6.70
100.00
0.00
H = 32, Nu
II, Flow 0/8
50.12 + 2.29
69.33 = 4.57
30.96 = 2.55
35.11 + 6.33
$100.00 = 0.00
H 32, 12,.0 1
111, Flow = 8/0
462.85 + 169.59
43.86 = 7.00
20.62 = 2.41
30.58 + 4.59
100.000 :: 0.00
H N
Flow 16/0
569.28 219.42
$4.12
22.49 1.98
31,49 4.90
$100.00 0.00
H = 32, 1
111, Flow 0/4
49.83 + 1.25
67.93 = 9.50
27.61 = 3.98
31.87 1: 6.17
1000000 :: 0.00
NH
Flow
$1.26
H1 N
50.79
18.84 1: 2.13
26.45
0.00
11 N
IV, Flow 16/0
49.91 1. 1.54
45.15 1. 7.53
23.00 1. 2.26
27.90 1. 3.51
100.000
0.00
49.82 +
4.29
34.17 +:
100.00
0.00
H N IV, Flow 0/8
51.66 + 1.79
65.31
9.06
28.34 3,83
38.84 + 6.46
100.000
0.00
H Nu
Flow 8/0
$2.95 + 1.36
39.37 = 7.16
17.74 1.36
28.73 5.78
$100.00
0.00
H N Vlolow
146.99 + 94,70
56.50
3.61
24.44
2,00
34,28
4.16
*100.00
0.00
H Nu Flow 0/4
51.42 + 1.68
74.23
7.35
27.64 1.85
35.90
6.06
$100.00
0.00
11 32, N V. Flow 0/8
49.31 1 1.81
10.30 76.36
31.86 4.07
41.32 1 4.75
1100.00
0.00
30

---

Published as a conference paper at ICLR 2022
Table 16: MNIST - OOD detection with AUC-ROC scores. Bold numbers indicate best score among single-pass models. Starred numbers indicate best scores among all models. Gray numbers indicate that R-PriorNet has seen samples from the Bashion-MNNST dataset during training.
Epist.
Aleaa.
Epist.
OODom
OODom Epist.
Dropout
98.12 0.05
97.16 0.11
*99.26
96.77 + 0.25
15.19 1.70
88.16 0.59
Ensemble
98.17 + 0.07
98.03 0.05
99.15 + 0.07
98.04 + 0.11
11.83 181
81.53 0.38
NatPE:
98.25 1. 0.27
0.03 99.48
98.79 1 0.33
*99.61 0.07
1100.00
0.00
*100.00 1: 0.00
R-PriorNet
*99.44 +: 0.09
*99.59 + 0.08
100.00 +: 0.00
100.00 +: 0.00
99.44 0.16
1.82 + 0.67
Eni>"
98.21 0.16
98.65
0.15
99.06 0.20
99.21 1: 0.17
56.31
2.78
4.60 1.89
PostNet
0.05
0.06
0.35
98.57 0.34
*100.00 0.00
*100.00 :
NatPN
99.11 + 0.17
99.25
0.09
99.13 1: 0.24
99.45 1 0.11
99.98 0.01
*100.00 1: 0.00
Table 17: CIFAR-10 OOD detection with AUC-ROC scores. Bold numbers indicate best score among single-pass models. Starred numbers indicate best scores among all models. Gray numbers indicate that R-PriorNet has seen samples from the SVHN dataset during training.
SVHN Alea.
SVHN Epist.
CelebA Alea.
CelebA Epist.
OODom Alca.
OODom Epist.
Dropout
84.67 + 142
75.79 + 0.86
75.95 + 3.61
75.00 + 3.21
21.75 + 6.36
78.81 + 6,91
Eusemble
88.08
85.70 +
78.80 + 0.82
77.63 0.61
40.53 9.95
96.71 2.31
NatPE
88.73 + 0.26
0.82 *86.73
*80.46 0.22
*85.75 + L.0
1.37 92.45
*99.56 + 0.11
R-PriorNet
99.94 +: 0.01
99.98 0.00
74.69 + 2.39
70.63 6.14
64.45 = 10.72
59.61 13.23
EnD
*89.56 + 0.67
84.36 + 0.84
77.94
+ 1.62
78.14 + 1.66
53.05 6.07
4.42 + 2.57
PostNet
85.52 1: 0.58
84,25 1: 0.90
75.68
1: 2.05
77.96 2.05
93.00 2.46
97.22 0.86
NatPN
85.24 1: 0.98
81.74 1: 1.05
77.98 1: 1,22
81.62 3.15
*96.94 1: 1.53
97.41 1.63
Table 18: Bike Sharing OOD detection with AUC-ROC scores. Bold numbers indicate best score among single-pass models. Starred numbers indicate best scores among all models. Normal and Poisson Regression are treated separately.
Winter Epist.
Spring Epist.
Autuma Epist.
00Dem Epist.
Dropout-A
$3.98 + 1.60
51.24 0.96
$3.89 1.16
*100.00 0.00
Ensemble-N
81.53 + 1.11
67.07 = 0.55
67.78 + 1,31
*100.00 = 0.00
EvReg-V
55.26 + 2.14
53.76 + 1.35
$2.39 + 1.31
47.68 + 17.67
NatPN-A
*87.67 = 3.13
*68.68 + 2.58
#71.70
3,23
*100.00
0.00
Dropout-Pot
55,30 +: 0.58
50.75 U.56
59.05 + 1.15
$100.00
Ensemble-Poi
95.31 1. 0.41
75.62 0.85
78.93 1. 1.35
$100.00
0.00
NatPN-Poi
*96.67
*78.45
*82.42 1.73
$100.00
31


Reviews:
[{"review_id": "sFMXznifODv", "reviewer": "Reviewer_QsKU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Pros \n\n- Utilizing normalizing flows for density estimation for calibration and OOD detection is intuitive \n- NatPN can be used for both regression and classification tasks. The majority of previous works neglect regression tasks and focus solely on classification. \n- NatPN maintains a fast inference time as compared to other sampling based BNN’s\n- NatPN achieves high aleatoric uncertainty on OOD datasets, which performs well against baselines compared against. \n\n# Cons\n\n- Section 2: The authors state that modelling distributions over the weights results in ‘pathological’ behavior. This is vague and hard to understand, what is the pathology?\n- Section 2 (sampling free methods): parameterizing conjugate prior distributions which $\\rightarrow$ parameterizing conjugate prior distributions, which (even though I believe this fixes the problem with the sentence, it likely needs to be rewritten because the long list of references breaks the flow over three lines…)\n- It is not immediately clear to me why the 2nd term of the loss in equation 5 is necessary. If each component of the model behaves as expected, wont $p( x | \\omega )$ remove the need for entropy regularization?\n- There are recent advances in single pass uncertainty such as SNGP [1] which are nor compared to. At least for classification tasks, I would be interested to see the performance difference between the proposed method and SNGP. \n- Table 5 shows results for models which are trained on the Kin8nm and Concrete UCI datasets, and evaluated in terms of epistemic uncertainty on other UCI datasets. These datasets all have different input dimensions, so how does a model trained with a specific input dimension evaluate a dataset with another dimension? \n\n# Minor\n\n- section 4.2 or training OOD $\\rightarrow$ or training on OOD\n\n\n# References\n\n[1] Liu, J. Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., & Lakshminarayanan, B. (2020). Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006.10108.", "summary_of_the_paper": "The authors propose NatPN, a model which can estimate predictive uncertainty for classification and regression tasks. NatPN predicts the parameters of the posterior distribution which belongs to the exponential family. Contrary to other algorithms, NatPN requires no OOD data for training, and is able to evaluate the uncertainty in a single forward pass utilizing normalizing flows for density estimation.", "summary_of_the_review": "There are a few concerns which are highlighted in the sections above, but overall the paper is a strong submission which solves a relevant problem in a novel and intuitive way. My score reflects everything stated in my review and is likely to be updated based on the authors response to my questions above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "W6fMvBmWe-", "reviewer": "Reviewer_qjTd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Edit: Postrebuttal Update: The authors clarify most points in the rebuttal/include them in the new version. I increase my score from 5 $\\to$ 6._\n_________________\n\n## Strengths\n- While prior work in this direction of parameterizing a _'prior'_ distribution as part of the model focused mostly on the classification task offering the clear Categorical+Dirichler pair, this paper's focus on the exponential family allows for a clear generalization. \n\n## Weaknesses\n- While the paper title is phrased as targeting the predictive uncertainty, the paper's discussion and experiments solely focus on the epistemic/aleatoric decomposed setup (apart from some in-distribution calibration results). While the distinction between reducible and irreducible uncertainty is nice within a model, to better understand its performance or as a guiding signal for an active learning task, the paper explores no direction where decomposition of the predictive uncertainty actually is relevant.\n- Related to the last point, the experimental comparisons are also all against methods that allow for a clean distinction instead of focusing on a broader class of models that target a predictive uncertainty independent of whether they allow for a theoretical/practical decomposition of such a term (e.g. a simple deterministic net with temperature scaling for calibration). \n- The exponential family offers an extension away from the standard classification case. The paper has experiments with regression and count data, but as the authors acknowledge, the Poisson likelihood seems to perform poorly in the chosen data set. Having an experiment where the explicit modeling of the counts as counts was necessary would be a lot stronger.\n- The paper is closely related to prior work by Charpentier et al. (2020), down to using the same input examples in Figure 2. \n- ~~The related work discusses BNNs in the sampling-based paragraph but lacks a similar discussion of BNNs in the sampling-free methods~~ _Edit: Wrong claim from my side. I overlooked some references in the sampling-free paragraph as the keyword BNN was missing._ \n\n\n## Questions and minor comments\n- Q1: In the contribution, the authors mention (point (3)) that the density is added \"to the last predictor layer\", a term that is never mentioned again. Can the authors comment on what they mean by that?\n- Q2: As mentioned above on the epistemic/aleatoric point, a stated goal is to aim  \"at accurately modelling both aleatoric and epistemic uncertainty\". Can the authors comment on that goal and why they target it as the paper lacks discussions/explorations on the usefulness of the distinction between the two?\n- The experiments compare against Amini et al. (2020) in the regression case but lack a comparison in the classification against Sensoy et al. (2018) Amini et al.'s closest relation and predecessor. That work also has the benefit of not OOD data during training as the PriorNets do.\n- The BNN discussion comments on the huge computational budget of Izmailov et al. (2021) as a downside. This comment ignores that this cost was due to their goal of inferring the true posterior as optimally as possible via a costly HMC approach. However, that work includes many cheaper methods (be they variational inference-based or MCMC-based, such as SGLD, SGHMC) that demonstrate good performance for a much cheaper computational cost.\n- Table 1 contains **(1),(2)** whose meaning is only resolved several pages later. At the same time, the page includes three different sets of (1),(2) (twice in two separate lists, and once each as equation numbers). A restructuring might be helpful to guide the reader towards the desired meaning in table 1. \n- While the term is only taken from prior work, I would strongly encourage the authors to rename their objective to something other than \"the Bayesian loss\". It is an objective whose optimum is given by the posterior, which is nice, but apart from that, there is no such thing as _THE Bayesian loss_ or even \"the principled Bayesian loss\"..\n- NatPN Ensembles are discussed, but the NatPE abbreviation is missing from the paragraph and never formally introduced\n- In equation (6), as soon as $\\lambda \\neq 1$, L is not actually proportional to the right-hand side anymore.\n\n### Minor Appendix comments\n- App B: The ELBO loss refers again to the appendix \n- App B: There appears a discussion on a prior over $y$, while talking about distributions over $\\theta$\n- In the appendix $P(\\theta)$ and $P(y|\\theta)$ seem to be used interchangeably \n- The appendix states that the entropy of the posterior is used to estimate the predictive uncertainty, but the posterior predictive uncertainty should actually be computed after marginalization over the posterior.\n\n### Typos\n- Throughout the paper, please follow the ICLR style guide properly. E.g. captions belong above tables and below figures.\n- Table 1 refers to eq (7) which is in the appendix. It should probably mean (5), i.e. the same loss just in the main paper\n- End of page 3: Bishop (2006) -> (Bishop, 2006).\n- page 4: predictive posterior distribution -> posterior predictive distribution\n- page 5: using each NatPN member ~~all~~ separately\n- The abbreviation OOD is introduced, but ID is not and needs to be guessed from the context\n\n\n__________\nSensoy et al., Evidential Deep Learning to Quantify Classification Uncertainty, NeurIPS 2018\n", "summary_of_the_paper": "The paper targets the task of getting useful predictive uncertainty, as measured by in-distribution calibration and out-of-distribution detection capability. Towards this, the authors focus on the distributions from the exponential family, which allow for a closed-form posterior form whose parameters are predicted via a neural net, together with a flexible density provided by a normalizing-flow net for OOD detection. \n", "summary_of_the_review": "While the paper provides a useful extension and next step to the prior work by Charpentier et al. (2020), the theoretical contribution seems minor and without great empirical improvements. Explorations in areas where the proposed setup with the uncertainty decomposition becomes relevant could strengthen it a lot.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U-c3KYV6QSy", "reviewer": "Reviewer_pzkS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Update:** changed my score (see comment) from 6 to 8.\n\n**Strengths**:\n\nThe paper makes three claims. \nFirstly, NatPN can be applied to all problems with an exponential family likelihood distribution. The paper provides strong evidence that this is the case. They perform multiple experiments on classification tasks and two experiments on regression and count prediction. The experiments are thorough and tested on multiple datasets each. \nSecondly, they claim that empirically NatPNs outperform other state-of-the-art methods in different uncertainty-related metrics such as calibration and OOD detection. \nAcross all experiments, their method beats most competitors or provides competitive estimates across multiple metrics. Their experiments provide more than enough evidence for this claim. \nThirdly, they provide a theoretical statement on the OOD behaviour of NatPNs in the limit. Their theorom shows that under mild assumptions NatPNs recover the prior far away from the training data. \n\nOverall, the experiments convince me of the main goal of the paper, namely that NatPN provides well-calibrated uncertainty estimates for multiple data types. \n\nFurthermore, there are other miscellaneous strengths. The paper is well written and the figures are (mostly) of high quality. The paper doesn’t shy away from talking about its limitations. The method is fast and easy to set up which is especially beneficial for practitioners. All of these are important strengths of the paper that are often undervalued.\n\nAll in all, the paper provides a significant contribution to an important problem. \n\n**Weaknesses:**\n\nThere are two main (and easily fixable) weaknesses.\n\na) I think the role of the normalizing flow is underexplained. It is stated multiple times that the normalizing flow provides the evidence updates and its purpose is to estimate epistemic uncertainty. The remaining questions for me are 1. From which space to which does the NF map the latent variable z? 2. Why is the arrow in Figure 2 from a Gaussian space into the latent space, rather than from the latent space to n^(i)? I thought the main purpose was to influence n^(i)? 3. Which experiments show that the normalizing flow contributes meaningfully to the epistemic uncertainty (see b))?\n\nb) Figure 1 does a good job of showing the intuition behind NatPNs but it lacks some components and a discussion in the text. The authors choose to show aleatoric (un-)certainty and predictive certainty respectively but don’t show epistemic (un-)certainty. Technically, you could deduce epistemic uncertainty from aleatoric and predictive uncertainty but it would be easier to compare and follow your argument if it was made explicit. Furthermore, I would like to see an explicit discussion of the results. Why, for example, is the difference between aleatoric and predictive uncertainty so low? Is there no or little epistemic uncertainty in this setting? There are two things that would convince me more regarding this problem: a) an additional toy experiment similar to Figure 1 which includes more epistemic uncertainty, e.g. with fewer data points. This could show that the epistemic uncertainty is well-calibrated. b) An argument for why the epistemic uncertainty is (presumably) so low in your setting. a) and b) are not mutually exclusive, doing both would convince me more. \n\n**There are a couple of minor improvements:**\nFigure 1 is not referenced in the main text.\nI find it hard to spot the difference w.r.t the symbols in Figure 1. Maybe just making it less crowded would already improve visibility.\nIn the last paragraph of 3.1, you mention “warm-up” and “fine-tuning”. It would be helpful to explain these concepts briefly in one additional sentence or provide references.\n\n**What would raise my score?**\n\nI would raise my score by 1 or 2 points if my main weaknesses are well addressed or if evidence is provided that my criticism is the consequence of a misunderstanding. \n\nI would raise my score even further if I’m convinced of the high significance of this work. This will be mostly dependent on the estimate of more expert reviewers but I’m also open to arguments by the authors. \n\n", "summary_of_the_paper": "This paper proposes Natural Posterior Networks (NatPNs), a technique to provide uncertainty-estimation to tasks where the likelihood is an exponential family, e.g. classification or regression. NatPNs leverage the properties of exponential families, i.e. conjugacy and closed-form posterior predictive for fast and elegant Bayesian Inference. This is combined with a Bayesian treatment of the loss function that allows for end-to-end training. Furthermore, it uses a normalizing flow to account for the epistemic uncertainty. The normalizing flow density is supposed to increase the evidence on the training data and thereby decrease it everywhere else. The posterior distribution is on the outputs not on the weights of the NN which makes it easier to train. The paper provides one theorem for why the posterior uncertainty is meaningful and multiple experiments showing practical improvements for multiple exponential families and datasets.  \n\nNatPNs are an extension of Posterior Networks. However, the differences are significant and well explained in the paper. From my perspective, the contributions are clearly novel enough to justify this new paper. \n", "summary_of_the_review": "**Recommendation:**\n\n6: marginally above the acceptance threshold; Tendency towards 8, if concerns addressed. \n\n**Summary:**\n\nThe paper presents a novel method and provides empirical and theoretical evidence for its effectiveness. There are two parts of the paper that are currently insufficiently explained. Firstly, the technical setup of the normalizing flow and how it is used to create epistemic uncertainty is underexplained IMO. Secondly, the toy experiments in Figure 1 should be explained and discussed in more detail. I'm specifically currently uncertain why the epistemic uncertainty is not shown and presumably quite low in this setting. \n\nI think these concerns can be easily addressed by the authors and would be willing to increase my score to an acceptance (8) if they are addressed well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MJPWTRIKHk", "reviewer": "Reviewer_x9Pt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Relevance**:\nUncertainty estimation is a highly active research area. Methods that don’t require calculating an average over multiple forward passes are arguably particularly relevant for practical applications. While most machine learning problems, at least in a research setting, are typically classification or regression problems, generalizations beyond those settings are of interest for the community.\n\n**Novelty**:\nWhile the paper builds off PostNet (Charpentier et al., 2020), which is clearly referenced, the generalization to non-classification tasks is novel. The paper further proposes an alternative scheme of using a single normalizing flow for density estimation on the features, which is computationally more scalable for a higher number of classes. The overall contribution seems solid in terms of novelty, although there might be related work that I’m not aware of, as I have been following this branch of the literature only superficially.\n\n**Clarity**:\nOverall the paper is clear regarding the proposed method, although I found it helpful to also read the PostNet paper. However, the typesetting is too dense, presumably due to the space constraints. In particular in section 3 I would suggest using subsection and subsubsections to structure the text instead of paragraphs. Similarly pages 7 and 8 could benefit from more spacing, the use of bold font inside of paragraphs does not seem ideal to me (in particular when some terms also get underlined, visually I find this page distracting). I’m not sure if an additional page will be available during the discussion period, otherwise I would suggest moving most of the detailed discussion of the setup into the appendix, most of the datasets and metrics are fairly standard.\n\n**Empirical evaluation**:\nThe classification experiments and metrics are mostly standard and the regression problems appear to overlap with (Amini et al., 2020). I would have ideally liked to see the paper push a little bit further in the direction of problems that are not classification or regression, but it is good to see that there is a count prediction problem and that OOD detection is improved by a Poisson observation model. \n\nIn an orthogonal direction, I would also be interested in seeing results for a classification problem with a higher number of classes, as this is the setting where the difference to PostNet of using a single normalizing flow instead of one per class becomes more relevant. I would be interested in seeing if either approach runs into limitations in terms of fitting the data. Ideally this would be shown for ImageNet (if PostNet can be scaled to 1000 classes), but I understand that this may be too much to ask over the discussion period and would be happy with Cifar100 results -- I assume this should be runnable with negligible overhead based off the Cifar10 experiments.\n\n**Other notes/questions**:\n* How sensitive is the method to the value of $\\lambda$ (the entropy hyperparameter)? An ablation study would be useful, I would ideally like to see a plot showing how the different metrics vary across values of $\\lambda$ with some dataset for each likelihood model (normal/categorical/poisson).\n* Please place table captions above the tables as per the formatting instructions.\n* Page 7, final line: missing transition between sentences?\n* There are some minor inconsistencies in the reference formatting (full first names vs only initials, title capitalization).", "summary_of_the_paper": "This paper proposes a single-pass uncertainty estimation method for neural networks by predicting the update to the natural parameters of the conjugate prior to the likelihood of the data distribution and optimizing a corresponding ‘Bayesian loss’. In that it generalizes PostNet (Charpentier et al., 2020) beyond classification. The paper reports competitive or improved performance across a range of datasets and against both standard and recent baselines.", "summary_of_the_review": "The paper presents a solid and well-motivated generalization of prior work. There is some room for improvement in terms of the writing as well as some additional experiments, however, I would already consider this a solid submission as it is. Therefore I am **leaning towards an accept**", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1f5EhwkRK21", "reviewer": "Reviewer_cKhx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper is well written and clear.\n- The theoretical contribution, Theorem 1, is a nice addition to the empirical results.\n- NatPN performs on par or better than similar prior methods from the literature.\n- NatPN extends the Posterior Network to the exponential family distributions which encompasses important tasks such as regression with Gaussian likelihood and counting with Poisson likelihood.\n- NatPN uses a single normalizing flow for classification rather than one flow per class as per Posterior Networks, which in theory enables scaling to datasets with many classes.\n\nWeaknesses:\n\n- NatPN has the theoretical advantage of being able to scale to many classes given that it uses a single normalizing flow. This is a significant potential advantage over Posterior Networks. However the paper only evaluates on classifiation tasks with 10 classes: CIFAR-10, MNIST, FMNIST. It would be very useful to verify the scalability of the method to > 10 classes on datasets such as CIFAR-100 and Imagenet which have well defined associated OOD tasks e.g. CIFAR-100 vs. SVHN, CIFAR-100 corrupted, Imagenet-C/A/R/V2 and are standard benchmarks in the OOD literature.\n- Please also include OOD performance e.g. accuracy, RMSE, etc. for the near OOD datasets e.g. CIFAR-10 corrupted. Methods which are can provide good OOD detection should also be capable of generalizing to near OOD data points.\n- The work is nice, but somewhat incremental vs. the Posterior Network. The extension to exponential families is natural. The use of a single normalizing flow for classification seems more significant to me, but it is unclear if this causes new training difficulties (see questions below).\n-  The empirical results do not show a clear gain from NatPN vs. similar competing methods.\n\nAdditional suggested improvements:\n\n- Comparing against other deterministic/single-pass methods from the literature that do not require normalizing flows would be a significant improvement to the experiments, in particular SNGP (Liu et al., 2020) and DUQ (van Amersfoort et al., 2020).\n- When comparing NatPN to the Posterior Network it is hard to disentangle the performance changes for classification coming from the use of a single normalizing flow rather than C normalizing flows vs. the new update rules. Could you add an ablation to disentangle these contributions? For example, could you maintain the old Posterior Network update rules but use a single normalizing flow or could you use C normalizing flows but use the NatPN update rules per flow? This would help assessing the relative importance of the contributions of this paper for classification tasks.\n\nQuestions:\n\n- Could the authors comment on the disadvantages of not scaling the number of normalizing flows with the number of classes as per PostNet?\n- The second sentence quoted is not clear to me: \"On the other hand, a single normalized density is trained to output the evidence update n (i) = NH P(z (i) | ω) accounting for the epistemic uncertainty. The intuition is that increasing the evidence on training data during training forces the evidence everywhere else (incl. far from training data) to decrease thanks to the density normalization constraint.\" Could the authors clarify?\n- It is not clear to me where the Bayesian NatPN Ensemble posterior update for n^{post, (i)} comes from? How is this not effectively (m - 1) times overcounting the effective training samples given that each ensemble member is presumably trained on the same dataset from a different random initialization?\n- Please comment further or point to the point in the appendix where this is clarified: “Additionally, we observed that “warm-up” training and “fine-tuning” of the density helped to improve uncertainty estimation for more complex flows and datasets. Thus, we trained the normalizing flow density to maximize the likelihood of the latent representations before and after the joint optimization while keeping all other parameters fixed.” If a significant change in the training procedure for NatPN vs. Posterior Networks and other baselines is required then this is a significant disadvantage of the method.\n- What is the definition of the “predicted evidence” for all models? In particular for baseline models: dropout and deep ensembles?    \n\nNits:\n\n- For comparison purposes it would be useful to include results for a standard/deterministic network trained without dropout (with standard MAP parameter estimation).\n- NatPE is never defined. Assumed it is the Bayesian NatPN Ensemble.\n- NatPE should be included in the batched inference time table. Also nit: this should be a Table {number} not Figure 4.\n- I am surprised the Dropout baseline inference time is so slow in Figure 4. It seems reasonable to compute the inference time assuming that the sampling of the dropout masks can be parallelized along the batch dimension. From the numbers it looks like what is being measured is the inference time if the 5 dropout masks are sampled serially. In addition, given this efficient implementation, 5 dropout masks seems low for a fair comparison to a deep ensemble, a useful additional result would be to include a Dropout baseline with num_samples=128 or similar.", "summary_of_the_paper": "The paper introduces a new method Natural Posterior Network (NatPN), an extension to the Posterior Network (Charpentier et al., 2020). NatPN aims to tackle the problem of enabling calibrated uncertainty estimates for in and out-of-distribution inputs for exponential family likelihoods parameterized by deep neural networks. In particular NatPN maps input samples into a latent space on which a normalizing flow is defined. A mapping from this latent space and its corresponding likelihood under the normalizing flow to the parameters of exponential family update rule is defined. The exponential family posterior tends towards the prior parameters for data points far from the training data and prior has little influence for data points within the training distribution.\n\nNatPN makes two significant changes to the Posterior Network 1) it generalizes the method to the exponentially family, allowing for regression and counting tasks beyond just classification as per the Posterior Network and 2) it uses a single normalizing flow rather than class specific normalizing flows as per the Posterior Network, in theory this aids the scalability of the method to datasets with many classes. In addition a minor change is made to the update rule of the posterior parameters.\n\nThe paper presents extensive empirical results on classification, regression and counting tasks across several relatively small scale datasets. NatPN is shown to perform on par or slightly better than similar methods from the literature.", "summary_of_the_review": "The paper is well written and seems technically correct. The proposed method NatPN extends the Posterior Network to the exponential family distributions and has the advantage of using a single normalizing flow rather than C flows for classification tasks. The empirical results are backed up with an interesting theoretical guarantee under what seem to be reasonable conditions.\n\nA number of aspects of the paper are unclear to me, I have listed these questions above.\n\nGiven the slightly incremental contribution of the extension of Posterior Networks to the exponential family distributions the paper would benefit from additional empirical validation e.g. with the above suggested ablation, the addition of SNGP and DUQ baselines and the inclusion of OOD performance metrics. Additionally, given that I see the use of a single normalizing flow as a main contribution of this paper I would like to have the question regarding the modified training procedure clarified and I would like to see empirical validation of the scaling to CIFAR-100 and/or Imagenet.\n\nNonetheless, the paper is a solid contribution and I believe worthy of acceptance given the information currently available.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for producing higher quality uncertainty estimates by mapping the predictions from an arbitrary (e.g. deep learning) model to an exponential family distribution.  This is achieved by using the model to map from the inputs to a low-dimensional latent space and then using a normalizing flow to map to the parameters of the distribution.  The authors show empirically that this improves over a variety of baselines on a number of OOD and uncertainty quantification tasks.  This paper received 5 reviews who all agreed that the paper should be accepted (6, 6, 8, 8, 8).  The reviewers in general found the method novel compared to existing literature, compelling and the results strong.  Multiple reviewers asked for experiments with higher dimensional output distributions (e.g. CIFAR 100) and had concerns regarding the "entropy regularization" term (akin to the beta term in a beta VAE, this is a constant applied to the entropy term).  The reviewers seemed satisfied with the author response, however, and the concensus decision is to accept.
Decision: Accept (Spotlight)