Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
Authors: Marius Hobbhahn
Agustinus Kristiadi
Philipp Hennig
Venue:  (2021)
Keywords: Bayesian Deep Learning
Approximate Inference

Abstract: In Bayesian Deep Learning, distributions over the output of classification neural networks are approximated by first constructing a Gaussian distribution over the weights, then sampling from it to receive a distribution over the categorical output distribution. This is costly. We reconsider old work to construct a Dirichlet approximation of this output distribution, which yields an analytic map between Gaussian distributions in logit space and Dirichlet distributions (the conjugate prior to the categorical) in the output space. We argue that the resulting Dirichlet distribution has theoretical and practical advantages, in particular, more efficient computation of the uncertainty estimate, scaling to large datasets and networks like ImageNet and DenseNet. We demonstrate the use of this Dirichlet approximation by using it to construct a lightweight uncertainty-aware output ranking for the ImageNet setup.

Under review as a conference paper at ICLR 2021
FAST PREDICTIVE UNCERTAINTY FOR CLASSIFICATION WITH BAYESIAN DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In Bayesian Deep Learning, distributions over the output of classification neural networks are approximated by first constructing Gaussian distribution over the weights, then sampling from it to receive a distribution over the categorical out- put distribution. This is costly. We extend existing work to construct Dirichlet approximation of this output distribution, yielding an analytic map between Gaus- sian distributions in logit space and Dirichlet distributions in the output space. We argue that the resulting Dirichlet distribution has theoretical and practical advantages, in particular more efficient computation of the uncertainty estimate. scaling to large datasets and networks like ImageNet and DenseNet. We demon- strate the use of this Dirichlet approximation by using to construct a lightweight uncertainty-avee aware output ranking for the ImageNet setup.
INTRODUCTION
Quantifying the uncertainty of Neural Networks' (NNs) predictions is important in safety-critical applications such as medical-diagnosss (Begoli et al., 2019) and self-driving vehicles (McAllister al. 2017 Michelmore et af., 2018). Architectures for classification tasks produce probability distribution as their output, constructed by applying the softmax to the point-estimate output of the penultimate layer. However, has been shown that this distribution is overconfident (Nguyen et al.) 2015 Hein et al. 2019) and thus cannot be used for predictive uncertainty quantification.
Approximate Bayesian methods provide quantified uncertainty over the network's parameters and thus the outputs in tractable fashion. The commonly used Gaussian approximate posterior (MacKay] 1992al Graves 2011; Blundell et al. [015 Ritter et al. 2018) approximately induces Gaussian distribution over the logits of NN (Mackay) 1995). But the associated predictive dis- tribution does not have an analytic form. It is thus generally approximated by Monte Carlo (MC) integration requiring multiple samples. Predictions in Bayesian Neural Networks (BNNs) are thus generally expensive operations.
In this paper, we re-consider old but largely overlooked idea originally proposed by David JC MacKay (1998) in a different setting (arguably the inverse of the Deep Learning setting) which transforms a Dirichlet distribution into Gaussian. Dirichlet distributions are generally defined on the simplex. But when its variable is defined on the inverse softmax's domain, its shape effec- tively approximates a Gaussian. The inverse of this approximation, which will be called the Laplace Bridge here (Hennig et al., (012), analytically maps a Gaussian distribution onto a Dirichlet dis- tribution. Given a Gaussian distribution over the logits of a NN, one can thus efficiently obtain an approximate Dirichlet distribution over the softmax outputs (Figure I].
Our contributions are: We re-visit MacKay's s derivation with particular attention to a symmetry con- straint that becomes necessary in our "inverted" use of the argument from the Gaussian to the Dirich- let family. We then validate the quality of this approximation both theoretically and empirically, and demonstrate significant speed-up over MC-integration. Finally, we show use-case, leveraging the analytic properties Dirichlets to improve the popular top-k metric through uncertainties.
Section 2 provides the mathematical derivation. Section discusses the Laplace Bridge in the con- text of NNs. We compare it to the recent approximations of the predictive distributions of NNs in Section 4 Experiments presented in Section

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(C)
(d)
(c)
Figure 1: Densities on the simplex of the true distribution (top row, computed by MC integration) and "Laplace Bridge" approximation constructed in this paper (bottom row). For column (a) and (b), two different Gaussians were constructed, such that the resulting MAP estimate the same, but the uncertainty differs. For (c), (d) and (e) the same mean with decreasing uncertainty used. We find that in all cases the Laplace Bridge a good approximation and captures the desired properties.
2 THE LAPLACE BRIDGE
Laplace approximation are popular and light- weight method approximate general probability distribution (xx) with Gaussian N (x| Σ) when q(x) is twice differentiable and the Hessian at the mode positive definite. sets μ mode of q, and Σ -(V log q(x)),+)== the inverse Hessian of log at that mode. This scheme can work well if the true distribution is unimodal and defined on the real vector space.
The Dirichlet distribution, which has the density function
r (2 K k-1 Cik)
Dir(T/a)
II m
1 k= [(ar)
defined on the probability simplex and can be "multimodal" in the sense that the distribution diverges in the k-corner of the simplex when ar 1. Both issues preclude Laplace approximation, at least in the naïve form described above. However, MacKay (1998) noted that both can be fixed, elegantly, by change of variable (see Figure (2. Details of the following argument be found in the supplements. Consider the K K-diimensioal variable π ~ Dir(T(a defined as the softmax of E K
(2) = exp(zz)
(2)
E exp(2))
for all k = 1,..., K. We will call z the logit of π. When expressed as a function of z, the density of the Dirichlet in has be multiplied by the absolute value of the determinan of the Jacobian
dn Oz
det
II Tx(2),
(3)
thus removing the terms the exponent:
EK k= Oik
K
Dirz(T(())e = II 1 I(ax)
II (z)
(4)
This density of z, the Dirichlet distribution in the sofimax basis, can be accurately approximated by a Gaussian through Laplace approximation, yielding an analytic map from the parameter space IR to the parameter space of the Gaussian (μ R and symmetric positive definite
'For clarity: Laplace approximations are also one out several possible ways construct a Gaussian approximation to the weight posterior of a NN, by constructing second-order Taylor approximation of the empirical risk at the trained weights. This is not the way they are used in this section. The Laplace Bridge is agnostic to how the input Gaussian distribution is constructed. It could, e.g. also he constructed as variational approximation, the moments of Monte Carlo samples. See also Section
2

---

Under review as a conference paper at ICLR 2021
Laplace in standard besis
Leplace in logit basis
Transoormation back to standard basis
D
D
N
N
pla
plajdd
0.5
11.
Figure 2: (Adapted from Hennig et al. (2012)). Visualization of the Laplace Bridge for the Beta distribution (ID special case of the Dirichlet). Left: "Generic" Laplace approximations of standard Beta distributions by Gaussians. Note that the Beta Distribution (red curve) does not even have a valid approximation because the Hessian is not positive semi-definite. Middle: Laplace approxima- tion to the same distributions after basis transformation through the softmax (4). 'he transformation makes the distributions "more Gaussian" (i.e. uni-modal, bell-shaped, with support the real line), thus making the Laplace approximation more accurate, Right: The same Beta distributions, with the back-transformatio of the Laplace approximations from the middle figure the simplex, yield- ing a much improved approximate distribution. In particular, in contrast to the left-most image, the dashed lines now actually are probability distributions (they integrate
the simplex).
RK*K), given by
Nk =log Ork
log on
(5)
K
Ese = ERt
(6)
K
akk
K
u=l
The corresponding derivations require care because the Gaussian parameter space evidently larger than that of the Dirichlet and not fully identified by the transformation. A pseudo-inverse of this map was provided by Hennigg et al. (2012). It maps the Gaussian parameters to those of the Dirichlet as
1 ΣE
OKk
K2
(this equation ignores off-diagonal elements of E, more discussion below). Together, Eqs. 5 and will here be used for Bayesian Deep Learning, and jointly called the Laplace Bridge. A full derivation of the Laplace Bridge can be found in B and Even though the Laplace Bridge implies a reduction of the expressiveness of the distribution, we show in Section that this map is still sufficiently accurate.
Figure I shows the quality of the resulting approximation. We consider multiple different μe, E in three dimensions. We exhaustively sample from the Gaussian and apply the softmax. The result- ing histogram is compared to the PDF of the corresponding Dirichlet. The first part of the figure emphasizes that point estimate insufficient. Since the mean for the Dirichlet is the normalized parameter vector a, the parameters (a, [2, 2, 6] and a2 [11, 11, 51] yield the same point estimate even though their distributions are clearly different. The second part shows how the Laplace Bridge maps w.r.t decreasing uncertainty.
THE LAPLACE BRIDGE FOR BNNS
Let fo N + K be an L-layer Neural Network parametrized by E R.P. with a Gaussian approximate posterior Ni μe, Eo). For any input x E R.N. one way to obtain an approximate Gaussian distribution on the pre-softmax output (logit vector) fx( =: z is as
q(z)x) R [22)fmu ((x) [x
where J(x) is the P x K Jacobian matrix representing the derivative linearized approximation of the network which doesn't account for Inarities often
dz
Mackay
1995). This
80

---

Under review as a conference paper at ICLR 2021
computationally infeasible. For the experiments with larger networks therefore use last-layer Laplace approximation of the network as successfully used, e.g. by Snoek et al. (2013); Wilson et al. 2016) Brosse al. (2020); Kristiadi et al. 22220). is given by
(((/x) = N (Z)pwo ((x) [
where (ax) denotes the output of a forward pass through the first L-1 layers, (w is the maximum likelihood estimate for the weights of the last layer, and Ewi is the inverse of the negative Hessian
w.r.t. the loss Ew - aaww }-1
8
Approximating the density of the softmax of this Gaussian random variable as a Dirichlet, using the Laplace Bridge, analytically approximates the predictive distribution in single step, as opposed to many samples. From Eq. 77. this requires O(K) computations to construct the K parameters ag of the Dirichlet. In contrast, MC-integration has computational costs of ((MJ), where M is the number of samples and J is the cost of sampling from ((z(x) (typically J is of order y² after an initial O(K*) operation for a matrix decomposition of the covariance). The Monte Carlo approximation has the usual sampling error ((1/ MM), while the Laplace Bridge has fixed but small error (empirical comparison in Section 5.2)
We now discuss several qualitative properties of the Laplace Bridge relevant for the uncertainty quantification use case in Deep Learning. For output classes of "comparably high" probability, as defined below, the variance Var(TA (a) under the Laplace Bridge increases with the variance of the underlying Gaussian. In this sense, the Laplace Bridge approximates the uncertainty information encoded in the output of a BNN.
Proposition 1 (proof in supplements). Let Dir(TT(o) be obtuined via the Laplace Bridge from Gaussian (Z) Σ) over R.K. Then, for each = 1,.... K, letting Oxp == Lize on if
ag > 1 (V9a2 10a4x + axt -1)
then the variance Var(TTla)) of the k-th component of is increasing in Egs-
Intuitively, this result describes the condition that needs to fulfilled such that the variance of the resulting Dirichlet scales with the variance of the k-th component of the Gaussian. It can be seen as a proxy for a high-quality approximation. An empirical evaluation showing that this condition fulfilled in most cases can be found in the supplements.
Further benefits of this approximation arise from the convenient analytical properties of the Dirichlet exponential family. For example, a point estimate of the posterior predictive distribution is directly given by the Dirichlet's mean,
OKK
Ex
(10)
Σ 04
11
Additionally, Dirichlets have Dirichlet marginals: If p(T)) = Dir(T(aa), then p 11,
Tj. k>
Dir
01, 02
Ojj
Okk
(11)
additional benefit of the Laplace Bridge for BNNs is that is more flexible than an MC-integral. If we let p(m (T) be the distribution over the MC-integral can be seen as a "point-cstimate" of this distribution since it approximates Ex. In contrast, the Dirichlet distribution Dir(T (x) approximates the distribution p( (T). Thus, the Laplace Bridge enables tasks that can be done only with a distribution but not point estimate. For instance, could ask "what is the distribution of the first L classes?" when one is dealing with K-class (L < K) classification. Since the marginal distribution can be computed analytically [1.. the Laplace Bridge provides convenient yet cheap way of answering this question.
soft == [e*1 / e* e=*// E e*1 e²] then
RELATED WORK
In Bayesian Neural Networks, analytic approximations of posterior predictive distributions have attracted great deal of research. In the binary classification case, for example, the probit ap- proximation has been proposed already in the 1990s (Spiegelhalter and Lauritzen 1990) MacKay
4

---

Under review as a conference paper at ICLR 2021
1992b). However, while there exist some bounds (Titsias 2016) and approximations of the expected log-sum-exponen function (Ahmed and Xing 2007 Braun and McAuliffe 2010), in the multi-class case, obtaining good analytic approximation of the expected softmax function under Gaussian measure is still considered an open problem. The Laplace Bridge can be used to produce a close analytical approximation of this integral. It furthers the trend of sampling-fres solutions within Bayesian Deep Learning (Wu et al., 2018 Haussmann et al., 2019] etc.). The crucial difference is that, unlike these methods, the Laplace Bridge approximates the full distribution over the softmax outputs of a deep network.
Recently, has been proposed to model the distribution of softmax outputs of a network directly. Similar to the Laplace Bridge, Malinin and Gales (2018) 2019): [Sensoy et al. (2018) proposed to use the Dirichlet distribution to model the posterior predictive for non-Bayesian networks. They further proposed novel training techniques in order to directly learn the Dirichlet. In contrast, the Laplace Bridge tackles the problem of approximating the distribution over the softmax outputs of the
ubiquitous Gaussian-aprroximatee Bayesian networks (Graves 2011 Blundell et al. [2015] Louizos and Welling 2016 Sun et al. 2017] etc) without any additional training procedure. This allows the Laplace Bridge to be used with pre-trained networks and emphasizes its non-invasive nature.
EXPERIMENTS
We conduct three main experiments. First, we compare the Laplace Bridge to the MC-integral in the example application of out-of-distribution (OOD) detection (Section 5.1). Secondly, we compare their computational cost and contextualize the speed-up for the prediction process Section [.2]
Thirdly, in Section [.3] we present analysis ImageNet (Russakovsky et al. (214]) to demonstrate the scalability of the LB and the advantage of having a full Dirichlet distribution over softmax outputs. For preliminary experiment to showcase useful features of the LB the MNIST dataset and code for all experiments we refer the reader to Appendix [
All experiments were conducted using different forms of Laplace approximations the weight space. For the smaller experiments, a full (all-layer) Laplace approximation with a diagonal Hessian (LeCun al. 1990) was used. For the experiments with larger networks, the Laplace approxi- mation has been applied only to the last-layer of the network. This scheme has been successfully used by (Snoek al. (2015): Wilssn al.] (2016); Brosse et al. (2020), etc. and it has been shown theoretically to mitigate overconfidence problems in ReL.U networks (Kristiadi et al. 2020). For the last-layer experiments we use diagonal and frrttizer-factorizzed (Ritter et al. 2018)) (KFAC) approximations of the Hessian, since inverting the exact Hessian is too costly. A detailed mathe- matical explanation and setup of the experiments can be found in appendix D While the Laplace Bridge could also be applied to different approximations of a Gaussian posterior predictive such as Variational Inference (Graves) 2011 Blundell et al. 2015). we used a Laplace approximation in our experiments construct such approximation. This is for two reasons: (i) it is one of the fastest ways to get a Gaussian posterior predictive and (ii) it can be applied to pre-trained networks which is especially useful for ImageNet experiments. Nevertheless, we want to emphasize again that the Laplace Bridge can be applied to any Gaussian over the outputs independent of the way it was gener- ated. Despite the overlap in nomenclature, the Laplace Bridge not restricted Gaussians arising from a Laplace approximation of the network.
5.1 OOD DETECTION
We compare the performance of the Laplace Bridge to the MC-integral (Diagonal and KFAC) on a standard OOD detection benchmark suite, test whether the Laplace Bridge gives similar results to the MC sampling methods and compare their computational overhead. Following prior literature, http--aaiiuumcoooiiieeee
(MMC) and area under the ROC-eurve (AUROC) metrics (Hendrycks and Gimpel, 2016). For an in-distribution dataset, higher MMC value is desirable while for the OOD dataset we want a lower MMC value (optimally, 1/K in K-class classification problems). For the AUROC metric, the higher the better, since it represents how good a method is for distinguishing in- and dut-tf-diitributiin datasets.
we use the standard
The test scenarios are: (i) A two-layer convolutional network trained the MNIST dataset. To approximate the posterior over the parameter of this network, a full (all-layer) Laplace approxima- tion with diagonal Hessian is used. The OOD datasets for this case FMNIST (Xiao et al.)
5

---

Under review as a conference paper at ICLR 2021
Table 1: OOD detection results. The Laplace Bridge (LB) wins most comparisons with Diagonal sampling and draws even with KFAC sampling w.r.t. both metrics. However, the LB is around 400 times faster on average which the key property of the LB. 1000 samples were drawn from the Gaussian over the outputs motivated by Figure 3 The MNIST experiments were done with Laplace approximation of the entire network while the others only used the last layer. The same table with error bars is in Appendix D
Diag Sampling
Ding LB
KFAC Sampling
KFAC
Time in J.
Train
Test
MMC L AUROCT
MMC 4 AUROC f
MMC AUROCT f MMCJ AUROC f Sampling
LB
MNIST
MNIST
0.942
0.987
26.8
0.062
MNIST
FMNIST
0.397
0.992
0.363
0.996
26.8
0.062
MNIST
BOLMNIST
0,543
0.960
0.649
0.961
50.3
0.117
MNIST
0.637
26.9
0.062
CIFAR-10
CIFAR-10
0.948
0.966
0.857
0.966
6.58
0.017
CIFAR-10
CIFAR-100
0.708
0.889
0.742
0.80 0.866
0.562
0.880
0.741
0.866
6.59
0.016
CIFAR-10
SVHN
0.643
0,933
0.647
0.934
0.484
0.939
0.648
0.934
17.0
0.040
SVHN
SVHN
0
0,993
0,947
0.993
17.1
0.042
SVHN
CIFAR-100
0.595
0.984
0.526
0.985
0.460
0.986
0.527
0.985
6.62
0.017
SVHN
CIFAR-10
0.593
0.984
0.520
0.987
0.458
0.986
0,520
0.987
6.62
0.017
CIFAR-100
CIFAR-100
0.762
0,590
0.757
0.593
6.76
0.016
CIFAR-100 100
CIBAR-10
0.467
0.788
0,206
0.791
0.463
0.788
0.209
0.791
6.71
0.017
CIFAR- SVHN
0.795
0.170
0.453
0.798
0.173
17.3
2017). notMNIST (Bulatov 2011 and KMNIST (Clanuwat et al. 2018 (ii) For larger datasets, and CIFAR- 100 (Krizhevsky) et al. 2014), we use Res.Net-18 network (He al. 2016). Since this network is large, Equation (8) in conjunction with full Laplace approximation is too costly. We, therefore, use a last-layer Laplace approximation to obtain the approximate diagonal and KFAC Gaussian posterior. The OOD datasets for CIFAR-10, SVHN, and CIFAR- 100 are SVHN and CIFARI00; CIFAR- and CIFAR- 100; and SVHN and CIFAR- 10. respectively. In all scenarios, the networks are well-trained with 99% accuracy on MNIST, 95.4% on 1I 76.6% on CIFAR-100, and 100% on SVHN. For the sampling baseline, we use 1000 posterior samples to compute the predictive distribution which is motivated by Figure 3 We use the mean of the Dirichlet to obtain a comparable approximation to the MC-integral. The results are presented in Table 1 The Laplace Bridge yields, on average, better results than diagonal sampling and ties with KFAC sampling w.r.t both metrics. However, the Laplace Bridge around 400 times faster than both sampling based methods and is therefore preferable.
T.e. CIFAR-I0 (Krizhevsky al. 2014), SVHN (Netzer et al. 2011 al.]|2011
Further comparisons with ensemble networks and other methods to approximate the integral of a softmax-Gaussian can be found in Appendix [ In both cases the LB is comparable or better in terms of OOD detection while being faster and yielding a full distribution.
5.2 TIME COMPARISON
We compare the computational cost of the density-estimated Psample distribution via sampling and the Dirichlet obtained from the Laplace Bridge PLB for approximating the true Pine over softmax- Gaussian sample:"] Different amounts of samples are drawn from the Gaussian, the softmax is applicd and the KL divergence between the histogram of the samples with the true distribution is computed. We use KL-divergences DKL. (Pune (Paue |Pample) and (Puuel (PP.n. respectively, measure similarity between approximations and ground truth while the number of samples for Psample is in- creased exponentially. The true distribution Pire is constructed via MC with 100k samples. The experiment is conducted for three different Gaussian distributions over IR3, Since the softmax ap- plied to a Gaussian does not have an analytic form, the algebraic calculation of the approximation error is not possible and an empirical evaluation via sampling is the best option. The logistic-Normal (Aitchison and Shen 1980) analytic expected value. The fact that there is no analytic solution is part of the justification for using the Laplace Bridge in the first place.
different distribution and not useful for our purposes it has no
"Le. samples are obtained by first sampling from Gaussian and transforming it via the softmax function.

---

Under review as a conference paper at ICLR 2021
Laplace Bridge
40-
Laplace Bridge
KL Divergence
Monte Carlo
KL Divergence
Monte Carlo
20-
10:
Number of Samples
Time
Figure 3: KL-divergence plotted against the number of samples (left) and wall-clock time (right). Monte Carlo density estimation becomes as good the Laplace Bridge after around 750 to 10000 samples and takes at least 100 times longer. The three lines represent three different samples.
Figure 3] suggests that the number of samples required such that the distribution Psample approximates the true distribution Pine good as the Dirichlet distribution obtained via the Laplace Bridge is large, i.e. somewhere between 500 and 10000. This translates to a wall-clock time advantage of at least a factor of 100 before sampling becomes competitive in quality with the Laplace Bridge.
To better understand how weight and non-invasive the Laplace Bridge is, we have timed differ- ent parts of the process for our setup. Training ResNet-18 on CIFARIO over 130 epochs takes us 71 minutes and 30 seconds. Computing a Hessian for the network from the training data can be done with (Dangel et al.. 2019) the cost of one backward pass or around 29 seconds. No other changes to the training procedure are necessary to apply the LB. Since the Laplace Bridge only applies to
Table 2:
Contextualization of the timings for the entire predictive process. We find that the LB provides a significant speed-up of the process as a whole.
# samples in brackets
Forward
MC(1000)
MC(100)
MC(10)
Laplace Bridge
Time in seconds
0.448 1 0.004
8.183 1 0.080
0.867
0.004
0.117 1 0.001
0.014 1 0.000
Fraction of overall time
0.0590.3400.79/0.97
0.95
0.66
0.21
0.03
the last step of the prediction pipeline it is important to compare its time to that of a forward pass through the rest of the network. Re-using the ResNet-18 18 and CIFAR10 setup we measure the time in seconds for a forward pass, for the application of the Laplace Bridge, and for the sampling proce- dure with 10, 100, and 1000 samples. The resulting sum of timings for the entire test is given in T). We find that sampling is between 95% (for 1000 samples) and 21 % (for 10 samples) of the entire prediction while the LBB is only 3%. These results show that the time accelerations through the LB are meaningful improvement for the prediction process whole.
5.3 UNCERTAINTY-AWAN OUTPUT RANKING ON IMAGENET
Classification tasks on large datasets with many classes, like ImageNet, are not often done in a Bayesian fashion since the posterior inference and sampling are expensive. The Laplace Bridge, in conjunction with the last-layer Bayesian approximations, can be used to alleviate this problem. Furthermore, having full distribution over the softmax outputs of BNN gives rise new possi- bilities. For example, one could subsume all classes which have sufficiently overlapping marginal distributions into one if they are semantically similar as illustrated Figure
Another possibility is to improve the standard classification metrics. Large classification tasks like ImageNet are often compared along a top-5 metric, i.e. it is tested whether the correct class is within the five most probable estimates of the network. Although widely accepted, this metric has some pathologies. Consider two examples: i) Assume the network has classify a hypothetical image of "raptor" and it is confident that the label is either "hawk" or an "cagle". Then all probability mass should be distributed between those two classes. ii) Assume the network has to classify an image of which it is confident that it is "fish" but it is uncertain between ten different possible fish species. In both cases a static rule (always 5) is ill-suited.
Leveraging the probabilistic output provided by the Laplace Bridge, we propose simple decision rule that can handle such examples and is more fine-grained due to its awareness of uncertainty. One may call such rule uncertainty-awen top-k; pseudocode for the algorithm is given in the
7

---

Under review as a conference paper at ICLR 2021
gove
spece hanibekk compirr
Figure 4: Upper row: images from the "laptop" class of ImageNet. Bottom row: Beta marginals of the top-k predictions for the respective image. In the first column. the overlap between the marginal of all classes is large, signifying high uncertainty, i.e. the prediction is "I do not know". In the second column, "notebook" and "laptop" have confident, yet overlapping marginal densities and therefore yield top-2 prediction: "either a notebook or laptop". In the third column "desktop computer", "screen" and "monitor" have overlapping marginal densities, yielding top-3 estimate. The last case shows a top-1 estimate: the network is confident that "laptop" the only correct label.
supplements. Instead of taking the top-k as decision threshold for arbitrary h take the uncertainty/confidencc of the model to inform the decision. This is more flexible and therefore able to handle situations in which different numbers of classes are plausible outcomes. The Dirich- let distribution obtained from the Laplace Bridge provides this capability. In particular, since the marginal distribution over each component of Dirichlet distribution is a Beta(o,, Ej a,). this can be done analytically and efficiently. The proposed decision rule uses the area overlap between the marginal distributions of the sorted outcomes. This is similar to hypotheses testing, i.c. t-tests (Nickerson) 2000) or its Bayesian alternatives (Masson 2011). If, for example, two Beta densities overlap more than 5%, we cannot say that they are different distributions with high confidence. All distributions that have sufficient overlap should become the new top-k estimate. Pseudo code for the uncertainty aware top-k ranking can be found in Algorithm| [ Figure 4 shows four examples from the "laptop" class of ImageNet. We evaluate this decision rule on the test set of ImageNet. The
40000
Count
20000
2
4
6
10
Top-k
Figure 5: A histogram of ImageNet predictions' length using the proposed uncertainty- top-k. Most test images a top-1 prediction, indicating high confidence. There are some top-2, top-3, and top-10 predictions, showing increased uncertainty. All predictions >> 10 are grouped in the 10-bin for visibility.
overlap is calculated through the inverse CDF of the respective Beta marginals. The original top-1 accuracy of DenseNet ImageNet is 0.744. Mcanwhile, the uncertainty- -wware top-k accuracy is 0.797, where l is on average 1.688. A more detailed analysis of the distribution of top-k estimates (see Figure shows that most of the predictions given by the uncertainty- a top-1 prediction. This means that using uncertainty does not imply adding meaningless classes to the prediction. However, there are some non-negligible cases where k equals to 2, 3, 10 where all
metric still yielded
Also known the quantile function percent point function
8

---

Under review as a conference paper at ICLR 2021
values larger than 10 are in the 10 bin for visibility. This indicates that whenever the class labels ambiguous, our method can detect it, and thus yields a significantly higher accuracy.
Algorithm -ncertainty-aware top-k
Input: A Dirichlet parameter a E RK obtained by applying the Laplace Bridge to the Gaussian over the logit of an input, percentile threshold T e.g. 0.05, a function class_of that returns the underlying class of sorted index.
á sort descending(a)
Il with the highest confidence II initialize top-k, must include least closs
ag E; 04
C = {class_of(1]
for i = 2,..., K do
F Beta(â;,1, a
the previous marginal # the current marginal CDF # left & percentile of the previous marginal If right Ig percentile the current marginal
F; = Betal ag - ā;)
4-1 = FT (T)2)
r F. (1 T/2)
if ri > 4-11 then
C CU {class_of (i))
Il overlap, end algoritiun overlop derected, add the current class
else
break end if' end for
Output: C
return the resulting top-k prediction
CONCLUSION
We have adapted an old overlooked approximation scheme for new use in Bayesian Deep Learn- ing. Given Gaussian approximation to the weight-space posterior of neural network (which can be constructed by various means, including another Laplace approximation), and an input, the Laplace Bridge analytically maps the marginal Gaussian prediction on the logits onto Dirich- let distribution over the softmax vectors. The associated computational cost of O(K) for K-class prediction compares favorably that of Monte Carlo sampling. The proposed method both theoret- ically and empirically preserves predictive uncertainty, offering an attractive, low-cost, high-quality alternative Monte Carlo sampling. conjunction with low-cost, last-layer Bayesian approxima- tion, it can be useful in real-time applications wherever uncertainty is required especially because it reduces the cost of predicting posterior distribution at test time.

---

Under review as a conference paper at ICLR 2021
REFERENCES
E. Begoli, T. Bhattacharya, and D. Kusnezov. The need for uncertainty quantification in machinc-assisted medical decision making. Nat Mach Intell, 1:20-23, 2019.
Rowan McAllister, Yarin Gal, Alex Kendall, Mark van der Wilk, Amar Shah, Roberto Cipolla. and Adrian Weller. Concrete problems for autonomous vehicle safety: Advantages bayesian deep learning. In IJCAI, 2017.
Rhiannon Michelmore, Marta Kwiatkowska, and Yarin Gal. Evaluating uncertainty quantification in end-to-end autonomous driving control. CoRR, abs/1811.00 1.0817, 0,817,20 2018.
Anh Nguyen. Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predic- tions for unrecognizable images. In CVPR, 2015.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- high-confidence predictions far away from the training data and how to mitigate the problem. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
David J. C. MacKay. A practical bayesian framework for backpropagation networks. Neurul Comput. 4(3): 448 472, May 1992a. ISSN 0899-7667.
Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. S. 7emel, P. L.. Bartlett, F. Pereira, and K. Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2348823366 Curran Associates, Inc., 2011.
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. ArXiv, 2015.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations. 2018
David J C. Mackay. Probable networks and plausible predictions review of practical bayesian methods for supervised neural networks. Network: Computation in Neural Systems, 6(3):469-505, 1995.
David J.C. MacKay, Choice of basis for laplace approximation. Machine Learning, 33(1):77 86, Oct 1998. ISSN 1573-0565.
Hennig, D. Stern. R. Herbrich, and T. Graepel. Kernel topic models. In Fifteenth International Conference on Artificial Intelligence and Staristics, volume of JMLR Proceedings, pages 51 511-519. JorR. 2012.
Jasper Snock, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Pat- wary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, vol- ume 37 of Proceedings of Machine Learning Research, pages 2171-2180, Lille, France, 07-09 Jul 2015. PMI.R
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning, In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 370-378, Cadiz, Spain, 09-11 May 2016. PMLR.
Nicolas Brosse, Carlos Riquelme, Alice Martin, Sylvain Gelly, and Éric Moulines. On last-layer algorithms for classification: Decoupling representation from uncertainty estimation. arXiv preprint arXiv:2001.088 0800499 2020.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just bit, fixes overconfidence relu networks. arXiv preprint arXiv:2002. 2020.
David J Spiegelhalter and Steffen L Lauritzen. Sequential updating of conditional probabilities directed graphical structures. Networks, 20(5):579-605, 1990.
David IC MacKay. The evidence framework applied to classification networks. Neural computation, 4(5): 720-736, 1992b.
Michalis Titsias. One-vs-each approximation sofumax for scalable estimation of probabilities. In NIPS, 2016.
Amr Ahmed and Eric Xing. On tight approximate inference of the logistic-normal topic admixture model. In Proceedings of the 11th Tenth International Workshop Artificial Intelligence and Statistics, 2007.
10

---

Under review as a conference paper at ICLR 2021
Michael Braun and Jon McAuliffe. Variational inference for large-scale models of discrete choice. Journal of the American Statistical Association, 105(489):324-335 2010.
Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E. Turner, José Miguel Hernández-Lobate and Alexan- der L. Gaunt. Fixing variational bayes: Deterministic variational inference for bayesian neural networks. arXiv, abs/1810.03958, 2018.
Manuel Haussmann. Sebastian Gerwinn, and Melih Kandemir. Bayesian evidential deep learning with pac regularization, 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pages 7047-7058, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. In Advances in Neural Information Processing Systems, pages 14520-14531, 2019.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classification uncer- tainty. In Advances in Neural Information Processing Systems, pages 3179-3189, 2018
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In ICMI., 2016.
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertainty in bayesian neural networks. In Artificial Intelligence and Statistics, pages 1292, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjecv Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. CoRR, 0bs//1409.055 0075, 2014.
Yann LeCun. John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky, editor, Advances in Neural Information Processing Systems 2, pages 598-605. Morgan-Kauffaam, 1990.
Dan Hendrycks Kevin Gimpel. A baseline for detecting misclassified and distr--iistiitttinn examples neural networks. arXiv, abss/610.00116. 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv, 0ts//708.0074777 2017.
Yaroslav Bulatov. notmnist dataset, 2011. URL http:://grrollwwbbbb..ogeee...//
Intmmisst stanmiisttdtttet..... .tml
dataset.........
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv, abs/1812.01718,, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar- 10 dataset. online: http://www. cs. toronto edufkriz/cifar. html, 55, 2014.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
J. Aitchison and S. M. Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2): 261-272, 1980. ISSN 00063444. bttp: //www//wwww/....//.///////./. jstor. wttp://bbb///ee/e34/77
Felix Dangel. Frederik Kunstner, and Philipp Hennig. Backpack: Packing more into backprop. arXiv preprint arXiv:1912.10985, 2019.
Raymond S Nickerson. Null hypothesis significance testing: a review of an old and continuing controversy. Psychological methods, 5(2):241, 2000.
Michael E. J. Masson. A tutorial on a practical bayesian alternative to mull-hypothesis significance testing. Behavior Research Methods, 43(3)x679-690, Sep 2011. ISSN 1554-3528.
James Martens and Roger Grosse. Optimizing neural networks with frctoree approximate curvature. In ICML, 2015.
Mark N. Gibhs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, University of Cambridge, September 1997.
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
FIGURES
The parameters of Figure 1 are the ji and Y mapped from (2, and 11,,11,51 51) through the Laplace Bridge for a) and b) respectively. The values have been chosen have the same mode of the Dirichlet distribution clearly yield different uncertainties. For c). d) and they share μ = 1, -1) and vary with 10 Iz, 1 13 and 0.1 I3 where I3 is the three-dimensional identity matrix.
The parameters of Figure are from left to right o, 3 = (0.8, 0.9) (4,2),,
CHANGE OF VARIABLE FOR PDF
Let z be n-dimensional continuous random variable with joint density function Px. I G(x), where G differentiable function, then y has density Py:
ply) = 5 det
dG (z) dz
(12)
where the differential is the Jacobian of the inverse of G evaluated y. This procedurc, also known 'change of basis', the core the Laplace bridge since it used transform the Dirichlet into the softmax basis.
PROOF FOR PROPOSITION
Proof. Considering that Ox is decreasing function of E by definition t is sufficient to show that under the hypothesis, the derivative of 8 Var(Tx (r) is negative.
By definition, the variance Var(Ta is
ta togg))
Var(Te/a:)
OAk + Cleck +
The derivative is therefore
dau Var(mala)
azs axpee + area au a2 + 1)) (au + (ax axec 1)2
Solving das
a Var(TTl(x) for ar yields
On 4 (V9az, 10 atple -
1
Therefore, under this hypothesis, Var(T& (x) is decreasing function of
EXPERIMENTAL EVALUATION OF THE PROPOSITION
To test how often condition fulfilled we count its frequency. The fact that the condition fulfilled implies good approximation. The fact that the condition not fultilled does not automatically imply a bad approximation.
B APPENDIX (DERIVATION OF LB)
Assume we have Dirichlet in standard basis with parameter vector a and probability density function:
K au r (>
Dir(T =
ok
(13)
1 = [(ax)
We aim transform the hasis of this distribution via the softmax transform be in the base m::
12

---

Under review as a conference paper at ICLR 2021
frequency
MNIST
MNIST
MNIST
FMNIST
MNIST
KMNIST
CIFAR-I0
CIFAR-10
0.998
CIFAR-I0
CIFAR-100
0.925
CIFAR-10
SVHN
SVHN
CIFAR-100
0.668
SVHN
CIFAR-10
0.653
CIFAR-100 CIFAR-100
CIFAR-100
0.662
SVHN
0,166
Table 3
(2) exp(2x)
(14)
Σ exp(z1)
Usually, to transform the basis would need the inverse transformation H H²) described in the main paper. However, the softmax does not have an analytic inverse. Therefore David JC MacKay uses the following trick. Assume we know that the distribution in the transformed basis is:
(E ax Dir, (TT)))) = I I(ax)
Il Ta()))
(15)
then we can show that original distribution is result the basis transform by softmax.
The Dirichlet in the softmax basis: We show that the density over и shown Equation 15 transforms into
the Dirichlet over z. First, we consider the special case where æ is confined to a / dimensional subspace satisfying E,
In this subspace represent by an I dimensional vector such that
n. = a. i, mi
(16)
1-1
(17)
and similarly represent by dimensional vector y::
Pi
1, i,...,
(18)
I-1
21
(19)
then we can find the density over oo (which proportional the required density over z) from the density over
x (which is proportional to the given density over лт) by finding the determinant of the (I 1) x (1 - 1)
Jacobian J given by
8z 0.p.
Oz, θig Oi, Dipt
Ja.
Σ
6. ZiZki + ZiZs z.(6.1 (Za zi))
(20)
13

---

Under review as a conference paper at ICLR 2021
We define additional - dimensional helper vectors 2 y from lincar algebra. follows that
24 and rie and use del (1- "-a""y")
det J det. nz 1
r-1
2i x (1 - n2 =x)2
(21)
i=1
- Σ 2
2: x
X;
Therefore, using Equation 15
find that
P(T)) det.J
P(z)
(22)
This result is true for any constant e since it can be put into the normalizing constant. Thereby we make sure that the integral of the distribution is 1 and we have valid probability distribution.
C APPENDIX (DERIVATION OF INVERSION)
Through the figures of the ID Dirichlet approximation in the main paper we have already established that the mode of the Dirichlet lies at the mean of the Gaussian distribution and therefore T(y) a. Additionally, the elements of must sum zero. These two constraints combined yield only one possible solution for μ..
log Clk
log cx
(23)
Calculating the covariance matrix 2 is more complicated but layed out in the following. The logarithm of the Dirichlet is, to additive constants
log Py (ylaa
Σ
(24)
k
Using the softmax of shown in Equation 14 we find the elements of the Hessian L
Lu a(dartf ft)
(25)
where á : Σx on 2% value π at mode, Analytically inverting L done via lengthy derivation using fact that can write XBX and inverting it with the schur-complement This process results in the inverse the Hessian
K Σ
]
L
Eat Ol
(26)
K
Ohk
On
K
Ohu
We mostly interested in the diagonal elements, since desire a sparse encoding for computational reasons and we otherwise needed to map a K covariance matrix x Dirichlet parameter vector which would be very overdetermined mapping. Note that K is a scalar not matrix. The diagonal elements can be calculated as
1 O%
E
(27)
K2
To invert this mapping we transform Equation
23
to
K
1/K
(28)
14

---

Under review as a conference paper at ICLR 2021
by applying the logarithm and re-ordering some parts. Inserting this into Equation 27 and re-arranging yields
K
I
1/K
e
+
(29)
Eee
K
K
which can be re-inserted into Equation 28 to give
2
OK
Eg
K
K
Mk
(30)
which is the final mapping. With Equations 23 and 27 we are able to map from Dirichlet to Gaussian and with Equation 30 we are able to map the inverse direction.
D APPENDIX (EXPERIMENTAL DETAILS)
The exact experimental setups, i.c. network architectures, learning rates, random seeds, found in the accompanying anonymized GitHub repository This section used to justify some the decisions we made during the process in more detail, highlight some miscellaneous interesting things and showcase the additional experiments promised in the main paper.
EXPERIMENTAL SETUP
The (F-, K-, not MNNIST networks have been trained with Adam, a learning rate of le 6 and batch size of 32 over 6 epochs. The batch size chosen be so low because the Jacobian for the full-layer Laplace approximation would otherwise not fit into memory. CIFARIO, CIFARI00 SGD, start learning rate 0.1, momentum
SVHN have been trained with 0.9, weight decay of 5e and a batch size 128. applicd a learning rate schedule that multiplics the learning rate with 0.1 at epoch 30, 80 and 130 for CIFAR 10, 100, 150, 200 for CIFAR 100 and at 10, 20 and 25 for SVHN. (F-, K-, not-)MNIST has been trained small 2 conv-layer the ImageNet experiments we used the pre-trained weights from Py and only computed the Hessian information with BACKpack (Dangel et al. 2019) in one backward pass. All prior uncertainties for the Laplace approximation have been chosen such that the resulting MMC is not more than 5 10% away from the MAP estimate to have comparable results.
linear layer setup and all other OOD experiments
conducted with ResNet-18.
MATHEMATICAL DESCRIPTION OF THE SETUP
In principle, the Gaussian the weights required by the Laplace Bridge for BNNs be constructed by any Gaussian approximate Bayesian method such as variational Bayes (Graves) 2011 Blundell
2015 and Laplace approximations for NNs (MacKay 1992a; Ritter al. (018)). We will Tocus on the Laplace approximation, which uses the same principle as the Laplace Bridge. However, in the Laplace approximation for neural networks, the posterior distribution over the weights of network is the one that is approximated as Gaussian, instead of Dirichlet distribution
the outputs in the Laplace Bridge.
Given dataset D
{(x,, t.) 12 and a prior p(0), let
D x pl (D p(0)
II (x.I)D
ply = :(0,),,,,
(31)
be the posterior the parameter 0 of L-layer network fo. Then can get an approximation of the posterior DD) by fitting a Gaussian Ai(0) Σg) where
μ4 = OMAP
Σg = (- 2 oour log (00
That is, we fit Gaussian centered the mode PMAP of p(ODD) with the covariance determined by the curvature that point. We assume that the prior p(0) zero-mean isotropic Gaussian /V o I²I) and the likelihood function the Categorical density
PD)))
II (x.t)DD
Cat(y = (fsoftmax(fo(x)).
15

---

Under review as a conference paper at ICLR 2021
For various applications in Deep Learning approximation with full Hessian is often computationally too expensive. Indeed, for each input x R one has to do K backward passes to compute the Jacobian [x). Morcover, it requires an ((PK) storage which is also expensive since P is often in the order of millions. A cheaper alternative is to fix all but the last layer of fe and only apply the Laplace approximation on Wr, the last layer's weight matrix. This scheme has been used successfully by [Snoek et al. (2015); (ilson et al. (2016);
Brosse al. 2020), etc. and has been shown theoretically that it can miligate overconlidence problems ReLO networks (Kristiadi al. 2020). In this case, given the approximate last-layer posterior
p W (D) z (vec live (MA Hww):
(32)
one can efficiently compute the distribution over the logits. That is, Ict Φ Rx + R be the first L --1 1 layers of fo. seen as a feature map. Then, for each x E IR,, the induced distribution over the logit W (0(x) =: z is given by
x(zx) = ((x (Mx//Wiise (x @ I)H (6(() @ I)),
(33)
where denotes the Kronecker product.
An even more efficient last-layer approximation can be obtained using Kronecker-factoret matrix normal distribution (Louizos and Welling 2016, (Snn et al. 2017 Ritter et al. 2018). That is, we assume the posterior distribution to be
p ID) MW((WM))))))) ( MAP, U, V),
(34)
where U E _ and V E R are the Kronecker factorization of the inverse Hessian matrix H w
Martens and Grosse 2015). In this case, for any R one can easily show that the distribution over logits given by
p(z = (zz (x)),
(hexx))) (Ø(x) Vo(())U),
(35)
which is easy to implement and computationally cheap. Finally, and even more efficient, is last-layer approx- imation scheme with a diagonal Gaussian approximate posterior, i.e. the so-called mean-field approximation. In this case, we assume the posterior distribution to be
p (DD = NVevcc (eec (MA)), diag(o"))
(36)
where 2² obtained via the diagonal of the Hessian of the tog-posterion w.r.t. vec( E) ) at vee (W MAP)).
UNCERTAINTY ESTIMATES ON MNIST
Mean Variance
0.10
Dist
Out Dist
0.05
6
MNIST Class
Figure 6: Average variance of the Dirichlet distributions of each MNIST class. The in-distribution uncertainty (variance) is nearly nil, while distrribtiiiono variance is higher. This implies usabil- ity for OOD detection.
To give a good preliminary intuitive understanding of the LBs usefulness we empirically investigate the ap- proximation quality of the Laplace Bridge in "real-world" BNN the MNIST datasct. A CNN with 2 convolutional and fully-connected layers trained on the first three digits of MNIST (the digits 1, and 2). Adam optimizer with learning rate 1e-3 and weight decay 5e-4 used. The batch size 128. obtain the posterior over the weights of this network, we perform a full (all-layer) Laplace approximation using Back- PACK (Dangel et al. 2019) to get the diagonal Hessian. The network is then evaluated on the full test set of MNIST (containing all ten classes). We present the results in Figure 6 We show for each le = I,..., K, the average variance 1. Σp, Var( (fo())) of the resulting Dirichlet distribution over the softmax outputs, where D& is the number of test points predicted with label k. The results show that the variance of the Dirichlet distribution obtained via the Laplace Bridge is useful for uncertainty quantification: OOD data can be casily detected since the mean variance of the first three classes is nearly zero while that of the others is higher.
Dy
OOD DETECTION
Every experiment has been conducted with 5 different seeds. In the tables the mean and standard deviations are presented. The reason why the sampling pracedure for the CIFAR-10 and CIFAR-100 case similarly
16

---

Under review as a conference paper at ICLR 2021
fast even though draw from 10- 100-dimensional Gaussian is because the sampling procedures were parallelized on a GPU. All prior uncertainties over the weights were chosen such that the MMC of the sampling averages was around 5% lower than the MAP estimate.
To compare the OOD detection capabilities of the LB with other more standard methods we first evaluate it compared to an ensemble of 100 different sets of weights. As shown in Table we find that the Laplace Bridge yields comparable results while being more than a thousand times faster.
Table 4: Results for sampling from all weights instead of just the last layer. Number of samples was 100. Sampling all weights and doing forward pass seems be worse than using the Laplace Bridge even though it takes much longer.
Sampling (100)
Laplace Bridge
Train
Test
MMC +
AUROCT 1
Time in +
MMC
AUROC 7
Time in
MNIST MNIST
MNIST
0.989= 0.000
76.3
0.987 + 0.000
0,062
FMNIST
0.465 0.004
0.992 1 0.000
76.3
0.363 1: 0.000
0.996 0.000
0.062
MNIST BOMNIST
0.648 0.001
0.955 + 0.001
42.6
0.649
0.000
0.961
0.000
0.117
MNIST
KMNIST
0.632 0.001
0.967 0.001
76.0
0.637 + 0.000
0.973 + 0.000
0.062
However, we think the Laplace Bridge should primarily not be compared to other OOD detection algorithms such as those listed in Hendrycks and Gimpel (2016) but rather to algorithms that approximate the integral of the softmax transform applied to Gaussian samples since they have comparable speed. The two algorithms we compare to the IBB are the Extended MacKay approach Gibbs
Gibbss [997] (sce equation 5.33 in (Gibbs)) which uses
ʃ exp(a Σ, exp(a(t)
(a 60
exp(7 ( Σ, (xp((((())a())
Z
exp
Σ
(37)
as an approximation. The logit output of class is denoted by a its average by a()) its variance by ( and r(O) = 1/v1++ -. 1/88. The second approximation is from Wu et al. (2018) (equation 23 in appendix) and we will refer to it as the "second-order delta posterior predictive (SODPP)" Il is given by
ply) = P 1+ 1+p
Ep + 2 Ep 1+p
diag(E) diag(E)
(38)
where p is the vector of logit outputs. Σ is its covariance matrix, and @ represents the outry-wise product. The results in Table [3] suggest that the LB shows slightly better performance than the other two approaches. Additionally, the LB not only yields an approxination of the integral but additionally a fully parameterized Dirichlet distribution of the output classes and is therefore preferable.
Table 5: Comparison of the Laplace Bridge Dirichlet mean with other methods to compute the in- tegral over a softmax-Gaussian. We find that the LB slightly better than the other two methods while additionally not only providing an estimate for the integral but also giving a fully parameter- ized Distribution.
Extended MacKay
SODPP
Laplace Bridge
Train
Test
MMC
AUROCT 1
MMC
AUROC
MMC
AUROC
MNIST MNISTT
MNIST
0
0.001
0.979 0.001
0.987 1: 0.001
FMNIST
0,557 + 0.022
0.985 + 0.002
0,546 + 0.024
0.983 : 0.003
0.491 + 0.035
0.990 + 0.002
MNIST
BOLMNIST
0.737
0.015
0,928
0.017
0.754
0.012
MNIST
KMNIST
0.673
+ 0.006
0.967
+ 0.002
0.642 + 0.006
0.965 : 0.003
0.710 + 0.008
0.967 + 0.003
CIFAR- 10
CIFAR 10
0.970 1 0.001
0.970 1. 0.001
0.963 1. 0.001
CIFAR-10
CIFAR-100
0
0.003
0.002
0.793
0.010
0.002
0.756
0.003
CIFAR-10
SVHN
0.714 + 0.036
0.933 + 0.012
0.710 + 0.036
0.932 + 0.012
0.638 + 0.049
0.931 + 0.017
SVHN
SVHN
0.976 + 0.002
0.985 + 0.002
0.993 + 0.000
SVHN
CIFAR-100
0.566 + 0.007
0,984 0.002
0.508 0.008
0,984 0.002
0.616 + 0.015
0.981 0.004
SVHN
CIFAR-10
0.563 + 0.007
0.985 1 0.002
0,505 + 0.009
0.985 1 0.002
0.618 = 0.017
0.981 + 0.004
CIFAR-100
CIFAR-100
0.748 + 0.000
0.758 + 0.000
0.593 + 0.000
CIFAR-100
CIFAR-10
0.440 + 0.000
0.790 0. 000
0.455
0.000
0.787 0.0 0.000
0.209 + 0.000
0.791 0.000
CIFAR-100
SVHN
0.427 + 0.000
0.802 + 0.000
0.450 = 0.000
0,793 : 0.000
0.173 + 0.000
0.815 # 0.000
TIME COMPARISON
Every experiment has been conducted with 5 differrnt seeds. The presented curves are the averages over these 5 experiments with emrorbrr. The reason why taking one sample is slower than two is because of the way random numbers are generated for the normal distribution. For further information read up on the Box-Mueller Transform.
17

---

Under review as a conference paper at ICLR 2021
Table OOD detection results. Same table as in the main experiments but with error estimates. The Laplace Bridge (LB) wins most comparisons with Diagonal sampling and draws even with KFAC sampling w.r.t both metrics. However, the LB is around 400 times faster on average. 1000 samples were drawn from the Gaussian over the outputs motivated by Figure 3 The (F-, K-, not-)MNIST experiments were done with a Laplace approximation the entire network while the others only used the last layer. Five runs with different seeds per experiment were conducted. Standard deviation is reported to the third decimal.
Vrats
Tol
333CC
AUROC
MMCC
AUROC
MNCC
ALBOCT
JIHC.
AUROCT
Sampling
1.5
MIJST
MNNST
14042 (1EET
0.947 16000
25.1
1L862
MNNST
EMINST
11.01
16000
0.101
MNNST
KMNIST
(1001
83744
1U550
01913
0100
11162
CBAA--D
CIEAR-D
(1.00
6/966 18000
6387 1 0011
0.96 L 0100)
6.15
ILEIT
CIFAKID
CHARR100
ILT86
11.00
0,89
16000
0.001
0.111
0.000
11886
0,993
SVEN
CIFAR-100
0,555 0.000
0.584 5000
0.526 0,500
0,955
0.000
0,400 0/044
0 0.000
0.527 = 0,0C
0963 0/03
0UTT
SVIEN
CIFAR-I0
1U593
0,000
0,064 4 20003
0.529 6: 9,000
0 = 0.000
0,488
0/914
0J955 9 0.001
0,530 0.000
0307 2: 0.000
6.52
0.80T
CRAR-110
CIPAR-10
SVHIS
0.000
0.755 2: D000
0,878 1U550
0.000
0.55 0/011
0.799 (U800
0.333 0.00)
0215 0000
0.641
UNCERTAINTY-AWARE OUTPUT RANKING ON IMAGENET
The prior covariances for the Laplace approximation of the Hessian the weights were chosen such that uncertainty estimate of the Laplace bridge MMC over the outputs was not more than 5% lower than MAP estimate. The length of list generated by our uncertainty aware method was chosen such that contained at least one and maximally ten samples. Originally we wanted to choose the maximal length according to the size of the largest category (e.g. fishes dogs) but the class tree hierarchy of ImageNet does not answer this question mcaningfully. We chose because there no reasonable hins larger than when looking histogram. Pscudo code for the uncertainty aware top-k ranking can be found in Algorithm [
18


Reviews:
[{"review_id": "u4Rll5ng0N0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis article improves the efficiency of Bayesian neural networks (BNNs). It follows the sampling-free solutions within Bayesian deep learning (Wu et al., 2018; Haussmann et al., 2019, etc.). The difference is the proposed Laplace Bridge that approximates the  full distribution over the softmax outputs of a neural network. The Laplace Bridge tackles the problem of approximating the distribution over the softmax outputs of the ubiquitous Gaussian-approximated BNNs without any additional training  procedure. This allows the Laplace Bridge to be used with pre-trained networks and emphasizes its non-invasive nature.\n\nMajor comments: \n- Overall, I find the paper is easy to follow and the experimental evaluation shows promising results. But my major concern is about the novelty of this work, given the fact that the basic idea was originally proposed by MacKay (1998) in a different  setting which transforms a Dirichlet distribution into a Gaussian and the inverse of this approximation, called the Laplace Bridge, was already proposed in the literature (Hennig et al., 2012).\n- I would recommend the authors to provide more details on the training of BNNs within the Laplace Bridge, including the hyperparameters used for the training stage.\n\nMinor comments:\n- It is recommended to further check with the wording, e.g., the line 4 of the title of Figure 4 “In the column” and adding the description of units of x/y axis. \n- The size of some figures appears too small, for example Fig. 4, which may hinder readability.\n\nAt the moment, I recommend a weak reject as the main weakness is the novelty, but I could be open to increasing my score if my concerns are addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "n8fztjDo3mY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to use Laplace bridge as a building block to map the distribution of logits to the distribution of post-softmax predictions. The authors also provide some theoretical analysis on when such Laplace approximation behaves reasonably.\n\nThe idea of constructing a Dirichlet distribution to describe the output distribution for classification in Bayesian neural networks is not new. For example, natural-parameter networks (NPN) [1] support exponential family distributions including Dirichlet distributions as the output of each layer. The idea of Dirichlet distributions is extended and made more explicit in [2,3]. \n\nThe idea of using the Laplace bridge in BNN is interesting and can potentially be useful. I like the demonstration in Figure 1, showing that Laplace bridge is a good approximation. \n\nOne of my major concerns is that a large trunk of related works are missing, especially those directly related to the use of Dirichlet distributions in BNN’s output. Lack of comparison and description makes it difficult and potentially misleading for reviewers to evaluate the proposed method’s technical merit. Besides, another line of highly related works is on combining Laplace approximation and BNN, such as [5].\n\nEq. 8 is incorrect. Since f_theta is an L-layer network, with the parameter theta, the mean of the network’s output is not simply \\mu_theta^T x. This is because of the multi-layer structure with nonlinearity. Therefore Eq. 8 only holds when L=1.\n\nAnother potentially misrepresentation of related works is that: note that David MacKay’s proposed BNN treatment also involves Laplace approximation. Given that the Laplace bridge idea also originated from David MacKay, it would be better to clarify such related works and the differences between the proposed method and David MacKay’s works.\n\nAnother point that is unclear from the paper is the computation of the Hessian matrix in a large neural network. Given that a neural network typically has a large number of parameters, it would be important to provide details on related implementation in practice. \n\nThe authors emphasize one of the advantages of the proposed method is that it is sampling-free and therefore significantly speeds up computation. Note that there is a rich literature of sampling-free BNN (e.g., [1,2,3,4]) in recent years.\n\nAlthough results on Table 2 make sense, the table itself is a bit misleading. (1) Even with LB, one still need a FF pass to compute the prediction. (2) the 29-second overhead should be made clear in the caption.\n\nThe experiment in Figure 4 is interesting, but it is a bit unclear what the point is and whether similar conclusion can be drawn using non-Bayesian NN, since softmax also provides uncertainty estimates for classification tasks naturally. Alternatively, it would be helpful to demonstrate the proposed method’s advantages in terms of obtaining calibrated uncertainty using metrics such as ECE or Brier Score.\n\nAs I said, this could potentially be a very interesting paper if related works are handled and compared to thoroughly, especially highly relevant ideas proposed before [1,2,3]. Otherwise it is difficult to evaluate the actual novelty and advance this paper provides.\n\nMinor:\nTypos in the caption of Figure 4.\n\n[1] Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016\n[2] Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers, ICLR 2018\n[3] Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, CVPR 2019\n[4] Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks, ICML 2015\n[5] Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks, ICML 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iz8ku_21_m", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an approach called the Laplace Bridge to approximate predictive uncertainty in Bayesian neural networks. The approach is essentially based on first a change of variable, followed by a Laplace approximation. They provided a theoretical result for this approach, which essentially shows that for \\alpha_k large enough, the variance of \\pi_k given \\alpha is increasing with the variance \\Sigma_kk.  They performed some experiments to essentially show the computational speedup of Laplace Bridge against more cumbersome MC based competitors. \n\nThe pros of this approach is obvious: it is remarkably simple, and provides a sizable speed up computationally when compared to the cumbersome existing tools for Bayesian Neural Networks, and it uses well-established tools such as the Dirichlet and Laplace approximation.  The authors did a good job explaining the motivation and the paper is written in a clear manner. The experiments appear to demonstrate that this should work well in practice. \n\nThere remains, however, several cons that I believe the authors need to address: \n\nFirst, the author should comment on how this is a *substantial* and *sufficiently novel* contribution to the field? The Laplace Bridge idea/derivations seem to come, for the most part, from Mckay and Hennig et al’s  (2012) etc prior work. The Laplace approximation idea is a common tool, and using the Dirichlet distribution for uncertainty quantification in neural networks for classification has been amply explored in other prior work (e.g. Sensoy 2018, Malinin etc). To play Devil’s advocate, it does seem like the authors are just applying to the BNN context something that we know works fast (Laplace Approximation) and comparing it to something that we know works slow (MC Sampling). \n\nSecond, the authors should try to justify with more rigor/guarantees on how good the approximation is. I do think the computational speed up in the empirical evaluations are big. However, currently, it appears that the Laplace Bridge is simply a heuristic, with minimal quantification/guarantees on the approximation error etc. For example, in Figure 2 (Right), in the very simple case of approximating a beta distribution (transformed back to the standard basis) , one sees that near 0 and 1, the approximation appears to be doing very poorly. So currently, the justification that Laplace Bridge is a good approximation comes mainly from visual plots (figure 2) and empirical evaluations. More rigor in the quantification of approximation error would be ideal. \n\nMinor comments: \nIn the first line of section 2, “...light-weight method to approximate a general probability distribution…”, I believe that q has to satisfy certain smoothness conditions, so the phrase “a general probability distribution” appears to be too general. \n\nOverall, the idea is simple and appears to lead to substantial speedups in practice/experiments, and the paper is written clearly , but authors should address issues of 1. where the substantial novelty/contribution comes from and 2. more rigorous/quantitive guarantees for approximation error. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "GOHiGJ3g5a6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review Summary\n--------------\n\nOverall this paper offers a simple idea -- the laplacian bridge, revived from earlier work by MacKay (1998) and Henning et al (2012) -- that produces useful yet affordable predictive posterior estimates. I thought the fundamental approach is sound and does seem to be an elegant way to apply some older ideas that do indeed seem to be overlooked. My chief concerns are that the presentation contains many distracting errors (e.g. as written Eq 8, the primary equation of their method, is fundamentally wrong but I think this is a typo) and many steps lack justification. I think the core idea is promising, and a strong author response might convince me that the paper should be accepted.\n\nPaper Summary\n-------------\n\nThe paper presents the idea of the \"Laplacian Bridge\", a deterministic mapping provided in Eqs. 5-7 for converting between the parameters of a K-dimensional multivariate Normal distribution and the parameters of a K-dimensional Dirichlet distribution in the *softmax* basis. The mapping is \"pseudo\"-invertible (e.g. it works from Normal to Dirichlet and from Dirichlet to Normal, but only approximately). The Dirichlet to Normal direction is derived by performing a Laplace approximation to a Dirichlet written in the softmax basis. As cited, changing the parameterization to softmax is an ideal from MacKay in the 1990s. The alternative direction (Normal to Dirichlet) in Eq. 7 is a construction due to Henning et al (2012), where given a known mean parameter \\mu (a K-dim vector) and covariance \\Sigma (a KxK matrix), we can obtain a Dirichlet concentration parameter vector \\alpha (a vector of size K). \n\nWhile this abstract construction is known, the present paper uses it specifically to approximate the posterior predictive of a multi-class Bayesian neural network. Given an approximate posterior q(w|\\mu, \\Sigma) over weights, we \n\nThe approximate posterior of pre-softmax activations z (real valued) given input features x is:\n\n    p(z | x) = \\int_w p(z | x, w) q(w | \\mu, \\Sigma) dw\n    \t\t \\approx N( z | m, S)\n\nwith expressions for mean m and covariance S given in Equation 8. Note that even after determining parameters m and S, in order to make predictions at input x we need to draw several samples of vector z, feed each through a softmax, and average the results. Drawing a single sample of vector z from a normal with covariance S usually has cost O(K^2).\n\nThis paper advocates for the following approximation of the post-softmax activations \\pi(z):\n\n\tp( \\pi(z) | x) = Dirichlet( laplace_bridge(m, S) )\n\nwhere the parameters of the dirichlet are determined by the laplace bridge construction in Eq. 7.\n\nThe contribution is that the expected value of the class probabilities \\pi under the Dirichlet has closed form, so predictions can be made without *any* sampling, just by computing the posterior mean. The total prediction cost is O(K) once m and S are determined.\n\nOne new theoretical result is presented in proposition 1, showing that under some conditions, the variance of the k-th entry of a probability vector constructed using the LB method increases with \\Sigma_kk (the relevant variance of this entry under the Normal). \n\nThe first set of results (Sec 5.1) compares the LB to baselines for using the posterior predictive to assess out-of-distribution. The Laplace Bridge tends to have better results than diagonal sampling and ties with KFAC sampling w.r.t both metrics, while being around 400 times faster than both sampling-based methods.\n\nThe second results section (Sec. 5.2) does a detailed timing comparison, indicating that computing a Hessian matrix (needed to determine the covariance \\Sigma) takes less than a minute for one backward pass using the BackPACK modification of the backprop algorithm, while the training is more like 1 hour. \n\nThe final results section (Sec. 5.3) looks at a new \"uncertainty-aware\" top-k prediction technique, that makes use of the fact that under a Dirichlet over many classes, we can always get the marginal of a single class in closed-form as a Beta.\n\nStrengths\n---------\n* Simplicity of the method\n* Speed of the method (both big-oh runtime complexity and wallclock runtime wins)\n* I liked Figure 3 showing the tradeoffs in KL approximation quality as number of MC samples increases\n* I like the idea of uncertainty-aware top-k in Sec. 5.3. Dynamically selecting the number of \"top classes\" based on model predictions is a nice idea, as is the use of Beta marginals.\n\nWeaknesses\n----------\n* Some typos so distracting to mislead non-expert reviewers about the fundamental method (see Eq 8)\n* Other presentation issues are present throughout (e.g. often refering to primary literature instead of concisely explaining the justification for an equation)\n* Should discuss connections to the logistic normal distribution\n* The uncertainty-aware top-k idea needed more description in the main paper. Hiding the core method in the supplement seems to undercut its contribution.\n* Proposition 1 seems distracting and not strictly needed to tell the main story of the paper\n\n\nMajor Presentation Concerns\n---------------------------\n\nEquations 5-7\n* Need more explanation of where Eq. 7 comes from. Is there a justification for this \"pseudo-inverse\"?\n\nEquation 8\n* This is an approximation of another integral. I think readers would benefit from seeing this other integral written out.\n* The mean parameter here is a typo, right? x is a vector of size N, but \\mu_\\theta has size P, so the inner product doesnt make sense as written. Shouldn't the mean parameter be f(x; \\mu_\\theta) ... the output of the network when weights are set to mu?\n* What is the justification for Eq 8? I guess in MacKay's original derivation, the assumption here is a local linearization of the output of the network as a function of parameters (e.g. first-order Taylor approximation), and this makes the predictive distribution a Gaussian. Though do you need to model z | w, x as a random variable here to make this work? I suppose you can say this is Gaussian with vanishing small variance. Would like to see discussion improved to explain this.\n\nTable 2 needs some clarity improvements:\n* Are these timings for performing predictions for a single image? A minibatch? Please give some context.\n\nFigure 5:\n* Why are there many top 10 predictions, but not as many top 4 or top 8? I would expect a priori that these strictly decrease.\n\n\nTechnical Concerns\n------------------\n\n## T1: The paper seems to be unaware of the logistic normal distribution\n\nE.g. in Sec. 5.2, the authors state \"the softmax applied to a Gaussian does not have an analytic form\", but should this just be the well-known logistic Normal distribution, which has a known pdf that is possible to evaluate analytically:\n\nhttps://en.wikipedia.org/wiki/Logit-normal_distribution#Multivariate_generalization\n\nPerhaps the issue is that the Logit-normal produces a K-dimensional probability vector by using K-1 dimensional means and covariances (to maintain an invertible mapping), while the present paper seems to use means/covariances of size K. Or perhaps the issue is that this logistic normal distribution does not have closed-form expressions for its mean or other moments. Either way, the discussion should be more clear.\n\nI recommend the authors explain more carefully and cite a primary reference (e.g. J. Aitchison and S. M. Shen Biometrika 1980).\n\n\n## T2: What is the point of proposition 1?\n\nI feel that proposition 1 is added just to make the paper feel more original that just smartly using ideas from previous work.\nI don't see any of the later experiments relying on this proposition. Why include it?\n\n\nExperimental Concerns\n---------------------\n\n## E1: In Sec. 5.1, should we expect LB to deliver improved classification over MC with many samples?\n\nTo me, the convincing argument here is that LB is much faster than the MC sampling. I wish the experiments focused on this.\n\nHowever, it seems that Table 1 is trying to make an additional point, that the LB approximation of the posterior predictive will yield *better* OOD classifier performance than MC sampling. To this I am skeptical. Why should we care that you get 0.001 better AUROC? Aren't you showing the same distribution, sampled in two different ways (each with its own possible error)?  To me, the right question to ask is, which is a better approximation of the given distribution, not which gets better OOD performance. I think the convincing thing would be to show the existing results together with a more accurate MC estimate from many more samples (100k?), and try to argue whether the LB results are a more accurate *approximation*. \n\n\nMinor Line-by-line concerns\n---------------------------\n\nBetween Equations 2-3:\nInstead of \"has to be multiplied by the Jacobian determinant\", should be:\n\"has to be multiplied by the absolute value of the determinant of the Jacobian\"\n\nFigure 1\n* Can you indicate the parameters used to generate each subfigure? Perhaps in the supplement if needed.\n* Could make the point masses at the corners of the simplex a bit easier to see in the middle plots.\n\nFigure 2 (left)\n* Is the Hessian of blue and black curves p.s.d.?\n* Can make clear that the black and blue dashed curves do integrate to one, just not on the interval 0-1 but over the whole real line\n\nSec. 5 first paragraph: \nYou say \"First,\" and \"Thirdly,\" but are missing a \"Secondly,\" I found it hard to parse\n\nFigure 4\n* Font size on axes could be increased", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a mechanism for fast sampling from the posterior over the weights of the last layer of neural network, by approximating the logits as a Gaussian through equation 8. This is based on earlier work by MacKay, but has some new empirical investigations. This is a very difficult case.

In its favor:
* Despite the main ideas coming from older work by MacKay, they are interesting and relevant and worth re-surfacing.
* The experiments demonstrate some improvements in OOD detection over a diagonal Laplace approximation to the last layer, and is competitive in performance with a KFAC Laplace last layer approximation but much faster at test-time.
* The authors provided early and thoughtful responses and actively tried to have a discussion with reviewers. It is a pity that the reviewers did not participate in this discussion. 

Concerns:
* While interesting, it is unclear if the proposed method actually has much practical utility in its current form. The method is presented as a fast approach for uncertainty in Bayesian deep networks. But Eq. 8 requires such significant computations to form that whatever is gained by the fast sampling may not make-up for the cost of forming the approximation itself. This computational burden is why the approach is only applied to a last layer. Table 2 and some of the surrounding discussion helps with alleviating these concerns and is most appreciated. But many basic questions persist: (i) do we really need many samples to achieve good performance, especially from a posterior over only a last layer? (we see the KL divergence decreases, but what about performance on interesting problem as a function of sample size?) (ii) in terms of total runtime-accuracy would this be competitive with using Bayesian methods over all the parameters, even if these methods are taking fewer samples? It would be easy to try. (iii) Besides OOD detection, how does this approach generally affect accuracy or calibration? (iv) How would this method compare to a basic baseline like retraining the last layer several times and ensembling? (v) could anything be done to significantly accelerate the computations in forming Eq. 8? While not all of the answers to these questions need to be favorable to the Laplace bridge for acceptance, it would certainly improve the paper to at least address most of the questions explicitly. At the end of the paper, an online setting is mentioned, which I think would be amenable to this approach --- it could be good to explore this direction.
* It is disappointing that the reviewers did not communicate with the authors, despite commendable efforts from the authors. However, the paper continued to lack a clear champion. Given the persisting lukewarm reception of reviewers,  and some of the practical concerns above, it would help to have some "stand-out" result, especially since the methodology, while interesting and relevant, is not new. That does raise some expectations for the experiments.

This is not an easy case. The paper has merits. And it's possible some of the concerns could be addressed by simply more clearly rationalizing design decisions (why would we use this approach in its current form over full Bayesian methods, which are now quite fast, with fewer posterior samples?).  At the same time it's clear the paper in its current form is not resonating with reviewers, and there are concerns about the practical applicability and limitations. It's on the borderline. Having some stand-out results could really help this paper realize its potential.
Decision: Reject