Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks
Authors: Agustinus Kristiadi
Runa Eschenhagen
Philipp Hennig
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian neural networks
predictive calibration
normalizing flows

Abstract: Monte Carlo (MC) integration is the _de facto_ method for approximating the predictive distribution of Bayesian neural networks (BNNs). But, even with many MC samples, Gaussian-based BNNs could still yield bad predictive performance due to the posterior approximation's error. Meanwhile, alternatives to MC integration are expensive. In this work, we experimentally show that the key to good MC-approximated predictive distributions is the quality of the approximate posterior itself. However, previous methods for obtaining accurate posterior approximations are expensive and non-trivial to implement. We, therefore, propose to refine Gaussian approximate posteriors with normalizing flows. When applied to last-layer BNNs, it yields a simple, cost-efficient, _post hoc_ method for improving pre-existing parametric approximations. We show that the resulting posterior approximation is competitive with even the gold-standard full-batch Hamiltonian Monte Carlo.

Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks
Agustinus Kristiadi University of Tübingen
Runa Eschenhagen University of Tübingen
agustinus boustimus.kristiddinn kristiadi@uiiii tuebingete de
runa. runa.eschenhageeee eschenhccheaaeeeeeu beuggbggeeeuuutttuuuue..e.ddd
Philipp Hennig
University of Tübingen and MPI for Intelligent Systems, Tübingen philipp
Abstract
Monte Carlo (MC) integration is the de facto method for approximating the predic- tive distribution of Bayesian neural networks (BNNs). But, even with many MC samples, Gaussian-base BNNs could still yield bad predictive performance due to the posterior approximation's error. Meanwhile, alternatives MC integration tend to be more expensive and biased. In this work, we experimentally show that the key good MC-approximated predictive distributions the quality of the approxi- mate posterior itself. However, previous methods for obtaining accurate posterior approximations expensive and non-trivial to implement. We, therefore, propose to refine Gaussian approximate posteriors with normalizing flows. When applied last-layer BNNs, it yields simple post hoc method for improving pre-existing parametric approximations. We show that the resulting posterior approximation is competitive with even the gold-standard full-batch Hamiltonian Monte Carlo.
1 Introduction
Predictive uncertainty is crucial in safety-critical systems [1]. Yet. commonly-used neural network (NN) predictive systems are overconfident [2, 3]. Approximate Bayesian inference of NNs, resulting in Bayesian neural networks (BNNs), is a principled way of mitigating this issue [4]. Indeed, even crude approximations BNNs' posteriors, such as the Laplace approximation [5], can lead to good predictive uncertainty quantification (UQ) performance [6], provided the predictive distributions are correctly computed [7].
A prediction a BNN amounts to an integration of the likelihood w.r.t. the (approximate) posterior measure. Due the non-linearity of NNs, no analytic solution the integral exists, even when the likelihood and the approximate posterior are both Gaussian. A low-cost, unbiased, stochastic approximation can be obtained via Monte Carlo (MC) integration: obtain S samples from the approximate posterior and then compute the empirical expectation of the likelihood w.r.t. these samples. While MC integration accurate for large S, because of the sheer size of modern NB),Ns, virtually all BNNs use small S (typically 10 to 30, see [8-14, etc.)). Due to its well-known error scaling of @ (S), intuitively, MC integration with a small S is inadequate for an accurate prediction-yeet yeet this has been studied in depth for BNNs. Furthermore, while linearization of an NN around point estimate in the parameter space is an alternative to MC integration [7, 15-17], it generally costly due to the computation per-example Jacobian matrices.
In this work, we study the quality of MC integration for making predictions in BNNs. We show that few-sample MC integration is inaccurate for even an "easy" integral such when the domain of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The MC Standard Error to (xx 15,10") da
Calibration on F-MNIST
0.2
0.3
0.2
LA 10000)
0.1
Error 0.08
HMC 10)
HMC
0
1000 2000
3000
4000
5000
ECE 4
NEL
Metrics
Figure 1: Left: Few-sample MC integration is inaccurate for computing classification predictive distribution: the standard error of the MC integration of the logistic- logistic-Gaussian integral is large for the commonly used (few) numbers samples. Right: But one can still obtain good predictive performance with small S if high-quality posterior approximation, e.g. HMC, used.
integration the output space of cinary-classffication BNNs, cf. Fig. (left). Further, we show that its alternative, the network linearization, disagrees with large-sample MC integration, making it unsuitable as general-purpont predictive approximation method, i.e. when the Gauss-Newton matrix is not employed.' Meanwhile, as indicated by full-batch Markov Chain Monte Carlo methods [18, 19] or even (the Bayesian interpretation of) ensemble methods [20], few-sample MC integration might still be useful for making predictions, provided that the approximate posterior measure is "close enough" to the true posterior-cf. Fig. (right). This implies that one should focus on improving the accuracy of posterior approximations.
Nevertheless, prior methods for obtaining expressive posteriors [e.g., 10, 21] require either significant modification to the NN or storing many copies of the parameters, Morcover, they require training from scratch and thus introduce significant overhead, which can be undesirable in practical applications. Therefore, we propose a post hoc method for "refining" Gaussian approximate posterior by leveraging normalizing flows [22]. Contrary the existing normalizing flow methods with a priori base distribution (e.g. (0, Z)), the proposed refinement method converges faster with shorter flows, making cheaper than naive application of normalizing flows in BNNs. When used conjunction with last-layer BNNs, which have been shown to be competitive to their all-layer counterparts [6], the proposed method is simple, cheap, yet competitive to even the gold-standard Hamiltonian Monte Carlo in terms of predictive performance.
To summarize, our contributions are as follows:
ii) We highlight the deficiencies of both few- and many-sample MC integration for computing BNNs' predictive distributions.
(ii) We argue that one must be careful when applying analytic alternatives to MC integration, such as linearization and the probit approximation since they can yield unintended effects.
(iii) We propose a widely-applicable technique for refining parametric posterior approximations by leveraging normalizing flows, which yields cost-efficient post hoc method when used in conjunction with last-layer BNNs.
(iv) We validate the method via extensive experiments and show that refined posteriors are competi- tive with the much more expensive full-batch Hamiltonian Monte Carlo.
Preliminaries
2.1 Bayesian neural networks
Let D (xi, Me He be an i.i.d. dataset, sampled from some distributions p(x) on I2 and p(y () on IRc A Bayesian neural network (BNN) is a neural network (NN) fe : IR" + Re with a random parameter p(0) on Rd and likelihood p(y fo(x)), along with the associated posterior p(0) D) x p()) [_= p(ys fo(z))). However, the exact posterior p(O D) is intractable in general
'The generalized
matrix is the exact Hessian matrix of linearized network.

---

due to its normalization constant. Approximate BNNs, which approximate p(0 D) with only its samples or with simpler parametric distributions, must thus be employed in most cases.
2.1.1 Posterior approximations
The Laplace approximation [LA, 5] is one of the simplest approximation methods for BNNs. The main idea to construct a local Gaussian approximation to p(0 D), centered at a maximum posteriori (MAP) estimate OMAP == arg max, p(0 D), with the covariance matrix given by the negative inverse-Hessian (-V p( DD) ( Since the MAP estimation is the standard training procedure for NNs, the LA can be efficiently applied on top of pre-trained NNs [6], especially due to recent efficient second-order optimization libraries (23, 24].
Variational Bayes [VB, 25] assumes a family of simple approximate distributions over the parameter space, e.g. M := (g(0)= == N/ μr, E): (1,, Σ) E ]Rd+d"), and finds q E M that is the closest to p(0 D) by minimizing the reverse Kullback-Leibler (KL) divergence DKL (((),, p(0 (D) over M. Unlike the LA, VB is not constrained to capture just the local mode of the leg-posterior landscape. However, VB is not post hoc and is generally not as straightforward to implement.
Unlike the previous two approximations, Markov Chain Monte Carlo (MCMC) methods do not obtain parametric upproximations to the posterior. Instead, they collect samples from asymptotically the true posterior and use them to approximate integrals under the posterior measure. A popular MCMC method the Hamiltonian Monte Carlo (HMC) method [18, 19], where sampling processes are casted as Hamiltonian dynamics. Nevertheless, MCMC methods are generally expensive since they require full batches of data and storage of many copies of the networks' parameters.
2.1.2 Predictive approximations
Let g(0) be an approximate posterior of NN fo under D. Given a test point 2., the prediction y. is distributed as p(y. x., D) JR. p(y. fo( )) ((0) d0. However, even when both p(y. fa()) and (0) are Gaussians, this integral does not have an analytic solution due the nonlinearity of fa- Further approximations on top of the posterior approximation are thus necessary.
Monte Carlo integration The Monte Carlo (MC) integration is simple technique approximate integrals under probability measures. Specifically, in our case,
with 02 g()) Vs = I....,S. S, p(u=1 Ex, D) = S Σ p(u+ (x.));
(1)
where S is the number of samples from ((f). It is casy to show that MC integration is unbiased with error given by the standard error around the mean that scales like ((1/VS), see e.g. Murphy [26, Section 2.7.3]. That is, MC integration (slowly) becomes more accurate as the number of samples increases. In the realm of BNNs, however, S is often chosen to be small number, e.g. S = 20, due to the computational cost of evaluating Jo.
Analytic approximations Since Gaussian N(A) μ., Σ) is the de facto choice for (0), an analytic
approximation of the marginal output distribution' p(f(x) x, D) can be obtained by linearizing fo around Specifically, we perform a first-order Taylor expansion fo(x) R fu((x) + Jf(x) (0 1)) where (x) E Rgx is the Jacobian matrix of the network output w.r.t. 0 at and denotes matrix multiplication. Under this scheme, denoting f. = f(x.), we thus have for each test point x.
p(f. w.. D) R= Wf. (z.),, (z.) (T.) =: N(f
One can then use this distribution obtain the predictive distribution p(y- x,, D) v S ply I f(f D) df., which again can be approximated via MC integration.
Nevertheless, there exist analytic approximations to this integral [27, 15, 28, 29]. Specifically for binary classification where ply f-) = (f.) for the logistic function one can use the so-called probit approximation [27, 15]: pạy 1 x., D. (ff(x.)) 7/8 S/x))))) Moreover, for
2 Where 0 has been marginalized out.
Here, S(x.) E Rso since fxx) is real-valued random variable.
3

---

MC 10000
30
0.20
0.018
0.018
0.185
0.15
20
0.017
0.10
10
0.00
-50
100
-100
-100
-50
m
773
771
Figure 2: The (absolute) errors of the MC integration and the probit approximation for computing I [m, 8) across different values of m and s. A trapezoid method with 20000 evaluation points is used as gold-standard baseline. Red dots indicate the maximum enrors. White indicates (near) zero error.
the multi-class case, we can use the multi--class probit approximation [MPA, 28]:
ply xx, D) x softmax (fu(==// f/8 diag(S(zz))
(2)
where the vector division is taken component-wise.
2.2 Normalizing flows for Bayesian inference
Let F Rd + Rd be diffeomorphism" in the sense of C** (R2) for fixed k 2 N. If
p is density on ied then the density of the random variable 0 F()) is given by p(0)
p(F (Ø)) det --) where Jr-1 = JF is the Jacobian matrix of F-1 A normalizing flow (NF) is method exploiting this relation [22, 30, 31]. Specifically, is way of constructing a sophisticated diffeomorphism F, by composing several simple parametric ones. The resulting diffeomorphism thus transforms simple density into complicated one in tractable manner.
Let Fs. Rd + Idd be a diffeomorphism parametrized by Φi E IR* with a known inverse F=1 and known d Jacobian matrix JF,, for / 1, I,...,f. Writing F6 Fe Fsz and á = (di)f= then the change-of dendsstt formula becomes
Ps ( = FF (0))
II det JF (0)
Given target posterior density p", the goal of normalizing flow is then to estimate s.t. Po is close to p" In Bayesian inference, the reverse KL--divergence DKL(Ps.P7 is often used.
3 Pitfalls of BNNs' Approximate Predictive Distributions
In this section, we study the failure modes BNNs, especially last-layer ones, for predictive uncertainty calibration. We shall show that there
parts contributing the inaccuracy in the predictive distribution of BNN: (i) the weight-space approximation and (ii) the integration method do Bayesian model averaging. We shall observe that while the accuracy the integration method is impactful (i.e., the number of samples in MC integration), it is less important than the quality of the posterior approximation itself. That is, higher-quality weight-space approximations even in last-layer BNNs BNNs-yield yield samples that are more efficient: Few-sample MC integration can already yield good predictive performance in this case.
3.1 Accurate MC integration requires many samples
We begin analysis with the simplest yet practically relevant case. Let x, R and f. : f(.)) with p(f. C. D) = N(= m, 82) for some m E R and s2 E RRy. We are interested in computing
A smooth function with smooth inverse.
All densities considered in this paper assumed be the Lebesgue measure.

---

the integral (cf. Fig. 3)
I(m, 8) == p xx, D) (ff) m, mm,²)df. df..
Note that this integral is prevalent in practice, e.g. in linearized or last-layer classification BNNs [4, 7, 17, 32].
We compare MC integration with S = 100 against the probit approximation, using a trapezoid quadrature with 20000 evaluation points to represent a gold-standard base-
0.8
line. That is, we compute the discrepancy {m(m,)) 0.4
(mm, +]
a(f-)
(. s)] where 7(m, s) is [m,,s) s) computed either with MC integration or the probit approximation, and I. (m,, s) obtained via the trapezoid method.
0.2
W/J. (m.ss))
-10
-5
5
10
The results are in Fig. 2: Even with S = 100 samples- Figure 3: The intuition of I [(m, s).
larger than the usual = 10-30-MC integration inaccurate. Its error can be as high as 0. 18, which is substantial considering / (m, s) E [o, 1]. As S increases beyond 10000, MC integration improves and eventually overtakes the probit approximation, as to be expected given its theoretical guarantees. This highlights the flaw of few-sample MC integration-accureee MC integration generally requires large number of samples.
3.2 sampl MC integration not sufficient
However, even with large S, MC integration can still fail to yield good predictive performance in BNNs. This can happen when the approximation q()) used in (1) is in- accurate approximation of p(0 virtuallyy the case for every parametric BNN. Thankfully, there is some evidence that the error of MC integration might be relatively small in comparison the error generated by crude posterior approximations. As an extreme example, Deep Ensemble [20] and its variants, even though they perform MC inte- gration with a small number of samples (usually S = 5), generally yield better approximations to the predictive dis- tributions. This is perhaps due to their multimodality, i.e.
Table 1: The expected calibration er- ror (ECE) and negative liglliiel (NLL) of LA (S = 10000) and HMC
(S= = (100
Methods
ECE 1
NLL 4
F-MNIST
LA
10,5
0.311
HMC
3.4
0.275:+00044
CIFAR-10
LA
4.9.10.2
0.161+:0.001
HMC
4.2+:0.2
0.158+0.001
due to their finer-graines posterior approximations.
To show this more concretely, consider the following experiment. We take Fashion-MNIST (F- MNIST) pre-trained LeNet and CIFAR-10 10 pre- pre-trained 1id For each case, perform a last-layer Laplace approximation with the exact Hessian and a full-batch NUTS-HMC [19], under the same prior and likelihood. We find in Table 1 that HMC, even with few samples, yields better- calibrated predictive distribution than the LA with three orders of magnitude more samples. This finding validates the bidely-believed wisdoms [9, 10, 14, 32, etc.] that highly accurate posterior approximations are most important for BNNs. Furthermore, this also shows that with a fine-grained posterior approximation, the predictive performance of a BNN is less sensitive the number of MC samples, leading to better test-time efficiency.
3.3 Analytic alternatives to MC integration are not the definitive answers
Of course, fine-grained posterior approximations are expensive. So, natural question is whether one can keep a cheap but crude posterior approximation by replacing MC integration. Network linearization seems to be prime candidate to replace MC integration in the case of Gaussian approx- imations [7, 15]. But it poses several problems: First, it requires relatively expensive computation of the per-example Jacobian Jz(2,), where for each test point I., one must store the associated e X d matrix, where d could easily be tens of millions (e.g. in ResNets) and could be in the order of thousands (e.g. ImageNet). Second, while Immer et al. [7] argued that network linearization the correct way to make predictions in applicable. We show this in Fig. 4: Everything else being equal, MC integration (S 10000) and lincarization yield different results especially in terms of predictive uncertainty. Since should
Gaussian approximations, is not generally
5

---

2
0.6
8
MC
0.4
0
4
0.2
0
(a) Predictive distributions.
(b) Densities of pred. means.
(c) Densities of pred. std. dev.
Figure 4: Predictive distributions, computed via MC integration (S 10000) and network lineariza- tion, of BNN with weight-space Gaussian approximate posterior (an all-layer LA with the exact Hessian on a two-layer NN under a toy regression dataset).
prefer MC integration in this many-sample regime, linearization is thus not accurate for the general cases.
Moreover, we show that the multi-class probit approxi- mation (MPA), which often used on top of a linearized output distribution p(f(x.) Zx. D) to obtain the pre- dictive distribution p(y, Txx D) [6, 17, 32), can also obscure the predictive performance, albeit often for the better [6, 32). To that end, we use the same experiment setting Section 3.2 and Table 1, and compare the MPA against MC integration. The results are in Table 2.
Table Calibration of MC integration
S 10000 and the multi-class probit approximation.
Methods
ECE L
NLL
F-MNIST
MPA
3.3+
0.281+t0.002
MC
10.5+00.4
0.311+00000
Indeed, we that the MPA yields better-calibrated pre- dictive distributions. However, this result should be taken with grain of salt: The MPA ignores the off-diagonal ele- ments of the covariance of p(f(z, 2x, D). as shown in (2)-furhher further detail is in Appendix A. That is, the MPA "biases" the predictive distribution ply Ix, D) since it generally assumes that f- has lower uncertainty than it actually has. And since p(f- Tv, D) is usually induced by the LA VB which are often underconfident the bias MPA towards overconfidence thus counterbalance aances the underconfidence of p(f= F., D). Therefore, in this case, the MPA yield better-calibrated predictive distributions than MC inte- gration [32, 6]. Nevertheless, it can also fail even simple cases such as in Fig. 5,6 where the MPA yields underconfident predictions even near the training data (notice the lighter shade in Fig. bottom). Moreover, the structured nature of the error of the MPA (cf. Fig. 2 for the binary case) might also contribute the where the MPA differs from MC integration. Both examples above thus highlights the need for careful consideration when analytic alternatives to MC integration are employed.
CIFAR-10
MPA
3.8+0.1
0.161+0.001
MC
4.9+0.22
0.161+0.001
large networks [7, 33, 34],
Refining Gaussian Approximate Posteriors
The previous analysis indicates the importance accurate approx- imate posteriors q(f) for BNNs' predictive distributions. However, the current de facto way of obtaining accurate posterior approxi- mations, HMC [18, 19], are very expensive since they require a full-batch of data in their updates [35]. While mini-batch versions of HMC exist, they do seem to yield as good of results their full-batch counterparts. Indeed, Daxberger et al. [6] showed that a well-tuned last-laye last-layer LA can outperform state-of state-of-the-ar all-layer stochastic- graadient MCMC method [21]. Furthermore, these sample- based methods- both the full- and mini-batch versions, along with deep ensembles and their variants-aae are costly in terms of storage since one effectively must store copies of the network's parameters. We, therefore, propose simple post hoc technique for "refining"
0.75
0.75
0.5
Figure 5: Confidence estimate MC integration (top) and the MPA (bottom).
An all all-r full-Hessian LA thyee-layer tanh network used.

---

Gaussian approximations using normalizing flows. The resulting method is thus analytic yet can produce high-quality samples.
Let So be a NN equipped with a Gaussian approximate posterior ((0) = (N μ. Σ) (e.g., via a LA, VB, SWAG [13]) the parameter space under dataset DD Given NF Fg of length with parameter 6, we obtain the refined posterior by
To q( (0)) ddet JF, (0)
Then, refinement of g(0) amounts to minimizing the reverse KI KL-divergence to the true posterior, using the evidence lower bound (ELBO) as proxy (HH below the entropy functional):
arg max
E 8~78
[ogr((D) (a] + log log
[o
(4)
Given a refined posterior Os- (0) and a test point x", we can obtain the predictive distribution via MC integration:
p(y" x² D)
1 S
Ex v (1
where 0, = Fo. (0a): 0, 0()) =-=
L...,..., S.
Due to the expressiveness of NEs [36], we can expect based on the previous analysis that a large S is not necessary here obtain good predictive performance. We shall validate this in Section 6. Last but not least, this refinement technique is especially useful for last-layer BNNs since their parameter spaces typically have manageable dimensions, cf. Section 6.4.
5 Related Work
Normalizing flows have previously been used for approximate Bayesian inference in BNNs. An obvious way do so, based on the flexibility of NFs in approximating any density [36], would be to apply NF on top of the standard normal distribution N(@ 0, I) to approximate p(0 DD, see e.g. Izmailov et al. [38] and the default implementation of variational approximation with NF in Pyro [39]. We show in Section 6.3 that the subtle difference that we make-using
approximate posterior instead of an a priori distribution- more cost-effective. a more sophisticated model, Louizos and Welling [10] combine VB with NF by assuming compound distribution on each NN's weight matrix and using the NF obtain an expressive mixing distribution. However, their method requires training both the BNN and the NF jointly from scratch. In adjacent field, Maroñas et al. [40] use NFs to transform Gaussian process priors.
Posterior refinement in approximate Bayesian inference has recently been studied. Immer et al. [7] proposes to refine the LA using a Gaussian- based VB and Gaussian processes. However, this implies that they still assume Gaussian posterior. To obtain non- Gaussian posterior, Miller et al. [41] form a mixture-of of-Gaussians approximation by iteratively adding component distributions. But, every iteration, their methods require a full ELBO optimization, making it costly for BNNs. A lower-cost. LA-based alternative to their work has also been proposed by Eschenhagen et al. [32]. These methods have high storage costs since they must store many high-dimensional Gaussians. By contrast, our method only does an ELBO optimization once and only requires the storage of the base Gaussian and the parameters of NF. In similar vein, Havasi et al. [42] perform iterative ELBO optimization for refining a sample of a variational approximation. But, this inner-loop optimization needs to be conducted each time sample is drawn, e.g., during MC integration at test time. Our method, on the other hand, only requires sampling from a Gaussian and evaluations of a NE, cf. (4).
While our results also reaffirm the widely held belief that single-mode mode Gaussian approximations are inferior to more fine-grained counterparts, our conclusion differs slightly from that of Wilson and Izmailov [43]: They argue "[...] Ultimately, the goal is to accurately compute the predictive distri- bution, rather than find a generally accurate representation of the posterior. propose multi-modal approximation to the true posterior. Here, we show that finding an accurate -oon-Gaussian) weight-space posterior approximation can still be a worthy goal, even when only utilizing single mode of the true posterior.
[...]" and subsequently
E.g.. WideResNets" last- features' dimensionality typically range from 256 2048 [37].
7

---

(a) LA (0.040)
(b) VB (0.016)
(c) LA+Refine (0.002)
(d) HMC
Figure 6: Comparison of approximate posterior densities, visualized via kernel density estimation, on a 2D logistic regression problem. Black contour: The exact posterior contour, up to normalizing constant. Colored contour: The kernel density estimate obtained from the posterior samples of each method. Number: The MMD distance to HMC's samples; lower is better.
Experiments
Code available at: https: htpps://githhb.... http/////////tttto////////////// http://ww...///aaaaa/a/aaaaeaaaa See Appendix and Appendix C for additional details.
6.1 Setups
Datasets We validate our method using standard classification datasets: Fashion-MNIST (F- MNIST), CIFAR-10, and CIFAR-100." For the out-of-distribution (OOD) detection task, we use three standard OOD test sets for each in-distribution dataset, Appendix B.2 for the full list. Finally, for the toy logistic regression experiment, a dataset of size 50 is generated by sampling from bivariate, bimodal Gaussian.
Network architectures For the F-MNIST experiments, we use the LeNet-5 architecture [44]. Mcanwhile, for the CIFAR experiments, use the WideResNet architecture with depth of 16 and widen factor of 4 [WRN-16-4, 37]. For the NF, we use the radial flow [22] which is among the simplest and cheapest non-trivial NF architectures.
Baselines We focus on last-layer Bayesian methods to validate the refinement technique." We the LA obtain the base distribution for refinement method following recent practice [6], we tune the prior precision via post hoc marginal likelihood maximization. The No-U-Turn- Sampler Hamiltonian Monte Carlo [HMC, 18, 19] with 600 samples is used baseline-aas all HMC results presented in this paper are well-converged in term of the Gelman- Rubin diagnostic [45], see Appendix B. Furthermore, we compare
gold-standard
method against recent, all-layer BNN baselines: variational Bayes with the Flipout estimator [VB, 46) and cyclical arochastic-gradiin HMC [CSGHMC, 21]. For all methods, distribution, except for HMC and CSGHMC where we use S 600 and S = 12, respectively. We use prior precisions of 510 and 40 for the last-layer F-MNIST and CIFAR experiments, respectively. These prior precisions are obtained via grid search on the respective HMC baseline, maximizing validation log likelihood. Additionally, we use MAP and its temperature-scal scaled version (MAP-Temp) as non- Bayessan baselines. More implementation details are in Appendix B.
MC integration with 20 samples obtain the predictive
Metrics We use standard metrics measure both calibration and deD-detection performance. For the former, we use the negative log-like ligllikll (NLL) and the expected calibration error (ECE, 47). For the latter, we employ the false-positive rate 95% true-positive rate (FPR95). Finally, to measure the closeness between an approximate posterior to the true posterior, we use the maximum-mean discrepancy [MMD, 48] distance between the said approximation' 's samples the HMC samples, i.e.
we use HMC as proxy to the true posterior.
6.2 Toy example
We visualize different approximations to the posterior of the toy logistic regression problem in Fig. 6. Note that the true posterior density is non-symmetric and non-Gaussian.
Additionally, see Appendix C results on text classifications. Results for all- all-rayer network are in Table 6 (Appendix C).

---

Table 3: In-distribution calibration performance. All-layer baselines marked with asterisk.
F-MNIST
CIFAR-10
CIEAR-100
Methods
MMD І
NLL
MMD
NLL
MMD
NLL
MAP
1.093+00000
0.8116+0.0049
0.438+0.001
0.1698+000009
0.416+00.001
0.9365=0.0063
MAP-Temp
1.09310.003
0.2694::0.0025
0.43810.0001
0.15452000007
0.41611.000
0.9155=0.0047
VB"
0.2673:10.0016
0,2823:10.0020
1,2638 0.0059
CSGHMM
0.2854110.0018
0.2101110.0033
0.9892_000000
LA
0.418+0.002
0.3076++0.004
0.209
0.1672++0.000
0.063++0.000
6.9865=00005
LA-Refine-1
0.356:+00004
0,2752 000311
0.346=0.000
0.1616::0.0000
0.063++0000
0.9548=0.0062
LA-Refine-5
0.022.10.002
0,2699 10.0028
0.29010.000
0,1582_10.000 0.000
0.018.10.000 0.0000
0.9073
LA-Refine-10
0.013:10.002
0.2701
0.130=10.001
0.1577::0.00
0.019:10.000
0.9037=0.0058
LA-Refine-30
0.012:+0.02
0.2001
0.002+:0.000
0.0008
0.020++0.000
0.9035=000055
HMC
0.000:00.000
0.2699=0.0028
0.000=00000
0.1581=0.00088
0.000=00.000
0.8849=0.0047
The LA matches the weight-space posterior mean, but is inaccurate the further away from it. It even assigns probability mass on what are supposed be low-density regions. While VB yields a more accurate result than the LA, it still assigns some probability mass on low-density regions due to the symmetry of the Gaussian approximation. Furthermore, it is unable to match the posterior's mode well. The proposed refinement method, on the other hand, is able to make the LA more accurate it yields skewed, non-Gaussian approximation, similar to HMC.
We further quantify the previous observation using the MMD distance between each approximation 's samples and HMC's samples. The LA, as expected, obtain the worst weight-space MMD. While VB is better than the LA with an MMD, the refined LAs achieve even better MMDs. This quantifies the previous visual observation.
6.3 Image classification
We present the calibration results in Table 3 using the LA and HMC as basclines, which represent two "extremes" in the continuum of posterior approximations. As has pre- viously shown by c.g. Guo al. [49], the vanilla MAP approximation yields uncalibrated, low-quality predictive distributions in terms of NLL and ECE. While the LA is cheap way to improve MAP predictive, its predictive performance is still lagging behind HMC. By refining it with NE, the LA becomes even better and closer to the HMC predictive. We also note that one does not need complicated or long (thus expensive) NF to achieve these improvements. Furthermore, we observe a positive correlation between sured via MMD) and predictive quality. Considering that
Table 4: OOD detection in terms of FPR95 (in percent, lower better), av- eraged over three test sets and five seeds. All-layer baselines are usterisk-marker
Methods
CIFAR-10
CIFAR-100
VB* CSGHMC"
62.9=2.00
80.8+1.0
58.7#1.6
19.3+1.0
LA
49.2+22.4
79.6:1:.00
47.7+-2.1
77.3+-0.7
LA- 5
46.8:12.2
77.810.7
LA-Refine-1 10
46.2:22.3
77.81007
approxiitia quality (mea-
LA- LA-Refine-30 30
46.1:22.3
77.910.8
20 is used, this validates hypothesis that can "get away" with fewer MC samples when accurate weight-space posterior approximations are employed.
HMC
46.0+t2.3
77.8+60.9
Moreover, we present diitrii-iittiiitionn (OOD) data de- tection in Table 4. We observe that while the last-layer LA bascline can already be better than all-layer basclines as also observed by Daxberger et al. [6] [6]-refining it can yield better results: Even with a small NF, e.g., = 5, the OOD detection performance of the refined LA is close to that of the gold-standart HMC.
1.5
NLL-
LA
150
Time (6)
100
50
FANIST
CHFR--O
Finally, we show that is indeed desirable to do refinement, i.e. using an approximate posterior as the base distribution of the NF, instead of starting from scratch, i.e. starting from a data-independent distribution such as (0, 7). As shown in Fig. 7, starting from approximate posterior
10
15
20
Flow Length
Figure 7: Calibration and wall-clock re- finement time flow length.

---

yields better predictive distributions faster than when (0, I) is used as the base distribution of the NF. This is particularly important since the computational cost of NF depends on its length: We see a 53% increase in training time from & = 5 to & = 10-the latter is required for the priori NE approximation to yield similar predictive performance to the refined posterior.
6.4 Costs
The proposed refinement technique is post hoc and cheap when applied to last-layer BNNs. Suppose one already has last-layer Gaussian approximate posterior. Using standard consumer GPU (Nvidia RTX 2080Ti), each epoch length-5 NF's optimization takes around 3.4 seconds. From our experiments, we found that low number epochs (we use 20) is already sufficient for improving a crude approximate posterior. Thus, the entire refinement process is quick, especially when compared MAP estimation, ELBO optimization, or HMC sampling. Additional details in Appendix C.3.
Concluding Remarks
7.1 Limitations
NF is a composition diffeomorphisms is bijective and invertible. Since we use the NF to transform the density on the parameter space of NN, which has tens of millions of dimensions, the proposed refinement framework is thus costly for all-layer BNNs or when the number of last-layer parameters is prohibitively large. Interesting future works to alleviate this limitation are to refine posterior in the subspace the parameter space [38] or in the "pruned" parameter space [50].
Moreover, by refining a Gaussian with NF, one loses the convenient properties associated with Gaussians, making the resulting BNN unsuitable for cheaply addressing tasks such as continual learning [51] and model selection [52]. Nevertheless, this is not unique our method-any sample- based method such as MCMC and ensembles are unsuitable for such tasks.
7.2 Conclusion
We have shown that MC integration tends to produce underperforming predictive distributions under crude posterior approximation, even with a large number of samples. While recently analytic approximations such as network linearization and the probit approximation have been extensively used BNNs, we show that one must be careful with them since they are biased and do not have theoretical guarantees like MC integration. Inspired by the high predictive performance of full-batch HMC- when a low number of samples are used-we confirm the widely-held belief that a finer-grained posterior approximation is a key to achieving better predictive performance in BNNs. To that end, we proposed a post hoc method for last-layer parametric BNNs, based on normalizing flows, that can cheaply refine crude posterior approximations. In conjunction with the post hoc last-layer Laplace approximation, the proposed method can thus give practitioners similar predictive performance to full-batch, expensive HMC in cost-efficient manner.
Acknowledgments and Disclosure of Funding
The authors gratefully acknowledge financial support by the European Research Council through ERC StG Action 757275 / PANAMA; the DFG Cluster of Excellence "Machine Learning- New Perspectives for Science",, EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039A); and funds from the Ministry of Science, Research and Arts of the State of Baden- Moden-Württemlee Württemberg The authors are also grateful Felix Dangel for feedback. AK is also grateful the International Max Planck Research School for Intelligent Systems (IMPRS-IS IS) for support.
10

---

References
[1] Alexander A Alemi, Ian Fischer, and Joshua V Dillon. Uncertainty in the variational information bottleneck. In UAL, 2018.
[2] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
[3] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why ReLU networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In CVPR. 2019.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, even just a bit, fixes overconfi- dence ReLU networks. ICML, 2020.
[5] David JC MacKay. A practical Bayesian framework backpropagation networks. Neural Computation, 4(3). 1992.
[6] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless Bayesian deep learning. In NeurlPS, 2021.
[7] Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local linearization. In AISTATS, 2021.
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. JCML, 2015.
[9] Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In ICMI, 2016.
[10] Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In ICML, 2017.
[11] Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable Laplace approximation for neural networks. In ICI.R 2018.
[12] Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard Turner, and Rio Yokota. Practical deep learning with Bayesian principles. In NeurlPS, 2019.
[13] Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry Vetrow, and Andrew Gordon Wilson, A simple baseline for Bayesian uncertainty in deep learning. In NeurlPS, 2019.
[14] Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In
ICML, 2020.
[15] David JC MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5), 1992.
[16] Andrew YK Foong, Yingzhen Li, José Miguel Hernández-Lobane and Richard Turner. In-between uncertainty in Bayesian neural networks. arXiv, 2019.
[17] Agustimus Kristiadi, Matthias Hein, and Philipp Hennig. An infinite-feature extension for Bayesian ReLU nets that fixes their asymptotic overconfidence. In NewrIPS, 2021.
[18] Radford M Neal et al. MCMC using Hamiltonian dynamics. Hondbook of Markov Chain Monte Carlo, 2 (11), 2011.
[19] Matthew D Hoffman, Andrew Gelman, et al. The No-U-Tum sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. JMLR, 15(1), 2014.
[20] Balaji Lakshminnaryaaan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
[21] Rugi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for Bayesian deep learning. In ICLR, 2020.
[22] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.

---

[23] Felix Dangel. Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into backprop. In ICLR, 2020.
[24] Kazuki Osawa. ASDL: Automatic order differentiation (for Fisher, gradient covariance, Hessian, Jacobian, and kernel) library. httpps///githb: com www.ukkosswaaasaaaaaae. 2021.
[25] Geoffrey Hinton Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In COLT, 1993.
[26] Kevin P. Murphy, Machine Learning: A Probabilistic Perspective. The MIT Press, 2012.
[27] David J Spiegelhalter and Steffen Lauritzen. Sequential updating of conditional probabilities on directed graphical structures. Networks, 20(5), 1990.
[28] Mark Gibbs. Bayesian Gaussian Processes for Regression and Classification. PhD thesis, University of Cambridge, 1998.
[29] Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for classification with Bayesian deep networks. arXiv preprint arXiv:2003.01227, 2020.
[30] Oren Rippel and Ryan Preseott Adams. High-dimensional probability estimation with deep density models. arXiv preprint 130.:5332..122 2013.
[31] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation.
In ICLR Workshop, 2015.
[32] Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of Laplace approximations for improved post-hoe uncertainty in deep learning. In NeurlPS Workshop of Bayesian Deep Learning, 2021.
[33] Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. ICLR, 2019.
[34] Sanae Lotfi, Pavel Izmailov, Gregory Benton, Micah Goldblum, and Andrew Gordon Wilson. Bayesian model selection, the marginal likelihood, and generalization. arXiv preprint arXiv:2202. 11678, 2022.
[35] Pavel Izmailov, Sharad Vikram, Matthew D Hoffmann, and Andrew Gordon Wilson. What are Bayesian neural network posteriors really like? In ICML, 2021.
[36] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. JMLR, 22(57), 2021.
[37] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[38] Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Subspace inference for Bayesian deep learning. In UAI, 2019.
[39] Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D Goodman. Pyro: Deep universal probabilistic programming. JMLR, 20(1), 2019.
[40] Juan Maroñas, Oliver Hamelijnck, Jeremias Knoblauch, and Theodoros Damoulas. Transforming Gaussian processes with normalizing flows. In AISTATS, 2021.
[41] Andrew C Miller, Nicholas Foti, and Ryan P Adams. Variational boosting: Iteratively refining posterior approximations. In ICML, 2017.
[42] Marton Havasi, Jasper Snoek, Dustin Tran, Jonathan Gordon, and José Miguel Hemúánezz-Lobato, Refining the variational posterior through iterative optimization. Entrupy, 23(1475), 2021.
[43] Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In NeurlPS, 2020.
[44] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 I), 1998.
[45] Andrew Gelman and Donald B Rubin. Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 1992.
12

---

[46] Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo-independent weight perturbations mini-batches. In ICLR, 2018.
[47] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht, Obtaining well calibrated probabilities using Bayesian binning. In AAAI, 2015.
[48] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. JMLR, 13(1), 2012.
[49]
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
[50] Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antorán, and José Miguel Hernández-Lobato. Bayesian deep learning via subnetwork inference. In ICML., 2021.
[51] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured Laplace approximations for overcoming catastrophic forgetting. In NIPS, 2018
[52] Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scalable marginal likelihood estimation for model selection in deep learning. In ICMI,, 2021.
[53] Convolution of Gaussians and the Probit integral. https://agustinng www.seii.isiittieeeeieeeeeeeeeee httpa////eeeeebbbg//222//// 06/25/ 2/5//oo-orobb////// Accessed: 2022-09-26.
[54] Zhiyun La, Eugene le, and Fei Sha. Mean-field approximation to Gaussian-softmas integral with applica- tion uncertainty estimation. arXiv preprint arXiv:2006.07584 2020.
[55] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
[56] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
[57] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[58] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In NewrlPS, 2019.
[59] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
[60] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig, Being bit frequentist improves Bayesian neural networks. In AISTATS, 2022.
[61] Kyunghyun Cho, Bart Van Merriënboer. Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Holger Sehwenk, and Yoshua Bengio. Learning phrase reprosentations using RNN encoder-decoder for statistical machine translation. EMNLP, 2014.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 7.1.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] Appendix B.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Section 6.4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "xFkngTNg2od", "reviewer": "Reviewer_5mzJ", "summary": "The paper considers the problem of posterior approximation in Bayesian neural networks. First, the authors show that the accurate approximation with Monte Carlo requires a large number of samples, far exceeding the number of samples typically used by practitioners. Next, the authors show that (1) a large number of samples from a poor posterior approximation and (2) crude analytic approximations to the MC integral both lead to suboptimal performance in Bayesian neural networks. Finally, the authors suggest to use a variational refinement of the Laplace approximation with a normalizing flow to approximate the posterior in last layer BNNs, and show good performance. ", "questions": "**Question 1**. Where do you expect the proposed method to be most useful? It appears that the primary strength is improving the uncertainty calibration for deep neural networks. How does it compare to specialized calibration methods, e.g. temperature scaling? \n\n**Question 2**. How much is the proposed method affected by the size of the last layer, and the number of classes in particular? Is it cheap / computationally feasible to apply it to an ImageNet-scale model? ", "limitations": "No issues", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength 1**. The authors show many interesting observations, e.g. Figure 3 suggesting that 100 samples is insufficient for an MC approximation to even a very simple model, and Figure 4 showing the discrepancy between the MC approximation and analytic approximation to the posterior predictive.\n\n**Strength 2**. The proposed approach is practical, as it is a cheap post-processing step to a pre-trained neural network. The empirical results are promising. \n\n**Strength 3**. It is nice that the authors included MMD distance estimates in Table 3 and showed that the refined posterior is closer to the HMC posterior. \n\n**Weakness 1**. The narrative of the paper is not very focused. In particular, it is not obvious, why the paper discusses the number of samples required for an MC approximation and the quality of analytic linearization-based approximations. The proposed method doesn't follow logically from these experiments, and the connection between these two parts of the paper is relatively weak.\n\n**Weakness 2**. The empirical evaluation is not very strong. In terms of in-distribution performance, the authors only compare the proposed method to last-layer baselines, and achieve an improvement in uncertainty / NLL, but not accuracy (according to appendix Table 5). The results on out-of-distribution detection (appendix Table 7) show improvement over full parameter-space Bayesian methods, but do not include specialized out-of-distribution detection methods.\n\nOverall, the paper shows interesting observations, and some promising results. However, the story feels disconnected, and the empirical observation doesn't prove that the proposed method will be useful for practitioners.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dFQ6I689GQw", "reviewer": "Reviewer_4ug5", "summary": "First, the authors of this paper propose an experimental analysis of failure modes of a given way of building Bayesian Neural Networks (Bayesian NNs, BNNs), namely, MC integration over samples from an approximate posterior. \nThen, they propose a cheap and accurate method, which avoids these failure modes: replace the last layer of a NN by a variational layer, whose distribution is generated by a trained Normalizing Flow (which spans a much larger space of distributions than the commonly used product of Gaussians), then perform MC integration on top of that. They compare experimentally their method to the \"gold standard\" HMC.\n\n## Failure modes of MC methods\n\nThe authors recall (and prove experimentally) that MC methods do not lead to good predictive performances if the underlying approximation of the Bayesian posterior is too rough. For instance: Laplace Approximation (LA) and Variational Bayes (VB) ; notably, these approximations of the posterior assume Gaussian and independent parameters.\n\nOne of the pitfalls is also the lack of exploration of the parameter space: deep ensembles perform way better, even with very few samples.\n\n## Last-layer Bayesian NNs\n\nThe authors emphasize the idea of building \"Bayesian\" NNs only by replacing the usual last layer by a *Bayesian* last layer. Thus, the proposed methods relies heavily on \"Laplace redux–effortless Bayesian deep learning\" (Daxberger et al., 2021) [6]. Actually, the authors propose an improvement of [6]: instead of performing a simple LA on the last layer, the authors add Normalizing Flow (NF) to it, in order to increase the variety of reachable distributions.", "questions": "The phenomenon observed in [6] and by the authors of the current paper, i.e., the importance of the last layer when doing approximate Bayesian inference, is not understood:\n * According to the authors, which set of experiments could help us to understand it? \n * Is this a way for the NN to calibrate itself automatically, which would be impossible if the distribution of the final layer weights was constrained to Gaussians? \n * How would the NN react to other constrained distributions (Laplace, Weibull, etc.)? Is this related to the thickness of the tails?\n\nThe theoretical questions brought by the experiments done by the authors are a strength of the paper. These points should probably be discussed somewhere in the paper, with possibly some preliminary experiments.", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe authors motivate their approach by an analysis of failure of common methods. So, the proposed method is justified, at least experimentally.\n\n## Weaknesses\n\nThe main weakness of this paper is its significance, compared to what is already known and used.\n\nFor instance, it is already known that sampling NNs from the same \"basin of attraction\" is not working very well compared to HMC and deep ensembles. So, the poor results of VB + MC or LA + MC in Section 3 are not surprising. For instance, see the exhaustive paper \"Bayesian deep learning and a probabilistic perspective of generalization\", Wilson and Izmailov, 2020.\n\nAbout the proposed new method, this is a variation of one of the methods proposed in [6], which is actually an extensive study of the LA in many setups, with practical consideration and available code. \n\n## Other\n\nThe experiments are limited to F-MNIST and CIFAR-10 and CIFAR-100. Since the results (tables 3, 4, 5) are clear and convincing on many aspects (calibration error, distance to the gold standard HMC, OOD detection), this should not be a problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ub_jHE6u0km", "reviewer": "Reviewer_qker", "summary": "The paper addresses the problem of scalable Bayesian inference for large-scale neural networks. The core idea of the paper is to first do a (last layer) Laplace approximation of the posterior of the network weights and then refine it using normalizing flows. That is, the paper proposes to use a Laplace approximation as base distribution in normalizing flows instead of a fixed distribution, e.g. a standardized Gaussian. \n\nThe contribution of the paper is as follows:\n- a discussion of shortcomings of common strategies for computing predictive distributions (by averaging over the approximate posterior). Specifically, they contrast linearization and probit approximations with Monte Carlo integration.\n- propose to use a Laplace approximation as base distribution in a normalizing flows approximation\n- a set of thorough numerical experiments to support the claims.\n", "questions": "- Can the explanation described in lines 190-200 be further justified/substantiated? For example by inspecting/visualizing the pairs of approximate posterior and approximate posterior predictive in the two cases?\n- How much tuning was needed for optimizing the flows? Did you use the same optimization procedure/search parameters for all problems?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes to combine two existing approximate inference techniques: Laplace approximations and normalizing flows. The idea is not revolutionary, but it is simple, effective, and original to the best of my knowledge. The paper is very well-structured, well-written and easy to follow. The claims of the paper are not justified theoretically, but they are supported by a set of high-quality numerical experiments.\n\n\nThere are a few things that could be slightly clarified:\n\n- In my perspective, the quality of an approximate posterior predictive distribution depends on two things: 1) the quality of the posterior distribution of the model and 2) the accuracy of the integration method used for averaging over the posterior distribution. Therefore, using high-fidelity integration methods (e.g. MC integration with S -> infinity) will not save you if the quality of the posterior approximation is sufficiently poor. I think this is in line with the message of the paper, but it appears a bit convoluted in the discussion of the effect of the MC sample size on the accuracy of the predictive distribution (in my opinion)\n- The difference between the top and bottom plot in Figure 5 could be emphasized visually and/or in the text. It took me a while to realize the difference.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GHfRnt_jHE0", "reviewer": "Reviewer_mJ2u", "summary": "The paper studies empirically the current limitations of posterior estimation in Bayesian Neural Networks, focusing on Monte Carlo integrations as a simple golden standard and comparing its problems and limitations to the more recent techniques based on network linearization.\nThe authors then propose a post-hoc posterior refinement technique for posteriors to be estimated through MC integration and validate it empirically against the other analyzed techniques.", "questions": "1. Why Figure 5 should convince the reader that MPA is underconfident?\n\n2. Why figure 2.c should convince the reader that linearization-based approaches are being overconfident? Is there a ground truth I'm missing?\n\n3. Did you try different flows? What should convince the reader that this is a design choice not worth discussing or that it does not have a big impact on the final performance of the method?\n\n**********---------------------**********\n\nAfter the author's response I've increased my score to 6.", "limitations": "Big picture limitations were appropriately addressed.\nHowever, I think the paper would greatly benefit from a more structured section or paragraph detailing the computational limitations, especially when compared to other baselines and related work. This doesn't need to be an empirical study of all methods computational costs, but just a clearer outline of the costs implied by each of the mentioned methods.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nOverall, the paper is well written. The problem is clearly stated and so are the main approaches to it.\nThe proposed method is simple to understand and looks computationally cheap enough given the advantages it brings, and section 3 has some compelling arguments on why current methods should not be trusted.\n\nWeaknesses\nSection 3.3 feels confusing. It doesn't a good job in convincing the reader that linearization-based alternatives are bad enough to disregard a priori. Figure 5 failed in convincing me that the qualitative estimate of MC or MPA was either better or worse than the competitor.\nEmpirically, the range of experiments is satisfying, however the fact that the tested models are only a Le-Net and a WideResNet leaves the reader wondering how much the results would change with different type of architectures, e.g. even a simple Gated Recurrent Unit on Emotions or Newsgroup-20 or a perceptron on the same proposed datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept