Laplace Redux - Effortless Bayesian Deep Learning
Authors: Erik Daxberger
Agustinus Kristiadi
Alexander Immer
Runa Eschenhagen
Matthias Bauer
Philipp Hennig
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian Deep Learning
Laplace Approximation
Uncertainty Quantification
Software Library

Abstract: Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection. The Laplace approximation (LA) is a classic, and arguably the simplest family of approximations for the intractable posteriors of deep neural networks. Yet, despite its simplicity, the LA is not as popular as alternatives like variational Bayes or deep ensembles. This may be due to assumptions that the LA is expensive due to the involved Hessian computation, that it is difficult to implement, or that it yields inferior results. In this work we show that these are misconceptions: we (i) review the range of variants of the LA including versions with minimal cost overhead; (ii) introduce "laplace", an easy-to-use software library for PyTorch offering user-friendly access to all major flavors of the LA; and (iii) demonstrate through extensive experiments that the LA is competitive with more popular alternatives in terms of performance, while excelling in terms of computational cost. We hope that this work will serve as a catalyst to a wider adoption of the LA in practical deep learning, including in domains where Bayesian approaches are not typically considered at the moment.

Laplace Redux - Effortless Bayesian Deep Learning
Erik Daxberger Agustinus Kristiadi*
Alexander Immer* Philipp Hennig
Runa Eschenhagen "
Matthias Bauer
University of Cambridge MMI for Intelligent Systems, Tübingen Uniiersity of Tübingen
DDepartment of Computer Science, ETH Zurich MMax Planck ETH Center for Learning Systems "DeepMind, London
Abstract
Bayesian formulations of deep learning have been shown to have compelling theo- retical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection. The Laplace approximation (LA) is classic, and arguably the simplest family of approximations for the intractable posteriors of deep neural networks. Yet, despite its simplicity, the LA not as popular as alternatives like variational Bayes or deep ensembles. This may be due assumptions that the LA expensive due the involved Hessian computation, that it is difficult to implement, that it yields inferior results. In this work we show that these are misconceptions: we (i) review the range variants the LA includ- easy-to-us,
ing versions with minimal cost overhead; (ii) introduce laplace, software library for PyTorch offering user-friendly access all major flavors of the LA; and (ii) demonstrate through extensive experiments that the LA is competitive with more popular alternatives in terms of performance, while excelling terms of computational cost. We hope that this work will serve catalyst to wider adoption of the LA in practical deep learning, including domains where Bayesian approaches not typically considered at the moment.
laplace library: https: http://ttw...mmmeeee/////a/aaaaa Experiments: https: //gtthub..ttttt/ hatn//uuammmaaplaaeeeeaaa
Introduction
Despite their successes, modern neural networks (NNs) still suffer from several shortcomings that limit their applicability in some settings. These include (i) poor calibration and overconfidence, especially when the data distribution shifts between training and testing [1]. (ii) catastrophic forgetting of previously learned tasks when continuously trained on new tasks [2], and (iii) the difficulty of selecting suitable NN architectures and hyperparameters [3]. Bayesian modeling [4, 5] provides principled and unified approach to tackle these issues by (i) equipping models with robust uncertainty estimates [6], (ii) enabling models learn continually by capturing past information [7], and (iii) allowing for automated model selection by optimally trading off data fit and model complexity [8]-
Even though this provides compelling motivation for using Bayesian neural networks (BNNs) [9], they have not gained much traction in practice. Common criticisms include that BNNs are difficult
Equaa contributors: author ordering sampled uniformly random. Correspondence to: ead548cam.c ac. uk, agustinus. aentiiiiiinni-tuuineeeee runa. www.eenhggeneettueeeeeeee uni sri-tuebingen.d..
alexander. inmereinf ethz.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) MAP Estimation
(b) Laplace Approximation
(c) Prediction
Sappl
Figure 1: Probabilistic predictions with the Laplace approximation in three steps. (a) We find MAP estimate (yellow star) via standard training (background contours log- nog-pooterior landscape on the two-dimensional PCA subspace of the SGD trajectory [30]. (b) We locally approximate the posterior landscape by fitting Gaussian centered at the MAP estimate (yellow contours), with covariance matrix equal to the negative inverse Hessian of the loss at the MAP-this the Laplace approximation (LA). (c) We use the LA make predictions with predictive uncertainty estimates- here, the black curve is the predictive mean, and the shading covers the 95% confidence interval.
to implement, finicky to tune, expensive to train, and hard to scale to modern models and datasets. For instance, popular variational Bayesian methods [10-12, etc.] require considerable changes to the training procedure and model architecture. Also, their optimization process is slower and typically more unstable unless carefully tuned [13]. Other methods, such deep ensembles [14], Monte Carlo dropout [6]. and SWAG [15] promise to bring uncertainty quantification to standard NNs in simple manners. But these methods either require significant cost increase compared single network, have limited empirical performance, or an unsatisfying Bayesian interpretation.
In this paper, argue that the Laplace approximation (LA) is simple and cost-efficient, yet compet- itive approximation method for inference Bayesian deep learning. First proposed in this context by MacKay [16]. the LA dates back to the 18th century [17]. It locally approximates the posterior with a Gaussian distribution centered at a local maximum, with covariance matrix corresponding to the local curvature. Two key advantages of the LA are that the local maximum is readily available from standard maximum a posteriori (MAP) training of NNs, and that curvature estimates can be casily and efficiently obtained thanks to recent advances in second-order optimization, both in terms of more efficient approximations to the Hessian [18-20] and easy-to-use software libraries [21]. Together, they make the LA practical and readily applicable to many already- trained NNs the LA essentially enables practitioners to turn their high-performing point-estimate NNs into BNNs easily and quickly, without loss of predictive performance. Furthermore, the LA to the marginal likelihood may even be used for Bayesian model selection or NN training [8, 22). Figure 1 provides an intuition of the LA-we first fit point estimate of the model and then estimate a Gaussian distribution around that.
Yet, despite recent progress in scaling and improving the LA for deep learning [23-29], it is far less widespread than other methods. This is likely due to misconceptions, like that the LA is hard to implement due to the Hessian computation, that it must necessarily perform worse than the competitors due its local nature, quite simply that it is old and too simple. Here, show that these indeed misconceptions. Moreover, we argue that the LA deserves a wider adoption in both practical and research-oriented deep learning. To this end, our work makes the following contributions:
1. We first survey recent advances and present the key components of scalable and practical Laplace approximations in deep learning (Section 2).
2. We then introduce laplace, an easy-to-use RyTorch-based library for "tuming NN into BNN" via the LA (Section 3). laplace implements wide range of different LA variants.
3. Lastly, using laplace, we show in an extensive empirical study that the LA is competitive alternative approaches, especially considering how simple and cheap it is (Section 4).
2 The Laplace Approximation in Deep Learning
The LA can be used in two different ways to benefit deep learning: Firstly, we can use the LA to approximate the model's posterior distribution (see Eq. (5) below) to enable probubilistic predictions (as also illustrated in Fig. 1). Secondly, we can use the LAA to approximate the model evidence (see Eq. (6)) enable model selection (e.g. hyperparameter tuning).

---

The canonical form of (supervised) deep learning is that of empirical risk minimization. Given, e.g. an i..d. classification dataset D == {(r, E IR IRM, Yn E ) the weights 0 E I2' of an L-layer NN fo R - RC are trained to minimize the (regularized) empirical risk, which typically decomposes into a sum over empirical loss terms E(Ta, Un: ff) and a regularizer r(0),
MAP = arg minger" L(D; 0) = arg mingeR" (r(0) + En= ((
(I)
From the Bayesian viewpoint, these terms can be identified with i.i.d. log- likelihoods and a log-prior, respectively and, thus, AMA is indeed maximum a-posteriori (MAP) estimate:
(I yn: 8) log p(yn fo(x)) and r(0) log p(0)
For example, the widely used weight regularizer r(0) 3 21000 (a.k.a. weight decay) corresponds to centered Gaussian prior p(0) = (0: 0, 22), and the cross-entropy loss amounts to a categor-
ical likelihood. Hence, the exponential of the negative training loss unnormalized posterior. By normalizing it, we obtain
amounts to an
p(0 ( ž p(D 0) p(0) = = exp(=<(D;0),
z: p(D p(0) do
(3)
with an intractable normalizing constant Z. Laplace approximations [17] use a second-order expan- sion of L around OMAP construct a Gaussian approximation to p(0 (D). Le. consider:
[(D;0) & L(D;OMAP) + { - ( (V)C(D;))Axa) (0- 0MAP),
(4)
where the first-order term vanishes at OMAP. Then we can identify the Laplace approximation as
Laplace posterior approximation
p(0 (D) N(; OMAP: 23)
with
>' CDD 0) (lssss))
The normalizing constant Z (which is typically referred as the marginal likelihood or evidence) is useful for model selection and can also be approximated as
Laplace approximation of the evidence D/2 (det 1)1/
x exp( (D(;; MM^)) D
(6)
See Appendix A for more details. Thus, obtain the approximate posterior, we first need to find the argmax OMAP of the log-posterion function, i.e. do "standard" deep learning with regularized empirical risk minimization. The only additional step to compute the inverse of the Hessian matrix at 6MPP (see Figure 1(b)). The LA can therefore be constructed post-hoc to a pre--reetttt pre-trained network, even one downloaded off-the off-the-shelf. As we discuss below, the Hessian computation can be offloaded to recently advanced automatic differentiation libraries [21]. LAs are widely used to approximate the posterior distribution in logistic regression [31], Gaussian process classification [32, 33), and also for Bayesian neural networks (BNNs), both shallow [34] and deep [23]. The latter is the focus of this work.
Generally, any prior with twice differentiable log-density can be used. Due to the popularity of the weight decay regularizer, we assume that the prior is a zero-mean Gaussian p(0) = (0; 0, 721) unless stated otherwise. 2 The Hessian CDD 0) lewe then depends both on the (simple) log-prior regularizer and the (complicated) ligllikeldoood empirical risk:
V2L(D; 0) = - 2 >m V log I (xx
naive implementation of the Hessian infeasible because the second term in Eq. (7) scales quadratically with the number of network parameters, which can be in the millions even billions [35, 36). In recent years, several works have addressed scalability, as well as other factors that affect approximation quality and predictive performance of the LA. In the following, we identify, review, and discuss four key components that allow LAs to scale and perform well on modern deep architectures. See Fig. for an overview and Appendix B for a more detailed version of the review and discussion.
Four Components of Scalable Laplace Approximations for Deep Neural Networks
Inference over all Weights or Subsets of Weights
In most cases, is possible to treat all weights probabilistically when using appropriate approxima- tions of the Hessian, as we discuss below in ( Another simple way to scale the LA to large NNs
2 Onne can also consider a pay-layer even per parameter weight decay, which corresponds to a more general, but still comparably simple Gaussian prior. In particular, the Hessian of this prior is still diagonal and constant.
3

---

Weights to be treated probabilistically with Laplace
Deterministic neural network fo
(a) All
(b) Sabnetwork
(c) Last Layer
Optional: Train / usual (MAP)
Laplace
ightt- aubnetsork I "aaat__ayer''
Hyperparumeter tuning method
Approximation the
Untrained fo
(a) Full
(b) LRank
(c) KFAC
(d) Diag.
Laplace C..,
beseisn, structure- full' "full' 'loxrankk "Kroa' ('däng!)
(a) Online Laplace
Thined fa
(Approximate) predictive p(ylfo(=.);
Classification
Regression
Moute Carlo
Monte Carlo
Probit approx.
Exact predictive
Laplace bridge
Laplace
la(x, aipk.approz=
'probit" ('bridee"
Four key components scale and apply the LA to a neural network fo (with randomly- initialized or pre-trained weights 0), with corresponding laplace code. ①1 We first choose which part of the model we want to perform inference over with the LA. 2 We then select how to approximate the Hessian. (3 We can then perform model selection using the evidence:
Figure 2:
(a) If we started with an untrained model fe. we can jointly train the model and use the evidence to tune hyperparameters online. (b) If we started with pre-trained model, can use the evidence to tune the hyperparameters post-hoc. Here, shades represent the loss landscape, while contours represent LA log-po tee--ootiioos-ftceeee faded contours represent intermediate iterates during hyperparameter tuning to obtain the final log-posterion (thick yellow contours). (4 Finally, to make predictions for a new input Ir, we have several options for computing/approxiiatiin the predictive distribution p(y ())
(without Hessian approximations) is the subnetwork LA [27], which only treats subset of the model parameters probabilistically with the LA and leaves the remaining parameters at their MAP-estimated values. An important special case of this applies the LA to only the last linear layer an L-layer NN, while fixing the feature extractor defined by the first L layers at its MAP estimate [37, 28]. This last-layer LA is cost-effective yet compelling both theoretically and in practice [28].
Hessian Approximations and Their Factorizations
One advance in recond-orded optimization that the LA can benefit from positive semi-definie definie approximations to the (potentially indefinite) Hessian of the liglliielsooods of NNs in the second term of Eq. (7) [38]. The Fisher information matrix [39], abbreviated the Fisher and defined by
F = N
Eğmp(yl fo(x,)) [(V, a log P(ÿ (o(xr)))) bns ) (Vo log fo(xx)) (mass)))
(8)
one such choice.? One can also use the generalized Gauss-Newton matrix (GGN) matrix [41]
-V G GN ((x)) V log p(y.. f) Jx,))"
where J(xx)) = Vefe(In) leuNp is the NN's Jacobian matrix. As the Fisher and GGN are equivalent for common likeliielssodd [38], we will henceforth refer them interchangeably. In deep LAs, they have emerged as the default choice [23, 24, 28, 29, 27, 26, etc.).
Iff instead of taking expectation in (8), we use the training label we call the matrix the empirical Fisher. which is distinct from the Fisher [38, 40).

---

As F' and G are still quadratically large, we typically need further factorization assumptions. The most lightweight diagonal factorization which ignores off-diagonal elements [42, 43]. More expressive alternatives are block-diagonal factorizations such as Kronecker-fuctored approximate curvature (KFAC) [18-20], which factorizes each within-layer Fisher* as Kronecker product of two smaller matrices. KFAC has been successfully applied to the LA [23, 24] and can be improved by low-rank approximations of the KFAC factors [29] by leveraging their eigendeccompositions [44]. Finally, recent work has studied/enabled low-rank approximations of the Hessian/Fisher [45-47].
3 Hyperparameter Tuning
As with all approximate inference methods, the performance of the LA depends on the (hy- per)parameters of the prior and likelihood. For instance, it is typically beneficial to tune the prior variance 7² used for inference [23, 28, 27, 26, 22]. Commonly, this is done through cross-validation, c.g. by maximizing the validation log-likelihoon [23, 48] or, additionally, using -ut-of-distribution data [28, 49]. When using the LA, however, marginal likelihood maximization (a.k.a. empirical Bayes or the evidence framework [34, 50]) constitutes a more principled alternative to tune these hyperparameterss and requires no validation data. Immer et al. [22] showed that marginal likelihood maximization with LA can work in deep learning and even be performed in an online manner jointly with the MAP estimation. Note that such approach is not necessarily feasible for other approximate inference methods because most do not provide an estimate of the marginal likelihood. Other recent approaches for hyperparameter tuning for the LA include Bayesian optimization [51] or the addition of dedicated, trainable hidden units for the sole purpose of uncertainty tuning [49].
Approximate Predictive Distribution
To predict using a posterior (approximation) p(0 'D), we need to compute p(y (x.). D) = I p(y foxx)) p(0) D)) de for any test point X. E RR"," which is intractable in general. The sim- plest but most general approximation to p(y I., D) is Monte Carlo integration using S samples (0,)² from p(0| D): ply (().)) & S- p(y Jo, (2.)). However, for LAS with GGN and Fisher Hessian approximations Monte Carlo integration can perform poorly [48, 26). Immer et al. [26] attribute this to the inconsistency between Hessian approximation and the predictive and suggest to use a linearized predictive instead, which can also be useful for theoretic analyses [28]. For the last-layer LA, the Hessian coincides with the GGN and the linearized predictive is exact.
E==
The predictive of a linearized neural network with a LA approximation to the posterior p(0) (D) z ((;; 0MAP. Σ) results in a Gaussian distribution on neural network outputs f. := fxx)) and therefore enables simple approximations or even a closed-form solution. The distribution on the outputs is given by p(f= [ur., DD s Nf..; Sowss (x.,s ). J (ar.) (xr))) and is typically significantly lower- dimensional (number of outputs C instead of parameters D). It can also be inferred entirely in function space as Gaussian process [25, 26). Given the distribution on outputs f.. the predictive distribution can be obtained by integration against the likelihood: p(yl x., D) = ] p(y) f-)p(f= z,, D) d0. In the case of regression with a Gaussian likelihood with variance σ², the solution can even be obtained analytically: ply x., D) R Nw; Jonne (x=), (Jr..) + I). For non-Gaussian likelihoods, e.g. in classification, a further approximation is needed. Again, the simplest approximation to this is Monte Carlo integration. In the binary case, we can employ the probit approximation [31, 16] which approximates the logistic function with the probit function. In the multi-class case, we can use its generalization, the extended probit approximation [52]. Finally, first proposed for non-BNN BNN applications [53, 54], the Laplace bridge approximates the softmax- Noftmax-Gaussial Gaussian integral via a Dirichlet distribution [55]. The key advantage is that it yields distribution of the integral solutions.
3 laplace: A Toolkit for Deep Laplace Approximations
Implementing the LA is non-trivial, as it requires efficient computation and storage of the Hessian. While this is not fundamentally difficuul, there exists no complete, easy-to-use, and standardized im- plementation of various LA flavors-instead it is common for deep learning researchers repeatedly re-implement the LA and Hessian computation with varying efficiency [56-58, etc.). An efficient implementation typically requires hundreds of lines of code, making it hard to quickly prototype
'The elements F or G corresponding the weight Wi C 0 of the (-th layer of the network.
5

---

from laplace import Laplace
from laplace inport Laplace
2
2
pre-trained model model 1oad__ap_mmdel1.)
Load or pre- pre-trdsned model = ma__dol)))
Define and fet LA varient usth custon seftings
# Fst default, recommended LT variant;
1s Laplace(uodel, "claseification"
7 # Last-layer KFAC LA
peplacemmddel 'regression') 9 la. wwl.fftttiinnaaaaaeaa
10 fo.fittrrii__odder pracpiiiioon
11 la www.ootiioepprrr.rcmom praaiaaa..pptrr..eeeeiioooeeeeed val_lodder-val_oodee
11 # Differentiate merginal ltkelshood w.r.t.
12
12 prior precssion and observation noise ml Suglix(prior_precieet www.priiiinpprirrrrrr.. prec, eigmo_noise-chs_noii
13
14 Make prediction with custom predictive approz.
14
pred la(x, prod_type glm', link_approx 'probist"" 'probit')
backeard()
Fit diagonal LA over all weights of pre-trained classification model, do post-hoc tuning of the prior precision hyperparameter us- ing vooss-vali,tion, and make prediction for input æ with the probit approximation.
Listing 1:
Fit KFAC LA over the last layer of pre- or un un-trained regression model and differen- tiate its marginal likelihood w.r.t. some hyperpa- rameters for post-hoc hyperparameter tuning or online empirical Bayes (see Immer et al. [22]).
Listing 2:
with the LA. To address this, we introduce laplace: simple, easy-to-use, extensible library for scalable LAs of deep NNs PyTorch [59]. laplace enables all sensible combinations of the four components discussed in Section 2 see Fig. 2 for details. Listings and 2 show code examples.
The core of laplace consists of efficient implementations of the LA's key quantities: (i) posterior (i.e. Hessian computation and storage), (ii) marginal likelihood, and (iii) posterior predictive. For (i), take advantage of advances in automatic differentiation, we outsource the Hessian computation to state-of-the-art, optimized second-order optimization libraries: BackPACK [21] and ASDL [60]. Moreover, we design 1 laplace ace in a modular manner that makes easy to add new backends and approximations in the future. For (ii), we follow Immer et al. [22] in our implementation of the LA's marginal likelihood both online and post-hoc marginal likelihood tuning. cf. Listing 2. Note that laplace also supports standard pross-validation for hyperparameter tuning [23, 28), as shown in Listing Finally, for (iii), laplace supports all approximations to the posterior predictive distribution discussed in Section 2-it thus provides the user with flexibility in making predictions, depending on the computational budget.
is thus both efficient and differentiable and allows the user to implement
Default behavior To abstract away from large number of options available (Section 2), we provide the following default choices based on extensive experiments (Section 4); they should be applicable and perform decently in the majority of use cases: we assume a pre-trained network and treat only the last-layer weights probabilistically (last-layer LA), use the KFAC factorization of the GGN and tune the hyperparameters post-hoc using empirical Bayes. To make predictions, we use the closed-form Gaussian predictive distribution for regression and the (extended) probit approximation for classification. Of course, the user can pick custom choices (Listings and 2).
Limitations Because laplace employs external libraries (BackPACK [21] and ASDL [60]) as backends, inherits the available choices of Hessian factorizations from these libraries. For instance, the LA variant proposed by Lee et al. [29] can currently not be implemented via laplace, because neither backend supports eigenvalue-corrected KFAC [44] (yet).
Experiments
We benchmark various LAs implemented via laplace. Section 4.1 addresses the question of "which are the best design choices for the LA", ; in light of Figure 2. Section 4.2 shows that the LA is competitive to strong Bayesian baselines in in-distribution, dataset-shift, and diitt--iistiibtttinn (OOD) settings. We then showcase some applications of the LA in downstream tasks. Section 4.3 demonstrates the applicability of the (last-layer) LA on various data modalities and NN architectures (including transformers [61])- settingss where other Bayesian methods are challenging to implement. Section 4.4 shows how the LA can used as an easy-to-use yet strong baseline continual learning. In all results, arrows behind metric names denote if lower (L) or higher (f) values are better.

---

CIFAR-I0 DA
CIFAR-10
Table 1: OOD detection performance aver- aged over all test sets (see Appendix C.2 for details). Confidence is defined as the max. of the predictive probability vector [62] (e.g. Confidence((O.7, 0.2. 0.1l) = 0.7). LA and especially LA* reduce the overconfidence of MAP and achieve better results than the VB, CSGHMC (HMC), and SWAG (SWG) baselines.
0.93
0.88
AUROC
0.84
MAP
0.91
0.80
online
pust-hoc
0.76
0.92 0.93
Acc. ID
ID
Figure 3: In- vs. out-of-distribation (II) and OOD, resp.) performance configurations (dots), each being a combination of settings for 1) subset-of-weights, 2) covariance structure, 3) hyperparameter tuning, and 4) predic- tive approximation (see Appendix C.1 for details), "DA" stands for "data augmentation". Post-hoc per- forms better with DA and a strong pre-trained net- work. while online performs better without DA where optimal hyperparameters are unknown.
CIFAR-10 of different LA
Confidence
AUROC 1
Methods
MNIST
CIFAR-10
MNIST
CIFAR-10
MAP
75.0+0.4
76.11:122
96,5+0.11
92.1=0.5
DE
65.7::0.3
65.4104
97.5+000
94.0=0.1
VB
73.2.10..8
58.81.0.7
95.8.10.2
88.7.0.3
HMC
69.2::1.7
69.41:066
96.140022
90.6=0022
SWG
75.8.10.3
68.11.2.3
96.5.10.1
91.3100.
67.5:t0.4
69.0::1.3
96.2+2022
92.2-0055
LA*
56.1.10.5
55.7.11.2
96.4.10.2
92.4.10.5
4.1 Choosing the Right Laplace Approximation
In Section 2 we presented multiple options for each component of the design space of the LA. resulting in a large number of possible combinations, all of which are supported by laplace. Here, we try to reduce this complexity and make suggestions for sensible default choices that cover common application scenarios. To this end, we performed a comprehensive comparison between most variants; measured in- and -uitt--dittiibutiin performance on standard image classification benchmarks (MNIST, FashionMNISTT. CIFAR-10) but also considered the computational complexity of each variant. We provide details of the comparison and list of the considered variants in Appendix C.1 and summarize the main arguments and take-aways in the following.
Hyperparameter tuning and parameter inference. We can apply the LA purely post-hoc (only tune hyperparameters of pre-trained network) or online (tune hyperparameters and train the network jointly, as e.g. suggested by Immer al. [22]). We find that the online LA only works reliably when it is applied to all weights of the network. In contrast, applying the LA post-hoc only on the last layer instead of all weights typically yields better performance due to less underfitting, and is significantly cheaper. For problems where a pre-trained network or optimal hyperparameters are available, e.g. for well-studied data sets, we, therefore, suggest using the post-hoc variant on the last layer. This LA has the benefit that it has minimal overhead over standard neural network forward pass (cf. Fig. 5) while performing on par or better than state-of-the-att approaches (cf. Fig. 4). When hyperparameters are unknown or no validation data is available, we suggest training the neural network online by optimizing the marginal likelihood, following Immer et al. [22] (cf Section 4.4). Figure 3 illustrates this on CIFAR- 10: for CIFAR-10 10 with data augmentation, strong pre-traie pre-trained networks and hyperparameters are available and the post-hoc methods directly profit from that while the online methods merely reach the same performance. On the less studied CIFAR- 10 without data augmentation, the online method can improve the performance over the post-hoc methods.
Covariance approximation and structure. Generally, we find that more expressive covariance approximation improves performance, as would be expected. However, a full covariance is in most cases intractable for full networks or networks with large last layers. The KFAC structured covariance provides good trade-off between expressiveness and speed. Diagonal approximations perform significantly worse than KFAC and are therefore not suggested. Independent of the structure, we find that the empirical Fisher (EF) approximations perform better on out-of of-distribution detection tasks while GGN approximations tend perform better on in-distribution metrics.
Predictive distribution. Considering in- and out- -iftiiiiiinttion (OOD) performance as well as cost, the probit provides the best approximation to the predictive for the last-layer LA. MC integration can sometimes be superior for OOD detection but at an increased computational cost. The Laplace bridge has the same cost as the probit approximation but typically provides inferior results in our experiments. When using the LA online to optimize hyperparameters, we find that the resulting MAP
7

---

10a
100%
MAP
DE
BMC
0.6
7.3
SwG LA LX
0.4
10
99%
5
2.5
0.2
10
98%
0
NLL
ECE
& Acc.
0
50
100
150
50
100
150
100
96%
0.3
10-1
93%
0.2
0.1
10-2
90%
0
NLL J
ECE
% Acc. 1
Metries
Shift Intensity
Shift Intensity
(a) In-Distribution
(b) Distribution-shift NLL
(c) Distribution-shit ECE
Figure 4: Assessing model calibration (a) on in-distribution data and (b,c) under distribution shift, for the MNIST (top row) and CIFAR-I0 (bottom row) datasets. For (b,c), we use the
(top) (bottom) benchmarks [63, 64]. In (a), we report accuracy and, to measure
and Corrupted-CIFAR calibration, negative log-likeliine oigliiklihood (NLL) and expected calibration error (ECE)-a) all evaluated on the standard test sets. In (b) and (c), we plot shift intensities against NLL and ECE, respectively. For Rotated-MNIST (top). shift intensities denote degrees of rotation of the images, while for Corrupted- CIFAR-10 (bottom), they denote the amount of image distortion (see [63, 64) for details). (a) On in-distribution data, LA is the best-calibrated method in terms ECE, while also retaining the accuracy of MAP (unlike VB and CSGHMC). (b,c) On corrupted data, all Bayesian methods improve upon MAP significantly. Even though post-hoc all LAs achieve competitive results, even to DE. In particular, LA* achieves the best results, at the expense of slightly worse in-distribution calibration this trade-off between in- and -ittr--distribution performance has been observed previously [65].
predictive provides good performance in-distribution, but a probit or MC predictive improves OOD performance.
Overall recommendation. Following the experimental evidence, the default in laplace a post- hoc KFAC last-layer LA with a GGN approximation to the Hessian. This default is applicable to all architectures that have Cully-connected last layer and can be easily applied to pre-trained networks. For problems where trained networks are unavailable or hyperparameters are unknown, the online KFAC LA with GGN empirical Fisher provides a good baseline with minimal effort.
4.2 Predictive Uncertainty Quantification
We consider two flavors of LAs: the default flavor of laplace (LA) and the most robust one in terms of distribution shift found in Section 4.1 lastlaar,,, with full empirical Fisher Hessian approximation, and the probit approximation). We compare them with the MAP network (MAP) and various popular and strong Bayesian baselines: Deep Ensemble [DE, 14], meannfiield variational Bayes Hamiltonian Monte Carlo
[VB. 11. 12] with the flipout estimator [66], cyclical stochastic-gradies [CSGHMC HMC, 67], and SWAG [SWG, 15]. For each baseline, we use the hyperparameters recommended in the original paper- Appendix A for details. First, Fig. 4 shows that LA and LA* are, respectively, competitive with and superior to the baselines in trading- off between mn-distriuution calibration and dataset-shift robustness. Second, Table shows that LA and LA* achieve better results on diitr-i-ditiiitiinnnn (OOD) detection than even VB, CSGHMC, and SWG.
The LA shines even more when we consider its (time and memory) cost relative to the other, more complex baselines. Fig. 5 we show the wall-clock times of each method relative to MAP's for training and prediction. As expected, DE, VB. and CSGHMC are slow to train and in making predictions: they are between two to five times more expensive than MAP. Meanwhile, despite being post-hoc, SWG is almost twice expensive MAP during training due to the need for sampling and updating its batch normalization statistics. Moreover, with 30 samples, as recommended by its authors [15], it is very expensive at prediction time more than ten times more expensive than MAP.
8

---

MAP
DE
Temp. Scaling
X LA
OOD
ID
ODD
ID
00D
OOD
ID
0OD
0.3
0.70
0.70
NILL
0.50
0.2
X
0.65
0.25
0.1
0.2
0.4
40
ECE/ ECE/Calle Calib.
0.3
0.10
0.1
20
0.1
0.0
(a)
Camelyon17
(b) FMow
(c) CivilComments
(d) Amazon
(e) PovertyMap
Figure Assessing real-worke distribution shift robustness on five datasets from the WILDS bench-
mark [68], covering different data modalities, model architectures, and output types. Camelyon17:
Tissue slide image tumor classification across hospitals (DenseNet-121 [69]). FMoW: Satellite image land use classification across regions/years D2nseNNet1121.. CivilCommments: Online comment toxicity classification across demographics (DistiBERT [70]). Amazon: Product review sentiment classification across users (DistilBERT), PovertyMap: Satellite image asset wealth regression across countries (ResNet-18 [35]). We plot means + standard errors the NLL (top) and ECE (for classifi- cation) regression calibration error [71] (bottom). The in-distribution (left panels) and OOD (right panels) dataset splits correspond to different domains (e.g. hospitals for Camelyon17), LA is much better calibrated than MAP, and competitive with temp. scaling and DE, especially the OOD splits.
Meanwhile, LA (and LA the cheapest of all methods considered: it only incurs negligible overhead on top of the costs of MAP. This is similar for the memory consump- tion (see Table 5 in Appendix C.5). This shows that the LA significantly more memory- and compute-efficiee all the other methods, adding minimal overhead over MAP inference and prediction. This makes the LA particularly attractive for practitioners, especially in low-resource vironments. Together with Fig. 4 and Table 1, this justifies default flavor in lapl acl, and importantly, shows that Bayesian deep learning does not have to be expensive.
Relative Time
Thining
than
MAP
DE
VB
HMC SWG
LA
Figure 5: Wall-clock time costs relative to MAP. LA introduces negligible over- head over MAP, while all other baselines are significantly more expensive.
4.3 Realistic Distribution Shift
So far, our experiments focused on comparably simple benchmarks, allowing us to comprehensively assess different LA variants and compare to more involved Bayesian methods such as VB, MCMC, and SWAG. In more realistic settings, however, where we want to improve the uncertainty of complex and costly-to-train models, such as transformers [61], these methods would likely be difficult get to work well and expensive to run. However, one might often have access to a pre-trained model, allowing for the cheap use of post-hoe methods such as the LA. To demonstrate this, we show how laplace can improve the distribution shift robustness of complex pre-trained models in large-scale settings. To this end, we use WILDS [68], a recently proposed benchmark of realistic distribution shifts encompassing a variety of real-world datasets across different data modalities and application domains. While the WILDS models employ complex (e.g. convolutional or transformer) architectures feature extractors, they all feed into linear output layer, allowing to conveniently and cheaply apply the last-layer LA. As baselines, we consider: 1) the pre- are-trained MAP models [68], 2) post-hoc temperature scaling of the MAP models (for classification tasks) [1]. and 3) deep ensembles [14]. More details on the experimental setup are provided in Appendix C.3. Fig. 6 shows the results on five
We simply construct deep ensembles from the various pre- pre-trained models provided by Koh et al. [68].

---

different WILDS datasets (see caption for details). Overall, Laplace is significantly better calibrated than MAP, and competitive with temperature scaling and ensembles, especially on the OOD splits.
4.4 Further Applications
Beyond predictive uncertainty quantification, the LA is useful in wide range of applications such as Bayesian optimization [37], bandits [72], active learning [34, 73], and continual learning [24]. The laplace library con- veniently facilitates these applications. As an example, we demonstrate the performance of the LA on the stan- dard continual learning benchmark with the Permuted- MNIST dataset, consisting of ten tasks each containing pixel-permuted MNIST images [74]. Figure 7 shows how the all-layer diagonal and Kronecker-factee overcome catastrophic forgetting. forgettiiinn
0.9
Accuracy
0.8
MAP LA
VB (VOGN)
0.7
LA
4
8
10
Task
Figure Continual learning results on Permuted-MNIST. MAP fails catastroph- ically as more tasks are added. The Bayesian approaches substantially out- perform MAP, with LA-KFAC perform- ing the best, closely followed by VOGN.
LAs can In this experiment, we update the LAs after each task as suggested by Ritter al. [24] and improve upon their result by tuning the prior pre- cision through marginal likelihood optimization during training, following Immer et al. [22] (details in Appendix C.4). Using this scheme, the performance after 10 tasks is around 96% accuracy, outperforming other Bayesian approaches for continual learning [7, 75, 76]. Concretely, we show that the KFAC LA, while much simpler when applied via laplace, can achieve better performance to a recent VB baseline [VOGN, 13]. Our library thus provides an easy and quick way of constructing a strong baseline for this application.
5 Related Work
The LA is fundamentally a local approximation that covers a single mode of the posterior; similarly, other Gaussian approximations such as mean-field variational inference [11-13] SWAG [15] also only capture local information. SWAG uses the first and second empirical moment of SGD iterates to form diagonal plus low-rank Gaussian approximation but requires storing many NN copies and applying a (costly) heuristic related to batch normalization at test time. In contrast, the LA directly uses curvature information of the loss around the MAP and can be applied post-hoc pre-trained NNs.
In contrast to local Gaussian approximations, (stochastic- gradient) MCMC methods [77, 78, 67, 79, 80, etc.] and deep ensembles [14] can explore several modes. Nevertheless, prior works-alss validated in our experiments Section 4 i-idiiaaee that using single mode might not be as limiting in practice as one might think. Wilson and Izmailov [81] conjecture that this is due to the complex, nonlinear connection between the parameter space and the function (output) space NNs. Moreover, while unbiased compared to its simpler alternatives, MCMC methods are notoriously expensive in practice and, thus, often require further approximations such as distillation [82, 83]. Finally, note that both the LA as well as SWAG can be extended to ensembles of modes in post-hoc manner [84, 81].
6 Conclusion
In this paper, we argued that the Laplace approximation is a simple yet competitive and versatile method for Bayesian deep learning that deserves wider adoption. To this end, we reviewed many recent advances to and variants of the Laplace approximation. including versions with minimal cost overhead that can be applied post-hoc to pre-trained off-the-shelf models. demonstrated that the Laplace approximation is on par with other approaches that approximate the intractable network posterior, but at typically much lower computational cost. A particularly simple variant that only treats some weights probabilistically can even be used in the context of pre-trained transformer models to improve predictive uncertainty. As an efficient implementation not straightforward, we introduced laplace, a modular and extensible software library for PyTorch offering user-friendly access to all major flavors of the Laplace approximation. In this way, Laplace approximations provide drop-in in Bayesian functionality for most types of deep neural networks.
comprehensive evaluation we
10

---

Acknowledgments and Disclosure of Funding
We thank Kazuki Osawa for providing early access to his automatic second-order differentiation (ASDL) library for PyTorrc and Alex Botev for feedback on the manuscript. We also thank the anonymous reviewers for their helpful suggestions for our paper.
E.D. acknowledges funding from the EPSRC and Qualcomm. A.I. gratefully acknowledges funding by the Max Planck ETH Center for Learning Systems (CLS), R.E., A.K. and P.H. gratefully acknowledge financial support by the European Research Council through ERC StG Action 757275 PANAMA; the DFG Cluster of Excellence "Machine Learning New Perspectives for Science". EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S8809A and funds from the Ministry of Science, Research and Arts of the State of Baden-W Württemberg. A.K. is grateful to the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support.
References
[1] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinherger. On Calibration of Modern Neural Networks. In ICML, 2017.
[2] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bart Brabska-Barwinska, al. Overcoming Catastrophic Forgetting Neural Networks. Proceedings of the National Academy of Sciences, 114(13), 2017.
[3] Frank Hutter. Lars Kotthoff, and Joaquin Vanschoren. Automated Cachine Learning: Methods, Systems. Challenges. Springer Nature, 2019.
[4] David Barber. Bayesian Reasoning and Machine Learning. Cambridge University 2012.
2012.
[5] Zoubin Ghahramani. Probabilistic Machine Learning and Artificial Intelligence. Nature, 521(7553), 2015.
[6] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In JCML, 2016.
[7] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational Continual Learning. In ICLR, 2018.
David JC MacKay. Probable Networks and Plausible Predictions-4 Review of Practical Bayesian Methods Supervised Neural Networks. Network: Computation Neural Systems, 1995.
[9] Yarin Gal. Uncertainty deep learning. University of Cambridge, 2016.
[10] Geoffrey Hinton and Drew Van Camp. Keeping the Neural Networks Simple by Minimizing Description Length of the Weights. In COLT, 1993.
[11] Alex Graves. Practical Variational Inference for Neural Networks. In NIPS, 2011.
[12] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Networks. In JCML, 2015.
[13] Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical Deep Learning with Bayesian Principles. In NeurIPS, 2019.
[14] Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncer- tainty Estimation using Deep Ensembles. In NIPS, 2017.
[15] Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A Simple Baseline for Bayesian Uncertainty in Deep Learning. In NeurlPS, 2019.
[16] David JC. MacKay. Bayesian Interpolation. Neural computation, 4(3), 1992.
[17] Pierre-Simon Laplace. Mémoires de Mathématique et de Physique, Tome Sixieme. 1774.
[18] Tom Heskes. On "Natural" Learning and Pruning in Multilayered Perceptrons. Neural Computation, 12
(4), 2000.

---

[19] James Martens and Roger Grosse. Optimizing Neural Networks with Kronecker-Factored Approximate Curvature. In ICML, 2015.
[20] Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton Optimisation for Deep Learning. In ICMI, 2017.
[21] Felix Dangel, Frederik Kunstner, and Philipp Hennig. Backpack: Packing More into Backprop. In ICLR, 2020.
[22] Alexander Immer. Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scal- able Marginal Likelihood Estimation for Model Selection Deep Learning. ICML, 2021.
[23] Hippolyt Ritter, Aleksandar Botev, and David Barber. A Scalable Laplace Approximation for Neural Networks. In ICLR, 2018.
[24] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting. In NIPS, 2018.
[25] Mohammad Emtiyaz E Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate Inference Turns Deep Networks Into Gaussian Processes. In NeurlPS, 2019.
[26] Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving Predictions of Bayesian Neural Net- works via Local Linearization. In AISTATS, 2021.
[27] Erik Daxberger, Eric Nalisnick, James Urquhart Allingham, Javier Antorán, and José Miguel Hernández- Lobato. Bayesian Deep Learning via Subnetwork Inference. In ICMI, 2021.
[28] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, Even Just a Bit, Fixes Overconfi- dence in ReLU Networks. In ICML, 2020.
[29] Jongseok Lee, Matthias Humt, Jianxiang Feng, and Rudolph Trichel. Estimating Model Uncertainty of Neural Networks in Sparse Information Form. In ICML, 2020.
[30] Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Subspace Inference for Bayesian Deep Learning. In UAI, 2019.
[31] David J Spiegelhalter and Steffen L Lauritzen. Sequential Updating of Conditional Probabilities on Directed Graphical Structures, Networks, 1990.
[32] Christopher KI Williams and David Barber. Bayesian Classification with Gaussian processes. IEEE Transactions Pattern Analysis and Machine Intelligence, 20(12), 1998.
[33] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes in Machine Learning. The MIT Press, 2005.
[34] David JC MacKay. The Evidence Framework Applied to Classification Networks. Neural computation. 1992.
[35] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
[36] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LLMM LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint wxXiv:199.080553 2019.
[37] Jasper Snock, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayaman Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In ICML, 2015.
[38] James Martens. New insights and perspectives on the natural gradicnt method. Journal of Machine Learning Research, 21616)):177, 2020.
[39] Shun-Ichi Amari. Natural Gradient Works Efficiently in Learning. Neural computation, 10(2), 1998.
[40] Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the Empirical Fisher Approximation for Natural Gradient Descent. In NewrIPS, 2019.
[41] Nicol N Schraudolph. Fast Curvature Matrix-Vects Products for Second- -rddr Gradient Descent. Neural computation, 14(7), 2002.
12

---

[42] Yann LeCun, John S Denker, and Sara A Solla. Optimal Brain Damage. In NIPS, 1990.
[43] John S Denker and Yann LeCun. Transforming Neural-Net Output Levels to Probability Distributions. In NIPS, 1990.
[44] Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis. In NIPS, 2018.
[45] David Madras, James Atwood, and Alex D' Amour. Detecting extrapolation with local ensembles. In JCLR, 2020.
[46] Wesley Maddox, Gregory Benton, and Andrew Gordon Wilson. Rethinking parameter counting in deep models: Effective dimensionality revisited. arXiv preprint arXiv:2003.024 0211:1999 2020.
[47] Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-of-distributio detection for doep neural networks. arXiv preprint arXiv:2102.12567, 2021.
[48] Andrew YK Foong, Yingzhen Li, José Miguel Hemmnndez-Lohato, and Richard E Turner. 'In-Between' Uncertainty in Bayesian Neural Networks. arXiv preprint arXiv: /S57, 2019.
[49] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Learnalee Uncertainty under Laplace Approxima- tions. UAI, 2021.
[50] José M Bernardo and Adrian FM Smith. Bayesian Theory. John Wiley & Sons, 2009.
[51] Matthias Humt, Jongseok Lee, and Rudolph Triebel. Bayesian Optimization Meets Laplace Approximation for Robotic Introspection. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Long-Term Autonomy Workshop, 2020.
[52] Mark N Gibbs. Bayesian Gaussian Processes for Regression and Classification. Ph. D. Thesis, Department of Physics, University of Cambridge, 1997.
[53] David JC MacKay. Choice of Basis for Laplace Approximation. Machine learning, 33(1), 1998.
[54] Philipp Hennig, David Stern, Ralf Herbrich, and Thore Craepel. Kernel Topic Models. In AISTATS, 2012.
[55] Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast Predictive Uncertainty for Classification with Bayesian Deep Networks. arXiv preprint arXiv:2003.01227 2020.
[56] Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. Code repo for "A Simple Bascline for Bayesian Deep Learning". https://github. http://ggthuuu.coo///... con/wjmaddox/sua gaussian, 2019.
[57] Agustinus Kristiadi. laytr Laplace approximation code examples. last_ Last_layer__aplace. 2020.
[58] Jongseok Lee and Matthias Humt. Official Code: Estimating Model Uncertainty Neural Networks in Sparse Information Form, ICML2020. attps://gitae com/DLR- RM// curvature, 2020.
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py Torch: An Imperative Style, PerfoPeerfomancc Deep Learning Library. In NeurlPS, 2019.
[60] Kazuki Osawa. ASDL: Automatic second order differentiation (for fisher. gradient covariance, hessian, jacobian, and kernel) library. https: : /gtbbbttt....... wtw/ /wwaaakkkaaa//aaaaaataaaa ghjkl, 2021.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In NIPS, 2017.
[62] Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and cattof-Distribution Examples in Neural Networks. In ICLR, 2017.
[63] Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corrup- tions and Perturbations. In ICLR, 2019.
[64] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshninarayanan, and Jasper Snock. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift. In NeurIPS, 2019.
13

---

[65] Zhiyun Lu, Eugene le. and Fei Sha. Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation. arXiv preprint 2rXiv:2006.07584, 2020.
[66] Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient Pseudo--Indeenndnn Weight Perturbations Mini-Batches. In ICLR, 2018.
[67] Rugi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning. In ICLR, 2020.
[68] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang. Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. WIL.DS: Benchmark of In-The- Wild Distribution Shifts. arXiv preprint arXiv22012.07421, 2020.
[69] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Connected Convo- lutional Networks. In CVPR, 2017.
[70] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a Distilled Version of Bert: Smaller, Faster, Cheaper and Lighter. In 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing NeurlPS, 2019.
[71] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate Uncertainties for Deep Learning Using Calibrated Regression. In ICML, 2018.
[72] Olivier Chapelle and Lihong Li. An Empirical Evaluation of Thompson Sampling. In NIPS, 2011.
[73] Mijung Park, Greg Horwitz, and Jonathan Pillow. Active Leamming of Neural Response Functions with Gaussian Processes. In NIPS, 2011.
[74] lan J Goodfellow, Mchdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An Empirical Investigation of Catastrophic Forgetting in Gradient- -Baeed Neural Networks, arXiv preprint arXiv:1312..0. 1312,2211, 2013,
[75] Michalis K Titsias, Jonathan Schwarz, Alexander de G Matthews, Razvan Pascanu, and Yee Whye Tch. Functional Regularisation for Continual Learning with Gaussian Processes. In ICLR, 2020.
[76] Pingho Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard E Turner, and Moham- mad Emtiyaz Khan. Continual Deep Learning by Functional Regularisation of Memorable Past. In NeurIPS, 2020.
[77] Max Welling and Yee W Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In ICML, 2011.
[78] Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swigtkowski, Linh Tran, Stephan Mandt, Jasper Snock, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How Good is the Bayes Posterior in Deep Neural Networks Really? ICML, 2020.
[79] Pavel Izmailov, Sharad Vikram, Matthew D Hoffmmnn, and Andrew Gordon Wilson. What Are Bayesian Neural Network Posteriors Really Like? In ICMI, 2021.
180] Adrià Garriga-Alonso and Vincent Fortuin, Exact langevin dynamics with stochastic gradients. arXiv preprint 10w.:21022000911111111 2021.
[81] Andrew G Wilson and Pavel Izmailov. Bayesian Deep Learning and a Probabilistic Perspective of General- ization. In NewrlPS, 2020.
[82] Anoop Korattikara, Vivek Rathod, Kevin Murphy, and Max Welling. Bayesian Dark Knowledge. In NIPS, 2015.
[83] Kuan-Chich Wang, Paul Vicol, James Lueas, Li Gu, Roger Grosse, and Richard Zemel. Adversarial Distillation of Bayesian Neural Network Posteriors. In ICML, 2018.
[84] Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of Laplace Ap- proximations for Improved Post-Hoc Uncertainty in Deep Learning. NeurlPS Workshop on Bayesian Deep Learning, 2021.
[85] Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In ICLR, 2019.
[86] David JC MacKay, A Practical Bayesian Framework For Backpropagation Networks. Neural computation. 1992.
14

---

[87] Sebastian Farquhar, Lewis Smith. and Yarin Gal. Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations, In NeurIPS, 2020.
[88] Arjun K Gupta and Daya K Nagar. Matrix Variate Distributions. Chapman and Hall, 1999.
[89] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, (3):211-218 1-218, 1936.
[90] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[91] Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E. Turner, Jose Miguel Hermandez-Loban and Alexander L. Gaunt. Deterministic Variational Inference for Robust Bayesian Neural Networks. In ICLR, 2019.
[92] Amr Ahmed and Eric P Xing. Seeking The Truly Correlated Topic Posterior-On On Tight Approximate Inference of Logistic-Normal Admixture Model. In AISTATS, 2007.
[93] Michael Braun and Jon McAuliffe. Variational Inference for Large-Scale Models of Discrete Choice. Journal of the American Statistical Association, 105(489), 2010.
[94] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 11:2278-22332 1998.
[95] Sergey Zagoruyko and Nikos Komodakis.
Wide Residual Networks. In BMVC, 2016.
[96] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR, 2017.
[97] Ranganath Krishnan and Piero Esposito. Bayesian-Torch: Torch: Bayesian Neural Network Layers for Uncertainty Estimation. https://githe:
http://yyysaaattrch...... 2020.
[98] Florian Wenzel, Jasper Snock, Dustin Tran, and Rodolphe Jenatton. Hyperparameter Ensembles for Robustness and Uncertainty Quantification. In NewrIPS, 2020.
[99] Ferenc Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, page 201717042, 2018
15


Reviews:
[{"review_id": "qzhJV2ZEtd5", "reviewer": "Reviewer_vbUc", "summary": "This paper describes a software library for scalable Laplace approximations in PyTorch, enabling more efficient implementations of all of its various forms and variants (from KFAC approximations to marginal likelihood optimization to different approximations at test time).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "tldr: I tend to weakly reject this paper because I'm really ambivalent about accepting what is essentially a research software package with experiments.\n\n## Originality\n\n*Are the tasks or methods new?* Not really, the software package is. The methods themselves have been around for a while.\n\n*Is the work a novel combination of well-known techniques?* The software package is.\n\n*Is it clear how this work differs from previous contributions?* The authors do a good job of distinguishing their implementation from others.\n\n*Is related work adequately cited?* Mostly, ref [71] is mis-cited and uses HMC (although they compare to SGMCMC as well), while the GGMC method of [Garriga-alonso and Fortuin, AABI, '20](https://arxiv.org/abs/2102.01691) probably also deserves a citation due to its recent popularity.\n\n\n## Quality\n\n*Is the submission technically sound?* Yes, I think so. The software package seems pretty well put together.\n\n*Are claims well supported (e.g., by theoretical analysis or experimental results)?* \n- Fig 5 is a pretty unfair comparison, as, deep ensembles, HMC and SWAG are using sampling (either during train / test or both), while the Laplace approximations studied use deterministic predictions. \n- The comparisons between (HMC, SWAG, and LA*) seem a lot more mixed for OOD detection from Table 1 than is depicted in the paper. It looks like LA* is mostly significantly less confident (some type of extra tuning?) than any method, but that this doesn't translate into better AUROC. \n\n*Are the methods used appropriate?* Mostly so given this space, a few comments:\n- I would have preferred to see comparisons to full Hessian / Fisher Laplace approximations on say MNIST since this should be doable (implement scalable Hessian vector or Fisher vector products and then use a low rank approximation). This shouldn't be too difficult in the authors' software framework and might be a convincing demonstration of the modularity a bit more.\n\nFrom what I can see, you either use a low rank KFAC Fisher approximation or a last layer one. A dense approximation would be helpful to see if it provides improved benefits.\n\n- Given the high amount of modularity in MC approximation, one pretty clear missing piece of comparison is that of probit approximation + SWAG (or Laplace bridge + SWAG but it Laplace bridge seems to be dominated by other methods so less necessary). After all, in Fig 5 the wall-clock time is devoted to the computational cost of the number of samples...\n- The comparisons to CSGHMC in the appendix should absolutely be moved to the main text, and shouldn't be buried that deep.\n\n*Is this a complete piece of work or work in progress?* Complete work.\n\n*Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?* Pretty much.\n\n## Significance\n\n*Are the results important?* Yes, if this gains steam, then it'll probably be very useful.\n\n*Are others (researchers or practitioners) likely to use the ideas or build on them?* Yes, releasing a software package for scalable laplace approximations should definitely be useful.\n\n*Does the submission address a difficult task in a better way than previous work?* Not really.\n\n*Does it advance the state of the art in a demonstrable way?* Yes, I think so.\n\n*Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?* \n\n## Discussion & Questions\n\nI'm very ambivalent about accepting this paper because almost the entire contribution is the software package. Included in the citations are references for research works that do, give or take maybe one experiment, most of the comparisons in this paper. My score is pretty much based on this ambivalence, with some concerns about comparisons addressed above.\n\nThinking some more about this, there's fairly few research software packages (essentially) that end up published particularly because it's so difficult to evaluate software packages. From my understanding, the few packages that have ended up published ([Autoconj](https://arxiv.org/abs/1811.11926), [neural-tangents](https://openreview.net/forum?id=SklD9yrFPS)) have included what is effectively state of the art engineering capabilities as part of their contributions.\n\nOverall, I really like the message of the paper and think that it could be a boon to the community to see this type of demonstration published. First, I like the comparisons demonstrating that something as straightforward as Laplace approximations (and really even last layer ones) can easily outperform variational methods. Second, I like the demonstration that MC sampling is often outperformed by deterministic probit style approximations.\n\n### Writing / Minor Comments\n\n- I really dislike that I'm saying this, but please editorialize your captions some more. Include a bit of takeaway for each figure caption --- at least \"what is the one sentence summary of this figure?\"\n- Example: Fig 7. Permuted MNIST: all Bayesian approaches outperform MAP estimation in continual learning, with LA-KFAC performing the best, closely followed by VOGN.\n- The caption on Fig 3 is especially rough.\n\n\n-------\n\n## Review of Software Package\n\nAfter discussion with the area chair, I decided to take a more detailed look at the software package, which seems to be of reasonably high quality. A quick review of the package itself is here:\n\nI thank the authors for taking the time to develop the package and for putting a bunch of different Laplace approximations all in the same place in a mostly modular fashion. However, I think that it's still pretty oriented around research code, rather than practicability, but this might be a personal feeling. Ultimately, I'm still not sure that people would use it outside of their current implementations of Hessian vector products or their current implementations of KFAC, without a lot of advertisement, although as a researcher, I'd probably take a look at it.\n\n- Can you please describe what the (presumably anonymized) backend `asdfghjkl` is?\n- While the implementation is described as modular, from a peek through the package, it's not clear to me how one would implement full rank Hessians via say the torch functional interface (https://pytorch.org/docs/stable/generated/torch.autograd.functional.hvp.html), rather than backpack? Would this require setting up an entirely different backend in order to do so?\n- Are there any restrictions on layers that the backpack backend has? I recall that originally backpack couldn't handle some types of convolutional layers (I believe).\n- As a comment, I find that the package is rather focused on KFAC style approximations, rather than other types of approximations (for example, those based around low rank approximations of the Hessian / Fisher defined via matrix vector multiplications or even layer-wise versions of these). Thus, it seems tricky to see how to extend beyond these types of approximations, making the package more limited in scope than is promised.\n- `torch.kron` is in pytorch now, might want to update to that.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "qkOJmIbL6pI", "reviewer": "Reviewer_R8DH", "summary": "The paper considers the problems of approximate Bayesian inference in deep learning. The paper argues that the Laplace approximation does not receive the attention it deserves in the community and that the Laplace approximation is a good trade-off between simplicity, computational complexity, and predictive performance. The paper then provides a review of the relevant theory and the relevant decision for setting up Laplace approximations in deep learning. The main contribution of the paper is a software package for computing Laplace approximations based on pre-trained model. The paper concludes with a large empirical study comparing Laplace approximations to a range of other relevant methods.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem - how to make Bayesian inference feasible for large-scale deep networks? I really enjoyed reading the paper. While the paper does not contain much novelty in terms of methodological development, I believe this paper and the software package could be of great interest to many researchers and practitioners in the field. \n\nThe first sections of the paper (introduction, background on the Laplace approx, description of the software) are in generally very well-structured, well-written, and easy to follow. However, I found the structure of the experimental section quite confusing and it took me a while to decipher the information because of this. For example, the authors refer to Figs. 3, 4 and 5 in section 4.1, but all the abbreviations used in the figures are not introduced before section 4.2. Besides, there are several experimental details that are unclear to me (see comments below). Therefore, it think the experimental section could benefit significantly from a restructuring.\n\nThe claims of the paper are substantiated to a reasonable degree based on theoretical and empirical evidence from the literature as well as by a fairly comprehensive empirical study based on several different datasets. However, it would have been even more interesting if the authors had chosen to include more adhoc methods like Monte Carlo Dropout, etc., and more advanced VI methods for completeness. But that being said, I think the empirical results are already quite extensive.\n\n\n\nOther comments:\n\n- Figure 3: What does the individual dots in Fig. 3 represent?\n\n- Figure 4: It is still a bit unclear what exactly the axis in (b) and (c) are showing. Please elaborate on this. How is the shift intensity defined and what is on the y-axis?\n\n- What is the definition of \"Confidence\" defined in Table 1?\n\n- In several experiments, the authors compared results for models with fixed hyperparameters vs learned hyperparameters. However, it is very unclear to me which parameters the authors consider to be in the set of hyperparameters. For example, which quantities are considered as hyperparameters in Fig 3? And how are they initialized when learning them through the approximate marginal likelihood?\n\n- The paper provides a set of general recommendations based on the specific experiments. It would be nice if the authors would include a discussion of how well those recommendations will generalize to other types of problems.\n\n- In line 97 the authors argue that it can be sufficient to approximate a subset of weights using the Laplace approximation because it has been shown that some networks can be pruned without sacrificing test accuracy. However, I fail to see how this justifies only modeling uncertainty for a subset of the weights. When pruning networks, you don't simply ignore the uncertainty - you completely remove the weights. Could the authors please elaborate a bit on this?\n\n**Post rebuttal**\nUpdated score from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Nb5sj29WzKV", "reviewer": "Reviewer_T5ck", "summary": "The paper discusses Laplace approximations in the context of approximate Bayesian inference for deep neural networks. \nThe Authors propose a new library, laplace, that allow practitioners to easily extend loss-trained models (MAP) to Bayesian. \nThe new library allows also the user to choose different approximations for the Hessian, different strategies for the Bayesian extension (e.g. by considering all or just a subset of weights) and a variety of procedures to choose/learn the hyper-parameters.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nMethodologically the contributions of this work are very thin as the Laplace approximation is a well known and simple way to do approximate Bayesian inference.\nEven the application of LA to deep neural networks is not new (see [21,25] in paper).\nAt the same time, I agree with the Authors when they argue on the poor adoption rate of LA, especially when compared against more popular variational and MCMC methods, and it's true that this might be due to missing reference implementations easily available.\n\nFor this reason, I have to positively evaluate the empirical benchmark, the analysis of different approximations and the source code (which is well documented and which promises to be very user-friendly).\nRealistically, this paper will not be groundbreaking in terms of contributions but it provides good baselines for future research in BDL.\n \nI was playing a bit with the code and the 1D regression example and overall it looks easy to tinker the different choices/parameters. \nI have found a major limiting design choice of the library, though: unless I've missed it in the code/documentation, it seems that only Gaussian prior can be used. Indeed it looks like the Gaussian prior is hard-coded in implementation of the approximation (and of the marginal likelihood).\nIn practice, there is evidence that heavy-tail priors (like student-t) can deliver better performance than Gaussian [a,b], and the fact that there is no option for the user to choose the prior is a problem. \nSame comment applies also for choice of the likelihood (only Gaussian and multi-class).\n\n## Minor\nOne minor remark: it would be better to have a minimal version for the requirements. I had an older version of PyTorch installed and some functions were not available, causing the script to fail (the setuptools didn't update the packages due to the missing minimal version).\n\n[a] Tran et al. All you need is a good functional prior.\n[b] Fortuin et al. Bayesian Neural Network Priors Revisited\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are only partially addressed.\nIn my opinion the is one important discussion that is missing and it regards the memory consumption, for CPU and GPU implementations. \nAll Bayesian methods are known to be extremely challenging for both timing and memory usage: the Authors do a good job from the timing perspective (see Figure 4) but they neglect to address this other point of view.\nAs a future practitioner, especially if I'm working with constrained resources, e.g. very small GPU or edge devices, I would be very interested in a proper comparison between approximations/performance versus memory usage.\nThis would be very helpful to really understand all the different trade-off I should consider for if I intend to use LA. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "7P8AL7htVSV", "reviewer": "Reviewer_oDXG", "summary": "The authors bring to light the recent progress in approximate Bayesian inference via the Laplace approximation, and note that despite its simplicity, its adoption has not kept pace for modern deep learning methods. The work describes a consolidation of recent work into a user-friendly PyTorch-based software library called `laplace`. LA can be applied post-hoc to existing deep learning models. With this package in place, the authors provide practical recommendations for reasonable defaults when used in practice.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is very well-timed given the modern evidence of Laplace approximations for Bayesian inference being effective. As the authors identify, despite its conceptual simplicity, the developments have not kept pace with modern times in terms of easy-to-use software packages. In that sense, this is much appreciated. I have specific comments and question below.\n\n- In terms of writing, while the introduction is clear, but for a paper that aims to convince the reader about the simplicity of implementing scalable LA, Section 2 does not inspire confidence. I think this may only be a matter of organizing and contextualizing the approximations available better. \n\n- To me, the most important part of the paper is the experiments section. I think the most important aspect missing in the study is the sensitivity of the approximations. How sensitive is the final performance to the Hessian approximations? Clearly, this has to be consequential?\n\n- Something that would also be interesting to see with this broad-scoped analysis is whether we see the Occam's razor effect during the marginal likelihood optimization still happen even with the approximation?\n\n- Another important point worth noting is that Multi-SWAG observed improvements using mutli-modal approximations of SWAG. Have the authors investigated this with Laplace approximation? Given that Laplace approximation is a finer approximation of the rather empirical curvature approximation with SWAG, do we really get gains there? Is it worth using LA to more finely capture geometry?\n\nOverall, this is well-timed paper to remind the community of important progress in scalable LA. I do think, however, that there are major missed opportunities in terms of the qualitative analysis of the method beyond benchmarking (which is no doubt diverse and reasonably exhaustive), especially in terms of the sensitivity and other qualitative improvements that LA provides over Deep Ensembles or SWAG. I will vote for a weak accept, but I would be more than happy to be convinced otherwise since this is an important effort towards consolidation of scalable LA.\n\n### Minor Comments\n\n- For Figure 3, it may be helpful to the eye to have both graphs with the same scales to make comparison visually easier.\n- It would be good have a detailed caption summarizing the message in Figure 4 (and if space is a constraint, perhaps move bulk of the explanation to the figure).\n- Lines 31-32 should mention SWAG [1], which appears to satisfy good empirical performance at the cost of a single network. The authors do compare to SWAG later as well anyways.\n\n[1]:: https://arxiv.org/abs/1902.02476\n\n---\n\n**UPDATE**: See comment below. Updating score from 6 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A well-written paper proposing a software package.  The paper additionally gives a compact review on Laplace approximation (LA) for DNN  and benchmark experiments where LA works better or is comparable to more accurate approximation methods.  The software is easy to use but reviewers have concerns about its flexibility (e.g., prior is hardcoded and cannot be changed) and maintainability.  We hope that the authors would try to keep "Laplace" useful for the community.
Decision: Accept (Poster)