Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift
Authors: Florian Seligmann
Philipp Becker
Michael Volpp
Gerhard Neumann
Venue: NeurIPS 2023 poster (2023)
Keywords: bayesian deep learning
distribution shift
calibration
TL;DR: We present a large-scale evaluation of modern, scalable Bayesian deep learning algorithms under realistic distribution shift with a focus on ensembles of single-mode posterior approximations.

Abstract: Bayesian deep learning (BDL) is a promising approach to achieve well-calibrated predictions on distribution-shifted data. Nevertheless, there exists no large-scale survey that evaluates recent SOTA methods on diverse, realistic, and challenging benchmark tasks in a systematic manner. To provide a clear picture of the current state of BDL research, we evaluate modern BDL algorithms on real-world datasets from the WILDS collection containing challenging classification and regression tasks, with a focus on generalization capability and calibration under distribution shift. We compare the algorithms on a wide range of large, convolutional and transformer-based neural network architectures. In particular, we investigate a signed version of the expected calibration error that reveals whether the methods are over- or underconfident, providing further insight into the behavior of the methods. Further, we provide the first systematic evaluation of BDL for fine-tuning large pre-trained models, where training from scratch is prohibitively expensive. Finally, given the recent success of Deep Ensembles, we extend popular single-mode posterior approximations to multiple modes by the use of ensembles.   While we find that ensembling single-mode approximations generally improves the generalization capability and calibration of the models by a significant margin, we also identify a failure mode of ensembles when finetuning large transformer-based language models.
  In this setting, variational inference based approaches such as last-layer Bayes By Backprop outperform other methods in terms of accuracy by a large margin, while modern approximate inference algorithms such as SWAG achieve the best calibration.

Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift
Florian Seligmann" *
Philipp Becker
Karlsruhe Institute of Technology
Karlsruhe Institute of Technology
FZI Research Center for Information Technology
Michael Volpp
Gerhard Neumann
Karlsruhe Institute of Technology Bosch Center for Artificial Intelligence
Karlsruhe Institute of Technology
FZI Research Center for Information Technology
Abstract
Bayesian deep learning (BDL) is promising approach to achieve well-calibrated predictions on distribution-shifted data. Nevertheless, there exists no large-scale survey that evaluates recent SOTA methods on diverse, realistic, and challenging benchmark tasks in systematic manner. To provide a clear picture of the current state of BDL research, we evaluate modern BDL algorithms on real-world datasets from the WILDS collection containing challenging classification and regression tasks, with focus on generalization capability and calibration under distribution shift. We compare the algorithms on wide range of large, convolutional and transformer based neural network architectures. signed version of the expected calibration error that reveals whether the methods are over- or underconfident, providing further insight into the behavior of the methods. Further, we provide the first systematic evaluation of BDL for fine-tuning large pre-trained models, where training from scratch prohibitively expensive. Finally, given the recent success of Deep Ensembles, we extend popular single-mode posterior approximations to multiple modes by the use of ensembles. While we find that ensembling single-mode approximations generally improves the generalization capability and calibration of the models by a significant margin, we also identify failure mode of ensembles when finetuning large transformer-basse models. In this setting, variational inference based approaches such as last-layer Bayes By Backprop outperform other methods in terms of accuracy by large margin, while modern approximate inference algorithms such as SWAG achieve the best calibration.
particular, we investigate a
language
Introduction
Real-world applications of deep learning require accurate estimates of the model's predictive uncer- tainty [2, 21, 38]. This is particularly relevant in safety-critical applications of deep learning, such as medical applications [95] and self-driving cars [27]. Therefore, we want our models to be calibrated: model should confident about its prediction if and only if the prediction will likely be correct. Only then it is sensible to rely on high-confidence predictions, and, e.g., to contact a human expert in the low-confidence regime [4]. Calibration is particularly relevant when models are evaluated on
(o.o.d.) data, i.c. inputs that very differrnnt from the training data, and hence, the model cannot always make accurate predictions. However, typical deep neural networks are highly overconfident on o.o.d. data [28, 72].
"Conresoondence to florian.seligmanee @ wit.deet..iteddu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Bayesian deep learning (BDL) promises to fix this overconfidence problem by marginalizing over the posterior of the model's parameters. This process takes all explanations that are compatible with the training data into account. As desired, explanations will disagree on o.o.d. data, so that predictions will have low confidence in this regime. While computing the exact parameter posterior in BDL is infeasible, many approximate inference procedures exist to tackle this problem, aiming at making BDL applicable to real-world problems. Yet, recent BDL algorithms are typically only evaluated on the comparatively small and curated MNIST [52]. UCI [16], and CIFAR [48] datasets with artificial o.o.d. splits. Existing BDL surveys [19, 20, 29, 72] concentrate on a few popular but relatively old algorithms such as Bayes By Backprop. Deep Ensembles, and Monte Carlo Dropout. In the light of recent calls for more realistic benchmarks of ofute-of-the-ar (SOTA) algorithms [1] with some experts going as far calling the current state of BDL a "replication crisis" we aim to provide a large-scale evaluation of recent BDL algorithms on complex tasks with large, diverse neural networks.
Contributions. i) We systematically evaluate a comprehensive selection of modern, scalable BDL algorithms on large image- and text- classification and regression datasets from the WILDS collection [47] that originate from real-world, safety-critical applications of deep learning (Section 5). In the spirit of Ovadia et al. [72], we focus generalization capability and calibration on o.o.d. data, but consider more diverse and modern algorithms (Section 3) on realistic datasets with distribution shift. In particular, we include recent advances in variational inference such as natural gradient descent (iVON [53]) and low-rank posterior approximations (Rank-1 VI [17). Furthermore, we use modern neural network architectures such as various ResNets [32], a DenseNet [35], and transformer architecture [87]. ii) We present the first systematic evaluation of BDL for finetuming large pre-trained models, setting that has recently gained attention in the context of BDL [79, 81]. We show that using BDL for finetuning gives significant performance boost across a wide variety of tasks compared standard deterministic finetuning (Section 5). iii) Inspired by the success of Deep Ensembles [51], we systematically evaluate the benefit of ensembling single-mode posterior approximations [4] (Section 5). iv) We use a signed extension of the expected calibration error (ECE) called the signed expected calibration error (sECE) that can differentiate between overconfidence and underconfidence, allowing us to better understand in which ways models are miscalibrated (Section 4), v) We compare the posterior approximation quality of the considered algorithms using the HMC samples from Izmailov et al. [37] (Section 5) and show that modern single-mode BDL algorithms approximate the parameter posterior better than Deep Ensembles, with further gains being achieved by ensembling these algorithms. Overall, our work is similar in spirit to Band et al. [4], but we compare the algorithms more diverse datasets and focus pure calibration metries, thereby revealing failure modes of SOTA BDL algorithms that are not yet present in the literature. We provide code for all implemented algorithms and all evaluations
2 Related Work
Several recent publications [1, 25] review the SOTA in uncertainty quantification using Bayesian models without providing experimental results. Yao al. [93] compare wide range of Markov Chain Monte Carlo [31] and approximate inference [6] methods on toy classification and regression datasets. Ovadia et al. [72] perform large-scale experimental evaluation of small selection of popular BDL. algorithms on o.o.d. data and conclude that Deep Ensembles [51] perform best while stochastic variational inference [26] performs worst. Filos al. [19] use a similar selection of algorithms but only evaluate on a single, large computer vision task not considering o.o.d. data. Foong et al. [20] artificially create o.o.d. splits for UCI datasets [16] and again find that variational inference performs worse than the Laplace approximation [57]. Mukhoti and Gal [63] and Gustafsson et al. [29] compare Monte Carlo Dropout [22] and Deep Ensembles in the context of semantic segmentation and depth completion, but, again, do not consider o.o.d. data. Nado et al. [65] evaluate many popular BDL algorithms on a small number of mostly artificially created o.o.d. image and text classification tasks. The work of Band et al. [4] is the most similar to ours, as they evaluate several BDL. algorithms, including ensembles of single-mode posterior approximations, on two large image- image-classificall classification datasets and consider o.o.d. data. Compared to Band et al. [4], we evaluate different set of algorithms such as SWAG [59] and natural gradient descent variational inference [53]
bettps://githul/ http:///ggthuuu.ccc/eeeeeeeeeeee htw.ecawwcccoo/eeeeeeegg///////e Deep.deeeeeeeeeeeeeeeeee Ensembles

---

on a more diverse selection of datasets and network architectures, including transformer-basee models and finetuning tasks, thereby revealing new failure modes of SOTA BDL methods. Competitions such as Wilson et al. [90] and Malinin et al. [60] also provide insights into the performance of different algorithms. However, the employed algorithms are typically highly tuned and modified for the specific tasks and thus of limited use to assess the general quality of the underlying methods in more diverse settings. Importantly, all winners Wilson et al. [90] use ensemble-based algorithms.
3 Bayesian Deep Learning Algorithms
We assume a neural network with parameters that models the likelihood p(y x. 0) of an output y given input æ. Treating 0 as random variable, the parameter posterior p(0) DD) given training dataset 'D {(2r,, yi) I,....N} of input-output pairs is defined by Bayes' theorem as
P1: T., p()) JTL, plu: a 0) p(0) do
p(0 D)
where p(0) a prior over parameters. The posterior p(0 D) assigns higher probability to parameter vectors that fit the training data well and conform to our prior beliefs. Using p(0 D), prediction given an input vector æ is defined as
/ plu x. 0) p(0 D) do
ply x. D) =
E 0~p(0//)
[p(y x, 0)]
This so-called Bayesian model average (BMA) [6, 58, 64, 89] encompasses the information of all explanations of the training data that are consistent with the parameter posterior. The BMA is especially valuable when dealing with large neural networks that are typically underspecified by the training data, where marginalizing over parameters can mitigate overfitting and promises significant accuracy and calibration gains [89]. While recent work has shown problems with certain types of covariate shift [36] and o.o.d. data [11]. BDL not only promises calibration but also generalization gains [89].
3.1 Scalable Approximations for Bayesian Deep Learning
As computing the marginalization integral defin- ing the normalization constant the parame- ter posterior (Equation (1)) is intractable for
(00D
Heppy
neural networks, we have to resort to approx- imations. Approximate inference algorithms ap- proximate the posterior, either by sampling from it by computing approximate distribution. Sampling-basee Markov Chain Monte Carlo (MCMC) methods [31] such as Hamiltonian Monte Carlo (HMC) [67] sample directly from the true posterior and are therefore asymptoti- cally exact. However, they are computationally very expensive and hence typically intractable in the context of BDL. Deterministic methods such variational inference construet local approx- imations at a mode of the parameter posterior and are generally more computationally perfor- mant than MCMC [37]. as they transform the posterior inference problem into an optimization problem that can be efficiently solved with stan- dard gradient- based optimization techniques [6, 58, 64]. Therefore, we focus on these algorithms in this work. This framework also encompasses standard deep learning, which is equivalent to a "Maximum A Posteriori" (MAP) estimate, i.e., point estimate the posterior maximum. In this section, we give a brief overview of the algorithms that we evaluate. See Appendix for more detailed explanations and Appendix D for implementation details.
(a) MAP
(b) Singlc-Mode
(c) Ensemble
MultiX
Figure 1: Posterior Approximation Types. MAP approximates a single posterior mode with point estimate, while probabilistic singlc-mode approxi- mations additionally capture the shape of the mode. Deep Ensembles approximate multiple modes with a mixture of point estimates. Likewise, MultiX employs mixture of single-mode approximations to capture the shape of multiple modes. Figure adapted from Wilson and Izmailov [89].
3

---

Variational Inference. Variational inference (VI) minimizes the Kullback-Leibler divergence [50]
between the approximate posterior and the true posterior [26]. Bayes By Backprop (BBB) [7]
approximates the posterior with diagonal Gaussian distribution and optimizes the mean and variance parameters with Stochastic Gradient Descent (SGD) [42]. Rank-1 variational inference (Rank-1 VI)
[17] in contrast uses low-rank posterior approximation, which reduces the number of additional parameters and allows the use of multiple components in the low-rank subspace. The improved Variational Online Newton (IVON) algorithm [53] still uses diagonal Gaussian posterior but uses second-order information to better optimize the distribution parameters with natural gradients. Stein Variational Gradient Descent (SVGD) [55] is non non-parametri. VI algorithm that approximates the posterior with multiple point estimates. SVGD is similar to a Deep Ensemble (see below) but adds repulsive forces between the particles to push them away from each other in parameter space.
Other Algorithms. Lakshminarayanan et al. [51] introduce Deep Ensembles that approximate the posterior with a few, typically five to ten, independently trained MAP models. As such, Deep Ensembles were originally considered competing approach to Bayesian models [51] but can be viewed as Bayesian as they form sum of delta distributions that approximate the posterior [89]. We follow this interpretation. The Laplace approximation [57] approximates the posterior with second- order Taylor expansion around the parameters of a MAP model. We only consider the last-layer Laplace approximation [13] with diagonal and Zronecker-factoriae which Daxberger et al. [13] find to achieve the best tradeoff between performance and calibration. Monte Carlo Dropout (MCD) [22] utilizes the probabilistic nature of dropout units that are part of many common network architectures to construct an approximation of posterior mode. Stochastic Weight Averaging- Gaussian (SWAG) [59] periodically stores the parameters during SGD training and uses them to build low-rank Gaussian posterior approximation. Finally, we evaluate Spectrally- Normalized Gaussian Processes (SNGP) [54] a Bayesian baseline that does not infer distribution over the model's 's parameters, but replaces the last layer by a Gaussian Process. The results for SNGP therefore not directly comparable to the other algorithms' results.
[74] posterior approximations,
3.2 MultiX
While single-mode posterior approximations such as BBB and SWAG capture the shape of a single mode of the parameter posterior, Deep Ensembles cover multiple modes but approximate each with a single point estimate. Hence, ensembling single-mode approximations promises
better posterior coverage and therefore improved uncertainty estimates (see Figure 1). This concept is not new: Tomezak al. [86] experiment with ensemble of BBB models small datasets. Cobb al. [10] use an ensemble of Concrete Dropout [23] models and Filos et al. [19] use MCD models. Both report accuracy improvements compared Deep Ensemble. Wilson and Izmailov [89] introduce MultSSWAG, an ensemble of SWAG [59] models. The winning teams of Wilson et al. [90] also show that ensembling Bayesian neural networks yields good posterior approximations. Similar Band et al. [4] and Mehrtens et al. [61], we ensemble all considered single-mode posterior approximations (Section 3.1) except for SNGP assess the performance gains on algoritht basis. We use the term "MultiX" to refer to an ensemble of models trained with algorithm "X". We make an exception for "MultiMAP", which we keep referring as Deep Ensemble for consistency with the existing literature.
4 Calibration Metries
A calibrated model is defined model that makes confident predictions if and only if they will likely be accurate. While this definition directly implies a calibration metric for classification tasks [66]. it has to be adapted for regression tasks, "being accurate" is not binary property in the regression case.
4.1 Unsigned Calibration metrics
Calibrated Classification. The calibration of a classification model can be measured with the expected calibration error (ECE) [28, 66). By partitioning the interval [0, 1] into M equally spaced bins and grouping the model's 's predictions into those bins based on their confidence values, we can calculate the average accuracy and confidence of each bin. The expected calibration error is then

---

given by ECE = Σm= (Bml/10') (tll/p"jjacc((,laa (m)) conf(Bm)) where Bn, is the set of predictions in the m-th bin, and acc (n,) and conf Bm) are the average accuracy and confidence of the predictions in Bm (see Appendix B for details). An ECE of zero indicates perfect calibration.
Calibrated Regression. The confidence intervals of the predictive distribution can be used to measure the calibration of regression model. Selecting M confidence levels Pm allows the compu- tation of a calibration error based on the observed probability (pn calculated as the fraction of predictions that fall into the Pm -moffiieeeece interval of their respective predictive distributions: QCE = 1/as m= (Pots(mm) - pm We refer this as the quantile calibration error (QCE), which simply replaces the quantiles in the definition of the calibration error from Kuleshov et al. [49] by confidence intervals. Using the confidence intervals allows a simpler interpretation of the resulting reliability diagrams (see Appendix B).
4.2 Signed Calibration Metrics
Models can be miscalibrated in two distinct ways: Overconfident models make inaccurate predictions with high confidence, and underconfident models make accurate predictions with low confidence. Arguably, overconfidence is worse in practice when applicants want to rely on the model's confidence to assess whether they can trust prediction, for example in safety-critical applications of deep learning. However. none of the presented metries can differentiate between overconfidence and underconfidence. Until now, this information was only apparent in reliability diagrams [28]. We propose two simple extensions of the ECE and the QCE that condense the information about overconfidence and underconfidence into single scalar value by removing the absolute values: SECE and sQCE. We define these signed calibration metrics as
M
|mml D
M Σ (Pobs(Pm) - Pmm)-
SECE Σ
(acc(Bm)) -con)) Bm)) and sQCE M
positive signed calibration error indicates that model makes predominantly underconfident predictions and negative signed calibration error indicates predominantly overconfident predictions. Perfectly calibrated models have a sECE/sQCE of zero. For models that are overconfident for some inputs but underconfident for others the signed calibration metrics may be zero, even though the model is not perfectly calibrated. This typically not an issue in practice, as our experiments in Appendix C. show that the absolute value of the signed metrics is usually very close to the absolute value of the corresponding unsigned metric, as most models are either overconfident underconfident for nearly all predictions. Nevertheless, we always report the signed calibration metrics together with the unsigned calibration metrics to avoid any ambiguity.
5 Empirical Evaluation
For our comparison of the BDL algorithms introduced in Section 3.1, we focus on i) the ability of the models to generalize to realistic distribution data, ii) the calibration of the models under distribution shift, and iii) how well the models approximate the true parameter posterior. To assess the generalization capability and calibration of the models under realistic distribution shift, use a subset of the WILDS dataset collection [47]. We assess the posterior approximation quality by comparing the model's predictive distributions to those of the HMC approximation provided by Izmailov et al. [37] for CIFAR-10 [48]. See Section 5. and Section 5.2 for details about the task, Section 5.3 for generalization results, Section 5.4 for calibration results, and Section 5.5 for posterior approximation quality results. All results for WILDS are directly comparable to the respective non-BDL o.o.d. detection algorithms on the WILDS leaderboard since we strictly follow their training and evaluation protocol. Whenever the BDL models' accuracy is competitive with the best performing algorithm on the WILDS leaderboard, the WILDS result is marked in the respective plot. We also report results for a subset of the smaller UCI [16] and UCI-Gap [20] tabular regression datasets in Appendix G. Appendix F contains information about the used computational resources and training times. Details regarding hyperparameters, training procedures, and additional results can be found in Appendix G. The results on all datasets are reported with a 95% confidence interval.
'https:/ htttps::/wiids.ss stanford..d....... wwt.eeaaeebbrrrd//////// last accessed August 31, 2023
5

---

5.1 The WILDS Datasets
WILDS consists of ten diverse datasets that originate from real-world applications of deep learning in which models need to perform well under distribution shift. Standard o.o.d. datasets such as MNIST- C [62]. CIFAR- 10-C [33] and UCI-Gap [20] create distribution-shiin
data by selectively removing data from the training split or artificially adding data corruptions onto the data in the evaluation split. WILDS represents real-world distribution shifts and is therefore more suitable for an application- oriented evaluation of BDL. We systematically evaluate all considered algorithms (see Section 3. on six of the ten datasets: The image-basee based regression task POVERTY MAP, the image classification tasks [WILDCAM, FMOW, and RxRxI, and the text classification tasks COMMENTS and AMAZON. Aside from POVERTYMAP, all datasets are finetuning tasks, where we initialize the model's parameters from model that has been pre-trained a similar task. We also evaluate some algorithms on the CAMELYON 7 image classification dataset but find that the performance degradation on the o.o.d. evaluation split is to large part consequence of the use of batch normalization rather than the o.o.d. data, making the dataset less interesting for a fair comparison on o.o.d. data (see Appendix E. for details).
As we want to evaluate the posterior approximation, generalization, and calibration capability of all models given the true parameter posterior, none of our models use the metadata (e.g. location, time) associated with the input data, nor do we consider approaches that are specifically designed for o.o.d. generalization or augment the dataset, for example by re-weighting underrepresented classes, contrary to the algorithms evaluated by Koh et al. [47].
Large-Scale Regression. POVERTYMAP-WI.S WIIDDD [94] is an image-based regression task, where the goal to better target humanitarian aid in Africa by estimating the asset wealth index of an area using satellite images, As the task is significantly easier when buildings are visible in the images, the evaluation set is split into images containing urban and images containing rural areas. The accuracy of the models is evaluated on both splits by the Pearson coefficient between their predictions and the ground truth, and the worst Pearson coefficient is used as the main evaluation metric. All models are based on a ResNet- 18 [32]. See Figure for the Pearson coefficient and sQCE on the o.o.d. evaluation split and Appendix G.3.2 for further details.
Finetuning of CNNs. IWILDCAM-WILDD [5] is an image classification task that consists of animal photos taken by camera traps across the world. The model's task is to determine which of 182 animal species can be seen in the image. As rare animal species, which are of special interest to researchers, are naturally underrepresented in the dataset, the macro Fl score is used to evaluate the predictive performance. The o.o.d. evaluation split consists of images from new camera locations. All models are based on ResNet-50 [32]. See Figure 2a for the macro F1 score and sECE on the o.o.d. evaluation split and Appendix G.3.3 for further details. FMOW-WILDS (Functional Map of the World) [9] is an image classification task, where the inputs are satellite images and the class is one of 62 building and land use categories. The o.o.d. evaluation split consists of images from different years than the images in the training set. Models are separately evaluated on five geographical regions of the world, with the lowest accuracy taken as the main evaluation metric. All models are based on a DenseNet- 121 [35]. See Figure 2b for the accuracy and SECE for the region of the o.o.d. evaluation split the models perform worst on and Appendix G.3.4 for further details. RXRxI-WIILD [85] is an image classification task, where the inputs are three-channel images of cells, and the classes are 1 1139 applied genetic treatments. The o.o.d. evaluation split is formed by images from different experimental batches than the training data. Following Koh et al. [47], we only use three of the six available input channels to limit the computational complexity of the models. This makes the task considerably harder and leads to the low accuracy of the models, but makes our results comparable to those of Koh et al. [47]. All models are based on ResNet-50 [32]. See Figure 2c for the accuracy and SECE on the o.o.d. evaluation split and Appendix G.3.5 for further details.
F Finetuning of Transformers. where the model's task is to classify whether a given comment is toxic or not. The comments are grouped based on whether they mention certain demographic groups, such as LGBTQ or Muslim identities. Models are evaluated based on the group on which they achieve the lowest accuracy on the evaluation set. All models are based on the DistilBERT architecture [76]. See Figure 4a for the accuracy and sECE on the group of the o.o.d. evaluation split the models perform worst on and Appendix G.3.6 for further details. AMAZON-WILDD [68] consists of textual product reviews, where
WWW.CCOMEETS-WILDD [8] is binary text classification dataset,

---

-13
1000
-03
-15
0.22
1.20
0.25
10.0
0.22
10.4
1.30
1328
100
5.11
0.1
Scare
Ascenay Beginel
Accansy
(a) /WILDCAM-W WILDS
(b) IMOW-WILDS
(c) RXRXII-IILD
MAP
MCD
SWAG
BBB
Loglace
IVON
Rank-I VI
SNGP
Deep Ensemble
MulaMCD
MaldSWAG
MulaBBB
MultiLaplaze
MuliiVON
SVCD
WILDS Leader
Figure 2: Accuracy Metries SECE the o.o.d. evaluation splits of the image classification finetuning tasks WVI.DCAM-WII.D FMOW-WILDS, and RxRx1- Note the split y-axis in Figure 2c. All MultiX algorithms approximation. Except for RxRx1, BBB is better calibrated than MCD and SWAG. iVON's calibration is inconsistent: On (WILCCAM is better calibrated than BBB, but FMOW it barely performs better than MAP. On RxRx1, the VI algorithms except for SVGD are significantly less accurate than all other models. We experimented with different hyperparameters Appendix G.3.5. Laplace is very well calibrated on IWILDCAM, but underperforms on FMoW and RxRxI. SVGD performs very similarly to MAP regarding both metrics, even though it uses multi-mode posterior approximation.
accurate and better calibrated than any single-mode
the task is predict the star rating from one five. The o.o.d. evaluation split consists of reviews from reviewers that are not part of the training split. Models are evaluated based on the accuracy of the reviewer the 10% quantile. All models are based DistilBERT [76]. See Figure 4b for the accuracy and SECE on the o.o.d. evaluation split and Appendix G.3.7 for further details.
5.2 The Corrupted CIFAR-10 Dataset
CIFAR-10- [33] is a corrupted version of the evaluation split of the image classification dataset CIFAR- [48], where images are corrupted with increasing levels of noise, blur, and weather and digital artifacts. We compare the considered algorithms on the standard evaluation split of CIFAR- 10 as well as the corruption levels 3, and of CIFAR-10-C Following Izmailov et [37], all of our models on CIFAR-10-CC are based on the ResNet-20 architecture. See Appendix G.3.6 for details.
5.3
Generalization to Realistic Distribution Shift
We measure the generalization capability of the models with the task-specific accuracy metrics proposed by Koh et al. [47] that are based on the real-world origin of the respective tasks. The metrics typically emphasize the performance on groups classes that are underrepresented in the training data, as avoiding bias against these groups is crucial in safety-critical applications of BDL.
Except for the text classification tasks, MultiX always generalizes better than single-mode poste- rior approximations. Overall, the relative ordering of the MultiX models depends on the dataset and in many cases does not correlate with the relative ordering of the corresponding single-man mode approximations.
Large-Scale Regression. All models achieve similar Pearson coefficients, with MultiX being slightly more accurate. The Deep Ensemble is competitive with the best performing algorithm the WILDS leaderboard with a Pearson coefficient of 0.52 compared to 0.53 of C-Mixup [92]. However, due to the large standard errors resulting from the different difficulties of the folds, the results are not significant. Note that Koh et al. [47] report similarly large standard errors.
Finetuning of CNNs. Confirming the overall trend, MultiX models generalize better than single- mode models, with MultiSWAG and MultMCD performing particularly well. Except for RxRx1 the
7

---

models perform competitively with the best models from the WILDS leaderboard. On IWILDCAM, the single-mode posterior upproximations SWAG and MCD are competitive with the Deep Ensemble. SVGD performs similarly to MAP, even though it is based on an ensemble, likely due to the repulsive forees pushing the particles away from the well- performnng pre-trained model. While the VI algorithms' accuracy is similar to the accuracy of MAP on IWILDCAM and FMOW, all VI algorithms except SVGD perform significantly worse than the non-V VI algorithms on RxRxI. Laplace is well calibrated on IW IWILDCAM, but significantly less accurate than the other algorithms on FMOW and RxRxI. This seems to represent a fundamental approximation failure of Laplace, and not only a sampling issue, since increasing the number of samples lead to only small increase in accuracy (see Appendix G.3.4 and Appendix G.3.5).
Finetuning of Transformers. BBB and Rank- I VI are the most accurate models on both tasks, with no benefit from the multiple components of Rank-1 VI. Interestingly. iVON is significantly less accurate than BBB, even though it is also based on mean-field VI, indicating that the nat- ural gradient- based training is disadvantageous on the transformer- based BERT architecture. To see whether the better performance of BBB is due to less regularization compared MAP, also experiment with a smaller weight decay factor for MAP CIVI.COMMENT While we find that the accuracy increases, BBB is still more accurate (see Appendix G.3.6). Finally, we also check whether the better performance of BBB is due to its last-layer nature. We ex- periment with last-layer versions of MCD and SWAG on AMAZON (see Appendix G.3.7), but find that both are still significantly less accurate than BBB.
MAP
Deep Ensemble
-022
MCD
Mutr'MCC
SWAG
-0.25
MutiS WAG
BBB
-033
MulaBBB
Rank-1 VI
-0.35
Laplace
Multil aphace
SVGD
-0.4
SNGP
0.45
0.55
WILDS
Wors U/R Pearson
POVERTY MAP-WILDS: Worst ur- ban/rural Pearson coefficient between the model's predictions and the ground truth plotted against the sQCE on the o.o.d. test split of the image-basee re- gression task. All models achieve similar, but noisy [47], Pearson coefficients, indicating similar gener- alization capabilities. Multi-mode approximations are consistently better calibrated than single-mode approximations (note that Rank-1 VI's components and SVGD's particles give them multi-mode ap- proximation capabilities). Regarding calibration, the relative ordering of the single-mode models does not translate to the MultiX models: BBB is among the best-calibrated single-mode models, but MultiBBB is the worst calibrated MultiX model. Laplace and SWAG are very similarly calibrated, therefore the data points of SWAG are hidden be- hind the data points of Laplace. iVON performs significantly worse than the other algorithms and therefore excluded.
Figure 3:
MultiX is no more accurate than the correspond- ing single-mode approximation, contrary to the results on all other datasets. We suspect that this effect is to large part due the finetuning nature of the tasks, where all ensemble members start close each other in parameter space and therefore converge to the same posterior mode. Note that the failure of ensembles is most likely due to the task and the network architecture and not due the training procedure: While we train for fewer epochs than on the image classification tasks, the datasets are larger. On IWILDCAM we perform 97k parameter updates, compared to 84k parameter updates on CIVI.COMMENTS.
5.4 Calibration under Realistic Distribution Shift
We measure calibration with the sECE for classification tasks and with the sQCE for regression tasks (see Section 4). We additionally report the unsigned ECE/QCE and the log-likelihood for the regression task in Appendix G.
MultiX is almost always less overconfident than single-mode approximations. When all models are already comparatively well calibrated, MultiX tends to become underconfident. Thus, we find that MultiX is typically only less confident, but not automatically better calibrated than songle-mode approximations. On the transformer-basee text classification tasks, MultiX is almost never better calibrated than the respective single-mode approximation.
8

---

We experi- Large-Scale Regression. MultiX, when based on probabilistic single-mode approximation, is generally better calibrated than the Deep Ensemble. SVGD is better calibrated than the Deep Ensemble, showing the benefit of the repulsive forces between the particles. Rank-1 VI is the best calibrated model, indicating that multi-modality over all parameters as with the Deep Ensemble is not necessary to capture the multi-modality of the parameter posterior.
Finetuning of CNNs. Again, we find that MultiX gen- erally performs better than a Deep Ensemble. However, MultiSWAG in particular is more overconfident than the Deep Ensemble, even though SWAG is better calibrated than MAP. BBB is better calibrated than other single- mode approximations such as SWAG and MCD. Rank-1 VI performs similar to BBB on all tasks, indicating that the low-rank components are not sufficient to capture the multi- molliiy of the parameter posterior in the finetuning setting. On IWILDCAM, Laplace is the best calibrated single-mode approximation, and correspondingly Multi- Laplace is the most underconfident multi-mode approxi- mation. This result is unique to IWIIDCAM, as Laplace tends to be overconfident on the other image classification datasets.
MAP
Deep Ensemble
MCD
MulEMCD
SWAG
MultiSWAG
Laplace
Multil Muluiaplace
BBB
MulrBBB
IVON
MultiVON
Rank-1 VI
SVGD
SNGP
WILDS Leader
0.2
<ECE Woml Grup)
-0.55
-0.
3.36
Finetuning of Transformers. Except for MultiBBB on AMAZON, ensembles are similarly calibrated than the spective single-mode approximations. SWAG is the least confident model on both tasks, which leads to undercon- fidence on AMAZON. MCD's s calibration is inconclusive, as it better calibrated than MAP on AMAZON, but more overconfident on VI are not better calibrated than MAP and on AMAZON significantly more overconfident than MAP.
-014
-0.45
Accurcy (Worst Group)
(a) CIVII D.VI..COMENTS-WLLL.
BBB and Rank-1
0.05
5.5 Posterior Approximation Quality
50C0
While approximate inference is commonplace in BDL, the large size of the neural networks typically makes it computationally intractable to measure how well model approximates the true parameter posterior. Following Wil- son et al. [90] and using the HMC samples provided by Izmailov et al. [37], we measure how well the models ap- proximate the predictive distribution of HMC by the total variation (TV) between the model's predictions and HMC and the top-1 agreement with HMC on CIFAR-10-(C) [33, 48]. While the TV in the predictive space indica- tive of the parameter posterior approximation quality, it does not allow for definite conclusions about the param- eter space. Figure 5 displays the TV of the evaluated models under increasing levels of image corruption. For further results regarding the accuracy, SECE, ECE, and 1 agreement with HMC see Appendix G.2.
0.44
0.45
0.52
100% Accuracy 1
(b) AMAZON-WILDS
Figure 4: Text classification with pre- trained transformers. Except for Multi- BBB on AMAZON-WILDS, MultiX per- forms nearly identically to the corre- sponding single-mode approximation. VI improves the accuracy of the mod- els. MCD the least accurate model on CIVIL COMMENTS ment with different dropout rates in Ap- pendix G.3.6 but find that MCD never outperforms MAP.
Overall, good probabilistic single-mode approximation the most important factor for good posterior approxi- mation. MultiX, when based on probabilistic single-mode approximations, consistently approximates the parameter posterior better than single-mode-only approximations and the Deep Ensemble. MultiVVON approxi- mates the posterior best across all corruption levels as measured by the TV, with WultiSWAG being close contender. Even single-mode approximations such as MCD and SWAG achieve better TVs

---

MAP
MCTD
SWAC MukuSWAG
BRR
Eaplace
NDN MalaaVON
Rank-1 VI
SNOP
Essemble
M
Cocmuttion
0.12
0.14
0.16
0.15
0.2
0.22
U.24
0.26
0.28
0.3
0.32
0.34
0.36
0.38
Total Variation HMC
Figure 5: CIFAR-IO-(C): Total variation (TV, smaller is better) between the model's predictions and HMC's predictions on the evaluation split CIFAR- 10 (corruption 0) and under increasing levels of image corruption by noise, blur, and weather and digital artifacts (corruption 1,3,5) 5). MultiX consistently approximates the posterior better than the corresponding single-mode approximations, with MultiiVON and MultiSWAG achieving the smallest TV. The single-mode approximations SWAG, MCD, and iVON approximate the posterior better than the Deep Ensemble.
under data corruption than the Deep Ensemble. As expected, MAP has the highest TV, with only small improvement made by Laplace.
6 Conclusion
We presented a comprehensive evaluation wide range of modern, scalable BDL algorithms, using distribution-shife data based on real-world applications of deep learning. We focused on the generalization capability, calibration, and posterior approximation quality under distribution shift. Overall, our analysis resulted in the following takeaway messages:
1. Finetuning only the last layers of pre-trained models with BDL algorithms gives significant boost of generalization accuracy and calibration on realistic distribution-shifted data, while incurring comparatively small runtime overhead. These models are in many cases petitive to or even outperform methods that are specially designed for OOD generalization such as IRM [3] and Fish [80].
2. For CNNs, ensembles are more accurate and better calibrated on OOD data than single- mode posterior approximations by a wide margin, even when initializing all ensemble members from the same pre-trained checkpoint with only the last layers differently initialized, cheekrrindchhekooiiin
i.e. when not using the standard protocol of randomly initializing all ensemble members. Ensembling probabilistic single-mode posterior approximations such as SWAG or MCD yields only small additional increase in accuracy and calibration.
3. When finetuning large transformers, ensembles, which typically considered be the SOTA in BDL, yield no benefit. Compared to all other evaluated BDL. algorithms, classical mean-field variational inference achieves significant accuracy gains under distribution shift.
Limitations. While we evaluate on wide range of datasets from different domains and using different network architectures, the choice of tasks is still limited. In particular, we do not consider LSTMs [34] Ovadia et al. [72] do. Given the limitations of WILDS [47], evaluate on single large-scale regression dataset. As both text classification experiments use DistilBERT [76]. is conceivable that the failure of ensembles is limited to this particular architecture. We do not include algorithms that are based on function-space priors [30, 56, 75, 84]. Except for the results on CIFAR-10-(C) and POVERTY MAP, all results were obtained by finetuning pre-trained models and are therefore only valid in this setting. The HMC samples used in Section 5.5 have been criticized for not faithfully representing the true parameter posterior due to low agreement between the predictions of different chains [78].
Broader Context. Bayesian deep learning aims to provide reasonable uncertainty estimates in safety-critical applications. Hence, we do not expect any societal harm from our work, as long as ensured by proper evaluation that accuracy and calibration requirements are met before deployment.
10

---

Acknowledgments
This work was supported by funding from the pilot program Core Informatics of the Helmholiz Association (HGF). The authors acknowledge support by the state of Beden-Württemberrg through bwHPC, as well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden- Württemberg and by the German Federal Ministry of Education and Research.
References
[1] Moloud Abdar et al. "A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges". In: Inf. Fusion 76.C (2021), pp. 243-297.
[2] Dario Amodei al. "Conerete Problems Al Safery". In: arXiv preprint arXiv:
(2016).
[3]
Martin Arjovsky al. Invariant Risk Minimizationn. 2020. arXiv: 1907 02893.289.
[4]
Neil Band al. Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks. 2021.
[5] Sara Beery, Elijah Cole, and Arvi Gjoka. "The iWildCam 2020 Competition Dataset". In: CoRR abs/2004. 10340 (2020). arXiv: 2004.10340.
[6] Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verte Verlag, 2006.
[7] Charles Blundell et Weight Uncertainty Neural Networks. ICML'15. JMIR.org, 2015,
pp. 1613-1622.
[8] Daniel Borkan al. "Nuanced Metries for Measuring Unintended Bias with Real Data for Text Classification" In: Companion Proceedings of The 2019 World Wide Web Conference.
WWW 19. 2019, pp. 491-500.
[9] Gordon Christic et al. "Functional Map of the World". In: CVPR. 2018.
[10] Adam Cobb et al. "An Ensemble of Bayesian Neural Networks for Exoplanetary Atmo-
spheric Retrieval". In: The Astromomical Journal 158.1 (2019), p. 33.
[11] Francesco D' Angelo and Christian Henning. On out-of-distribution detection with Bayesian
neural networks. 2022. arXiv: $110.06020
[12] A. P. Dawid. "The Well-Calibrated Bayesian". In: Journal of the American Statistical Associa-
tion 77.379 (1982), pp. 605-610.
[13]
Erik Daxberger et al. "Laplace -fforr-Efo Bayesian Deep Learning". In: NeurIPS. 2021. [14] Jia Deng et al. "ImageNet: A Large-Scale Hierarchical Image Database". In: CVPR09. 2009.
[15] Alexey Dosovitskiy et al. "An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale". In: International Conference on Learning Representations. 2021.
[16] Dheeru Dua and Casey Graff. UCI Machine Learning Repository. 2017. URL: http: :// archive ics. uci edu/ml.
[17] Michael Dusenberry et al. "Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors". In: Proceedings of the 37th International Conference on Machine Learning. Vol. 119, PMLR, 2020. pp. 2782-2792.
[18] Sebastian Farquhar, Lewis Smith, and Yarin Gal. "Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations". In: Advances in Neural Information Processing Systems. Vol. 33. 2020, pp. 4346-4357.
[19] Angelos Filos et al. "A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks". In: arXiv preprint 1912v (2019).
[20] Andrew Y. Foong et al. "'In-Between' Uncertainty Bayesian Neural Networks". In: arXiv
preprint arXiv: /906. / 1537 (2019).
[21] Yarin Gal. "Uncertainty in Deep Learning". PhD thesis. University of Cambridge, 2016.
[22] Yarin Gal and Zoubin Ghahramani. "Dropout Bayesian Approximation: Representing Model Uncertainty in Deep Learning". In: Proceedings of The 33rd International Conference on Machine Learning. Vol. 48. 2016. pp- 1050-1059.
[23] Yarin Gal, Jiri Hron, and Alex Kendall, "Concrete Dropout". In: Proceedings of the 31 Inter- national Conference Neural Information Processing Systems. NIPS' 17. Curran Associates Inc., 2017, pp. 3593.

---

[24] Jacob R Gardner et al. "GPyTorch: Blackbox
Gaussian Process Inference with GPU Acceleration". In: Advances in Neural Information Processing Systems. 2018
[25] Jakob Gawlikowski et al. "A survey of uncertainty in deep neural networks". In: arXiv preprint
arXiv:2107.034: 00344 (2021).
[26] Alex Graves. "Practical Variational Inference for Neural Networks". In: Advances in Neural
Information Processing Systems. Vol. 24. Curran Associates, Inc., 2011.
[27] Sorin Grigorescu et al. "A survey of deep learning techniques for autonomous driving". In:
Journal of Field Robotics 37 (Nov. 2019). DOI: 10. 1002/r0b.bb.../ 21918.
[28] Chuan Guo et al. "On Calibration of Modern Neural Networks". In: Proceedings of the 34th International Conference on Machine Learning. Vol. 70. PMLR, 2017, pp. 1321-1330.
[29] Fredrik Gustafsson, Martin Danelljan, and Thomas B Schon. "Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020, pp. 318-319.
[30] Danijar Hafner et al. "Noise Contrastive Priors for Functional Uncertainty". In: arXiv preprint arXiv: 180.009289 (2018).
[31] W. K. Hastings. "Monte Carlo Sampling Methods Using Markov Chains and Their Applica- tions". In: Biometrika 57.1 (1970), pp. 97-109.
[32] Kaiming He et al. "Deep Residual Learning for Image Recognition". In: June 2016, pp. 770- 778.
[33] Dan Hendrycks and Thomas Dietterich. "Benchmarinng Neural Network Robustness to Com- mon Corruptions and Perturbations". In: arXiv preprint /903./1990.1120 (2019). Sepp Hochreiter and Jürgen Schmidhuber. "Long Short-term Memory". In: Neural computation
[34]
9 (Dec. 1997), pp. 1735-80.
[35] G. Huang et al. "Densely Connected Convolutional Networks". In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, 2017, pp. 2261- 2269.
[36] Pavel Izmailov et al. "Dangers of Bayesian model averaging under covariate shift". In: Ad- vances in Neural Information Processing Systems 34 (2021), pp. 3309-3322.
[37] Pavel Izmailov et al. "What Are Bayesian Neural Network Posteriors Really Like?" In: Proceedings of the 38th International Conference on Muchine Learning. Vol. 139, PMLR,
2021, pp.
[38] Alex Kendall and Yarin Gal. "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?" In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS' 17. Curran Associates Inc., 2017, pp. 5590-
[39] Mohammad Emtiyaz Khan and Hävard Rue. "The Bayesian Learning Rule".. In: arXiv preprint 1rXiv:2107.044: (2021).
[40] Mohammad Emtiyaz Khan al. "Fast and Scalable Bayesian Deep Learning by Weight- Perturbation in Adam". In: ICML. 2018.
[41] Mohammad Emtiyaz Khan et al. "Faster Stochastic Variational Inference Using Proximal- Gradient Methods with General Divergence Functions". In: Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence. 16 AUAI Press, 2016, pp. 319-328.
[42] J. Kiefer and J. Wolfowitz. "Stochastic Estimation of the Maximum of a Regression Function". In: The Annals of Mathematical Statistics 23.3 (1952), pp. 462-466.
[43] Diederik Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In: Interna- tional Conference on Learning Representations (Dec. 2014).
[44] Diederik P. Kingma and Max Welling. "Auto-Encoding Variational Bayes". In: CoRR abs/1312.611 14 (2014).
[45] Durk P Kingma, Tim Salimans, and Max Welling. "Variational Dropout and the Local Repa- rameterization Trick". In: Advances in Neural Information Processing Systems. Vol. 28. Curran Associates, Inc., 2015,
[46] Pang Wei Koh al. WILDS Leaderboard. 2020. URL: https: h/tp:://wwwll/ldee stanford. edu/ 1 leaderboard/ (visited on 05/08/2023).
[47] Pang Wei Koh et al. "WILDS: A Benchmark of En-the-Wild Distribution Shifts". In: Pro- ceedings of the 38th International Conference on Machine Learning. Vol. 139. PMLR, 2021, pp. 5637-5664.
12

---

[48] Alex Krizhevsky. Learning multiple layers features from tiny images. Tech. rep. 2009.
[49] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. "Accurate Uncertainties for Deep Learning Using Calibrated Regression". In: Proceedings of the 35th International Conference Machine Learning. Vol. 80. PMLR, 2018, pp.
[50] S. Kullback and R. A. Leibler. "On Information and Sufficiency". In: Ann. Math. Statist. 22.1
(1951), pp. 79-86.
[51] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles". In: Proceedings of the 31st Inter- national Conference Neural Information Processing Systems. NIPS 17. Curran Associates Inc., 2017. pp. 6405-6416.
[52] Y. LeCun et al. "Gradient-based learning applied to document recognition". In: Proceedings of the IEEE 86.11 1 (1998), pp. 2278-2324.
[53] Wu Lin, Mark Schmidt, and Mohammad Emtiyaz Khan. "Handling the Fositive-Definiie Con- straint in the Bayesian I Learning Rule". In: Proceedings of the 37th International Conference on Machine Learning ICML''20. JMLR.org. 2020.
[54]
Jeremiah Zhe Liu et al. "Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness". In: NIPS 20. Curran Associates Inc., 2020.
[55] Qiang Liu and Dilin Wang. "Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm". In: Advances in Neural Information Processing Systems. Vol. 29, Curran Associates, Inc., 2016.
[56] Chao Ma and José Miguel Hermández-Lobato. "Functional Variational Inference based on Stochastic Process Generators", In: Advances in Neural Information Processing Systems.
Vol. 34. 2021, pp. 21795-21807.
[57] David J. C. MacKay. "A Practical Bayesian Framework for Backpropagation Networks". In:
Neural Comput. 4.3 (1992). pp. 448-472.
[58] David C. MacKay. "Bayesian Methods for Adaptive Models", PPD thesis. California Institute of Technology, 1992.
[59] Wesley J Maddox et al. "A Simple Baseline for Bayesian Uncertainty in Deep Learning". In: Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., 2019. [60] Andrey Malinin et al. "Shifts: A Dataset of Real Distributional Shift Across Multiple Large- Scale Tasks". In: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Vol. 1. 2021.
[61] Hendrik A. Mehrtens et al. Benchmarking common uncertainty estimation methods with histopathlogical images under domain shift and label noise. 2023. arXiv: 2301. .01544.
[62] Norman Mu and Justin Gilmer. "MNIST-C: A Robustness Benchmark for Computer Vision". In: arXiv preprint arXiv: / /906.02337 abs/1906.02337 (2019).
[63]
Jishnu Mukhoti and Yarin Gal. "Evaluating bayesian deep learning methods for semantic segmentation". In: arXiv preprint arXiv: (2018).
[64] Allan H. Murphy and Edward S. Epstein. "Verification of Probabilistic Predictions: A Brief Review".. In: Journal of Applied Meteorology (1962-1982) 6.5 (1967), pp. 748-755.
[65] Zachary Nado et al. "Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning". In: arXiv preprint arXiv:2106. 04015 (2021).
[66] Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht. "Obtaining Well Cali- brated Probabilities Using Bayesian Binning". In: AAAI 15, 2015, pp. 2901-2907.
[67] Radford M. Neal. Bayesian Learning for Neural Networks. Springer- Verlag, 1996. ISBN:
0387947248.
[68] Jianmo Ni, Jiacheng Li, and Julian McAuley. "Justifying Recommendations using Distantly- Labeled Reviews and -aninGdd Aspects". In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP)) Association for Computational Linguistics, 2019, pp. 188-197.
[69] Jeremy Nixon et al. "Measuring Calibration in Deep Learning". In: arXiv preprint 1rXiv:1904.00663 (2019).
[70] Kazuki Osawa et al. "Practical Deep Learning with Bayesian Principles". In: Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., 2019.
13

---

[71] Ian Osband. "Risk versus Uncertainty in Deep Learning: Bayes, Bootstrap and the Dangers of Dropout". In: Workshop on Bayesian Deep Learning, NIPS (2016).
[72] Yaniv Ovadia et al. "Can you trust your model's s uncertainty? Evaluating predictive uncertainty under dataset shift". In: Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc.. 2019.
[73] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for Activation Functions, 2017. arXiv: 1710. 05941.
[74]
Hippolyt Ritter, Aleksandar Botev, and David Barber. "A Scalable Laplace Approximation for Neural Networks". In: International Conference Leurning Representations. 2018.
[75] Tim G. J. Rudner al. "Tractable Function-Space Variational Inference in Bayesian Neural Networks". In: Advances in Neural Information Processing Systems. Vol. 35. 2022, pp. 22686- 22698.
[76] Victor Sanh al. "DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter". In: arXiv preprint arXiv: /9/0.0/ 108 (2019).
[77] Steffen Schneider et al. "Removing covariate shift improves robustness against common corruptions". In: CoRR abs/2006. 16971 (2020).
[78] Mrinank Sharma et al. "Do Bayesian Neural Networks Need To Be Fully Stochastic?" In: Proceedings of The 26th International Conference on Artificial Intelligence and Statistics. Vol. 206. Proceedings of Machine Learning Research, PMLR, 2023, pp. 7694-7722.
[79] Mrinank Sharma et al. Incorporating Unlabelled Data into Buyesian Neural Networks. 2023.
aiv: 2304. 01762.
[80] Yuge Shi et al. "Gradient Matching for Domain Generalization". In: International Conference on Learning Representations. 2022.
[81] Ravid Shwartz-Ziv al. "Pre-Train Your Loss: Easy Bayesian Transfer Learning with Infor- mative Prior". In: First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022. 2022.
[82] Saurabh Singh and Shankar Krishnan. "Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks". In: 2020 IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR). 2020, pp. 11234-11243.
[83] Nitish Srivastava et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting". In: 15.1 (2014), pp. 1929-1958.
[84] Shengyang Sun et al. "Functional Variational Bayesian Neural Networks". In: arXiv preprint vrXiv:1903.05770 (2019). [85] James Taylor al. "Rxrxl: An image set for cellular morphological variation across many experimental batches". In: International Conference on Learning Representations (ICLR).
Vol. 22. 2019, p. 23.
[86] Marcin B Tomezak, Siddharth Swaroop, and Richard E Turner. "Neural network ensembles and variational inference revisited"". In: 1st Symposium on Advances in Approximate Bayesian Inference. 2018, pp. 1-11.
[87] Ashish Vaswani et al. "Attention is All you Need". In: Advances in Neural Information Processing Systems. Vol. 30. 2017.
[88] Yeming Wen al. "Flipout: Efficient Sheudo-Independen Weight Perturbations on Mini- Batches". In: International Conference on Learning Representations (ICLR). 2018. [89] Andrew Gordon Wilson and Pavel Izmailov. Bayesian Deep Learning and Probabilistic Perspective of Generalization. NIPS' 20. Curran Associates Inc., 2020.
[90] Andrew Gordon Wilson et al. Evaluating Approximate Inference in Bayesian Deep Learning.
2021. URI.: http::tttts::/iiallloooamae www.ooaaaeelllab.bb..aa. io/neurips bdl___dl______ wwwppetiiion/elliiiiiiiiiiiiii.t It [91] Thomas Wolf et al. "Transformers: State-of-the-AA Natural Language Processing". In: Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online: Association for Computational Linguistics, Oct. 2020, pp. 38- 45.
pdf.
[92] Huaxiu Yao et al. "C-Mixup: Improving Generalization in Regression". In: Advances in Neural Information Processing Systems. Vol. 35. 2022, pp. 3361-3376.
14

---

[93] J. Yao et al. "Quality of Uncertainty Quantification for Bayesian Neural Network Inference". In: Proceedings at the International Conference on Machine Learning: Workshop on Uncertainty & Robustness in Deep Learning (ICML). 2019.
[94]
Christopher Yeh et al. "Using publicly available satellite imagery and deep learning under- stand economic well-being in Africa". In: Nature Communications 11.1 (May 2020).
[95] Wei Zhao et al. "3D Deep Learning from CT Scans Predicts Tumor Invasiveness of Subcen- timeter Pulmonary Adenocarcinomas" In: Cancer Research 78.24 (2018), pp. 6881-6889.
15

---

A Approximate Inference
This section provides further details on the algorithms introduced in Section 3.1.
A.1 Variational Inference
Variational inference (VI) minimizes the KL divergence [50] between the true posterior p(0) D) and the approximate posterior q0 D) [6]. While the KL divergence cannot be computed by itself, as the true posterior is unknown, it can still be minimized by maximizing the evidence lower bound (ELBO) given the parameter prior p(0):
ELBO-
E 0~q(0|D)
pog 0) KI. [a(0 (D) || (())
Maximizing the ELBO means maximizing the likelihood of the training data, therefore fitting the data well, while staying close the parameter prior [6].
Bayes By Backprop (BBB). BBB [7] is application of VI to deep neural network. BBB approximates the parameter posterior with a diagonal Gaussian distribution that cannot model covariances between parameters. The per-parameter means and variances are learned with standard Stochastic Gradient Descent (SGD) [42] using the negative of the ELBO as the loss function. The ELBO by itself is not differentiable as it depends the randomly chosen parameters. However, the reparameterization trick [44] applics to diagonal Gaussians and allows us to use the negative ELBO as the loss function. Further runtime performance improvements are possible by using the local reparameterization trick [45] or Flipout [88].
While there have been reports of BBB performing well when used on neural networks [7, 86), the current consensus of the research community seems to be that BBB falls short when compared to e.g. ensembles [20, 72, 89], even though it has been shown that the diagonal Gaussian posterior not significantly less expressive than a posterior that models covariances [18, 20J. In recent years significant work has been done to improve the performance of VI in deep learning setting. To assess whether these improved algorithms can compete with SOTA Bayesian algorithms, we also evaluate promising improvements on posterior parameterizations (Rank-1 1 VI, SVGD) and optimization procedures (iVON).
Rank-1 Variational Inference (Rank-1 VI). Rank-1 VI [17] enhances the posterior approximation of BBB by approximating a full-rank covariance matrix with a low-rank approximation. Rank-1 VI learns a diagonal Gaussian distribution over two vectors per layer, whose outer product is then element-wise multiplied to a learned point estimate of the layer's 's weights, The bias vector is kept as point estimate. The limited number of additional parameters allows Rank-1 VI to learn multi- component Gaussian distribution for the two low-rank vectors, which gives Rank-1 1 VI ensemble-like properties. Rank-1 VI both less expressive than BBB with the mean field approximation in the sense that it has fewer variational parameters, and is more expressive as it can model covariances between parameters within a layer and can express multi-modality in a limited way.
Improved Variational Online Newton (iVON). The usage of SGD for the optimization of varia- tional parameters is problematic, as these parameters form a complex, non-euclidean manifold [41]. Natural gradient descent (NGD), recently formalized as the Bayesian learning rule [39], exploits this structure to speed up training. VOGN [41, 70] applies NGD to neural networks but has scaling problems, it requires per-example gradients in minibatch training. iVON, based on the improved Bayesian learning rule [53], no longer has this problem. While iVON still uses the mean-field approximation of BBB, it is expected converge faster, and, importantly, halves the number of trainable parameters by implicitly learning per-parameter variances.
Stein Variational Gradient Descent (SVGD). SVGD [55] a non -an-maraettiic VI algorithm that does not assume the posterior to be of particular shape but approximates it with p particles (i.e. point estimates). The particles can viewed as members of Deep Ensemble [51], and the use of VI adds a repulsive component to the loss function based on the RBF kernel distance between the parameters of the partieles. While this repulsive component can prevent the particles from converging to the same posterior mode, it prohibits the independent training of the particles.
16

---

A.2 Other Algorithms
Deep Ensembles. Lakshminarayanan et al. [51] introduce Deep Ensembles that combine the predictions of multiple independently trained neural networks to improve uncertainty estimates. Originally, Deep Ensembles have been seen as a competing approach to Bayesian algorithms [51]. However, ensembles can be considered to be a Bayesian algorithm that approximates the posterior with a sum of delta distributions [89]. We consider all ensembles to Bayesian: While they are missing the principled posterior approximation approach of VI, basically hoping that the members converge to different posterior modes, the approach results in a posterior approximation that is in many cases better than the approximation of for example BBB (Section 5, [72, 89]).
Ensembles are usually considered SOTA in uncertainty estimation [72, 89]. However, the training time scales linearly in the number of ensemble members. This makes them highly expensive in cases where training a single member is already expensive, such as with large networks, and opens the space for new, cheaper posterior approximations.
Monte Carlo Dropout (MCD). MCD [22] uses dropout [83] form Bernoulli distribution over network parameters. The dropout rates are typically not learned, but the dropout units that are present in many network architectures are simply applied during the evaluation of the model. This very cheap posterior approximation has been criticized for not being truly Bayesian [71]. Despite this criticism, is still widely used, including in practical applications [10]. When the dropout rate is learned, MCD can be considered to implicitly perform VI [21].
Stochastic Weight Averaging- Gausssin (SWAG). SWAG [59] forms its posterior approximations from the parameter vectors that are traversed during the training of a standard neural network. During the last epochs of SGD training, SWAG periodically stores the current parameters of the neural network to build a low-rank Gaussian distribution over model parameters. While SWAG has only very small performance overhead during training, storing the additional parameters requires significant amount of additional memory, and sampling parameters from the low-rank Gaussian distribution incurs performance overhead during evaluation.
Laplace Approximation. The Laplace approximation [57] builds a local posterior approximation from a second-order Taylor expansion around MAP model. We always use the last-layer Laplace ap- proximation and switch between a full-rank posterior, diagonal posterior, and a Kronecker-factoree factorized posterior [74] depending post-hoc algorithm that we consider: It can be fitted on top of an existing MAP model by performing single pass on the training dataset.
the task. In this configuration, the Laplace approximation is the only
B Unsigned Calibration Metrics
As mentioned the main paper (Section 4), calibrated model makes confident predictions if and only if they will likely be accurate. Based on this definition, we can directly derive a calibration metric for classification models: The expected calibration error (ECE) [28, 66). In the regression case, neither "accuracy" nor "confidence" are well-defined properties of prediction. The notion of calibration must therefore be adapted for regression tasks. addition, the log marginal likelihood is commonly used to jointly evaluate the accuracy and the calibration in regression tasks. See Appendix G.3.2 for details.
Calibrated Classification. In the classification case, each data point has an associated distribution Y over the possible labels. Y represents the inherent aleatoric uncertainty of the label. Given a prediction = arg max, ply æx, D) made with confidence P max, p(y xx, D), the model is perfectly calibrated if and only if
(ú Y 1 =p Vp E 10
holds for every data point [12, 28, 66]. Informally speaking, this means that if the model makes 100 predictions with a confidence of 0.8, 80 of these predictions should be correct. The expected difference between the left and the right side of Equation (5) is called the expected calibration error (ECE) of the model:
ECE
[ (U Y 9 p) rl]
(6)
p=(((11))
17

---

Obareal Contieneeeen
Obhrne Pattinaatt
0.4
0.2
15.8
12.0
0.5
0.2
11.4
0.8
105
Diperted Contidencc Lavel
(a) Overconfident Quantile
(b) Overconfident CI
Figure 6: Reliability plots of fictional, overconfident regression models when using (a) quantiles and (b) confidence intervals (CI).
implies two propertics of a well-calibrated model: If the accuracy low, the confidence should also be low. This means that the model must not be overconfident in its predictions. Conversely, the accuracy is high, the contidence should also be high, meaning that the model must not be underconfident in its predictions.
In practice, model does not make enough predictions of the same confidence to calculate the calibration етrоr exactly. Therefore, the model's 's predictions grouped into M equally spaced bins Bn based on their confidence values, and the average accuracy and confidence of each bin are used to calculate the ECE [28, 66]:
an evaluation set D'' are commonly
Bm D
ECE Σ
lacc conf(B,.))
where Bm is the set of predictions in the m-th bin, and acc(Bm) and conf( (m)) are the average accuracy and confidence of the predictions in Bm
acc(Bm) =
Σ
1(y = arg max ply x x. D))
(8)
[mm]
Pm
y'
conf(Bm) =
maxply xx
(9)
Bml (..
1/'
(10)
An ECE of zero indicates perfect calibration. We always use ten bins (M 10).
A main problem of the ECE is that bins with few predictions in them may exhibit a high variance [69]. Therefore, Nixon al. [69] proposed extension of the ECE that uses bins of adaptive width.
Calibrated Regression. The confidence intervals of the predictive distribution can be used to measure the calibration of regression model [49]. The probability of the ground-truth output y laying inside of the p-confidence interval of the predictive distribution of the model for input should be exactly p. Formally, we say a regression model is perfectly calibrated on an evaluation dataset if and only if
((Qx(() - S Q1)x(x)) =
v(I, y) D'
(11)
holds for every q-qunttile (x) of the predictive distribution for input æ with p' = (1-p1/2.
Selectively evaluating Equation (11) for M confidence values Pm allows the practical computation of a quantile calibration emror (QCE) on an evaluation dataset D
M
QCE
Σ
(pm Pota Potaa (pm)))
(12)
M m=1
with
Pobs (Pm) (obe(Pm)
Σ
((0 (
(13)
D
(x)N)E
18

---

The QCE simply replaces the quantiles in the definition of the calibration enror from Kuleshov al. [49] by confidence intervals. Using the confidence intervals allows simpler interpretation of the resulting reliability diagrams: With the calibration error proposed by Kuleshov et al. [49], the reliability diagram of perfectly calibrated regression model is horizontally mirrored version of the reliability diagram of perfectly calibrated classification model, as there are too many ground-truth values below the lower quantiles of their predictive distributions, and too few above the higher quantiles (Figure 6a). Using confidence intervals for the reliability diagram results in a plot that can be interpreted in the same way as reliability diagram of a classification model (Figure 6b). We always use ten equally-spare spaced confident levels between 0 and 1 (M 10).
C Signed Calibration Metrics
As described in the main paper, our signed calibration metrics (sECE and sQCE) may be zero even though the model not perfectly calibrated. However, show that this typically not issue in practice, as for most models nearly all predictions are overconfident or nearly all predictions are underconfident. The reliability diagrams in Figure confirm this for representative selection of overconfident and underconfident models. We always report the unsigned calibration metrics in Appendix G in addition the signed calibration metrics mentioned in the main paper. The unsigned metrics are in almost all cases very close to the absolute value of the signed metric, resulting in the same relative ordering of the algorithms. On the other hand, the sECE provides valuable insights into the underconfidence of some algorithms such as MultiSWAG on CIFAR-16 10 and SWAG on AMAZON- WILDS.
19

---

0.8
0.8
Accuricy
Accuracy
0:2
0
Confidence
Confidence
(a) MAP on the o.o.d. evaluation split of 1WILDCAM-WEL WILDS. SECE: -0.0457. ECE:
(b) MultiLaplace on the o.o.d. evaluation split
of IWILDCAM- AM-LIDD sECE: 0.04551 ECE: 0.0501
0.0463
2000
200e
0.8
Accuracy
Accurccy
0.2
0.
0
0:2
0.6
0.8
Confidence
Confidence
(c) MAP the group with the worst accu- racy RWV..COMMENTS-LL.. SECE: 00.3162 ECE: 0.3162
(d) SWAG on the o.o.d. evaluation split of AMAZON-WILDSS. SECE: -0.0405, ECE: 0.0408
Figure 7: Reliability diagrams of different models variety of datasets. No data point is drawn for empty bins. The number of predictions in each bin denoted at the top of each plot. The dashed line corresponds perfectly calibrated model. In all cases, either nearly all of the model's predictions are overconfident, or nearly all are underconfidentt. Therefore, the ECE is close to the absolute value of the sECE, indicating that the sECE is reasonable calibration metric.
D Implementation Details
Except for Laplace, we implement all algorithms ourselves as PyTorch [24] optimizers. The implementation of the algorithms as well as code to reproduce all experiments available at https::/ggt http//ggttttb.tt thub. com/bdl wwwtthorr/beyood-eneeeeee rial on the usage of our implementation.
where we also provide short tuto-
Bayes By Backprop. We use the local reparameterization trick [45]. it standard today [20, 72, 89], we do not use the scale mixture prior introduced by BBB's original authors [7], but unit Gaussian prior. For experiments on CIFAR-10, we make the parameters of the Filter Response Normalization layers variational.
Rank-1 1 VI. Following Dusenberry et al. [17], we keep the bias of each layer as point estimate. We also keep the learned parameters of batch normalization and Filter Response Normalization layers as point estimates. We use five components in most cases which is close to the four components recommended by Dusenberry et al. [17] and make Rank-1 VI directly comparable other ensemble- based models that use five members.
20

---

IVON. We adapt the data augmentation factor that Osawa et al. [70] introduce for VOGN [40] to iVON, We do not use the tempering parameter from VOGN.
Laplace. We use the Laplace library from Daxberger et al. [13] due to the difficulty of implementing second-order optimization in PyTorch. In all cases except for ZWVI.COMMENTSWIIEE we use a Kronecker-factorizee last-layer Laplace approximation. On CIVILCOMMENTS-WIIE diagonal last-layer Laplace approximation as the Kronecker-factorizee approximation frequently leads to diverging parameters. We do not use the GLM approximation as proposed by Daxberger al. [13] but use Monte Carlo sampling to stay consistent with the other evaluated algorithms. In all experiments we use the Laplace library's functions to tune the prior precision after fitting the Laplace approximation.
SWAG. While the authors of SWAG argue that SWAG benefits from a special learning rate schedule [59]. they do not use such schedule in most of their experiments with SWAG and MultiSWAG [89]. Correspondingly. we use the same schedule with SWAG as with any other algorithm. We 30 parameter samples for building the mean and the low-rank covariance matrix of SWAG. On MWWI.COMMENTS-WIL.. we only use 10 parameter samples due to the storage size of the samples.
E Batch Normalization, Distribution Shift, and Bayesian Deep Learning
SECIE
-02
00.4
0.5
8.5
0.7
0.5
Ascrage
Gtatic BN)
Evtatic
BBB (Atalic BN)
Figure 8: CAMELYONI7-WIILD:: Average accuracy vs. sECE on the o.o.d. test split. The models that use no running statistics (static BN) are significantly more accurate and better calibrated, while exhibiting a smaller variance.
Schneider et al. [77] find that significant part of the accuracy loss o.o.d. data is due to changing batch statistics that cannot be adequately normalized by the running batch normalization statistics that are based on the training data. The authors propose re-initialize the running statistics subset of the evaluation dataset.
We are able to reproduce the issue with o.o.d. data on the CAMELYON17-WILD dataset from the WILDS collection [47] (Figure 8). The o.o.d. evaluation set of CAMELYON17 has been generated by selecting the images that were most visually distinct from the other images. In addition, the employed ResNet-20 [32] architecture includes batch normalization layers. We find that using only batch statistics, thereby essentially using the batch normalization layers in training mode during evaluation, entirely alleviates the i.d. o.o.d. performance gap on CAMELYON 17, as well as the large standard deviations the o.o.d. dataset. Coincidentally, the WILDS leaderboard [46] shows that models that do not include batch normalization, such as a model based on the vision transformer [15], or that use extensive data augmentation, perform best.
The running statistics of batch normalization layers also pose problems with Bayesian neural networks that sample parameters, as the running statistics depend on the parameters of the neural network. Wilson and Izmailov [89] therefore propose to recalculate each parameter sample. This is not necessary in our case as we never use running statistics for normalization layers. By doing so we also avoid the aforementioned distribution-shift problem without requiring additional o.o.d. data during evaluation, and do not add any computation overhead.
batch normalization statisties for
21

---

Computational Resources
We use single NVIDIA Testa V100, A100, and H100 GPUs for all tasks from Wilds [47] and CIFAR- 10-(C) [33, 48]. See Table for the GPUs that we use on the individual datasets well as the runtime of MAP. Table 2 displays the relative runtime of the BDL algorithms. In total, we estimate that the evaluation required about 1600 of GPU time, which about 25% were consumed during implementation, testing and hyperparameter optimization. Training and hyperparameter optimization of the UCI models was performed on single CPU in about 20 h. Table 3 shows the GPU memory overhead of the BDL algorithms.
Dataset
GPU
Runtime of MAP
CIFAR-10 10 DOVERYYMAP-WILDY (WI.DCAM-WILDS
NVIDIA VI00
50 min
NVIDIA V100
50 min
NVIDIA A100
150 min
FMOW-WILDS
NVIDIA V100
150 min
RXRXI-WILDS
NVIDIA V100
140 min
CIVILCOMMENTS-
NVIDIA A100
60 min
AMAZON-WILDS
NVIDIA H100
90 min
Table 1: Hardware and runtime for MAP for each dataset. The results are rounded to the next 10
minutes.
Model
MAV
/WILDCAM
FMoW
RxRx1
CIVILCOMMENTS
AMAZON
CIFAR- 10
1.0
1.0
1.0
MCD
1.0
1.0
1.0
1.0
SWAG
1.3
1.5
L.0
1.3
Laplace
1.0
1.0
1.0
1.0
1.0
1.0
1.0
BBB
5.7
5.0
LL BBB
1.6
2.0
3.7
1.8
2.0
Rank-1
3.9
Rank-1
2.0
1.9
iVON
2.8
LVON
3.0
2.9
3.6
5.6
6.6
SVGD
9.2
4.9
7.3
8.9
9.2
10.0
8.0
Table 2: Runtime of different algorithms relative MAP. The numbers CIFAR-10 conser- vative estimates as exact numbers were no longer available. Note that the runtime also depends on whether able to mixed precision training, which not possible with the VI algorithms. The training time of a MultiX model with members is n times the training time of the respective single-mode approximation. LL Last-Layer.
Model
Memory Overhead
MAP
1.0
MCD
1.0
SWAG
1.0
Laplace
1.0
BBB
Rank-1 VI
1+ #components /parameter count
iVON
2
SVGD
#particles
Table 3: GPU memory requirements of different algorithms relative to MAP. The numbers are estimates are based on a theoretical analysis of the algorithms, not on measurements. The memory consumption of MultiX is the same as for the respective single-mode approximation, since all members can be trained indenpendently.
No additional memory overhead due separate optimizer
22

---

G Additional Experimental Results
G.1 UCI Datasets
We report results for both the standard and the gap splits [20] on the HOUSING and ENERGY datasets from the UCI machine learning repository [16]. On ENERGY. we can reproduce the catastrophic failure of VI both with BBB and Rank-1 VI, but not with iVON which performs still similarly to MultiSWAG. Overall, we find that the benefit of ensembles less clear than on the larger WILDS datasets, which emphasizes the importance of evaluating Bayesian algorithms on large datasets.
Hy perpaaaaamesrss. All hyperparameters were optimized through grid search on the validation set. Note that for the gap splits the validation set is not part of the gap. We considered 40, 100 and 200 epochs, learning rates of 0.01 and 0.001 and (where applicable) weight decay factors of 10-4 and 10-5. For BBB, the prior standard deviations are 0.1, 1.0 and 10.0 and we scale the KL divergence in the ELBO by 0.2, 0.5, and 1.0, with colder temperatures generally leading to better results. For iVON, we consider prior precisions of 10, 100, and 200, with 200 being selected in most cases. BBB and IVON use five Monte Carlo samples during training. For SWAG, we consider 60, 100, and 150 epochs, use 30 parameter samples and start sampling after 50%. 75%, or 90% of the training epochs were completed. For Laplace, we always use a last-layer approximation with a full covariance matrix. We use the Adam optimizer [43] to optimize the log- liiklihood//LBBB and learn the output standard deviation jointly with the parameters. We use 1000 parameter samples for each prediction.
Model
LML
MSE
QCE
SQCE
MAP
-2,613 0.051
10.707 = 0.79
0.036
0.001
-0.018 = 0.012
Deep Ensemble
2.350 1: 0.015
6.084 1 0.413
0,099 = 0.0016
0.099 1: 0.006
MCD
-2,86
18.531
1.101
0.011
0.019
MuliMCD
2.328 0.010
5.789
0.077
0.101 - 0.002
0.101 1: 0.002
SWAG
-2.195 0.011
6.011 0.214
0,128 0.003
0.128 1 0.003
MuliSWAG
2.001
7.071
0.011
0,130 0.002
0.139 0.002
LL Laplace
-2,515 2..15:00.05..
9.498
1
0.099 = 0.020
0.099 +: 0.1120
L.. Multil aplace
2.467 0.0031
5.850
0.395
0.157 0.005
0.157 0.005
BBB
-2475 1:
7.716 1.053
0.033
0.004
-0,006 4 0.012
MuliBBB
-,529 0.003
1.212 + 0.160
0.172 + 0.005
0.172 + 0.005
Rank-1 VI
-2.531 0.113
0.008
IVON
-2.033 0.006
9.863 + 0.318
0.215 0.003
0.215 + 0.003
SVGD
-2611 0.017
8.397 # 0.642
0,143 0.010
0.1 0.010
Table 4: UCI-HOUSING (standard splits)
Model
LML
MSE
OCE
=QCE
MAP
-2350 : 207
11.775 = 3.635
0.010 = 0.013
-0.012
Deep Ensemble
-2767 1: 0.133
13.012 1. 3.034
0.054 1 0.011
0.045 1: 0.017
MCD
-2.892 0.210
0.031 0.007
0.0018
MuliMCD
2.730 1: 0.132
12.760 1: 2.838
0.054 0.013
1 0.019
SWAG
-2.743 0.042
12.940 + 1.742
0.113+
0,112 0,017
MuldSWAG
0.091
11.941
0.020
0.020
LL. Laplace
-28733 0.111
15.294
1: 3.687
0.074 1 0.018
0.072 1 0.020
L.. Multi aplace
-28322 0.1
13.445 +: 3.209
0.106 + 0.020
0,104 + 0,022
BBB
3,829 1.009
17.238
0.114
MuldBBB
14.071 + 2.525
0.065 + 0.018
0.064 + 0.022
Rank-I VI
-2,806 :: 0.199
13.513 + 2.930
0,067 1 0.025
0.018 + 0.043
0.023
17.904
0,152
0.015
SVGD
2.855 + 0.184
14.848 + 3.170
0.039 1 0.014
0.003 + 0.025
Table 5: UCI-HOUSING (gap splits)
Model
LML
MSE
QCE
sQCE
0.009
1.760 0.289
0.051
-0.020 ::
Deep Ensemble
1,235 0.003
0.177
0.007
0.270 =: 0.002
0.270 :: 0.002
MCD
-1.009 0.079
1.779 0.252
0.049 1049:
0.022 + 0.032
MuliMMD
1,296 0.005
0.212 0.015
0.260 0.003
0.280 0.003
SWAG
2.127
2.198 + 0.274
0.000
0
MuliSWAG
2.143+0..00
2.454 + 0.018
0.220 - 0.001
0.220
LL Laplace
-16533 1.03:.0.000
0.608 0.100
0.019
1245:0.001
L.. Multil aplacce
-1.006 + 0-006
0.235 + 0.033
0.316 0.008
0.316 0.008
BBB
0.976 0.123
0.113
0.030
MuliBBB
1.02 1: 0.021
0.309 + 0.075
0.0022
0.210 - 0.012
Rank-1 VI
1.029 + 0.166
0.459 0.145
0.054 0.019
0.019 + 0.036
IVON
-2.463 0.006
6.620
+ 0.191
0.161 0.010
0.161 + 0.010
SVGD
-1,322 : 0.010
0.550 2 0.121
0.159 = 01027
0.159 - 0.127
Table 6: UCI-ENERGY (standard splits)
23

---

Model
LML
MSE
QCE
sQCE
МAР
-7.233 7.533
34.14
41.620
0.277 = 0003
0.019:0.145
Deep Ensemble
-1.60 1.068
31.419 1: 36.845
0.272 1: 0.060
0.072 # 0.207
MCDD
10.290 10.685
48.401
58 682
0.065
0.206
MultiMDD
SWAG
3.655 + 1: 1.469
30.372
28.902
0.218 + 01844
0.011 0.183
MuluSWAG
3.110 = 0.815
25.362
128
0.192 : 0.058
0.034 0.152
Laplace
1.. Multil aplace
5.540 2.088
38.452
01.667
0.270 + 0.046
0.142 0.127
BBB
61.268 79.182
13.670
02 833
0.199 :: 0.131
-0.101
0.184
MuliHBH
Rank-1
-72/112 92.191
19,099
60 606
0.191 : 0.133
-0.109
0.178
IVON
-3.367 1: 0.903
21.546
13.347
0.109 +: 0.025
0.038 0.074
SVGD
46.757
56-551
Table 7: UCI-ENERGY (gap splits)
24

---

G.2 CIFAR-10
Following Wilson et [90]. we train ResNet-22 [32] with Swish activations [73] and Filter Re- sponse Normalization [82]. The use of Filter Response Normalization instead of batch normalization, which only uses batch statistics, eliminates the problems mentioned in Appendix We train all models except iVON with SGD and a learning rafe of 0.05 and Nesterov momentum of strength 0.9 for 300 epochs. We use the learning rate schedule from Maddox et al. [59]: The learning rate is kept at its initial value for the first 150 epochs, then linearly reduced to learning rate of 0.005 epoch 270 at which
kept constant for the remaining 30 epochs. For MCD, we use dropout rate of 0.1 and insert dropout units after every linear and convolutional layer of the ResNet-20. For BBB, we temper the KL divergence the ELBO with factor 0.2. Rank-1 VI uses an untempered posterior and four components. BBB and iVON use two Monte Carlo samples during training. The Laplace approximation based on a diagonal last-layer approximation. iVON is also trained for 300 epochs with learning rate of 1 10 prior precision of 50, and data augmentation factor of 10 (see Osawa et al. [70] for details), but uses no learning rate schedule. We found these changes to be necessary ensure that iVON performs well, likely because iVON is much more similar Adam [43] than to SGD and therefore needs smaller learning rate. Following Nado et al. [65]. SNGP uses spectral normalization factor of 6.0 and mean field factor of 20. We did perform any additional tuning of the mean field factor. We always use 50 parameter samples during evaluation.
Figure 9 displays the accuracy, ECE, SECE, agreement with HMC, and TV compared to HMC. MultiX models tend to become underconfident. Table shows detailed numerical results for all algorithms and corruption levels.
25

---

Model
Accunccy
<CCE
Agreement
'MAP"
0.325 = 0.000
0.030 = 0.001
0006
= 0.001
0:296 = 0.006
0.306 :: 0.002
0.172 = 0.001
Deep Ensemble
0.944 0.001
0.010 0.001
0.003 + 0.000
0.174 0.001
0.923 0.001
0.111 = 0..00
MCD
0.927
0.001
0:07 0.001
0.216 0.005
0.920 0.003
0.132 0.002
MuhiMCDD
0.941 + 0.001
0.31 + 0.001
0:031
+ 0.001
0.186 + 0.001
0.300 + 0.002
0.116 = 0.001
SWAG
0.921 0.002
0.012 0.003
0.012 0.003
0,250 4 0.002
0.910
0.002
0.130 = 0.001
MaldSWAG
0.930 0.001
0.002
0.099 0.002
0.002
0.927
0.001
0.001
11. Laplaee
0.9244 + 0.001
0.040 0.001
0.040 + 0.001
0.282 + 0.006
0.906 + 0.002
0.189- + 0.001
LL MultiLaplace
0.945 + 0.001
0.012 = 0.002
0.007 0.000
0.174 0.002
0.923 0.001
0.142 = 0.000
BDB
0.898 0.003
00016
0,900
0.001
MuldBBB
0.929 0.000
0.018 = 0.002
0.018 = 0.002
0.228 : 0.002
0.931 2: 0.001
0.122 = 0.001
RankI-VI
0.881 0.000
0.041 0.002
0.041 0.002
0.363 0.005
0.910 0.003
0.116 0.002
IVON
0.842 0.004
0.003
0.024 0.003
0.464 0.011
0.874
0.004
0.003
VONGiVON
0,881 +: 0.003
0.077 = 0.003
0.077 2 0.003
0.368 + 0. 000
0,921 :: 0.002
0.100 107 = 0.001
SVGD
0.927 0.001
0.018 0.001
0.003 + 0.001
0.255 0.001
0.924
0.002
0. 035 0.001
SNGP
0.917 0.003
0.076 0.008
0.076 0.006
0.380 0.013
0.903 0.002
0.178 0.002
HMC
0.503
0.069
0.068
0.320
1.000
0,000
aa) Standard Evaluation Split (Corruption Level 0)
Model
Accuracy
BCB
sECE
NLL
Agrecment
TV
MAP
0.372.1000
0,080 2 0.0012
-0,0801 0.3012
0.518 0.110
0,815 =2 0.000
0.239 1 0.000
Deep Ensemble
0.903 0.001
0.014 0.001
0.000 +: 0.001
0.305 1: 0.006
0.870
0.001
MCD)
0.872 + 0.004
0.011 + 0.003
-0.004 + 0.006
0.400 + 0.011
0.865 +: 0.003
0.184 + 0.002
MultiMDD
0.850 + 0.003
0,028 0.002
0.026 + 0.002
0.335 + 0.005
0,850 + 0.002
0.158 + 0.001
0.006
0.047
0,856
MalESWAG
0,900 2: 0.000
0,117 0.002
0.111 1: 0.002
0.383 1: 0.002
0.878 -: 0.001
0.147 1: 0,000
LL Laplace
0.873:20.002
0,071 : 0.004
-0.071 1: 0.001
0.199 : 0.017
0.819 :: 01.00
0.239 + 0.002
LL MaltiLaplace
0.903 0.005
0,016 0.005
-0,001 1: 0.005
0.514 1: 0.003
0.859 :: 0.004
0.199 0.001
BRB
0.839 0.003
0,883 0.003
0.003 0.003
0.682 0.029
0.1002
0.222 0.000
MallBBB
0.878 +:
0.018 + 0.005
0.006 + 0.009
0.394 +: 0.012
0.850 =: 0.005
0.170 + 0.004
RankI-VI
0.843-2:0000
0.842 0.004
0.042 + 0.004
0.484 + 0.015
0.800 +: 0.1022
0.165 + 0.003
IVON
0.795 0.795-10.022
0,025
0.010 0.010
0.396
0.008
MallaVON
0,845 2: 0.000
0,083 0.003
0.081 : 0.0003
0.304 + 0.018
0.863 -: 0.00S
0.145 + 0.003
SVGD
0.563 1: 0.001
0,021 4 0.003
0..007
1 0.001
0.4 + 0.010
0.875
=: 0.001
0.192 1 0.000
SNGP
0.967 0.015
0.074 0.009
0.069 0.012
0.580 0.034
0.844 0.010
0.231 0.004
HMC
0.834
0.066
0.064
0.508
1,000
0.000
(b) Corruption Level 1
Model
Accuracy
ECB
SECE
NLL
Agroement
TV
MAP
0.806 + 0.005
0.128 + 0.1015
0.1288 + 0.004
0.863 + 0.018
0.778 + 0.002
0.300 - 0.0022
Deep Ensemble
+
-0:027
0,805
0.253
MCD
0.777 0.0011
0.047 2: 0.004
-0.047 1: 0.005
0.733 1: 0.051
0.796 1: 0.010
0.292 -: 0.001
MultiMCD
0.805 + 0.003
0.003
0,000 1: 0.003
0.584 + 0.012
0.823 + 0.005
0.196 + 0.001
SWAG
0.806
0:031
0.009
0.788
0.226 0.003
MuldSWAG
0.839 + 0.002
0.117 = 0.001
0.117 2 0.004
0.556 1 0.002
0.818 1: 0.001
0.186 = 0.001
LL. Laplace
0.804 1: 0.003
0.120 4: 0.002
0.119 :: 0.002
0 + 0.021
0.777 1 0.005
0.308 : 0.003
L. Multilaplace
0.850 + 0.012
0.008
0.008
0.498
0.007
0.003
BBB
0.735 2: 0.009
0.151 0.008
-0.151 = 0.008
1.296 1 0.072
0.774 1: 0.001
0.256 = 0.002
MuliBDB
0.786 3: 0.014
0.033 1: 0.010
0.026 2: 0.014
0.741 1 0.045
0,830 1: 0.006
0.204 = 0.003
RankI-VI
0.774 0.008
0.002
0.019
0.684 0.027
0.802 0.006
0.005
IVON
0.725 : 0.004
0.028 0.001
-B.022 = 0.005
0,809 + 0.036
0.756 :: 0.016
0.237 = 0, 8.008
MaldiVON
0.783 1: 0.008
0.069 2 0.009
0.068 :: 0.010
0.666 1: 0.006
0,821 1: 0.002
0.179 = 0.003
SVGD
0.804 0.004
0.002
0.008 0.002
0.821 0.024
0,818 0.001
0.208 0.001
SNGP
0.785 + 0.015
0.067 0.007
0.044 0.010
0.837 + 0.042
0.767
: 0.008
092 = 0.004
HMC
0.724
0.020
0.017
0.833
1.000
0.000
(c) Corruption Level 3
Model
Accuracy
ECE
SECE
NLL
Agrecment
TV
MMPP
0,089 0.0016
0.217
-0.217
1.011
0.083 0.000
: 0.003
Deep Ensemble
0.733 =: 0.006
0.075 1: 0.007
-0.075
1: 0.007
0.937 0.036
0.718 :: 0.004
0.310 +: 0.003
MCD
0,629 + 0.009
0.141 + 0.011
0.141 + 000000
1.312 + 0.004
0.725 + 0.012
0.277
+ 0.003
MuluMCD
0,006
1.633 0.022
0.760 0.0001
SWAG
0.606 1: 0.005
0.018 : 0.009
-0.012 - 0.008
0.927 0.025
0.699 : 0.008
0.277 1 0.006
MaldSWAG
0.728 = 0.004
0.0122 0.001
0.082 2 0.001
0.8-11 + 0.009
0.7 : 0.00+
0.225 - 0.002
LL Laplace
0,650 0.006
0.263 1: 0.005
-0.203 1 0.005
L4300 1: 0.026
0.655 0.003
0.350 0.002
I.J. aplace
0.781 = 0.013
0.073
0.072 + 0.011
0:941 0.064
0.722 0.008
0.303 0.002
BBB
0.584 :: 0.011
0.268 = 0.014
-0 -0.268 = 0.014
2,413
0.126
0. 0.698 =: 0.008
0.353 - 0.003
MaluBBB
0.621 0.011
0,114 1: 0.016
-0.111 0.016
1411 0,105
0.757 0.010
0.217 1 0.008
RankI-VI
0.673 0.016
0.010
0.024 0.014
1,10 0.056
0.719 0.013
0.010
IVON
0.617 =: 0.011
0.086 + 0.006
-0.086
= 0.006
1.201 + 0.061
0.678 =: 0.008
0.250 = 0.007
MalGiVON
0.657 6.012
0.025 + 0.005
0,000 + 0.009
0.998 0.017
0.752 0.012
0.221 1 0.006
SVGD
0.666 0.004
0.117 0.009
= 0.009
L.557 0.043
0.742 = 0.007
0.002
SNGP
0.657 + 0.006
0.084 + 0.007
D.017 = 0.008
1.256 +: 0.022
0.681 = 0.008
0.363 + 0.005
HMC
0.592
0.055
-0.051
1.255
1.000
0,300
(d) Corruption Level 5
Table CIFAR-10. 10: Detailed results on the standard evaluation split and the corruption levels 1, 3, and 5 of SIFAR-10-C.
26

---

Compuion
3
0.55
0..6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Accuracy
Cormpttinn
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0.22
0.21
0.26
0.28
BCE
Comptton
00.25
-0.2
00.15
-0.1
0..05
0.05
0.1
SECE
Cormpton
0.68
0.7
0.72
0.74
0.76
0.78
0.8
0.82
0.84
0.86
0.88
0.9
0.92
Agreement with HMC
Comppion
0.12 0.14 0.16 0.18
0.22
0.24 0.26 0.28
0.32
0.34 0.36
0.38
Variation
Bacend:: MuLENCD
MaNiSWAG
MaIBUB
MALIVONN
SVGD
1930
Figure 9: CIFAR-10-(C): All results for the corruption intensities 0, 3, and 5. The corruption intensities are denoted on the y-axis. A negative SECE indicates overconfidence, a positive sECE indicates underconfidence. The plot for the TV is repeated from Figure 5.
G.3 WILDS
We strictly follow the training and evaluation protocol of Koh al. [47] by reusing their data folds for training, validation, and testing. We use the hyperparameters proposed by Koh et al. [47] where applicable, and set the other hyperparameters of the respective algorithms. If the standard values lead to unexpectedly bad results, we tune the hyperparameters through a grid search. Hyperparameter tuning was performed on the i.d. validation and, where available, o.o.d. validation splits, but never on testing splits. In particular, we select the prior precision of iVON through grid search over the values 1, 10, 100, and 500 per model architecture. We find the prior precision of iVON to be hard to tune, as iVON frequently diverges for comparatively small prior precisions such as and 10. BBB works always well with the standard unit prior. We also experiment with other priors but find no difference in performance except on RxRxI -WILDS (see Figure 14). BBB and iVON use two Monte Carlo samples during training. See the sections below for the hyperparameters that were chosen the individual datasets. We use mixed precision training whenever possible. The VI algorithms as well as the Laplace approximations are mostly trained without mixed precision, as this leads to unstable training.
standard values suggested by the developers
SNGP uses the same learning rate, weight decay, and number of epochs as the other algorithms. Following the recommendations by Liu et al. [54] and the tuning done by Nado et al. [65]. we use spectral normalization factor of 6.0 for the computer vision tasks and 0.95 for the text classification tasks. On the image classification tasks, SNGP performs significantly better when limiting the input
27

---

dimension of the Gaussian Process to 128 or 256 instead of using the output dimension of the previous network layer.
We use 10 posterior samples per prediction during evaluation to constrain the computational overhead of the Bayesian algorithms, which is generally sufficient to capture the predictive distribution [72]. Note that our results are not directly comparable to the results of Daxberger et al. [13]. as they build their Deep Ensembles and Laplace approximations from the pretrained models provided by Koh et al.
[47]. When comparing our results with the best performing algorithms on the WILDS leaderboard, we only consider the algorithms on the "overall leaderboard", i.e. the algorithms that conform the official submission guidelines of Koh et al. [47].
G.3.1 CamelyonI7-WILDS
Following Koh al. [47], we train DenseNet-122 [35] with SGD for epochs with a learning rate of 0.001, weight decay 0.01 and momentum 0.9. SWAG collects 30 parameter samples during the last epoch.
G.3.2 Poverty-AA--ILLDS
We train ResNet-18 [32] using the same hyperparameters as Koh et al. [47] where applicable: learning rate of 10-3 and no weight decay. We only train for 100 epochs as all models were converged after that. SWAG collects 30 parameter samples starting at epoch 50. For BBB, we scale the KL. divergence down with factor 0.2, as this significantly improves the MSE. Rank-1 VI uses an unscaled KL divergence. The ensembles, Rank-1 VI and SVGD use five members/componentts We optimize the log likelihood of the training data and represent the aleatoric uncertainty with fixed standard deviation of 0, as this is the value MAP converges to when jointly optimizing the standard deviation and the model's parameters. For the final evaluations, we do not optimize the standard deviation, this leads to unstable training with the VI algorithms. Following Koh et al. [47], aggregate all results over the five folds of POVERT MAP, with one seed per fold.
As mentioned in the main paper. IVON performs significantly worse than the other algorithms. We conducted grid search over prior precisions 1, 10, 100 and 500 with a single seed per value, and found that for 1 and 10 iVON diverges, for 100 iVON achieves an o.o.d. Pearson coefficient on the "A" split of 0.21 and for 500 it achieves Pearson coefficient of 0.25. Most likely due to their underfitting the non-diverged models are comparatively well calibrated with sECEs of -0.21 for prior precision of 100 and 0.24 for a prior precision of 500.
Log Marginal Likelihood. The log marginal likelihood is commonly used to jointly evaluate the accuracy and calibration of a regression model. On an evaluation dataset D', the log marginal likelihood (LML) is given by
LMI. = PDD' D) = log p(D' | O)p(0 DD d0 v log Σ p(D' Om),
(14)
where the 0n are samples from the parameter posterior. When only few predictions are available because sampling parameters On evaluating the likelihood p(D' 0,)) is expensive, the LML may become very noisy. We therefore also report the per-sample log marginal likelihood
psLML=
Σ
logp(y, x;; D)
(2r, (D)
log / p(y: x;, O)p( (D) do
Σ
(15)
(2) (2)))))
log Σ p(y: xi, 0w),
(2)
which has lower variance than the LML. We present the results for the LML, the psLML, the urban/rural Pearson coefficient (see Section 5.1), and the sQCE in Figure 10 and Figure 11. Table shows detailed numerical results.
28

---

Model
UVR
QCE
MAP
0.487 = 0.074
119522002
11.915
= 2.012
0.267
0.00
0,322
0.012
-0.322
2 0X012
Deep Ensemble
0.520 1: 0.075
6.126 1.422
12.113
2.074
0.249 0.045
0.283
0.027
00.233 4 0.027
MCD
0.491 + 0.079
-6.888 1.720
-12.175-
2.398
0.259
0.316 0.028
-0816 0.023
MultMCD
0.316 1: 0.078
5.053 1.188
12.290
2.409
0.253 1: 0.052
0.271 1 0.035
0.271 0.035
SWAG
0.477 1: 0.078
-12.551
0.280 0,040
0.386
0.012
WAG
0.512 = 0.078
-6.010 1.373
-12.167
1.989
0.250=0.014
0.280 = 0.026
-0.250
0.026
LL Laplace
0.473 1: 0.072
112.599
2.19
-12.999
= 2.191
0.250 = 0.044
0.387 2: 0.012
-0.367 1: 0.012
MultiLaplace
-12.324
0.026
BBB
0.500 1: 0.072
-7.881 2.2200
-12.075
2.470
0.264 1: 0.054
0.33 1 0.036
0.933 0.036
MullBBB
0.518 + 0.074
6.257 1.482
-11,498
2.299
0.262 = 0,048
0.309 0.027
-0,000 0.027
Rank-1
IVO
0.249
-4.657
19.7777
0.347
0.236
-0.236
SVGD
0.497 + 0.070
-5.416 = 1.270
12.524
2.224
0.251 = 0,011
0.235 0.026
-0 255 0.026
SNGP
0.031
0.357
0.009
(a) O.o.d. Evaluation Split
Model
Pearvon
MLLML
MAP
0,673 0.019
1.455 - 0.761
7.445 1. 0.761
0.177 0.015
0,349 + 0.008
0,348 + 0.00S
Essemble
0.703 0.002
-3.438
0.744
-0.228
MCD
0.695 + 0.010
-3,001
-7.237 +: 0.673
0.162
0.012
0.267 0.014
-0.267 = 0.014
MuluMCD
0.711 0.021
-2,680
-7.383
0.615
0.156
0.20 0.011
-0.220 0.014
-7.752
1:552:0..5
MuluSWAG
0.705 + 0.023
-3.331 = 00/399
-7.221 0.625
0.155
0.223 0.008
-0.223 = 008
Laplace
0.664 0.018
-7.823 0.737
-7.823 1: 0.737
1.184100.115
0.35T 0.006
-0.357
0.44000000 0.0066
aplaac
3.234
0.530
-7.474
0.828
0.20 0.004
-0.206
BBB
0,680 + 0.019
4,508 = 0.269
-7.224
1 0.754
0.169 1 0.014
0 0.014
0.296 1: 0.014
MuliBRR
0.694
-3,619-- 0.344
0.586
0.25
00225
Rank-1
0.609 = 0.011
22,077
0.323
-9.630 1.166
0.173 0.016
0.162
0.016
0.162
0.016
IVON
0.571
3,888
-1,8322-
0.284 +
0.201
-0.201
0.691
-7.561
-0,200
SNGP
0.692 + 0.013
-7133 = 0.639
7.135 + 0.639
0.170 0.013
0,300 0.008
-0.300 - 0.008
(b) Ld.. Evaluation Split
Table 9: POVERTY MAP-WILD:: Detailed results the evaluation splits. iVON underperforms, with a Pearson coefficient of 0.249 on the o.o.d. split and a Pearson coefficient of 0.571 on the i.d. split. All models achieve the same LML and MSE within 95% confidence interval.
WILDS Leuder
procompll Log Morginal seelihoo
MAP
Deep Linsemble
Log Margina L.ellood
-0.25
SWAG
SQCE
-8-
MuleSWAG
-03
MuliBBB
Rank:
MutiLaplize
0.4
09.63
0..5
0..55
0.0
0.66
0.45
0.55
0U.6
0.5
0.4
01.65
0.5
0.55
0..0
SNGP
Winel U.R Pearson
Wond LVR Pexrou
Wind U.R Pauason
Figure 10: POVERTYMAP-WIL.D Worst urban/rural Pearson coefficient vs. sQCE, LML, and PsLML on the o.o.d. evaluation split. Ensemble- models consistently outperform single-mode models (note that Rank-1 VT's components and SVGD's particles give them ensemble- like properties). The psLML is less noisy than the LML and results a ranking of the algorithms that is more consistent with the s sQCE and the Pearson coefficient. Laplace and SWAG perform nearly equivalently, therefore the data points of SWAG are hidden behind the data points of Laplace. iVON performs significantly worse than the other algorithms and is therefore excluded.
29

---

persaamle Lag Marginal Likelthood
Log Marginal L.belhhood
5QCT:
-0.25
-03
-0.35
-11
0.64
0.06
0.68
0.7
0.72
0.74
0.51
0.66
0.68
0.7
0.72
0.74
0.84
0.06
0.68
0.7
0.72
0.74
Woest U/R Peanson
Worst Peurson
U/R Peanson
Deep Ensemble
MulIMCD
Rank-1
SVGD
Figure 11: POVERTYMAP-WILDD Worst urban/rural pearson coefficient SQCE, LML and LsLML on the i.d. evaluation split. The WILDS leaderboard [46] does not report the i.d. pearson coefficient.
G.3.3 WWILDCAM-WILDS
Following Koh al. [47], we finetune ResNet-50 [32], pretrained ImageNet [14], for 12 epochs with the Adam optimizer [43]. For each model, we replace the linear classification layer of the ResNet- 50 by randomly initialized one of the appropriate output dimension. We use the hyperparameters that Koh et al. [47] found to work best based on their grid search: A learning rate of 3 10 and no weight decay. For MCD, we try dropout rates of 0.1 and 0.2 and select 0.1 due to slightly better macro F1 score on the evaluation split. iVON uses prior precision of 100, as optimized by grid search. We use three seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the three evaluation runs. Figure 12 shows the results on the o.o.d. evaluation split that are not presented in the main paper. Table 10 displays detailed numerical results on the o.o.d. evaluation split and on the i.d. validation split.
0.12
0.36
BCE
SECE
0008
0.1
id.
0.06
0.58
0.7
0.72
0.74
0.76
0.3
0.35
0.4
0.45
0.5
0.77
0.76
0.78
0.82
0.84
Accuracy
Accuracy
Laplace
Lender
Deep Ensemble
MulLMCD
MuleSWAG
MultiLaplace
MultBBB
SVGD
SNGP
Figure 12: WWILD AM- WILDS: Macro score, accuracy, sECE and ECE on the o.o.d. evaluation split and the i.d. validation split (see Figure 2a for Macro F1 vs. sECE on the o.o.d. evaluation split). MultiX is less accurate than single-mode approximations on the i.d. split, but better calibrated.
30

---

Scare
Accumacy
SECE
0:700
01/08 1001
0.0015
1:511
Deep Fissemble
0.312 0.007
0.019
-0015
0.005
1,88
0.0000
MCD
0.274 0.024
0.710 0.021
0,158 1..013
-0.15
1.001
1.451
0.073
MulsMCD
0.316 0.012
0.763
0.006
0.024
0.001
0008
0.007
1,021
0.0.2
SWAG
0.300 0:008
0.235 0:005
0.117 0.00
-0.17 0.005
1.317
0.030
MulsSWAG
0.337 0.005
0.762
0.001
0,033 0.000
0,033 1 0.001
1.009
0.001
LL SWAG
0.254 1: 0.003
0.721 0.025
0.104 0.019
-0.144 1 0.020
1.296
0.091
LL Laplace
0.227) = 0.310
0LEN 0.15
0J053 0.015
0.052 + 0.017
1.567 0.083
L.L. MukkiLaglase
0.334 = 11.007
0.789 00044
0.046 BJIKS
0,046 2 0.001
1.197
0.022
L. RAR
0.282 + 0.011
0718 0.009
0.007 0.008
-0.055 + 0.005
1.543 0.054
L. MukiBR
0.312 + 0.008
0.748 0.022
0.015
0.003
0.0012 + 0.002
1.151 0.0
Ronk-1 VI
0.265 + 0.008
0,780 0.066
0.075 0.001
-0.766 0.011
1.198
0.03
ALLVVON
0.255 1 0.000
0.725 0.010
0.058 0.015
-0.081 1 0.014
1.531 0.040
LL MaliiVNN
0.250 1 0.006
0.763
0.003
0.019 0:0.00
0.011 + 0.003
1.000 0.006
SVGD
112200 4: 11:02
0.723 0.008
0,146 0.004
-0.165 2 0,004
1,619 0.017
LVGD
0,355 4 01.18
0.787 1 0.014
0.118 0.000
-0.117 1 0.003
1.447 0.045
SNOI
0.270 = 0.000
0.707 + 0.015
0,100 0.015
-0.16
1,450
0.005
(a) O.o.d. Test Split
Model
Macro Score
Accunacy
ECE
SECB
NLL
MAP
0,320 1 0.01Y
0S15 0107
0,101 1 0.00
-0.01 1 0.00
1.121
0.087
Deep Ensemble
0.388 x 0.005
0.752 1: 0107
0.020 1.0001
-0.015 0.000
1,057
0.0.2
MCD)
0:657 + 0.010
0811 + 0.002
0.100-+00.01
-0.103++.0001
1,106
0.041
MaliMCD
0,311 1 00001
0.762 0.006
0,013 0.003
-0.008 1. 0.006
1,024 0.036
SWAG
0.491 1 0.011
0.832 0.005
0,087 1 0.002
-0087 1 0.002
0.987 0.0.5
MaliSWAG
11333 = 0.011
0.761
0.002
0,033 = 0.002
-0035 + 0.012
1,08
0.002
LL SWAG
01:65 = 1..M
0.119 2: 0.05
0 0.012
-.888 2 0.012
1,02
0.001
LL Laplace
11.656 = 11.17
0.100 0.005
0,022 1 0.000
-0025 1 0.030
1,45 0.038
1.1. Mukril aplace
0.489 = 0,012
0.836 + 0.001
0.027 0.000
0,027 + 0.003
0,839
0.012
L.L BBB
0,642 = 0.011
0816 0.005
0,075 0.000
-0.075 0.003
1,143 0.030
LL MukeBBB
0.316 0.006
0,449 0.0002
0.015 0.003
-0.011 1 0.003
1,165 0.000
Rank-I VI
0.612 5 0.005
0
0.052 0.001
-.0012
0,950
0.052
IVON
0,447 2 0.315
0.112 0.005
0,076
-0.075
1,002
MuliiVON
0,254 1 0.004
0,765 0.005
0,019 1 0.000
0.003
1,035
SVGD
0,630 1 0.024
0.15 1
0,106 0.00s
-0.105 1 0.00S
1,500
0.136
SVGD
11308
0.001
1.185
SNGP
0.639 11007
0,821
0.006
1,081
(b) I.d. Validation Split
Table 10: IWILDCAM-WILDS Detailed results on the evaluation splits. LL Last-Layer. For the MultiX models, the entire model is ensembled.
G.3.4 FMOW-WILDS
Following Koh et al. [47], we finetune DenseNet-121 [35], pretrained on ImageNet [14], for 50 epochs with the Adam optimizer [43] with a batch size of 64 and a learning rate of 10-4 that decays by factor of 0.96 per epoch. For each model, we replace the linear classification layer of the DenseNet-121 by randomly initialized one of the appropriate output dimension. iVON uses a prior precision of 100. We use five seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the five evaluation runs.
We report in the main paper that the Laplace approximation underfits, with a worst-region accuracy of 0.217 + 0.012 and sECE of -0.583 + 0.015 on the o.o.d. test split. Similarly, MultiLaplace only achieves a worst-region accuracy of 0.301 + 0.004 and sECE of 0.123 + 0.004 on the o.o.d. evalu- ation split. The accuracy doesn't I change when using 100 posterior samples during evaluation, but increases to 0.243 for 1000 posterior samples. However, using so many samples incurs significant computational overhead. Note that the better results of Daxberger et [13] are most likely due to their usage of models pretrained with ERM. Figure 13 shows additional results for the other models across all regions on the o.o.d, evaluation split, as well as the ECE on the worst region.
31

---

BCB (Woss Region)
SECT (All) Regions)
RCE (All Rogians)
-0.2
0.4
0.2
0.3
-03
0.1
0.25
0.3
0.55
0.4
0.45
0.5
0.55
0.4
0.45
0.5
0.55
Accuracy (Worst Region)
Accurncy (All Regions)
Aecurayy (All Regions)
MAP
MCD
SWAG
BBB
IVON
Rank-1 VI
WILDS Lender
Deep Ensemble
MaleMCD
MaInSWAG
MobBBB
MolaiVON
SVGD
SNGP
Figure 13: FMOW-WILDS: Accuracy, SECE and ECE the o.o.d. evaluation split for the region with the lowest accuracy and across all regions (see Figure 2b for accuracy vs. sECE on the worst region). All models are underconfident when evaluated across all regions, but MultiX is less underconfident.
Model MAP
WR Accurccy
WR BCE
WR JECE
WR MLL
Ang Accurcc 0.518 2 0.000
Avg BCE
Avg 4BCE
Arg NEL
0.300 ILB
0.825 0.000
-0.52 0,00
5,499 2..117
0.501
0.022
-1.55
0.002
3,905 0,025
Dorp Envrmbee
01033
0:71 0.04
-0.771 0.000
3.466-1
0.50
0100
0.06
0/06
0/00
DWAG
0.315 (L932
0.301 = 0,007
0.500
4:318 0.007
0.531
0.227 0.003
0.227
0.035
8.150 0.055
MallSWAG
0:33 0.903
0.270 0.005
-0.27
0.00
3.313 0.016
0.500 0.00
0.121 0.00
-6:121
0190
2,016
0.0SS
0.005
0.315 0.945
-0.36
0.00
5/885 1.0.3
1.000
-0.31
01933
3:271 0.05
Laplac
0.280=00.00
0.005
0000
5.346
0.643
0.449
0.252
LL Munta
0.301 01094
0.183 2: 0.004
0.123 0.004
4,086 208847
0,317
0.002
0.000 =: 0.002
0.030 =: 0.002
2.744 07017 0.17
11. Muhhlaplace (100 Samples)
0.:112 -093
0.123 2: 0.003
-0.125
0.005
1.688 0.000
0.517 3 0.00
0.00 = 0.002
0.020
0.002
2.26 0005
RBB
0:418 0.000
-0.165
0.3365
0.50
0.65
Mark-1 VI
01356 0/977
0.497
-0.97
1:455 0.1447
0.512 0.000
0.225
-0.325
2 011T
LL IVON
0.350 0.025
0.314 0,230
-0.14 0,00
4,357 0357220013 6.112
0.56 0.00
0.523 0,002
-0.555 2::15:21000 0.022
2.107 0.25
LL MuliVON
0.511 0.004
0211 = 00004
0024 0.00%
3.117 0.025
0.300 0.00
0.112 0JM2
-.1112 0002
2050 0.00
SVGID
0.007
0.3%6 0.000
-0.22 0.000
5.3
0.117 0.00
-0331 0193
3355 0.01
Table 11: FMOW-WILDS: WILDS: Detailed results the o.o.d. evaluation split the worst region as measured by the accuracy on each region and across all regions. For the MultiX models, the entire model is ensembled, but the single-mode approximation only applied to the classification head. WR = Worst Region, I.I Last-Layer.
G.3.5 RXRXI-WILDS
Following Koh et al. [47], we finetune ResNet-50 [32], pretrained on ImageNet [14], for 90 epochs with the Adam optimizer [43]. For each model, replace the linear classification layer of the ResNet-50 by a randomly initialized one of the appropriate output dimension. Following Koh et al. [47], a learning
of 10-4 and weight decay 10-5, For MCD, we try dropout rates of 0.1 and 0.2 and select 0.1 due to a slightly better accuracy on the evaluation split. iVON uses prior precision of 100 optimized by grid search. We five seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the five evaluation runs.
Similar to FMOW, Laplace underperforms accuracy-wise compared to the non-Vi algorithms. While do find significant inerease in accuracy to 0.061 = 0.002 when using 100 posterior samples, Laplace still performs worse than even MAP. However, Laplace is better calibrated with an sECE of -0.028 = 0.001.
32

---

01.02
SECC2
-1.00
-0.1
-0.12 1/
Accuncy
10,3-
1.0
0.5.A 1.00
L.0.A 0.30
Ronk-1
Figure 14: EXRX1-WILDD BBB and Rank-1 VI under different prior variances a and posterior temperatures A. We multiply À to the KI. divergence the ELBO during training to reduce the regularization strength. However, neither small prior variances nor colder posteriors make BBB competitive with the non-VI algorithms.
Model
BCE
4BC
a.dd
o.d BCE
JBCE
"IAF""
100S
-1-352
1000
KBB5P 0.121
CUKT 1 0001
0 12R21X000 3.03
-I152 103155
Dapp Enommible
0.156 0.00
0.08 0.001
-0.00 0.00
5,211
0.012
0.122 1 1.0000
0.01 1 0.000
-0051
0,002
5.677 = 0.019
MCD
0.106 = 0.001
0.807 0.003
00.27 0.000
0,021 = 0.055
0.863 = 0.00
0.288 2-306
0.004
7:501 2.043
0.158
6:227 0.008
0.121 0.000
0.000
0.008
-0.260
MahiswaG
0.167 0.001
04655
-0.02 0.00
5:590 0.000
0.126
0.001
0.855 4:000
-0078 0.001
5,514 1-011
LL Laplace
DU122 0.001
0.007 2 0.001
-0.55 0.001
16.250 0.566
0.010
2
0:200:0:000
0.090
0.001
15,500 1: 0.67
LL Laslace (100 Sacpless
0.77 2 0.000
0,001 0.034
0.011 0.000
1.024 0.200
GUMi 2.002
0.087 2 2.007
0828 0,007
5,007 2 0.27
0015 0.001
-0.00 0.000
6.017 0.018
0 0.00
0.015 0.00
-0015 0.000
5.577 0012
0.007 0.002
0.012 = 0.017
0.001 0.00
0,015
00,00
LL BBB
0.000:000000
0,005
0.02
11 6A 1.0 0.2)
0.044 0.001
(.102 0.002
-0.30 0.002
7.595 0.044
0.098 0001
0.317 3.000
-0.117 0.000
7.007 0.050
Rari: VI
BUGS 0.001
0,063 0.00)
0,088 0.001
7,200 0.023
0.043 0011
0,075 2.000
0005 0,200
7:077
0.04
LL IVON
0005 = 0.000
0.008 = 0.000
0.000 0.000
7.176= 0012
0 =
0.000 0.001
0.009
0.0.01
7:511 0.03
0.102
0.51 0.005
-0.354
6.00
8.254 0.000
0 0.022
0.382 0.005
-0.922 0,006
0.088
SNOP
Table 12: RXRXI-WILDS: Detailed results on the i.d. and the o.o.d. evaluation split. LL= Last-LaLay For the MultiX models, the entire model is ensembled, but the single-mode approximation is only applied to the classification head. We evaluate multiple hyperparameter combinations for LL BBB, as the standard parameters do not perform well. The failure of VI is equally present with LL BBB, LL Rank-1 I VI, and LL iVON.
G.3.6 CIVILCONMENTT-WIIDD
We the pretrained DistilBERT [76] model from HuggingFace transformers [91] with classifica- tion head consisting of two linear layers with a ReLU nonlinearity and Dropout unit with a drop rate of 0.2 between them. Following Koh et al. [47], we finetune the pretrained checkpoint with learning rate of 1 10-5 and, where applicable, a weight decay factor of 1- 10-2 for three epochs using the Adam optimizer [43]. SWAG collects ten parameter samples during the last two epochs of training. iVON uses prior precision of 500, as optimized by a grid search. We use five seeds for all non-ensembled models. The ensembles are build from four of the five single-model versions, leaving out a different member per model to create five different ensembled models of four members each.
We note in the main paper that MCD results in less accurate and more overconfident models. We investigate this further by experimenting with different dropout rates in Figure 15. While a dropout rate of 0.1 had no impact, dropout rates of 0.05 and 0.01 lead progressively better accuracy and calibration, coming close to MAP. However, there is still no accuracy or calibration benefit to be gained from using MCD.
G.3.7 AMAZON-WILDS
We use the pretrained DistilBERT [76] model from HuggingFace transformers [91] with classifica- tion head consisting of two linear layers with a ReLU nonlinearity and Dropout unit with a drop rate of 0.2 between them. Following Koh et al. [47], we finetune the pretrained checkpoint with learning rate of 10-5 and, where applicable, a weight decay factor of 10-2 using the Adam optimizer [43]. Contrary to Koh al. [47], we finetune for five epochs, we find that the validation accuracy
33

---

-0.35
SECE
-0.45
0.35 0.08 Woest Group Accuracy
0.2
p 0.05
0.01
p=
Figure 15: CIVILCOMMENTS WILDS: Accuracy and sECE for different MCD dropout rates p. While smaller dropout rates improve the accuracy, the models are still less accurate and more overconfident than MAP.
Model MAP
WG Accuracs
WG ECE
WG&ECE
WG NIL
Ave Aucuracs
Avg ECE
Avp JECE
Jug. NLL
0.420 2 [0121
G.51 20.00
L 00200
10100 2 [001
0.0.2 000015
G.012 00.013
0,207 I U.000
Derp Fraemhi
0.119 0.08
6.39 0.009
00.00
1/166
0916 0.000
0.0:0-1
-0006 00001
6:301 0.000
MCD 0.3)
0:325 0.021
0.012
-0.15
0 0.000
0.005
0.006
0.000
0,204 0.000
MCD 6.05)
0.354 1 0.018
0.505 0.024
-6.503
0.0004
1 0.065
0918 3 0.000
0.065 2: 0.002
-0.005
0.004
0,208
0.001
MCD (p 0.01)
0.266 2: 0.05
0.0055
-0.374-00 0.000
1.452 2 0:314
D917 5 0450
0.0:1 0011==0..00 0.001
-0.011
0.0013
0,205 0.201
MulsMCD (p 0.2)
0.228 2: 0.005
06612:00.000 0.000
-0.412 0.0007
L 0.022
0.90 2 0.000
0.000 0.0002
0.000 0..00
0.203
0.000
XWAG:
8:148:1 0177
0.197
1.0004
-0.184
0.027
0,871 0.000
0 0.074
0:132:4:001 0.017
0.153 00:104
0.206 0.19
LL
0.015
0.547 0.0008
0.001
0.207
T. Matil
01:20 0.006
0.346 0.005
-0.548 0.0.00
1.411 0.000
0.16 0.000
0.010 0.001
-4.009 0.001
0.301 0.000
LL BBB
0.587 +: 0.003
0.362 0.000
0.361 0.003
2,102 2: 0.278
0.018 22 DU622
0.036
00050=00.002 0.002
0.066 0.002
0,333 = 0.017
LL MalnBSB
0.525 2: 0.0012
0.338 0.001
-0.55% 0.0.11
1,501 1 000000
0822 3 0.000
0.011 0.001
-6.11
0.000
0.365 2 0.00
Rask-1 VI
0.540 2: 0.028
0:273 0.0000
0.273 0.273 0.0000
2.00 107
1017 = 0002
0.000
0.000 0.0002
0.319 = 0.00
0.045
6.421
-6.421 60.04
2 0.03
0.002
-0.04
0.300
McliVONN
0.015
0.002
SWORD
0.354 0.068
0.079
0:007
1 0.100 154
9 D003
0:005
-0.005
0.205 0.002
SNOP
0.334 2: 0.030
0.368 0.085
-0.588 0.000
1.341 0,078
0.919 2 0.001
0.0.4
-0.014 0.005
0,206 a: 0.004
Figure 16: CIVILCOMMENTS-WLDD Detailed results on the o.o.d. evaluation split for the worst group (WG, determined by the accuracy on each group) and averaged over all groups. L.I. Last- Layer.
is still increasing after three epochs. SWAG collects 30 parameter samples during the last two epochs of training. We also experiment with last-layer versions of SWAG and MCD, but find both to perform very similar to MAP (see Table 13). iVON uses a prior precision of 500, as optimized by a grid search. We six seeds for all non-ensembled models. The ensembles are build from five of the six single-model versions, leaving out a different member per model to create five different ensembled models of five members each.
Madd
DCT
cod JECT
14. 10-a
BCT
14 JECE
1ANEL
11E SIRE
11 SEOT
2110
MCD
0.17 6 m/01
0.07 0.003
6100
4 m00
03911.03 0.35
0.774-4 5m1
0575 0391
0102 0.0
-0,00 6,13
E20 Eve0
MCD
01111 210305
11305 10311
2101
11302
0151
1030 500
1121
01462 146520000
1100
5380
01311
11:37 21002
SWAN
0222 0020
0555 1355-1853
-0344 0000
5000
0357 13377
01:1
-3100 31100
02-14
NO
01002 11030
0509 -1000
5164
1002 1000
1131
0330
01:3 0100
-3001 3100
02004 (000
0.33 2 0.000
0.0000
0110 2310
-00G 5000
0778 0.001
0.355 2 0.000
0.66 0.000
0179 010
-3702 33160
0,548 0.022
0.08 21000
2000
-008 000
0.24 5100
0S5: 100
0137 -00
-3100 1105
SYCD
-03.5 EURC
830 6311
B 5000
0.522 023
0535
-3104 11265
5200
0581 0100
5166
-000 6006
(1000 0.22
B-87 2003
0385 6000
0136 0.5
-3100 43166
Table 13: AMAZON-WILDS:: Detailed results the i.d. and the o.o.d. evaluation splits. I.I
Last-Layer.
34


Reviews:
[{"review_id": "WXgIE9PVdJ", "reviewer": "Reviewer_CBPk", "summary": "This paper compares a variety of Bayesian Deep Learning methods on the WILDS dataset collection comprised of regression and classification tasks with the aim of evaluation generalization performance under distribution shifts. In addition to previously proposed single-mode approximation methods, the authors extend the methodology to sampling ensembles of these kinds of methods and show an improvement on various tasks. Additionally, the authors adapt existing calibration metrics to a signed version that is able to reflect under- or over confidence more directly. \n\nGeneral comment:\n- As you state, your paper is in line with the work of (Band et al. 2021) which was submitted to and accepted at the Neurips Benchmark track. The [Neurips website](https://nips.cc/Conferences/2023/CallForDatasetsBenchmarks) states that it is suited among other criteria for \"Systematic analyses of existing systems on novel datasets yielding important new insight.\" which to me perfectly describes this work and in conclusion seems to be a better fit than the main track.", "questions": "- I am slightly confused by how the methodology and code defines a predictive distribution or predictive uncertainty. It appears that you use a Gaussian Mixture Model approach for the Deep Ensembles, but in the appendix you state that \"We optimize the log likelihood of the training data and use a fixed standard deviation of 0.1, as this is the value MAP converges to when jointly optimizing the standard deviation and the model's parameters\". Is this supposed to be an estimate of the aleatoric uncertainty that is added to the epistemic uncertainty you obtain when computing a variance over the sampled point predictions?\n- Why can a pretrained model like resnet18 not be used for the povertyMAP task, but instead you train a model from scratch? At least I was not able to find loading an image-net pretrained weights in the code and there is lots of papers in the earth observation domain that show that image-net outperforms training from scratch even on satellite based optical imagery (as a latest example [Corley et al 2023](https://arxiv.org/pdf/2305.13456.pdf))\n- from my understanding a strength of the Laplace approximation is that it yields an uncertainty estimate on top of the MAP estimate and it should theoretically yield the same mean prediction as the MAP model which should be reflected in the same accuracy metric, however, this is not the case in almost all experiments, like Figure 2 and 3. I understand that you are using a sampling approach but perhaps you are not using enough samples? A striking example seems Figure 3 where you state that Laplace underperforms on FMoW and RxRx1 while a Deep Ensemble that is also based on MAP estimates performs well.\n- As you state in your conclusion, BBB performs quiet well on a variety of tasks but it appears they are always beat by the much simpler MC-Dropout approach in terms of the TV metric when compared to the gold standard HMC uncertainty estimation (Figure 5). I was wondering whether you could comment on this discrepancy and whether you have a possible explanation for this?", "limitations": "The limitations are accurately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- extensive experiments across image classification and regression tasks as well as text based classification\n- released code can be run without errors and provides adequate comments about running the code\n- the methodology of the paper builds on established datasets and methods and provides a useful benchmark of BDL on more realistic datasets", "weaknesses": "- I believe the paper could benefit from a small discussion about why Bayesian Deep Learning in general should be well suited to OOD tasks since Bayesian methods through Bayes rule assume that the test data follows the same distribution as the training data. A relevant discussion seems to be this paper for example (https://arxiv.org/abs/2110.06020) \n- I think the training/evaluation setup over the different folds and how those folds are created should be explained more explicitly\n- The wide variety of methods you compare is a great strength of the paper, however, I was wondering why you did not include Deep Kernel Learning (DKL) ([Wilson et al. 2015](https://arxiv.org/abs/1511.02222)) or an \"improvement\" of it like DUE ([Amersfoort et al. 2021](https://arxiv.org/abs/2102.11409)) since their experiments show good generalization performance on a variety of tasks and are also theoretically grounded in Gaussian Processes which many regard as a \"gold standard\" in Bayesian Machine Learning and UQ\n- you state that you provide the first systematic evaluation of BDL for finetuning large pre-trained models, but I would argue that the common practice of using imagenet weights for example is also finetuning? Or are you saying your experiments are the first systematic evaluation with respect to binary text classification?\n- the performance of \"generalizability\" seems to focus on accuracy metrics such as the pearson coefficient or F1 score for classification, however, isn't a more suitable metric the NLL or other proper scoring rules [Gneiting and Raferty 2007](https://www.tandfonline.com/doi/abs/10.1198/016214506000001437) because accuracy metrics ignore the predictive uncertainty? I am aware that these are provided for the regression task in the appendix but should they not be part of the main analysis?\n\nExtra comments:\n- On some tables in the Appendix I am not clear about the use of bold face, for example in Table 8 the entire MSE column is bold, or I misread the table?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive and detailed feedback. We hope the following points help to resolve all your questions regarding our work. If you have any further questions, please let us know so that we can clarify things further. Please also see the global response for a list of takeaway messages, results for SNGP, and a list of all citations.\n\n**NeurIPS Benchmark track.** We agree with you that the Benchmark track would have also been a good fit for our paper. Before submitting the paper, we reviewed both the main track and the Benchmark track and decided for the main track, since the NeurIPS Call for Papers for the main track explicitly lists \"Evaluation (e.g., methodology, meta studies, replicability and validity)\" as a submission topic and the Benchmark track's Call for Papers states \"It is also still possible to submit datasets and benchmarks to the main conference\".\n\n**Discussion about the usefulness of BDL.** Thank you very much for the suggestion. We added the proposed discussion to the introduction, mentioning both the theoretical results of [12], the intuition about the Bayesian Model Average given by [4], and the good OOD calibration results that BDL can achieve in practice as reported by practical evaluations [1, 4, 6], including our own.\n\n**Training/Evaluation Setup.** We added an explicit description of the folds and training/test setup to Appendix G.3. For all WILDS tasks, we used the same splits as [3] to ensure that our results are directly comparable with their results. We used in-distribution validation splits and, where available, OOD validation splits for hyperparameter optimization, in line with the protocol of [3].\n\n**Deep Kernel Learning.** Thank you for the suggestion! We added SNGP, a scalable variant of DKL, as an additional baseline. Please see our global response for further details.\n\n**Initialization from ImageNet weights.** We agree with you that initializing from ImageNet weights can also be considered finetuning. However, to the best of our knowledge, our evaluation is the first to systematically evaluate finetuning across a wide range of BDL algorithms and tasks. For example, neither of the BDL evaluations of [8] and [5] consider finetuning, and [2] do so only in a very limited fashion. We clarify this point in the revised version of our manuscript.\n\n**NLL Metric.** We decided to specifically separate predictive accuracy and uncertainty, as we find the NLL to be hard to interpret, given that it mixes both the accuracy and calibration into a single metric. Especially when considering practical applications of BDL, we believe that it is relevant to be able to assess accuracy and uncertainty independently, which is why we decided to provide these metrics in the main text and move the NLL to the appendix. In our revised paper, we more prominently reference the NLL and also include it for all tasks where it is not yet present in Appendix G. We also provide the extended tables for FMoW and CivilComments in the uploaded PDF file (Table 1).\n\n**Bold face in tables.** The bold face is intended to highlight the best performing methods within two times standard error. However, there are a few tables in the appendix where we incorrectly highlighted all numbers. Thanks for catching this! We will fix it in the revised version.\n\n**Predictive Distribution & Ensembles.** As you correctly state, we treat ensembles as mixture models where each mixture component receives equal weight. The standard deviation in the NLL of PovertyMap does indeed represent the aleatoric uncertainty. We clarified this in Appendix G.3.2.\n\n**Pretraining on PovertyMap.** We agree with you that a pretrained model would most likely have improved the model's accuracy. However, our goal was to stay consistent with [3] whenever possible, so that our results can be fairly compared to their non-Bayesian baselines. In particular, [3] do not use a pretrained model on PovertyMap (see Appendix E.8.1 of [3]).\n\n**Performance of Laplace.** Following your remarks, we add an extended discussion of how we made sure that our implementation works correctly in our revised manuscript and also in the uploaded PDF file. To check for sampling issues, we evaluated Laplace with 1000 posterior samples on FMoW and 100 on RxRx1 and found an increase in accuracy, with Laplace still performing worse than MAP (see Figure 2 in the uploaded PDF file). We will rerun the Laplace evaluations for all tasks and include the results in the revision of the paper, with an explicit mention of the different number of posterior samples. Further, we will add a discussion of the sampling issues in the appendix. To further check our implementation, we evaluated the Laplace model at its parameter posterior mean on FMoW and recovered the accuracy of MAP.\n\n**BBB vs. MCD.** The agreement with HMC reported in Figure 9 and Table 7 in our appendix may give an insight into this: BBB's and HMC's predictions more frequently differ from each other than MCD's and HMC's, which leads to a high TV in case of differing predictions. Further, our evaluation shows that the results on CIFAR, which is the only task where HMC samples are available to evaluate the TV, cannot be transferred to the text classification tasks where BBB performs particularly well compared to MCD.\n"}, {"review_id": "9HJz9TU5gU", "reviewer": "Reviewer_1hhR", "summary": "The paper conducts a large-scale benchmark of Bayesian deep learning (BDL) methods for distribution-shifted data. The focus is on evaluating the quality of the posterior approximation, generalization ability, and calibration using signed versions of calibration metrics to distinguish between under- and overconfidence. The BDL algorithms are evaluated on convolutional and transformer-based architectures, mostly on regression and classification tasks from the WILDS collection. For most tasks, the experiments show improved generalization and calibration when extending single-mode approximations to multiple modes through ensembling. The authors identify a limitation of ensemble-based methods when finetuning transformer-based language models. In this task, accuracy and calibration are not significantly improved over single-mode approximations.", "questions": "1) In the conclusion, the authors state that “[...] we demonstrated that BDL is in many cases competitive with algorithms that are specifically designed for o.o.d. generalization.” To justify that conclusion, additional comparisons to such algorithms, either from existing works or through experimental evaluation, should be added. Except for a hint to the WILDS leaderboard for the large-scale regression experiments, comparisons to specialized methods that are designed for o.o.d. generalization are missing from the main text.\n\n2) One main result of the paper is that ensembling-based BDL methods do not systematically improve upon the single-mode counterparts in finetuning of transformers-based language models. The authors hypothesize that this finding may be due to the nature of the finetuning task. There, the initialization from a pre-trained model may lead to a lack of diversity among the ensemble members. Considering the results of the “Finetuning for CNNs” section, where ensemble methods yield clear improvements, the hypothesis is not fully supported. To clarify the reason for the lack of performance of ensemble-based methods, results on a second architecture in addition to DistillBERT would be insightful. Experimental results to identify the mechanism among task, architecture (BERT-specific?), training procedure, or others, would further strengthen the submission.\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Although there are similar experimental surveys of BDL algorithms, the paper benchmarks a wider range of state-of-the-art BDL algorithms applied to modern neural network architectures. Considering the success of ensembling in applications and recent competitions, the choice of focusing on the evaluation of ensembling single-mode BDL algorithms is relevant. The lack of improvement in the generalization and calibration of such methods in transformer-based finetuning tasks is insightful. The introduced signed calibration metrics are useful for making the distinction between over- and underconfident models. The authors have included code with their submission, which is useful for reviewing the details of the experiments and algorithmic implementations. The experiments are described with sufficient clarity and detail.", "weaknesses": "1) The paper focuses largely on finetuning tasks, with the exceptions of corrupted CIFAR-10 and the large-scale regression task. The limitation should be described in the “Limitations” section or amended by including further experiments of models that are trained from scratch. Some of the algorithms have specific weaknesses or strengths that become only evident in such settings. For instance, ensemble-based methods may become computationally restrictive. Methods like iVON, SVGD, and BBB that modify the training objective or introduce noise to regularize the training may lead to base models with improved generalization or calibration. When focusing on finetuning of pre-trained models, these effects are neglected to some extent.\n\n2) The experiment on the RXRX1- WILDS datasets in Fig. 3 lacks results for several algorithms. The figure caption and main text do not justify this choice. The exception is the Laplace approximation, where underperformance is noted as a reason.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you a lot for your helpful and positive feedback. We are happy that you find our choice of algorithms to be \"relevant\", and the failure of ensembles on transformer-based finetuning tasks to be \"insightful\". We answer your remaining questions below. Please also see the global response for a list of takeaway messages, results for SNGP, and a list of all citations.\n\n**Finetuning tasks.** We added the following paragraph to the limitations: \n*Except for the results on CIFAR-10-(C) and PovertyMap-wilds, all results were obtained by finetuning pre-trained models and are therefore only valid in this setting.*\nNote that even when finetuning, training an ensemble requires training multiple models independently with the associated computational cost, since we finetune all layers of the models. Of course, training is still significantly less computationally expensive than when training all members from scratch.\n\n**Missing results for RxRx1 in Figure 3.** We excluded the VI algorithms for the same reason as Laplace: They are significantly less accurate than the other methods and including them in the figure would make the figure hardly readable.\nThe results can be found in the appendix in Figure 14. We added a reference to this figure in the caption of Figure 3 and made the reason for exclusion clearer.\n\n**Comparison with additional baselines.** Thank you very much for the suggestion. Since the WILDS leaderboard [3] contains extensive baselines for all WILDS tasks, including the classification tasks, we will include a small selection of results from WILDS and a more prominent reference to the leaderboard to better support our conclusion.\n\n**Additional results on transformers other than BERT.** Thank you very much for the suggestion. We agree with you that results on an additional architecture would be helpful, however we do find this to be infeasible given the significant computational cost of training ensembles. We decided on the two WILDS tasks due to the availability of baselines for this tasks on the WILDS leaderboard [3]. However, we do have additional results available for models trained with only very small regularization, as we hypothesized that this would allow the ensemble members to better diversify. Please see Figure 3 in the uploaded PDF. On CivilComments, we found that the accuracy of MAP increases when only using a weight decay factor of $10^{-4}$ (though not to the accuracy level of BBB), but the Deep Ensemble still does not perform better than a single model.\n"}, {"review_id": "LQTQrwxRAw", "reviewer": "Reviewer_Sjhm", "summary": "This paper presents a systematic comprehensive evaluation of a wide range of scalable Bayesian deep learning (BDL) algorithms in distribution-shited real-world data scenarios. A signed version of calibration error is presented, which allows for the identification of overconfidence and underconfidence rather than only absolute calibration. The study bolsters the ongoing effort in the BDL community to establish the practicality of BDL algorithms in safety-critical settings.", "questions": "-Can you summarise the implications for practitioners who use large-scale models and need to make decisions under distribution shifts?\\\n-Do you believe there is an inherent trade-off between accuracy, efficiency and calibration? \\\n-In your opinion, what are the key research directions to further improve the generalization and calibration capabilities of BDL in real-world distribution-shifted data scenarios?\\\n-How do frequentist methods like conformal prediction factor into this study?", "limitations": "See the weaknesses section above.\n\n-Figures 3 and 4 are a bit too small, consider using a horizontal legend for example.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In my opinion, this line of work has huge potential but is in dire need of a \"reality check\" in order to make tangible, impactful progress in real-world safety-critical settings. I believe this paper makes real progress in that direction. Indeed, the primary motivating statement behind uncertainty estimation is one which appeals to its utility in safety-critical applications, i.e. how and when can we trust our predictions. However, perhaps ironically, much emphasis has been placed on reiterating the former aspect by proposing more sophisticated uncertainty estimation algorithms, and not enough emphasis has been placed on establishing their utility and reproducibility in the very safety-critical settings they were designed for.\n\nThe paper is well-written and I found the empirics to be thorough and convincing. I believe this work will help guide the BDL community toward more practicable outcomes so I recommend acceptance.", "weaknesses": "One weakness of this type of empirical review is that - although appropriately broad and comprehensive for a conference submission - the empirical study can always be extended to other tasks which may or may not change the conclusions. Another weakness is the lack of novelty from a methodological perspective; although I do appreciate the proposed usage of a signed calibration error to identify under- or overconfidence without having to look at reliability diagrams.\n\nI would suggest including an itemized list of the main takeaways from the empirical study in the conclusion/discussion section of this paper. I think this would help organize the findings of the study and would provide easy-to-follow actionable advice for BDL practitioners going forward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you very much for your helpful and positive feedback. We are thrilled that you find that our paper \"makes real progress\" towards practically applicable BDL and that it is \"thorough and convincing\". We answer your remaining questions below. Please also see the global response for a list of takeaway messages, results for SNGP, and a list of all citations.\n\n**Itemized list of takeaways and implications for practitioners.** Thank you very much for the suggestion. We added a list of takeaway messages to the conclusion and provide it here on OpenReview as part of the global response.\n\n**Trade-off between accuracy, efficiency, and calibration.** Given our experiments, we do see a \"pick only two of the three options\" situation in the current state of the art. Given our current understanding of BDL, accurate posterior approximations should achieve accuracy and calibration gains, but are currently very expensive to obtain. However, we think it is conceivable that future algorithms may allow cheap but accurate Bayesian inference.\n\n**Key research directions.** We think that not only from our research it has become clear that multi-modal posterior approximations are central to BDL. While MultiX works well, it is computationally expensive and therefore unlikely to be of practical use in the current world of models with ever-growing parameter counts. Therefore, we think cheap multi-modal posterior approximations are one of the most important future research directions. In a similar spirit, we think post-hoc BDL algorithms such as the Laplace approximation have significant potential to be used in practice to provide uncertainty estimates on top of already trained models. More broadly, finetuning with Bayesian principles as evaluated by us is promising due to its relative computational cheapness, but requires further research, maybe even in the form of BDL algorithms specifically designed for finetuning. We will include some discussion along these lines in the conclusion of the paper revision.\n\n**Frequentist methods.** Frequentist methods are certainly an important and in many cases well working approach to uncertainty quantification. However, we see them as an orthogonal approach to the Bayesian methods that we evaluate, which build on the idea of inferring a posterior over the neural network's parameters. For practical applications of uncertainty quantification, of course all types of methods should be considered, including frequentist/deterministic uncertainty quantification methods. Please also see our global response for an extended discussion of our algorithm selection.\n\nFinally, we appreciate your suggestions regarding Figures 3 and 4 and will include this feedback in our revision.  "}, {"review_id": "0j4MnXsU53", "reviewer": "Reviewer_hcj9", "summary": "This paper presents an empirical comparison between various Bayesian methods/approaches on out-of-distribution (OOD) data. The authors focus on non-MCMC based methods (such as Bayes By Backprop, SWAG, and Laplace approximation). Methods are compared on challenging OOD benchmarks from the WILDS collection and using several network architectures. All compared methods are evaluated in two flavours, first as a single-mode posterior and second in an ensemble of models. Results are somewhat inconclusive, different methods tend to work better in terms of calibration and generalization on different datasets.", "questions": "* I may be wrong, but from the main text it seems that you evaluated BBB on the layer only, why not use it on the full network?\n* The reference format is a bit odd. For instance, not all authors names are written. Please check that you adhere to NeurIPS guidelines. ", "limitations": "The authors addressed limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* I think that the paper makes a nice contribution to the community. It may be hard to pick a model that perform well on OOD data in the Bayesian model zoo, and this paper makes another step towards a better understanding of this question. \n* These types of projects are hard to manage. There are many choices to be made in terms of methods, datasets, evaluation metrics, etc. I think that the authors made good choices in all of these aspects. For instance, they picked methods that reflect typical Bayesian methods people use, and they focused on 2-3 evaluation metrics that capture both uncertainty quantification and generalization capabilities adequately.\n* For the most part, the paper is written well. The authors justify many of their choices.\n* Evaluating the ensemble version of single-mode-posteriors is a good idea and was mandatory in my opinion.\n* The results seem reproducible, fully experimental details were given and the code was provided.", "weaknesses": "* I find it hard to understand the key takeaways. I think the authors should present the main conclusion from each experiment and the key takeaways from all experiments (e.g., inside a text box with some background color, or in bold).\n* As the results are inconclusive, e.g.,  Rank-1 VI is best calibrated on PovertyMAP, but less so on IWILDCAM, FMOW, and RXRX1, I believe it would have been beneficial to suggest a possible explanation (or perform an empirical analysis) for the reasons that make a method work better on some datasets and worse on others.\n* An important evaluation metric that is missing in my opinion is in terms of computational complexity (memory and time of each method). It may be important information when comparing the methods as well.\n* The authors evaluate last-layer Bayesian methods, which is important and great. An important method that I find missing is deep kernel learning [1, 2]. It is also a popular last-layer Bayesian method and, from my experience, it tends to work better than most of the compared methods.\n* In terms of exposition:\n  * I think that the authors should explain at the beginning of Section 5 or 5.1 that the analyses of the results are given in Sections 5.3 and 5.4. It is not clear until reaching there.\n  * In the figures, the font of the tick labels, axis labels, and method names should be larger.\n\n[1]  Wilson, A. G., Hu, Z., Salakhutdinov, R., & Xing, E. P. (2016, May). Deep kernel learning. In Artificial intelligence and statistics (pp. 370-378). PMLR.\n\n[2]  Calandra, R., Peters, J., Rasmussen, C. E., & Deisenroth, M. P. (2016, July). Manifold Gaussian processes for regression. In 2016 International joint conference on neural networks (IJCNN) (pp. 3338-3345). IEEE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your overall positive remarks. We are happy that you think \"that the paper makes a nice contribution to the community\" and that we made \"good choices\" regarding methods, datasets, and metrics. We address your remaining questions below. Please also see the global response for a list of takeaway messages, results for SNGP, and a list of all citations.\n\n**Key takeaways.** We added an itemized list of takeaway messages as part of the conclusion. Please see our global response for the takeaway message.\n\n**Explanation of performance differences.** Please note that our results are largely conclusive within groups of similar tasks, e.g. the image classification tasks iWildCam, FMoW, and RxRx1, and the text classification tasks CivilComments and Amazon. For Rank-1 VI in particular, we strongly suspect that the good performance on CIFAR (classification) and PovertyMap (regression) and the bad performance on the other datasets indicates that the multi-modal rank-1 factors are only sufficiently multi-modal when performing inference on the full network, but not when performing inference on only the last layers. Following your suggestions, we add a more clear discussion of this point in the revised manuscript.\n\n**Computational complexity.** We agree that computational complexity is an important metric. Please note that we report the runtime of each method on each dataset in the appendix (Section F, Tables 1 and 2). We will add a similar comparison for the memory consumption as given below:\n\n\n| Method     | Memory consumption relative to MAP                                 |\n|------------|--------------------------------------------------------------------|\n| MCD        | 1                                                                  |\n| SWAG       | 1                                                                  |\n| LL Laplace | $\\sim 1$                                                           |\n| Ensembles  | 1 (since the models can be trained independently)                  |\n| BBB        | 2                                                                  |\n| Rank-1 VI  | $\\sim 1$ + #components $\\cdot \\sqrt{\\text{parameter count}}$ |\n| SVGD       | #particles                                                      |\n| iVON       | $\\sim 2$, but no additional memory overhead due to an optimizer                                                      |\n\nThe memory overhead is of course lower when using the algorithms only on the last layer(s).\n\n**Deep Kernel Learning.** Thank you for the suggestion. We added SNGP, a scalable variant of DKL, as an additional baseline. Please see our global response for further details and a general remark regarding our algorithm selection.\n\n**Overview of Section 5.** Thank you very much for the suggestion. We added a short overview over the section as well as references to the subsections at the beginning of Section 5.\n\n**Font size in figures.** Again, thank you very much for the suggestion. We will take it into account for the next revision of the paper.\n\n**Last-layer BBB.** On CIFAR-10-(C) and PovertyMap, we trained BBB on the full network. On the other tasks from WILDS, we trained BBB only on the classification heads of the respective models. This was mainly done to limit BBB's runtime, which is of major concern given its computational overhead.\n\n**Reference format.** Thank you very much for the notice. We will check all references in the paper and ensure that they comply with the NeurIPS guidelines in our revision.\n"}, {"review_id": "BiIv68FjmK", "reviewer": "Reviewer_eT5S", "summary": "This paper evaluates several BDL algorithm on real-world datasets (and real-world distribution shifts) in terms of generalisation (accuracy/correlation metrics) and uncertainty estimation quality (calibration/signed calibration). ", "questions": "See also the weaknesses for the list of questions. Some additional questions are:\n\n1. What are the differences between WILDS and e.g., CIFAR-10, other than the \"realistic\" dataset shift? Would you be more satisfied with BDL people evaluating on Imagenet-C?\n\n2. How are hyperparameters selected?\n\n3. How do you tune the prior precision for the Laplace approximation? In my experience, it can be quite difficult to tune well. I tried to look in the appendix, but it said that I didn't have permission to view the file.\n\n4. Line 328 says Bayes-by-Backprop has a \"last-layer nature\". Why? People perform BBB on the full network.", "limitations": "Seems fine to me; see weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. I think there is a need for benchmarking BDL techniques, and the paper tries to tackle that problem.\n\n2. The point about batch normalisation leading to really poor o.o.d. data performance is interesting, and makes sense. This could be emphasised more as a contribution of the work (if it is not already known, I'm not sure!).", "weaknesses": "Overall, I vote for rejection. I am not convinced that this paper is of interest to the community, and ready for publication yet. However, I am open to changing my mind if I see evidence from the authors. I've included suggestions and questions for the authors that would enable me to raise my score. It's possible I misunderstood key details; I am more than happy to be corrected.\n\n1. I do not think the paper makes a substantial enough contribution to be accepted at NeurIPS. The contribution of the signed calibration metric is nice but small. If there was a clear takeaway recommendation from the experiment results, that would be a useful contribution to the community, but as I can understand, there isn't one. Despite this, the experiment section is mostly descriptive and does not provide greater insight and understanding about when one method is better than another. Either making the argument that the experiment section makes clearer and using the results to provide greater insight and understanding would help me increase my score. For example, the result that Bayes-by-Backprop performs very well here is interesting and surprising to me. Why is this the case? VI usually underfits, so does VI regularise to the base model more strongly? Which would be good in the pre-trained setup? This result contradicts other work ([1]) where VI often performs poorly and underfits. Please explain the contradiction. \n\n2. Missing reference to Nado et. al [1]. The need for baselines is clear, but what does this work offer over this work? I'm quite surprised that I didn't see the reference in the paper. For what it's worth, I think the Nado et al. baselines are also limited (only every evaluating marignal uncertainty, nothing about sequential decision making), so I think good work remains to be in this domain, but I don't think the paper offers much over this work. Would you please explain to me, in clear and precise terms, the contribution of this submission relative to this work? \n\n3. Missing baselines: give the point of this paper is the comparison (at least), it is important to be comprehensive in terms of the baselines covered. I would like to see SNGP, DUQ, and some frequentist approaches like temperature rescaling. This is a clear weakness of the paper. Further, not including function-space approaches is a major limitation: the paper claims to do a thorough investigation, and leaves out an important class of methods.\n\n4. A key point of the paper is that existing evaluations e.g., on CIFAR-10-C are unrealistic because the distribution shift is constructed and not real world. I think this point is interesting, but I would like to see evidence to support it. For example, if one would make different (and importantly different) conclusions based on benchmarking on CIFAR-10-C vs on WILDS. Further, other BDL papers do look at WILDS datasets e.g. [4], in case you weren't aware of this. Evaluating on WILDS is not sufficient for novelty, in my view.\n\n5. \"modern single-model BDL algorithms approximate the parameter posterior better than deep ensembles ...\" Well, for one, your evaluation looks at total variation in predictive space, so you can make no claim about the __parameter__ posterior. And, furthermore, recent work [5] shows that the HMC chains from the Izmailov et al paper have not converged well, so I don't think a claim about \"approximating the posterior\" is valid, you can only claim to approximate the HMC chains. And since the HMC chains perform badly out of distribution, it is not clear that lower total variation is better! See [5] on that, which argues that we might not want to be doing full network inference in the first place.\n\n6. The writing of the experiment section is very poor. The figures aren't referenced in the text, the writing is mostly around the results, but offers little in terms of understanding, insight, conclusions, or discussion. I found this very hard to read, and I am left confused: what are your take away messages? What is the contribution of the work?\n\n7. A claim is made \"BDL is in many cases competitive with algorithms that are specifically designed for OOD generalisation\" (Line 393,394). This does not seem to be supported: what benchmarks did you compare to that are specifically designed for OOD data?\n\n\nMinor points:\n- \"typical deep neural networks are highly confident on o.o.d. data\". I'd argue this is a bit different with large pretrained models.\n- Since your results used pre-trained model, [6, 7] are relevant citations because they justify pre-training in a Bayesian setting.\n\n[1] Nado, Zachary, et al. \"Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning.\" arXiv preprint arXiv:2106.04015 (2021).\n\n[2] Liu, Jeremiah, et al. \"Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.\" Advances in Neural Information Processing Systems 33 (2020): 7498-7512.\n\n[3] Van Amersfoort, Joost, et al. \"Uncertainty estimation using a single deep deterministic neural network.\" International conference on machine learning. PMLR, 2020.\n\n[4] Daxberger, Erik, et al. \"Laplace redux-effortless bayesian deep learning.\" Advances in Neural Information Processing Systems 34 (2021): 20089-20103.\n\n[5] Sharma, Mrinank, et al. \"Do Bayesian Neural Networks Need To Be Fully Stochastic?.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2023.\n\n[6] Shwartz-Ziv, Ravid, et al. \"Pre-train your loss: Easy bayesian transfer learning with informative priors.\" Advances in Neural Information Processing Systems 35 (2022): 27706-27715.\n\n[7] Sharma, Mrinank, et al. \"Incorporating Unlabelled Data into Bayesian Neural Networks.\" arXiv preprint arXiv:2304.01762 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your effort in reviewing our paper. We are happy that you found our results interesting and that you think there is a need for benchmarking BDL techniques. We will address your concerns regarding our evaluation below and refer to our global response, which summarizes our takeaway messages, presents preliminary results for SNGP, discusses our algorithm selection in detail, and provides the list of citations.\n\nOverall, we believe our results are highly relevant to the BDL community since we compare many current SOTA algorithms (MultiX, natural gradient VI) on a diverse range of realistic tasks (finetuning, transformer-based models, large-scale regression) and find surprising results (cf. our takeaway messages). Our results allow a direct comparison with the non-Bayesian OOD generalization baselines of [2]. Reviewers Sjhm, 1hhr, hcj9, and CBPk agree with us on the potential impact of the results.\n\n**Batch normalization.** We discuss this effect in more detail in Appendix E and refer to [1] which demonstrates the problem with Batch Normalization on more datasets.\n\n**Size of the contribution.** We agree that clear and actionable takeaway messages are very useful and will add them, as stated in the global response, to our revised conclusion. As we wanted to focus on the evaluation of a wide range of methods on many diverse datasets, we decided against investigating a single phenomenon in depth. This approach follows similar large-scale evaluations published at NeurIPS both within BDL [5,8] and other domains [14,15] (see the references below). Further, NeurIPS 2023 explicitly invites \"Evaluation (e.g., methodology, meta studies, replicability and validity)\" submissions. \n\n**Performance of BBB.** Please note that the performance of BBB on the image classification tasks is in line with the results of [2] on similar tasks. Further, the regularization of MAP on the text classification tasks is strong (weight decay of ${10^{-2}}$, as mandated by [3]). Reducing the weight decay makes MAP more accurate on CivilComments, but still less accurate than BBB. As shown in Figure 3 in the uploaded PDF, reducing the regularization of BBB by introducing a tempering factor does not significantly change BBB's accuracy. These results indicate that BBB's performance is at most partially due to a difference in regularization, and otherwise represents an actual benefit of performing mean-field VI. We will add an extended discussion of this topic to the paper revision.\n\n**Distinction to Nado et al.** We thank you for the suggestion to cite [2]. We added a reference and short summary in the related work section. We believe that our paper offers a number of important new results compared to their work:\n\n- A systematic evaluation of ensembles of probabilistic single-mode posterior approximations (MultiX). Reviewers hcj9 and 1hhR agree with us that this evaluation is highly relevant. Further, we include approaches not considered in [2], e.g., the Laplace approximation and iVON (natural gradient VI). \n- A systematic evaluation of BDL algorithms for the finetuning of pre-trained models.\n- The inclusion of a large-scale regression task. To the best of our knowledge, such an evaluation is yet completely missing from the BDL literature, including [2], who only evaluate on a number of small UCI datasets. \n- Insights into the posterior approximation quality by comparing to the HMC samples from [6].\n- Strict adherence to the evaluation protocol of [3], which allows for a fair comparison with the non-Bayesian baselines on the WILDS leaderboard. \n\n**Missing baselines.** Following your suggestion, we add a comparison with SNGP during the revision. We include preliminary results in the uploaded PDF file (Figure 1a). Furthermore, we discuss our algorithm selection in the global response and add this discussion to the appendix. \n\n**Realism of evaluations.** We want to clarify that we do not think that evaluations on CIFAR-10-C are unrealistic, but one of our main research questions is whether the conclusions on these artificial distribution shifts transfer to real-world distribution shifts. Regardless of our concrete results, we see this as an important point by itself, as it has direct implications for the applicability of BDL. We do find novel and surprising results (c.f. takeaway messages). Please also note that while [9] evaluates on WILDS, they finetune models from [2], originally trained with algorithms that are specially designed for domain generalization. This prevents a fair comparison of the algorithms' performance in isolation. Further, they consider a significantly smaller number of algorithms.\n\n**Comparison with HMC.** We changed the formulations in the paper to be more cautious about the implications in parameter space. We now refer to [7] to acknowledge the problems with the HMC samples, but argue that they currently still provide the best available way to assess the posterior approximation quality on a complex task.\n\n**Competitiveness with special OOD generalization algorithms.** We added an explicit reference to the WILDS leaderboard [3] which provides an extensive selection of OOD generalization algorithms. \n\n**Overconfidence of large pretrained models.** We agree that such issues are less severe on large models, yet, our results for MAP on FMoW (Figure 3b) and CivilComments (Figure 4) show that these models can still be highly overconfident on OOD data. \n\n**Hyperparameters.** We selected hyperparameters based on a combination of grid search and default values, and tuned the prior precision of Laplace based on the marginal likelihood of the training data.\n\nFinally, we appreciate your suggestions regarding our writing style and additional citations and will include this feedback in our revision.\n\n[14] Veilleux, Olivier, et al. \"Realistic evaluation of transductive few-shot learning\" NeurIPS 2021\n\n[15] Setlur, Amrith, et al. \"Two sides of meta-learning evaluation: In vs. out of distribution\" NeurIPS 2021\n"}]

Author rebuttal:
We would like to thank all reviewers for their time and efforts in providing detailed and insightful feedback, which we will incorporate in our revision.

**We are pleased that the reviewers are convinced that our paper "provides a useful benchmark of BDL on more realistic datasets" (Reviewer CBPk) and is "appropriately broad and comprehensive for a conference submission" (Reviewer Sjhm). The reviewers think our results are "insightful" (Reviewer 1hhR) and "convincing" (Reviewer Sjhm), and that the paper "makes real progress" (Reviewer Sjhm) towards real-world applicability of BDL.**

Reviewer eTS5 was hesitant to recommend acceptance due to similarities to other BDL evaluation papers and a focus on the evaluation itself, rather than a deep investigation of singular results. Therefore, we would like to emphasize that our work contains a number of novel results which we judge to be highly relevant for practitioners as summarized below.

## Key Takeaways
As requested by reviewers Sjhm, hcj9, and eTS5, we would like to summarize our key takeaways, which we expect to have direct implications for practitioners and method developers and have been unknown or not thoroughly shown until now. Since we evaluated on diverse, realistic distribution-shifted data, the results are likely directly applicable to many real-world applications of BDL.

- Finetuning only the last layers of pre-trained models with BDL algorithms gives a significant boost of generalization accuracy and calibration on realistic distribution-shifted data, while incurring a comparatively small runtime overhead. These models are in many cases competitive to or even outperform methods that are specially designed for OOD generalization such as IRM [12] and Fish [13].
- For CNNs, ensembles are more accurate and better calibrated on OOD data than single-mode posterior approximations by a wide margin, even when initializing all ensemble members from the same pre-trained checkpoint with only the last layers differently initialized, i.e. when not using the standard protocol of randomly initializing all ensemble members. Ensembling probabilistic single-mode posterior approximations such as SWAG or MCD yields only a small additional increase in accuracy and calibration. 
- When finetuning large transformers, ensembles yield no benefit. Compared to all other evaluated BDL algorithms, classical mean-field variational inference achieves significant accuracy gains under distribution shift. We want to emphasize the importance of this point: Our results show that ensembles, which are typically considered to be the SOTA in BDL, do not work well for finetuning large transformers, a setting that is highly relevant given the recent success of large language models.

In the paper revision, we added these points prominently as part of the conclusion of the paper.

## Algorithm Selection
**We follow the requests of reviewers eTS5, CBPk, and hcj9 and add SNGP [10], a scalable variant of DKL, as an additional baseline for the WILDS tasks. We provide preliminary results in the additionally uploaded PDF file (Figure 1).** Please note that these results are not directly comparable to our other results and the WILDS baselines due to necessary modifications of the architecture, in particular the addition of a GP. Due to the limited timeframe, we could so far only evaluate with the default hyperparameters from [19]. We are working on completing the results for all WILDS datasets.

Furthermore, as multiple reviewers requested additional algorithms, we want to clarify our selection of algorithms. We selected the algorithms based on the following criteria, that ensure scalability of the algorithms and allow a fair comparison with the non-Bayesian baselines of WILDS:

- Algorithms must scale to large parameter counts and datasets. This excludes MCMC-based methods.
- It must not be required to modify the underlying neural network. This ensures that the comparison with the WILDS baselines is fair, and excludes GP-based methods such as DKL, SNGP, and DUE.
- Algorithms must approximate the parameter posterior of the underlying neural network. This allows us to compare the algorithms with the HMC samples from [6], and excludes deterministic UQ methods such as DUQ.
- Algorithms must be applicable to both classification and regression problems. This excludes e.g. fBNN.

We belief that these criteria make for a fair comparison that is relevant to practitioners. In particular, reviewers hcj9, Sjhm, and 1hhR agree with us on the relevance of the selected algorithms.

[1] Schneider, Steffen et al. "Improving robustness against common corruptions by covariate shift adaptation" NeurIPS 2020

[2] Nado, Zachary, et al. "Uncertainty Baselines: Benchmarks for uncertainty \& robustness in deep learning." arXiv 2021

[3] Koh, Pang Wei, et al. "WILDS: A Benchmark of in-the-Wild Distribution Shifts" ICML 202

[4] Wilson, Andrew G., et al. "Bayesian Deep Learning and a Probabilistic Perspective of Generalization" NeurIPS 2020

[5] Band, Neil, et al. "Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks" NeurIPS 2021 Datasets and Benchmarks Track

[6] Izmailov, Pavel, et al. "What Are Bayesian Neural Network Posteriors Really Like?", ICML 2021

[7] Sharma, Mrinank, et al. "Do Bayesian Neural Networks Need To Be Fully Stochastic?" AISTATS 2023

[8] Ovadia, Yaniv, et al. "Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift" NeurIPS 2019

[9] Daxberger, Erik, et al. "Laplace redux - effortless bayesian deep learning" NeurIPS 2021

[10] Liu, Jeremiah, et al. "Simple and principled uncertainty estimation with deterministic deep learning via distance awareness" NeurIPS 2020 

[11] D'Angelo et al. "On out-of-distribution detection with Bayesian neural networks" arXiv 2022

[12] Arjovsky, Martin, et al. "Invariant risk minimization" arXiv 2019

[13] Yuge, Shi et al. "Gradient Matching for Domain Generalization" ICLR 2022


Meta-review: This paper performs a large-scale experimental study of Bayesian NNs under distribution shift, focusing on the WILDS benchmarks.  Of particular note are the experiments involving transformers and multi-modal (i.e. ensembled) versions of popular techniques (e.g. Laplace).  The work reports findings such as the benefits of last-layer fine-tuning, CNN's amenability to ensembling, and the lack of benefit of ensembles of transformers.  The reviewers are generally in favor of acceptance (1 x strong accept, 1 x accept, 2 x weak accept, 1 x borderline reject).  Reviewer eT5S's, the only to favor rejection, primary criticisms are (i) lack of contribution due to no novel methodology, (ii) other papers have previously evaluated on WILDS, and (iii) lack of clear takeaways / recommendations.  While these criticisms do have some merits, I side with the other reviewers that the scale of the experimental evaluation, including the aforementioned findings, does make for a sufficiently interesting contribution.  Moreover, the work goes well beyond previous work that has evaluated on WILDS and performed other largescale benchmarking, e.g. with the analysis of transformers and of multi-modal approximations.

The authors should address the above concerns in the camera-ready version, as promised.  Moreover, many of the plots can be improved in the size of their text and general format.  They are generally small and hard to read without zooming into the pdf.
Decision: Accept (poster)