SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	Accuracy	ECE	sECE	NLL	Agreement	TV
MAP	0.925 ± 0.001	0.045 ± 0.001	-0.045 ± 0.001	0.296 ± 0.006	0.906 ± 0.002	0.172 ± 0.001
Deep Ensemble	0.944 ± 0.001	0.010 ± 0.001	0.003 ± 0.000	0.174 ± 0.001	0.923 ± 0.001	0.144 ± 0.001
MCD	0.927 ± 0.002	0.008 ± 0.001	0.007 ± 0.001	0.216 ± 0.005	0.920 ± 0.003	0.132 ± 0.002
MultiMCD	0.941 ± 0.001	0.031 ± 0.001	0.031 ± 0.001	0.186 ± 0.001	0.930 ± 0.002	0.116 ± 0.001
SWAG	0.921 ± 0.002	0.042 ± 0.003	0.042 ± 0.003	0.250 ± 0.002	0.910 ± 0.002	0.130 ± 0.001
MultiSWAG	0.940 ± 0.001	0.099 ± 0.002	0.099 ± 0.002	0.258 ± 0.002	0.927 ± 0.001	0.107 ± 0.001
LL Laplace	0.924 ± 0.001	0.040 ± 0.001	-0.040 ± 0.001	0.282 ± 0.006	0.906 ± 0.002	0.169 ± 0.001
LL MultiLaplace	0.945 ± 0.001	0.012 ± 0.002	0.007 ± 0.001	0.174 ± 0.002	0.923 ± 0.001	0.142 ± 0.000
BBB	0.898 ± 0.003	0.046 ± 0.002	-0.046 ± 0.002	0.387 ± 0.011	0.900 ± 0.001	0.161 ± 0.001
MultiBBB	0.929 ± 0.001	0.018 ± 0.002	0.018 ± 0.002	0.228 ± 0.002	0.930 ± 0.001	0.122 ± 0.001
Rank1-VI	0.881 ± 0.003	0.041 ± 0.002	0.041 ± 0.002	0.363 ± 0.005	0.910 ± 0.003	0.116 ± 0.002
iVON	0.842 ± 0.004	0.025 ± 0.003	0.024 ± 0.003	0.464 ± 0.011	0.874 ± 0.004	0.145 ± 0.003
MultiiVON	0.881 ± 0.003	0.077 ± 0.003	0.077 ± 0.003	0.388 ± 0.002	0.921 ± 0.002	0.107 ± 0.001
SVGD	0.927 ± 0.001	0.018 ± 0.001	0.003 ± 0.001	0.255 ± 0.001	0.924 ± 0.002	0.135 ± 0.001
SNGP	0.917 ± 0.003	0.076 ± 0.006	0.076 ± 0.006	0.380 ± 0.013	0.903 ± 0.002	0.178 ± 0.002
HMC	0.903	0.069	0.068	0.320	1.000	0.000
(a) Standard Evaluation Split (Corruption Level 0)
Model	Accuracy	ECE	sECE	NLL	Agreement	TV
MAP	0.872 ± 0.002	0.080 ± 0.002	-0.080 ± 0.002	0.518 ± 0.010	0.848 ± 0.001	0.238 ± 0.001
Deep Ensemble	0.903 ± 0.001	0.014 ± 0.001	-0.003 ± 0.001	0.305 ± 0.006	0.870 ± 0.003	0.196 ± 0.001
MCD	0.872 ± 0.004	0.011 ± 0.003	-0.004 ± 0.006	0.400 ± 0.011	0.865 ± 0.003	0.184 ± 0.002
MultiMCD	0.890 ± 0.003	0.028 ± 0.002	0.026 ± 0.002	0.335 ± 0.005	0.880 ± 0.002	0.158 ± 0.001
SWAG	0.876 ± 0.004	0.047 ± 0.006	0.047 ± 0.005	0.388 ± 0.004	0.856 ± 0.005	0.179 ± 0.001
MultiSWAG	0.900 ± 0.001	0.117 ± 0.002	0.117 ± 0.002	0.383 ± 0.002	0.878 ± 0.001	0.147 ± 0.000
LL Laplace	0.873 ± 0.004	0.074 ± 0.004	-0.074 ± 0.004	0.499 ± 0.017	0.849 ± 0.002	0.239 ± 0.002
LL MultiLaplace	0.903 ± 0.005	0.016 ± 0.003	-0.001 ± 0.005	0.314 ± 0.003	0.859 ± 0.004	0.199 ± 0.001
BBB	0.839 ± 0.003	0.083 ± 0.003	-0.083 ± 0.003	0.682 ± 0.029	0.844 ± 0.002	0.222 ± 0.000
MultiBBB	0.878 ± 0.008	0.018 ± 0.005	0.006 ± 0.009	0.394 ± 0.012	0.880 ± 0.005	0.170 ± 0.004
Rank1-VI	0.843 ± 0.004	0.042 ± 0.004	0.042 ± 0.004	0.484 ± 0.015	0.860 ± 0.002	0.165 ± 0.003
iVON	0.795 ± 0.010	0.025 ± 0.006	0.010 ± 0.010	0.596 ± 0.021	0.827 ± 0.011	0.189 ± 0.008
MultiVON	0.845 ± 0.004	0.083 ± 0.003	0.081 ± 0.003	0.504 ± 0.018	0.863 ± 0.008	0.145 ± 0.003
SVGD	0.883 ± 0.001	0.021 ± 0.003	-0.007 ± 0.001	0.432 ± 0.010	0.875 ± 0.001	0.192 ± 0.000
SNGP	0.867 ± 0.015	0.074 ± 0.009	0.069 ± 0.012	0.560 ± 0.034	0.844 ± 0.010	0.231 ± 0.004
HMC	0.834	0.066	0.064	0.508	1.000	0.000
(b) Corruption Level 1
Model	Accuracy	ECE	sECE	NLL	Agreement	TV
MAP	0.805 ± 0.005	0.128 ± 0.005	-0.128 ± 0.004	0.863 ± 0.019	0.778 ± 0.002	0.309 ± 0.002
Deep Ensemble	0.838 ± 0.004	0.027 ± 0.004	-0.027 ± 0.004	0.518 ± 0.020	0.805 ± 0.002	0.253 ± 0.001
MCD	0.777 ± 0.011	0.047 ± 0.004	-0.047 ± 0.005	0.733 ± 0.051	0.796 ± 0.010	0.232 ± 0.001
MultiMCD	0.805 ± 0.003	0.011 ± 0.003	0.000 ± 0.003	0.594 ± 0.012	0.823 ± 0.005	0.196 ± 0.001
SWAG	0.806 ± 0.004	0.032 ± 0.003	0.031 ± 0.004	0.593 ± 0.009	0.783 ± 0.005	0.228 ± 0.003
MultiSWAG	0.839 ± 0.002	0.117 ± 0.004	0.117 ± 0.004	0.556 ± 0.002	0.818 ± 0.001	0.186 ± 0.001
LL Laplace	0.804 ± 0.003	0.120 ± 0.002	-0.119 ± 0.002	0.839 ± 0.021	0.777 ± 0.005	0.308 ± 0.003
LL MultiLaplace	0.850 ± 0.012	0.026 ± 0.008	-0.015 ± 0.009	0.498 ± 0.035	0.800 ± 0.002	0.251 ± 0.003
BBB	0.735 ± 0.009	0.154 ± 0.008	-0.154 ± 0.008	1.296 ± 0.072	0.774 ± 0.004	0.286 ± 0.002
MultiBBB	0.786 ± 0.014	0.033 ± 0.010	-0.026 ± 0.014	0.741 ± 0.045	0.830 ± 0.006	0.204 ± 0.003
Rank1-VI	0.774 ± 0.008	0.024 ± 0.002	0.019 ± 0.006	0.684 ± 0.027	0.802 ± 0.006	0.210 ± 0.005
iVON	0.725 ± 0.014	0.028 ± 0.004	-0.022 ± 0.005	0.809 ± 0.036	0.756 ± 0.016	0.237 ± 0.008
MultiiVON	0.783 ± 0.008	0.069 ± 0.009	0.068 ± 0.010	0.666 ± 0.006	0.821 ± 0.002	0.179 ± 0.003
SVGD	0.804 ± 0.004	0.038 ± 0.002	-0.038 ± 0.002	0.821 ± 0.024	0.818 ± 0.001	0.238 ± 0.001
SNGP	0.785 ± 0.015	0.067 ± 0.007	0.044 ± 0.010	0.837 ± 0.042	0.767 ± 0.008	0.295 ± 0.004
HMC	0.724	0.020	0.017	0.833	1.000	0.000
(c) Corruption Level 3
Model	Accuracy	ECE	sECE	NLL	Agreement	TV
MAP	0.689 ± 0.006	0.217 ± 0.006	-0.217 ± 0.006	1.494 ± 0.019	0.683 ± 0.005	0.390 ± 0.003
Deep Ensemble	0.733 ± 0.006	0.075 ± 0.007	-0.075 ± 0.007	0.937 ± 0.036	0.718 ± 0.004	0.310 ± 0.003
MCD	0.629 ± 0.009	0.141 ± 0.011	-0.141 ± 0.010	1.312 ± 0.094	0.725 ± 0.012	0.277 ± 0.003
MultiMCD	0.666 ± 0.007	0.069 ± 0.007	-0.069 ± 0.007	1.063 ± 0.021	0.760 ± 0.001	0.234 ± 0.001
SWAG	0.696 ± 0.005	0.018 ± 0.009	-0.012 ± 0.008	0.927 ± 0.025	0.699 ± 0.006	0.277 ± 0.006
MultiSWAG	0.728 ± 0.004	0.082 ± 0.004	0.082 ± 0.004	0.841 ± 0.009	0.740 ± 0.004	0.228 ± 0.002
LL Laplace	0.690 ± 0.006	0.203 ± 0.005	-0.203 ± 0.005	1.430 ± 0.026	0.685 ± 0.003	0.380 ± 0.002
LL MultiLaplace	0.731 ± 0.013	0.073 ± 0.011	-0.072 ± 0.011	0.941 ± 0.064	0.722 ± 0.008	0.303 ± 0.002
BBB	0.584 ± 0.011	0.268 ± 0.014	-0.268 ± 0.014	2.413 ± 0.126	0.698 ± 0.008	0.353 ± 0.003
MultiBBB	0.621 ± 0.014	0.114 ± 0.016	-0.114 ± 0.016	1.411 ± 0.105	0.757 ± 0.010	0.247 ± 0.009
Rank1-VI	0.673 ± 0.016	0.028 ± 0.010	-0.024 ± 0.014	1.010 ± 0.056	0.719 ± 0.013	0.262 ± 0.010
iVON	0.617 ± 0.011	0.086 ± 0.006	-0.086 ± 0.006	1.201 ± 0.061	0.678 ± 0.008	0.290 ± 0.007
MultiiVON	0.657 ± 0.012	0.025 ± 0.005	0.000 ± 0.009	0.998 ± 0.047	0.752 ± 0.012	0.224 ± 0.006
SVGD	0.666 ± 0.004	0.117 ± 0.009	-0.117 ± 0.009	1.557 ± 0.043	0.742 ± 0.007	0.286 ± 0.002
SNGP	0.657 ± 0.006	0.084 ± 0.007	-0.017 ± 0.008	1.256 ± 0.022	0.681 ± 0.008	0.353 ± 0.005
HMC	0.592	0.055	-0.054	1.225	1.000	0.000
(d) Corruption Level 5
Table 8: CIFAR-10: Detailed results on the standard evaluation split and the corruption levels 1, 3, and 5 of CIFAR-10-C.
Figure 9: CIFAR-10-(C): All results for the corruption intensities 0, 1, 3, and 5. The corruption intensities are denoted on the y-axis. A negative sECE indicates overconfidence, a positive sECE indicates underconfidence. The plot for the TV is repeated from Figure 5.
G.3 WILDS
We strictly follow the training and evaluation protocol of Koh et al. [47] by reusing their data folds for training, validation, and testing. We use the hyperparameters proposed by Koh et al. [47] where applicable, and set the other hyperparameters to standard values as suggested by the developers of the respective algorithms. If the standard values lead to unexpectedly bad results, we tune the hyperparameters through a grid search. Hyperparameter tuning was performed on the i.d. validation and, where available, o.o.d. validation splits, but never on testing splits. In particular, we select the prior precision of iVON through a grid search over the values 1, 10, 100, and 500 per model architecture. We find the prior precision of iVON to be hard to tune, as iVON frequently diverges for comparatively small prior precisions such as 1 and 10. BBB works always well with the standard unit prior. We also experiment with other priors but find no difference in performance except on RXRx1-wILDS (see Figure 14). BBB and iVON use two Monte Carlo samples during training. See the sections below for the hyperparameters that were chosen on the individual datasets. We use mixed precision training whenever possible. The VI algorithms as well as the Laplace approximations are mostly trained without mixed precision, as this leads to unstable training.
SNGP uses the same learning rate, weight decay, and number of epochs as the other algorithms. Following the recommendations by Liu et al. [54] and the tuning done by Nado et al. [65], we use a spectral normalization factor of 6.0 for the computer vision tasks and 0.95 for the text classification tasks. On the image classification tasks, SNGP performs significantly better when limiting the input
dimension of the Gaussian Process to 128 or 256 instead of using the output dimension of the previous network layer.
We use 10 posterior samples per prediction during evaluation to constrain the computational overhead of the Bayesian algorithms, which is generally sufficient to capture the predictive distribution [72]. Note that our results are not directly comparable to the results of Daxberger et al. [13], as they build their Deep Ensembles and Laplace approximations from the pretrained models provided by Koh et al. [47]. When comparing our results with the best performing algorithms on the WILDS leaderboard, we only consider the algorithms on the "overall leaderboard", i.e. the algorithms that conform to the official submission guidelines of Koh et al. [47].
G.3.1 Camelyon17-WILDS
Following Koh et al. [47], we train a DenseNet-121 [35] with SGD for 5 epochs with a learning rate of 0.001, weight decay 0.01 and momentum 0.9. SWAG collects 30 parameter samples during the last epoch.
G.3.2 PovertyMAP-WILDS
We train a ResNet-18 [32] using the same hyperparameters as Koh et al. [47] where applicable: A learning rate of  $10^{-3}$  and no weight decay. We only train for 100 epochs as all models were converged after that. SWAG collects 30 parameter samples starting at epoch 50. For BBB, we scale the KL divergence down with a factor of 0.2, as this significantly improves the MSE. Rank-1 VI uses an unscaled KL divergence. The ensembles, Rank-1 VI and SVGD use five members/components. We optimize the log likelihood of the training data and represent the aleatorc uncertainty with a fixed standard deviation of 0.1, as this is the value MAP converges to when jointly optimizing the standard deviation and the model's parameters. For the final evaluations, we do not optimize the standard deviation, as this leads to unstable training with the VI algorithms. Following Koh et al. [47], we aggregate all results over the five folds of POVERTYMAP, with one seed per fold.
As mentioned in the main paper, iVON performs significantly worse than the other algorithms. We conducted a grid search over prior precisions 1, 10, 100 and 500 with a single seed per value, and found that for 1 and 10 iVON diverges, for 100 iVON achieves an o.o.d. Pearson coefficient on the "A" split of 0.21 and for 500 it achieves a Pearson coefficient of 0.25. Most likely due to their underfitting the non-diverged models are comparatively well calibrated with sCEEs of  $-0.21$  for a prior precision of 100 and  $-0.24$  for a prior precision of 500.
Log Marginal Likelihood. The log marginal likelihood is commonly used to jointly evaluate the accuracy and calibration of a regression model. On an evaluation dataset  $\mathcal{D}'$ , the log marginal likelihood (LML) is given by
$$
\mathrm {L M L} = \log p (\mathcal {D} ^ {\prime} \mid \mathcal {D}) = \log \int p (\mathcal {D} ^ {\prime} \mid \boldsymbol {\theta}) p (\boldsymbol {\theta} \mid \mathcal {D}) \mathrm {d} \boldsymbol {\theta} \approx \log \sum_ {n} p (\mathcal {D} ^ {\prime} \mid \boldsymbol {\theta} _ {n}), \tag {14}
$$
where the  $\theta_{n}$  are samples from the parameter posterior. When only few predictions are available because sampling parameters  $\theta_{n}$  or evaluating the likelihood  $p(\mathcal{D}'\mid \boldsymbol{\theta}_n)$  is expensive, the LML may become very noisy. We therefore also report the per-sample log marginal likelihood
$$
\begin{array}{l} \operatorname {p s L M L} = \sum_ {\left(\boldsymbol {x} _ {i}, \boldsymbol {y} _ {i}\right) \in \mathcal {D} ^ {\prime}} \log p (\boldsymbol {y} _ {i} \mid \boldsymbol {x} _ {i}, \mathcal {D}) \\ = \sum_ {\left(\boldsymbol {x} _ {i}, \boldsymbol {y} _ {i}\right) \in \mathcal {D} ^ {\prime}} \log \int p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {x} _ {i}, \boldsymbol {\theta}\right) p (\boldsymbol {\theta} \mid \mathcal {D}) d \boldsymbol {\theta} \tag {15} \\ \approx \sum_ {\left(\boldsymbol {x} _ {i}, \boldsymbol {y} _ {i}\right) \in \mathcal {D} ^ {\prime}} \log \sum_ {n} p \left(\boldsymbol {y} _ {i} \mid \boldsymbol {x} _ {i}, \boldsymbol {\theta} _ {n}\right), \\ \end{array}
$$
which has a lower variance than the LML. We present the results for the LML, the psLML, the urban/rural Pearson coefficient (see Section 5.1), and the sQCE in Figure 10 and Figure 11. Table 9 shows detailed numerical results.
Model	Worst U/R Pearson	psLML	LML	MSE	QCE	sQCE
MAP	0.487 ± 0.074	-11.945 ± 2.042	-11.945 ± 2.042	0.267 ± 0.041	0.382 ± 0.012	-0.382 ± 0.012
Deep Ensemble	0.520 ± 0.075	-6.126 ± 1.422	-12.113 ± 2.074	0.249 ± 0.043	0.283 ± 0.027	-0.283 ± 0.027
MCD	0.491 ± 0.079	-6.868 ± 1.720	-12.175 ± 2.398	0.259 ± 0.049	0.316 ± 0.023	-0.316 ± 0.023
MultiMCD	0.516 ± 0.078	-5.053 ± 1.518	-12.290 ± 2.409	0.253 ± 0.052	0.271 ± 0.035	-0.271 ± 0.035
SWAG	0.473 ± 0.078	-12.551 ± 1.994	-12.621 ± 2.000	0.280 ± 0.040	0.386 ± 0.012	-0.386 ± 0.012
MultiSWAG	0.512 ± 0.078	-6.010 ± 1.373	-12.167 ± 1.989	0.250 ± 0.042	0.280 ± 0.026	-0.280 ± 0.026
LL Laplace	0.473 ± 0.072	-12.599 ± 2.191	-12.599 ± 2.191	0.280 ± 0.044	0.387 ± 0.012	-0.387 ± 0.012
LL MultiLaplace	0.516 ± 0.080	-5.614 ± 1.271	-12.324 ± 2.141	0.251 ± 0.044	0.265 ± 0.026	-0.265 ± 0.026
BBB	0.500 ± 0.072	-7.881 ± 2.290	-12.075 ± 2.470	0.264 ± 0.054	0.333 ± 0.036	-0.333 ± 0.036
MultiBBB	0.518 ± 0.074	-6.257 ± 1.462	-11.498 ± 2.299	0.252 ± 0.048	0.309 ± 0.027	-0.309 ± 0.027
Rank-1 VI	0.509 ± 0.069	-3.568 ± 1.053	-13.276 ± 1.978	0.246 ± 0.043	0.212 ± 0.028	-0.212 ± 0.028
iVON	0.249 ± -	-4.657 ± -	-19.787 ± -	0.347 ± -	0.236 ± -	-0.236 ± -
SVGD	0.497 ± 0.070	-5.416 ± 1.270	-12.524 ± 2.224	0.254 ± 0.041	0.255 ± 0.026	-0.255 ± 0.026
SNGP	0.456 ± 0.072	-12.688 ± 1.556	-12.688 ± 1.556	0.281 ± 0.031	0.357 ± 0.009	-0.357 ± 0.009
(a) O.o.d. Evaluation Split
Model	Worst U/R Pearson	psLML	LML	MSE	QCE	sQCE
MAP	0.673 ± 0.019	-7.445 ± 0.761	-7.445 ± 0.761	0.177 ± 0.015	0.348 ± 0.008	-0.348 ± 0.008
Deep Ensemble	0.703 ± 0.022	-3.438 ± 0.569	-7.093 ± 0.744	0.155 ± 0.014	0.228 ± 0.010	-0.228 ± 0.010
MCD	0.695 ± 0.010	-3.604 ± 0.483	-7.237 ± 0.673	0.162 ± 0.012	0.267 ± 0.014	-0.267 ± 0.014
MultiMCD	0.711 ± 0.024	-2.680 ± 0.358	-7.383 ± 0.615	0.156 ± 0.012	0.220 ± 0.014	-0.220 ± 0.014
SWAG	0.664 ± 0.021	-7.719 ± 0.716	-7.752 ± 0.723	0.183 ± 0.014	0.355 ± 0.003	-0.355 ± 0.003
MultiSWAG	0.705 ± 0.023	-3.331 ± 0.439	-7.221 ± 0.625	0.155 ± 0.012	0.223 ± 0.008	-0.223 ± 0.008
LL Laplace	0.664 ± 0.018	-7.823 ± 0.737	-7.823 ± 0.737	0.184 ± 0.015	0.357 ± 0.006	-0.357 ± 0.006
LL MultiLaplace	0.702 ± 0.023	-3.234 ± 0.539	-7.474 ± 0.828	0.157 ± 0.014	0.206 ± 0.004	-0.206 ± 0.004
BBB	0.680 ± 0.019	-4.508 ± 0.269	-7.224 ± 0.754	0.169 ± 0.014	0.286 ± 0.014	-0.286 ± 0.014
MultiBBB	0.694 ± 0.014	-3.619 ± 0.344	-7.159 ± 0.586	0.161 ± 0.011	0.256 ± 0.012	-0.256 ± 0.012
Rank-1 VI	0.669 ± 0.011	-2.077 ± 0.323	-9.630 ± 1.166	0.173 ± 0.016	0.162 ± 0.016	-0.162 ± 0.016
iVON	0.571 ± -	-3.888 ± -	-15.832 ± -	0.284 ± -	0.201 ± -	-0.201 ± -
SVGD	0.694 ± 0.026	-3.057 ± 0.611	-7.564 ± 1.017	0.159 ± 0.018	0.200 ± 0.008	-0.200 ± 0.008
SNGP	0.692 ± 0.013	-7.135 ± 0.639	-7.135 ± 0.639	0.170 ± 0.013	0.300 ± 0.008	-0.300 ± 0.008
(b) I.d. Evaluation Split
Table 9: POVERTYMAP-WILDS: Detailed results on the evaluation splits. iVON underperforms, with a Pearson coefficient of 0.249 on the o.o.d. split and a Pearson coefficient of 0.571 on the i.d. split. All models achieve the same LML and MSE within a  $95\%$  confidence interval.
Figure 10: POVERTYMAP-WILDS: Worst urban/rural Pearson coefficient vs. sQCE, LML, and psLML on the o.o.d. evaluation split. Ensemble-based models consistently outperform single-mode models (note that Rank-1 VI's components and SVGD's particles give them ensemble-like properties). The psLML is less noisy than the LML and results in a ranking of the algorithms that is more consistent with the sQCE and the Pearson coefficient. Laplace and SWAG perform nearly equivalently, therefore the data points of SWAG are hidden behind the data points of Laplace. iVON performs significantly worse than the other algorithms and is therefore excluded.
Figure 11: POVERTYMAP-WILDS: Worst urban/rural pearson coefficient vs. sQCE, LML and psLML on the i.d. evaluation split. The WILDS leaderboard [46] does not report the i.d. pearson coefficient.
G.3.3 IWILDCAM-WILDS
Following Koh et al. [47], we finetune a ResNet-50 [32], pretrained on ImageNet [14], for 12 epochs with the Adam optimizer [43]. For each model, we replace the linear classification layer of the ResNet-50 by a randomly initialized one of the appropriate output dimension. We use the hyperparameters that Koh et al. [47] found to work best based on their grid search: A learning rate of  $3 \cdot 10^{-5}$  and no weight decay. For MCD, we try dropout rates of 0.1 and 0.2 and select 0.1 due to a slightly better macro F1 score on the evaluation split. iVON uses a prior precision of 100, as optimized by a grid search. We use three seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the three evaluation runs. Figure 12 shows the results on the o.o.d. evaluation split that are not presented in the main paper. Table 10 displays detailed numerical results on the o.o.d. evaluation split and on the i.d. validation split.
Figure 12: iWILDCAM-WILDS: Macro F1 score, accuracy, sECE and ECE on the o.o.d. evaluation split and the i.d. validation split (see Figure 2a for Macro F1 vs. sECE on the o.o.d. evaluation split). MultiX is less accurate than single-mode approximations on the i.d. split, but better calibrated.
Model	Macro F1 Score	Accuracy	ECE	sECE	NLL
MAP	0.280 ± 0.020	0.708 ± 0.016	0.140 ± 0.015	-0.140 ± 0.015	1.514 ± 0.094
Deep Ensemble	0.312 ± 0.007	0.752 ± 0.007	0.019 ± 0.002	-0.015 ± 0.005	1.068 ± 0.016
MCD	0.274 ± 0.024	0.710 ± 0.021	0.138 ± 0.013	-0.138 ± 0.013	1.461 ± 0.074
MultiMCD	0.316 ± 0.012	0.763 ± 0.006	0.014 ± 0.004	-0.008 ± 0.007	1.026 ± 0.012
SWAG	0.302 ± 0.009	0.733 ± 0.005	0.117 ± 0.006	-0.117 ± 0.006	1.317 ± 0.032
MultiSWAG	0.337 ± 0.005	0.762 ± 0.001	0.033 ± 0.001	-0.033 ± 0.001	1.009 ± 0.001
LL SWAG	0.294 ± 0.033	0.721 ± 0.023	0.104 ± 0.019	-0.104 ± 0.020	1.295 ± 0.091
LL Laplace	0.270 ± 0.010	0.694 ± 0.015	0.053 ± 0.015	-0.052 ± 0.017	1.567 ± 0.083
LL MultiLaplace	0.304 ± 0.007	0.739 ± 0.004	0.046 ± 0.005	0.046 ± 0.005	1.197 ± 0.012
LL BBB	0.282 ± 0.011	0.718 ± 0.009	0.097 ± 0.006	-0.093 ± 0.005	1.543 ± 0.054
LL MultiBBB	0.312 ± 0.008	0.748 ± 0.002	0.015 ± 0.003	-0.012 ± 0.002	1.164 ± 0.011
Rank-1 VI	0.265 ± 0.009	0.750 ± 0.006	0.078 ± 0.018	-0.076 ± 0.017	1.198 ± 0.043
LL iVON	0.265 ± 0.009	0.725 ± 0.010	0.088 ± 0.015	-0.084 ± 0.014	1.331 ± 0.049
LL MultivON	0.299 ± 0.006	0.763 ± 0.003	0.019 ± 0.001	0.011 ± 0.003	1.036 ± 0.006
SVGD	0.260 ± 0.022	0.723 ± 0.008	0.146 ± 0.004	-0.146 ± 0.004	1.619 ± 0.017
LL SVGD	0.265 ± 0.018	0.737 ± 0.014	0.118 ± 0.003	-0.117 ± 0.003	1.447 ± 0.045
SNGP	0.275 ± 0.010	0.707 ± 0.013	0.160 ± 0.015	-0.160 ± 0.015	1.459 ± 0.095
(a) O.o.d. Test Split
Model	Macro F1 Score	Accuracy	ECE	sECE	NLL
MAP	0.460 ± 0.017	0.813 ± 0.007	0.104 ± 0.007	-0.104 ± 0.007	1.121 ± 0.087
Deep Ensemble	0.308 ± 0.005	0.752 ± 0.007	0.020 ± 0.001	-0.015 ± 0.005	1.067 ± 0.012
MCD	0.457 ± 0.010	0.814 ± 0.002	0.100 ± 0.011	-0.100 ± 0.011	1.105 ± 0.041
MultiMCD	0.311 ± 0.001	0.762 ± 0.006	0.013 ± 0.003	-0.008 ± 0.006	1.024 ± 0.016
SWAG	0.491 ± 0.011	0.832 ± 0.003	0.087 ± 0.002	-0.087 ± 0.002	0.987 ± 0.015
MultiSWAG	0.333 ± 0.011	0.761 ± 0.002	0.033 ± 0.002	-0.033 ± 0.002	1.008 ± 0.002
LL SWAG	0.465 ± 0.043	0.819 ± 0.016	0.088 ± 0.012	-0.088 ± 0.012	1.012 ± 0.060
LL Laplace	0.456 ± 0.017	0.810 ± 0.005	0.028 ± 0.009	-0.026 ± 0.010	1.045 ± 0.058
LL MultiLaplace	0.489 ± 0.012	0.836 ± 0.001	0.027 ± 0.003	0.027 ± 0.003	0.839 ± 0.012
LL BBB	0.442 ± 0.011	0.816 ± 0.005	0.075 ± 0.003	-0.075 ± 0.003	1.143 ± 0.030
LL MultiBBB	0.316 ± 0.006	0.749 ± 0.002	0.015 ± 0.003	-0.011 ± 0.003	1.165 ± 0.009
Rank-1 VI	0.442 ± 0.005	0.819 ± 0.001	0.082 ± 0.011	-0.082 ± 0.011	0.960 ± 0.052
LL iVON	0.447 ± 0.015	0.812 ± 0.005	0.076 ± 0.008	-0.076 ± 0.008	1.002 ± 0.028
LL MultiVON	0.294 ± 0.004	0.763 ± 0.003	0.019 ± 0.003	0.010 ± 0.003	1.035 ± 0.005
SVGD	0.439 ± 0.024	0.813 ± 0.009	0.106 ± 0.008	-0.105 ± 0.008	1.303 ± 0.136
LL SVGD	0.453 ± 0.018	0.822 ± 0.012	0.094 ± 0.010	-0.094 ± 0.009	1.135 ± 0.234
SNGP	0.459 ± 0.007	0.820 ± 0.004	0.106 ± 0.006	-0.106 ± 0.006	1.081 ± 0.048
(b) I.d. Validation Split
Table 10: iWILDCAM-WILDS: Detailed results on the evaluation splits. LL = Last-Layer. For the MultiX models, the entire model is ensembled.
G.3.4 FMOW-WILDS
Following Koh et al. [47], we finetune a DenseNet-121 [35], pretrained on ImageNet [14], for 50 epochs with the Adam optimizer [43] with a batch size of 64 and a learning rate of  $10^{-4}$  that decays by a factor of 0.96 per epoch. For each model, we replace the linear classification layer of the DenseNet-121 by a randomly initialized one of the appropriate output dimension. iVON uses a prior precision of 100. We use five seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the five evaluation runs.
We report in the main paper that the Laplace approximation underfits, with a worst-region accuracy of  $0.217 \pm 0.012$  and sECE of  $-0.583 \pm 0.015$  on the o.o.d. test split. Similarly, MultiLaplace only achieves a worst-region accuracy of  $0.301 \pm 0.004$  and sECE of  $0.123 \pm 0.004$  on the o.o.d. evaluation split. The accuracy doesn't change when using 100 posterior samples during evaluation, but increases to 0.243 for 1000 posterior samples. However, using so many samples incurs a significant computational overhead. Note that the better results of Daxberger et al. [13] are most likely due to their usage of models pretrained with ERM. Figure 13 shows additional results for the other models across all regions on the o.o.d. evaluation split, as well as the ECE on the worst region.
Figure 13: FMOW-wILDS: Accuracy, sECE and ECE on the o.o.d. evaluation split for the region with the lowest accuracy and across all regions (see Figure 2b for accuracy vs. sECE on the worst region). All models are underconfident when evaluated across all regions, but MultiX is less underconfident.