SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model	WR Accuracy	WR ECE	WR sECE	WR NLL	Avg Accuracy	Avg ECE	Avg sECE	Avg NLL
MAP	0.310 ± 0.008	0.526 ± 0.009	-0.526 ± 0.009	5.439 ± 0.117	0.518 ± 0.003	0.353 ± 0.002	-0.353 ± 0.002	3.503 ± 0.025
Deep Ensemble	0.342 ± 0.003	0.271 ± 0.004	-0.271 ± 0.004	3.446 ± 0.007	0.569 ± 0.001	0.128 ± 0.001	-0.128 ± 0.001	2.141 ± 0.006
MCD	0.307 ± 0.009	0.520 ± 0.011	-0.520 ± 0.011	5.400 ± 0.118	0.515 ± 0.002	0.349 ± 0.004	-0.349 ± 0.004	3.489 ± 0.036
MultiMCD	0.353 ± 0.005	0.253 ± 0.005	-0.253 ± 0.005	3.477 ± 0.030	0.571 ± 0.000	0.122 ± 0.001	-0.122 ± 0.001	2.150 ± 0.007
SWAG	0.308 ± 0.009	0.501 ± 0.007	-0.500 ± 0.007	4.913 ± 0.074	0.520 ± 0.003	0.327 ± 0.003	-0.327 ± 0.003	3.150 ± 0.035
MultiSWAG	0.338 ± 0.003	0.270 ± 0.003	-0.270 ± 0.003	3.243 ± 0.016	0.570 ± 0.001	0.124 ± 0.001	-0.124 ± 0.001	2.016 ± 0.008
LL SWAG	0.305 ± 0.005	0.516 ± 0.003	-0.516 ± 0.003	5.085 ± 0.036	0.516 ± 0.003	0.343 ± 0.003	-0.343 ± 0.003	3.271 ± 0.028
LL Laplace	0.212 ± 0.008	0.590 ± 0.009	-0.590 ± 0.009	8.249 ± 0.435	0.371 ± 0.014	0.449 ± 0.012	-0.449 ± 0.012	5.947 ± 0.320
LL Laplace (100 Samples)	0.213 ± 0.006	0.588 ± 0.008	-0.588 ± 0.008	8.246 ± 0.355	0.369 ± 0.012	0.449 ± 0.010	-0.449 ± 0.010	5.953 ± 0.262
LL MultiLaplace	0.301 ± 0.004	0.123 ± 0.004	-0.123 ± 0.004	4.086 ± 0.047	0.517 ± 0.002	0.059 ± 0.002	0.020 ± 0.002	2.744 ± 0.017
LL MultiLaplace (100 Samples)	0.301 ± 0.003	0.123 ± 0.003	-0.123 ± 0.003	4.088 ± 0.039	0.517 ± 0.002	0.059 ± 0.002	0.020 ± 0.002	2.748 ± 0.016
LL BBB	0.306 ± 0.008	0.448 ± 0.010	-0.448 ± 0.010	6.674 ± 0.343	0.509 ± 0.003	0.293 ± 0.003	-0.293 ± 0.003	4.251 ± 0.053
LL MultiBBB	0.339 ± 0.006	0.233 ± 0.008	-0.233 ± 0.008	4.174 ± 0.085	0.561 ± 0.001	0.102 ± 0.001	-0.102 ± 0.001	2.617 ± 0.008
Rank-1 VI	0.296 ± 0.007	0.497 ± 0.004	-0.497 ± 0.004	4.645 ± 0.147	0.512 ± 0.003	0.328 ± 0.003	-0.328 ± 0.003	2.995 ± 0.037
LL iVON	0.300 ± 0.009	0.514 ± 0.009	-0.514 ± 0.009	4.557 ± 0.112	0.505 ± 0.003	0.348 ± 0.002	-0.348 ± 0.002	3.107 ± 0.023
LL MultiiVON	0.341 ± 0.004	0.241 ± 0.004	-0.241 ± 0.004	3.177 ± 0.023	0.560 ± 0.001	0.112 ± 0.002	-0.112 ± 0.002	2.060 ± 0.009
SVGD	0.310 ± 0.007	0.526 ± 0.009	-0.526 ± 0.009	5.542 ± 0.083	0.517 ± 0.004	0.354 ± 0.003	-0.354 ± 0.003	3.559 ± 0.041
SNGP	0.234 ± 0.013	0.294 ± 0.033	-0.294 ± 0.033	3.419 ± 0.201	0.412 ± 0.012	0.179 ± 0.028	-0.179 ± 0.028	2.473 ± 0.126
Table 11: FMOW-WILDS: Detailed results on the o.o.d. evaluation split on the worst region as measured by the accuracy on each region and across all regions. For the MultiX models, the entire model is ensembled, but the single-mode approximation is only applied to the classification head. WR = Worst Region, LL = Last-Layer.
G.3.5 RXRx1-WILDS
Following Koh et al. [47], we finetune a ResNet-50 [32], pretrained on ImageNet [14], for 90 epochs with the Adam optimizer [43]. For each model, we replace the linear classification layer of the ResNet-50 by a randomly initialized one of the appropriate output dimension. Following Koh et al. [47], we use a learning rate of  $10^{-4}$  and weight decay  $10^{-5}$ . For MCD, we try dropout rates of 0.1 and 0.2 and select 0.1 due to a slightly better accuracy on the evaluation split. iVON uses a prior precision of 100 as optimized by a grid search. We use five seeds per model and build all ensembles by training six models independently and leaving out a different model for each of the five evaluation runs.
Similar to FMOw, Laplace underperforms accuracy-wise compared to the non-VI algorithms. While we do find a significant increase in accuracy to  $0.061 \pm 0.002$  when using 100 posterior samples, Laplace still performs worse than even MAP. However, Laplace is better calibrated with an sECE of  $-0.028 \pm 0.001$ .
Figure 14: RXRx1-wILDS: BBB and Rank-1 VI under different prior variances  $\sigma$  and posterior temperatures  $\lambda$ . We multiply  $\lambda$  to the KL divergence in the ELBO during training to reduce the regularization strength. However, neither small prior variances nor colder posteriors make BBB competitive with the non-VI algorithms.
Model	i.d. Accuracy	i.d. ECE	i.d. sECE	i.d. NLL	o.o.d. Accuracy	o.o.d. ECE	o.o.d. sECE	o.o.d. NLL
MAP	0.105 ± 0.002	0.232 ± 0.015	-0.232 ± 0.015	6.669 ± 0.121	0.083 ± 0.001	0.262 ± 0.015	-0.262 ± 0.015	7.197 ± 0.149
Deep Ensemble	0.156 ± 0.001	0.066 ± 0.001	-0.026 ± 0.002	5.211 ± 0.012	0.122 ± 0.000	0.071 ± 0.001	-0.061 ± 0.002	5.677 ± 0.019
MCD	0.106 ± 0.001	0.257 ± 0.003	-0.257 ± 0.003	6.924 ± 0.035	0.083 ± 0.001	0.288 ± 0.004	-0.288 ± 0.004	7.503 ± 0.043
MultiMCD	0.158 ± 0.001	0.069 ± 0.001	-0.035 ± 0.001	5.327 ± 0.003	0.121 ± 0.000	0.081 ± 0.001	-0.073 ± 0.000	5.836 ± 0.008
SWAG	0.110 ± 0.001	0.269 ± 0.009	-0.269 ± 0.009	6.947 ± 0.088	0.086 ± 0.001	0.301 ± 0.010	-0.301 ± 0.010	7.549 ± 0.109
MultiSWAG	0.161 ± 0.001	0.075 ± 0.002	-0.042 ± 0.001	5.299 ± 0.009	0.126 ± 0.001	0.085 ± 0.001	-0.078 ± 0.001	5.824 ± 0.017
LL Laplace	0.012 ± 0.001	0.097 ± 0.001	-0.097 ± 0.001	15.280 ± 0.546	0.010 ± 0.001	0.099 ± 0.001	-0.099 ± 0.001	15.890 ± 0.579
LL Laplace (100 Samples)	0.077 ± 0.000	0.034 ± 0.004	-0.011 ± 0.009	6.624 ± 0.206	0.061 ± 0.002	0.037 ± 0.007	-0.028 ± 0.007	6.909 ± 0.271
LL BBB (σ = 1.0, λ = 1.0)	0.046 ± 0.001	0.032 ± 0.002	-0.032 ± 0.002	6.657 ± 0.019	0.038 ± 0.000	0.045 ± 0.002	-0.045 ± 0.002	6.837 ± 0.012
LL BBB (σ = 0.5, λ = 1.0)	0.040 ± 0.001	0.007 ± 0.002	-0.006 ± 0.003	6.632 ± 0.017	0.035 ± 0.001	0.015 ± 0.003	-0.015 ± 0.003	6.737 ± 0.015
LL BBB (σ = 0.1, λ = 1.0)	0.036 ± 0.002	0.010 ± 0.001	0.003 ± 0.004	6.618 ± 0.017	0.031 ± 0.001	0.008 ± 0.002	-0.006 ± 0.004	6.681 ± 0.020
LL BBB (σ = 1.0, λ = 0.2)	0.054 ± 0.001	0.102 ± 0.002	-0.102 ± 0.002	7.598 ± 0.044	0.046 ± 0.001	0.117 ± 0.002	-0.117 ± 0.002	7.957 ± 0.050
Rank-1 VI	0.053 ± 0.001	0.068 ± 0.001	-0.068 ± 0.001	7.389 ± 0.023	0.043 ± 0.001	0.078 ± 0.000	-0.078 ± 0.000	7.577 ± 0.014
LL iVON	0.003 ± 0.000	0.008 ± 0.000	-0.008 ± 0.000	7.176 ± 0.012	0.003 ± 0.000	0.009 ± 0.001	-0.009 ± 0.001	7.213 ± 0.013
SVGD	0.102 ± 0.001	0.354 ± 0.005	-0.354 ± 0.005	8.254 ± 0.080	0.081 ± 0.002	0.382 ± 0.005	-0.382 ± 0.005	8.936 ± 0.088
SNGP	0.089 ± 0.006	0.245 ± 0.023	-0.245 ± 0.023	6.588 ± 0.272	0.067 ± 0.005	0.273 ± 0.019	-0.273 ± 0.019	7.070 ± 0.221
Table 12: RXRX1-wILDS: Detailed results on the i.d. and the o.o.d. evaluation split. LL = Last-Layer. For the MultiX models, the entire model is ensembled, but the single-mode approximation is only applied to the classification head. We evaluate multiple hyperparameter combinations for LL BBB, as the standard parameters do not perform well. The failure of VI is equally present with LL BBB, LL Rank-1 VI, and LL ivON.
G.3.6 CIVIL COMMENTS-WILDS
We use the pretrained DistilBERT [76] model from HuggingFace transformers [91] with a classification head consisting of two linear layers with a ReLU nonlinearity and a Dropout unit with a drop rate of 0.2 between them. Following Koh et al. [47], we finetune the pretrained checkpoint with a learning rate of  $1 \cdot 10^{-5}$  and, where applicable, a weight decay factor of  $1 \cdot 10^{-2}$  for three epochs using the Adam optimizer [43]. SWAG collects ten parameter samples during the last two epochs of training. iVON uses a prior precision of 500, as optimized by a grid search. We use five seeds for all non-ensembled models. The ensembles are built from four of the five single-model versions, leaving out a different member per model to create five different ensemble models of four members each.
We note in the main paper that MCD results in less accurate and more overconfident models. We investigate this further by experimenting with different dropout rates in Figure 15. While a dropout rate of 0.1 had no impact, dropout rates of 0.05 and 0.01 lead to progressively better accuracy and calibration, coming close to MAP. However, there is still no accuracy or calibration benefit to be gained from using MCD.
G.3.7 AMAZON-WILDS
We use the pretrained DistilBERT [76] model from HuggingFace transformers [91] with a classification head consisting of two linear layers with a ReLU nonlinearity and a Dropout unit with a drop rate of 0.2 between them. Following Koh et al. [47], we finetune the pretrained checkpoint with a learning rate of  $10^{-5}$  and, where applicable, a weight decay factor of  $10^{-2}$  using the Adam optimizer [43]. Contrary to Koh et al. [47], we finetune for five epochs, as we find that the validation accuracy
Figure 15: CIVIL COMMENTS-WILDS: Accuracy and sECE for different MCD dropout rates  $p$ . While smaller dropout rates improve the accuracy, the models are still less accurate and more overconfident than MAP.
Figure 16: CIVIL COMMENTS-WILDS: Detailed results on the o.o.d. evaluation split for the worst group (WG, determined by the accuracy on each group) and averaged over all groups. LL = Last-Layer.
Model	WG Accuracy	WG ECE	WG sECE	WG NLL	Avg Accuracy	Avg ECE	Avg sECE	Avg NLL
MAP	0.420 ± 0.021	0.353 ± 0.025	-0.353 ± 0.025	1.455 ± 0.086	0.916 ± 0.001	0.012 ± 0.003	-0.012 ± 0.003	0.207 ± 0.001
Deep Ensemble	0.419 ± 0.008	0.349 ± 0.010	-0.349 ± 0.010	1.416 ± 0.032	0.916 ± 0.000	0.010 ± 0.001	-0.010 ± 0.001	0.204 ± 0.000
MCD (p=0.2)	0.326 ± 0.023	0.417 ± 0.030	-0.417 ± 0.030	1.391 ± 0.074	0.918 ± 0.000	0.007 ± 0.005	0.006 ± 0.006	0.204 ± 0.001
MCD (p=0.1)	0.325 ± 0.021	0.418 ± 0.027	-0.418 ± 0.027	1.390 ± 0.070	0.918 ± 0.000	0.007 ± 0.005	0.006 ± 0.005	0.204 ± 0.001
MCD (p=0.05)	0.364 ± 0.018	0.393 ± 0.024	-0.393 ± 0.024	1.430 ± 0.065	0.918 ± 0.000	0.005 ± 0.002	-0.003 ± 0.004	0.203 ± 0.001
MCD (p=0.01)	0.396 ± 0.015	0.374 ± 0.023	-0.374 ± 0.023	1.452 ± 0.114	0.917 ± 0.000	0.011 ± 0.003	-0.011 ± 0.003	0.206 ± 0.001
MultiMCD (p=0.2)	0.326 ± 0.005	0.412 ± 0.007	-0.412 ± 0.007	1.363 ± 0.022	0.919 ± 0.000	0.009 ± 0.002	0.009 ± 0.002	0.203 ± 0.000
SWAG	0.448 ± 0.021	0.197 ± 0.041	-0.184 ± 0.027	0.872 ± 0.050	0.877 ± 0.024	0.152 ± 0.024	0.152 ± 0.024	0.396 ± 0.019
MultiSWAG	0.429 ± 0.016	0.183 ± 0.018	-0.183 ± 0.018	0.819 ± 0.011	0.901 ± 0.002	0.184 ± 0.002	0.184 ± 0.002	0.388 ± 0.004
LL Laplace	0.424 ± 0.016	0.348 ± 0.018	-0.347 ± 0.018	1.438 ± 0.065	0.916 ± 0.001	0.011 ± 0.002	-0.011 ± 0.002	0.207 ± 0.001
LL MultiLaplace	0.420 ± 0.008	0.348 ± 0.010	-0.348 ± 0.010	1.411 ± 0.032	0.916 ± 0.000	0.010 ± 0.001	-0.009 ± 0.001	0.204 ± 0.000
LL BBB	0.537 ± 0.032	0.362 ± 0.032	-0.361 ± 0.033	2.192 ± 0.278	0.918 ± 0.002	0.056 ± 0.002	-0.056 ± 0.002	0.333 ± 0.017
LL MultiBBB	0.525 ± 0.012	0.338 ± 0.012	-0.338 ± 0.012	1.801 ± 0.078	0.922 ± 0.000	0.041 ± 0.001	-0.041 ± 0.001	0.265 ± 0.003
Rank-1 VI	0.540 ± 0.028	0.373 ± 0.030	-0.373 ± 0.030	2.065 ± 0.179	0.917 ± 0.002	0.060 ± 0.002	-0.060 ± 0.002	0.319 ± 0.007
LL iVON	0.480 ± 0.045	0.421 ± 0.048	-0.421 ± 0.048	2.198 ± 0.263	0.919 ± 0.003	0.054 ± 0.002	-0.054 ± 0.002	0.299 ± 0.014
LL MultiVON	0.465 ± 0.011	0.396 ± 0.015	-0.396 ± 0.015	1.752 ± 0.073	0.924 ± 0.001	0.039 ± 0.001	-0.039 ± 0.001	0.240 ± 0.002
SVGD	0.384 ± 0.068	0.380 ± 0.079	-0.379 ± 0.079	1.393 ± 0.154	0.915 ± 0.003	0.011 ± 0.005	-0.008 ± 0.008	0.208 ± 0.002
SNGP	0.394 ± 0.039	0.388 ± 0.036	-0.388 ± 0.036	1.341 ± 0.078	0.919 ± 0.001	0.014 ± 0.006	-0.014 ± 0.006	0.206 ± 0.004
is still increasing after three epochs. SWAG collects 30 parameter samples during the last two epochs of training. We also experiment with last-layer versions of SWAG and MCD, but find both to perform very similar to MAP (see Table 13). iVON uses a prior precision of 500, as optimized by a grid search. We use six seeds for all non-ensembed models. The ensembles are build from five of the six single-model versions, leaving out a different member per model to create five different ensembled models of five members each.
Model	o.o.d. 10 Accuracy	o.o.d. Accuracy	o.o.d. ECE	o.o.d. sECE	o.o.d. NLL	i.d. 10 Accuracy	i.d. Avg Accuracy	i.d. ECE	i.d. sECE	i.d. NLL
MAP	0.453 ± 0.010	0.655 ± 0.003	0.067 ± 0.006	-0.067 ± 0.006	0.815 ± 0.007	0.477 ± 0.008	0.678 ± 0.002	0.049 ± 0.007	-0.049 ± 0.007	0.755 ± 0.005
Deep Ensemble	0.453 ± 0.000	0.659 ± 0.001	0.058 ± 0.002	-0.058 ± 0.002	0.800 ± 0.001	0.480 ± 0.000	0.682 ± 0.000	0.040 ± 0.002	-0.040 ± 0.002	0.742 ± 0.001
MCD	0.447 ± 0.013	0.657 ± 0.002	0.020 ± 0.011	-0.019 ± 0.012	0.789 ± 0.004	0.472 ± 0.012	0.678 ± 0.001	0.015 ± 0.006	-0.002 ± 0.013	0.741 ± 0.003
MultiMCD	0.451 ± 0.005	0.660 ± 0.001	0.012 ± 0.003	-0.012 ± 0.003	0.780 ± 0.001	0.475 ± 0.007	0.682 ± 0.000	0.007 ± 0.002	0.005 ± 0.003	0.733 ± 0.000
LL MCD	0.451 ± 0.008	0.656 ± 0.003	0.069 ± 0.008	-0.069 ± 0.008	0.816 ± 0.011	0.478 ± 0.011	0.679 ± 0.002	0.051 ± 0.009	-0.051 ± 0.009	0.756 ± 0.009
SWAG	0.436 ± 0.011	0.639 ± 0.006	0.032 ± 0.003	0.031 ± 0.004	0.840 ± 0.014	0.460 ± 0.010	0.658 ± 0.006	0.047 ± 0.004	0.047 ± 0.004	0.807 ± 0.015
MultiSWAG	0.443 ± 0.005	0.646 ± 0.001	0.040 ± 0.001	0.040 ± 0.001	0.828 ± 0.002	0.469 ± 0.005	0.667 ± 0.001	0.057 ± 0.001	0.057 ± 0.001	0.796 ± 0.003
LL SWAG	0.452 ± 0.012	0.656 ± 0.003	0.048 ± 0.009	-0.048 ± 0.009	0.802 ± 0.006	0.474 ± 0.010	0.679 ± 0.002	0.031 ± 0.008	-0.030 ± 0.009	0.747 ± 0.005
LL Laplace	0.455 ± 0.009	0.654 ± 0.003	0.067 ± 0.006	-0.067 ± 0.006	0.816 ± 0.006	0.482 ± 0.009	0.678 ± 0.002	0.048 ± 0.007	-0.048 ± 0.007	0.756 ± 0.004
LL MultiLaplace	0.453 ± 0.000	0.659 ± 0.001	0.058 ± 0.001	-0.058 ± 0.001	0.800 ± 0.001	0.480 ± 0.000	0.682 ± 0.000	0.040 ± 0.002	-0.040 ± 0.002	0.742 ± 0.001
LL BBB	0.527 ± 0.006	0.695 ± 0.007	0.154 ± 0.005	-0.154 ± 0.005	0.898 ± 0.019	0.560 ± 0.000	0.730 ± 0.006	0.128 ± 0.004	-0.128 ± 0.004	0.778 ± 0.015
LL MultiBBB	0.533 ± 0.000	0.709 ± 0.002	0.105 ± 0.001	-0.105 ± 0.001	0.748 ± 0.001	0.573 ± 0.000	0.746 ± 0.001	0.079 ± 0.002	-0.079 ± 0.002	0.648 ± 0.002
Rank-1 VI	0.527 ± 0.005	0.695 ± 0.003	0.173 ± 0.004	-0.173 ± 0.004	0.923 ± 0.015	0.558 ± 0.004	0.729 ± 0.003	0.147 ± 0.004	-0.147 ± 0.004	0.801 ± 0.010
LL iVON	0.458 ± 0.010	0.661 ± 0.002	0.053 ± 0.010	-0.053 ± 0.010	0.794 ± 0.008	0.484 ± 0.009	0.684 ± 0.002	0.037 ± 0.010	-0.037 ± 0.010	0.737 ± 0.006
LL MultiVON	0.459 ± 0.007	0.665 ± 0.001	0.045 ± 0.002	-0.045 ± 0.002	0.779 ± 0.001	0.484 ± 0.005	0.687 ± 0.001	0.029 ± 0.003	-0.029 ± 0.002	0.724 ± 0.001
SVGD	0.456 ± 0.010	0.661 ± 0.003	0.049 ± 0.005	-0.049 ± 0.005	0.793 ± 0.011	0.477 ± 0.010	0.682 ± 0.002	0.035 ± 0.005	-0.034 ± 0.005	0.740 ± 0.008
SNGP	0.451 ± 0.013	0.661 ± 0.001	0.053 ± 0.008	-0.053 ± 0.008	0.800 ± 0.002	0.487 ± 0.013	0.685 ± 0.001	0.036 ± 0.008	-0.035 ± 0.008	0.737 ± 0.001
Table 13: AMAZON-WILDS: Detailed results on the i.d. and the o.o.d. evaluation splits. LL = Last-Layer.