SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Monte Carlo Dropout (MCD). MCD [22] uses dropout [83] to form a Bernoulli distribution over network parameters. The dropout rates are typically not learned, but the dropout units that are present in many network architectures are simply applied during the evaluation of the model. This very cheap posterior approximation has been criticized for not being truly Bayesian [71]. Despite this criticism, it is still widely used, including in practical applications [10]. When the dropout rate is learned, MCD can be considered to implicitly perform VI [21].
Stochastic Weight Averaging-Gaussian (SWAG). SWAG [59] forms its posterior approximations from the parameter vectors that are traversed during the training of a standard neural network. During the last epochs of SGD training, SWAG periodically stores the current parameters of the neural network to build a low-rank Gaussian distribution over model parameters. While SWAG has only a very small performance overhead during training, storing the additional parameters requires a significant amount of additional memory, and sampling parameters from the low-rank Gaussian distribution incurs a performance overhead during evaluation.
Laplace Approximation. The Laplace approximation [57] builds a local posterior approximation from a second-order Taylor expansion around a MAP model. We always use the last-layer Laplace approximation and switch between a full-rank posterior, diagonal posterior, and a Kronecker-factorized posterior [74] depending on the task. In this configuration, the Laplace approximation is the only post-hoc algorithm that we consider: It can be fitted on top of an existing MAP model by performing a single pass on the training dataset.
B Unsigned Calibration Metrics
As mentioned in the main paper (Section 4), a calibrated model makes confident predictions if and only if they will likely be accurate. Based on this definition, we can directly derive a calibration metric for classification models: The expected calibration error (ECE) [28, 66]. In the regression case, neither "accuracy" nor "confidence" are well-defined properties of a prediction. The notion of calibration must therefore be adapted for regression tasks. In addition, the log marginal likelihood is commonly used to jointly evaluate the accuracy and the calibration in regression tasks. See Appendix G.3.2 for details.
Calibrated Classification. In the classification case, each data point has an associated distribution  $Y$  over the possible labels.  $Y$  represents the inherent aleatoric uncertainty of the label. Given a prediction  $\hat{y} = \arg \max_y p(y\mid x,\mathcal{D})$  made with confidence  $\hat{p} = \max_y p(y\mid x,\mathcal{D})$ , the model is perfectly calibrated if and only if
$$
\mathbb {P} (\hat {y} = Y \mid \hat {p} = p) = p \quad \forall p \in [ 0, 1 ] \tag {5}
$$
holds for every data point [12, 28, 66]. Informally speaking, this means that if the model makes 100 predictions with a confidence of 0.8, 80 of these predictions should be correct. The expected difference between the left and the right side of Equation (5) is called the expected calibration error (ECE) of the model:
$$
\operatorname {E C E} = \underset {p \sim \mathcal {U} ([ 0, 1 ])} {\mathbb {E}} [ | \mathbb {P} (\hat {y} = Y \mid \hat {p} = p) - p | ] \tag {6}
$$
(a) Overconfident Quantile
(b) Overconfident CI
Figure 6: Reliability plots of fictional, overconfident regression models when using (a) quantiles and (b) confidence intervals (CI).
It implies two properties of a well-calibrated model: If the accuracy is low, the confidence should also be low. This means that the model must not be overconfident in its predictions. Conversely, if the accuracy is high, the confidence should also be high, meaning that the model must not be underconfident in its predictions.
In practice, a model does not make enough predictions of the same confidence to calculate the calibration error exactly. Therefore, the model's predictions on an evaluation set  $\mathcal{D}'$  are commonly grouped into  $M$  equally spaced bins  $B_{m}$  based on their confidence values, and the average accuracy and confidence of each bin are used to calculate the ECE [28, 66]:
$$
\mathrm {E C E} \approx \sum_ {m = 1} ^ {M} \frac {\left| B _ {m} \right|}{\left| \mathcal {D} ^ {\prime} \right|} \left| \operatorname {a c c} \left(B _ {m}\right) - \operatorname {c o n f} \left(B _ {m}\right) \right|, \tag {7}
$$
where  $B_{m}$  is the set of predictions in the  $m$ -th bin, and  $\mathrm{acc}(B_m)$  and  $\mathrm{conf}(B_m)$  are the average accuracy and confidence of the predictions in  $B_{m}$ :
$$
\operatorname {a c c} \left(B _ {m}\right) = \frac {1}{\left| B _ {m} \right|} \sum_ {\left(\boldsymbol {x}, y\right) \in B _ {m}} \mathbf {1} \left(y = \underset {y ^ {\prime}} {\arg \max } p \left(y ^ {\prime} \mid \boldsymbol {x}, \mathcal {D}\right)\right) \tag {8}
$$
$$
\operatorname {c o n f} \left(B _ {m}\right) = \frac {1}{\left| B _ {m} \right|} \sum_ {\left(\boldsymbol {x}, y\right) \in B _ {m}} \max  _ {y ^ {\prime}} p \left(y ^ {\prime} \mid \boldsymbol {x}, \mathcal {D}\right) \tag {10}
$$
An ECE of zero indicates perfect calibration. We always use ten bins  $(M = 10)$ .
A main problem of the ECE is that bins with few predictions in them may exhibit a high variance [69]. Therefore, Nixon et al. [69] proposed an extension of the ECE that uses bins of adaptive width.
Calibrated Regression. The confidence intervals of the predictive distribution can be used to measure the calibration of a regression model [49]. The probability of the ground-truth output  $\pmb{y}$  laying inside of the  $\rho$ -confidence interval of the predictive distribution of the model for input  $\pmb{x}$  should be exactly  $\rho$ . Formally, we say a regression model is perfectly calibrated on an evaluation dataset  $\mathcal{D}'$  if and only if
$$
\mathbb {P} \left(Q _ {\rho^ {\prime}} (\boldsymbol {x}) \leq \boldsymbol {y} \leq Q _ {1 - \rho^ {\prime}} (\boldsymbol {x})\right) = \rho \quad \forall (\boldsymbol {x}, \boldsymbol {y}) \in \mathcal {D} ^ {\prime} \tag {11}
$$
holds for every  $q$ -quantile  $Q_{q}(\pmb{x})$  of the predictive distribution for input  $\pmb{x}$  with  $\rho' = (1 - \rho) / 2$ .
Selectively evaluating Equation (11) for  $M$  confidence values  $\rho_{m}$  allows the practical computation of a quantile calibration error (QCE) on an evaluation dataset  $\mathcal{D}'$
$$
\mathrm {Q C E} = \frac {1}{M} \sum_ {m = 1} ^ {M} | \left(\rho_ {m} - p _ {\mathrm {o b s}} \left(\rho_ {m}\right)\right) | \tag {12}
$$
$$
p _ {\mathrm {o b s}} (\rho_ {m}) = \frac {1}{| \mathcal {D} ^ {\prime} |} \sum_ {(\boldsymbol {x}, \boldsymbol {y}) \in \mathcal {D})} \mathbf {1} \left(Q _ {\rho^ {\prime}} (\boldsymbol {x}) \leq \boldsymbol {y} \leq Q _ {1 - \rho^ {\prime}} (\boldsymbol {x})\right). \tag {13}
$$
The QCE simply replaces the quantiles in the definition of the calibration error from Kuleshov et al. [49] by confidence intervals. Using the confidence intervals allows a simpler interpretation of the resulting reliability diagrams: With the calibration error proposed by Kuleshov et al. [49], the reliability diagram of a perfectly calibrated regression model is a horizontally mirrored version of the reliability diagram of a perfectly calibrated classification model, as there are too many ground-truth values below the lower quantiles of their predictive distributions, and too few above the higher quantiles (Figure 6a). Using confidence intervals for the reliability diagram results in a plot that can be interpreted in the same way as a reliability diagram of a classification model (Figure 6b). We always use ten equally-spaced confident levels between 0 and 1 ( $M = 10$ ).
C Signed Calibration Metrics
As described in the main paper, our signed calibration metrics (sECE and sQCE) may be zero even though the model is not perfectly calibrated. However, we show that this is typically not an issue in practice, as for most models nearly all predictions are overconfident or nearly all predictions are underconfident. The reliability diagrams in Figure 7 confirm this for a representative selection of overconfident and underconfident models. We always report the unsigned calibration metrics in Appendix G in addition to the signed calibration metrics mentioned in the main paper. The unsigned metrics are in almost all cases very close to the absolute value of the signed metric, resulting in the same relative ordering of the algorithms. On the other hand, the sECE provides valuable insights into the underconfidence of some algorithms such as MultiSWAG on CIFAR-10 and SWAG on Amazon-WILDS.
(a) MAP on the o.o.d. evaluation split of IWILDCAM-WILDS. sECE: -0.0457, ECE: 0.0463
(b) MultiLaplace on the o.o.d. evaluation split of iWILDCAM-WILDS. sECE: -0.04501, ECE: 0.0501
(c) MAP on the group with the worst accuracy on CIVILCOMMENTS-WILDS. sECE: -0.3162, ECE: 0.3162
(d) SWAG on the o.o.d. evaluation split of AMAZON-WILDS. sECE: -0.0405, ECE: 0.0408
Figure 7: Reliability diagrams of different models on a variety of datasets. No data point is drawn for empty bins. The number of predictions in each bin is denoted at the top of each plot. The dashed line corresponds to a perfectly calibrated model. In all cases, either nearly all of the model's predictions are overconfident, or nearly all are underconfident. Therefore, the ECE is close to the absolute value of the sECE, indicating that the sECE is a reasonable calibration metric.
Except for Laplace, we implement all algorithms ourselves as PyTorch [24] optimizers. The implementation of the algorithms as well as code to reproduce all experiments is available at https://github.com/bdl-authors/beyond-ensembles, where we also provide a short tutorial on the usage of our implementation.
Bayes By Backprop. We use the local reparameterization trick [45]. As it is standard today [20, 72, 89], we do not use the scale mixture prior introduced by BBB's original authors [7], but a unit Gaussian prior. For the experiments on CIFAR-10, we make the parameters of the Filter Response Normalization layers variational.
Rank-1 VI. Following Dusenberry et al. [17], we keep the bias of each layer as a point estimate. We also keep the learned parameters of batch normalization and Filter Response Normalization layers as point estimates. We use five components in most cases which is close to the four components recommended by Dusenberry et al. [17] and make Rank-1 VI directly comparable to other ensemble-based models that use five members.
iVON. We adapt the data augmentation factor that Osaka et al. [70] introduce for VOGN [40] to iVON. We do not use the tempering parameter from VOGN.
Laplace. We use the Laplace library from Daxberger et al. [13] due to the difficulty of implementing second-order optimization in PyTorch. In all cases except for CIVIL COMMENTS-WILDS, we use a Kronecker-factorized last-layer Laplace approximation. On CIVIL COMMENTS-WILDS, we use a diagonal last-layer Laplace approximation as the Kronecker-factorized approximation frequently leads to diverging parameters. We do not use the GLM approximation as proposed by Daxberger et al. [13] but use Monte Carlo sampling to stay consistent with the other evaluated algorithms. In all experiments we use the Laplace library's functions to tune the prior precision after fitting the Laplace approximation.
SWAG. While the authors of SWAG argue that SWAG benefits from a special learning rate schedule [59], they do not use such a schedule in most of their experiments with SWAG and MultiSWAG [89]. Correspondingly, we use the same schedule with SWAG as with any other algorithm. We use 30 parameter samples for building the mean and the low-rank covariance matrix of SWAG. On CIVIL COMMENTS-WILDS, we only use 10 parameter samples due to the storage size of the samples.
E Batch Normalization, Distribution Shift, and Bayesian Deep Learning
Figure 8: CAMELYON17-WILDS: Average accuracy vs. sECE on the o.o.d. test split. The models that use no running statistics (static BN) are significantly more accurate and better calibrated, while exhibiting a smaller variance.
Schneider et al. [77] find that a significant part of the accuracy loss on o.o.d. data is due to changing batch statistics that cannot be adequately normalized by the running batch normalization statistics that are based on the training data. The authors propose to re-initialize the running statistics on a subset of the evaluation dataset.
We are able to reproduce the issue with o.o.d. data on the CAMELYON17-WILDS dataset from the WILDS collection [47] (Figure 8). The o.o.d. evaluation set of CAMELYON17 has been generated by selecting the images that were most visually distinct from the other images. In addition, the employed ResNet-20 [32] architecture includes batch normalization layers. We find that using only batch statistics, thereby essentially using the batch normalization layers in training mode during evaluation, entirely alleviates the i.d. - o.o.d. performance gap on CAMELYON17, as well as the large standard deviations on the o.o.d. dataset. Coincidentally, the WILDS leaderboard [46] shows that models that do not include batch normalization, such as a model based on the vision transformer [15], or that use extensive data augmentation, perform best.
The running statistics of batch normalization layers also pose problems with Bayesian neural networks that sample parameters, as the running statistics depend on the parameters of the neural network. Wilson and Izmailov [89] therefore propose to recalculate the batch normalization statistics for each parameter sample. This is not necessary in our case as we never use running statistics for normalization layers. By doing so we also avoid the aforementioned distribution-shift problem without requiring additional o.o.d. data during evaluation, and do not add any computation overhead.
F Computational Resources
We use single NVIDIA Tesla V100, A100, and H100 GPUs for all tasks from Wilds [47] and CIFAR-10-(C) [33, 48]. See Table 1 for the GPUs that we use on the individual datasets as well as the runtime of MAP. Table 2 displays the relative runtime of the BDL algorithms. In total, we estimate that the evaluation required about  $1600\mathrm{h}$  of GPU time, of which about  $25\%$  were consumed during implementation, testing and hyperparameter optimization. Training and hyperparameter optimization of the UCI models was performed on a single CPU in about  $20\mathrm{h}$ . Table 3 shows the GPU memory overhead of the BDL algorithms.
Dataset	GPU	Runtime of MAP
CIFAR-10	NVIDIA V100	50 min
POVERTYMAP-WILDS	NVIDIA V100	50 min
IWILDCAM-WILDS	NVIDIA A100	150 min
FMOW-WILDS	NVIDIA V100	150 min
RXRX1-WILDS	NVIDIA V100	140 min
CIVILCOMMENTS-WILDS	NVIDIA A100	60 min
AMAZON-WILDS	NVIDIA H100	90 min
Table 1: Hardware and runtime for MAP for each dataset. The results are rounded to the next 10 minutes.
Model	POVERTYMAP	IWILDCAM	FMOW	RXRx1	CIVILCOMMENTS	AMAZON	CIFAR-10
MAP	1.0	1.0	1.0	1.0	1.0	1.0	1.0
MCD	1.0	1.0	1.0	1.0	1.0	1.0	~ 1.0
SWAG	1.3	1.5	1.0	1.2	1.3	1.5	~ 1.0
Laplace	1.0	1.0	1.0	1.0	1.0	1.0	~ 1.0
BBB	5.7	-	-	-	-	-	~ 5.0
LL BBB	-	1.6	2.0	3.7	1.8	2.0	-
Rank-1 VI	3.9	-	-	-	-	-	~ 4.0
LL Rank-1 VI	-	2.0	2.0	3.7	1.9	2.0	-
iVON	2.8	-	-	-	-	-	~ 3.0
LL iVON	-	3.0	2.9	3.6	5.6	6.6	-
SVGD	9.2	4.9	7.3	8.9	9.2	10.0	~ 8.0
Table 2: Runtime of different algorithms relative to MAP. The numbers on CIFAR-10 are conservative estimates as exact numbers were no longer available. Note that the runtime also depends on whether we are able to use mixed precision training, which was not possible with the VI algorithms. The training time of a MultiX model with  $n$  members is  $n$  times the training time of the respective single-mode approximation.  $\mathrm{LL} =$  Last-Layer.
Model	Memory Overhead
MAP	1.0
MCD	1.0
SWAG	~ 1.0
Laplace	~ 1.0
BBB	~ 2
Rank-1 VI	~ 1 + #components · √parameter count
iVON	~ 25
SVGD	~ #particles
Table 3: GPU memory requirements of different algorithms relative to MAP. The numbers are estimates are based on a theoretical analysis of the algorithms, not on measurements. The memory consumption of MultiX is the same as for the respective single-mode approximation, since all members can be trained independently.
${}^{5}$  No additional memory overhead due to a separate optimizer
G Additional Experimental Results
G.1 UCI Datasets
We report results for both the standard and the gap splits [20] on the HOUSING and ENERGY datasets from the UCI machine learning repository [16]. On ENERGY, we can reproduce the catastrophic failure of VI both with BBB and Rank-1 VI, but not with iVON which performs still similarly to MultiSWAG. Overall, we find that the benefit of ensembles is less clear than on the larger WILDS datasets, which emphasizes the importance of evaluating Bayesian algorithms on large datasets.
Hyperparameters. All hyperparameters were optimized through a grid search on the validation set. Note that for the gap splits the validation set is not part of the gap. We considered 40, 100 and 200 epochs, learning rates of 0.01 and 0.001 and (where applicable) weight decay factors of  $10^{-4}$  and  $10^{-5}$ . For BBB, the prior standard deviations are 0.1, 1.0 and 10.0 and we scale the KL divergence in the ELBO by 0.2, 0.5, and 1.0, with colder temperatures generally leading to better results. For iVON, we consider prior precisions of 10, 100, and 200, with 200 being selected in most cases. BBB and iVON use five Monte Carlo samples during training. For SWAG, we consider 60, 100, and 150 epochs, use 30 parameter samples and start sampling after  $50\%$ ,  $75\%$ , or  $90\%$  of the training epochs were completed. For Laplace, we always use a last-layer approximation with a full covariance matrix. We use the Adam optimizer [43] to optimize the log-likelihood/ELBO and learn the output standard deviation jointly with the parameters. We use 1000 parameter samples for each prediction.
Model	LML	MSE	QCE	sQCE
MAP	-2.643 ± 0.054	10.707 ± 0.794	0.036 ± 0.001	-0.018 ± 0.012
Deep Ensemble	-2.330 ± 0.015	6.034 ± 0.413	0.099 ± 0.006	0.099 ± 0.006
MCD	-2.836 ± 0.095	13.531 ± 1.401	0.044 ± 0.008	-0.031 ± 0.019
MultiMCD	-2.328 ± 0.010	5.789 ± 0.077	0.101 ± 0.002	0.101 ± 0.002
SWAG	-2.495 ± 0.014	6.044 ± 0.214	0.128 ± 0.003	0.128 ± 0.003
MultiSWAG	-2.511 ± 0.002	7.071 ± 0.011	0.139 ± 0.002	0.139 ± 0.002
LL Laplace	-2.515 ± 0.058	9.498 ± 2.347	0.099 ± 0.020	0.099 ± 0.020
LL MultiLaplace	-2.467 ± 0.031	5.850 ± 0.395	0.157 ± 0.005	0.157 ± 0.005
BBB	-2.475 ± 0.096	7.716 ± 1.053	0.033 ± 0.004	-0.006 ± 0.012
MultiBBB	-2.529 ± 0.003	7.212 ± 0.160	0.172 ± 0.005	0.172 ± 0.005
Rank-1 VI	-2.531 ± 0.103	8.983 ± 1.451	0.033 ± 0.008	0.006 ± 0.016
iVON	-2.793 ± 0.006	9.853 ± 0.318	0.215 ± 0.003	0.215 ± 0.003
SVGD	-2.614 ± 0.017	8.397 ± 0.642	0.143 ± 0.010	0.143 ± 0.010
Table 4: UCI-HOUSING (standard splits)
Model	LML	MSE	QCE	sQCE
MAP	-2.850 ± 0.207	14.775 ± 3.635	0.040 ± 0.013	-0.012 ± 0.024
Deep Ensemble	-2.767 ± 0.183	13.012 ± 3.034	0.054 ± 0.011	0.045 ± 0.017
MCD	-2.892 ± 0.210	15.488 ± 3.661	0.034 ± 0.007	-0.001 ± 0.018
MultiMCD	-2.730 ± 0.132	12.760 ± 2.838	0.054 ± 0.013	0.045 ± 0.019
SWAG	-2.743 ± 0.042	12.940 ± 1.742	0.113 ± 0.017	0.112 ± 0.017
MultiSWAG	-2.694 ± 0.047	11.941 ± 1.874	0.118 ± 0.020	0.117 ± 0.020
LL Laplace	-2.873 ± 0.117	15.294 ± 3.687	0.074 ± 0.018	0.072 ± 0.020
LL MultiLaplace	-2.832 ± 0.106	13.445 ± 3.209	0.106 ± 0.020	0.104 ± 0.022
BBB	-3.829 ± 1.009	17.238 ± 7.435	0.114 ± 0.024	-0.113 ± 0.024
MultiBBB	-2.734 ± 0.115	14.071 ± 2.925	0.065 ± 0.018	0.054 ± 0.026
Rank-1 VI	-2.806 ± 0.193	13.513 ± 2.930	0.067 ± 0.025	0.018 ± 0.043
iVON	-2.930 ± 0.023	17.904 ± 2.300	0.152 ± 0.015	0.151 ± 0.015
SVGD	-2.855 ± 0.184	14.848 ± 3.170	0.039 ± 0.014	-0.003 ± 0.025
Table 5: UCI-HOUSING (gap splits)
Model	LML	MSE	QCE	sQCE
MAP	-1.702 ± 0.094	1.760 ± 0.289	0.051 ± 0.020	-0.020 ± 0.036
Deep Ensemble	-1.235 ± 0.003	0.177 ± 0.007	0.270 ± 0.002	0.270 ± 0.002
MCD	-1.709 ± 0.079	1.779 ± 0.252	0.049 ± 0.016	-0.022 ± 0.032
MultiMCD	-1.236 ± 0.005	0.212 ± 0.015	0.260 ± 0.003	0.260 ± 0.003
SWAG	-2.127 ± 0.029	2.198 ± 0.274	0.210 ± 0.006	0.210 ± 0.006
MultiSWAG	-2.143 ± 0.002	2.454 ± 0.018	0.220 ± 0.001	0.220 ± 0.001
LL Laplace	-1.653 ± 0.026	0.608 ± 0.110	0.245 ± 0.019	0.245 ± 0.019
LL MultiLaplace	-1.606 ± 0.016	0.235 ± 0.033	0.316 ± 0.008	0.316 ± 0.008
BBB	-0.976 ± 0.123	0.413 ± 0.103	0.055 ± 0.017	0.030 ± 0.032
MultiBBB	-1.022 ± 0.021	0.309 ± 0.075	0.210 ± 0.012	0.210 ± 0.012
Rank-1 VI	-1.029 ± 0.166	0.459 ± 0.145	0.054 ± 0.019	0.019 ± 0.036
iVON	-2.463 ± 0.006	6.620 ± 0.191	0.161 ± 0.010	0.161 ± 0.010
SVGD	-1.322 ± 0.040	0.550 ± 0.121	0.159 ± 0.027	0.159 ± 0.027
Table 6: UCI-ENERGY (standard splits)
Model	LML	MSE	QCE	sQCE
MAP	-7.723 ± 7.553	34.444 ± 41.620	0.247 ± 0.065	0.043 ± 0.195
Deep Ensemble	-4.360 ± 3.066	31.419 ± 36.845	0.272 ± 0.060	0.072 ± 0.207
MCD	-10.299 ± 10.685	48.491 ± 58.682	0.261 ± 0.065	0.041 ± 0.206
MultiMCD	-6.744 ± 6.151	41.030 ± 49.269	0.272 ± 0.061	0.073 ± 0.207
SWAG	-3.655 ± 1.469	30.372 ± 28.902	0.218 ± 0.084	0.011 ± 0.183
MultiSWAG	-3.110 ± 0.815	25.362 ± 22.428	0.192 ± 0.059	0.034 ± 0.152
LL Laplace	-7.009 ± 4.256	45.505 ± 37.787	0.247 ± 0.040	0.116 ± 0.119
LL MultiLaplace	-5.549 ± 2.983	38.452 ± 31.657	0.270 ± 0.046	0.142 ± 0.127
BBB	-64.268 ± 79.182	43.670 ± 52.833	0.199 ± 0.131	-0.101 ± 0.184
MultiBBB	-22.150 ± 25.068	50.502 ± 58.432	0.236 ± 0.110	-0.073 ± 0.202
Rank-1 VI	-72.412 ± 92.191	49.099 ± 60.606	0.191 ± 0.133	-0.109 ± 0.178
iVON	-3.367 ± 0.903	21.546 ± 13.347	0.109 ± 0.025	0.038 ± 0.074
SVGD	-9.945 ± 10.449	46.757 ± 56.551	0.227 ± 0.067	0.037 ± 0.182
Table 7: UCI-ENERGY (gap splits)
G.2 CIFAR-10
Following Wilson et al. [90], we train a ResNet-20 [32] with Swish activations [73] and Filter Response Normalization [82]. The use of Filter Response Normalization instead of batch normalization, which only uses batch statistics, eliminates the problems mentioned in Appendix E. We train all models except iVON with SGD and a learning rate of 0.05 and Nesterov momentum of strength 0.9 for 300 epochs. We use the learning rate schedule from Maddox et al. [59]: The learning rate is kept at its initial value for the first 150 epochs, then linearly reduced to a learning rate of 0.005 at epoch 270 at which it is kept constant for the remaining 30 epochs. For MCD, we use a dropout rate of 0.1 and insert dropout units after every linear and convolutional layer of the ResNet-20. For BBB, we temper the KL divergence in the ELBO with a factor of 0.2. Rank-1 VI uses an untempered posterior and four components. BBB and iVON use two Monte Carlo samples during training. The Laplace approximation is based on a diagonal last-layer approximation. iVON is also trained for 300 epochs with a learning rate of  $1 \cdot 10^{-4}$ , a prior precision of 50, and a data augmentation factor of 10 (see Osawa et al. [70] for details), but uses no learning rate schedule. We found these changes to be necessary to ensure that iVON performs well, likely because iVON is much more similar to Adam [43] than to SGD and therefore needs a smaller learning rate. Following Nado et al. [65], SNGP uses a spectral normalization factor of 6.0 and mean field factor of 20. We did not perform any additional tuning of the mean field factor. We always use 50 parameter samples during evaluation.
Figure 9 displays the accuracy, ECE, sECE, agreement with HMC, and TV compared to HMC. MultiX models tend to become underconfident. Table 8 shows detailed numerical results for all algorithms and corruption levels.