On Uniform, Bayesian, and PAC-Bayesian Deep Ensembles

## Abstract

Abstract Bayesian neural networks address epistemic uncertainty by learning a posterior distribution over model parameters. Sampling and weighting networks according to this posterior yields an ensemble model referred to as Bayes ensemble.
Ensembles of neural networks (deep ensembles) can profit from the cancellation of errors effect: Errors by ensemble members may average out and the deep ensemble achieves better predictive performance than each individual network.
We argue that neither the sampling nor the weighting in a Bayes ensemble are particularly well-suited for increasing generalization performance, as they do not support the cancellation of errors effect, which is evident in the limit from the Bernstein-von Mises theorem for misspecified models.
In contrast, a weighted average of models where the weights are optimized by minimizing a PAC-Bayesian generalization bound can improve generalization performance. This requires that the optimization takes correlations between models into account, which can be achieved by minimizing the tandem loss at the cost that hold-out data for estimating error correlations need to be available.
The PAC-Bayesian weighting increases the robustness against correlated models and models with lower performance in an ensemble. This allows us to safely add several models from the same learning process to an ensemble, instead of using early-stopping for selecting a single weight configuration.
Our study presents empirical results supporting these conceptual considerations on four different classification datasets. We show that state-of-the-art Bayes ensembles from the literature, despite being computationally demanding, do not improve over simple uniformly weighted deep ensembles and cannot match the performance of deep ensembles weighted by optimizing the tandem loss, which additionally come with non-vacuous generalization guarantees.

## 1 Introduction

Combining different deep neural networks to an ensemble model is a common way to increase generalization performance (Bishop and Bishop, ; Goodfellow et al., ).
Such deep ensembles can profit from the cancellation of errors effect (Eckhardt and Lee, ):
When the individual networks perform better than random guessing and make independent errors, the errors in an additive ensemble (or committee) average out and the ensemble tends to outperform the individual models (Hansen and Salamon, ; Perrone and Cooper, ; Dietterich, ).
Many strategies to create the networks for an ensemble have been explored (e.g. Buschjäger et al., ; D’Angelo and Fortuin, ; Dziugaite and Roy, ; Garipov et al., ; Huang et al., ; Jiang et al., ; Lee et al., ; Liu and Yao, ; Masegosa et al., ; Monteith et al., ; Ortega et al., ; Pérez-Ortiz et al., ), however, simple deep ensembles, which solely rely on repeating the neural network training with random weight initializations and stochastic training to generate ensemble members, often already improve generalization (Lakshminarayanan et al., ).

One way to build a neural network ensemble is based on the predictive posterior of Bayesian neural networks (Adlam et al., ; Aitchison, ; Chen et al., ; Farquhar et al., ; Gal and Ghahramani, ; Grünwald, ; Izmailov et al., ; Kapoor et al., ; Kendall and Gal, ; MacKay, ; Maddox et al., ; Monteith et al., ; Nabarro et al., ; Neal, , , ; Pearce et al., ; Ritter et al., ; Welling and Teh, ; Wenzel et al., , ; Wilson and Izmailov, ; Zhang et al., ), and this approach has gained attention recently (Bachmann et al., ; Fortuin et al., ; Rudner et al., ; Wiese et al., ). A Bayesian neural network learns a posterior distribution over the network weights modeling epistemic uncertainty (MacKay, ; Neal, ). Averaging networks sampled from this posterior yields an approximation of the Bayesian model average (BMA) and is referred to as Bayes ensemble.
This approach comes with great expectations, for example, Domingos ()
stated ‘Given the “correct” model space and prior distribution, Bayesian model averaging is the optimal method for making predictions; in other words, no other approach can consistently achieve lower error rates than it does.’
In this work, *we argue that neither the sampling nor the weighting in a Bayes ensemble are particularly well-suited for increasing generalization performance in theory and practice.*
The Bernstein-von Mises theorem (assuming identifiability of the model) shows that the BMA converges towards the maximum likelihood point estimate with growing dataset size (Kleijn and Van der Vaart, ). In the limit, it will eventually concentrate on a single model without exploiting ensemble diversity. Thus, the Bayes ensemble typically does not leverage the cancellation of errors effect. This is particularly problematic because the considered model space in the BMA setting is typically not ‘correct’: The Bayes ensemble itself is in general outside the model space from which its members are sampled – and in the space of ensemble models the Bayes ensemble cannot be expected to be optimal.

PAC-Bayesian methods (McAllester, ) provide an alternative approach to incorporate prior information about models and to improve the weighting of ensemble members.
We bring forward optimizing the weighting of deep ensemble members using a PAC-generalization bound based on the tandem loss, which accounts for pairwise correlations between networks (Masegosa et al., ). This allows us to maintain diversity and increase generalization performance of the deep ensemble.
The PAC-Bayesian weight optimization is especially useful when intermediate weight configurations (snapshots) from model training are considered, in which case the optimization performs model selection taking both individual network performance and ensemble diversity into account. The price to pay for the improved weighting is additional hold-out data for the optimization of the weighs, however, then the resulting PAC-Bayesian bound provides a rigorous performance guarantee.
The main contributions of this study can be summarized as:

- •
We discuss the conceptual differences between simple, Bayes and PAC-Bayesian deep ensembles, stressing that Bayes ensembles do not consider the cancellation of errors effect, which explains some observations in the current literature.
- •
Unbiased experiments on four different datasets show that complex Bayesian approximate inference methods can often be surpassed by more efficient simple deep ensembles.
Bagging decreased predictive performance, showing the trade-off between the degree of randomization and single-network performance given limited training data.
- •
We show that the optimization of a PAC-Bayesian generalization bound using the tandem loss can improve the predictive performance of deep ensembles and provides non-vacuous generalization guarantees.
- •
We demonstrate that the inclusion of intermediate checkpoints from the same training run in a neural network ensemble can increase its predictive performance, especially when the PAC-Bayesian bound optimization weights their contributions.

In the next section, we will briefly summarize the background on Bayesian model averaging, the cancellation of errors effect, deep ensembles, and PAC-Bayesian majority voting.
Section 3 will bring these topics together: It contrasts uniform, Bayes, and PAC-Bayesian deep ensembles and formulates our main hypotheses. These hypotheses will be empirically studied in Section 4 before we conclude.

## 2 Background

We consider data $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n}$ drawn i.i.d. from a distribution $p$.
The loss of a hypothesis/model $h$ on $(x,y)$ is denoted by $\ell(h(x),y)$. The generalization error (risk) of $h$ is given by $L(h)=\mathbb{E}_{(x,y)\sim p}[\ell(h(x),y)]$ and the empirical risk by $\hat{L}_{\mathcal{D}}(h)=\frac{1}{n}\sum_{i=1}^{n}\ell(h(x_{i}),y_{i})$.
We write $h_{w}$ to stress that a model is parameterized by $w\in\mathbb{R}^{d}$.
We consider ensembles of $M$ models $h_{i}=h_{w_{i}}\in\mathcal{H}$, $i=1,\dots,M$, weighted by $\rho\in\mathbb{R}^{M}$.
For regression, the ensemble regressor is $h_{\rho}(x|h_{1},\dots,h_{M})=\sum_{i=1}^{M}\rho_{i}h_{i}(x)$.
For classification, we distinguish two ways of combining the predictions of the ensemble members.
We assume a predictive distribution $p_{i}(y|x)=p(y|x,w_{i})$ associated with each hypothesis $h_{w_{i}}(x)$.
The ensemble prediction $h_{\rho}(x|h_{1},\dots,h_{M})$ can be either defined by the $\rho$-weighted average of these distributions
$\mathrm{AVG}_{\rho}(x)=\arg\max_{y}\sum_{i=1}^{M}\rho_{i}p_{i}(y|x)$ or by the
$\rho$-weighted majority vote
$\mathrm{MV}_{\rho}(x)=\arg\max_{y}\sum_{i=1}^{M}\rho_{i}\mathds{1}[y=\arg\max_
{y^{\prime}}p_{i}(y^{\prime}|x)]$, where $\mathds{1}[\cdot]$ is 1 if its argument is true and zero otherwise. Uniform weighted aggregations are denoted by $\mathrm{AVG}_{u}$ and $\mathrm{MV}_{u}$.

##### Bayesian model average.

In Bayesian inference, we update our prior belief about the parameters $w$ of a model predicting $y$ given $x$ according to $p(y|x,w)$
to a posterior distribution
$p(w|\mathcal{D})$ based on training data $\mathcal{D}$.
Marginalizing $p(y|x,w)$ over this posterior
gives the posterior predictive distribution

$$ $\displaystyle p(y|x,\mathcal{D})$ $\displaystyle=\int p(y|x,w)p(w|\mathcal{D})\mathrm{d}w,$ (1) $$

also known as Bayesian Model Average (BMA), Bayes ensemble or posterior predictive. It predicts by averaging over all possible models weighted by their
posterior probability. The BMA explicitly models aleatoric uncertainty (given by $p(y|x,w)$) and epistemic uncertainty in the form of $p(w|\mathcal{D})$.
As the integral in (1) is in general intractable, it is approximated in practice by an average of $M$ models sampled from the posterior:

$$ $\displaystyle p(y|x,\mathcal{D})$ $\displaystyle\approx\frac{1}{M}\sum_{m=1}^{M}p(y|x,w_{m}),\quad w_{m}\sim p(w| \mathcal{D})$ (2) $$

It is in general difficult to sample from $p(w|\mathcal{D})$,
which can, for example, be addressed by
Markov chain Monte Carlo (MCMC) sampling (Chen et al., ; Neal, , ; Welling and Teh, ; Wenzel et al., ; Zhang et al., ), Monte Carlo Dropout (Folgoc et al., ; Gal and Ghahramani, ),
and Laplace approximation (MacKay, ; Ritter et al., ).

##### Cancellation of errors.

The cancellation of errors effect refers to the fact that,
when individual ensemble members perform better than random guessing and make independent errors, their errors average out and the combined model
outperforms the individual predictors
(Eckhardt and Lee, ; Hansen and Salamon, ).
Let us assume binary classification and that each ensemble member has an error probability lower bounded by $p_{\max}<\frac{1}{2}$. Given $M$ models, the generalization error of the ensemble is upper bounded by
$\exp\left(-{2\big{(}\frac{M+1}{2}-{M}p_{\max}\big{)}^{2}}{M^{-1}}\right)$, see Appendix A.3.
That is, in the idealized (and not realistic) setting of independent errors, the generalization error vanishes with increasing $M$ if the classifiers are better than random guessing.
The corresponding regression case relating expected mean-squared error and correlation of the model outputs is discussed in deep learning textbooks by (Goodfellow et al., , sec. 7.11) and (Bishop and Bishop, , sec. 9.6).

##### Deep ensembles.

Deep ensembles (Lakshminarayanan et al., ) are ensembles with deep neural networks as members. More specifically, we refer to a deep ensemble if the networks all share the same structure.
These networks are typically the result of independent training processes.
The diversity is a result of random initialization and the stochastic optimization. This randomness alone can yield well-performing ensembles, while additional randomization of the training data by bootstrap aggregation (bagging) bears the risk of the ensemble being worse than a single network trained on all data (Lakshminarayanan et al., ; Lee et al., ).
Other ways to create deep ensemble members include the use of a cyclical learning rate schedule and taking the intermediate checkpoints as snapshot ensembles (SSEs) (Huang et al., ) as well as searching for other ensemble members in neighborhood of a single pre-trained network (fast geometric ensembling (Garipov et al., )).
While being introduced as an alternative to Bayesian approaches,
Bayesian interpretations of deep ensembles can be found in recent work (D’Angelo and Fortuin, ; Wilson and Izmailov, ).

##### PAC-Bayesian majority voting.

PAC-Bayesian analysis (McAllester, ; Valiant, ) provides bounds on the generalization error of Gibbs classifiers
defined by a distribution $\rho$ over a (subset of a) hypothesis space $\mathcal{H}$
given empirical risks for the hypotheses.
A Gibbs
classifier draws a hypothesis $h$ according to $\rho$ at random for each input $x$ and returns the prediction $h(x)$.
PAC-Bayesian bounds hold for all distributions $\rho$ over $\mathcal{H}$ simultaneously. This allows us to directly optimize a PAC-Bayesian bound in terms of $\rho$, e.g., by gradient-based methods (Dziugaite and Roy, ; Masegosa et al., ; Masegosa, ; Ortega et al., ; Pérez-Ortiz et al., ; Thiemann et al., ).
In a weighted majority voting classifier $\mathrm{MV}_{\rho}$ (e.g., a deep ensemble), the randomized prediction is replaced by a $\rho$-weighted vote by all hypotheses (Germain et al., , ). PAC-Bayesian bounds can be applied to $\mathrm{MV}_{\rho}$ by bounding $L(\mathrm{MV}_{\rho})$ by twice the risk of the corresponding Gibbs classifier (Germain et al., ; Langford and Shawe-Taylor, ; Masegosa et al., ). Gibbs classifiers ignore interactions between the models (such as cancellation of errors), and optimization of the weighting using bounds on the Gibbs classifier directly does typically not increase generalization performance of the ensemble (e.g., Lorenzen et al., ). This can be addressed by second order PAC-Bayesian bounds (Germain et al., ; Lacasse et al., ; Masegosa et al., ). In particular,
derive a bound in terms of the
tandem loss $L(h,h^{\prime})=\mathbb{E}_{p(x,y)}[\mathds{1}(h(x)\neq y\land h^{\prime}(x)
\neq y)]$, which takes pairwise correlations into account:
For any probability distribution $\pi$ on $\mathcal{H}$ that is independent of $\mathcal{D}$ and any $\delta\in]0,1[$,
with probability at least $1-\delta$ over a random draw of $\mathcal{D}$ with $n$ elements, for all distributions $\rho$ on $\mathcal{H}$ and all $\lambda\in]0,2[$
simultaneously:

$$ $L(\mathrm{MV}_{\rho})\leq 4\left(\frac{\mathbb{E}_{\rho^{2}}[\hat{L}_{\mathcal {D}}(h,h^{\prime}])}{1-\lambda/2}+\frac{2\mathrm{KL}(\rho\|\pi)+\ln(2\sqrt{n}/ \delta)}{\lambda(1-\lambda/2)n}\right)$ (3) $$

Here $\mathrm{KL}(\rho\|\pi)$ denotes the Kullback–Leibler divergence between prior $\pi$ and posterior $\rho$. The bound can be efficiently optimized w.r.t. to $\rho$ and $\lambda$ (Thiemann et al., ).

We consider the optimization of $\rho$ (i.e., the weights in a weighted ensemble) given $M$ models as proposed by Masegosa et al. (). One could instead also turn a second order PAC-Bayesian bound into an objective function for optimizing $\rho$ and the parameters of the $M$ models simultaneously.
This has been done by Ortega et al. () to optimize deep ensembles. However, because of the simultaneous optimization of the neural networks’ parameters and their weighting, the formal guarantees by the PAC-Bayesian bounds get lost. Furthermore, did not find improvements compared to uniformly weighted deep ensembles for larger networks (e.g., ResNet20, He et al., ) in their empirical evaluation.

## 3 Uniform, Bayesian, and PAC-Bayesian deep ensembles

##### Bayes ensembles and cancellation of errors.

As outlined above, general ensemble methods and Bayesian neural networks have different motivations.
In a weighted additive ensemble, a finite set of
hypotheses from $\mathcal{H}$ are combined to a new model $h_{\rho}\in\mathcal{H}_{\text{E}}$
from the hypothesis class

$$ $\mathcal{H}_{\text{E}}=\left\{h_{\rho}(x|h_{1},\dots,h_{M})\,\bigg{|}\,M\in{ \mathbb{N},\rho\in\mathbb{R}^{M},h_{i}\in\mathcal{H}}\right\}.$ (4) $$

As discussed in Section 2, $h_{\rho}$ can generalize better than the individual $h_{1},\dots,h_{M}$ if the ensemble members are diverse in the sense that their errors are not strongly correlated.
Typically, $\mathcal{H}\subset\mathcal{H}_{\text{E}}$, and for most ensemble methods this is key.
The ensemble members selected from $\mathcal{H}$ are often referred to
as weak learners and are not expected to have a very low risk (generalization performance)
as long as they are better than random guessing: The desired model is expected to be in $\mathcal{H}_{\text{E}}\setminus\mathcal{H}$. Boosting with decision stumps is a prime example (Freund et al., ).

In Bayesian machine learning, the posterior distribution $p(h|\mathcal{D})$ over $\mathcal{H}$ models the uncertainty of identifying a desired model $h^{*}\in\mathcal{H}$ given finite training data $\mathcal{D}$ and the prior over $\mathcal{H}$.(^1^11The desired model in $\mathcal{H}$ is the model in $\mathcal{H}$ closest to the true model $h_{p}$ underlying the data generating distribution $p$. We do neither assume $h_{p}\in\mathcal{H}$ nor $h_{p}\in\mathcal{H}_{\text{E}}$.) In Bayesian deep learning with a fixed neural network architecture $h_{w}$ parameterized by weights $w\in\mathbb{R}^{d}$, we have $\mathcal{H}=\{h_{w}|w\in\mathbb{R}^{d}\}$ and the posterior over the hypotheses is modelled by a distribution $p(w\,|\,\mathcal{D})$ over the weights $w$.
To simplify the discussion, let us assume that the models are parameterized in a way that ensures identifiability, that is, different parameters generate different hypotheses.
With increasing training data set size $n=|\mathcal{D}|$ the uncertainty about the desired model decreases, and
$p(w\,|\,\mathcal{D})$ should concentrate on $w^{*}$ (i.e., the weights of the desired model $h^{*}=h_{w^{*}}$).
This is formally shown by the Bernstein-von Mises theorem (Kleijn and Van der Vaart, ; Van der Vaart, ), which was first discovered by Laplace (), taken up by Bernstein () and von Mises () independently,
and proven in a more general form by
Le Cam (), who coined the name.
Let the parametric model $h_{w}$ be twice differentiable with non-singular Fisher information matrix $\mathcal{I}(w^{*})$ at $w^{*}$, identifiable and continuous on a compact parameter space.
Further, let the prior measure be absolutely continuous in a neighbourhood of $w^{*}$ with a continuous density non-zero at $w^{*}$.
Let us consider the sequence of posterior distributions $p_{n}(w\,|\,\mathcal{D}_{n})$
and maximum likelihood estimates
$\hat{w}_{n}$ given $\mathcal{D}_{n}$ for data sets $\mathcal{D}_{n}\subset\mathcal{D}_{n+1}$.
The Bernstein-von Mises theorem implies

$$ $\displaystyle\lVert p_{n}(w|\mathcal{D}_{n})-\mathcal{N}(\hat{w}_{n},{n}^{-1} \mathcal{I}^{-1}(w^{*}))\rVert_{\text{TV}}\overset{p_{w^{*}}}{\rightarrow}0.$ (5) $$

That is, for increasing data set size the posterior converges in probability to a normal distribution $\mathcal{N}(\hat{w}_{n},{n}^{-1}\mathcal{I}^{-1}(w^{*}))$, where $\|\cdot\|_{\text{TV}}$ denotes the total variation norm (see Van der Vaart, ). The normal distribution is centered on the maximum likelihood estimate with covariance ${n}^{-1}\mathcal{I}^{-1}(w^{*})$, that is, the distribution gets more and more concentrated with increasing data set size.
Under the same assumptions, this implies that the expectation of the Bayes ensemble (2)
converges to the maximum likelihood estimate $h_{{w^{*}}}\in\mathcal{H}$ for $n\to\infty$ (while the Bayes ensemble is not necessarily in $\mathcal{H}$ for finite $n$).

Thus, from the perspective on general ensemble learning and the goal of finding a model with minimum risk, a sample from the posterior
$p(w\,|\,\mathcal{D})$ does not appear to be a particularly good choice for ensemble members.
As the sample will be distributed around the maximum likelihood estimate $h_{w^{*}}\in\mathcal{H}$, the ensemble members cannot be expected to be very diverse, and the lack of diversity reduces the cancellation of errors effect. The Bayes ensemble will tend to the model with the highest likelihood in $\mathcal{H}$ – and not to the model with the highest likelihood in $\mathcal{H}_{\text{E}}$.
One may well argue that the presented asymptotic argument is of little practical relevance if
$\mathcal{H}$ already has a high capacity (e.g., consists of overparameterized deep neural networks), because then
$h_{w^{*}}\in\mathcal{H}$ may be close to the optimal solution. However, the important point is that even for finite $n$, the $M$ ensemble members can be expected to be quite similar in their predictive behavior.
Thus, we formulate the following hypothesis:

###### Hypothesis 1

The Bayes ensemble is not a particularly good way of selecting and weighting networks in a deep ensemble.

While this hypothesis appears to be evident given the theoretical considerations above, many studies evaluate the performance of the Bayes ensemble by measuring generalization performance on a test set and provide favorable comparisons to baselines suggesting the posterior predictive as a way to maximize generalization performance. This study presents unbiased comparisons of Bayes ensembles with simple deep ensemble approaches to test Hypothesis 1.

A Bayesian way to finding the best generalizing ensemble is to lift the Bayesian reasoning from $\mathcal{H}$ to $\mathcal{H}_{\text{E}}$. This has been demonstrated by Monteith et al. (), who apply BMA not to $h_{w}\in\mathcal{H}$, but to $h_{\rho}\in\mathcal{H}_{\text{E}}$, where $\mathcal{H}_{\text{E}}$ are all ensembles of a constant size $M$. In such an approach, the posterior distribution contracts around the maximum likelihood estimate of the best weighting of ensemble members.
Thus, we go from a distribution over the weights of a single neural network to a distribution over all parameters of the $M$ networks and the ensemble weights. This growth in the number of dimensions renders the approach computationally infeasible in larger settings, however.

###### Hypothesis 1

##### PAC-Bayesian deep ensembles.

If Hypothesis 1 holds true, this does neither mean that we cannot do better than uniform averaging given $M$ models from $\mathcal{H}$ (as already pointed out in early works on ensembles, e.g., Krogh and Sollich, ) nor that we cannot make use of priors and Bayesian reasoning to improve ensembles.
For example, there is no need to put weight on an ensemble member that is worse than random guessing or always errs when another ensemble member errs (i.e., it can never correct an error). Given the results by Masegosa et al. (), it appears to be promising to weight ensemble members based on PAC-Bayesian generalization bounds optimized using the tandem loss.
However, the bounds require for each ensemble member $h_{i}$ that there are data points $\mathcal{D}_{i}$ not used for training $h_{i}$ and that for each pair of models $h_{i}$ and $h_{j}$
the overlap $\mathcal{D}_{i}\cap\mathcal{D}_{j}$ is large enough for a good enough estimate of the tandem loss:
In (3), we compute
$\hat{L}_{\mathcal{D}_{i}\cap\mathcal{D}_{j}}(h_{i},h_{j})$ for
each hypotheses pair $h_{i}$ and $h_{j}$ and $n$ becomes $n=\min_{i}|\mathcal{D}_{i}|$.
For random forests as considered by Lorenzen et al. () and Masegosa et al. (), data sets $\mathcal{D}_{i}\subset\mathcal{D}$ naturally arise from the bagging procedure and the proposed bounds can be computed and optimized for free. When applied to deep ensembles, we have to pay by leaving data out when training the ensemble members, which can be expected to reduce the performance of the individual networks. Taking this into account, we hypothesize:

###### Hypothesis 2

PAC-Bayesian weighting optimized using the tandem loss can improve the generalization performance of a deep ensemble, in particular if extra data for computing the bound are available, and provide non-vacuous performance guarantees.

In most deep ensemble approaches, each ensemble member is the result of an independent training process, where the weight configuration is chosen by early-stopping (requiring some hold-out data) or simply after a predefined number of learning iterations (which is then also a crucial hyperparameter).
In contrast, Wenzel et al. ()
sample from the posterior using a single
stochastic gradient Markov chain Monte Carlo
(SG-MCMC) sequence and SSEs combine networks from a single learning process (Huang et al., ). That is, one process creates all ensemble members.
If the latter part of Hypothesis 2 is true, one need not worry about adding models to the ensemble that do not perform particularly well and/or are correlated.
The PAC-Bayesian weighting performs a model selection taking both individual performance and diversity into account, which should provide protection from networks not contributing positively to a ensemble.
This makes it safe to add
several weight configurations (checkpoints) from a single learning process to an ensemble.
It also renders using a hold-out validation dataset for early-stopping the neural network training unnecessary: It should neither be harmful to add underfitted nor overfitted models. The early-stopping data can instead be used to optimize the PAC-Bayesian weighting:

###### Hypothesis 3

Optimizing the weighting using the tandem loss allows inclusion of several models from a training run in a way that efficiently improves performance and makes early-stopping unnecessary.

###### Hypothesis 2

###### Hypothesis 3

## 4 Experiments and results

### 4.1 Experimental setup

Four different datasets and four neural network architectures were considered.(^2^22We consumed approximately 320 GPU days for all experiments on a local cluster.) We evaluated a CNN LSTM (Yenter and Verma, ) on the IMDB binary classification benchmark (Maas et al., ) and compared it to a state-of-the-art Bayesian approximate inference method from Wenzel et al. () (referred to as cSGHMC-ap), who combine stochastic gradient Langevin dynamics (SGLD) (Chen et al., ; Welling and Teh, )
with a cyclical learning rate schedule (Zhang et al., ), and adaptive preconditioning (Li et al., ; Ma et al., ). Furthermore, we considered the multi-class data sets CIFAR-10 and CIFAR-100 (Krizhevsky, ) with ResNet110 (He et al., ) and WideResNet28-10 (WRN28-10) (Zagoruyko and Komodakis, ) architectures, for which Ashukha et al. () and Vadera et al. () provide reference results for a variety of Bayesian methods. Lastly, the ResNet50 was evaluated on the EyePACS dataset containing diabetic retinopathy diagnoses distinguishing five degrees of severity. It was introduced by Band et al. () as a benchmark for Bayesian approximate inference methods. The authors provide results for a variety of Bayesian neural networks
and ensembles of those, creating “deep ensembles of Bayes ensembles”. As we are concerned with a comparison to Bayes ensembles, we focus on the results of the individual Bayesian methods as reference. The hyperparameters for each experiment are given in Appendix A.3.

To optimize the weighting while retaining the original training data set size, test-time cross validation as introduced by Ashukha et al. () was employed. We used 50% of the hold-out data for bound optimization and the other half for testing. This was then repeated with the two subsets switching roles, and the results were averaged to decrease the variance for the unbiased performance estimate, see Appendix A.2 for details.

For each dataset and model architecture, a simple deep ensemble was constructed based on the reference paper’s single-run hyperparameters. Intermediate checkpoints were stored and either included (referred to as all from now) or ignored in the final ensemble (last). Further, ensembles with sub-sampled training sets (bagging) as well as snapshot ensembles (SSE) using a cyclical learning rate were formed. Finally, the trade-off between ensemble size and number of training epochs was studied in a sequential setting in Appendix A.4.2.

We trained $M_{\text{total}}$ networks for each setting, where $M_{\text{total}}$
was 30, 50, and 10 for CIFAR, IMDB, and EyePACS, respectively.
Ensembles of size $M$ were created by sampling without replacement from these models. This was repeated five times for every setting, and we report mean and standard deviation $\sigma$ over these five trials.
Generalization bounds are reported for $\delta=0.05$.

Figure: Figure 1: Mean test accuracy $\hat{A}$ $\pm\sigma$ vs. ensemble size $M$ over five ensembles for uniformly and PAC-Bayesian weighted deep ensembles ($\mathrm{AVG}_{u}$ and $\mathrm{AVG}_{\rho}$), using either only the last or all training checkpoints, and the best single model (SGD). References are Bayesian ensembles cSGHMC-ap (Wenzel et al., ), cSGLD (Ashukha et al., ), *MC-Dr*opout and *MC-Dr*opout *D*eep *E*nsemble (i.e., ensemble of Bayesian ensembles), as well as simple *D*eep *E*nsembles for EyePACS from Band et al. (). Numbers in brackets indicate the ensemble sizes of the baselines. Additional results for other settings are shown in Figure A.4 in the appendix.
Refer to caption: x1.png

### 4.2 Results

##### Bayesian vs. uniform deep ensembles (Hypothesis 1 ).

We compared the uniformly weighted deep ensembles across all datasets and model architectures with Bayesian ensembles from the literature.
Our results shown in Table 1 and figures 1
and A.4
demonstrate that uniformly weighted deep ensembles can match the performance of models based on BMA. On IMDB, a uniform deep ensemble of size $M\geq 20$ matched the reported performance from Wenzel et al. (), and equivalent results were observed for all other datasets and architectures. On CIFAR-100 with a ResNet110, the uniform deep ensemble outperformed the Bayesian reference from Ashukha et al. () by 1.3ppt. Interestingly, the SGD and deep ensemble results from are notably better than ours. As the authors base their Bayesian methods on pre-trained SGD solutions, we can assume that their Bayesian reference performances could also profit from the better accuracies achieved in their training environment. Thus, it is remarkable that our uniform deep ensembles nevertheless matched the Bayesian references, and we would expect an even better performance if compared in the same environment.

In accordance with the literature, bagging reduced the performance, see Appendix A.4.1 for details. The experiments on the trade-off between $M$ and the number of training epochs showed that, even in a sequential setting with a strict limit on the number of overall training epochs, deep ensembles could match the BMA approach, see Appendix A.4.2 for details.

Table: Table 1: Mean test accuracies $\hat{A}$ over five ensembles, ensemble size is given in brackets (for other ensemble sizes see figures A.4, A.5, A.6, and A.7 in the appendix). The *Simple* results refer to simple deep ensembles with SGD hyperparameters from the reference papers. The subscript $\rho$ indicates that the weighting of the ensemble members was based on minimizing a PAC-Bayesian tandem bound. The *Bayesian reference* results are taken from Ashukha et al. () (CIFAR ResNet110 & WRN28-10), Band et al. () (EyePACS) and Wenzel et al. () (IMDB, CIFAR-10 ResNet20).

##### Bayesian vs. PAC-Bayesian deep ensembles (Hypothesis 2 ).

Optimizing the weighting by minimizing the second-order PAC-Bayesian bound matched the uniform performance in all cases $\pm 0.1$ppt, while even improving the ensemble performance slightly on IMDB (Table 1: $\hat{A}(\mathrm{AVG}_{u})$ last vs. $\hat{A}(\mathrm{AVG}_{\rho})$ last). At the same time, the PAC generalization guarantees in Table 2 – which still hold after optimization – tightened dramatically, most significantly on CIFAR-10 ResNet20 by 31.8ppt, from 0.425 to 0.743. For ResNet110 on CIFAR-100, the optimization was necessary to get a non-trivial bound, increasing from 0.0 to 0.127. The PAC-Bayesian guarantees hold for $\mathrm{MV}_{\rho}$ as aggregation method, and $\mathrm{AVG}_{\rho}$ usually performs slightly better in practice. However, Table A.4 in the appendix shows how similar the two aggregation methods behave. That is, a slight decrease in performance can give rigorous generalization bounds.

Computing and optimizing the PAC-Bayesian bound requires additional hold-out data, which biases the comparison with uniform weighting.
Instead of using extra data, one could use bagging (Lorenzen et al., ; Masegosa et al., ).
The results of our bagging experiments are presented in Appendix A.4.1.
For our neural networks ensembles and rather small data sets,
bagging decreased the performance, which is in line with the literature (Lakshminarayanan et al., ; Lee et al., ).

##### Bayesian vs. PAC-Bayesian deep snapshot ensembles (Hypothesis 3 ).

We evaluated snapshot ensembling (Huang et al., ) as well as simply taking checkpoints from the original learning rate schedule. In SSEs, going from one to all intermediate checkpoints with optimized weighting improved predictive performance in all but one cases.
On IMDB, this improvement was most pronounced with 13.3ppt (Table 1: $\hat{A}(\mathrm{AVG}_{\rho})$ last vs. $\hat{A}(\mathrm{AVG}_{\rho})$ all). An SSE with 8 members (and 10 snapshots per member) gave the best accuracy with a training budget ($8\cdot 50=400$ epochs) below the budget of the Bayesian reference method (500 epochs).
SSEs include networks when the learning rate is lowest, just before re-starting the learning rate cycle. However, including checkpoints following the original schedule also matched the baseline of taking only the last checkpoint across all experiments. This straightforward approach improved performance on IMDB and EyePACS (Table 1: $\hat{A}(\mathrm{AVG}_{\rho})$ last vs. $\hat{A}(\mathrm{AVG}_{\rho})$ all). This is in contrast to the uniform weighting results, where adding the snapshots decreased the performance compared to only taking the last network.
As hypothesized, it is the tandem loss bound optimization that allows us to include worse performing models in the ensemble (at no additional training cost), while maintaining or improving performance. Figures A.2 and A.3 in the appendix depict the weighting from minimizing the tandem loss (and a first order) PAC-generalization bound. They show that the second order
objective function avoids
putting all weight on a single hypothesis.

**Table 2: Test accuracies for $\mathrm{AVG}_{\rho}$ and $\mathrm{MV}_{\rho}$ and PAC-Bayesian bounds ($\delta=0.05$) for all *Simple* experiments (see Table 1). An overview over all experiments is shown in Figure A.4 in the appendix. The bounds were computed for the *all* setting, except for IMDB, where checkpoints were taken in a separate experiment due to early-stopping in *Simple*; $\text{MV}_{\text{u}}$ and $\mathrm{MV}_{\rho}$ refer to majority voting with uniform and optimized $\rho$, respectively.**
| Model | Dataset | Bound<br>$\text{MV}_{\text{u}}$ | Bound<br>$\mathrm{MV}_{\rho}$ | $\hat{A}(\mathrm{AVG}_{\rho})$<br>last | $\hat{A}(\mathrm{MV}_{\rho})$<br>last | $\hat{A}(\mathrm{AVG}_{\rho})$<br>all | $\hat{A}(\mathrm{MV}_{\rho})$<br>all |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CNN LSTM | IMDB | 0.576 | 0.594 | 0.873 | 0.872 | 0.873 | 0.874 |
| ResNet20 | CIFAR-10 | 0.425 | 0.743 | 0.938 | 0.938 | 0.938 | 0.938 |
| ResNet110 | CIFAR-10 | 0.559 | 0.792 | 0.956 | 0.956 | 0.956 | 0.955 |
| WRN28-10 | CIFAR-10 | 0.794 | 0.833 | 0.967 | 0.966 | 0.967 | 0.966 |
| ResNet110 | CIFAR-100 | 0.0 | 0.127 | 0.793 | 0.791 | 0.793 | 0.791 |
| WRN28-10 | CIFAR-100 | 0.172 | 0.277 | 0.83 | 0.829 | 0.829 | 0.829 |
| ResNet50 | EyePACS | 0.686 | 0.695 | 0.911 | 0.909 | 0.912 | 0.912 |

## 5 Discussion and conclusions

Bayesian model averaging (BMA) is not meant to maximize predictive performance but to get a better idea about the maximum likelihood estimate of the parameters of a single model.
Still, the Bayes ensemble is often – implicitly and explicitly – brought forward as an approach to improve generalization performance. Several lines of research study the generalization performance of the Bayes ensemble and try to improve it, where performance is measured in terms of accuracy on a test set.
Here we argued conceptually and showed empirically
that the predictive posterior is not a particularly good basis for selecting and weighting the networks in a deep ensemble.
A proper Bayesian way to build an ensemble would be to apply Bayesian inference to the space of ensembles (Monteith et al., ), but this is computationally expensive and – also for that reason – is not what is typically proposed in the research directions our study addresses.
A simple deep ensemble can be expected to perform on par with BMA based approaches and it can be created more efficiently. It does not require sampling from the posterior and training is embarrassingly parallel.

Optimizing the weighting of the ensemble members using the tandem loss (Masegosa et al., ) and additional data can improve uniform deep ensembles.
Using a second order bound to derive the optimization objective is crucial, because minimizing a first order bound will lead to a lack of diversity similar to BMA (see also Lorenzen et al., ).
The PAC-Bayesian weighting allows us to include several models from a training run, which can improve performance efficiently and make early-stopping unnecessary. It has already been argued by Sollich and Krogh () that having models overfitted to a subset of the training data in a properly weighted ensemble need not be harmful.
While no hold-out data for early-stopping is needed, it is a limitation of the weighting approach that additional hold-out data is required.
For the rather small data sets and the models considered in this study, experiments with bagging showed that leaving out training data impaired the performance of the ensemble (as also observed by Lakshminarayanan et al., , and Lee et al., ),
even though bagging increases diversity.
However, the data used for optimization of the weighting provide at the same time a rigorous, non-trivial upper bound on the generalization performance – and getting such formal guarantees generally requires independent data. The PAC-Bayesian bounds still hold after optimization.
In our case, this optimization was necessary to elevate some of the bounds from being trivial to non-vacuous guarantees on the ensemble performance.
The bound presumes model combination by majority voting as prediction method, however, in our experiments we did not observe a big difference in generalization performance between majority voting and averaging predictive distributions.
Thus, PAC-Bayesian optimization of the ensemble weights is highly recommended if not all data available for model construction is needed for training so that a subset can be employed for optimization and getting performance guarantees.

## Acknowledgments and Disclosure of Funding

CI acknowledges support through the Global Wetland Center,
funded by the Novo Nordisk Foundation under grant number NNF23OC0081089 in the call “Challenge Programme 2023 – Prediction of Climate
Change and Effect of Mitigating Solutions”, and the Pioneer Centre for AI, DNRF grant number P1.

## Appendix A Appendix / supplemental material

### A.1 Experimental details

The experimental setups including hyperparameters were taken from the respective cited references and corresponding source code.
Thus, no hyperparameter tuning was performed, except for the EyePACS dataset, where the hyperparameters from the literature did not lead to convergence of the models and the learning rate was decreased by a factor of 100.

##### IMDB.

For IMDB, the described setup from Wenzel et al. () was copied as closely as possible. The dataset from the tensorflow.keras.datasets API was used with 20,000 word features and a maximum sequence length of 100. 20,000 training samples with 5,000 random validation samples being used for training with early-stopping due to overfitting. The test set included all 25,000 samples from the original test set. The CNN LSTM model and code were taken from the Keras example(^3^33[https://github.com/keras-team/keras/blob/1a3ee8441933fc007be6b2beb47af67998d50737/examples/imdb_cnn_lstm.py](https://github.com/keras-team/keras/blob/1a3ee8441933fc007be6b2beb47af67998d50737/examples/imdb_cnn_lstm.py)) and extended with a Gaussian prior $\mathcal{N}(0,I)$ for regularization, as done by the authors in their code repository(^4^44[https://github.com/google-research/google-research/tree/master/cold_posterior_bnn](https://github.com/google-research/google-research/tree/master/cold_posterior_bnn)). As optimizer, SGD with Nesterov momentum of 0.98 and a batch size of 32 with a constant learning rate was utilized. For the cyclic learning rate in all snapshot ensemble (SSE) experiments, the original schedule from Huang et al. () was used.

##### CIFAR.

The CIFAR datasets were taken from the tensorflow.keras.datasets API with the original train and test split. Similar to Wenzel et al. (), no early-stopping with a validation set was employed. The ResNet20 model was taken from the Keras example (^5^55[https://github.com/keras-team/keras/blob/1a3ee8441933fc007be6b2beb47af67998d50737/examples/cifar10_resnet.py](https://github.com/keras-team/keras/blob/1a3ee8441933fc007be6b2beb47af67998d50737/examples/cifar10_resnet.py)). As in the example, all experiments used data augmentation during training with random left/right flipping and random cropping with $4px$ of shift horizontally and vertically. The hyperparameters for the ResNet20 were adopted from Wenzel et al. (), while they were taken from Ashukha et al. () for the ResNet110 and Wide ResNet28-10. As optimizer, SGD with Nesterov momentum of 0.9 and a batch size of 128 with a step-wise decreasing learning rate ($\eta$) was employed (epoch, $\eta$-multiplier): (80, 0.1), (120, 0.01), (160, 0.001), (180, 0.0005). For the ResNet110 and WRN28-10, a linearly decreasing learning rate schedule starting at half of the total number of epochs was utilized, as reported by Ashukha et al. () and introduced by Garipov et al. ():

$$ $\displaystyle\eta(i)=\begin{cases}\eta_{\mathrm{init}},&i\in[0,0.5\cdot\mathrm {epochs}]\\ \eta_{\mathrm{init}}\cdot(1.0-0.99\cdot(i/\mathrm{epochs}-0.5)/0.4),&i\in[0.5 \cdot\mathrm{epochs},0.9\cdot\mathrm{epochs}]\\ \eta_{\mathrm{init}}\cdot 0.01,&\mathrm{otherwise}\\ \end{cases}$ (A.6) $$

**Table A.3: Hyperparameters for all experiments. *Ep.b.* (Epoch budget) refers to the experiments in section A.4.2.**
| Model | Experiment | Val. set | $lr_{\mathrm{init}}$ | CP | epochs | L2-reg. | LR Sched. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| (Dataset) |  |  |  |  |  |  |  |
| IMDB<br>(CNN LSTM) | Simple | 20% | 0.1 | 1 | 50 | $\mathcal{N}(0,I)$ | constant |
| Checkp. | 20% | 5 | 5 | constant |  |  |  |
| Bagging | Bagging | 1 | 50 | constant |  |  |  |
| SSE | 20% | 5 | 50 | cyclic |  |  |  |
| Ep.b. | 20% | 1 | variable | constant |  |  |  |
| ResNet20<br>(CIFAR-10) | Simple | 0% | 0.1 | 5 | 200 | 0.002 | step |
| Bagging | Bagging | 0.1 | 5 | 200 | step |  |  |
| SSE | 0% | 0.2 | 5 | 200 | cyclic |  |  |
| Ep.b. | 0% | 0.1 | 1 | variable | step |  |  |
| ResNet110,<br>WRN28-10<br>(CIFAR-10,<br>CIFAR-100) | Simple | 0% | 0.1 | 5 | 300 | 0.0003 | linear |
| Bagging | Bagging | 5 | 300 | linear |  |  |  |
| SSE | 0% | 5 | 300 | cyclic |  |  |  |
| Ep.b. | 0% | 1 | variable | linear |  |  |  |
| ResNet50<br>(EyePACS) | Simple | 0% | $2.3\cdot 10^{-4}$ | 6 | 90 | $1.07\cdot 10^{-4}$ | step |
| Bagging | Bagging |  |  |  |  |  |  |

##### EyePACS.

The EyePACS dataset was included in the 2015 Kaggle diabetic retinopathy detection competition (Dugas et al., ). Diabetic retinopathy is the leading cause of blindness in the working-age population of the developed world and estimated to affect over 92 million people. The dataset contains high-resolution labeled RGB images of human retinas with varying degrees of diabetic retinopathy on a five-grade scale, from none (0), to mild (1), moderate (2), severe (3) and proliferating (4) development of the disease. It consists of 35126 training, 10906 validation and 42670 test images, each labeled by a medical expert. In order to binarize the labels, Band et al. () follow previous work and classify moderate or worse manifestation as sight-threatening (grades 2-4), and remaining grades 0-1 as non sight-threatening. The dataset with binary labels is unbalanced, such that 19.6% of the training and 19.2% of the test set have a positive label, which is why the cross-entropy objective is weighted by the inverse of the global class distribution. Furthermore, the images exhibit different kinds of noise (artifacts, focus, exposure) and are expected to show label noise due to misjudgement of the medical personnel (Dugas et al., ). The preprocessing of images by Band et al. () follows the winning entry of the original Kaggle challenge (Dugas et al., ). The images are first rescaled such that the retinas have a radius of 300 pixels, smoothed using local Gaussian blur with a kernel standard deviation of 100 pixels and clipped to 90% to remove boundary effects. Finally, they are resized to 512x512 pixels and stored. For our preprocessing, the implementation from Nado et al. () was used(^6^66[https://github.com/google/uncertainty-baselines/blob/main/uncertainty_baselines/datasets/diabetic_retinopathy_dataset_utils.py](https://github.com/google/uncertainty-baselines/blob/main/uncertainty_baselines/datasets/diabetic_retinopathy_dataset_utils.py)). Except the preprocessing, no data augmentation was used. With the optimal SGD hyperparameters from Band et al. (), the resulting models failed to converge in our case, which is why we reduced the learning rate and switched the optimizer to Adam. The batch size was kept at 32. For the learning rate schedule, a step-wise decrease was employed with a reduction by factor $0.2$ at 30 and 60 epochs.

### A.2 Test-time cross-validation

Ashukha et al. () describe the problem of requiring a validation set (in their case for scaling the logit outputs of neural networks with a temperature parameter), while the benchmark datasets only feature a training and test dataset (e.g. CIFAR). In that case, when splitting the training set into a training and validation subset, the performance may drop compared to methods that make use of the full training data because the reduced training data set is a worse description of the task. In contrast, when splitting off a validation set from the test data, one still obtains an unbiased estimate of the generalization error, but with higher variance due to smaller test dataset size. In order to reduce the variance, Ashukha et al. () propose test-time cross-validation: splitting the test dataset randomly and averaging the results of the generalization estimates to reduce variance. We follow this approach and divide the test dataset randomly in half. One half is used for the tandem loss bound optimization, while the other serves as a generalization estimate. Afterwards, the datasets’ roles are switched, and the risk estimates are averaged.
In our case, this setting does not result in a fair comparison. All methods should make use of the same amount of data, and algorithms that do not need a validation set could use the additional data for training, which can improve performance, in particular if data are scarce.
However, one can envision a scenario where the additional data are not available during training but only later when deploying the model
(e.g., local fine-tuning of centrally trained models).

### A.3 Cancellation of independent errors

Because this known result is often stated without proof, we provide an upper bound for the error probability of the majority vote for an ensemble of binary classifiers with independent errors.
We consider $M$ binary classifiers $h_{i},\dots,h_{M}$ mapping to $\{0,1\}$ and the majority vote classifier given by:

$$ $h_{\text{MV}}(x)=\begin{cases}1&\text{if }\sum_{i=1}^{M}h_{i}(x)\geq n/2\\ 0&\text{otherwise}\end{cases}$ (A.7) $$

We define the random variables
$A_{i}=\mathds{1}[h_{i}(x)\neq y]$ indicating a mistake by $h_{i}$ and
$S_{M}=\sum_{i=1}^{M}A_{i}$.
The probability $P(h_{\text{MV}}(x)\neq y)$ that the majority vote classifier makes a mistake
is the probability that $P(S_{M}>M/2)$ or alternatively $P(S_{M}\geq(M+1)/2)$.

We assume for all $i=1,\dots,M$ that the risk is bounded by a constant

$$ $P_{(x,y)\sim\ p}(h_{i}(x)\neq y)=\mathbb{E}\{A_{i}\}\leq p_{\max}<\frac{1}{2}$ (A.8) $$

and that the $A_{i}$ are independent.
Then

$$ $\mathbb{E}\bigg{\{}\sum_{i=1}^{M}A_{i}\bigg{\}}=\mathbb{E}\{S_{M}\}\leq Mp_{\max}$ (A.9) $$

and we have

$$ $\displaystyle P(S_{M}\geq(M+1)/2)$ $\displaystyle=P(S_{M}-\mathbb{E}\{S_{M}\}\geq(M+1)/2-\mathbb{E}\{S_{M}\})$ (A.10) $\displaystyle\leq P(S_{M}-\mathbb{E}\{S_{M}\}\geq(M+1)/2-Mp_{\max}).$ (A.11) $$

With
Hoeffding’s inequality and $\varepsilon=(M+1)/2-Mp_{\max}$ we get
the desired bound

$$ $\displaystyle P(S_{M}\geq(M+1)/2)\leq P\{S_{M}-\mathbb{E}\{S_{M}\}\geq\varepsilon\}$ $\displaystyle\leq\exp\left(-\frac{2\varepsilon^{2}}{M}\right)$ (A.12) $\displaystyle=\exp\left(-\frac{2\big{(}\frac{M+1}{2}-Mp_{\max}\big{)}^{2}}{M} \right).$ $$

### A.4 Further experimental results

Figure: Figure A.2: IMDB: Weight distribution per member for first order (FO, Lorenzen et al., ) and tandem weighting (TND) for the *Simple* setting considering only a single network per training process (left) and when adding checkpoints (right). The number of training runs was 40. Four intermediate checkpoints were added, giving a total of five weight configurations per training process.
Refer to caption: x5.png

Figure: Figure A.3: CIFAR-10 ResNet20: Weight distribution per member for first order (FO) and tandem weighting (TND) for the *Simple* setting (left) and *SSE* (right), both including all checkpoints (24 training processes, five checkpoints per process).
Refer to caption: x7.png

**Table A.4: Test accuracies for $\mathrm{AVG}_{\rho}$ and $\mathrm{MV}_{\rho}$ and PAC-Bayesian bounds for all experiments. The bounds were computed for the *all* setting, except for IMDB, where checkpoints were taken in a separate experiment due to early-stopping in *Simple*; $\text{MV}_{\text{u}}$ and $\mathrm{MV}_{\rho}$ refer majority voting with uniform and optimized $\rho$, respectively.**
| Model<br>(Dataset) | Experiment | Uniform<br>Bound | TND<br>Bound | $\hat{A}(\mathrm{AVG}_{\rho})$<br>last | $\hat{A}(\mathrm{MV}_{\rho})$<br>last | $\hat{A}(\mathrm{AVG}_{\rho})$<br>all | $\hat{A}(\mathrm{MV}_{\rho})$<br>all |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CNN LSTM<br>(IMDB) | Simple | 0.585 | 0.59 | 0.872 | 0.873 | - | - |
| Checkpointing | 0.576 | 0.594 | 0.873 | 0.872 | 0.873 | 0.874 |  |
| Bagging | 0.579 | 0.585 | 0.869 | 0.867 | - | - |  |
| SSE | 0.459 | 0.575 | 0.741 | 0.74 | 0.874 | 0.872 |  |
|  | Ep.b. (500) | 0.571 | 0.574 | 0.873 | 0.872 | - | - |
| ResNet20<br>(CIFAR-10) | Simple | 0.425 | 0.743 | 0.938 | 0.938 | 0.938 | 0.938 |
| Bagging | 0.387 | 0.691 | 0.929 | 0.928 | 0.929 | 0.928 |  |
| SSE | 0.593 | 0.612 | 0.894 | 0.893 | 0.908 | 0.909 |  |
| Ep.b. (1500) | 0.718 | 0.718 | 0.937 | 0.934 | - | - |  |
| ResNet110<br>(CIFAR-10) | Simple | 0.559 | 0.792 | 0.956 | 0.956 | 0.956 | 0.955 |
| Bagging | 0.555 | 0.763 | 0.949 | 0.948 | 0.949 | 0.948 |  |
| SSE | 0.785 | 0.802 | 0.955 | 0.954 | 0.954 | 0.953 |  |
| Ep.b. (1500) | 0.799 | 0.8 | 0.954 | 0.953 | - | - |  |
| Ep.b. (300) | 0.735 | 0.735 | 0.942 | 0.939 | - | - |  |
| Wide<br>ResNet28-10<br>(CIFAR-10) | Simple | 0.794 | 0.833 | 0.967 | 0.966 | 0.967 | 0.966 |
| Bagging | 0.764 | 0.808 | 0.959 | 0.958 | 0.959 | 0.958 |  |
| SSE | 0.843 | 0.848 | 0.963 | 0.964 | 0.964 | 0.964 |  |
| Ep.b. (1500) | 0.832 | 0.834 | 0.966 | 0.965 | - | - |  |
| Ep.b. (300) | 0.805 | 0.806 | 0.961 | 0.957 | - | - |  |
| ResNet110<br>(CIFAR-100) | Simple | 0.0 | 0.127 | 0.793 | 0.791 | 0.793 | 0.791 |
| Bagging | 0.0 | 0.0 | 0.771 | 0.769 | 0.771 | 0.768 |  |
| SSE | 0.083 | 0.125 | 0.79 | 0.787 | 0.794 | 0.793 |  |
| Ep.b. (1500) | 0.091 | 0.099 | 0.783 | 0.777 | - | - |  |
| Ep.b. (300) | 0.0 | 0.0 | 0.749 | 0.728 | - | - |  |
| Wide<br>ResNet28-10<br>(CIFAR-100) | Simple | 0.172 | 0.277 | 0.83 | 0.829 | 0.829 | 0.829 |
| Bagging | 0.041 | 0.173 | 0.812 | 0.811 | 0.812 | 0.811 |  |
| SSE | 0.276 | 0.289 | 0.818 | 0.818 | 0.826 | 0.827 |  |
| Ep.b. (1500) | 0.271 | 0.276 | 0.829 | 0.822 | - | - |  |
| Ep.b. (300) | 0.233 | 0.233 | 0.815 | 0.811 | - | - |  |
| ResNet50<br>(EyePACS) | Simple | 0.686 | 0.695 | 0.911 | 0.909 | 0.912 | 0.912 |
| Bagging | 0.613 | 0.638 | 0.894 | 0.893 | 0.896 | 0.895 |  |

Figure: Figure A.4: Mean test accuracy $\hat{A}$ $\pm\sigma$ over five ensembles for uniformly and PAC-Bayesian weighted deep ensembles ($\mathrm{AVG}_{u}$ and $\mathrm{AVG}_{\rho}$), using only the last or all training checkpoints, and best single model (SGD). References are Bayesian ensembles cSGHMC-ap (Wenzel et al., ), cSGLD (Ashukha et al., ), *MC-Dr*opout and *MC-Dr*opout *D*eep *E*nsemble (i.e., ensemble of Bayesian ensembles), as well as simple *D*eep *E*nsembles for EyePACS from Band et al. (). Numbers in brackets indicate the ensemble sizes.
Refer to caption: x9.png

Figure: Figure A.5: Snapshot ensemble (SSE) experiments: Mean ensemble accuracy (5 ensembles, $\pm\sigma$)
Refer to caption: x14.png

#### A.4.1 Bagging

In the bagging experiments, we used different training data for each ensemble member to increase diversity.
We drew bootstrap samples from the training data uniformly at random with replacement. The sample size was equal to the size of the training data.
The experiments illustrate the effect of decreasing the training data volume (e.g., to use the data for bound optimization).

The individual network performance suffered from omitting unique training data points, and
although the networks are assumed to be more diverse, the
resulting ensembles performed worse performance across all experiments. The results are visualized in Figure A.6, which shows that none of the ensembles, neither uniform nor optimized in their weighting, could match the Bayesian reference. This is in line with the literature (Lakshminarayanan et al., ; Lee et al., ), which finds the same decrease in performance for neural network ensembles.
PAC-Bayesian optimization is therefore best suited for large datasets with additional data that is not essential for training.
On a positive note, random initialization and stochastic training appears to be enough randomization for creating diverse ensembles.

Figure: Figure A.6: Bagging: Mean ensemble accuracy (5 runs, $\pm\sigma$ highlighted)
Refer to caption: x20.png

#### A.4.2 Training time comparison

It is not straight-forward to compare the computational resources required by the different approaches due to their different training and inference procedures.
The BMA based approaches require training of the neural network and sampling the posterior, where the latter can be time consuming.
In contrast, minimization of the PAC-Bayesian bound is highly efficient and its computational costs can be neglected.
Simple deep ensembles are embarrassingly parallel,
while methods that consider ensemble members from one process cannot be fully parallelized.

But even for simple ensembles, there can be a trade-off between number of networks and the number of training iterations spent on each network.
To explore this trade-off, we considered the worst case scenario for deep ensemble training and assume that there is no parallelization. Furthermore, we assumed that the sampling from the posterior does not take any time. That is, for methods based on BMA only the training epochs for optimizing the network are counted, not the sampling.
In this sequential setting, we asked, given an overall budget of $B$ training epochs, how does the performance of our deep ensembles depend on the number $M$ of networks when each network is trained for $\lfloor B/M\rfloor$ epochs.
For the experiments taken from Wenzel et al. (), we set
$B=1500$ for CIFAR-10 and $B=500$ for IMDB, corresponding to the number of training epochs used for the Bayes ensemble.
The results are shown in Figure A.7. Even in this biased comparison,
there were choices of $M$ for which the simple deep ensemble outperformed the Bayesian approach.

Figure: Figure A.7: Epoch budget in a sequential training setting: Mean ensemble accuracy (5 ensembles, $\pm\sigma$). The number of epochs is given by $\lfloor B/M\rfloor$, where $b=1500$.
Refer to caption: x28.png

### A.5 Licenses

Our work built on several, publicly available code repositories and datasets. The IMDB training code including the CNN LSTM model was based on the [Keras example](https://github.com/keras-team/keras/blob/1a3ee8441933fc007be6b2beb47af67998d50737/examples/imdb_cnn_lstm.py), similarly to the CIFAR training code from the CIFAR-10 [Keras example](https://github.com/keras-team/keras/blob/1a3ee8441933fc007be6b2beb47af67998d50737/examples/cifar10_resnet.py), including all ResNets. Both examples are distributed under the Apache 2.0 license. The Gaussian prior for the CNN LSTM on IMDB was based on [the code from Wenzel et al. (2020a)](https://github.com/google-research/google-research/tree/master/cold_posterior_bnn), also under Apache 2.0 license. The preprocessing routine of the EyePACS dataset was taken and adapted from [Google’s uncertainty-baselines repository](https://github.com/google/uncertainty-baselines/blob/main/uncertainty_baselines/datasets/diabetic_retinopathy_dataset_utils.py), again under Apache 2.0 license. The Wide ResNet (Zagoruyko and Komodakis, ) implementation was taken from a publicly available [GitHub repository](https://raw.githubusercontent.com/titu1994/Wide-Residual-Networks/master/wide_residual_network.py). Finally, the implementation of the tandem loss PAC-Bayesian bound optimization (Masegosa et al., ) was taken and extended from the [official implementation by the authors](https://github.com/StephanLorenzen/MajorityVoteBounds).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: None
soundness: 2
strengths: 1. Overall, this paper is well-written and the main hypotheses or claims are clearly presented.
2. Experimental results support these hypotheses.
summary: This paper rethinks three methods (i.e., uniform, Bayesian, and PAC-Bayesian) for deep ensembles. The authors argue that neither the sampling nor the weighting in a Bayes ensemble is well-suited for increasing generalization performance. In comparison, a weighted average of models obtained by minimizing a PAC-Bayesian generalization bound can improve the performance. Empirical results are conducted to support these hypotheses.
weaknesses: 1. Theoretical results are lacking to support these hypotheses or claims.
2. The authors have not proposed a new method or theory, resulting in little contribution.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: * Using the Bernstein-von Mises theorem, the authors require some assumptions mentioned in line 219. Can the the authors clarify if these assumptions hold for the deep networks class?
soundness: 3
strengths: - Studies deep ensembles which is an important and interesting problem
- Brings together different types of ensembles studied in the literature together in a unified framework.
- Includes empirical results supporting the theoretical claims
summary: The paper examines the effectiveness of different ensemble techniques—uniform, Bayesian, and PAC-Bayesian—specifically applied to deep neural networks. The authors contend that Bayesian ensembles, which typically average models based on posterior distributions, are suboptimal for maximizing generalization performance due to limitations in supporting the "cancellation of errors" effect. This issue arises because Bayesian ensemble members tend to converge to similar predictions rather than compensating for each other's errors. Instead, the authors propose a PAC-Bayesian approach that optimizes ensemble weights by minimizing a tandem loss over the validation set. This method accounts for model correlations and aims to increase robustness by improving generalization, even when ensemble members are generated from similar training processes. The empirical results on several benchmark datasets show that PAC-Bayesian ensembles with optimized weighting  using the validation set outperform Bayesian ensembles.
weaknesses: * Using the Bernstein-von Mises theorem theorem to show that a Bayes ensemble corresponds to a single model and hence cannot capture the cancellation of errors effect is interesting. But as mentioned by the paper, this has already been shown by the Masegosa paper already.
* Not clear what this paper is adding over the Masegosa paper
* This work claims that “Our results on four datasets show that complex Bayesian approximate inference methods can often be surpassed by more efficient simple deep ensembles.” but that has already been known empirically. This is not a novel observation.
* “• We show that the optimization of a PAC-Bayesian generalization bound using the tandem loss can improve the predictive performance of deep ensembles and provides non-vacuous generalization guarantees.”. Not clear what are they adding over the Masegosa paper.
* ”We demonstrate that the inclusion of intermediate checkpoints from the same training run in a neural network ensemble can increase its predictive performance, especially when the PAC-Bayesian bound optimization weights their contributions.” This has also been shown by snapshot ensembles as already mentioned by this paper. It is not clear what is the novel contribution in this regard also. 
*  The main contribution of the paper seems to be using the validation set to optimize the weights of the ensemble models which does not seem to be too novel.
* Overall, in the end the authors are able to match the performance of Pac-bayesian ensembles to that of deep ensembles after using the hold-out data. There does not seem to be any gains in using their approach.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: I will here outline my reasons for the listed weaknesses.

* With regards to lack of novelty, I here specifically outline why the listed main contributions in my opinion are not novel:
    1. As far as I am aware, the concept of reweighing ensemble members post model training is not new. If I have understood the proposed method correctly, then this is what is commonly referred to as _stacking_ in ensembles. See e.g. [1]. 
    2. "We discuss the conceptual differences... ". The discussion on why deep ensembles outperform single mode bayesian neural networks (referred to as Bayes ensembles) is commonly known, understood and researched within the Bayesian neural network community. See e.g. [2]. This also relates to the third listed main contribution which has previously been shown and is widely understood.
* In my opinion some key metrics are lacking. Currently the authors only report accuracy, but I think negative log-likelihood or calibration metrics (ECE) are equally important and of interest to the Bayesian community. Currently there are no metrics showcasing uncertainty quantification of the proposed methods which is of high interest in these types of works. Could the authors possible include some of these metrics?
* Could the authors discuss why they write "As outlined above, general ensemble methods and Bayesian neural networks have different motivations" in line 195-196? In my opinion they are motivated for the same reason: improved generalisation, robustness and uncertainty quantification. One simply is multimodal (ensembles) whilst the other (typically) only covers a single mode (BNNs).
* I strongly disagree with the sentence in lines 480-481 "Bayesian model averaging (BMA) is not meant to maximize predictive performance but to get a better idea about the maximum likelihood estimate of the parameters of a single model.". To me, the exact point of BMA is to improve model uncertainty quantification and predictive performance. Could the authors elaborate on this?

Minor comments/questions:
1. In line 133 the authors write "It is in general difficult to sample from $p(w|D)$, which can .. ", but I suspect the authors meant to say that it is difficult to approximate $p(w|D)$?
2. The setup in the results section is a little bit confusing to me when reading the paper. I would suggest that the authors perhaps first present experimental setup, and then discuss results (but this is an opinion).
3. The naming convention of "Bayes ensembles" is confusing to me. I have not seen this terminology before and to me it indicates an ensemble of Bayesian neural networks. Are the authors in fact just referring to Bayesian neural networks?

[1]: Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors, Yao et. al, 2018.

[2]: Bayesian Deep Learning and a Probabilistic Perspective of Generalization, Wilson and Izmailov, 2020.
soundness: 2
strengths: * The paper is well written.
* The authors have generally performed a thorough literature review.
* The method and general area of research is of high interest in the community (methods for aiding model generalisation, robustness and uncertainty quantification).
summary: In this paper, the authors present their PAC-Bayesian Deep Ensembles which is a method for reweighing members in the posterior predictive of deep ensembles. The method in essence utilises a PAC-Bayesian bound to compute posterior predictive model weights of a deep ensemble through gradient optimisation on a hold-out dataset. In contrast to [1] which also investigates PAC-Bayesian bounds for posterior predictive weighting, the authors stress the use of second order PAC-Bayesian analysis. The authors also discuss why deep ensembles often outperform single mode Bayesian Neural Networks (referred to as Bayes ensembles in the paper), and motivate PAC-Bayesian bounds from this. The authors evaluate the methods on a number of datasets and compare to other methods.


[1]: Diversity and Generalization in Neural Network Ensembles, Ortega et. al, 2021
weaknesses: I will outline the weaknesses here and elaborate on them in the questions.

* Lack of novelty. This is the main reason for the low score.
* Lacking experimental evaluation metrics. Given the lack of novelty, I would strongly argue that experimental evaluation should be much larger for this work to be accepted.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In eq. 3, is $\hat{L}_{\mathcal{D}}$ defined a the approximation of $L(h,h’)$ over $p(x,y)$ by sampling from $\mathcal{D}$? What is its expectation w.r.t. $\rho^2$? This shorthand notation should be defined.
2. The use of intermediate checkpoints has been extensively discussed in previous work such as snapshot ensembles. What are the contributions of this work regarding this point? Can the authors elaborate on their fourth contribution claimed in L96-99?
3. In L255-256, “we go from a distribution over the weights of a single neural network to a distribution over all parameters of the M networks and the ensemble weights." How is this implemented in practice? Can the authors give a clear formulation to demonstrate the difference?
4. Regarding the experiments, I have the following clarification questions:
    - In Figure 1: What are the differences between $\mathrm{AVG}_u$ and DE? From the definition in L117, $\mathrm{AVG}_u$ takes the uniform average, which is the vanilla approach of deep ensembles.
    - In Figure 1: Why are the uniform deep ensembles (DE) horizontal lines in Figure 2? It should also be affected by the number of ensemble members. 
    - Why are the results of $\mathrm{AVG}_u$ omitted from Table 2 compared with Table 1?
soundness: 3
strengths: 1. This work focuses on an interesting problem of understanding the mechanism of ensembles. 
2. It carries out extensive reviews of theoretical works regarding ensembles and provides a theoretical analysis of why Bayesian ensembles fail to select better weights compared with uniform ensembles. 
3. The optimization of weights through the second-order bound and tandem loss is an interesting approach.
summary: This work focuses on the ensemble problem with uniform, Bayesian, and PAC-Bayesian variants. It summarizes and concludes that Bayes ensembles are not superior to uniform ensembles as it does not consider the cancellation of errors effect.  As for PAC-Bayesian ensembles, the authors leverage the tandem loss as the bound of the weighted majority vote. Then they propose to optimize the weights $\rho$ through the tandem loss.
weaknesses: 1. The derivation of eq. 3 is missing and the authors do not provide much insight into this equation. I suggest that the presentation and motivation can benefit from elaborating more on how this upper bound should be understood and what the implications could be.
2. The presentation does not properly highlight the contribution of this work. Section 3 summarizes many existing works and approaches while leaving its own contribution unclear. This section should be reorganized for a better presentation.
3. The proposed methodology is not properly described. For instance, the objective function is only discussed in one sentence in L185, without any formulation or definitions.
4. The main conclusion of this work is that the Bayesian Ensemble is no better than the simple deep ensemble approach. However, simple, uniform deep ensembles are already the conventional approach. Thus the contributions in this conclusion are pretty limited. As for the PAC-Bayesian ensemble, from the experimental results, it appears that the differences are very marginal. What are the benefits of optimizing this framework compared with the simple and conventional approach like deep ensembles? 

[Minor]

1. $\hat{L}_{\mathcal{D}}$ in eq. 3 is not defined. The audience has to go through the reference to find out the definition. Also, the parenthesis and the bracket are reversed. 
2. The colors of curves of the “-last” and “-all” variants should be more distinguishable.