Deep Ensemble as a Gaussian Process Posterior
Authors: Zhijie Deng
Feng Zhou
Jianfei Chen
Guoqiang Wu
Jun Zhu
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep ensemble
Bayesian deep learning
Gaussian process
functional variational inference
uncertainty estimation

Abstract: Deep Ensemble (DE) is a flexible, feasible, and effective alternative to Bayesian neural networks (BNNs) for uncertainty estimation in deep learning. However, DE is broadly criticized for lacking a proper Bayesian justification. Some attempts try to fix this issue, while they are typically coupled with a regression likelihood or rely on restrictive assumptions. In this work, we propose to define a Gaussian process (GP) approximate posterior with the ensemble members, based on which we perform variational inference directly in the function space. We further develop a function-space posterior regularization mechanism to properly incorporate prior knowledge. We demonstrate the algorithmic benefits of variational inference in the GP family, and provide strategies to make the training feasible. As a result, our method consumes only marginally added training cost than the standard Deep Ensemble. Empirically, our approach achieves better uncertainty estimation than the existing Deep Ensemble and its variants across diverse scenarios.

Under review as a conference paper at ICLR 2022
DEEP ENSEMBLE AS A GAUSSIAN PROCESS APPROXIMATE POSTERIOR
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Ensemble is a flexible and effective alternative to Bayesian neural networks for uncertainty estimation in deep learning. However, Deep Ensemble is broadly criticized for lacking proper Bayesian justification. Some attempts try to fix this issue, while they are typically coupled with the regression likelihood or rely on restrictive assumptions. In this work, we propose to define Gaussian process (GP) approximate posterior with Deep Ensemble, based on which we perform variational inference directly in function space. We further develop function- -unetion-space posterior regularization mechanism to properly incorporate prior knowledge. We provide strategies make the training feasible, and demonstrate the algorithmic benefits of variational inference in the GP family. As result, our method consumes only marginally added training cost than the standard Deep Ensemble. Empirically, our approach achieves better uncertainty estimation than Deep Ensemble and its variants across diverse scenarios.
INTRODUCTION
Bayesian treatment of deep neural networks (DNNs) is promised to enjoy principled Bayesian uncertainty while unleashing the capacity DNNs, with Bayesian neural networks (BNNs) as popular
examples (MacKay 1992 Hinton & Van Campl 1993; Neal 1995 Graves] 201 1). Nevertheless,
despite the surge advance in BNNs (Louizos Welling 2016 Zhang et al. 2018), many of
existing BNNs still face challenges in accurate and scalable inference (Sun et al. 2019), and exhibit limitations in uncertainty estimation and -iittt-iiittibuttion robustness (Ovadia et al. 2019).
Alternatively. Deep Ensemble (DE) (Lakshminarayanan et al. 2017) trains multiple independent, candoml--initialin DNNs for ensemble, presenting higher flexibility and effectiveness than BNNs. However, it is hard interpret DE Bayesian approach which seeks for the Bayesian posterior of a certain model, and there is no guarantce that the uncertainty estimates given by DE are reliable. To chase Bayesian justification for DE, recent works like RMS (Lu & Van Roy, 2017 Osband et al. 2018 Pearce et al. 2020) and NTKGP (He et al.) 2020 refine DE to be a "sample-then-opiime" approach (Matthews al. 2017). Nonetheless, these works rely on the regression likelihood and often make strong assumptions like linearised and/or infinite-width models.
To build a Bayesian refinement of DE without reliance restrictive assumptions, we propose to use DE to define a Gaussian process (GP) approximate posterior, which is dubbed as DE-GP for short. The imposition of GP form on the approximate posterior empowered by the equivalence between GPs and BNNs (Khan et al. 2019 Neal] 1996 Lee et al. 2018 Garriga-Alom et al.) 2018 Matthews et al., 2018 Novak al. 2018 [ al., 2020), as well the associated computational
benefits when adopting a GP prior without loss of generality.
Concretely, we evaluate the empirical mean and covariance of the ensemble members and use them to specify DE-GP. This is inspired by the formula of neural network Gaussian process (NN-GP) (Novak al. 2018). Nevertheless, we clarify that we define parameteric and learnable GP kernel with finitely many and adaptive basis functions, in contrast to the NN-GP kernels built with infinitely many yet fixed basis functions. The flexibility of DE-GP assured by the expressiveness of DNN ensemble members- by tuning them it is enabled evolve
rich variety of kernels.
Given these setups, we perform function-space variational inference (VI) for learning. We maximize the functional evidence lower bound (fELBO) (Sun et al. 2019 Rudner et al.) (021)) to push DE-GP towards the true posterior over functions. We further present a posterior regularization scheme in function space to conveniently incorporate structural priors, and present concrete example how

---

Under review as a conference paper at ICLR 2022
Tax Ravution (1)
TTtw per Mucetee (0
NX-GP
DE-GP (our)
DR
RMS
Training speed
Figure Prediction on data from y = 2 /4 c,e (O0.0.1). The two rows correspond to multilayer perceptrons (MLPs) with 2 hidden layers of 64 units and MLPs with 3 hidden layers of 512 units, respectively. The first four columns correspond to four methods. We plot the training data (red dots), mean predictions (dark blue curves), and uncertainty (shaded regions). For NN-GP, MLPs. The other methods train 50 MLPs, with the training speed depicted the last column. As network size rises, DE-GP behaves consistently, while DE and RMS suffer from degeneracy, DE-GP is on par with the non-parameteric NN-GP in aspect of uncertainty quality, with only marginally added overheads upon DE.
performs analytic GP regression without training
to control function complexity. By VI, we are enabled to handle classification problems directly and exactly, without casting them into regression ones (He et al.] 2020). In practice, DE-GP can be easily implemented on top of standard DE, while only introducing minimally added cost.
The function-space Bayesian inference enables DE-GP to ameliorate the pathologies induced by the wee-paraameterization nature of DNNs (Sun et al. 2019 Wang al. 2019). Thus DE-GP can offer calibrated uncertainty estimates, especially when adopting large networks (see Fig. T. We empirically demonstrate that DE-GP outperforms strong baselines various regression datasets, and presents superior uncertainty estimates and out-of-distributis accuracy in challenging classification tasks. DE-GP also shows promise in solving contextual bandit problems, where the uncertainty plays a vital role in guiding exploration.
robustness without compromising
RELATED WORK
Bayesian neural networks (BNNs). Bayesian treatment of DNNs is an emerging topic in deep
learning yet with long history (Mackay 1992. Hinton & Van Camp 1993; Neal, [995 Graves
2011). BNNs can be learned by virtue variational inference (Blundell et al. 2015]] Hernández.
Lobato & Adams 2015; Louizos & Welling, 2016 Zhang et al. 2018 Khan et al. [018 Deng
et al. (020). Laplace approximation (Mackay 1992] Ritter et al. [018). Markov chain Monte
Carlo (Welling & Teh, 2011 Chen et al. 2014 Zhang al. 2019), particle-optimizaton based variational inference (Liu & Wang 2016), Monte Carlo dropout (Gal & Ghahramani] (016), and other methods (Maddox et al. 2019 Izmailov et al., 2020). To avoid the difficulties of posterior inference in weight space, some recent works advocate performing Bayesian reasoning in function 2019). In function space, BNNs of infinite or even finitely width equal to Gaussian processes (GPs) Neal 1996 Lee et al. 2018 Novak et al.) Khan et 2019). which provides valuable insights for this work.
space (Sun al. 2019 Rudner et al. 202T Wang et al.
2018
He al. 2020
Deep Ensemble (DE) [Lakshminarayanaal et al., 2017) is a qualified alternative to BNNs for uncer- tainty estimation (Ovadia al. 2019)., yet lacks proper Bayesian interpretation. Wilson & Izmailov (2020) conceptually interpreted DE as a method that approximates the Bayesian posterior predictive distribution, but is hard to judge whether the approximation is reliable or not in practice. Recently, a suite of works based on the notion of developed to make DE Bayesian. For example, RMS (Lu Van Roy, 2017] [Osband et al., 2018 Pearce et al., (2200) regularizes the ensemble members towards randomised priors to obtain posterior samples after training, while it typically assumes linear data likelihood which is impractical for deep models and classification tasks. He et al. (2020) proposed to add randomised function to each en- semble member to realise a function dunction-space Bayesian interpretation, but the method is asymptotically exact in the infinite width limit and is limited to regressions. By contrast, DE-GP works without restrictive assumptions. In parallel, D' Angelo & Fortuin (202T) proposed to add a repulsive term to DE to achieve Bayesian inference, which yet relies unscalable gradient estimators.
(Matthews et al. (017) have been
2

---

Under review as a conference paper at ICLR 2022
3 BACKGROUND
3.1 BAYESIAN NEURAL NETWORKS
Consider learning on dataset D YX {(z,, y.)eE. with æ; A' data and y, as C-dimensional targets. Let g(,, w) X' DC denote the function represented by DNN with weights w. Bayesian neural networks (BNNs) treat w as random variables and chase the posterior p(w by imposing prior p(w). Among the family, variational BNNs (VBNNs) are particularly appealing as they cast posterior inference as an optimization problem (Graves) 2011 Blundell et al. 2015; Louizos & Welling 2016), where variational distribution q is introduced approximate the true posterior by maximizing the evidence lower bound (EI.O)
Eq(w) (lgg))logp(7 p(D (D w)] DEL (l)
BNNs marginalize over the posteriors to obtain the posterior predictive for new data w":
plylæ" D) Ep(pp))p(yl w) Eggne) (1"" w) Σ plul==, w.),
where w. ~ q( ()) 8 ..... S. Such a procedure is expected to propagate the embedded model uncertainty into the prediction. However in practice, most of the existing BNN approaches face obsta- cles in precise posterior inference due to non- mon-triviil and convoluted posterior dependencies (Louizos) & Welling, 2016 Zhang al. 2018 Shi et al.) 2018, Sun al. 2019), and deliver unsatisfactory
uncertainty estimation and out-of duitt--distribution (OOD) robustness OOadiia et al. 2019).
3.2 DEEP ENSEMBLE
As workaround of BNNs, Deep Ensemble (DE) Lakshminarayanan et al.
2017) deploys set of
M DNNs {g(:, ()((;, w for ensemble. {un:} are independently trained to interpret the data from different angles, under maximum likelihood estimation (MLE) principle in standard DE, maximum
a posteriori (MAP) principle in regularized DE (rDE):
M
DD
M
log Dgg((D)es (D)(:(:)))))) w)
IIX
LIEE
> M
1
log p(D)w:: pow)(
(3)
Due to the randomness in network initialization and stochastic gradient descent (SGD), DE can effectively explore the non-convex, multimodal loss landscape of DNNs (Fort al., 2019 Wilson & Izmailov 2020), thus has shown promise in various uncertainty estimation tasks (Lakshminarayanan) ct al., 2017; Ovadia et al., 2019). However, due to the lack of a Bayesian justification, there is no guarantee that the uncertainty estimates given by DE are reliable.
3.3
RELATION OF THE BAYESIAN POSTERIORS OF DNNS TO GAUSSIAN PROCESSES
early shown that shallow BNNs converge to Gaussian processes (GPs) in the infinite width limit (Neal 1996). Recently, there is ongoing effort to extend the result to multiple layers (Lee et al. Matthews et al. 2018), convolutional architectures Garriga-Alonss et al. 1018.Nova et al. and beyond Yang 2019). Briefly, infinitely wide BNN amount to the NN-GP:
2018
2018
(4) where p ((w) is layerwise isotropic Gaussian and the DNN specifying g has infinite width. Notably, is is a compositional, matrix-valued kernel, with values in the space of C x C' matrices, and can be analytically estimated in some cases (Novak et al. 2018 2019). Due the i.i.d. weights, the kernel evaluation K xxx", "x*))))) x²) of some data x" is scaled identity matrix.
gP(O,, m(x, )) with x(x, x²) == Fw--p()) [g(x,, w)) un)
Based on the canonical results of GP regression, we know the posteriors of infinitely wide BNNs still GPs (in regression tasks), corresponding to the Bayesian training of the last readout layer (Arora al. 2019 Lee al. 2019 Despite golden standard for small regression problems, analytical pos- terior inference for NN-GP faces challenges when handling contemporary DNN architectures (Novak) al. 2018), large data (Shi et 2019) and classification problems.
Interestingly, finitely wide BNNs with Gaussian posteriors on weights can also be connected to GPs (Khan et al. 20099; Rudner al. (021). We then arrive at the hypothesis that the Bayesian posteriors of DNNs are closely related to GPs. So, we specify GP approximate posterior with DE.
3

---

Under review as a conference paper at ICLR 2022
4 METHODOLOGY
4.1 DEEP ENSEMBLE AS A GAUSSIAN PROCESS APPROXIMATE POSTERIOR
Basically, we utilize DE's ensemble members (referred to as basis functions hereinafter) to define GP approximate posterior as follows:
<(f: Wirry WM) (P(ff (o
my(x) (a) == är Σ/ g:(2)),
(5)
(x x²)
T7 E (gi(m) - (x)) (gx( my (=x))) AIc,
where refers to g(: ws). Namely, the empirical mean and central covariance of finitely many, adaptive basis functions are leveraged to specify DE-GP. (a, x') is a linear, matrix-valued kernel with small scaled identity matrix Ale appended to avoid singularity. The variations in (g æ') are confined to having up to M rank, echoing the recent investigations showing that low-rank approximate posteriors for DNNs conjoin effectiveness and efficiency (Maddox et al. (0199] Izmailov et al. 2020; Dusenberry et al. 2020). Note that the popular Nyström method (Williams & Seeger low-rank matrix approximate the original kernel matrix.
2001 also
Akin to the kernels Wilssnn et al. 2016), the DE-GP kernels are highly flexible, and may automati- cally discover the underlying structures of nigh-dimensional data without manual participation.
4.2 VARIATIONAL INFERENCE IN FUNCTION SPACE
We do function-space variational inference push DE-GP towards the posterior over functions.
Prior We have high freedom to determine which prior to use attributed to the variational inference paradigm. Without loss of generality, use the MC estimate of NN-GP (MC NN-GP) (Novak et al. 2018) as the prior due to its accessibility. Concretely, supposing a finitely wide DNN composed of
feature projector h(. w) X + CC and linear readout layer with weight variance ou and bias variance of the MC NN-GP is defined as p(f) = GP(f k(x, x')), where
5
k(x,, x') ((x, Lo, with k(x,
Σ Max w.)" h(('', (w)..
(6)
SČ
Ig refers to the indentity matrix of size C x C and w, are i.i.d. samples from the Gaussian prior on weights p(w). Similar setups can be found in some related works like (Wang al. 2019).
There may be a subtle difference between the MC NN-GP and the NN-GP, so what we are actually doing is not approximating the exact NN-GP posterior with DE-GP. Also of note that the NN-GP prior and the DE-GP posterior can be defined with various architectures.
IELBO Following Sun al. (2019 we maximize the functional ELBO (FELBO) to optimize DE-GP:
max Evcrs ((D))) - Dxu(g(f))ll(())) (()
Notably, there is KL divergence between GPs, which, on its own, is challenging cope with. Fortunately, as proved by (Sun et al. (2019), we can take the KL divergence between the marginal distributions of function evaluations as a substitute for it, giving rise to a more tractable objective:
max C = K
Σ
Bonnlogp(uulf(z.))) - DKL D )pp()))){))
(
[dt (MLDLEED
where X denotes a measurement set including all training inputs X, and fx is the concatenation of the vectorized outputs of f for X, i.e., px E C
4.3 POSTERIOR REGULARIZATION IN FUNCTION SPACE
Though Bayesian learning in function space enables direct imposition of prior knowledge function properties like smoothness and periodicity (Sun et al. 2019), sometimes the prior knowledge cannot be trivially designed as prior distribution over functions. For example, the functions
4

---

Under review as a conference paper at ICLR 2022
periodical only over limited input range, etc. In these cases, posterior regularization (Ganchev et al. 2010 Zhu et al. 2014) provides a principled workaround to impose structural constraints.
Briefly, we apply posterior regularization to functional variational inference by solving:
max s.t.: q()) E Q.
<())
Q - (()))) < 0} is valid set defined in terms of functional Ω which delivers some statistic of interest of a function
For tractable optimization, slack the constraint as penalty:
max L' = s())
L 3 (a)
(10)
where 3 is trade-off coefficient.
We next present an example on how to impose prior knowledge on function (hypothesis) complexity on the DE-GP approximate posterior using this paradigm For the sake of brevity, we assume binary classification scenario in the following discussion where E {-1,1} and f.g. : X + R. We measure the classification error one datum. We assume an underlying distribution μ = fu y) supported on A' {-1, } for generating the training data D, based on which we can define the true risk of a function (hypothesis) f: R(J)= == E(a.))) (f (a)) y). We set (()) == R()))(()))) in the seek of posterior over functions that can generalize well.
use 0-1 loss {(f()),)) y) = [f(=))
By definition, hypothesis sample M Ex gx))) + ((x) with ((x) ~ GP(O, (x, x²)). If sign( (x)) * y. it is impossible
q(f) GP(m, (xy x²)) be decomposed as
f(x) = 1 that sign( (z)) y, .... sign(gar (2)) y, and sign({(æ)) y all hold. In other words,
{(((z).)) = Σ (((()(()) m)1 ())
(11)
We can further re- re-parameterizee ((x) as ((x) = E (x((()) - m,( + VAeo, e: (0, 1), 0, .... M, which is essentially a vallevalued random function symmetric around Thus, for any (xz, μt, we have (g(r)((((),,)) = E ((((x)))))) v) = 1/2. As a result,
M
Bach R(f) S B
(((( (x), v) + =p [1/2] = [(()))] + 1/2.
(12)
Namely, the expected generalization error of the approximately posteriori functions be bounded from above by those of the DNN basis functions Recalling the theoretical and empirical results showing that DNNs' generalization error R(g.) can be decreased by controlling model capacity in
terms of norm-base based regularization minw; ||m,,12 (Neyshabur et al. 2015 2017 Bartlett et al. 2017
Jiang et al. 2019), we opt to maximize the following refined IELBO for learning DE-GP:
max
LDE-GP
Σ
B.cr)llog((ul(((..))) D (aaall(())lp(())) - 3 Σ 1| ă.
(13)
Wi... 10
Intuitively, the weight space of an werrpparamterrized DNN contains many redundant interpretations for the data, thus pushing the weights towards zero would not hurt data fitting but boost generalization. In practice, when processing with shallow architectures, / can be set as 0 to remove the constraint on model capacity; while when using deep architectures (e.g.. ResNet (He jet al., (016)), we can set /3 according to commonly used weight decay coefficient for better generalization. Comparisons in Fig, 2 and Table supporr this.
2iree
20000
20000
5008
52
7/
221
1.5
150
37a
202
Epech
Figure 2: Comparison on the L2 norm of weights. We traine models on CIFAR-10 with ResNet-20.. DE-GP (3 = 0) finds solutions with high complexity and poor test accuracy (see Table []. yet DE-GP (8 0.1) settles this.
4.4 THE ALGORITHM
We outline the training procedure of DE-GP in Algorithm [ and elaborate some details below.
Here assume one-dimensional outputs for Q for notation compactness.
The motivation that, as observed, the fELBO cannot cause proper regularization effects on weights w, for deep architectures, thus the learned DE-GP may suffer from high-complexity hasis functions.
A similar conclusion he drawn in multi-class classification scenario (see Appendix A
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Deep Ensemble as Gaussian process posterior (DE-GP).
1: Input: D: dataset; { M DNNs; k: prior kemnl;; a, 8: trade-off coefficients; v: distribution for
sampling extra measurement points; U: numher of MC samples for estimating the expected log-likelihond
2: while not converged do
3: D. = (X., Y.) C, X ~ v, X. = (X., X.)
4: gi = 9 E R X.I0 == I...,, M
5:
m = M Σ, 8i, kg
K..
Ei - m m )g. -m; )T
TT + AI AIx X,IC
k*.,, = , X.) <RIx, X.IC
L1 = Lim Σ (2,
log (fi (a:)), ) ft N(m [, kg x,)
8:
L2 DRL (M x,,,.,,1I1000 k xx),,))), = Σ; wm
9:
wi wi + 1/ 10 (Ci aL2 - BLs), I.....
4.4.1 MINI-BATCII TRAINING
In deep learning scenarios, DE-GP should proceed by mini-batch training. At each step. we manufac- ture stochastic measurement set with mini-batch D, (X,, Y, from the training data D and random samples X, from a continuous distribution v (e.g., a uniform distribution) supported on X. Then, adapt the objective defined in Eq. (13) to the following form:
CDE-GP
Σ (BLIAEED
EecDDogp(us (mx))) aDKula(p* (Pxkll((x)))(((x).))) )1 Σ
|| 1
(14)
where X, indicates the union of X, and X., When 1 and 3 = O, LDE-GP is lower bound of log p(D)) according to (Sun et al. 2019). Yet, given that we do not perform intensive hyper-parameter tuning for the NN-GP prior, it is reasonable to make tunable between the data fitting and the priori inductive bias.
to better trade off
4.4.2 AN EXACT AND EFFICIENT ESTIMATION OF THE MARGINAL KL DIVERGENCE
The marginal distributions of function evaluations are multivariate Gaussian by the definition of GP:
q(f = NN(xx. mm X. k. Xx),, p(P%x) = -N(P ko,k,, X.....).
(15)
with the kernel matrices k X. X. K. E xxX as the joints of pair-wise outcomes.
Therefore, the marginal KL divergence and its gradients can be estimated exactly without resorting to some approximations (Rudner et al.) 2021). What's more, as discussed in Section (33] there is simple structure in the prior kernel matrices, so we can write them in the form of Kronecker product:
k K, as (o2 x. + +o & Ic.
(16)
where &. E (g|xIx/(X.) corresponds to the evaluation of kernel k. Hence we can exploit the property of Kronecker product to inverse X in O(xx 1S) complexity.
Besides, as kxx k, low-rank, we can leverage the Woodbury matrix identity (Woodbury 1950 and matrix determinant lemma (Harville 1998) to efficiently compute the inverse and determinant of kx X, in O(XX,CMM time complexity given that usually M << X, C (e.g., 10 < 256C).
4.5 DISCUSSION
Diversity. The diversity among the ensemble members in function space is explicitly encouraged by the KL divergence between variational and the prior in fELBO. Nonetheless, the expected log- likelihood in fELBO enforces each ensemble member to yield the same, correct outcomes for the training data. Thereby, the diversity mainly exists in the regions far away from the training data (see Fig. and Fig. in Appendix B.2). Yet, the diversity in DE does not have a clear theoretical support.
Efficiency. Compared the overhead introduced by DNNs, the effort of estimating the KL diver- gence between Gaussians is negligible. The added cost DE-GP primarily arises from the extra measurement points and the evaluation of the prior kernels. In practice, we use small batch size

---

Under review as a conference paper at ICLR 2022
1.0
Nowrage Towe MRL
Boston
Power
Wine
Kindram
Protein
Boston
Power
Wine
Kinfinm
Protein
Figure 3: Comparison on average test NLL and RMSE on UCI regression problems. The lower the better.
for the extra measurement points. We build the prior NN-GP kernels with cheap architectures and perform MC estimation in parallel. Eventually, DE-GP is only marginally slower than DE in training.
Weight sharing. DE-GP does not care about how the basis functions are parameterized. so we can perform weight sharing among the basis functions, for example, using shared feature extractor and M independent MLP classifiers to construct M basis functions (Deng al.) 202T). With shared weights, DE-GP is still likely to be reliable because our learning principle induces diversity in function space regardless of the weights. Experiments in Section 5.3 validate this.
Limitations. Despite being Bayesian in principle, DE-GP is likely to be less flexible than DE and other variants because all the involved basis functions need be updated simultaneously.
EXPERIMENTS
We perform extensive evaluation prove that DE-GP yields better uncertainty estimates than the baselines, while preserving non-degraded predictive performance. The baselines include DE, rDE, NN-GP, RMS, etc. In all experiments, estimate the prior kernel with 10 MC samples and set the sampling distribution for extra measurement points v as the uniform distribution over the data region.
5.1 ILLUSTRATIVE REGRESSION
We build two regression problems with data from y= 22/4 (0.00.1) and sin 5x + c. ~ (O, 0.2). respectively. We consider two architectures: MLPs with 2 hidden layers of 64 units and MLPs with 3 hidden layers of 512 units, where ReLU activation is used. For NN-GP, we analytically estimate the GP kernel and perform GP regression without training MLPs. For DE-GP, DE, rDE, and RMS, we train 50 MLPs. For MC dropout, train MLP with the commonly used 0.2 dropout rate. We use the default values for a and B, i.c., a = and = O.
Fig. D and Appendix B.2 show the results on the two problems. We also provide the comparison on training speed in Fig. I As shown, DE-GP delivers calibrated uncertainty estimates across settings, on par with the con-paraettric Bayesian baseline NN-GP. Yet, the baselines like DE, rDE, and RMS suffer from degeneracy issue as the dimension of weights increases. Though NN-GP outperforms other methods, the involved analytical GP regression may have scalability and effectiveness issues when facing modern architectures (Novak et al. 2018) while DE-GP does not suffer from them.
5.2
UCI REGRESSION
We then assess DE-GP on 5 UCI real-valued regression problems. The used architecture is a MLP with 2 hidden layers of 256 units and ReLU activation. 10 networks are trained for DE, DE-GP and other variants. For DE-GP, set B = 0 and tune according to validation sets.
We perform cross validation with 5 splits. Fig. 3 shows the results. DE-GP surpasses or approaches the baselines across scenarios in aspects of both test negative log- likeeiioooood (NLL) and test root mean square error (RMSE). DE-GP even beats NN-GP, which is probably attributed to that the variational family specified by DE enjoys the beneficial inductive bias of practically sized SGD-trained DNNs, and DE-GP can flexibly trade off between the likelihood and the prior by tuning a.
5.3
IMAGE CLASSIFICATION ON FASHION-MNIST AND CIFAR-10
In the classification experiments, augment the data log- likeliood (i.e., the first term in Eq. 14 with a trainable temperature to tackle oversmoothing and avoid underconfidence. f
Fashion-MNIST. We use widened LeNet5 architecture with batch normalizations (BNs) (Toffe & Szegedy 2015) for the Fashion-MNIST dataset (Xiao et al. 2017). Considering the inefficiency of
"We can make the temperature Bayesian variable, but it unnecessary as our model is already Bayesian.
7

---

Under review as a conference paper at ICLR 2022
Test Error vs. Ensemble Size
Test NLL vs. Ensemble Size
FashicnMMNST +NIST Error vs, Uncertainty
26-07
Negatie iikelloood
0.25
Cheesifictionn Error
0.24
6:23
64-
64
6:11
6:00
0.1
0.2
rDF
DE-GP
DE
RMS
Ensemble
Ensemble size
Uncertainty Threshold 1
Figure 4: (Left): Test error varies w.r.t. ensemble size on Pashion-MNIST. (Middle): Test NLL varies w.r.t. ensemble size on Fashion-MNIST. MNIST. (Right): Test error versus uncertainty plots for methods trained on Fashion-MNIST and tested on both MasHion-MNIST and MNIST. Ensemble size is fixed as 10.
Table I: Test accuracy comparison on CIFAR- 10. Results are summarized over trials.
Architecture
DE-GP = 0.1J
DE-GP 0)
DE
rDE
RMS
ResNet-20
10.0%%
93.7110.06%
93.43 94.04.0..0%%
94.58 10.05%
93.63-10.07%
ResNet-55
95.55.10.04%
94.24100.7%%
95.5610.06%
94.45.10.03%
NN-GP, we mainly compare DE-GP to DE, rDE, and RMS. We a as well as the regularization coefficients for rDE and RMS all as 0,1 according to validation accuracy. For DE-GP, we use 3 = 0 given the limited capacity of the architecture. The in mi-distrbbution performance is averaged over 3 runs. Fig. (Lef)) and Fig. 4 (Middle) display how ensemble size impacts the test results. Surprisingly, the test error of DE-GP is even lower than the baselines.
Besides, to compare the quality of uncertainty estimates, we use the trained models to make prediction and quantify epistemic uncertainty for both the in-distribution test set and the out-of-distribution (OOD) MNIST test set. All predictions on OOD data are regarded as wrong. The epistemic uncertainty is estimated by the mutual information between the prediction and the variable function:
1 I), N H
( 3
> p(lL(()) - S Σ H (p(vlf((x))) s L...,,,,
8
(17)
where H indicates Shannon entropy, with f. w.) for DE, rDE, and RMS, and f. ~ q(;: w1, ... Wx) for DE-GP. We normalize the uncertainty estimates into [0, 1]. For each threshold [0, 1], we plot the average test eтror for data with < т uncertainty in Fig. (Right) It is prominent that under various uncertainty thresholds, DE-GP makes fewer mistakes than the baselines, implying that DE-GP succeeds to assign relatively higher uncertainty for the OOD data.
CIFAR-10. Next, we apply DE-GP to the real-world image classification task CIFAR-10 10 (Krizhevsky et al. 2009). We consider the popular ResNet architectures (He et al. (016) including ResNet-20 and ResNet-56. The ensemble size is fixed as 10. We split the data as training set, validation set, and test set of size 45000, 5000. 10000, respectively. We set 3 0.1, equivalent with the regularization coefficient on weight in rDE. We set a = 0.1 according to an ablation study in Appendix [3.] We use lite ResNet-20 20 architecture without BNs and residual connections to set up the prior NN-GP kernel for both the ResNet-20 and ResNet-56 based variational posteriors.
We present the in-distribution test accuracy in Table T and the error versus uncertainty plots on the combination of CIFAR-10 10 and SVHN test sets in Fig. 5 It is noteworthy that DE-GP is on par with the practically-used, competing rDFE in aspect of test accuracy. DE-GP (.3 0) shows unsatisfactory test accuracy, verifying the necessity of performing posterior regularization to penalize function complexity when using ultra-deep networks. The error versus uncertainty plots are similar those for Mashion-MNIST, substantiating the universality of DE-GP.
We further test the trained methods on CIFAR- 10 corruptions (Hendryeks Dietterich 2018), ), challenging OOD generalization/rrbusnees benchmark for deep models. As shown in Fig. 6 and Appendix B.3 DE-GP reveals smaller Expected Calibration Error (ECF) (Guo et al., 2017) ) and lower NI. at various levels of skew, reflecting its ability to make conservative predictions under corruptions and good OOD robustness.
More results for the deeper ResNet-110 10 architecture and the more challenging CIFAR-100 benchmark are provided in Appendix B.3 and Appendix B.4
8

---

Under review as a conference paper at ICLR 2022
2.6
0.6
2.-
0.4
DE-GP
DE-GP
02-
DE
0-2
rDE
rDE
61
0.0
1.0
Uncertainty Threshold
Uncertainty Threshold
Figure 5: Test error versus uncertainty plots for methods trained on CIFAR-10 and tested on both CIFAR- --0 and SVHN with ResNet-20 (Left) or ResNet-56 (Right) architecture.
0.5
DC-CP
DE-GP
12B
DZ
0.4
DE
12e
0.3
AMS
03
RMS
0.1
Skow Intensity
S kow Intensity
Figure 6: Expected Calibration Error CIFAR-10 corruptions for models trained with ResNet-20 (Left)
ResNet-56 (Right) architecture. We summarize the results across 19 types of skew each box.
Weight Sharing. We build a ResNet-20 with 10 clas- sification heads and shared feature extraction module to evaluate the methods under weight sharing. We set a larger value for for DE-GP induce higher magnitudes of functional diversity. The test accuracy (over 6 trials) and error versus uncertainty plots on CIFAR-I0 are illustrated in Fig. 7 We exclude RMS from the comparison as it assumes d. basis functions which may be incompatible with weight sharing. DE-GP benefits from the diversity in function space, hence performs better than DE and rDE, which purely hinge the diversity in weight space.
10.
1DE
Uncertainty Threshold
Figure 7: In-distribution test accuracy (Left) and error versus uncertainty plots on the com- bination CIFAR-10 and SVHN (Right) under weight sharing. (ResNet-20)
5.4 CONTEXTUAL BANDIT
Finally, we apply DE-GP to contextual bandit, an impor- tant decision-making task where the uncertainty helps guide exploration. Following (Osband et al. 2016). we use DE-GP to achieve efficient exploration inspired by Thomp- sampling. We reuse most of the settings for UCI regres- sion. We leverage the GenRL library to build two contex- tual bandit problems Covertype and Mushroom (Riquelme et al. 2018). The cumulative reward is depicted in Fig. g As desired, DE-GP offers better uncertainty estimates and
DE-GP
DE
IDE
RMS
Randim
Coverlype
Mushroom
2000
Cumulative Roward
2009 128 3807 Round
2005 J0DK 4b0
Figure 8: Cumulative reward varies w.r.t round Covertype (Left) and Mushroom (Right). Random corresponds the Uniform algorithm. Summarized over 5 trails.
hence beats the baselines by clear margins. The potential of DE-GP in more reinforcement learning and Bayesian optimization scenarios deserves future investigation.
CONCLUSION
In this work, we attempt to build a Bayesian refinement of Deep Ensemble by drawing inspiration from the connection between BNNs and GPs. We propose leverage the ensemble members to specify an adaptive GP approximate posterior, and perform variational inference directly in function space. We inherit some theoretical results from existing works on functional variational inference, and further develop a posterior regularization scheme to conveniently induce prior knowledge on function properties. We demonstrate how to penalize the function complexity based on this scheme and empirically show the necessity of doing so. The whole algorithm be implemented easily and efficiently. Extensive experiments validate the effectiveness our method. We hope this work may shed light the development of better Bayesian deep learning approaches.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This work proposes Bayesian refinement of Deep Ensemble. Its potential positive impacts in the society are evident: its ability to enable better uncertainty estimation while maintaining predictive performance is crucial in industry, c.g., automatic driving, disease analysis, and financial applications. In this scenarios, the uncertainty estimates could be used to reject uncertain predictions, and raise the requirement of inviting humans into the decision process, As fundamental research in machine learning, the negative consequences are not obvious. Though theory any technique can be misused, it is not likely to happen at the current stage.
REPRODUCIBILITY STATEMENT
It is easy to reproduce the experiments given the code in the supplementary material.
REFERENCES
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32: 8141-8150, 2019.
Peter Bartlett, Dylan J Foster, and Matus Telgarsky. spectrally-normalize margin bounds for neural networks. Advances in Neural Information Processing Systems, 014-6240-64445 2017.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference Muchine Learning, pp. 1613-1622, 2015.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference machine learning, pp. 1683-1691. PMLR, 2014.
Francesco D' D'Angelo and Vincent Fortuin. 1rXiv:2106.1162, 2021.
Repulsive deep ensembles are bayesian. arXiv preprint
Zhijie Deng, Hao Zhang, Xiao Yang, Yinpeng Dong, and Jun Zhu. Bayesadapter: Being bayesian, inexpensively and reliably, via bayesian fine-tuning. arXiv preprini arXiv:2010.01979, 2020.
Zhijie Deng, Xiao Yang. Shizhen Xu, Hang Su, and Jun Zhu. Libre: A practical bayesian approach to adversarial detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 972-982, June 2021.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snock, Katherine Heller, Balaji Lakshmi- narayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp. 2082-279222 PMLR, 2020.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv
preprint arXiv: 112.v:191..02757, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050-1059, 2016.
Kuzman Ganchev, Joao Graça, Jennifer Gillenwater, and Ben Taskar. Posterior regularization for structured latent variable models. The Journal of Machine Learning Research, 11 11:2001-2049, 2010.
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. In International Conference Learning Representations, 2018.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348-2356, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMI.R, 2017.
David A Harville. Matrix algebra from statistician's perspective, 1998.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. In Advances in Neural Information Processing Systems, volume 33, 2020.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1022 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common coruuttions and perturbations. In International Conference on Learning Representations, 2018.
José Miguel Hernández-Lofe Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. International Conference on Machine Learning, pp. 1861-1869, 2015.
Geoffrey Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the description length of the weights. In ACM Conference on Computaaionaa Learning Theory, 1993.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pp. 448-456. PMLR, 2015.
Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Subspace inference for bayesian deep learning. In Uncertainty in Artificial Intelligence, pp. 1169- 1179, 2020.
Yiding Jiang, Behnam Neyshabur, Hosscin Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2019.
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable Bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning. pp. 1016-26255. 2018.
Mohammad Emtiyaz E Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate inference turns deep networks into gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances Neural Information Processing Systems, pp. 6402 6413, 2017.
Jaehoon Lee. Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl- Dickstein. Deep neural networks as gaussian processes. International Conference on Learning Reprexenta- tionx, 2018.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32:8572-8583, 2019.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose Bayesian inference algorithm. In Advances in Neural Information Processing Systems, pp. 2378 2288222662222 2016.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning, pp. 1708- 1716, 2016.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In NIPS, 2017.
David JC MacKay, A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3): 448 472, 1992.
David John Cameron Mackay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems. pp. 13153-13164, 2019.
Alexander G de G Matthews, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. thenle-tttn---imiiee posterior sampling for bayesian linear models. In NeurlPS Workshop Advances in Approximate Bayesion Inference, 2017.
11

---

Under review as a conference paper at ICLR 2022
Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Chahramani. Gaussian process behaviour wide deep neural networks. International Conference on Learning Representations, 2018.
Radford M Neal. Bayesian Learning for Neural Networks, PhD thesis, University Toronto, 1995.
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pp. 29-53. Springer, 1996.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401. PMLR, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro. Exploring generalization in deep learning. Advances in Neural Information Processing Systems, 30:5947-59555 5956, 2017.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jachoon Lee, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2018.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander Alemi, Jascha Sohl-Dickstein, and Samuel Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representutions, 2019.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4400 4034,,4044440000 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neurul Information Processing Systems, 31, 2018
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Schastian Nowazin, Joshua Dillon, Balaji Iakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32:13991-14002. 2019.
Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approximately bayesian ensembling. In International conference on artificial intelligence and statistics, pp. 234-244. PMLR, 2020.
Carlos Riquelme, George Tucker, and Jasper Snock. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference Learning Representations, 2018.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018 Coof-enfeeeecee Track Proceedings. volume 6. International Conference on Representation Learning, 2018
Tim G. Rudner, Zonghao Chen, and Yarin Gal. Rethinking function-space variational inference in bayesian neural networks. In Third Symposium on Advances in Approximate Bayesian Inference, 2021. URL httpa
wet.iiiww..ttt.ee.....o..e..eeee //ppnrevviww.oo net http:ntt//oooeeeee worum Ktt
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. In International Conference on
Learning Representations, 2018
Jiaxin Shi, Mohammad Emtiyaz Khan, and Jun Zhu, Scalahle training of inference networks for gaussian-process models. In International Conference on Machine Learning, pp.
PMLR, 2019.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. In International Conference on Learning Representations, 2019.
Ziyu Wang, Tongzheng Ren, Jun Zhu, and Bo Zhang. Function space particle optimization for Bayesian neural networks. In International Conference Learning Representations, 2019.
Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-II), pp. 681-688, 2011.
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In Proceedings of the 14th annual conference neural information processing rystems, number CONF, pp. 682-688, 2001.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, volume 33, pp. 4697-4708, 2020.
12

---

Under review as a conference paper at ICLR 2022
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Max Woodbury. Inverting modified matrices. Statistical Research Group. 1950.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmiist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 2017.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural langent kernel derivation. arXiv preprint arXiv:1902.0476) 2019.
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International Conference Machine Learning, pp. 5847-5856, 2018.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference Learning Representations, 2019.
Jun Zhu, Ning Chen, and Fric P Xing. Bayesian inference with posterior regularization and applications to infinite latent syms. The Journal of Machine Learning Research, 15(1):1799-1885 2014.
13

---

Under review as a conference paper at ICLR 2022
A PENALIZE FUNCTION COMPLEXITY IN MULTI-CLASS CLASSIFICATION
In the multi-class classification scenario where y E (1,2..., C) and J. X - RRC, we use the loss (2:) 2) [(a)))ll f(m]lv') to measure prediction error where f(x) denotes j-th coordinate f()) The distinct difference between this scenario and the binary classification scenario is that in this setting, ((xr) is vector-valued function:
((x)
Σ
<ilg.(a) - m,( + vdeo.
(18)
VM
where 6o Ic), and 6 N(0 1), 1,1==,.,, M. We then make mild assumption simplify the analysis.
Assumption 1. For any (x,y)= E μ. the elements on the diagonal of (a, x) have the same value.
This assumption implies that for any j.3 {I, Cj.
- m. (x)))) +x= 1 M Σ -
M (((()))i)-m((((() A.
(19)
Le.,
M Σ mm,
M - m
(20)
Therefore, ((x) possesses the same variance across its output coordinates and becomes a random guess classifier. Based on this, we have ( y) E. Ee......s ((((2)))))))))))))(())))))))())) (C 1)/C. We can then derive similar conclusion to that in Section
The validity of Assumption 1. When the data dimension |X| is high and the number of training data n is finitely large, with zero probability the sampled data (æt, y) μ resides in the training set. Therefore, only the KL divergence term of the fELBO explicitly affects the predictive uncertainty at a. Because the NN-GP prior possesses diagonal structure, it is hence reasonable to make the above assumption.
B MORE OF EXPERIMENTS
We provide more experimental details and results this section. all experiments, the number of MC samples for estimating the expected log-likelihood (i.e., U in Line 7 of Algorithm is set as 256. Unless otherwise stated, we set the regularization constant as 0.05 times of the average eigenvalue of the central covariance matrices, and set the weight and bias variance for defining the NN-GP prior kernel at each layer as 2/ fan___t in and 0.01, where fan_in is the number of input features, as suggested by He initialization (He al. 2015).
B.1 DETAILED SETTINGS
Illustrative regression. For the problem on y p ( 0.1), we randomly sample 20 data points from 1.5, -0.6] U (0.6, 1.5). For the problem on y = sin 5x + C.C ~ (O, we use 6 data points {00.8 0.1,, 0.02, 0.2, 0.6. 0.8) following (Pearce et al. 2020). For optimizing the DNN basis functions, we use an Adam (Kingma & Bal 2015) optimizer with 0.003 learning rate. The optimization takes 300 iterations. We set hidden size as 64 for the architecture of the prior NN-GP kernel for speedup. The regularization constant À set as le 4 times of the average eigenvalue of the central covariance matrices.
UCI regression. We pre-process the UCI data by standard normalization. We set the variance for data noise and the weight variance for the prior kernel following (Pearce al. 2020). The batch size for stochastic training is 256. We use an Adam optimizer to optimize for 1000 epochs. The learning rate is initialized as 0.01 and decays by 0.99 every 5 epochs.
FaShion-MNIST classification. The used architecture is Conv(32, 3, D--BN-ReLU-MuxPooll Conv(64, 3,
Reet-innrr (10), where Conv(r, y, 2) represents
14

---

Under review as a conference paper at ICLR 2022
NN-GP
DE-GP (our)
DE
DE
RMS
MC dropout
Figure 9: Prediction on toy data from y = sin 5z + e, 6 ~ (0, 0.2). The two rows correspond to MLPs with 2 hidden layers of 64 units and MLPs with 3 hidden layers of 512 units, respectively. DE-GP provides calibrated uncertainty estimates and is consistently behaved mode complexity increases.
Nightbetogg Mo
Skew intensity
Skow intensity
Figure 10: Negative log- likeliieihood on CIFAR-10 10 corruptions for models trained with ResNet-20 (Left) or ResNet-56 (Right) architecture. We summarize the results across 19 types of skew in each box.
a 2D convolution with x output channels, kernel size y, and padding 2. The batch size for training data is 64. The batch size for extra measurement points is O. We use an SGD optimizer to optimize for 24 epochs. The learning rate initialized as 0.1 and follows a cosine decay schedule. Adam optimizer with 1e learning rate to optimize the temperature. We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty, because the involved computation is only the cheap softmax transformation the sampled function values.
use an
CIFAR-10 classification. We perform data augmentation including random horizontal flip and random crop. The batch size for training data is 128. The batch size for extra measurement points is 0. We use a SGD optimizer with 0.9 momentum to optimize for 200 epochs. The learning rate is initialized as 0.1 and decays by 0.1 100-th and 150-th epochs. We use an Adam optimizer with le - learning rate to optimize the temperature. We use 1000 MC samples to estimate the posterior predictive and the epistemic uncertainty. Suggested by (Ovadia al. 2019 He al., (2020), we train models on CIFAR- 10, and test them on the combination of CIFAR- 10 and SVHN test sets. This is standard benchmark for evaluating the uncertainty on OOD data.
Contextual bandit. We use MIPs with 2 hidden layers of 256 units. The batch size for training data 512. The batch size for extra measurement points is O. We update the model (i.e., the agent) for 100 1 and 8 0 without tuning. DE, rDE,
epochs with an Adam optimizer every 50 rounds. We set and RMS all randomly choose an ensemble member at per iteration, but our method randomly draws sample from the GP for decision. This is actually emulating Thompson Sampling and advocated by Bootstrapped DQN (Osband et al. 2016). "Random" baseline corresponds to the Uniform algorithm.
B.2 MORE RESULTS ON ILLUSTRATIVE REGRESSION
We depict the results on y sin 5z E, \O 0.2) problem in Fig which are consistently with those in Fig.
B.3 MORE RESULTS ON CIFAR-16 10 CLASSIFICATION
We plot the negative log-likelihood and test accuracy on 10 corruptions for models trained with ResNet-20 and ResNet-56 in Fig. 10 and Fig. [T] As shown, DE-GP outperforms the baselines in aspect of negative ligeiio,, but yields similar test accuracy the baselines. Recapping the results in Fig. 6 DE-GP indeed has improved OOD robustness, but may still face problems OOD generalization.
15

---

Under review as a conference paper at ICLR 2022
0.8
A8
RN5
RMS
0.2
02
Skow intensity
S Skow intansity
Figure 11: Test accuracy CIFAR- 10 corruptions for models trained with ResNet-20 (I.eft) ResNet-56 (Right) architecture. We summarize results across types of skew each box.
Eipected Calbrraton Enor
0.4
IDE RKS
DE FPGG
0.2
rDC
RMS
Skew intensity
Uncertainty Threshold
Figure 12: (Left): Expected Calibration Error on CIFAR-10 corruptions for models trained with ResNet- 110 architecture. We summarize the results across 19 types of skew in each box. (Right): Test emor versus uncertainty plots for methods trained on CIFAR-10 10 and tested on both CIFAR-10 and SVHN with ResNet-1 10 architecture. Ensemble size is fixed as 5 for these experiments.
We then conduct experiments with the deeper ResNet- architecture. Due to resource constraint, use 5 ensemble members. The other settings roughly the same as those for ResNet-56. The results are offered Fig. [2] which validate the effectiveness DE-GP for large networks.
B.4 RESULTS ON CIFAR-100
We further perform experiments on the more challenging CIFAR-100 benchmark. We present the in-distribution test accuracy of DE-GP as well as the baselines in Table We can see that DE-GP (3 0.1) is still par with rDE. We depict the of CIFAR-100 and SVHN test sets in Fig. I3 It is shown that the uncertainty estimates yielded by DE-GP for OOD data are more calibrated than the baselines. We further test the trained methods on CIFAR- 100 corruptions (Hendrycks & Dietterich 2), and present the comparisons in aspects of test accuracy and NLL in Fig. 14 is evident that DE-GP reveals lower NLL, than the baselines at various levels of skew.
versus uncertainty plots the combination
Table 2: Test accuracy comparison on CIFAR-
Architecture ResNet-20
DE DE-GP 0.1)
DE
rDE
RMS
76.59%
74.14%
76.81%
75.08%
ResNet-56
79.51% %
76.46%
79.21% %
76.77%
B.5 ABLATION STUDY ON a
We have conducted an ablation study (using ResNet-20 CIFAR-10). The results presented in Table We can see that DE-GP not sensitive to the value of a. We practice set a = 0.01 in the CIFAR experiments. We did not smaller it may result in colder posteriors and in turn worse uncertainty estimates.
Table 3: Ablation study on for DE-GP (8 = 0.1) (using ResNet-20 on CIFAR-10).
a
0.1
0.05
0.01
0.005
Accuracy
94.67-00.09%
94.66=0.07%
94.67+0.04%
94.83+0. 10%
B.6 ABLATION STUDY ON THE ARCHITECTURE OF PRIOR KERNEL.
We perform an ablation study on the architecture for defining the prior MC NN-GP kernel, with the results listed in Table Surprisingly, using the cheap ResNet-20 architecture results in DE-GP with
16

---

Under review as a conference paper at ICLR 2022
9.7
0.7
0.8-
0.6
DE-GP
DE-GP
0.3
DE
22
0.2
rDE
rDE
81
1.0
Uncertainty Threshold
Uncertainty Threshold
Figure 13: Test emor versus uncertainty plots for methods trained on CIFAR- 100 and tested on both CIFAR-I00 100 and SVHN with ResNet-20 (Left) or ResNet-56 (Right) architecture.
DE-GP
DE
IDE
/BE
AMS
Nogalle
Skew intensity
Skow intensity
0.8
0.6
0.4
DE-GP
DE-GP
IDE
AMS
RMS
Skew intensity
Skew intersity
Figure 14: First row: test NLL on CIFAR-100 -100 corruptions for models trained with ResNet-20 (Left) ResNet-56 (Right) architecture. Second row: accuracy CTFAR-100 corruptions for models trained with ResNet-20 (Left) or ResNet-56 (Right) architecture. summarize the results across types skew each box.
better test accuracy. We deduce this is because deeper prior architecture induces more complex, black-box correlation for the function, which may lead to over-regularization
Table 4: Ablation study on the architecture of the prior MC NN-GP kernel.
Prior kernel architecture
ResNet-20
ResNet-56
ResNet-110
DE-GP GP architecture
ResNet-56 (10 ensemble member)
95.50%
95.28%
ResNet- ensemble member)
95.54%
94.87%
17


Reviews:
[{"review_id": "n0ROPoYBOPZ", "reviewer": "Reviewer_5XQ4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed methods is elegant and provides Bayesian justification on top of DE.\n- Extensive experiment evaluation.\n- DE-GP shows good performance compared to DE.\n\nSome questions:\n- Algorithm: How to choose $\\lambda$? \n- Algorithm Line 7: what is U?\n- why is \"the diversity on the points far away from the training data is further fostered\"?\n- why not search over $\\beta$ since it seems to change for different experiments?\n- How sensitive is the performance with the changes of the hyperparameters $\\alpha, \\beta$? \n\nMinor comments:\n- Page 5: Require revising \"given the fact that a voting is incorrect when at least one of the individuals makes a mistake\"\n", "summary_of_the_paper": "The paper presents a novel perspective on deep ensembles (DE) that aims at providing a Bayesian justification to DE. The proposed approach builds a Gaussian process (GP) using the ensemble members and performs variational inference in the functional space. Moreover, the paper introduces a regularization method that works directly in the function space. The proposed algorithm DE-GP shows better results compared to DE an other well known Bayesian deep learning methods.\n", "summary_of_the_review": "Overall, I find the contributions presented in the paper novel and important to the Bayesian deep learning researchers to know about. Such contributions open the door for more works adapting Bayesian deep learning methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b6VdLHH-Kui", "reviewer": "Reviewer_zZzc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n-\tThe paper is very well-structured overall and a pleasure to read. I particularly enjoyed the sections describing the contributions, as well as the brief discussion in Section 3.5 that summarises some of the key properties of the proposed method.\n-\tThe dilemma on whether ensemble methods are just as effective as fully Bayesian methods has been the subject of several papers and discussion panels in recent years - the insights and contributions presented in this paper are thus timely and relevant, and should be of interest to the community. \n-\tThe experiments section is very thorough, and covers a vast set of problems ranging from synthetic examples for visual exposition of the model’s features, regression datasets, and classification problems. The paper also contains an additional experiment on a contextual bandit problem that I particularly appreciated, as it shows how the improved uncertainty estimates returned by this method can be beneficial when used in iterative decision making tasks. The experimental set-ups are well-explained in each case, and I appreciated that the results were all computed from scratch using implementations of existing methods (as opposed to simply copying in results from other papers that could possibly have been computed over slightly different dataset folds and experimental settings).\n\n### Weaknesses\n\n-\tAlthough I previously praised the paper for being well-organised, there are several instances where the writing can be improved. There are quite a few typos and grammatical errors scattered throughout the paper which could easily have been fixed with a proper read-through before submission. The introduction is particularly disappointing and I would recommend rewriting. There are also a few phrases which are ambiguous to a point and I would also reconsider. For example, the reference to ‘graceful BNNs’ in the conclusion is quite odd.\n-\tThe related work section appearing towards the end of the paper does a good job in exhaustively listing the various papers working on areas related to this topic, but offers very little insight into their actual contributions. Perhaps it may be more suitable to place this section earlier in the paper such that any contributions can be more clearly referenced against earlier work?\n-\tWhile interesting, the principal contributions of the paper are fairly incremental, and the authors themselves concede leveraging insights ‘inherited’ from other works as the foundation for some of the key contributions. Even so, this does not dent my overall opinion of the paper.\n-\tThe references need to be properly cleaned up - journal versions and conference proceedings should be cited instead of Arxiv editions of certain papers, while words such as Gaussian need to consistently appear as capitalized in paper titles.\n", "summary_of_the_paper": "The motivation for using fully Bayesian methods over ensemble methods has been a contentious topic in recent years - while ensemble methods are prized for their relative simplicity, they lack the theoretical framework that grounds fully Bayesian approaches. In this work, the authors propose an interpretation of Deep Ensemble models (DE) under a variational Bayesian framework. In particular, the authors demonstrate how VI can be carried out in the function space, and leverage posterior regularisation on functions to incorporate prior knowledge into the model architecture. The benefits of the proposed approach are verified via an extensive evaluation covering a variety of different problem settings, whereby it appears that the DE-GP consistently yields predictions having superior uncertainty calibration, and without compromising on predictive accuracy.", "summary_of_the_review": "The paper tackles an interesting subject in an intuitive manner, and the extensive experimental evaluation showcases the performance of the model in several different settings. On the downside however, some of the writing needs to be heavily improved, while the connections to related work should also be reconsidered to further highlight the original contributions of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "_fKPqTr31V", "reviewer": "Reviewer_uL2a", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Methodology\n\nThe method is, as far as I understand, the following: \n1. We define an NN-GP Gaussian process prior.\n2. We define a variational Gaussian process family, where the mean and covaraince are parameterized through the deep ensemble.\n3. We introduce a modified ELBO, where (a) we use a tunable weight for the KL, (b) we have an extra regularization term on the weights of the ensemble members and (3) the KL is calculated using the marginal distributions on the train set and additional randomly sampled inputs.\n\nThe authors claim that this is \"theoretically sound\" and \"principled\" \"Bayesian\" method. However, essentially, we are using a variational GP model. If we ignore the $||w_i||_2^2$ term in EQ 13, then the optimal solution of the ELBO optimization problem is recovering the NN-GP model. However, the method seems to work better than the NN-GP in practice, and, if the goal was to approximate NNGP, why wouldn't we just use NNGP? Is the idea that the specific family in which we approximate the posterior of the NN-GP the key component here, and what we want is the closest possible deep ensemble to the NNGP posterior? The motivation isn't fully clear to me.\n\n## Section 3.3\n\nI think section 3.3 is very confusing. Essesntially, the whole purpose of this section is to introduce a regularizer on the norm of the weights of the ensemble components, $||w_i||_2^2$. However the authors introduce it by first talking about constrained VI, than saying that our constraint will be that the accuracy on the train set should be good, then bounding the accuracy on train with the accuracy of ensemble components, and then saying that the accuracy of ensemble components is bounded by the norm of the weights. \n\nI have a few issues with this section:\n1. I think your constrained in the constrained VI should not depend on the data. Restricting your model class to only the functions that get good accuracy on your train set doesn't seem like a valid Bayesian procedure. Accuracy on train should be a part of your likelihood, not constraint on the model class.\n2. I don't think there are bounds on DNN performance that are based just on the $L_2$  norm of the weights.\n3. The regularization on the weights is a simple-enough idea, I think it would be much more clear if you just said that you are adding an extra weight decay term without drawing parallels to constrained VI and generalization bounds.\n\n## Making deep ensembles Bayesian\n\nOne of the goals that the authors repeat multiple times in the paper is \"make deep ensembles a Bayesian method\". In light of [this recent blog-post](https://cims.nyu.edu/~andrewgw/deepensembles/) I would like to ask the authors what exactly constitutes a Bayesian method, and why the proposed procedure is \"more Bayesian\" than the original deep ensembles?\n\nThe recent paper [1] also proposes a way of making deep ensembles \"more Bayesian\", so I think it should be discussed in the paper.\n\n## Experimental results\n\nI think the experimental results are the main strength of the paper. It seems like the method proposed by the authors performs well in a few practical settings, including CIFAR-10 and CIFAR-10-C. However, the performance is only marginally better than regularized deep ensembles on the in-distribution CIFAR-10 benchmarks.\n\nI think the experiments could still be made stronger by including other realistic datasets, perhaps CIFAR-100. \n\n## Questions\n\n- Do you train the ensemble from scratch using Algorithm 1, or do you pretrain the ensemble components first?\n\n\n[1] Repulsive Deep Ensembles are Bayesian\nFrancesco D'Angelo, Vincent Fortuin", "summary_of_the_paper": "The paper develops a deep ensemble-based Gaussian process model and a variational inference procedure to train it. As a result, the authors obtain a method for improved uncertainty and predictions with deep ensembles.", "summary_of_the_review": "The paper presents a method with promising empirical results. However, the authors frame the main contribution of the paper as making a principled a theoretically sound version of deep ensembles. I do not fully agree with this interpretation given the details of the method. So, I vote for a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RNknkdX384Q", "reviewer": "Reviewer_hBTZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "NOVELTY & SIGNIFICANCE\n\nFirst, I am not sure why this paper is titled deep ensemble as GP posterior while the proposed method instead uses deep ensemble to parameterize the GP posterior. I was under the impression that this paper is about exposing deep ensemble as the natural posterior mean of a GP setup because it aims to provide a Bayesian justification for DE via GP but this is unfortunately not the case -- one can in fact parameterize the GP with any deep network as its kernel but that is not the same as exposing them as inference results of a GP setup, thus providing a Bayesian justification. Given this, the first contribution claimed is kind of moot.\n\nFor the same reason as above, this work also appears incremental to me. It is pretty much a special case of GP with deep kernel but perhaps the point here is the DE is a fine choice for parameterization that can improve performance. As for the second contribution that involves the variational inference and regularization scheme, I have to unfortunately say the same thing that these are also too incremental as they are, as a matter of fact, vanilla inheritance from prior work with little to no adaptation: \n\nThe variational expression is obviously inherited from (Sun, 2019) as the authors mentioned and it is also known even before that by the prior work of Titsias in AISTATS-09 where an ELBO of GP is derived based on a set of inducing points. The regularization is simply a L2 penalty that was well-rationalized by prior work. \n\nGiven the above, I think the key contribution of this work is the empirical investigation of a potential parameterization for a GP with DE, which is fine but marginal given the reported results in Table 1 (i.e. the improvement over DE and rDE is pretty much marginal)\n\nSOUNDNESS\n\nAll derivations appear correct to me.\n\nEXPERIMENT\n\nAs I mentioned above, the results in Table 1 appear positive but are somewhat marginal (in one setting, rDE is still better). In fact, given the marginal differences, maybe it's worth running for more trials to have more stable estimates -- at this point, every model is within the error bar of almost every other model so it is not clear if the performance is really improved. \n\nOn another note, results in the contextual bandit look significant -- perhaps the calibrated uncertainty will show its effect best in an exploit-explore decision making scenario. But I am not sure how the authors set up the uncertainty variance for other non-Bayesian baseline in this experiment? Could the authors clarify this in the rebuttal?\n\nAs a more minor note, using bar plot in Fig 7 is kind of inflating the result differences. The difference between DE-GP and rDE is less than 0.5% and we do not know if this is significant without showing the deviation bar -- if it is similar to what was reported in Table 1 then it is not significant.", "summary_of_the_paper": "This paper presents a GP model with a deep kernel defined in terms of a (finite) deep ensemble (DE). A variational approximation and a regularization scheme were introduced to optimize the GP. The proposed methods were then demonstrated on several benchmark regression/classification datasets. \n\nHere, the claimed contributions are (1) a DE-parameterized GP posterior that provides a Bayesian justification for DE; and (b) a regularization scheme that improves the generalization of the trained GP.", "summary_of_the_review": "This is an empirical work that aims to demonstrate the uncertainty calibration of DE-GP and its improved performance over a set of DE variants. The results appear somewhat marginal where reported performance of DE-GP is within the error range of other models. More trial runs are necessary to obtain more reliable estimate. Also, I disagree with the claim this work exposes a Bayesian rationalization for DE since this is not what it is doing. Instead, a GP parameterized by DE is introduced and its performance is analyzed empirically.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HGxrtdG8fM", "reviewer": "Reviewer_Bycy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Contributions & Weaknesses\nThe paper shows empirically that DE-GP achieves somewhat better RMSE and accuracy compared to most baselines across a variety of synthetic, regression, and low dimensional vision tasks. However, these gains do not appear significant or well motivated. It is not clear what pain points of deep ensembles \"not being Bayesian'' are being  addressed and what concrete value the DE-GP method brings the other forms of approximate inference against which the method is compared. Like the other Bayesian methods of approximate inference, DEs have empirically shown closer approximations to the predictive posterior distribution, so it is unclear what is meant when seeking \"a more proper Bayesian exposition for DE\", and whether this distinction of DEs being \"non-Bayesian\" is necessary. If the goal is higher fidelity Bayesian inference, this need not be achieved by incorporating a GP. It makes sense to want to improve the generality of approximate models across various tasks, but significance could be better emphasized by specifying an argument against other state of the art methods, either theoretically or empirically, or by highlighting significant advantages / capabilities of the proposed method. \n\nTheoretically, there is little theoretical novelty in setting up the variational inference optimizing framework and defining the mean and covariance of basis functions, as these seem to be trivial adaptations of prior works. In terms of empirical results, there does not seem to be significant differences between NN-GP, DE-GP, and DE in most cases. Figures 1 and 9 are also not super informative in terms of emphasizing differences or giving insight into why or how DE-GP is competent at exploring and averaging uncertainty under a single mode, as noted that ensembles may not always select the best points from each mode. It is also difficult to justify better calibration purely from the visual.\n\n## Correctness & Clarity\nThe paper is well written overall. The proofs and math are clear and in line with prior works (e.g. Theorem 1 of [3] is applied for developing the objective in section 3.2). It could be better specified which parts of the pipeline are original work to emphasise contribution, as it currently only states which parts are borrowed. \n\nIn various parts of the paper, the method is motivated by the fact that DEs \"lack a proper Bayesian justification\". Could this be explained in more specificity? This is confusing and the paper does not clarify how and why certain methods are considered Bayesian or not. It is stated that the objective of the paper is to make DEs more Bayesian, but also noted that doing so via DE-GP may be at the cost of \"trad[ing]-off between theoretical soundness and flexibility in practical usage\". Flexibility should be clarified and justified for the DE-GP, otherwise, it cannot be used as evidence against employing DE in Bayesian inference problems.\n\n## Relation to prior work\nGenerally well-covered relevant works session, cites major works that would be good additional baselines. Methods are developed from past works on basis functions and GPs.\n\n## Additional feedback (comments, suggestions for improvement) \n- Section 3.1 claims that the chosen linear, matrix value kernel is comparable to recent low rank approximations to original kernel matrices. The authors should provide comparisons with methods like Batch Ensemble [1], Rank-1 BNNs [2], etc. to demonstrate this claim. \n- It would be nice to do an ablation to better motivate why VI is necessary and important to consider in Bayesian inference. There should be better characterizations of local approximations including Laplace approximation as well as global behaviours with MCMC, HMC or SGLD. Additionally, negative log likelihood appears better than others, but there is no comparison with NN-GP on CIFAR-10 for example. It would also be good to vary the ensemble size and compare it with NN-GP.\n- As uncertainty quantification is a major focus of most BNN / GP papers, additional methods and baselines should be included in the comparisons: mean field V I (motivating variational inference), AugMix (insights into the effects of data augmentations), ResNet architectures with and without batch normalization (analyze inductive biases contributed by architectures and generally how far off from most performant models).\n- The paper claims that DE-GP endows high complexity in function space. Can you show why this matters? For example, it would be more exciting and interesting to use a mixture of Gaussians as basis and show that you can / cannot achieve non-Gaussian and/or multimodal posteriors.\n- Further uncertainty and robustness analysis can be justified. It could be informative to measure the quality of robustness through additional experiments on tasks that encourage robustness to novel modalities (e.g. textures in vision tasks, corruptions / covariate shapes etc.)\n- Since the GP has a linear kernel, the method is identical to a Bayesian linear layer after a mean-only layernorm stage. Describing this model as an ensemble is misleading as well, since the components are jointly trained in a single objective as opposed to separately.\n\n## Questions for the authors\n1. More emphasis should be placed to detail the problem the paper is hoping to solve. Is it with regards to inference, prior selection, or something else more specifically?\n2. How do the computational costs of training DE-GPs compare to baseline and recent methods? \n3. How is expressivity of the posterior affected by the choice of prior process? Does a \"good\" choice of prior also allow for better tradeoff between cost and uncertainty estimation?\n4. Do you think a choice of non-stationary kernel as prior processes could induce better inductive biases and hence be conducive to stronger OOD detection?\n5. Have you tried experimenting with different ways of doing model aggregation in the ensemble framework?\n\n### References\n[1] https://arxiv.org/abs/2002.06715\n[2] https://arxiv.org/pdf/2005.07186.pdf\n[3] https://arxiv.org/pdf/1903.05779.pdf \n[4] https://arxiv.org/pdf/1711.00165.pdf \n", "summary_of_the_paper": "The authors propose using Deep Ensemble (DE) as basis functions to train a Gaussian Process (GP) with the primary motivation of making DEs more Bayesian. The Evidence Lower BOund (ELBO) being maximized is now optimized via variational inference (VI) over the functional space of bases (i.e. fELBO from [3] Sun et al.) defining the mean and covariance of a GP similar to NN-GP [4]. The paper evaluates the proposed method on standard UCI regression and small-scale MNIST / CIFAR10 image classification tasks in terms of accuracy, likelihood, and uncertainty estimation.", "summary_of_the_review": "Overall this paper demonstrates theoretical soundness and evaluates against standard benchmarks in the space of models with uncertainty estimation. However, it lacks some originality as well as comparisons with additional baselines and ablations to better separate its significance and reason for performing better / worse. There is also no clear indication of the problem being addressed and why being Bayesian matters.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use deep ensembles to parameterize a variational Gaussian process posterior, and uses an additional L2 penalty on parameters of the neural networks, and an (MC) NN-GP prior (although the prior is a free design choice). Reviewers appreciated aspects of the paper, finding there to be a minor improvement in uncertainty calibration over regularized deep ensembles, and nice results for the contextual bandit experiments. Ultimately, however, after a healthy and active exchange between reviewers and authors, four out of five reviewers are voting to reject the paper. There is a belief that the paper can be substantially improved from its current form, by carefully accommodating reviewer feedback, but it is not currently at a stage ready for publication.

There were common themes in the concerns expressed by several reviewers. Many reviewers found the technical contributions incremental. Parametrizing a GP using deep ensembles, or adding L2 regularization, is not itself a major technical contribution, and the variational framework leans heavily on Sun et. al (2019) and work that came before it from Titsias (2009). Similarly, the theoretical contributions were found to be incremental. 

These concerns about the technical contributions may have been counterbalanced if the experimental results had been outstanding or the framing of the paper perceived to be very clear and well justified. However, the experimental results had a mixed reception, with several reviewers noting accuracy was not in fact much better than the simpler regularized deep ensembles, despite some improvements in uncertainty calibration. One reviewer liked the bandit experiments, but wished there was a deeper exploration of this application domain. The current experimental results do not seem to warrant the relative complexity of the approach over simple regularized deep ensembles. 

Additionally, several reviewers found the framing and presentation of the paper needing significant work. The introduction of the L2 regularization terms, for example, was perceived to be overly complex, involving several steps that were not well-motivated.  

Several reviewers also found the motivation about making deep ensembles Bayesian unconvincing. A procedure being sensitive to initialization, or unreliable in certain settings, does not mean it does not perform approximate Bayesian inference. For example, variational methods and Laplace approximations can depend on initialization, and could get stuck in poor local optima. Quoting papers referring to deep ensembles as non-Bayesian is also not an argument in itself. The blog post linked by a reviewer is clearly pushing back against these claims, and does address points raised in the discussion, such as unimodal approximations and theoretical guarantees. As reviewers have also noted, several papers have now provided plain deep ensembles with a Bayesian justification, and these papers should be acknowledged. It could be reasonable to argue that your paper makes deep ensembles _more_ Bayesian, and you could potentially try to measure this claim in a concrete way. Or you could simply argue that your approach helps reduce sensitivity to initialization, and represents solutions with lower posterior density, which can be helpful practical contributions and don't need to be tied to claims about the method being Bayesian.

Please thoughtfully reflect on the reviewer comments in updated versions of the paper. The reviewers put a lot of effort into providing feedback and engaging during the rebuttal period. While the paper has some nice features, there is significant room for improvement on several fronts: technical innovation, experimental investigation, and framing. Improving the framing will help, but working further to also address other concerns will likely be needed to sway reviewers.
Decision: Reject