Mixture Distributions for Scalable Bayesian Inference
Authors: Pranav Poduval
Hrushikesh Loya
Rajat Patel
Sumit Jain
Venue:  (2020)
Keywords: uncertainty estimation
Deep Ensembles
Adverserial Robustness

Abstract: Bayesian Neural Networks (BNNs) provides a mathematically grounded framework to quantify uncertainty. However BNNs are computationally inefficient,
thus are generally not employed on complicated machine learning tasks. Deep
Ensembles were introduced as a Bootstrap inspired frequentist approach to the
community, as an alternative to BNN’s. Ensembles of deterministic and stochastic networks are a good uncertainty estimator in various applications (Although,
they are criticized for not being Bayesian). We show Ensembles of deterministic
and stochastic Neural Networks can indeed be cast as an approximate Bayesian
inference. Deep Ensembles have another weakness of having high space complexity, we provide an alternative to it by modifying the original Bayes by Backprop (BBB) algorithm to learn more general concrete mixture distributions over
weights. We show our methods and its variants can give better uncertainty estimates at a significantly lower parametric overhead than Deep Ensembles. We
validate our hypothesis through experiments like non-linear regression, predictive
uncertainty estimation, detecting adversarial images and exploration-exploitation
trade-off in reinforcement learning.

Under review as a conference paper at ICLR 2020
MIXTURE DISTRIBUTIONS FOR SCALABLE BAYESIAN INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian Neural Networks (BNNs) provides a mathematically grounded frame- work to quantify uncertainty. However BNNs are computationally inefficient, thus are generally not employed on complicated machine learning tasks. Deep Ensembles were introduced as Bootstrap inspired frequentist approach the community, an alternative tic networks are a good uncertainty estimator in various applications (Although, they are criticized for not being Bayesian). We show Ensembles of deterministic and stochastic Neural Networks can indeed be cast as an approximate Bayesian inference. Deep Ensembles have another weakness of having high space com- plexity. we provide an alternative to by modifying the original Bayes by Back- prop (BBB) algorithm to learn more general concrete mixture distributions over weights. We show our methods and its variants can give better uncertainty es- timates significantly lower parametric overhead than Deep Ensembles. We validate our hypothesis through experiments like non-linear regression, predictive uncertainty estimation, detecting adversarial images and exploration-exploittiin trade-off in reinforcement learning.
BNN's. Ensembles of deterministic and stochas-
INTRODUCTION
Neural Networks models have been applied diverse fields from weather forecasting, to au- tonomous vehicle driving, to online advertisement and many more (Goodfellow et al. (2016)). How- vanilla ceed-forward NNs susceptible to the problem of ever-fitting In addition to this, NNs trained using Maximum Likelihood Estimation (MLE), or Maximum A Posteriori (MAP) can- not provide an estimate the uncertainty in predicted value and tend to produce overconfident results on out-of-distribution test data. Thus we aim to build deep learning frameworks which are more robust, secure and reliable- specifically, in the face of uncertainty we would like the model to be able to say "I Don't 't Know!"
A principled approach build such models is through Bayesian inference, where instead of a point estimate for the network parameters, we infer the posterior distribution the weights given the data. The predictive distribution of unseen data (x",y") ) is given by ((v"(=") p(y"(2", W D))WW where D) is the true posterior computed from Bayes rule. Un- fortunately, since modern NNs have an exponentially large number of parameters and do not lend themselves to being integrated into mathematical equations, exact inference remains intractable.
Several methods for approximate Bayesian inference have been investigated. Until now, Hamil- tonian Monte Carlo (HMC) (Neal (2012)) has been considered the gold standard for approximate Bayesian inference. However, HMC requires explicit storage of samples from the posterior and is typically not scalable to larger networks and datasets. Thus methods based on Variational Inference (VI) have gained popularity recently for the task of approximate inference. VI relies on using a surrogate posterior (ga(W)) )) as an approximation for the true posterior. One of the first application of VI in NNs by Hinton & Van Camp (1993) but the optimization remained intractable for most Bayesian Neural Networks (BNNs). Graves (2011) revisited similar ideas and proposed a simple but biased estimator for performing VI with fuly factorized posterior (mean field assumption).
algorithm (BBB) also employed VI using an unbiased estima- tor for Variational loss, fully factorized Gaussian posterior and a non-Gaussian prior [Blun-] dell et al. (2015). Interestingly, Gal & Ghahramani (2016) showed links between Bernoulli al. (2014), Hinton et al. (2012) and approximate inference in Deep Gaussian Process (Damianuu & Lawrence (2013)), thus allowing for the extraction of uncertainty estimates of a NN In a principled way. Lourzos & Welling (2016) arrived at the same conclusion through structured posterior approximations via matrix variate Gaussian Posterior (Gupta & Nagar (2018) and local reparameterization trick (Kingma et al. (015)).
Recently, Koyee-by-Bacckpee

---

Under review as a conference paper at ICLR 2020
Bayesian methods are far more computationally expensive, compared to standard NNs, making Deep Ensembles a very attractive alternative as they scale well to large datasets and models. The intuition behind Deep Ensembles is straightforward, NNs initialized and trained independently can be expected to output similar predictions on training data, while disagreeing on points away from training set. Despite their empirical success in uncertainty estimation and adversarial robustness [aakshmiarayyaaan et al. (2017). Strruss al. ). Deep Ensembles have lacked theoretic sup- port from the Bayesian perspective. Thus, it IS interesting see how ensembles of deterministic and stochastic NNs can be placed into the VI framework. Another drawback of Deep Ensembles is their heavy parametric overhead. We modify the BBB algorithm to learn mixture distributions rather than the unimodal Gaussian distribution, which reduces this burden. We further show that this way, Deep Ensembles be linked with the Bayesian framework. In essence, using our methods, are able to achieve accuracy and uncertainty estimates (with lower parametric cost) which are never worse than Deep Ensembles, BBB, Concrete Dropout
Contributions:
We design an efficient and simple way of sampling from mixture distributions which incorporates the "local reparameterization trick "ork""(Knng "(Kingma et al. (2015) to obtain low variance gra- dients for tuning of parameters in mixture distributions. This also of independent interest for designing more complex posteriors using any mixture distributions besides the ones used in this pa- per. We also fit ensembles of deterministic and Bayesian NNs into VI framework with no changes made to the original algorithm. We improve upon the uncertainty estimates of Deep Ensembles by using mixture distribution with relatively less number of parameters.
Distribution of the paper- In section we show how the original BBB algorithm can be modified for generalized mixture distributions while remaining back-prop compatible. For simplicity, we stick to a mixture of Gaussian's s. In section 3, we provide links between Ensembles and VI with mixture distributions. Section 4 contains experiments that demonstrate that our methods generate superior uncertainty estimates compared to unimodal distributions Deep Ensembles.
2 CONCRETE MIXTURE OF GAUSSIAN
A feed-forward network trained with gradient descent will arrive at point estimates. However, in the case of BNNs, the weights not point estimates but probability distribution. Let us revisit the problem of finding a functional relationship = fw (z) between input x and output y. given a labelled training dataset D = ((11,)) ), (2r2, ..).... find a distribution over the parameters p(W D), given the input data. Given this posterior, we can predict test output y" given a new test input x" by marginalizing the likelihood over the space of parameters W.
..... } from a Bayesian standpoint. Our task is to
However, even for modest sized NNs, the number of uncertain parameters prohibit this calculation analytically. We require approximate inference methods in such cases. We define an approximating variational distribution qe (W) with parameters 0. We would minimize the Kullback-Leible (KL) Divergence with respect to the parameters @ to maximize the similarity between proposed posterior and the true posterior. Note that the problem of minimizing KI. Divergence can be reformulated to minimizing of another term usually called the Variational free energy in literature Friston et al.
2000d,,Blunddell [Blundell al. (2015). Evaluating the objective of maximizing this is computationally expen- sive for large datasets therefore. Thus we split the entire data into M mini batches D D2 ..... DM The unbiased Monte Carlo (MC) approximation the minibatch cost it given as:
(2007).
W (o LVI-Mo(0) = Σ log(p(u.l (x;))) + M
iD
In order to use stochastic backpropogation, we need the MC sample W to be reparameterized as g(0, e). We shall see how this is possible for Mixture of Gaussians (MoG) next.
2.1 SAMPLING FROM CONCRETE MIXTURE OF GAUSSIAN
Blundell et al. (2015) proposed the Brres-by-Backkppn algorithm for tuning of weight posterior's parameters. For simplicity, the posterior was assumed to be fully factorized Gaussian. The as- sumption of fully factorized posterior is also known as the Mean Field assumption in the literature.
qo (W)+ II N(w o?) 0 = (M, (;=(...)
N Total no. of weights (2)
Louizos & Welling (016)) used a matrix variate Gaussian posterior Gupta & Nagar 20188) to cap- ture correlations between weights of same layer. Both of the above assumption do not account for multiple modes in true posterior. A natural choice for posterior that capture multiple modes is
2

---

Under review as a conference paper at ICLR 2020
mixture distribution, where each mixture component would correspond to separate mode of the true posterior.
We show method sample from for mixture Gaussian (MoG) for independent weights and mix- ture matrix variate Gaussian (MoMG) for correlated weights, while still being able to use the local reparameterization trick. This method can be extended for any mixture of distributions in general.
To sample weight from a mixture of K Gaussians i.e.
go(w) Σラ= Ej=1,, NyN(;; P5, of),
Σj= Pj = 1, we assume a latent random variable which governs the component of the mixture that generates the value of w. We assume the latent random variable to be uniform random variable i.e. Uniform[0.1]. The latent random variable has Pj probability of being in the range [p--1.P; + P)-1), similarly it has Ps+1 probability of being in between [pj + P)-1, Pj + P--1 + sample from the j'th distribution
PJ-1) and so on, thus if falls in the range ( + Py-1) N(w; My of). For example when sampling from mixture of three Gaussian, we will have to first sample two random variables 2p1 and tps which can be seen as correlated Bernuulli random variables.
u ~ Uniform 00, 1], Epp = Lu<pi 2ptto = Lucpitp2 and e~ N(0,1) Ep, (mi o1c) (1 2p1 (2p 02c) (1 =p1 age)
Unfortunately, in the above case where we sample w from a multimodal distribution, local reparametrization trick cannot be applied. This is because the Bernoulli random variables 2p1 and Ep--ps can't be parameterized relaxation, the Concrete distribution Gal et 2017 Maddison M et al. (2016). This distribution can be viewed as softmax relaxation of the max function used in Gumbel-max trick Gal et al. (2017). represent the discrete random variable žp1 as Epp g(0 e). Thus we replace Zy
the form of g0,,c) c). So, we replace them with their continuous
This allows and +p in equation with:
Zp, 2 2p. sigmoid( sigmoid(". (log(p):) log(1 p1) +lg(uu) - log(1 - u))
tp1 20 2p1tp2 = sigmoid( 1 (log(p1 + p2) log(1 - P1 - P2) + log(u) - -log(1 - -u)) (4)
where, is the temperature term and is usually kept as 0.1 or 0.01 (<<1 to guarantee most of the mass of the pdf is around 0 and 1). So, we have now successfully reparameterized w g(0, e) where 0 (f1.011 "tt.:/4....2/..0033.. PP))).). To prevent parametric explosion, we assume p1, P2 are same for the entire layer. This is similar to assuming that Dropout parameter is the same for entire layer. Thus now, with mean field assumption we can define the posterior of the weight matrix W from mixture of K Gaussians as:
go (W)
II Pij Noys Piy, oig,
Σ Pij =1, 0 = -(Pij, Mijs (k..,,
To model the correlations among the weights it is possible for the treat the entire weight matrix W of a layer, as a MoMG Gupta Nagar (0188 i.c. p(W) = Σ, P.MNNMMM, U,, Vi), where
p(W) = MNNMM V.) exp( LV (W'
VWW T, (W" -M,))
(6)
(2m))r//1V/////10//>
Mj is ar x e matrix and is the mean of the distribution. U; is a X r matrix that provides covariance of the rows and Vi is с matrix that governs the covariance of the columns of the matrix. It has been shown by Gupta et al. in Gupta & Nagar (018) that matrix variate distributions are essentially multivariate Gaussian.
p(vec(W")) = NNvee(M.), V, & U.), @ is the Kronecker product
Thus, now instead of treating each weight independently we are now able to sample directly from its joint distribution which is assumed to be a MoMG. For sampling from a mixture of 2 matrix variate
3

---

Under review as a conference paper at ICLR 2020
(U 2,) (1 Už EV, where Ei,; ~ N(), We We replace
Gaussian, W = 2 ( 2p with its continuous relaxed version E.. thus making possible to use the local reparameterizaatio trick. We assume U; = diag(o?;) and Vi diag(o ) i.e. independent rows and columns in this paper. This allows us to reduce the number of parameters as compared to MoG, while still modelling correlations between weights. Now that we are able to sample from MoG and MoMG need to be able to approximate Dxz. (go (g)(W)))))))) where we assume Gaussian prior i.e. p(W) N(0, 1)) We use Monte Carlo approximations similar to the one used in Blundell et al. 22155) for optimising the DEL (go () (W)).)(W))
For all following sections. we interchangeably use the name Concrete MoG (CMoG) and Concrete MoMG (CMoMG) for MoG and MoMG respectively.
DEEP ENSEMBLE AS BAYESIAN INFERENCE
Deep Ensemble is a simple non-Bayesian framework which is able to predict uncertainty on out-of- distribution samples. Deep Ensemble algorithm involves training M randomly initialized networks on independently sampled batches. Owing to its simplicity and scalability, Deep Ensemble is widely used for uncertainty estimation. This departure from Bayesian methodology capture uncertainty of concern since the Bayesian framework is a principled and widely accepted approach to capture uncertainty. Thus, it is indeed surprising to see that no simplification made to the original Deep Ensemble algorithm described in Lakshminarayanan et al. (2017), we can show equivalence with the VI framework.
In order to show the equivalence with VI, we assume the approximate posterior is a mixture of points or Gaussian (with a - 0) in high dimensional probability space. For example in case the network has only two layers with parameters W1 and W2 respectively, the joint distribution is ((W, W2 = pN(M,, ( I² + (1 - NM2, (DN(M,,²2I(C I² + 0), where 0 is the set of all variational parameters which in this case are M1 and M2. We fix probability p
Note, M₁ [M11, M22] is the concatenation of the variational parameters of layer I and layer 2, and similarly, M2 = [M21, Ma2] is the concatenation of the variational parameters of layer I and layer 2 (Assuming
have NN of only two layers for simplicity). For example if the first point selected while sampling, we will have W1 = Mn and W2- M1.. we will see that minimizing the KL divergence this posterior with the true posterior is mathematically equivalent to the Deep Ensemble algorithm described above. For sampling, the W and W2 from the above posterior, we use the same sampling technique described for MoG, except that here same random variable 2p is tied to all variational parameters.
u ~ Uniform(0, 1] and Zp luep
W1 = Mp (1 (1-21)) M21,
W₂ = 2p + +(1 (1- 222
In MoG for every weight we had to sample a separate 2, independently for each weight. This random variable tying is the only difference in sampling between the two, and this observation is critical tool linking Deep Ensemble to VI Framework. Next we try to minimize the unbiased Monte Carlo (MC) estimate of the mini-batch variational free energy. Minimizing the mini-batch cost for mini-batch 1,2
(1) (ww),, Liv- log(p (w))))(()))) +
1 M log(g(((000)) 1
log(p(w()))
M
Note we have taken a MC estimate of the KI. divergence too, but following Hoffman et al. (003)Kingun & Welling (2014) convergence to the same limit as the expectation of variational free energy is still guaranteed. Now assume a Gaussian prior i.c. P(w(()) = N(O,1-21) Note
for p 2 there are equal chance of having (i(() = MI or (i)) = M2, suppose for this batch i,
wi)) = =M1
(M)) + M log( 1 +
1 11
= log(((
M 1
PIMM
(10)
We further simplify loss function for regression case, and fit Gaussian with variance -- (also called model precision), to the negative log- likeliiodood of the data i.. p(y,lo,, M1) N(ys 0j, (T)). So, after dividing the whole equation by we have
4

---

Under review as a conference paper at ICLR 2020
Li- 1 2
Σ JED
1l8, 05112 +
1 MT 2
log( 1 2T
1
(1I)
M
Minimizing the above equation be seen as training single NN with parameters Mj on mini batch i with 1.2 regularization À 2M Equivalently, there will be another mini-batch] j where w( M2. and minimizing the MC estimate of the variational free energy, in this case, will be equivalent to training NN with parameters M2 with the same regularization as before. This is equivalent to a Deep Ensemble of 2 NNs with parameters M1 and M2 which were randomly ini- tialized and trained on independently sampled mini batches as was in the original algorithm. The predictive distribution of unknown label y is given as g(v" ((v(<x)) = p(y" [x", w)g(W]))dl)))
{p(y"a", M2) + [x="" M2), which is mixture of Gaussian and matches the predictive distri- bution of Deep Ensemble. Thus independently training an Ensemble of NN's can be seen training a "hypothetical" Bayesian NN's with a specific mixture distribution.
Although we have shown equivalence for a specific case of 2 NNs, it can easily generalized to N NNs, by considering an approximate posterior a mixture of N points and taking p = N. Note that we can tune p's too by using the same concrete relaxation that we took in CMoG. We called this modified algorithm Concrete Ensemble.
Stein Variational Gradient Descent (SVGD) (Liu & Wang (2066) is particle based Bayesian ap- proach and close to Deep Ensembles. Both SVGD and Deep Ensembles can be viewed as particle filters trying to approximate an underlying distribution. The mini- SVGD update is given as 0-1 = 0 + N 1 Vaz log(p(0)1 + Vork(0). 0.)]. Ensembles have a mode seeking nature, whereas SVGD seek to maintain some level of diversity. SVGD is far less popular despite their similarity with Deep Ensembles because practice, different initializations for large scale NNs suffice for capturing different modes, and thus, the additional time complexity overhead of O(N2) for computing the kernel is not justified by empirical results.
More specifically, if fo(w) > Ei [i(w; 0z), where 0, are independent variational parameters, and (w) could be any arbitrary distribution. Let us define (() as the Variational Free Energy of the mixture distribution fo( and L(0,) as the Variational Free Energy of the individual mixture
component fs 0:). We can approximately bound L()) as; N Σ L(0.) ( < L()) s
& E I(O.),, where H(p) ) is the entropy. So, individually optimizing 0, 's (since they are indepen- dent of each other) is also minimizing the variational free energy of the mixture distribution, and interestingly increasing the ensemble size increases the tightness of the bound,The upper bound is arrived at using the convexity of KI. divergence and the lower bound is obtained by further lower bounding the variational approximation to KI. divergence between mixture distributions (Durrieu et al. (2012)). See Appendix A.2 for more details
Simple arguments that will allow us to extend this proof to ensemble of arbitrary architecture of Neural Networks have been added to Appendix
EXPERIMENTS
The following networks are used in experiments: Normal feed forward Multi Layer Perceptron (MLP), Deep Ensemble of NN (Ensemble) (EnsembleyLakkhmmnnnn (Lakshminarryaaan et al. a 2017). Concrete Deep En- semble of 3 NN (CEnsemble), Concrete Mixture of 3 Gaussain (CMoG) with p's tied for each layer (3 p's per layer), Concrete Mixture of 2 Matrix Variate Gaussian (CMoMG), Bayes by Backprop (BBB) (Blundell et al. (2015) and Concrete Dropout (CDropout) Gal et al. 2017
4.1 REGRESSION TASK
We use Concrete mixture of Gaussian, and Concrete Ensemble described above fit y 23 for -4 < 4, and observe the estimation for -6 < < 6. We observe our algorithms (CMoG and CMoMG) can predict uncertainty in out of training data points much better compared to the uncertainty estimation with vanilla NN or previous uncertainty estimation frameworks like MC- Dropouts Gal (0016). Ensemble, BBB, and Concrete dropouts. For both CMoG (we consider a mixture of 2 and Gaussian) and Concrete Ensemble, we use fully connected feed forward NN with two hidden layers of size 100 each and ReLU activation, and performed forward pass ten times, with result averaging for training, and for testing, performing a forward pass on trained NN forty times. We perform this experiment thrice and average the results. Figure ] compares uncertainty estimation using CMoG and Concrete Ensemble with previous frameworks for the same architecture as described above.
5

---

Under review as a conference paper at ICLR 2020
Figure 1: Uncertainty estimates (=1 std) ) for out of distribution samples on toy regression data
-15
t
1000
t= 2500
t 5000
Figure 2: Time evolution of
of one of the weight in the CMoG-BNN with mixture
of Gaussians
4.2 UNCERTAINTY ESTIMATION ON FULLY CONNECTED NETWORKS
In this subsection, we show advantage of using multimodal posterior for fully connected feed for- ward networks. We first describe results for our methods on the classification task of MNIST digit dataset (LeCun et al. (0TO0). We use fully connected network with two hidden layers having 1000 and 600 units respectively and ReLU activation. The last layer is a softmax layer with ten units. Refer appendix section for accuracy obtained and no. of parameters used.
Now, we assess how our methods behave when test data does not lie on the training manifold. Over- confident predictions on such data are one of the most important challenges for reliable deployment of deep learning methods in the real world. The expectation is that if the input image does not lie the training manifold, then the predictive uncertainty should be high. We use the same architecture as described above for the remaining subsection.
The first experiment we do evaluate the predictive uncertainty of model trained on MNIST dataset on Fashion MNIST dataset (Xiao et al. (2017)). This is similar to MNIST-NoIMNIST experiment al. (2017). 7). As the images in Fashion MNIST dataset completely different than MNÍST, ideally a trained model should be very uncertain about its prediction. Figure [3] shows the histogram of entropy of the output of trained models using different methods on test MNIST, and test Fashion MNIST data. MLP and Concrete Dropout are overconfident on predictions for Fashion MNIST data. Deep Ensemble and CEnsemble show increase in no. of data points showing high entropy. BBB shows significant improvement over previously described methods. CMoMG shows comparable results with BBB even after having more modes indicating that the assumption of independent rows and columns may not be best for capturing uncertainty. CMoG gives the best results by far, which are also near ideal expectations.
done by Lakshminaraaaaan
For second and third experimént we use more sophisticated methods analyzing behavior of pre- dictive uncertainty of model as done
Li & Gal (011)). We don't include MLP and Concrete Dropout for these experiments for better clarity (they perform significantly worse). The second ex- periment considers single gradient step Fast Gradient Sign (FGS) method (Goodfellow et al. (201))) input image by mov-
The method generates adversarial input by adding small perturbations
ing the direction which reduces predicted class probability.
Xade sgn V max log p(y(x))
(12)
7

---

Under review as a conference paper at ICLR 2020
14:17
MEST
2%
36
45.
Chinle
300
Figure 3: MNIST-FashhioMMIISST Histogram (200 bins) of entropy obtained on MNIST and Fash- ion MNIST test data
The step size Yj is varied from to 0.5. Figure shows the accuracy and predictive entropy for different values of y. CMoG again gives the highest predictive entropy which helps in identifying adversarial inputs. CMoMG has the slowest decaying accuracy vs step-size curve which makes it more robust to adversarial attacks. For example, second column of images(1) 0.13) in right panel of Figure are visually close to class '2' and CMoMG gives higher accuracy for that 7) than any other method. Ensemble based methods are again outperformed by BBB, CMoG and CMoMG. The second attack we consider is the targeted version of FGS (Goodfellow et al. (2014)). The attack
2
2
2
2
2
1
3
Predectvve
5
2
2
2
2
2
2
2
2
2
7
Slop Sice
Figure 4: Untargeted Attack: Classification accuracy and predictive entropy vs step-size. The ad-
versarial images are shown(top bottom) for BBB, CEnsemble, Ensemble, CMoG and CMoMG
here implemented iteratively (Kurakin et al. (2016) which produces adversarial examples by even finer perturbations than single gradient step and without completely destroying the content of original image. The predictive probability of target class ("0' in our case) is maximized for all non-target class images using
adt
X
ado = Xodo + 17 sgn( V xta log P(Utarget (xdd))
(13)
We fix v) to 0.01 and iteratively update for 100 iterations. The results obtained are shown in Figure CMoMG has the slowest decaying and slowest rising true class and target class accuracy respectively making it more robust against targeted attack. CMoG has highest rise in predictive entropy, making more suited for identifying adversarial attack. We note that for iteration 30-100, CMoMG has better ability to identify an adversarial input. Both these methods having multi-modal posterior perform better than BBB and Ensemble based methods.
More detailed analysis is required for examining the performance of our methods on adversarial attacks, but conclude from the experiments that multi-moda posterior based methods are well suited for capturing uncertainty.
4.3 UNCERTAINTY ESTIMATION ON ALEXNET AND RESNET
We implement multimodal posterior for AlexNet Krizhevsky et al. 2012 and ResNet (He et al. (2166) getting better uncertainty estimates. The posterior used is only for fully connected layers betore final layer thus, with less of parameters than Deep Ensemble. All the other layers except fully connected layer have been used with spatial concrete dropout. Refer appendix section for accuracy obtained and no. of parameters used.
We repeat the MNIST-FashhonMNNIS' experiment done in subsection 4,2 for AlexNet. Figure L()) shows the result obtained. We have shown CDF for better comparison. Note that, even with slightly less no. of parameters CMoG provides considerable improvement Deep Ensembles while BBB

---

Under review as a conference paper at ICLR 2020
2
2
2
2
2
2
2
2
2
2
2
Fird Den
2
2
2
2
2-
2
2
2
2
0
1
2
2
Figure 5: Targeted Attack: Classification accuracy for target class '0" and class and predictive
entropy vs. iterations. The adversarial images are shown(top to bottom) for BBB, CEnsemble, Ensemble, CMoG, and CMoMG
and CMoMG gives comparable uncertainty with significantly less no. of parameters. This effect will only improve for larger and deeper networks.
Labio
Figure 6: Predictive Entropy CDF: AlexNet on FashionMNIST (L), ResNet on SVHN (R)
We repeat the same experiment but now for CIFAR-10 (Krizhevsky al.) SVHN Netzer et al. (20TT), ), using ResNet-20. Figure/6) shows the uncertainty obtained SVHN dataset after train- ing the model on CIFAR- 10 dataset. clear improvement
uncertainty estimation can observed for CMoG over Deep Ensembles with significantly less no. of parameters. This further proves that we can indeed get better uncertainty estimates for deep networks by using multimodal posterior on few layers with only small increase in parameters over normal network.
4.4 CONTEXTUAL BANDIT
We consider an online learning task (Mushroom Bandit Task Collier Llorens 2018), Blundell et al.] (2015)) to get clear handle on the usefulness of uncertainty estimates. In an online setup, the agent plays her action based on policy π. Based on this action, the environment reveals stochastic reward. The objective of online learning is to Update the policy
data is revealed maximize the cumulative reward received. Since the reward revealed is a noisy reward, online learning algorithms face a fundamental dilemma: exploration v/s exploitation. Typically, online learning algorithms explore in high uncertainty environments and exploit in low uncertainty setups.
more and
We train a contextual neural bandit to learn the dependency between the context and optimal action per the pseudo-algorithm given in Collier & Llorens (2018). We train the NN based BBB, CMoG, Ensemble, CEnsemble, and CMoMG. The regret variation is shown in figure with the results have been averaged over three trials.
The MoG agent achieves balance between eating and ignoring mushrooms more quickly than an ensemble network and incurs the least regret. Similarly, Matrix Variate based CMoMG can estimate uncertainty better than BBB and learns the context much faster. Thus, can that the ability capture multiple modes provides us superior estimates of uncertainty, allowing us to balance exploration-explottaiio a better way.
300
Greerie
20
Figure 7: Cumulative Regret Variation for Mushroom Bandit Task
8

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION
In this paper, we show equivalence between Ensembles of Deterministic and Bayesian NN and VI. We have also generalized this proof to an ensemble of NN's with arbitrary architecture. Bayesian NN's with multi-modal posterior have been shown give superior uncertainty estimates on out of distribution samples, be more robust to adversarial attacks and learn better trade-offs between explo- ration and exploitation, while maintaining comparable or better test accuracy as standard methods. We also show that the parametric overhead of Ensembles can be reduced by using Bayesian NN's with mixture distribution.
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint 4rX1v:1505.05424. 2015.
Mark Collier and Hector Urdiales Llorens. Deep contextual multi-armed bandits. drXiv:1807.09889, 2018.
arXiv preprint
Andreas Damianou and Neil Lawrence. Deep gaussian processes. In Artificial Intelligence and Statistics, pp. 207-215, 2013.
J-L Durrieu, J-Ph Thiran, and Finnian Kelly. Lower and upper bounds for approximation of the kullback-leibler divergence between gaussian mixture models. In 2012 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4833-4834 leee, 2012.
Karl Friston, Jérémie Mattout, Nelson Trujillo-Barreto, John Ashburner, and Will Penny. Variational free energy and the laplace approximation. Neuroimage, 34(1):220-234, 2007.
Yarin Gal. Uncertainty in deep learning. PhD thesis, PhD thesis, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in Neural Information Processing Systems, pp. 3581-3590 2017.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint vrXiv:14/2.6577 2014.
Alex Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pp. 2348-2356 2011.
Arjun K Gupta and Daya K Nagar. Matrix variate distributions. Chapman and Hall/CRC, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaus- sian mixture models. In 2007 IEEE international Conference Acoustics, Speech and Signal Processing-ICASSPPO volume 4, pp. IV-317. IEEF, 2007.
Geoffrey Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the descrip- tion length of the weights. In in Proc. of the 6th Ann, ACM Conf. on Computational Learning Theory. Citeseer, 1993.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint arXiv:1207.0580 2012.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley, Stochastic variational infer- ence. The Journal of Machine Learning Research, 14(1):1303-133 1347, 2013.
Diederik P Kingma and Max Welling. Stochastic gradient vb and the variational auto-encoder. In Second International Conference Learning Representations, ICLR, 2014.

---

Under review as a conference paper at ICLR 2020
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, pp. 2575-2583, 2015.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL URL http: /wwww es toronto. cabuukkriz/a citzz/cifarr html
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q- Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-
1105. Curran Associates, Inc.. 2012. URL www.wwiittthh-hoooeee -ttp--eeeoooooooooo.a...... wtw.:uuuttooonannuuuaaaaaa neuraa-networks.
ni CC/ cc/pper
4824 minagenet-classiiiaa...
pdi
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprini
00.00077.,,,,, 2016.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [On- line]. Available: http://yann. lecun. com/exdb/mmist, 2:18, 2010.
Yingzhen Li and Yarin Gal. Dropout inference in bayesian neural networks with alpha-divergences. In Proceedings of the 34th International Conference on Machine Gearning-Volumne 70, pp. 2052- 2061. JMLR. org, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. In Advances in neural information processing systems, pp. 2378-2386, 2016.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference Machine Learning, pp. 1708-1716, 2016.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.0077 2016.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NISS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL
housemumbers. F
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014.
Thilo Strauss, Markus Hanselmann, Andrej Junginger, and Holger Ulmer. Ensemble methods as defense to adversarial perturbations against deep neural networks, 2018. URL https://
@genneeview.net//Formm wttprrummmdirka11ff33
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 2017,
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 ACCURACY AND NUMBER OF PARAMETERS
Table reppots the test accuracy obtained on MNIST with Fully connected network and results were averaged over three trials. The accuracy of CMoG, CEnsemble and CMoMG are better or compa- rable to benchmark MIP, BBB, Deep Ensemble and Concrete Dropout with the same architecture.
Table 1: Test Accuracy MNIST
Method
Test Accuracy
No. of parameters (in Millions)
MLP
98.28
1.39
Concrete Dropout
98.8
1.39
Deep Ensemble
98.51
4.17
BBB
98,44
2.78
Concrete Ensemble
98.57
4.17
CMoG
98.6
8.34
CMoMG
98.66
2.79
Table reports the test accuracy obtained MNIST with AlexNet and results averaged over three tnals.
Table 2: Test Accuracy on MNIST
Method (Dataset)
Test Accuracy
No. of parameters (in Thousands)
CMoG
99.36
44.94
BBB
99.26
21.58
Deep Ensemble
99.29
48.25
CMoMG
99.18
29.38
Table 3 reports the test accuracy obtained on CIFAR-10 with Resnet-20 and results were averaged three trials.
Table Test Accuracy on CIFAR-10
Method (Dataset)
Test Accuracy
No. parameters (in Millions)
CMoG
91.9
0.273
BBB
91.01
0,271
Deep Ensemble
92.1
0.81
Table rrpport the Expected Calibration Error (ECE) on CIFAR-10 with Resnet-20 and results were averaged over three trials.
Table ECE on CIFAR-10
Method
ECE
CMoG
0.019
MoG (categorical p)
0.023
Deep Ensemble
0.018
11

---

Under review as a conference paper at ICLR 2020
A.2 BOUNDING THE VARIATIONAL FREE ENERGY WITH RESPECT TO MIXTURE DISTRIBUTIONS
Variational free energy has components the Expected loss and the KL divergence between the assumed posterior and the prior. Now because the Expectation over a mixture distribution equal to the sum over expectation of individual components all we need to do is bound the KL divergence between mixture distribution and true posterior in terms of KL divergence between individual mixture components and the true posterior.
Thus we want bound the KL divergence between f(x) and g(x) where
fo(x) Σ (,)((;)0)))
is the weight of the component individual in the mixture distribution.
simple upper bound can be achieved using convexity of KL distance as
(KI.(a(
(.D))((((r
(14)
Keeping Ps fixed as & will give us the desired upper bound
We can also find the lower bound using the Variational Approximation (Hershey & Olsen by first splitting the KL divergence into components-
2007
(f=))()))())) = Es(r)log(((=)) (rco)log(o(>))
(15)
We can use Jensen's 's inequality to get the appropriate lower bound for the first term as-
Σ [1(0))) [1)))ogg( Σ Pj
Lsi fj)dx
Ez(log([(x))
Øj
2 Enio / fi() (1)
Maximizing the lower bound wrt Φjs under the constraint dy 0 and Σ, ف3z 1, i.j get
(lg((f((z)
Σ
log( Σ Dje
(io-(tt)))))))) - Σ p.H(fi)
j=l
By similar arguments we can derive-
N ps log(e PRo(filo)) Σ p. H(A)
Fy (log(g(z))
Subtracting the two gives the Variational Approximation (Hershey Olsen (007)) KL diver- gence
N pje Dxz(A1165)
Dvar (fll)) Σ p. D.D +
E Pe log
Durrieu et al. (2012) empirically showed this approximation to be closer to Dxi(fllg) than most other approximations, thus Drar (fllg) DxI(]19))
12

---

Under review as a conference paper at ICLR 2020
We further lower bound, this Variational approximation by ignoring all * i terms inside log Vog(Ei111) Pye D2z(f.115))
Dkr(f Dvar (fllg)
Σ (fill)( + Σ pc log(p:)
Fixing pi 's as N gives us the required lower bound
N Σ (kllg) H1 )
DxI(f)lg) Dvar (fllg)
N
We know Variational free Energy w.r fo(w) is L(O) Ein
+
DEL (fo(w))lp(w)
We then the derived upper and lower bounds for the KL divergence between mixture distribution bound the Variational free energy as required.
A.3 GENERALIZATION OF PROOF TO MODEL AGNOSTIC
In an ensemble it is often the case when we take an ensemble of arbitrary NN's rather than NN's with same architecture. So it is desirable for generalization of the proof given in Section 3 to this case.
To do so we add "pseudo parameters" to each NN such that the dimensionality of the pa- rameters in each NN is extended become the same and assume this to be N. We now define prior on the R such that the support of both the approximate posterior Mixture of Gaussian and Gaussian prior is the same.
We call these added parameters "pseudo" because we want them to be hanging parameters that do not effect the neural network output (imagine additional hidden layer whose output is detached from the next layer, thus increasing the dimensionality of the network without changing it's original output).
Thus using this trick we have reduced the problem of ensemble of arbitrary NN's to the problem of NN's with same parametric dimensionality. Now we can define mixture distribu- Σim p:9(w(0.) where 0; corresponds the variational parameters of 7th NN and each 0g now has the same N dimensionality thanks to the previously mentioned trick.
tion posterior as q(w(6)
Note now will comprise of two sets of variational parameter (real parameters) and 0" (pseudo parameters). An interesting observation that can be made that the optimal solution for the is collapsing to the prior.
L0)) = E
+ DxI (o
Now for optimal solution w.r.t d" we have
V or E F loggo V Dx ()
Note Var gq [0) log(p the prior, c.g. if ( Gaussian and the prior p(w) N(0, I) then we have q(w(PP) N(O. 7) as the optimal solution. For the q( (ww(6") the optimization shall remain exactly the same as in Section 3. Implying ensembles of NN's with different architecture too be framed inside Variational Framework.
0 by design, so clearly the optimal solution for 0P is to collapse
13


Reviews:
[{"review_id": "rkeat7C6FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes to use either relaxed mixture distributions or relaxed mixtures of matrix Gaussians as the approximate posterior for Bayesian neural networks. Naturally, taking the mixture variance to zero allows a stretched interpretation of ensembling to become Bayesian as well. Experiments are performed on small neural networks on regression tasks, as well as uncertainty estimation experiments on deep networks. Finally, a downstream task of bandit optimization (the mushroom experiment) is performed. \n\npost-rebuttal: I'd like to sincerely thank the authors of the paper for their enlightening discussions about their work. I'm inclined to think that this paper has a future, and definitely encourage the authors to resubmit - taking into account the reviews here. In particular, it would be greatly helpful if they cleaned up some of the writing and explanation, as our discussion points to.\n\nHowever, my stance on the codebase remains mostly unchanged - while a notebook ended up being released with their method, it was too close to the deadline and in a bit too rough of shape to really be able to poke through their implementation. \nFrom a quick pass, their implementation seems to be correct - although rather than attempting to base their code on the old torch/lua implemenations, I'd still suggest using a pre-written version. However, without the completely trained numbers, I cannot in good conscience vote to accept the paper. \n\nTldr: I vote to reject this paper for several reasons. The interpretation of ensembling as variational inference is both flawed and well known. Mixture distributions have been previously proposed for variational inference. There are significant enough weaknesses in the empirical results that make me question the authors’ own implementations. I will increase my score if these issues are resolved.\n\nOriginality:\n1)\tIt is well known (and immediately obvious) that one can interpret ensembles as a Bayesian method – with a mixture of delta functions around the independently trained models. Furthermore, the Bayesian bootstrap (Rubin, 1983) is a Bayesian interpretation of the bootstrap (identical under many conditions), while re-weighted ensembles are Bayesian models (Newton, 1995; Newton, 2018) that converge to the true posterior under many conditions – intriguingly, this version of re-weighting could potentially have good uncertainty quantification properties. For the specific case of stacking (which is closely related to ensembling), there are interpretations of stacking as Bayesian model combination (Yao et al, 2018, Clyde & Iversen, 2013). As a result, this view of ensembling as a Bayesian method is not novel.\n\n2)\tAdditionally, the derivation of dropout as Bayesian inference (which relies on a similar interpretation on taking the variance parameter in the Gaussian to zero) is known to be flawed (Hron et al, 2018); this flaw is directly applicable to the interpretation of ensembling as variational inference as described in Section 3 of this paper. The issue is fundamental: when taking the variance parameter to zero, the supports of the variational distribution and the true distributions have a mismatch, causing the KL divergence to become infinite. As a result, one cannot minimize the objective (as it is infinite by design).\n\nClarity:\n1)\tThe proposed methods are clearly explained in the setting under which one might be able to re-implement them, particularly in the description of the approximate posteriors.\n\npost rebuttal: to my knowledge, this was never really addressed. In particular, \"why do we want to interpet ensembling in a Bayesian manner?\" is still an open question to me.\n2)\tWhy is that we should expect (approximate) Bayesian procedures to be better calibrated than frequentist ones? It may be the case that if the integrals are accurately estimated (e.g. we have reached the true posterior and the model used for the data is in fact accurate), then we should expect Bayesian procedures to be well calibrated. However, this does not seem to be the case. A reference or argument for why (even in the model incorrect & approximate inference cases) in the introduction would greatly enhance the strength of the paper.\n\npost rebuttal: I think that this issue was addressed in a reasonable manner through the rebuttal, but will require a bit of rewriting.\n3)\tFrom a quick pass through Appendix A.2, the derivation is quite unstructured and difficult to follow. It is tough to see exactly what is being meant by the probabilities in the first line. \nAdditionally, the phrasing of “We further lower bound…” seems to suggest that the bound being used is not the well known bound of Durrieu et al, 2012, but a looser bound. If so, what is the approximation accuracy of this bound in relation to the other bounds compared against in this setting?\n\n4)\tIn sum, three different methods are proposed: concrete mixtures of Gaussians, concrete ensembles, and concrete mixtures of matrix variate Gaussians. Throughout the experiments, it seems like __one__ of these methods always wins on the task at hand. However, it does not always seem to be the same method. Is there a recommendation for practitioners as to which method to use – or is this simply task dependent? Specifically, is there an understanding as to why each method performs vastly differently on each task?\nA priori, I would expect the concrete mixture of matrix variate Gaussians to always perform best because it does seem to be able to model multivariate posteriors the best. However, it seems to perform worse on several of the predictive uncertainty tasks.\n\n5)\tFor the uncertainty experiments, it would be nice to have more qualitative numbers – for example the expected calibration error – rather than just the plots on in and out of distribution example. After all, not only do we want to be able to recognize out of distribution examples, but we want to be able to trust the probabilities that are output. \n\n6)\tIn Figure 4, thank you for providing the adversarial attacks used to fool the networks – the step size parameter doesn’t give a great intuition, and the images are welcome and useful.\n\n\nSignificance:\npost rebuttal: I think that this issue was satisfactorily addressed.\n1)\tMixture distributions as the variational family have already been proposed – at least once – see Miller et al, 2017 for an example that additionally uses the reparameterization trick. There, updates to the posterior are performed via selectively adding mixture components when necessary. Given that they also run experiments on neural networks, it would be nice to see a comparison to that method. \n\nQuality:\n1)\tI am very concerned by the performance of the ResNet20 models trained on CIFAR10 in the paper. The reported results for deep ensembles (an ensemble of _three_ independently trained models) in Appendix A are 85.23% accuracy. By comparison, the number in the original paper (He et al, 2015) is 91.25% for a _single_ model. Furthermore, the first link to a PyTorch repo from a Google search (https://github.com/akamaster/pytorch_resnet_cifar10) comes up with a re-implementation that gets 91.63% (again for a single model).\n\n__This issue alone is enough for me to vote to reject the paper until the authors’ implementations are fixed and run with deep ensembled ResNets that get similar accuracy. I hope to see the implementation issues fixed in the rebuttal period. __\n\n2)\tFurthermore, in Appendix 2, I see a potential issue in using the KL divergence and would like the authors to clarify.\nIf the secondary lower bound is being used to approximate the KL divergence between the mixture distributions, it becomes very unclear if what is being optimized is in fact a __lower bound__ on the log probability. After all, the standard ELBO is written as:\n\tLog p(y) \\geq ELBO := E_q(\\log p(y | \\theta)) – KL(q(\\theta) || p(\\theta)) \nImplying that another lower bound on the KL would be greater than the ELBO and thus not necessarily a lower bound on the log marginal likelihood.\n\n3)\tBottom of page 7: “The posterior is used only for fully connected layers…” Why is this the case? Alternative Bayesian methods can be used on the full network.\n\n\nMinor Comments:\n-\tPlease attempt to use the math_commands.tex in the ICLR style file to write math. This makes the math standardized throughout.\n-\tFigure 1 is especially unclear and the legends and captions should be made considerably larger. Zooming in at 800% seems to be necessary to even be able to make out which method is which. Consider, placing all methods on a single plot and then coloring/dashing them separately. Additionally, please compare to HMC on this task (and an equivalent RBF GP) to show the uncertainties from the “gold standard” methods.\n-\tPlease additionally make the figure legends larger for the rest of the figures. \n-\tWhat is the meaning of time in Figure 2? I assume that it means training time, but it is not especially clear from the caption. With that being said, I think that it is quite interesting that the marginals do seem to look multi-modal, even at the end of training.\n-\tLine before Eq 7: please use \\citet if possible to cite Gupta & Nagar, rather than stating the clunky Gupta et al. \\citep{Gupta & Nagar}.\n-\tEq 2: please do not use \\hspace{-…} to save space in equations, so that the equation does not overwrite lines before.\n-\tPage 2: “Deep Ensembles have lacked theoretical support …” In order to make this claim, you must first explain why one ought to be Bayesian in the first place. See above for the history of interpreting ensembling from a Bayesian perspective. There is no a priori reason why one would not just wish to have frequentist justifications of ensembling (and potentially even calibration).\n-\tPlease do not capitalize Ensembling or Variational Inference throughout.\n\nReferences:\nClyde & Iversen, Bayesian model averaging in the M-open framework. In Bayesian Theory and Applications, 2013. DOI:10.1093/acprof:oso/9780199695607.003.0024\n\nHe, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition, CVPR, 2016. https://arxiv.org/abs/1512.03385\n\nHron, Matthews & Ghahramani, Variational Bayesian dropout: pitfalls and fixes, ICML, 2018. https://arxiv.org/abs/1807.01969\n\nMiller, Foti & Adams, Variational Boosting: Iteratively Refining Posterior Approximations, ICML, 2017. https://arxiv.org/abs/1611.06585\n\nNewton & Raftery, Approximate Bayesian Inference with the Weighted Likelihood Bootstrap. JRSS:B, 1994. https://www.jstor.org/stable/2346025?seq=1#metadata_info_tab_contents\n\nNewton, Polson & Xu, Weighted Bayesian Bootstrap for Scalable Bayes, 2018; https://arxiv.org/abs/1803.04559\n\nRubin, The Bayesian Bootstrap, 1981. Annals of Statistics. https://projecteuclid.org/euclid.aos/1176345338\n\nYao, Vehtari, Simpson, & Gelman, Using Stacking to Average Bayesian Predictive Distributions, Bayesian Analysis, 2018; https://projecteuclid.org/euclid.ba/1516093227\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syg9jcy6cH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nIn this works, the authors propose to use a concrete mixture of Gaussians as a variational distribution. The authors show that the deep ensemble method can be viewed as a special case of a mixture of Gaussians with a categorical prior.\nI think the main contribution is to use the concrete distribution as a mixture prior q(z) instead of a categorical prior.\nHowever, there are some concerns. The following points should be addressed to get a higher rating. \n\n(1) Several papers consider the problem of learning a mixture of Gaussians in the VI framework ([1,2,3,4,5]). The deep ensemble method considered in this paper is just one of the existing ensemble approaches. The related work section should be updated to discuss the novelty of this work.\n\n(2) In the deep ensemble method, the categorial prior q(z) is held fixed. However, it is possible to update the categorical prior q(z). \nIn this work, the proposed distribution is q(w) = sum_{c=1}^{K} Concrete(z=c|\\theta_z) Gauss(w|z=c,\\theta_{w_c}).\nBy using the concrete prior, the gradient can be computed by the local reparametrization trick for q(w|z) and the reparametrization trick for q(z). \nHowever, even when q(z) is the categorical prior,  the local reparametrization trick for q(w|z) is still valid if the variational parameters for each mixture component are not tied.  The main difference is the reparametrization trick can not be used for q(z). \nThe authors should show why the proposed variational distribution is better than the mixture of Gaussians with a categorical prior. \n \n\nReferences\n[1] O. Arenz, M. Zhong, and G. Neumann. \"Efficient Gradient-Free Variational Inference using Policy Search.\" ICML. 2018.\n[2] A. C. Miller, N. J. Foti, A. D’Amour, and R. P. Adams. Variational boosting: Iteratively refining posterior approximations. ICML, 2017. \n[3] O. Zobay. Variational Bayesian inference with gaussian-mixture approximations. Electronic Journal of Statistics, 8(1):355–389, 2014.\n[4] Lin, Wu, Mohammad Emtiyaz Khan, and Mark Schmidt. Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations. ICML 2019.\n[5] F. Guo, X. Wang, K. Fan, T. Broderick, and D. B. Dunson. Boosting variational inference. arXiv:1611.05559v2, 2016. \n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJee9LcZqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors interpret Deep Ensemble as a special type of variational inference. Based on Bayes by Backprop which uses a Gaussian approximation to the posterior, the authors propose to use a mixture of Gaussians. The proposed methods have been tested on a regression task and Bayesian neural networks.\n\nThe authors argue that Deep ensemble is equivalent to variational inference with a mixture of Gaussians approximation with variance going to 0. It is not surprising that Deep ensemble is equivalent to variational inference with learning the mean only.  I’m not sure how useful this argument is since learning the distribution, not only the mean, is the key factor of being Bayesian.\n\nUsing a mixture of Gaussians rather than a Gaussian in Bayes by Backprop is a natural extension and therefore the novelty seems low. \n\nIn the experiments, I’m surprised to see that Bayes by Backprop fails to give any uncertainty estimate on the simple regression experiment. From the figure, BBB seems to be very confident about its prediction. However, it has been demonstrated in the literature that BBB is able to perform fairly well on this kind of regression. Can the authors give explanations on why it performs badly here? Also, I do not see why it is important to model multiple modes on this task. I believe a Gaussian approximation is able to work well. The multimodality here is likely induced by the symmetric parameterization of neural networks and trying to capture this kind of multimodality will be meaningless and even problematic. By looking at Figure 2, it seems like the posterior of the proposed method is close to being unimodal. Why is it beneficial to use a mixture of Gaussians under this situation?\n\nThe axis and labels in the figures are very small and hard to read.\nEq. (2) is overlapped with the above text.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use mixture distributions to improve uncertainty estimates in BNNs. Ensemble methods are interpreted as a Bayesian mixture posterior approximation. To reduce the computation, a modification to BBB is provided based on a concrete mixture distribution.

Both R1 and R3 have given useful feedback. It is clear that interpretation of ensemble as a Bayesian posterior is well known, and some of them also have theoretical issues. The experiment to clearly comparing proposed mixture posterior to more commonly used mixture distribution is also necessary. 

Due to these reasons, I recommend to reject this paper. I encourage the authors to use reviewers feedback to improve the paper.
Decision: Reject