Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary
In this works, the authors propose to use a concrete mixture of Gaussians as a variational distribution. The authors show that the deep ensemble method can be viewed as a special case of a mixture of Gaussians with a categorical prior.
I think the main contribution is to use the concrete distribution as a mixture prior q(z) instead of a categorical prior.
However, there are some concerns. The following points should be addressed to get a higher rating. 

(1) Several papers consider the problem of learning a mixture of Gaussians in the VI framework ([1,2,3,4,5]). The deep ensemble method considered in this paper is just one of the existing ensemble approaches. The related work section should be updated to discuss the novelty of this work.

(2) In the deep ensemble method, the categorial prior q(z) is held fixed. However, it is possible to update the categorical prior q(z). 
In this work, the proposed distribution is q(w) = sum_{c=1}^{K} Concrete(z=c|\theta_z) Gauss(w|z=c,\theta_{w_c}).
By using the concrete prior, the gradient can be computed by the local reparametrization trick for q(w|z) and the reparametrization trick for q(z). 
However, even when q(z) is the categorical prior,  the local reparametrization trick for q(w|z) is still valid if the variational parameters for each mixture component are not tied.  The main difference is the reparametrization trick can not be used for q(z). 
The authors should show why the proposed variational distribution is better than the mixture of Gaussians with a categorical prior. 
 

References
[1] O. Arenz, M. Zhong, and G. Neumann. "Efficient Gradient-Free Variational Inference using Policy Search." ICML. 2018.
[2] A. C. Miller, N. J. Foti, A. D’Amour, and R. P. Adams. Variational boosting: Iteratively refining posterior approximations. ICML, 2017. 
[3] O. Zobay. Variational Bayesian inference with gaussian-mixture approximations. Electronic Journal of Statistics, 8(1):355–389, 2014.
[4] Lin, Wu, Mohammad Emtiyaz Khan, and Mark Schmidt. Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations. ICML 2019.
[5] F. Guo, X. Wang, K. Fan, T. Broderick, and D. B. Dunson. Boosting variational inference. arXiv:1611.05559v2, 2016. 




review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The authors interpret Deep Ensemble as a special type of variational inference. Based on Bayes by Backprop which uses a Gaussian approximation to the posterior, the authors propose to use a mixture of Gaussians. The proposed methods have been tested on a regression task and Bayesian neural networks.

The authors argue that Deep ensemble is equivalent to variational inference with a mixture of Gaussians approximation with variance going to 0. It is not surprising that Deep ensemble is equivalent to variational inference with learning the mean only.  I’m not sure how useful this argument is since learning the distribution, not only the mean, is the key factor of being Bayesian.

Using a mixture of Gaussians rather than a Gaussian in Bayes by Backprop is a natural extension and therefore the novelty seems low. 

In the experiments, I’m surprised to see that Bayes by Backprop fails to give any uncertainty estimate on the simple regression experiment. From the figure, BBB seems to be very confident about its prediction. However, it has been demonstrated in the literature that BBB is able to perform fairly well on this kind of regression. Can the authors give explanations on why it performs badly here? Also, I do not see why it is important to model multiple modes on this task. I believe a Gaussian approximation is able to work well. The multimodality here is likely induced by the symmetric parameterization of neural networks and trying to capture this kind of multimodality will be meaningless and even problematic. By looking at Figure 2, it seems like the posterior of the proposed method is close to being unimodal. Why is it beneficial to use a mixture of Gaussians under this situation?

The axis and labels in the figures are very small and hard to read.
Eq. (2) is overlapped with the above text.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: Summary: This paper proposes to use either relaxed mixture distributions or relaxed mixtures of matrix Gaussians as the approximate posterior for Bayesian neural networks. Naturally, taking the mixture variance to zero allows a stretched interpretation of ensembling to become Bayesian as well. Experiments are performed on small neural networks on regression tasks, as well as uncertainty estimation experiments on deep networks. Finally, a downstream task of bandit optimization (the mushroom experiment) is performed. 

post-rebuttal: I'd like to sincerely thank the authors of the paper for their enlightening discussions about their work. I'm inclined to think that this paper has a future, and definitely encourage the authors to resubmit - taking into account the reviews here. In particular, it would be greatly helpful if they cleaned up some of the writing and explanation, as our discussion points to.

However, my stance on the codebase remains mostly unchanged - while a notebook ended up being released with their method, it was too close to the deadline and in a bit too rough of shape to really be able to poke through their implementation. 
From a quick pass, their implementation seems to be correct - although rather than attempting to base their code on the old torch/lua implemenations, I'd still suggest using a pre-written version. However, without the completely trained numbers, I cannot in good conscience vote to accept the paper. 

Tldr: I vote to reject this paper for several reasons. The interpretation of ensembling as variational inference is both flawed and well known. Mixture distributions have been previously proposed for variational inference. There are significant enough weaknesses in the empirical results that make me question the authors’ own implementations. I will increase my score if these issues are resolved.

Originality:
1)	It is well known (and immediately obvious) that one can interpret ensembles as a Bayesian method – with a mixture of delta functions around the independently trained models. Furthermore, the Bayesian bootstrap (Rubin, 1983) is a Bayesian interpretation of the bootstrap (identical under many conditions), while re-weighted ensembles are Bayesian models (Newton, 1995; Newton, 2018) that converge to the true posterior under many conditions – intriguingly, this version of re-weighting could potentially have good uncertainty quantification properties. For the specific case of stacking (which is closely related to ensembling), there are interpretations of stacking as Bayesian model combination (Yao et al, 2018, Clyde & Iversen, 2013). As a result, this view of ensembling as a Bayesian method is not novel.

2)	Additionally, the derivation of dropout as Bayesian inference (which relies on a similar interpretation on taking the variance parameter in the Gaussian to zero) is known to be flawed (Hron et al, 2018); this flaw is directly applicable to the interpretation of ensembling as variational inference as described in Section 3 of this paper. The issue is fundamental: when taking the variance parameter to zero, the supports of the variational distribution and the true distributions have a mismatch, causing the KL divergence to become infinite. As a result, one cannot minimize the objective (as it is infinite by design).

Clarity:
1)	The proposed methods are clearly explained in the setting under which one might be able to re-implement them, particularly in the description of the approximate posteriors.

post rebuttal: to my knowledge, this was never really addressed. In particular, "why do we want to interpet ensembling in a Bayesian manner?" is still an open question to me.
2)	Why is that we should expect (approximate) Bayesian procedures to be better calibrated than frequentist ones? It may be the case that if the integrals are accurately estimated (e.g. we have reached the true posterior and the model used for the data is in fact accurate), then we should expect Bayesian procedures to be well calibrated. However, this does not seem to be the case. A reference or argument for why (even in the model incorrect & approximate inference cases) in the introduction would greatly enhance the strength of the paper.

post rebuttal: I think that this issue was addressed in a reasonable manner through the rebuttal, but will require a bit of rewriting.
3)	From a quick pass through Appendix A.2, the derivation is quite unstructured and difficult to follow. It is tough to see exactly what is being meant by the probabilities in the first line. 
Additionally, the phrasing of “We further lower bound…” seems to suggest that the bound being used is not the well known bound of Durrieu et al, 2012, but a looser bound. If so, what is the approximation accuracy of this bound in relation to the other bounds compared against in this setting?

4)	In sum, three different methods are proposed: concrete mixtures of Gaussians, concrete ensembles, and concrete mixtures of matrix variate Gaussians. Throughout the experiments, it seems like __one__ of these methods always wins on the task at hand. However, it does not always seem to be the same method. Is there a recommendation for practitioners as to which method to use – or is this simply task dependent? Specifically, is there an understanding as to why each method performs vastly differently on each task?
A priori, I would expect the concrete mixture of matrix variate Gaussians to always perform best because it does seem to be able to model multivariate posteriors the best. However, it seems to perform worse on several of the predictive uncertainty tasks.

5)	For the uncertainty experiments, it would be nice to have more qualitative numbers – for example the expected calibration error – rather than just the plots on in and out of distribution example. After all, not only do we want to be able to recognize out of distribution examples, but we want to be able to trust the probabilities that are output. 

6)	In Figure 4, thank you for providing the adversarial attacks used to fool the networks – the step size parameter doesn’t give a great intuition, and the images are welcome and useful.


Significance:
post rebuttal: I think that this issue was satisfactorily addressed.
1)	Mixture distributions as the variational family have already been proposed – at least once – see Miller et al, 2017 for an example that additionally uses the reparameterization trick. There, updates to the posterior are performed via selectively adding mixture components when necessary. Given that they also run experiments on neural networks, it would be nice to see a comparison to that method. 

Quality:
1)	I am very concerned by the performance of the ResNet20 models trained on CIFAR10 in the paper. The reported results for deep ensembles (an ensemble of _three_ independently trained models) in Appendix A are 85.23% accuracy. By comparison, the number in the original paper (He et al, 2015) is 91.25% for a _single_ model. Furthermore, the first link to a PyTorch repo from a Google search (https://github.com/akamaster/pytorch_resnet_cifar10) comes up with a re-implementation that gets 91.63% (again for a single model).

__This issue alone is enough for me to vote to reject the paper until the authors’ implementations are fixed and run with deep ensembled ResNets that get similar accuracy. I hope to see the implementation issues fixed in the rebuttal period. __

2)	Furthermore, in Appendix 2, I see a potential issue in using the KL divergence and would like the authors to clarify.
If the secondary lower bound is being used to approximate the KL divergence between the mixture distributions, it becomes very unclear if what is being optimized is in fact a __lower bound__ on the log probability. After all, the standard ELBO is written as:
	Log p(y) \geq ELBO := E_q(\log p(y | \theta)) – KL(q(\theta) || p(\theta)) 
Implying that another lower bound on the KL would be greater than the ELBO and thus not necessarily a lower bound on the log marginal likelihood.

3)	Bottom of page 7: “The posterior is used only for fully connected layers…” Why is this the case? Alternative Bayesian methods can be used on the full network.


Minor Comments:
-	Please attempt to use the math_commands.tex in the ICLR style file to write math. This makes the math standardized throughout.
-	Figure 1 is especially unclear and the legends and captions should be made considerably larger. Zooming in at 800% seems to be necessary to even be able to make out which method is which. Consider, placing all methods on a single plot and then coloring/dashing them separately. Additionally, please compare to HMC on this task (and an equivalent RBF GP) to show the uncertainties from the “gold standard” methods.
-	Please additionally make the figure legends larger for the rest of the figures. 
-	What is the meaning of time in Figure 2? I assume that it means training time, but it is not especially clear from the caption. With that being said, I think that it is quite interesting that the marginals do seem to look multi-modal, even at the end of training.
-	Line before Eq 7: please use \citet if possible to cite Gupta & Nagar, rather than stating the clunky Gupta et al. \citep{Gupta & Nagar}.
-	Eq 2: please do not use \hspace{-…} to save space in equations, so that the equation does not overwrite lines before.
-	Page 2: “Deep Ensembles have lacked theoretical support …” In order to make this claim, you must first explain why one ought to be Bayesian in the first place. See above for the history of interpreting ensembling from a Bayesian perspective. There is no a priori reason why one would not just wish to have frequentist justifications of ensembling (and potentially even calibration).
-	Please do not capitalize Ensembling or Variational Inference throughout.

References:
Clyde & Iversen, Bayesian model averaging in the M-open framework. In Bayesian Theory and Applications, 2013. DOI:10.1093/acprof:oso/9780199695607.003.0024

He, Zhang, Ren & Sun, Deep Residual Learning for Image Recognition, CVPR, 2016. https://arxiv.org/abs/1512.03385

Hron, Matthews & Ghahramani, Variational Bayesian dropout: pitfalls and fixes, ICML, 2018. https://arxiv.org/abs/1807.01969

Miller, Foti & Adams, Variational Boosting: Iteratively Refining Posterior Approximations, ICML, 2017. https://arxiv.org/abs/1611.06585

Newton & Raftery, Approximate Bayesian Inference with the Weighted Likelihood Bootstrap. JRSS:B, 1994. https://www.jstor.org/stable/2346025?seq=1#metadata_info_tab_contents

Newton, Polson & Xu, Weighted Bayesian Bootstrap for Scalable Bayes, 2018; https://arxiv.org/abs/1803.04559

Rubin, The Bayesian Bootstrap, 1981. Annals of Statistics. https://projecteuclid.org/euclid.aos/1176345338

Yao, Vehtari, Simpson, & Gelman, Using Stacking to Average Bayesian Predictive Distributions, Bayesian Analysis, 2018; https://projecteuclid.org/euclid.ba/1516093227

review_title: Official Blind Review #3

Meta-review: This paper proposes to use mixture distributions to improve uncertainty estimates in BNNs. Ensemble methods are interpreted as a Bayesian mixture posterior approximation. To reduce the computation, a modification to BBB is provided based on a concrete mixture distribution.

Both R1 and R3 have given useful feedback. It is clear that interpretation of ensemble as a Bayesian posterior is well known, and some of them also have theoretical issues. The experiment to clearly comparing proposed mixture posterior to more commonly used mixture distribution is also necessary. 

Due to these reasons, I recommend to reject this paper. I encourage the authors to use reviewers feedback to improve the paper.
Decision: Reject