Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose to approximate the posterior over weights by minimizing KL(p||q) where p denotes the true posterior and q denotes a diagonal Gaussian approximation. The authors collect multiple samples from the posterior by running SGLD and learn the variational approximation by updating the mean and variance of the weights in an incremental fashion (which removes the need to store all previous samples). In practice, the authors use Adam + small gradient noise with fixed standard deviation instead of a properly tuned SGLD. Overall, it's a quite simple idea to learn a distribution over weights and   looks promising for outlier detection. It under-performs compared to explicit ensemble, however the proposed method is more memory efficient and requires just training 1 model.

Questions:
- Do you use all the samples or do you have some sort of "burn-in" phase where you ignore the first few samples as they could be quite bad? Did you try "thinning" the samples, i.e. use only every K-th sample or so?
- how do you choose d_x in (4)?
- There has been some work on expectation propagation for Bayesian neural networks which also minimize KL(p||q). See "Stochastic expectation propagation" by Li et al. 2015 and "Distributed Bayesian learning with stochastic natural-gradient expectation propagation and the posterior server" by Hasenclever et al. 2015. It would be nice to discuss connections to these works.
- The snapshot ensemble code is available online at https://github.com/gaohuang/SnapshotEnsemble. It'd be worth understanding why/when it underperforms.
review_title: Interesting idea

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose to approximate inference in Bayesian neural networks by minimising the KL divergence between the true posterior and the approximating distribution KL(p|q) (rather than the usual VI objective KL(q|p)). The objective used (known as EP's KL objective) is known to result in better uncertainty estimates. The authors then propose to approximate the resulting intractable expectation over the posterior by Monte Carlo sampling, with samples produced from SGLD. This neat idea of summarising MCMC approximations using sufficient statistics originating from a variational approximation has some interesting implications, for example avoiding the memory requirements of storing many samples of the model parameters. This technique is then used in outlier detection, and demonstrated to slightly improve over MC sampling with dropout VI, and slightly under-perform compared to full ensembles that replicate many copies of the parameters (requiring much more memory to store the models). 

My biggest concern with the paper, though, is that the authors don't actually use SGLD in the experiments since they "found [SGLD hyper parameters] hard to tune", and used the Adam optimiser instead. This means that the samples accumulated are not from the posterior of the model at question, but just *a* collection of network weights, casting a heavy shade on any possible interpretation of the experiment results. 

I like the ideas presented in the submission, and would encourage the authors to repeat their experimental evaluation, also comparing their method to more sensible baselines such as fully factorised Gaussian VI inference in Bayesian neural networks (which is closely related to the suggested technique).
review_title: Neat idea

Decision: Accept