Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper uses recent progress in the understanding and approximation of curvature matrices in neural networks to revisit a venerable area- that of Laplace approximations to neural network posteriors. The Laplace method requires two stages - 1) obtaining a point estimate of the parameters followed by 2) estimation of the curvature. Since 1) is close to common practice it raises the appealing possibility of adding 2) after the fact, although the prior may be difficult to interpret in this case. A pitfall is that the method needs the point estimate to fall in a locally quadratic bowl or to add regularisation to make this true. The necessary amount of regularisation can be large as reported in section 5.4.

The paper is generally well written. In particular the mathematical exposition attains good clarity. Much of the mathematical treatment of the curvature was already discussed by Martens and Grosse and Botev et al in previous works. The paper is generally well referenced. 

Given the complexity of the method, I think it would have helped to submit the code in anonymized form at this point.There are also some experiments not there that would improve the contribution. Figure 1 should include a comparison to Hamiltonian Monte Carlo and the full Laplace approximation (It is not sufficient to point to experiments in Hernandez-Lobato and Adams 2015 with a different model/prior). The size of model and data would not be prohibitive for either of these methods in this instance. All that figure 1 shows at the moment is that the proposed approximation has smaller predictive variance than the fully diagonal variant of the method. 

It would be interesting (but perhaps not essential) to compare the Laplace approximation to other scalable methods from the literature such as that of Louizos and Welling 2016 which uses also used matrix normal distributions. It is good that the paper includes a modern architecture with a more challenging dataset. It is a shame the method does not work better in this instance but the authors should not be penalized for reporting this. I think a paper on a probabilistic method should at some point evaluate log likelihood in a case where the test distribution is the same as the training distribution. This complements experiments where there is dataset shift and we wish to show robustness. I would be very interested to know how useful the implied marginal likelihoods of the approximation where, as suggested for further work in the conclusion.

review_title: Competent work building on recent literature. Mixed results and perhaps not currently fulfilling its full potential as a paper.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper proposes a novel scalable method for incorporating uncertainty estimate in neural networks, in addition to existing methods using, for example, variational inference and expectation propagation. The novelty is in extending the Laplace approximation introduced in MacKay (1992) using a Kronecker-factor approximation of the Hessian. The paper is well written and easy to follow. It provides extensive references to related works, and supports its claims with convincing experiments from different domains.

Pros:
-A novel method in an important and interesting direction.
-It is a prediction method, so can be applied on existing trained neural networks (however, see the first con).
-Well-written with high clarity.
-Extensive and convincing experiments.

Cons:
-Although it is a predictive method, it's still worth discussing how this method relates to training. For example, I suspect it works better when the model is trained with second-order method, as the resulting Taylor approximation (eq. 2) of the log-likelihood function might have higher quality when both terms are explicitly used in optimisation.
-The difference between using KFAC and KFRA is unclear, or should be better explained if they are identical in this context. Botev et al. 2017 reports they are slightly different in approximating the Gaussian Newton matrix.
-Acronyms, even well-known, are better defined before using (e.g., EP, PSD).
-Need more details of the optimisation method used in experiments, especially the last one.
review_title: Novel idea, well-written, and supported with extensive experiments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a Laplace approximation to approximate the posterior distribution over the parameters of deep networks. 

The idea is interesting and the realization of the paper is good. The idea builds upon previous work in scalable Gauss-Newton methods for optimization in deep networks, notably Botev et al., ICML 2017. In this respect, I think that the novelty in the current submission is limited, as the approximation is essentially what proposed in Botev et al., ICML 2017.  The Laplace approximation requires the Hessian of the posterior, so techniques developed for Gauss-Newton optimization can straightforwardly be applied to construct Laplace approximations.

Having said that, the experimental evaluation is quite interesting and in-depth. I think it would have been interesting to report comparisons with factorized variational inference (Graves, 2011) as it is a fairly standard and widely adopted in Bayesian deep learning. This would have been an interesting way to support the claims on the poor approximation offered by standard variational inference. 

I believe that the independence assumption across layers is a limiting factor of the proposed approximation strategy. Intuitively, changes in the weights in a given layer should affect the weights in other layers, so I would expect the posterior distribution over all the weights to reflect this through correlations across layers. I wonder how these results can be generalized to relax the independence assumption. 


review_title: Good paper but what about novelty?

Meta-review: This paper gives a scalable Laplace approximation which makes use of recently proposed Kronecker-factored approximations to the Gauss-Newton matrix. The approach seems sound and useful. While it is a rather natural extension of existing methods, it is well executed, and the ideas seem worth putting out there.

Decision: Accept (Poster)