Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies empirically the current limitations of posterior estimation in Bayesian Neural Networks, focusing on Monte Carlo integrations as a simple golden standard and comparing its problems and limitations to the more recent techniques based on network linearization.
The authors then propose a post-hoc posterior refinement technique for posteriors to be estimated through MC integration and validate it empirically against the other analyzed techniques.
strengths_and_weaknesses: Strengths
Overall, the paper is well written. The problem is clearly stated and so are the main approaches to it.
The proposed method is simple to understand and looks computationally cheap enough given the advantages it brings, and section 3 has some compelling arguments on why current methods should not be trusted.

Weaknesses
Section 3.3 feels confusing. It doesn't a good job in convincing the reader that linearization-based alternatives are bad enough to disregard a priori. Figure 5 failed in convincing me that the qualitative estimate of MC or MPA was either better or worse than the competitor.
Empirically, the range of experiments is satisfying, however the fact that the tested models are only a Le-Net and a WideResNet leaves the reader wondering how much the results would change with different type of architectures, e.g. even a simple Gated Recurrent Unit on Emotions or Newsgroup-20 or a perceptron on the same proposed datasets.
questions: 1. Why Figure 5 should convince the reader that MPA is underconfident?

2. Why figure 2.c should convince the reader that linearization-based approaches are being overconfident? Is there a ground truth I'm missing?

3. Did you try different flows? What should convince the reader that this is a design choice not worth discussing or that it does not have a big impact on the final performance of the method?

**********---------------------**********

After the author's response I've increased my score to 6.
limitations: Big picture limitations were appropriately addressed.
However, I think the paper would greatly benefit from a more structured section or paragraph detailing the computational limitations, especially when compared to other baselines and related work. This doesn't need to be an empirical study of all methods computational costs, but just a clearer outline of the costs implied by each of the mentioned methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: First, the authors of this paper propose an experimental analysis of failure modes of a given way of building Bayesian Neural Networks (Bayesian NNs, BNNs), namely, MC integration over samples from an approximate posterior. 
Then, they propose a cheap and accurate method, which avoids these failure modes: replace the last layer of a NN by a variational layer, whose distribution is generated by a trained Normalizing Flow (which spans a much larger space of distributions than the commonly used product of Gaussians), then perform MC integration on top of that. They compare experimentally their method to the "gold standard" HMC.

## Failure modes of MC methods

The authors recall (and prove experimentally) that MC methods do not lead to good predictive performances if the underlying approximation of the Bayesian posterior is too rough. For instance: Laplace Approximation (LA) and Variational Bayes (VB) ; notably, these approximations of the posterior assume Gaussian and independent parameters.

One of the pitfalls is also the lack of exploration of the parameter space: deep ensembles perform way better, even with very few samples.

## Last-layer Bayesian NNs

The authors emphasize the idea of building "Bayesian" NNs only by replacing the usual last layer by a *Bayesian* last layer. Thus, the proposed methods relies heavily on "Laplace redux–effortless Bayesian deep learning" (Daxberger et al., 2021) [6]. Actually, the authors propose an improvement of [6]: instead of performing a simple LA on the last layer, the authors add Normalizing Flow (NF) to it, in order to increase the variety of reachable distributions.
strengths_and_weaknesses: ## Strengths

The authors motivate their approach by an analysis of failure of common methods. So, the proposed method is justified, at least experimentally.

## Weaknesses

The main weakness of this paper is its significance, compared to what is already known and used.

For instance, it is already known that sampling NNs from the same "basin of attraction" is not working very well compared to HMC and deep ensembles. So, the poor results of VB + MC or LA + MC in Section 3 are not surprising. For instance, see the exhaustive paper "Bayesian deep learning and a probabilistic perspective of generalization", Wilson and Izmailov, 2020.

About the proposed new method, this is a variation of one of the methods proposed in [6], which is actually an extensive study of the LA in many setups, with practical consideration and available code. 

## Other

The experiments are limited to F-MNIST and CIFAR-10 and CIFAR-100. Since the results (tables 3, 4, 5) are clear and convincing on many aspects (calibration error, distance to the gold standard HMC, OOD detection), this should not be a problem.
questions: The phenomenon observed in [6] and by the authors of the current paper, i.e., the importance of the last layer when doing approximate Bayesian inference, is not understood:
 * According to the authors, which set of experiments could help us to understand it? 
 * Is this a way for the NN to calibrate itself automatically, which would be impossible if the distribution of the final layer weights was constrained to Gaussians? 
 * How would the NN react to other constrained distributions (Laplace, Weibull, etc.)? Is this related to the thickness of the tails?

The theoretical questions brought by the experiments done by the authors are a strength of the paper. These points should probably be discussed somewhere in the paper, with possibly some preliminary experiments.
limitations: -
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers the problem of posterior approximation in Bayesian neural networks. First, the authors show that the accurate approximation with Monte Carlo requires a large number of samples, far exceeding the number of samples typically used by practitioners. Next, the authors show that (1) a large number of samples from a poor posterior approximation and (2) crude analytic approximations to the MC integral both lead to suboptimal performance in Bayesian neural networks. Finally, the authors suggest to use a variational refinement of the Laplace approximation with a normalizing flow to approximate the posterior in last layer BNNs, and show good performance. 
strengths_and_weaknesses: **Strength 1**. The authors show many interesting observations, e.g. Figure 3 suggesting that 100 samples is insufficient for an MC approximation to even a very simple model, and Figure 4 showing the discrepancy between the MC approximation and analytic approximation to the posterior predictive.

**Strength 2**. The proposed approach is practical, as it is a cheap post-processing step to a pre-trained neural network. The empirical results are promising. 

**Strength 3**. It is nice that the authors included MMD distance estimates in Table 3 and showed that the refined posterior is closer to the HMC posterior. 

**Weakness 1**. The narrative of the paper is not very focused. In particular, it is not obvious, why the paper discusses the number of samples required for an MC approximation and the quality of analytic linearization-based approximations. The proposed method doesn't follow logically from these experiments, and the connection between these two parts of the paper is relatively weak.

**Weakness 2**. The empirical evaluation is not very strong. In terms of in-distribution performance, the authors only compare the proposed method to last-layer baselines, and achieve an improvement in uncertainty / NLL, but not accuracy (according to appendix Table 5). The results on out-of-distribution detection (appendix Table 7) show improvement over full parameter-space Bayesian methods, but do not include specialized out-of-distribution detection methods.

Overall, the paper shows interesting observations, and some promising results. However, the story feels disconnected, and the empirical observation doesn't prove that the proposed method will be useful for practitioners.
questions: **Question 1**. Where do you expect the proposed method to be most useful? It appears that the primary strength is improving the uncertainty calibration for deep neural networks. How does it compare to specialized calibration methods, e.g. temperature scaling? 

**Question 2**. How much is the proposed method affected by the size of the last layer, and the number of classes in particular? Is it cheap / computationally feasible to apply it to an ImageNet-scale model? 
limitations: No issues
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the problem of scalable Bayesian inference for large-scale neural networks. The core idea of the paper is to first do a (last layer) Laplace approximation of the posterior of the network weights and then refine it using normalizing flows. That is, the paper proposes to use a Laplace approximation as base distribution in normalizing flows instead of a fixed distribution, e.g. a standardized Gaussian. 

The contribution of the paper is as follows:
- a discussion of shortcomings of common strategies for computing predictive distributions (by averaging over the approximate posterior). Specifically, they contrast linearization and probit approximations with Monte Carlo integration.
- propose to use a Laplace approximation as base distribution in a normalizing flows approximation
- a set of thorough numerical experiments to support the claims.

strengths_and_weaknesses: The paper proposes to combine two existing approximate inference techniques: Laplace approximations and normalizing flows. The idea is not revolutionary, but it is simple, effective, and original to the best of my knowledge. The paper is very well-structured, well-written and easy to follow. The claims of the paper are not justified theoretically, but they are supported by a set of high-quality numerical experiments.


There are a few things that could be slightly clarified:

- In my perspective, the quality of an approximate posterior predictive distribution depends on two things: 1) the quality of the posterior distribution of the model and 2) the accuracy of the integration method used for averaging over the posterior distribution. Therefore, using high-fidelity integration methods (e.g. MC integration with S -> infinity) will not save you if the quality of the posterior approximation is sufficiently poor. I think this is in line with the message of the paper, but it appears a bit convoluted in the discussion of the effect of the MC sample size on the accuracy of the predictive distribution (in my opinion)
- The difference between the top and bottom plot in Figure 5 could be emphasized visually and/or in the text. It took me a while to realize the difference.


questions: - Can the explanation described in lines 190-200 be further justified/substantiated? For example by inspecting/visualizing the pairs of approximate posterior and approximate posterior predictive in the two cases?
- How much tuning was needed for optimizing the flows? Did you use the same optimization procedure/search parameters for all problems?
limitations: Yes.
review_title: null

Meta-review: metareview: The paper proposes a method to refine Gaussian approximations of the posterior in Bayesian computations by using the normalizing flow. Such Gaussian approximations are usually cheap to obtain, via Laplace approximation or variational Bayes. The method proposed by the authors outperform standard MC approaches and is competitive with the most sophisticated ones (Hamiltonian MC), while cheaper.

The reviewers praised the experimental results. They also liked the nice explanations and illustrations of the failure of the standard MC approaches. Some remarks about the limited novelty of this discussion with respect to existing works (e.g. Wilson and Izmailov, 2020) were satisfactorily addressed by the authors during the discussion with the reviewers. Overall, the reviewers agreed that, while the writing of the paper could be improved in parts, the discussion and the method proposed in this paper are a nice contribution to Bayesian learning, and will be useful to the community. I will therefore recommend to accept the paper. I encourage the authors to take into account the comments of the reviewers (especially Rev. 5mzJ) when preparing the camera-ready version of the paper.
confidence: Certain
Decision: Accept