Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a software library for scalable Laplace approximations in PyTorch, enabling more efficient implementations of all of its various forms and variants (from KFAC approximations to marginal likelihood optimization to different approximations at test time).
main_review: tldr: I tend to weakly reject this paper because I'm really ambivalent about accepting what is essentially a research software package with experiments.

## Originality

*Are the tasks or methods new?* Not really, the software package is. The methods themselves have been around for a while.

*Is the work a novel combination of well-known techniques?* The software package is.

*Is it clear how this work differs from previous contributions?* The authors do a good job of distinguishing their implementation from others.

*Is related work adequately cited?* Mostly, ref [71] is mis-cited and uses HMC (although they compare to SGMCMC as well), while the GGMC method of [Garriga-alonso and Fortuin, AABI, '20](https://arxiv.org/abs/2102.01691) probably also deserves a citation due to its recent popularity.


## Quality

*Is the submission technically sound?* Yes, I think so. The software package seems pretty well put together.

*Are claims well supported (e.g., by theoretical analysis or experimental results)?* 
- Fig 5 is a pretty unfair comparison, as, deep ensembles, HMC and SWAG are using sampling (either during train / test or both), while the Laplace approximations studied use deterministic predictions. 
- The comparisons between (HMC, SWAG, and LA*) seem a lot more mixed for OOD detection from Table 1 than is depicted in the paper. It looks like LA* is mostly significantly less confident (some type of extra tuning?) than any method, but that this doesn't translate into better AUROC. 

*Are the methods used appropriate?* Mostly so given this space, a few comments:
- I would have preferred to see comparisons to full Hessian / Fisher Laplace approximations on say MNIST since this should be doable (implement scalable Hessian vector or Fisher vector products and then use a low rank approximation). This shouldn't be too difficult in the authors' software framework and might be a convincing demonstration of the modularity a bit more.

From what I can see, you either use a low rank KFAC Fisher approximation or a last layer one. A dense approximation would be helpful to see if it provides improved benefits.

- Given the high amount of modularity in MC approximation, one pretty clear missing piece of comparison is that of probit approximation + SWAG (or Laplace bridge + SWAG but it Laplace bridge seems to be dominated by other methods so less necessary). After all, in Fig 5 the wall-clock time is devoted to the computational cost of the number of samples...
- The comparisons to CSGHMC in the appendix should absolutely be moved to the main text, and shouldn't be buried that deep.

*Is this a complete piece of work or work in progress?* Complete work.

*Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?* Pretty much.

## Significance

*Are the results important?* Yes, if this gains steam, then it'll probably be very useful.

*Are others (researchers or practitioners) likely to use the ideas or build on them?* Yes, releasing a software package for scalable laplace approximations should definitely be useful.

*Does the submission address a difficult task in a better way than previous work?* Not really.

*Does it advance the state of the art in a demonstrable way?* Yes, I think so.

*Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?* 

## Discussion & Questions

I'm very ambivalent about accepting this paper because almost the entire contribution is the software package. Included in the citations are references for research works that do, give or take maybe one experiment, most of the comparisons in this paper. My score is pretty much based on this ambivalence, with some concerns about comparisons addressed above.

Thinking some more about this, there's fairly few research software packages (essentially) that end up published particularly because it's so difficult to evaluate software packages. From my understanding, the few packages that have ended up published ([Autoconj](https://arxiv.org/abs/1811.11926), [neural-tangents](https://openreview.net/forum?id=SklD9yrFPS)) have included what is effectively state of the art engineering capabilities as part of their contributions.

Overall, I really like the message of the paper and think that it could be a boon to the community to see this type of demonstration published. First, I like the comparisons demonstrating that something as straightforward as Laplace approximations (and really even last layer ones) can easily outperform variational methods. Second, I like the demonstration that MC sampling is often outperformed by deterministic probit style approximations.

### Writing / Minor Comments

- I really dislike that I'm saying this, but please editorialize your captions some more. Include a bit of takeaway for each figure caption --- at least "what is the one sentence summary of this figure?"
- Example: Fig 7. Permuted MNIST: all Bayesian approaches outperform MAP estimation in continual learning, with LA-KFAC performing the best, closely followed by VOGN.
- The caption on Fig 3 is especially rough.


-------

## Review of Software Package

After discussion with the area chair, I decided to take a more detailed look at the software package, which seems to be of reasonably high quality. A quick review of the package itself is here:

I thank the authors for taking the time to develop the package and for putting a bunch of different Laplace approximations all in the same place in a mostly modular fashion. However, I think that it's still pretty oriented around research code, rather than practicability, but this might be a personal feeling. Ultimately, I'm still not sure that people would use it outside of their current implementations of Hessian vector products or their current implementations of KFAC, without a lot of advertisement, although as a researcher, I'd probably take a look at it.

- Can you please describe what the (presumably anonymized) backend `asdfghjkl` is?
- While the implementation is described as modular, from a peek through the package, it's not clear to me how one would implement full rank Hessians via say the torch functional interface (https://pytorch.org/docs/stable/generated/torch.autograd.functional.hvp.html), rather than backpack? Would this require setting up an entirely different backend in order to do so?
- Are there any restrictions on layers that the backpack backend has? I recall that originally backpack couldn't handle some types of convolutional layers (I believe).
- As a comment, I find that the package is rather focused on KFAC style approximations, rather than other types of approximations (for example, those based around low rank approximations of the Hessian / Fisher defined via matrix vector multiplications or even layer-wise versions of these). Thus, it seems tricky to see how to extend beyond these types of approximations, making the package more limited in scope than is promised.
- `torch.kron` is in pytorch now, might want to update to that.




limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper discusses Laplace approximations in the context of approximate Bayesian inference for deep neural networks. 
The Authors propose a new library, laplace, that allow practitioners to easily extend loss-trained models (MAP) to Bayesian. 
The new library allows also the user to choose different approximations for the Hessian, different strategies for the Bayesian extension (e.g. by considering all or just a subset of weights) and a variety of procedures to choose/learn the hyper-parameters.

main_review: 
Methodologically the contributions of this work are very thin as the Laplace approximation is a well known and simple way to do approximate Bayesian inference.
Even the application of LA to deep neural networks is not new (see [21,25] in paper).
At the same time, I agree with the Authors when they argue on the poor adoption rate of LA, especially when compared against more popular variational and MCMC methods, and it's true that this might be due to missing reference implementations easily available.

For this reason, I have to positively evaluate the empirical benchmark, the analysis of different approximations and the source code (which is well documented and which promises to be very user-friendly).
Realistically, this paper will not be groundbreaking in terms of contributions but it provides good baselines for future research in BDL.
 
I was playing a bit with the code and the 1D regression example and overall it looks easy to tinker the different choices/parameters. 
I have found a major limiting design choice of the library, though: unless I've missed it in the code/documentation, it seems that only Gaussian prior can be used. Indeed it looks like the Gaussian prior is hard-coded in implementation of the approximation (and of the marginal likelihood).
In practice, there is evidence that heavy-tail priors (like student-t) can deliver better performance than Gaussian [a,b], and the fact that there is no option for the user to choose the prior is a problem. 
Same comment applies also for choice of the likelihood (only Gaussian and multi-class).

## Minor
One minor remark: it would be better to have a minimal version for the requirements. I had an older version of PyTorch installed and some functions were not available, causing the script to fail (the setuptools didn't update the packages due to the missing minimal version).

[a] Tran et al. All you need is a good functional prior.
[b] Fortuin et al. Bayesian Neural Network Priors Revisited

limitations_and_societal_impact: Limitations are only partially addressed.
In my opinion the is one important discussion that is missing and it regards the memory consumption, for CPU and GPU implementations. 
All Bayesian methods are known to be extremely challenging for both timing and memory usage: the Authors do a good job from the timing perspective (see Figure 4) but they neglect to address this other point of view.
As a future practitioner, especially if I'm working with constrained resources, e.g. very small GPU or edge devices, I would be very interested in a proper comparison between approximations/performance versus memory usage.
This would be very helpful to really understand all the different trade-off I should consider for if I intend to use LA.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors bring to light the recent progress in approximate Bayesian inference via the Laplace approximation, and note that despite its simplicity, its adoption has not kept pace for modern deep learning methods. The work describes a consolidation of recent work into a user-friendly PyTorch-based software library called `laplace`. LA can be applied post-hoc to existing deep learning models. With this package in place, the authors provide practical recommendations for reasonable defaults when used in practice.
main_review: I think this paper is very well-timed given the modern evidence of Laplace approximations for Bayesian inference being effective. As the authors identify, despite its conceptual simplicity, the developments have not kept pace with modern times in terms of easy-to-use software packages. In that sense, this is much appreciated. I have specific comments and question below.

- In terms of writing, while the introduction is clear, but for a paper that aims to convince the reader about the simplicity of implementing scalable LA, Section 2 does not inspire confidence. I think this may only be a matter of organizing and contextualizing the approximations available better. 

- To me, the most important part of the paper is the experiments section. I think the most important aspect missing in the study is the sensitivity of the approximations. How sensitive is the final performance to the Hessian approximations? Clearly, this has to be consequential?

- Something that would also be interesting to see with this broad-scoped analysis is whether we see the Occam's razor effect during the marginal likelihood optimization still happen even with the approximation?

- Another important point worth noting is that Multi-SWAG observed improvements using mutli-modal approximations of SWAG. Have the authors investigated this with Laplace approximation? Given that Laplace approximation is a finer approximation of the rather empirical curvature approximation with SWAG, do we really get gains there? Is it worth using LA to more finely capture geometry?

Overall, this is well-timed paper to remind the community of important progress in scalable LA. I do think, however, that there are major missed opportunities in terms of the qualitative analysis of the method beyond benchmarking (which is no doubt diverse and reasonably exhaustive), especially in terms of the sensitivity and other qualitative improvements that LA provides over Deep Ensembles or SWAG. I will vote for a weak accept, but I would be more than happy to be convinced otherwise since this is an important effort towards consolidation of scalable LA.

### Minor Comments

- For Figure 3, it may be helpful to the eye to have both graphs with the same scales to make comparison visually easier.
- It would be good have a detailed caption summarizing the message in Figure 4 (and if space is a constraint, perhaps move bulk of the explanation to the figure).
- Lines 31-32 should mention SWAG [1], which appears to satisfy good empirical performance at the cost of a single network. The authors do compare to SWAG later as well anyways.

[1]:: https://arxiv.org/abs/1902.02476

---

**UPDATE**: See comment below. Updating score from 6 to 7.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers the problems of approximate Bayesian inference in deep learning. The paper argues that the Laplace approximation does not receive the attention it deserves in the community and that the Laplace approximation is a good trade-off between simplicity, computational complexity, and predictive performance. The paper then provides a review of the relevant theory and the relevant decision for setting up Laplace approximations in deep learning. The main contribution of the paper is a software package for computing Laplace approximations based on pre-trained model. The paper concludes with a large empirical study comparing Laplace approximations to a range of other relevant methods.

main_review: The paper addresses an important problem - how to make Bayesian inference feasible for large-scale deep networks? I really enjoyed reading the paper. While the paper does not contain much novelty in terms of methodological development, I believe this paper and the software package could be of great interest to many researchers and practitioners in the field. 

The first sections of the paper (introduction, background on the Laplace approx, description of the software) are in generally very well-structured, well-written, and easy to follow. However, I found the structure of the experimental section quite confusing and it took me a while to decipher the information because of this. For example, the authors refer to Figs. 3, 4 and 5 in section 4.1, but all the abbreviations used in the figures are not introduced before section 4.2. Besides, there are several experimental details that are unclear to me (see comments below). Therefore, it think the experimental section could benefit significantly from a restructuring.

The claims of the paper are substantiated to a reasonable degree based on theoretical and empirical evidence from the literature as well as by a fairly comprehensive empirical study based on several different datasets. However, it would have been even more interesting if the authors had chosen to include more adhoc methods like Monte Carlo Dropout, etc., and more advanced VI methods for completeness. But that being said, I think the empirical results are already quite extensive.



Other comments:

- Figure 3: What does the individual dots in Fig. 3 represent?

- Figure 4: It is still a bit unclear what exactly the axis in (b) and (c) are showing. Please elaborate on this. How is the shift intensity defined and what is on the y-axis?

- What is the definition of "Confidence" defined in Table 1?

- In several experiments, the authors compared results for models with fixed hyperparameters vs learned hyperparameters. However, it is very unclear to me which parameters the authors consider to be in the set of hyperparameters. For example, which quantities are considered as hyperparameters in Fig 3? And how are they initialized when learning them through the approximate marginal likelihood?

- The paper provides a set of general recommendations based on the specific experiments. It would be nice if the authors would include a discussion of how well those recommendations will generalize to other types of problems.

- In line 97 the authors argue that it can be sufficient to approximate a subset of weights using the Laplace approximation because it has been shown that some networks can be pruned without sacrificing test accuracy. However, I fail to see how this justifies only modeling uncertainty for a subset of the weights. When pruning networks, you don't simply ignore the uncertainty - you completely remove the weights. Could the authors please elaborate a bit on this?

**Post rebuttal**
Updated score from 6 to 7.
limitations_and_societal_impact: Not applicable
review_title: null

Meta-review: A well-written paper proposing a software package.  The paper additionally gives a compact review on Laplace approximation (LA) for DNN  and benchmark experiments where LA works better or is comparable to more accurate approximation methods.  The software is easy to use but reviewers have concerns about its flexibility (e.g., prior is hardcoded and cannot be changed) and maintainability.  We hope that the authors would try to keep "Laplace" useful for the community.
Decision: Accept (Poster)