Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper defines bayesian neural network models with hierarchical prior distributions. These models are trained efficiently using collapsed variational bounds, in which some of the variables of the hierarchical ELBO are integrated out. The authors show that these models outperform their hierarchical counterparts in a number of experiments and standard benchmarks.
main_review: I found this paper to be a very interesting read, and a solid contribution to the fundamental research question on how to best model uncertainty in deep neural networks.

Using hyperpriors makes a lot of sense to me, since as it is also shown in the experiments, fixing the values of the hyperparameters as normally done is not a robust solution.
To the best of my knowledge, this is the first work that extends MF-VI for bayesian deep learning using ideas on collapsed bounds previously introduced for hierarchical bayesian models.
By using distribution from the exponential family the authors show a scalable and effective way to learn such models.

Because of its nature, the paper is quite dense from the theoretical point of view. In most parts derivations are well explained, but there are some places that require clarification:

1. line 87: "where for clarity we omit. the parameterization.." this actually caused me some confusion since I did not know why you would do this. You should say that you do it since it will be implicitly defined.

2. line 107: "straightforward to derive..." missing a reference for readers not familiar with MF-VI (Bishop's PRML book could be a good one for example)

3. line 146: why are there 2 different priors over W?

4. line 148: distribution G in p($\tau$) is not defined, i assume it is a Gamma distribution?

5. line 215: what is p? is it the dropout probability?

Overall, I believe this paper can have a big impact on the area of bayesian deep learning, so I vote for accepting it.


___________________________________________
Reply to author's rebuttal.
Thanks for your reply, I will leave the score unchanged and argue for acceptance.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces an improvement over the typical ELBO by deriving a tighter bound to the log marginal likelihood. It further shows that the ELBO can be used to learn appropriate hyperparameters which can help in situations where MF-VI would have underfit. A few variants of this method are derived where some combination of prior means/variances are learned. These methods were shown to be competitive with and at times outperform existing methods in experiments on standard benchmarking datasets. 
main_review: Other work in this area has largely centered on improvements on gradients of the ELBO this work focuses on an improvement on the bound of the log marginal likelihood. These improved bounds are shown mathematically with sufficient detail. This work builds on existing ideas from Bayesian hierarchical models. The idea of learning hyperparameters seems to be related in spirit to empirical Bayes applied to neural networks. [1]

It is hard to read the algorithm as it is currently written in paragraph form; breaking this into its own section/some sort of indentation would help.. The provided example right after this was illustrative and helpful.

Just above equation (2) there is a missing period “.”: “0, ELBO lower bounds log marginal likelihood $\log p(D|µ_p,σ^2_p) ≥ L(µ_q,σ^2_q;µ_p,σ^2_p)$ Replacing” → “0, ELBO lower bounds log marginal likelihood $\log p(D|µp,σ2 p) ≥ L(µ_q,σ^2_q;µ_p,σ^2_p)$. Replacing”


[1] G. Casella, “An Introduction to Empirical Bayes Data Analysis,” The American Statistician, vol. 39, no. 2, pp. 83–87, 1985, doi: 10.2307/2682801.


----------------------
Thanks for the reply. After further discussion and thought I have lowered my rating to marginally above the acceptance threshold. The idea is good but the structure and writing is weak.


limitations_and_societal_impact: The authors have not addressed this, but this decision is appropriate for the content.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Although marginal likelihood can be used to select hyper-parameters, the ELBO is generally regarded as too weak a bound to do this effectively.
The authors propose applying collapsed variational bounds to variational inference of BNNs.
They show why this maintains the tightness of the bound for parameter inference while also providing a tightish bound for hyper-parameter optimization by exploiting the independence of the two.
They empirically show that their method performs well in a range of settings, including in tests of ood uncertainty.
main_review: # Overall review
I found this paper difficult to assign a score to, and I remain quite uncertain.
On the one hand, I believe that you have proposed a sensible approach which fixes a moderately important problem.
You have also provided extensive experimental support for your hypotheses.
On the other hand, I found the paper quite hard to read and somewhat poorly organized.
I feel that it would benefit a lot from being redrafted in a way that makes it clearer what its contributions are and how they are being examined.
For that reason I am currently recommending borderline rejection, but I could easily be persuaded upwards if other reviewers think particularly highly of the significance of the work.

# Major comments

There is a lot of redundancy.
The abstract could probably be pruned by a third.
Much of the introduction is relatively boiler-plate and does not focus on the specific contribution that *you* are making.
I did not quite understand what the role of the "Preliminaries" section was. Probably you could take a small part of this and place it at the start of what is now S3.
Similarly, I would reserve a discussion of over-pruning until the experiments section and make a note in the introduction that your method resolves an over-pruning issue in VI.
When I read it I had no idea why this was being mentioned and Figure 1 is somewhat a distraction at this stage.

I think some of the details of the derivations in S3 could be moved to an appendix, with more space devoted to an explanation of why you are doing things and what the intuitive purpose is and less on the details.

The captions on many figures seem incomplete, with only some subfigures discussed explicitly.
The figures are all far too small and difficult to read.

I would be inclined to pick your experiments more carefully for the main body so that you can devote more time/space to explaining and interpreting each thoroughly.
They feel rushed and crowded.

# Minor remarks

Regarding the introduction: I'm easily persuaded that a tighter bound to the marginal likelihood would be good, so a lot of this background isn't super necessary.
I would have liked more of a sketch of what your approach is going to be.

Please make your references hyperlinks.
Especially when you use a numerical citation style it is almost impossible to figure out what you are citing otherwise.

The spacing of the paper is very tight, perhaps some of the style-sheet was accidentally overridden?
This makes it much harder to read.
limitations_and_societal_impact: These are adequately addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper applies the idea of the collapsed variational bound studied by Teh et al., 2007 to BNNs. The bound is constructed by assuming that VB's Gaussian prior mean and variance are also part of the inference. (In contrast, one commonly sets them to some fixed values, e.g. $\mathcal{N}(0, 1)$.) The collapsed bound seems to have a similar computational cost but is tighter than the standard ELBO.

Extensive experiments show that the proposed objective works well, even on large networks, the regime which mean-field VB tends to underfit. It is also shown that the objective can mitigates the over-pruning effect that plagues VB.
main_review: I think the proposed method is interesting and is backed up by extensive experiments. This could be the answer to the difficulty of training VB due to hyperparameter selection. Nevertheless, I think the experiments can be expanded more, esp. regarding uncertainty quantification, OOD detection. Currently, the authors only show MNIST-FMNIST OOD detection with MLP. I think this is inadequate and the authors should use the standard OOD detection suites (MNIST, FMNIST, SVHN, CIFAR10, CIFAR100 in-distributions against 3+ OOD test sets) with a more commonly used network instead (ResNets, Wide-ResNets).

With that being said, the major issue with this paper is that it is *not* well-written. There is no paragraph break *at all* in this paper, making it a continuous blob of text. (Instead, the authors used \paragraph ``with its whitespace removed.) Maybe this due to some "LaTex hacks" that the authors used to make the text fit the page limit. If this is so, I suggest the authors not use them.
All in all, the presentation of this paper needs to be improved significantly and I urge the authors to do so meticulously. 

**Questions:**
* Based on the discussion in L.80-84: How does the collapsed bound compare to EM?

**Suggestions:**
* Use paragraph break and use whitespace judiciously
* Use display math judiciously, instead of putting long formulas inline
* Put L.120-123 inside LaTex's algorithm env. Inline, it is very confusing to read
* Figures are very hard to read. I suggest the author to use tikz

**Minor suggestions:**
* L.46: "..., *the* ELBO lowers bound ...", also add a period before "Replacing ..."
* Fig. 3: Add y-label to the second row
* Tab. 1: What is "test ER"? (It's discussed in the next page, but it should be much closer to Tab. 1)
* Tab. 2, 3: Unclear which one is test-NLL, which one is test-ER.


---

**Post-Rebuttal**

While I like the idea, I still think that it would be beneficial for the paper to be given more careful treatment in its writing and presentation---this will ultimately change the paper substantially and thus, I believe, is more suitable for resubmission. My updated score (weak reject) reflects this ambivalence.
limitations_and_societal_impact: The authors discussed the limitations of their method in the paper.
review_title: null

Meta-review: This paper tightens the standard variational bound optimized in Bayesian Deep Learning by drawing on collapsed variational inference. The authors consider the prior parameters as latent variables and derive a hierarchical variational inference procedure in which the top-level latent variables are marginalized out. The authors showed strong empirical performance of their method compared to a variety of baselines.

The paper provides a non-trivial methodological contribution to the Bayesian deep learning community. Its mathematical derivations are not simple but generally well-explained (the authors sometimes don’t clearly distinguish between latent variables and variational parameters at times).  The main point of criticism was suboptimal structuring and writing. I strongly encourage the authors to make their paper more accessible by following the detailed advice that the reviewers provided. Overall, this is a very good paper. 
Decision: Accept (Poster)