Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a change to an established approach (Sun et al., 2019) for performing inference in Bayesian neural networks, which works directly in function-space rather than weight-space. The previous work by Sun et al. (2019) derive an ELBO with an intractable KL divergence between processes, so to make the problem tractable, they estimate the gradients of this KL term using the spectral Stein gradient estimator. This estimator, however, has been shown to be less efficient for high-dimensional distributions. 
As an alternative approach, the current paper's authors propose a simple estimator of the KL term, which is based on a Taylor expansion of the random functions induced by BNNs and an MC estimate over context points from the input space.
The authors empirically show how their method leads to state-of-the-art uncertainty estimation and predictive performance on several datasets.

strengths_and_weaknesses: **Strengths**

This is an impressive paper; incredibly well-written and thorough, with a good and concise related work section. The proposed approach is a novel and significant contribution to the field, which will undoubtedly be of great interest to the NeurIPS community. 

The proposed approach is carefully derived in a way that makes it possible for the reader to follow along all the way. The paper contains a great empirical assessment with an impressive amount of details for making the work reproducible. When a setting or a choice of, say, a hyperparameter or distribution is not immediately clear, the authors carefully discuss the options. This might be the most polished and detailed paper I have ever reviewed.

The proposed approach also works well in practice, and several studies of the effect of hyperparameters and priors (in the supplementary) provide valuable insights.

**Weaknesses**

It's really quite hard to find any substantial weaknesses in the paper.
In my view, the greatest weakness is that the authors do not compare their approach against that of Sun et al. (2019) despite framing the entire paper as an improvement over this. It would have been interesting to see the two approaches compared side-by-side on problems that both methods can handle.

An additional (minor!) weakness of the proposed approach is that it introduces a few extra choices to be made and hyperparameters to be optimised compared to the method by Sun et al. (2019). The authors do, however, provide suggestions and thorough discussions of how to make these choices.

**Comments, typos, and various minor things**

* Line 100: $\mathcal Q_\theta$ -> $\mathcal Q_{q_\Theta}$.
* Line 111: The word "poses" doesn't seem to fit in this sentence.
* In the equation following line 138, the matrix $\mathbf S$ is only defined in the appendix.
* In line 175, if $\Sigma$ is the variance, I think the reparametrisation should be written (informally) using $\Sigma^{½}$ or using the Cholesky decomposition $L$.
* Line 207: $SM$ should be $SK$, I suppose.
* In lines 246-247, I find the phrasing of "predictions will be significantly higher" a bit confusing. I suppose it refers to the predictive uncertainty, which should be higher, but I'm not entirely sure.
* I suppose the ablation studies are actually more like hyperparameter tuning studies rather than actual ablation studies.

questions: 1. You mention that you choose a prior variance of 10, and based on the experiment in section 5.4 it is clear how this choice is seemingly the best. I am a little confused about whether this is the prior over weights of the BNNs or over functions they represent, however. I suppose it is over the weights, but the text keeps referring to the induced prior over functions. Which prior variance does $\Sigma_0$ refer to?

2. Further to the question above, if $\Sigma_0$ is the variance of the prior over the weights, did you try to estimate the induced variance over functions as well? Given that the datasets are normalised, an induced function-space variance of more than 1 would be peculiar.

3. In lines 154-157, you mention that, theoretically, the approximation to the KL divergence will be better for smaller variances. While a variance of 10 empirically works the best, did you investigate the quality of the approximation for this value? I'm wondering whether the variance of 10 might actually result in a rather poor approximation that happens to work well and whether this is somehow related to the cold posterior effect (this is more of a speculative/discussion point rather than a question and I might well be misunderstanding something here!).

4. Did you empirically compare the performance of your approach to that of Sun et al. (2019) on problems that are within reach of both approaches?

limitations: The authors have adequately addressed the limitations of their work. A potential negative societal impact has not been discussed, but it is not necessary given the paper's theoretical nature.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The manuscript proposes a scalable functional-space variational inference method by incorporating prior information. The main contribution of the paper is to propose a KL divergence estimator between a posterior and variational distribution over functions by approximating the first-order Tayler series expansion of the mean of the parameters.

strengths_and_weaknesses: The proposed approach seems intuitive and the proof seems reasonable. The results demonstrate the benefits of the approach.

Weakness:
-  The diagonalization approximation to the covariance matrices in Section 3.2 is a simplification that deviates from the original approach from Sun et. al. 
- The proposed method are presented only image classification problems.
-  Authors need to compare other approaches that scale functional variation inference to even larger models: please refer to the following manuscript.
"Carvalho, Eduardo DC, et al. "Scalable uncertainty for computer vision with functional variational inference." CVPR'2020

- Other approaches which are not functional variational inference but provide well-calibrated uncertainty estimation need to be compared.

Karandikar, Archit, et al. "Soft calibration objectives for neural networks." Advances in Neural Information Processing Systems 34 (2021): 29768-29779.
Krishnan, Ranganath, and Omesh Tickoo. "Improving model calibration with accuracy versus uncertainty optimization." Advances in Neural Information Processing Systems 33 (2020): 18237-18248.



 
questions: What are the potential ways to apply this approach to non-Gaussian priors?
How does the proposed method work for other problems such as Bandits, and Bayesian Optimization?
limitations: The results presented are convincing for the image classification task, but you need to compare against other variational inference approaches proposed in the literature. Some references are provided above.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a tractable method for function space Variational inference that allows the method to be applied to realistic real-world settings. The authors achieve this by means of two approximations. Firstly, rather than calculating a KL divergence between two posteriors for which we do not know the closed-form density, they calculate the KL divergence between locally accurate linear approximations to the posteriors, whose density *is* known. Secondly, they use a rough approximation to the supremum over the aforementioned KL divergence, which they argue is enough since it encourages the variational distribution to match the prior on a set of context points.
strengths_and_weaknesses: ## Strengths

**Originality**: the paper proposes a novel solution to a challenging and important problem in the BNN literature. I particularly enjoyed the idea of using local linearization to approximate the KL divergence between the approximate posterior and prior! 

**Clarity**: for the most part I found the paper well written and explained. There were a few issues, which I detail later.

## Weaknesses

**Quality**: While the experimental evaluation is quite extensive (many benchmarks and many baselines – very nice!), I do have some concerns and suggestions, which are also listed below. My main concern is that the experiments do not add to the understanding of the performance of the method–the relative strengths and weaknesses are not apparent, and the impact of the various approximations is unclear. There is also a lack of some important details that make it difficult to interpret the results of the experiments. 

**Significance**: I have a number of concerns about the approximations and assumptions in this work, which have not been addressed in the text. Without having these concerns addressed it is hard for me to make any judgement about the potential significance of the work. My primary concern is with the choice of prior. This concern and others are listed in the questions below. 

questions: ## Major

0. [Significance] To me it seems very strange to do function space inference when the function space prior does not have any correlation between inputs. Under this modelling assumption, we are saying that we do not expect one input to tell us anything about the other input. This issue is sidestepped in FSVI due to the fact that NNs have some smoothness properties and as such the approximate posterior does not treat all inputs as independent. Nevertheless, one should strive to have a correct model and then apply approximations which are hopefully not harmful, rather than have an incorrect model that is fixed by applying approximations. 

1. [Experimental] It would be great to have a qualitative comparison between FSVI and HMC for some of the to datasets, for example those in Figure 1. Providing only the fits for FSVI without a comparison isn't particularly illustrative, and since these problems are small, HMC can be run to get a (somewhat) good idea of the 'true' posterior. 

2. [Experimental] On a similar note, comparisons to Sun, et al. and Ma, et al. on some smaller settings (perhaps the UCI datasets used by Sun et al.). This would allow the reader to understand what is being gained/lost (other than improved tractability) when using FSVI rather than these previous approaches to function space BNNs.

3. [Signifigance] On lines 168-170, the authors suggest that the choice of context distribution can be viewed as a problem-specific modelling choice. I think that this is a very interesting idea, however, the idea is somewhat vague. Would it be possible to elaborate on this and give a few concrete examples? 

4. [Experimental/Significance] On lines 170-171, the authors mention that $S$ and $K$ are hyperparameters to be optimised on a validation set. How impactful are these hyperparameters? Do the authors have any concrete pieces of advice for choosing values/ranges? A sensitivity analysis would be very helpful. 

5. [Experimental] The ablation study for the choice of context distribution, while a great inclusion, was somewhat unhelpful because there was no accompanying discussion and the plot labels were somewhat mysterious. It would also be interesting to see the impact of swapping the context set and OOD dataset in the OOD results from Tables 1 and 2.

6. [Experimental/Significance] How restrictive is the mean-field variational assumption (c.f., line 174)? Would it be possible to include a small ablation study? 

7. [Experimental/Significance] Ditto for the diagonalization discussed on lines 179-180.

8. [Experimental]. For all of the baselines used in Section 5, a description of what hyperparameters were optimised and how (and additionally how hyperparameters which were not optimised were chosen) needs to be supplied. Without this information, it is not clear what to make of the results. 

9. [Experimental] Re Figure 2. Why is the brier score being used to measure predictive accuracy, rather than accuracy itself? Also, it seems like FSVI is underfitting, since the accuracy is low for small rotations. However, this seems to contradict the results in Table 1. Is there a difference in the experimental setup that I do not appreciate? Finally, would it be possible to see a plot like this for corrupted CIFAR10? This would be much more informative than a single column in Table 2.

10. [Experimental] Figure 4 is somewhat incomprehensible because there are too many lines. I would suggest removing the ensemble results (other than deep ensemble) since they are not important for the key message. 

11. [Experimental/Significance] It seems to me that the conclusion that the correlation between OOD-AUROC and ECE is much less strong for CIFAR10 than FashionMNIST, which calls into question the generality of the finding. Is it possible to run this experiment for additional datasets?

## Minor

A. On line 27, I would suggest citing MacKay in addition to, or instead of, Neal for BNNs.

B. On lines 29-31, the authors suggest that BNNs underperform non-Bayesian methods. However, this is a very strong statement. For example, the Linearised Laplace approximation is known to outperform deep ensembles in some settings; see https://arxiv.org/pdf/2106.14806.pdf and http://proceedings.mlr.press/v139/daxberger21a/daxberger21a.pdf. Similarly, HMC BNNs can also outperform deep ensembles in certain settings http://proceedings.mlr.press/v139/izmailov21a/izmailov21a.pdf.

C. On line 87, the prior is introduced as though it was present in the equation above, which is confusing. Perhaps the authors meant to describe the posterior here? 

D. In eqn 1, the authors switch to a KL between the approximate posterior and *prior* without any explanation, which could confuse readers. 

E. Proposition 1 does not need to be a proposition. As far as I understand, it is a well-known result (e.g., in the linearised Laplace approximation community). Similarly, Corollary 1 does not need to be a corollary. This is also a well-known result, as evidenced by the trivial proof. In my opinion, presenting these results in such a way only serves to make the paper less approachable. 

F. I found the use of $g$ and $\tilde{g}$ in Corollary 1 confusing – the symbols are too similar!

G. Lines 188-189, 198-199 are repeats of 171-172 and can be removed. 

H. In what sense is CIFAR100 diverse?  (line 195)

I. Would it be possible to provide a more detailed description of the procedure for generating the monochrome images? I am afraid that the description in the text was not clear to me at all. 

J. On line 2017, I believe that there is a typo. Should 'S*M*' be 'S*K*'?

K. It would be great to have a slightly expanded discussion of the related work. In particular, the relation of FSVI to the approaches of, (Benjamin et al., 2019; Pan et al., 2020; Titsias et al., 2020) could be expanded upon. 

L. Similarly, a discussion of how the results from (Foong et al., 2019) for weight space BNNs related to function space BNNs, would be helpful. 

M. As far as I understand the debate about the properties of mean-field variational distributions in deep BNNs has not been settled, so the statement on line 233 implying otherwise is somewhat strong. I would rather say that "Farquhar et al., (2020) suggest that deep BNNs do not suffer from this issue."

N. On lines 246-247, do the authors mean that the *uncertainty* of the predictions will be significantly higher? 

O. Lines 258-260, seem to be redundant with the previous paragraph. 

P. I assume that the Kaggle dataset on line 316 is the EyePACS dataset on line 313?

limitations: The authors have not made an effort to explicitly describe the limitations of their work. For example, they have not discussed the sensitivity of their proposed method to its hyperparameters, nor have they described the trade-offs between their method and existing functional BNN methods. However, I am confident that this issue can easily be addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This works presents the function-space variational inference in tractable manner. Specifically, the existing function-space variational inference [1] has computational issue in computing KL terms between variational and prior distributions over function-space. This work first presents the tractable form of KL terms by approximating the distributions over function-space linearly by the Taylor approximation, and then employs the estimate of supreme of KL terms by using the context sets, which results in the function-space variational inference in tractable manner. Through experiments, authors validate that the proposed inference could perform the reliable uncertainly estimation, and outperform the existing Bayesian deep learning model for predictive performance and uncertainty estimation.

[1] Functional variational bayesian neural networks, ICLR 19 
strengths_and_weaknesses: $\textbf{Strengths}$

$\bullet$ This work proposes the tractable function-space variational inference, which can easily be computed.

$\bullet$ This work has been sufficiently validated that the proposed inferences leads to the reliable uncertainly estimation and superior predictive performance on a variety of the datasets.

$\textbf{Weaknesses}$

$\bullet$ The technical novelty looks incremental.

$\bullet$ As described in methodology section and shown in Table 1 and 2, the predictive performances seem to be largely affected depending on how to chose the context points and distribution. Thus, the performance could be unstable depending on users who might have deep understanding of given task or not. Thus, I think that explaining the intuition on how to choose context distribution should be described in detail.
questions: $\bullet $ Q1. could you explain some intuition on why authors choose such context distributions in Table 1 and 2? what if the KMNIST or CIFAR100 is not available, which datasets do the authors use? Please explain why authors chose such datasets.  
limitations: See section Weakness above.
review_title: null

Meta-review: metareview: The review process for this manuscript is complex. The reviewers are not in consensus. Most of them have engaged considerably with the original submission as well as the significant updates that the authors have made to the manuscript post submission. In my opinion, new full covariance rank results are what make the paper interesting and these were presented after the original submission. Normally, I would find this not to be fair as the reviewers are not obligated to read such a big revision to a submitted article. But at least two reviewers have engaged with the revision considerably and I feel like the paper is stronger than what the current scores imply. The last holdout reviewer maintains a few outstanding low-confidence concerns about the paper—I do not think these should hold back the manuscript from being presented and discussed at the conference.

I am voting to accept this paper in spite of its low score, but recommend that the authors correct their behavior. Such a large revision to a manuscript puts an enormous tax on the review process; this is basically a "journal level" edit to the submission and normally this would require a second round of review. 
confidence: Less certain
Decision: Accept