Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the behavior of Bayesian model averaging under covariate shift. This is mainly an experimental paper, in which the authors explain the poor generalization ability of Bayesian inference by possible linear dependencies in the input features that cause a lack of posterior contraction, and they show why alternative approximate inference procedures are not affected. They also propose a way to fix the issue via novel priors.
main_review: I really enjoyed the paper, which is clear, well written, and brings novel ideas to the community. It is in line with the recent work of Izmailov et al. (2021) that showed that exact Bayesian inference in deep learning (via Hamiltonian Monte Carlo) generalizes poorly although approximate Bayesian inference is widely used in this setting. Supported with extended numerical experiments, this work validates trough extensive experiments on different types of covariate shift the assumption that the issue of averaging is due to dependencies in the features of the train data distribution that make some model parameters do not affect the predictions on the train data (in which case the posterior coincides with the prior). Hence, the authors use this finding to design new priors that assign low variance to relevant weights. I really appreciated the investigation and the results of the paper.

I have a question regarding the technique used in the paper. The authors say that they use full-batch HMC, and alternative approximate inference techniques such as SWAG and MC Dropout are studied in Appendix G. Both fix the issue, but the influence of the prior in variational inference plays a particular role, in opposite to SWAG for instance which approximates the posterior distribution as a multivariate Gaussian with the SWA solution as its mean. Would the poor generalization issue of the exact posterior be fixed as well for Gaussian Gaussian approximations / in which case, what would be the influence of the prior ?

Also, I would have enjoyed a more general discussion relating the current study with the slightly different notion of misspecification. Could the authors elaborate on this ?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies an important and interesting problem that the BNN with HMC inference performs worse than point estimation under covariate shift. The paper conducts theoretical analysis and contributes this to the linear dependencies in the input feature. A data-dependent prior is proposed for improving the robustness under covariate shift.
main_review: Originality:
This paper is of high originality, as it provides a new perspective to explore the reason of poor robustness of HMC. The EmpCov prior is reasonable with previous analysis.

Quality, clarity:
The paper is well-written and easy to read. I like that every claim is back up by empirical results. The graphs are clear and helpful in understanding the paper.

Significance:
Although this paper provides useful analysis and methods, I would question that if the analysis and method only work on small datasets like MNIST where linear dependence in features are prevalent. How about more complex datasets with rich data?
The data dependent approach might not be scalable to the size of data, as the covariance should be calculated.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: **Post-rebuttal update.** Thank you for your response. I did not have any major concerns with this work, and have also read the other reviews and the author responses and did not find anything concerning. Thus I am keeping the rating unchanged.

This work investigates the performance of Bayesian model averaging (BMA) prediction for NN models under covariate shift. It
* provides further empirical evidence that, for toy models on image classification tasks, BMA may perform poorly on covariate shifted data even though it has competitive performance on clean data.
* provides a (partial) explanation by observing the difference of the posterior and MAP estimate of the first-layer parameters: the MAP estimate shrinks prediction on unobserved "subspaces" (meaning orthogonal linear subspace for Gaussian prior, and zero-variance feature dimensions for general factorized prior) to zero, while the posterior must behave like prior on these subspaces, leading to more noisy feature extractions and deteriorated performance.
* proposes to use as the first-layer prior the $N(0, \Sigma+\epsilon^2 I)$ distribution, where $\Sigma$ is the empirical data covariance.  Experiments show that the prior leads to improved robustness.
main_review: This work provides new insight on an important problem.  The idea, as summarized above, is obvious (in hindsight) and quite convincing, and is well supported by experiments.  The explanation is incomplete as it does not provide much intuition for the subsequent layers (besides the difference on "dead neurons"), and the experiments are limited in scale, but both issues are understandable and acknowledged in discussion.

I do not have any major concern with this work. There are a few issues / questions however:

* There should be more discussion on the past observations that the distribution of NN weights at lower layers are more heavy-tailed (e.g., Fortuin et al. [13]).  Even though the observations are made in different contexts, and experiments here show the Laplace prior does not alleviate the problem, the high-level intuition is similar, and it is possible that a more extreme sparsity-promoting prior is needed to address this issue (e.g. the spike-and-slab prior).

* The most interesting / practically relevant results are Prop 2-3, but they only applies to Gaussian priors. Non-Gaussianity clearly breaks the proof, but does it lead to qualitatively different behavior for feature extraction at the first layer (or is its performance deterioration due to different mechanisms)? It will be helpful to re-do Figure 4 using Laplace or student-t priors for this purpose.
limitations_and_societal_impact: The most important limitations are the limited scale of the experiments, and the lack of complete explanation for the subsequent layers / nonlinear dependencies. They are acknowledged in the paper, and for the latter, some toy-ish scenarios are analyzed in the appendix.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper explores the performance of Bayesian neural networks (BNNs) under covariate shift, comparing primarily to MAP and deep ensembles. The authors explain why, when using standard priors, the test accuracy of HMC BNNs degrades quicker than MAP and deep ensembles when the models are tested on increasing levels of corruption. Using the insights from their explanation, the authors provide a new prior, 'EmpCov', that attempts to address this.
main_review: Overall, I am quite conflicted on this paper. On the positive side, it is well-written with clear explanations of the propositions and results, and the proposed 'EmpCov' prior does appear to work quite well while being a relatively simple fix. From what I can tell, the results and derivations are correct. The figures are also well-made and the authors provide extensive appendices covering a large array of related topics.

However, I am not convinced by the framing of the central issue of the paper. The authors present the degradation of test *accuracy* that BNNs exhibit under covariate shift specifically as an issue that should be surprising (e.g. lines 26-29). They claim that a BNN should be more robust to covariate shift than a deterministic network, in particular with respect to test accuracy, and test accuracy is the main focus of the work. However, I do not see any reason that this should be the case. Moreover, after searching the literature (in particular the works the paper cites), I could not find any work that specifically claimed that the test accuracy of BNNs *should* be higher than those of deterministic NNs. 

For me, it makes sense that the accuracy would be worse, as the BNN will increasingly match the prior for test points that are increasingly far from train points, and the prior predicts a uniform distribution over classes. In my view the main advantage of using a BNN, which is the claim that is often repeated in the literature, is that it should have better-calibrated uncertainty than a deterministic NN, so that the end user is aware when it has encountered an example it has not seen enough data to reliably classify. Indeed, in Fig. 6 in the appendix (page 17), we see that the standard Gaussian prior BNN far outperforms MAP networks on test log likelihoods (test LLs) and expected calibration error (ECE), and has approximately equal performance to the proposed EmpCov prior BNNs. This indicates that standard BNNs (at least with HMC) are much better calibrated than MAP networks, which on average overconfidently predict the correct class more.

Nevertheless, the paper does clearly explain one of the potential mechanisms by which standard BNNs provide predictions closer to uniform for far-away test points, and the proposed prior is able to obtain similar LLs and ECE to standard BNNs with improved accuracy. Therefore, the work clearly has value.

Overall, the paper does a good job investigating the worsened accuracy that BNNs obtain under covariate shift, exploring a mechanism for the worsened accuracy and providing a new prior that addresses this mechanism. However, I believe the framing is not quite correct, in that it is not clear why one would necessarily expect the accuracy of standard BNNs to be better than those of MAP networks on out-of-distribution data. Moreover, standard BNNs are clearly better calibrated when it comes to uncertainty. If the standard BNNs had worse test LLs and ECE then the framing would be far more convincing, but this is clearly not the case. Therefore, I hope the authors would address this with further clarification and by focusing the discussion more on uncertainty calibration (which is the main desired benefit of using a BNN in covariate shift settings). Indeed, it seems as though EmpCov prior can effectively bring together the higher accuracy of the MAP networks and the improved calibration of more standard BNN priors, and I think the paper would be much more effective if it focused more on this rather than claiming that it is surprising that BNNs have reduced accuracy over MAP in OOD regions.

As I stated at the beginning of the review, I am quite borderline on this paper, and so I am looking forward to hearing the authors' response as well as what the other reviewers think.

Additional questions/comments:
- I am not entirely convinced by the arguments concerning linear models given in lines 326 - 330. The predictive mean and MAP solutions only coincide for regression, as the predictive variance will affect the of the output of e.g. a softmax for classification. Along these lines, I would be curious how the authors feel the NNGP [1, 2] would perform in comparison to the finite-width BNNs. I would expect for the MLPs at least that the behavior would be quite similar, because the MLPs used in the paper are 256 units wide and relatively shallow, which should be wide enough to obtain NNGP behavior.
- In many cases neural networks are trained without explicit regularization in the form of an L2 penalty. Would the authors expect these MLE networks to perform worse in terms of test accuracy on corrupted data? Following the logic of the paper it would seem they should.
- Izmailov et al. (2021) show that beyond MAP-based methods, MFVI and SGLD also seem to outperform HMC when it comes to covariate shift. Additionally, the test LLs of MAP seem to be much more even with HMC than what you have observed. Do you have any theories why this may be?

Minor/typos:
- line 120: footnotes should appear after punctuation
- line 227: 'Propositions' shouldn't be plural
- line 930: should be 'then' rather than 'than'

[1] https://arxiv.org/abs/1804.11271
[2] https://arxiv.org/abs/1711.00165
limitations_and_societal_impact: The authors briefly describe some of the main limitations of their work. I am satisfied by their exploration of these limitations, and agree that this work is a sufficient first exploration of these issues. They state that the main negative societal impact of their work was the high energy usage that went into the HMC experiments. As the work is focused on fundamental properties of BNNs, I believe this is adequate and do not believer there are any other direct negative societal impacts.
review_title: null

Meta-review: The reviewers have provided thoughtful and constructive comments. They have responded to the authors' feedback and the most active reviewers who are championing this manuscript continue to lean towards acceptance. I personally find that the authors are presenting an interesting set of contributions in the under-represented and -valued field of model evaluation. I hope the authors will take the reviewers' comments to heart and encourage them to incorporate their thoughts in preparing the camera-ready version of their manuscript.
Decision: Accept (Poster)