Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a fast way to get Bayesian posterior by using a pretrained deterministic model. Specifically, the authors first train a standard DNN model and then use it to initialize the variational parameters. Finally the variational parameters are optimized through standard variational inference (VI) training. To further improve uncertainty estimate, the authors propose an uncertainty regularization which maximizes the prediction inconsistency on out-of-distribution (OOD) data. Experiments including image classification and uncertainty estimates are conducted to demonstrate the proposed method.

The idea of this paper is quite simple: initialize the mean in variational parameters by a pretrained DNN. Thus the method is cheap and simple enough to use broadly in practice. The authors did reasonable empirical tests. I especially appreciate the ablation study which helps understand the method a lot. The paper is well-written and easy to follow.

I mainly have the following concerns about the paper. 

- One of the main motivations to use a pretrained DNN is that BNN learned from scratch is worse than its corresponding DNN. I feel this claim is misleading. Many papers have shown that BNNs trained from scratch outperform DNNs. Particularly the paper [Wenzel et.al. 2020] which the authors cited to support their claim clearly shows that BNN (with reasonable temperature) is significantly better than DNN in predictive performance. But I do agree that the proposed method is cheaper than training BNNs from scratch. 

- The experimental results verify the effectiveness of the proposed method. But the results of the proposed method seem to be worse than BNNs training from scratch (e.g. the ImageNet results in [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning, ICLR 2020] are much better). This also supports my first point, that the main benefit of the proposed method is to get the Bayesian posterior fast and cheaply, instead of to improve BNNs’ performance. I think the authors should revise the claim to be more precise.  

- The proposed method is closely related to [Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes, AAAI 2020] which also uses a pretrained DNN as the initialization of variational parameters. How does the proposed method compare to it theoretically and empirically? The main idea seems quite similar; the only difference is that the proposed method does not use the pretrained model as prior in VI. Due to the similarity, I think a comparison is necessary. 
 
- The authors argue that BNN training suffers from suboptimal local optima. Could the authors provide evidence/citations to support this claim? I do not think it is true. Perhaps it is true only for a few BNN methods such as BNN using naive VI.

- As the proposed method is essentially a VI method, it would be interesting to see comparisons to SOTA VI methods.

Overall I'm positive about this paper and would be happy to increase my score if my concerns are addressed. 

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper explores the variational training of Bayesian neural networks. It proposes to improve the quality of the inferred variational posterior and computational efficiency of the procedure by (i) better initialization (mean parameters are initialized at the MAP) (ii) reducing variance in the Monte Carlo approximated evidence lower bound by increasing the number of weight samples (one per datapoint in a batch) (iii) a posterior regularization encouraging higher uncertainty on adversarially generated or other “near OOD” data.  

The authors use the term BayesAdapter to refer to the process of running black-box variational inference from a fully factorized variational approximation with mean initialized at the MAP estimate and randomly initialized variances. The fact that variational inference (especially those employing fully factorized approximations) are susceptible to poor local optima and that better initializations can help navigate these local optima is widely known. The fact that better initializations can lead to somewhat improved posterior approximations is not surprising. Such initializations are also standard practice when employing stochastic gradient MCMC techniques and Laplace approximations (where it is a requirement). Reducing variance by increasing the number of weight samples to one per datapoint in a batch is another straightforward idea, and it is unclear whether it can be claimed as a contribution of the current paper. Kingma et al., in their local re-parameterization considered a variant with per data samples as well. The uncertainty regularization is indeed novel and appears effective (but the experiments illustrating its benefits need to be better explained). 

Given the modest methods contributions, the empirical section needs to be particularly strong to demonstrate that the combination of these incremental improvements provides meaningful empirical advantages. To their credit, the authors demonstrate their approach on several large datasets and do provide experiments for vetting different aspects of the proposed extensions to variational BNN training. However, many experiments are missing details and some are lacking key comparisons. Overall this section could be significantly strengthened. 

* Tables 1 and 2 need to include comparisons against deep ensembles and multi-SWAG (https://arxiv.org/pdf/2002.08791.pdf). If the goal of this paper is to claim that variationally trained BNNs (with the proposed improvements) are useful in practice, a natural question to ask is whether they are competitive with far simpler ensembling approaches that are able to account for the multimodality of the posterior surface, unlike variational BNNs.
* How was the calibration threshold $\gamma$ chosen for these experiments? How sensitive is the performance to this choice? Ideally, the authors would include results with different settings of $\gamma$. How were the prior precisions selected (which determine $\lambda$ set?  My main worry is that the marginal improvements provided by Bayesadapter variants over BNNs disappear when making slightly different parameter choices. It would be great if the authors can demonstrate that this isn’t the case. 
* Section 4.2 needs more details about the experimental setup. Did the 1000 / 10000 OOD training/test examples include both images created via PGD and SNGAN (for CIFAR 10) and PGD and BigGAN (for imagenet)? If so, how many from each source? If not, it would be interesting to see cross performance — using PGD images for training and SNGAN images for testing. 
* In Table 4, it doesn’t make sense to include ECE numbers from a different architecture trained via SWAG. These numbers are not comparable. Also, interestingly, both BayesAdapter variants have lower ECE scores than vanilla BNN on CIFAR, suggesting poorer calibration. Do the authors have an explanation for this? 

Based on concerns about both novelty and experiments I am currently leaning towards a reject, but could be convinced otherwise based on the authors’ response and additional comparisons.
review_title: Some interesting ideas; limited novelty; experimental section could be improved.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed one simple and effective way to trainBayesian neural networks (BNN). 

Pros:
1. The proposed method is quite simple and cheap to realize, compared to previous Bayesian methods.  
2. Extensive experiments on a diverse set of challenging benchmarks have been conducted, which shows several promising results of the proposed method.
3. The proposed idea is novel which distinguishes from most of previous efforts, which try to train BNN from scratch using Bayesian methods. As described in this paper, most of previous methods, though paying much additional efforts than deterministic ones, do not lead to expected results, even with non-diagonal covariance matrices. The BayesAdapter, however, pays little efforts and obtains improvements even with diagonal covariance matrices. From this perspective, this is an encoraging result. 

Cons:
1. The results of comparison in Table 1 only repot the average result in 3 runs (3 is kind of small). However, it is better to show the std metric of the result to make the comparison more convincing because the improvement of BayesAdapter in average value is in fact not very apparent, especially compared with MAP. If the variance of the result is large, then there will be large overlap between different methods and thus it is not reasonable to claim that there is an apparent advantage over previous methods. 

2. In evaluating the result of BayesAdapter,  MC samples are used. What if only using the mean value of the posterior?  Compared to deterministic methods like MAP, inference using MC is more costly.   In addition, it is suggested to provide some visualizations of the posterior distribution after BayesAdapter. 

3. Based on results in Table 3, BayesAdapter- performs similar as baselines, which indicates that the improvement comes from calibrating the uncertainty estimation. This leads to another question: what if we also use such calibration for the baseline methods. It would be interesting to make such a comparison. 
review_title: Providing a simple method to realize potential advantages of Bayesian neural networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Contributions**

This paper proposes a post-hoc approach to obtain model uncertainty estimates from vanilla pre-trained NNs through MFVI fine-tuning. Namely, 1) the authors re-cast the KL divergence in the VI objective as weight decay applied to the variational parameters 2) the authors propose a variance reduction technique for the reparametrisation trick 3) the authors explicitly train their model to produce large model uncertainty on Out of Distribution (OOD) inputs. 
Empirically, the proposed methods seems to retain the strong performance, and much of the simplicity, of point-estimate NNs while providing enhanced robustness in terms of uncertainty estimation. 


**originality and significance** 

To my knowledge, most of the proposed techniques (or variants of them) have appeared before in the literature or are simple extensions of existing approaches: Re-casting MFVI as SGD [Khan et. al., 2018], Decorrelation of reparametrisation gradients across batch elements [Wen et. al., 2018], Training on OOD measurement points to produce large uncertainty [Hendrycks et. al., 2018 and Hafner et. al., 2018] .
However, this work refines these ideas and puts them into a single framework which seems to produce strong results. I view this as a noteworthy contribution which might bring Bayesian Deep Learning closer to real world deployments. 

 **clarity** 

Most ideas are presented clearly. The paper is well structured and easy to follow. Some passages are slightly ungrammatical but never does this impede the transmission of ideas.

**pros**
* Presents useful practices to make BNNs more mainstream with strong empirical performance.
* Authors provide code for an efficient implementation of exemplar reparametrisation. 
* The proposed technique for OOD detection bypasses typical pathologies of MFVI [Ovadia  et. al., 2019] by explicitly optimising variational parameters to produce large model uncertainties OOD.

**cons**
* Exemplar reparametrisation is very similar to Flipout [Wen et. al., 2018]. A comparison of the two would be appreciated. 
* The proposed technique for OOD detection is not very principled and has provides no guarantees. It seems empirically successful however. 
* The experimental setup is not very clear, even when reading the supplementary sections concerning experimental setup. Some questions I was left with:
	* There are many hyperparameters, how did you find all of them? —Especially the weight decay coefficients.  Are the standard deviations implied by these priors interesting / meaningful in any way? 
	* A single Mutual Information threshold is provided. Is this one used for uncertainty calibration training on all tasks? Is it also used when classifying inputs as in-distribution or OOD? If this is the case, it is possible that models without uncertainty calibration training would benefit from using a different threshold. A better metric might be ROC-AUC, as it is threshold agnostic.
* The only baselines provided are other VI approaches. SWAG is known to be a decently strong baseline but it is not evaluated for OOD detection peroformance. The current  state of the art baseline for uncertainty quantification is deep ensembles [Ovadia  et. al., 2019]. These are much more expensive. However, it might be interesting to see how they compare.
* The authors repeat all experiments 3 times but only provide mean results. In some cases, like tables 1 and 2, the values presented are similar across methods. I think that errorbars (standard deviation across 3 runs) would be very informative to the reader. 

**Other comments and questions:**
* Typo in title: Bayesian, not Bayeisian
* The maximum predictive entropy (and thus mutual information) will depend on dimensionality of output space (number of classes). In your experimental section, you say you set a single threshold for all models. Could you further comment on this?

**References**
[Wen et. al., 2018]  https://arxiv.org/pdf/1803.04386.pdf
[Khan et. al., 2018] http://proceedings.mlr.press/v80/khan18a/khan18a.pdf
[Ovadia  et. al., 2019] https://papers.nips.cc/paper/9547-can-you-trust-your-models-uncertainty-evaluating-predictive-uncertainty-under-dataset-shift.pdf
[Hendrycks et. al., 2018] https://openreview.net/forum?id=HyxCxhRcY7
[Hafner et. al., 2018] https://arxiv.org/abs/1807.09289
review_title: This paper refines existing ideas and places them into a simple framework that produces good results. 

Meta-review: This paper aims at improving the adoption of Bayesian NNs by providing a practical and user friendly variational inference method. The main ideas consist of two parts:
1. Warm-start the variational inference from a pre-trained deterministic NN. It takes advantage of existing deep learning library features for easy implementation including weight decay, batch matrix multiplication, etc.
2. Calibrating uncertainty estimation for out-of-domain detection using adversarial examples.

Pros:
1. A practical way of implementing DNN variational inference with reduced variance, without sacrificing classification accuracy of the pretrained NN model.
2. Significantly better OOD detection accuracy compared to other BNN approaches without taking OOD into account explicitly.

Cons:
1. During discussion, it becomes clear that most of the techniques have been proposed similarly in the literature. Krishnan, 2020 applied BNN starting from MAP of NN, Flipout (Wen et. al., 2018) applies instance-wise sampling, Hendrycks et. al., 2018 and Hafner et. al., 2018 improves detection accuracy by training on OOD examples. The novelty of the proposed method is therefore limited.
2. There's not much benefit on the classification performance compared to the initial MAP and is inferior to MCMC-based SOTA BNNs. One of the reviewers considers the SGLD-type approach may be more appealing to ML practitioners with the overhead of VI in training additional variance parameters.
3. The authors argue MCMC-based BNN methods cannot achieve good performance without temperature scaling. But the main performance improvement of the paper is in the OOD detection with uncertainty regularization that modifies the posterior as well. The method of training with OOD samples is orthogonal to applying Bayesian inference to NNs, and the detection performance is limited to the distribution close to examples during training.

This paper falls on the borderline for acceptance. With the goal of improving adoption of BNN in practice, it is not convincing yet making mean field VI easier to implement could realize it without achieving competitive performance.

Decision: Reject