BayesAdapter: Being Bayesian, Inexpensively and Robustly, via Bayesian Fine-tuning
Authors: Zhijie Deng
Xiao Yang
Hao Zhang
Yinpeng Dong
Jun Zhu
Venue:  (2021)
Keywords: Bayesian neural networks
Bayesian fine-tuning
uncertainty estimation
OOD detection

Abstract: Despite their theoretical appealingness, Bayesian neural networks (BNNs) are falling far behind in terms of adoption in real-world applications compared with normal NNs, mainly due to their limited scalability in training, and low fidelity in their uncertainty estimates. In this work, we develop a new framework, named BayesAdapter, to address these issues and bring Bayesian deep learning to the masses. The core notion of BayesAdapter is to adapt pre-trained deterministic NNs to be BNNs via Bayesian fine-tuning. We implement Bayesian fine-tuning with a plug-and-play instantiation of stochastic variational inference, and propose exemplar reparameterization to reduce gradient variance and stabilize the fine-tuning. Together, they enable training BNNs as if one were training deterministic NNs with minimal added overheads. During Bayesian fine-tuning, we further propose an uncertainty regularization to supervise and calibrate the uncertainty quantification of learned BNNs at low cost. To empirically evaluate BayesAdapter, we conduct extensive experiments on a diverse set of challenging benchmarks, and observe satisfactory training efficiency, competitive predictive performance, and calibrated and faithful uncertainty estimates. 

Under review as a conference paper at ICLR 2021
BAYESADAPTER: BEING BAYESIAN, INEXPENSIVELY AND ROBUSTLY, VIA BAYESIAN FINE-TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
Despite their theoretical appealingness. Bayesian neural networks (BNNs) are falling far behind in terms adoption in real-world applications compared with deterministic NNs, mainly due to their limited scalability in training and low fi- delity in uncertainty estimates. In this work, we develop a new framework, named BayesAdapter, address these issues and bring Bayesian deep learning the masses. The core notion of BayesAdapter is to adapt pre-trained deterministic NNs be BNNs via Bayesian fine-tuning. We implement Bayesian fine-tuning with plug-and and-play instantiation of stochastic variational inference, and propose exemplar reparameterizattion to reduce gradient variance and stabilize the fine- tuning. Together, they enable training BNNs as if one were training deterministic NNs with minimal added overheads. During Bayesian minssiaininetteee fine-tuning, we further pro- pose an uncertainty regularization to supervise and calibrate the uncertainty quan- tification of learned BNNs at low cost. To empirically evaluate BayesAdapter, we conduct extensive experiments a diverse set of challenging benchmarks, and observe satisfactory training efficiency, competitive predictive performance, and calibrated and faithful uncertainty estimates.
INTRODUCTION
Much effort has been devoted to developing flexible and efficient Bayesian deep models to make accurate, robust, and well-calibrated decisions (MacKay, 1992; Neal, 1995; Graves, 2011; Blun- dell et al., 2015), with Bayesian neural networks (BNNs) as popular examples. The principled uncertainty quantification inside BNNs is critical for realistic decision-making, well evaluated in scenarios ranging from model-based reinforcement learning (Depeweg et al., 2016) and active learn- ing (Hernández-Loohe & Adams, 2015), to healthcare (Leibig et al., 2017) and autonomous driv- ing (Kendall & Gal, 2017). BNNs are also known be capable of resisting weet-ftting.
However, there are fundamental obstacles posed in front of ML practitioners when trying to push the limit of BNNs to larger datasets and deeper architectures: (i) The scalability of the existing BNNs is generally restrictive owing to the essential difficulties learning a complex, non-degenerate dis- tribution over parameters in a high-dimensional and over-parameterized space (Liu & Wang, 2016; Louizos & Welling, 2017; Sun et al., 2019). (iz) The Bayes posteriors learned from scratch are of- systematically worse than their point-estimate counterparts in terms of predictive performance when "cold posterior" strategies are not applied (Wenzel et al., 2020). (iii) It is shown that the BNNs have the possibility to assign low (epistemic) uncertainty for realistic (e.g., adversarial examples), rendering their uncertainty estimates unreliable in safety-critical sce- narios (Grosse et al., 2018).
(OOD) data
To solve these problems, we present scalable workflow, named BayesAdapter, to learn more re- liable BNNs. In a holistic view, we unfold the learning of a BNN into two steps: deterministic pre-training of the deep neural network (DNN) counterpart of the BNN followed by several-round Bayesian fine-tuning This enables us to learn a principled BNN with slightly more efforts than training regular DNN, and provides pre- are-trained DNNs (e.g., those on Py Torch Hub). The converged parameters of the deterministic model serve strong start point for Bayesian fine-tuning, allowing to bypass extensive local
with the opportunities to embrace qualified off-the-shelf

---

Under review as a conference paper at ICLR 2021
Pre-train the DNN
Fine-tune the BNN
mubysinizeri
def stepceo, 1schdats
pucioco reightt desay
grad mu.grad Lambda data
Training samples
Training sample
OOD sumples
goad, updatte van 50b,
Ag x8
= wB
olase Peloptinioors
up buth in weiglt drroy
h h Bwya
def steprpet, laanda, non, trwttnnng datali
Trassorm
af puctoon iaporencial seligtt decay"
grad pet- grid Tanbds explpess. data
by Wy
inteurate a conttunt pestiien
Innaaline
grd.. Log-likelihood
atad.
updatte pai.data
training daca.
Predictions
Uncertainty reg.
descustipal. dats, grad, melt.ie)
Figure 1: The workflow BayesAdapter. We assume three-layer model for simplicity. We at first pre- train a DNN counterpart of the target BNN via maximum a posteriori (MAP) estimation, then transform it to be BNN by replacing the point-estimate parameters with diagonal Gaussian centered at them, from which the parameter samples weight decay for the Gaussian mean and variance, and perform fine-tuning to fit the data under uncertainty regularization based on autodiff libraries.
drawn for computation. After that, we build separate optimizers with built-in
optimum suffered by a direct learning of BNN'. To render the fine-tuning in the style of training normal NNs, we resort to stochastic variational inference (VT) to update the approximate posterior. We develop optimizers with built-in weight decay for the parameters of the variational distribution to absorb the regularization effects from the prior, and develop exemplar reparametrization duce the gradient variance. Moreover, to make the uncertainty estimation of the learned models reliable, we propose additionally, explicitly regularize the model behave uncertainly on repre- sentative foreseeable OOD data during fine-tuning. This regularization takes the form of a margin loss. and is readily applicable most of the existing BNNs. Figure depicts the whole frame- work of BayesAdapter. Extensive empirical studies validate the efficiency and effectiveness of our workflow. In summary, our contributions are as follows:
1. We propose BayesAdapter, to quickly and cheaply adapt pre-trained DNN to be Bayesian without compromising performance when facing new tasks.
2. We provide an easy-to-use instantiation of stochastic VI, which allows learning a BNN as if training deterministic NN and frees the users from tedious details of BNN.
3. We augment the fine-tuning with a generally applicable uncertainty regularization term to rectify the predictive uncertainty according collection of OOD data.
4. Extensive studies validate that BayesAdapter is scalable; the delivered BNN models are high-quality; and the acquired uncertainty quantification calibrated and transferable.
2 BAYESADAPTER
In this section, we first motivate BayesAdapter by drawing a connection between maximum a poste- riori (MAP) and Bayesian inference. We then describe the proposed procedure Bayesian fine-tuning, and practical and robust implementation of stochastic VI to realize it. Figure illustrates the over- all workflow of BayesAdapter.
2.1 FROM DNNS TO BNNS
Let D {<x, yi) )= be a given training set, where æ; Rd and yi E y denote the input data and label, respectively, A DNN model can be fit via MAP as following:
HICX VA
> [log p(ylla la; [x) + log.
(I)
We use w RpP denote the high- nigh-dimensional model parameters, and p(y| zz; w) the predictive distribution associated with the model. The prior term p(w). when taking the form of an isotropic Gaussian, reduces to the common I.2 weight decay regularizer in optimization. Despite the wide adoption, DNNs are known to be prone to over-fitting, generating over-confident predictions, and are unable convey valuable information on the trustwrrtiiess of their predictions. Naturally, Bayesian neural networks (BNNs) come into the picture to address these limitations.
Here the BNN mainly refers mean-field variational BNNs, and the results in See 4.1 testify this point.
2

---

Under review as a conference paper at ICLR 2021
Typically, BNN imposes prior p(w) on model parameters, which is put together with the likelihood p(D|w) to infer the posterior p(w(D). Among the wide spectrum of BNN algo- rithms (MacKay, 1992; Neal, 1995; Graves, 2011; Blundell et al., 2015; Liu & Wang, 2016; Gal & Ghahramani, 2016; Louizos & Welling, 2017), variational BNNs are particularly promising due to their ease of training compared with other BNN variants. Formally, variational BNNs derive 0- --paameterizee varitional distribution q( (w O) to approximate the true posterior (D) by maxi- mizing the evidence lower bound (ELBO) (scaled by 1/n):
max E. (p(=))))
Σ log p(ul) mn) DxL TI
Ee
where Len is the expected log- log-likelihood and L. is the complexity loss. By casting posterior inference into optimization, Eq. (2) makes the training of BNNs more approachable. However, most existing BNNs trained under such a criterion exhibit limitations in scalability and performance (Osawa et al., 2019a; Wenzel al., 2020) compared with their deterministic counterparts, mainly attributed to the higher difficulty of learning high-dimensional distributions than point estimates, and challenges in finding non-degenerated optima of highly nonlinear functions characterized by NNs.
Given that MAP converges to the mode of the Bayesian posterior, it might be plausible to adapt pre- trained deterministic DNNs to be Bayesian economically. Following this hypothesis, we propose to repurpose the converged parameters w" of MAP, and use to instantiate q(w w 0) as a Gaussian N(w; 0) with 0 (n,, Σ) where μ is initialized as w and Σ ppxp denotes the covariance. Then, we arrive at BNN with posterior predictive:
plyl D) Ex(e:.2))((y);; 5
Σ p(ylæ; lz; w where w- ~ N(w: 14, E), 8 1,..., (3)
Eq. (3) is also called Bayes ensemble, where μ is perturbed, and the predictions from multiple likely models are assembled. Σ controls the magnitude of perturbation. A classic method to generate an informative Σ is by Laplace approximation (Bleistein & Handelsman, 1986). but it is more like postprocessing procedure, lacking the flexibility to jointly adapt the mean and covariance of the Gaussian posterior w.r.t. data, and its naive implementation without strong assumptions may be computationally prohibitive. Instead, suggest more practical workflow that fine-tunes the approximate posterior N(w: μa, Σ) by maximizing the ELBO with randomly initialized Σ.
2.2 BAYESIAN FINE-TUNING IN THE STYLE OF FINE-TUNING DNNS
We develop practical learning algorithms under the stochastic VI scheme to fine-tune the imperfect variational posterior, and to cope with contemporary MI. frameworks. In the following, we discuss how to deal with each term in Eq. (2). Algorithm gives an overview of BayesAdapter.
Complexity loss Le. Without losing generality, we assume an isotropic Gaussian prior p(w) N(w; O, I). Then the complexity loss derived as:
C. = -1 (N(w; μ. Σ) [ww; 0, ai)) 14 44 + tr()
det2
20mn +
+
where r and det are matrix trace and determinant, respectively. is constant. The gradients of Co w.r.t 4e and E can be estimated precisely as:
VDC. o OE'-L1
VuCe
1
(5)
20īn
Eq. (5) indicates that max, Le amounts to applying weight decay regularizer with coefficient À = ogn on μ. which can be conveniently optimized by leveraging the built-in weight decay modules in ML frameworks such as TensorFlow (Abadi al., 2016) PyTorch (Paszke al., 2019).
Directly computing VzL. involves matrix inversion. Implementing the posterior as matrix-variate Gaussian is an alternative, while existing algorithms for matrix- varriae Gaussian posterior typically exhibit high complexity in time or memory, limited compatibility with contemporary NN build- ing block operations (e.g., convolution), and struggle to scale with data-parallel distributed train- ing (Louizos& & Welling, 2016; Sun et al., 2017; Osawa et al., 2019b). To simplify the implemen- tation and boost scalability, we assume fully factorized Gaussian variational by devising Σ as
We use BNNs equivalently with variational BNNs in the following text when there is ambiguity.
3

---

Under review as a conference paper at ICLR 2021
diag(exp(2<b)) (2x)))2)). where 2o E R' is the parameter to be optimized along with μ (i.e., 0 (H, (H,,²)), 2))) Injecting this into Eq. (5) gets a more concise gradient estimator: VeL. = ing that max L. adds exponential weight decay of & with coefficient X. which can be realized by modifying only two lines of code on top of de facto DI. frameworks (see Figure 1).
mean-
Expected log-likelihood Len- With the complexity loss expressed as weight decay, now develop efficient ways for calculating the La at the end of forward pass, and for performing backpropagation afterwards. In particular, we derive a Monte Carlo (MC) estimation of Lell based on reparameter- ization (Kingma & Welling, 2013): we sample a p-dimensional Gaussian noise 6 ~ (N I), then obtain the sampled parameter for the whole mini-batch B of data via w ext exp((e) ()),, given which we upproximate Cell with C'u1 = [8] BB log p( [ w). The gradients of μa and sp can be derived automatically with autodiff libraries, thus the training resembles that of normal DNNs.
However, gradients derived by C.u might exhibit high variance, caused by sharing one set of sam- pled parameters across all the training instances in B. Local reparameterization is proposed to reduce the variance, but it requires at least 2x bawwwrd FLOPS than vanilla reparameter- ization (refer Kingma et al. (2015) for more details). Flipout (Wen et al., 2018) is alternative solution. But it is only suitable for perturbation based MC estimation and its modeling assumptions make Flipout unable to handle complex variational posterior like a FLOW (Louizos & Welling, 2017), or an implicit model (Shi et al., 2018b). Besides, it is still as slow as local reparameter- ization. To mitigate these issues, we propose exemplar reparametrization (ER) which samples separate set of parameters for every exemplar in the minibatch. Formally, for Ya;; E B, we draw (f()) = ja + exp(1))e(() where e()) ~ (0, I), and approximate the expected log-likelihood by fau = BI E log p(u: ((ys(a,;; (+)))
Obviously, ER is distribution agnostic, and is readily applicable to various variational distri- butions. While ER generates more parame- ters at training. they are mostly temporary, and
assuso shipe it (b) de by Miz au, podr Io, d, R, KJ
oun convis, W, padding, groupe):
convis(s, w. stride, padcing, groupe)
def Enyenadapter.com ConvE,,
stxide, padding, groupe)
pasple batch of parasetero (b, o, 1, J. A7
the resultant computational FLOPS are prov- ably identical to that of the vanilla reparame- terization. The challenge of ER is to cope with nowadays ML frameworks and maintain com- puting efficiency, because off -heflf compu- tation kernels in autodiff libraries typically as- sume a batch of instances share common of parameters. We present an example in Fig- ure 2 on how the standard convolution io op can be converted into its exemplar version without compromising computational efficiency. The key insight here is that multiple exemplar convolutions can be expressed as a group convolution, which can be performed in parallel using single group convolution kernel, leveraging the optimized implementations provided by various device-propriety kernel backends (e.g. cuDNN (Chetlur et al., 2014)). Other common operators such as matrix multiplication are straightforward to handle (refer to Appendix A).
ou exp(pel) casdatb, +listteu shapo))
rashape v en have enape (oro, 1, b, 8]
* - v. ( and cier1) restape x to have shape (1, Des, hy by
x xfflatten(ttarr_ daten end_dimel) (-nngueeeee))) parcomm conne in paralles
y convidix, w. atzide, padding, gzouparb)
rastape reante
y.vimo(s, en mta.ehpe])), (- y. y_shapa(1])
Figure 2: The comparison between standard convolu- tion and convolution used in BayesAdapter.
With this insight, BayesAdapter enables to obtain a BNN with only minor computational cost in addition to pre-training, and can immediately benefit from the availability of higher-performance computational kernels (e.g., more powerful group convolution kernel).
3 CALIBRATE THE UNCERTAINTY ESTIMATION
So far we have developed an inexpensive fine-tuning procedure to obtain BNNs from deterministic NNs. While BNNs can offer uncertainty estimates, these uncertainty measures are highly non- smooth due to the con-convy of NNs- out-of-distribution (OOD) regions, but become vulnerable on OOD samples close to the normal ones (Grosse et al., 2018), rendering BNNs unable to react to potentially harmful inputs. We quantify this phenomenon in Section 4.2. To address this problem, we next develop methods to further calibrate the uncertainty estimation of naively trained BNNs. Inspired by recent work on OOD detection (Wang et al., 2020; Durall et al., 2020). we propose to additionally incorporate uncertainty regularization on top of the above fine-tuning procedure. The idea is to force BNNs to generate inconsistent predictions for each sample from a cheaply collected OOD sample set, so that they acquire the ability to yield high uncertainty for OOD samples with similar fingerprints.
- they might exhibit high uncertainty for data from faraway
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: BayesAdapter
Input:
normal training set 'D in the size of n, OOD training set Dt, weight decay coefficient for both the pre-training and the fine-tuning, threshold 7, learning rates (r Irv, fine-tuning epochs T
Pre-train the DNN counterpart of the target BNN 'D by MAP; denote the converged parameters as J4
Create randomly initialized parameters to; make the computation modules Bayesian (see Figure 2)
3 Build optimizers opt, and o (see Figure 1) with learning rate Ir and Iri. for μa and ub respectively
for epoch 1. 2, ....
[st] for mini-batch B= = {(x,, (6))i= in D, mini- batch B¹ = {<x| Ji=1 in DJ do
Build the whole mini-batch (211.... 28, u!...., 2 Tiss "i..... (1111 ([s1)]
and feed it into the model
Given the predictive distribution and labels [y. ] compute Ciu and Cune
Derive the gradients of Ceu + Lune w.r. je and aa via AutoGrad
Update the parameters je and ab with optimizers , and opt.
To achieve this, we start by defining a differentiable uncertainty metric in terms of mutual informa- tion, following Smith & Gal (2018):
T(u. = H E moll: - H (pluloe; ()) where wn ~ -N t5). 1,..... (6)
H is the Shannon entropy. I highly correlates with softmax variance (Smith & Gal, 2018), and measures the epistemic uncertainty which describes uncertainty in the model and can be used to identify OOD instances. Then, assuming access to an OOD dataset Di = = [=]);= we enforce the model to behave uncertainly on each of them by optimizing margin loss with threshold y;
/ min (I(ee, ulæe!, D),1).
max fun: [B
(7)
where Bt refers to a mini-batch of OOD data. For efficiency, we adopt S = 2 MC samples for esti- mating I(w, y| u/ D) in Eq. (7) in the training. While this loss has a seemingly opposite form from the maasistency-promottin loss in semi-supervised learning (SSL) (Laine & Aila, 2016), they share the same design philosophy: Lune maximizes the prediction inconsistency of OOD instances so as to distinguish them from in-distribution instances, while SSL minimizes the prediction inconsistency of unlabeled data so to classify them without labels. Put it in the context of autonomous driving: if the model trained on data only containing scenes in regular weather, we can take a small set of scene data of extreme weather, e.g., tornado and sandstorm, to regularize the training following Eq. (7). Then the model will learn to identify these abnormal scenes based on predictive uncertainty, thus can refuse to make unreliable decisions in these scenes.
Constructing the OOD dataset DI is flexible and application-specific. In discriminative tasks, two types of OOD data of particular concerns are adversarial and fake samples, which can be both collected trivially following procedures described below.
Adversarial samples. Directly generating adversary samples following methods like PGD (Madry et al., 2017) might be expensive. We propose a more cost-effective alternative based on a key obser- vation: given valid perturbation space [-6m 5m]" 1d where 8m, is the maximum norm under the loc threat model, we can see that uniform noises 5 ~ (- 5,,)² radically encompass the adversarial perturbations which usually reside at local optimas. Thus into uncertainty training to direct the model to behave uncertainly on randomly contaminated data, bypassing the potential cost of generating real adversary samples. The results in Sec 4.2 surprisingly confirm the effectiveness of uniform noises, and imply strong connection between uniform noises and adversarial ones, which deserves future investigation.
add uniformly perturbed samples
Fake samples. Fake samples can be obtained by utilizing pretrained state-of-the-ar GANs (Miyato et al., 2018; Brock et al., 2018), DeepFake (Deepfakes, 2018), and FaceSwap (Faceswap. 2018). We use only 1000 random fake samples for Bayesian fine-tuming on diverse benchmarks.
For both, we empirically find the proposed uncertainty regularization is data efficient - with access to proxy set of adversarial samples and a small set of fake samples, the model can acquire reliable, transferable uncertainty quantification.
4 EXPERIMENTS
In this section, we evaluate BayesAdapter diverse set of challenging benchmarks.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Predictive performance comparison image classification. NLL denotes the negative log ongliikeihhod.
CIFAR- 10 Reide-Re
ImageNet (ResNet-50)
Method
TOPI (%) f
NLL
TOPI (%)
TOP5 (%)
NLL
MAP
96.92
0.1312
76.13
92.86
0.9618
Deep Ensemble
97.40
0.0869
SWAG
96.32
0.1122
Laplace Approx.
96.41
0.1204
75.89
92.70
0,9739
MC dropout
96.95
0.1151
74.88
92.32
0,9884
BNN
97.02
0.0975
75.97
92.95
0,9435
BayesAdapter-
97.09
0.0945
76.49
93.10
0.9337
BayesAdapter
96.02
0.100420.0002
76.26:#0.06
92.96=+0.03
0.9428==0.0020
Table Accuracy 1 comparison on open set face recognition with MobileNetV2 architecture.
Method
LFW
CPLFW
CALFW
CFP-FF
CFP-FP
VGGFace2
AgeDB-30
MAP
98.2%
84.0%
87.6%
97.8%
92.7%
91.7%
85.3%
MC dropout
98.2%
83.6%
87.3%
97.8%
92.8%
92.6%
86.0%
BNN 97.8%
82.4%
85.7%
96.8%
91.4%
90.5%
83.8%
BayesAdapter- 98.4%
84.4%
87.3%
97.7%
92.7%
92.3%
85.1%
BayesAdapter 98.5% %
84.1%
87.4%
97.9%
93.1%
92.5%
84.8%
Settings. We first conduct experiments on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng al., 2009) using vide-ResNee-28-11 (Zagoruyko & Komodakis, 2016) and ResNet-50 (He al., 2016), respectively. Besides, we train face recognition models on CASIA (Yi et al.. 2014) with Mo- bileNetV2 (Sandler al., 2018), and perform comprehensive evaluation on face verification datasets including LFW (Huang et al., 2007), CPLFW (Zheng & Deng, 2018), CALFW (Zheng et al., 2017), CFP (Sengupta et al., 2016). VGGFace2 (Cao al., 2018), and AgeDB-30 (Moschoglou et al., 2017). We pre-train DNNs following standard protocols, and perform Bayesian fine-tuning for 40, 12, and epochs with weight decay coefficients (i.e., A) 2e-4, le-4, and 5e-4 on these benchmarks respectively. We set uncertainty threshold 7 according an observation that the normal examples usually present < 0.75 mutual information uncertainty, across the studied scenarios. Then we use y 0.75 in the regularization to push both the adversarial and fake data to exhibit distinguish- able uncertainty from the normal data. We initialize ab uniformly from [-6, -5] and use 20-step PGD as validation adversaries. On the three benchmarks, the perturbation budgets 8, set 0.031, 16/255, and 16/255, and the fake samples are obtained from SNGAN (Miyato et al., 2018). Big- GAN (Brock et al., 2018), and DeepFake. We perform intensive data augmentation for fake training data with a random strategy including Gaussian blur, JPEG compression, etc. We defer more details to Appendix B. We run every experiment 3 times on 8 RTX 2080-TIs and report the average.
We compare the full BayesAdapter to extensive baselines including: (1) MAP. (2) Laplace Approx.: Laplace Approximation with diagonal Fisher information matrix, (3) MC dropout (detailed in Appendix B), (4) BNN: BNNs trained from scratch by solving Eq. (2) without uncer- tainty regularization, (5) BayesAdapter:: variant BayesAdapter without uncertainty regulariza- tion. We also include Deep Ensemble (Lakshminarayanan et al., 2017), one of the state-of-the-an BNNs, and SWAG (Maddox et al., 2019), whose performance not worse than SGLD (Welling & Teh, 2011), KFAC Laplace (Ritter et al., 2018), and temperature scaling (Guo et al., 2017), into the comparison on CI
Baselines.
Metrics. We concern (i) the posterior predictive performance with S = 100 MC samples: (ii) the average precision (AP) of directly using the uncertainty estimated by Eq. (6) (S = 20) to distinguish OOD test samples (labeled 1) from normal test samples (labeled 0). Eq. (6) of the deterministic baseline MAP is 0, so we take the predictive entropy as an alternative uncertainty measure for MAP.
4.1 PREDICTIVE PERFORMANCE
We compare the prediction performance, which is of central importance in practice, of various meth- ods in Table 1 and 2. Deep Ensemble shows outperforming classification performance because the ensemble candidates can investigate diverse function modes, but orders of magnitude more expensive than BayesAdapter. BayesAdapter- notably surpasses the MAP, especially in NLL, ver- ifying the modeling superiority of a Bayesian formulation and highlights the practical value of our workflow. Laplace Approx. is consistently worse than MAP. In all settings, BNN is significantly defeated by BayesAdapter, confirming our claim that performing Bayesian fine-tuning from the
CCurrently, we have scaled Deep Ensemble and SWAG up ImageNet due resource constraints.

---

Under review as a conference paper at ICLR 2021
Nernal Advesss
Nernal Adoesaiial
Namaal
Narual
Falc
Fale
440
5-
0.5
Matual
8.6
0.33
Monal Infernaii
Merual lndcaaiiee
Monull Information
(a) Bayeshdapter. CIEAR-10
(b) Bayeshdapper. CIFAR-10
(c) BayezAdapter, ImugeNet
(d) BayesAdapter- ImageNet
Figure The histograms for the mutual information uncertainty of normal data and OOD data given by models
trained w/ and w/o uncertainty regularization.
Table 3: Comparison on the quality of uncertainty estimates in terms of average precision (AP) f of uncertainty based binary classification. (CIFAR-10 10 and ImageNet)
CIFAR-I0
ImageNet
Method
Adversarial (PGD)
Fake (SNGAN)
Adversarial (PGD)
Fake (BigGAN)
MAP
0.307
0,800
0.308
0.010
Deep Ensemble
0.427
0.812
SWAG
0.316
0,816
Laplace Approx.
0,308
0,800
0.311
0.015
MC dropout BNN
0,308
0.803
0.309
0.012
0.307
0.799
0.310
0.021
BayesAdapter-
0.307
0.806
0.387
0.013
BayesAdapter
0.903++0.00
0.9941:00001
0.964
0.848=0.037
converged deterministic checkpoints is beneficial to bypass the local optimas potentially encoun- tered by direct Bayesian inference. The popular baselines MC dropout and SWAG show weaker performance on ImageNet and CIFAR-10, 10, respectively, revealing limited upplicability. Also of note that no method shows dominant performance on face recognition, probably due to the diversity of these validation sets. Across these tasks, BayesAdapter is slightly worse than its regularization-fre version BayesAdapter- This is reasonable since such a regularization enforces the model to trade partial capacity for fidelity of uncertainty estimates. Nevertheless, BayesAdapter is substantially better than its fine-tuning start point MAP and the BNN trained from scratch in most settings.
Speedup. Adyesr is a much more economical way obtain BNNs. To interpret the speedup of BayesAdapter over BNN, we assume deterministic ResNet-50 takes one unit time for one epoch of training on ImageNet, and observe Bayesian ResNet-50 takes 2.1t for one epoch. Thus, BNN trained from scratch consumes 189t for 90-epoch training, while BayesAdapter- need t x 90 + 2.1f 12 115.2f, saving 73.8t (around 40%) training time than BNN.
4.2 QUALITY OF UNCERTAINTY ESTIMATES
We study the effects of the proposed uncertainty regularization by visualizing the predictive uncer- tainty of BayesAdapter and BayesAdapter- on validation data in Figure 3. On both CIFAR-10 and ImageNet, BayesAdapter yields evidently higher uncertainty for OOD data than normal data, while BayesAdapter- is on the contrary, showing it can effectively calibrate the predictive uncertainty.
To precisely evaluate its efficacy, we quantitatively assess the quality of the predictive uncertainty of various methods by estimating AP, which reflects if the model knows what it knows. As stated, take adversarial samples crafted by PGD and fake samples from GANs and DeepFake as proxies of harmful OOD data. We report the results in Table 3 and Table 6 in Appendix C. As shown, SWAG, Laplace Approx.. MC dropout, BNN, and BayesAdapter- perform all as bad as MAP across settings, except that MC dropout is capable of partially detecting OOD data face tasks Despite impressive prediction accuracy, Deep Ensemble also yields unreliable uncertainty estimates on these two kinds of challenging OOD data. These results echo our concern on the reliability of existing BNNs' predictive uncertainty. By contrast, BayesAdapter, which is fine-tuned upon MAP for only several rounds based on low-cost supervisions, achieves near-perfect results in detecting OOD instances on CIFAR-10 10 and face recognition, and also detects most of the OOD instances on ImageNet (see Appendix D for some samples).
4.3 ABLATION STUDY
'In practice, BoyesAdapter would a little slower than RayesAdapter- due to the incorporation of the OOD training set, but still much more efficient than BNN. We speculate this may relate where add dropout in the NN architecture, but leave it for future study.
7

---

Under review as a conference paper at ICLR 2021
Model calibration. Model calibration is other important aspect of the uncertainty estima- tion. Suggested by pionecring works, we take the Expected Calibration Error (ECE) (Guo et al., 2017) the measure of calibration, and report the ECE of the studied methods in Table 4, The ECE of BayesAdapter is on par with the MC dropout, Deep Ensemble, and BNN baselines, but BayesAdapter can meanwhile offer much better uncertainty for detecting risky OOD data.
Table Comparison on ECE 4.
Method
CIFAR- 10
ImageNet
MAP
0,0198
0.0373
Deep Ensemble
0.0057
SWAG
0.0088
Laplace Approx.
0,0106
0.0375
MC dropout
0.0119
0.0152
BNN 0.0055
0.0183
BayesAdapter-
0.0094
0.0159
RayesAdapter 0.0057
0.0165
Transferability uncertainty quantification. One may wonder the uncertainty quantification learned according to specialized OOD data can generalize to other OOD data. To figure out this prob- lem, we evaluate the BayesAdapter trained on CIFAR-10 10 on 10000 samples from PGGAN (Karras et al., 2017) whose patterns are unseen during training. We compute their uncertainty and calculate the AP metric, obtaining 0.932. As comparison, the APs of MAP, Deep Ensemble, SWAG, Laplace Approx., MC dropout, BNN, BayesAdapter- on such data are 0.789, 0.797, 0.809, 0.800, 0.792, 0.793, 0.803 respectively. On the other hand, we craft adversarial examples by the fast gradient sign method (FGSM) (Goodfellow al., 2014) against the ResNet-152 DNN model with 1000 valida- tion images from ImageNet. Then we estimate the AP on these instances, and obtain 0.011, 0. 125, 0.027, 0.202, 0.019, and 0.882 for MAP. Laplace Approx., MC dropout, BNN, BayesAdapter. and BayesAdapter respectively. These studies validate the transferability of our uncertainty estimation.
The effectiveness of exemplar reparameterization. We build a toy model with only a Bayesian convolutional layer, fixing the model input and target output, and computing the variance of stochas- tic gradients across 500 runs. We average the gradient variance of μ and &b over all their coordinates, and observe that standard reparameterization typically introduces more than 100x variance than ex- emplar reparumeterization despite with the same FLOPS.
Ablation study on uncertainty threshold % We perform an ablation study regarding 7 on CIFAR- 10 to evaluate the hyper-parameter tolerance of the proposed method. Table presents the results. The results reveal that values of r [0.75, 1.0] may be good choices for OOD detection, and also
Table 5: Ablation study 7.
7
Acc.
(PGD)
AP (fake)
0.25
96.93%
0.915
0.910
0.50
96.70%
0.948
0.981
0.75
96.82%
0.993
0.994
1.00
96.74%
0.991
0.994
echo the observation that normal examples usu- ally present < 0.75 uncertainty.
1.50
96.79%
0.944
0.988
The impacts of ensemble number. We draw the change of test accuracy w.r.t. the number of MC samples S for estimating Eq. (3) in Figure The model trained by BayesAdapter on ImageNet. The points on the red line represent the individual accuracies of the 100 parameter samples. The yellow dashed line refers to the de- terministic inference with only the Gaussian mean. The green line displays the effects of Bayes ensemble - the predictive performance increases from < 74% to > 76% quickly before seeing 20 parame- ter samples, and gradually saturate after that. That is why we use 20 samples for estimating uncertainty and crafting adversarial samples.
76
Commmnn (00
Individual Easemble
Deterministo
The number of MC sample
Figure 4: The change of test ac- curacy w.r.t. the number of MC samples for estimating Eq. (3).
Uncertainty-basee rejective decision. In practice, we expect our models can reject to predict for data with relatively large uncer- tainty, and only care about the data that they are certain about. this spirit, sort the validation data of ImageNet w.r.t. the uncer- tainty provided by BayesAdapter, and split them into 10 buckets of equal size. We depict the average accuracy of each bucket in Fig- ure 5. As expected, our BNN is more accurate for instances with smaller uncertainty. Quantitatively, there are 95% instances with uncertainty less than 0.45, and their accuracy is 78,66 78.6%; there are 90% instances with uncertainty less than 0.37, and their accuracy is 80.7%; there are 80% instances with uncertainty less than 0.25, and their accuracy is 84.8%.
tour Hut Figure 5: Comparison on acCu- racy for instance buckets of equal size but with rising uncertainty.
8

---

Under review as a conference paper at ICLR 2021
5 RELATED WORK
Fruitful works have emerged in the BNN community in the last decade (Graves, 2011; Welling & Teh, 2011; Blundell al., 2015; Kingma & Welling. 2013; Balan et al., 2015; Liu & Wang, 2016; Kendall & Gal, 2017). However, most of the existing works cannot achieve the goal of practica- bility. For example, Liu & Wang (2016); Louizos & Welling (2016; 2017); Shi al. (2018a); Sun et al. (2019) trade learning efficiency for flexible variational posteriors, leading to restrictive scala- bility. Khan et al.; Zhang al.; Osawa et al. build Adam-like optimizers to do variational inference, but their parallel training throughput and compatibility with data augmentation are inferior to SGD. Empirical Bayes methods, e.g., Monte Carlo (MC) dropout (Gal & Ghahramani, 2016), deep ensem- ble (I aaakshinaraaannan et al., 2017), and SWAG (Maddox et al., 2019), usually maintain impressive predictive performance, but suffer from degenerated uncertainty estimates (Fort et al., 2019) or ex- pensive training/storage cost. What's 's worse, the existing works usually evaluate on impractical OOD data (Louizos & Welling, 2017; Pawlowski et al., 2017) to show the promise of Bayesian principle. Instead, we offer new evaluation standard in this work, which may benefit the following works.
Laplacian approximation (Bleistein Handelsman, 1986; Ritter et al., 2018) a known approach to transform a DNN to BNN, but it is inflexible due to its postprocessing nature and some strong assumptions made for practical concerns. Alternatively, BayesAdapter works in the style of fine- tuning, which is more natural and economical for deep networks. Bayesian modeling the last layer of a DNN is proposed recently (Kristiadi et al., 2020), and its combination with BayesAdapter de- serves an investigation. BayesAdapter connects MOPED (Krishnan et al.) in that their variational configurations are both based on MAP. Yet, beyond MOPED, BayesAdapter is further designed to achieve good fre-ffiendllness, improved learning stability, and trustable uncertainty estimation, by virtue of optimizers with built-in weight decay, exemplar reparameterization, and uncertainty reg- ularization, respectively, which significantly boost the practicability of BayesAdapter, especially in real-world and scage settings.
CONCLUSION
In this work, propose scalable BayesAdapter framework to learn practical BNNs. Our idea is to learn a BNN by first pre-training its DNN counterpart and then performing Bayesian fine- tuning. In Bayes, we develop a plug-and-play instantiation of stochastic VI, and propose exemplar reparancterization to reduce the gradient variance. We also propose a generic uncertainty regularization calibrate the uncertainty quantification given low-cost supervisions. We evaluate BayesAdapter in diverse realistic scenarios and report promising results.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large- scale machine learning. In 12th {USENIX} symposium on operating systems design and imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Anoop Korattikara Balan, Vivek Rathod, Kevin P Murphy, and Max Welling. Bayesian dark knowl- edge. In Advances in Neural Information Processing Systems, pp.
2015.
Norman Bleistein and Richard A Handelsman. Asymptotic expansions of integrals. Courier Corpo- ration, 1986.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622, 2015.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Qiong Cao, Li Shen, Weidi Xie, Omkar Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). pp. 67-74. IEEE, 2018.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXXiv1410.000 2014.

---

Under review as a conference paper at ICLR 2021
Deepfakes, 2018. httpss//ggttaub..cm//eeeeeettdtp
Accessed: 2018-10-29
Deng, W. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Stefan Depeweg, José Miguel Hernández-Lobato, Finale Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with Bayesian neural networks. arXiv preprint arXiv: 2016.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Can based gener- ative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7890-7899, 2020.
Faceswap, 2018. httpppss//ggaacccccmmm//rreeeees 10-29.
Accessed: 2018-
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv: 191.00771, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference Machine Learning, pp. 1050-1059, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Alex Graves. Practical variational inference for neural networks. In Advances Neural Information Processing Systems, pp. 2348-2356, 2011.
Kathrin Grosse, David Pfaff, Michael Thomas Smith, and Michael Backes. uncertainty in adversarial settings. arXiv preprint arXiv:181..02606 2018.
The limitations of model
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. arXiv preprint (eXXvv/706..45999 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learn- ing of Bayesian neural networks. In International Conference on Machine Learning, pp. 1861- 1869, 2015.
Gary Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Mlllrr, Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Technical report, 2007.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03360 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10199. 2017.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, pp. 5574-5584, 2017.
Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivas- tava. Fast and scalable Bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning, pp. 2616-2625, 2018.
Diederik Kingma and Max Welling.
Auto-encoding variational Bayes.
arXiv preprint
1312 2013.
10

---

Under review as a conference paper at ICLR 2021
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. In Advances in Neural Information Processing Systems, pp.
2015.
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Specifying weight priors in bayesian deep neural networks with empirical bayes.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in relu networks. arXiv preprint wrXiv:2022.10116 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1rXiv:1610.02222., 2016.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6800 6413, 2017.
Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, and Siegfried Wahl. Lever- aging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7(1):1-14, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose Bayesian inference algorithm. Advances in Neural Information Processing Systems, pp. 2378-2386, 2016.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning, pp. 1708-1716, 2016.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In International Conference on Machine Learning, pp. 2218-2227, 2017.
David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural Compu- tation, 4(3):488 472, 1992.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, pp. 13153-13164, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, arXiv preprint 1rXiv:1706.0608. 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 1rXXv:1802.05957 2018.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 51-59, 2017.
Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with Bayesian principles. arXiv preprint 1106vv:99060220.. 2019a.
Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno, Akira Naruse, Rio Yokota, and Satoshi Matsuoka. Large-scale distributed second-order optimization using tronecker-factone approximate curva- ture for deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 22359-12367 2019b.

---

Under review as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in neural information processing systems, pp- 8026-8037, 2019.
Nick Pawlowski, Andrew Brock, Matthew Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint urXiv: 2017.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conferent Track Proceedings, volume 6. International Conference on Representation Learning, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chigh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 4510-4520, 2018
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 1-9. IEEE, 2016.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. In International Conference on Learning Representutions, 2018a.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. A spectral approach to gradient estimation for implicit distributions. arXiv preprint arXiv:1806.029) 2018b.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detec- tion. arXiv preprint arXiv:/803.085333 2018.
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertainty in Bayesian neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 1283-1292, 2017.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. International Conference on Learning Representations, 2019.
Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Niebner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387-2395, 2016.
Sheng-Yi Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Con-generated images are surprisingly easy to spot... for now. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, volume 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-I/), pp. 681-688, 2011.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo- independent weight perturbations on mini-batches. arXiv preprint 1rXiv:1803.04388 2018
Florian Wenzel, Kevin Roth, Bastiaan S Veeling. Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02400. 2020.
Dong Yi, Zhen Lei,, Shengcai Liao, and Stan Li. Learning face representation from scratch. arXiv preprint #rXiv:1411.7923 2014.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.00116. 2016.
Wide residual networks.
arXiv preprint
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International Conference Machine Learning, pp. 5847-5856, 2018.
12

---

Under review as a conference paper at ICLR 2021
Tianyue Zheng and Weihong Deng. Cross-port posse Ifw: A database for studying cross nition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5, 2018.
cross-posse pass-pose face recog-
Tianyue Zheng, Weihong Deng, and Jiani Hu. Cross-age Ifw: A database for studying cross-age face recognition in unconstrained environments. arXiv preprint 08197, 2017.
13

---

Under review as a conference paper at ICLR 2021
A THE EXEMPLAR VERSION OF POPULAR OPERATORS
As introduced in Sec 2.2, the regular convolution can be elegantly converted into an exemplar ver- sion by resorting to group convolution. The other popular operators are relatively easy to handle. For example, we substitute the qualified batch matrix multiplication, which is highly optimized in the well-known autodiff libraries, for matrix multiplication. For the affine transformation in batch normalization (Ioffe & Szegedy, 2015), we can at first sample dedicated affine weight and bias for every exemplar in the batch, then perform transformation with these two batches of parameters by just not broadcasting on the batch dimension.
B MORE EXPERIMENTAL DETAILS
The only two important Syyer--arametee are the weight decay coefficient À and the uncertainty threshold y. Other hyper-parameters for defining PGD or specifying learning rates, etc., all follow standard practice in the DL community. The number of fake data training (1000) and the number of MC samples for evaluation (S) are flexible and not tuned.
For X, we keep it consistent between pre-training and fine-tuning (stated in Algorithm 1), without elaborated tuning, for example, À = 2e for the 2ele-ResNNtt22-11 10 architecture CIFAR- 10, 1e - 4 for ResNet-50 architecture on ImageNet, and À = 5e for MobileNet-Y V2 architecture on CASIA. These values correspond to isotropic Gaussian priors with of as 0. 0.0078, and 0.0041 on CIFAR-10, ImageNet, and CASIA, respectively. is notable that for a "small" dataset like CIFAR-10, flatter prior is preferred. While on larger datasets with stronger data evidence, we need a sharper prior for regularization.
For 7, use 7 0.75 for training across all the scenarios. But it is not used for OOD detection in the testing phase. For estimating the results of OOD detection, we use the non non-parametric metric average precision (see the metric part of Section 4), which is the Area Under the Precision- Recall Curve and is more suitable than the ROC-AUC metric when there is class imbalance.
For the pre-training, we follow standard protocols available online. On CIFAR-10, we perform CutOut (DeVries & Taylor, 2017) transformation upon popular resize/crop/flip transformation for data augmentation. On ImageNet, we leverage the ResNet-50 checkpoint on PyTorch Hub as the converged deterministic model. On face tasks, we train MobileNetV2 following popular hyper- parameter settings, and the pre-training takes 90 epochs. We use the same weight decay coefficients in both the pre-training and the fine-tuning.
For the fine-tuning, we set I to decay at 1/4, 1/2, and 3/4 of the total fine-tuning steps from 0.1, and set lr, to be the final value of tr on the CIFAR-10. ImageNet, and face recognition benchmarks. We add a coefficient 3 before the Lune term in Line 8 of Algorithm 1 for Bayesian fieettinng on ImageNet to achieve better uncertainty calibration. For models face recognition, features before the last FC layer of the MobileNetV2 architecture to conduct feature distance-based face classification in the validation phase, due to the open-set nature of the validation data. The Bayes ensemble is similarly achieved by assembling features from multiple runs as the final feature for estimating predictive performance. But we still adopt the output from the last FC layer for uncertainty estimation (i.c., calculating Eq. (6)).
utilize the
The training perturbation budget is identical to the evaluation budget on CIFAR-I0 and ImageNet. But we set the budget of the uniform noise used for training in face tasks to be 1/4 of the evaluation budget make the models more sensitive to the perturbed data. We adopt PGD for generating adversarial samples in the validation phase. Concretely, we attack the posterior predictive objective, 20 MC samples. On CIFAR- 10, we set 8m 0.031 and perform PGD for 20 steps with step size at 0.003. On ImageNet and face recognition, set d. 16/255 and perform PGD for 20 steps with step size at 1/255.
i.e., Eq. (3), with
Regarding the fake data, we craft 1000 fake samples for training and 10000 ones for evaluation with SNGAN (Miyato et al., 2018) on CIFAR-10; we craft 1000 fake samples for training and 1000 ones for evaluation with BigGAN (Brock et al., 2018) on ImageNet; we randomly sample 1000 fake samples for training and 10000 ones for evaluation from DeepFakes (Deepfakes, 2018), FaceSwap (Faceswap, 2018 8) and Face2Face (Thies et al., 2016) on face recognition.
14

---

Under review as a conference paper at ICLR 2021
Table 6: Comparison the quality of uncertainty estimates in terms of AP upper part is for adversarial instances the other part for DeepFake.
face recognition tasks. The
Method
LFW
CPLFW
CALFW
CFP FF
CFP FP
VGG2 FP
AGEDB 30
MAP
0.191
0.192
0.191
0.211
0.205
0.200
0.199
MC dropout
0.965
0.946
0.959
0.965
0.949
0.954
0.952
BNN
0.399
0,282
0.429
0.390
0.271
0.291
0.327
BayesAdapter-
0.189
0,189
0.189
0.193
0.191
0.191
0.190
BayesAdapter
0.998
0.981
0.999
0.999
0.983
0.990
0.995
MAP 0.389
0,456
0,375
0,394
0.454
0.519
0.437
MC dropout
0.846
0.664
0,862
0.874
0.685
0.733
0.785
BNN
0.621
0.399
0.648
0.559
0.355
0.469
0.516
BayesAdapter-
0.273
0.273
0.273
0.251
0.248
0.309
0.275
BayesAdapter 0.998
0.987
0.999
0.999
0.986
0.994
0.996
As for the MC dropout, we add dropout-O.. 0.3 (0.3 denotes the dropout rate) before the second con- volution the residual blocks in Mide-ResNet-22-- 10, dropout-0.2 after the second and the third convolutions in the bottleneck blocks in ResNet-50, and dropout-0.2 before the last fully connected (FC) layer in MobileNetV2.
For reproducing Deep Ensemble, we train 5 MAPs separately, and assemble them for prediction and uncertainty quantification. For reproducing SWAG, we take use its official implementation, and leverage 20 MC samples for decision making.
C MORE RESULTS FOR UNCERTAINTY ESTIMATION
We provide the comparison the quality of uncertainty estimates face recognition in Table 6. is an immediate observation that BayeAdapter outperforms the extensive baselines significantly, and can detect almost all the OOD instances across the validation datasets. By contrast, BayeAdapter- MAP, and BNN are similarly unsatisfactory. Surprisingly. MC dropout exhibits some capacity to detect adversarial instances and DeepFake ones in the face tasks. Comparing these results with those of MC dropout on CIFAR-10 10 and ImageNet, we speculate that such results may stem from the location of deploying dropout in the architecture, which deserves future investigation.
D VISUALIZATION OF SOME OOD DATA
We provide some random samples of the OOD data used for evaluation in Figure 6. Obviously, these samples pretty realistic and challenging.
E VISUALIZATION OF THE LEARNED POSTERIOR
We plot the parameter posterior of the first convolutional kernel in ResNet-50 architecture learned by Adapte on ImageNet. The results are depicted in Figure 7. The learned posterior variance seems be disordered, unlike the mean. We leave more explanations as future work.
15

---

Under review as a conference paper at ICLR 2021
Figure Some random samples of the OOD data used for evaluation. The first row refers the fake sam- ples from BigGAN on ImageNet. The second row refers to the adversarial examples generated by PGD on ImageNet. The third row refers to the fake samples from DeepFake.
16

---

Under review as a conference paper at ICLR 2021
Figure 7: Left: the mean of the Gaussian posterior. Right: the variance of the Gaussian posterior. These correspond to convolutional kernel with 64 output channels and 3 input channels, where every output channel plotted as separate image.
17


Reviews:
[{"review_id": "hUiMEzKBIpd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Contributions**\n\nThis paper proposes a post-hoc approach to obtain model uncertainty estimates from vanilla pre-trained NNs through MFVI fine-tuning. Namely, 1) the authors re-cast the KL divergence in the VI objective as weight decay applied to the variational parameters 2) the authors propose a variance reduction technique for the reparametrisation trick 3) the authors explicitly train their model to produce large model uncertainty on Out of Distribution (OOD) inputs. \nEmpirically, the proposed methods seems to retain the strong performance, and much of the simplicity, of point-estimate NNs while providing enhanced robustness in terms of uncertainty estimation. \n\n\n**originality and significance** \n\nTo my knowledge, most of the proposed techniques (or variants of them) have appeared before in the literature or are simple extensions of existing approaches: Re-casting MFVI as SGD [Khan et. al., 2018], Decorrelation of reparametrisation gradients across batch elements [Wen et. al., 2018], Training on OOD measurement points to produce large uncertainty [Hendrycks et. al., 2018 and Hafner et. al., 2018] .\nHowever, this work refines these ideas and puts them into a single framework which seems to produce strong results. I view this as a noteworthy contribution which might bring Bayesian Deep Learning closer to real world deployments. \n\n **clarity** \n\nMost ideas are presented clearly. The paper is well structured and easy to follow. Some passages are slightly ungrammatical but never does this impede the transmission of ideas.\n\n**pros**\n* Presents useful practices to make BNNs more mainstream with strong empirical performance.\n* Authors provide code for an efficient implementation of exemplar reparametrisation. \n* The proposed technique for OOD detection bypasses typical pathologies of MFVI [Ovadia  et. al., 2019] by explicitly optimising variational parameters to produce large model uncertainties OOD.\n\n**cons**\n* Exemplar reparametrisation is very similar to Flipout [Wen et. al., 2018]. A comparison of the two would be appreciated. \n* The proposed technique for OOD detection is not very principled and has provides no guarantees. It seems empirically successful however. \n* The experimental setup is not very clear, even when reading the supplementary sections concerning experimental setup. Some questions I was left with:\n\t* There are many hyperparameters, how did you find all of them? —Especially the weight decay coefficients.  Are the standard deviations implied by these priors interesting / meaningful in any way? \n\t* A single Mutual Information threshold is provided. Is this one used for uncertainty calibration training on all tasks? Is it also used when classifying inputs as in-distribution or OOD? If this is the case, it is possible that models without uncertainty calibration training would benefit from using a different threshold. A better metric might be ROC-AUC, as it is threshold agnostic.\n* The only baselines provided are other VI approaches. SWAG is known to be a decently strong baseline but it is not evaluated for OOD detection peroformance. The current  state of the art baseline for uncertainty quantification is deep ensembles [Ovadia  et. al., 2019]. These are much more expensive. However, it might be interesting to see how they compare.\n* The authors repeat all experiments 3 times but only provide mean results. In some cases, like tables 1 and 2, the values presented are similar across methods. I think that errorbars (standard deviation across 3 runs) would be very informative to the reader. \n\n**Other comments and questions:**\n* Typo in title: Bayesian, not Bayeisian\n* The maximum predictive entropy (and thus mutual information) will depend on dimensionality of output space (number of classes). In your experimental section, you say you set a single threshold for all models. Could you further comment on this?\n\n**References**\n[Wen et. al., 2018]  https://arxiv.org/pdf/1803.04386.pdf\n[Khan et. al., 2018] http://proceedings.mlr.press/v80/khan18a/khan18a.pdf\n[Ovadia  et. al., 2019] https://papers.nips.cc/paper/9547-can-you-trust-your-models-uncertainty-evaluating-predictive-uncertainty-under-dataset-shift.pdf\n[Hendrycks et. al., 2018] https://openreview.net/forum?id=HyxCxhRcY7\n[Hafner et. al., 2018] https://arxiv.org/abs/1807.09289", "extra_scores": {}, "extra_text": {}}, {"review_id": "dtybQUEcJ1L", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores the variational training of Bayesian neural networks. It proposes to improve the quality of the inferred variational posterior and computational efficiency of the procedure by (i) better initialization (mean parameters are initialized at the MAP) (ii) reducing variance in the Monte Carlo approximated evidence lower bound by increasing the number of weight samples (one per datapoint in a batch) (iii) a posterior regularization encouraging higher uncertainty on adversarially generated or other “near OOD” data.  \n\nThe authors use the term BayesAdapter to refer to the process of running black-box variational inference from a fully factorized variational approximation with mean initialized at the MAP estimate and randomly initialized variances. The fact that variational inference (especially those employing fully factorized approximations) are susceptible to poor local optima and that better initializations can help navigate these local optima is widely known. The fact that better initializations can lead to somewhat improved posterior approximations is not surprising. Such initializations are also standard practice when employing stochastic gradient MCMC techniques and Laplace approximations (where it is a requirement). Reducing variance by increasing the number of weight samples to one per datapoint in a batch is another straightforward idea, and it is unclear whether it can be claimed as a contribution of the current paper. Kingma et al., in their local re-parameterization considered a variant with per data samples as well. The uncertainty regularization is indeed novel and appears effective (but the experiments illustrating its benefits need to be better explained). \n\nGiven the modest methods contributions, the empirical section needs to be particularly strong to demonstrate that the combination of these incremental improvements provides meaningful empirical advantages. To their credit, the authors demonstrate their approach on several large datasets and do provide experiments for vetting different aspects of the proposed extensions to variational BNN training. However, many experiments are missing details and some are lacking key comparisons. Overall this section could be significantly strengthened. \n\n* Tables 1 and 2 need to include comparisons against deep ensembles and multi-SWAG (https://arxiv.org/pdf/2002.08791.pdf). If the goal of this paper is to claim that variationally trained BNNs (with the proposed improvements) are useful in practice, a natural question to ask is whether they are competitive with far simpler ensembling approaches that are able to account for the multimodality of the posterior surface, unlike variational BNNs.\n* How was the calibration threshold $\\gamma$ chosen for these experiments? How sensitive is the performance to this choice? Ideally, the authors would include results with different settings of $\\gamma$. How were the prior precisions selected (which determine $\\lambda$ set?  My main worry is that the marginal improvements provided by Bayesadapter variants over BNNs disappear when making slightly different parameter choices. It would be great if the authors can demonstrate that this isn’t the case. \n* Section 4.2 needs more details about the experimental setup. Did the 1000 / 10000 OOD training/test examples include both images created via PGD and SNGAN (for CIFAR 10) and PGD and BigGAN (for imagenet)? If so, how many from each source? If not, it would be interesting to see cross performance — using PGD images for training and SNGAN images for testing. \n* In Table 4, it doesn’t make sense to include ECE numbers from a different architecture trained via SWAG. These numbers are not comparable. Also, interestingly, both BayesAdapter variants have lower ECE scores than vanilla BNN on CIFAR, suggesting poorer calibration. Do the authors have an explanation for this? \n\nBased on concerns about both novelty and experiments I am currently leaning towards a reject, but could be convinced otherwise based on the authors’ response and additional comparisons.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cf1x6SffANo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed one simple and effective way to trainBayesian neural networks (BNN). \n\nPros:\n1. The proposed method is quite simple and cheap to realize, compared to previous Bayesian methods.  \n2. Extensive experiments on a diverse set of challenging benchmarks have been conducted, which shows several promising results of the proposed method.\n3. The proposed idea is novel which distinguishes from most of previous efforts, which try to train BNN from scratch using Bayesian methods. As described in this paper, most of previous methods, though paying much additional efforts than deterministic ones, do not lead to expected results, even with non-diagonal covariance matrices. The BayesAdapter, however, pays little efforts and obtains improvements even with diagonal covariance matrices. From this perspective, this is an encoraging result. \n\nCons:\n1. The results of comparison in Table 1 only repot the average result in 3 runs (3 is kind of small). However, it is better to show the std metric of the result to make the comparison more convincing because the improvement of BayesAdapter in average value is in fact not very apparent, especially compared with MAP. If the variance of the result is large, then there will be large overlap between different methods and thus it is not reasonable to claim that there is an apparent advantage over previous methods. \n\n2. In evaluating the result of BayesAdapter,  MC samples are used. What if only using the mean value of the posterior?  Compared to deterministic methods like MAP, inference using MC is more costly.   In addition, it is suggested to provide some visualizations of the posterior distribution after BayesAdapter. \n\n3. Based on results in Table 3, BayesAdapter- performs similar as baselines, which indicates that the improvement comes from calibrating the uncertainty estimation. This leads to another question: what if we also use such calibration for the baseline methods. It would be interesting to make such a comparison. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5quiF9Q1TYK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a fast way to get Bayesian posterior by using a pretrained deterministic model. Specifically, the authors first train a standard DNN model and then use it to initialize the variational parameters. Finally the variational parameters are optimized through standard variational inference (VI) training. To further improve uncertainty estimate, the authors propose an uncertainty regularization which maximizes the prediction inconsistency on out-of-distribution (OOD) data. Experiments including image classification and uncertainty estimates are conducted to demonstrate the proposed method.\n\nThe idea of this paper is quite simple: initialize the mean in variational parameters by a pretrained DNN. Thus the method is cheap and simple enough to use broadly in practice. The authors did reasonable empirical tests. I especially appreciate the ablation study which helps understand the method a lot. The paper is well-written and easy to follow.\n\nI mainly have the following concerns about the paper. \n\n- One of the main motivations to use a pretrained DNN is that BNN learned from scratch is worse than its corresponding DNN. I feel this claim is misleading. Many papers have shown that BNNs trained from scratch outperform DNNs. Particularly the paper [Wenzel et.al. 2020] which the authors cited to support their claim clearly shows that BNN (with reasonable temperature) is significantly better than DNN in predictive performance. But I do agree that the proposed method is cheaper than training BNNs from scratch. \n\n- The experimental results verify the effectiveness of the proposed method. But the results of the proposed method seem to be worse than BNNs training from scratch (e.g. the ImageNet results in [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning, ICLR 2020] are much better). This also supports my first point, that the main benefit of the proposed method is to get the Bayesian posterior fast and cheaply, instead of to improve BNNs’ performance. I think the authors should revise the claim to be more precise.  \n\n- The proposed method is closely related to [Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes, AAAI 2020] which also uses a pretrained DNN as the initialization of variational parameters. How does the proposed method compare to it theoretically and empirically? The main idea seems quite similar; the only difference is that the proposed method does not use the pretrained model as prior in VI. Due to the similarity, I think a comparison is necessary. \n \n- The authors argue that BNN training suffers from suboptimal local optima. Could the authors provide evidence/citations to support this claim? I do not think it is true. Perhaps it is true only for a few BNN methods such as BNN using naive VI.\n\n- As the proposed method is essentially a VI method, it would be interesting to see comparisons to SOTA VI methods.\n\nOverall I'm positive about this paper and would be happy to increase my score if my concerns are addressed. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims at improving the adoption of Bayesian NNs by providing a practical and user friendly variational inference method. The main ideas consist of two parts:
1. Warm-start the variational inference from a pre-trained deterministic NN. It takes advantage of existing deep learning library features for easy implementation including weight decay, batch matrix multiplication, etc.
2. Calibrating uncertainty estimation for out-of-domain detection using adversarial examples.

Pros:
1. A practical way of implementing DNN variational inference with reduced variance, without sacrificing classification accuracy of the pretrained NN model.
2. Significantly better OOD detection accuracy compared to other BNN approaches without taking OOD into account explicitly.

Cons:
1. During discussion, it becomes clear that most of the techniques have been proposed similarly in the literature. Krishnan, 2020 applied BNN starting from MAP of NN, Flipout (Wen et. al., 2018) applies instance-wise sampling, Hendrycks et. al., 2018 and Hafner et. al., 2018 improves detection accuracy by training on OOD examples. The novelty of the proposed method is therefore limited.
2. There's not much benefit on the classification performance compared to the initial MAP and is inferior to MCMC-based SOTA BNNs. One of the reviewers considers the SGLD-type approach may be more appealing to ML practitioners with the overhead of VI in training additional variance parameters.
3. The authors argue MCMC-based BNN methods cannot achieve good performance without temperature scaling. But the main performance improvement of the paper is in the OOD detection with uncertainty regularization that modifies the posterior as well. The method of training with OOD samples is orthogonal to applying Bayesian inference to NNs, and the detection performance is limited to the distribution close to examples during training.

This paper falls on the borderline for acceptance. With the goal of improving adoption of BNN in practice, it is not convincing yet making mean field VI easier to implement could realize it without achieving competitive performance.

Decision: Reject