Attacking Bayes: On the Adversarial Robustness of Bayesian Neural Networks
Authors: Yunzhen Feng
Tim G. J. Rudner
Nikolaos Tsilivis
Julia Kempe
Venue: Accepted by TMLR ()

Abstract: Adversarial examples have been shown to cause neural networks to fail on a wide range of vision and language tasks, but recent work has claimed that {\em Bayesian} neural networks (BNNs) are inherently robust to adversarial perturbations. In this work, we examine this claim. To study the adversarial robustness of BNNs, we investigate whether it is possible to successfully break state-of-the-art BNN inference methods and prediction pipelines using even relatively unsophisticated attacks for three tasks: (1) label prediction under the posterior predictive mean, (2) adversarial example detection with Bayesian predictive uncertainty, and (3) semantic shift detection. We find that BNNs trained with state-of-the-art approximate inference methods, and even BNNs trained with Hamiltonian Monte Carlo, are highly susceptible to adversarial attacks. We also identify various conceptual and experimental errors in previous works that claimed inherent adversarial robustness of BNNs and conclusively demonstrate that BNNs and uncertainty-aware Bayesian prediction pipelines are {\em not} inherently robust against adversarial attacks.

Published in Transactions on Machine Learning Research (07/2024)
Attacking Bayes:
On the Adversarial Robustness of Bayesian Neural Networks
Yunzhen Feng" New York University
yf2231 Onyu. edu
Tim G. J. Rudner*
ruw. edu
New York University
Nikolaos Tsilivis
nt2231 Onyu. edu
New York University
Julia Kempe
jk1 Dnyy. edu
New York University
Reviewed on OpenReview: https: www.review..eeeeeeeeeee net. forum? id= 17Vj17VBmu
Abstract
Adversarial examples have been shown to cause neural networks to fail on a wide range of vision and language lasks, but recent work has claimed that Bayesian neural networks (BNNS) are inherently robust to adversarial perturbations. In this work, we examine this claim. To study the adversarial robustness of BNNs, we investigate whether it is possible to successfully break state-of-the-art BNN inference methods and prediction pipelines using even relatively unsophisticated attacks for three tasks: (1) label prediction under the posterior predictive mean, (2) adversarial example detection with Bayesian predictive uncertainty, and (3) semantic shift detection. We find that BNNS trained with state-of-the-arr approximate inference methods, and even BNNS trained with Hamiltonian Monte Carlo, are highly susceptible to adversarial attacks. We also identify various conceptual and experimental errors in previous works that claimed inherent adversarial robustness of BNNS and conclusively demonstrate that BNNs and incertainyy-aaat Bayesian prediction pipelines are not inherently robust against adversarial attacks.
1 Introduction
Modern machine learning systems have been shown to lack robustness in the presence of adversarially chosen inputs so-called adversarial examples. This vulnerability was first observed in computer vision by Szegedy et al. (2014) and has since been shown to be consistent across various benchmarks and environments, including standard academic benchmarks (Carlini and Wagner, 2017b; Goodfellow et. al., 2015; Papernot et al., 2017), well as in less controlled environments (Alzantot et al., 2018; Kurakin al., 2017).
Several recent works-largee outside of the mainstream adversarial examples literature- have initiated the study of adversarial robustness of Bayesian neural networks (BNNS; MacKay, 1992; Neal, 1996; Murphy, 2013; Papamarkou et al., 2024) and claim to provide empirical and thcoretical evidence that BNNS are able to detect adversarial examples (Rawat et al., 2017; Smith and Gal, 2018) and to defend against gradient-based attacks on predictive accuracy to higher degree than their deterministic counterparts (Bortolussi al., 2022; Carbone et al., 2020; Zhang al., 2021). This has led to a growing body of work that operates under the premise of "inherent adversarial robustness" of BNNs, alluding to this "well-known" fact starting point (e.g,, De Palma et al., 2021; Pang et al., 2021; Yuan et al., 2021; Zhang el al., 2021).
In this paper, we investigate the claim that BNNs are inherently robust to adversarial attacks and able to detect adversarial examples.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Label Prediction
Adversarial Example Detection
Semantic Shift Detection
100
100
Average Selective Accuracy (%)
Clean
Adversarial
80
800
Accuracy (%)
60
20
20
MCD
HMC
Figure 1: Left (Label Prediction): Accuracy and robust accuracy test and adversarial inputs for CNNs trained on MNIST. Center & Right (Adversarial Example and Semantic Shift Detection): Average selective prediction accuracy (ASA) for adversarial examples and semantically shifted inputs on MNIST (with MNSS as the semantic shifted data). Note that ASA has a lower bound of 12.5%. Simple PGD attacks break methods all prediction pipelines. For further details, see Section
There are good reasons to suspect that, in principle, BNNs may be more adversarial robust than deterministic neural networks. BNNs offer principled way to quantify model's predictive uncertainty by viewing the network parameters random variables and inferring a posterior distribution over the network parameters using Bayesian inference. This way-unlike deterministic neural networks RNNS can provide uncertainty estimates that reflect limitations of the learned predictive model. Such epistemic uncertainty has been successfully applied to various tasks to build more reliable and robust systems (Neal, 1996; Gal and Ghahramani, 2016). Improvements in BNNs have been spurred by recent advances in approximate Bayesian inference, such as function-space variational inference (FSVI; Rudner al., 2022b), that yield significantly improved uncertainty estimation. It is, therefore, plausible that BNN's predictive uncertainly may be able to successfully identify adversarial examples and provide poteenttilll significant level of inherent adversarial robustness.
To evaluate empirical claims about inherent adversarial robustness of BNNS (c.g., in Smith and Gal, 2018; Bortolussi et al., 2022; Carbone et al., 2020), we examined open-source code provided by authors of prior works, identified implementation errors, and found that there is no evidence of inherent adversarial robustness of BNNs after fixing these errors. To validate this finding, we trained BNNS using well-established and tatt--o--thoartt approximate inference methods and evaluated their robustness. this evaluation, we focused three key tasks to assess the adversarial robustness of RNNs: 1) classification under the posterior predictive mean; 2) adversarial example detection, and 3) semantic shift detection. Semantic shift detection is a staple application of BNNs, previously not considered in the context of adversarial attacks. Across datasets, we found that none of the BNNs were able to withstand even relatively simple adversarial attacks. A summary of a representative subset of our results is shown in Figure 1.
To summarize, our key contributions are as follows:
We re-examine prior evidence the literature on robustness of BNNs for (i) adversarial example detection (Smith and Gal, 2018) and (ii) adversarial example classification (Bortolussi et al., 2022; Carbone et al., 2020; Zhang et al., 2021) and find that prior works do not convincingly demonstrate robustness against adversarial attacks. In particular, we find that results indicative of adversarial robustness in BNNs presented in previously published works are due to implementation errors and cannot be replicated once the errors are fixed. We extract common pitfalls from these findings and provide a set of guiding principles to evaluate the robustness of DNNs with suitable attack methods (Section 4).
2. We conduct thorough evaluations of DNNS trained with well-established and state-of-the-art approximate inference methods (HMC, Neal (2010); PSVI, Blundell et al. (2015); MCD, Gal and Ghahramani (2016); FSVI, Rudner et al. (2022b)) on benchmarking tasks such as MNIST, FashionMNIST, and CIFAR-10. We demonstrate that: (i) classification under the posterior prodictive mcan completely fails under adversarial attacks; (ii) adversarial example detection fails even under altacks targeting accuracy only; and (iii) in semantic shift detection (MNIST vs. FashionMNIST, CIFAR-10 vs. SVHN), adversarial attacks fool BNNs into rejecting in-distribution samples (Section 5). This work is the first to demonstrate this failure mode.
In summary, our analysis suggests that that-using recognized adversarial testing protocols BNNs do not demon- strate meaningful level of inherent adversarial nobustness.
2

---

Published in Transactions on Machine Learning Research (07/2024)
2 Background and Preliminaries
We consider supervised learning tasks with N i.idd (independent and identically distributed) data realizations
S = {x("), (() y())))))))))) y (xs.s) of inputs x E & and labels Y E y with input space X C R and label space
y S {0, 13 K for classification with K classes.
2.1 Adversarial Examples
An adversarial example (AE) for classifier = softmax((())) an input that is indistinguishable from a "natural" one (measured in some metric- typically an lp ball), yet it is being misclassified by ((-). In particular, if we let D be distribution over X' x y' and Bx encode our notion of indistinguisaabiityy from x, we will consider the following adversarial risk:
P ((xa))D) [EX E Bx: 0xx) vl.
(1)
This is sometimes referred to as the consland in the ball definition of adversarial examples (Gourdeau el al., 2019), because we assume that the magnitude of perturbation c is small enough that does not result in a change of the ground truth label. Please see the works of Diochnos et al. (2018); Gourdeau et al. (2019) for more discussion on different definitions of adversarial robustness in classification settings. All the papers we evaluated adopt. the same definition the experiment. Adversarial examples deep learning systems were first observed in Szegedy et al. (2014). Since then many approaches in generating such examples have been proposed called adversarial allacks (Carlini and Wagner, 2017b; Chen et al., 2017; Goodfellow et al., 2015; Papernot et al., 2017: Kurakin et al., 2017) and subsequently methods for shielding models against
defenses (Goodfellow et al., 2015; Madry et al., 2018; Papernot et al., 2016) been developed. Many such defenses have been found to be breakable either by carefully implementing already known attacks or by adapting to the defense (Carlini and Wagner, 2017b; Carlini et al., 2019b; Tramèr et al., 2020).
Formally, the process of generating an adversarial example x + 17 for a classifier 2(-) and natural input. under the &o distance involves the solution of the following optimization problem:
n = arg max |nll [((((x + n)).9).
for some e >> O that quantifies the dissimilarity between the two examples. There is some flexibility in the choice of the loss function C, but typically it is chosen to be the cross-entropy loss. In general, this is a non-convex problem and one can resort to first-order methods (Goodfellow et al., 2015):
X = x sign (VxC()(x),
(3)
or iterative versions for solving it (Kurakin et al., 2017; Madry et al., 2018). The former method is called Fast Gradient Sign. Method (FGSM) and the latter Projected Gradient Descent (PGD; standard 10, 20, or 40 iterates are denoted by PGD10, PGD20, PGD40, respectively). When classifier is stochastic, the adversarial defense community argues that the attack should target the loss of the expected prediction (Expectation over Transformation; Athalye et al., 2018a;b):
x = x sign (V
(4)
where the expectation is over the randomness at prediction time. Note that some works use expectation of gradients instcad (c.g., Gao et al., 2022). In the common case of convex loss functions, this is a weaker attack, however. One common pitfall when evaluating robust ness with gradient-based atlacks is what has been called obfuscated gradients (Athalye et al., 2018a), when the gradients become too uninformative (or zero) to solve the optimization Equation (2). A suite of alternative adaptive attack benchmarks (AutoAttack) has been developed to allow for standardized robustness evaluation (Carlini et al., 2019a; Croce and Hein, 2020; Croce et al., 2021)
3

---

Published in Transactions on Machine Learning Research (07 (07/2024)
2.2 Bayesian Neural Networks
Consider neural network ((;;)) defined in terms of stochastic parameters θ RP, For an observation model Pr|x.e and prior distribution over parameters pe, Bayesian inference provides mathematical formalism for finding the posterior distribution over parameters given the observed data, Pejs (MacKay, 1992; Neal, 1996). However, since neural networks are non-lincar in their parameters, exact inference over the stochastic network parameters is analytically intractable.
Full-batch Hamiltonian Monte Carlo. Hamiltonian Monte Carlo (μмс) is a Markov Chain Monte Carlo method that produces asymptotically exact samples from the posterior (Neal, 2010) and is commonly referrod to as the "gold standard" for inference in BNNS. However, HMC does not scale to large neural networks and is in practice limited to models with only few 100,000 parameters (Izmailov et al., 2021).
Variational inference. Variational inference is an approximate inference method that seeks to avoid the intractability of exact inference and the limitations of HMC by framing posterior inference as a variational optimization problem. Specifically, we can obtain a BNN defined in terms of a variational distribution over parameters ge by solving the optimization problem
min DKL(Ce Pajs) - ee
max F(ge),
(5)
where F(ge) is the variational objective
F(ge) = Eg logppoooosxx XD, 0] -Dxr ||
pP),
(6)
and Qo a variational family of distributions (Wainwright and Jordan, 2008). In the remainder of the paper, will drop subscripts unless needed for clarity.
Unlike IMC, variational inference is not guaranteed to converge to the exact posterior unless the variational objective is convex in the variational parameters and the exact posterior is a member of the variational family. Various approximate inference methods have been developed based on the variational problem above. These mothods make different assumptions about the variational family Qe and, therefore, result in different posterior approximations. Two particularly simple methods are Monte Carlo Dropoul (MCD; Gal and Ghahramani, 2016) and Porameter-Span Variational Inference under a mean-fiell assumption (PSVI; also referred to as Bayes-by-Backprop: Blundell et al., 2015; Graves, 2011). These methods enable stochastic (i.e., mini-batch- based) variational inference, be scaled to large neural networks (Hoffman et al., 2013), and can be combined with sophisticated priors (Klarner et al., 2023; Rudner et al., 2023; Lopez et al., 2023; Rudner et al., 2024b;a; Klarner et al., 2024). More recent work on Fumction-Space Variational Inference in BNNS (FSVI; Rudner et al., 2022b;a) frames variational inference as optimization over induced functions and has been demonstrated to result in state-of-the-at predictive uncertainty estimates in computer vision tasks,
Uncertainty in Bayesian neural networks. To reason about the predictive uncertainty of BNNS, we decompose the tolal uncerlainly of predictive distribution into its constituent parts: The alealoric uncertainty of model's predictive distribution is the uncertainty inherent in the data (according to the model), and a model's epistemic uncertainty (or model uncertainty) denotes its uncertainty based on constraints on the learned model (e.g., due to limited data, limited model capacity, inductive biases, optimization routines, etc.). Mathematically, we can then express a model's predictive uncertainty as
H ((E lp(y x, 0;
Ege [H(ply x, O;
+ 7(: Model Uncertainty
(7)
Total Uncertainty
Expected Data Uncertainty
where H(-) the entropy functional and Z(Y: ②) is the mutual information CaaiionnSSaaaa (Shannon and Weaver, 1949;
Cover and Thomas, 1991; Depeweg et al., 2018)
4

---

Published in Transactions on Machine Learning Research (07/2024)
2.3 Selective Prediction
Selective prediction modifies the standard prediction pipeline by introducing rejection class, 1, via galing mechanism defined by selection function s: X' + R that determines whether a prediction should be made for a given input point x E X' (El-Yaniv and Wiener, 2010). For a rejoction threshold т, the prediction model is then given by
ppy| xx 8 SST
(ply f()((())))(())
(8)
otherwise,
with ply| f) = Eug (p(y 1, f)l, where (()) is an approximate posterior for BNNS and a Dirac delta distribution, ( 800 0²), for deterministic models with learned parameters 0r. A variely of methods have been proposed to find selection function (Rabanser et al., 2022). BNNs offer automatic mechanism for doing so, since their posterior predictive distributions do not only reflect the level of noise in the data distribution via the model's aleatoric uncertainty which can also be captured by deterministic neural networks- but also the level of uncertainty due to the model itself, for example, due to limited access to training data or an inflexible model class, via the model's epistemic uncertainty. As such, the total uncertainty of BNN's posterior predictive distribution reflects both uncertainty that can be derived from the training data and uncertainty about model's limitations. The selective prediction model is then
(ply 1,, I) H(ply I (x)))) [ply|x x, f) H(ply x; f)))s
otherwise,
that is, a point x E X' will be placed into the rejection class if the model's predictive uncertainty is above a certain threshold. To evaluate the predictive performance of a prediction model (p(y) i s)),, we compute the predictive performance of the classifier p(y x; f) over a range of thresholds т. Successful selective prediction models obtain high cumulative accuracy over many thresholds.
There is rich body of work in the Bayesian deep learning literature that evaluates selective prediction in settings where some test data may exhibit semantic shifts (e.g., Band et al., 2021; Nado et al., 2022; Tran et al., 2022), arguing that the capability of BNNS to represent epistemic uncertiinty renders them particularly well-suited for semantic shift, detection.
3 Related Work
We begin by surveying existing work on inherent adversarial robustness of BNN prediction pipelines. Since the focus of this work is to investigate claims about inherent robustness of BNNs, an overview of works that altempt to explicitly incorporate adversarial training into BNN training (Liu et al., 2018; Doan et al., 2022) has been relegated to Appendix B.
To align this survey with our own results, we first outline prior work on adversarial example (AE) detection with BNNs, before proceeding to robustness of classification with BNNs. Note that while AE detection seems an easier task than robust classification, recent work (Tramer, 2022) shows that there is reduction from detection to classification (albeit a computationally inefficient one), which means that claims of a high-confience AE detector should receive equal scrutiny as robust classification claims would. In particular, after nearly a decade of work by an ever-growing community, only robustness results achieved with vevesaaiaal training (Madry al., 2018) have stood the test of time and constitute today's benchmark (Croce al., 2021) to establish empirical robustness against Community-standart perturbation strength. Note that there is an interesting body of work achieving certifiable adversarial robustness, but the robustness guarantees they achieve apply only for much smaller perturbation strengths.
Adversarial example detection with Bayesian neural networks. A first set of early works has investigated model confidence on adversarial samples by looking at Bayesian uncertainty estimates using the intuition that adversarial examples lie off the true data manifold. Feinman et al. (2017) give first schome using uncertainty estimates in dropout neural networks, claiming AE detection, which is subsequently broken in Carlini and Wagner (2017a) (who, incidentally break most. AE detection schemes of their time). Rawat.
5

---

Published in Transactions on Machine Learning Research (07/2024)
et al. (2017) analyze four Bayesian methods and claim good AE detection using various uncertainty estimates, but analyze only weak FGSM attacks on MNIST". Smith and Gal (2018) claim to provide empirical evidence that epistemic uncertainty of MCD could help detect stronger adversarial attacks (FGSM and BIM, a variant of PGD) on a more sophisticated cats-and-dogs dataset (refuted in Section 4). Bekasov and Murray (2018) evaluate AE detection ability of MCMC and PSVI, but do so only a simplified synthetic data setting. The first to demonstrate adversarial vulnerability of Baycsian AE detection (though not for BNNS) are Grosse et al. (2018), who attack both accuracy and uncertainty of the Gaussian Process classifier (see Appendix B.1). Several works leave the BNN-inference framework and thus are not our focus: by way of example Deng et al. (2021) design Bayesian tack-on module for AE, detection and Li et al. (2021) add Gaussian noise to all parameters in deterministic networks to generate distributions on each hidden representation for AE detection. To the best of our knowledge, our work is the first to demonstrate adversarial vulnerability of modern Bayesian inference methods (while examining and refuting previous claims about robustness of BNNs).
Robustness of Bayesian neural network predictions. Several recent works have hypothesized that BNN posteriors enjoy enhanced robustness to adversarial attacks, compared to their deterministic counterparts. Carbone et al. (2020); Bortolussi et al. (2022), guided by (correct) theoretical considerations of vanishing gradients on the data manifold in the infinite data limit for BNNS, claim to observe robustness to gradient- based attacks like FGSM and PGD for HMC and PSVI using a simple CNN and a fully-connected network (reevaluated and refuted in Section 4) Uchendu et al. (2021) examine the robustness of VGG and DenseNet. with Variational Inference and claim marginal improvement over their deterministic counterparts. Pang et al. (2021) evaluate Bayesian VGG networks for two inference methods (standard Bayes-by-Backpeee Flipout), claiming evidence for surprising adversarial robustness*. Cardelli et al. (2019), De Palma et al. (2021), Grosse et al. (2021); and Patane et al. (2022) study the adversarial robustness of GPS. None of these works benchmark recent Bayesian inference methods like FSVI, which are state-of-the-arl for prediction and semantic shift detection. Zhang et al. (2021) propose regularization that could improve adversarial robustness. Our ovaluation both in their and in the standard attack parameter setting shows no significant. robustness gain with their regularization (see Section 4). The first in the literature to give negative evidence for robust of BNNS is Blaas (2021), comparing a BNN trained with HMC to a deterministic network on MNIIT and INSh but observing no difference in robustness.
and
Robustness of semantic shift detection. To the best of our knowledge, there is no prior work examining the adversarial robustness of Bayesian distribution shift detection pipelines. Appendix B surveys some additional prior work.
4 Examination of Prior Claims About Bayesian Neural Networks Robustness
Here, we examine (and refute) all papers (to the best of our knowledge) that make adversarial robustness claims about BNNs that have publicly accessible code and have not. been previously refuted (Smith and Gal, 2018; Bortolussi et al., 2022; Carbone et al., 2020; Zhang et al., 2021). Each of them provides a different failure mode that will help illustrate our recommendations for evaluating robustness of BNNS at the end of this section. We note that in the adversarial robustness community, a model is considered robust only when it can resist adversarial perturbations generated with any method, as long as these perturbations are within the constraint set. As we will see, a common failure mode is careless attack evaluation, for instance because of double application of the softmax function in Equation (3) through inappropriate use of standard packages.
Adversarial example detection with epistemic uncertainty. Smith and Gal (2018) examine adversarial detection with MCD using a ResNet-50 on the large-scale ASSIRA Cats & Dogs datasct consisting of clean test images, adversarial samples on that test set, and noisy lest images with the same perturbation levels as the adversarial ones, where they use PGD10 in their attacks. They present empirical findings claiming that epistemic uncertainty in particular may help detect adversarial examples. However, after investigating their
It is widely accepted that many proposed adversarial defenses fail to generalize from MNIST (Carlini and Wagner, 2017s).
We do not contest the theoretical analysis in the infinite limit, but observe that it does not seem to support the empirical phenomenon.
Both Uchendu al. (2021) and Pang et al. (2021) have code released assess these claims. Pang al. (2021) also distinguishes between "variational inference" and "Bayes-by- Blayes-by-Backprop although Bnyes-by-Backprop variational inference methor. We adversarially break Rayes- hy-ye--b-Backke Backprop (L.., PSV1) in Section 5.
6

---

Published in Transactions on Machine Learning Research (07 (07/2024)
100
PGD10 Total
Accuracy (%)
PGD10 Epistemic
60
PGD40 Total
PGD40 Epistemic
TransferPGD+ + Total
TransferPGD+ + Epistemic
20
0
20
40
60 80 100
% of Data Rejected by Uncertainty
Figure 2: Selective Accuracy for the AE detection in Smith and Gal (2018). TOTAL and EPISTEMIC refer to the thresholding uncertainty. A decrease in accuracy as the rejection rate increases indicates that the model rejects more clean than adversarial samples the rejection threshold decreases. Even for the weakest attack model accuracy alone, the essentially flat curve demonstrates that detection is no better than random. There is no advantage in using epistemic uncertainty rather than total uncertainty.
code, we find several problems: leakage of batch statistics at test time, faulty cropping of noisy images and the "double-softmax" problem resulting in improper evaluation (more details in Appendix C). This Icads the BNN to accept the noisy images and reject the clean and adversarial ones, resulting in misleading, overly optímistic ROC curves. After correcting these errors, no evidence of successful AE detection from selective prediction remains. Figure shows the updated selective accuracy using both total and epistemic uncertainty thresholds. A successful AE detector would have monotonically increasing accuracy curve, whereas we see flat decreasing curves meaning no better than random detection and no advantage of epistemic over total uncertainty. Lastly, we implement stronger attacks (PGD40 and TransferPGD+, see Appendix C) to remonstrate the complete failure of this AE detection method.
Robustness of Bayesian neural network accuracy. Carbone et al. (2020) and Bortolussi et al. (2022) present empirical results claiming robustness of BNN accuracy duc to vanishing gradicnts of the input with respect to the posterior. They implement FGSM and PGD for BNNS with HMC and PSVI on MNIST and FashionMNISTT to show robust accuracy. However, when examining the publicly accessible code, we found that instcad of using Equation (4) to calculate gradients, they compute expectations of gradicnts. Combined with large logit values before the softmax layer that lead to numerical overflow and result in zero gradients for large fraction of samples, this leads to inadvertent gradient masking. In addition, we also found the double-softmax problem mentioned above. After correcting and rescaling logits to avoid numerical issues (see Appendix C), their models are entirely broken by PGD40, see Table 1. Note that our critique specifically addresses the empirical evidence in Carbone et al. (2020), and not at all their theoretical analysis.
Regularization for robust accuracy of Bayesian neural networks. Zhang et al. (2021) propose to defend against. adversarial attacks by adding a regularization term and present empirical results for enhanced accuracy robustness for MNIST and CIFAR (most for non-standard settings of attack parameters like smaller radius). We find that the adversarial gradient is erroncously computed on singular randomly
Table 1: Robust accuracy (in %) on MNIST for BNNS trained with HMC and PSVI. We show results reported in Carbone et al. (2020); Bortolussi et al. (2022) and our reevaluation.
Table 2: Robust accuracy (in %) MNIST for the "robust" BNN proposed by Zhang et al. (2021). We show results for an MLP trained on MNIST using the original and a revised evaluation protocol.
Clean
FGSM
PGD40
Radius
FGSM
PGD15
Reported Reevaluated
96.0
97.0
Original setting
44.2311.32
11.38
HMC
= 0.16
95.8910.23
11. 11.62.12.4
2. 2.1910.12
Reevaluated
24.0410.95
4.22.0.11
Reported
93.6
93.8
Original setting
25.3020.91
3.64
PSVI
Reevaluated
97.35-0.18
51.45+3.77
0.07+0.05
0.3
Reevaluated
8.63+1.62
0.04
7

---

Published in Transactions on Machine Learning Research (07 (07/2024)
selected model during the inference stage, instead of the expected prediction in Equation 4. Furthermore, the typer-parameters utilized for the attack are not aligned with established standards. After fixing these flaws, the robustness claims do not hold any more, as shown in Table 2.
4.1 Recommendations for Evaluating the Robustness of Bayesian Neural Networks
Having examined these three robustness claims, we draw several conclusions about pitfalls and failure modes, and list detailed recommendations to avoid them when allacking Bayesian pipelines:
1. When considering randomness, use the strongest attack appropriate for the model, all other things being equal. In the case of stochastic models, attack the loss of the average (Equation (4)), rather than the average loss (at least for convex losses).
2. Beware of double softmax. All prior works examined in this paper provide implementations that apply the softmax function twice. Nearly all BNNS output the probability prediction, and is necessary to remove the softmax function from the loss to effectively apply standard attack implementations.
3. Fix all normalization layers but enable all stochastic network components (such dropout) al test time.
4. Monitor gradient values throughout the attack to avoid numerical issues. The gradients when attacking accuracy should never be vanishing to zero.
5. Increase the radius of perturbation
assess whether the model can be broken or not. In general, if the perturbation is large enough, all models should fail to be robust (e.g. the semantics of an input change if we perturb it by much). If the model remains robust, check for the existence of vanishing or uninformative gradients in the attacks (Athalye et al., 2018a). If the model still remains robust, attempt to attack a deterministic network using the parameters from the posterior or several fixed samples.
6. If PGD atlacks fail, consider using refined attack benchmarks like AutoAttack (Croce and Hein, 2020).
7. Design attacks appropriate for the model pipeline. Consider how to break the model based on its design, such as largeting both accuracy and uncertainty estimation (like the PGD+ allack we introduce in Section 5). Adaptiving attacks to the model quantities can provide more confidence in robustness results.
5 Empirical Evaluation of Adversarial Robustness in Bayesian Neural Networks
Here we present our findings on the lack of adversarial robustness of BNN pipclines for the three tasks: 1) classification with posterior prediction mean (Section 5.1), 2) AE detection (Section 5.2), and 3) OOD detection (Section 5.3). We evaluate four Bayesian Inference methods, HMC, MCD, PSVI, and FSVI for three datasets: MNIST, FashionMNIST, and CIFAR-10. We implement two architectures, a four-layer CNN and ResNet-18. All hyperparameter details can be found in Appendix E. Reproducibility, Code to reproduce our results can be found at
Threat model and generation of adversarial perturbations with FGSM, PGD, and PGD+. We consider a full while-bor attack: The adversary has knowledge of the entire model and its parameters, and can obtain samples from its posterior, which is the white-bax model considered in most prior work (Carbone et al., 2020; Bortolussi et al., 2022; Zhang et al., 2021). We apply FGSM and PGD with 40 iterations attack expected accuracy (to break robustness in Section 5.1) or uncertainty (see Equation (7)) (to fool OOD detectors in Section 5.3) as per Equation (4). To devise stronger attack on AE detectors, create a combined attack, PCD+: the idea is to produce adversarial examples that both fool the classifier (to drive accuracy to zero) and have low uncertainty (lower than the clean samples), resulting in poor predictive accuracy, worsening for higher rejection rates. To this end, PGD+ first attacks BNN accuracy in the first 40 iterates (using the BNN prediction as its label) and, from this starting point, computes another 40 iterates to attack uncertainty within the allowed c-ball around the original point. PGD+ does not need the ground truth to attack uncertainty (unlike the uncertainty attack in Galil and El-Yaniv (2021)). We settle on PGD+ after observing empirically that it is stronger than PGD80 targeting accuracy targeting linear combination of
also
8

---

Published in Transactions on Machine Learning Research (07/2024)
Table 3: Robustness of BNN and DNN to adversarial attacks.
The table shows robust accuracy (in %).
MNIST (c 0.3)
MNsHi
0.1)
CIFAR-10
(c 8/255)
Methods
Clean
PGSM
PGD
Clean
PGSM
PGD
Clean
FGSM
PGD
HMC
CNN
99.2620.00 20.44 0.57 00.0
92.33=0.04
32.7940.888
6.73+0.50
CNN
99.3910.04
10.19
0.52 1.003
93.04 10.00
18.45.2.78
1..14
MCD
ResNet- 18
99.39+0.05
10. 10.1
5.20=0.05
94.27=0.05
7.58=0.29
4.40=0.11
94.12=0.07 26.45+0.15
4.23
CNN
99.36+0.00 03
14.24
0.53=0.01
92.4770.07
28.20=6.94
5.72+0.18
SCIMC
RosNet-18
99.5610.01
3.53
0.39
10.54
27.72
13.71
5.5610.07
89.48
21.4510.23
7.0110.07
CNN
99.0710.0r 07
12.18
0.40.1..1
92.47
29.11
5..1
SCLD
ResNet-11
99.53 00
14.97
0.42=0.00
92.61
=3.83
5.72+0..2
90.16=0.17
22.40+0.37
6.69
CNN
99.21+0.02
2.60++.02
0.64==..2
92.58=0.01
13,95=0.88
5.50
PSVI
ResNet-18
99.59+0.0e 2.07=0.25
0.36=0.02
94.22=00.8
16.86=5.64
4.32+0.10
94.75+0.26 37.17+1.11
5.25+2.27
CNN
99.27+0.01 11.9
0.60=0.06
92.58=0.25
23.93=1.87
5.45+00.28
PSVI
ResNet- 18
99.5810.02 02 6.4513333
0.09
93.62
28,231 63
4.6010.02
93.4810.18
43.8510.04
5.1810.27
CNN
99.3420.01 37.18
1.10
22.27
34.73 62
8.0220.23
Deterministic
ResNet-18
99.56+0.02
3.64++..4 0.39
93.93=0.11
11.12=0.78
4.64+0.08
93.46=0.10 23.95 4.74+0.10
accuracy and uncertainty, as its two stages are specifically designed to fool AE detectors. Throughout, we use 10 samples from the posterior for each iteration of the attack (see Equation (4)). We find it unnecessary to increase the attack sample size for our attacks and hence adopt this number for computational efficiency. We only apply gradient- based atlacks; since these are sufficient. for breaking all BNNS, we do not need to implement AutoAttack (Croce and Hein, 2020). We showcase for attacks with standard attack parameters: c= 0.3 for MNIST, c 0.1 for FashionMNIST, = 8/255 for CIFAR-10. These perturbation magnitudes are the ones most commonly adapted in the robustness literature, in particular since they lead to complete misclassification by standard models. For completeness, we also include comprehensive results on smaller in Appendix F. We only consider total uncertainty in our attack and for selective prediction, as we found no evidence of an advantage in using epistemic uncertainty.
Metrics: We report accuracy from posterior mean prediction with 100 samples. Our notion of robusiness for BNNs is the natural one that aligns with deployment of BNNS: Robust accuracy is given by the fraction of correctly classified adversarial samples when predicting the class with the largest posterior mean prediction probability. As summary statistics for the selective prediction curves we use average selective accuracy (ASA), that is, the area under the selective accuracy curve computed with rejection rates from 0% to 99% in integer increments; and average negative log-likelihood (ANLL), for which we average NLL of all non-rejected samples for the same rejection grid (see Appendix D).
5.1 Assessing Robust Accuracy of Bayesian Neural Networks
Table 3 shows the predictive accuracies from our evaluation. It demonstrates significant deterioration in predictive accuracy when evaluated on adversarial examples even for the weakest attacks (FGSM) with a complete breakdown for PGD for all methods and datasets considered. Note that for deterministic neural networks, robust accuracy under adversarial attacks approaches 0% while for our attacks on BNNS it is in the low single digits (still below the 10% accuracy for random guessing). Since the goal of this work is to evaluate claims of significant adversarial robustness of BNNS, have not optimized our attacks to drive accuracy to approach zero but believe this to be possible. Compared with the deterministic model, we observe no significant difference in robust accuracy. In Appendix F, we provide the robustness curve for a range of smaller adversarial radii and observe no significant difference either.
5.2 Assessing Robust Adversarial Example Detection
A AE delection selling: We evaluate all AE detectors on Lest data consisting of 50% clean samples and 50% adversarially perturbed samples, using total uncertainty for the rejection as described in Section 2.2. In the case of perfect accuracy on clean data and 0% accuracy on adversarial samples, a perfect AE detoctor would start with 50% accuracy at 0% rejection rate, increasing linearly to 100% accuracy at 50% rejection rate, for maximum ASA of 87.5%. A completely defunct AE detector, on the other hand, would start at 50%
9

---

Published in Transactions on Machine Learning Research (07/2024)
Table 4: Selective Prediction (left) and average selective accuracy (right) for semantic shift detection with BNNS. Results on MNIST, FMNIST are with a CNN, while results on CIFAR-10 are with a ResNet-18.
Selective Prediction
Average selective accuracy
Clean
Noisy
FGSM
PGD
PGD+
Clean
Noisy
FCSM
PGD
HMC
99.9840.00
99.95
86.67:1011 63. 73
60.1620.38 33
83.9210.02
83.6510.11
61.02=1.54
47.1612.05
MCD
99.99=0.00 99.9720.00 0.000 83.33
24.58 2:
1.73
83.8820.10
13.84
55.22=0.04
20.01
SCHMC
99.9940.00 99.98 00 83.7310.23
23.3
27-70=1.20
99.3620.04
83.9520.10
66.23=2.50 50
24.4811.92
MNIST
SCLD
99.97=0.01 99.97=0.01
77.92+0.90
13.1929.52
37.67=13.35
83.4340.36
83.4550.35
54.14=6.34
28. 28.866213.6
99.98=0.00 99.98=0.00 oo
74.98:40.83
15.28
20.28=0.70
83.9240.01
83.8310.04
48.21
18.03=0.54
FSVI
99.98=0.00 98.81=1.00
88.53
28.79
17.52=0.66
84.10=0.01
84.17=0.03
60.00+11.60
15.45+00.1
HмC
98.9910.00 00.00000000000 98.7610.00 01
76.22..041
41.30.0..0 47.7310.67
60
77.57.0022 10.04
61.90
45. 15.14
MCD
99.1810.00 01 99.0710.01
15.31
31.9210.77
28.8910.44 45
77.1410.61 08.26
40.52:3342
21.4010.77
FMNIST
PSVI
98.98.0.00
98.8610.00 os
62.03:12.02
30.6810.58
24.92.10. 49
75.26:1040 0..66
46.54-2.23
16.8110.17
FSVI
98.5840.00 04 97.9310.24 28 69.5911.48
10.94:1.91
24.17:1..8 82
76.7510.55 79.97
28.94=2.00 68
15. 15.1510.03
MCD
99.37 99.35+t.00
+6.97+0.21
19.73+0.2 22.85+0.49
24
77.51+0.42 75.31 78.18
+0.01
SGHMC
97 97.9140.01 97.90+0.01
73.98+0.32
41.04
32.20+0.12
76.56+0.0s 76.56+0.07
58.50=0.48
18.35
CIFAR
SGLD
98.36+40.06 06 98.36+0.06
+2.23++0.24
+..92+0.91
27.83++.33
26 7..44
55.88
16.49 1..0
-10
PSVI
99.40 99.3640.04
0..5
09.7
19.89+1.51
79.16+
78.30
72.97
15.33 00.9
FSVI
09.06.10.0
00.00 99.0440.07 05.16 20. 0..38 20.09
80.5440.17 80.72
15.24 0..14
and reject clean samples first, reaching 0% accuracy at 50% rejection rate for minimum ASA of 12.5%. A random detector would yield a horizontal curve with ASA 50%. To benchmark, we also show ASA for 100% clean Lest data ("Clean") and for a 50-50 mix of clean and noisy data where we add a pixel-wise Gaussian perturbation with the same standard deviation as the radius in our adversarial perturbations ("Noisy").
Results: Table 4 lists our results for ASA, with all methods failing under attack, coming quite close to the idealized minimum ASA of 12.5%. Figure 3 (MNIST with CNN) (and Figures 8, 5, 6, 7 in Appendix D for the other datasets and architectures) illustrate the selective accuracy curve for the benchmarks and our three altacks, FGSM, PGD and PGD+, and show a histogram of uncertainties for adversarial samples. Table 5 in Appendix D further lists ANLL. Our results show that our iterative attacks, PGD and PCD+, essentially completely fool AE detection. Particularly interesting is the fact that PGD samples, optimized against predictive accuracy only (but not against uncertainty), already manage to fool the uncertainty thresholding, resulting in decreasing accuracy curves! The principled two-stage PGD+ attack that also targets uncertainty directly further lowers detection performance and decreases uncertainty further for all adversarial samples. We note that the weaker FGSM attack is not successful: As the histograms show, uncortainty does not fall below that of clean samples and the selective accuracy curve increases with an increased rejection rate.
DROPOUT,
105
120
Soloc Accino
70
60-
33
50
30
60
80
200
20
150
23
60
10
BD
1.50
10
100
ficecced by Uncollarty Histogram
of Dusa Recctod by Tosal Unseriaitty
Rajected Uncortainty
Data Rejected by Total Uncatainty
10f
10
Histogram
10a-
Histogram
Histogram
101
18%.
107
101
102
187.
102.
Tonal Unceriaiity
TM6 Uncuriaiity
TUKJl Uncortaityy
T:l2 Unceraaryy
Clean
Clean and Noley
Clean and FGSM
Clean and PGD
Clean and PGD+
Clean
FGSM
PGD
PGD+
Figure& Adversarial example detection statistics for all four methods on MNIST with a four-layer CNN architecture. Higher curves correspond to better adversarial example detection. The adversarial attacks are able to significantly deteriorate OOD detection in all settings and for all methods.
10

---

Published in Transactions on Machine Learning Research (07/2024)
5.3 Assessing Robust Semantic Shift Detection
Semantic shift detection a common application of BNNS, where they outperform state-of-the-art deterministic uncertainty quantification methods for NNS (Band et al., 2021; Rudner et al., 2022b; Tran et al., 2022; van Amersfoort et al., 2020). Building this body of work, should not be constrained to attacking predictive accuracy, since key application of Bayesian neural networks is to enable effective semantic shift detection. As such, it is important to inquire whether it is possible to not just attack the BNN's accuracy but also their uncertainty estimates.
argue that attacking Bayesian neural networks
Setting: Our semantic shift datasets for MNIST, Fashion and CIFAR-10 are FashionMNIST, MNIST, and SVHN, respectively, each of them giving zero accuracy, The test set contains half in-distribution (ID) and half semantically-shiited
(OOD) samples, hence selective accuracy curves start at 50% accuracy. We attack only the OOD samples with FGSM and PGD, targeting the uncertainty. Since this is alroady sufficient to reject most ID samples, we do not also attack the ID samples, though we could do so.
Results: Selective accuracy curves are shown in Figure (for MNIST, using a CNN). See 8, 5, 6, 7 in Appendix D for the additional datasets and architectures. Our results resemble what we have seen for AE detection. The PGD attack nearly complctely fools the detector to reject all ID samples, reaching close 0% accuracy for 50% rejection rate and indeed most methods give ASA close to the lower bound of 12.5%, with only HMC showing higher ASA. Still, even for HMC ASA is below 50, meaning it performs worse than random rejection. As before, FGSM attacks are too weak to turn the direction of the selective accuracy curve.
6 Discussion and Conclusions
In our empirical analysis, we have presented evidence that refutes claims in the literature that BNNs enjoy some natural inherent robustness to adversarial attacks and that they can be successfully deployed for adversarial example detection. We benchmarked set of contemporary BNN inference methods to further corroborate this finding, In addition, we are the first to demonstrate that uncertainty- detection of semantic shifts with BNNs is as vulnerable to relatively simple allacks as conventional prediction lasks: Slightly perturbing the semantically shifted samples can lead the model to reject in-distribution samples instead.
Bayesian adversarial training. We thus hope to draw the attention of the Bayesian deep lcarning community lowards devising Bayesian defenses against adversarial attacks. The gold standard to create robust deterministic models is adversarial training (Madry et al., 2018). Transposing this idea to BNNs poses some interesting challenges and opportunities since BNNs allow tailoring priors but may face distinct difficulties when optimizing the posterior adversarially. Some prior work has touched upon adversarially training BNNS or using BNNS to improve adversarial training of deterministic models (Doan et al., 2022; Liu et al., 2018; Uchendu et al., 2021; Wicker et al., 2021). (We provide a short review of related work in this area in Appendix B.) Yet, while relevant, these works have not proposed defenses that preserve proper Bayesian inference, and they do not assess their methods on Bayesian prediction pipelines that include uncertainty quantification: They add modifications that depart from Bayesian inference, subtly but improperly change the variational objective, and only consider accuracy-based adversarial robustness. Finding a principled and conceptually simple approach to defending BNNs (and possibly providing enhanced
robust models)
MNIST. HMC, CNN
INT MC DROPOUT, CNN
MNISS. PSVI, CNN
MNIST. CNN
165
100
1:5
Sccodive Accrrayy10
93
20
20
03
of Duia Reoddad by Total Uncoflarity
of DuGa Recollod Tosal Unsorialitty
of Dota Rijectood
Total Uneataiiyy
Duta Rejociod by Toll Uncanaaryy
Clean and DOD
Clean and Noisy DOD Clean and FGSM OOD
Clean and PGD OOD
..... Random Rajection
Figure 4: Semantic shift detoction statistics for HMC, MCD, PSVI and FSVI for MNIST with CNN. Higher curves correspond to better OOD detection. The adversarial attacks able to significantly deteriorate OOD detection in all settings and for all methods.
11

---

Published in Transactions on Machine Learning Research (07/2024)
constitutes exciting avenue for future research. Such work should explicitly focus defending Bayesian prediction pipelines using adversarial training approaches consistent with Bayesian inference.
Adversarial robustness and inference quality. In our empirical evaluation, we found that even BNNS trained with HMC, the gold standard for inference in BNNs, do not withstand adversarial attacks and exhibit a signíficant deterioration in robust accuracy, average selective accuracy, and semantic shift detection when either their predictions or their predictive uncertainty are attacked. However, our analysis of HMC is limited to CNN architecture with only 100,000 parameters since training larger BNNs with HMC is computationally infeasible without super computer- grade hardware, leaving the adversarial robustness of larger BNNs trained with HMC an open question. As would be expected, the different approximate inference methods examined in this work were less robust. than IMC on the uncertainly-awant selective prediction metrics, and for ResNet-18 18 models, FsVI-a a state-of-the-ar approximate inference method- is more robust against strong attacks on the uncertainty-aware selective prediction metrics than PSVI and MCD, two well-cstablished but empirically worse methods (sce Table 4).
white-box threat model for BNNs. It would be interesting to explore vulnerability of BNNS against transfer attacks from models that do not have access to the posterior, especially in light of recent progress in deriving downstream BNNS by creating priors from rained publicly-available models (Shwartz-Ziv et al., 2022; Tran et al., 2022). Lastly, selective prediction works well for clean and noisy data but easily broken by adversarial attacks of standard strength. An interesting direction for future research would to explore the effect of varying the perturbation strength on adversarial example detection.
Transfer-attack threat models. Most works have focused
Acknowledgments
YF, NT, and JK acknowledge support through NSF NRT training grant award 1922658. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
12

---

Published in Transactions on Machine Learning Research (07/ (204))
References
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'icht Tsuji,, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, Brussels, Belgium, October 31- November 4, 2018, pages 2890-2896. Association for Computational Linguistics, 2018.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Slockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 274-283. PMTR,, 2018a.
Anish Athalye, Logan Engstrom, Andrew Tlyas, and Kevin Kwok. Synthesizing robust adversarial examples. In Jemnifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 284 283. PMLR, 2018b.
Neil Band, Tim G. J. Rudner, Qixuan Feng, Augelos Filos, Zachary Nado, Michael W. Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal. Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks. 2021.
Artur Bekasov and Tain Murray. Bayesian adversarial spheres: Bayesian inference and adversarial examples in noiseless setting. arXiv preprint arXiv:1811.13333 2018.
Arno Blaas and Stephen J Roberts. The effect of prior lipschitz continuity on the adversarial robustness of Bayesian neural networks. arXiv preprint arXiv:2101. 026606900,0289,, 2021.
Arno C Blaas. On the adversarial robustness of Boyesian machine learning models. PhD thesis, University of Oxford, 2021.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural networks. volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, Lille, France, 07-09 Jul 2015. PMLR.
Luca Bortolussi, Ginevra Carbone, Luca Laurenti, Andrea Patane, Guido Sanguinetti, and Matthew Wicker. On the robustness of Bayesian neural networks to adversarial attacks. arXiv preprint arXi::2207.06154, 2022.
Ginevra Carbone, Matthew Wicker, Luca Laurenti, Andrea Patane', Luca Bortolussi, and Guido Sanguinetti. Robustness of Bayesian neural networks to gradient-based attacks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 15602 15613. Curran Associates, Inc., 2020.
Luca Cardelli, Marta Kwiatkowska, Luca Laurenti, and Andrea Patane. Robustness guarantecs for Bayesian inference with gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7759-7768, 2019.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, ATSec '17, page 3 14, New York, NY, USA, 2017a. Association for Computing Machinery.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pages 39-57. IEEE Computer Society, 2017b.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint 1rXis:1902.00605, 2019a.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan J. Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. CoRR, abs/1902.06705, 2019b.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. 700: zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Bhavani M. Thuraisingham, Battista Biggio, David Mandell Freeman, Brad Miller, and Arunesh Sinha, editors, Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, A ATSec@CCS 2017, Dallas, TX, USA, November 3, 2017, pages 15 26. ACM, 2017.
Adam D Cobb and Brian Jalaian. Scaling hamiltonian monte carlo inference for Bayesian neural networks with symmetric splitting. Uncertainty in Artificial Intelligence, 2021.
Thomas M. Cover and Joy A. Thomas. Elements of information theory. Wiley, New York, 1991.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirly-ffth Conference on Neural Information Processing Systems Dalasels and Benchmarks Track (Round 2), 2021.
Giacomo De Palma, Bobak Kiani, and Seth Lloyd. Adversarial robustness guarantees for random deep neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2522-2534. PMLR, 18 24 Jul 2021.
Zhijic Deng, Xiao Yang, Shizhen Xu, Hang Su, and Jun Zhu. Libre: practical Bayesian approach to adversarial detection. In Proceedings of the IEFE/CVF Conference on Compuler Vision and Pullern Recognition, pages 972 982, 2021.
Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Jemnifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1193, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Dimitrios Diochnos, Saeed Mahloujifar, and Mohammad Mahmoody. Adversarial risk and robustness: General definitions and implications for the uniform distribution. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Besachiancch, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NewrIPS 2018, December 9-8, 2018, Montréal, Canada, pages 10380-10389, 2018.
Bao Gia Doan, Fhsan M. Abbasnejad, Javen Qinfeng Shi, and Damith Ranasinghe C. Bayesian learning with information gain provably bounds risk for robust adversarial defense. In Kamalika Chaudhuri, Stelanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 33309-5323... PMLR, 17-23 Jul 2022.
Ran EI- El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11 11(53):6055-164) 2010.
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. Detecting adversarial samples from artifacts, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICMI. 2016, pages 1050-1059, 2016.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Ido Galil and Ran El-Yaniv. Disrupting deep uncertainty estimation without harming accuracy. Advances in Neural Information Processing Systems,
2021.
Yue Gao, Ilia Shumailov, Kassem Fawaz, and Nicolas Papernot. On the limitations of stochastic pre-pit pre-prreessing defenses. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Newral Information Processing Systems, 2022.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, Sun Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, and James Worrell. On the hardness of robust classi- fication. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-BBuc Emily B. Fox, and Roman Garnett, oditors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7444-74533 2019.
Alex Graves. Practical variational inference for neural networks. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS'11, page 2011. Curran Associates Inc. ISBN 978161839599..
Red Hook, NY, USA,
Kathrin Grosse, David Pfaff, Michael Thomas Smith, and Michael Backes. The limitations of model uncertainty in adversarial settings. arXiv preprint 1Ki:1882002606 2018.
Kathrin Grosse, Michael T Smith, and Michael Backes. Killing four birds with one gaussian process: the relation between different lest-time attacks. In 2020 25th International Conference on Pollern Recognition (ICPR), pages 4696 4703. IEEE, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770 778. IEEE Computer Society, 2016. doi: (0.1109CCVP.20011
J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI'13, pages 282 282-290, Arlington, Virginia, United States, 2013. AUAT Press.
James Hensman, Alexander G. de G. Matthews, and Zoubin Ghahramani. Scalable variational Gaussian process classification. In AISTATS, 2014.
Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(1):1303 1347, May 2013. ISSN 1532-4435.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are Bayesian neural network posteriors really like? In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages (629-4644 PMLR, 18 24 Jul 2021.
Leo Klarner, Tim G. J. Rudner, Michael Reutlinger, Torsten Schindler, Garrett M. Morris, Charlotte Deane, and Yee Whye Tch. Drug Discovery under Covariate Shift with Domain-Infrmmed Prior Distributions over Functions. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2023.
Leo Klarner, Tim G. J. Rudner, and Yee Whye Teh. Garrett M. Morris, Charlotte Deane. Domain-awark guidance for out-of-distribation molecular and protein design. In Proceedings of the 41th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2024.
Anna-Kathrin Kopetzki, Bertrand Charpentier, Danicl Zügner, Sandhya Giri, and Stephan Günnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? In International Conference on Machine Learning, pages 5707-5718. PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (07/2024)
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Regresentations, TCLR 2017, Toulon, France, April 24-26, 2017, Workshop Trock Proceedings, 2017.
Yao Li, Tongyi Tang, Cho-Jui Hsieh, and Thomas Lee. Detecting adversarial examples with Bayesian neural network. arXiv preprint ar arXiv:2105 05.08020, 2021.
Xuanqing Liu, Yao Lá, Chongruo Wu, and Cho-Jui Hsieh. Adv-BNN: Improved adversarial defense through robust Bayesian neural network. In International Conference on Learning Representations, 2018.
Julian Lechuga Lopez, Tim G. J. Rudner, and Farah Shamout. Informative priors improve the reliability of multimodal clinical data classification. In Machine Learning for Health Symposium Findings, 2023.
David J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Comput.. 4(3): 448-472, May 1992. ISSN 0899-7667. doi: 10.1162/neco.1990 199........9....44...
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Represenlations, 2018.
Kevin P. Murphy. Machine learning: A probabilistic perspective. MIT Press, Cambridge, Mass. [u.a.]. 2013.
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael W. Dusenberry, Scbastian Farquhar, Qixuan Feng, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim G. J. Rudner, Faris Sbahi, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran. Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning, 2022.
Radford M Neal. Bayesian Learning for Neural Networks. 1996.
Radford M. Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 54:113-162, 2010.
Yutian Pang, Sheng Cheng, Jueming Hu, and Yongming Liu. Evaluating the robustness of Bayesian neural networks against different types of attacks. arXiv preprint Sti:::1066002222 2021.
Theodore Papamarkou, Maria Skoularidon, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hemnig, Jose Miguel Hernandez Lobato, Aliaksandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nemeth, Michael A. Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Teh, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position paper: Bayesian deep learning in the age of large-scale ai. In Proceedings of the 41th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2024.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In IEEF Symposium on Security and Privacy, SP 2016, Son Jose, CA, USA, May 22-26, 2016, pages 582-597. IEEE Computer Society, 2016.
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Ramesh Karri, Ozgur Sinanoglu, Ahmad- Reza Sadeghi, and Xun Yi, editors, Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pages 506 519. ACM, 2017.
Andrea Patane, Arno Blaas, Luca Laurenti, Luca Cardelli, Stephen Roberts, and Marta Kwiskk Adversarial robustness guarantees for gaussian processes. Journal of Machine Learning Research, 23:1 55, 2022.
Stephan Rabanser, Anvith Thudi, Kimia Hamidich, Adam Dziedzic, and Nicolas Papernot. Selective classification via neural network training dynamics, 2022.
16

---

Published in Transactions on Machine Learning Research (07/2024)
C. E. Rasmussen and C. K. L. Williams. Gaussian Processes for Machine Learning. MIT Press, Massachusetts Institute of Technology, Cambridge, MA, USA, 2006. URL www. www.CaussianProcess.oom grgmgl...... gpml.
Ambrish Rawat, Martin Wistuba, and Maria-Irina Nicolae. Adversarial phenomenon in the eyes of Bayesian deep learning, 2017.
Tim G. J. Rudner, Freddie Bickford Smith, Qixuan Feng, Yee Whye Teh, and Yarin Gal. Continual Learning via Sequential Function-Space Variational Inference. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2022a.
Tim G. J. Rudner, Sanyam Kapoor, Shikai Qiu, and Andrew Gordon Wilson Function-Space Regularization in Neural Networks: A Probabilistic Perspective. In Proceedings of the 40th International Conference on Machine Learning, Proceodings of Machine Learning Rescarch. PMLR, 2023.
Tim G. J. Rudner, Xiang Pan, Yucen Lily Li, Ravid Shwartz-Ziv, and Andrew Gordon Wilson. Fine-tuning with uncertainty-aware priors makes vision and language foundation models more reliable. In TCML, 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024a.
Tim G. J. Rudner, Ya Shi Zhang, Andrew Gordon Wilson, and Julia Kempe. Mind the GAP: Improving robustness to subpopulation shifts with group-aware priors. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 2024b.
Tim G.J. Rudner, Zonghao Chen, Yee Whye Teh, and Yarin Gal. Tractable Function-space Variational Inference in Bayesian Neural Networks. In Advances in Neural Information Processing Systems, 2022b.
Vikash Schwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, and Prateek Mittal. Better the devil you know: An analysis of evasion attacks using out-of-distribution adversarial examples. arXiv preprint arXiv:1905.01190 2019.
Claude E. Shannon and Warren Weaver. The Mathematical Theory of Communication. University of Illinois Press, Urbana and Chicago, 1949.
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and An- drew Gordon Wilson. Pre-train your loss: Easy Bayesian transfer learning with informative priors. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 560-569. AUAT Press, 2018
Edward Snelson and Zoubin Ghahramani. Sparse Gaussian processes using pseudo-inputs. In Y. Weiss, B. Schölkopf, and J. C. Platt, editors, Advances in Neural Information Processing Systems 18, pages 1257 1264. MIT Press, 2006.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, 2014.
Florian Tramer. Detecting adversarial examples is (Nearly) as hard as classifying them. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learwing Research, pages 11692-217082 PMLR, 17-23 Jul 2022.
Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Hugo Larochelle, Mare' Aurclio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
17

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snock, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions, 2022.
Adaku Uchendu, Daniel Campoy, Christopher Menarl, and Alexandra Hildenbrandt. Robustness of Bayesian neural networks to white-box adversarial attacks. In 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (ATKE), pages 72-80. IEEE, 2021.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference Machine Learning, 2020.
Martin J Wainwright and Michael Jordan Graphical Models, Exponential Fomílies, and Variational Inference. Now Publishers Inc., Hanover, MA, USA, 2008. ISBN 1601981848.
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q Weinberger, and Andrew Gordon Wilson. Exact Gaussian processes on million data points. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 14648 14659. Curran Associates, Inc., 2019.
Matthew Wicker, Luca Laurenti, Andrea Patane, Zhuotong Chen, Zheng Zhang, and Marta Kwiatkowska. Bayesian inference with certifiable adversarial robustness. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Stofislics, volume 130 of Proceedings of Machine Learning Research, pages 2431 2439. PMLR, 13 15 Apr 2021.
Nanyang Ye and Zhanxing Zhu. Bayesian adversarial learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associales, Inc., 2018.
Matthew Yuan, Matthew Robert Wicker, and Luca Laurenti. Gradient-free adversarial attacks for Bayesian neural networks. In Third Symposium on Advonces in Approzimate Bayesian Inference, 2021.
Huimin Zeng, Zhenrui Yue, Yang Zhang, Ziyi Kou, Lanyu Shang, and Dong Wang. On attacking oul-domain uncertainty estimation in deep neural networks. In Lud De Raedt, editor, Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAI-22, pages -893-4899. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, 1CMT, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Reseorch, pages 7472 7482. PMLR, 2019.
Jiaru Zhang, Yang Hua, Zhengui Xue, Tao Song, Chengyu Zheng, Ruhui Ma, and Haibing Guan. Robust Bayesian neural networks by spectral expectation bound regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 3815 3824, June 2021.
Roland S Zimmermann. Comment on 'Adv-BNN: Improved adversarial defense through robust Bayesian neural network". arXiv preprint ar 1070009077,0885 2019.
18

---

Published in Transactions on Machine Learning Research (07/2024)
Supplementary Material
Table of Contents
A Reproducibility
20
B Further Background and Related Work
20
B.1 Gaussian Processos
20
B.2 Investigating Bayesian Neural Network Priors
20
B.3 Robustness of Semantic Shift Detection
................................
20
B.4 Adversarial Training and Bayesian Neural Networks
................................
20
B.5 Adversarial Training Bayesian Inference
21
C Details of Evaluation of Prior Work
21
C.1 Attacking Adversarial Example Detection in Smith and Gal (2018)
21
C.2 Attacking Robustness in Carbone et al. (2020) and Bortolussi et al. ...2............................
24
C.3 Attacking Robustness via. Regularization Zhang et (2021)
D Further Results
27
D.1 Tabular Results
................................
27
D.2 Additional Figures
27
E Experimental Details
30
F Varying the Strength of the Adversarial Attacks
31
G Limitations
31
19

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Appendix A
Reproducibility
Code to reproduce our results can be found at:
Appendix B
Further Background and Related Work
Here summarize prior work that, while not directly relevant to our results, provides interesting comple- mentary analyses.
B.1 Gaussian Processes
Gaussian processes (GPS; Rasmussen and Williams, 2006) are non-parametric alternative to BNNs. They correspond to anfinitely-wide stochastic neural networks (Neal, 1996) and allow for exact posterior inference in small-data regression tasks but require approximate inference methods to be applied to classification tasks and to datasets with more than few ten thousand data points (Snelson and Ghahramani, 2006; Hensman et al., 2013; 2014). While recent work has enabled exact posterior inference for GPS larger datasets using approximate matrix inversion methods (Wang et al., 2019), classification, and especially prediction tasks with high-dimensional input data such as images, require parametric feature extractors and approximate methods, for which there are no guarantees that the approximate posterior distribution faithfully represents the exact posterior, and underperform neural networks and BNNS.
B.2 Investigating Bayesian Neural Network Priors
Blaas and Roberts (2021) ask how the prior could affect adversarial robustness for BNNS. For PSVI on a three-layer fully-comnected network, they observe a trade-ofT between accuracy and robusinnss (reminiscent. of the rccuracy-rbbusteess trade-off for deterministic NNs (Zhang et al., 2019)): small priors yield small Lipschitz constant and thus better robustness but cannot fit the data; for large priors the opposite is true.
B.3 Robustness of Semantic Shift Detection
The profound advantage of Bayesian neural network is that it can provide both aleatoric and epistemic uncertainty estimations because of the probabilistic representation of the model. In particular, large body of work leverages this to create Bayesian pipelines for semantic shift detection. To the best of our knowledge, there is no work prior to ours examining Bayesian OOD pipclines for their robustness against adversarial attacks and our work is the first to do so. There is prior work on robustness of semantic shift detection with deterministic models which we briefly survey here: Sehwag et al. (2019) attack the semantic shift detection of deterministic models with calibration and temperature scaling. Kopetzki et al. (2021) attack both accuracy and semantic shift detection for Dirichlet-based models. Zeng al. (2022) atlack out-domain uncertainty estimation for deep ensembles and uncertainty estimation using radial basis function kernels and GPs. All these works break the semantic shift detection capabilities of the models.
In a different though related work, Galil and Yaniv (2021) attack in-distribution data to increase the uncertainty of correctly classified images and decrease the uncertainty of incorrectly classified images, thus targeting selective accuracy in an in-distribution setting for deterministic networks as well as MCD. Their attack requires knowledge of the ground truth for the instances it attacks, which might be difficult to attain.
B.4 Adversarial Training and Bayesian Neural Networks
Some works have used Bayesian methods to improve robustness of deterministic models. For instance, Ye and Zhu (2018) introduce uncertainty over the generation of adversarial examples to improve adversarial training.
Liu et al. (2018) propose to create robust models with a version of adversarial training for BNNs guided by the intuition that randomness in the model parameters could enhance adversarial training optimization
20

---

Published in Transactions on Machine Learning Research (07/2024)
against adversarial attacks. However, Zimmermann (2019) refutes these claims by observing that robustness significantly diminishes when using expected gradients to attack. Moreover, it is unclear whether the observed remaining robustness claims solely come from the adversarial training components in the algorithm. Uchendu et al. (2021) directly combine adversarial training with BNN by training on iteratively generated adversarial examples, and observe small improvements in nosust Doan el al. (2022) make the interesting observation that direct adversarial training of BNNs as in Liu al. (2018) might lead to mode collapse of the posterior and propose a new "information gain" objective for more principled adversarial training of BNNS.
We also argue that Liu et al. (2018) and Doan et al. (2022) depart from the standard Bayesian inference framework. The foundation of variational inference is laid out in the equivalence given in Equations (5) and (6) in Section 2.2. In Liu et al. (2018) and Doan et al. (2022), the data uD in log loggyjx,e (yD|<> 0; is replaced by the adversarial examples generated on the fly and an additional regularization term is introduced. Since the training data is being modified, the oquivalence above not given anymore and the solution to the variational optimization problem under the modified data does not approximate the exact posterior in the original model, moving outside the realm of standard Bayesian inference. Therefore, we call for future work on a principled and conceptual approach to defending Bayesian inference pipelines.
An interesting work deals with certifiable robustness Wicker et al. (2021). It also modifies the standard variational objective to adversarially train a BNN, thus cleverly optimizing a posterior with improved robustness. However, as before, this approach departs from the scope of Bayesian inference, since changing the variational objective in this way biases the objective and will not lead to the best approximation to the posterior within the variational family. In other words, the approach in Wicker et al. (2021) may superficially look like approximate Bayesian inference but is not. Moreover, we point out that their code also applies the incorrect double softmax and expected gradient computations to generate the attack. These errors lead to a robust accuracy for HMC on FMNIST with e = 0.1 of 40%, which is significantly higher than the robust accuracy we would observe after fixing these errors (around 6%). This further emphasizes the importance of ro-ovaluating previous robustness ovaluations in publishod works and for heoding the recommendations for robustness evaluations put forth our work.
B.5 Adversarial Training as Bayesian Inference
Bayesian inference in neural networks involves finding a conditional distribution p(0 yD, yD, (the posterior) given a likelihood er (VDlED, 0: and a prior p(0). Here, D is the observed data, and there are various ways to find approximations to the posterior (e.g., variational inference, HMC, etc.). Liu et al. (2018) and Doan et al. (2022) depart from the standard Bayesian inference framework of finding p(0 yD.
More specifically, in the adversarial training methods proposed by [1] and [2]. the data sp in logpy|x,e (yD 0; f) is replaced by the adversarial examples generated in an adaptive way during training and an additional regularization term is introduced into the objective functions. Each of these modifications means that the proposed methods do not seek to find ((OlyD,*D) anymore. First, modifying the training data implics that the true postcrior is continually evolving as a function of the very model that is being trained, which is not consistent with finding p(0|yp, TDD) Second, changing the optimization objective by adding an ad-hoc regularization term changes the optimum and, as such, implies that the solution to the optimization problem is not the posterior p (p))
Both approaches, therefore, diverge from the standard Bayesian inference problem of finding p(0 TD).
Appendix C Details of Evaluation of Prior Work
Here we provide further details on implementation issues in prior work. This section complements Section 4.
C.1 Attacking Adversarial Example Detection in Smith and Gal (2018)
In addition to implementing the intended PGD10 attack on the BNNs, we additionally provide evaluations for two stronger attacks, PGD (=PGD40) and TransferPGD+. We use the same Lx0 perturbation as in their results, 10/255. PGD directly attacks the BNN accuracy for 40 iterations with the gradient of the loss of the
21

---

Published in Transactions on Machine Learning Research (07/2024)
expectation, as discussed in Section 2.1. At each iteration use 10 samples from the posterior. Figure 2 shows that PGD10, and more so PGD already fool the AE detector leading it to reject more clean samples than adversarial samples, as witnessed by the decreasing selective accuracy curve. However, when tracking gradients for PGD40 we find some degrec of gradient diminishing. Therefore, we design a stronger attack called TranslerGGD+. As ils name indicates, it is a two phase altack that consists of transfer alackk that warm-starts subsequent PGD+. That is, we attack the deterministic non-dropout version of the BNN first. (by turning off dropout when computing the loss), and then perform 40 iterations of PGD on the uncertainty estimation of the network starting from the transferred adversarial examples generated in the first phase, again with 10 posterior samples per iteration. We use the label predicted by the BNN to avoid label leakage. TransferPGD+ can break the accuracy of the BNN to 2% and fools the AE detector successfully.
Implementation issues: After investigating the code privided by Smith and Gal (2018), we find two mutually reinforcing problems. First, they evaluate the clean, adversarial, and noisy data separately but do not set nomaalioomiiintionn layers evaluation mode. (The first highlighted line in Listing 1). As result, is possible that the detector may use batch statistics among the different sample groups distinguish them. Secondly, when creating the noisy data, re-centered images with initial values in [0, 255] mistakenly clipped to [0, 1], making all noisy data points very similar to each other (since most information is clipped to cither O or 1), and leading to very small epistemic uncertainty for noisy images. See Listing 4. This leads the BNN to accept the noisy images and reject the clean and adversarial ones, resulting in misleading, overly optimistic ROC curves (this effect was further amplified once we fixed the batch-normallzztion issue). Moreover, care needs to be applied when using standard packages, developed for vanilla NNs, to BNNs. Deterministic models tend to operate logits and the softmax and cross-entropy calculations are combined, and therefore standard attack packages apply softmax function these pre-loss outputs. BNNS, on the other hand, average the probability predictions from posterior samples after the softmax function, and hence directly produce probabilities. A direct application of standard attack packages to BNNs would apply the softmax function to class probabilities (a "double softmax problem"), thereby making the class probabilities more uniform and weakening the attack strength (Listing 1 and Listing 2). See Listing for the softmax the standard package, Cleverhans. The model already implements softmax in Listing 1. To fix this problem, we change the softmax function in Cleverhans to the bnn loss from Listing 3.
Note that Smith and Gal (2018) show the ROC curve and provide AUROC values, while we have chosen to show svective accuracy curves throughout our work. AUROC and ASA are incomparable, we do not show comparison of these metrics here. Both curves quantify the performance of the AE detector and our findings (see Figure 2) show failure to detect AE.
line 62 def noeiine_ddee__eeeeeet
2
X. sete get__learning_hhae (True)
3.
rn50 Resllet500weiihhta= 'imagenet', include_top- "False)Faase))
a Dropou (rate=0.5) 5) (rn50. (rn50.output))
a Dense (2, activation= ') (a)
model keras models Model (inputt==== iin.iitttrr....iuut.. outputs=a)
The first highlighted line shows that the batch normalization layers are set to be True, which should be done in training mode but not for evaluation. The second highlighted line shows the softmax operation in the model. Code is copied from https://b master/ cats and_adddde _oddd_dggppyyyy (commit dbc7ec5).
Listing 1:
22

---

Published in Transactions on Machine Learning Research (07 (07/2024)
line 9 fast_grrddent.mettooll
model_fn,
x,
eps, norm,
loss_fn-None,
clip_min=lone,
clip_max-None,
y=None, targeted=False,
canity_checks=aalee,
):
line 46 if loss_fn is None:
loss_fn www.sparre__eee pparree_oftaarccrosseeetttpoopp logits
Listing 2: The second softmax when performing the attack from httpp://ggtthu...c/////l///eeree httpc///etaaan//o//////e////eeee http://////aaaattttttt/tt//////t tettackk/tast_graddentt pytptdd._et.dyyy....m Cleverhans uses the same cross-entropy loss across its different versions, also see line 142 in v3.1 at
https///tttubb.com/cllvvrrrat http://hhw....oooooooooooooooooa att.://wrr.onaaaaaaa/a/ee/aaaaar attacke/fast_gradient: method.py
def bnn, Lan_loss([]) (labels, logita): labels tf one (labels, 2)
return tf math, reduce_sum (tf math
labels)
Listing 3: The NLI loss to avoid double softmax.
line noise random random (sizeeexxxi- phape)
noise /- (dists np. linalg
woall..oormoooo norm (noise. reshape (x_plus__p______epp pllpppluusssoeeeeeeeeeppppppllle Sos.ussnesesssspeeC.. [0], 1), axis-1)) None, None, None]
3
[:,
polls_nnosse +- noise
x_plus_noise np. clwp(xxppus_noiiee noise, 0, 1)
line 344 attackk_paraeeee params
"method"; "fgm",
"eps": 5, "clip_min": -103.939, "clip_max"; 131.32,
"ord": np inf,
Listing The incorrect clipping added the noisy images. The first highlighted line shows the clipping of noisy images to and However, from the other highlighted lines, one can that the images actually are the [0,255] scale.
23

---

Published in Transactions on Machine Learning Research (07/2024)
C.2 Attacking Robustness in Carbone et al. (2020) and Bortolussi et al. (2022)
As we discuss in Section 4, the BNNs trained in this evaluation tend to have large values before the softmax layer, resulting in gradient vanishing, Therefore, we renormalize the logits by 100 to fix numerical issues when attacking the trained model. We evaluate these adversarial examples on the unnormalized network. With the double softmax corrected (see Listings and 6) for issues with the uncorrected code, one can use standard PGD on the loss of the expectation as Equation (4) break the accuracy nearly 0% on MNIST with perturbation radius 0.3.
line 69 class BlY(PoooM))
line 121 def guide x_data, y_data=None):
dists = +
for key, value in self www.eeteetttteeeeietteeeeeeeeeee i)ems():
loc pyro- www.ppaarraaaaaaa (roorrrr torch. wandn_like (valuu))))
scale pyro patppaan(ssrt"eeeee_eccllll
torch randn likee
distr Hormal(1oc=oc,, acalersoftplus (scalee dista ufaate((str (aen(attrrkyy)y)ee :distr))
10
11
lifted_module p random_module ("module", self basenet, dists)
12
13
with pyro plate plate("data) ("data", len(x_data)):
14
logits cefted_module(x_dde
18
preds nnf soft (logits, dim--1)
return preds
Listing The first softmax in http:///thhcooooooo////////////t (commit 71843ba).
line Tgsm_attackknet, image, label, Hyperparane-None, n_ samples-NNne,, wwg_postorior-Palee))
2
epsilon hyperparama ("porparran("essioooo] hyperparams not None else 0.3
image requires_grad True
output net. forwad (input--inn,. o_eamples-n_sampled wwg.pesterrorrray.ppttrioo
loss torch. nn. CroasEntropyLose (output, label) net. net.zero_gra zet.zero_gra())
loss. backward)
image_grad image. btage.grad.om .ata
perturbed_ imagg image epailon * mmage_grad.sigge
perturbed torch. cechh.clmppppp. (parturbdd _imgee, 0, 1)
return perturbed_imaee
Listing 6: The second softmax in http:///tthhaak///aarr/////ttttt adversariallttacks.pp (commit 84e39ee).
24

---

Published in Transactions on Machine Learning Research (07/2024)
C.3 Attacking Robustness via Regularization in Zhang et al. (2021)
We use the publicly available code to train Bayesian MLP MNIST and then use the evaluation code to assess its adversarial robustness for different values of perturbation budget e. We find that the adversarial gradient is erroncously computed on singular randomly selected model during the inference stage, instead of the expected prediction in Equation (4) (see Listing and Listing 8 for our proposed fix). Furthermore, the hyper-parameters utilized for the attack are not aligned with established standards. Specifically, the PGD stepsize set to 10,000 (Listing 9), while the usual stepsize is smaller than the adversarial budget e. We set this stepsize to 1/10 of the overall c.
In Table 2, using their code, we have reevaluated the results in Zhang et al. (2021) in their setting (with the double-softmax) and after fixing it, for 0.16, where their work claimed the largest benefit of regularization, and the more standard c = 0.3.
line 31 def (gam(model, X, y, norm, epsilon):
delta torch. zeroe_likede (euuireesgrad-Tree)
X2 nor= delta) cuda()
outputs model mX2e
loss mn mn. CrossEntropyLoss( (outputs, c))
loss backward()
return epsilon * delta. iitaa.grad..e. grad.sign()
The highlighted line shows that the adversarial altack only performed on one sample from the model. The adversarial attack is weak due to not attacking the full model. The code is copied from eetllleellllaaapppp (commit e5b17ce).
Listing 7:
def igam(model, X, y, norm, epailon):
delta torch. zeros zeros_like requires_grad-Trus
norm(X delta) cuda()
outputs, outputs torch nn functional softmax dim=-1)
model. model. samples
[fiaed code] logit MC samp les
# [fieed code] application of softmaz to logst MC samples
outputa- outputs. mean (dim=0)
[fimed code] Monte Carlo estimate of posterior predictive mean (yptpuu)) t)))
outputs outputs[0]
loss = mn. MLLLoss (torch (torch. log + 1e-10), y. cuda())
# [fimed code] cross entropy loss computation with mean of softmaz predictions as input loss. backward()
return epsilon delta, grad.sign()
Listing 8: Our modification: We changed the code to directly attack the expected prediction in
Equation (4).
25

---

Published in Transactions on Machine Learning Research (07 (07/2024)
line 57 aadhiladd__nniiseaeeeee adythhaddaooiisaattacc noise_ratios):
assert attack ['pg'', 'fgam ']
cost_dev 0.0
err_dev 0.0
erra 0
for noise_ratio in noise_ratios:
ab_samples 0.0
for J, (x, y) in mnuerate
if attack = 'pgd':
x_noise
pgd (net.mmddddl,, x, MNIST_ moise_ratio, 1e5, 15)
line 42 def (gddel x, y, norm, epsilon, alpha, num_iter):
delta torch. www.seeereeeeeeee
delta data. uniform_ (-epsilon, epsilon)
for (ange(mum_tter))
norm delta) cudaa)
17
outputs model (X2)
type (outpuus))) type(()::
18
outputs outputs [0]
19
20
loss nn CrossEntropyLLoss (outputs, y c)da())
21
loss. backward
delta. (delta alpha delta. grad. .tt...iaaaaaa......... sign()) (110)))cllpppmmmm cwamp(-epsileeeeeee epsilon) delta. grad zero_()
return delta. ()tach()
Listing 9: The code shows the wrong step size for PGD in the PGD code. The code was copied from http:////s.aaaaaa..aa/.///////// http:///BB__eallatiinnn... (commit e5b17ce).
26

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Appendix D
Further Results
D.1 Tabular Results
Table 5 shows the average negative log likelihood (ANLL) for AE detection on Section 5.2. The NLL is an evaluation metric of interest, since it reflects the degree of confidence in a prediction and penalizes underconfident correct predictions as well as overconfident wrong predictions. For classification models, the NLL given by the cross-entropy loss between the onc-hot labels and the predicted class probabilities.
Table 5: Detecting Adversarial Examples with BNNS. Average Negative Log-Likelihood.
Clean
Noisy
FGSM
PGD
PGD +
HMC
0.20+0.00
0.1040.04
1.51+0.15
4.06+0.23
4.45=0.51
MCD
0.00 +0.00
0.00 +0.00
1.70 +0.04
10.42 40.17
10.03 40.24
CNN
PSV1
0.00
0.00
2.48#0.12
8.75+0.29
10.84
MNIST
FSVT
0.02 #0.00
0.08 #0.03
0.42 =0.09
7.48 20.36
10.34 #0.22
0.00 =0.00
2.34 40.37
1.60
10.60 #0.00
11.56
ResNet-18
PSVI
0.00 10.00
1.09 10.17
1.53 10.17
11.11 10.05
11.58 1:065
FSVI
0.04:0.05
0.59
0.84
9.12:0066
10.29
MCD
0.05 +0.00
0.05 +0.00
2.77 +0.07
9.34 #0.11
9.78 =0.07
CNN
PSVT
0.05 =0.00
0.05 #0.00
4.15 20.30
9.23 10.07
10.08 10.06
FSVI
0.08 20.00
0.12 20.01
1.83 20.12
8.75 10.28
9.73 20.25
FMNIST
MCD
0.06 20.00
0.09 20.00
3.58 20.27
11.01 #0.06
11.06 #0.03
ResNet-18
PSVI
0.10 #0.01
0.15 40.01
1.63
9.84
10.01
FSVI
0.08 10.01
0.12 10.01
1.69 10.19
10.65 10.03
10.86 10.06
MCD
0.04 10.00
0.04 10.00
2.67
10.03
10.64 10.07
11.08 1:0.3
CIFAR10
R8
PSVI
0.04 10.01
0.05 10.01
1.61
10.35
10.18 11.38
10.44 11.08
FSVI
0.06 10.01
0.06 10.01
1.08 10.03
10.18 10.00
10.62 #0.07
D.2 Additional Figures
In addition to the selective prediction curves shown in Sections 5.2 and 5.3, we also generate the full sets for each Seeautt-methhd--rchiteeeeee setting. The results shown the following pages in
Figure 5 for MNIST with a ResNet-18,
Figure 6 for FashionMNIST with a CNN,
Figure for FashionMNIST with a ResNet-18, and
Figure for CIFAR-10 with ResNet-18.
27

---

Published in Transactions on Machine Learning Research (07 (07/2024)
MNIST, MC DROPOUT, ResNet18
MNISTT PSVI, ResNor18
MNIIS: FSVI. ResNet1
70
70
60
60
50
60
Data Sejected Unnertainty
o4 Total uncetaeity
of Dats Sejected Uincerittity
Histggram
Histogram
Histogram
10a
10
10a
10'
10
19'
132
10a
12a
0.0
1.0
15
0.0
0.5
2.0
2'5
0.0
0.5
1.0
3
2.9
DROPOUT, Restatt1
MNIST, PSVI, Restict18
ResNiett
100
100
90-
Acsunny
80
70
80
50
40
30
Data Rojected by Tota uncortality
of Data Rejected by Total unceraaiity
% Dora Rdddted Toal Uncoraiitty
Noiay
Clean and FGSM
00D
Random Rejection
Figure 5: Adversarial example and semantic shift detection statistics for MCD, PSVI, and FSVI on MNIST with a ResNet-18 18 architecture
PNISST HMC
DVNEST MC DACROUT
PURRST CAN
DNMEET, CWN
24
4. Rocceii ceuetarly
Bull Raniaa
listggram
Hatogram
10
1P2
Clean
Clean and Naisy
Clean and FGSM
Glan and PGD
Gean and PGD
Clean
FGSM
PGD
PGD
300
FMNIST, HMC,
300
DROPOUT, ENN
120
PSV,
1.0
FSVI CNN
5.3
Selvettum
27
100
70
20
101
25
100
Noisy Uncerbiitt
Rejection
Figure 6: Adversarial example and semantic shift detection statistics for HMC, MCD, PSVI, and ESVI on MNIST with a CNN architecture
28

---

Published in Transactions on Machine Learning Research (07 (07/2024)
FMNIST, MC DROPOUT, ResNet18
FMNIST, PSVI. ResNer18
FMNIST, FSVL ResNer18
70
70
60
60
50
30
50
60
Data Sejected Total Unnertersty
of Rejacted Total uncetaeityy
of Dets Sejected Uineersttity
Histggram
Histogram
Histogram
10a
10
10a
10a
10
19'
13a
10a
12a
0.0
0.5
1.0
15
0.0
0.5
2'5
0.0
0.5
1.6
3
2.0
Clean
Noray
Clean and
FMNIST, MC DROPOUT, ResNat18
EMNIST, PSV), ResNiet18
FNNIST, FSVL ResNet18
100
100
90-
70
80
40
30
Data Rojectod by Total uncortality
of Data Rejected by Total Uncertariyy
of Dora Rddcted Tool Uncoraiiity
000
Noiay
Clean and FGSM
PGD 00D
Random Rejection
Figure 7: Adversarial example and semantic shift detection statistics for MCD, PSVI, and ESVI on FNSH with a ResNet-18 18 architecture
CIFAR1O, DROPOUT, ResNet18
CIFAR1O, PSVI, ResNet18
CIFAR10, ResNet18
100
50
50
40
30
30
20
40
60
20
20
4)
60
Uncertaisty
Rejected
Hotogram
Histogram
Histogram
10a
10
10a
10
104
10a
3.0
0.5
1.0
15
2.5
2.5
0.0
0.5
100
2.0
25
0.0
0.5
1.5
1.5
2.0
2.5
Uncortaiity
Uncertamly
Clean
Clean and Noiay
Clean and
Clean and PGD
Clean and PGD+
Ciaan
FGSM
PGD
PGD
ResNet18
CIFAR10, PSVI,
120
100
1.0
50
90
80
70
20
40 60
100
20
1000
20 Random Rejection
40
60
80
1.0
Clcanand and 000
Clean and Nolgy 00D
Clean and FGSM OCD
Clean and PGD 00D
Figure 8: Adversarial example and semantic shift detection statistics for MCD, PSVI, and PSVI on CIFAR-10 with a Res.Nt- 18 architecture.
29

---

Published in Transactions on Machine Learning Research (07/2024)
Appendix E
Experimental Details
Here, we provide detailed description of our experimental setup in Section 5.
Architectures For CNN models, CNN is shown in Table 6. We use the standard ResNet-18 from He et al. (2016). For MCD, we add dropout after every activation function for both CNN and ResNet-18.
four-layer CNN for all the experiments. The architecture of the
Hyperparameters We implement HMC using the Hamiltorch package from Cobb and Jalaian (2021) and apply to MNIST with a CNN model due the lack of scalability. To optimize GPU memory usage, we use 10,000 training and 5,000 validation samples from the MNIST dataset. We deploy 100 samples burnin and generate another 100 samples from the posterior. For each sample, train the model for 20 steps with 0.001 as the step size. Such configuration already yields HMC BNN with around 96% accuracy the test set. The hyperparameters for FSVI, PSVI, and MCD are shown in Table 7, Table 8, and Table 9.
Table 6: The Architecture of the four-layer CNN
Nontvoott_eetureeee kernel_size= size= 3)) ReLU()
max_pool
(2, 2), strides= (2, 2), padding="VALID") nn.Conv (atttooot_fftuuees-eeee Bernel_sizese (3, 3))
ReLU()
max_pool Siindov_shape-(2 2), strides= (2, 2), padding="VAAIID)
reshape flatten
un.fc (out mm.ffcoott_eeturresee features=256
ReLU()
nn.fc (out,
Table 7: Hyperparametens for FSVI
"MNIST+ReeNE
FMNIST-CNN
MNIST MNIST+ReeNett ResNet-18
MNNST+CNN
Prior Var
100,000
100,000
1,000,000
100,000
1,000,000
Prior Mean
0
0
0
0
0
Epochs
200
30
200
10
200
Batch Size
128
128
128
128
128
Context Batch Size
128
128
16
128
16
Learning Rate
0.005
0.005
0.05
0.005
0.05
Momentum
0.9
0.9
0.9
0.9
0.9
Weight Decay
0
0
0
0
0
Alpha
0.05
0.05
0.05
0.05
0.05
Reg Scale
Table 8: Hyperparametors for PSVI
CIFARIO+
FMNIST 1esNet-18
FMNIST+CNN
Restett CNIST+CNN
Prior Var
Prior Mean
Epochs
200
50
200
200
Batch Size
128
128
128
128
128
Learning Rate
0.005
0.005
0.005
Momentum
0.9
0.9
0.9
0.9
0.9
Weight Decay
Alpha
0.05
Reg Scale
30

---

The shadow represents the Published in Transactions on Machine Learning Research (07/2024)
Table 9: Hyperparameters for MCD
CIFAR10 ResNet-18
-HINIST+ReNNette
+
MNIST ResNeet-18
MNIST CNN CIN
Prior Precision
0.0005
0.0005
0.0001
0.0005
0.0001
Dropout Rate
0.1
0.1
0.1
0.1
0.1
Epochs
200
30
200
10
200
Batch Size
128
128
128
128
128
Learming Rate
0.005
0.005
0.05
0.005
0.05
Momentum
0.9
0.9
0.9
0.9
0.9
Weight Decay
0
0
0
Alpha
0.05
0.05
0.05
0.05
0.05
Reg Scale
Appendix F
Varying the Strength of the Adversarial Attacks
In addition to the robust accuracy results presented in the main text, conducted detailed analysis to understand the robustness across varying attack radii ranging from minimal to the radius discussed in the main manuscript. Our examinations reveal that robust accuracy demonstrates substantial variation at smaller radii, influenced by differences in random seeds and syper-parameter choices. This insight may explain the community's 's preference for employing a standardizod, larger radius to assess robustness. To shed light on robustness at smaller radii, we adjusted the prior variance, learning rate, weight decay, and the number of training epochs. We depict the upper and lower bounds of robustness in Figure 9 and Figure 10 for CNNs and ResNets, respectively. Notably, our investigations do not identify marked superiority of BNNs over deterministic NNs at smaller radii. At the maximum radius examined, variations in hyper-parameters yield negligible differences in robust accuracy, indicating that models universally exhibit vulnerability at this scale.
With HERST
PM
900
92-
00-
0201
0:00
0.01
Figure 9: Adversarial robustness for CNNs with different attack budgets. The shadow represents the robustness range with different hyperparameters.
Becket with S1857
Roctett with FUM
Reoteet with CBiR
54b
P%
%0
M4
0279
013
0.03
PCD Botic
Figure 10: Adversarial robustness for ResNets with different attack budgets. robustness range with different hyperparameters.
Appendix G Limitations
While we took careful steps to make the study presented in this paper exhaustive and include wide range of representative method and datasets, as with any empirical study, it is possible that some of our findings may not carry to other datasets, neural network architectures, and BNN approximate inference methods.
31


Reviews:
[{"review_id": "jTFKttnODi", "reviewer": "Reviewer_L8Kz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n**S1.** Scrutinizing claims with reproducibility concerns is always welcome. Even more so, when the experimental findings are against the original claims.\n\n**S2.** A good selection of datasets has been used for demonstration purposes.\n\n\n**Weaknesses**\n\n**W1.** I would like the paper to be more formal so that everything can be put into perspective better.\n\nFirst of all, the authors should use a formal definition for adversarial examples.  In Section 2.1 they provide an intuitive definition near the beginning of the section, indicating that adversarial examples cause the learnt model to misclassify the perturbed inputs.  However, in (1) they define an adversarial perturbation with respect to the label y of the original (unperturbed) input x and calculate the loss with y being constant in the ball of radius epsilon around x.  While this is a common practice, especially for domains with images, it nevertheless carries an assumption that the label of the ground truth remains constant around all the possible x + eta inputs that are in the vicinity of the original input x. In general this does not have to be the case.  This assumption is known as \"corrupted inputs\" definition of adversarial examples, or \"constant in the ball\" definition of adversarial examples.  There are other definitions that have been studied in the literature, including \"prediction change\" (where the prediction of the model at the perturbed input is compared to the prediction of the model at the original input) and \"error region\" definition (or, \"exact in the ball\" definition) which compares the predicted label of the model at the perturbed point to the true label at that particular perturbed point.  While this latter definition puts misclassification into the right perspective, nevertheless, it has the drawback of comparing the predicted label with the true label at the same point (not, with the label at the original point).  For more on these please see [C1, C3].  As the authors can actually see from these papers, there are examples, even for non-Bayesian classifiers, where the different definitions that have been used in the literature can lead to conclusions of vastly different robustness guarantees (e.g., a model can appear to be robust using the constant in the ball definition, but vulnerable using the exact in the ball definition, and the other way around).\n\nBased on the above, this particular paper, using (1), is effectively using the \"corrupted inputs\" definition; aka \"constant in the ball\" definition for adversarial examples.\n\nWhile I am not familiar with BNNs and therefore my overall opinion should be taken with a grain of salt, my understanding is that Bayes classifiers try to make predictions that minimize the loss with respect to the ground truth generating distribution P.  As such I cannot believe under the \"error region\" definition (not used in the submitted paper), that any Bayesian classifier can be vulnerable to adversarial perturbations.  The catch is that I have in mind the \"error region\" definition, which is not the one used in this paper.  It is unclear however, but most likely true, if the \"corrupted inputs\" definition is used in the papers cited by the authors for which they explore the reproducibility of their work.  It is also intuitive now, that under the \"corrupted inputs\" definition, Bayesian classification should *not* be robust and thus BNNs should indeed be vulnerable as the authors of this paper claim.  To me this is obvious since one can push an input to a region where the generating probability distribution changes the label, and since the Bayesian classifier will align its prediction with this new label, it will also be the case that it will predict a different label compared to the original label of the unperturbed input.\n\nThere is also additional related work that the authors could cite and put adversarially robust Bayesian classification in perspective beyond the definitional papers [C1, C3].  Here are some examples -- but again, I am not up-to-date with the latest on robust Bayesian classification -- that the authors should include and integrate their findings on Bayesian classifiers, using different definitions of adversarial examples: [C2, C4, C5, C6].\n\n\n\n** REFERENCES **\n\n**[C1]** Dimitrios I. Diochnos, Saeed Mahloujifar, Mohammad Mahmoody:\nAdversarial Risk and Robustness: General Definitions and Implications for the Uniform Distribution. NeurIPS 2018: 10380-10389\n\n**[C2]** Sadia Chowdhury, Ruth Urner:\nRobustness Should Not Be at Odds with Accuracy. FORC 2022: 5:1-5:20\n\n**[C3]** Pascale Gourdeau, Varun Kanade, Marta Kwiatkowska, James Worrell:\nOn the Hardness of Robust Classification. J. Mach. Learn. Res. 22: 273:1-273:29 (2021)\n\n**[C4]** Eitan Richardson, Yair Weiss:\nA Bayes-Optimal View on Adversarial Examples. J. Mach. Learn. Res. 22: 221:1-221:28 (2021)\n\n**[C5]** Pranjal Awasthi, Natalie Frank, Mehryar Mohri:\nOn the Existence of The Adversarial Bayes Classifier. NeurIPS 2021: 2978-2990\n\n**[C6]** Muni Sreenivas Pydi, Varun S. Jog:\nThe Many Faces of Adversarial Risk: An Expanded Study. IEEE Trans. Inf. Theory 70(1): 550-570 (2024)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores experimentally the adversarial robustness of Bayesian artificial neural networks (BNNs).  In particular, recent work has provided evidence that BNNs are *inherently robust* to adversarial perturbations.  Along these lines, the authors explore the code of recent papers making such claims on three datasets: MNIST, FashionMNIST, and CIFAR-10.  The authors' findings are that after correcting some mistakes in the code of the papers that have made such claims, the authors are able to drive the robust accuracy to really low values, thus indicating, contrary to the popular claim, that BNNs are in fact vulnerable to adversarial perturbations.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- At the very least the authors should devote discussion on the different definitions for adversarial examples and should compile a table with some checkmarks that indicate which paper is using which definition.  \n\n- The authors should also integrate in their literature review all the above-mentioned papers which give insight and perspective to the various results on Bayesian classification and consequently on BNNs as well. \n\n- The authors should submit the code that they have and make it publicly available so that their results can be reproduced by others.  Currently the authors cite several GitHub repositories in the appendix, but their repository should be cited in the main part of the paper  eventually.", "broader_impact_concerns": "I am not sure.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8j7QNqmNXE", "reviewer": "Reviewer_BTeo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n\nThe paper is clearly written and the finding that BNN is vulnerable to adversarial attacks are relevant to this journal. \n\nThe experiment is solid and support the main claim. \n\nThe recommendations for evaluating the robustness of BNNs are good practical experience.\n\n\n**Weaknesses**:\n\nDiscussions and experiments using Stochastic Gradient Langevin Dynamics (SGLD) and SGHMC are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an empirical study into the adversarial vulnerability of Bayesian neural networks (BNNs). Four BNN methods, i.e., HCM, MCD, PSVI and FSVI are extensively tested in terms of label prediction, adversarial example detection and semantic shift detection. The experiment result shows that BNNs are in fact quite vulnerable to adversarial attacks if the evaluation is done properly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper claims that existing works like Adv-BNN have not proposed defenses that preserve proper Bayesian inference, and they do not assess their methods on Bayesian prediction pipelines that include uncertainty quantification. I believe it is possible to also test the uncertainty of Adv-BNN in some naive way. It is more convincing to include some experiments to test those works' uncertainty. \n\nWhat is the exact meaning of consistent with Bayesian inference? Like a posterior distribution is needed?", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jytOzI7S3y", "reviewer": "Reviewer_jv9d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The manuscript is exceptionally well-written. It is clear, well-organized, and refreshing to read.\n- The question being addressed is of significant interest to the machine learning community, not just to people working on adversarial robustness. This paper could have broad impact.\n- The conclusions contradict claims made by previous work.  \n- The section on relevant work is clear. I am only familiar with some of the literature cited, but as far as I know, the authors give a fair and thorough review.\n- Specific flaws are identified in the public code of previous work and helpful guidelines are provided for how to avoid such mistakes. This is a service to the community. \n- Experimental results strongly support the claims. \n- Experiments use multiple BNN approaches, which strengthens the argument. \n- The discussion section clearly states the limitations of this work and future directions of research.\n\nWeaknesses: None.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines the claim that bayesian neural networks (BNNs) are inherently robust to adversarial examples (AEs). The authors review three prominent papers that provide support for the claim, and looking through the publicly available code, they find flaws in the experiments. The authors provide a set of specific recommendations for avoiding such flaws in future experiments, and perform their own experiments. These experiments show that BNNs are highly susceptible to AEs, even when using the latest Bayesian inference methods such as Hamiltonian Monte Carlo (HMC) and function-space variational inference (FSVI). The experiments demonstrate susceptibility to three types of attacks that target: (1) classification accuracy; (2) adversarial example detection; and (3) semantic shift detection.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In Section 4.1, I was unable to understand the rational for Recommendation 5. I am not familiar with the paper cited, so a little more explanation would be helpful.\n\nSection 4.1, Recommendation 7 has a typo: \"Adaptiving\".", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper thoroughly investigates the widely accepted claim that Bayesian deep learning can greatly increase robustness against adversarial attacks. Contrary to previously reported findings in the literature, the paper finds that Bayesian neural nets are not inherently robust against adversarial attacks, even relatively simple attacks consistently succeed on state-of-the-art at either perturbing classification results, or adversarial example detection and semantic shift detection (via epistemic uncertainty estimates). Most importantly, the paper thoroughly analyses why the results contradict previously published (and widely accepted) findings - and finds that there are a number of very subtle implementation issues in some previous publications that compound to suggest increased robustness against adversarial attacks.

The paper is very well written and the analysis and experiments are conducted very thoroughly. Furthermore, the paper raises and rectifies an important issue, and shows that the story regarding robustness of Bayesian neural nets w.r.t. adversarial examples is more complex. Quite impressively, the paper points out the precise lines in the source code of 4 previously published (influential) papers that led to distorted results and caused overestimation of the inherent robustness of Bayesian neural nets. The paper then shows, that once these errors are fixed, results change significantly and some of the previously drawn conclusions no longer hold). I have rarely seen this level of thoroughness, and I agree with one of the reviewers (L8Kz) that recommended a 'Reproducibility Certification' for this paper. Overall I think the work is clearly ready for publication and very interesting to a substantial part of TMLR's audience and suggest to accept the paper as is.
Decision: Accept (poster)