Sufficient Conditions for Robustness to Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks

1 INTRODUCTION . Adversarial examples , inputs to machine learning models that an adversary designs to manipulate model output , pose a major concern in machine learning applications . Many hypotheses have been suggested in the literature trying to explain the existence of adversarial examples . For example , Tanay & Griffin ( 2016 ) hypothesise that these examples lie near the decision boundary , while Nguyen et al . ( 2015 ) hypothesise that these examples lie in low density regions of the input space . However , adversarial examples can lie far from the decision boundary ( e.g . “ garbage ” images ( Nguyen et al. , 2015 ) ) , and using a simple spheres dataset it was shown that adversarial examples can exist in high density regions as well ( Gilmer et al. , 2018 ) . In parallel work following Nguyen et al . ( 2015 ) ’ s low-density hypothesis , Li ( 2018 ) empirically modelled input image density on MNIST and successfully detected adversarial examples by thresholding low input density . This puzzling observation , seemingly inconsistent with the spheres experiment in ( Gilmer et al. , 2018 ) , suggests that perhaps additional conditions beyond the ability to detect low input density have led to the observed robustness by Li ( 2018 ) . Suggesting two sufficient conditions , here we prove that an idealised model ( in a sense defined below ) can not have adversarial examples , neither in low density nor in high density regions of the input space . We concentrate on adversarial examples in discriminative classification models , models which are used in practical applications . To formalise our treatment , and to gain intuition into the results , we use tools such as discriminative Bayesian neural network ( BNN ) classifiers ( MacKay , 1992 ; Neal , 1995 ) together with their connections to modern techniques in deep learning such as stochastic regularisation techniques ( Gal , 2016 ) . This pragmatic Bayesian perspective allows us to shed some new light on the phenomenon of adversarial examples . We further discuss which models other than BNNs abide by our conditions . Our hypothesis suggests why MC dropout-based techniques are sensible for adversarial examples identification , and why these have been observed to be consistently effective against a variety of attacks ( Li & Gal , 2017 ; Feinman et al. , 2017 ; Rawat et al. , 2017 ; Carlini & Wagner , 2017 ) . We support our hypothesis mathematically and experimentally using HMC and dropout inference . We construct a synthetic dataset derived from MNIST for which we can calculate ground truth input densities , and use this dataset to demonstrate that model uncertainty correlates to input density , and that under our conditions this density is low for adversarial examples . Using our new-found insights we develop a new attack for MC dropout-based models which does not require gradient information , by looking for “ holes ” in the epistemic uncertainty estimation , i.e . imperfections in the uncertainty approximation , and suggest a mitigation technique as well . We give illustrative examples using MNIST ( LeCun & Cortes , 1998 ) , and experiment with real-world cats-vs-dogs image classification tasks ( Elson et al. , 2007 ) using a VGG13 variant ( Simonyan & Zisserman , 2015 ) . 2 RELATED LITERATURE . There has been much discussion in the literature about the nature of “ adversarial examples ” . Introduced in Szegedy et al . ( 2013 ) using gradient crafting techniques for image inputs , these were initially hypothesised to be similar to the rational numbers , a dense set within the set of all images1 . Szegedy et al . ( 2013 ) ’ s gradient based crafting method performed a targeted attack , where an input image is perturbed with a small perturbation to classify differently to the original image class . Follow-up research by Goodfellow et al . ( 2014 ) introduced non-targeted attacks , where a given input image is perturbed to an arbitrary wrong class by following the gradient away from the image label . This crafting technique also gave rise to a new type of adversarial examples , “ garbage ” images , which look nothing like the original training examples yet classify with high output probability . Goodfellow et al . ( 2014 ) showed that the deep neural networks ’ ( NNs ) non-linearity property is not the cause of vulnerability to adversarial examples , by demonstrating the existence of adversarial examples in linear models as well . They hypothesised that NNs are very linear by design and that in high-dimension spaces this is sufficient to cause adversarial examples . Later work studied the linearity hypothesis further by constructing linear classifiers which do not suffer from the phenomenon ( Tanay & Griffin , 2016 ) . Instead , Tanay & Griffin ( 2016 ) argued that adversarial examples exist when the classification boundary lies close to the manifold of sampled data . After the introduction of adversarial examples by Szegedy et al . ( 2013 ) , Nguyen et al . ( 2015 ) developed crafting techniques which do not rely on gradients but rather use genetic algorithms to generate “ garbage ” adversarial examples . Nguyen et al . ( 2015 ) further hypothesised that such adversarial examples have low probability under the data distribution , and that joint density models p ( x , y ) will be more ‘ robust ’ because the low marginal probability p ( x ) would be indicative of an example being adversarial . Nguyen et al . ( 2015 ) argued that this mitigation is not practical though since current generative models do not scale well to complex high-dimensional data distributions such as ImageNet . Li ( 2018 ) recently extended these ideas to non-garbage adversarial examples as well , and lent support to the hypothesis by showing on MNIST that a deep naive Bayes classifier ( a generative model ) is able to detect targeted adversarial examples by thresholding low input density . Parallel work to Li ( 2018 ) has also looked at the hypothesis of adversarial examples having to exist in low input density regions , but proposed that adversarial examples can exist in high density regions as well . More specifically , Gilmer et al . ( 2018 ) construct a simple dataset composed of a uniform distribution over two concentric spheres in high dimensions , with a deterministic feed-forward NN trained on 50M random samples from the two spheres . They propose an attack named “ manifold attack ” which constrains the perturbed adversarial examples to lie on one of the two concentric spheres , i.e . in a region of high density , and demonstrate that the attack successfully finds adversarial examples with a model trained on the spheres dataset . This demonstration that there could exist adversarial examples on the data manifold and in high input density regions falsifies the hypothesis that adversarial examples must exist in low density regions of the input space , and is seemingly contradictory to the evidence presented in Li ( 2018 ) . We will resolve this inconsistency below . A parallel line to the above research has tried to construct bounds on the minimum magnitude of the perturbation required for an image to become adversarial . Fawzi et al . ( 2018 ) for example quantify “ robustness ” using an introduced metric of expected perturbation magnitude and derive an upper bound on a model ’ s robustness . Fawzi et al . ( 2018 ) ’ s derivation relies on some strong assumptions , for example assuming that it is feasible to compute the distance between an input x and the set { x : f ( x ) > 0 } for some classifier f ( x ) . Papernot et al . ( 2016 ) further give a definition of a robust model , extending the definitions of Fawzi et al . ( 2018 ) to targeted attacks , and propose a model to satisfy this definition . In more recent work , Peck et al . ( 2017 ) extend on both these ideas ( Fawzi et al. , 2018 ; Papernot et al. , 2016 ) , and propose a lower bound on the robustness to perturbations necessary to change the classification of a neural network . Peck et al . ( 2017 ) also make strong assumptions in their premise , assuming the existence of an oracle f∗ ( x ) able to assign a “ correct ” label for each input x ∈ RD . This assumption is rather problematic since it implies that any input has 1This was refuted in ( Goodfellow et al. , 2014 ) ; Below we will see another simple theoretical argument refuting this hypothesis . a “ correct ” class , including completely blank images which have no objects in them . Lastly , Hein & Andriushchenko ( 2017 ) , working in parallel to ( Peck et al. , 2017 ) , use alternative assumptions and instead offer a bound relying on local Lipschitz continuity . Following the perturbation bounds literature , in this work we will use similar but simpler tools , relying on the continuity of the classifier alone . Contrary to the generative modelling perspective , we will concentrate on discriminative Bayesian models which are much easier to scale to highdimensional data ( Kendall & Gal , 2017 ) . Such models capture information about the density of the training set as we will see below . We will define our idealised models under some strong assumptions ( as expected from an idealised model ) , in a similar fashion to previous research concerned with provable guarantees . However below we will also give empirical support demonstrating the ideas we develop with practical tools . The class of models which satisfy our conditions postulated below includes models other than BNNs , such as RBF networks and nearest neighbour in feature space . Even so , we will formalise our arguments in ‘ BNN terminology ’ to keep precise and rigorous language . After laying out our ideas , below we will discuss which other models our results extend to as well . 3 BACKGROUND . A deep neural network for classification is a function f : RD 7→ Y from an input space RD ( e.g . images ) to a set of labels ( e.g . { 0 , 1 } ) . The network f is parametrised by a set of weights and biases ω = { Wl , bl } Ll=1 , which are generally chosen to minimize some empirical risk E : Y × Y 7→ R on the model outputs and the target outputs over some dataset X = { xi } Ni=1 , Y = { yi } Ni=1 with xi ∈ RD and yi ∈ Y . Rather than thinking of the weights as fixed parameters to be optimized over , the Bayesian approach is to treat them as random variables , and so we place a prior distribution p ( ω ) over the weights of the network . If we also have a likelihood function p ( y | x , ω ) that gives the probability of y ∈ Y given a set of parameter values and an input to the network , then we can conduct Bayesian inference given a dataset by marginalising ( integrating out ) the parameters . Such models are known as Bayesian neural networks ( MacKay , 1992 ; Neal , 1995 ) . The conditional probability of the model parameters ω given a training set X , Y is known as the posterior distribution . Ideally we would integrate out our uncertainty by taking the expectation of the predictions over the posterior , rather than using a point estimate of the parameters ( e.g . MAP , the maximiser of the posterior ) . For deep Bayesian neural networks this marginalisation can not be done analytically . Several approximate inference techniques exist , and here we will concentrate on two of them . Hamiltonian Monte Carlo ( HMC , Neal , 1995 ) is considered to be the ‘ gold-standard ’ in inference , but does not scale well to large amounts of data . It has been demonstrated to give state-of-the-art results on many small-scale tasks involving uncertainty estimation in non-tractable models ( Neal , 1995 ) . A more pragmatic alternative is approximate variational inference , e.g . with dropout approximating distributions ( Gal , 2016 ) . This technique is known to scale to large models , preserving model accuracy , while giving useful uncertainty estimates for various down-stream tasks ( Kendall & Gal , 2017 ) . However , dropout approximate inference is known to give worse calibrated approximating distributions , a fact we highlight below as well . Bayesian neural networks are tightly connected to Gaussian processes ( Rasmussen & Williams , 2006 ) , and in fact the latter Gaussian processes can be seen as the infinite limit of single hidden layer Bayesian neural networks with Gaussian priors over their weights ( Neal , 1995 ) . Both can quantify “ epistemic uncertainty ” : uncertainty due to our lack of knowledge . In terms of machine learning , this corresponds to a situation where our model output is poorly determined due to lack of data near the input we are attempting to predict an output for . This is distinguished from “ aleatoric uncertainty ” ( which we will refer to below as ambiguity ) which is due to genuine stochasticity in the data ( Kendall & Gal , 2017 ) : This corresponds to noisy data , for example digit images that can be interpreted as either 1 or 7 ; no matter how much data the model has seen , if there is inherent noise in the labels then the best prediction possible may be a high entropy one ( for example , if we train a model to predict fair coin flips , the best prediction is the max-entropy distribution P ( heads ) = P ( tails ) ) . An attractive measure of uncertainty able to distinguish epistemic from aleatoric examples is the information gain between the model parameters and the data . Recall that the mutual information ( MI ) between two random variables ( r.v.s ) X and Y is given by I ( X , Y ) = H [ X ] − EP ( y ) [ H [ X | Y ] ] = H [ Y ] − EP ( X ) [ H [ Y | X ] ] withH [ X ] the entropy of r.v . X . In terms of machine learning , the amount of information we would gain about the model parameters r.v . ω if we were to receive a label realisation for the r.v . y for a new input x , given the dataset D , is then given by the difference between the predictive entropy H [ y | x , D ] and the expected entropy Ep ( ω|D ) [ H [ y | x , ω ] ] : I ( ω , y | D , x ) = H [ y | x , D ] − Ep ( ω|D ) [ H [ y | x , ω ] ] . ( 1 ) Being uncertain about an input point x implies that if we knew the label at that point we would gain information . Conversely , if the function output at an input is already well determined , then we would gain little information from obtaining the label . Thus , the MI is a measurement of the model ’ s epistemic uncertainty ( in contrast to the predictive entropy which is high when either the epistemic uncertainty is high or when there is ambiguity , e.g . refer to the example of a fair coin toss ) . Note that the MI is always bounded between 0 and the predictive entropy . To gain intuition into the different types of uncertainty in BNNs we shall look at BNN realisations in function space with a toy dataset . Our BNN defines a distribution over NN parameters , which induces a distribution over functions from the input space to the output space . Drawing multiple function realisations we see ( Fig . 1 ) that all functions map the training set inputs to the outputs , but each function takes a different , rather arbitrary , value on points outside the train set . Assessing the discrepancy of these functions on a given input allows us to identify if the tested point is near the training data or not . In classification , having high enough discrepancy between the pre-softmax functions ’ values for a fixed input leads to lower output probability when averaged over the postsoftmax values . Thus any input far enough from the training set will have low output probability .

Summary: The paper studies the adversarial robustness of Bayesian classifiers. The authors state two conditions that they show are provably sufficient for "idealised models" on "idealised datasets" to not have adversarial examples. (In the context of this paper, adversarial examples can either be nearby points that are classified differently with high confidence or points that are "far" from training data and classified with high confidence.) They complement their results with experiments.