Title: ARE GENERATIVE CLASSIFIERS MORE ROBUST TO ADVERSARIAL ATTACKS?

Abstract: There is a rising interest in studying the robustness of deep neural network classifiers against adversaries, with both advanced attack and defence techniques being actively developed. However, most recent work focuses on discriminative classifiers, which only model the conditional distribution of the labels given the inputs. In this paper, we propose and investigate the deep Bayes classifier, which improves classical naive Bayes with conditional deep generative models. We further develop detection methods for adversarial examples, which reject inputs with low likelihood under the generative model. Experimental results suggest that deep Bayes classifiers are more robust than deep discriminative classifiers, and that the proposed detection methods are effective against many recently proposed attacks.

1. INTRODUCTION:

Deep neural networks have been shown to be vulnerable to adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014) . The latest attack techniques can easily fool a deep neural network with imperceptible perturbations (Goodfellow et al., 2014; Papernot et al., 2016b; Carlini & Wagner, 2017a; Kurakin et al., 2016; Madry et al., 2018; Chen et al., 2017a) , even in the black-box case, where the attacker does not have access to the network's weights (Papernot et al., 2017b; Chen et al., 2017b; Alzantot et al., 2018a) . Adversarial attacks are serious security threats to machine learning systems, threatening applications beyond image classification (Carlini & Wagner, 2018; Alzantot et al., 2018b) .

To address this outstanding security issue, researchers have proposed defence mechanisms against adversarial attacks. Adversarial training, which augments the training data with adversarially perturbed inputs, has shown moderate success at defending against recently proposed attack techniques (Szegedy et al., 2013; Goodfellow et al., 2014; Tramèr et al., 2017; Madry et al., 2018) . In addition, recent advances in Bayesian neural networks have demonstrated that uncertainty estimates can be used to detect adversarial attacks (Li & Gal, 2017; Feinman et al., 2017; Louizos & Welling, 2017; Smith & Gal, 2018) . Another notable category of defence techniques involves the usage of generative models. For example, Gu & Rigazio (2014) used an auto-encoder to denoise the inputs before feeding them to the classifier. This denoising approach has been extensively investigated, and the "denoisers" in usage include generative adversarial networks (Samangouei et al., 2018) , PixelCNNs (Song et al., 2018) and denoising auto-encoders (Kurakin et al., 2018) . These developments rely on the "off-manifold" conjecture, that is, that adversarial examples are far away from the data manifold, although Gilmer et al. (2018) has challenged this observation with a synthetic example. Surprisingly, much less recent work has investigated the robustness of generative classifiers (Ng & Jordan, 2002) against adversarial attacks for multi-class classification, where such classifiers explicitly model the conditional distribution of the inputs given labels. Typically, a generative classifier produces predictions by comparing between the likelihood of the labels for a given input, which is closely related to the "distance" of the input to the data manifold associated with a class. Therefore, generative classifiers should be robust to many recently proposed adversarial attacks if the "off-manifold" conjecture holds for many real-world applications. Unfortunately, many generative classifiers in popular use, including naive Bayes and linear discriminant analysis (Fisher, 1936) , perform poorly on natural image classification tasks, making it difficult to verify the "off-manifold" conjecture and the robustness of generative classifiers with these tools. In recent work, k-nearest neighbors (Cover & Hart, 1967) , a method which shares many similarities with generative classi-fiers, has been significantly improved in handling natural images by leveraging deep feature representations (Papernot & McDaniel, 2018) . To the best of our knowledge, an approach which targets a similar contribution has not yet been proposed for generative classifiers.

Are generative classifiers more robust to recently proposed adversarial attack techniques? To answer this, we improve the naive Bayes algorithm by using conditional deep generative models, and evaluate the conjecture on the proposed generative classifier. In summary, our contributions include:

• We propose deep Bayes as an extension of naive Bayes, in which the conditional distribution of an input, given a label, is parameterised by a deep latent variable model (LVM).

We learn the LVM with the variational auto-encoder algorithm (Kingma & Welling, 2013; Rezende et al., 2014) , and approximate Bayes' rule using importance sampling. • We propose three detection methods for adversarial perturbations. The first two use the learned generative model as a proxy of the data manifold, and reject inputs that are far away from it. The third computes statistics for the classifier's output probability vector on training data, and rejects inputs that lead to under-confident predictions. • We evaluate the robustness of the proposed generative classifier on the MNIST multi-class and CIFAR binary classification tasks. We also improve the robustness of deep neural networks on CIFAR-10 multi-class classification, by fusing discriminatively learned visual feature representations with the proposed generative classifiers. We further compare the generative classifiers with a number of discriminative classifiers, including Bayesian neural networks and discriminative latent variable models. 

2. DEEP BAYES: CONDITIONAL DEEP LVM AS:

= {(x (n) , y (n) )} N

n=1 is generated as the following:

(x (n) , y (n) ) ∼ p D (x, y) ⇔ y (n) ∼ p D (y), x (n) ∼ p D (x|y).

A (Bayesian) generative classifier first builds a generative model p(x, y) = p(x|y)p(y), and then, in prediction time, predicts the label y * of a test input x * using Bayes' rule,

p(y * |x * ) = p(x * |y * )p(y * ) p(x * ) = softmax C c=1 [log p(x * , y c )] ,

where softmax C c=1 denotes the softmax operator over the c axis. Therefore, the output probability vector is computed analogously to many deep discriminative classifiers which use softmax activation in the output layer, so many existing attacks can be tested directly. However, unlike discriminative classifiers, the "logit" values prior to softmax activation have a clear meaning here, which is the (approximated) log joint distribution log p(x * , y c ) of input x * conditioned on a given label y c . Therefore, one can also analyse the logit values to determine whether the unseen pair (x * , y * ) is legitimate, a utility which will be discussed further in later sections.

Naive Bayes is perhaps the most well-known generative classifier; it assumes a factorised distribution for the conditional generator, i.e. p(x|y) = D d=1 p(x d |y). However, this factorisation assumption is inappropriate for e.g. image and speech data. Fortunately, we can leverage the recent advances in generative modelling and apply a deep generative model for the joint distribution p(x, y). We refer to such generative classifiers that use deep generative models as deep Bayes classifiers.

In this paper, we consider a deep latent variable model (LVM) p(x, z, y), which will be used for classification. In this case, the conditional distribution is p(x|y) = p(x,z,y)dz p(x,z,y)dzdx . Importantly, this leads to a conditional distribution p(x|y) that is not factorised (even when p(x|z, y) is), which is much more powerful than naive Bayes. Since maximum likelihood is intractable, we follow Kingma & Welling (2013) and Rezende et al. (2014) to introduce an amortised approximate posterior q(z|x, y), and train both p and q by maximising the variational lower-bound:

EQUATION

)

After training, the predicted class probability vector y * for a future input x * is computed by an approximation to Bayes' rule with importance sampling:

EQUATION

We evaluate the effect of the following factorisation structures on the robustness of the induced classifier from the generative model p(x, z, y) (see Figure 1 ).

p(x, z, y) = p(z)p(y|z)p(x|z, y) (GFZ) p(x, z, y) = p D (y)p(z|y)p(x|z, y) (GFY) p(x, z, y) = p(z)p(y|z)p(x|z) (GBZ) p(x, z, y) = p D (y)p(z|y)p(x|z) (GBY) p(x, z, y) = p D (x)p(z|x)p(y|z, x) (DFX) p(x, z, y) = p(z)p(x|z)p(y|z, x) (DFZ) p(x, z, y) = p D (x)p(z|x)p(y|z) (DBX)

We use the initial character "G" to denote generative classifiers and "D" to denote discriminative classifiers. Models with the second character as "F" have a fully connected graph, while "B" models enforce the usage of the latent code z as a bottleneck. The last character of the model name indicates the first node in topological order. Model DFZ is somewhat intermediate, as it builds a generative model for the inputs x but also directly parameterises the conditional distribution p(y|x, z). We do not test other architectures under this nomenclature, as either the graph contains directed cycles (e.g. x → y → z → x), or z is the last node in topological order (e.g. x → y, (x, y) → z) so that the marginalisation of z does not affect classification.

3. DETECTING ADVERSARIAL ATTACKS WITH GENERATIVE CLASSIFIERS:

We propose detection methods for adversarial examples using generative classifier's logit values.

As an illustrating example, consider a labelled dataset of "cat" and "dog" images. If an adversarial image of a cat x adv is incorrectly labelled as "dog", then either this image is ambiguous, or, under a perfect generative model, the logit log p(x adv , "dog") will be significantly lower than normal. This means we can detect attacks using the logits log p(x * , y c ), c = 1, ..., C computed on a test input x * , by comparing them with the logits computed on legitimate training inputs.

Concretely, the proposed detection algorithms are as follows. We aim to reject both unlabelled input x that have low probability under p D (x), and labelled data (x, y) that have low p D (x, y) values.

• Marginal detection: rejecting inputs that are far away from the manifold. One can select a threshold δ and reject an input x if -log p(x) > δ. To determine the threshold δ, we can compute dD = E x∼D [-log p(x)] and σ D = V x∼D [log p(x)], then set δ = dD + ασ D . It is also possible to compute the statistics dp , σ p on the images generated by the generative model accordingly. • Logit detection: rejecting inputs using joint density.

Given a victim model y = F (x), one can reject x if -log p(x, F (x)) > δ y . We can use the mean and variance statistics dc , σ c computed on log p(x, y c ) and select δ yc = dc +ασ c . When D is selected as the KL-divergence, we call this detection method KL detection.

Other divergence/distance measures such as total variation (TV) can also be used.

For better intuition, we visualise the detection mechanisms in Figure 2 with a synthetic "two rings" binary classification example. In this case we sample 2 × 1000 training data points as:

(x, y) ∼ p D ⇔ y ∼ Bern(0.5), θ ∼ Uniform(0, 2π), x|y ∼ N (x; c y + r y [cos(θ), sin(θ)] T , σ 2 I).

We consider a generative classifier p(x, y) = p(x|y)p D (y) = N (x; µ y , σ 2 I)Bern(0.5) with µ y = proj(x, ring y ) = arg min x∈R 2 , || x-cy||2=ry ||x -x|| 2 . The δ thresholds are selected to achieve 10% false positive rates on training data. From the visualisations we see that inputs that are far away from the model manifold are rejected by marginal/logit detection. At the same time, logit detection rejects data points that are not on the manifold of the given class. KL/TV detection does not construct manifold-aware acceptance regions, which is as expected since the proposed divergence detection method does not require the classifier to be generative. However, both detection methods have some success in rejecting uncertain predictions, especially for TV, which also rejects ambiguous inputs (see the ring-cross regions in the last two plots). Combining all three methods, we see that the rejected inputs are either far away from the manifold, or are ambiguous. Again we emphasise that a suitable generative model is required to make the detection methods work in practice, since the data distribution p D (x, y) is approximated by p(x, y).

Detection methods using logit values have also been proposed in Li & Gal (e.g. 2017) ; Feinman et al. (e.g. 2017) . However it is unclear whether the logits values in discriminative classifiers have a clear semantic meaning, while the logit values in deep Bayes represent the log probability of generating the input x given the class label y = y c . Our approach is also distinct from previous "denoising" approaches (e.g. Song et al., 2018; Samangouei et al., 2018; Kurakin et al., 2018) that require training an additional generative model separately. The deep Bayes classifiers share the same generative model as the detection methods, meaning that detected adversarial examples are indeed far away from the classifier's manifold (which is also an approximation to the data manifold). Therefore the claim "detection neural networks can be bypassed" (Carlini & Wagner, 2017a; Athalye et al., 2018) does not directly transfer to our approach, and we can use the marginal and logit detection methods to directly verify the "off-manifold" adversarial example conjecture.

4. EXPERIMENTS:

We carry out a number of tests on the proposed deep Bayes classifiers (3), where q(z|•) and p(z|•) are factorised Gaussians, and the conditional probability p(x|•), if required, is parameterised by an 2 loss. Besides the LVM-based classifiers, we further train a Bayesian neural network (BNN) with Bernoulli dropout (dropout rate 0.3), as it has been shown in Li & Gal (2017) and Feinman et al. (2017) that BNNs are more robust than their deterministic counterparts. The constructed BNN has 2x more channels than LVM-based classifiers, making the comparison slightly "unfair", as the BNN layers have more capacity. We use K = 10 Monte Carlo samples for all the classifiers. The experiments are performed under various threat model settings. We further evaluate the transferability of crafted adversarial examples across different classifiers in the same way as done in Papernot et al. (2016a) . We only provide visualisations in the main text; full table results can be found in the appendix. Readers are also referred to the appendix for further experiments, including a quantitative analysis of the bottleneck effect on robustness and detection.

4.1. MNIST:

The first set of experiments evaluate the robustness of generative classifiers on MNIST. Here the image pixel values are normalised to [0, 1] , and the LVM-based classifiers have dim(z) = 64.

We first perform white-box zero-knowledge attacks, i.e. the attacker can differentiate through the classifier to craft adversarial examples, but he/she is not aware of the existence of the detector. Then we perform white-box perfect-knowledge attacks, where the attacker can differentiate through both the classifier and the detector, and he/she knows the usage of random z samples by the VAE-based classifiers (Biggio et al., 2013; Carlini & Wagner, 2017b) . Lastly we consider grey-box and blackbox attacks, and evaluate the robustness of generative classifiers against transferred attacks.

White-box attacks (zero-knowledge, ∞ ) Results for ∞ attacks are reported in Figure 3 , and in general generative classifiers perform better in terms of victim accuracy and minimum perturbation of the attacks. 1 By contrast, DFX & DFZ are not robust to the weakest attack (FGSM) even when = 0.2 (where the adversarial examples are still visually close to the original digit "7"). Interestingly, DBX is the most robust against FGSM & MIM2 , which agrees with the preliminary tests in Alemi et al. (2017) . But DBX is less robust to PGD, and here GBZ is a clear winner. These results show that the bottleneck is sometimes beneficial for better robustness of MNIST classifiers.

For detection, generative classifiers have successfully detected the adversarial examples with ≥ 0.3, which is reasonable as the visual distortion is already significant. Importantly, generative classifier's victim accuracy decreases as the ∞ distortion increases, but at the same time the TP rates for marginal and logit detection also increase. Therefore these ∞ attacks fail to find near-manifold adversarial examples that fool both the classifier and the detection methods. DFZ, as an intermediate between generative and discriminative classifiers, has worse robustness results, but has good detection performance for the marginal and logit metrics. This is because with softmax activation, the marginal distribution p(x) is dropped, but in marginal/logit detection p(x) is still in use.

White-box attack (zero-knowledge, 2 ) For the CW 2 attack, we performed a hyperparameter search for the loss-balancing parameter c ( 2 distortion increases with larger c) in {0.1, 1, 10, 100, 1000} on the first 100 test images, and we found c = 10 returns the highest success rate. Results are reported in Table 1 . Although being successful on fooling many classifiers, CW failed on attacking GBZ. Also, the mean 2 distortions of the successful attacks on generative classifiers are significantly larger. Furthermore, we found the success of the attack on other generative classifiers is mainly due to the ambiguity of the crafted adversarial images. As visualised in Figure 4 (also see Figure D.1 in appendix), the induced distortion from CW leads to ambiguous digits which sit at the perceptual boundary between the original and the adversarial classes. With the KL detection method, all classifiers achieve > 95% detection rates, which is as expected as the default CW attack configuration, by construction, generates adversarial examples that lead to minimal difference between the logit values of the most and the second most probable classes.

White-box attack (perfect-knowledge, ∞ ) The PGD-based perfect-knowledge ∞ attack is designed following (Carlini & Wagner, 2017b) : we construct an (approximate) Bayes classifier p k (y|x) using (3) for each set of samples {z k c } C c=1 , and minimize the following with PGD:

EQUATION

The detection statistic Φ(x + η, y) is -log p(x + η) for marginal detection, and δ is the corresponding threshold computed on training data. For logit/KL detection, the detection statistics and thresholds are constructed accordingly. We label the three attacks against the marginal, logit and KL based detection schemes '-PKM', '-PKL' and '-PKK' respectively. When we only have knowledge of the K samples and not the detection method we call this attack '-PK0' (ie λ detect =0 in Eq. 4).

Results are visualised in Figure 5 . We see that although the attacker can reduce detection levels, this comes with the trade-off of increasing accuracy, suggesting that a perfect-knowledge adversary cannot break both the classifier and detector working in tandem.

Sanity checks on gradient masking Athalye et al. (2018) claimed that if a successful defence against white-box attacks is due to gradient masking, then this defence is likely to be less effective against grey-box/black-box attacks, as they do not differentiate through the victim classifier and the defence mechanism (Papernot et al., 2017b) . Therefore, we consider two transfer attacks based on distilling the victim classifier using a "student" CNN which has no gradient masking. The two attacks differ in their threat models: in the grey-box setting the attacker has access to the output probability vectors of the classifiers on the training data, while in the black-box setting the attacker has access to queried labels only. For the latter black-box setting, we follow Papernot et al. (2017b) to train a substitute CNN using Jacobian-based dataset augmentation, and we refer to appendix B.1 for the detailed algorithm. The grey-box substitutes achieve > 99% agreement with the victims on test data, and the black-box substitutes obtained ∼ 96% accuracy on test data. Figure 6 : Accuracy and detection rates against distillation-based attacks on MNIST. The higher the better. We only present generative classifiers' results here, for full results see appendix E.

Figure 6 shows the accuracy and detection metrics on transferred ∞ attacks crafted on the substitute models, with a comparison to their white-box counterparts. Note that these crafted attacks achieve ∼ 100% success rates on fooling the substitute models when ≥ 0.2 (see appendix E). However, they do not transfer very well to the generative classifiers. Importantly, for a fixed setting, the whitebox attacks achieve significantly higher success rates (i.e. lower victim accuracies) than their grey-/black-box counterparts, and the gap is at least > 20% for ≤ 0.3 (see Table 2 ). We further present the mean minimum ∞ perturbation in Table 2 , and we see that the minimum perturbation obtained by grey-/black-box attacks are significantly higher than those obtained by white-box attacks.

All these results suggest that the robustness of generative classifiers is unlikely to be caused by gradient masking. Again on the detection side, the detection rates increase as the perturbation size increase, and they are near 100% for = 0.3. This means that most of the successfully transferred adversarial images are off the generative classifier's manifold (as a proxy to the data manifold).

SPSA (evolutionary strategies):

We consider another black-box setting that only assumes access to the logit values of the prediction given an input. We use the SPSA ∞ attack (Uesato et al., 2018) , which is similar to the white-box zero-knowledge attacks, except that gradients are numerically 3 clearly show that SPSA performs much worse on generative classifiers when compared to white-box PGD. Again this means gradient masking is unlikely to be responsible for the improved robustness of generative classifiers, as utilising the exact gradient yielded improved results.

Cross-model attack transferability Finally, we report in Figure 7 the transferability results of the crafted adversarial examples between different models (Papernot et al., 2016a ). Here we take adversarial examples crafted in the white-box setting with PGD and MIM ( = 0.3), and transfer successful attacks to other classifiers. The transfer is effective between generative classifiers but not from generative to discriminative (and vice versa). The attacks crafted on DBX do not transfer in general, while at the same time, DBX is the least robust model in this case. Furthermore, the generative classifiers obtain very high detection rates on all transferred attacks (> 95%). In summary, generative classifiers are more robust against the tested transfer attacks across different models.

4.2. CIFAR PLANE-VS-FROG BINARY CLASSIFICATION:

We consider the same set of evaluations on CIFAR-10, in order to validate the robustness of generative classifiers on natural images (c.f. Carlini & Wagner, 2017b) . Unfortunately, we failed to train fully generative classifiers with comparable test accuracies to discriminative CNNs (typically > 80%): the clean accuracies for GFZ & GFY are all < 50%. Even when using the conditional PixelCNN++ (Salimans et al., 2017) (which uses much deeper networks), the clean accuracy on the test data is 72.4%. Instead, we consider a simpler binary classification problem and construct a dataset containing CIFAR images from the "airplane" and "frog" categories. The images in this dataset are scaled to [0, 1] . On this dataset, the generative classifiers use dim(z) = 128 and obtain > 90% clean test accuracy (see appendix). The attacks are performed on the test images that all models initially correctly classify, leading to a test set of 1577 instances. Due to the page limit, we only present white-box zero-knowledge attacks here, and discuss further attacks in the appendix.

White-box attacks (zero-knowledge, ∞ ) We present the white-box ∞ attack results in Figure 8 , where the distortion strengths are selected as ∈ {0.01, 0.02, 0.05, 0.1, 0.2}. Again, generative classifiers are more robust than the discriminative ones, and GBZ is the most robust, much better than the others when ≥ 0.05. BNN is significantly better than other discriminative VAE-based classifiers, presumably due to higher randomness. Detection results are less satisfactory: marginal/logit detection fail to detect attacks with = 0.1 (which attain both high success rate and induce visually perceptible distortion). KL detection performs better, and interestingly, discriminative classifiers dominate in this metric. These results suggest that the 2 likelihood might not be best suited for modelling natural images (c.f. Larsen et al., 2016; van den Oord et al., 2016) . Still, the minimum distortion required to fool generative classifiers are much higher than that for discriminative ones, indeed the visual distortion of the adversarial examples on generative classifiers are more significant. White-box attack (zero-knowledge, 2 ) As the test set is relatively small, we directly perform CW attacks on the test data with c ∈ {0.1, 1, 10, 100, 1000}. Results are visualised in Figure 9 . The generative classifiers are significantly more robust than the others (with the best being GBZ), and the mean 2 distortions computed on successful attacks are also significantly higher. The TP rates are low for marginal and logit detection, which is reasonable as the crafted images are visually similar to the clean ones. Note that the distortion for the attacks on generative classifiers is perceptible. These results indicate that this CW attack is ineffective when attacking generative classifiers.

4.3. FULL CIFAR-10: COMBINING DEEP BAYES AND DISCRIMINATIVE FEATURES:

The final experiment examines the robustness of CIFAR-10 multi-class classifiers, with the generative classifiers trained on discriminative visual features. To do this, we download a pretrained deep convolutional net3 with VGG16-like architecture (Simonyan & Zisserman, 2014) , and use its feature representation φ(x) as the input to the VAE-based classifiers: p(y|x) = p(y|φ(x)). The classifiers in test include GBZ, GBY and DBX. We use fully-connected neural networks for these classifiers, and select from VGG16 the 9 th convolution layer (CONV9) and the first fully connected layer after convolution (FC1) as the feature layers to ensure ∼ 90% test accuracy (see appendix).

Results on white-box zero-knowledge ∞ attacks are visualised in Figure 10 . Compared with the VGG16 baseline, we see clear improvements in robustness and detection for all VAE-based classifiers. In particular, the generative classifiers GBZ and GBY are overall better than DBX. More importantly, generative classifiers based on CONV9 features are significantly more robust than those based on FC1 features. In contrast, for DBX, which is discriminative, the robustness results are very similar, indicating that the level of feature representation has little effect. These results suggest that one can achieve both high clean accuracy and better robustness against adversaries by combining discriminatively learned visual features and generative classifiers.

Figure 10 : Accuracy and detection rates against white-box zero-knowledge ∞ attacks on CIFAR-10. The higher the better. Note that results for the DBX classifiers are almost identical.

5. DISCUSSION:

We have proposed deep Bayes as a generative classifier that uses deep latent variable models to model the joint distribution of input-output pairs. We have given evidence, on multiple classification tasks, that generative classifiers are more robust to adversarial attacks than discriminative classifiers. Furthermore, the logit in generative classifiers has a well defined meaning and can be used to detect attacks, even when the classifier is fooled.

Our results corroborate with the Bayesian neural network literature, in particular (Li & Gal, 2017; Feinman et al., 2017; Carlini & Wagner, 2017b) , in showing that modelling unobserved variables are effective for defending against adversarial attacks4 . Concurrent to us, Schott et al. ( 2018) also demonstrated the robustness of generative classifiers on MNIST, in which the logits are computed by a tempered version of the variational lower-bound. However, their approach requires thousands of random z samples and tens of optimisation steps to approximate log p(x|y) for every input-output pair (x, y), making it less scalable than our importance sampling technique to large datasets and big architectures. Indeed, we have scaled our approach to CIFAR-10, a natural image dataset, and the robustness results are consistent with those on MNIST. In addition, we have also shown that the structure of the graphical model has a significant impact on robustness: deep LVM-based generative classifiers generally outperform the (randomised) discriminative ones.

While we have given strong evidence to suggest that generative classifiers are more robust to current adversarial attacks, we do not wish to claim that these models will be robust to all possible attacks. Aside from many recent attacks being designed specifically for discriminative neural networks, there is also evidence for the fragility of generative models; e.g. naive Bayes as a standard approach for spam filtering is well-known to be fragile (Dalvi et al., 2004; Huang et al., 2011) But even with this note of caution, we believe this work offers exciting avenues for future work.

Using generative classifiers offers an interesting way to evaluate generative models and can drive improvements in their ability to tackle high-dimensional datasets, where traditionally generative classifiers have been less accurate than discriminative classifiers (Efron, 1975; Ng & Jordan, 2002) .

In addition, the combination of generative and discriminative models investigated in this paper is a compelling direction for future research. Overall, we believe that progress on generative classifiers can inspire better designs of attack, defence and detection techniques.

A MODEL ARCHITECTURES:



MNIST experiments:

The VAEs are constructed with convolutional encoders and deconvolutional generators. More specifically, the encoder network for q(z|x, y) is the same across all VAE-based classifiers. It starts with a 3-layer convolutional neural network with 5 × 5 filters and 64 channels, with a max-pooling operation after each convolution. Then, the convolutional network is followed by a MLP with 2 hidden layers, each with 500 units, to produce the mean and variance parameters of q. The label y is injected into the MLP at the first hidden layer, as a one hot encoding (i.e. for MNIST, the first hidden layer has 500+10 units). The latent dimension is dim(z) = 64.

The p models' architectures are the following: GFZ: For p(y|z) we use a MLP with 1 hidden layer composed of 500 units. For p(x|y, z) we used an MLP with 2 hidden layers, each with 500 units, and 4 × 4 × 64 dimension output, followed by a 3-layer deconvolutional network with 5 × 5 kernel size, stride 2 and [64, 64, 1] channels.

GFY: We use an MLP with 1 hidden layer composed of 500 units for p(z|y), and the same architecture as GFZ for p(x|y, z).

DFZ: We use almost the same deconvolutional network architecture for p(x|z) as GFZ's p(x|y, z) network, except that the input is z only. For p(y|x, z) we use almost the same architecture as q(z|x, y) except that the injected input to the MLP is z and the MLP output is the set of logit values for y.

DFX: We use the same architecture as G3 for p(y|x, z). The network for p(z|x) is almost identical except that there is no injected input to the MLP, and the network returns the mean and variance parameters for q(z|x).

DBX: We use GFZ's architecture for p(y|z) and DFX's architecture for p(z|x).

GBY: We use GFY's architecture for p(z|y) and DFZ's architecture for p(x|z).

GBZ::

We use GFZ's architecture for p(y|z) and DFZ's architecture for p(x|z).

The BNN has almost the same architecture as the encoder network q, except that it uses 2x the hidden units/channels, and the last layer is 10 dimensions. Note that here we used dropout as it is convenient to implement, and we expect better approximate inference methods (such as stochastic gradient MCMC) to return better results for robustness and detection.

CIFAR plane-vs-frog experiments:

The model architectures are almost the same as used in MNIST experiments, except that the hidden layer dimensions for the MLP layers are increased to 1000. For the encoder q, the channels are increased to [64, 128, 256] . For the p models, the deconvolutional networks have different channel values, [128, 64, 3] , and the MLP before the deconvolution outputs a 4 × 4 × 256 vector (before reshaping). The BNN has 2x the channels but still uses 1000 hidden units. 

CIFAR-10 experiments:



B ATTACK SETTINGS:

We use the Cleverhans package to perform attacks. We use the default hyper-parameters, if not specifically stated.

PGD::

We perform the attack for 40 iterations with step-size 0.01.

MIM::

We perform the attack for 40 iterations with step-size 0.01 and decay factor 1.0.

CW-2 ::

We use learning rate 0.01 and confidence 0, and we optimise the loss for 1000 iterations.

SPSA::

We use almost the same hyper-parameters as in Uesato et al. (2018) except for the number of samples for gradient estimates. In detail, we perform the attack for 100 iterations with perturbation size 0.01, Adam learning rate 0.01, stopping threshold -5.0 and 2000 samples for each gradient estimate.

B.1 JACOBIAN-BASED DATASET AUGMENTATION:

The black-box distillation attack is based on Papernot et al. (2017b) , which trains a substitute CNN using Jacobian-based dataset augmentation. Assume y = F (x) is the output one-hot vector of the victim, and p(x) is the probability vector output of the substitute model, then at the t th outerloop, we train the substitute CNN on dataset D t = {(x n , y n )} with queried y n for 10 epochs, and augment the dataset by

EQUATION

We initialise D 1 with 200 × 10 datapoints from the MNIST test set, select λ = 0.1, and run the algorithm for 6 outer-loops. On MNIST, this results in 64, 000 queried inputs, and ∼ 96% accuracy of the substitute model on test data. On CIFAR binary classification, we use 200 × 2 datapoints for the inital query set D 1 , resulting in 12, 800 queries in total. The substitutes achieved almost the same accuracy as their corresponding victim models on clean test datapoints.

C FURTHER EXPERIMENTS C.1 FURTHER EXPERIMENTS ON CIFAR BINARY CLASSIFICATION:

White-box attack (perfect-knowledge, ∞ ) Figure C.1 shows the perfect knowledge attack on the CIFAR binary classification task. Again we see that although the attack is effective for the detection schemes, it comes with the price of decreased mis-classification rates. Interestingly GBY seems to be robust to this attack, where the accuracies on the crafted adversarial examples increase.

Sanity checks on gradient masking:

We conducted the same sets of transferred ∞ attack experiments, and presents the results in Figure C .2 and Table C .1. Again that these crafted attacks achieve ∼ 100% success rates on fooling the substitute models when ≥ 0.1. Similar to the MNIST experiments, these adversarial examples do not transfer very well to the generative classifiers, and for a fixed setting, the white-box attacks achieve significantly higher success rates than their grey-/black-box counterparts (with the gap at ≤ 0.1 around 30%). Furthermore, the minimum perturbation obtained by grey-/black-box attacks are significantly higher than those obtained by white-box attacks. For detection, the detection rates are relatively low at ≤ 0.1 (where the classifiers achieved high accuracy). But the detection rates increase significantly for = 0.2, where the victim accuracies also drop.

All these results suggest that the robustness of generative classifiers is unlikely to be caused by gradient masking. Also most of the successfully transferred adversarial images are off the generative classifier's manifold (as a proxy to the data manifold). Table C .1: Mean minimum ∞ perturbation (in red, computed on ∈ {0.01, 0.02, 0.05, 0.1, 0.2}) and victim accuracy (in blue, for ≤ 0.1) for ∞ attacks on CIFAR plane-vs-frog binary classification. We manually assign the min. perturbation = 0.3 to inputs that all attacks failed to find adversarial perturbations. C .2, with a comparison to the white-box PGD attacks. Again we see that SPSA fails to attack generative classifiers, and the bottleneck discriminative classifier DBX is significantly more robust than the discriminative ones with fully-connected graphical models.

Cross-model attack transferability Finally, we present the CIFAR cross-model attack transferability results in Figure C .3, and here we select = 0.1 instead. Again, transferred attacks are less effective across the victim models. However, the TP rates for logit detection are significantly lower than in the MNIST case (also see Figure 8 ). Nevertheless, the detection rates for the "discriminative to generative" transfer are considerably higher. Combined with the accuracy results, we see that discriminative models as substitutes are ineffective in the transferred attack setting. generative classifiers on adversarial images decreases to zero, the detection rates with marginal/logit detection increases to 100%. This clearly shows that the three attacks tested here cannot fool the generative classifiers without being detected. We present in Table D .1 the clean accuracy on CIFAR plane-vs-frog test images (2000 in total) . We present in Table D .2 the clean accuracy on CIFAR-10 test images. Under review as a conference paper at ICLR 2019 Table E .1: FGSM white-box zero-knowledge attack results on MNIST. Under review as a conference paper at ICLR 2019 Table E .4: Perfect knowledge attacks on MNIST. This is done using the PGD attack with = 0.2. 'ZK' stands for zero knowledge, 'PK0' where you have knowledge of the K samples but not of the detection system. 'PKM', 'PKL', and 'PKK' are attacks where you have knowledge of the K samples and of the marginal, logit and KL detection mechanisms respectively. DFZ 83.9 52.1 2.9 0.0 0.0 6.6 3.8 2.2 3.9 60.6 6.9 4.2 2.4 3.3 60.4 57.5 62.3 54.8 66.7 99.8 GFZ 98.0 93.9 67.7 21.7 3.5 3.3 6.6 5.6 7.8 32.9 5.0 9.5 6.0 8.2 33.0 37.5 45. Under review as a conference paper at ICLR 2019 Table E .13: Grey-box PGD attack results on CIFAR plane-vs-frog binary classification. substitute acc victim acc TP logit 0.01 0.02 0.05 0.10 0.20 0.01 0.02 0.05 0.10 0.20 0.01 0.02 0.05 0.10 0.20 GFZ 96.3 87.9 42.1 4.7 0.1 97.1 94.8 83.0 44.8 4.9 8.7 11.1 5.8 8.5 75.1 GBZ 92.9 82.5 36.9 4.9 0.0 95.5 94.7 91.4 81.7 49.5 14.8 12.6 13.9 21.9 97.8 GFY 96.8 87.9 34.1 3.5 0.1 96.9 95.6 84.6 49.1 6.9 2.6 2.0 6.7 10.3 81.1 GBY 95.1 85.5 32.0 2.6 0.1 96.8 95.5 85.7 56.2 10.9 21.8 14.8 9.2 13.4 97.6 DFX 95.5 80.5 10.5 0.2 0.0 99.1 96.5 77.3 24.0 0.2 N/A N/A N/A N/A N/A DBX 95.6 81.1 34.5 2.2 0.0 99.5 98.3 89.3 58.9 9.9 N/A N/A N/A N/A N/A DFZ 96.3 81.5 10.7 0.5 0.0 99.1 96.9 74.9 21.9 0.3 4.5 16.6 6.4 9.9 80.2 Table E .15: Grey-box CW attack results on CIFAR plane-vs-frog binary classification. substitute acc victim acc TP logit 0.10 1.00 10.00 100.00 1000.00 0.10 1.00 10.00 100.00 1000.00 0.10 1.00 10.00 100.00 1000.00 GFZ 98.7 61.9 0.0 0.0 

D ADDITIONAL RESULTS:



We visualise in:



:

The minimum perturbation is computed across all settings. If none of the attack is successful on an input, we manually assign the minimum perturbation of that input as max + 0.1 (0.6 for MNIST).

Note that MIM on DBX with = 0.9 achieves 100% success rate (i.e. victim accuracy 0%).

https://github.com/geifmany/cifar-vgg, 93.59% test accuracy

In Bayesian neural networks, the network weights are treated as unobserved/latent variables.

Figures and Tables:

A GENERATIVE CLASSIFIER Denote p D (x, y) the data distribution for the input x ∈ R D and label y ∈ {y c |c = 1, ..., C}, where y c denotes the one-hot encoding vector for class c. For a given x ∈ R D we can define the ground-truth label by y ∼ p D (y|x) if x ∈ supp(p D (x)). (1) We assume the data distribution p D (x, y) follows the manifold assumption: for every class c, the conditional distribution p D (x|y c ) has a low-dimensional manifold support M c = supp(p D (x|y c )). Therefore the training dataset D

Figure 1: A visualisation of the graphical models, including both Generative and Discriminative ones, as well as Fully connected and Bottleneck ones. The last character indicates the first node in the topological order of the graph. The colour encoding is the same as those in experiments.

Figure 2: Visualising detection mechanisms. The scattered dots are training data points, with different classes shown in different colours (red for c = 0 and blue for c = 1). Same labels are manually assigned for inputs when the detection method requires y. Decision regions are shown in the corresponding colours. Input points in the shaded area are rejected by detection.

Figure 3: Accuracies (column 1), detection rates (columns 2-4) and minimum inf perturbation (column 6) against white-box zero-knowledge ∞ attacks on MNIST. The higher the better. The second from right most column visualises crafted adversarial examples on an image of digit "7", with ∞ distortion growing from 0.1 to 0.5. The adversarial attacks in test include both ∞ and 2 untargeted attacks from CleverHans 2.0 library (Papernot et al., 2017a): fast gradient sign method (FGSM, Goodfellow et al., 2014), projected gradient descent (PGD, Madry et al., 2018), momentum iterative attack (MIM, Dong et al., 2017) and Carlini & Wagner 2 (CW, Carlini & Wagner, 2017a). Two metrics are reported: accuracy of the classifier on crafted adversarial examples, and detection rate on adversarial examples that have successfully caused the classifier to misclassify. This detection rate is defined as the true positive (TP) rate of finding an adversarial example, and the detection threshold is selected to achieve a 5% false positive rate on clean training data.

Figure 4: Visualising the clean inputs and the CW adversarial examples crafted on GFZ, digits in red rectangles show significant ambiguity.

Figure 5: Accuracy and detection rates of DBX, GBY, and GBZ against PGD-based perfectknowledge attack ( = 0.2) on MNIST. The solid area denotes accuracy and the hatched area denotes detection rate with each considered detector. Zero knowledge attacks are labelled '-ZK', other attack labels are described in the main text.

Figure 7: Results on cross-model transfer attacks on MNIST. The horizontal axis corresponds to the source victim that the adversarial examples are crafted on, and the vertical axis corresponds to the target victim that the attacks are transferred to. The higher (i.e. the lighter) the better. estimated using the logit values from the victim classifier. Results in Table3clearly show that SPSA performs much worse on generative classifiers when compared to white-box PGD. Again this means gradient masking is unlikely to be responsible for the improved robustness of generative classifiers, as utilising the exact gradient yielded improved results.

Figure 8: Accuracy, detection rates and minimum inf perturbations against white-box zeroknowledge ∞ attacks on the CIFAR plane-vs-frog dataset. The higher the better. The second from right most column visualises crafted adversarial examples on an image of a plane, with ∞ distortion ∈ {0.01, 0.02, 0.05, 0.1, 0.2}.

Figure 9: Accuracy and detection rates against white-box zero-knowledge CW attacks on the CI-FAR plane-vs-frog dataset. The higher the better.

Figure C.1: Accuracy and detection rates of DBX, GBY, and GBZ against PGD-based perfectknowledge attack ( = 0.1) on CIFAR binary task. The solid area denotes accuracy and the hatched area denotes detection rate with each considered detector. Zero knowledge attacks are labelled '-ZK', other attack labels are the same as for the MNIST plot (Figure 5).

Figure C.2: Accuracy and detection rates against distillation-based attacks on CIFAR plane-vs-frog binary classification. The higher the better. We only present generative classifiers' results here, for full results see appendix E.

Figure C.3: Results on cross-model transfer attacks on the CIFAR plane-vs-frog dataset. The horizontal axis corresponds to the source victim, and the vertical axis corresponds to the target victim. The higher (lighter) the better.

Figure D.1 the crafted adversarial images using white-box CW attack.

Figure D.1: Visualising the clean inputs of MNIST and the CW adversarial examples crafted on all the classifiers.

.8 40.5 26.2 20.4 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 93.0 94.2 95.4 95.5 96.5 GFZ 94.2 74.5 38.9 12.9 5.7 43.6 79.8 100.0 100.0 100.0 56.4 89.6 99.9 100.0 100.0 89.2 91.7 92.2 92.5 92.2 GBZ 92.5 80.3 62.0 42.4 27.2 37.0 81.7 100.0 100.0 100.0 57.6 93.5 100.0 100.0 100.0 91.6 90.9 90.3 91.1 91.5 GFY 94.3 74.8 46.5 21.7 10.5 53.1 92.6 100.0 100.0 100.0 66.2 97.9 100.0 100.0 100.0 90.8 93.1 93.8 94.2 94.2 GBY 93.6 76.4 47.5 22.3 10.7 41.9 84.5 100.0 100.0 100.0 57.7 92.5 100.0 100.0 100.0 89.3 92.7 92.8 92.9 92.9 DFX 70.1 14.8 1.0 0.4 0.6 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 93.6 93.8 93.8 93.5 94.3 DBX 91.6 77.8 58.1 44.5 36.0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 92.6 93.6 95.2 96.3 97.0 DFZ 75.2 19.0 2.4 1.1 1.0 12.6 50.6 100.0 100.0 100.0 26.9 65.7 100.0 100.0 100.0 93.1 95.1 94.5 94.9 94.8

0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 92.1 92.1 88.7 20.4 0.2 GFZ 86.7 37.7 7.7 1.2 0.3 43.2 71.8 91.4 99.4 100.0 55.8 94.8 99.6 100.0 90.3 92.1 90.9 90.4 89.7 GBZ 85.1 57.4 32.5 19.3 11.6 33.7 50.3 84.4 99.7 100.0 66.3 91.5 99.8 100.0 90.0 91.5 91.5 91.7 91.8 GFY 79.7 27.4 5.6 1.2 0.3 58.1 87.9 98.0 100.0 100.0 68.7 92.2 99.3 100.0 100.0 92.6 92.5 90.7 90.7 85.4 GBY 86.7 35.9 9.0 1.7 0.4 45.3 76.1 94.6 99.7 100.0 56.9 79.3 95.6 99.9 100.0 90.1 92.1 91.6 91.7 91.4 DFX 47.6 0.7 0.0 0.0 0.0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 92.8 89.8 13.3 31.4 42.8 DBX 58.0 18.3 6.0 1.3 0.2 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 92.7 94.0 94.1 93.7 84.8 DFZ 49.6 1.0 0.0 0.0 0.0 11.8 44.3 94.1 100.0 100.0 25.9 57.6 94.5 99.9 100.0 93.9 89.8 12.1 22.9 28.0

0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 93.2 89.6 9.2 17.5 17.5 DBX 66.7 28.7 19.7 17.2 17.2 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 93.5 94.3 94.6 94.7

.2 58.5 14.5 6.3 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 58.6 55.1 65.5 54.0 58.5  GFZ 97.1 94.7 81.8 56.4 31.4 11.4 10.1 5.9 17.4 99.3 11.4 10.4 6.8 17.2 99.3 58.6  40.1 35.9 41.0 50.6 GBZ 95.1 93.5 87.1 74.9 62.0 26.0 18.6 15.5 38.6 99.6 26.3 20.3 19.1 41.9 99.6 35.5 41.5 47.0 43.7 45.6 GFY 96.5 94.2 80.7 56.7 32.0 9.6 9.2 6.9 18.8 99.0 17.6 13.1 8.1 20.8 99.1 46.6 43.7 36.1 39.5 49.3 GBY 96.1 92.9 82.0 60.5 36.3 17.2 15.1 8.1 28.5 99.2 20.2 18.1 10.0 31.1 99.2 49.7 48.0 39.2 38.7 47.3 DFX 83.8 42.2 0.7 0.0 0.0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 63.3 66.6 48.4 97.7 100.0 DBX 90.9 78.6 50.7 31.7 18.3 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 50.7 58.7 56.8 59.7 57.2

0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 41.7 59.4 55.7 68.4 98.9

7 0.3 0.0 0.0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 64.1 64.7 69.6 100.0 100.0 DBX 83.3 34.6 4.2 0.7 0.0 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 59.8 59.5 65.1 Under review as a conference paper at ICLR 2019 Table E.10: Grey-box PGD attack results on MNIST. 4 N/A N/A N/A N/A N/A DBX 80.6 5.1 0.0 0.0 0.0 93.2 59.2 22.5 10.9 8.2 N/A N/A N/A N/A N/A DFZ 69.6 3.4 0.3 0.0 0.0 91.9 57.2 21.4 6.3 2.6 33.6 55.1 91.7 100.0 100.0

.4 28.3 0.9 N/A N/A N/A N/A N/A DBX 95.7 80.5 30.5 1.7 0.0 99.5 98.2 91.1 65.1 21.8 N/A N/A N/A N/A N/A DFZ 96.5 81.9 10.5 0.3 99.1 96.9 76.9 26.2 1.3 4.5 16.5 6.1 8.5 59.6

.7 92.6 64.8 9.9 N/A N/A N/A N/A N/A DBX 85.5 76.4 42.5 4.5 0.0 99.6 99.0 94.8 77.5 24.5 N/A N/A N/A DFZ 85.5 73.5 21.1 1.5 0.0 99.5 99.2 93.8 70.1 15.1 0.0 5.0 8.9 15.9 85.1



Table content: <table/>

Mean minimum ∞ perturbation (in red, computed on ∈ {0.1, 0.2, 0.3, 0.4, 0.5}) and victim accuracy (in blue, for ≤ 0.3) for ∞ attacks on MNIST. We manually assign the min. perturbation = 0.6 to inputs that all attacks failed to find adversarial perturbations.

Table content: <table><tr><td>Attack</td><td>GFZ</td><td>GBZ</td><td>GFY</td><td>GBY</td></tr><tr><td colspan="4">PGD(white-box) 0.23 / 7.71% 0.30 / 30.78% 0.21 / 5.52%</td><td>0.23 / 8.89%</td></tr><tr><td colspan="2">MIM(white-box) 0.24 / 9.02%</td><td>0.21 / 4.97%</td><td>0.22 / 6.72%</td><td>0.21 / 1.54%</td></tr><tr><td>PGD(grey-box)</td><td colspan="4">0.37 / 51.08% 0.36 / 50.64% 0.38 / 53.29% 0.36 / 48.66%</td></tr><tr><td>MIM(grey-box)</td><td colspan="4">0.34 / 43.00% 0.33 / 40.94% 0.34 / 46.64% 0.33 / 40.06%</td></tr><tr><td colspan="5">PGD(black-box) 0.40 / 61.93% 0.42 / 66.75% 0.38 / 56.35% 0.43 / 68.50%</td></tr><tr><td colspan="5">MIM(black-box) 0.36 / 50.44% 0.38 / 59.86% 0.36 / 48.07% 0.39 / 61.78%</td></tr></table>

Accuracy and detection rates against black-box SPSA attack ( = 0.3), with a comparison to white-box PGD on 1000 randomly sampled test datapoints. The higher the better.

Table content: <table><tr><td/><td>GFZ</td><td>GBZ</td><td>GBY</td><td>GFY</td><td colspan="2">DFX DBX</td><td>DFZ</td></tr><tr><td>PGD victim acc</td><td colspan="3">4.0% 29.7% 7.4%</td><td colspan="3">2.3% 0.0% 5.4%</td><td>0.0%</td></tr><tr><td colspan="7">SPSA victim acc 68.2% 79.0% 71.0% 55.9% 0.0% 46.3% 11.0%</td></tr><tr><td>SPSA TP logit</td><td colspan="5">91.5% 92.4% 95.7% 98.5% N/A</td><td>N/A</td><td>65.8%</td></tr></table>

, and very recentlyTabacof et al. (2016);Kos et al. (2017);Creswell et al. (2017) also designed attacks for (unconditional) VAE-type models. However, generative classifiers can be made more robust too, to counter these weaknesses.Dalvi et al. (2004) have shown that generative classifiers can be made more secure if aware of the attack strategy, andBiggio et al. (2011; 2014)  further improved naive Bayes' robustness by modelling the conditional distribution of the adversarial inputs. These approaches are similar to the adversarial training of discriminative classifiers, and efficient ways for doing so with generative classifiers can be an interesting research direction.

Table content: <table/>

The pre-trained VGG16 network is downloaded from https:// github.com/geifmany/cifar-vgg, where the CONV9 and FC1 layers correspond to: We use an MLP of layers [dim(z), 1000, 1000, dim(y)] for p(y|z) and an MLP of layers [dim(z), 1000, 1000, dim(φ(x))] for p(φ(x)|z).GBY: We use an MLP of layers [dim(y), 1000, dim(z) × 2] for p(z|y) and GBZ's architecture for p(φ(x)|z).

Table content: <table><tr><td>CONV9: https://github.com/geifmany/cifar-vgg/blob/master/</td></tr><tr><td>cifar10vgg.py#L82</td></tr><tr><td>FC1: https://github.com/geifmany/cifar-vgg/blob/master/</td></tr><tr><td>cifar10vgg.py#L109</td></tr><tr><td>The VAE-based classifiers build fully connected networks on top of the extracted features, and use</td></tr></table>

2: Accuracy and detection rates against black-box SPSA attack ( = 0.05) on CIFAR plane-vs-frog, with a comparision to white-box PGD. The higher the better.

Table content: <table><tr><td/><td>GFZ</td><td>GBZ</td><td>GBY</td><td>GFY</td><td colspan="2">DFX DBX</td><td>DFZ</td></tr><tr><td colspan="7">PGD victim acc 67.7% 83.9% 67.9% 67.3% 0.3% 4.2% 0.4%</td></tr><tr><td colspan="7">SPSA victim acc 96.4% 95.2% 96.4% 96.3% 0.4% 87.5% 5.1%</td></tr><tr><td>SPSA TP logit</td><td colspan="5">10.0% 17.1% 15.5% 10.0% N/A</td><td>N/A</td><td>4.5%</td></tr></table>

C.2 QUANTIFYING THE EFFECT OF THE BOTTLENECK LAYERWe see from the main text that classifiers with bottleneck structure may be preferred for resisting adversarial examples. To quantify this bottleneck effect, we train on MNIST models DBX, GBZ and GBY with z dimensions in {16, 32, 64, 128} (the main text experiments use dim(z) = 64).The clean test accuracy is shown in TableC.3, showing that all models in test perform reasonably well.We repeat the same white-box zero-knowledge ∞ attack experiments as done in the main text, where results are presented in FigureC.4 and Tables E.25, E.26 and E.27. It is clear that for discriminative classifiers, DBX, the models become less robust as the bottleneck dimension dim(z) increases. Interestingly DBX classifiers seem to be very robust against FGSM attacks, which agrees with the results inAlemi et al. (2017). For the generative ones, we also observe similar trends (although much less significant) of decreased robustness for GBY classifiers, and for GBZ the trend is unclear, presumably due to local optimum issues in optimisation. In summary, GBZ classifiers are generally more robust compared to GBY classifiers. More importantly, when the accuracy of Figure C.4: Accuracy and detection rates against white-box zero-knowledge ∞ attacks on MNIST, with varied bottleneck layer sizes.

Table content: <table><tr><td/><td colspan="6">D.1: Clean test accuracy on CIFAR plane-vs-frog classification.</td><td/></tr><tr><td>BNN</td><td>GFZ</td><td>GFY</td><td>DFZ</td><td>DFX</td><td>DBX</td><td>GBZ</td><td>GBY</td></tr><tr><td colspan="8">97.00% 91.60% 91.20% 94.85% 95.65% 96.00% 89.35% 90.65</td></tr><tr><td colspan="8">Table C.3: Clean test accuracy on MNIST classification (with varied bottleneck layer sizes).</td></tr><tr><td/><td colspan="7">dim(z) = 16 dim(z) = 32 dim(z) = 64 dim(z) = 128</td></tr><tr><td>DBX</td><td colspan="2">99.11%</td><td>99.01%</td><td colspan="2">98.98%</td><td>98.91%</td><td/></tr><tr><td>GBZ</td><td colspan="2">97.11%</td><td>97.08%</td><td colspan="2">97.45%</td><td>96.62%</td><td/></tr><tr><td>GBY</td><td colspan="2">98.82%</td><td>98.95%</td><td colspan="2">98.72%</td><td>98.75%</td><td/></tr></table>

2: Clean test accuracy on CIFAR-10 classification.

Table content: <table><tr><td colspan="7">VGG16 GBZ-FC1 GBY-FC1 DBX-FC1 GBZ-CONV9 GBY-CONV9 DBX-CONV9</td></tr><tr><td>93.59%</td><td>92.55%</td><td>93.21%</td><td>93.49%</td><td>91.76%</td><td>88.33%</td><td>93.21%</td></tr></table>

2: PGD white-box zero-knowledge attack results on MNIST.

Table content: <table/>

3: MIM white-box zero-knowledge attack results on MNIST.

Table content: <table/>

TP marg. TP logit TP KL acc. TP marg. TP logit TP KL acc. TP marg. TP logit TP KL acc. TP marg. TP logit TP KL acc. TP marg. TP logit TP KLTable E.5: FGSM white-box zero-knowledge attack results on CIFAR plane-vs-frog binary classification.

Table content: <table><tr><td/><td>ZK</td><td/><td/><td/><td/><td>PK0</td><td/><td/><td>PKM</td><td/><td/><td>PKL</td><td/><td/><td/><td>PKK</td></tr><tr><td>acc. DBX 18.3</td><td>N/A</td><td>N/A</td><td>94</td><td>17</td><td>N/A</td><td>N/A</td><td>93.1 N/A</td><td>N/A</td><td>N/A</td><td>N/A N/A</td><td>N/A</td><td>N/A</td><td colspan="2">N/A 35.9</td><td>N/A</td><td>N/A</td><td>93.9</td></tr><tr><td>GBZ 57.4</td><td>50.3</td><td>66.3</td><td colspan="2">91.5 35.5</td><td>82.2</td><td>90.7</td><td>91.4 47.9</td><td>36.5</td><td>67.4</td><td>91.3 44.6</td><td>50.5</td><td>69.9</td><td>91.0</td><td>45</td><td>73.0</td><td>84.8</td><td>91.5</td></tr><tr><td>GBY 35.9</td><td>76.1</td><td>79.3</td><td>92.1</td><td>22</td><td>90.8</td><td>91.9</td><td>92.1 43.8</td><td>47.0</td><td>61.9</td><td>91.1 52.0</td><td>56.4</td><td>66.9</td><td colspan="2">90.6 32.8</td><td>89.2</td><td>91.8</td><td>93.3</td></tr></table>

6: PGD white-box zero-knowledge attack results on CIFAR plane-vs-frog binary classification.

Table content: <table/>

11: Grey-box MIM attack results on MNIST.

Table content: <table/>

12: Grey-box CW attack results on MNIST. .00 10.00 100.00 1000.00 0.10 1.00 10.00 100.00 1000.00 0.10 1.00 10.00 100.00 1000.00

Table content: <table><tr><td colspan="2">substitute acc</td><td/><td>victim acc</td><td>TP logit</td><td/></tr><tr><td>0.10 1GFZ 98.4 4.6 0.0</td><td>0.0</td><td>0.0</td><td>98.8 96.9 96.0 93.4</td><td>90.2 53.4 44.2 41.6 40.6</td><td>43.7</td></tr><tr><td>GBZ 98.4 63.2 0.0</td><td>0.0</td><td>0.0</td><td>97.4 95.4 92.9 88.6</td><td>84.5 45.9 39.7 33.0 31.2</td><td>38.6</td></tr><tr><td>GFY 98.1 0.9 0.0</td><td>0.0</td><td>0.0</td><td>99.0 97.6 97.0 95.7</td><td>94.0 61.0 55.2 49.2 49.5</td><td>52.7</td></tr><tr><td>GBY 98.3 5.3 0.0</td><td>0.0</td><td>0.0</td><td>98.7 96.8 96.0 93.8</td><td>91.2 54.2 41.5 37.6 38.7</td><td>43.6</td></tr><tr><td>DFX 85.1 0.0 0.0</td><td>0.0</td><td>0.0</td><td>97.6 93.3 91.2 90.4</td><td>89.8 N/A N/A N/A N/A</td><td>N/A</td></tr><tr><td>DBX 97.3 46.0 0.4</td><td>0.0</td><td>0.0</td><td>98.7 93.9 88.4 85.5</td><td>76.3 N/A N/A N/A N/A</td><td>N/A</td></tr><tr><td>DFZ 80.5 0.0 0.0</td><td>0.0</td><td>0.0</td><td>97.2 94.6 92.3 91.5</td><td>91.2 32.9 28.6 28.1 36.5</td><td>44.5</td></tr></table>

14: Grey-box MIM attack results on CIFAR plane-vs-frog binary classification.

Table content: <table/>

Under review as a conference paper at ICLR 2019 TableE.19: Black-box PGD attack results on CIFAR plane-vs-frog binary classification.

Table content: <table><tr><td/><td/><td>0.0</td><td>98.6 95.8 94.3 93.8</td><td colspan="2">93.2 11.7 10.2 10.9</td><td>9.9</td><td>8.5</td></tr><tr><td>GBZ 96.3 69.8 3.3</td><td>0.0</td><td>0.0</td><td>95.9 95.7 94.7 94.7</td><td colspan="3">94.5 13.5 12.7 12.3 13.7</td><td>14.4</td></tr><tr><td>GFY 98.4 54.4 0.0</td><td>0.0</td><td>0.0</td><td>98.1 96.1 95.3 95.1</td><td>94.3</td><td>0.0 2.2 3.7</td><td>4.6</td><td>7.6</td></tr><tr><td>GBY 97.6 52.9 2.2</td><td>0.0</td><td>0.0</td><td>97.7 95.6 94.5 94.3</td><td colspan="3">93.7 25.8 15.7 14.2 13.7</td><td>18.0</td></tr><tr><td>DFX 64.4 0.0 0.0</td><td>0.0</td><td>0.0</td><td>98.5 97.2 97.1 96.7</td><td colspan="3">95.5 N/A N/A N/A N/A</td><td>N/A</td></tr><tr><td>DBX 91.3 56.2 0.3</td><td>0.0</td><td>0.0</td><td>99.7 99.0 99.0 98.9</td><td colspan="3">98.1 N/A N/A N/A N/A</td><td>N/A</td></tr><tr><td>DFZ 80.3 0.0 0.0</td><td>0.0</td><td>0.0</td><td>99.2 97.9 97.5 97.3</td><td>95.4</td><td>0.0 15.7 14.6</td><td>8.8</td><td>6.1</td></tr></table>

20: Black-box MIM attack results on CIFAR plane-vs-frog binary classification.

Table content: <table/>

21: Black-box CW attack results on CIFAR plane-vs-frog binary classification. .00 10.00 100.00 1000.00 0.10 1.00 10.00 100.00 1000.00 0.10 1.00 10.00 100.00 1000.00

Table content: <table><tr><td colspan="2">substitute acc</td><td/><td>victim acc</td><td/><td colspan="2">TP logit</td><td/></tr><tr><td>0.10 1GFZ 96.3 41.5 0.5</td><td>0.0</td><td>0.0</td><td>98.7 94.7 93.4 93.3</td><td colspan="2">92.3 13.3 8.6 7.7</td><td>7.7</td><td>8.9</td></tr><tr><td>GBZ 94.9 73.3 1.4</td><td>0.0</td><td>0.0</td><td>95.9 95.3 94.6 94.5</td><td colspan="3">94.5 13.5 11.8 11.8 13.2</td><td>13.1</td></tr><tr><td>GFY 96.4 62.0 1.3</td><td>0.0</td><td>0.0</td><td>98.1 96.7 94.8 94.7</td><td>94.1</td><td>0.0 2.8 6.5</td><td>6.1</td><td>6.8</td></tr><tr><td>GBY 93.5 33.6 1.7</td><td>0.0</td><td>0.0</td><td>97.7 95.9 95.4 95.4</td><td colspan="3">95.0 25.8 16.4 15.9 18.6</td><td>19.8</td></tr><tr><td>DFX 90.3 10.9 0.0</td><td>0.0</td><td>0.0</td><td>99.9 98.8 98.5 98.5</td><td colspan="3">97.9 N/A N/A N/A N/A</td><td>N/A</td></tr><tr><td>DBX 87.2 31.8 0.0</td><td>0.0</td><td>0.0</td><td>99.9 97.1 94.5 93.9</td><td colspan="3">94.7 N/A N/A N/A N/A</td><td>N/A</td></tr><tr><td>DFZ 84.9 22.7 0.0</td><td>0.0</td><td>0.0</td><td>99.9 99.0 98.3 98.3</td><td>97.8</td><td>0.0 3.8 2.3</td><td>2.3</td><td>11.0</td></tr></table>

Peer review (iclr2019):
 In this work the authors propose and analyse generative models as defences against adversarial examples. In addition, three detection methods are introduced and an extension to deep features is suggested. My main concerns are as follows (details below): * Important prior work is not mentioned. * Evaluation with direct attacks is only based on (very few) gradient-based techniques, many results are not reliable. * There are signs of gradient masking (the common problem of robustness evaluation, in particular of only gradient-based techniques are used). * The way detection rates are taken into account in the perfect knowledge scenario is confusing. ### Style I like the idea of testing many different factorisation structures. However, that comes with the drawback that one needs to constantly check back what the abbreviations mean. Together with the three detection methods, the manuscript is quite confusing at times and should definitely be streamlined. One suggestion: remove the detection methods: I did not find any real conclusion about them but they are definitely side-tracking users away from the main results. ### Prior work There is at least one closely related prior work not mentioned here: the analysis by synthesis model [1]. This model uses a variational auto encoder to learn class conditional distributions and shows high robustness on MNIST. Please make clear what your contribution is over this paper (other than testing several other factorisations). ### Evaluation problems The robustness of models should be evaluated on different direct attacks ranging from gradient-based to score-based (e.g. NES [2]) to decision-based attacks [3]. Please take a look at [1] to see how a very extensive evaluation might look like. The results can be astonishingly different for different attacks, and so basing conclusion on only one or two attacks is dangerous (in particular if you only use gradient-based ones). One can also see that in your results, just check the variations you get between MIM and PGD. Also, rather then discussing (and showing in detail) results for individual attacks, the minimum adversarial distance for a given sample that can be found by any attack is much more comparable between models (which can also streamline the manuscript). One can see signs of gradient masking in your results. For example, in Figure 3 the MIM attacks levels out at 20% for the DBX model. That can happen for iterative attacks if the gradient is masked. Similarly, in Figure 5 DBX-ZK (zero knowledge) is better in both accuracy and detection rate than DBX-PKK (which takes the KL-detection method into account and should thus either be better in accuracy or detection rate). More generally, the perfect knowledge case, in which the attacker knows about the detector, should only count samples as adversarials which evade the detector and change the model decision. Thus, the detection rate should be zero. Otherwise I have no idea what trade-off between accuracy and detection rate you are actually targeting and how to compare the results. Also, some intermediate results are conflicting with each other. E.g. in 4.1 you state “the usage of bottleneck is beneficial for better robustness”, but for L2 this is not true. Also, I am not sure how conclusive the grey-box and black-box scenarios really are: since the substitute is basically a DFX or DFZ, it’s unsurprising that adversarials transfer best to those two models. ### Minor * In 4.1 you say “as they fail to find near manifold adversarials”, but I don’t see how there can be L-infty adversarials on MNIST that are on-manifold (remember, MNIST pixel values are basically binary). Plus, in the zero-knowledge scenario there is nothing that enforces staying on this manifold. * Result presentation (Figure 3/5 & Table 1) is very different for different attack scenarios, which makes them hard to compare. Please unify. * Is the L2 distance you report in Table 1 the mean (or median) distance to adversarial examples. If so, GBZ (for which you state that C&W “failed on attacking” has actually a smaller mean adversarial distance than some other models (for which C&W is actually quite successful). * Grey-box scenario doesn’t make a lot of sense: since the substitute is basically a DFX or DFZ, it’s unsurprising that adversarials transfer best to those two models. A similar confounder makes the black-box results difficult to interpret. * Also, taking into account that the paper is two pages longer and thus calls for higher standards Taken together, I find the general direction of the paper very interesting and I’d definitely encourage the authors to go further. At the current stage, however, I feel that (1) contributions are not sufficiently delineated to prior work, (2) the evaluation is not convincingly supporting the claims and that (3) the manuscript needs to be streamlined (both in terms of text and figures). [1] Schott et al. (2018) “Towards the first adversarially robust neural network model on MNIST” (https://arxiv.org/abs/1805.09190) [2] Ilyas et al. (2018) “Black-box Adversarial Attacks with Limited Queries and Information” ( [https://arxiv.org/abs/1804.08598)](https://arxiv.org/abs/1804.08598)) [3] Brendel et al. (2018) “Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models” (https://arxiv.org/abs/1712.04248)