Title: ADVERSARIAL EXAMPLES FOR GENERATIVE MODELS

Abstract: We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning architectures are known to be vulnerable to adversarial examples, but previous work has focused on the application of adversarial examples to classification tasks. Deep generative models have recently become popular due to their ability to model input data distributions and generate realistic examples from those distributions. We present three classes of attacks on the VAE and VAE-GAN architectures and demonstrate them against networks trained on MNIST, SVHN and CelebA. Our first attack leverages classification-based adversaries by attaching a classifier to the trained encoder of the target generative model, which can then be used to indirectly manipulate the latent representation. Our second attack directly uses the VAE loss function to generate a target reconstruction image from the adversarial example. Our third attack moves beyond relying on classification or the standard loss for the gradient and directly optimizes against differences in source and target latent representations. We also motivate why an attacker might be interested in deploying such techniques against a target generative network.

1. INTRODUCTION:

Adversarial examples have been shown to exist for a variety of deep learning architectures. 1 They are small perturbations of the original inputs, often barely visible to a human observer, but carefully crafted to misguide the network into producing incorrect outputs. Seminal work by Szegedy et al. (2013) and Goodfellow et al. (2014) , as well as much recent work, has shown that adversarial examples are abundant and finding them is easy.

Most previous work focuses on the application of adversarial examples to the task of classification, where the deep network assigns classes to input images. The attack adds small adversarial perturbations to the original input image. These perturbations cause the network to change its classification of the input, from the correct class to some other incorrect class (possibly chosen by the attacker). Critically, the perturbed input must still be recognizable to a human observer as belonging to the original input class. 2Deep generative models, such as Kingma & Welling (2013) , learn to generate a variety of outputs, ranging from handwritten digits to faces (Kulkarni et al., 2015) , realistic scenes (Oord et al., 2016) , videos (Kalchbrenner et al., 2016) , 3D objects (Dosovitskiy et al., 2016) , and audio (van den Oord et al., 2016) . These models learn an approximation of the input data distribution in different ways, and then sample from this distribution to generate previously unseen but plausible outputs.

To the best of our knowledge, no prior work has explored using adversarial inputs to attack generative models. There are two main requirements for such work: describing a plausible scenario in which an attacker might want to attack a generative model; and designing and demonstrating an attack that succeeds against generative models. We address both of these requirements in this work.

One of the most basic applications of generative models is input reconstruction. Given an input image, the model first encodes it into a lower-dimensional latent representation, and then uses that representation to generate a reconstruction of the original input image. Since the latent representation usually has much fewer dimensions than the original input, it can be used as a form of compression. The latent representation can also be used to remove some types of noise from inputs, even when the network has not been explicitly trained for denoising, due to the lower dimensionality of the latent representation restricting what information the trained network is able to represent. Many generative models also allow manipulation of the generated output by sampling different latent values or modifying individual dimensions of the latent vectors without needing to pass through the encoding step.

These properties of input reconstruction generative networks suggest a variety of different attacks that would be enabled by effective adversaries against generative networks. Any attack that targets the compression bottleneck of the latent representation can exploit natural security vulnerabilities in applications built to use that latent representation. Specifically, if the person doing the encoding step is separated from the person doing the decoding step, the attacker may be able to cause the encoding party to believe they have encoded a particular message for the decoding party, but in reality they have encoded a different message of the attacker's choosing. We explore this idea in more detail as it applies to the application of compressing images using a VAE or VAE-GAN architecture.

2. RELATED WORK AND BACKGROUND:

This work focuses on adversaries for variational autoencoders (VAEs, proposed in Kingma & Welling (2013) ) and VAE-GANs (VAEs composed with a generative adversarial network, proposed in Larsen et al. (2015) ).

2.1. RELATED WORK ON ADVERSARIES:

Many adversarial attacks on classification models have been described in existing literature (Goodfellow et al., 2014; Szegedy et al., 2013) . These attacks can be untargeted, where the adversary's goal is to cause any misclassification, or the least likely misclassification (Goodfellow et al., 2014; Kurakin et al., 2016) ; or they can be targeted, where the attacker desires a specific misclassification. Moosavi-Dezfooli et al. (2016) gives a recent example of a strong targeted adversarial attack. Some adversarial attacks allow for a threat model where the adversary does not have access to the target model (Szegedy et al., 2013; Papernot et al., 2016) , but commonly it is assumed that the attacker does have that access, in an online or offline setting (Goodfellow et al., 2014; Kurakin et al., 2016) . 3Given a classifier f (x) : x ∈ X → y ∈ Y and original inputs x ∈ X , the problem of generating untargeted adversarial examples can be expressed as the following optimization: argmin x * L(x, x * ) s.t. f (x * ) = f (x), where L(•) is a chosen distance measure between examples from the input space (e.g., the L 2 norm). Similarly, generating a targeted adversarial attack on a classifier can be expressed as argmin x * L(x, x * ) s.t. f (x * ) = y t , where y t ∈ Y is some target label chosen by the attacker.

These optimization problems can often be solved with optimizers like L-BFGS or Adam (Kingma & Ba, 2015) , as done in Szegedy et al. (2013) and Carlini & Wagner (2016) . They can also be approximated with single-step gradient-based techniques like fast gradient sign (Goodfellow et al., 2014) , fast gradient L 2 (Huang et al., 2015) , or fast least likely class (Kurakin et al., 2016) ; or they can be approximated with iterative variants of those and other gradient-based techniques (Kurakin et al., 2016; Moosavi-Dezfooli et al., 2016) .

An interesting variation of this type of attack can be found in Sabour et al. (2015) . In that work, they attack the hidden state of the target network directly by taking an input image x and a target image x t and searching for a perturbed variant of x that generates similar hidden state at layer l of the target network to the hidden state at the same layer generated by x t . This approach can also be applied directly to attacking the latent vector of a generative model.

A variant of this attack has also been applied to VAE models in the concurrent work of Tabacof et al. (2016) 4 , which uses the KL divergence between the latent representation of the source and target images to generate the adversarial example. However in their paper, the authors mention that they tried attacking the output directly and that this only managed to make the reconstructions more The VAE is used as a compression scheme to transmit a latent representation of the image from the sender (left) to the receiver (right). The attacker convinces the sender to compress a particular image into its latent vector, which is sent to the receiver, where the decoder reconstructs the latent vector into some other image chosen by the attacker.

blurry. While they do not explain the exact experimental setting, the attack sounds similar to our L VAE attack, which we find very successful. Also, in their paper the authors do not consider the more advanced VAE-GAN models and more complex datasets like CelebA.

2.2. BACKGROUND ON VAES AND VAE-GANS:

The general architecture of a variational autoencoder consists of three components, as shown in Figure 8 . The encoder f enc (x) is a neural network mapping a high-dimensional input representation x into a lower-dimensional (compressed) latent representation z. All possible values of z form a latent space. Similar values in the latent space should produce similar outputs from the decoder in a well-trained VAE. And finally, the decoder/generator f dec (z), which is a neural network mapping the compressed latent representation back to a high-dimensional output x. Composing these networks allows basic input reconstruction x = f dec (f enc (x)). This composed architecture is used during training to backpropagate errors from the loss function.

The variational autoencoder's loss function L VAE enables the network to learn a latent representation that approximates the intractable posterior distribution p(z|x):

EQUATION

q(z|x) is the learned approximation of the posterior distribution p(z|x). p(z) is the prior distribution of the latent representation z. D KL denotes the Kullback-Leibler divergence. E q [log p(x|z)] is the variational lower bound, which in the case of input reconstruction is the cross-entropy H[x, x] between the inputs x and their reconstructions x. In order to generate x the VAE needs to sample q(z|x) and then compute f dec (z).

For the VAE to be fully differentiable while sampling from q(z|x), the reparametrization trick (Kingma & Welling, 2013) extracts the random sampling step from the network and turns it into an input, ε. VAEs are often parameterized with Gaussian distributions. In this case, f enc (x) outputs the distribution parameters µ and σ 2 . That distribution is then sampled by computing z = µ+ε √ σ 2 where ε ∼ N (0, 1) is the input random sample, which does not depend on any parameters of f enc , and thus does not impact differentiation of the network.

The VAE-GAN architecture of Larsen et al. (2015) has the same f enc and f dec pair as in the VAE. It also adds a discriminator f disc that is used during training, as in standard generative adversarial networks (Goodfellow et al., 2014) . The loss function of f dec uses the disciminator loss instead of cross-entropy for estimating the reconstruction error.

3. PROBLEM DEFINITION:

We provide a motivating attack scenario for adversaries against generative models, as well as a formal definition of an adversary in the generative setting.

3.1. MOTIVATING ATTACK SCENARIO:

To motivate the attacks presented below, we describe the attack scenario depicted in Figure 1 . In this scenario, there are two parties, the sender and the receiver, who wish to share images with each other over a computer network. In order to conserve bandwidth, they share a VAE trained on the input distribution of interest, which will allow them to send only latent vectors z. The columns are, in order: the original image, the reconstruction of the original image, the adversarial example, the predicted class of the adversarial example, the reconstruction of the adversarial example, the predicted class of the reconstructed adversarial example, the reconstruction of the reconstructed adversarial example (see Section 4.5), and the predicted class of that reconstruction. The attacker's goal is to convince the sender to send an image of the attacker's choosing to the receiver, but the attacker has no direct control over the bytes sent between the two parties. However, the attacker has a copy of the shared VAE. The attacker presents an image x * to the sender which resembles an image x that the sender wants to share with the receiver. For example, the sender wants to share pictures of kittens with the receiver, so the attacker presents a web page to the sender with a picture of a kitten, which is x * . The sender chooses x * and sends its corresponding z to the receiver, who reconstructs it. However, because the attacker controlled the chosen image, when the receiver reconstructs it, instead of getting a faithful reproduction x of x (e.g., a kitten), the receiver sees some other image of the attacker's choosing, xadv , which has a different meaning from x (e.g., a request to send money to the attacker's bank account).

There are other attacks of this general form, where the sender and the receiver may be separated by distance, as in this example, or by time, in the case of storing compressed images to disk for later retrieval. In the time-separated attack, the sender and the receiver may be the same person or multiple different people. In either case, if they are using the insecure channel of the VAE's latent space, the messages they share may be under the control of an attacker. For example, an attacker may be able to fool an automatic surveillance system if the system uses this type of compression to store the video signal before it is processed by other systems. In this case, the subsequent analysis of the video signal could be on compromised data showing what the attacker wants to show.

While we do not specifically attack their models, viable compression schemes based on deep neural networks have already been proposed in the literature, showing promising results Toderici et al. (2015; 2016) .

3.2. DEFINING ADVERSARIAL EXAMPLES AGAINST GENERATIVE MODELS:

We make the following assumptions about generating adversarial examples on a target generative model, G targ (x) = f dec (f enc (x)). G targ is trained on inputs X that can naturally be labeled with semantically meaningful classes Y, although there may be no such labels at training time, or the labels may not have been used during training. G targ normally succeeds at generating an output x = G targ (x) in class y when presented with an input x from class y. In other words, whatever target output class the attacker is interested in, we assume that G targ successfully captures it in the latent representation such that it can generate examples of that class from the decoder. This target output class does not need to be from the most salient classes in the training dataset. For example, on models trained on MNIST, the attacker may not care about generating different target digits (which are the most salient classes). The attacker may prefer to generate the same input digits in a different style (perhaps to aid forgery). We also assume that the attacker has access to G targ . Finally, the attacker has access to a set of examples from the same distribution as X that have the target label If the attacker only needs to generate one successful attack, the problem of determining if an attack is successful can be solved by manually reviewing the x * and xadv pairs and choosing whichever the attacker considers best. However, if the attacker wants to generate many successful attacks, an automated method of evaluating the success of an attack is necessary. We show in Section 4.5 how to measure the effectiveness of an attack automatically using a classifier trained on z = f enc (x).

4. ATTACK METHODOLOGY:

The attacker would like to construct an adversarially-perturbed input to influence the latent representation in a way that will cause the reconstruction process to reconstruct an output for a different class. We propose three approaches to attacking generative models: a classifier-based attack, where we train a new classifier on top of the latent space z and use that classifier to find adversarial examples in the latent space; an attack using L VAE to target the output directly; and an attack on the latent space, z. All three methods are technically applicable to any generative architecture that relies on a learned latent representation z. Without loss of generality, we focus on the VAE-GAN architecture.

4.1. CLASSIFIER ATTACK:

By adding a classifier f class to the pre-trained generative model5 , we can turn the problem of generating adversaries for generative models back into the previously solved problem of generating adversarial examples for classifiers. This approach allows us to apply all of the existing attacks on classifiers in the literature. However, as discussed below, using this classifier tends to produce lower-quality reconstructions from the adversarial examples than the other two attacks due to the inaccuracies of the classifier.

Step 1. The weights of the target generative model are frozen, and a new classifier f class (z) → ŷ is trained on top of f enc using a standard classification loss L classifier such as cross-entropy, as shown in Figure 3 . This process is independent of how the original model is trained, but it requires a training corpus pulled from approximately the same input distribution as was used to train G targ , with ground truth labels for at least two classes: y t and y t, the negative class.

Step 2. With the trained classifier, the attacker finds adversarial examples x * using the methods described in Section 4.4.

Using f class to generate adversarial examples does not always result in high-quality reconstructions, as can be seen in the middle column of Figure 5 and in Figure 11 . This appears to be due to the fact that f class adds additional noise to the process. For example, f class sometimes confidently misclassifies latent vectors z that represent inputs that are far from the training data distribution, resulting in f dec failing to reconstruct a plausible output from the adversarial example.

4.2. L VAE ATTACK:

Our second approach generates adversarial perturbations using the VAE loss function. The attacker chooses two inputs, x s (the source) and x t (the target), and uses one of the standard adversarial methods to perturb x s into x * such that its reconstruction x * matches the reconstruction of x t , using the methods described in Section 4.4.

The adversary precomputes the reconstruction xt by evaluating f dec (f enc (x t )) once before performing optimization. In order to use L VAE in an attack, the second term (the reconstruction loss) of L VAE (see Equation 1) is changed so that instead of computing the reconstruction loss between x and x, the loss is computed between x * and xt . This means that during each optimization iteration, the adversary needs to compute x * , which requires the full f dec (f enc (x * )) to be evaluated.

4.3. LATENT ATTACK:

Our third approach attacks the latent space of the generative model.

Single latent vector target. This attack is similar to the work of Sabour et al. (2015) , in which they use a pair of source image x s and target image x t to generate x * that induces the target network to produce similar activations at some hidden layer l as are produced by x t , while maintaining similarity between x s and x * .

For this attack to work on latent generative models, it is sufficient to compute z t = f enc (x t ) and then use the following loss function to generate adversarial examples from different source images x s , using the methods described in Section 4.4:

EQUATION

L(•) is a distance measure between two vectors. We use the L 2 norm, under the assumption that the latent space is approximately euclidean.

We also explored a variation on the single latent vector target attack, which we describe in Section A.1 in the Appendix.

4.4. METHODS FOR SOLVING THE ADVERSARIAL OPTIMIZATION PROBLEM:

We can use a number of different methods to generate the adversarial examples. We initially evaluated both the fast gradient sign Goodfellow et al. (2014) method and an L 2 optimization method. As the latter produces much better results we focus on the L 2 optimization method, while we include some FGS results in the Appendix. The attack can be used either in targeted mode (where we want a specific class, y t , to be reconstructed) or untargeted mode (where we just want an incorrect class to be reconstructed). In this paper, we focus on the targeted mode of the attacks.

L 2 optimization. The optimization-based approach, explored in Szegedy et al. ( 2013) and Carlini & Wagner (2016) , poses the adversarial generation problem as the following optimization problem:

argmin x * λL(x, x * ) + L(x * , y t ).

(3) As above, L(•) is a distance measure, and L is one of L classifier , L VAE , or L latent . The constant λ is used to balance the two loss contributions. For the L VAE attack, the optimizer must do a full reconstruction at each step of the optimizer. The other two attacks do not need to do reconstructions while the optimizer is running, so they generate adversarial examples much more quickly, as shown in Table 1 .

4.5. MEASURING ATTACK EFFECTIVENESS:

To generate a large number of adversarial examples automatically against a generative model, the attacker needs a way to judge the quality of the adversarial examples. We leverage f class to estimate whether a particular attack was successful. 6Reconstruction feedback loop. The architecture is the same as shown in Figure 3 . We use the generative model to reconstruct the attempted adversarial inputs x * by computing:

EQUATION

Then, f class is used to compute:

ŷ = f class (f enc (x * )).

(5) The input adversarial examples x * are not classified directly, but are first fed to the generative model for reconstruction. This reconstruction loop improves the accuracy of the classifier by 60% on average against the adversarial attacks we examined. The predicted label ŷ after the reconstruction feedback loop is compared with the attack target y t to determine if the adversarial example successfully reconstructed to the target class. If the precision and recall of f class are sufficiently high on y t , f class can be used to filter out most of the failed adversarial examples while keeping most of the good ones.

We derive two metrics from classifier predictions after one reconstruction feedback loop. The first metric is AS ignore-target , the attack success rate ignoring targeting, i.e., without requiring the output class of the adversarial example to match the target class:

EQUATION

N is the total number of reconstructed adversarial examples; 1 ŷi =y i is 1 when ŷi , the classification of the reconstruction for image i, does not equal y i , the ground truth classification of the original image, and 0 otherwise. The second metric is AS target , the attack success rate including targeting (i.e., requiring the output class of the adversarial example to match the target class), which we define similarly as:

EQUATION

Both metrics are expected to be higher for more successful attacks. Note that AS target ≤ AS ignore-target . When computing these metrics, we exclude input examples that have the same ground truth class as the target class.

5. EVALUATION:

We evaluate the three attacks on MNIST (LeCun et al., 1998) , SVHN (Netzer et al., 2011) and CelebA (Liu et al., 2015) , using the standard training and validation set splits. The VAE and VAE-GAN architectures are implemented in TensorFlow (Abadi & et al., 2015) . We optimized using Adam with learning rate 0. In this section we only show results where no sampling from latent space has been performed. Instead we use the mean vector µ as the latent representation z. As sampling can have an effect on the resulting reconstructions, we evaluated it separately. We show the results with different number of samples in Figure 22 in the Appendix. On most examples, the visible change is small and in general the attack is still successful.

5.1. MNIST:

Both VAE and VAE-GAN by themselves reconstruct the original inputs well as show in Figure 9 , although the quality from the VAE-GAN is noticeably better. As a control, we also generate random noise of the same magnitude as used for the adversarial examples (see Figure 13 ), to show that random noise does not cause the reconstructed noisy images to change in any significant way. Although we ran experiments on both VAEs and VAE-GANs, we only show results for the VAE-GAN as it generates much higher quality reconstructions than the corresponding VAE.

We use a simple classifier architecture to help generate attacks on the VAE and VAE-GAN models.

The classifier consists of two fully-connected hidden layers with 512 units each, using the ReLU activation function. The output layer is a 10 dimensional softmax. The input to the classifier is the 50 dimensional latent representation produced by the VAE/VAE-GAN encoder. The classifier achieves 98.05% accuracy on the validation set after training for 100 epochs.

To see if there are differences between classes, we generate targeted adversarial examples for each MNIST class and present the results per-class. For the targeted attacks we used the optimization method with lambda 0.001, where Adam-based optimization was performed for 1000 epochs with a learning rate of 0.1. The mean L 2 norm of the difference between original images and generated adversarial examples using the classifier attack is 3.36, while the mean RMSD is 0.120. 2 show that the targeted classifier attack successfully fools the classifier. Classifier accuracy is reduced to 0%, while the matching rate (the ratio between the number of predictions matching the target class and the number of incorrectly classified images) is 100%, which means that all incorrect predictions match the target class. However, what we are interested in (as per the attack definition from Section 3.2) is how the generative model reconstructs the adversarial examples. If we look at the images generated by the VAE-GAN for class 0, shown in Figure 4 , the targeted attack is successful on some reconstructed images (e.g. one, four, five, six and nine are reconstructed as zeroes). But even when the classifier accuracy is 0% and matching rate is 100%, an incorrect classification does not always result in a reconstruction to the target class, which shows that the classifier is fooled by an adversarial example more easily than the generative model.

Numerical results in Table:

Reconstruction feedback loop. The reconstruction feedback loop described in Section 4.5 can be used to measure how well a targeted attack succeeds in making the generative model change the reconstructed classes. Table 4 in the Appendix shows AS ignore-target and AS target for all source and target class pairs. A higher value signifies a more successful attack for that pair of classes. It is interesting to observe that attacking some source/target pairs is much easier than others (e.g. pair (4, 0) vs. (0, 8)) and that the results are not symmetric over source/target pairs. Also, some pairs do well in AS ignore-target , but do poorly in AS target (e.g., all source digits when targeting 4). As can be seen in Figure 11 , the classifier adversarial examples targeting 4 consistently fail to reconstruct into something easily recognizable as a 4. Most of the reconstructions look like 5, but the adversarial example reconstructions of source 5s instead look like 0 or 3.

For generating adversarial examples using the L VAE attack, we used the optimization method with λ = 1.0, where Adam-based optimization was performed for 1000 epochs with a learning rate of 0.1. The mean L 2 norm of the difference between original images and generated adversarial examples with this approach is 3.68, while the mean RMSD is 0.131.

We show AS ignore-target and AS target of the L VAE attack in Table 5 in the Appendix. Comparing with the numerical evaluation results of the latent attack (below), we can see that both methods achieve similar results on MNIST.

To generate adversarial examples using the latent attack, we used the optimization method with λ = 1.0, where Adam-based optimization was performed for 1000 epochs with a learning rate of 0.1. The mean L 2 norm of the difference between original images and generated adversarial examples using this approach is 2.96, while the mean RMSD is 0.105.

Table 3 shows AS ignore-target and AS target for all source and target class pairs. Comparing with the numerical evaluation results of the classifier attack we can see that the latent attack performs much better. This result remains true when visually comparing the reconstructed images, shown in Figure 5 .

We also tried an untargeted version of the latent attack, where we change Equation 2 to maximize the distance in latent space between the encoding of the original image and the encoding of the adversarial example. In this case the loss we are trying to minimize is unbounded, since the L 2 distance can always grow larger, so the attack normally fails to generate a reasonable adversarial example. Additionally, we also experimented with targeting latent representations of specific images from the training set instead of taking the mean, as described in Section 4.3. We show the numerical results in Table 3 and the generated reconstructions in Figure 15 (in the Appendix). It is also interesting to compare the results with L VAE , by choosing the same image as the target. Results for L VAE for the same target images as in Table 3 are shown in Table 6 in the Appendix. The results are identical between the two attacks, which is expected as the target image is the same -only the loss function differs between the methods.

5.2. SVHN:

The SVHN dataset consists of cropped street number images and is much less clean than MNIST. Due to the way the images have been processed, each image may contain more than one digit; the target digit is roughly in the center. VAE-GAN produces high-quality reconstructions of the original images as shown in Figure 17 in the Appendix.

For the classifier attack, we set λ = 10 -5 after testing a range of values, although we were unable to find an effective value for this attack against SVHN. For the latent and L VAE attacks we set λ = 10.

In Table 10 we show AS ignore-target and AS target for the L 2 optimization latent attack. The evaluation metrics are less strong on SVHN than on MNIST, but it is still straightforward for an attacker to find a successful attack for almost all source/target pairs. Figure 2 

5.4. SUMMARY OF DIFFERENT ATTACK METHODS:

Table 1 shows a comparison of the mean distances between original images and generated adversarial examples for the three different attack methods. The larger the distance between the original image and the adversarial perturbation, the more noticeable the perturbation will tend to be, and the more likely a human observer will no longer recognize the original input, so effective attacks keep these distances small while still achieving their goal. The latent attack consistently gives the best results in our experiments, and the classifier attack performs the worst.

We also measure the time it takes to generate 1000 adversarial examples using the given attack method. The L VAE attack is by far the slowest of the three, due to the fact that it requires computing full reconstructions at each step of the optimizer when generating the adversarial examples. The other two attacks do not need to run the reconstruction step during optimization of the adversarial examples.

6. CONCLUSION:

We explored generating adversarial examples against generative models such as VAEs and VAE-GANs. These models are also vulnerable to adversaries that convince them to turn inputs into surprisingly different outputs. We have also motivated why an attacker might want to attack generative models. Our work adds further support to the hypothesis that adversarial examples are a general phenomenon for current neural network architectures, given our successful application of adversarial attacks to popular generative models. In this work, we are helping to lay the foundations for understanding how to build more robust networks. Future work will explore defense and robustification in greater depth as well as attacks on generative models trained using natural image datasets such as CIFAR-10 and ImageNet. 

A APPENDIX A.1 MEAN LATENT VECTOR TARGETED ATTACK:

A variant of the single latent vector targeted attack described in Section 4.3, that was not explored in previous work to our knowledge is to take the mean latent vector of many target images and use that vector as x t . This variant is more flexible, in that the attacker can choose different latent properties to target without needing to find the ideal input. For example, in MNIST, the attacker may wish to have a particular line thickness or slant in the reconstructed digit, but may not have such an image available. In that case, by choosing some images of the target class with thinner lines or less slant, and some with thicker lines or more slant, the attacker can find a target latent vector that closely matches the desired properties.

In this case, the attack starts by using f enc to produce the target latent vector, z t , from the chosen target images, x (t) .

EQUATION

In this work, we choose to reconstruct "ideal" MNIST digits by taking the mean latent vector of all of the training digits of each class, and using those vectors as x t . Given a target class y t , a set of examples X and their corresponding ground truth labels y, we create a subset x (t) as follows:

EQUATION

)

Both variants of this attack appear to be similarly effective, as shown in Figure 15 and Figure 5 . The trade-off between the two in these experiments is between the simplicity of the first attack and the flexibility of the second attack. 

:

Adversarial examples are even easier to produce against most other machine learning architectures, as shown in Papernot et al. (2016), but we are focused on deep networks.

Random noise images and "fooling" images(Nguyen et al., 2014) do not belong to this strict definition of an adversarial input, although they do highlight other limitations of current classifiers.

See Papernot et al. (2015) for an overview of different adversarial threat models.

This work was made public shortly after we published our early drafts.

This is similar to the process of semi-supervised learning inKingma et al. (2014), although the goal is different.

Note that f class here is being used in a different manner than when we use it to generate adversarial examples. However, the network itself is identical, so we don't distinguish between the two uses in the notation.

Figures and Tables:

Figure1: Depiction of the attack scenario. The VAE is used as a compression scheme to transmit a latent representation of the image from the sender (left) to the receiver (right). The attacker convinces the sender to compress a particular image into its latent vector, which is sent to the receiver, where the decoder reconstructs the latent vector into some other image chosen by the attacker.

Figure 2: Results for the L 2 optimization latent attack (see Section 4.3) on the VAE-GAN, targeting a specific image from the class 0. Shown are the first 12 non-zero images from the test SVHN data set.The columns are, in order: the original image, the reconstruction of the original image, the adversarial example, the predicted class of the adversarial example, the reconstruction of the adversarial example, the predicted class of the reconstructed adversarial example, the reconstruction of the reconstructed adversarial example (see Section 4.5), and the predicted class of that reconstruction.

001 and other parameters set to default values for both the generative model and the classifier. For the VAE, we use two architectures: a simple architecture with a single fully-connected hidden layer with 512 units and ReLU activation function; and a convolutional architecture taken from the original VAE-GAN paper Larsen et al. (2015) (but trained with only the VAE loss). We use the same architecture trained with the additional GAN loss for the VAE-GAN model, as described in that work. For both VAE and VAE-GAN we use a 50-dimensional latent representation on MNIST, a 1024-dimensional latent representation on SVHN and 2048-dimensional latent representation on CelebA.

Figure4: Results for the L 2 optimization latent attack on the VAE-GAN, targeting the mean latent vector for 0. Shown are the first 12 non-zero images from the test MNIST data set. The columns are, in order: the original image, the reconstruction of the original image, the adversarial example, the predicted class of the adversarial example, the reconstruction of the adversarial example, the predicted class of the reconstructed adversarial example, the reconstruction of the reconstructed adversarial example (see Section 4.5), and the predicted class of that reconstruction.

Figure 5: Left: representative adversarial examples with a target class of 0 on the first 100 nonzero images from the MNIST validation set. These were produced using the L 2 optimization latent attack (Section 4.3). Middle: VAE-GAN reconstructions from adversarial examples produced using the L 2 optimization classifier attack on the same set of 100 validation images (those adversaries are not shown, but are qualitatively similiar, see Section 4.1). Right: VAE-GAN reconstructions from the adversarial examples in the left column. Many of the classifier adversarial examples fail to reconstruct as zeros, whereas almost every adversarial example from the latent attack reconstructs as zero.

Figure 6: Left: VAE-GAN reconstructions of adversarial examples generated using the L 2 optimization L VAE attack (single image target). Right: VAE-GAN reconstructions of adversarial examples generated using the L 2 optimization latent attack (single image target). Approximately 85 out of 100 images are convincing zeros for the L 2 latent attack, whereas only about 5 out of 100 could be mistaken for zeros with the L VAE attack.

using the latent attack and a lambda value of 0.5 (L 2 norm between original images and generated adversarial examples 9.78, RMSD 0.088) and the corresponding VAE-GAN reconstructions. Most of the reconstructions reflect the target image very well. We get even better results with the L VAE attack, using a lambda value of 0.75 (L 2 norm between original images and generated adversarial examples 8.98, RMSD 0.081) as shown in Figure21.

Figure 7: Summary of different attacks on CelebA dataset: reconstructions of original images (top), reconstructions of adversarial examples generated using the latent attack (middle) and L VAE attack (bottom). Target reconstruction is shown on the right. Full results are in the Appendix.

Figure 8: Variational autoencoder architecture.

Figure 10: Untargeted FGS L VAE Attack: VAE reconstructions (left) and VAE-GAN reconstructions (right). Note the difference in reconstructions compared to Figure 9. Careful visual inspection reveals that none of the VAE reconstructions change class, and only two of the VAE-GAN reconstructions change class (a 6 to a 0 in the next-to-last row, and a 9 to a 4 in the last row). Combining FGS with L VAE does not seem to give an effective attack.

Figure 11: L 2 Optimization Classifier Attack: Reconstructions of the first 100 adversarial examples targeting 4, demonstrating why the AS target metric is 0 for all source digits.

Figure 12: Untargeted FGS Classifer Attack: Adversarial examples (left) and their reconstructions by the generative model (right) for the first 100 images from the MNIST validation set. Top results are for VAE, while bottom results are for VAE-GAN. Note the difference in quality of the reconstructed adversarial examples.

Figure 13: Original images with random noise added (top) and their reconstructions by VAE (bottom left) and VAE-GAN (bottom right). The magnitude of the random noise is the same as for the generated adversarial noise shown in Figure 12. Random noise does not cause the reconstructed images to change in a significant way.

Figure 15: L 2 Optimization Latent Attack (single latent vector target): VAE-GAN reconstructions of adversarial examples generated using the latent attack with target classes 0 and 7 using two random targets in latent space per target class. Original examples which already belong to the target class are excluded. The stylistic differences in the reconstructions are clearly visible.

Figure16: L 2 Optimization Latent Attack (single latent vector target): t-SNE plot of the latent space, with the addition of green circles representing the adversarial examples for target class 0. In this plot, it appears that the adversarial examples cluster around 6 (yellow) and 0 (red).

Figure 17: Original Inputs and Reconstructions: The first 100 images from the SVHN validation set (left) reconstructed by VAE-GAN (right).

Figure 18: L 2 Optimization Latent Attack (single latent vector target): Nearest neighbors in latent space for generated adversarial examples (target class 0) on the first 100 images from the MNIST (left) and SVHN (right) validation sets.

Figure 19: Original images in the CelebA dataset (left) and their VAE-GAN reconstructions (right).

Figure 20: L 2 Optimization Latent Attack on CelebA Dataset (single latent vector target): Adversarial examples generated for 100 images from the CelebA dataset (left) and their VAE-GAN reconstructions (right).

Figure 21: L 2 Optimization L VAE Attack on CelebA Dataset (single image target): Adversarial examples generated for 100 images from the CelebA dataset (left) and their VAE-GAN reconstructions (right).

The VAE-GAN classifier architecture used to generate classifier-based adversarial examples on the VAE-GAN. The VAE-GAN in the dashed box is the target network and is frozen while training the classifier. The path x → f enc → z → f class → ŷ is used to generate adversarial examples in z, which can then be reconstructed by f dec .

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>VAE-GAN</td></tr><tr><td>x</td><td>Encoder f enc</td><td>z</td><td>Decoder f dec</td><td>x</td><td>Discriminator f disc</td><td>(0, 1)</td></tr><tr><td/><td/><td/><td>Classifier f class</td><td>y</td><td/><td/></tr><tr><td>Figure 3:</td><td/><td/><td/><td/><td/><td/></tr></table>

supports this evaluation. Visual inspection shows that 11 out of the 12 adversarial examples reconstructed as 0, the target digit. It is worth noting that 2 out of the 12 adversarial examples look like zeros (rows 1 and 11), and two others look like both the original digit and zero, depending on whether the viewer focuses on the light or dark areas of the image (rows 4 and 7). The L 2 optimization latent attack achieves much better results than the L VAE attack (see Table11 and Figure6) on SVHN, while both attacks work equally well on MNIST. Comparison of mean L 2 norm and RMSD between the original images and the generated adversarial examples for the different attacks. Time to attack is the mean number of seconds it takes to generate 1000 adversarial examples using the given attack method (with the same number of optimization iterations for each attack).

Table content: <table><tr><td>5.3 CELEBA</td></tr><tr><td>The CelebA dataset consists of more than 200,000 cropped faces of celebrities, each annotated</td></tr></table>

material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016. Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In Proceedings of the 1st IEEE European Symposium on Security and Privacy, 2015. Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial manipulation of deep representations. CoRR, abs/1511.05122, 2015. URL http://arxiv.org/abs/1511. 05122. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.

Table content: <table><tr><td>Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Celik, and Ananthram Swami. Practical black-box attacks against deep learning systems using adversarial examples. arXiv preprint arXiv:1602.02697, 2016.</td></tr><tr><td>Sara Sabour,</td></tr></table>

L 2 Optimization Classifier Attack on MNIST: f class accuracy on adversarial examples against the VAE-GAN for each target class (middle row) and the matching rate between the predictions f class made and the adversarial target class (bottom row). The adversarial examples successfully fool f class into predicting the target class almost 100% of the time, which makes this attack seem like a strong attack, but the attack actually fails to generate good reconstructions in many cases. Reconstructions for target classes 0 and 4 can be seen in Figure4and Figure11.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 85.54% (34.94%)</td><td>Target 2 100.00% (100.00%)</td><td>Target 3 100.00% (13.25%)</td><td>Target 4 75.90% (75.90%)</td><td>Target 5 96.39% (92.77%)</td><td>Target 6 100.00% (100.00%)</td><td>Target 7 96.39% (91.57%)</td><td>Target 8 0.00% (0.00%)</td><td>Target 9 100.00% (83.13%)</td></tr><tr><td>1</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>100.00% (93.60%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>100.00% (98.40%)</td></tr><tr><td>2</td><td>100.00% (100.00%)</td><td>97.37% (55.26%)</td><td>-</td><td>100.00% (55.26%)</td><td>97.37% (88.60%)</td><td>95.61% (74.56%)</td><td>100.00% (100.00%)</td><td>99.12% (94.74%)</td><td>100.00% (0.00%)</td><td>100.00% (92.98%)</td></tr><tr><td>3</td><td>100.00% (100.00%)</td><td>90.65% (89.72%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (91.59%)</td><td>94.39% (94.39%)</td><td>100.00% (100.00%)</td><td>85.05% (84.11%)</td><td>100.00% (0.00%)</td><td>90.65% (88.79%)</td></tr><tr><td>4</td><td>100.00% (100.00%)</td><td>97.27% (67.27%)</td><td>100.00% (100.00%)</td><td>100.00% (18.18%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>100.00% (100.00%)</td></tr><tr><td>5</td><td>100.00% (100.00%)</td><td>96.55% (80.46%)</td><td>100.00% (100.00%)</td><td>2.30% (2.30%)</td><td>100.00% (96.55%)</td><td>-</td><td>100.00% (100.00%)</td><td>98.85% (89.66%)</td><td>100.00% (0.00%)</td><td>95.40% (94.25%)</td></tr><tr><td>6</td><td>100.00% (100.00%)</td><td>87.36% (80.46%)</td><td>100.00% (100.00%)</td><td>100.00% (11.49%)</td><td>100.00% (97.70%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (98.85%)</td><td>100.00% (0.00%)</td><td>100.00% (96.55%)</td></tr><tr><td>7</td><td>100.00% (100.00%)</td><td>90.91% (82.83%)</td><td>100.00% (100.00%)</td><td>100.00% (16.16%)</td><td>100.00% (79.80%)</td><td>100.00% (98.99%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (0.00%)</td><td>100.00% (100.00%)</td></tr><tr><td>8</td><td>100.00% (100.00%)</td><td>89.77% (71.59%)</td><td>100.00% (100.00%)</td><td>100.00% (35.23%)</td><td>100.00% (97.73%)</td><td>89.77% (62.50%)</td><td>100.00% (100.00%)</td><td>98.86% (92.05%)</td><td>-</td><td>98.86% (96.59%)</td></tr><tr><td>9</td><td>100.00% (100.00%)</td><td>95.65% (75.00%)</td><td>100.00% (100.00%)</td><td>100.00% (18.48%)</td><td>100.00% (97.83%)</td><td>100.00% (95.65%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>-</td></tr></table>

L 2 Optimization Latent Attack on MNIST (single latent vector target): AS ignore-target (AS target in parentheses) after one reconstruction loop for different source and target class pairs on the VAE-GAN model. The latent representation of a random image from the target class is used to generate the target latent vector. Higher values indicate more successful attacks against the generative model.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 40.96% (1.20%)</td><td>Target 2 6.02% (4.82%)</td><td>Target 3 10.84% (7.23%)</td><td>Target 4 75.90% (0.00%)</td><td>Target 5 6.02% (3.61%)</td><td>Target 6 28.92% (28.92%)</td><td>Target 7 37.35% (20.48%)</td><td>Target 8 6.02% (1.20%)</td><td>Target 9 10.84% (3.61%)</td></tr><tr><td>1</td><td>99.20% (77.60%)</td><td>-</td><td>7.20% (5.60%)</td><td>1.60% (1.60%)</td><td>85.60% (0.00%)</td><td>8.00% (5.60%)</td><td>28.80% (28.00%)</td><td>8.80% (7.20%)</td><td>3.20% (1.60%)</td><td>69.60% (0.80%)</td></tr><tr><td>2</td><td>85.96% (80.70%)</td><td>3.51% (2.63%)</td><td>-</td><td>29.82% (23.68%)</td><td>78.95% (0.00%)</td><td>72.81% (20.18%)</td><td>72.81% (46.49%)</td><td>35.09% (8.77%)</td><td>41.23% (12.28%)</td><td>68.42% (2.63%)</td></tr><tr><td>3</td><td>93.46% (83.18%)</td><td>26.17% (12.15%)</td><td>27.10% (16.82%)</td><td>-</td><td>67.29% (0.00%)</td><td>66.36% (62.62%)</td><td>87.85% (22.43%)</td><td>50.47% (27.10%)</td><td>23.36% (8.41%)</td><td>33.64% (8.41%)</td></tr><tr><td>4</td><td>100.00% (82.73%)</td><td>70.00% (48.18%)</td><td>28.18% (10.91%)</td><td>84.55% (17.27%)</td><td>-</td><td>66.36% (31.82%)</td><td>95.45% (71.82%)</td><td>62.73% (37.27%)</td><td>20.91% (0.91%)</td><td>51.82% (44.55%)</td></tr><tr><td>5</td><td>93.10% (89.66%)</td><td>21.84% (1.15%)</td><td>68.97% (11.49%)</td><td>28.74% (18.39%)</td><td>3.45% (0.00%)</td><td>-</td><td>20.69% (19.54%)</td><td>80.46% (41.38%)</td><td>22.99% (2.30%)</td><td>44.83% (12.64%)</td></tr><tr><td>6</td><td>29.89% (28.74%)</td><td>44.83% (1.15%)</td><td>24.14% (3.45%)</td><td>59.77% (11.49%)</td><td>77.01% (0.00%)</td><td>10.34% (8.05%)</td><td>-</td><td>62.07% (8.05%)</td><td>8.05% (0.00%)</td><td>75.86% (4.60%)</td></tr><tr><td>7</td><td>79.80% (65.66%)</td><td>77.78% (26.26%)</td><td>20.20% (8.08%)</td><td>8.08% (4.04%)</td><td>100.00% (0.00%)</td><td>56.57% (23.23%)</td><td>97.98% (17.17%)</td><td>-</td><td>38.38% (1.01%)</td><td>17.17% (10.10%)</td></tr><tr><td>8</td><td>94.32% (84.09%)</td><td>96.59% (18.18%)</td><td>60.23% (42.05%)</td><td>57.95% (43.18%)</td><td>100.00% (0.00%)</td><td>93.18% (80.68%)</td><td>100.00% (57.95%)</td><td>100.00% (34.09%)</td><td>-</td><td>87.50% (26.14%)</td></tr><tr><td>9</td><td>98.91% (79.35%)</td><td>97.83% (33.70%)</td><td>26.09% (1.09%)</td><td>17.39% (2.17%)</td><td>100.00% (0.00%)</td><td>22.83% (21.74%)</td><td>100.00% (30.43%)</td><td>47.83% (43.48%)</td><td>31.52% (4.35%)</td><td>-</td></tr></table>



Table content: <table/>

L 2 Optimization L VAE Attack on MNIST (single image target): AS ignore-target (AS target in parentheses) for different source and target class pairs using adversarial examples generated on the VAE-GAN model. Higher values indicate more successful attacks against the generative model.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 90.36% (14.46%)</td><td>Target 2 100.00% (100.00%)</td><td>Target 3 100.00% (98.80%)</td><td>Target 4 100.00% (61.45%)</td><td>Target 5 91.57% (90.36%)</td><td>Target 6 100.00% (96.39%)</td><td>Target 7 68.67% (50.60%)</td><td>Target 8 100.00% (91.57%)</td><td>Target 9 98.80% (37.35%)</td></tr><tr><td>1</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (99.20%)</td><td>100.00% (100.00%)</td><td>100.00% (97.60%)</td><td>100.00% (96.00%)</td><td>100.00% (100.00%)</td><td>100.00% (96.00%)</td></tr><tr><td>2</td><td>100.00% (100.00%)</td><td>84.21% (60.53%)</td><td>-</td><td>100.00% (100.00%)</td><td>90.35% (71.93%)</td><td>100.00% (85.96%)</td><td>88.60% (88.60%)</td><td>97.37% (76.32%)</td><td>94.74% (94.74%)</td><td>97.37% (35.09%)</td></tr><tr><td>3</td><td>100.00% (100.00%)</td><td>75.70% (66.36%)</td><td>100.00% (100.00%)</td><td>-</td><td>94.39% (52.34%)</td><td>99.07% (99.07%)</td><td>98.13% (82.24%)</td><td>64.49% (53.27%)</td><td>100.00% (96.26%)</td><td>67.29% (31.78%)</td></tr><tr><td>4</td><td>100.00% (100.00%)</td><td>100.00% (52.73%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (97.27%)</td><td>100.00% (100.00%)</td><td>100.00% (99.09%)</td><td>100.00% (100.00%)</td><td>85.45% (83.64%)</td></tr><tr><td>5</td><td>100.00% (100.00%)</td><td>96.55% (40.23%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>93.10% (59.77%)</td><td>-</td><td>100.00% (95.40%)</td><td>93.10% (71.26%)</td><td>96.55% (96.55%)</td><td>83.91% (51.72%)</td></tr><tr><td>6</td><td>100.00% (100.00%)</td><td>97.70% (70.11%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (91.95%)</td><td>100.00% (100.00%)</td><td>-</td><td>97.70% (67.82%)</td><td>100.00% (98.85%)</td><td>95.40% (50.57%)</td></tr><tr><td>7</td><td>100.00% (100.00%)</td><td>85.86% (58.59%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (98.99%)</td><td>100.00% (97.98%)</td><td>100.00% (79.80%)</td><td>-</td><td>100.00% (98.99%)</td><td>100.00% (96.97%)</td></tr><tr><td>8</td><td>100.00% (100.00%)</td><td>69.32% (44.32%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>54.55% (53.41%)</td><td>96.59% (96.59%)</td><td>95.45% (92.05%)</td><td>73.86% (52.27%)</td><td>-</td><td>42.05% (29.55%)</td></tr><tr><td>9</td><td>100.00% (100.00%)</td><td>100.00% (44.57%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>96.74% (95.65%)</td><td>100.00% (97.83%)</td><td>100.00% (100.00%)</td><td>100.00% (97.83%)</td><td>100.00% (100.00%)</td><td>-</td></tr></table>

L 2 Optimization L VAE Attack (mean reconstruction target): AS ignore-target (AS target in parentheses) for all source and target class pairs using adversarial examples generated on the VAE-GAN model. The mean reconstruction image for each target class (over all of the images of that class in the training set) is used as the target reconstruction. Higher values indicate more successful attacks against the generative model.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 85.54% (34.94%)</td><td>Target 2 100.00% (100.00%)</td><td>Target 3 100.00% (13.25%)</td><td>Target 4 75.90% (75.90%)</td><td>Target 5 96.39% (92.77%)</td><td>6 100.00% (100.00%)</td><td>Target 7 96.39% (91.57%)</td><td>Target 8 0.00% (0.00%)</td><td>Target 9 100.00% (83.13%)</td></tr><tr><td>1</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>100.00% (93.60%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>100.00% (98.40%)</td></tr><tr><td>2</td><td>100.00% (100.00%)</td><td>97.37% (55.26%)</td><td>-</td><td>100.00% (55.26%)</td><td>97.37% (88.60%)</td><td>95.61% (74.56%)</td><td>100.00% (100.00%)</td><td>99.12% (94.74%)</td><td>100.00% (0.00%)</td><td>100.00% (92.98%)</td></tr><tr><td>3</td><td>100.00% (100.00%)</td><td>90.65% (89.72%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (91.59%)</td><td>94.39% (94.39%)</td><td>100.00% (100.00%)</td><td>85.05% (84.11%)</td><td>100.00% (0.00%)</td><td>90.65% (88.79%)</td></tr><tr><td>4</td><td>100.00% (100.00%)</td><td>97.27% (67.27%)</td><td>100.00% (100.00%)</td><td>100.00% (18.18%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>100.00% (100.00%)</td></tr><tr><td>5</td><td>100.00% (100.00%)</td><td>96.55% (80.46%)</td><td>100.00% (100.00%)</td><td>2.30% (2.30%)</td><td>100.00% (96.55%)</td><td>-</td><td>100.00% (100.00%)</td><td>98.85% (89.66%)</td><td>100.00% (0.00%)</td><td>95.40% (94.25%)</td></tr><tr><td>6</td><td>100.00% (100.00%)</td><td>87.36% (80.46%)</td><td>100.00% (100.00%)</td><td>100.00% (11.49%)</td><td>100.00% (97.70%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (98.85%)</td><td>100.00% (0.00%)</td><td>100.00% (96.55%)</td></tr><tr><td>7</td><td>100.00% (100.00%)</td><td>90.91% (82.83%)</td><td>100.00% (100.00%)</td><td>100.00% (16.16%)</td><td>100.00% (79.80%)</td><td>100.00% (98.99%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (0.00%)</td><td>100.00% (100.00%)</td></tr><tr><td>8</td><td>100.00% (100.00%)</td><td>89.77% (71.59%)</td><td>100.00% (100.00%)</td><td>100.00% (35.23%)</td><td>100.00% (97.73%)</td><td>89.77% (62.50%)</td><td>100.00% (100.00%)</td><td>98.86% (92.05%)</td><td>-</td><td>98.86% (96.59%)</td></tr><tr><td>9</td><td>100.00% (100.00%)</td><td>95.65% (75.00%)</td><td>100.00% (100.00%)</td><td>100.00% (18.48%)</td><td>100.00% (97.83%)</td><td>100.00% (95.65%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (0.00%)</td><td>-</td></tr><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 40.96% (10.84%)</td><td>Target 2 65.06% (65.06%)</td><td>Target 3 53.01% (46.99%)</td><td>Target 4 62.65% (54.22%)</td><td>Target 5 36.14% (36.14%)</td><td>Target 6 59.04% (59.04%)</td><td>Target 7 46.99% (46.99%)</td><td>Target 8 13.25% (12.05%)</td><td>Target 9 44.58% (27.71%)</td></tr><tr><td>1</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (96.80%)</td></tr><tr><td>2</td><td>96.49% (96.49%)</td><td>60.53% (59.65%)</td><td>-</td><td>95.61% (95.61%)</td><td>78.07% (75.44%)</td><td>98.25% (71.05%)</td><td>94.74% (90.35%)</td><td>71.05% (69.30%)</td><td>52.63% (50.88%)</td><td>75.44% (42.98%)</td></tr><tr><td>3</td><td>100.00% (100.00%)</td><td>87.85% (66.36%)</td><td>90.65% (90.65%)</td><td>-</td><td>85.98% (73.83%)</td><td>95.33% (95.33%)</td><td>79.44% (53.27%)</td><td>65.42% (64.49%)</td><td>59.81% (46.73%)</td><td>70.09% (58.88%)</td></tr><tr><td>4</td><td>99.09% (99.09%)</td><td>67.27% (66.36%)</td><td>96.36% (96.36%)</td><td>100.00% (81.82%)</td><td>-</td><td>100.00% (98.18%)</td><td>93.64% (93.64%)</td><td>98.18% (95.45%)</td><td>97.27% (92.73%)</td><td>39.09% (39.09%)</td></tr><tr><td>5</td><td>100.00% (100.00%)</td><td>79.31% (51.72%)</td><td>100.00% (83.91%)</td><td>70.11% (70.11%)</td><td>80.46% (72.41%)</td><td>-</td><td>73.56% (73.56%)</td><td>87.36% (73.56%)</td><td>55.17% (52.87%)</td><td>75.86% (65.52%)</td></tr><tr><td>6</td><td>97.70% (97.70%)</td><td>68.97% (50.57%)</td><td>96.55% (96.55%)</td><td>95.40% (71.26%)</td><td>73.56% (73.56%)</td><td>87.36% (77.01%)</td><td>-</td><td>88.51% (72.41%)</td><td>90.80% (55.17%)</td><td>91.95% (35.63%)</td></tr><tr><td>7</td><td>100.00% (97.98%)</td><td>83.84% (83.84%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>93.94% (90.91%)</td><td>98.99% (96.97%)</td><td>88.89% (81.82%)</td><td>-</td><td>100.00% (86.87%)</td><td>50.51% (50.51%)</td></tr><tr><td>8</td><td>100.00% (100.00%)</td><td>96.59% (78.41%)</td><td>100.00% (100.00%)</td><td>98.86% (95.45%)</td><td>94.32% (86.36%)</td><td>98.86% (98.86%)</td><td>98.86% (93.18%)</td><td>98.86% (73.86%)</td><td>-</td><td>87.50% (78.41%)</td></tr><tr><td>9</td><td>100.00% (100.00%)</td><td>100.00% (76.09%)</td><td>100.00% (100.00%)</td><td>98.91% (96.74%)</td><td>100.00% (100.00%)</td><td>100.00% (98.91%)</td><td>97.83% (97.83%)</td><td>98.91% (98.91%)</td><td>97.83% (94.57%)</td><td>-</td></tr></table>

L 2 Optimization Latent Attack (mean latent vector target): AS ignore-target (AS target in parentheses) for all source and target class pairs using adversarial examples generated on the VAE-GAN model. The mean latent vector for each target class (over all of the images of that class in the training set) is used as the target latent vector. Higher values indicate more successful attacks against the generative model.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 95.18% (9.64%)</td><td>Target 2 100.00% (100.00%)</td><td>Target 3 98.80% (93.98%)</td><td>Target 4 100.00% (48.19%)</td><td>Target 5 91.57% (89.16%)</td><td>Target 6 100.00% (89.16%)</td><td>Target 7 73.49% (43.37%)</td><td>Target 8 100.00% (87.95%)</td><td>Target 9 100.00% (25.30%)</td></tr><tr><td>1</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (92.80%)</td><td>100.00% (97.60%)</td><td>100.00% (98.40%)</td><td>100.00% (76.00%)</td><td>100.00% (100.00%)</td><td>100.00% (90.40%)</td></tr><tr><td>2</td><td>98.25% (98.25%)</td><td>83.33% (48.25%)</td><td>-</td><td>100.00% (100.00%)</td><td>88.60% (43.86%)</td><td>99.12% (63.16%)</td><td>74.56% (71.93%)</td><td>99.12% (63.16%)</td><td>93.86% (92.98%)</td><td>99.12% (21.05%)</td></tr><tr><td>3</td><td>99.07% (98.13%)</td><td>57.01% (42.99%)</td><td>99.07% (99.07%)</td><td>-</td><td>82.24% (36.45%)</td><td>89.72% (88.79%)</td><td>99.07% (61.68%)</td><td>57.01% (37.38%)</td><td>98.13% (92.52%)</td><td>67.29% (18.69%)</td></tr><tr><td>4</td><td>100.00% (100.00%)</td><td>100.00% (37.27%)</td><td>100.00% (100.00%)</td><td>100.00% (99.09%)</td><td>-</td><td>100.00% (80.00%)</td><td>98.18% (93.64%)</td><td>100.00% (94.55%)</td><td>100.00% (99.09%)</td><td>86.36% (80.00%)</td></tr><tr><td>5</td><td>100.00% (100.00%)</td><td>97.70% (19.54%)</td><td>100.00% (98.85%)</td><td>98.85% (98.85%)</td><td>85.06% (44.83%)</td><td>-</td><td>95.40% (88.51%)</td><td>93.10% (45.98%)</td><td>96.55% (96.55%)</td><td>87.36% (34.48%)</td></tr><tr><td>6</td><td>100.00% (100.00%)</td><td>96.55% (58.62%)</td><td>100.00% (98.85%)</td><td>100.00% (98.85%)</td><td>100.00% (86.21%)</td><td>100.00% (97.70%)</td><td>-</td><td>100.00% (56.32%)</td><td>100.00% (96.55%)</td><td>95.40% (43.68%)</td></tr><tr><td>7</td><td>100.00% (100.00%)</td><td>80.81% (40.40%)</td><td>100.00% (100.00%)</td><td>100.00% (98.99%)</td><td>100.00% (92.93%)</td><td>100.00% (87.88%)</td><td>100.00% (62.63%)</td><td>-</td><td>100.00% (97.98%)</td><td>100.00% (88.89%)</td></tr><tr><td>8</td><td>100.00% (100.00%)</td><td>44.32% (18.18%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>30.68% (28.41%)</td><td>78.41% (76.14%)</td><td>89.77% (81.82%)</td><td>75.00% (38.64%)</td><td>-</td><td>22.73% (15.91%)</td></tr><tr><td>9</td><td>100.00% (100.00%)</td><td>98.91% (17.39%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>97.83% (92.39%)</td><td>100.00% (89.13%)</td><td>100.00% (92.39%)</td><td>98.91% (94.57%)</td><td>100.00% (100.00%)</td><td>-</td></tr></table>

L 2 Optimization L

Table content: <table/>

L 2 Optimization L VAE Attack on MNIST (single image target): AS ignore-target (AS target in parentheses) for different source and target class pairs using adversarial examples generated on the VAE-GAN model. Higher values indicate more successful attacks against the generative model.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 92.77% (38.55%)</td><td>Target 2 100.00% (100.00%)</td><td>Target 3 100.00% (66.27%)</td><td>Target 4 100.00% (34.94%)</td><td>Target 5 100.00% (22.89%)</td><td>Target 6 100.00% (100.00%)</td><td>Target 7 79.52% (63.86%)</td><td>Target 8 97.59% (90.36%)</td><td>Target 9 100.00% (62.65%)</td></tr><tr><td>1</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (99.20%)</td><td>100.00% (90.40%)</td><td>100.00% (0.80%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td></tr><tr><td>2</td><td>97.37% (97.37%)</td><td>97.37% (57.02%)</td><td>-</td><td>100.00% (87.72%)</td><td>98.25% (42.11%)</td><td>100.00% (50.88%)</td><td>100.00% (99.12%)</td><td>(89.47%)</td><td>89.47% (89.47%)</td><td>100.00% (81.58%)</td></tr><tr><td>3</td><td>100.00% (100.00%)</td><td>89.72% (85.05%)</td><td>100.00% (100.00%)</td><td>-</td><td>62.62% (48.60%)</td><td>91.59% (45.79%)</td><td>100.00% (99.07%)</td><td>95.33% (90.65%)</td><td>97.20% (94.39%)</td><td>90.65% (79.44%)</td></tr><tr><td>4</td><td>100.00% (100.00%)</td><td>95.45% (67.27%)</td><td>100.00% (100.00%)</td><td>100.00% (73.64%)</td><td>-</td><td>100.00% (30.00%)</td><td>100.00% (100.00%)</td><td>100.00% (99.09%)</td><td>100.00% (99.09%)</td><td>99.09% (99.09%)</td></tr><tr><td>5</td><td>100.00% (100.00%)</td><td>98.85% (79.31%)</td><td>100.00% (100.00%)</td><td>73.56% (73.56%)</td><td>83.91% (34.48%)</td><td>-</td><td>100.00% (100.00%)</td><td>90.80% (87.36%)</td><td>100.00% (100.00%)</td><td>87.36% (82.76%)</td></tr><tr><td>6</td><td>100.00% (100.00%)</td><td>86.21% (79.31%)</td><td>100.00% (100.00%)</td><td>100.00% (88.51%)</td><td>95.40% (71.26%)</td><td>10.34% (10.34%)</td><td>-</td><td>100.00% (83.91%)</td><td>100.00% (97.70%)</td><td>100.00% (70.11%)</td></tr><tr><td>7</td><td>100.00% (100.00%)</td><td>91.92% (79.80%)</td><td>100.00% (100.00%)</td><td>100.00% (87.88%)</td><td>100.00% (63.64%)</td><td>100.00% (58.59%)</td><td>100.00% (100.00%)</td><td>-</td><td>100.00% (100.00%)</td><td>100.00% (100.00%)</td></tr><tr><td>8</td><td>100.00% (100.00%)</td><td>88.64% (73.86%)</td><td>100.00% (100.00%)</td><td>100.00% (46.59%)</td><td>95.45% (44.32%)</td><td>96.59% (31.82%)</td><td>100.00% (100.00%)</td><td>96.59% (94.32%)</td><td>-</td><td>95.45% (79.55%)</td></tr><tr><td>9</td><td>100.00% (100.00%)</td><td>96.74% (72.83%)</td><td>100.00% (100.00%)</td><td>100.00% (59.78%)</td><td>66.30% (63.04%)</td><td>100.00% (28.26%)</td><td>100.00% (100.00%)</td><td>98.91% (98.91%)</td><td>100.00% (100.00%)</td><td>-</td></tr></table>

L 2 Optimization Latent Attack on SVHN (single latent vector target): AS ignore-target (AS target in parentheses) after one reconstruction loop for different source and target class pairs on the VAE-GAN model. The latent representation of a random image from the target class is used to generate the target latent vector. Higher values indicate more successful attacks against the generative model.

Table content: <table><tr><td>Source 0</td><td>Target 0 -</td><td>Target 1 64.29% (40.00%)</td><td>Target 2 78.57% (61.43%)</td><td>Target 3 92.86% (80.00%)</td><td>Target 4 84.29% (57.14%)</td><td>Target 5 98.57% (98.57%)</td><td>Target 6 94.29% (38.57%)</td><td>Target 7 88.57% (54.29%)</td><td>Target 8 95.71% (11.43%)</td><td>Target 9 95.71% (25.71%)</td></tr><tr><td>1</td><td>76.80% (70.72%)</td><td>-</td><td>74.59% (67.40%)</td><td>93.37% (88.95%)</td><td>75.69% (65.19%)</td><td>98.34% (97.79%)</td><td>86.74% (24.86%)</td><td>46.96% (36.46%)</td><td>96.13% (4.97%)</td><td>96.13% (28.73%)</td></tr><tr><td>2</td><td>82.93% (65.85%)</td><td>57.93% (42.68%)</td><td>-</td><td>90.24% (86.59%)</td><td>53.66% (46.34%)</td><td>99.39% (98.17%)</td><td>82.93% (14.02%)</td><td>71.34% (57.32%)</td><td>71.34% (6.71%)</td><td>24.39% (23.17%)</td></tr><tr><td>3</td><td>92.17% (64.35%)</td><td>58.26% (41.74%)</td><td>83.48% (68.70%)</td><td>-</td><td>84.35% (49.57%)</td><td>96.52% (95.65%)</td><td>53.91% (23.48%)</td><td>90.43% (56.52%)</td><td>93.04% (5.22%)</td><td>93.91% (33.91%)</td></tr><tr><td>4</td><td>74.44% (55.56%)</td><td>47.78% (43.33%)</td><td>70.00% (61.11%)</td><td>86.67% (77.78%)</td><td>-</td><td>100.00% (98.89%)</td><td>93.33% (35.56%)</td><td>90.00% (36.67%)</td><td>85.56% (14.44%)</td><td>94.44% (27.78%)</td></tr><tr><td>5</td><td>75.31% (50.62%)</td><td>59.26% (43.21%)</td><td>88.89% (58.02%)</td><td>97.53% (88.89%)</td><td>72.84% (53.09%)</td><td>-</td><td>37.04% (18.52%)</td><td>80.25% (41.98%)</td><td>32.10% (6.17%)</td><td>92.59% (30.86%)</td></tr><tr><td>6</td><td>67.44% (47.67%)</td><td>56.98% (27.91%)</td><td>84.88% (55.81%)</td><td>86.05% (79.07%)</td><td>65.12% (39.53%)</td><td>94.19% (94.19%)</td><td>-</td><td>90.70% (33.72%)</td><td>58.14% (10.47%)</td><td>87.21% (22.09%)</td></tr><tr><td>7</td><td>87.34% (63.29%)</td><td>55.70% (48.10%)</td><td>79.75% (74.68%)</td><td>92.41% (79.75%)</td><td>69.62% (41.77%)</td><td>97.47% (89.87%)</td><td>93.67% (18.99%)</td><td>-</td><td>91.14% (7.59%)</td><td>97.47% (17.72%)</td></tr><tr><td>8</td><td>98.33% (63.33%)</td><td>78.33% (38.33%)</td><td>80.00% (63.33%)</td><td>100.00% (88.33%)</td><td>93.33% (48.33%)</td><td>98.33% (96.67%)</td><td>96.67% (35.00%)</td><td>96.67% (50.00%)</td><td>-</td><td>95.00% (31.67%)</td></tr><tr><td>9</td><td>87.88% (66.67%)</td><td>72.73% (43.94%)</td><td>92.42% (80.30%)</td><td>93.94% (86.36%)</td><td>80.30% (51.52%)</td><td>95.45% (93.94%)</td><td>98.48% (27.27%)</td><td>92.42% (62.12%)</td><td>93.94% (9.09%)</td><td>-</td></tr></table>

L 2 Optimization L VAE Attack on SVHN (single image target): AS ignore-target (AS target in parentheses) after one reconstruction loop for different source and target class pairs on the VAE-GAN model. The latent representation of a random image from the target class is used to generate the target latent vector. Higher values indicate more successful attacks against the generative model.

Table content: <table/>

Peer review (iclr2017):
After the rebuttal: The paper contains an interesting set of results (mainly produced after the initial submission), but novelty is limited, and presentation is suboptimal. For me now the biggest problem now is that the title and the content do not correspond. The authors clearly attack deterministic encoder-decoder models (as described in 3.2), which are not at all the same as generative models, even though many generative models make use of this architecture. A small experiment with sampling is interesting, but does not change the overall focus of the paper. This inconsistency in not acceptable. The whole issue could be resolved for example by simply replacing *generative models* by *encoder-decoder networks* in the title. Then I would tend towards recommending acceptance. ------ Initial review: The paper describes three approaches to generating adversarial examples for deep encoder-decoder generative networks (trained as VAE or VAE-GAN), and shows a comparative analysis of these. While the phenomenon of adversarial examples in discriminative models is widely known and relatively well studied, I am not aware of previous work on adversarial examples for generative networks, so this work is novel (there is a concurrent work by Tabacof et al. which should be cited, though). The paper has significantly improved since the initial submission; still, I have a number of remarks on presentation and experimental evaluation. I am in the borderline mode, and may change my rating during the discussion phase. Detailed comments: 1) The paper is 13 pages long - significantly over the recommended page limit of 8 pages. Reviewers have to read multiple papers, multiple versions of each, it is a lot of work. Large portions of the paper should be shortened and/or moved to the appendix. It is job of the authors to make the paper concise and readable. *in our attempts to be thorough, we have had a hard time keeping the length down* is a bad excuse - it may be hard, but has to be done. 2) I intentionally avoided term *generative model* above because it is not obvious to me if the attacks described by the authors indeed attack generative models. To clarify, the authors train encoder-decoders as generative models (VAE or VAE-GAN), but then remove all stochasticity (sampling) and prior on the latent variables: that is, they treat the models as deterministic encoders-decoders. It is not a big surprise that a deterministic deep network can be easily tricked; it would be much more interesting to see if the probabilistic aspect of generative models makes them more robust to such attacks. Am I missing something? I would like the authors to clarify their view on this and adjust the claims in the paper if necessary. 3) The paper is motivated by possible attacks on a data channel which uses a generative network for compressing information. Description of the attack scenario in 3.1 does not look convincing to me. It takes a huge amount of space and I do not think it adds much to the paper. First, experiments on natural images are necessary to judge if the proposed attack could succeed in a realistic scenario and second, I am not aware of any existing practical applications of VAEs to image compression: attacking JPEG would be much more practical. 4) Experiments are limited to MNIST and, in the latest version, SVHN (which is very nice). While no good generative models of general natural images exist, it is common to evaluate generative models on datasets of faces, so this would be another very natural domain for testing the proposed approach. Smaller remarks: 1) Usage of *Oracle* in 3.2 does not look appropriate - oracle typically has access to (part of) ground truth, which is not the case here as far as I understand. 2) Beginning of section 4: *All three methods work for any generative architecture that relies on a learned latent representation z* - *are technically applicable to* would be more correct than *work for*. 3) 4.1: *confidentally*