Title: Adversarial examples for generative models
Authors: Jernej Kos
Ian Fischer
Dawn Song
Venue: ICLR 2017
Keywords: Computer vision
Unsupervised Learning

Abstract: We explore methods of producing adversarial examples on deep generative models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning architectures are known to be vulnerable to adversarial examples, but previous work has focused on the application of adversarial examples to classification tasks. Deep generative models have recently become popular due to their ability to model input data distributions and generate realistic examples from those distributions. We present three classes of attacks on the VAE and VAE-GAN architectures and demonstrate them against networks trained on MNIST, SVHN and CelebA. Our first attack leverages classification-based adversaries by attaching a classifier to the trained encoder of the target generative model, which can then be used to indirectly manipulate the latent representation. Our second attack directly uses the VAE loss function to generate a target reconstruction image from the adversarial example. Our third attack moves beyond relying on classification or the standard loss for the gradient and directly optimizes against differences in source and target latent representations. We also motivate why an attacker might be interested in deploying such techniques against a target generative network.

TL;DR: Exploration of ways to attack generative models with adversarial examples and why someone might want to do that.

Review discussion:
Figure 7 suggests that your adversarial approach is systematically distorting the representation such that the reconstructed images belong to a different class. Have you compared the nearest neighbor of your perturbed embeddings to the embeddings in your training data set?
Additionally, have you quantified the degree to which the reconstructed class might be mis-classified as a different class?
We have not done the nearest neighbor experiment yet, but it’s an excellent suggestion -- we hope to have that in our next draft.  We have just uploaded a new draft that substantially expands and clarifies our results.  One of the major additions is showing how the reconstructions are classified and using that as the basis for some metrics about how the different approaches perform, which sounds very much like what you are suggesting.  Please take a look!
Hello,
An interesting paper! Below are some questions. They are based on the yesterday version of the paper, and I see some of them are addressed in the new version which just appeared.
But unfortunately I do not have time to carefully re-read the paper now, so just let me know if some of these do not apply any more.
1. Do you believe the method would be applicable in the attack scenario you described? The added perturbations are very visible.
2. To me the most interesting aspect of this paper is that the proposed method does _not_ work very well, meaning that generative models are "difficult to fool". Do you agree with this intuition? Have you considered this perspective?
3. Have you experimented with other datasets? Ideally natural images, but at least faces?
4. Why are you only using the fast gradient sign method for generating adversarial examples? Wouldn't you get better results with iterative optimization?
5. Don't you think viewing the figures would be more convenient if you grouped the results by samples, not by methods? As in Figure 7.
Thanks!
Thanks for the questions! In this work, we’ve chosen to explore the attacks in breadth and depth, which has caused us to remain focused on MNIST for the time being.
However, exploring adversarial attacks on natural image datasets like CIFAR and ImageNet is in our plan for upcoming work (although we don’t mention that in the paper yet).  Using MNIST has some advantages in terms of clarity of results, but, as your questions indicate, the attacks end up having different behavior than they will on natural image datasets, so we can’t claim to be sure that our results will generalize. I would hypothesize that there will be a few major differences once we start on ImageNet:
The attacks will be much less noticeable.
The generative reconstructions (on clean and adversarial images) will be of much lower quality.
Mean latent vector attacks will not generate as good results with large numbers of images, so we will be limited to a very small number of images for that class of attacks.
Individual question responses are below:
1. We agree that the attacks are quite visible on MNIST.  That has been the case for most attacks in the literature, however (e.g., with the fast gradient sign, in order to get high attack success rates on MNIST, we’ve found that it’s necessary to have an epsilon of >= 0.3, which is much more noticeable than our L2 optimization attacks).  If we have a generative model that works well with natural images, then we think the various attack scenarios are quite plausible.  The new draft goes into more detail on motivating the attack (Section 3.1) -- let us know if you think it is unclear or problematic in some way.
2. We kept hoping that there would be an interesting defense story in the results, but we haven’t been able to convince ourselves yet that it’s there. Adversarial examples on MNIST are almost always easy to detect, since the dataset is so clean.
3. Not yet, but we would like to.
4. The new version focuses almost entirely on optimization attacks, which do indeed perform much better than FGS.
5. Yes, we should probably include more figures with different inputs in each row and attack details for the different attacks in each column. Thanks for the suggestion!
Looking at Figure 8 (first version of paper, untargeted FGS classifier attack), it seems that maybe only 10% of MNIST digits are incorrectly reconstructed. Do the authors think this method is a success in achieving what they defined in the problem statement, i.e., "reconstruction looks more like something from another class."?
How is this paper different from the "Adversarial Images for Variational Autoencoders" paper, in terms of both the ideas and the performance? Specially, I think the idea of "latent attack" is very similar in both papers. I think there has to be a section in the paper to explain the difference between these two works.
MNIST dataset is the only dataset that is considered, which is not very convincing.
The paper is much longer than the standard page-limit, which makes it hard to follow.
Thanks for the comments!  Responses are below:
- Yes, the untargeted FGS attacks are not convincing.  We include them because FGS is a common baseline attack in the literature, but we focus on the optimization attacks because those are much more successful.
- We only learned of this workshop paper at NIPS, so we didn’t have a reference to it in previous drafts.
It is worth noting that our first public draft was uploaded to OpenReview before their first public version (according to arxiv) by almost a month, so this is an example of contemporaneous work.  It appears that the attack they present in their paper is similar to our latent attack, although they use a KL divergence rather than the L2 distance.  Since they only provide one set of examples for the two datasets, it is somewhat hard to compare their attack to our three attacks.  We’ve added a brief reference to that paper in our related work section in the latest draft.
- We had some minor technical issues that prevented us from including SVHN previously, but those are now resolved, so we have included SVHN in our latest draft.  Please take a look!
- Yes, in our attempts to be thorough, we have had a hard time keeping the length down.  We may need to rework some of the data presentation to shorten the main body.
After the rebuttal:
The paper contains an interesting set of results (mainly produced after the initial submission), but novelty is limited, and presentation is suboptimal.
For me now the biggest problem now is that the title and the content do not correspond. The authors clearly attack deterministic encoder-decoder models (as described in 3.2), which are not at all the same as generative models, even though many generative models make use of this architecture. A small experiment with sampling is interesting, but does not change the overall focus of the paper. This inconsistency in not acceptable. The whole issue could be resolved for example by simply replacing "generative models" by "encoder-decoder networks" in the title.
Then I would tend towards recommending acceptance.
Initial review:
The paper describes three approaches to generating adversarial examples for deep encoder-decoder generative networks (trained as VAE or VAE-GAN), and shows a comparative analysis of these.
While the phenomenon of adversarial examples in discriminative models is widely known and relatively well studied, I am not aware of previous work on adversarial examples for generative networks, so this work is novel (there is a concurrent work by Tabacof et al. which should be cited, though).  The paper has significantly improved since the initial submission;
still, I have a number of remarks on presentation and experimental evaluation. I am in the borderline mode, and may change my rating during the discussion phase.
Detailed comments:
1) The paper is 13 pages long - significantly over the recommended page limit of 8 pages. Reviewers have to read multiple papers, multiple versions of each, it is a lot of work. Large portions of the paper should be shortened and/or moved to the appendix. It is job of the authors to make the paper concise and readable. "in our attempts to be thorough, we have had a hard time keeping the length down" is a bad excuse - it may be hard, but has to be done.
2) I intentionally avoided term "generative model" above because it is not obvious to me if the attacks described by the authors indeed attack generative models. To clarify, the authors train encoder-decoders as generative models (VAE or VAE-GAN), but then remove all stochasticity (sampling) and prior on the latent variables: that is, they treat the models as deterministic encoders-decoders. It is not a big surprise that a deterministic deep network can be easily tricked; it would be much more interesting to see if the probabilistic aspect of generative models makes them more robust to such attacks. Am I missing something? I would like the authors to clarify their view on this and adjust the claims in the paper if necessary.
3) The paper is motivated by possible attacks on a data channel which uses a generative network for compressing information. Description of the attack scenario in 3.1 does not look convincing to me. It takes a huge amount of space and I do not think it adds much to the paper.
First, experiments on natural images are necessary to judge if the proposed attack could succeed in a realistic scenario and second, I am not aware of any existing practical applications of VAEs to image compression: attacking JPEG would be much more practical.
4) Experiments are limited to MNIST and, in the latest version, SVHN (which is very nice).
While no good generative models of general natural images exist, it is common to evaluate generative models on datasets of faces, so this would be another very natural domain for testing the proposed approach.
Smaller remarks:
1) Usage of "Oracle" in 3.2 does not look appropriate - oracle typically has access to (part of) ground truth, which is not the case here as far as I understand.
2) Beginning of section 4: "All three methods work for any generative architecture that relies on a learned latent representation z" - "are technically applicable to" would be more correct than "work for".
3) 4.1: "confidentally"
Thanks for the review!  The new draft is shorter, reducing it to 11 pages while also adding some new results.
In this draft we have added experimental results using stochastic sampling. They are referenced in the second paragraph of Section 5 with a figure in the appendix. The results show that for most examples, sampling doesn’t meaningfully change the reconstructed adversarial examples, and the attack is similarly successful even when sampling is being used.
We have also added some citations to clarify where the attack scenario sits in the current literature.  There are a few recent publications exploring using deep networks as compression models, similar to what we describe in Section 3.1, for example: https://arxiv.org/abs/1511.06085, and https://arxiv.org/abs/1608.05148.  We are not attempting to directly attack these networks in this work, however -- we are instead motivating why this type of attack is relevant to current work.
Thanks for the suggestion to use faces!  We have added experiments on CelebA in the latest draft.  The results in Section 5.3 show that the attacks are equally effective even in the more complex domain.  We have also addressed your other comments in this draft.
This paper considers different methods of producing adversarial examples for generative models such as VAE and VAEGAN.
Specifically, three methods are considered: classification-based adversaries which uses a classifier on top of the hidden code, VAE loss which directly uses the VAE loss and the "latent attack" which finds adversarial perturbation in the input so as to match the latent representation of a target input.
I think the problem that this paper considers is potentially useful and interesting to the community. To the best of my knowledge this is the first paper that considers adversarial examples for generative models. As I pointed out in my pre-review comments, there is also a concurrent work of "Adversarial Images for Variational Autoencoders" that essentially proposes the same "latent attack" idea of this paper with both L2 distance and KL divergence.
Novelty/originality: I didn't find the ideas of this paper very original. All the proposed three attacks are well-known and standard methods that here are applied to a new problem and this paper does not develop *novel* algorithms for attacking specifically *generative models*.
However I still find it interesting to see how these standard methods compare in this new problem domain.
The clarity and presentation of the paper is very unsatisfying. The first version of the paper proposes the "classification-based adversaries" and reports only negative results. In the second set of revisions, the core idea of the paper changes and almost an entirely new paper with a new co-author is submitted and the idea of "latent attack" is proposed which works much better than the "classification-based adversaries".
However, the authors try to keep around the materials of the first version, which results in a 13 page long paper, with different claims and unrelated set of experiments. "in our attempts to be thorough, we have had a hard time keeping the length down" is not a valid excuse.
In short, the paper is investigating an interesting problem and apply and compare standard adversarial methods to this domain, but the novelty and the presentation of the paper is limited.
Thanks for the review, and for pointing out the NIPS workshop paper. That work was not known at the time of our submission to ICLR, as their paper was published online one month after our initial ICLR submission, and a couple of weeks after our first major revision. We have added a paragraph explaining the differences with their paper. It appears that their paper comes to a different conclusion regarding the L_vae attack -- we find it to be effective on MNIST, SVHN and CelebA, while they say that they couldn’t get it to work. We also present a more in-depth study of the topic, which considers more attacks, more generative model architectures, and more datasets (with our addition of CelebA results in the latest draft).
We agree that ultimately we were able to determine that existing attacks could be modified to apply to the new domain of generative models, and so it may feel self-evident in retrospect.
However, prior to our work, it was unknown in the literature whether the stochasticity of generative models would confer robustness, or what an attack on a generative model would look like, or even under what scenarios an attacker might want to attack a generative model.  With our work, the research community knows that generative models are effectively as vulnerable to adversarial examples as deterministic classifiers, and that there are realistic scenarios where such attacks could matter.  In our opinion, this is an important result, even though it didn’t require discovering substantially different mathematical approaches (and perhaps it is even more important because the attacks transferred so naturally, underscoring the vulnerability of current architectures).
It’s true that we did a lot of work on this paper after the initial deadline.  ICLR’s unusual submission and review process permits and encourages authors to continue revising their work after submission, which clearly has some advantages and disadvantages for all parties involved.  Presumably norms and expectations around how much papers change after submission will become more settled over time.  Thanks for your patience through the various revisions and additions we have made!  In this version, we have shortened and streamlined the paper, reducing it to 11 pages while also adding some new results.
Comments:
"This contrasts to adversarial attacks on classifiers, where any inspection of the inputs will reveal the original bytes the adversary supplied,
which often have telltale noise"
Is this really true?  If it were the case, wouldn't it imply that training "against" adversarial examples should easily make a classifier robust to adversarial examples (if they all have a telltale noise)?
Pros:
-The question of whether adversarial examples exist in generative models, and indeed how the definition of "adversarial example" carries over is an interesting one.
-Finding that a certain type of generative model *doesn't have* adversarial examples would be a really significant result, finding that generative models have adversarial examples would also be a worth negative result.
-The adversarial examples in figures 5 and 6 seem convincing, though they seem much more overt and noisy than the adversarial examples on MNIST shown in (Szegedy 2014).  Is this because it's actually harder to find adversarial examples in these types of generative models?
Issues:
-Paper is significantly over length at 13 pages.
-The beginning of the paper should more clearly motivate its purpose.
-Paper has "generative models" in the title but as far as I can tell the whole paper is concerned with autoencoder-type models.  This is kind of annoying because if someone wanted to consider adversarial attacks on, say, autoregressive models, they might be unreasonably burdened by having to explain how they're distinct from a paper called "adversarial examples for generative models".
-I think that the introduction contains too much background information - it could be tightened.
What you say about the telltale noise is correct -- as Goodfellow et al. 2014 show, it is possible to train a network using the adversarial examples and substantially reduce (but not eliminate) the network’s vulnerability to adversarial examples.
Additionally, another ICLR submission this year showed a way to train a separate network to classify images as adversarial or non-adversarial with high accuracy.  All that being said, that paragraph is one of the paragraphs that we’ve removed from the new version of the paper in order to reduce the length.  The paper is now 11 pages plus references and includes some new results.
Since we are the first to study adversarial examples on generative models in depth, we think it is appropriate to use “generative models” in the title. The reason for focusing on VAE-like models is that in the paper we propose a plausible scenario how such a model can be used in a compression scheme, which can then be attacked by an adversary. To the best of our knowledge, other types of generative models, such as autoregressive models, do not offer such a clear attack scenario, where adversarial examples may be used by an attacker.

Meta-review:
role: PC
data: comment: The main idea in this paper is interesting, of considering various forms of adversarial examples in generative models. The paper has been considerably revised since the original submission. The results showing the susceptibility of generative models to attacks are interesting, but the demonstrations need to be more solid and on other datasets to be convincing.

Decision: Reject