Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ======== update ========
I have read the authors' response and it has addressed most of my concerns. I am glad to see the authors' experiments on online adversarial training. 

However, there is one additional concern that I didn't realize previously. Currently the performance of adversarial training is measured in "success rates". However it seems to me this success rates were not computed using human evaluation (since the authors claim once the classifier is finished training, the attack success rate can be larger than 99%). I would have changed my score to 8 if either 1) some adversarial images from the generator were included after finishing adversarial training or 2) success rates using human evaluation is reported. Unfortunately, I only realized this after the author rebuttal period, and the authors didn't have the chance to address this.

That being said, I feel this paper still presents interesting contribution to the field. I am still largely in favor of the acceptance of this paper, and will remain my rating of 6 for now. If this paper gets accepted, I strongly encourage the authors to address the concern I mentioned above in their camera ready.


======== original reviews ========

This paper proposes a novel method on generating unrestricted adversarial examples by finetuning GANs. The authors have conducted comprehensive experiments on evaluating the advantages of their approach. They demonstrated that their attack is harder to mitigate using adversarial training, produces unrestricted adversarial examples faster than existing methods, and can generate some unrestricted adversarial examples for complex high-dimensional datasets such as ImageNet.

I feel although the approach is straightforward, the authors have done a good job in motivating the method and have demonstrated its advantages via a good cohort of experiments. I like how the authors motivated finetuning in section 3.2, and I am glad that the authors have conducted ablative experiments to support their arguments in section 4.4. The experiments on adversarial training are especially interesting, since previous work hasn't considered this straightforward defense against unrestricted adversarial attacks. I am also glad that the authors can generate unrestricted adversarial examples for data as complicated as ImageNet images using the latent technique in GANs. Although still not perfect, some of the unrestricted adversarial examples on ImageNet are surprisingly good to the sense that they may be used as practical attacks.

The writing is great, and it is a pleasure to read this paper. 

I do have some suggestions and questions for further improvement of the paper, and I strongly recommend the authors to address those before publication.

- Section 3 is lacking an explicit form of the combined objective function. Currently some loss functions such as $l_ordinary$, $l_targeted$, $l_d$ and $l_finetune$ are only defined in Figure 1 but not in the main text. It is not clear their explicit mathematical form.

- In section 3.2, it is better to also mention the ablative study you did later in section 4.4. 

- In section 4.1, the authors showed nearest neighbors to some of the unrestricted adversarial examples they generated. It is more convincing to have some quantitive results of that. For example, what is the average minimum distance to training data for a group of 10000 unrestricted adversarial examples? In addition, what is the distance function used in computing nearest neighbors? Did you use Euclidean distance? If so, it would be better to also have results using distances computed in the feature space of a pre-trained convolutional network.

- In section 4.2, the adversarial training was done by alternating two phases of training rounds. I am wondering whether this makes the classifier harder to adapt to the newly generated unrestricted adversarial examples? Can you use some procedure more similar to traditional adversarial training, i.e., the attacker and the classifier are learned together at each step? 

- Song et al. require 100-500 iterations to generate an adversarial example, whereas your approach only need one iteration. Why is your approach 400 to 2000x more efficient? What is the additional reason that speeds up your approach?

- In section 4.5 line 1, the word "replies" was repeated twice.




review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a GAN architecture that generates realistic adversarial
inputs that fool a targeted classifier.  Adversarial inputs are unrestricted:
they may be any realistic images that humans will often classify as real
examples of the intended class, whereas the target model misclassifies them.
The novelty is that they finetune the generator itself during training, the
method can be applied to a variety of GAN architectures, and the method is fast.

Tricks used to successfully train the GAN are clearly described, and the
experimental evaluation was of good scope, covering a good selection of
experiments.  I particularly enjoyed the short Section 4.2 and Fig 7a+b, where
they show that a local defense can always be fooled somewhere else along
the input manifold of that class.

While the modifications to existing solutions may at first seem minor, they
have significant impact in applicability, effectiveness and speed of generating
unrestricted adversarial images.  So I think this paper can be accepted.

I had a bit of avoidable confusion in the introductory sections.  Figure 1
describing the GAN is never referred to.  It includes components not exactly
agreeing with my naive expectations from surrounding text.  Are any Fig. 1 features
optional?  It would help to highlight the novel elements in Fig. 1.  Or does
Fig.1 correspond perhaps to the combined GAN elements in Section 4 ("In our
experiments, we combine three ...").  My uncertainty was really relieved only
by the time I got to Related Work and Appendix E :(

The main claims seemed well supported by experiments, apart from claim 3
(applicability to "any" checkpointed GAN codebase). Might the scope of their
approach also be clarified by clearly identifying required and optional GAN
components in Fig. 1?

---- misc comments ----
Some sentences were long and difficult to parse:
- 4.1: "Our method generates..., else ...."  Perhaps make the else clause a second sentence.
- 4.2: "Image quality as measured ...." length and references made this difficult to read. Can you
rewrite as separate shorter sentences?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes using GANs to generate unrestricted adversarial examples. They seek to generate examples that are adversarial for a specific classifier, and they do so by using class-conditional GANs and a fine-tuning loss. The fine-tuning loss consists of both the ordinary GAN loss (to fool the discriminator) as well as an adversarial loss (which rewards the GAN for generating examples misclassified by the specific classifier). The authors perform various experiments on their generated examples to check for realism and how adversarial the generated images are.

I would reject this paper for two key reasons. First, I feel that the contributions are not significant enough (in comparison to the prior work of Song et. al). Second, I feel that some of the methods (and some of the writing) are not too principled.

In my opinion, unrestricted adversarial examples are significant if they can be made to be realistic. If our current deep learning models often mislabeled very realistic images, that would properly expose a big failure mode of our current models. However, if our machine learning models perform poorly on images that look fake/generated 40% of the time (which is what the authors state) and don’t look too realistic to humans, it is less worrying.

In comparison to Song et. al, the authors state that their methods result in very similar results in terms of realism and how adversarial their images are (arguably, Song et. al actually produces better results in terms of being adversarial). In my opinion, the authors’ claimed improvements are not significant enough, because I think realism should be the primary metric to evaluate this field. Improving speed of generation is nice, and being able to bypass a simple adversarial training procedure is interesting but not significant unless this insight is expanded upon. The results on MNIST in Fig. 5 and Fig. 6 are not too convincing, as simpler attacks that generate (arguably) more realistic images like translations and rotations [1] or L1/L2 attacks [2] (since the networks are trained for L_inf robustness) can also degrade accuracy. Finally, I can also think of another reasonable baseline that I would have liked to see the authors compare their method against. Because the authors want to attack a specific network, they could have (1) generated realistic images using a pre-trained GAN (2) used a norm-bounded attack on the specific classifier and the generated GAN images. These images could be even more realistic if the norm-bound of the attack is fairly small, and would still be able to attack specific classifiers.

Finally, I am confused by the comparison to a not-fine-tuned GAN in Fig. 14/Fig. 15 and would appreciate a clarification so that I can understand the results. For example, what does it mean for intended true label = 9, target label = 0 to have 90% success in Fig. 15? Does this mean that when you try to generate a 9 with the GAN, the classifier misclassifies it as a 0 90% of the time? In particular, I’m struggling to understand what the target label is for the case of the not-fine-tuned GAN.

Secondly, I feel that there are many instances in the paper where the methods used are not explained in a principled way. For example, one of the key parts of this work is the fine-tuning loss function. Why does the loss function involve multiplying the ordinary GAN loss (with some additional transformation applied to it which seems unnecessary) with the adversarial loss? It seems most reasonable add the adversarial loss and the ordinary GAN loss (without the additional transformation). Is the stochastic loss selection procedure necessary? If all these peculiarities of the method are necessary, it seems that the success of this method is quite brittle.

Additional feedback:

- In the intro, I think citing [3] in addition to Xu et. al is more appropriate.
- You should refer to Figure 1 somewhere in the text of your work
- In section 3.2, you can use “cosine similarity” to describe what you are doing faster.
- When you talk about “global optima of realistic adversarial examples” and “local optimal of unrealistic adversarial examples,” it sounds weird. I would try to reword this because I don’t think you are trying to make a precise mathematical statement but it sounds like one when you write it this way.
- In Table 1, I would format the numbers better to be vertically aligned
- You should provide a citation for MixTrain on page 5

[1] https://arxiv.org/abs/1712.02779,
[2] https://arxiv.org/abs/1905.01034
[3] https://arxiv.org/abs/1802.00420
review_title: Official Blind Review #3

Meta-review: This paper presents an interesting method for creating adversarial examples using a GAN.  Reviewers are concerned that ImageNet Results, while successfully evading a classifier, do not appear to be natural images.  Furthermore, the attacks are demonstrated on fairly weak baseline classifiers that are known to be easily broken.  They attack Resnet50 (without adv training), for which Lp-bounded attacks empirically seem to produce more convincing images.  For MNIST, they attack Wong and Kolter’s "certifiable" defense, which is empirically much weaker than an adversarially trained network, and also weaker than more recent certifiable baselines.

Decision: Reject