Adaptive Generation of Unrestricted Adversarial Inputs
Authors: Isaac Dunn
Hadrien Pouget
Tom Melham
Daniel Kroening
Venue:  (2020)
Keywords: Adversarial Examples
Adversarial Robustness
Generative Adversarial Networks
Image Classification

Abstract: Neural networks are vulnerable to adversarially-constructed perturbations of their inputs. Most research so far has considered perturbations of a fixed magnitude under some $l_p$ norm. Although studying these attacks is valuable, there has been increasing interest in the construction of—and robustness to—unrestricted attacks, which are not constrained to a small and rather artificial subset of all possible adversarial inputs. We introduce a novel algorithm for generating such unrestricted adversarial inputs which, unlike prior work, is adaptive: it is able to tune its attacks to the classifier being targeted. It also offers a 400–2,000× speedup over the existing state of the art. We demonstrate our approach by generating unrestricted adversarial inputs that fool classifiers robust to perturbation-based attacks. We also show that, by virtue of being adaptive and unrestricted, our attack is able to bypass adversarial training against it.

Under review as conference paper at ICLR 2020
ADAPTIVE GENERATION OF UNRESTRICTED ADVERSARIAL INPUTS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks are vulnerable to deveesariillyyconsttee inputs, Most rescarch so far has considered perturbations of fixed magnitude under some I, norm. Although studying these attacks is valuable, there has been increasing interest in the construction of and robustness to -unrestrictdd attacks, which are not constrained to small and rather artificial subset of all possible adversarial inputs. We introduce novel algorithm for generating such unrestricted adversarial inputs which, unlike prior work, is adaptive: it is able to tune its attacks to the classifier being targeted. also offers 200002 speedup over the existing state of the art. We demonstrate our approach by generating unrestricted ad adversarial inputs that fool classifiers robust to perturbation- based attacks. We also show that, by virtue of being adaptive and unrestricted, our attack is able to defeat adversarial training against it.
perturbations of their
INTRODUCTION
Despite their dramatic successes in other respects, neural networks are well-known to not be adver- sarially robust. Szegedy al. (2014) discovered that neural networks are vulnerable what they termed adversarial inputs: by adding carefully-chosen perturbations to corrccly-classified inputs, the accuracy of any neural network could be almost arbitrarily decreased. Since then, the machine learning community has rightly focused great deal of research effort on this phenomenon. Many early efforts to train more robust models initially appeared promising, but have since been shown to be vulnerable to new algorithms for constructing adversarial perturbations (Xu et al., 2019; Athalye al., 2018). As re- sult, more attention has been given to methods that provide formal guarantees about performance in the presence of adversarial perturbations (Liu et al., 2019), with the state of the art now providing non-tri trivial guarantees for the MNIST test set (Wong& & Kolter, 2018; Croce et al., 2018; Wang et al., 2018).
However, almost all of this work has focused exclusively adversarial perturbations whose magnitude constrained by an l, norm. There is a growing acknowledgement that this threat model is somewhat contrived: such examples are not realistic security concern and also occupy vanishingly small fraction of the set of potential adversarial inputs. Therefore, there is burgeoning interest in adversarial attacks that are unrestricted, in the sense that they do not necessarily derive from a perturbation of a natural input (Brown et al., 2018; Song et al., 2018b).
The main contribution of this paper is novel and general method to generate unrestricted adversarial inputs. In short, the training procedure for generative adversarial networks (GANs) is modified so that the generator network rewarded for producing data that are both realistic and deceive fixed target network. Our approach has four advantages over prior work:
1. Our method is adaptive in that it adjusts itself to best attack the specific network being targeted. For instance, adversarial training is ineffective against our approach.
2. Our method is efficient (offering a 400-2000x speedup over prior work).
3. Our method can easily be applied any existing conditional GAN codebase and checkpoints, regardless of architecture, training procedure, or application domain.
4. Our method therefore demonstrably scales to ImageNet.

---

Under review as conference paper at ICLR 2020
2 BACKGROUND: GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are a class of generative machine learning models involving the simultaneous training of two neural networks: generator g and discriminator d. Specifically, given a dataset D of samples drawn from a probability distribution Pp. the generator g learns to transform random noise drawn from a simple distribution Pa into an approximation of pp- The discriminator network d learns to predict whether a given example æ is drawn from the data distribution pD or was generated by g. The generator and the discriminator are adversarial because they train simultaneously, with each being rewarded for out out-performing the other.
GANs' training behaviours are notoriously temperamental, and many modifications the original al- gorithm have been proposed (Goodfellow, 2017). The Wasserstein GAN variant (Arjovsky et al., 2017) aims to provide more reliable gradient by designing the discriminator (renamed 'critic") to approxi- mate the Wasserstein distance between the distribution generated by go and the data distribution pp. An additional 'gradient penalty' loss term a can be added to implement the constraint that the function be I-Lipschitz continuous (Gulrajani et al., 2017). The loss functions for this Wasserstein GAN with gradient penalty (WGAN-GP) are: = E, Ez (.ggt-
and La= -Lg + Ez~po [-d(x)]+ gp
where the gradient penalty Lg = Ez-p, [d sampling uniformly from the linear interpolations between generated samples and examples from PD-
1)²]. where p1 denotes the distribution
The original proposal for a conditional generative adversarial network (CGAN) learns to generate samples from conditional distribution (Mirza & Osindero. 2014) 4) by simply passing the intended label y for the generated image to both the generator and the discriminator. An extension of this approach is the auxiliary classifier generative adversarial network (ACGAN) (Odena et al., 2017), in which the discriminator is modified to also predict the label y for the input data. Both the generators are trained to maximise the log-likelln likelihood of the correct label in addition to optimising their usual objective.
3 GENERATING UNRESTRICTED ADVERSARIAL INPUTS
Suppose we have a trained target classifier network f: X + R that attempts to approximate an oracle function o: O-YY (where O _ X the oracle's 's domain) by outputting a confidence f()) E R for each class e E Y. As Song et al. (201 8b) do, we define an unrestricted adversarial example to be any input 2 E O such that the classifier's 's prediction is incorrect: argmax, (xr) # o(x). Unlike Song et al., we consider the domain of the oracle to be any input with a recognisable class, not just realistic inputs. Gilmer et al. (2018) offer a number motivations for why unrealistic but recongisable adversarial examples could be interesting, including security concerns and improving models" abilities to generalize. Nevertheless, we do carefully evaluate how realistic our results are in Section 4.1.
Unrestricted adversarial examples are superset of conventional perturbation based adversarial examples (which are restricted to lie within a fixed distance of some correctly-cassified input from test dataset). While providing vastly larger space of candidates, difficulty arises in determining that the classification is incorrect; we can no longer rely on the oracle pracle-provided labels from the test dataset. We leverage generative models solve this problem.
3.1 OUR PROCEDURE
We begin by taking any conditional GAN, with generator loss londinary; the use of a conditional GAN allows us to determine the correct label y of our generated unrestricted adversarial examples. We then introduce loss terms which incentivise the generator g to create adversarial examples for classifier f. A targeted attack for desired true label y and target label t y should output an image which humans would regard to have label y yet is classified as / by the classifier. We introduce loss term, Iuurgendd which is minimised when the conditional generator output g(2,y) is classified in this way: harggred maxe of (g(2)))) - (g((2)))) For an untargeted attack, we use a loss term that is minimised for any misclassification of the example: Luntargeted = f (g(2z)) ),, maxe If (g(2,y) (.)) ee Note that these new terms assume that the true labels of the generated data g( g(z.y) do indeed match the intended labels y, an assumption empirically validated in Section 4.1.
Our procedure is to alter the generator's 's training objective simultancously. thereby training the generator to output data which are both realistic and also fool the target network. Section 3.3 provides further details.
to minimise both londinary and 4 Itrgette
2

---

Under review as conference paper at ICLR 2020
0.1
Nomallood Projctiion
-0.
Norel medd Projection
1500
-0.4
0.2
-03
0.6
-044
0.7
0.8
4.6
4.7
488
Cradient Stop
Firetuning step
Beginning from randomly- initialised GAN.
(b) Adversarially finetuning pretrained GAN,
Figure 1: Projecting normalised gradient vectors from londinary and (an)targeted onto one another.
3.2 CHALLENGE: CONFLICTING GRADIENTS
Intuition suggests that the gradient from lontnnryy may be pointing in different direction to the gradient from (unnttrrenedd since making an image adversarial seems likely to make it less realistic, not more. A simple experiment suffices verify this intuition. We compute the cosine similarity between the gradients of our two loss terms each step, i.c.
[Vlndde Vinusoyeelll Figure la shows that this projection tends towards -1; for reference, if the gradient vectors were selected uniformly at random, the magnitude of this projection would very rarely exceed 0.001. In other words, training progresses, the gradients from these terms tend towards pointing in opposite directions. This makes joint optimisation using gradient descent approach challenging.
Vlusioon Vlusiton-Viiiiinn Vianiionend
3.3 STRATEGIES To OVERCOME TRAINING CHALLENGES
We empirically evaluate the effect of each technique described below in our ablative experiments reported in Section 4.4.
Realistic pretraining is widely accepted that real image data occupy relatively low-dimensional and contiguous manifold (Goodfellow et al., 2016, p. 160). Conversely, we know that adversarial examples pervade the full input space: it appears that there is adversarial example nearby nearly any point in the input space. Therefore, a generator that is pretrained using only londinary before adversarially finetuning by introducing loss terms from a random initialisation. By beginning our search in regions realistic examples, we'n more likely to find adversarial examples that sufficiently realistic. Besides the generated images being subjectively better, Figure lb shows that the gradients conflict to a much lesser extent. Note that any existing conditional GAN architecture, pretrained checkpoint and training algorithm could be used here, allowing our method to leverage the significant advances being made in this area.
additional loss term is more successful than using both
Amalgamation loss terms following per-example loss term:
Rather than naively summing lontiary and luuntrrgettd we use the
Innenune = 8( (londinary) ( ks) where s(1) [+exp(!) ifl O.
2+( otherwise.
Here, hyperparameter similar that in the Carlini & Wagner (2017) attack: it controls the confidence of the generated adversarial examples. If the difference between the desired logit and the next- eext-greatest logit is less than K, the generator is linearly rewarded for improving this gap (gaining confidence); beyond a difference of (once example is 'good enough") )) the reward exponentially decreases. We use x: == for our experiments as we not require strong misclassifications.
The gradients from the two loss terms are in conflict, and in practice the lumnargeted gradient dominates. The proportion of misclassified generated inputs rises quickly to almost 100%, but the generated images were noticeably unrealistic, meaning their correct label may change. To address this, we introduce the 'attack rate" μ.. During adversarial finetuning, the finetuning loss term is used at each step only with probability μæ; with probability 1 -4, the pretraining loss Condinory
Stochastic loss selection
3

---

Under review as conference paper at ICLR 2020
Target label
0
2
4
5
7
8
9
None
0
0
O
0
U
O Y 2
O
O
/
/
0 2 3 4
Z
2
3
2
.2 9 4 9 6
2
2
Intended true label
3
+
3
3
3
3
3
4
4
4.
4
4
5
6
S
5
5 V
L
"Lorzea'
6
6
6
6
h
6
C
6
6
7
7
1
7
7 7
7
7 8
7 7
7
"Ohrrbe
8
2
8
8
6 7
8
9
7
9
9
9
9
?
9
Figure 3: Selected successful targeted unre- stricted adversarial examples on ImageNet, gener- ated by BigGAN (Brock et al., 2019) finetuned to attack ResNet-152 (He et al., 2016).
Figure 2: Randomly-sclectee images generated by a GAN finetuned to attack Wong & Kolter's (2018) classifier, which is robust perturbations.
only) is used. As desired, this new hyperparameter allows the success rate of the generated unrestricted adversarial examples be traded off with their realism.
EXPERIMENTAL EVALUATION
Our method aims to generate unrestricted adversarial inputs in way that adapts the targeted classifier. We therefore conducted experiments to check whether the generated examples were in fact unrestricted, adversarial, and adapted to the classifier. We then address some questions regarding the performance and generality of our approach, including realism.
The MNIST dataset (LeCun et al., 1998) is the main focus of the experimental evaluation, because this is the most challenging domain for the generation of realistic adversarial inputs. State-of-the-art classifiers perform very well, with around 0.2% test error (Kowsari et al., 2018; Wan et al., 2013), In particular, attempts to create robust classifiers have also been most successful this dataset, perhaps due to its simplicity (Shafahi al., 2018). We target five pretrained classifiers provably robust to adversarial perturbations: there is guaranteed to be no adversarial input within distance of p% of test inputs under the 2o norm. All five are the current state-of-the art in this domain, trained by Wong & Kolter (2018), and Wang et al. (2018). See Appendix C for details.
In our experiments, we combine three well-established generator architectures: a Wasserstein GAN with gradient penalty (WGAN-GP) (Gulrajani et al., 2017), conditional GAN (Mirza & Osindero, 2014) and an auxiliary classifier GAN (Odena et al., 2017). Full details are given in Appendix E.
Once GAN adversarially finetuned for each of the 10 target labels, and for the untargeted trained, the generators were used to produce examples for all intended true labels, which were then filtered that the classifier label matched the target. Images were generated until 200 filtered examples were generated or until 100 seconds had elapsed. Interestingly, this led to no adversarial examples with intended true label '0' and target classification 'I', this case is omitted. Figure 2 and Appendix B give examples of generated images for which the computed label matches the target classification.
4.1 EFFICACY OF ATTACKS
We claim that our method generates unrestricted adversarial examples, which are somewhat realistic. We empirically verify each claim in turn.
Since our method does not work by perturbing existing data, only a simple sanity check was required to verify that the generated images are not close images in the training set, could be caused by over- fittingg.... We selected ten generated inputs that are visually similar to the training set, and computed the shortest distances between the images and all images in the training The selected images
4

---

Under review as conference paper at ICLR 2020
Table 1: Comparison of typical perturbation magnitudes from the literature and ours.
Table 2: Ten selected unrestricted adver- sarial inputs used for Table 1.
Metric
Nearest neighbour
Typical perturbation magnitude
2
3
4
seen
lo
508
<40 (Ruan et al., 2018)
5
6
7
8
9
l
22.8
<5 (Lu et al., 2018)
l2
3.28
-1.5 (Schott et al., 2018)
1oo
0.838
-0.1 (Wong & Kolter, 2018)
Target label
Target label
0
1
2
3
4
5
6
7
8
9
None
0
1
2
3
4
5
6
7
8
9
None
96
94
90
85
96
97
99
85
89
95
0
40
60
56
34
46
51
40
36
63
51
66
88
69
97
89
74
91
81
87
1
37
52
36
51
81
40
53
35
49
Intended true label
Intended true label
2
69
89
82
58
82
70
64
79
49
75
2
30
37
43
40
42
35
37
55
32
54
3
43
84
81
68
74
46
82
54
71
53
3
39
39
43
34
40
40
42
45
48
40
84
67
86
74
75
96
79
82
77
76
4
51
50
34
38
37
46
42
41
43
40
5
58
75
70
78
79
52
82
69
81
75
5
32
34
32
36
43
42
36
37
55
51
82
90
95
73
84
84
86
94
84
82
6
51
39
45
36
57
46
57
40
46
75
75
88
82
76
95
88
92
59
80
7
47
48
53
33
42
58
41
52
44
39
8
76
85
91
76
98
97
77
75
91
83
8
29
46
47
55
44
48
36
39
42
60
9
77
68
90
84
95
92
88
95
95
90
9
38
34
50
49
54
53
53
69
57
67
Mean
70
81
85
81
79
88
78
82
82
76
80
Mean
40
41
45
44
43
47
47
43
48
45
50
Figure 4: The success rates of the adversarial attacks by finetuned GANs (the computed label matches the target label and the true label remains the same).
Figure 5: How often adversarial images are not identified as being generated. If the generated images were completely realistic, the expected result would be 90.
given in Table 2. Table shows that they are much further from any training example than would be the case with a perturbation-basel attack.
Next, we evaluate the whether our method is successful in generating adversarial inputs. We need to check that the true label matches the intended true label for each example, or the generator could simply be producing images that visually match the target class. To check this we had Amazon's s MTurk workers classify the generated images. For cost reasons, Kolter's 's provably-robus network (Wong& & Kolter, 2018). We used a sample size of 100 judges for each intended true label/target label pair for each experiment. Figure 4 shows the proportion of inputs for which not only does the label computed by the classifier match the target label, but the human-judged true label matches the intended true label specified the generator. The mean number of correct labels for the untargeted attack is 80%. This can be considered to be the success rate of our attack.
only did this targeting Wong and
We now investigate if the generated examples are realistic. A set of inputs is realistic with respect to dataset if a human cannot reliably identify to which set an example belongs. To check this, we again used MTurk workers. After familiarising themselves with examples from the training dataset, each worker had to pick which image out of ten was most likely to have been generated. Figure 5 shows the proportion of the time that generated images were not identified as such.
For comparison, we repeated these experiments but attacking a non-robust classifier network. The untargeted success rate was 90% (vs. 80% against the robust classifier), and 60% (vs. 50%) were not identified as being generated. Similar differences were seen in targeted attacks; see Appendix I.
4.2 ADAPTIVITY TO ADVERSARIAL TRAINING DEFENCES
In our experiments far we have evaluated our method against pretrained classifiers that are provably robust to adversarial perturbations. We now investigate whether standard adversarial training (Madry al., 2018) against attack in particular is effective. As method is adaptive, it is not immediately
5

---

Under review as conference paper at ICLR 2020
3.2
10
6,4
0.4-
0.4
08
100
253
Traiting Round
Thakiing Tound
Traiking stop
(a) Adversarial training against our attack, first few training rounds.
(b) Adversarial training against our attack, later training rounds.
(c) Adversarial training against Song al. (2018b).
2.0
AlackS.ccm Pale
0.8
6.2
6.2
5.0
200000
400000
000002
810000
1000002
1200000
1400000
2600036
6.0
6000
8200
:2000
16000
(d) Online adversarial training performance.
(e) Attack performance after online training.
Figure 6: Plots showing attack efficacy in the presence of adversarial training.
clear how to integrate method into the adversarial training framework. The two obvious possibilities are finetuning the generator at each step of training, or alternately training the classifier and finetuning the generator. We explore these below 'online' and 'offline" adversarial training, respectively.
Online Adversarial Training We update both the generator and classifier each training step. We run adversarial training for over 1.6 million training steps, and Figure 6d shows that even during training we maintain roughly 80% attack success rate. Once we have finished training the classifier, we attack it normally using our method. Despite having been adversarially trained, we are still able to fool the classifier >99% of the time within 16,000 steps, shown in Figure 6e.
Offline Adversarial Training Starting with pretrained GAN and classifier, iterate 'training rounds' consisting of two phases. First, GAN is adversarially finetuned (starting from the pretrained GAN each time) for fixed period to attack the classifier. Second, 80,000 generated unrestricted adversarial examples are added to the existing training dataset, and the classifier continues training until almost 100% accuracy is achieved. Figure 6a shows that, for the first few training rounds, adversarial finetuning is successful: the proportion of examples generated which fool the classifier increases over 80%. Figure 6b shows the same story 30 rounds (and hence hundreds of thousands of classifier gradient steps) in.
We find that the classifier able to defend against the kinds attacks previously produced by the genera- tor. However, the generator's opportunity to udversarially finetune again allows it to generate ad adversarial examples in a new 'blindspot' of the classifier. For more details on these experiments, see Appendix
4.3 SCALING TO IMAGENET
To demonstrate the scalability of our method, we apply it to the notoriously large and complex ImageNet-I1 KK dataset, using the author's 'officially unofficial' published code and checkpoints for the current state-of-the-arr BigGAN (Brock al., 2019). In the untargeted case, our method is able to finetune this BigGAN to fool the classifier >99% of the time within 40 gradient steps (compared to the 10 taken to train from scratch). Our main focus, though, is on the much more challenging targeted attack. We found that typically, the order of 100 gradient steps were required for > 10% of generated examples to be classified (top- 1) as the target class. Compared to MNIST, each ImageNet gradient step takes about 100x longer compute, but the 100x decrease in the number of gradient

---

Under review as conference paper at ICLR 2020
steps required compensates for this, resulting in a similar compute time overall. Image quality as measured by Inception Score (Salimans et al., 2016) typically decreased from 70. This is slightly better than mid-2018 ofate-of-the-art of 52 (Zhang et al., 2019) or the mid-2017 state-of- ti-thhe-attt of 12 using WGAN-GP (Shmelkov et al., 2018) 8).. We speculate that the GAN were finetuned for significantly longer, the gradient from the discriminator would learn regain some of this lost realism. Figure shows selected samples of generated adversarial examples; Appendix A has a more extensive collection.
4.4 ABLATIVE STUDIES AND BASELINE
No Adversarial Finetuning We evaluate the extent to which adversarial finetuning is effective by comparing a GAN which has not been finetuned. As expected, this is much less successful at producing adversarial examples. The proportion of its outputs which are misclassified matches that of the test set (around 2% for MNIST), whereas adversarial finetuning can easily increase this proportion to well over 99%. Furthermore, only 66% of its misclassified outputs had maintained their true class, as opposed to 80% for an adversarially finetuned GAN. Further results can be found Appendix K.
Naïve Adversarial Finetuning We investigate the effect of the training strategies described in Section 3.3 by removing each in turn. We find that without pretraining, our method does eventually converge, but with much less realistic results. also find that the attack rate is an effective technique for trading off adversarial success rate with image realism; without this technique, the attack rate is in effect set 1, which again produces less realistic results. Note that unrealistic images affect the attack's success rate because the true class of the images is more likely to change. Lastly, we find that choosing loudinary +
luunkargeeedd as the generator's s loss results in catastrophic collapse of the training, likely because /fanjtargeseed is optimised for too heavily. Full details can be found in Appendix L.
Naïve Baseline Finally we compare our method to baseline where data is generated using an ordinary GAN and then adversarially perturbed using projected gradient descent with norm bound = under 4 With success rate of under 1% on data which were originally classified correctly, we find this approach is no better than perturbations on the test set; this is as expected, since the generator learns the data distribution.
4.5 THREATS TO VALIDITY
The evaluation of the success of the attacks relies on data provided by the MTuur workers. We therefore employed measures to safeguard the quality of this data, described in Appendix H. We also believe that our method will generalise to any dataset and domain for which GANs can be trained successfully, However, this has only been demonstrated on two image classification tasks (albeit dissimilar in nature). Lastly, intuition suggests that our method will be able adapt to find unrestricted adversarial examples for most fixed defence methods, since it is so free to generate inputs without the constraints that current defence methods rely upon. However, have only demonstrated it explicitly for the most popular standard defence; we leave it to future work to find defence against our adaptive approach.
5 RELATED WORK
5.1 COMPARISON TO THE STATE OF THE ART
We compare our method to that of Song et al. (2018b), the current state of the art in generating unrestricted adversarial examples. Like ours, this method leverages a pretrained GAN. differs. however, in how adversarial examples are then produced. Instead of adversarially finetuning the generator, searches for an input to the generator that both deceives the target network and are confidently correctly classified by the discriminator's 'ss auxiliary classifier (an ACGAN (Odena et al., 2017) is required in this case). The GAN training is therefore blind to the target network.
Our model achieves similar success rates in generating unrestricted adversarial examples: our success rate of 80% (cf. Section 4.1) 1) is roughly comparable to that of Song et al. (2018b), 88.8%. For comparison, repeated the realism experiments from Section 4.1, with the difference that judges were asked to identify the one generated image from a choice of two. In this case, Song et al. report that participants select the generated image as the more realistic 21.8% of the time while for untargeted
7

---

Under review as conference paper at ICLR 2020
attack, this figure is 24%; completely realistic image would be chosen 50% of the time. Full results are given in Appendix J.
Beyond achieving comparable attack success rates, our approach has four significant advantages over prior work. Firstly: adaptivity. In Section 4.2 we have shown that our model is capable of iteratively adapting to adversarially trained classifier. By contrast, Song et al." 's method performs poorly against adversarial training because the GAN is not trained with respect to a target classifier, remaining fixed after the attack begins. Therefore, if a classifier learns be correct in the space their algorithm searches, it will no longer be able to generate images different enough to be adversarial. Figure 6c shows that standard adversarial training quickly and effectively defends against Song et al. 's attack, while fails against ours. Secondly: efficiency. Once trained, our method requires only single forward pass to generate adversarial examples. Song et al. require 100-500 iterations, each with 4 passes: forward and backward through both the generator and classifier. Our method is therefore 400-2,000 more efficient. Lastly: scale and versatility. Section 4.3 shows that our model scales to ImageNet, a dataset with dimensionality 16x greater than the largest Song et al. demonstrate on. Our method has the further benefit that we can use any pretrained conditional GAN, such as BigGAN (Brock et al., 2019). Song et al. depend on an auxiliary classifier for larger datasets, which BigGAN does not provide.
5.2 OTHER RELATED WORK
Wang et al. (2019) independently propose a method which is superficially similar to ours: they also train a GAN to directly generate adversarial examples. Instead of using the ordinary GAN loss to maintain realism, they use new loss term. This term, 11gpretrained - g( llpp penalises the generator g given input z proportional to the deviation caused by finetuning from the original output. Wang et al.' 's choice of loss term has the unfortunate effect of preventing the generator from generating either unrestricted adversarial examples or examples which are sure to fall within an 1p-norm ball of realistic input. By contrast approach allows for truly unrestricted and adaptive examples. Furthermore: we evaluate against state-of-the art provably-robut networks rather than ad-hoc classifiers; we do not assume that all the true labels remain the same (which is unlikely), and conduct user study to test this; and demonstrate that our approach scales beyond MNIST (to ImageNet).
Sharif et al. (2019) train network to generate patterned spectacles, which, when added to an image of a face, cause misclassification. They also adapt this approach to MNIST using an approach quite similar ours. However, this only achieves a success rate of 0.83% after filtering to "only the digits that where likely to be comprehensible by humans" against a classifier which was state-of-the of-the-at art in 2017. In contrast, we achieve around 80% accuracy against current state-of-the-art robust classifiers.
A wide range of work trains networks to generate adversarial perturbations (Hayes & Danezis, 2018; Baluja & Fischer, 2018; Xiao et al., 2018; Song et al., 201 8a; Poursaeed al., 2018). While these must also balance realism and adversarial success, the key difference is that we generate unrestricted adversarial examples, allowing attacks to succeed when constrained perturbations provably fail.
Hu et al. (2019) introduce search for pairs of nearby unrestricted adversarial examples, but unfor- tunately cannot ensure that their true label meaningful; if the search starting point random, it is overwhelmingly likely not to be. If instead it is a known input, the examples are not unrestricted.
6 CONCLUSION
We have introduced an algorithm which trains a GAN to generate unrestricted adversarial inputs; we demonstrate that these, as expected, are successful against state-of-the-art classifiers robust to perturbation attacks. The key novelty in our attack procedure is that it entails the tuning of the weights of the generator to target a specific network. As a result, it can be considered adaptive: we have shown that, while prior work is quickly mitigated by standard adversarial training, our attack adapts to find new way of fooling the classifier. In addition, once the generator is adversarially finetuned, it becomes an endless supply of cheap adversarial examples: generation of adversarial examples requires single forward pass rather than execution of any optimisation algorithm, resulting in a 400-2000x speedup over the state of the art. We have also demonstrated that any existing GAN codebase can easily be used by adapting BigGAN to generate unrestricted adversarial examples for ImageNet.
8

---

Under review as conference paper at ICLR 2020
REFERENCES
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Tch (eds.), International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 214-223. PMLR, 2017. URL
mml.. pwwssppees///e.e / v70/ attp//rjoovsyyyaa...
Anish Athalye, Nicholas Carlini, and David A Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Jennifer G Dy and Andreas Krause (eds.), International Conference on Machine Learning (ICML), volume 80, pp. 274-283, 2018. hamm..
URL http: pttt:://proceediiie httpp/ppoceednnn...eeee mlr wwwppresssseeee v80 aareas/v80/athalyeeee
Shumeet Baluja and lan Fischer. Learning to attack: Adversarial transformation networks. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), AAAI Conference Artificcil Intelligence, pp.
2687-2695. AAAI Press, 2018, URL https: www. aaai. org/
org/oca// /cs//////////// windexx.x...... wnndx.ppp/aae.. AAAI/
AAAI18 / papeer/pppp//////ee /vvivvieeee ew 6529.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: http:s://epenreveww net wtt.://rrmmiiiie -oee:/ooouu
Tom B Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Francis Christiano, and Ian Goodfellow. Unrestricted adversarial examples. CoRR, abs/ 1809.0, 2018. URL http:
org//org///... wwtpaabbs./111./// 08352.....
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP. pp. 39-57. IEEE Computer Society, 2017. ISBN 978-1-5080 1-500-553333. doi: 10.1109/SP2017.99. URL
https:://oi.... http//oo..org//////// .wrg/orggooo wwt.or///......///
2017.49.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of ReLU networks via maximization of linear regions. CoRR, abs/1810.07481, 2018. URL
http://arxiv> www.lv...rg/bbbse.ee
07481.
Justin Gilmer, Ryan P Adams, Ian J Goodfellow, David Andersen, and George E Dahl. Motivating the rules of the game for adversarial example research. CoRR, abs/1807.0, 2018. URL http:://rae
lan J Goodfellow, NIPS 2016 tutorial: Generative adversarial networks. CoRR, abs/1701.0. 2017.
htt:tp///aaxxi org/ httpabb///11 001 60.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D Lawrence, and Kilian Q Weinberger (eds.), Advances Neural Information Processing Systems (NeurIPS), pp. 2672-2680, 2014. URL http::/ nips. wwtsscccppppee ccc/paper/5433-gnee. vaweerrraaiee-vvvveririiaaaaaa
lan J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. Deep Learning. Adaptive computation and machine learning. MIT Press, 2016. ISBN 978-0-262-03561-3, URL
org/.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M Wallach, Rob Fergus, V Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 5769-57799 2017. URL http:/ /tppppppp.....//////////////.e.. httpssccpppppeee115555 wwwe-iimrrvvereeee traeeddddrriinnngg of www.ooooowwwwwwnnnnoooose gans..
Jamie Hayes and George Danezis. Learning universal adversarial perturbations with generative models. In 2018 IEEE Security and Privacy Workshops, pp. 43-49. IEEE Computer Society, 2018. doi: 10.1 1 109/SPW.201 Wwt.S/.....0000000 000155 URL htR.sttss:
1109/ SPW 201 8. 00015.

---

Under review as conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034. IEEE Computer Society, 2015. ISBN 978-1-4673-8399 doi: 10.1 1091ICCV.215. 123. URL wwt.tttp/ttt.....mm /doi org/ 1109/0011110 TCVV 201 5 123.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 N0.100//CP..2016.. URL https: doii org/1o0r10000 1109/ CVt///CPRR222 2016...221.... 90.
Hanbin Hu, Mit Shah, Jianhua Z. Huang, and Peng Li. Global adversarial at- tacks for assessing deep learning robustness. http://arxiv. wttp.v.oog//b///105....
CoRR, abs/1906.077200 2019. URL
Kamran Kowsari, Mojtaba Heidarysafa, Donald E. Brown, Kiana Jafari Meimandi, and Laura E.
Barnes. RMDL: random multimodel deep learning for classification. CoRR, 0bs//105500890 2018.
Yann LeCun, Corinna Cortes, and Chris Burges. MNIST handwritten digit database, 1998. URL http: / /yann. lecun. com/ hoom/exb////// /mnnst/
Changliu Liu, Tomer Amon, Christopher Lazarus, Clark Barrett, and Mykel J. Kochender- fer. Algorithms for verifying deep neural networks. CoRR, abs/1903.06753. 2019. URL
http:/// http:://axxvv.ooooee abs www.aa/s/ss.. 1903, 06758.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. International Conference on Learning Representations (ICLR), 2017.
Pei-Hsuan Lu, Pin-Yu Chen, Kang-Cheng Chen, and Chia-Mu Yu. tion of MagNet defense against T--based adversarial examples. In IEEF/FFP In- ternational Conference on Dependable Systems and Networks Workshops (DSN), pp. 200-214, IEEE Computer Society, 2018. doi: 10.1 10.1109/DSN- WWW.W/DSN-WW21..... W.2018.00065. http://det ieeecompteeroocetyy wwwpieett.orr/////e helgg//..1119//// 1109/DSN W. 0..208.00066
On the limita-
URL
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. CoRR, abs/1706.06083, 2017. URL horg/abs http://arxix: org/ abs 1tbbs/170 /0706.7.000 06083.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Interna- fional Conference Learning Representations (ICLR). OpenReview.net, 2018. URL ht https:///ppeneviale net http//ommiddrruuaaaa.aa
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, abs/1411.1, 2014. URL
1784.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In Doina Precup and Yee Whye Teh (eds.), International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 2642-2651. PMLR, 2017. URL http::///// ptwc/eooceeeeeeee .... htrroeeeeegg..mrpprrsss////// odena 7a html.
press//w0
Omid Poursaeed, Isay Katsman, Bicheng Gao, and Serge J. Belongie. Generative Computer Vision and Pat-
adversarial perturbations. In 2018 IEEE Conference tern Recognition, CVPR 2018, Salt Lake City. UT, USA, June 18-22, 2018, pp. 4422 4431. IEEE Computer Society, 2018. doi: 10.1109/CVPR.22018.002 http:://openaaccsss.teeee htt.://cceeeettteeee.oooooooottt http:/0////tmmm/ouuueseeee aenerreeiveeaaeeerrraaaaaaattaar
URL
1a100118ppppprrteee html.
Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neural networks with provable guarantees for LO norm. CoRR, abs/1 01111804.0550 05805, 2018. URL http: //wxiiiiitt/ rxive org abs 8804....... 05805.
10

---

Under review as conference paper at ICLR 2020
Tim Salimans, Ian J Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Daniel D Lee, Masashi Sugiyama, Ulike von Luxburg, Isabelle Guyon, and Roman Garnett Advanees in Neural Information Processing Systems (NeurlPS), pp. 2226-2234, 2016. URL http:/ /ttp://aappps..e n mips. cc/ paper/ 1125-imgroved--ecchnigg. forees-----ttriinnn----e-
Lukas Schott, Jonas Rauber, Wieland Brendel, and Matthias Bethge. Towards the first ad- versarially robust neural network model on MNIST. CoRR, abs/1805.091990 2018. URL
http:// www.oorgg////....o..oo....o...ae http/abs//8800 0t.bs//885.0000 .991900
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein.
Are adversarial examples inevitable?
CoRR, abs/1800.02104, 2018. URL http:
0214.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. A general framework for adversarial examples with objectives. ACM Trans. Priv. Secur, 22(3):16:1-16:: 10.1145/3317611. URL https:/ /doi wrg/ 11111.33311166611
2019. doi:
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part 1I, volume 11206 of Lecture Notes in Computer Science, pp. 218-234. doi: http:::/ddo..gggg/g 10t.111000970000 8-117778-000000-1----------1--1- 14.
Springer, 2018. ISBN 978- 118-1-030-01211-1
URL
Qing Song, Yingqi Wu, and Lu Yang. Attacks on state-of-the-art face recognition using attentional adversarial attack generative network. CoRR, abs/1811. 2018a. URL
http://arxiv. www.oorgg//bbbbbeeeee
/bs.1111..
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnet (eds.), Advances in Neural Infor- mation Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada., pp. 8322 -333, 2018b. URL https://arret org/
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), 2014. URL http:://rxtii
abs/1122 6199.
Li Wan, Matthew D. Zeiler, Sixin Zhang. Yann LeCun, and Rob Fergus. Regularization of neural networks using dropeonnect. In International Conference on Machine Learning (ICML), volume 28 of Proceedings of Machine Learning Research, pp. 1058-1066. PMLR, 2013. URL http: http:///poceenneeeeeeeeee mlr....................... www.s/ressseeeeee waal3............/...... html.
Shiqi Wang, Yizheng Chen, Ahmed Abdou, and Suman Jana. Mixtrain: Scalable training of formally robust neural networks. CoRR, abs/181 11100225, 2018.
Xiaosen Wang, Kun He, and John E. Hoperoft. AT-GAN: generative attack model for ad- versarial transferring generative adversarial nets. CoRR, abs/1904.07793, 2019. URL
http://arxiv. org/ htg/abs/////eo 01///999.07777
Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the con- vex outer adversarial polytope. In Jennifer G Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning (ICML), volume 80 of JMLR Workshop and Conference Proceedings, pp. 5283-5292. JMLR.org, 2018. URL http: http://poceeeeegeeeeee//// mlr. www.mmmws.....tttmt..m..m.. ////80//wwoo http///ollll....
Yuxin Wu and Kaiming He. Group normalization. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14 2018, Proceedings, Part XIII, volume 11217 of Lecture Notes in Computer Science, pp. 3-19. Springer, 2018. doi: 10. https: /oi. .org/ 10. 1007 1007 978- -30- 0120001666111
URL

---

Under review as conference paper at ICLR 2020
Chaowei Xiao, Bo Bi, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. In Jérôme Lang (ed.), International Joint Conferences on Artificial Intelligence (IJCAI), pp. 3905-3911. ijcaiiorg, 2018. ISBN 978-0-9992411-222 doi: jca i 0010/15433
10.24963/jjci..211 20t 8/543. URL https: /ttas:///o/s /doi.. org/ 0 24963/
Han Xu, Yao Ma, Haochen Liu, Debayan Deb, Hui Liu, Jiliang Tang, and Anil Jain. Adversarial attacks and defenses in images, graphs and text: review. CoRR, abs/1909.08072, 2019. URL
https:// wwwia. org/ wttp/////////ooo/eo/////eeeeee// attp///ss/1990 1909. 08072.
Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention generative adversarial networks. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363, PMLR, 2019. http: / /tt://proooc proceed i ngs. ml wrwpsreesvvveeee zhangl 9d, html.
12

---

Under review conference paper at ICLR 2020
A SAMPLES OF IMAGENET UNRESTRICTED ADVERSARIAL EXAMPLES
Randomly-selected successful targeted unrestricted adversarial examples generated using adversarially finetuned BigGANs (Brock et al., 2019). The targeted classifier is ResNet- 152 (He et al., 2016), the highest-accuracy pretrained classifier packaged with PyTorch. Besides setting attack rate at 0.1, all configuration and hyperparameters are as described in the BigGAN 'officially unofficial' codebase.
homed ratriesnuke
corly-coated retriever
admiral
bendhouse
kile
spotted sa salmmnndrr
terriiin
alligtton liand
chest
born
lifeboat
solar cisc
green
night
contipade
thresher
unieycle
lady
howler
combinaation lock
camic book
butteruut aquass
candion
hdy
sombreeo
curn
capitulum
Figure 8: Successful targeted unrestricted adver- sarial examples for target class 'slug'.
Figure Successful targeted unrestricted adver- sarial examples for target class 'tabby cat'.
cook
black widow
nautillus
bittern
cock
dhole
squirell monkey
ballooe
bluetick
engllish setier
SUsSEN spumiel
briard
casile
garisge truck
urgam
palace
eskimo
stundard
ladle
mailbag
bench
revolver
shower
stupa
paddle wheel
Uastand apple
eggiing
pooker
triumphal arch
To ailor Lower
yurt
traflic light
Figure Successful targeted unrestricted adver- sarial examples for target class 'orange'.
Figure 10: Successful targeted unrestricted ad- versarial examples for target class 'church'.
hhttps://gii httpss://gitthbb.cmm///poootttt
13

---

Under review conference paper at ICLR 2020
A.1 COMPARISON TO NO ADVERSARIAL FINETUNING
The results above are not as high-quality as those reported in the BigGAN paper (Brock et al., 2019). There causes of this which need to be disentangled: the effect of adversarial finetuning, and the limitations of the BigGAN implementation. One limitation is that the checkpoint we use has a much lower Inception Score than is reported in the paper. Another that our limited access expensive hardware forces us to use a batch size of 15; small batch sizes "Icad to inaccurate estimation of the batch statistics, and reducing batch normalisation' 's batch size increases the model error dramatically" (Wu & He, 2018, p.
We therefore provide samples from the BigGAN running on our machine, 15 iterations after the checkpoint. Figure I la shows set of samples taken after 15 iterations of continuing ordinary training; Figure 1 llb shows the output of the generator for the same input after 15 iterations of adversarial finetuming, instead.
To our eyes, the samples in Figure I are similar in quality to those Figure 1lb and on the previous page.
(a) No adversarial finetuning: ordinary train- ing continues.
(b) With 15 gradient steps adversarial fine- tuning.
Figure 11: Samples for fixed inputs to the BigGAN implementation we use, taken 15 gradient steps after the checkpoint we begin adversarial finetuning from.
14

---

Under review conference paper at ICLR 2020
B
SAMPLES OF MNIST UNRESTRICTED ADVERSARIAL EXAMPLES
(a) Intended true label '0'.
(b) Intended true label 'I'.
2
2
2
2
2
2
7
2
3
3
5
3
5
5
3
3
3
3
?
3
2
3
J.
3
3.
1
3
3
3
3
3
3
3
5
6
-2
2
3
3
3
2
)
0
5
3
2
3
3
3
3
3
5
0
2
2
2
2
1
2
3 ).
3
3
3
3
3
3
5
5
3
2
7
2
3
2
2
2
3
5
3
5
3
5
5
3
(c) Intended true label '
(d) Intended true label '3'.
4
4
/
4
;
S
$
5
5
5
:
5
3
S
4
5
5
5
6
5
5
5 5
4
5
5
6
3
5
5
5 4
4
5
3
5 6
4
4
4
5
5
S
5
J
5
6
9
5
(c) Intended true label '4'
(f) Intended true label '5'.
6
7
1
+
7
9
7
6
6
6
6
7 7
7
6
6
7 7
7
6
b
7 7. 7
7
7
6
7
7
1
/
(g) Intended true label '6'.
(h) Intended true label
S
3
8
3
8
8
8
3
8
S 8
Q
7
9
9
1
9
7
7
4
9
8
3
8
6
3
3
8
8
9
G
8
9
9
9
7
9
9
3
3
8
6
8
9
9
7
G
9
4
9
9
9
9
8
4
3
6
3
7
8
8
8
9
9
9
9
7
7
9
9
7
3
8
3
8
8
$
6
7
9
7
7
9
6
9
C
9
9
(i) Intended true label '8'.
Intended label '9'.
Figure 12: Examples generated by one adversarially-fime -inettned GAN to perform an untargeted attack on Wong & Kolter's (2018) classifier, which is provably robust perturbation attacks.
15

---

Under review conference paper at ICLR 2020
C TARGETED CLASSIFIERS
All targeted classifiers (other than "simple fully-connected") are provably robust to adversarial perturbations the sense that there is guaranteed to be adversarial input within distance of p% of test inputs under the loo norm.
Table 3: Descriptions of and references to target classifiers used.
Our Name
Abbreviation
Architecture
Wong & Kolter (2018)
W&K
0.1
94.2
2 convolutional layers followed by 2 dense layers
MixTrain (Wang et al., 2018)
MT-A
0.1
97.1
convolutional layers followed by 1 dense layer
"MNIST_smal
Model A
MixTrain (Wang et al., 2018)
MT-B
0.3
60.1
'MNIST "MNIST_Small': small': convolutional layers followed by 1 dense layer
Model B
MixTrain (Wang et al., 2018)
MT-C
0.1
96.4
'MNIST convolutional lay- ers followed by 2 dense layers
Model C
MixTrain (Wang al., 2018)
MT-D
0.3
58.4
'MNIST large' 4 convolutional lay- ers followed by 2 dense layers
Model D
Simple mully-Connected
Simple
N/A
N/A
Three fully-connected layers of size 256, 128 and 32 with LeakyReLU activations
D ADVERSARIAL TRAINING EXPERIMENT
The classifier trained during adversarial training (both the architecture and hyperparameters) is the one used Madry et al. (2017), 7), and in particular from their associated MNIST Adversarial Examples Challenge.
For the offline experiments with our own model, we first pretrain the generator: We then continue in 'training rounds'. First, fine-tune against the classifier for 5000 gradient steps, using the hyperparam- eters from Table 6, but with an attack rate of 0.4. Next, we produce 80,000 attacked training examples (using an untargeted attack), which are added to the pool of all examples generated so far. Then, the clas- sifier is trained on the entirety of the pool of samples 30 times, with a batch size of 128. Once training round completed we start again, resetting the GAN to how was before the adversarial finetuning.
For the experiments with Song et al. 's (2018b) model, we run 300 training gradient steps for the Madry al. classifier, with batch size of 64. At each step, the training data produced by Song et al. model. We use their code and the hyperparameters they provide for untargeted attacks in Table 4 of their appendix.
16

---

Under review as conference paper at ICLR 2020
E MNIST EXPERIMENTS: ARCHITECTURES AND HYPERPARAMETERS
The WGAN-GP (Gulrajani et al., 2017) and ACGAN (Odena et al., 2017) architectures were the starting points for the design of these neural networks. Only small amount of manual hyperparameter tuning was performed.
The generator is convolutional neural network, conditioned on class label.
The discriminator network is a combination conditional WGAN-GP critic, which learns an approxi- mation of the Wasserstein distance between the generated and training-son set conditional distributions, and an auxiliary classifier, which predicts the likelihood of the possible values of ft(z). We combined these two architectures in an attempt strengthen the gradient provided to the generator, helping to generate data which are both realistic and for which the true (i.e., human- human-judged) labels match the intended true labels. The critic given the true label of the data h(x) improve its training, but the auxiliary classifier must not have access to this information since its purpose is predict it. We therefore split discriminator d into three sub-networks. Network dg: X + IR effectively preprocesses the input, passing an intermediate representation to the critic network d1 IR' x Y - R and the auxiliary classifier network d2 : R -> RIY In our experiments, both dj and d2 were single fully-connected layers of the appropriate dimension. The loss terms from the WGAN-GP and ACGAN algorithms are simply summed. The auxiliary classifier helps the training converge, but is not necessary.
17

---

Under review conference paper at ICLR 2020
Table 4: Architecture for generator network,
Layer Type
Kernel Strides
Feature Maps
Batch Norm.
Dropout Activation
Fully-Connected
N/A
N/A
No
0
ReLU
64
Transposed Convolution
5
2 2
Yes
0.35
LeakyReLU
32
Transposed Convolution
5
2x2
Yes
0.35
LeakyReLU
Transposed Convolution
x
2x2
Yes
0.35
LeakyReLU
4
Fully-Connected
N/A
N/A
784
No
0
Tanh
Table 5: Architecture for discriminator subnetwork, dg.
Layer Type
Kernel
Strides
Feature Maps
Batch Norm.
Dropout
Activation Function
Convolution
3x3
2x2
8
No
0.2
LeakyReLU
Convolution
3x3
1x1
16
No
0.2
LeakyReL
Convolution
3x3
2x2
32
No
LeakyReLU
Convolution
3x3
1x1
64
No
0.2
LeakyReLU
Convolution
3x3
2x2
128
No
0.2
LeakyReLU
Convolution
3x3
1x1
256
No
0.2
LeakyReLU
Table 6: Hyperparameters for all networks.
Hyperparameter
Value
Attack rate
= 0.1
Learning rate
0.000005
Adam betas
31 0.6,32=0.999
Leaky ReLU slope
0.2
Minibatch size
100
Dimensionality of latent space
128
Weight initialisation
Normally distributed as described by He et al. (2015)
Coefficient of gradient penalty loss term
1=10
18

---

Under review conference paper at ICLR 2020
F
VISUAL EFFECT OF ADVERSARIAL FINETUNING
0
2
3
5
6
7
8
9
0
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9 9 9 9
0
/
2
3
4
5
6
7
8
9
O
/
2
3
4
5
6 6 6
7
8 8
O
/
}
3
4
5
6 6 6
7
8 8
9 9 9 9
O
1
2
3
5
7 7
O
1
2
3
4
5
7 7
0
0
2
3
4
5
8
0
2
3
4
5
8
0
1
2
3
4
5
6
7
8
9
0
2
3
4
5
6
7
8
Samples from pretrained generator.
(b) After 5,000 iterations of finetuning.
0
2
3
/
5
6
8
9
0
2
3
4
6
7
2
9
0
/
2
3
7
5
6
7
8
9 9 9
0
/
2
3
4
5
6
7
g
9
O
/
2
3
4
5
6
7
8 8
O
/
}.
3
4 4
S
7
9
9
01
2
3
4
5
6 6
7 7
O
1
2
3
8
9 9
0
1
7
3
1
5
8
Q
0
/
7
3
4
5
8
0
2
3
4
5
67
t
9
0
2
3
4
5
6
9
9
(c) After 10,000 iterations of finetuning.
(d) After 20,000 iterations of finetuning.
0
3
4
5
6
7
8
9
0.1 2
3
4
5
7
?
0
/
2
3
4
5
4
7
8 8 8
9
b.
2
3
4
5
6
7
3
9
0
/
1
/
5
6
7
9
0 011
6
6
7
8
9 9 9
O
2
3
4
8
6
9 9 9
C
2
3
4
$
6
7
8
0
7
3
4
5
6
7
8
Q
/
2
3
4
5
6
7
8
0
2
3
5
6
7
0
0
3
4
5
6
4
8
4
(f) After 45,000 iterations of finetuning. We ended finetuning at this stage.
(e) After 30,000 iterations of finetuning.
Figure 13: A sequence of images tracking the output of the generator network for one fixed random sample in latent space adversarial finetuning takes place. Five samples are given for each intended true label. The finetuning is an untargeted attack against Wong & Kolter's (2018) provably-robust network.
19

---

Under review conference paper at ICLR 2020
G TRANSFERABILITY OF ADVERSARIAL EXAMPLES
Perturbation based adversarial examples typically somewhat generalise between models (Szegedy et al., 2014; Liu al., 2017). That inputs crafted using white-box access fool one model often fool a dif- ferent model. This means that black-box attacks are possible, if the attacker has a different trained model for the same task. To evaluate whether our method could be used in the same way, we generated about 20,000 untargeted unrestricted adversarial inputs for each target classifier, and measured the misclassifi- cation rates on this set for the other models. The high variance of the results, shown Table 7, suggests that successful transfer may depend more on the networks in question than on our generation algorithm. Table 7: The percentage of adversarial examples targeting each classifier which are also adversarial for the others. See Appendix C for descriptions of the classifiers.
To
W&K MT-A
MT-B
MT-C
MT-D Simple
W&K
20.2
18.4
9.0
60.7
16.8
MT-A
19.5
14.1
13.3
55.2
4.7
From
MT-B
5.2
4.8
1.6
57.8
2.6
MT-C
25.8
47.6
13.9
67.8
12.1
MT-D
5.9
7.3
9.4
4.3
1.7
Simple
2.7
2.6
2.6
1.3
48.0
H SAFEGUARDING MTURK DATA QUALITY
The evaluation of our method relies entirely on the quality of the data provided by the MTurk workers. We therefore took a number of measures to ensure that participants understood the instructions and completed the tasks diligently:
Only workers with good track records were permitted to participate.
The instructions specified that particular answers should be given to specified questions to prove that the instructions had been read carefully. Approximately 10% of work was rejected for failing this check.
For the image labelling tasks, some images with known labels were included to check that the right labels were being given. Reassuringly, almost work was rejected for failing this check.
For the identification of the generated images, bonus nearly doubling the pay per image was given for each correctly-idention image, providing an extra incentive to try hard.
To provide disincentive to high-speed random clicking, minimum time spent answering each question was enforced.
If more than 1% of questions were left unanswered, we interpreted this as a sign of carelessness and did use any of the data from that task.
20

---

Under review as conference paper at ICLR 2020
1 RESULTS FOR NON-ROBUST TARGET NETWORK
These results are targeting a simple convolutional neural network with LeakyReI.U activations and three hidden layers of size 256, 128 and 32, trained until convergence.
Target label
Target label
3
6
7
9
None
2
3
4
7
None
93
96
96
99
93
95
97
94
97
96
76
44
65
70
77
89
56
75
76
92
100
92
97
96
88
96
96
93
82
89
95
99
93
76
98
98
98
Intended true label
2
73
86
82
80
87
92
84
87
75
81
2
35
45
94
55
40
55
75
75
48
68
88
83
87
81
88
81
89
96
90
92
64
66
66
46
71
33
73
84
80
Intendie ITue
84
53
69
90
90
81
87
85
66
41
68
56
66
85
77
84
89
77
89
88
79
94
88
88
83
64
68
80
76
71
69
63
65
82
73
6
96
83
92
95
93
95
93
100
96
94
6
82
63
66
43
91
79
78
81
60
85
7
93
59
89
95
85
94
80
99
94
92
7
73
47
82
80
72
92
100
92
86
77
8
96
86
97
93
98
93
90
92
92
91
87
69
81
77
79
65
88
62
81
85
93
76
96
97
91
89
93
98
90
79
75
76
96
82
74
Mean 88
79
89
91
90
91
88
91
92
91
90
Mean
70
56
75
72
72
74
73
73
76
78
80
Figure 14: The success rates of the adversarial attacks by finetuned GANs. More precisely. of generated images for which the computed label output by the classifier matches the target label, the percentage which are truly adversarial (in the sense that the true label of the image matches the intended true label passed to the generator network) is reported.
Figure 15: The success rates of the adversarial attacks by pretrained but not finetuned GAN. More precisely, generated images for which the computed label output by the classifier matches the target label, the percentage which are truly adversarial (in the sense that the true label of the image matches the intended true label passed to the generator network) is reported.
Target label
Target label
3
4
5
7
8
9
None
2
3
5
6
7
None
45
48
46
43
38
61
51
44
59
53
59
56
61
65
63
55
51
54
63
64
74
72
62
78
83
73
75
79
56
62
76
76
70
72
80
77
74
Intended true label
Intended true label
2
552
43
53
41
52
35
48
54
40
55
2
53
55
66
54
48
50
62
64
52
61
3
64
41
62
43
60
29
51
56
58
50
68
54
62
48
64
52
64
65
71
59
45
49
36
45
55
49
69
65
57
54
55
63
52
60
73
5
48
46
44
63
62
60
49
57
61
50
5
69
64
58
62
57
73
53
60
62
63
6
76
48
44
46
54
54
38
62
54
58
6
63
64
58
59
67
71
62
63
63
67
7
51
32
60
59
54
54
46
61
65
65
7
54
63
71
61
71
62
52
67
74
77
8
62
53
62
57
56
56
54
50
67
57
65
60
60
71
57
60
81
55
69
67
47
42
51
60
72
69
54
66
71
67
64
51
65
64
81
71
76
68
68
Mean
57
44
54
55
55
54
52
55
59
61
60
Mean
62
58
60
62
64
64
62
62
63
65
68
Figure 16: Measures of how realistic the adver- sarial images generated by finetuned GANs are. More precisely, the proportion of generated inputs for which the classified label matches the target label which were not identified as being gener- ated when placed amongst nine images from the training dataset. If the generated images were completely realistic, the expected result would be 90.
Figure 17: Mcasures of how realistic the adversar- ial images generated by pretrained but not fine- tuned GAN are. More precisely, the proportion of generated inputs for which the classified label matches the target label which were not identified as being generated when placed amongst nine im- ages from the training dataset. If the generated images were completely realistic, the expected result would be 90.
21

---

Under review conference paper at ICLR 2020
J SIDE-BY-SIDE IMAGE COMPARISON RESULTS
Each figure shows the number of human judgements out of 100 which correctly identified the unrestricted adversarial input in side-by-side comparison with image drawn from the dataset. If the generated images were completely realistic, the expected result would be 50.
Target label
Target label
0
2
3
4
5
6
7
8
9
None
0
2
3
4
5
6
7
8
9
None
23
29
24
30
25
20
22
17
22
28
43
37
42
42
41
36
43
38
45
36
19
16
26
25
21
24
29
40
38
40
43
37
34
Intended true label
Intended true label
24
25
18
25
17
26
17
19
29
45
30
38
36
35
39
44
41
31
40
3
22
24
17
31
28
25
24
24
30
20
35
35
38
36
45
34
42
44
38
36
4
25
25
28
20
21
28
22
17
23
19
30
42
42
40
36
35
47
42
38
42
5
23
16
24
27
29
23
19
27
21
21
35
43
40
35
34
34
42
36
37
44
19
25
21
19
25
20
28
18
23
46
42
37
42
4.5
38
36
23
22
26
17
24
29
16
21
32
38
42
42
3.5
466
4.5
8
25
25
21
21
24
23
24
25
28
233
41
41
44
46
37
37
43
42
44
9
18
21
22
27
27
24
23
28
23
28
9
45
41
35
44
40
40
52
36
49
Mean
22
23
22
23
24
24
25
23
23
22
24
Mean
38
40
39
38
40
44
40
39
Figure 18: Results against Wong & Kolter (2018)
Figure 19: Results against Wong & Kolter (2018) generated by a pretrained but not finetuned GAN.
generated by adversarially finetuned GANs.
Target label
Target label
3
4
5
7
8
9
None
1
2
3
5
6
7
8
9
None
23
22
22
24
27
23
25
31
26
34
16
21
27
31
32
31
28
30
36
41
29
28
36
26
31
27
25
24
38
22
25
Intended true label
Intended true label
2
24
28
31
22
24
30
26
31
20
28
2
24
24
29
28
26
21
28
25
22
27
3
30
26
30
20
20
25
19
21
25
23
3
23
27
26
26
29
23
22
29
31
31
27
25
20
28
27
25
26
27
30
27
18
23
31
28
24
29
32
37
33
29
25
24
25
22
22
28
24
30
24
28
23
32
30
22
27
17
30
28
20
32
23
29
37
23
23
21
28
30
26
29
25
26
7
18
24
30
32
25
24
22
34
26
34
7
23
33
22
29
26
25
24
27
28
32
8
28
29
19
23
22
28
27
26
26
37
8
31
20
21
26
29
31
3
26
33
30
29
21
20
31
28
33
21
44
30
38
27
26
26
22
32
26
31
26
30
Mean
27
25
27
26
25
26
25
27
29
25
30
Mean
27
24
26
26
29
28
26
25
29
28
29
Figure 20: Results against an ordinary neural net- work generated by adversarially finetuned GANs.
Figure 21: Results against an ordinary neural net- work generated by pretrained but not finetuned GAN.
22

---

Under review as conference paper at ICLR 2020
K ADVERSARIAL FINETUNING ABLATION STUDY
We perform the attack procedure described in Section 4, but using a GAN which has not been adversarially finetuned. That is, we use the generator to generate many examples, and filter keep all those which are misclassified (the untargeted case) or misclassified with particular label (the 'targeted' case, although note that of course the un-finetuned generator has no notion of target). We then evaluate the proportion of these filtered examples which correctly maintain their intended true class, a necessary condition for an adversarial attack. We also report the proportion of these filtered examples which are not correctly identified by humans judges as being generated (out of selection of ten).
For instance, consider the result for intended true label 9 and target label We first use the conditional GAN to produce set of images that are intended to be 9s. We then filter this set and keep only those that are classified as by the classifier. Finally, we report below the percentage of these for which the true label (determined by humans) is indeed 9 (55%), and the percentage of these which fool human judges into believing that they are real test data (51%).
Target label
Target label
3
4
8
9
None
1
2
6
None
73
91
62
62
91
88
50
59
75
48
63
53
62
56
46
40
37
54
49
51
80
06
31
47
61
77
57
34
52
50
37
27
52
47
46
42
Intended true label
Intended true label
2
13
62
53
32
30
30
52
50
19
60
2
42
39
51
39
30
36
54
49
40
43
29
69
60
26
60
12
79
22
28
71
41
47
55
42
55
47
53
41
47
51
42
43
70
70
48
73
65
49
52
46
45
50
51
57
18
46
55
61
54
29
59
51
60
5
38
50
43
56
47
54
44
51
48
51
6
50
60
80
87
74
80
74
54
59
66
6
49
52
47
50
46
55
38
62
46
48
7
22
43
82
63
3
64
00
62
38
40
7
39
57
59
36
49
4.5
32
41
.57
.59
8
70
80
75
75
50
80
79
52
50
59
52
57
43
39
51
59
55
74
88
92
69
88
51
51
66
53
73
Mean
37
60
72
63
58
65
43
67
50
54
66
Mean
45
50
52
50
51
50
44
50
47
49
53
Figure 22: Adversarial success rates for non- finetuned generator, as described above.
Figure 23: Realism rates for adversarial images produced by non-finetuned generator, as de- scribed above. If the generated images were com- pletely realistic, the expected result would be 90.
23

---

Under review conference paper at ICLR 2020
L TRAINING STRATEGIES ABLATION STUDIES
We run several ablation studies for the components described in Section 3.3. We find that while these strategies are necessary, they make examples more realistic. This in turn improves the chance that the true class of an example remains the same, increasing our success rate.
L.1
PRETRAINING
0
3
3
7/
8
7 5 ?
0
1
3
3
3
6 7 $ 8
9
0
3
4
5
6
7
9
3
3
4
3
5 9
O
5
7.
3
7 2
9
Q
3
4
6
12
1
8
7
3
y
7. 9
0
7
7
5
F
/
3
6
7
5
0 0155 4
6 3. 11 6.6
5
0
3
4
6
0
F
2
4
5
6
7.
9
0 3. 5
2
3
3
2
1
3
3
4
5
5
7
0 19.4 1 5 7
3
1
4 3
3
5 7 89
5
2
7
5
4
7
5
9 9
/ 2. 3
5759
/ 2
0
2
3
7
5
7 7 7
3 3
1 6
3
/
~
4
0
1
6
6
3
1
5
5
0
/
3
7 7659
5
0
3
8
0
3
3
6 7
0
/ 3
0
3
5
7
(a) Results after having long as pretraining, 705,000 iterations
(b) Results after having long as Figure 13, 750,000 iterations
Results training had verged, over 2.5 million iterations
On MNIST pretrained the GAN for 705,000 iterations, and then finetuned for another 45,000. Here, we show results using the same setup as in Figure 13. We show the results after the number of iterations equivalent to pretraining, to pretraining and finetuning, and at convergence. The final results both took longer and are visually less convincing than comparable results in Figure 13.
Figure 24:
L.2
NAÎVE LOSS FUNCTION
5
4
5 3
8
3
5
5
7
3
1 3
$ B
/
5
2
1
8
7
5
G
1
3
5
7
3
5
7
7
\
$
7
8
5
3
4
7
2
5
4
5
5
(a) After 500 adversarial finetum- ing steps; misclassified.
(b) After 1500 adversarial finetun- ing steps; 91% 'misclassified'.
After 10,000 adversarial fine- tuning steps; 'misclassified'.
Adversarial finetuning with naïve generator loss lordinary + ummtrggted As expected, custom loss function as described Section 3.3 significantly improves convergence to generator weights which continue to generate images realistic enough maintain their true classes.
Figure 25:
24

---

Under review conference paper at ICLR 2020
L.3
ATTACK RATE
0
2
3
4
5
3 9
0 / I
2 8
5 5
6
7 6 7 7 3
9
4
5
4
5
0
2
3
4
3
7
f 9
5
3
4 5
8 8 ! 7
9
O /
3
4
5
7
8
O
2
3
4/
5
8 9 8 9
7
O I 3
3
0
/
3
4 /
6
6
2
3
4
S
7
0 1
6
A
4
6
O A
3
/
0 3
0
7
0
/
223
5
6
7
0 0175
6 6 6 5666677
5
7 89
0 1
2 3 3
0
2
3
4
5
6
7
8
0
/
3
3
4
99
D 1 2
H
2
3
4
5
7
8 8
0 /
/ 2 2
8
4
<
9 7 7 S
O 1 O /
3
4
3
4
7 7
0 0
3
6
7
3
5
8
9
x
9
O 1
6
6
(a)
μr 0.01: misclassified
(b) p= 0.
10;
16% misclassified
(c) 0.25: 68% misclassified
/
2
3
5
7 5 5 8 9 8 9
9
0 /
3
3
4
<
6 7 5 9
0
/
3 3
5 6
7
8
3
3
4
5
7
7
C
1
3
2
4
7
8
9
D. / 5
5
3.
6
3
2
3
4
7 6
9
0
0125
R
7
$
9
O 1 2 3.
5
6
8
0
2
3
7
5"
0
01351
1
5
7
8
/
556
3
0
2
3
9
6 7
9
G.
1
1
4
4
7
5
9
0 _
5 1
3
0
2
3
4
0
1
5
4
5
6
7
8
0
13354666
3
5
0
2
3
4
5
7 7
9
0
i
3
4
0
7 7
8.9
0
2
3
3
9
2
3
5
6
9
6 f
3
~
79
G
3
2
3
9
3
3
7 9
0
2
3
4
2
3
4
6
0
2
3
5 6 2
0
1
2.
5
(d) μr 0.50; 94% misclassified
(c) μ= 0.75; 98% misclassified
p= >99% misclassified
Figure 26: The effect of attack rate on image quality and proportion misclassified, using otherwise the same setup as in Figure 13. An attack rate of equivalent to not having the attack rate. As expected, lower attack rates mean higher visual quality, but less successful attack.
25


Reviews:
[{"review_id": "rkgqAJxpFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a GAN architecture that generates realistic adversarial\ninputs that fool a targeted classifier.  Adversarial inputs are unrestricted:\nthey may be any realistic images that humans will often classify as real\nexamples of the intended class, whereas the target model misclassifies them.\nThe novelty is that they finetune the generator itself during training, the\nmethod can be applied to a variety of GAN architectures, and the method is fast.\n\nTricks used to successfully train the GAN are clearly described, and the\nexperimental evaluation was of good scope, covering a good selection of\nexperiments.  I particularly enjoyed the short Section 4.2 and Fig 7a+b, where\nthey show that a local defense can always be fooled somewhere else along\nthe input manifold of that class.\n\nWhile the modifications to existing solutions may at first seem minor, they\nhave significant impact in applicability, effectiveness and speed of generating\nunrestricted adversarial images.  So I think this paper can be accepted.\n\nI had a bit of avoidable confusion in the introductory sections.  Figure 1\ndescribing the GAN is never referred to.  It includes components not exactly\nagreeing with my naive expectations from surrounding text.  Are any Fig. 1 features\noptional?  It would help to highlight the novel elements in Fig. 1.  Or does\nFig.1 correspond perhaps to the combined GAN elements in Section 4 (\"In our\nexperiments, we combine three ...\").  My uncertainty was really relieved only\nby the time I got to Related Work and Appendix E :(\n\nThe main claims seemed well supported by experiments, apart from claim 3\n(applicability to \"any\" checkpointed GAN codebase). Might the scope of their\napproach also be clarified by clearly identifying required and optional GAN\ncomponents in Fig. 1?\n\n---- misc comments ----\nSome sentences were long and difficult to parse:\n- 4.1: \"Our method generates..., else ....\"  Perhaps make the else clause a second sentence.\n- 4.2: \"Image quality as measured ....\" length and references made this difficult to read. Can you\nrewrite as separate shorter sentences?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxptTW6tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "======== update ========\nI have read the authors' response and it has addressed most of my concerns. I am glad to see the authors' experiments on online adversarial training. \n\nHowever, there is one additional concern that I didn't realize previously. Currently the performance of adversarial training is measured in \"success rates\". However it seems to me this success rates were not computed using human evaluation (since the authors claim once the classifier is finished training, the attack success rate can be larger than 99%). I would have changed my score to 8 if either 1) some adversarial images from the generator were included after finishing adversarial training or 2) success rates using human evaluation is reported. Unfortunately, I only realized this after the author rebuttal period, and the authors didn't have the chance to address this.\n\nThat being said, I feel this paper still presents interesting contribution to the field. I am still largely in favor of the acceptance of this paper, and will remain my rating of 6 for now. If this paper gets accepted, I strongly encourage the authors to address the concern I mentioned above in their camera ready.\n\n\n======== original reviews ========\n\nThis paper proposes a novel method on generating unrestricted adversarial examples by finetuning GANs. The authors have conducted comprehensive experiments on evaluating the advantages of their approach. They demonstrated that their attack is harder to mitigate using adversarial training, produces unrestricted adversarial examples faster than existing methods, and can generate some unrestricted adversarial examples for complex high-dimensional datasets such as ImageNet.\n\nI feel although the approach is straightforward, the authors have done a good job in motivating the method and have demonstrated its advantages via a good cohort of experiments. I like how the authors motivated finetuning in section 3.2, and I am glad that the authors have conducted ablative experiments to support their arguments in section 4.4. The experiments on adversarial training are especially interesting, since previous work hasn't considered this straightforward defense against unrestricted adversarial attacks. I am also glad that the authors can generate unrestricted adversarial examples for data as complicated as ImageNet images using the latent technique in GANs. Although still not perfect, some of the unrestricted adversarial examples on ImageNet are surprisingly good to the sense that they may be used as practical attacks.\n\nThe writing is great, and it is a pleasure to read this paper. \n\nI do have some suggestions and questions for further improvement of the paper, and I strongly recommend the authors to address those before publication.\n\n- Section 3 is lacking an explicit form of the combined objective function. Currently some loss functions such as $l_ordinary$, $l_targeted$, $l_d$ and $l_finetune$ are only defined in Figure 1 but not in the main text. It is not clear their explicit mathematical form.\n\n- In section 3.2, it is better to also mention the ablative study you did later in section 4.4. \n\n- In section 4.1, the authors showed nearest neighbors to some of the unrestricted adversarial examples they generated. It is more convincing to have some quantitive results of that. For example, what is the average minimum distance to training data for a group of 10000 unrestricted adversarial examples? In addition, what is the distance function used in computing nearest neighbors? Did you use Euclidean distance? If so, it would be better to also have results using distances computed in the feature space of a pre-trained convolutional network.\n\n- In section 4.2, the adversarial training was done by alternating two phases of training rounds. I am wondering whether this makes the classifier harder to adapt to the newly generated unrestricted adversarial examples? Can you use some procedure more similar to traditional adversarial training, i.e., the attacker and the classifier are learned together at each step? \n\n- Song et al. require 100-500 iterations to generate an adversarial example, whereas your approach only need one iteration. Why is your approach 400 to 2000x more efficient? What is the additional reason that speeds up your approach?\n\n- In section 4.5 line 1, the word \"replies\" was repeated twice.\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJeOnu0jYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes using GANs to generate unrestricted adversarial examples. They seek to generate examples that are adversarial for a specific classifier, and they do so by using class-conditional GANs and a fine-tuning loss. The fine-tuning loss consists of both the ordinary GAN loss (to fool the discriminator) as well as an adversarial loss (which rewards the GAN for generating examples misclassified by the specific classifier). The authors perform various experiments on their generated examples to check for realism and how adversarial the generated images are.\n\nI would reject this paper for two key reasons. First, I feel that the contributions are not significant enough (in comparison to the prior work of Song et. al). Second, I feel that some of the methods (and some of the writing) are not too principled.\n\nIn my opinion, unrestricted adversarial examples are significant if they can be made to be realistic. If our current deep learning models often mislabeled very realistic images, that would properly expose a big failure mode of our current models. However, if our machine learning models perform poorly on images that look fake/generated 40% of the time (which is what the authors state) and don’t look too realistic to humans, it is less worrying.\n\nIn comparison to Song et. al, the authors state that their methods result in very similar results in terms of realism and how adversarial their images are (arguably, Song et. al actually produces better results in terms of being adversarial). In my opinion, the authors’ claimed improvements are not significant enough, because I think realism should be the primary metric to evaluate this field. Improving speed of generation is nice, and being able to bypass a simple adversarial training procedure is interesting but not significant unless this insight is expanded upon. The results on MNIST in Fig. 5 and Fig. 6 are not too convincing, as simpler attacks that generate (arguably) more realistic images like translations and rotations [1] or L1/L2 attacks [2] (since the networks are trained for L_inf robustness) can also degrade accuracy. Finally, I can also think of another reasonable baseline that I would have liked to see the authors compare their method against. Because the authors want to attack a specific network, they could have (1) generated realistic images using a pre-trained GAN (2) used a norm-bounded attack on the specific classifier and the generated GAN images. These images could be even more realistic if the norm-bound of the attack is fairly small, and would still be able to attack specific classifiers.\n\nFinally, I am confused by the comparison to a not-fine-tuned GAN in Fig. 14/Fig. 15 and would appreciate a clarification so that I can understand the results. For example, what does it mean for intended true label = 9, target label = 0 to have 90% success in Fig. 15? Does this mean that when you try to generate a 9 with the GAN, the classifier misclassifies it as a 0 90% of the time? In particular, I’m struggling to understand what the target label is for the case of the not-fine-tuned GAN.\n\nSecondly, I feel that there are many instances in the paper where the methods used are not explained in a principled way. For example, one of the key parts of this work is the fine-tuning loss function. Why does the loss function involve multiplying the ordinary GAN loss (with some additional transformation applied to it which seems unnecessary) with the adversarial loss? It seems most reasonable add the adversarial loss and the ordinary GAN loss (without the additional transformation). Is the stochastic loss selection procedure necessary? If all these peculiarities of the method are necessary, it seems that the success of this method is quite brittle.\n\nAdditional feedback:\n\n- In the intro, I think citing [3] in addition to Xu et. al is more appropriate.\n- You should refer to Figure 1 somewhere in the text of your work\n- In section 3.2, you can use “cosine similarity” to describe what you are doing faster.\n- When you talk about “global optima of realistic adversarial examples” and “local optimal of unrealistic adversarial examples,” it sounds weird. I would try to reword this because I don’t think you are trying to make a precise mathematical statement but it sounds like one when you write it this way.\n- In Table 1, I would format the numbers better to be vertically aligned\n- You should provide a citation for MixTrain on page 5\n\n[1] https://arxiv.org/abs/1712.02779,\n[2] https://arxiv.org/abs/1905.01034\n[3] https://arxiv.org/abs/1802.00420", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting method for creating adversarial examples using a GAN.  Reviewers are concerned that ImageNet Results, while successfully evading a classifier, do not appear to be natural images.  Furthermore, the attacks are demonstrated on fairly weak baseline classifiers that are known to be easily broken.  They attack Resnet50 (without adv training), for which Lp-bounded attacks empirically seem to produce more convincing images.  For MNIST, they attack Wong and Kolter’s "certifiable" defense, which is empirically much weaker than an adversarially trained network, and also weaker than more recent certifiable baselines.

Decision: Reject