Title: Adversarial Machine Learning at Scale
Authors: Alexey Kurakin
Ian J. Goodfellow
Samy Bengio
Venue: ICLR 2017
Keywords: Computer vision
Supervised Learning

Abstract: Adversarial examples are malicious inputs designed to fool machine learning models.
They often transfer from one model to another, allowing attackers to mount black
box attacks without knowledge of the target model's parameters.
Adversarial training is the process of explicitly training a model on adversarial
examples, in order to make it more robust to attack or to reduce its test error
on clean inputs.
So far, adversarial training has primarily been applied to small problems.
In this research, we apply adversarial training to ImageNet.
Our contributions include:
(1) recommendations for how to succesfully scale adversarial training to large models and datasets,
(2) the observation that adversarial training confers robustness to single-step attack methods,
(3) the finding that multi-step attack methods are somewhat less transferable than single-step attack
      methods, so single-step attacks are the best for mounting black-box attacks,
      and
(4) resolution of a ``label leaking'' effect that causes adversarially trained models to perform
      better on adversarial examples than on clean examples, because the adversarial
      example construction process uses the true label and the model can learn to
      exploit regularities in the construction process.



Review discussion:
The key contribution of scalable adversary training is "examples to be grouped into batches containing both normal and adversary examples".
Could you explain the why it is important?
Also I suggest to compare with other adversary training algorithm, such as in [1], training without clear sample. In experiment of [1], with VGG network, similar regularization effect was found, and they provided proof for linear model.
[1] Huang, Ruitong, et al. "Learning with a strong adversary." CoRR, abs/1511.03034 (2015).
We added a citation to “Learning with a Strong Adversary” and discussed a comparison of the findings in the paper.
“The key contribution of scalable adversary training is ‘examples to be grouped into batches containing both normal and adversary examples’. “
Recommendation to use both clean and adversarial examples in the minibatch was one of the observations made in our experiments.
However there are other contributions of the paper (as stated in Sec 1), including successful adversarial training on ImageNet dataset,  the discovery of the “label leaking” effect, empirical study of transferability and importance of model capacity.
Regarding training with only adversarial examples:
This strategy corresponds to the method described in “Explaining and Harnessing Adversarial Examples”, Goodfellow et al, 2014, with alpha set to 0 rather than 0.5. It is not a completely distinct training procedure.
When writing “Explaining and Harnessing Adversarial Examples”, Ian introduced the alpha parameter after finding that the alpha=0 variant did not perform acceptably. “Improving back-propagation by adding an adversarial gradient” (Nøkland 2015) advocated setting alpha to 0 before “Learning with a Strong Adversary” did so.
In the present work, training on only adversarial examples corresponds to k=32 (given that minibatch size of 32). For our experiments on ImageNet we have done hyperparameter search to find optimal value of k. We observed that increase of k lead to decrease of accuracy on clean examples and does not necessarily provide significant improvement of accuracy on adversarial examples.
Overall for our experiments we have chosen k=16.
Note that in “Learning with a Strong Adversary,” the VGG network is not applied to ImageNet, but rather to MNIST and CIFAR-10. Our most important contribution is experimentation at ImageNet scale.
Hi,
I have two questions about the experiments.
1. For the results in Table 3, what is the value of \epsilon for the adversarial training?
2. For the results in Figure 2, when \epsilon is fixed (say \epsilon=8), have you compared how many successful adversarial examples are generated (and thus passed to the target network)? Will it be another factor of this transferability rate?
Also, do we have an estimation of the variance of the estimation of the transferability rate? How does this variance depend on the number of the successful adversarial examples?
Another question about a statement.
In the paragraph below Table 3, what does the following sentence mean?
"The most labels are leaked wit \epsilon=0.3 *****, the model leaks 79 labels on the test set"
How this number 79, is calculated? Shouldn't all the true label have been leaked, since they all have been used to construct adversarial examples.
Lastly, for the observations in section 4.2, 4.3, and 4.4, are they particular for only large scale dataset, or similar phenomenons happen in small scale dataset?
> For the results in Table 3, what is the value of \epsilon for the adversarial training?
During adversarial training \epsilon was chosen randomly for each example in the minibatch from truncated normal distribution (as described in the last paragraph of section 3). This includes results in Table 3.
> For the results in Figure 2, when \epsilon is fixed (say \epsilon=8), have you compared how many successful adversarial examples are generated (and thus passed to the target network)? Will it be another factor of this transferability rate?
Transfer rate is computed as “number of examples fooling both source and target networks” / “number of examples fooling only source network”.
So yes, number of successful adversarial examples for source network is indeed one of the factors affecting transferability rate.
However conclusions of section 4.4 are valid if you take into account this observation.
In particular if you fix epsilon and compare only numerator of transfer rate for different methods then it will be higher for one step methods, lower for basic iterative method and the lowest for iterative least likely class.
Overall numerator of transfer rate will behave similarly to error rate on adversarial images generated for one network and classified by another. These numbers are discussed in appendix C, table 6 and figure 5.
If your question is about change of epsilon with fixed method.
In such case both numerator and denominator of transfer rate will increase with increase of epsilon.
However we observed that numerator (number of transferred examples) increasing much faster than denominator (number of examples which fooled source network).
For example when epsilon growth from 8 to 16, denominator increases by less than 1% while numerator increases by more than 20%.
We added this clarification to the paper.
> Also, do we have an estimation of the variance of the estimation of the transferability rate? How does this variance depend on the number of the successful adversarial examples?
We estimated that 95% confidence interval for the transfer rate is within ±1.3 percent point of value reported in Table 4. 1.3% is worst case estimation, for many values in this table actual 95% confidence interval is even smaller.
> In the paragraph below Table 3, what does the following sentence mean? "The most labels are leaked wit \epsilon=0.3 *****, the model leaks 79 labels on the test set" How this number 79, is calculated?
N_correct_adv_label - N_correct_adv_pred
where N_correct_adv_label is the number of test examples that are
correctly classified when perturbed using the true label to generate
the perturbation, and N_correct_adv_pred is the number of test
examples that are correctly classified when perturbed using the
model's prediction of the label.
> Shouldn't all the true label have been leaked, since they all have been used to construct adversarial examples.
No, every label had the opportunity to be leaked, but not all of them
were successfully leaked.
We updated the paper to clearly define the term "leak." We say a
label has been leaked if and only if the model classifies an
adversarial example correctly when that adversarial example is
generated using the true label but misclassifies a corresponding
adversarial example that was created without using the label.
We think your question is based on a less restrictive definition of
"leak", where a label is considered leaked if the perturbation is a
function of the label.
Recall that the model gets to observe only the adversarial example
x_adv, not the original example x_clean, so the trivial way to recover
the label by enumerating all possible calls to
make_adversarial_example(x_clean, label) is not possible.
> Lastly, for the observations in section 4.2, 4.3, and 4.4, are they particular for only large scale dataset, or similar phenomenons happen in small scale dataset?
Label leaking was observed on small dataset (MNIST) as well after careful re-examination of results (section 4.2).
We haven’t specifically studied how \epsilon affect transferability and how model capacity affect robustness to adversarial examples on small dataset.
But there are no fundamental reason why it would be different.
In response to this question, we ran the MNIST tutorial for cleverhans 1.0 (the paper for 1.0 isn’t available yet, but 0.1 is here: https://arxiv.org/abs/1610.00768) for 100 epochs with different depths of model.
Default depth, included with cleverhans:
Clean train, clean eval: 0.9922
Clean train, adv eval: 0.0503
Adv train, clean eval: 0.9928
Adv train, adv eval: 0.9347
Double depth:
Clean train, clean eval:  0.9915
Clean train, adv eval: 0.1754
Adv train, clean eval: 0.9924
Adv train, adv eval:  0.9683
We see that increasing the depth of the model increased the resistance to adversarial examples, whether or not adversarial training was used.
Great work! Just one question:
Table 4 shows that FGSM produces examples that are most transferable to other models, implying they are the examples mostly likely to be used in an 'attack'.
However, you mention that adversarial training on FGSM examples does not work due to the label leaking effect. Could you comment on how robust models trained on step l.l. (or another comparable method) adversarial examples are to FGSM examples? Do you have any other proposals for how to deal with making models robust to these kinds of examples?
To clarify, FGSM could be used for adversarial *training* and it does provide robustness to adversarial examples.
However we do not recommend to use FGSM with true labels *to evaluate the model* because it might be unclear whether an increase of accuracy is caused by label leaking or due to actual increase in robustness.
> Could you comment on how robust models trained on step l.l. (or another comparable method) adversarial examples are to FGSM examples?
We discovered that it’s enough to train model on “step l.l.” method to become robust to FGSM adversarial examples. Quantitative results are provided in Appendix A, fig. 3.
>  Do you have any other proposals for how to deal with making models robust to these kinds of examples?
Yes, we briefly discussed in the paper the following ways to increase robustness to adversarial examples:
1. As discussed in section 4.3., a dramatic increase of model capacity, far beyond the model sizes currently in use, might lead to increased robustness to adversarial examples.
2. As shown in appendix D, certain non-linear activation function (so called ReluDecay) could increase robustness to various types of adversarial examples.
However these two methods do not yet provide full robustness to adversarial examples.
Also, they work better in conjunction with adversarial training.
This paper is a well written paper. This paper can be divided into 2 parts:
1.Adversary training on ImageNet
2.Empirical study of label leak, single/multiple step attack, transferability and importance of model capacity
For part [1], I don’t think training without clean example will not make reasonable ImageNet level model. Ian’s experiment in “Explaining and Harnessing Adversarial Examples” didn't use BatchNorm, which may be important for training large scale model. This part looks like an extension to Ian’s work with Inception-V3 model. I suggest to add an experiment of training without clean samples.
For part [2], The experiments cover most variables in adversary training, yet lack technical depth.  The depth, model capacity experiments can be explained by regularizer effect of adv training;  Label leaking is novel; In transferability experiment with FGSM, if we do careful observe on some special MNIST FGSM example, we can find augmentation effect on numbers, which makes grey part on image to make the number look more like the other numbers.
Although this effect is hard to be observed with complex data such as CIFAR-10 or ImageNet, they may be related to the authors' observation "FGSM examples are most transferable".
In this part the authors raise many interesting problems or guess, but lack theoretical explanations.
Overall I think these empirical observations are useful for future work.
Thanks for the review.
As you suggested we added appendix E, which contains experiments of training with various values of fractions of clean/adversarial examples, including experiments with no clean examples.
As could be seen from these results, increasing the fraction of adversarial examples in the minibatch leads to a decrease of accuracy on clean examples and to an increase of accuracy on adversarial examples. Having more than half of adversarial examples in the minibatch (corresponding to k=16 in our case) leads to almost no increase of accuracy on adversarial examples, at the same time it could lead to up to 1% decrease of accuracy on clean examples. This was the reason we chose k=16 for the main experiments done in the paper.
Regarding part [2], do you have specific things in mind where we should add better theoretical explanation?
This paper investigate the phenomenon of the adversarial examples and the adversarial training on the dataset of ImageNet.
While the final conclusions are still vague, this paper raises several noteworthy finding from its experiments.
The paper is well written and easy to follow.
Although I still have some concerns about the paper (see the comments below), this paper has good contributions and worth to publish.
Pros:
For the first time in the literature, this paper proposed the concept of ‘label leaking’.
Although its effect only becomes significant when the dataset is large, it should be carefully handled in the future research works along this line.
Using the ratio of 'clean accuracy' over ‘adversarial accuracy’ as the measure of robust is more reasonable compared to the existing works in the literature.
Cons:
Although the conclusions of the paper are based on the experiments on ImageNet, the title of the paper seems a little misleading. I consider Section 4 as the main contribution of the paper. Note that Section 4.3 and Section 4.4 are not specific to large-scale dataset, thus emphasizing the ‘large-scale’ in the title and in the introduction seems improper.
Basically all the conclusions of the paper are made based on observing the experimental results.
Further tests should have been performed to verify these hypotheses. Without that, the conclusions of the paper seems rushy.
For example, one dataset of imageNet can not infer the conclusions for all large-scale datasets.
Thanks for the review.
We would like to add that while sections 4.3 and 4.4 are not specific to large-scale dataset, all experiments in these sections were done on a large scale dataset.
Thus these results are valid for large scale datasets as well.
In other words these results do cover the case of at least one large scale datasets.
Also as a way to address cons we could consider changing title of the paper to limit its scope.
Possible new titles which we though of:
a) “Adversarial machine learning at scale: a case study”
b) “Adversarial object recognition at scale”
c) “An example of adversarial machine learning at scale”
This paper has two main contributions:
(1) Applying adversarial training to imagenet, a larger dataset than previously considered
(2) Comparing different adversarial training approaches, focusing importantly on the transferability of different methods. The authors also uncover and explain the label leaking effect which is an important contribution.
This paper is clear, well written and does a good job of assessing and comparing adversarial training methods and understanding their relation to one another. A wide range of empirical results are shown which helps elucidate the adversarial training procedure. This paper makes an important contribution towards understand adversarial training and believe ICLR is an appropriate venue for this work.
Thank for the review.

Meta-review:
role: PC
data: comment: This paper is an empirical study of adversarial training in a large-scale regime. 
 Its main contributions are to demonstrate that Imagenet models can be made robust to adversarial examples using relatively efficient constructions (so-called 'one-step' methods). Along the way, the authors also report the 'label leaking' effect, a flaw in previous adversarial constructions that limits the effectiveness of adversarial examples at regularizing training. 
 
 The reviewers were consensual that this contribution is useful to the field, and the label leaking phenomena is an important aspect that this paper helps address and mitigate. The authors responded promptly to reviewers questions. Despite limited novelty and lack of quantitative analysis, I recommend accept as a poster.

Decision: Accept (Poster)