Title: Simple Black-Box Adversarial Perturbations for Deep Networks
Authors: Nina Narodytska
Shiva Kasiviswanathan
Venue: ICLR 2017
Keywords: Computer vision
Deep learning

Abstract: Deep neural networks are powerful and popular learning models that achieve state-of-the-art pattern recognition performance on many computer vision, speech, and language processing tasks. However, these networks have also been shown susceptible to carefully crafted adversarial perturbations which force misclassification of the inputs. Adversarial examples enable adversaries to subvert the expected system behavior leading to undesired consequences and could pose a security risk when these systems are deployed in the real world.

In this work, we focus on deep convolutional neural networks and demonstrate that adversaries can easily craft adversarial examples even without any internal knowledge of the target network. Our attacks treat the network as an oracle (black-box) and only assume that the output of the network can be observed on the probed inputs. Our first attack is based on a simple idea of adding perturbation to a randomly selected single pixel or a small set of them. We then improve the effectiveness of this attack by carefully constructing a small set of pixels to perturb by using the idea of greedy local-search. Our proposed attacks also naturally extend to a stronger notion of misclassification. Our extensive experimental results illustrate that even these elementary attacks can reveal a deep neural network's vulnerabilities. The simplicity and effectiveness of our proposed schemes mean that they could serve as a litmus test while designing robust networks.


TL;DR: Simple, but highly effective, adversarial attacks on deep neural networks even in the absence of any internal knowledge about the network

Review discussion:
The paper seems to be an incremental twist on the already large field of adversarial images. What's the big advantage?
At top of page for (end of first paragraph) you mention that one advantage is that you don't rely on the transferability assumption and that that makes it much more applicable. How so? Can you give examples?
What do you mean by a stronger notion of missclassification?
1) Understanding the robustness of modern computer vision techniques to adversarial inputs is a fundamental challenge for building practical AI systems.
While there exist quite a few white-box adversarial attacks (where the adversary has detailed knowledge of the network architecture and its parameters), the situation in the black-box setting was less well understood.  Note that black-box model is a much more realistic and applicable threat model in practice. We make important progress on this problem by proposing black-box attacks that are highly effective in generating adversarial images with little perturbation. One of main takeaway from our results is that, perhaps worryingly, it is much easier to generate adversarial images even in the harder black-box setting than was previously known.
2) While empirical evidence exists for the transferability assumption, it should be noted that there is a degradation in effectiveness in the attacks when relying on this assumption, and in some cases this degradation can be upwards of 30% (see Section 5 in Papernot et al., “Practical Black-Box Attacks against Deep Learning Systems”, 2016).  A very recent result (Liu et al., “Delving into Transferable Adversarial Examples and Black-box Attacks”, 2016) has highlighted that this assumption should be treated carefully.
Also, previously proposed black-box attacks that relied on this assumption have an additional overhead of gathering data and training a substitute network. By crafting adversarial images directly for a target network, our proposed attacks overcome these above shortcomings.
3) A feature of our attack is that it can be carried even on a network that is based on top-k classification. We utilize a stronger notion of misclassification (that we refer to as k-misclassification), where the goal of an adversary, given an image, is to craft an adversarial image on which even the top-k predictions of the network fails to capture the true label of the original image.
While the standard setting with k=1 has been investigated before, to the best of our knowledge, these are the first attacks which consider this stronger misclassification notion with k > 1.
We will modify the discussion in the paper to clarify these points.
The notion of an adversarial example here appears to be somewhat weak in that it need only cause a misclassification (to any other class) rather than a misclassification to a particular desired class.  It is perhaps not surprising that some form of locally corrupted input can produce misclassification.
But if the result of misclassification (target class) cannot be controlled, is there a realistic scenario under which this would be a security threat?
The previous adversarial attacks in the literature have focused on both untargeted and targeted misclassification.
It should be noted that in a binary classification setting these two notions are the same (e.g., consider an image spam detection system that is deciding whether an image is spam/not spam). Even in the multi-class setting, an adversary that succeeds in effective untargeted misclassification is already very powerful (e.g., consider an adversary that can fool an autonomous driving system into not following posted traffic signs). Many such practical scenarios exist where any misclassification attack can lead to a serious security threat. We would also like to point that we achieve a stronger notion of untargeted misclassification than previously considered, which we refer to as k-misclassification, where the adversary ensures that it fools even a network that relies on top-k classification accuracy. To the best of our knowledge, ours are the first adversarial attacks on deep neural networks achieving k-misclassification for k >1.
Also, while an adversary can easily perform a local perturbation in selected pixels with access to the gradient information, our results, somewhat surprisingly, demonstrates that local perturbations schemes are very effective even in the harder black-box setting where the adversary has no knowledge about the network or its gradient.
The authors propose a method to generate adversarial examples w/o relying on knowledge of the network architecture or network gradients.
The idea has some merit, however, as mentioned by one of the reviewers, the field has been studied widely, including black box setups.
My main concern is that the first set of experiments allows images that are not in image space. The authors acknowledge this fact on page 7 in the first paragraph. In my opinion, this renders these experiments completely meaningless. At the very least, the outcome is not surprising to me at all.
The greedy search procedure remedies this issue. The description of the proposed method is somewhat convoluted. AFAICT, first a candidate set of pixels is generated by using PERT.
Then the pixels are perturbed using CYCLIC.
It is not clear why this approach results in good/minimal perturbations as the candidate pixels are found using a large "p" that can result in images outside the image space. The choice of this method does not seem to be motivated by the authors.
In conclusion, while the authors to an interesting investigation and propose a method to generate adversarial images from a black-box network, the overall approach and conclusions seem relatively straight forward. The paper is verbosely written and I feel like the findings could be summarized much more succinctly.
We thank the reviewer for the suggestions about writing. As we discuss in the paper, the results in Section 4 (first set of experiments) are not proposed as a standalone practical attack, but to understand the robustness of modern deep learning networks to single pixel perturbations. To the best of our knowledge, this is the first study investigating the robustness of deep networks along this direction. We build on findings from this section to propose our main local-search based algorithm (Section 5), which always generates adversarial images in the original image space. The PERT procedure helps in identifying the highly salient pixels that provides an implicit approximation to the gradient, whereas the CYCLIC procedure performs the actual perturbation by adding very little noise to the selected pixels.
In our opinion, simplicity of our approach should be considered as a virtue (as is common in the security literature), as it reflects on the ease of carrying out an effective adversarial attack even for black-box scenarios. Our findings lead to a better understanding on the (lack of) robustness of deep networks to simple perturbation schemes.
The paper presents a method for generating adversarial input images for a convolutional neural network given only black box access (ability to obtain outputs for chosen inputs, but no access to the network parameters).
However, the notion of adversarial example is somewhat weakened in this setting: it is k-misclassification (ensuring the true label is not a top-k output), instead of misclassification to any desired target label.
A similar black-box setting is examined in Papernot et al. (2016c).  There, black-box access is used to train a substitute for the network, which is then attacked.  Here, black-box access in instead exploited via local search.  The input is perturbed, the resulting change in output scores is examined, and perturbations that push the scores towards k-misclassification are kept.
A major concern with regard to novelty is that this greedy local search procedure is analogous to gradient descent; a numeric approximation (observe change in output for corresponding change in input) is used instead of backpropagation, since one does not have access to the network parameters.
As such, the greedy local search algorithm itself, to which the paper devotes a large amount of discussion, is not surprising and the paper is fairly incremental in terms of technical novelty.
The reviewer is correct that the local search produces a numerical approximation to the gradient, an observation that we also state in this paper. This fact is also highlighted by our experiments that show our approach successfully identifies those pixels in an image that have the largest influence on the outcome of the network.
However, a priori it is not obvious how one could efficiently use local search to find a good approximation of the gradient. There is no direct prior work that we are aware of achieving this.
In a non-binary classification setting, targeted and k-misclassification notions are irreducible to each other so one of them is not stronger than the other. In a binary setting both notions are identical. In most practical scenarios any misclassification notion will suffice. We will also like to point that our methods can be adapted to obtain targeted misclassification. To do so, in the local-search procedure, we replace our objective function that minimizes Pr(Image = true label)  to an objective function that maximizes Pr(Image = target label). The local search procedure itself remains identical. We plan to include these results in the revised version of the paper.
Paper summary:
This work proposes a new algorithm to generate k-adversarial images by modifying a small fraction of the image pixels and without requiring access to the classification network weight.
Review summary:
The topic of adversarial images generation is of both practical and theoretical interest. This work proposes a new approach to the problem, however the paper suffers from multiple issues. It is too verbose (spending long time on experiments of limited interest); disorganized (detailed description of the main algorithm in sections 4 and 5, yet a key piece is added in the experimental section 6); and more importantly the resulting experiments are of limited interest to the reader, and the main conclusions are left unclear.
This looks like an interesting line of work that has yet to materialize in a good document, it would need significant re-writing to be in good shape for ICLR.
Pros:
* Interesting topic
* Black-box setup is most relevant
* Multiple experiments
* Shows that with flipping only 1~5% of pixels, adversarial images can be created
Cons:
* Too long, yet key details are not well addressed
* Some of the experiments are of little interest
* Main experiments lack key measures or additional baselines
* Limited technical novelty
Quality: the method description and experimental setup leave to be desired.
Clarity: the text is verbose, somewhat formal, and mostly clear;
but could be improved by being more concise.
Originality: I am not aware of another work doing this exact same type of experiments.
However the approach and results are not very surprising.
Significance: the work is incremental, the issues in the experiments limit potential impact of this paper.
Specific comments:
* I would suggest to start by making the paper 30%~40% shorter. Reducing the text length, will force to make the argumentation and descriptions more direct, and select only the important experiments.
* Section 4 seems flawed. If the modified single pixel can have values far outside of the [LB, UB] range;
then this test sample is clearly outside of the training distribution; and thus it is not surprising that the classifier misbehaves (this would be true for most classifiers, e.g. decision forests or non-linear SVMs). These results would be interesting only if the modified pixel is clamped to the range [LB, UB].
* [LB, UB] is never specified, is it ? How does p = 100, compares to [LB, UB] ? To be of any use, p should be reported in proportion to [LB, UB]
* The modification is done after normalization, is this realistic ?
* Alg 2, why not clamping to [LB, UB] ?
* Section 6, “implementing algorithm LocSearchAdv”, the text is unclear on how p is adjusted; new variables are added. This is confusion.
* Section 6, what happens if p is _not_ adjusted ? What happens if a simple greedy random search is used (e.g. try 100 times a set of 5 random pixels with value 255) ?
* Section 6, PTB is computed over all pixels ?
including the ones not modified ? why is that ?
Thus LocSearchAdv PTB value is not directly comparable to FGSM, since it intermingles with #PTBPixels (e.g. “in many cases far less average perturbation” claim).
* Section 6, there is no discussion on the average number of model evaluations. This would be equivalent to the number of requests made to a system that one would try to fool. This number is important to claim the “effectiveness” of such black box attacks. Right now the text only mentions the upper bound of 750 network evaluations.
* How does the number of network evaluations changes when adjusting or not adjusting p during the optimization ?
* Top-k is claimed as a main point of the paper, yet only one experiment is provided. Please develop more, or tune-down the claims.
* Why is FGSM not effective for batch normalized networks ? Has this been reported before ? Are there other already published techniques that are effective for this scenario ? Comparing to more methods would be interesting.
* If there is little to note from section 4 results, what should be concluded from section 6 ?
That is possible to obtain good results by modifying only few pixels ? What about selecting the “top N” largest modified pixels from FGSM ? Would these be enough ? Please develop more the baselines, and the specific conclusions of interest.
Minor comments:
* The is an abuse of footnotes, most of them should be inserted in the main text.
* I would suggest to repeat twice or thrice the meaning of the main variables used (e.g. p, r, LB, UB)
* Table 1,2,3 should be figures
* Last line of first paragraph of section 6 is uninformative.
* Very tiny -> small
We thank the reviewer for suggestions about writing and will work on making the paper more concise. Below we address some of the major points raised by the reviewer:
1) The normalized images are in the range [-2,2] (LB=-2, UB=2). As we discuss in the paper, the results in Section 4 are not proposed as a standalone practical attack, but to understand the general robustness of modern deep learning networks. Our results show that the robustness is an issue in these networks even under single pixel perturbation. This effect is already observed with small values of p, and we vary p to large values just to get the complete picture. For a small p value, while the original image (training) and the adversarial image (test) distributions are not identical, they are also not too far apart as we perturb just one pixel (for low-resolution images).
In fact, one can analytically bound the distance between the original image distribution and adversarial image distribution under some assumptions (e.g., if the original images are drawn from multivariate Gaussian).
2) Normalization procedures are generally standard, so we assume that the adversary can carry out them.
3) A simple clamping approach in Algorithm 2 will generate pixels whose values are fixed to either LB or UB, whereas with our cyclic rounding approach we noticed that we get pixel values that are closer (in absolute sense) to their original values.
4) While adaptively changing p is not necessary, we noticed that it improves overall performance by both increasing the adversarial image generation success rate and decreasing the perturbation applied per image. We will clarify the discussion here.
5) PTB measures the mean absolute error between the image and its adversarial counterpart over successful adversarial images. The mean absolute error is typically computed over all pixels, hence our choice. We will change the description to clarify this.
6) We would add the average network evaluation numbers for local-search. We thank the reviewer for this suggestion.
7) While batch normalization reduces the effectiveness of both our scheme and the FGSM scheme, we noticed that the degradation for the FGSM scheme was significantly more prominent. We are not aware of previous results in the adversarial image generation literature that have factored in the effects of batch normalization.
8) Since the FGSM scheme adds same level of perturbation to all pixels, it is not possible to identify the top-N largest modified pixels here. We will work on creating more baselines. Note that black-box adversarial image generation is a new direction of research with limited previous literature.

Meta-review:
role: PC
data: comment: While this is an interesting topic, both the method description and experimental setup could be improved.

Decision: Reject