Title: Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models
Authors: Wieland Brendel *
Jonas Rauber *
Matthias Bethge
Venue: ICLR 2018
Keywords: adversarial attacks
adversarial examples
adversarials
robustness
security

Abstract: Many machine learning algorithms are vulnerable to almost imperceptible perturbations of their inputs. So far it was unclear how much risk adversarial perturbations carry for the safety of real-world machine learning applications because most methods used to generate such perturbations rely either on detailed model information (gradient-based attacks) or on confidence scores such as class probabilities (score-based attacks), neither of which are available in most real-world scenarios. In many such cases one currently needs to retreat to transfer-based attacks which rely on cumbersome substitute models, need access to the training data and can be defended against. Here we emphasise the importance of attacks which solely rely on the final model decision. Such decision-based attacks are (1) applicable to real-world black-box models such as autonomous cars, (2) need less knowledge and are easier to apply than transfer-based attacks and (3) are more robust to simple defences than gradient- or score-based attacks. Previous attacks in this category were limited to simple models or simple datasets. Here we introduce the Boundary Attack, a decision-based attack that starts from a large adversarial perturbation and then seeks to reduce the perturbation while staying adversarial. The attack is conceptually simple, requires close to no hyperparameter tuning, does not rely on substitute models and is competitive with the best gradient-based attacks in standard computer vision tasks like ImageNet. We apply the attack on two black-box algorithms from Clarifai.com. The Boundary Attack in particular and the class of decision-based attacks in general open new avenues to study the robustness of machine learning models and raise new questions regarding the safety of deployed machine learning systems. An implementation of the attack is available as part of Foolbox (https://github.com/bethgelab/foolbox).

TL;DR: A novel adversarial attack that can directly attack real-world black-box machine learning models without transfer.

Review discussion:
The authors identify a new security threat for deep learning: Decision-based adversarial attacks. This new class of attacks on deep learning systems requires from an attacker only the knowledge of class labels (previous attacks required more information, e.g., access to a gradient oracle). Unsurprisingly, since the attacker has so few information, such kind of attacks involves quite a lot trial and error. The authors propose one specific attack instance out of this class of attacks. It works as follows.
First, an initial point outside of the benign region is guessed.
Then multiple steps towards the decision boundary is taken, finally reaching the boundary (I am not sure about the precise implementation, but it seems not crucial; the author may please check whether their description of the algorithm is really reproducable).
Then, in a nutshell, a random walk on a sphere centered around the original, benign point is performed, where after each step, the radius of the sphere is slightly reduced (drawing the point closer to the original point), if and only if the resulting point still is outside of the benign region.
The algorithm is evaluated on the following datasets: MNIST, CIFAR, VGG19, ResNet50, and InceptionV3.
The paper is rather well written and structured. The text was easy to follow. I suggest that a self-contained description of the problem setting (assumptions on attacker and defender; aim?) shall be added to the camera-ready version (being not familiar with the area, I had to read a couple of papers to get a feeling for the setting, before reviewing this paper). As in many DL papers these days, there really isn't any math in it worth a mention;
so no reason here to say anything about mathematical soundness. The authors employ a reasonable evaluation criterion in their experiments: the median squared Euclidean distance between the original and adversarially modified data point. The results show consistent improvement for most data sets.
In summary, this is an innovative paper, proposing a new class of attacks that totally makes sense in my opinion.
Apart from some minor weaknesses in the presentation that can be easily fixed for the camera ready, this is a nice, fresh paper, that might spur more attacks (and of course new defenses) from the new class of decision-based attacks. It is worth to note that the authors show that distillation is not a useful defense against such attacks, so we may expect follow-up proposing useful defenses against the new attack (which BTW is shown to be about a factor of 10 in terms of iterations more costly than the SOTA).
Thanks a lot for your positive review!
We appreciate your nice summary of the attack. To find the first point „on“ (i.e. close to) the boundary, we just perform a line search between the initial point and the original. As you said, this is indeed not crucial and it would be perfectly fine to leave out this step and directly follow the actual algorithm of the Boundary Attack. We use the line search because it basically just gives us a better starting point and therefore speeds up the beginning of the attack.
Regarding your first sentence, we’d like to add that the Boundary Attack does not just pose a new threat, it also makes measuring the robustness of models more reliable. In the past virtually all defence strategies proposed in the literature did not actually increase the robustness of the model but only disabled the attack strategy. A prime example is gradient masking where the backpropagated gradient needed in gradient-based attacks is simply zero (e.g. defensive distillation). The Boundary Attack is robust to many such nuisances and will make it easier to spot whether a model is truly robust.
We improved the introduction to make the paper more self-contained and thus easier to to read for people not familiar with the area. We’d appreciate your feedback if something is still unclear.
This is a nice paper proposing a simple but effective heuristic for generating adversarial examples from class labels with no gradient information or class probabilities. Highly relevant prior work was overlooked and there is no theoretical analysis, but I think this paper still makes a valuable contribution worth sharing with a broader audience.
What this paper does well:
- Suggests a type of attack that hasn't been applied to image classifiers
- Proposes a simple heuristic method for performing this attack
- Evaluates the attack on both benchmark neural networks and a commercial system
Problems and limitations:
1. No theoretical analysis. Under what conditions does the boundary attack succeed or fail? What geometry of the classification boundaries is necessary? How likely are those conditions to hold? Can we measure how well they hold on particular networks?
Since there is no theoretically analysis, the evidence for effectiveness is entirely empirical. That weakens the paper and suggests an important area of future work, but I think the empirical evidence is sufficient to show that there's something interesting going. Not a fatal flaw.
2. Poor framing. The paper frames the problem in terms of "machine learning models" in general (beginning with the first line of the abstract), but it only investigates image classification. There's no particular reason to believe that all machine learning algorithms will behave like convolutional neural network image classifiers.
Thus, there's an implicit claim of generality that is not supported.
This is a presentation issue that is easily fixed. I suggest changing the title to reflect this, or at least revising the abstract and introduction to make the scope clearer.
A minor presentation quibble/suggestion: "adversarial" is used in this paper to refer to any class that differs from the true class of the instance to be disguised.
But an image of a dalmation that's labeled as a dalmation isn't adversarial -- it's just a different image that's labeled correctly. The adversarial process is about constructing something that will be mislabeled, exploiting some kind of weakness that doesn't show up on a natural distribution of inputs. I suggest rewording some of the mentions of adversarial.
3. Ignorance of prior work. Finding deceptive inputs using only the classifier output has been done by Lowd and Meek (KDD 2005) for linear classifiers and Nelson et al. (AISTATS 2010, JMLR 2012) for convex-inducing classifiers. Both works include theoretical bounds on the number of queries required for near-optimal adversarial examples. Biggio et al. (ECML 2013) further propose training a surrogate classifier on similar training data, using the predictions of the target classifier to relabel the training data. In this way, decision information from the target model is used to help train a more similar surrogate, and then attacks can be transferred from the surrogate to the target.
Thus, "decision-based attacks" are not new, although the algorithm and experiments in this paper are.
Overall, I think this paper makes a worthwhile contribution, but needs to revise the claims to match what's done in the paper and what's been done before.
Thanks a lot for your insightful comments!
1. We agree that a theoretical analysis of the decision boundaries of neural networks is an important area of future work. One assumption of the attack is that the boundary is fairly smooth, otherwise the linearity assumption in the proximity of the current adversarial wouldn’t hold. Other then that the success of the attack very much depends on the number and distribution of local minimas on the boundary, but that’s true for all attacks.
So just as the success of stochastic gradient descent is first and foremost an empirical result, so is the success of the Boundary Attack and it will be challenging (yet interesting) to get better theoretical insights in the future.
2. The Boundary Attack is by no means restricted to CNNs.
However, virtually all papers on adversarial attacks from recent years were evaluated on CNNs in computer vision tasks and so we followed this setup in order to be as accountable as possible.
Of course, we do not claim that the Boundary Attack will work equally well on all Machine Learning algorithms, but that’s something no attack can legitimately claim (including gradient-based attacks).
In other words, the Boundary Attack is in principle applicable to any machine learning algorithm but how well it will perform is an empirical question that future work will highlight. We’d thus choose to leave the title as is in order to stimulate more progress in this direction.
3. We thank you for these pointers to relevant prior work which have prompted us to adapt the claims of the manuscript accordingly. More concretely, we now delineate our work more clearly as the first decision-based attack that scales to complex machine learning algorithms (such as DNNs) and complex data sets (such as ImageNet).
We also added more explanations as to our definition of an “adversarial”.
In this paper, the authors propose a novel method for generating adversarial examples when the model is a black-box and we only have access to its decisions (and a positive example).  It iteratively takes steps along the decision boundary while trying to minimize the distance to the original positive example.
Pros:
- Novel method that works under much stricter and more realistic assumptions.
- Fairly thorough evaluation.
- The paper is clearly written.
Cons:
- Need a fair number of calls to generate a small perturbation.  Would like to see more analysis of this.
- Attack works for making something outside the boundary (not X), but is less clear how to generate image to meet a specific classification (X).  3.2 attempts this slightly by using an image in the class, but is less clear for something like FaceID.
- Unclear how often the images generated look reasonable.  Do different random initializations given different quality examples?
Thanks a lot for the positive review! We have added two new figures to address your comments (figures 5 and 6).
- It is true that the Boundary Attack needs a fair number of calls until it converges to the absolute minimal perturbation. In most practical scenarios, however, it is sufficient to generate adversarials that are perceptually indistinguishable from the original image in which case a much smaller number of iterations is necessary. We added figure 6 to show how fast the size of the perturbation is reduced by the Boundary Attack.
- The Boundary Attack is made to find minimal perturbations from a given image that is misclassified (or classified as a certain target). More graphically, it tries to find an image that is classified as a bus but clearly looks like a cat. In the case you mentioned - Face ID - one would try to find any image that is classified as a certain target.
In other words: you’d try to find an image that is classified as a bus, and it’s completely fine if it also looks like a bus to humans. That’s a totally different threat scenario that we do not consider in the paper.
- All results shown in the paper are for a single run of the Boundary Attack, there is no cherry picking. To make this point clearer we added figure 5 where we show the result of ten runs of the Boundary Attack on the same image with random initialization. The final perturbation ends up in one of two minima, and both are of similar quality (the size of the perturbations varies only by a factor of 2).
We have uploaded an updated version of the paper with the following changes:
1.) The Boundary attack has now been run to full convergence for all experiments reported in the paper. On some experiments, in particular on ImageNet, the performance considerably increased and now places the attack squarely between DeepFool and Carline & Wagner in all setups.
2.) We added a new figure showing the result of the Boundary Attack for repeated runs on the same sample. The attack converges to only two minima with similar distance to the original image (Figure 5).
3.) We added a new figure showing the distance to the original image as a function of model calls (Figure 6). The figure further highlights that relatively few model calls are already sufficient to find excellent adversarial examples. A large number of calls is only necessary to fully converge.
4.) We improved the introduction to be more self-contained and easier to read and added additional references to prior work.

Meta-review:
role: PC
data: comment: The reviewers all agree this is a well written and interesting paper describing a novel black box adversarial attack.   There were missing relevant references in the original submission, but these have been added.  I would suggest the authors follow the reviewer suggestions on claims of generality beyond CNN; although there may not be anything obvious stopping this method from working more generally, it hasn't been tested in this work.   Even if you keep the title you might be more careful to frame the body in the context of CNN's.

Decision: Accept (Poster)