user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Exploring the Space of Black-box Attacks on Deep Neural Networks

Paper content:
Existing black-box attacks on deep neural networks (DNNs) so far have largely focused on transferability, where an adversarial instance generated for a locally trained model can “transfer” to attack other learning models.
In this paper, we propose novel Gradient Estimation black-box attacks for adversaries with query access to the target model’s class probabilities, which do not rely on transferability.
We also propose strategies to decouple the number of queries required to generate each adversarial sample from the dimensionality of the input.
An iterative variant of our attack achieves close to 100% adversarial success rates for both targeted and untargeted attacks on DNNs.
We carry out extensive experiments for a thorough comparative evaluation of black-box attacks and show that the proposed Gradient Estimation attacks outperform all transferability based black-box attacks we tested on both MNIST and CIFAR-10 datasets, achieving adversarial success rates similar to well known, state-of-the-art white-box attacks.
We also apply the Gradient Estimation attacks successfully against a real-world content moderation classiﬁer hosted by Clarifai.
Furthermore, we evaluate black-box attacks against state-of-the-art defenses.
We show that the Gradient Estimation attacks are very effective even against these defenses.
The ubiquity of machine learning provides adversaries with both opportunities and incentives to develop strategic approaches to fool learning systems and achieve their malicious goals.
Many attack strategies devised so far to generate adversarial examples to fool learning systems have been in the white-box setting, where adversaries are assumed to have access to the learning model BID18 ; BID0 ; BID1 ; BID6 ).
However, in many realistic settings, adversaries may only have black-box access to the model, i.e. they have no knowledge about the details of the learning system such as its parameters, but they may have query access to the model's predictions on input samples, including class probabilities.
For example, we find this to be the case in some popular commercial AI offerings, such as those from IBM, Google and Clarifai.
With access to query outputs such as class probabilities, the training loss of the target model can be found, but without access to the entire model, the adversary cannot access the gradients required to carry out white-box attacks.Most existing black-box attacks on DNNs have focused on transferability based attacks BID12 ; BID7 ; BID13 ), where adversarial examples crafted for a local surrogate model can be used to attack the target model to which the adversary has no direct access.
The exploration of other black-box attack strategies is thus somewhat lacking so far in the literature.
In this paper, we design powerful new black-box attacks using limited query access to learning systems which achieve adversarial success rates close to that of white-box attacks.
These black-box attacks help us understand the extent of the threat posed to deployed systems by adversarial samples.
The code to reproduce our results can be found at https://github.com/ anonymous 1 .New
black-box attacks. We
propose novel Gradient Estimation attacks on DNNs, where the adversary is only assumed to have query access to the target model. These
attacks do not need any access to a representative dataset or any knowledge of the target model architecture. In the
Gradient Estimation attacks, the adversary adds perturbations proportional to the estimated gradient, instead of the true gradient as in white-box attacks BID0 ; Kurakin et al. (2016) ). Since
the direct Gradient Estimation attack requires a number of queries on the order of the dimension of the input, we explore strategies for reducing the number of queries to the target model. We also
experimented with Simultaneous Perturbation Stochastic Approximation (SPSA) and Particle Swarm Optimization (PSO) as alternative methods to carry out query-based black-box attacks but found Gradient Estimation to work the best.Query-reduction strategies We propose two strategies: random feature grouping and principal component analysis (PCA) based query reduction. In our
experiments with the Gradient Estimation attacks on state-of-the-art models on MNIST (784 dimensions) and CIFAR-10 (3072 dimensions) datasets, we find that they match white-box attack performance, achieving attack success rates up to 90% for single-step attacks in the untargeted case and up to 100% for iterative attacks in both targeted and untargeted cases. We achieve
this performance with just 200 to 800 queries per sample for single-step attacks and around 8,000 queries for iterative attacks. This is much
fewer than the closest related attack by . While they achieve
similar success rates as our attack, the running time of their attack is up to 160× longer for each adversarial sample (see Appendix I.6).A further advantage
of the Gradient Estimation attack is that it does not require the adversary to train a local model, which could be an expensive and complex process for real-world datasets, in addition to the fact that training such a local model may require even more queries based on the training data.Attacking real-world systems. To demonstrate the
effectiveness of our Gradient Estimation attacks in the real world, we also carry out a practical black-box attack using these methods against the Not Safe For Work (NSFW) classification and Content Moderation models developed by Clarifai, which we choose due to their socially relevant application. These models have
begun to be deployed for real-world moderation BID4 , which makes such black-box attacks especially pernicious. We carry out these
attacks with no knowledge of the training set. We have demonstrated
successful attacks ( FIG0 ) with just around 200 queries per image, taking around a minute per image. In FIG0 , the target
model classifies the adversarial image as 'safe' with high confidence, in spite of the content that had to be moderated still being clearly visible. We note here that due
to the nature of the images we experiment with, we only show one example here, as the others may be offensive to readers. The full set of images
is hosted anonymously at https://www.dropbox.com/s/ xsu31tjr0yq7rj7/clarifai-examples.zip?dl=0.Comparative evaluation
of black-box attacks. We carry out a thorough empirical
comparison of various black-box attacks (given in TAB8 ) on both MNIST and CIFAR-10 datasets. We study attacks that require zero
queries to the learning model, including the addition of perturbations that are either random or proportional to the difference of means of the original and targeted classes, as well as various transferability based black-box attacks. We show that the proposed Gradient
Estimation attacks outperform other black-box attacks in terms of attack success rate and achieve results comparable with white-box attacks.In addition, we also evaluate the effectiveness of these attacks on DNNs made more robust using adversarial training BID0 BID18 and its recent variants including ensemble adversarial training BID21 and iterative adversarial training BID9 . We find that although standard and
ensemble adversarial training confer some robustness against single-step attacks, they are vulnerable to iterative Gradient Estimation attacks, with adversar-ial success rates in excess of 70% for both targeted and untargeted attacks. We find that our methods outperform
other black-box attacks and achieve performance comparable to white-box attacks.Related Work. Existing black-box attacks that do
not use a local model were first proposed for convex inducing two-class classifiers by BID11 . For malware data, Xu et al. (2016)
use genetic algorithms to craft adversarial samples, while Dang et al. (2017) use hill climbing algorithms. These methods are prohibitively expensive
for non-categorical and high-dimensional data such as images. BID13 proposed using queries to a target
model to train a local surrogate model, which was then used to to generate adversarial samples. This attack relies on transferability.To
the best of our knowledge, the only previous literature on query-based black-box attacks in the deep learning setting is independent work by BID10 and . BID10 propose a greedy local search to generate
adversarial samples by perturbing randomly chosen pixels and using those which have a large impact on the output probabilities. Their method uses 500 queries per iteration, and
the greedy local search is run for around 150 iterations for each image, resulting in a total of 75,000 queries per image, which is much higher than any of our attacks. Further, we find that our methods achieve higher
targeted and untargeted attack success rates on both MNIST and CIFAR-10 as compared to their method. propose a black-box attack method named ZOO, which
also uses the method of finite differences to estimate the derivative of a function. However, while we propose attacks that compute an
adversarial perturbation, approximating FGSM and iterative FGS; ZOO approximates the Adam optimizer, while trying to perform coordinate descent on the loss function proposed by BID1 . Neither of these works demonstrates the effectiveness
of their attacks on real-world systems or on state-of-the-art defenses.
Overall, in this paper, we conduct a systematic analysis of new and existing black-box attacks on state-of-the-art classifiers and defenses.
We propose Gradient Estimation attacks which achieve high attack success rates comparable with even white-box attacks and outperform other state-of-the-art black-box attacks.
We apply random grouping and PCA based methods to reduce the number of queries required to a small constant and demonstrate the effectiveness of the Gradient Estimation attack even in this setting.
We also apply our black-box attack against a real-world classifier and state-of-the-art defenses.
All of our results show that Gradient Estimation attacks are extremely effective in a variety of settings, making the development of better defenses against black-box attacks an urgent task.Stephen

assistant: text: Query-based black-box attacks on deep neural networks with adversarial success rates matching white-box attacks