Title: QUERY-EFFICIENT META ATTACK TO DEEP NEURAL NETWORKS

Abstract: Black-box attack methods aim to infer suitable attack patterns to targeted DNN models by only using output feedback of the models and the corresponding input queries. However, due to lack of prior and inefficiency in leveraging the query and feedback information, existing methods are mostly query-intensive for obtaining effective attack patterns. In this work, we propose a meta attack approach that is capable of attacking a targeted model with much fewer queries. Its high queryefficiency stems from effective utilization of meta learning approaches in learning generalizable prior abstraction from the previously observed attack patterns and exploiting such prior to help infer attack patterns from only a few queries and outputs. Extensive experiments on MNIST, CIFAR10 and tiny-Imagenet demonstrate that our meta-attack method can remarkably reduce the number of model queries without sacrificing the attack performance. Besides, the obtained meta attacker is not restricted to a particular model but can be used easily with a fast adaptive ability to attack a variety of models. The code of our work is available at https://github.com/dydjw9/MetaAttack_ICLR2020/.

1. INTRODUCTION:

Despite the great success in various tasks, deep neural networks (DNNs) are found to be susceptible to adversarial attacks and often suffer dramatic performance degradation in front of adversarial examples, even if only tiny and invisible noise is imposed on the input (Szegedy et al., 2014) . To investigate the safety and robustness of DNNs, many adversarial attack methods have been developed, which apply to either a white-box (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017; Madry et al., 2018) or a black-box setting (Papernot et al., 2017; Brendel et al., 2018; Narodytska & Kasiviswanathan, 2017) . In the white-box attack setting, the target model is transparent to the attacker and imperceptible adversarial noise can be easily crafted to mislead this model by leveraging its gradient information (Goodfellow et al., 2015) . In contrast, in the blackbox setting, the structure and parameters of the target DNN model are invisible, and the adversary can only access the input-output pair in each query. With a sufficient number of queries, black-box methods utilize the returned information to attack the target model generally by estimating gradient (Chen et al., 2017; Ilyas et al., 2018a; Narodytska & Kasiviswanathan, 2017; Cheng et al., 2019) .

Black-box attack is more feasible in realistic scenarios than white-box attack but it is much more query-intensive. Such a drawback is largely attributed to the fact that returned information for each queried example is sparse and limited. During inferring attack patterns, existing black-box methods simply integrate the information between two sequential iterations brutally and ignore the implicit but profound message, thus not fully exploiting the returned information. Although query-efficient algorithms for generating attack examples are very meaningful in practice (Ilyas et al., 2018a) , how to enhance query-efficiency for black-box attack remains underexplored.

In this work, we address a query-efficiency concerned attack problem. Particularly, we consider only top-k probability scores accessible from the target black-box model. With this practical but challenging scenario, we aim at three important objectives: lower query number, higher success rate and smaller noise magnitude. We develop a meta-learning based attack method, which applies meta learning to obtaining prior information from the successful attack patterns, and uses the prior for efficient optimization. Specifically, we propose to train a meta attacker model through meta learning (Nichol et al., 2018) , inspired by its success in solving few-shot learning problems. We first deploy several existing classification models to get pairs of (images, gradients) with the max-margin logit classification loss. Then we use the data pairs of each classification model to train the meta attacker. After obtaining the attacker, we use it to attack a new black-box model for accelerating the search process for adversarial examples by optimizing it with coordinate-wise gradient estimation. Different from previous methods, we use the estimated gradient not only to update adversarial noise but to fine-tune the well-trained attacker. After few-shot fine-tuning, the attacker is able to simulate the gradient distribution of the target model.

We evaluate our method on MNIST, CIFAR10 and tiny-ImageNet datasets by comparing it with state-of-the-art black-box attack methods including Zoo (Chen et al., 2017) , Decision-Boundary (Brendel et al., 2018) , AutoZoom (Tu et al., 2019) , Opt-attack (Cheng et al., 2019) and Bandits (Ilyas et al., 2018b) . In both targeted and untargeted settings, our proposed method achieves comparable attack success rate and adversarial perturbation to all baselines but with a significantly reduced query number. The detailed experiment results demonstrate our superior query-efficiency.

2. RELATED WORK:

Classical white-box attack methods include Fast-Gradient Sign Method (FGSM) (Goodfellow et al., 2015) , IFGSM (Madry et al., 2018 ), DeepFool (Moosavi-Dezfooli et al., 2016) and C&W attack (Carlini & Wagner, 2017) , following a setting where detailed information about the target model (gradients and losses) is provided. Comparatively, the black-box setting better accords with the real world scenarios in that little information about the target model is visible to the attacker. The pioneer work on black-box attack (Papernot et al., 2017) tries to construct a substitute model with augmented data and transfer the black-box attack problem to a white-box one. However, its attack performance is very poor due to the limited transferability of adversarial examples between two different models. (Brendel et al., 2018) considers a more restricted case where only top-1 prediction classes are returned and proposes a random-walk based attack method around the decision boundary. It dispenses class prediction scores and hence requires extensive model queries. Zoo (Chen et al., 2017) is a black-box version of C&W attack, achieving a similar attack success rate and comparable visual quality as many white-box attack methods. However, its coordinate-wise gradient estimation requires extensive model evaluations. More recently, (Ilyas et al., 2018a) proposes a query-limited setting with L ∞ noise considered, and uses a natural evolution strategy (NES) to enhance query efficiency. Though this method successfully controls the query number, the noise imposed is larger than average. (Narodytska & Kasiviswanathan, 2017) proposes a novel local-search based technique to construct numerical approximation to the network gradient, which is then carefully used to construct a small set of pixels in an image to perturb. It suffers a similar problem as in (Chen et al., 2017) for pixel-wise attack. (Cheng et al., 2019) considers a hard-label black-box setting and formulates the problem as real-valued optimization that is solved by a zeroth order optimization algorithm. Ilyas et al. (2018b) reduce the queries by introducing two gradient priors, the time-independent prior and the data-dependent prior, and reformulating the optimization problem.

We then briefly introduce some works on meta-learning related to our work. Meta-learning is a process of learning how to learn. A meta-learning algorithm takes in a distribution of tasks, each being a learning problem, and produces a quick learner that can generalize from a small number of examples. Meta-learning is very popular recently for its fast adaptive ability. MAML (Finn et al., 2017) is the first to propose this idea. Recently, a simplified algorithm Reptile (Nichol et al., 2018) which is an approximation to the first-order MAML is proposed, achieving higher efficiency in computation and consuming less memory. With these superior properties, meta learning is applied to adversarial attack methods (Zgner & Gnnemann, 2019; Edmunds et al., 2017) . Zgner & Gnnemann (2019) try to attack the structure of a graph model in the training process to decrease the model generalization performance. Edmunds et al. (2017) investigate the susceptibility of MAML to adversarial attacks and the transferability of the obtained meta model to a specific task.

3. METHOD:



3.1. PRELIMINARIES: BLACK-BOX ATTACK SCHEMES:

We first formulate the black-box attack problem and introduce the widely used solutions. We use (x, t) to denote the pair of a natural image and its true label, and x and t to denote the adversarial perturbed version of x and the returned label by the target classification model M tar . The black-box attack aims to find an adversarial example x with imperceivable difference from x to fail the target model , i.e., t = t through querying the target model for multiple times. It can be formulated as

min x (x, M tar (x), t) s.t. x -x p ≤ ρ, #queries ≤ Q. (1)

Here • p denotes the p norm that measures how much perturbation is imposed. M tar (x) is the returned logit or probability by the target model M tar . The loss function (x, M tar (x), t) measures the degree of certainty for model M tar assigning the input x into class t. One common used adversarial loss is the probability of class t: (x, M tar (x), t) = p Mtar (t|x). The first constraint enforces high similarity between the clean image x and the adversarial one x adv and the second imposes a fixed budget Q for the number of queries allowed in the optimization.

In the white-box attack setting, the adversary can access the true gradient ∇ xt (x t ) and perform gradient descent xt+1 = xt -∇ xt (x t ). But in the black-box setting, the gradient information ∇ xt (x t ) is not attainable. In this case, the attacker can estimate the gradient using only queried information from model evaluation such as hard label, logits and probability scores. This kind of estimator is the backbone of so-called zeroth-order optimization approaches (Chen et al., 2017; Narodytska & Kasiviswanathan, 2017; Tu et al., 2019; Ilyas et al., 2018a; b) . The estimation is done via finite difference method (Chen et al., 2017; Narodytska & Kasiviswanathan, 2017; Tu et al., 2019) , which finds the k components of the gradient by estimating the inner products of the gradients with all the standard basis vector e 1 , ..., e k :

EQUATION

where step size h controls the quality of the estimated gradient. Another strategy is to reformulate the loss function (Ilyas et al., 2018a; b) . Instead of computing the gradient of (x) itself, the expected value of loss function (x) under the search distribution is minimized and when the search distribution of random Guassian noise is adopted, the gradient estimation problem transfers into a zeroth-order estimation problem,

∇E[ (x)] ≈ 1 σn n i=1 (x + σδ i )δ i (3)

where n is the amount of noise sampled from the distribution. After obtaining the estimated gradient, classical optimization algorithms (Nesterov, 2013; Johnson & Zhang, 2013 ) can be used to infer the adversarial examples. Though the estimated gradient may not be accurate, it is still proved useful enough in adversarial attack. The convergence of these zeroth-order methods is guaranteed under mild assumptions (Ghadimi & Lan, 2013; Nesterov & Spokoiny, 2017; Hazan et al., 2016) .

Since each model evaluation consumes a query, naively applying the above gradient estimation to black-box attack is quite query expensive due to its coordinate or noise sampling nature. Take the first strategy on tiny-Imagenet dataset for example. It consumes more than 20,000 queries for each image to obtain a full gradient estimate, which is not affordable in practice. In this work, we address such a limitation via developing a query-efficient meta-learning based attack model.

3.2. LEARNING OF META ATTACKER:

To reduce the query cost for black-box attack, we apply meta learning to training a meta attacker model, inspired by its recent success in few-shot learning problems (Finn et al., 2017; Nichol et al., 2018) . The meta attacker learns to extract useful prior information of the gradient of a variety of models w.r. 

for all T i do 4: Sample K samples from (X, G i ) for training, denoted as (X s , G s i ); 5: Evaluate ∇ θ L i (A θ ) = ∇ θ A θ (X s ) -G s i 2 2 with respect to (X s , G s i ); 6:

Update

θ i := θ -α∇ θ L i (A θ ); 7:

end for 8:

Update θ := θ + 1 n n i=1 (θ i -θ); 9: end while Output: Parameters θ of meta model A.

few queries. After obtaining such a meta attacker, we replace the zeroth-order gradient estimation in traditional black box attack methods with it to directly estimate the gradient.

We collect a set of existing classification models M 1 , ..., M n to generate gradient information for universal meta attacker training. Specifically, we feed each image x into the models M 1 , ..., M n respectively and compute losses 1 , ..., n by using following max-margin logit classification loss:

EQUATION

)

Here t is the groundtruth label and j indexes other classes. [M i (x)] t is the probability score of the true label predicted by the model M i , and [M i (x)] j denotes the probability scores of other classes. By performing one step back-propagation of losses 1 , ..., n w.r.t. the input images x, the corresponding gradients g i = ∇ x i (x), i = 1, ..., n are obtained. Finally, we collect n groups of data X = {x}, G i = {g i }, i = 1, ..., n to train the universal meta attacker.

We design a meta attacker A which has a similar structure as an autoencoder, consisting of symmetric convolution and de-convolution layers and outputs a gradient map with the same size as the input. Meta attacker model A is parameterized with parameters θ.

Due to the intrinsic difference between selected classification models, each obtained set (X, G i ) is treated as a task T i in meta attacker training. During the training process, for each iteration we only draw K samples from task T i and feedback the loss L i to update model parameters from θ to θ i . θ i is then computed through one or multiple gradient descents:

θ i := θ -α∇ θ L i (A θ ).

For a sensitive position of the meta attacker, the meta attacker parameters are optimized by combining each θ i across all tasks {T i } i=1,...,n , following the update strategy of Reptile (Nichol et al., 2018) in meta learning,

EQUATION

We adopt mean-squared error (MSE) as the training loss in the inner update,

EQUATION

)

The set (X s , G s i ) denotes the K samples used for each inner update from θ to θ i . Since the number of K sampled each time is very small, the update strategy above tries to find good meta attacker parameters θ as an initial point, from which the meta attacker model can fast adapt to new data distribution through gradient descent based fine-tuning within limited samples. Therefore, this characteristic can be naturally leveraged in attacking new black-box models by estimating their gradient information through a few queries. Detailed training process of our meta attacker is described in Algorithm 1.

3.3. QUERY-EFFICIENT ATTACK VIA META ATTACKER:

An effective adversarial attack relies on optimizing the loss function equation 1 w.r.t. the input image to find the adversarial example of the target model M tar . Differently, our proposed method applies the meta attacker A to predicting the gradient map of a test image directly.

Algorithm 2 Adversarial Meta Attack Algorithm Input: Test image x 0 with label t, meta attacker A θ , target model M tar , iteration interval m, selected top-q coordinates; 1: for t = 0, 1, 2, ... do 2:

if (t + 1) mod m = 0 then 3:

Perform zeroth-order gradient estimation on top q coordinates, denoted as I t and obtain g t ; 4:

Fine-tune meta attacker A with (x t , g t ) on Generate the gradient map g t directly from meta attacker A with x t , select coordinates I t ; 7: We use the obtained meta attacker model A to predict useful gradient map for attacking, which should be fine tuned to adapt to the new gradient distribution under our new target model. Particularly, instead of finetuning once, for each given image x, we update A by leveraging query information with the following periodic scheme. Suppose the given image is perturbed to x t ∈ R p at iteration t. If (t + 1) mod m = 0, our method performs zeroth-order gradient estimation to obtain gradient map g t for fine-tuning. As each pixel value for estimated gradient map consumes two queries, for further saving queries, we just select q of the p coordinates to estimate, q p, instead of the full gradient map through all p coordinates. The indexes of chosen coordinates are determined by the gradient map g t-1 obtained in iteration t-1. We sort the coordinate indexes by the value of g t-1 and select top-q indexes. The set of these indexes are denoted as I t . We feed image x t in iteration t into meta attacker A and compute the MSE loss on indexes I t , i.e.

I t by loss L = [A θ (x t )] It -[g t ] It

end if 8: Update [x ] It = [x t ] It + β[g t ] It ; 9: if M tar (x ) = t

L = [A θ (x t )] It -[g t ] It 2 2 .

Then we perform gradient descent for the MSE loss with a few steps to update the parameters θ of meta attacker A. For the rest iterations, we just use the periodically updated attacker A θ to directly generate the gradient g t = A θ (x t ). When we have the estimated gradient map g t in iteration t, we update to get the adversarial sample x t by

[x t ] It = [x t ] It + β[g t ] It

where β is a hyperparameter to be tuned. The details are summarized in Algorithm 2.

In our method, the following operations contribute to reducing the query number needed by the attacker. First, though we just use q coordinates to fine-tune our meta attacker A every m iterations, the meta attacker A is trained to ensure that it can abstract the gradient distribution of different x t and learn to predict the gradient from a few samples with simple fine-tuning. Secondly, the most query-consuming part lies in zeroth-order gradient estimation, due to its coordinate-wise nature. In our algorithm, we only do this every m iterations. When we use the finetuned meta attacker A directly, no query is consumed in gradient estimation in these iterations. Intuitively, larger m implies less gradient estimation computation and fewer queries. Besides, just as mentioned above, even in zeroth-order gradient estimation, only top-q coordinates are required. Normally, q is much smaller than dimension p of the input.

4. EXPERIMENTS:

We compare our meta attacker with state-of-the-art black-box attack methods including Zoo (Chen et al., 2017) , Decision-Boundary (Brendel et al., 2018) , AutoZoom (Tu et al., 2019) , Optattack (Cheng et al., 2019) , FW-black (Chen et al., 2018) and Bandits (Ilyas et al., 2018b) to evaluate its query efficiency. We also study its generalizability and transferability through a Meta transfer attacker, as detailed in the following sections.

4.1. SETTINGS:



Datasets and Target Models:

We evaluate the attack performance on MNIST (LeCun, 1998) for handwritten digit recognition, CIFAR10 (Krizhevsky & Hinton, 2009) and tiny-Imagenet (Russakovsky et al., 2015) for object classification. The architecture details of meta attack models on MNIST, CIFAR10 and tiny-Imagenet are given in Table 6 . For MNIST, we train a separate meta attacker model since the images have different channel numbers from other natural image datasets. For CIFAR10 and tiny-Imagenet, we use a common meta attacker model. On CIFAR-10, we choose ResNet18 (He et al., 2016) as the target model M tar and use VGG13, VGG16 (Simonyan & Zisserman, 2014) and GoogleNet (Szegedy et al., 2015) for training our meta attacker. On tiny-Imagenet, we choose VGG19 and ResNet34 as the target model separately, and use VGG13, VGG16 and ResNet18 for training the meta attacker together.

Attack Protocols For a target black-box model M tar , obtaining a pair of (input-output) is considered as one query. We use the mis-classification rate as attack success rate; we randomly select 1000 images from each dataset as test images. To evaluate overall noise added by the attack methods, we use the mean L 2 distance across all the samples noise(M tar ) = 1 n n i=1 x adv i,A,Mtar -x i 2 , where x adv i,A,Mtar denotes the adversarial version for the authentic sample x i .

Meta-training Details:

For all the experiments, we use the same architecture for the meta attacker A as shown in Table 6 . We use Reptile (Nichol et al., 2018) with 0.01 learning rate to train meta attackers. We use 10000 randomly selected images from the training set to train the meta-attackers in three datasets. The proportion of the selected images to the whole training set are 16%, 20%, and 10% respectively. Fine-tuning parameters are set as m = 5 for MNIST and CIFAR10, and m = 3 for tiny-Imagenet. Top q = 128 coordinates are selected as part coordinates for attacker fine-tuning and model attacking on MNIST; and q = 500 on CIFAR10 and tiny-Imagenet.

4.2. COMPARISON WITH BASELINES:

We compare our meta attacker with baselines for both the untargeted and targeted black-box attack on the three datasets. The results are reported in detail as below.

Untargeted Attack Untargeted attack aims to generate adversarial examples that would be misclassified by the attacked model into any category different from the ground truth one. The overall results are shown in Table 1 , in which, Meta transfer denotes that meta attacker trained on one dataset is used to attack target models on another dataset . Our method is competitive with baselines in terms of adversarial perturbation and success rate, but our query number is reduced.

We also compare the results of our method with Zoo (Chen et al., 2017) and AutoZoom (Tu et al., 2019) from a query-efficiency perspective. We use these models to conduct untargeted attack on CIFAR10 and tiny-Imagenet by limiting a maximum number of queries for each adversarial example and compare their success rate. The results are shown in Fig. 1 . We notice that for different query thresholds, the success rate of our method is always higher than Zoo and AutoZoom. This is possibly because the testing samples have different L 2 distances to the decision boundary. Higher success rate of our method indicates our meta attacker can predict correct gradient even when the query information is limited. These results give strong evidence on effectiveness of our proposed method for enhancing query efficiency. Targeted Attack Targeted attack aims to generate adversarial noise such that the perturbed sample would be mis-classified into any pre-specified category. It is a more strict setting than the untargeted (Brendel et al., 2018) 1.00 1.85 13,630 Opt-attack (Cheng et al., 2019) 1.00 1.85 12,925 AutoZoom (Tu et al., 2019) , 1.00 1.86 2,412 Bandits (Ilyas et al., 2018b) 0.73 1.99 3,771 Meta attack (ours)

1.00 1.77 749

CIFAR10 / Resnet18

Zoo (Chen et al., 2017) 1.00 0.30 8,192 Decision Boundary (Brendel et al., 2018) 1.00 0.30 17,010 Opt-attack (Cheng et al., 2019) 1.00 0.33 20,407 AutoZoom (Tu et al., 2019) 1.00 0.28 3,112 Bandits (Ilyas et al., 2018b) 0.91 0.33 4,491 FW-black (Chen et al., 2018) 1 (Brendel et al., 2018) 1.00 0.52 49,942 Opt-attack (Cheng et al., 2019) 1.00 0.53 71,016 AutoZoom (Tu et al., 2019) 1.00 0.54 8,904 Bandits (Ilyas et al., 2018b) 0 Zoo (Chen et al., 2017) 1.00 0.47 25,344 Decision Boundary (Brendel et al., 2018) 1.00 0.48 49,982 AutoZoom (Tu et al., 2019) 1.00 0.45 9,770 Opt-attack (Cheng et al., 2019) 1.00 0.52 60,437 Bandits (Ilyas et al., 2018b) 0.73 0.49 9,978 Meta transfer (ours) 0.99 0.56 3,540 Meta attack (ours) 0.99 0.53 3,268 one. For fair comparison, we define the target label for each sample -a sample with label gets the target label ( + 1) mod #classes. We deploy our meta attacker the same as above. The results on MNIST, CIFAR10 and tiny-ImageNet are shown in Table 2 . Similar to results of untargeted attack, we achieve comparable noise and success rate to baselines but with reduced query numbers.

4.3. MODEL ANALYSIS:



Meta Training:

We first test the benefits of meta training by comparing performance of a metatrained attacker with a Gaussian randomly initialized attacker without meta training on the three datasets. Fig. 3 shows their success rate, L 2 distortion and query count results for initial success. The meta pre-trained attacker achieves averagely 7% higher success rate with 16% lower L 2 distortion and 30% less queries, compared with the randomly initialized one. This justifies the contributions of meta training to enhancing query efficiency and also attack performance.

Guaranteed by fine-tuning, the randomly initialized attacker succeeds over many testing samples. The fine-tuning works like an inner training in meta training. With sufficient fine-tuning iterations, the randomly initialized attacker functions like a well-trained meta attacker. This explains the effectiveness of the randomly initialized meta attacker on many testing samples compromised by more queries. However, it could not gradient as accurate as the well-trained meta attacker during earlier iterations. Such inaccuracy leads to larger L 2 distortion at the beginning. On the contrary, the meta training process enables the well-trained meta attacker to fast-adapt to current testing samples. These results highlight the significant advantages of our meta model towards to black-box attack.

The process of meta training makes it familiar with gradient patterns of various models.

Generalizability Here we show that our meta attacker trained on one dataset can be transferred to other datasets. We conduct this experiment between CIFAR10 and tiny-Imagenet, denoted as Meta transfer in Table 1 and 2 . We first apply the meta attacker trained on CIFAR10 to attack VGG19, ResNet34 on tiny-Imagenet respectively, which are different from models used for training meta attacker. Note the meta attacker tested on CIFAR10 has no privileged prior and is not familiar with (Brendel et al., 2018) 0.64 2.71 19,951 AutoZoom (Tu et al., 2019) 0.95 2.52 6,174 Opt-attack (Cheng et al., 2019) 1.00 2.33 99,661 Meta attack (ours) 1.00 2.66 1,299

CIFAR10 / Resnet18

Zoo (Chen et al., 2017) 1.00 0.55 66,400 Decision Boundary (Brendel et al., 2018) 0.58 0.53 16,250 AutoZoom (Tu et al., 2019) 1.00 0.51 9,082 Opt-attack (Cheng et al., 2019) 1.00 0.50 121,810 FW-black (Chen et al., 2018) 0 neither tiny-Imagenet dataset nor the corresponding classification models. Similarly, we also use the meta attacker trained on tiny-Imagenet to attack the target ResNet18 model on CIFAR10. The results show the good generalizability and robustness of our proposed meta attacker.

Parameters Selection:

We test the selection of top-q and β on CIFAR10. We choose q ranging from 350 to 600 and give the results with β ranging from 3e-3 to 5e-3, as shown in Fig. 2 . When q increases, query number, success rate and L 2 will all increase. In order to balance the overall result, we choose q to be 500. When β increases, success rate and L 2 will increase and query number decreases. In order to balance success rate and query, we choose β to be 4e-3 in the experiment.

5. CONCLUSION:

We propose a meta-based black-box attack method that largely reduces demanded query numbers without compromising in attack success rate and distortion. We train a meta attacker to learn useful prior information about gradient and incorporate it into the optimization process to decrease the number of queries. Specifically, the meta attacker is finetuned to fit the gradient distribution of target model and each update is based on the output of finetuned meta attaker. Extensive experimental results confirm the superior query-efficiency of our method over baselines.

6. APPENDIX:

6.1 MORE EXPERIMENTAL RESULTS

6.1.1. COSINE SIMILARITY BETWEEN ESTIMATED GRADIENTS AND WHITE-BOX GRADIENTS:

To demonstrate the ability of estimating gradient of our meta attacker, we have conducted experiments with the Resnet-34 model on the tiny-Imagenet dataset to compare the cosine similarity between the estimated gradients from our proposed meta-attacker and the accurate white-box gradients. We also compare the cosine similarity between ZOO (Chen et al., 2017) estimated gradients and the white-box gradients as reference. The results of cosine similarity and required number of queries are shown in Table 3 . We can observe that our meta-attacker does fast-adapt to the target model and generate accurate gradients. Not only it estimates gradients with positive cosine similarity to the true gradients, but it also performs closer to the ZOO estimated results with the same small standard deviation. We have conducted experiments in section 4.3 to demonstrate the benefits of meta training. However, we only compare the performance of a meta-trained attacker with a Gaussian randomly initialized attacker. We conduct two more experiments on meta training here to further investigate the benefits of meta training. First we compare our meta-trained attacker with a vanilla autoencoder that learns to map images to gradients of one white-box model. The vanilla autoencoder has the same architecture with our meta-attacker, but it is trained in one white-box model. Then, we train a new meta-attacker in four black-box models, i.e., the gradients used for training are estimated via ZOO (Chen et al., 2017) . The experiment results are presented in Table 4 , 5 . The two experiments are conducted on 1000 randomly selected images from the MNIST testing set.

As for the four approach settings: "Ours (reported)" is the results we report in our paper; "ZOO estimated" is the meta-attacker trained from ZOO estimated gradients; "Vanilla Autoencoder" is the autoencoder maps image to gradient trained in one different MNIST classification model; "Initialised attacker" is the meta-attacker with randomly initialized weights. We can see the "ZOO estimated" model performs closer to our reported meta-attacker. However, the "Vanilla Autoencoder" performs much worse (with larger L2-norm and more queries), which performs similarly to randomly initialized meta-attacker. The two experiments verify the effectiveness of our first meta-training phase. 

:



Figures and Tables:

Figure 2: Top-q and β selection.

Figure 3: Comparison of randomly initialized and well-trained meta attackers.

Figure 4: Adversarial examples generated by our method on MNIST. The groundtruth images are shown in the diagonal and the rest are adversarial examples that are misclassified to the targeted class shown on the top.

Figure 5: Adversarial examples generated by our method on CIFAR10. The groundtruth images are shown in the diagonal and the rest are adversarial examples that are misclassified to the targeted class shown on the top.

Meta Attacker TrainingInput: Input images X, groundtruth gradients G i generated from classification models M i to serve as task T i ; 1: Randomly initialize θ; 2: while not done do

Table content: <table><tr><td>3:</td></tr></table>

then

Table content: <table><tr><td>10:</td><td>x adv = x ;</td></tr><tr><td>11:</td><td>break;</td></tr><tr><td>12:</td><td>else</td></tr><tr><td>13:</td><td>x t+1 = x ;</td></tr><tr><td>14:</td><td>end if</td></tr><tr><td colspan="2">15: end for</td></tr><tr><td colspan="2">Output: adversarial example x adv .</td></tr></table>

MNIST, CIFAR10 and tiny-ImageNet untargeted attack comparison: Meta attacker attains comparable success rate and L2 distortion as baselines, and significantly reduces query numbers.

Table content: <table><tr><td>Dataset / Target model</td><td>Method</td><td colspan="3">Success Rate Avg. L2 Avg. Queries</td></tr><tr><td/><td>Zoo (Chen et al., 2017)</td><td>1.00</td><td>1.61</td><td>21,760</td></tr><tr><td/><td>Decision Boundary</td><td/><td/></tr><tr><td>MNIST / Net4</td><td/><td/><td/></tr></table>

MNIST, CIFAR10 and tiny-ImageNet targeted attack comparison: Meta attack significantly outperforms other black-box methods in query numbers.

Table content: <table><tr><td>Dataset / Target model</td><td>Method</td><td colspan="3">Success Rate Avg. L2 Avg. Queries</td></tr><tr><td/><td>Zoo (Chen et al., 2017)</td><td>1.00</td><td>2.63</td><td>23,552</td></tr><tr><td/><td>Decision Boundary</td><td/><td/></tr><tr><td>MNIST / Net4</td><td/><td/><td/></tr></table>

Cosine similarity between estimated gradients and white-box gradients.

Table content: <table><tr><td/><td colspan="2">Ours meta-attacker</td><td colspan="2">ZOO (Chen et al., 2017)</td></tr><tr><td>Task Type</td><td>Similarity</td><td>Queries</td><td>Similarity</td><td>Queries</td></tr><tr><td colspan="2">Untargeted 0.356 ± 0.074</td><td>3,268</td><td colspan="2">0.395 ± 0.079 25,344</td></tr><tr><td>Targeted</td><td colspan="4">0.225 ± 0.090 12,897 0.363 ± 0.069 88,966</td></tr><tr><td colspan="5">6.1.2 VANILLA TRAINING AUTOENCODER AND META-ATTACKER LEARNING FROM</td></tr><tr><td colspan="2">ESTIMATED GRADIENTS</td><td/><td/><td/></tr></table>

MNIST untargeted attack comparison.

Table content: <table><tr><td>Method</td><td colspan="3">Success Rate Avg. L 2 Avg. Queries</td></tr><tr><td>Ours(reported)</td><td>1.000</td><td>1.78</td><td>1,103</td></tr><tr><td>ZOO estimated</td><td>1.000</td><td>1.77</td><td>1,130</td></tr><tr><td>Vanilla Autoencoder</td><td>1.000</td><td>1.93</td><td>1,899</td></tr><tr><td>Initialised attacker</td><td>0.912</td><td>1.96</td><td>1,721</td></tr></table>

MNIST targeted attack comparison.

Table content: <table><tr><td>Method</td><td colspan="3">Success Rate Avg. L 2 Avg. Queries</td></tr><tr><td>Ours(reported)</td><td>1.000</td><td>2.66</td><td>1,971</td></tr><tr><td>ZOO estimated</td><td>1.000</td><td>2.42</td><td>2,105</td></tr><tr><td>Vanilla Autoencoder</td><td>1.000</td><td>2.80</td><td>2,905</td></tr><tr><td>Initialised attacker</td><td>0.895</td><td>2.81</td><td>3,040</td></tr></table>

Peer review (iclr2020):
Summary: The authors of this paper propose a novel approach for query-efficient black-box adversarial attacks by leveraging meta-learning to approximate the gradient of given clean images. Paper strength: 1. The overall idea of this paper is interesting to improve the transferability of adversarial examples through meta-learning. 2. The paper is well-organized and easy to follow. 3. Adequate experiment results demonstrate that the meta-attacker is efficient to decrease the number of queries for the adversarial attacks. It is interesting to see the meta attacker trained on certain source domain can be transferred to other domains. Paper weakness: 1. The authors miss some important details about training meta attackers like how many images you choose to get the image and its gradient pairs. Are the images from the training set or test set. Is there an overlap between the meta-attacker training set and the test set? 2. What if you try to learn the adversarial perturbation directly instead of learning the gradient of images? 3. How about changing the meta learner to vanilla training and learning an autoencoder directly to map the image to its gradient? There could be a problem of how to incorporate the existing classifiers, but I think this could be an important baseline. 4. The authors try to use the gradient from the ZOO to guide the pre-trained meta attacker to adapt to a new classifier in Algorithm 2 (line 3). What if you take multi-step ZOO to give a much stronger prior to finetuning? 5. Take a look at the confusion matrix in [1] which studies the transferability of white-box attack and black-box attack. I think it could be more interesting if the authors change the ground truth from gradient to estimated gradient from ZOO for learning meta-attacker. Li, Yandong, et al. *NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks.* arXiv preprint arXiv:1905.00441 (2019).