QUERY-EFFICIENT META ATTACK TO DEEP NEURAL NETWORKS
Year: 2020

Abstract:
Black-box attack methods aim to infer suitable attack patterns to targeted DNN models by only using output feedback of the models and the corresponding input queries. However, due to lack of prior and inefficiency in leveraging the query and feedback information, existing methods are mostly query-intensive for obtaining effective attack patterns. In this work, we propose a meta attack approach that is capable of attacking a targeted model with much fewer queries. Its high queryefficiency stems from effective utilization of meta learning approaches in learning generalizable prior abstraction from the previously observed attack patterns and exploiting such prior to help infer attack patterns from only a few queries and outputs. Extensive experiments on MNIST, CIFAR10 and tiny-Imagenet demonstrate that our meta-attack method can remarkably reduce the number of model queries without sacrificing the attack performance. Besides, the obtained meta attacker is not restricted to a particular model but can be used easily with a fast adaptive ability to attack a variety of models. The code of our work is available at https://github.com/dydjw9/MetaAttack_ICLR2020/.

heading: 1 INTRODUCTION
text: Despite the great success in various tasks, deep neural networks (DNNs) are found to be susceptible to adversarial attacks and often suffer dramatic performance degradation in front of adversarial examples, even if only tiny and invisible noise is imposed on the input (Szegedy et al., 2014). To investigate the safety and robustness of DNNs, many adversarial attack methods have been developed, which apply to either a white-box (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017; Madry et al., 2018) or a black-box setting (Papernot et al., 2017; Brendel et al., 2018; Narodytska & Kasiviswanathan, 2017). In the white-box attack setting, the target model is transparent to the attacker and imperceptible adversarial noise can be easily crafted to mislead this model by leveraging its gradient information (Goodfellow et al., 2015). In contrast, in the blackbox setting, the structure and parameters of the target DNN model are invisible, and the adversary can only access the input-output pair in each query. With a sufficient number of queries, black-box methods utilize the returned information to attack the target model generally by estimating gradient (Chen et al., 2017; Ilyas et al., 2018a; Narodytska & Kasiviswanathan, 2017; Cheng et al., 2019).
Black-box attack is more feasible in realistic scenarios than white-box attack but it is much more query-intensive. Such a drawback is largely attributed to the fact that returned information for each queried example is sparse and limited. During inferring attack patterns, existing black-box methods simply integrate the information between two sequential iterations brutally and ignore the implicit but profound message, thus not fully exploiting the returned information. Although query-efficient algorithms for generating attack examples are very meaningful in practice (Ilyas et al., 2018a), how to enhance query-efficiency for black-box attack remains underexplored.
∗Equal contribution.
In this work, we address a query-efficiency concerned attack problem. Particularly, we consider only top-k probability scores accessible from the target black-box model. With this practical but challenging scenario, we aim at three important objectives: lower query number, higher success rate and smaller noise magnitude. We develop a meta-learning based attack method, which applies meta learning to obtaining prior information from the successful attack patterns, and uses the prior for efficient optimization. Specifically, we propose to train a meta attacker model through meta learning (Nichol et al., 2018), inspired by its success in solving few-shot learning problems. We first deploy several existing classification models to get pairs of (images, gradients) with the max-margin logit classification loss. Then we use the data pairs of each classification model to train the meta attacker. After obtaining the attacker, we use it to attack a new black-box model for accelerating the search process for adversarial examples by optimizing it with coordinate-wise gradient estimation. Different from previous methods, we use the estimated gradient not only to update adversarial noise but to fine-tune the well-trained attacker. After few-shot fine-tuning, the attacker is able to simulate the gradient distribution of the target model.
We evaluate our method on MNIST, CIFAR10 and tiny-ImageNet datasets by comparing it with state-of-the-art black-box attack methods including Zoo (Chen et al., 2017), DecisionBoundary (Brendel et al., 2018), AutoZoom (Tu et al., 2019), Opt-attack (Cheng et al., 2019) and Bandits (Ilyas et al., 2018b). In both targeted and untargeted settings, our proposed method achieves comparable attack success rate and adversarial perturbation to all baselines but with a significantly reduced query number. The detailed experiment results demonstrate our superior query-efficiency.

heading: 2 RELATED WORK
text: Classical white-box attack methods include Fast-Gradient Sign Method (FGSM) (Goodfellow et al., 2015), IFGSM (Madry et al., 2018), DeepFool (Moosavi-Dezfooli et al., 2016) and C&W attack (Carlini & Wagner, 2017), following a setting where detailed information about the target model (gradients and losses) is provided. Comparatively, the black-box setting better accords with the real world scenarios in that little information about the target model is visible to the attacker. The pioneer work on black-box attack (Papernot et al., 2017) tries to construct a substitute model with augmented data and transfer the black-box attack problem to a white-box one. However, its attack performance is very poor due to the limited transferability of adversarial examples between two different models. (Brendel et al., 2018) considers a more restricted case where only top-1 prediction classes are returned and proposes a random-walk based attack method around the decision boundary. It dispenses class prediction scores and hence requires extensive model queries. Zoo (Chen et al., 2017) is a black-box version of C&W attack, achieving a similar attack success rate and comparable visual quality as many white-box attack methods. However, its coordinate-wise gradient estimation requires extensive model evaluations. More recently, (Ilyas et al., 2018a) proposes a query-limited setting with L∞ noise considered, and uses a natural evolution strategy (NES) to enhance query efficiency. Though this method successfully controls the query number, the noise imposed is larger than average. (Narodytska & Kasiviswanathan, 2017) proposes a novel local-search based technique to construct numerical approximation to the network gradient, which is then carefully used to construct a small set of pixels in an image to perturb. It suffers a similar problem as in (Chen et al., 2017) for pixel-wise attack. (Cheng et al., 2019) considers a hard-label black-box setting and formulates the problem as real-valued optimization that is solved by a zeroth order optimization algorithm. Ilyas et al. (2018b) reduce the queries by introducing two gradient priors, the time-independent prior and the data-dependent prior, and reformulating the optimization problem.
We then briefly introduce some works on meta-learning related to our work. Meta-learning is a process of learning how to learn. A meta-learning algorithm takes in a distribution of tasks, each being a learning problem, and produces a quick learner that can generalize from a small number of examples. Meta-learning is very popular recently for its fast adaptive ability. MAML (Finn et al., 2017) is the first to propose this idea. Recently, a simplified algorithm Reptile (Nichol et al., 2018) which is an approximation to the first-order MAML is proposed, achieving higher efficiency in computation and consuming less memory. With these superior properties, meta learning is applied to adversarial attack methods (Zgner & Gnnemann, 2019; Edmunds et al., 2017). Zgner & Gnnemann (2019) try to attack the structure of a graph model in the training process to decrease the model generalization performance. Edmunds et al. (2017) investigate the susceptibility of MAML to adversarial attacks and the transferability of the obtained meta model to a specific task.

heading: 3 METHOD

heading: 3.1 PRELIMINARIES: BLACK-BOX ATTACK SCHEMES
text: We first formulate the black-box attack problem and introduce the widely used solutions. We use (x, t) to denote the pair of a natural image and its true label, and x̂ and t̂ to denote the adversarial perturbed version of x and the returned label by the target classification modelMtar. The black-box attack aims to find an adversarial example x̂ with imperceivable difference from x to fail the target model , i.e., t̂ 6= t through querying the target model for multiple times. It can be formulated as
min x̂ `(x̂,Mtar(x̂), t) s.t. ‖x̂− x‖p ≤ ρ, #queries ≤ Q. (1)
Here ‖ · ‖p denotes the `p norm that measures how much perturbation is imposed. Mtar(x̂) is the returned logit or probability by the target modelMtar. The loss function `(x̂,Mtar(x̂), t) measures the degree of certainty for model Mtar assigning the input x̂ into class t. One common used adversarial loss is the probability of class t: `(x̂,Mtar(x̂), t) = pMtar (t|x̂). The first constraint enforces high similarity between the clean image x and the adversarial one xadv and the second imposes a fixed budget Q for the number of queries allowed in the optimization.
In the white-box attack setting, the adversary can access the true gradient ∇x̂t`(x̂t) and perform gradient descent x̂t+1 = x̂t − ∇x̂t`(x̂t). But in the black-box setting, the gradient information ∇x̂t`(x̂t) is not attainable. In this case, the attacker can estimate the gradient using only queried information from model evaluation such as hard label, logits and probability scores. This kind of estimator is the backbone of so-called zeroth-order optimization approaches (Chen et al., 2017; Narodytska & Kasiviswanathan, 2017; Tu et al., 2019; Ilyas et al., 2018a;b). The estimation is done via finite difference method (Chen et al., 2017; Narodytska & Kasiviswanathan, 2017; Tu et al., 2019), which finds the k components of the gradient by estimating the inner products of the gradients with all the standard basis vector e1, ..., ek:
∇`(x) ≈ k∑ i=1 f(x+ hei)− f(x− hei) 2h ei, (2)
where step size h controls the quality of the estimated gradient. Another strategy is to reformulate the loss function (Ilyas et al., 2018a;b). Instead of computing the gradient of `(x) itself, the expected value of loss function `(x) under the search distribution is minimized and when the search distribution of random Guassian noise is adopted, the gradient estimation problem transfers into a zeroth-order estimation problem,
∇E[`(x)] ≈ 1 σn n∑ i=1 `(x+ σδi)δi (3)
where n is the amount of noise sampled from the distribution. After obtaining the estimated gradient, classical optimization algorithms (Nesterov, 2013; Johnson & Zhang, 2013) can be used to infer the adversarial examples. Though the estimated gradient may not be accurate, it is still proved useful enough in adversarial attack. The convergence of these zeroth-order methods is guaranteed under mild assumptions (Ghadimi & Lan, 2013; Nesterov & Spokoiny, 2017; Hazan et al., 2016).
Since each model evaluation consumes a query, naively applying the above gradient estimation to black-box attack is quite query expensive due to its coordinate or noise sampling nature. Take the first strategy on tiny-Imagenet dataset for example. It consumes more than 20,000 queries for each image to obtain a full gradient estimate, which is not affordable in practice. In this work, we address such a limitation via developing a query-efficient meta-learning based attack model.

heading: 3.2 LEARNING OF META ATTACKER
text: To reduce the query cost for black-box attack, we apply meta learning to training a meta attacker model, inspired by its recent success in few-shot learning problems (Finn et al., 2017; Nichol et al., 2018). The meta attacker learns to extract useful prior information of the gradient of a variety of models w.r.t. specific input samples. It can infer the gradient for a new target model using only a
Algorithm 1 Meta Attacker Training Input: Input images X, groundtruth gradients Gi generated from classification modelsMi to serve
as task Ti; 1: Randomly initialize θ; 2: while not done do 3: for all Ti do 4: Sample K samples from (X,Gi) for training, denoted as (Xs,Gsi ); 5: Evaluate∇θLi(Aθ) = ∇θ‖Aθ(Xs)−Gsi‖22 with respect to (Xs,Gsi ); 6: Update θ′i := θ − α∇θLi(Aθ); 7: end for 8: Update θ := θ + 1n ∑n i=1(θ ′ i − θ);
9: end while Output: Parameters θ of meta model A.
few queries. After obtaining such a meta attacker, we replace the zeroth-order gradient estimation in traditional black box attack methods with it to directly estimate the gradient.
We collect a set of existing classification modelsM1, ...,Mn to generate gradient information for universal meta attacker training. Specifically, we feed each image x into the modelsM1, ...,Mn respectively and compute losses `1, ..., `n by using following max-margin logit classification loss:
`i(x) = max [ log[Mi(x)]t −max
j 6=t log[Mi(x)]j , 0
] . (4)
Here t is the groundtruth label and j indexes other classes. [Mi(x)]t is the probability score of the true label predicted by the model Mi, and [Mi(x)]j denotes the probability scores of other classes. By performing one step back-propagation of losses `1, ..., `n w.r.t. the input images x, the corresponding gradients gi = ∇x`i(x), i = 1, ..., n are obtained. Finally, we collect n groups of data X = {x},Gi = {gi}, i = 1, ..., n to train the universal meta attacker. We design a meta attacker A which has a similar structure as an autoencoder, consisting of symmetric convolution and de-convolution layers and outputs a gradient map with the same size as the input. Meta attacker model A is parameterized with parameters θ. Due to the intrinsic difference between selected classification models, each obtained set (X,Gi) is treated as a task Ti in meta attacker training. During the training process, for each iteration we only draw K samples from task Ti and feedback the loss Li to update model parameters from θ to θ′i. θ ′ i is then computed through one or multiple gradient descents: θ ′ i := θ − α∇θLi(Aθ). For a sensitive position of the meta attacker, the meta attacker parameters are optimized by combining each θ′i across all tasks {Ti}i=1,...,n, following the update strategy of Reptile (Nichol et al., 2018) in meta learning,
θ := θ + 1
n n∑ i=1 (θ′i − θ). (5)
We adopt mean-squared error (MSE) as the training loss in the inner update,
Li(Aθ) = ‖Aθ(Xs)−Gsi‖22. (6) The set (Xs,Gsi ) denotes the K samples used for each inner update from θ to θ′i. Since the number of K sampled each time is very small, the update strategy above tries to find good meta attacker parameters θ as an initial point, from which the meta attacker model can fast adapt to new data distribution through gradient descent based fine-tuning within limited samples. Therefore, this characteristic can be naturally leveraged in attacking new black-box models by estimating their gradient information through a few queries. Detailed training process of our meta attacker is described in Algorithm 1.

heading: 3.3 QUERY-EFFICIENT ATTACK VIA META ATTACKER
text: An effective adversarial attack relies on optimizing the loss function equation 1 w.r.t. the input image to find the adversarial example of the target modelMtar. Differently, our proposed method applies the meta attacker A to predicting the gradient map of a test image directly.
Algorithm 2 Adversarial Meta Attack Algorithm Input: Test image x0 with label t, meta attacker Aθ, target model Mtar, iteration interval m,
selected top-q coordinates; 1: for t = 0, 1, 2, ... do 2: if (t+ 1) mod m = 0 then 3: Perform zeroth-order gradient estimation on top q coordinates, denoted as It and obtain gt; 4: Fine-tune meta attacker A with (xt, gt) on It by loss L = ‖[Aθ(xt)]It − [gt]It‖ 2 2; 5: else 6: Generate the gradient map gt directly from meta attacker A with xt, select coordinates It; 7: end if 8: Update [x′]It = [xt]It + β[gt]It ; 9: ifMtar(x′) 6= t then
10: xadv = x′; 11: break; 12: else 13: xt+1 = x′; 14: end if 15: end for Output: adversarial example xadv .
We use the obtained meta attacker model A to predict useful gradient map for attacking, which should be fine tuned to adapt to the new gradient distribution under our new target model. Particularly, instead of finetuning once, for each given image x, we update A by leveraging query information with the following periodic scheme. Suppose the given image is perturbed to xt ∈ Rp at iteration t. If (t+ 1) mod m = 0, our method performs zeroth-order gradient estimation to obtain gradient map gt for fine-tuning. As each pixel value for estimated gradient map consumes two queries, for further saving queries, we just select q of the p coordinates to estimate, q p, instead of the full gradient map through all p coordinates. The indexes of chosen coordinates are determined by the gradient map gt−1 obtained in iteration t−1. We sort the coordinate indexes by the value of gt−1 and select top-q indexes. The set of these indexes are denoted as It. We feed image xt in iteration t into meta attacker A and compute the MSE loss on indexes It, i.e. L = ‖[Aθ(xt)]It − [gt]It‖ 2 2. Then we perform gradient descent for the MSE loss with a few steps to update the parameters θ of meta attacker A. For the rest iterations, we just use the periodically updated attacker Aθ to directly generate the gradient gt = Aθ(xt). When we have the estimated gradient map gt in iteration t, we update to get the adversarial sample x′t by [x ′ t]It = [xt]It + β[gt]It where β is a hyperparameter to be tuned. The details are summarized in Algorithm 2.
In our method, the following operations contribute to reducing the query number needed by the attacker. First, though we just use q coordinates to fine-tune our meta attacker A every m iterations, the meta attacker A is trained to ensure that it can abstract the gradient distribution of different xt and learn to predict the gradient from a few samples with simple fine-tuning. Secondly, the most query-consuming part lies in zeroth-order gradient estimation, due to its coordinate-wise nature. In our algorithm, we only do this every m iterations. When we use the finetuned meta attacker A directly, no query is consumed in gradient estimation in these iterations. Intuitively, largerm implies less gradient estimation computation and fewer queries. Besides, just as mentioned above, even in zeroth-order gradient estimation, only top-q coordinates are required. Normally, q is much smaller than dimension p of the input.

heading: 4 EXPERIMENTS
text: We compare our meta attacker with state-of-the-art black-box attack methods including Zoo (Chen et al., 2017), Decision-Boundary (Brendel et al., 2018), AutoZoom (Tu et al., 2019), Optattack (Cheng et al., 2019), FW-black (Chen et al., 2018) and Bandits (Ilyas et al., 2018b) to evaluate its query efficiency. We also study its generalizability and transferability through a Meta transfer attacker, as detailed in the following sections.

heading: 4.1 SETTINGS
text: Datasets and Target Models We evaluate the attack performance on MNIST (LeCun, 1998) for handwritten digit recognition, CIFAR10 (Krizhevsky & Hinton, 2009) and tiny-Imagenet (Russakovsky et al., 2015) for object classification. The architecture details of meta attack models on MNIST, CIFAR10 and tiny-Imagenet are given in Table 6. For MNIST, we train a separate meta attacker model since the images have different channel numbers from other natural image datasets. For CIFAR10 and tiny-Imagenet, we use a common meta attacker model. On CIFAR-10, we choose ResNet18 (He et al., 2016) as the target modelMtar and use VGG13, VGG16 (Simonyan & Zisserman, 2014) and GoogleNet (Szegedy et al., 2015) for training our meta attacker. On tiny-Imagenet, we choose VGG19 and ResNet34 as the target model separately, and use VGG13, VGG16 and ResNet18 for training the meta attacker together.
Attack Protocols For a target black-box modelMtar , obtaining a pair of (input-output) is considered as one query. We use the mis-classification rate as attack success rate; we randomly select 1000 images from each dataset as test images. To evaluate overall noise added by the attack methods, we use the mean L2 distance across all the samples noise(Mtar) = 1n ∑n i=1 ‖xadvi,A,Mtar − xi‖2, where xadvi,A,Mtar denotes the adversarial version for the authentic sample xi.
Meta-training Details For all the experiments, we use the same architecture for the meta attacker A as shown in Table 6. We use Reptile (Nichol et al., 2018) with 0.01 learning rate to train meta attackers. We use 10000 randomly selected images from the training set to train the meta-attackers in three datasets. The proportion of the selected images to the whole training set are 16%, 20%, and 10% respectively. Fine-tuning parameters are set as m = 5 for MNIST and CIFAR10, and m = 3 for tiny-Imagenet. Top q = 128 coordinates are selected as part coordinates for attacker fine-tuning and model attacking on MNIST; and q = 500 on CIFAR10 and tiny-Imagenet.

heading: 4.2 COMPARISON WITH BASELINES
text: We compare our meta attacker with baselines for both the untargeted and targeted black-box attack on the three datasets. The results are reported in detail as below.
Untargeted Attack Untargeted attack aims to generate adversarial examples that would be misclassified by the attacked model into any category different from the ground truth one. The overall results are shown in Table 1, in which, Meta transfer denotes that meta attacker trained on one dataset is used to attack target models on another dataset . Our method is competitive with baselines in terms of adversarial perturbation and success rate, but our query number is reduced.
We also compare the results of our method with Zoo (Chen et al., 2017) and AutoZoom (Tu et al., 2019) from a query-efficiency perspective. We use these models to conduct untargeted attack on CIFAR10 and tiny-Imagenet by limiting a maximum number of queries for each adversarial example and compare their success rate. The results are shown in Fig. 1. We notice that for different query thresholds, the success rate of our method is always higher than Zoo and AutoZoom. This is possibly because the testing samples have different L2 distances to the decision boundary. Higher success rate of our method indicates our meta attacker can predict correct gradient even when the query information is limited. These results give strong evidence on effectiveness of our proposed method for enhancing query efficiency.
500 1000 3000 5000 Query Numbers
0.0
0.2
0.4
0.6
0.8
1.0
At ta
ck S
uc ce
ss R
at e
Cifar10 Resnet18 ZOO AutoZoom Meta attack (ours)
1000 4000 12000 20000 Query Numbers
0.0
0.2
0.4
0.6
0.8
1.0
At ta
ck S
uc ce
ss R
at e
tiny-Imagenet Resnet34
ZOO AutoZoom Meta attack (ours)
Figure 1: Comparison with limited queries. 300 350 400 450 500 550 600 650 top-q coordinates
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Su cc
es s R
at e
Success Rate, L2, Queries vs. top-q selection
Success Rate L2 Queries
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
L 2
0
500
1000
1500
2000
2500
3000
3500
4000
Qu er
ie s
2.5 3.0 3.5 4.0 4.5 5.0 5.5 1e 3
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Su cc
es s R
at e
Success Rate, L2, Queries vs. selection
Success Rate L2 Queries
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
L 2
0
500
1000
1500
2000
2500
3000
3500
4000
Qu er
ie s
Figure 2: Top-q and β selection.
Targeted Attack Targeted attack aims to generate adversarial noise such that the perturbed sample would be mis-classified into any pre-specified category. It is a more strict setting than the untargeted
one. For fair comparison, we define the target label for each sample —a sample with label ` gets the target label (`+1) mod #classes. We deploy our meta attacker the same as above. The results on MNIST, CIFAR10 and tiny-ImageNet are shown in Table 2. Similar to results of untargeted attack, we achieve comparable noise and success rate to baselines but with reduced query numbers.

heading: 4.3 MODEL ANALYSIS
text: Meta Training We first test the benefits of meta training by comparing performance of a metatrained attacker with a Gaussian randomly initialized attacker without meta training on the three datasets. Fig. 3 shows their success rate, L2 distortion and query count results for initial success. The meta pre-trained attacker achieves averagely 7% higher success rate with 16% lower L2 distortion and 30% less queries, compared with the randomly initialized one. This justifies the contributions of meta training to enhancing query efficiency and also attack performance.
Guaranteed by fine-tuning, the randomly initialized attacker succeeds over many testing samples. The fine-tuning works like an inner training in meta training. With sufficient fine-tuning iterations, the randomly initialized attacker functions like a well-trained meta attacker. This explains the effectiveness of the randomly initialized meta attacker on many testing samples compromised by more queries. However, it could not predict gradient as accurate as the well-trained meta attacker during earlier iterations. Such inaccuracy leads to larger L2 distortion at the beginning. On the contrary, the meta training process enables the well-trained meta attacker to fast-adapt to current testing samples. These results highlight the significant advantages of our meta model towards to black-box attack. The process of meta training makes it familiar with gradient patterns of various models.
Generalizability Here we show that our meta attacker trained on one dataset can be transferred to other datasets. We conduct this experiment between CIFAR10 and tiny-Imagenet, denoted as Meta transfer in Table 1 and 2. We first apply the meta attacker trained on CIFAR10 to attack VGG19, ResNet34 on tiny-Imagenet respectively, which are different from models used for training meta attacker. Note the meta attacker tested on CIFAR10 has no privileged prior and is not familiar with
neither tiny-Imagenet dataset nor the corresponding classification models. Similarly, we also use the meta attacker trained on tiny-Imagenet to attack the target ResNet18 model on CIFAR10. The results show the good generalizability and robustness of our proposed meta attacker.
Parameters Selection We test the selection of top-q and β on CIFAR10. We choose q ranging from 350 to 600 and give the results with β ranging from 3e-3 to 5e-3, as shown in Fig. 2. When q increases, query number, success rate and L2 will all increase. In order to balance the overall result, we choose q to be 500. When β increases, success rate and L2 will increase and query number decreases. In order to balance success rate and query, we choose β to be 4e-3 in the experiment.

heading: 5 CONCLUSION
text: We propose a meta-based black-box attack method that largely reduces demanded query numbers without compromising in attack success rate and distortion. We train a meta attacker to learn useful prior information about gradient and incorporate it into the optimization process to decrease the number of queries. Specifically, the meta attacker is finetuned to fit the gradient distribution of target model and each update is based on the output of finetuned meta attaker. Extensive experimental results confirm the superior query-efficiency of our method over baselines.

heading: ACKNOWLEDGEMENT
text: This research is partially supported by Programmatic grant no. A1687b0033 from the Singapore government’s Research, Innovation and Enterprise 2020 plan (Advanced Manufacturing and Engineering domain).
Hu Zhang (No. 201706340188) is partially supported by the Chinese Scholarship Council.
Jiashi Feng was partially supported by NUS IDS R-263-000-C67-646, ECRA R-263-000-C87-133, MOE Tier-II R-263-000-D17-112 and AI.SG R-263-000-D97-490.

heading: 6 APPENDIX

heading: 6.1 MORE EXPERIMENTAL RESULTS

heading: 6.1.1 COSINE SIMILARITY BETWEEN ESTIMATED GRADIENTS AND WHITE-BOX GRADIENTS
text: To demonstrate the ability of estimating gradient of our meta attacker, we have conducted experiments with the Resnet-34 model on the tiny-Imagenet dataset to compare the cosine similarity between the estimated gradients from our proposed meta-attacker and the accurate white-box gradients. We also compare the cosine similarity between ZOO (Chen et al., 2017) estimated gradients and the white-box gradients as reference. The results of cosine similarity and required number of queries are shown in Table 3. We can observe that our meta-attacker does fast-adapt to the target model and generate accurate gradients. Not only it estimates gradients with positive cosine similarity to the true gradients, but it also performs closer to the ZOO estimated results with the same small standard deviation.

heading: 6.1.2 VANILLA TRAINING AUTOENCODER AND META-ATTACKER LEARNING FROM ESTIMATED GRADIENTS
text: We have conducted experiments in section 4.3 to demonstrate the benefits of meta training. However, we only compare the performance of a meta-trained attacker with a Gaussian randomly initialized attacker. We conduct two more experiments on meta training here to further investigate the benefits of meta training. First we compare our meta-trained attacker with a vanilla autoencoder that learns to map images to gradients of one white-box model. The vanilla autoencoder has the same architecture with our meta-attacker, but it is trained in one white-box model. Then, we train a new meta-attacker in four black-box models, i.e., the gradients used for training are estimated via ZOO (Chen et al., 2017). The experiment results are presented in Table 4, 5 .
The two experiments are conducted on 1000 randomly selected images from the MNIST testing set. As for the four approach settings: ”Ours (reported)” is the results we report in our paper; ”ZOO estimated” is the meta-attacker trained from ZOO estimated gradients; ”Vanilla Autoencoder” is the autoencoder maps image to gradient trained in one different MNIST classification model; ”Initialised attacker” is the meta-attacker with randomly initialized weights. We can see the ”ZOO estimated” model performs closer to our reported meta-attacker. However, the ”Vanilla Autoencoder” performs much worse (with larger L2-norm and more queries), which performs similarly to randomly initialized meta-attacker. The two experiments verify the effectiveness of our first meta-training phase.

heading: 6.2 STRUCTURE OF META ATTACKER

heading: 6.3 STRUCTURE OF TARGET MODEL USED IN MNIST

heading: 6.4 ACCURACY OF TARGET MODELS ON ORIGINAL DATASETS
text: 6.5 ADVERSARIAL EXAMPLES GENERATED BY OUR METHOD

Review summaries:
The authors proposed a meta-learning based black-box adversarial attack method to reduce the number of queries. Their proposed meta attacker is trained via a meta-learning approach, and it could utilize historical query information and fast adapted to different target models. The authors conduct experiments in three datasets, and the results show that their proposed meta attacker is more efficient than other black-box attack methods.