Title: ADVERSARIAL IMITATION ATTACK

Abstract: Deep learning models are known to be vulnerable to adversarial examples. A practical adversarial attack should require as little as possible knowledge of attacked models. Current substitute attacks need pre-trained models to generate adversarial examples and their attack success rates heavily rely on the transferability of adversarial examples. Current score-based and decision-based attacks require lots of queries for the attacked models. In this study, we propose a novel adversarial imitation attack. First, it produces a replica of the attacked model by a two-player game like the generative adversarial networks (GANs). The objective of the generative model is to generate examples which lead the imitation model returning different outputs with the attacked model. The objective of the imitation model is to output the same labels with the attacked model under the same inputs. Then, the adversarial examples generated by the imitation model are utilized to fool the attacked model. Compared with the current substitute attacks, imitation attack can use less training data to produce a replica of the attacked model and improve the transferability of adversarial examples. Experiments demonstrate that our imitation attack requires less training data than the black-box substitute attacks, but achieves an attack success rate close to the white-box attack on unseen data with no query.

1. INTRODUCTION:

Deep neural networks are often vulnerable to imperceptible perturbations of their inputs, causing incorrect predictions (Szegedy et al., 2014) . Studies on adversarial examples developed attacks and defenses to assess and increase the robustness of models, respectively. Adversarial attacks include white-box attacks, where the attack method has full access to models, and black-box attacks, where the attacks do not need knowledge of models structures and weights.

White-box attacks need training data and the gradient information of models, such as FGSM (Fast Gradient Sign Method) (Goodfellow et al., 2015) , BIM (Basic Iterative Method) (Kurakin et al., 2017a) and JSMA (Jacobian-based Saliency Map Attack) (Papernot et al., 2016b) . However, the gradient information of attacked models is hard to access, the white-box attack is not practical in real-world tasks. Literature shows adversarial examples have transferability property and they can affect different models, even the models have different architectures (Szegedy et al., 2014; Papernot et al., 2016a; Liu et al., 2017) . Such a phenomenon is closely related to linearity and over-fitting of models (Szegedy et al., 2014; Hendrycks & Gimpel, 2017; Goodfellow et al., 2015; Tramèr et al., 2018) . Therefore, substitute attacks are proposed to attack models without the gradient information. Substitute black-box attacks utilize pre-trained models to generate adversarial examples and apply these examples to attacked models. Their attack success rates rely on the transferability of adversarial examples and are often lower than that of white-box attacks. Black-box score-based attacks (Chen et al., 2017; Ilyas et al., 2018a; b) do not need pre-trained models, they access the output probabilities of the attacked model to generate adversarial examples iteratively. Black-box decisionbased attacks (Brendel et al., 2017; Cheng et al., 2018; Chen et al., 2019) require less information than the score-based attacks. They utilize hard labels of the attacked model to generate adversarial examples.

Adversarial attacks need knowledge of models. However, a practical attack method should require as little as possible knowledge of attacked models, which include training data and procedure, models weights and architectures, output probabilities and hard labels (Athalye et al., 2018) . The disadvantage of current substitute black-box attacks is that they need pre-trained substitute models trained by the same dataset with attacked model T (Hendrycks & Gimpel, 2017; Goodfellow et al., 2015; Kurakin et al., 2017a) or a number of images to imitate the outputs of T to produce substitute networks (Papernot et al., 2017) . Actually, the prerequisites of these attacks are hard to obtain in real-world tasks. The substitute models trained by limited images hardly generate adversarial examples with well transferability. The disadvantage of current decision-based and score-based black-box attacks is that every adversarial example is synthesized by numerous queries.

Hence, developing a practical attack mechanism is necessary. In this paper, we propose an adversarial imitation training, which is a special two-player game. The game has a generative model G and a imitation model D. The G is designed to produce examples to make the predicted label of the attacked model T and D different, while the imitation model D fights for outputting the same label with T . The proposed imitation training needs much less training data than the T and does not need the labels of these data, and the data do not need to coincide with the training data. Then, the adversarial examples generated by D are utilized to fool the T like substitute attacks. We call this new attack mechanism as adversarial imitation attack. Compared with current substitute attacks, our adversarial imitation attack requires less training data. Score-based and decision-based attacks need a lot of queries to generate each adversarial attack. The similarity between the proposed method and current score-based and decision-based attacks is that adversarial imitation attack also needs to obtain a lot of queries in the training stage. The difference between these two kinds of attack is our method do not need any additional queries in the test stage like other substitute attacks. Experiments show that our proposed method achieves state-of-the-art performance compared with current substitute attacks and decision-based attack. We summarize our main contributions as follows:

• The proposed new attack mechanism needs less training data of attacked models than current substitute attacks, but achieves an attack success rate close to the white-box attacks.

• The proposed new attack mechanism requires the same information of attacked models with decision attacks on the training stage, but is query-independent on the testing stage. Adversarial Defenses To increase the robustness of models, methods for defending against adversarial attacks are being proposed. Adversarial training (Szegedy et al., 2014; Madry et al., 2018; Kurakin et al., 2017b; Tramèr et al., 2018) can be considered as a kind of data augmentation. It applies adversarial examples to the training data, resulting in a robust model against adversarial attacks. Defenses based on gradient masking (Tramèr et al., 2018; Dhillon et al., 2018) provide robustness against optimization-based attacks. Random transformation (Kurakin et al., 2017a; Meng & Chen, 2017; Xie et al., 2018) on inputs of models hide the gradient information, eliminate the perturbation. Buckman et al. (2018) proposed thermometer encoding based on one-hot encoding, it applied a nonlinear transformation to inputs of models, aiming to reduce the linearity of the model. However, most defenses above are still unsafe against some attacks (Carlini & Wagner, 2017a; He et al., 2017) . Especially Athalye et al. (2018) showed that defenses based on gradient masking actually are unsafe against attacks. Instead, some researchers focus on detecting adversarial examples. Some use a neural network (Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017) to distinguish between adversarial examples and clean examples. Some achieve statistical properties (Bhagoji et al., 2017; Hendrycks & Gimpel, 2017; Feinman et al., 2017; Ma et al., 2018; Papernot & McDaniel, 2018) to detect adversarial examples.

2. RELATED WORK:



3. IMITATION ATTACK:

In this section, we introduce the definition of adversarial examples and then propose a new attack mechanism based on adversarial imitation training.

3.1. ADVERSARIAL EXAMPLES:

X refers to the samples from the input space of the attacked model T , y true refers to the true labels of the samples X. T (y|X, θ) is the attacked model parameterized by θ. For a non-targeted attack, the objective of the adversarial attack can be formulated as:

EQUATION

where the and r are perturbation of the sample and upper bound of the perturbation, respectively.

To guarantee that is imperceptible, r is set to a small value in applications. 

3.2. ADVERSARIAL IMITATION TRAINING:

Inspired by the generative adversarial network (GAN) (Goodfellow et al., 2014) , we use the adversarial framework to copy the attacked model. We propose a two-player game based adversarial imitation training to replicate the information of attacked model T , which is shown in Figure 1 . To learn the characteristics of T , we define an imitation network D, and train it using disturbed input X = G(X) + X and the corresponding output label y T ( X) of attacked model. X denotes training samples here. The role of G is to create new samples that y T ( X) = y D ( X). Thus, D, G and T play a special two-player game. To simplify the expression but without loss of generality, we just analyze the case of binary classification. The value function of players can be presented as: 

G min D V G,D = -y T ( X) logD( X) -(1 -y T ( X) )log(1 -D( X)) subject to X = G(X) + X.

(2)

Note that the T is equivalent to the referee of this game. The two players G and D optimize their parameters based on the output y T ( X). We suppose that adversarial perturbation ≤ r 1 , and

∀ X = G(X) + X, G(X) ≤ r 2 , r 1 ≤ r 2 . If the D can achieve y D ( X) = y T ( X)

, our imitation attack will have the same success rate as the white-box attack without the gradient information of T . Therefore, for a well-trained imitation network, adversarial examples generated by D have strong transferability for T . A proper upper bound (r 2 ≥ r 1 ) of G(X) is the key points for training an efficient imitation network D. Especially in targeted attacks (T outputs the specified wrong label), if the characteristics of D is more similar to that of the attacked model, the transferability of adversarial examples will be stronger.

In the training stage, the loss function of D is J D = V G,D . Because G is more hard to train than D, sometimes the ability of D is much stronger than G, so the loss of G fluctuates during the training stage. In order to maintain the stability of training, the loss function of G is designed as J G = e -V G,D . Therefore, the global optimal imitation network D is obtained if and only if ∀ X, D( X) = y T ( X). At this point, J D = 0 and J G = e 0 = 1. The loss of G is always in a controllable value in training stage.

As we discussed before, if r 1 ≤ r 2 , the adversarial examples generated by a well-trained D have strong transferability for T . Because the attack perturbation of adversarial examples is set to be a small value, we constrain the G(X) in training stage to limit the search space of G, which can reduce the number of queries efficiently.

For training methodology, we alternately train the G and D in every mini-batch, and use L 2 penalty to constrain the search space of G. The procedure is shown in algorithm 1.

Algorithm 1 Mini-batch stochastic gradient descent training of imitation network.

1. : for number of training iterations do 2 ::

Sample mini-batch of m examples{X (1) , . . . , X (m) } from training set.

3. ::

Update the imitation model by descending its loss function : 4 :

-y T ( X)logD( X) -(1 -y T ( X))log(1 -D( X)).

Update the generative model by descending its loss function : 6 :

e -V G,D + α G( X) .

7. : end for:

In adversarial attacks, when the optimal D is obtained, the adversarial examples generated by D are utilized to attack T .

4. EXPERIMENTS:



4.1. EXPERIMENT SETTING:

In this subsection, we introduce the settings for our experiments.

Datasets: we evaluate our proposed method on MNIST (LeCun et al., 1998) and CIFAR-10 (Krizhevsky & Hinton, 2009) . Because we need to use data different with the training set of T to train the imitation network, we divided the test sets (10k) from MNIST and CIFAR-10 into two parts. One part contains 9500 images for training and another part contains 500 images for evaluating the attack performance.

Model architecture and attack method: in order to get the information of the attacked model T as little as possible, we only utilize the output label (not the probability) of the T to train the imitation network. The imitation network has no prior knowledge of the attacked model, which means it does not load any pre-trained model in experiments. For the experiments on MNIST, we design 3 different network architectures with different capacity (small network, medium network and large network) for evaluating the performance of our imitation attack with models having different capacity. We utilize the pre-trained medium network and VGG-16 (Simonyan & Zisserman, 2015) as the attacked model on MNIST and CIFAR-10, respectively. In order to compare the success rate of the proposed imitation attack with current substitute attack, we utilize 4 attack methods, FGSM, BIM, projected gradient descent (PGD) (Madry et al., 2018) , C&W to generate adversarial examples. For testing, we use AdverTorch library (Ding et al., 2019) to generate adversarial examples. On the other hand, for comparing the performance of our method with current decision-based attacks and score-based attacks, we utilize Boundary Attack (Brendel et al., 2017) , HSJA Attack (Chen et al., 2019) , SimBA-DCT Attack (Guo et al., 2019) as comparison methods. Note that score-based attacks require output probabilities of T , which contain much more information than labels.

Evaluation criteria: the goals of non-targeted attack and targeted attack are to lead the attacked model to output a wrong label and a specific wrong label, respectively. In non-targeted attacks, we only generate adversarial examples on the images classified correctly by the attacked model. In targeted attacks, we only generate adversarial examples on the images which are not classified to the specific wrong labels. The success rates of adversarial attack are calculated by n/m, where n and m are the number of adversarial examples which can fool the attacked model and the total number of adversarial examples, respectively.

4.2. EXPERIMENTAL ANALYSIS OF ADVERSARIAL ATTACK:

In this subsection, we utilize the proposed adversarial imitation training to train imitation models and evaluate the performance in terms of attack success rate.

To compare our method with substitute attack, We utilize the medium network and VGG-16 as attacked models on MNIST and CIFAR-10, respectively. Then we use the same train dataset to obtain a pre-trained large network (the architecture is also in Table 9 ) and ResNet-50 (He et al., 2016) to generate substitute adversarial examples. We obtain imitation networks by using the proposed adversarial imitation training. The large network and ResNet-16 are used as the model architectures of imitation networks on MNIST and CIFAR-10, respectively. The imitation models are only trained by 9500 samples of the test dataset, which are much less than the training sets of MNIST (60000 samples) and CIFAR-10 (50000 samples). The results of experiments are evaluated on the other 500 samples of the test dataset and are shown in Table 1 and Table 2 . The success rates of the proposed imitation attack far exceed the success rates of the substitute attack in all experiments.

The experiments of Table 1 and Table 2 show that the proposed new attack mechanism needs less training images than substitute attacks, but achieves an attack success rate close to the white-box attack. The experiments indicate that the adversarial samples generated by a well-trained imitation model have a higher transferability than the substitute attack.

To compare our method with decision-based and score-based attacks, we evaluate the performance of these attacks. We utilize 9500 images from the test set of MNIST and CIFAR-10 to train the imitation network, and use other 500 images from the test set as unseen data for our method. The other decision-based and score-based attacks are evaluated on the test set of MNIST and CIFAR-10 dataset. Note that score-based attacks require much more information (output probabilities) than decision-based attacks (output labels). The results on MNIST and CIFAR-10 are shown in Table 3 and Table 4 , respectively. Because our imitation attack only needs queries on the training stage, we evaluate performances of our method on its train and unseen sets. We set the iteration of adversarial imitation training to 1800, so the average number of query per image is 1800. We utilize BIM attack to generate adversarial examples as our imitation attack in this experiment.

This experiment shows that our imitation attack achieves state-of-the-art performance in decisionbased methods. Even compared with the score-based attack, our imitation attack outperforms it in terms of perturbation distance and attack success rate. More importantly, it also obtains good results on unseen data, which indicates our imitation attack can be applied to query-independent scenarios.

4.3. EXPERIMENTAL ANALYSIS OF NETWORK CAPACITY:

In the above subsection, we utilize a more complex network than the attacked model to replicate the attacked model. In this subsection, we study the impact of model capacity on the ability of imitation.

To evaluate the imitation performance of the network with less capacity than the attacked model, we train the small network, medium network, and large network to imitate the pre-trained medium network on MNIST dataset, and train VGG-13, VGG-16 and ResNet-50 to imitate VGG-16 on CIFAR-10. The performance of models with different capacities are shown in Table 5 and 6 . The results show that an imitation model with a lower capacity than the attacked model can also achieve a good imitation performance. Attack success rates of all imitation models far exceed the substitute attacks in Table 1 and 2 . Most experiments show that the larger capacity the imitation network has, the higher attack success rate it can achieve (FGSM, BIM, PGD in MNIST and BIM in CIFAR-10). However, some experiments show models having a larger capacity do not have a higher attack success rate (FGSM and PGD in CIFAR-10). We surmise that the performance of imitating an attacked model is not only influenced by the capacity of the imitation model D, but also influenced by the capability of the G.

4.4. EXPERIMENTAL ANALYSIS OF MODEL REPLICATION:

In this subsection, we only use 200 images (20 samples per class) to train the imitation networks and discuss characteristics of the model replication. 

:



Figures and Tables:

X = X + are the adversarial examples which can fool the attacked model T . θ refers to the parameters of the model T . For white-box attacks, they obtain gradient information of T to generate adversarial examples, and attack the T directly. For substitute attacks, they generate adversarial examples from a substitute model T , and transfer the examples to attack the T . The key point of a successful attack is the transferability of the adversarial examples. To improve the transferability of adversarial examples and avoid output query, we utilize an imitation network to imitate the characteristics of the T by accessing its output labels to improve the transferability of adversarial examples, which are generated by the imitation network. After the adversarial imitation training, adversarial examples generated by imitation network do not need additional query. In the next subsection, we introduce the proposed adversarial imitation training and imitation attack.

Figure 1: The proposed adversarial imitation attack. For the training stage, the objective of G is to generate samples X = G(X) + X and let y D ( X) = y T ( X). The objective of D is to guarantee y D ( X) = y T ( X). For the testing stage, the imitation model D is utilized to generate adversarial examples to attack T .

Figure 2: Visualization of some adversarial examples generated by the imitation model on MNIST. Original examples are in the first row. Examples from the second row to the third row are generated through FGSM, BIM, PGD, respectively.

Performance of the proposed imitation attack compared with the white-box attacks and substitute attacks on MNIST. The architectures of networks are shown in Table 9. "White-box": generate adversarial examples from the attacked medium network. "Substitute": generate adversarial examples from the pre-trained large network. "Imitation": generate adversarial examples from the imitation large network. The numbers in ( ) denote the average L F perturbation distance per image.

Table content: <table><tr><td>Attack</td><td>White-box</td><td>Non-targeted (%) Substitute</td><td>Imitation</td><td>White-box</td><td>Targeted (%) Substitute</td><td>Imitation</td></tr><tr><td colspan="7">FGSM 82.90 (5.17) 57.55 (5.17) 75.45 (4.78) 29.91 (5.25) 15.81 (5.25) 33.85 (5.17)</td></tr><tr><td>BIM</td><td colspan="6">86.52 (3.45) 45.47 (3.45) 70.62 (3.21) 39.96 (3.45) 14.03 (3.45) 28.51 (3.53)</td></tr><tr><td>PGD</td><td colspan="4">65.79 (3.76) 28.17 (3.76) 49.70 (3.68) 22.32 (3.84)</td><td>7.80 (3.84)</td><td>15.37 (3.84)</td></tr><tr><td>CW</td><td colspan="6">81.89 (3.06) 38.63 (2.90) 66.00 (3.14) 43.08 (1.88) 14.25 (1.56) 37.32 (1.88)</td></tr></table>

Performance of the proposed imitation attack compared with the white-box attacks and substitute attacks on CIFAR-10. The attacked model is VGG-16.

Table content: <table><tr><td>Attack</td><td>White-box</td><td>Non-targeted (%) Substitute</td><td>Imitation</td><td>White-box</td><td>Targeted (%) Substitute</td><td>Imitation</td></tr><tr><td colspan="5">FGSM 84.87(2.73) 39.29 (2.73) 81.30 (2.73) 41.42 (1.66)</td><td>8.22 (1.66)</td><td>32.65 (1.66)</td></tr><tr><td>BIM</td><td colspan="6">98.96 (1.08) 47.27 (1.01) 97.06 (1.14) 67.82 (0.92) 14.38 (0.83) 62.56 (0.95)</td></tr><tr><td>PGD</td><td colspan="4">67.44 (1.11) 30.25 (1.14) 67.02 (1.29) 28.17 (0.98)</td><td>6.39 (1.01)</td><td>25.57 (1.04)</td></tr><tr><td>CW</td><td colspan="6">97.69 (1.35) 38.66 (1.35) 70.40 (1.38) 68.60 (1.51) 20.78 (1.41) 45.43 (1.51)</td></tr></table>

Performance of the proposed imitation attack compared with the decision-based and scorebased attacks on MNIST. "Query": the average number of queries of attacks. "L F ": the average L F distance per image. "imitation-train": the performance of imitation attack on its training data. "imitation-unseen": the performance of imitation attack on other unseen data.

Table content: <table><tr><td>Attack</td><td>Query</td><td colspan="2">Non-targeted L F Success rate</td><td>Query</td><td colspan="2">Targeted L F Success rate</td></tr><tr><td/><td/><td/><td>Score-based</td><td/><td/><td/></tr><tr><td>SimBA</td><td colspan="2">752.32 5.41</td><td>96.78%</td><td>879.86</td><td>3.53</td><td>82.63%</td></tr><tr><td/><td/><td colspan="2">Decision-based</td><td/><td/><td/></tr><tr><td>Boundary</td><td colspan="2">2830.44 9.49</td><td>100.0%</td><td colspan="2">2613.31 11.05</td><td>81.30%</td></tr><tr><td>HSJA</td><td colspan="2">4679.57 6.59</td><td>100.0%</td><td colspan="2">2113.06 6.27</td><td>75,67</td></tr><tr><td>Imitation-train</td><td colspan="2">1800.00 4.94</td><td>99.62%</td><td colspan="2">1800.00 5.33</td><td>84.36%</td></tr><tr><td>Imitation-unseen</td><td>-</td><td>4.94</td><td>99.52%</td><td>-</td><td>5.41</td><td>82.62%</td></tr></table>

Performance of the proposed imitation attack compared with the decision-based and scorebased attacks on CIFAR-10.

Table content: <table><tr><td>Attack</td><td>Query</td><td colspan="2">Non-targeted L F Success rate</td><td>Query</td><td colspan="2">Targeted L F Success rate</td></tr><tr><td/><td/><td colspan="2">Score-based</td><td/><td/><td/></tr><tr><td>SimBA</td><td colspan="2">501.65 1.76</td><td>99.37%</td><td colspan="2">489.90 2.06</td><td>84.16%</td></tr><tr><td/><td/><td colspan="2">Decision-based</td><td/><td/><td/></tr><tr><td>Boundary</td><td colspan="2">1937.02 3.26</td><td>100.0%</td><td colspan="2">1837.01 6.02</td><td>35.59%</td></tr><tr><td>HSJA</td><td colspan="2">2206.55 2.59</td><td>100.0%</td><td colspan="2">1527.84 1.12</td><td>32.53%</td></tr><tr><td>Imitation-train</td><td colspan="2">1800.00 1.29</td><td>99.87%</td><td colspan="2">1800.00 1.19</td><td>87.76%</td></tr><tr><td>Imitation-unseen</td><td>-</td><td>1.29</td><td>99.58%</td><td>-</td><td>1.19</td><td>85.62%</td></tr></table>

Performance of the imitation networks with different capacity on MNIST. The attacked model is pre-trained medium network. The architectures of networks are shown in Table9. The adversarial examples generated by the same attack methods have the same perturbation distance.We train the imitation network using 200 images from MNIST and CIFAR-10 test set, and compare its performance with Practical Attack(Papernot et al., 2017) on other images from MNIST and CIFAR-10 test set. The results are shown in Table7 and 8. The practical attack uses the output labels of attacked models to train substitute models under the scenario, which they can make an infinite number of queries for attacked models. It is hard to generate adversarial examples to fool the attacked models by limited training samples. Note that what the substitute models imitate is the response for perturbations of the attacked model. A substitute model that can generate adversarial examples with a higher attack success rate is a better replica. Our adversarial imitation attack can produce a substitute model with much higher classification accuracy and attack success rate than Practical Attack for both non-targeted and targeted attacks in this scenario (with an infinite number of queries). We also show the performances of these two methods with limited query numbers in Figure6. Additionally, the imitation model with low classification accuracy still can produce

Table content: <table><tr><td>Attack</td><td colspan="6">Non-targeted (%) Small net Medium net Large net Small net Medium net Large net Targeted (%)</td></tr><tr><td>FGSM</td><td>69.82</td><td>63.98</td><td>75.45</td><td>33.18</td><td>28.06</td><td>33.85</td></tr><tr><td>BIM</td><td>61.17</td><td>57.75</td><td>70.62</td><td>21.36</td><td>18.71</td><td>28.51</td></tr><tr><td>PGD</td><td>42.45</td><td>39.84</td><td>49.70</td><td>10.45</td><td>10.47</td><td>91.67</td></tr></table>

Performance of the imitation networks with different capacity on CIFAR-10. The attacked model is pre-trained VGG-16. The adversarial examples generated by the same attack methods have the same perturbation distance.

Table content: <table><tr><td>Attack</td><td colspan="6">Non-targeted (%) VGG-13 VGG-16 ResNet-50 VGG-13 VGG-16 ResNet-50 Targeted (%)</td></tr><tr><td>FGSM</td><td>73.11</td><td>85.92</td><td>81.30</td><td>20.57</td><td>26.91</td><td>32.65</td></tr><tr><td>BIM</td><td>81.72</td><td>96.85</td><td>97.06</td><td>38.51</td><td>58.21</td><td>62.56</td></tr><tr><td>PGD</td><td>45.59</td><td>70.59</td><td>67.02</td><td>14.22</td><td>27.79</td><td>25.57</td></tr></table>

Comparisons between imitation model and other substitute models. "Accuracy": the classification accuracy on other images from test set. The numbers in ( ) denote the average L F perturbation distance per image.

Table content: <table><tr><td>Non-targeted (%)</td></tr></table>

Comparisons between imitation model and other substitute models. "Accuracy": the classification accuracy on other images from test set. adversarial attacks should have as little as possible knowledge of attacked model T . Current black-box attacks need numerous training images or queries to generate adversarial images. In this study, to address this problem, we combine the advantages of current black-box attacks and proposed a new attack mechanism, imitation attack, to replicate the information of the T , and generate adversarial examples fooling deep learning models efficiently. Compared with substitute attacks, imitation attack only requires much less data than the training set of T and do not need the labels of the training data, but adversarial examples generated by imitation attack have stronger transferability for the T . Compared with score-based and decision-based attacks, our imitation attack only needs the same information with decision attacks, but achieves state-of-the-art performances and is query-independent on testing stage. Experiments showed the superiority of the proposed imitation attack. Additionally, we observed that deep learning classification model T is easy to be stolen by limited unlabeled images, which are much fewer than the training images of T . In future work, we will evaluate the performance of the proposed adversarial imitation attack on other tasks except for image classification.

Table content: <table><tr><td>Non-targeted (%)</td></tr></table>

Peer review (iclr2020):
The authors propose to use a generative adversarial network to train a substitute that replicates (imitates) a learned model under attack. They then show that the adversarial examples for the substitute can be effectively used to attack the learned model. The idea is straightforward. The proposed approach leads to better success rates of attacking than other substitute-training approaches that require more training examples. Promising experimental results against decision and score-based attach schemes also demonstrate the effectiveness of the proposed approach. My main concern is that the comparison to other approaches seem unfair, because (1) Substitute models typically use all training data and query the learned model once per training example; the proposed approach uses fewer training data but needs 1800 queries per example. So it is not surprising that the proposed approach can be better than substitute models. As the authors focus on *practical* scenarios, the number of queries should be fixed as a constraint for all approaches to be fair. (2) Having said (1), there is not enough detail in this paper to understand the similarity and difference between the proposed approach and decision-based ones (which is claimed to have similar query complexity during training). The authors mix the results with score-based (requiring more information), making the results more confusing to the readers. The authors are encouraged to present more detailed discussions on the comparison with decision-based competitors. Also, (3) One thing that is missing from the experiments is how well the replica clones the original model. All the information in the experiments are somewhat *indirect* (success rate, test accuracy, etc.) to answer this question, but there is no direct evidence. Is a well-cloned replica really a better substitute to construct adversarial examples? For instance, for replica with different *cloning accuracy* (rather than test accuracy), is a better replica also a better substitute? The paper fails to answer this question that best matches its design motivation. The above are my main concerns. A minor one is (4) The abstract that directly uses notations like T, G and D is horribly hard to read.