Adversarial Imitation Attack
Authors: Mingyi Zhou
Jing Wu
Yipeng Liu
Xiaolin Huang
Shuaicheng Liu
Liaqat Ali
Xiang Zhang
Ce Zhu
Venue:  (2020)
Keywords: Adversarial examples
Security
Machine learning
Deep neural network
Computer vision

Abstract: Deep learning models are known to be vulnerable to adversarial examples. A practical adversarial attack should require as little as possible knowledge of attacked models T. Current substitute attacks need pre-trained models to generate adversarial examples and their attack success rates heavily rely on the transferability of adversarial examples. Current score-based and decision-based attacks require lots of queries for the T. In this study, we propose a novel adversarial imitation attack. First, it produces a replica of the T by a two-player game like the generative adversarial networks (GANs). The objective of the generative model G is to generate examples which lead D returning different outputs with T. The objective of the discriminative model D is to output the same labels with T under the same inputs. Then, the adversarial examples generated by D are utilized to fool the T. Compared with the current substitute attacks, imitation attack can use less training data to produce a replica of T and improve the transferability of adversarial examples. Experiments demonstrate that our imitation attack requires less training data than the black-box substitute attacks, but achieves an attack success rate close to the white-box attack on unseen data with no query. 

Under review as a conference paper at ICLR 2020
ADVERSARIAL IMITATION ATTACK
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep learning models known to be vulnerable to adversarial examples. A prac- tical adversarial attack should require as little as possible knowledge of attacked models. Current substitute attacks need pre-trained models to generate adversar- ial examples and their attack success rates heavily rely on the transferability of adversarial examples. Current score-based and decision- bbased attacks require lots of queries for the attacked models. In this study, we propose a novel adversarial imitation attack. First, it produces a replica the attacked model by a two-player game like the generative adversarial networks (GANs). The objective of the gen- erative model is to generate examples which lead the imitation model returning different outputs with the attacked model. The objective of the imitation model is to output the same labels with the attacked model under the same inputs. Then, the adversarial examples generated by the imitation model are utilized to fool the attacked model. Compared with the current substitute attacks, imitation attack can use less training data produce replica of the attacked model and improve the transferability of adversarial examples. Experiments demonstrate that our im- itation attack requires less training data than the black-box substitute attacks, but achieves an attack success rate close to the white-box box attack on unseen data with no query.
INTRODUCTION
Deep neural networks are often vulnerable to imperceptible perturbations of their inputs, causing incorrect predictions (Szegedy et al.] 2014). Studies on adversarial examples developed attacks and defenses to assess and increase the robustness of models, respectively. Adversarial attacks include white-box attacks, where the attack method has full access to models, and black-box attacks, where the attacks do not need knowledge of models structures and weights.
White-box attacks need training data and the gradient information of models, such as FGSM (Fast Gradient Sign Method) (Goodfellow et al.) 20155), BIM (Basic Iterative Method) (Kurakin et al.) 2017a) and JSMA (Jacobian- Saliency Map Attack) (Papernot et al., 2016b). However, the gradient information of attacked models is hard to access, the white-box attack not practical in real-world tasks. Literature shows adversarial examples have transferability property and they can affect different models, even the models have different architectures (Szegedy et al., 2014 Papernot let al. 2016a] et al. 2017). Such a phenomenon is closely related to linearity and over-fitting of models (Szegedy et al. 2014 Hendryeks & Gimpel 2017]: Goodfellow al.) [015 Tramer al., 2018). Therefore, substitute attacks are proposed to attack models without the gradient information. Substitute black-box attacks utilize pre-trained models generate adversarial examples and apply these examples to attacked models. Their attack success rates rely on the transferability of adver- sarial examples and are often lower than that of white-box attacks. Black-box score-based attacks (Chen et al.) 2017: [Tyas et al. 201 8a b) do not need pre-trained models, they access the output probabilities the attacked model to generate adversarial examples iteratively. Black-box decision- based attacks (Brendel et al., 2017 Cheng et al. 2018 Chen et al. 2019) require less information than the score-based attacks. They utilize hard labels the attacked model to generate adversarial examples.
Adversarial attacks need knowledge of models. However, a practical attack method should require as little possible knowledge of attacked models, which include training data and procedure, models weights and architectures, output probabilities and hard labels (Athalye et al.) 2018). The disadvan- tage of current substitute black-box attacks is that they need pre-trained substitute models trained by

---

Under review as a conference paper at ICLR 2020
the same dataset with attacked model T (Hendryeks & Gimpel 2017 Goodfellow al. 2013 Ku-
rakin et al. (017a) or number of images to imitate the outputs of T to produce substitute networks (Papernot et al. 2017). Actually, the prerequisites of these attacks are hard to obtain in real-world tasks. The substitute models trained by limited images hardly generate adversarial examples with well transferability. The disadvantage of current decision-based and score-hased black-box attacks is that every adversarial example is synthesized by numerous queries.
Hence, developing practical attack mechanism is necessary. In this paper, we propose an adver- sarial imitation training, which is special two-player game. The game has a generative model G and a imitation model D. The G is designed to produce examples to make the predicted label of the attacked model T and D different, while the imitation model D fights for oulputting the same label with T. The proposed imitation training needs much less training data than the T and does not need the labels of these data, and the data do not need to coincide with the training data. Then, the adversarial examples generated by are utilized fool the T like substitute attacks. We call this new attack mechanism adversarial imitation attack. Compared with current substitute attacks, our adversarial imitation attack requires less training data. Score-based and decision- bbasedd attacks need a lot of queries to generate each adversarial attack. The similarity between the proposed method and current score-basee and decision-
attacks is that adversarial imitation attack also needs to obtain lot of queries in the training stage. The difference between these two kinds of attack is our method do not need any additional queries in the test stage like other substitute attacks. Experi- ments show that our proposed method achieves state-of-the thrt performance compared with current substitute attacks and decision- based attack. We summarize our main contributions as follows:
The proposed new attack mechanism needs less training data of attacked models than cur- rent substitute attacks, but achieves attack success rate close to the white-box attacks.
The proposed new attack mechanism requires the same information of attacked models with decision attacks on the training stage, but is query-independent on the testing stage.
2 RELATED WORK
Adversarial Scenes Adversarial attacks happen in two scenes, namely the white-box and the black-box settings. In the white-box settings, the attack method has complete access to attacked models, such as models internal, training strategy and data. While in the black-box settings, the attack method has little knowledge of attacked models. The black-box attack utilizes the transfer- ability property of adversarial examples, only needs the labeled training data, but its attack success rate is often lower than that of the white-box attack method if attacked models have no defense. Actually, attack methods requiring lots of prior knowledge of attacked models are difficult to apply in practical applications (Athalye et al. 2018).
Adversarial Attacks Several methods for generating adversarial examples were proposed. Good- fellow et al. (2015) proposed a one-step attack called FGSM. On the basis of the FGSM, Kurakin et al. (2017a) came up with BIM, an iterative optimization-banl based attack. Another iterative attack Saboosvvi-ezzfode
et al. 2016) aims find an adversarial example that would cross the decision boundary. CCarlini & Wagner (2017b) provided stronger attack by simultane- ously minimizing the perturbation and the 7. norm of the perturbation. [ony et al. (2018) gener- ated adversarial examples through decoupling the direction and the norm of the perturbation, which also constrained by the LF norm. Liu et al.] (2017) showed that targeted adversarial examples hardly have transferability, they proposed ensemble-bate ples having stronger transferability. Papernot et al. (017)) proposed a practical black-box attack which accesses the hard label to train substitute models. For score-based attacks, Chen et al. (2017) proposed the zeroth-order based attack (ZOO) which uses gradient estimates to attack a black-bon model. [lyas et al. (2018b) improves the way estimate gradients. Guo al. (2019) proposed a simple black-box score-based attack on DCT space. For decision- attacks, Brendel et al.
called DeepFool
methods to generate adversarial exam-
attacks./Chen
(217)) first proposed decision-based attacks which do not rely on gradients. [Cheng et al. (201 and Chen et al. (2019) improve the query efficiency of the decision-basel attack.
Adversarial Defenses To increase the robustness of models, methods for defending against ad- versarial attacks are being proposed. Adversarial training (Szegedy et al. 2014 Madry al. 2018
2

---

Under review as a conference paper at ICLR 2020
Kurakin et al. 2017b Tramer et al. 2018) can be considered kind of data augmentation. It ap- plies adversarial examples to the training data, resulting in robust model against adversarial attacks. Defenses based on gradient masking (Tramèr al. 2018: Dhillon et al.] (2018) provide robustness against optimization-base attacks. Random transformation (Kurakin et al.) 201 7a Meng & Chen
2017 Xie al., 2018) inputs of models hide the gradient information, eliminate the perturba- tion. Buckman et al. (2018) proposed thermometer encoding based on one-hot encoding, it applied nonlinear transformation to inputs of models, aiming to reduce the linearity of the model. How- ever, most defenses above are still unsafe against some attacks (Carlini & Wagner, 2017a He et al.) 2017 Especially Athalye et al. (2018) showed that defenses based on gradient masking actually are unsafe against attacks. Instead, some researchers focus on detecting adversarial examples. Some use a neural network (Gong et al. 2017 Grosse et al. 2017 Metzen al. 2017 to distinguish between
adversarial examples and clean examples. Some achieve statistical properties (Bhagoji et al. 2017
Hendrycks Gimpel 2017; Feinman et al. 2017 Ma et al. 2018
Papernot & McDanicl 2018 to
detect adversarial examples.
IMITATION ATTACK
In this section, introduce the definition of adversarial examples and then propose new attack mechanism based on adversarial imitation training.
3.1 ADVERSARIAL EXAMPLES
X refers the samples from the input space of the attacked model T, Vtrue refers to the true labels of the samples X. Twy 0) is the attacked model parameterized by 0. For a non-targeted attack, the objective of the adversarial attack can be formulated as:
min||e/|
subject to argmax T(y, X X c, + Strue
Vi
(1)
and ||]| rr,
where the e and are perturbation of the sample and upper bound of the perturbation, respectively. To guarantee that imperceptible, adversarial examples which can fool the attacked model T. 0 refers to the parameters of the model T.
small value in applications. X X + e the
For white- box attacks, they obtain gradient information of T to generate adversarial examples, and attack the T directly. For substitute attacks, they generate adversarial examples from a substitute model T. and transfer the examples to attack the T. The key point of a successful attack is the transferability of the adversarial examples.
To improve the transfcrability of adversarial examples and avoid output query, we utilize an imi- tation network imitate the characteristics of the T by accessing its output labels improve the transferability of adversarial examples, which are generated by the imitation network. After the adversarial imitation training, adversarial examples generated by imitation network do not need ad- ditional query. In the next subsection, we introduce the proposed adversarial imitation training and imitation attack.
3.2 ADVERSARIAL IMITATION TRAINING
Inspired by the generative adversarial network (GAN) (Goodfellow et al. 2014). we use the ad- versarial framework to copy the attacked model. We propose a two-player game based adversarial imitation training to replicate the information of attacked model T, which is shown in Figure] [ To leam the characteristics
T, we define an imitation network D. and train it using disturbed input X = G(X) + X and the corresponding output label (X) of attacked model. X denotes training samples here. The role of G is to create new samples that yr (X) # yp (X). Thus, D, G and T play a special two-player game. To simplify the expression but without loss of generality, we just analyze the case of binary classification. The value function of players can be presented as:
3

---

Under review as a conference paper at ICLR 2020
X
Target model (T)
+ Yr
G
X
Generative model (G)
Imitation model (D) Yn
obj: Yr * y/
obj: yD Yr
Figure I: The proposed adversarial imitation attack. For the training stage, the objective of G is to generate samples X = G(X) X and let yn X) * (X). The objective of D is to guarantee yo X) (X) For the testing stage, the imitation model D is utilized to generate adversarial examples to attack T.
max min G D
Va. = -yr(() log - -(1 VT log(( - D(X))
(2)
subject to G G(X) + X.
Note that the is equivalent to the referee this game. The two players G and D optimize their parameters based on the output yr(() (X). We suppose that adversarial perturbation c S and A X G(X) X, ||G(X)] S r2, f1 S r2. If the D achieve (X) yr(X), our imitation attack will have the same success rate as the white-box attack without the gradient information of T. Therefore, for a well-trained imitation network, adversarial examples generated by D have strong transferability for T' A proper upper bound (r2 2 r1) of G(X) is the key points for training an efficient imitation network D. Especially in targeted attacks (T' outputs the specified wrong label), if the characteristics of D is more similar to that of the attacked model, the transferability of adversarial examples will stronger.
In the training stage, the loss function of D is JD = VG.D. Because G' is more hard to train than D, sometimes the ability of D is much stronger than G, so the loss of G fluctuates during the training stage. In order to maintain the stability of training, the loss function of G is designed as JG VG. Therefore, the global optimal imitation network is obtained and only
A X, DX = (X) At this point, TD = 0 and Ta = e = 1. The loss G is always controllable value in training stage.
As we discussed before, if rı S r2, the adversarial examples generated by a well-trained D have strong transferability for T. Because the attack perturbation of adversarial examples is set to be a small value, we constrain the ||G(X)] in training stage to limit the search space of G, which can reduce the number of queries efficiently.
For training methodology, we alternately train the G and D in every mini-batch, and use L2 penalty constrain the search space G. The procedure shown in algorithm
A Agorithm 1 Mini-batch stochastic gradient descent training of imitation network.
for number of training iterations do
Sample mini-batch of examples x
((m) from training set.
3
Update the imitation model by descending its loss function
4:
-r((X) )(oooo(XXi
(1 yr (X) D(X)).
5: Update the generative model by descending its loss function
6:
e -VGDD + G(X) ||
7: end for
In adversarial attacks, when the optimal is obtained, the adversarial examples generated by are utilized to attack T.
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 EXPERIMENT SETTING
In this subsection, we introduce the settings for our experiments.
Datasets: we evaluate our proposed method on MNIST (LeCun et al. 1998) and CIFAR-10 (Krzhevsky Hinton| 2009). Because we need to use data different with the training set of T to train the imitation network, we divided the test sets (10k) from MNIST and CIFAR-10 into two parts. One part contains 9500 images for training and another part contains 500 images for evaluat- ing the attack performance.
Model architecture and attack method: in order to get the information of the attacked model T as little possible, we only utilize the output label (not the probability) of the T to train the imitation network. The imitation network has no prior knowledge of the attacked model, which means it does not load any pre-trained model in experiments. For the experiments on MNIST, we design 3 different network architectures with different capacity (small network, medium network and large network) for evaluating the performance of our imitation attack with models having different capacity. We utilize the pre-trained medium network and VGG-16 (Simonyan & Zisserman 2013) as the attacked model on MNIST and CIFAR-10, respectively. In order to compare the success rate of the proposed imitation attack with current substitute attack, we utilize 4 attack methods, FGSM, BIM, projected gradient descent (PGD) (Madry et al.
2018), C&W W to generate adversarial examples. For testing, we use Torch'Torc library (Ding et al.
2019) to generate adversarial examples. On the other hand, for comparing the performance of our method with current decision-bart based attacks and score-based attacks, we utilize Boundary Attack (Brendel et al. 2017). HSJA Attack (Chen et al. 2019), SimBA- DCT Attack (Guo et al. 2019) as comparison methods. Note that score-based attacks require output probabilities of T, which contain much more information than labels.
the goals of non-targeted attack and targeted attack are to lead the attacked model to output wrong label and specific wrong label, respectively. In non-targeted attacks, only generate adversarial examples on the images classified correctly by the attacked model. In targeted attacks, we only generate adversarial examples the images which are not classified the specific wrong labels. The success rates of adversarial attack are calculated by n/m, where n and m are the number of adversarial examples which can fool the attacked model and the total number of adversarial examples, respectively.
Evaluation criteria:
4.2 EXPERIMENTAL ANALYSIS OF ADVERSARIAL ATTACK
In this subsection, utilize the proposed adversarial imitation training to train imitation models and evaluate the performance in terms of attack success rate.
To compare method with substitute attack, We utilize the medium network and VGG-16 as attacked models on MNIST and CIFAR-10, respectively. Then we use the same train dataset to obtain a pre-trained large network (the architecture is also in Table and ResNet-50 (He al. 2016) to generate substitute adversarial examples. We obtain imitation networks by using the proposed adversarial imitation training. The large network and ResNet-16 are used as the model architectures of imitation networks on MNIST and CIFAR-10, respectively. The imitation models are only trained by 9500 samples of the test dataset, which are much less than the training sets of MNIST (60000 samples) and CIFAR-10 (50000 samples). The results of experiments are evaluated on the other 500 samples of the test dataset and are shown Table T Table 2 The success rates of the proposed imitation attack far exceed the success rates of the substitute attack in all experiments.
The experiments of Table | and Table 2 show that the proposed new attack mechanism needs less training images than substitute attacks, but achieves an attack success rate close to the white-box attack. The experiments indicate that the adversarial samples generated by a well-trained imitation model have higher transferability than the substitute attack.
To compare our method with decision- based and score-based attacks, we evaluate the performance of these attacks. We utilize 9500 images from the test set of MNIST and CIFAR-10 to train the imitation network, and use other 500 images from the test set as unseen data for our method. The other decision-basee and score-based attacks are evaluated on the test of MNIST and CIFAR-10
5

---

Under review as a conference paper at ICLR 2020
Table 1: Performance of the proposed imitation attack compared with the white-box attacks and substitute attacks on MNIST. The architectures of networks are shown in Table 2 "White-box": generate adversarial examples from the attacked medium network. "Substitute": generate adversar- ial examples from the pre-trained large network. "Imitation": generate adversarial examples from the imitation large network. The numbers in denote the average LF perturbation distance per
image.
Non-targeted (%)
Targeted (%)
Attack
White-box
Substitute
Imitation
White-box
Substitute
Imitation
FGSM
82.90 (5.17)
57.55 (5.177
75.45 (4.78)
29,91 (5.25)
15.81 (5.25)
33.85 (5.17)
BIM
86.52 (3.45)
45.47 (3,45)
70.62 (3.21)
39.96 (3.45)
14.03 (3.45)
28.51 (3.53)
PGD
65.79 (3.76)
28.17 (3.76)
49.70 (3.68)
22.32 (3.84)
7.80 (3.84)
15.37 (3,84)
CW
81.89 (3.06)
38.63 (2.90)
66.00 (3.14)
43.08 (1.88)
14.25 (1.56)
37.32 (1.88)
Table 2: Performance of the proposed imitation attack compared with the white-box attacks and substitute attacks on CIFAR-10. The attacked model is VGG-16.
Non-targeted (%)
Targeted (%)
Attack
White-box
Substitute
Imitation
White-box
Substitute
Imitation
FGSM
84.87(2.73)
39.29 (2.73)
81.30 (2.73)
41.42 (1.66)
8.22 (1.66)
32.65 (1.66)
BIM
98.96 (1.08)
47.27 (1.01)
97.06 (1.14)
67.82 (0.92)
14.38 (0.83)
62.56 (0.95)
PGD
67.44 (1.11)
30.25 (1.14)
67.02 (1.29)
28.17 (0.98)
6.39 (1.01)
25.57 (1.04)
CW
97.69 (1.35)
38.66 (1.35)
70.40 (1.38)
68.60 (1.51)
20.78 (1.41)
45.43 (1.51)
dataset. Note that score-based attacks require much more information (output probabilities) than decision-based attacks (output labels). The results on MNIST and CIFAR-10 are shown in Table 3 and Table respectively. Because our imitation attack only needs queries on the training stage, we evaluate performances of our method on its train and unseen sets. We set the iteration of adversarial imitation training to 1800, so the average number of query per image is 1800. We utilize BIM attack to generate adversarial examples our imitation attack in this experiment.
This experiment shows that our imitation attack achieves state-of-the-ard performance in decision- based methods. Even compared with the score-based attack, our imitation attack outperforms it in terms of perturbation distance and attack success rate, More importantly, it also obtains good results on unseen data, which indicates our imitation attack can be applied to query- encry-independent scenarios.
4.3 EXPERIMENTAL ANALYSIS OF NETWORK CAPACITY
In the above subsection, we utilize a more complex network than the attacked model to replicate the attacked model. In this subsection, we study the impact of model capacity on the ability of imitation.
To evaluate the imitation performance of the network with less capacity than the attacked model, we train the small network, medium network, and large network to imitate the pre-trained medium network on MNIST dataset, and train VGG-13, VGG-16 and ResNer-50 to imitate VGG- on CIF The performance of models with different capacities are shown in Table 5 and 6 The results show that an imitation model with a lower capacity than the attacked model can also achieve good imitation performance. Attack success rates of all imitation models far exceed the substitute attacks in T and 2 Most experiments show that the larger capacity the imitation network has, the higher attack success rate it achieve (FGSM, BIM, PGD in MNIST and BIM in CIFAR- 10). However, some experiments show models having a larger capacity do not have a higher attack success rate (FGSM and PGD in CIF) We surmise that the performance of imitating an attacked model is not only influenced by the capacity of the imitation model D, but also influenced by the capability of the G.
4.4 EXPERIMENTAL ANALYSIS OF MODEL REPLICATION
In this subsection, we only use 200 images (20 samples per class) to train the imitation networks and discuss characteristics of the model replication.

---

Under review as a conference paper at ICLR 2020
Table 3: Performance of the proposed imitation attack compared with the decision-based and score- based attacks on MNIST. "Query": the average number of queries of attacks. "Lp": the average LF distance per image. "imitation-train" the performance of imitation attack on its training data. "imitation-unseed the performance of imitation attack on other unseen data.
Non-targeted
Targeted
Attack
Query
LF
Success rate
Query
LF
Success rate
Score-based
SimBA
752.32
5.41
96.78%
879.86
3.53
82.63%
Decision-based
Boundary
2830.44
9.49
100.0%
2613.31
11.05
81.30%
HSJA
4679.57
6.59
100.0%
2113.06
6.27
75,67
Imitation-train
1800.00
4.94
99.62 %
1800.00
5.33
84.36%
Imitation-unseen
4.94
99.52% %
5.41
82.62%
Table Performance of the proposed imitation attack compared with the decision-based and score- based attacks on C..AR-10.
Non-targeted
Targeted
Attack
Query
LF
Success rate
Query
LF
Success rate
Score-based
SimBA
501.65
1.76
99.37%
489.90
2.06
84. 16%
Decision-based
Boundary 1937.02
3.26
100,0%
1837.01
6.02
35.59%
HSJA
2206.55
2.59
100.0%
1527.84
1.12
32.53%
Imitation-train 1800.00
1.29
99.87%
1800.00
1.19
87.76%
Imitation-unseen
1.29
99.58%
1.19
85.62%
Table 5: Performance of the imitation networks with different capacity on MNIST. The attacked
model is pre-trained medium network. The architectures of networks are shown in Table 9 The adversarial examples generated by the same attack methods have the same perturbation distance.
Non-targeted (%)
Targeted (%)
Attack
Small net
Medium net
Large net
Small net
Medium net
Large net
FGSM
69.82
63.98
75.45
33.18
28.06
33.85
BIM
61.17
57.75
70.62
21.36
18.71
28.51
PGD
42.45
39.84
49.70
10.45
10.47
91.67
We train the imitation network using 200 images from MNIST and CIFAR-10 test set, and compare its performance with Practical Attack (Papernot et al. 2017 on other images from MNIST and CIFAR-10 set. The results are shown in Table and g The practical attack uses the output labels of attacked models to train substitute models under the scenario. which they can make an infinite number of queries for attacked models. the attacked models by limited training samples. Note that what the substitute models imitate is the response for perturbations of the attacked model. A substitute model that can generate adversarial examples with higher attack success rate is better replica. Our adversarial imitation attack can produce a substitute model with much higher classification accuracy and attack success rate than Practical Attack for both non-targeted and targeted attacks in this scenario (with an infinite number of queries). We also show the performances of these two methods with limited query numbers in Figure Additionally, the imitation model with low classification accuracy still can produce
hard generate adversarial examples to fool

---

Under review as a conference paper at ICLR 2020
Table 6: Performance of the imitation networks with different capacity CIFAR-10. The attacked
model is pre-trained VGG-16. The adversarial examples generated by the same attack methods have the same perturbation distance.
Non-targeted (%)
Targeted (%)
Attack
VGG-13
VGG-16
ResNet-50
VGG-13
VGG-16
ResNet-50
FGSM
73.11
85.92
81.30
20.57
26.91
32.65
BIM
81.72
96.85
97.06
38.51
58.21
62.56
PGD
45.59
70.59
67.02
14.22
27.79
25.57
Table Comparisons between imitation model and other substitute models. "Accuracy": the clas- sification accuracy on other images from test set. The numbers in () denote the average LF pertur- bation distance per image.
Non-targeted (%)
Training data
Accuracy
BIM
FGSM
PGD
Practical
200 (test set)
80.43
14.08 (4.94)
9.41 (5.10)
2.63 (3.81)
Imitation
200 (test set)
97.68
71.03 (4.94)
45.17 (5.12)
14.81 (3.78)
Targeted (%)
Training data
Accuracy
BIM
FGSM
PGD
Practical
200 (test set)
80.43
2.51 (4.84)
2.51 (5.14)
1.11 (4.60)
Imitation 200 (test set)
97.68
26.65 (5.01)
11.81 (5.23)
15.66 (4.69)
Table Comparisons between imitation model and other substitute models. "Accuracy": the clas- sification accuracy on other images from test set.
Non-targeted (%)
Training data
Accuracy
BIM
FGSM
PGD
Practical
200 (test set)
29.83
2.31 (1.66)
3.15 (1.65)
2.73 (1.07)
Imitation
200 (test set)
71.09
64.50 (1.32)
45.59 (1.65)
20.80 (1.05)
Targeted (%)
Training data
Accuracy
BIM
FGSM
PGD
Practical
200 (test set)
29.83
1.14 (1.69)
0.91 (1.65)
0.68 (1.26)
Imitation 200 (test set)
71.09
23.06 (1.23)
10.73 (1.65)
12.10 (1.17)
adversarial examples that have well transferability. It shows that our adversarial imitation training can efficiently steal the information from the attacked model of their classification characteristics under perturbations.
CONCLUSION
Practical adversarial attacks should have as little possible knowledge of attacked model T. Cur- rent black-box attacks need numerous training images or queries to generate adversarial images. In this study, address this problem, we combine the advantages of current black-box attacks and pro- posed new attack mechanism, imitation attack, to replicate the information of the T, and generate adversarial examples fooling deep learning models efficiently. Compared with substitute attacks, imitation attack only requires much less data than the training set of T' and do not need the labels of the training data, but adversarial examples generated by imitation attack have stronger transfer- ability for the T. Compared with score-based and decision-based attacks, our imitation attack only needs the same information with decision attacks, but achieves state-of-the-an performances and query-independent on testing stage. Experiments showed the superiority of the proposed imitation attack. Additionally, we observed that deep learning classification model T is easy to be stolen by limited unlabeled images, which much fewer than the training images of T. In future work, will evaluate the performance of the proposed adversarial imitation attack on other tasks except for image classification.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of se- curity: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, July 2018. URLhhttps:/arxivv org/ abs/
18C2. 00420
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Dimensionality reduction as a defense against evasion attacks on machine learning classifiers. arXiv preprint arXiv:1704. 02654, 2017.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based based adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint (rXiv:17/2.04220 2017.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations (ICLR), 2018. StRtsstts::///tt /ttttps://pperreviee wwwprrvviwwweetffooeooooeeeeeeee wwt. net/forum?id=S)
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3-14. ACM, 2017a.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017b.
Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. Hopskipjumpattack: A query-efficient decision- -haied attack, 2019.
Yu Chen, Huan Zhang, Yash Sharma. Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 15-26. ACM, 2017.
Minhao Cheng, Thong Le, Pun-Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsich. Query- efficient hard-label black-box attack: An optimization-balt based approach. arXiv preprint arXiv:1807.0445. 2018.
Guneet S. Dhillon, Kamyar Arizzadenesheli, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, Zachary C. Lipton, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In International Conference on Learning Representations, 2018. URL
htttps://openreiier net nott//oormm3ld--iuu444G2
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: adversarial robustness toolbox based on pytorch. arXiv preprint 1902v 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts, arXiv preprint
2017.
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. arXiv
preprint arXiv: 704.04444990 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference Learning Representations (ICLR), 2015.
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv preprint arXiv:1702.06288. 2017.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple Black-box adversarial attacks. In International Conference on Machine Learning, pp. 2484-2493. 2019.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In / 1Ith Workshop on Offensive Technologies (WOOT 2017), 2017.
Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. International Conference on Learning Representations (ICLR), 2017.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning, pp. 2142- 2151, aa.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint arXiv:1807.0797 2018b.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. International Conference on Learning Representations (ICIR), 2017a.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. Inter- national Conference Learning Representations (ICLR), 2017b.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278- 1998.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial ex- amples and black-box attacks. International Conference on Learning Representations (ICLR), 2017.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Michael E. Houle, Dawn Song, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. International Conference on Learning Representations (ICLR), 2018. URL https:
www. http//orummmidde d= gig.ilg
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations((CLRR)) 2018. URL htttps://oppnreste net.:/ewwweeeeeeeeeeeeeeeeeeeeee wwt.:eee//oooooe net/forumidd
rJzIBtZAb]
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135-147, ACM, 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. International Conference on Learning Representations (ICLR), 2017.
Seyed-Mohsen Deozavi- Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 180.::880000475 2018.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint 1oX5.::605.07277 2016a.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson. Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Sympo- sium on Security and Privacy (FumoS&P), pp. 372-387. IEFF, 2016b.
10

---

Under review as a conference paper at ICLR 2020
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference computer and communications security, pp- 506-519. ACM. 2017.
Jéröme Rony, Luiz G Hafemann, Luis S Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based 12 adversarial attacks and defenses. arXiv preprint arXiv: 1.09600, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. International Conference on Learning Representations (ICLR), 2014.
Florian Tramèr. Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018. URL https: httpp:////ppereveee... net forummf
FkZvSe-RZ
Cihang Xie, Jianyu Wang. Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations (ICLR), 2018. http:////pppppeeeeeeeeeeerrrraaa

---

Under review as a conference paper at ICLR 2020
A NETWORK ARCHITECTURES
Table Network architectures for MNIST. Convolutional kernel (A B, C) denotes the kernel size and channel number, respectively.
ConyBlock
Small net
Medium net
Large net
ConyLayer (A B)C)
ConvBlock (5 x 20)
ConvBlock 20)
ConvBlock (5 x 5,)
ReLU
ConvBlock (5 x 50)
ClovBlock 50)
ConvBloct (5 x 5, 50)
MaxPooling (2 x 2)
DenseLayer
ConvBlock (3 50)
ConvBlock (3 x 3, 50)
ReLU
DenseLayer
ConvBlock (3 x 3, 3,50)
DenseLayer Sigmoid
ReLU
DenseLayer
DenseLayer Sigmoid
ReLU
Denselayer Sigmoid
Table 10: Network architectures for generative model in our experiments.
Layers
Parameter
ConvLayer
kernel: 3 x 3, stride: 1, padding:1, channel: 128
BatchNorm
LeakyRel.U
ConvLayer
kernel: 3, stride: I, padding:1, channel: 512
BatchNorm
LeakyReLL
ConvLayer
kernel: 3, stride:
padding:1, channel: 256
BatchNorm
LeakyReLU
ConvLayer
kernel: 3 x 3, stride: 1, padding:1, channel: 128
BatchNorm
LeakyReI.U
ConvLayer
kernel: x 3, stride: 1, padding: channel: 64
BatchNorm
LeakyReLU
ConvLayer
kemel: 3 stride: 1, padding:1, 1, channel: 3
BatchNorm
LeakyReLU
12

---

Under review as a conference paper at ICLR 2020
B VISUALIZATION OF ADVERSARIAL EXAMPLES
In this section, we visualize the adversarial examples generated by the imitation model. The results are shown in Figure 2 and Figure 3 The experiments show that adversarial examples generated by the proposed imitation attack can fool the attacked model with a small perturbation.
Origin
8
24
FGSM
7
8
47
BIM
PGD 7
2844
7
Figure 2: Visualization some adversarial examples generated by the imitation model on MNIST.
Original examples are in the first row. Examples from the second row to the third row are generated through FGSM, BIM, PGD, respectively.
Origin
RGSM
BIM
PGD
Figure 3: Visualization of some adversarial examples generated by the imitation model CIFAR- 10. Original examples are in the first row. Examples from the second row to the third row are generated through FGSM, BIM, PGD, respectively.
13

---

Under review as a conference paper at ICLR 2020
C VISUALIZATION OF DISTURBANCES GENERATED BY GENERATOR IN TRAINING STAGE
In this section, we visualize the disturbances generated by generator in training stage.
5
9
4
4
4
S
2
7
8
5
5
6
2
I
8
6 9
0
f
(a) Original images
5
9
8
5
4
(b) Generated disturbances
(c) Training samples for the imitation network
Figure 4: Visualization of some disturbances generated by the generator in training stage. The clean images added with the disturbances are used train the imitation network.
14

---

Under review as a conference paper at ICLR 2020
D ACCURACY CURVE OF THE IMITATION MODEL ON TRAINING STAGE
1
0.9
0.8
0.7
accuracy
0.6
0.5
0.4
0.3
MINIST T
CIFAR-10
0.2
0
500
1000
1500
2000
iteration
Figure 5: Accuracy curve the imitation model on the 500 testing samples of MNIST and CIFAR- 10 in adversarial imitation training. The number of training samples is 9500. We show the raw data without filtering.
15

---

Under review as a conference paper at ICLR 2020
E COMPARISON OF ATTACK SUCCESS RATE BETWEEN THE PROPOSED METHOD AND THE PRACTICAL METHOD
70
imitation practical
60
attack success rate
50
40
30
20
10
0
0
50
100
150
200
250
epochs
Figure 6: Comparison of attack success rate between the proposed method and the practical method on the early training stage.
16


Reviews:
[{"review_id": "SkeKJn8y5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use a generative adversarial network to train a substitute that replicates (imitates) a learned model under attack. They then show that the adversarial examples for the substitute can be effectively used to attack the learned model. The idea is straightforward. The proposed approach leads to better success rates of attacking than other substitute-training approaches that require more training examples. Promising experimental results against decision and score-based attach schemes also demonstrate the effectiveness of the proposed approach.\n\nMy main concern is that the comparison to other approaches seem unfair, because\n\n(1) Substitute models typically use all training data and query the learned model once per training example; the proposed approach uses fewer training data but needs 1800 queries per example. So it is not surprising that the proposed approach can be better than substitute models. As the authors focus on \"practical\" scenarios, the number of queries should be fixed as a constraint for all approaches to be fair.\n\n(2) Having said (1), there is not enough detail in this paper to understand the similarity and difference between the proposed approach and decision-based ones (which is claimed to have similar query complexity during training). The authors mix the results with score-based (requiring more information), making the results more confusing to the readers. The authors are encouraged to present more detailed discussions on the comparison with decision-based competitors.\n\nAlso,\n\n(3) One thing that is missing from the experiments is how well the replica clones the original model. All the information in the experiments are somewhat \"indirect\" (success rate, test accuracy, etc.) to answer this question, but there is no direct evidence. Is a well-cloned replica really a better substitute to construct adversarial examples? For instance, for replica with different \"cloning accuracy\" (rather than test accuracy), is a better replica also a better substitute? The paper fails to answer this question that best matches its design motivation.\n\nThe above are my main concerns. A minor one is\n\n(4) The abstract that directly uses notations like T, G and D is horribly hard to read.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklEYLy0tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a GAN-based adv imitation attack that can use less training data to produce a replica of the model.\n\nThe idea of using a GAN in producing adv examples in quite interesting. But the proposed approach is closely related to the following paper:\n\nhttps://arxiv.org/pdf/1801.02610.pdf\n\nTherefore, I am not sure about the novelty of the proposed approach.\n\nAlso can authors comment on the stability of GAN's training? Are any stabilizing methods integrated in GANs being used? \n\nHow does the proposed approach relate to the adversarial distillation literature? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxYga09Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach to conduct adversarial attacks, where an imitation classifier is trained to mimic the behaviours of the targeted/attacked classifier and adversarial attacks can be generated with existing attack methods on the imitation classifier. The training of the imitation classifier only requires the predictions of the targeted classifier, which may fit better in practice with limited access to the targeted classifier.\n\nThe reasons that I am going towards accept are as follows:\n\n1. I feel that the idea of learning an imitation classifier is somehow novel and intuitive, which would improve the applicability of existing adversarial attacks in more realistic cases. In addition, using GAN framework in the training of the imitation classifier is also interesting.\n\n2. The experiments are quite comprehensive and promising, including the comparisons with gradient-based attacks as well as the decision-based attacks. In addition, the different model configurations of the imitation classifier are also reported.\n\nSuggestions:\n\n1. It is a bit unclear of the model configuration of the generator, which seems to be not introduced in details.\n\n2. It would be interesting to visualise what the generator generates.\n\nMinor:\n\n\"sometimes the ability of G is much stronger than G\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use a generative adversarial network to train a substitute that replicates (imitates) a learned model under attack. It then shows that the adversarial examples for the substitute can be effectively used to attack the learned model. The proposed approach leads to better success rates of attacking than other substitute-training approaches that require more training examples. The condition to get a well-trained imitation model is that a sufficient number of queries are obtained from the target model. This paper has valuable contributions by developing an imitation attacker. However, some key issues remain. In particular, I agree with R1 that the average number of queries per image is relatively high, even during training. In the rebuttal, the authors made the assumption that “suppose their method could make an infinite number of queries for target models”, which is unfortunately not realistic. Another point that I found confusing: at testing, I don’t see how you can use the imitation model D to generate adversarial samples (D is a discriminative model, not a generator); it should be G, right?

Decision: Reject