Black-Box Adversarial Attack with Transferable Model-based Embedding
Authors: Zhichao Huang
Tong Zhang
Venue:  (2020)
Keywords: adversarial examples
black-box attack
embedding

Abstract: We present a new method for black-box adversarial attack. Unlike previous methods that combined transfer-based and scored-based methods by using the gradient or initialization of a surrogate white-box model, this new method tries to learn a low-dimensional embedding using a pretrained model, and then performs efficient search within the embedding space to attack an unknown target network. The method produces adversarial perturbations with high level semantic patterns that are easily transferable. We show that this approach can greatly improve the query efficiency of black-box adversarial attack across different target network architectures. We evaluate our approach on MNIST, ImageNet and Google Cloud Vision API, resulting in a significant reduction on the number of queries. We also attack adversarially defended networks on CIFAR10 and ImageNet, where our method not only reduces the number of queries, but also improves the attack success rate.

Published as a conference paper at ICLR 2020
BLACK-BOX ADVERSARIAL ATTACK WITH TRANS- FERABLE MODEL-BASED EMBEDDING
Zhichao Huang, Tong Zhang The Hong Kong University of Science and Technology zhu zhuangbxe connect. hk,
ABSTRACT
We present a new method for black-box adversarial attack. Unlike previous meth- ods that combined transfer- and scored- -aseed methods by using the gradient or initialization low-dimensional embedding using pretrained model, and then performs efficient search within the embedding space to attack an unknown target network. The method produces adversarial perturbations with high level semantic patterns that are easily transferable. We show that this approach can greatly improve the query efficiency of black- adversarial attack across different target network architec- tures. We evaluate our approach on MNIST, ImageNet and Google Cloud Vision API, resulting in a significant reduction the number of queries. We also attack adversarially defended networks on CIFAR 10 and ImageNet, where our method not only reduces the number of queries, but also improves the attack success rate.
surrogate white-box model, this new method tries to learn a
INTRODUCTION
The wide adoption of neural network models in modern applications has caused major security concerns, as such models are known to be vulnerable to adversarial examples that can fool neural networks make wrong predictions (Szegedy al. 2014). Methods to attack neural networks can be divided into two categories based on whether the parameters of the neural network are assumed to be known the attacker: white-box attack and black-box attack. There are several approaches find adversarial examples for black-box neural networks. The transfer-based attack methods first pretrain a source model and then generate adversarial examples using a standard white-box attack method on the source model to attack an unknown target network (Goodfellow et al., 2015 Madry et al. 2018 Carlini & Wagner 2017 Papernot et al. 2016a). The score-based attack requires a loss-oracle, which enables the attacker to query the target network at multiple points to approximate its gradient. The attacker can then apply the white-box box attack techniques with the approximated gradient (Chen et al. 2017
llyas et al. 2018a Tu et al., (018).
major problem of the transfer-basee attack is that can not achieve very high success rate. And transfer-based attack is weak in targeted attack. On the contrary, the success rate of score-based attack has only small gap to the white-box attack but it requires many queries. Thus, it is natural to combine the black-box attack approaches, so that we can take advantage of pretrained white-box source neural network to perform more efficient search to attack an unknown target black-box model.
In fact, in the recent NeurIPS 2018 Adversarial Vision Challenge (Brendel al. 2018), many teams transferred adversarial examples from a source network as the starting point to carry out black-box boundary attack (Brendel et al., 2017). AAttack also used regression network initialization in the score-based attack (Li et al.) 2019a). The transferred adversarial example could be a good starting point that lies close to the decision boundary for the target network and accelerate further optimization. P-RGF (Cheng et al. 2019) used the gradient information from the source model to accelerate searching process. However, gradient information is localized and sometimes it is misleading. In this paper, we push the idea of using pretrained white-box source network to guide black-box attack significantly further, by proposing a method called TRansferable EMbedding based Black-box Attack (TREMBA). TREMBA contains two stages: (1) train encoder-decoder that can effectively generate adversarial perturbations for the source network with a low- -in-ensiiio embedding space; (2) apply NES (Natural Evolution Strategy) of (Wierstra al. 2014) to the

---

Published as a conference paper at ICLR 2020
low-dimensional embedding space of the pretrained generator to search adversarial examples for the target network. TREMBA uses global information of the source model, capturing high level semantic adversarial features that are insensitive to different models. Unlike noise-like perturbations, such perturbations would have much higher transferablity across different models. Therefore we could gain query efficieney by performing queries in the embedding space.
We note that there have been a number of earlier works on using generators to produce adversarial perturbations in the white-box setting (Baluja & Fischer 2018 Xiao al. 2018 Wang & Yu 2019). While black-box attacks were also considered there, they focused on training generators with dynamic distillation. These early approaches required many queries to fine-tune the classifier for different target networks, which may not be practical for real applications. While our approach also relies on a generator, we train as an encoder-decoder that produces a low-dimensional embedding space. By applying a standard black-box attack method such as NES on the embedding space, adversarial perturbations can be found efficiently for a target model.
It is worth noting that the embedding approach has also been used in AutoZOOM (Tu et al. 2018) However, it only trained the autoencoder to reconstruct the input, and did not take advantage of the information of a pretrained network. Although it also produces structural perturbations, these perturbations are usually not suitable for attacking regular networks and sometimes its performance even worse than direetly applying NES to the images (Cheng et al. 2019 Guo et al. 2019). TREMBA, on the other hand, tries to learn an embedding space that can efficiently generate adversarial perturbations for pretrained source network. Compared to AutoZOOM, our new method produces adversarial perturbation with high level semantic features that could hugely affect arbitrary target networks, resulting in significantly lower number of queries.
We summarize our contributions as follows:
We propose TREMBA, an attack method that explores a novel way to utilize the information of pretrained source network to improve the query efficiency of black-box attack target network.
2. We show that TREMBA can produce adversarial perturbations with high level semantic patterns, which are effective across different networks, resulting in much lower queries on MNIST and ImageNet especially for the targeted attack that has low transferablity.
3. We demonstrate that TREMBA can applied to SOTA defended models (Madry et al., 2018 Xie et al. 2018). Compared with other black-box attacks, TREMBA increases success rate by approximately 10% while reduces the number of queries by more than 50%.
2 RELATED WORKS
There have been a vast literature on adversarial examples. We will cover the most relevant topics including white-box attack, black-box attack and defense methods.
White-Box Attack White-box box attack requires the full knowledge of the target model. It was first discovered by (Szegedy et al. 2014) that adversarial examples could be found by solving opti- mization problem with LBBSGS (Nocedal) 1980). Later on, other methods were proposed to find adversarial examples with improved success rate and efficiency (Goodfellow al., 2015 Kurakin ct al. 2016 Papernot et al. 2016b Moosavi-Dezfooe et al.) (0166). More recently, it was shown that generators can also construct adversarial noises with high success rate (Xiao et al. 2018 Baluja &
Fischer 2018).
Black-Box Attack Black-box attack can be divided into three categories: transfer-based score-based and decision- based. is well known that adversaries have high transferablity across different networks (Papernot et al., 2016a). Transfer-based methods generate adversarial noises on a source model and then transter it to an unknown target network. It is known that targeted attack is harder than un- targeted attack for transfer-based methods, and using an ensemble of source models can improve the success rate (Liu et al. (0166). Score-based attack assumes that the attacker can query the output scores of the target network. The attacker usually uses sampling methods to approximate the true gradient (Chen et al. 2017 Ilyas et al. 2018201 Li et al.] 2019a Chen al. 2018). AutoZOOM tried to improve the query efficiency by reducing the sampling space with a bilinear transformation or an autoencoder (Tu et al.] 2018). (Ilyas al. 2018b) incorporated data and time prior accelerate attacking. In
2

---

Published as a conference paper at ICLR 2020
contrast the gradient based method, (Moon al. 2019) used combinatorial optimization to achieve good efficiency. In decision based attack, the attacker only knows the output label of the classifier. Boundary attack and its variants are very powerful in this setting (Brendel et al. 2017: Dong et al. 2019). In NeutIPS 2018 Adversarial Vision Challenge (Brendel et al. 2018), some teams combined transfer-based attack and decision-based attack in their attacking methods (Brunner al. 2018). And in similar spirit, . Attack also used regression network as initialization in score-based attack (Li al. 20 Gradient information from the surrogate model could also be used accelerate the scored based attack (Cheng et al. 2019).
Defense Methods Several methods have been proposed to overcome the vulnerability of neural networks. Gradient masking based methods add non non-differential operations in the model, interrupt- ing the backward pass of gradients. However, they are vulnerable to adversarial attacks with the approximated gradient (Athalye et al., 2018 Li et al.) 2019a). Adversarial training is the SOTA method that can be used to improve the robustness of neural networks. Adversarial training is a minimax game. The outside minimizer performs regular training of the neural network, and the inner maximizer finds a perturbation of the input to attack the network. The inner maximization process can be approximated with FGSM (Goodfellow et al., 2015), PGD (Madry et al., 2018), adversarial generator (Wang& & Yu 2019) etc. Moreover, feature denoising can improve the robustness of neural networks ImageNet (Xie et al. 2018).
BLACK-BOX ADVERSARIAL ATTACK WITH GENERATOR
Consider DNN classifier F(x). Let I E [0, 1 []di((()) be an input, and let F(x) be the output vector obtained before the softmax layer. We denote F(x); as the i-th component for the output vector and y as the label for the input. For un-targeted attack, our goal is find a small perturbation 8 such that the classifier predicts the wrong label, i.e. arg max F(x + 5) * y. And for targeted attack, we want the classifier to predicts the target label i.e. arg max F(x 8) = The perturbation 6 is usually bounded by bp norm: | S E, with small e > O.
Adversarial perturbations often have high transferablity across different DNNs. Given white-box source DNN F, with known architecture and parameters, we can transfer its white-box adversarial perturbation 8, to a black-box target DNN F; with reasonably good success rate. It is known that even if x 5, fails to be an adversarial example, 6, can still act as good starting point for searching adversarial examples using a score-based attack method. This paper shows that the information of F, can be further utilized to train a generator, and performing search on its embedding space leads to more efficient black-box attacks of an unknown target network Ft.
3.1 GENERATING ADVERSARIAL PERTURBATIONS WITH GENERATOR
Adversarial perturbations be generated by generator network G. We explicitly divide the generator into two parts: an encoder & and a decoder D. The encoder takes the origin input I and output latent vector E(x), where dim(z) < dim(x). The decoder takes the input and outputs an adversarial perturbation ò = e tanh (x())) with dim(ō) dim(x). In our new method, we will train the generator G so that 8 E (gr)) can fool the source network F,.
Suppose we have a training set { ...., (In, }n)}. where denotes the input and denotes its label. For un-targeted attack, we train the desired generator by minimizing the hinge loss used in the C& W attack (Carlini & Wagner 2017):
Cuntargr (2,, y:) = max Fa (((etann(G((x)) 21) max (Ettann((((())) + 2z)g -k). (1)
37
And for targeted, we use
Lingee (1:, 1) = max
max F5(5 tanh(G((:)) + ı;); F, tanh(g(a:)) x;)t, =1)),,x))
(2)
j+t
where denotes the targeted class and x is the margin parameter that can be used to adjust transfer- ability of the generator. A higher value of leads to higher transferability to other models (Carlini & | 2017). We focus on Eo norm in this work. By adding point-wise tanh function to an unnormalized output (z),, ). and scaling it with e, stanh(D(=)) is already bounded as |0]|
3

---

Published as a conference paper at ICLR 2020
Therefore we employ this transformation, that do not need to impose the infinity norm con- straint explicitly. While hinge loss is employed in this paper, we believe other loss functions such the cross entropy loss will also work.
3.2 SEARCH OVER LATENT SPACE WITH NES
Given new black-box DNN classifier (x(r). for which we can only query its output at any given point
x. As (Ilyas al. 2018a| Wierstra et al. 2014), we can employ NES approximate the gradient of
properly defined surrogate loss in order to find an adversarial example. Denote the surrogate loss by L. rather than calculating V ( y) directly, NES update by using VgBum~N(6.02) [T +w, y)), which can be transformed into EumN((s.02) [I + w, y) V. log(N (w]ō, o²)]]. The expectation can be approximated by taking finite samples. And we could use the following equation to iteratively update ó:
II (6. 1 sign( Σ [-c.c]
C(x + wk: V log log Na (wxx o))),
k=1
where is the learning rate, b is the minibatch sample size, wi is the sample from the gaussian distribution and [[[-s,s] represents clipping operation, which projects 8 onto the foe ball. The sign function provides an approximation of the gradient, which has been widely used in adversarial attack (yyas et al. 201 Sa Madry et al. 2018). However, it is observed that more effective attacks can be obtained by removing the sign function (Li et al. 2019b). Therefore in this work, we remove the sign function from Eqn and directly use the estimated gradient.
Instead of performing search on the input space, TREMBA performs search on the embedding space The generator G explores the weakness of the source DNN F, so that DD produces perturbations that can effective attack Fg. For a different unknown target network F,, we show that our method can still generate perturbations leading to more effective attack of F,, Given an input x and its label y. we choose a starting point z" = E(r). The gradient of z given by NES can be estimated as:
V [(x + tanh (D(²))) v) Ve.Eu~N(zt [L(x + tanh(D(v)). v)]
(4)
Clx - ((n=
where P4 is the sample from the gaussian distribution (2 o²) Moreover, 2¹ is updated with We do not need to stochastic gradient descent. The detailed procedure is presented in Algorithm 1
do projection explicitly since already satisñess || <.
Next we shall briefly explain why applying NES on the embedding space can accelerate the search process. Adversarial examples can be viewed as a distribution lying around a given input. Usually this distribution is concentrated on some small regions, making the search process relatively slow. After training on the source network, the adversarial perturbations of TREMBA would have high level semantic patterns that are likely to be adversarial patterns of the target network. Therefore searching over 2 is like searching adversarial examples in a lower dimensional space containing likely adversarial patterns. The distribution of adversarial perturbations in this space is much less concentrated. It thus much easier to find effective adversarial patterns in the embedding space.
4 EXPERIMENTS
We evaluated the number queries versus success rate of TREMBA on undefended network in two datasets: MNIST (LeCun et al. 1998) and ImageNet (Russakovsky al. 2015). Moreover, we evaluated the efficiency of our method on adversarially defended networks in CIFARIO (Krizhevsky & Hinton 2009) and ImageNet. We also attacked Google Cloud Vision API to show TREMBA can generalize to truly black-box model We used the hinge loss from Eqn for un-targeted and targeted attack respectively.
and 2 as the surrogate loss
We compared TREMBA to four methods: (1) NES: Method introduced by (Ilyss et al.) 2018a).
but without the sign function for reasons explained earlier. (2) Trans-NES: Take adversarial
Our code is available at httphsttttttttbbbbbbbb
FREMHA
4

---

Published as a conference paper at ICLR 2020
Algorithm 1 Black--ox adversarial attack on the embedding space
Input:
Target Network Fj; Input a and its label or the target class t; Encoder E; Decoder D; Standard deviation d:; Learning rate 77, Sample size by Iterations T; Bound for adversarial perturbation e
Output: Adversarial perturbation ò
1: 20 = {())
2: for -or111 to 7 do
3: Sample Gaussian noise 1/, 1/2, ... , (2
4: Calculate L; Luntarget (z, y) or (z,
5: Update 2t = 2--1 - E. V. Log.W(v (1i(2(-1,0)))
6: end for
7: return = E tanh((((z))
perturbation generated by PGD or FGSM on the source model to initialize NES. (3) AutoZOOM:
Attack target network with an unsupervised autoencoder described in (Tu et al. 2018). For fair comparisons with other methods, the strategy of choosing sample size was removed. (4) P-RGF: Prior-guided random gradient-free method proposed in (Cheng et al. 2019). The p version was compared. We also combined P-RGF with initialization from Trans-NESpop to form a more efficient method for comparison, denoted by Trans-P-RGF.
Since different methods achieve different success rates, need to compare their efficiency at different levels of success rate. For method with success rate 8,, the average number of queries is qi for all success examples. Let q" denote the upper limit of queries,
modified the average number of queries to be o.- = [maxx - s;) qº + 8; 9:l/ max, 8j. which unified the level of success rate and treated queries of failure examples as the upper limit the number of queries. Average queries sometimes could be misleading due to the the heavy tail distribution of queries. Therefore we plot the curve success rate at different query levels to show the detailed behavior of different attacks.
The upper limit on the number of queries was set to 50000 for all datasets, which already gave very high success rate for nearly all the methods. Only correctly classified images were counted towards success rate and average queries. And to fairly compare these methods, we chose the sample size to be the same for all methods. We also added momentum and learning decay for optimization. And we counted the queries one if its starting point successfully attacks the target classifier. The learning rate was fine-tuned for algorithms. We listed the hyperparameters and architectures of generators and classifiers in Appendix B and C
4.1 BLACK-BOL -BX ATTACK ON MNIST
We trained four neural networks on MNIST, denoted by ConvNetl, ConvNet1 ConvNet2 and FCNet. ConvNetl* and ConvNet1 have the same architecture but different parameters. All the network achieved about 99% accuracy. The generator was trained on ConvNet] * using all images from the training set. Each attack was tested on images from the MNIST test set. The limit of too was 0.2.
We performed un-targeted attack on MNIST. Table D lists the success rate and the average queries. Although the success rate of TREMBA is slightly lower than Trans-NES in ConvNetl and FCNet, their success rate are already close to 100% and TREMBA achieves about 50% reduction of queries compared with other attacks. In contrast to efficient attack ImageNet. P-RGF and Trans-P- P-RGF behaves very bad on MNIST. Figure 4.T shows that TREMBA consistently achieves higher success rate at nearly all query levels.
4.2 BLACK- -BX ATTACK ON IMAGENET
We randomly divided the ImageNet validation set into two parts, containing 49000 and 1000 images respectively. The first part was used the training data for the generator G, and the second part was used for evaluating the attacks. We evaluated the efficiency of all adversarial attacks on VGG19 (Simonyan & Zisserman 2014), Resnet34 (He et al. 2016), DenseNet121 (Huang et al. (017) and MobilenetV2 (Sandler et al. 2018). All networks were downloaded using torchvision package. We set 0.03125.
5

---

Published as a conference paper at ICLR 2020
Table 1: Success rate and average queries of un-targeted attack MNIST. 0.2
ConvNetl
ConvNet2
FCNet
Attack
Success
Queries
Success
Queries
Success
Queries
NES
97.88%
4380
90.32%
5428
99.98%
1183
98.65 %
2113
90.22%
4691
99.99%
818
Trans-NEss NESFGGM
98.34%
3592
91.32%
4218
99.99%
1540
AutoZOOM
93.39%
5874
91.21%
2645
99.69%
823
P-RGF
68.53%
16135
39.85%
29692
90.42%
8289
Trans-P P-RGF
66.34%
16428
27.57%
35576
68.39%
18818
TREMBA
98.00%
1064
92.63%
1359
99.75%
470
POMC
-01
Figure 1: Success of un-targeted attack at different query levels for undefended MNIST models.
Following (Liu ct al. 2016), used ensemble (VGG16, Resnet18, Squeezenet (landola et al. 2016) and Googlenet (Szegedy et al. 2015)) as the source model to improve transferablity (Liu et al. 2016) for both targeted and un-targeted attack. TREMBA, Trans-NES, P-RGF and Trans-P-RGF all used the same source model for fair comparison. We chose several target class. Here, we show the result of attacking class 0 (tench) in Table and Figure 2 And leave the result of attacking other classes in Appendix [ The average queries for TREMBA is about 1000 while nearly all the average queries for other methods more than 6000. TREMBA also achieves much lower queries for un- un-targeted attack on ImageNet. The result is shown in Appendix A.2 due to space limitation. And we also compared TREMBA with CombOpt (Moon al.
2019) the Appendix A.9
Figure 3 shows the adversarial perturbations of different methods. Unlike adversarial perturbations produced by PGD, the perturbations of TREMBA reflect some high level semantic patterns of the targeted class such the fish scale. As neural networks usually capture such patterns for classification, adversarial perturbation of TREMBA would be more easy to transfer than noise-like perturbation produced by PGD. Therefore TREMBA can search very effectively for the target network. More examples of perturbations of TREMBA are shown Appendix [A.3]
Choice of ensemble: We performed attack on different ensembles of source model, which is shown in Appendix A.4 TREMBA outperforms the other methods in different ensemble model. And more source networks lead to better transferability for TREMBA, Trans-NES and Trans-P-RGF.
Varying s: We also changed & and performed attack on E = 0.02 and 0.04. As shown in Appendix A.5 TREMBA still outperforms the other methods despite using the G trained on E = 0.03125. We also show the result of TREMBA for commonly used e = 0.05.
Sample size and dimension the embedding space: To justify the choice of sample size, per- formed a hyperparameter sweep over and the result is shown in Appendix A.6 And we also changed the dimension of the embedding space for AutoZOOM and Trans-P-RGi As shown in Appendix A.7 the performance gain of TREMBA does not purely come from the diminishing of dimension of the embedding space.
4.3 BLACK-BOL Box ATTACK ON DEFENDED MODELS
This section presents the results for attacking defended networks. We performed un -n-targeted attack two SOTA defense methods CIFAR10 and ImageNet. MNIST not studied since is already

---

Published as a conference paper at ICLR 2020
Table 2: Success rate and average queries of black-box targeted attack ImageNet. Targeted class is
class 0 (tench). 0.03125
VGG19
Resnet34
DenseNet121
MobilenetV2
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
NES
94.86%
12283
93.89%
14418
95.65%
12538
97.76%
10276
Trans-No NED
96.26%
6854
95.97%
8737
96.59%
8627
98.04%
9375
Trans-NES NES
90.85%
12885
91.81%
14090
93.61%
12859
97.48%
9983
AutoZOOM
25.80%
40195
26.25%
39681
31.98%
37628
27.03%
39689
P-RGF
96. 06.12%
6951
90.28%
10221
91.84%
11563
88.94%
14596
Trans-P-RGF
98.06%
2262
93.61 %
6309
94.69%
7263
91.60%
10048
TREMBA
98.47%
853
96.38%
1206
98.50%
1124
99.16%
1210
200
17
Figure The success rate of black-box adversarial targeted attack at different query levels for ImageNet models. The targeted class is tench
Taget Class
TRENDA
Trans
Tranu-P-iC
Figure 3: Visualization of adversarial perturbations targeted at tench
robust against very strong white-box attacks. For CIFARIO, the defense model was going through PGD minimax training (Madry et al. 2018). We directly used their model the source network" denoted by WResnet. To test whether these methods can transfer to defended network with a different architecture, ImageNet, we used the SOTA mode from (Xie et al. 2018). We used "ResNet152 Denoise" as the source model and transfered adversarial perturbations to the most robust "ResNeXt101 DenoiseAll". Following the previous settings, we set E 0.03125 for both CIFAR10 and ImageNet.
trained defended ResNeXt (Xie et al. 2017) using the method. For
As shown Table TREMBA achieves higher success rates with lower number of queries. TREMBA achieves about 10% improvement of success rate while the average queries are reduced by more than 50% on ImageNet and by 80% on CIFARIO. The curves Figure 4(a) and ((b) detailed behaviors. The performance of AutoZOOM surpasses Trans-NE: on defended models. We suspect that ow-frequency adversarial perturbations produced by AutoZOOM will be more suitable to fool the defended models than the regular networks. However, the patterns learned by AutoZOOM are still worse than adversarial patterns learned by TREMBA from the source network.

---

Published as a conference paper at ICLR 2020
Table Success rate of average queries of black-box un-targeted attack defended CIFAR10 and ImageNet model. Source network is WResNet and ResNet152 Denoise.
CIFARIO ResneXt
ImageNet RexneXt101 DenoiseAll
Attack
Success
Queries
Success
Queries
NES
32.17%
24521
29.72%
26526
Trans-NESPGD
32.92%
20735
32.84%
20446
Trans-NESRGsM
33.17%
20873
33.66%
18547
AutoZOOM
33.70%
14870
38.75%
14605
P-RGF
22.37%
25818
32.51%
17926
Trans-Pa Trans-P-RGF
20.88%
27222
31.03%
19262
TREMBA
42.73%
2528
49.59%
5985
TREMBAgs:
41.56%
4994
50.41 %
4771
Recheltt
Rssteeo0201 DenotreNii
Succesa Faato
Success
Queries
Queries
(a) CIFARIO
(b)
ImageNet
Figure 4: The success rate at different query levels for defended CIFARIO and ImageNet models. (a)CIFARIO; (b)ImageNet.
Table 4: Success rate and average queries of un-targeted attack of 10 images on Google Vision API.
Method NES
AutoZOOM
Trans-NESpp
P-RGF
Trans-P-RGF
TREMBA
Success 70.00%
20.00%
70.00%
50.00%
60.00%
90.00%
Queries 245
410
114
324
167
8
An optimized starting point for TREMBA: E(x) is already good starting point for attacking undefended networks. However, the capability of generator limited for defended networks (Wang
& Yu 2019). Therefore, 2g may not be the best starting point get from the defended source network. To enhance the usefulness of the starting point, we optimized z on the source network by gradient descent and found
z arg min max (F((taan())))) x), max F& tanh(D()))) +)); -)).-K)
(5)
The method is denoted by TREMBAosP (TREMBA with optimized starting point). Figure 4 shows TREMBAoSP has higher success rate at small query levels, which means its starting point is better than TREMBA.
4.4 ATTACK GOOGLE CLOUD VISION API
also attacked the Google Cloud Vision API, which was much harder to attack than the single neural network. Therefore we set 0.05 and perform un-targeted attack on the API, changing the topl label to whatever is not on topl before. We chose 10 images for the ImageNet dataset and set query limit to be 500 due high cost to the API. As shown Table TREMBA achieves much higher accuracy success rate and lower number of queries. We show the example of successfully attacked image in Appendix A.8
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
We propose a novel method, TREMBA, to generate likely adversarial patterns for an unknown network. The method contains two stages: (1) training an encoder-decoder to generate adversarial perturbations for the source network; (2) search adversarial perturbations on the low-dimensional embedding space of the generator for any unknown target network. Compared with SOTA methods, TREMBA learns an embedding space that is more transferable across different network architectures. achieves two six times improvements in black-box adversarial attacks on MNIST and ImageNet and it is especially efficient in performing targeted attack. Furthermore, TREMBA demonstrates great capability in attacking defended networks, resulting in a nearly 10% improvement on the attack success rate, with two six times of reductions in the number of queries. TREMBA opens up new ways to combine transfer-based and score-based attack methods to achieve higher efficiency in searching adversarial examples.
For targeted attack, TREMBA requires different generators to attack different classes. We believe methods from conditional image generation (Mirza & Osindero, 2014) may be combined with TREMBA to form a single generator that could attack multiple targeted classes. We leave as future work.
REFERENCES
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 2018. URL http://proceedings mlr
press/ / athalylee 8a ht.ll
Shumeet Baluja and lan Fischer. Learning to attack: Adversarial transoormatonooooo networks. In Proceed ings of AAAI-2018, 2018. URL http: http://wwsspppp http::/wppppcctt..ccoooopp/aaaaa pdf
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable
attacks against black-box machine learning models. arXiv preprint
2017.
Wieland Brendel, Jonas Rauber, Alexey Kurakin, Nicolas Papernot, Behar Veliqi, Marcel Salathé, Sharada P Mohanty, and Matthias Bethge. Adversarial vision challenge. In 32nd Conference on Neural Information Processing Systems (NIPS 2018) Competition Track, 2018. URL https
7/arxiv. wwwxivorrg/ab/////00 01976
Thomas Brunner, Frederik Diehl, Michael Truong Le, and Alois Knoll. Guessing smart: Biased sampling for efficient black-box adversarial attacks. arXiv preprint arXiv: 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Jinghui Chen, Jinfeng Yi, and Quanquan Gu. A wolfe framework for efficient and effective adversarial attacks. arXiv preprint arXiv: 10.2. 2018
Pin-Ye Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26. ACM, 2017.
Shuyu Cheng, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with transfer-based based prior. arXiv preprint arXiv:1906.000 ww... 06919, 2019.
Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu. Efficient decision-based black-box adversarial attacks on face recognition. arXiv preprint arXiv: (tt:::904.0444, 2019.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
arxiv. arxiv.org/abee wrgg/bs//////////////////////// 6572

---

Published as a conference paper at ICLR 2020
Chuan Guo, Jacob R Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Q Weinberger. Simple black-box adversarial attacks. arXiv preprint 1rXiv:1905.07121, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4747 07000-478, 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprimt 1602.0736, 2016.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2137-2146, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018a. PMLR.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint 1rXiv:1807.077 0797777 2018b.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint 16.00366 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324, 1998.
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong, Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. arXiv preprint arXiv: /0t5::/900000444 2019a.
Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. NATTACK: A STRONG AND UNIVERSAL GAUSSIAN BLACK-BOX ADVERSARIAL ATTACK, 2019b. URL https
//openreview net / forum? id ryyooxxX
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv: ItI1: 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant adversarial attacks. In International Conference on Learn- ing Representations, 2018. Utt/httpss://ppenreee
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 2014.
Seungyong Moon, Gaon An, and Hyun Oh Song. Parsimonious black-box adversarial attacks via efficient combinatorial optimization. arXiv preprint
2019.
Seyed- Mooosen Moosavi-Dezfool Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method fool deep neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation, 35(151):773-78,, 1980.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-bo box attacks using adversarial samples. arXiv preprint arXiv: (XXv:1605007277 2016a.
10

---

Published as a conference paper at ICLR 2020
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016b.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh
Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http::p://a....oooo///aaa/////// 1112. 6199
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Chun-Chen Tu, Paishun Ting, Pin-Yu Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: butoencoder-basee zeroth order optimization method for attacking black-box neural networks. arXiv preprint 1805. 2018.
Huaxia Wang and Chun-Nam Yu. A direct approach to robust deep learning using adversarial networks. In International Conference on Learning Representations, 2019. URL https:// openreview net. htte//orum??m00 11d=2111M05=FFF
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949-99))) 980, 2014.
Chaowei Xiao, Bo Bi,,, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks, 2018. URL HkrbyObC-
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. arXiv preprint 1rXiv:1812.03411,, 2018.
Saining Xic, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.

---

Published as a conference paper at ICLR 2020
A EXPERIMENT RESULT
A.1 TARGETED ATTACK ON IMAGENET
Figure 9 shows result of the targeted attack dipper, American chameleon, night snake, ruffed grouse and black swan. TREMBA achieves much higher success rate than other methods at almost all queries level.
A.2 UN-TARGETED ATTACK ON IMAGENET
We used the same source model from targeted attack the source model for un-targeted attack. report our evaluation results in Table and Figure Compared with Trans-P-RGF, TREMBA reduces the number of queries by more than half in ResNet34, DenseNet121 and MobilenetV2. Searching in the embedding space generator remains very effective even when the target network architecture differs significantly from the networks in the source model.
Table 5: Success rate and average queries of un-targeted attack on ImageNet. 5 = 0.03125
VGG19
Resnet34
DenseNet121
MobilenetV2
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
NES
100%
924
100%
1255
100%
1235
99.86%
872
TrESS-NESPGD
100 %
441
100 %
827
100%
838
100%
733
Trans-NESpost NESP
100 %
586
100%
982
100%
961
100%
648
AutoZOOM
98%
5184
96.25%
3754
94.56%
4567
95.38%
4213
P-RGF
100 %
277
99.72%
635
100%
709
99.72%
730
Trans-P-Re P-RF
100 %
130
99.86%
371
99.18%
806
99.86%
522
TREMBA
100% %
100
183
100%
172
100 %
- Traniiiiive
Figure 5: The success of un-targeted black-box adversarial attack at different query levels for undefended ImageNet models.
A.3 VISUALIZATION TARGETED PERTURBATION
Figure 10 shows some examples of adversarial perturbations produced by TREMBA. The first column one image of the target class and other columns are examples of perturbations (amplified by 10 times). It is easy discover some features of the target class in the adversarial perturbation such as the feather for birds and the body for snakes.
A.4 EXPERIMENTS ON DIFFERENT ENSEMBLES
We chose two more source ensemble models for evaluation. The first ensemble contains VGG16 and Squeezenet. And the second ensemble is consist of VGG16. Squeezenet and Googlenet. Figure 0 shows our result for targeted attack for ImageNet. We only compared Trans-NESpGp and Trans-P RGF since they are the best variants from Trans-NES and P-RGE
12

---

Published as a conference paper at ICLR 2020
110
26.
Suite
vontaven
50
104
344
127
Figure 6: We show the success rate at different query levels for targeted attack for different ensem- ble source networks. V represents VGGI6; S represents Squeezenet; G represents Googlenet; R represents Resnet18
A.5 VARYING
We chose 0.02 and e 0.04 and performed targeted attack on ImageNet. Although TREMBA used the same model that is trained on 0.03125, it still outperformed other methods, which shows that TREMBA can also generalize to different strength of adversarial attack with different e.
For the commonly used 0.05, TREMBA also performs well. The results are shown in Table 6 Table and Figure
A.6 VARYING SAMPLE SIZE
We performed hyperparameter sweep over on Densenet1 211 on un-targeted attack on ImageNet. b = 20 may not be the best choice Trans-NES, but it not the best for TREMBA, either. Generally, the performance is not very sensitive to b, and TREMBA will also outperform other methods even if we fine-tune the sample size for all the methods.
A.7 DIMENSION THE EMBEDDING SPACE
We slightly changed the architecture of the autoencoder by adding max pooling layers and changing the number of filters and perform un un-targeted attack on ImageNet. More specifically, we added additional max pooling layers after the first and the fourth convolution layers and changed the number of filters the last layer in the encoder to 8. Thus, the dimension of the embedding space would be 8 x 8 8. And also changed the factor of bilinear sampling in the decoder. The remaining
13

---

Published as a conference paper at ICLR 2020
(a) Targeted Attack 0.02
Pesnciti E.
(b) Targeted Attack & = 0.04
Figure 7: We show the success rate at different query levels for attack at different for ImageNet.
Table 6: Success rate and average queries of un un-ttteee attack on ImageNet. = 0.05
VGGT9
Resnet34
DenseNer121
MobilenetV2
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
NES
100%
651
100%
850
100%
840
99.86%
640
100% %
74
100% %
196
100%
235
100%
169
Trans-NES NESPGSM
100%
232
100%
401
100%
361
100%
272
AutoZOOM
99.72%
1743
99.58%
1481
99.32%
1730
99.29%
1672
P-RGF
100%
178
100%
328
100%
436
100%
402
Trans-P-RCH P-RFF
100% %
44
99.44%
418
98.09%
1049
100 %
157
TREMBA
100%
100%
27
100%
19
100%
Table 7: Success rate and average queries of targeted attack on ImageNet. = 0.05
VGG19
Resnet34
DenseNet121
Mobilenet
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
NES
99.03%
6364
98.89%
8003
99.32%
7525
99.728
5610
Trans-NESpcr
99.31%
1968
99.31% %
3549
99.46%
3731
99.86%
3223
Trans-NESpcse
99.03%
4997
98.33%
7298
98.23%
6874
99.169
5034
AutoZOOM
51.04%
30032
52.36%
28547
60.00%
25836
53.78%
28356
P-RGF 99.17%
3704
98.05%
5498
97.96%
5769
98.17%
6896
Trans-P-RGEP
99.58%
662
99.31%
1896
99.05%
2267
99. 16%
3192
TREMBA 99.72%
285
99.44%
443
99.72%
224
99.72% %
422
As shown in Table this autoencoder is even worse than the original autoencoder despite small dimension of the embedding space. In addition, we also changed to dimension of the data-dependent prior of Trans-P-RGF to match the dimension of TREMBA, whose performance is also not better than before. They show that simply diminishing the size of the embedding space may not lead to better performance. The performance gain of TREMBA comes beyond the effect of diminishing the dimension the embedding space.
settings are the same in Appendix A.2
A.8 EXAMPLES OF ATTACKING GOOGLE CLOUD VISION API
Figure shows one example of attacking Google Cloud Vision API. TREMBA successfully make the shark be classified as green. Compared with Trans-NESpcD. TREMBA hugely changes the
14

---

Published as a conference paper at ICLR 2020
(a) Un-targeted Attack 0.05
(b) Targeted Attack & = 0.05
Figure 8: We show the success rate at different query levels for targeted and un- un-targeted attack at 0.05 for ImageNet.
Table 8: Hyperparameter sweep b on Densenet 21 for un-targeted attack ImageNet
b = 10
= 30
b = 40
b == 50
Sweep over 6
Success
Queries
Success
Queries
Success
Queries
Success
Queries
NES
100%
1323
100%
1284
100%
1433
100%
1639
Trans-NESpcD
100%
915
100%
791
100%
707
100%
639
Trans-NESposA NESPosM
100%
1037
100%
916
100%
879
100%
886
AutoZOOM
90.9%
6052
96.2%
4148
97.1%
4066
97.3%
4366
P-RGF 99.73%
717
99.86%
860
99.86%
949
99.86%
1095
Trans-P-RGF 98.50%
1139
99.86%
479
99.86%
487
100%
427
TREMBA
100%
150
100%
205
100%
274
100%
299
Table Change dimension of the embedding space of AutoZOOM. The task is un-targeted attack on ImageNet.
VGG19
Resnet34
DenseNet121
MobilenetV2
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
AutoZOOM
64.35%
19684
71.94%
16931
68.44%
17871
71.15%
16134
Trans-P-RGF
99.86%
194
99.58%
508
99.59%
610
99.58%
705
labels of the image. It hard to say the overall classification of Trans-NESpGp is wrong. However, the labels of TREMBA are definitely not correct.
15

---

Published as a conference paper at ICLR 2020
12
(a) Dipper
V0311
American chamelcon
(c) Night snake
(d) Ruffed grouse
(e) Black swan
Figure 9: The success rate at different query levels for attack targeted at different class. Targeted classes are: (a)Dipper: (b)Amcrican chameleon; Night snake; (d)Ruffed grouse; (e)Black swan
16

---

Published as a conference paper at ICLR 2020
Figure 10: Visualization of adversarial perturbations for targeted attack on ImageNet. The first column shows one example of the target class. Other columns show the adversarial perturbations.
Siae
Tige-bbu
Litrslee Mireieeee
Lumitte
(a) Origin Image
(b) TREMBA
(c) Trans-NESpon
Figure 11: One example of adversarial image for attacking Google Cloud Vision API
17

---

Published as a conference paper at ICLR 2020
Table 10: Comparision between CombOpt and TREMBA for un-targeted attack Imagenet.
VGGT9
Resnet34
DenseNet121
MobilenetV2
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
CombOpt TREMBA
100 %
567
100%
499
100%
569
100% %
522
100% %
88
100%
183
100 %
172
100%
61
Table 11: Comparision between CombOpt and TREMBA for targeted attack Imagenet.
VGG19
Resnet34
DenseNet121
MobilenetV2
Attack
Success
Queries
Success
Queries
Success
Queries
Success
Queries
CombOpt 93.76%
9767
86%
8024
97.41%
6970
96.92%
8575
TREMBA 98.47% %
853
96.38% %
1206
98.50%
1124
99.16%
1210
A.9 COMPARISION BETWEEN TREMBA AND COMBOPT
CombOpt is one of the SOTA score-based black-box attack. We compared our method with it on the targeted and un-targeted attack on Imagenet. The targeted attack is 0 and E = 0.03125. As shown in Table IO and Table [T] TREMBA requires much lower queries than CombOpt. demonstrates the great improvement by combining the transfer-based and score-based attack.
B ARCHITECTURE OF CLASSIFIERS AND GENERATORS
B. CLASSIFIER
Table 12: Model architectures for the MNIST
ConvNetl
ConvNet2
FCNet
C6nv(64, 5, 5)+ReLU
Conv(16, 3, 3)+ReLU
FC(512)RRLL
MaxPool(2,2)
Conv(16, 3, 3)+ReLU FC(10)
Conv(64, 5, 5)+ReLU
(axPool(222)
MaxPool(2.2)
Conv(32, 3, 3)+ReLU
Dropout(O.25)
Conv(32, 3, 3)+ReLU
FC(128)+ReLU
Conv(32, 3, 3)+ReLU
Dropout(0.5)
MaxPool(2.2)
SC(10)+
FC(512)+ReLU FC(10)+Softma
lists the architectures of ConvNetl, ConvNet2 and FCNet. The architecture of Xt Table 12
used in CIFAR1O is from ht // gitu........ com cwm/prla. prlp 227 /ReeNeett....e pytorch| We set the depth to be 20, the cardinality to be and the widen factor to be 4. Other architectures of classifiers specified in the corresponding paper.
B.2 GENERATOR
Table 13 lists the architectures of generator for three datasets. For AutoZOOM, we find our architec- tures are not suitable and use the same generators in the corresponding paper.
C HYPERPARAMETERS
C.1 TRAINING GENERATOR
We trained the generators with learning rate starting at 0.01 and decaying half every 50 epochs. The whole training process was 500 epochs. The batch size was determined by the memory of GPU. Specifically, we set batch size to be 256 for MNIST and CIFAR10 defense model, 64 for ImageNet model. All large a will work well for our method and we chose x = 200.0. All the experiments were performed using pytorch NVIDIA RTX 2080Ti.
18

---

Published as a conference paper at ICLR 2020
Table 13: Architectures of encoder and decoder. ConvReI.UBN and DeconvRel.UBN represent convolution or deconvolution followed by ReLU and batch normalization. The parameters (c, m, n) used in ConVReLUBN or DeconvReLUB), mean channels with kernel size. MaxPool(m,,) n)
represents max pooling with (m, n) kernel size and (m, n)) stride.
MNIST
CIFARIO
ImageNet
ConvReLUBN((6.333
ConvReLUBN(16333
ConvReLUBN(I6.33
ConvReI
ConvReL.UBN(32.3.3)
ConvReL.UBN(22.3.33)
ConvReLJUBN(32.3333
ConvRel.UBN(32.3.33
MaxPool(2.2)
MaxPooo((..2)
MaxPool(2.2)
ConvReLUBN(64,33)
ConvReLUBB((32.33
ConvReLUBN(22.333
ConwReLUUBN(64.33.
ConvReLLUBN((6.3333
ConvReL.UBN(32.333
MaxPool((2,2)
Encoder
ConvReL.UBN(2.3333
ConvReLUBN(8,55
ConvReLUUBN(12.,
MaxPool(2,22)
(2xPool(2,2)
CanvReLUBN(122.333
MaxPool(2.2)
Con<ReL.UBN(32.335
(0n,ReLLUBN88333)
MaxPool(2.2)
ComvReLLBB(3323333
ConvReLUBN(323333)
ConvReLUBN(32.333
ConvReLUBN(332.33)
ConvReLUBN(123333
Deconv
DecomvReL.UBNG44..333
DeconvRe1.UBN644.33
ConvReLUBBN(12.33.
(3nv3eLUUBN(6433333
Con4ReLUBN((44333)
DeconVReLUBNN(12..33
Decoder
ConvReLLUBN(6433333
ConvReLUBN(64333)
ConvReL.UBN(128.333))
(econ.ReL.UBB((663333
DeconVReLUUBN(163.33
Decomv/el.UUBN644...33
Conv(1,1,1)
Conv(3,1,1)
ComvRel.UBN(32.33)
DeconvReLUBN(16.33
ConvReLUBN(3.1.I)
C.2 EVALUATION
Table 14 to 19 list the hyperparameters for all the algorithms. The learning rate was fine-tuned for all the algorithms. We set sample size 20 for all the algorithms for fair comparisons.
Table 14: Hyperparameters for NES
ImageNet
MNIST
CIFARI0
Un- terttrgeted
Targeted
Un-targeted Defense
Sample size (b)
20
20
20
20
Learning rate (n)
0.2
0.05
0.1
0.05
0.1
Table 15: Hyperparameters for Trans-NESpap and Trans-NESyOsm- White-box iteration, white-box margin and white-box learning rate mean the hyperparameters for generating the starting point on the source network for Trans-NESpcp-
ImageNet
MNIST CIFAR10
Un-targeted
Targeted
Un-targeted Defense
Sample size (b)
20
20
20
20
20
Learning rate (n)
0.2
0.05
0.1
0.05
0.1
White-box iteration
50
100
50
50
100
White-box margin(x)
100
100
100
100
100
White-box learning rate
0.05
0.1
0.01
0.005
0.1
19

---

Published as a conference paper at ICLR 2020
Table 16: Hyperparameters for AutoZOOM.
ImageNet
MNIST
CIFARIO
Un-targeted
Targeted
Un-targeted Defense
Sample size (b)
20
20
20
20
20
Learning rate (77)
5.0
20.0
5.0
3.0
5.0
Table 17: Hyperparameters for P-RGF and Trans-P-RGE
ImageNet
MNIST
CIFARIO
Un Un-targeted
Targeted
Un-targeted Defense
Sample size (b)
20
20
20
20
20
Learning rate (1)
0.1
0.05
0.005
0.003
0.005
White-box iteration
50
100
50
50
100
White- box margin(A)
100
100
100
100
100
White-box learning
0.05
0.1
0.01
0.01
0.1
Table 18: Hyperparameters for TREMBA.
ImageNet
MNIST
CIFARIO
Un-targeted
Targeted
Un-targeted Defense
Sample size (b)
20
20
20
20
Learning rate (1))
0.3
2.0
5.0
3.0
5.0
Table 19: Hyperparameters for TREMBAosP
CIFARIO Defense
ImageNet Defense
Sample size (b)
20
20
Learning rate (1)) White-box iteration
2.0
5.0
100
100
White-box margin(k)
100
100
White-box learning rate
1.0
2.0
20


Reviews:
[{"review_id": "rJgyVOG6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new method for black-box adversarial attacks which tries to learn a  low-dimensional embedding using a pretrained model and then performs efficient search within the embedding space to attack the target network. The proposed method can produce perturbation with semantic patterns are easily transferable. It can be used to improve the query efficiency in black-box attacks. \n\n- The main idea of this paper is quite simple, i.e., using a autoencoder model to capture encoding in the embedding space and then searching over the embedding space for possible attacks. While searching for adversarial examples in the embedding space is not something new, such as manifold attack or GAN-based attack, the authors claimed that by doing so, it can help reduce the query complexity of black-box attacks. However, it is not immediately clear to me why this can help reduce the query complexity, as intuitively, restricting the attack image space to an embedding space (or a manifold) will naturally increase the difficulty for finding adversarial examples. The authors’ explanation is not quite convincing to me since many adversarial examples are not necessarily on the embedding space.\n\n- Algorithm 1 is not clearly written and I do not understand the update rule in Algorithm 1. What is Li exactly? If Li means eq(1) or eq(2), it seems totally independent of sampled Guassian noise? Also, the update rule in Line 5 of Algorithm 1 is different from what is described in eq(4) or other black-box attack algorithms. Can the author explain the algorithm design with details?\n\n- In experiments section, the authors miss a few important black-box attack baselines. I would suggest the authors to further comment and compare with the following black-box attacks to better demonstrate the performance of the proposed algorithm.\n\nIlyas, Andrew, Logan Engstrom, and Aleksander Madry. \"Prior convictions: Black-box adversarial attacks with bandits and priors.\" ICLR 2019.\nMoon, Seungyong, Gaon An, and Hyun Oh Song. \"Parsimonious Black-Box Adversarial Attacks via Efficient Combinatorial Optimization.\" ICML 2019.\nChen, Jinghui, Jinfeng Yi, and Quanquan Gu. \"A Frank-Wolfe Framework for Efficient and Effective Adversarial Attacks.\" arXiv preprint arXiv:1811.10828 (2018).\n\n- Also can the authors further conduct experiments using more common choice of \\epsilon to help the reader get better understandings. For example, add ImageNet experiments with \\epsilon = 0.05.\n\nDetailed comments:\n\n- In section 3.2, the author suggests that by removing the sign function, the attacks can be more effective in (Li et al., 2019).  I didn’t find the corresponding argument in (Li et al., 2019). Can the authors be more specific on this argument?\n\n-------------------------\nI have read the response and it addressed my concerns. I will increase my score", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1l9pBNPqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new black-box adversarial attack method called TREMBA, in which the search for the “adversary” is done in a reduced space z. Summary of its contributions:\n-\tA attack method that improves query efficiency of black-box attack\n-\tProduces perturbations that are effective across different networks\n-\tImproves attack success over SOTA defended networks \n\nIn general, the paper is very well written, with clear mostly clear exposition and sufficient experimental verification. What follows are the itemized pros and cons (mostly just points that would be good to address):\n\n[pros]:\n-\tWell written\n-\tA good overview of previous methods and how the TREMBA fits within them\n-\tSufficient experimental validation\n\n[points to address]\n-\tIn Black-Box Attack method of related works, did you mean to say, “Targeted attack is much harder than un-targeted attack for transfer-based method.”?\n-\tYou ought to explain what NES is - Natural Evolution Strategies – and the general description of the method, as it is a major part of your algorithm (Section 3.2). It took two papers to find what NES stands for.\n-\tIn section 3.2 you write – “The sign function provides an approximation of the gradient, …” – is there a citation that should go with this claim? \n-\tMake sure to explain all of the variables in the paper, e.g. $\\omega_k$ in Eq. (3) or $\\nu_k$ in Eq. (4).\n-\tIS there a particular reason you chose the hinge loss to train the generator? Could you have used other losses instead?\n-\t“A higher value of $\\kappa$ laeds to higher transferability to other models” – maybe a citation required? Or else more intuition?\n-\tAdding specifics about ConvNet1, ConvNet2\n-\tHow did you set $\\epsilon$ ?\n-\tWould changing the sample size for each method improve the performance for respective methods?\n-\tIt would have been interesting to include a Future Works section\n-\tA more thorough discussion why the models works so well.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1x_HPraFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: The paper proposes a new framework (TREMBA) for black-box adversarial attack. The method utilizes a pretrained source network to learn a low dimensional embedding, it then searches efficiently within the embedding space (using NES) and produces an adversarial perturbation that can attack an unknown target network. A generator model first encodes an input to a latent vector and then decodes it to give an adversarial perturbation as an output. This generator is trained so that it can fool the source network and is then used to find the adversarial pattern when searching in the latent space. TREMBA produces perturbations with high level semantic patterns, and is easily transferable to different target architectures. The paper demonstrates its performance in terms of number of queries vs success rate on different datasets, Google cloud vision API and adversarially defended networks.\n\n- I like the exhaustive evaluation and comparative study done in the paper. It was especially interesting to see how TREMBA outperforms other techniques when attacking SOTA defended networks (on CIFAR 10 and Imagenet dataset). \n- When the method seems intuitive, it shows a novel way to combine transfer-based and score-based attack methods. \n- The motivation behind using low dim embedding space to accelerate adversarial pattern searching is also well explained in the paper. \n- The contributions are well-stated in the paper and definitely show an improvement over the past methods in not only reducing the number of queries but also improving the success rate of the attack. \n\nComments:\nThe loss function L_{target}(xi, t) on Page 3 has yi instead of t in the equation.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new black-box adversarial attack approach which learns a low-dimensional embedding using a pretrained model and then performs efficient search in the embedding space to attack target networks. The proposed approach can produce perturbation with semantic patterns that are easily transferable and improve the query efficiency in black-box attacks. All reviewers are in support of the paper after author response. I am very happy to recommend accept. 
Decision: Accept (Poster)