Blackbox Attacks via Surrogate Ensemble Search
Authors: Zikui Cai
Chengyu Song
Srikanth Krishnamurthy
Amit Roy-Chowdhury
M. Salman Asif
Venue: NeurIPS 2022 Accept (2022)
Keywords: surrogate ensemble
bilevel optimization
limited query attacks
surrogate ensemble
hard-label attacks

Abstract: Blackbox adversarial attacks can be categorized into  transfer- and query-based attacks. Transfer methods do not require any feedback from the victim model, but provide lower success rates compared to query-based methods. Query attacks often require a large number of queries for success. To achieve the best of both approaches, recent efforts have  tried to combine them, but still require hundreds of queries to achieve high success rates (especially for targeted attacks). In this paper, we propose a novel method for Blackbox Attacks via Surrogate Ensemble Search (BASES) that can generate highly successful blackbox attacks using an extremely small number of queries. We first define a perturbation machine that generates a perturbed image by minimizing a weighted loss function over a fixed set of surrogate models. To generate an attack for a given victim model, we search over the weights in the loss function using queries generated by the perturbation machine. Since the dimension of the search space is small (same as the number of surrogate models), the search requires a small number of queries. We demonstrate that our proposed method achieves better success rate with at least $30\times$ fewer queries compared to state-of-the-art methods on different image classifiers trained with  ImageNet (including VGG-19, DenseNet-121, and ResNext-50). In particular, our method requires as few as 3 queries per image (on average) to achieve more than a $90\%$ success rate for targeted attacks and 1--2 queries per image for over a $99\%$ success rate for untargeted attacks. Our method is also effective on Google Cloud Vision API and achieved a $91\%$ untargeted attack success rate with 2.9 queries per image. We also show that the perturbations generated by our proposed method are highly transferable and can be adopted for hard-label blackbox attacks. Furthermore, we argue that BASES can be used to create attacks for a variety of tasks and show its effectiveness for attacks on object detection models. Our code is available at https://github.com/CSIPlab/BASES.

Blackbox Attacks via Surrogate Ensemble Search
Zikui Cai, Chengyu Song, Srikanth Krishnamurthy, Amit Roy-Chowdhury, M. Salman Asif University of California Riverside
Abstract
Blackbox adversarial attacks can be categorized into transfer- and query-based attacks. Transfer methods do not require any feedback from the victim model, but provide lower success rates compared to query-based methods. Query attacks often require large number of queries for success. To achieve the best of both approaches, recent efforts have tried to combine them, but still require hundreds of queries achieve high success rates (especially for targeted attacks). In this paper, we propose a novel method for Blackbox Attacks via Surrogate Ensemble Search (BASES) that generate highly successful blackbox attacks using an extremely small number of queries. We first define a perturbation machine that generates a perturbed image by minimizing a weighted loss function over fixed set of surrogate models. To generate an attack for given victim model, we search over the weights the loss function using queries generated by the perturbation machine. Since the dimension of the search space is small (same as the number of surrogate models), the search requires a small number of queries. We demonstrate that our proposed method achieves better success rate with at least 30 fewer queries compared to state-of-thheart methods on different image classifiers trained with ImageNet (including VGG-19, DenseNet- 1221 and ResNext-50). In particular, our method requires as few as 3 queries per image (on average) to achieve more than 90% success rate for targeted attacks and 1-2 queries per image for over a 99% success rate for untargeted attacks. Our method is also effective on Google Cloud Vision API and achieved 91 01% untargeted attack success rate with 2.9 queries per image. We also show that the perturbations generated by our proposed method are highly transferable and can be adopted for hard-label blackbox attacks. Furthermore, we argue that BASES can be used to create attacks for a variety tasks and show its effectiveness for attacks object detection models. Our code is available at https: // thub. com/ CSIPlab/ BASES
Introduction
Deep neural network (DNN) models are known to vulnerable to adversarial attacks 4 Many methods have been proposed in recent years to generate adversarial attacks 22 5-11 (or to defend against such attacks 6 20) ). Attack methods for blackbox models can be divided into two broad categories: transfer- and query- based methods. Transfer-
methods generate attacks for some (whitebox) surrogate models via backpropagation and test if they fool the victim models [3] g. They usually agnostic to victim models they do not require readily any feedback; and they often provide lower success rates compared to query-based methods. On the other hand, query-based attacks achieve high success rate but the expense of querying the victim model several times to find perturbation directions that reduce the victim model loss (21 25 high success rate while keeping the number of queries small, is combine the transfer and query attacks. While there has been impressive recent work along this direction
One possible way to achieve a
26 10 27 the
*Corresponding authors: Zikui Cai (zcai0320ucr. edu) and M. Salman Asif (sasif@ucr.edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Perturbation Machine (FM)
Init
Generato
Loss landscape vs weights (N=3) models
Asa see-initinn
Quary
we ghte
Victim
N
up/ down?
Model
Coordinater wise update
Iteration
Fail
0.33
0:33
0,33
loss 1.5
W133
label Buttertly
0.3
:
loas 2.5
f2
(k)
Labil-Prinate
queries
Figure 1: BASES for score-based attack. (Top-left) We define perturbation machine (PM) using a fixed set of N surrogate models, each of which is assigned a weight value as w [wi...., ......,,vv. WN). The PM generates perturbed image x" (w) for given input image by minimizing the perturbation loss that is defined as function of w. To fool victim model, we update one coordinate in w at time while querying the victim model using x" (w) generated by the PM. We can view this approach as a bi-level optimization or search procedure; the PM generates a perturbed image with the given weights ==(w () in the inner level, while we update w in the outer level. (Bottom-left) We visualize weights and perturbed images for few iterations. We stop soon as the attack is successful (e.g. original label 'Butterfly' is changed target label 'Primate' for targeted attack). (Right) Victim loss values for different weights along Barycentric coordinates on the triangle. We start with equal weights (at the centroid) and traverse the space w to reduce loss (concentrate on model fa). Red color indicates large loss values (unsuccessful attack), and blue indicates low loss (successful attack).
still require hundreds of or more queries to be successful at targeted attacks. Such attacks are infeasible for limited-access settings where a user cannot query a model that many times [28
state-of-the-ard methods 24 |
Given this premise, we design a new method for blackbox attacks via surrogate ensemble search (BASES), combining transfer and query ideas, to fool a given victim model with higher success rates and fewer queries compared to state-of-the-ar methods. For example, our evaluation shows that BASES (on average) only requires 3 queries per image to achieve over 90% success rate for targeted attacks, which is at least 30x fewer queries compared to state-of-the-art methods [10 [ BASES consists of two key steps that can be viewed bilevel optimization steps. 1) A perturbation machine generates query for the victim model based on weights assigned the surrogate models. 2) The victim model's feedback is used to change weights of the perturbation machine to refine the query. Figure depicts these steps.
We first define a perturbation machine (PM) that generates a single perturbation to fool all the (whitebox) models the surrogate ensemble. We use a surrogate ensemble for two reasons: is known to provide better transfer attacks 29 7 The assumption is that if an adversarial image can fool multiple surrogate models, then it is very likely to fool victim model well. For the same reason, ensemble with different and diverse surrogate models provides better attack transfer. 2) Our main interest searching for perturbations that can fool the given victim model. single surrogate model provides a fixed perturbation; hence, it does not offer flexibility to search over perturbations. To facilitate search over perturbations, we define the adversarial loss for the PM as a function of weights assigned to each model in the ensemble. By changing the weights of the loss function, can generate different perturbations and steer in a direction that fools the victim model. It is worth noting that perturbations generated by a surrogate ensemble with an arbitrary set of weights often fools all the surrogate models, but they do not guarantee success on unseen victim models; therefore, searching over the weights space for surrogate models is necessary.
Since number of models the surrogate ensemble is small, search space is low dimensional and requires extremely small number of queries compared to other query-based approaches. In our method, we further simplify the search process by updating one weight element at a time, which is equivalent to coordinate descent, which has been shown to be effective in query based attacks

---

Since we search along one coordinate at time instead of estimating the full gradients, the method is extremely efficient in terms of query count. In particular, our method requires two queries per coordinate update but offers success rates as good as that given by performing a full gradient update step (as shown in Section D. Reducing the dimension of the search space while maintaining high success rate for query-based based attacks is an active area of research 2
21 23
10
27
and our proposed method pushes the boundary in this area.
We perform extensive experiments for (score-based) blackbox attacks using a variety of surrogate and blackbox victim models for both targeted and untargeted attacks. We select Py PyTorch Torchvision 80 as our model zoo, which contains 56 image classification models trained on ImageNet [31] that span a wide range of architectures. We demonstrate superior performance by large margin over state-of-the-arr approaches, especially for targeted attacks. Furthermore, we tested the perturbations generated by our method for attacks on hard-label classifiers. Our results show that the perturbations generated by our method are highly transferable. We present also present experiments for attacks on object detectors in the supplementary material, which demonstrate the effectiveness of our attack method for tasks beyond image classification.
The main contributions of this paper are as follows.
We propose a novel, yet simple method, BASES, for effective and query-efficient blackbox attacks. The method adjusts weights of the surrogate ensemble by querying the vietim model and achieves high fooling rate targeted attack with a very small number of queries.
We perform extensive experiments to demonstrate that BASES outperforms state-of-the-arr methods [26 I0 27 11 by large margin; over 90% targeted success rate with less than 3 queries, which at least 30x fewer queries than other methods.
We also demonstrate the effectiveness under a real-world blackbox setting by attacking Google Cloud Vision API and achieve 91 1 untargeted fooling rate with 2.9 queries (3x less than ITO..
The perturbations from BASES are highly transferable and can also be used for hard-label attacks. In this challenging setting, can achieve over 90% fooling rate for targeted and almost perfect fooling rate for untargeted attacks on a variety of models using less than and queries, respectively.
We show that BASES can be used for different tasks by creating attacks for object detectors that significantly improve the fooling rate over transfer attacks.
2 Related work
Ensemble-based transfer attacks. Transferable adversarial examples that can fool one model can also fool different model B
Transfer-based untargeted attacks are considered "easy" since the adversarial examples can disrupt feature extractors into unrelated directions (e.g.. in MIM I7). the fooling rate for some models can be as high as 87.9%). In contrast, transferbasedd targeted attacks often suffer from low fooling rates (e.g.. MIM shows transfer rate of about 20% at best). To improve the transfer rate, several methods use ensemble based approach. To combine the information from different surrogate models; [29] fuses probability scores and [17] proposes combining logits. While these methods have been effective, the most natural and generic approach to combine losses, which can be used for tasks beyond classification 35 MGAA [36] iteratively selects set of
4! 24
surrogate models from an ensemble and performs meta train and meta test steps to reduce the gap between whitebox and blackbox gradient directions. Simulator-Attack method [ uses several surrogate models to train a generalized substitute model as "Simulator" for the victim model; however, training such simulators is computationally expensive and difficult to scale to large datasets. Previous ensemble approaches typically assign equal weights for each surrogate model. In contrast, we update weights for different surrogate models based on the victim model feedback.
Query-based attacks. Unlike transfer-based attacks, query-based attacks do not make assumptions that surrogate models share similarity with the victim model. They can usually achieve high fooling rates even for targeted attacks (but at the expense of queries) [21 22 The query complexity is proportional to the dimension of the search space. Queries over the entire image space can be
extremely expensive [2I]. requiring millions of queries for targeted attack [22]. To reduce the query complexity, number of approaches have attempted to reduce the search space dimension or leverage
transferable priors or surrogate models to generate queries 23
39 SimBA-DCT [3 searches over the low DCT frequencies. P-RGF 26 utilizes surrogate gradients transfer-based
3

---

prior, and draws random vectors from www-dimesioonal subspace for gradient estimation. TREMBA [C trains perturbation generator and traverses over the low-dimensional latent space. ODS 2 optimizes in the logit space to diversify perturbations for the output space. GFCS [II] searches along the direction of surrogate gradients, and falls back ODS if surrogate gradients fail. Some
also reuse the query feedback to update surrogate models or "blackbox simulator', , but such a fine-tuning process provides very slight improvements. We summarize the typical search space and average number of queries for some sate-of-the-ar methods in Table I In our approach, we further shrink the search dimension to as low as the number models in the ensemble. Since search space is dense with adversarial perturbations, we show that moderate- size ensemble with 20 models can generate successful targeted attacks for a variety of victim models while requiring only 3 queries (on average), which is at least 30 time fewer than that of existing methods.
other methods [38 21
37
Method
3.1
Preliminaries
We use additive perturbation 11
to generate perturbed image as x* = = x ô, where ô denotes the perturbation vector of same size as input image To ensure that the perturbation is imperceptible to humans, we usually constrain its lp norm be less than a threshold, i..., || S E, where p is usually chosen from {2, )c}. Such adversarial attacks for a victim model f can be generated by minimizing the so-called adversarial loss function over such that the output [(r + d) as close to the desired (adversarial) output as possible. Specifically, the attack generator function maps the input image x to an adversarial image x² such that the output f' (x=' ) is either far/different from the original output y for untargeted attacks, closefidentical to the desired output y" for targeted attacks.
Let us consider multi-class classifier f() x + z, where: vector at the last layer. The logit vector can be converted to a probability vector p softmax(2). We refer such a classifier as a sccorebbased" or "soft-label" classifier. In contrast, "hard- label" classifier provides a single label index out of total of C' classes. We can derive the hard label from the soft labels as y = arg max, f())- For untargeted attacks, the objective is to find xr" such that arg max, f(x"). * y. For targeted attacks, the objective is to find x² such that arg max; [x*) = y", where y" is the target label.
[2], , =c] represents logit
Many efforts on adversarial attacks use iterative variants of the fast signed gradient method (FGSM) (2) because of their simplicity and effectiveness. Notable examples include I-FGSM (5), PGD 6 and MIM 77. We use PGD attack in our PM, which iteratively optimizes perturbations as
gr+1= I1. (6' sign(V;C(x
v)")))
where L. is the loss function and IT denotes a projection operator. There are many loss functions suitable for crafting adversarial attacks. We mainly employ the following margin loss, which has
been shown to be effective in C&W attacks 41:
C(f(x), = max max f(x), - -xx))x
(2)
37y'
where is the margin parameter that adjusts the extent to which the example is 'adversarial.' A larger x corresponds to a lower optimization loss. One advantage of C&W loss function is that its sign directly indicates whether the attack is successful or not (+ue value indicates failure, -ve value indicates success). Cross-entropy loss is also a popular loss function to consider, which has similar performance as margin loss (comparison results are provided in the supplementary material).
3.2 Perturbation machine with surrogate ensemble
Controlled query generation with PM. We define perturbation machine (PM) to generate queries for the victim model as shown in Figure U The PM accepts an image and generates a perturbation to fool all the surrogate models. Furthermore, we seek some control over the perturbations generated by the PM to steer them in a direction that fools the victim model. To achieve these goals, we construct the PM such that it minimizes a weighted adversarial loss function over the surrogate ensemble.
Adversarial loss functions for ensembles. Suppose our PM consists of N surrogate models given as F (f..... fN ) each of which is assigned weight in w W wN] such that Σ wi 1.

---

For any given image and w, we seek to find a perturbed image ="(w) that fools the surrogate ensemble. Below we discuss three possible weighted ensemble loss functions- optimization problems for targeted attack. Loss functions for untargeted attack can be derived similarly.
weighted probabilities
x² x=(w) arg min log (1g-
by softmax(f(z))),
(3)
weighted logits
x² x²(w) = arg min C
w. wxf((z))8"),
(4)
weighted loss
x² z'(w) arg min
Σ wi w;L((f(x),))
(5)
at
y" denotes the target label and 1y denotes its one-hot encoding. L represents some adversarial loss function (e.g., C&W loss). The first problem in 3 is the minimization of the softmax cross-entropy loss defined on the weighted combination of probability vectors from all the models in the ensemble [29 The second problem in f optimizes adversarial loss over a weighted combination of logits from the models 77. The third problem in ( optimizes weighted combination of adversarial losses over all models. The weighted loss formulation is the simplest and most generic ensemble approach that works not only for the classification task with logit or probability vectors, but also other tasks (e.g., object detection, segmentation) as long as the model losses can be aggregated [4 Here, we focus the weighted loss formulation, since it shows superior performance compared weighted probabilities and logits formulations in our experiments (additional experiments are presented the supplementary material).
Algorithm D presents a pseudocode for the PM module for a fixed set weights. The PM accepts an image and weights along with the surrogate ensemble and returns the perturbed image x* = x x after a fixed number of signed gradient descent steps (denoted as T) for the ensemble loss.
Algorithm 1 Perturbation Machine: (w) PM 6ma)
Input:
Input x and the target class y* (for untargeted attack y* # y );
Surrogate ensemble J {ʃ1, f2, JN];
Ensemble weights w = [wi, W2, .... WN
Initial perturbation iiii Step size A; Perturbation norm (l2/Ex)) and bound E;
Number of signed gradient steps T
Output: Adversarial perturbation 6, (w)
1: 8
2: for to T do
3: Calculate Lens Em w. (x
Ensemble loss
4: Update 8
sign VgLen)
D Gradient of ensemble via backpropagation > Project to the feasible set of los &g ball
5: Project 8 +- (d)
6: end for
7: x² (w) <- +
8: return x (w)
3.3 Surrogate ensemble search as bilevel optimization
Let us assume that we given blackbox victim model, fv. that seek to fool using perturbed image generated by the PM (as illustrated in Figure DT. Suppose the adversarial loss for the victim model is defined Ly. To generate perturbed image that fools the victim model, we want to solve the following optimization problem:
W = arg min (y(fv(x (w)), y").
(6)
The problem (6) is bilevel optimization that seeks to update the weight vector w the PM so that the generated x (ww) fools the victim model. The PM in Algorithm D can be viewed as a function that solves the inner optimization problem in our bilevel optimization. The outer optimization problem searches over w to steer the PM towards a perturbation that fools the victim model.
5

---

Blackbox Attacks via Surrogate Ensemble Search. Our objective is to maximize the attack success rate and minimize the number of queries on the victim model; hence, we adopt simple yet effective iterative procedure to update the weights w and generate sequence of queries. Pseudocode for our approach is shown in Algorithm 2 We initialize all entries w 1/N and generate the initial perturbed image x² (w) for input z. We stop if the attack succeeds for the victim model; otherwise, we update w and generate a new set perturbed images. We follow [2 and update in coordinate-wise manner, where at every outer iteration, we select 1ith index and generate two instances of w as w w by updating wn as Wn +7,, Wi 7, where 7) is step size. We normalize the weight vectors that the entries non-negative and add up to 1. We generate perturbations x" (w") ", ( using the PM and query the victim model. We compute the victim loss (or score) for {wrt, w"] and select the weights, the perturbation vector, and the perturbed images corresponding to the smallest victim loss. We stop if the attack is successful with any query.
BASES:
Algorithm 2 BASES: Blackbox Attack via Surrogate Ensemble Search
Input:
Input and the target class y* (for untargeted attack y* * y ):
Victim model fv: Maximum number of queries Q; Learning rate ny Perturbation machine (PM) with surrogate ensemble
Output: Adversarial perturbation 8,
1: Initialize 5 = 0: g = 0; w {1/N, , 1/N, ..... 1/N)
2: Generate perturbation via PM: 6, 5, (w) PM w,
bi first query with equal weights
3: Query victim model:
(r + 6)
4: Update query count: q - +
5: if arg max, y then
6: break
stop if attack successful
7: end if
8: while q Q do
9:
Update surrogate ensemble weights as follows.
outer level updates weights b cyclic or random order
10:
Pick surrogate index n
11:
Compute w w by updating wn, as wh + 17, Wn. 7,, respectively
12:
Generate perturbation x² (w+) ), (w" (w via PM
b inner level generates query 2 queries per coordinate
Query victim model: fo )), (x (w=
14:
Calculate victim model loss for {w+,, w as (w+),, Lw(ww)
Select w, 6, xx (w) for the weight vector with the smallest loss
16:
Increment q after every query, and stop if the attack is successful for any query
end while
18:
return 6
4 Experiments
4.1 Experiment setup
In this section, we present experiments on attacking the image classification task. Additional experiments on attacking object detection task can be found the supplementary material.
Surrogate and victim models. We present experiments for blackbox attacks mainly using pretrained image classification models from Pytorch Torchvision 30), which is compre- hensive and actively updated package for computer vision tasks. At the time of writing this paper, Torchvision offers 56 classification models trained on ImageNet dataset [3]] These models have different architectures and include the family of VGG 42 ResNet 43
SqueezeNet 4.. [555 ResNeXt 4., MobileNet (47 48] EfficientNet 44) Reg-
Net (50). Vision Transformer [51]. and ConvNeXt [52]. We choose different models the victim blackbox models for our experiments, as shown in Figures B and 4 To construct
an effective surrogate ensemble for the PM, we sample 20 models from different families: (VGG-16-BN, ResNet-18, SqueezeNet-1.1 GoogleNet, MNASNet-1.0 DenseNet-161. EfficientNet-Bii BO, RegNee-y-400, ResNeXt-101. Convnext-Shin -mall, ResNet-50, VGG-13, DenseNet-201, Inception-v3, ShuffleNet- MobileNett-v3-mmall Rede-ResNet-50, EfficientNet-B4, RegNet-x-4000 VIT-B-16). We vary ensemble

---

size N E {4.10, 10, 20) by picking the first N model from the set. In most of the experiments. our method uses N 20 models in the PM, unless otherwise specified. We also tested a different set of models pretrained on TinyImageNet dataset, the details of which are included in the supplementary material. To validate the effectiveness of our methods a practical blackbox setting, we also tested Google Cloud Vision API.
Comparison with other methods. We compare our method with some of the state-of-the-ar methods for score-based blackbox attacks. TREMBA I00 is powerful attack method that searches for perturbations by changing the latent code of generator trained using set of surrogate models. GFCS [TI is a recently proposed surrgate- -aseee attack method that probes the victim model using the surrogate gradient directions. We use their original code repositories [533 [4] For completeness, we also compare with two earlier methods, ODS [27 and P-RGF [26], that leverage transferable priors, even though they have been shown to be less effective than GFCS and TREMBA. Additional details about comparison with TREMBA and GFCS are provided in the supplementary material.
Dataset. We mainly use 1000 ImageNet-like images from the NeurIPS-17 challenge (55 56) which provides the ground truth label and a target label for each image. We also provide evaluation results on TinylmageNet in the supplementary material.
Query budget. In this paper, we move towards a limited-access setting, since for many real-life applications, legitimate users will not be able to run many queries (28]. In contrast with TREMBA and GFCS, which set the maximum query count to 10, 000 and 50, 000, respectively. we set the maximum count to be 500 and only run our method for 50 queries in the worst case. (TREMBA also uses only 500 queries for Google Cloud Vision API to cut down the cost.)
Perturbation budget. We evaluated our method under both foc and Eg norm bound, with commonly used perturbation budgets of lx < 16 and l2 < 2..00. 3128 on a 0-255 pixel intensity scale, where D denotes the number of pixels in the image. For attacking Google Cloud Vision API, we reduce the norm bound to &x < 12 to align with the setting in TREMBA. Results for Eg norm bound are provided in the supplementary material.
Targeted vs untargeted attacks. All the methods achieve near perfect fooling rates for untargeted attacks in our experiments. This is because untargeted attack on image classifiers not challenging [II, especially when the number of classes is large. Thus, we primarily report experimental results on targeted attacks in the main text and report results for untargeted attacks in the supplementary material. We use the target labels provided in the dataset (5S) in the experiments discussed in the main text. We provide additional analysis on using different target label selection methods such as "easiest" and 'hardest' according to the confidence scores in the supplementary material.
4.2 Score-based attacks
Targeted attacks. Figure 2 presents a performance comparison of five methods for targeted attacks on three blackbox victim models. Our proposed method provides the highest fooling rate with the least number of queries. P-RGF is found be ineffective (almost 0% success) for targeted attacks under low query budgets. TREMBA and GFCS are similar in performance; TREMBA shows better performance when query count is small, but GFCS matches TREMBA after nearly 100 queries. Nevertheless, our method clearly outperforms these two powerful methods by a large margin at any level of query count. We summarize the search space dimension D and query counts vs fooling rate of different methods under a limited (and realistic) query budget for both the targeted and untargeted attacks in Table Our method is the most effective in terms of fooling rate vs number of queries (and has the smallest search dimension). Additional results and details about fair comparison and fine taning of TREMBA and GFCS are provided in the supplementary material.
Surrogate ensemble size (N). To evaluate the effect of surrogate ensemble size on the performance of our method, we performed targeted blackbox attacks experiment on three different victim models using three different sizes of the surrogate ensemble: N E {4, 10, 20}. The results are presented in Figure in terms of fooling success rate vs number of queries. As we increase the ensemble size, the fooling rate also increases. With N = 20, the targeted attack fooling rate is almost perfect within 50 queries. Specifically, for VGG-19 with N = 20, we improve from 54% success rate at the first query (with equal ensemble weights) to 96% success rate at the end of 50 queries; this equates to 78% improvement. DenseNet-121 and ResNext-50 can achieve 100% fooling rate with N = 20. With DenseNeet-125 -121, using 10 surrogate models, we can achieve fooling rate of 98%. While
7

---

100
100
60-
Fooling Reeter
60-
Fooing
12a
10'
16
10
16:
10
105
101
307
Queries
Quertes
Queries
(a) VGG-19
(b) DenseNet-121
(c) ResNext-50
Figure Comparison of attack methods on three victim models under perturbation budget loc S for targeted attack. Our methad achieves high success rate (over 90%) with few queries (average of 3 per image).
Table 1: Number of queries fooling of different methods and the search space dimension D.
Number of queries (mean + std) per image and fooling rate
Method
D
VGG-19
DenseNet-121
ResNext-50
Targeted
Untargeted
Targeted
Untargeted
Targeted
Untargeted
156 + 113
164 + 112
166 E 116
P-RGF
7,500
93.5%
92.9%
92.5%
+ 107
+
+ 104
+
109
+ 98.9%
TREMBA
1,568
89.2%
99.7%
90.5%
99.5%
85.1%
261 125
+ 48 99.9%
266 123
+ 64
270 116
+ 65 98.4%
ODS 28
1,000
49.0%
49.7%
99.0%
42.7%
101 + 95
14 + 21 100.0%
76 + 75
16 36
86 86 87
15 18 99.7%
GFCS
1,000
89.1%
95.2%
99.9%
92.9%
3.0 5.4
5.4
1,2 2.4
1.8 + 2.7
1,2 1.8 99.9%
1.8 2.6
1.2 + 0.9
Ours
20
95.9%
99.8%
99.4%
99.7%
100.0%
using 4 models is challenging with respect to all victim models, improvement fooling rates when the number of queries increases.
rapid and significant
Comparison of whitebox (gradient) vs blackbox (queries). To check the effectiveness of our query-based coordinate descent approach for updating w, we compare its performance with the alternative approach of calculating the exact gradient of victim loss under the whitebox setting. The results are presented in Figure B as dotted lines. We observe that our blackbox query approach provides similar results as the whitebox version, which implies the coordinate-wis update of w is as good as a complete gradient update.
Fooiing Beece
Fooling Rabe
00-
80
854
40-
40
08:10
NN-4
20-
20
08.20
WB 70
WB-20
167
101
Querles
Querits
Querles
(a) VGG-19
(b) DenseNet-121
(c) ResNext-50
Figure 3: Comparison of targeted attack fooling rate with different number of ensemble models N {4, 10, 20} PM. Every experiment performed with whitebox gradient (denoted 'WB' with dotted lines) and blackbox score-based coordinate descent (denoted as 'BB' with solid lines). Experiment was run 100 images.
4.3 Hard-label attacks
The queries generated by our PM are highly transferable and can be used craft successful attacks for hard-label classifiers. To generate sequence of queries for hard-label classifiers, pick

---

100
Toolng (16)
Foo ing Pate 1N0
60
60
Reshet
EfficienNiet
Reglietw
20
20
V1--16
ConvNext Baso
10a
10'
Querles
(a) Targeted
(b) Untargeted
Figure 4: Performance of blackbox attack on 6 hard-label classifiers. Our method generates sequence of queries for targeted attack using VGG-19 vietim model while the PM has N 20 models in the surrogate ensemble. Experiment performed 100 images.
"surrogate victim' model and generate queries by updating w the same manner as the score-based attacks for Q iterations (without termination). We store the queries generated at every iterations in a query set {6) (6',....88 .2]. We test the victim hard-label blackbox model using x + by selecting from the set in sequential order until either the attack succeeds the queries finish.
In our experiments, we observed that this approach can achieve a high targeted attack fooling rate on variety of models. We present the results of our experiment in Figure where we report attack success rate vs query count for models: [MobileNet -V2, ResNet-34, ConvNeXt-Base EfficientNet-B22 RegNet-x-8, VIT-L- We used VGG-19 as the 'surrogate victim' model generate the queries using the PM with 20 surrogate models. Using the saved surrogate perturba- tions, we can fool all models almost 100% except for VIT-I-16 [55]] that is vision transformer and architecturally very different from the majority of surrogate ensemble models (thus difficult to attack). Nevertheless, the fooling rate increases from 18% - 63%, which is 250% improvement.
4.4 Attack on commercial Google Cloud Vision API
We demonstrate the effectiveness of our approach under practical blackbox setting by attacking the Google Cloud Vision (GCV) label detection API. GCV detects and extracts information about entities in image, across very broad group of categories containing general objects, locations, activities, animal species, and products. Thus, the label set very different from that of ImageNet, and largely unknown to us. We have no knowledge about the detection models in this API either. We randomly select 100 images from the aforementioned ImageNet dataset that are correctly classified by GCV, and perform untargeted attacks against GCV using 20 surrogate models with perturbation budget of fo < 12 to align with the setting in TREMBA 10
For each input image, GCV returns list of labels, which are usually the top 10 labels ranked by probability. Under the success metric of changing the top label to any other label, same as [10], our attack achieve fooling rate of 91% with only 2.9 queries per image on average, which is much lower than 8 queries TREMBA reported for similar experiment. We present some successful examples in Figure 5 We present additional results in the supplementary material that show attacks from classification can transfer to object detection models.
5 Conclusion and discussion
We propose novel and simple approach, BASES, to effectively perform blackbox attacks in query-efficient manner, by searching over the weight space of ensemble models. Our extensive experiments demonstrate that wide range of models are vulnerable to attacks at the fooling rate of over 90% with as few as 3 queries for targeted attacks. The attacks generated by our method are highly transferable and can also be used to attack hard-label classifiers. Attacks Google Cloud Vision API further demonstrates that our attacks are generalizable beyond the surrogate and victim models in our experiments.
Limitations. 1) Our method needs a diverse ensemble for attacks be successful. Even though the search space is dow-dimensional, the generated queries should span large space that they can

---

Antomettun Banking ighn
Vertabrate
Uhe ANoidie 2w
(a) Original Image- Bus
(b) Attacked Image
spod
Pollinster
Arthooped
Adoptation
79%
Vino
Roottle
705
(c)
Original Image Fly
(d) Attacked Image
Figure 5: Visualization of some successful attacks Google Cloud Vision.
fool any given victim model. This is not a major limitation for image classification task as a large number of models are available, but it can be a limitation for other tasks. 2) Our method relies on the PM to generate perturbation query for every given set of weights. The perturbation generation over surrogate ensemble is computationally expensive, especially as the ensemble size becomes large. In our experiments, one query generation with {4, 10. 20} surrogate models requires nearly (2.4s, 9.6s, 18s} per image on Nvidia GeForce RTX 2080 TI. Since our method requires small number of queries, the overall computation time of our method remains small.
Societal impacts. We propose an effective and query efficient approach for blackbox attacks. Such adversarial attacks can potentially be used for malicious purposes. Our work can help further explain the vulnerabilities of DNN models and reduce technological surprise. We also hope this work will motivate the community to develop more robust and reliable models, since DNNs are widely used in real-life or even safety-critical applications.
Acknowledgments. This material based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under agreement number HR001 12090096. Approved for public release; distribution is unlimited.
References
[1] Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Frhan, Tan Goodfellow; and Rob Fergus. Intriguing propertics of neural networks. International Conference on Learning Representations, 2014.
[2] lan Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples.
In International Conference on Learning Representations, 2015.
[3] Nicolas Papernot. Patrick McDaniel, and lan Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277 072.6600007777.... 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow. Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
[5] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples the physical world. arXiv preprint arXiv: 160.00233, 2016.
[6] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
[7] Yinpeng Dong. Fangzhou Liao. Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting Adversarial Attacks with Momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185-998 9193, 2018.
10

---

[8] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Puttern Recognition, pages 2730-2739, 2019.
[9] Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv: 1oXxv:9908.0622 1908.0628,, 2019.
[10] Zhichao Huang and Tong Zhang. Black-hox adversarial attack with transferable model-based embedding. In International Conference Learning Representations, 2019.
[11] Nicholas A. Lord, Romain Mueller, and Luca Bertinetto. Attacking deep networks with surrogate- based adversarial black- box methods is easy, In International Conference on Learning Representations, 2022.
[12] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick McDaniel. Ensemble ad- versarial training: Attacks and defenses. In International Conference on Learning Representations, 2018.
[13] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In Annual Network and Distributed System Security Symposium, 2017.
[14] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. International Conference on Learning Representations, 2017.
[15] Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 135-147, 2017.
[16] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369-385, 2018.
[17] Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations, 2018.
[18] Cihang Xie. Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 501-509, 2019.
[19] Kui Ren, Tianhang Zheng, Zhan Qin, and Xue Liu. Adversarial attacks and defenses in deep learning. Engineering, 6(3):346-3) 360, 2020.
[20] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In International Joint Conference on Artificial Intelligence. 2021.
[21] Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 15-26, 2017.
[22] Chun-Chen Tu, Paishun Ting, Pin- Yu Chen, Sijia Liu, Huan Zhang. Jinfeng Yi, Cho-Jui Hsich, and Shin- Ming Cheng. Autozoom: Autoencoder-basse zeroth order optimization method for attacking black-box neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 742-749, 2019.
[23] Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black- box adversarial attacks. International Conference on Machine Learning, pages 2484-2493. PMLR, 2019.
[24] Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. Qeha: Query-efficient houndary-basee blackhox attack. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1221-1230 2020.
[25] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box box adversarial attacks with limited queries and information. International Conference on Machine Learning, 2018.
[26] Shuyu Cheng, Yinpeng Dong. Tianyu Pang, Hang Su, and Jun Zhu. Improving black-box adversarial attacks with transfer-based based prior. In Advances in Neural Information Processing Systems, volume 32, 2019.

---

[27] Yusuke Tashiro, Yang Song, and Stefano Ermon. Diversity can be transferred: Output diversification for white-and black-box attacks. In Advances in Neural Information Processing Systems, volume 33, pages 4536-4548, 2020.
[28] Ian Goodfellow. research agenda: Dynamic models to defend against correlated attacks. arXiv preprint (0Xiv:9903.062900 2019.
[29] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations. 2017.
[30] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic di differentiation in pytorch. 2017.
[31] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. IEEE, 2009.
[32] Maosen Li, Cheng Deng, Tengjiao Li, Junchi Yan, Xinbo Gao, and Heng Huang. Towards transferable targeted attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 641-649, 2020.
[33] Zhaohui Che, Ali Borji. Guangtao Zhai, Suiyi Ling, Jing Li, and Patrick Callet. new ensemble adversarial attack powered by long-ton long-term gradient memories. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3405-3413, 2020.
[34] Zuxuan Wu, Ser-Nam Lim, Larry Davis, and Tom Goldstein. Making an invisibility cloak: Real world adversarial attacks on object detectors. In European Conference Computer Vision, pages 1-17. Springer, 2020.
[35] Zikui Cai, Xinxin Xie, Shasha Li, Mingjun Yin, Chengyu Song, Srikanth V Krishnamurthy, Amit K Ray-Chowdhury, and M. Salman Asif. Context-aware transfer attacks for object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[36] Zheng Yuan, Jie Zhang, Yunpei Jia, Chuanqi Tan, Tao Xue, and Shiguang Shan. Meta gradient adversarial attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7748-7757, 2021.
[37] Chen Ma, Li Chen, and Jun-Hai Yong. Simulating unknown target models for query-efficient black-box attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1135--1844, 2021.
[38] Fnu Suya, Jianfeng Chi. David Evans, and Yuan Tian. Hybrid batch attacks: Finding black- box adversarial examples with limited queries. USENIX Security Symposium, 2019.
[39] Shasha Li, Abhishek Aich, Shitong Zhu, M. Salman Asif, Chengyu Song, Amit Roy- Shy-Chhodhhury. and Srikanth Krishnamurthy. Adversarial attacks on black box video classifiers: Leveraging the power of geometric transformations. Advances in Neural Information Processing Systems, 34:2085-2096, 2021.
[40] Jiancheng Yang, Yangzhou Jiang, Xiaoyang Huang. Binghing Ni, and Chenglong 7hao. Learning black-box attackers with transferahle priors and query feedback. Advances in Neural Information Processing Systems, 33:12288-12299 12299, 2020.
[41] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pages 39 57. IEEE, 2017.
[42] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: /409.1556, 2014.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778-778, 2016.
[44] Forrest N landola, Song Han, Matthew W Moskewiez, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnct-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint 16Xiv:1602.07360. 2016.
[45] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4708, 2017.
12

---

[46] Saining Xie. Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[47] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 4520, 2018.
[48] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13244,132,,24 2019.
[49] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. pages 2105-6114 PMLR, 2019.
[50] Iliaa Radosavovic, Raj Prateek Kosaraju. Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428- 10436, 2020.
[51] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[52] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2022.
[53] Nicholas Lord, Romain Mueller, and Luca Bertinetto. Gfes. https://githt:l http://thhaabccm/ivieea/a//a 2022. [Creative Commons
4.0 Intemaatiioon license).
[54] Zhichao Huang and Tong Zhang. Tremba. http:://gggtuuuboooo//rraaaaaaat 2019. [No
license provided).
[55] Google Brain. Neurips 2017: Targeted adversarial attack. https://wwws hompeppl...comccoopttttioosssss nips- 2017-tareetedd
2017. [On Kaggle].
[56] Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS'17 17 Competition: Building Intelligent Systems, pages 195- 231..1..552 Springer, 2018
[57] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pages 484-501. Springer, 2020.
[58] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)211-252)) 2015.
[59] Jiawei Du, Hu Zhang, Joey Tianyi Zhou, Yang, and Jiashi Feng. Query-efficient meta attack to deep neural networks. In International Conference on Learning Representations, 2020. URL. https
//openreview.c
[60] Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. arXiv preprint IrXiv:1807 07998,, 2018
[61] Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie 7hao, Buyn Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 110..9900771117 2019.
[62] Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong. Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu. Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection. https: //github. 2.0].
http://ggttubboooeooeee www/opwn.cccco/ooo. mml mmmlab/mmettection 2019. [Apache License
13

---

[63] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-can: Towards real-time object detection with region proposal networks. In Advances neural information processing systems, pages 91-99, 2015.
[64] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv 1804.02767, 2018.
[65] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings the IEEE international conference computer vision, pages 2980-2988, 2017.
[66] Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye. FreeAnchor: Learning match anchors for visual object detection. Neural Information Processing Systems, 2019.
[67] Ze Yang, Shaohui Liu, Han Hu, Liwei Wang, and Stephen Lin. Reppoints: Point set representation for object detection. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 9657-9666 2019.
[68] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. arXiv:194.008550 2019.
Objects as points. In arXiv preprint
[69] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[70] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end object detection. In International Conference on Learning Representations, http://wwww..eeeeoooo///////////
2021. URL
[71] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference computer vision, pages 740- 755. Springer, 2014.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No] [N/A] You strongly encouraged include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datascts? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read ethics review guidelines ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state full set of assumptions of theoretical results? [N/A]
(b) Did you include complete proofs of all thcoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce main experimen- tal results (either the supplemental material or as a URLy? [Yes] Code is available at https://githbb.co https:/ggihha..com/S/Piaaaa wwwpCS///aa////ASS...
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)?
[Ye]]. We provide these details in the experiment section and supplementary material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experiments multiple times)? [No] Our method provides (almost) identical results for each run.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]. We provide these details in the experiment section and supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're us-
ing/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "pVQ0YtFbeO", "reviewer": "Reviewer_vphn", "summary": "This paper studies query efficient black-box attacks, which aims to generate black-box adversarial examples using least number of queries to the unknown victim model. The authors proposed to leverage ensemble of surrogate models to generate some candidate perturbations and then query the unknown model. The query feedback is then used to update the weights of individual surrogate models to produce better candidate perturbations for the next query. The empirical results demonstrate that the proposed approach can obtain reasonably high success with handful of queries. ", "questions": "Most of the suggestions for improving the paper are listed in the \"strengths and weaknesses\" section.", "limitations": "Yes. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The presentation of the paper is very clear and easy to follow. However, the paper lacks comprehensive review of the related works and also the adopted baselines are not convincing enough. Further, the experiments should be designed to illustrate the importance of each module in the attack process. More specifically,\n1) many closely related works (especially attacks that combine transfer and query based attacks) are missing from the paper. A few samples are listed below [1], [2], [3], [4] and the authors are encouraged to conduct a more comprehensive survey on related works and include more recent baselines. For example, the listed baselines [1], [2] can be directly adopted as baselines for this paper and the combination method proposed by Suya et al. [3] might be coupled with the strong (should still be the state-of-the-art) query based attack [4] to form another strong baseline. The key is also to ensure that all baselines are using the same model ensembles. Since the nature of the proposed attack in this paper is \"adaptive\" transfer attacks using query feedback, it is really interesting to see if it is always advantageous over attacks that combine transfer and query based attacks directly. \n[1] Yang et al., \"Learning Black-Box Attackers with Transferable Priors and Query Feedback\", NeurIPS 2021.\n[2] Ma et a.., \"Simulating Unknown Target Models for Query-Efficient Black-box Attacks\", CVPR 2021\n[3] Suya et al., \"Hybrid Batch Attacks: Finding Black-box Adversarial Examples with Limited Queries\", USENIX Security 2020.\n[4] Andriushchenko et al., \"Square Attack: a query-efficient black-box adversarial attack via random search\", ECCV 2020.\n2) It is quite surprising to me that, targeted attacks even using the baselines only cost less than 100 queries in most cases. It may be because the selected target class for misclassification is still not hard enough. A more comprehensive way of evaluation might be to include three types of target classes: 1) easiest class (second highest class in original confidence scores), 2) random class and 3) hardest class (lowest class in the original confidence scores). \n3) at least the impact of ensemble diversity should be carefully measured. The current attack setup assumes the attacker has access to models that are trained on the exact same training data as the black-box model and also the ensemble set contains diverse enough architectures, which may not always hold in practice. \n4) In the attack results, maybe it is also better to report the direct transfer rates when different ensembles have equal weights. With this, we can see how much improvement is obtained by updating the surrogate model weights. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NnRZIov0Z3Q", "reviewer": "Reviewer_cKYU", "summary": "This paper proposes a hybrid black-box attack on deep learning models that uses a combination of transfer and query-based attacks to reduce the number of queries while still maintaining a high targeted attack success rate. The main method involves creating a weighted ensemble of surrogate models, which is then used to generate local adversarial examples. These are tested against the victim model and the ensemble weights are updated using a form of coordinate descent based on the success of the locally generated adversarial examples. Experiments are carried out on non-robust models trained on the Imagenet dataset using standard Lp threat models. Both targeted and untargeted attacks are used. Real-world experiments on the Google Cloud Vision API also demonstrate the effectiveness of the method.", "questions": "- Line 262-263: Does the whitebox version involve taking gradients with respect to the weight vector $\\mathbf{w}$? How is this gradient computed?\n- Line 267-268: For the hard-label attacks using a surrogate victim model, is score based access to the surrogate model assumed? What is the link between the architecture of the surrogate and true victim models?\n- Comparison with existing hard label attacks: Is there a reason there is no direct comparison to existing hard-label attacks such as the Hop-Skip-Jump attack (https://arxiv.org/abs/1904.02144)?\n", "limitations": "- While I appreciate that the authors discuss the runtime of the perturbation machine as a possible source of computational expense, I would have liked to see direct timing comparisons between the proposed attack and existing black-box attacks. This is particularly important since the per-query time for this attack is likely to be higher than attacks using simpler queries.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The design of the proposed attack which involves optimizing over the weights in an ensemble of models to generate transferable adversarial examples ensures, by construction, that the number of queries used will be limited due to the significant amount of prior information leveraged to construct the adversarial examples. This is borne out by the empirical results, which use a small number of queries even for high-dimensional Imagenet data.\n+ The attack description is clear and well-written. The attack is easy to implement given a number of trained models with adversarial examples that transfer to the model under consideration.\n+ The attack is shown to be effective against different architectures.\n**Weaknesses**\n- My main concern with the paper is that the evaluation is rather thin. The experiments are carried out on a single dataset, and only models with non-robust training are considered. Also, only a single modality (image classification) and vision API (Google Cloud) is considered. For a more thorough evaluation, I would recommend that the authors do the following:\n     - Attempt black-box attacks on another modality such as object recognition or segmentation. This will also enable the use of a different dataset such as MS-COCO.\n    - Explore the space of attacks on robustly trained models. Does using robust models in the ensemble help or hinder the generation of effective adversarial examples? Does the number of queries needed to attack a robust model increase as the model's robustness increases (with respect to a specified threat model)?\n   - Carry out a comparative evaluation of the attack's effectiveness on different vision APIs. This is also important to determine the relation between the chosen surrogate models and the target model.\n   - Provide further explanations of the link between the models chosen in the ensemble and the target model. It appears that the models chosen to create the ensemble are very close to those targeted. What happens when the models in the ensemble are trained on a different dataset, for example, or a smaller subset of the same dataset?\n- The paper is missing a reference to very closely related work from Suya et al. (Hybrid Batch Attacks: Finding Black-box Adversarial Examples with Limited Queries).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8JM5Iv61AHX", "reviewer": "Reviewer_Emex", "summary": "This paper proposes an ensemble search strategy for black-box adversarial attacks, which updates the weight of each ensemble member according to the queried feedback from target models. The empirical evaluation is done on the NeurIPS 2017 competition dataset (with 1,000 ImageNet-like images), using different model architectures. The results show that the query efficiency is largely improved.", "questions": "Please answer the listed weaknesses. My final rate may depend on the authors' response. ", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear writing and explanation of the proposed ensemble search strategy.\n- The ensemble search strategy is simple but effective, leading to a significant empirical improvement.\n- Commercial Google Cloud Vision API is also tested.\n\nWeaknesses:\n- My main concern is that in Table 1, the average query times of the proposed method (Ours) is around 1~2, which means the ensemble weights are only slightly updated. This makes me wonder if the effectiveness comes from the ensemble rather than the weight searching strategy.\n- The architectures of the target model (e.g., VGG, DenseNet, ResNext in Figure 2) are included in the substitute ensemble, thus the ensemble search strategy is more like 'guessing the architecture of the target model'. This is not the common case in practice, because the substitute ensemble usually may not involve the architecture of the target model. More ablation studies on this aspect would be helpful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Jj6A296Wvk", "reviewer": "Reviewer_wjHs", "summary": "This paper proposes a new method in generating efficient adversarial examples by a set of surrogate models. They apply PGD over the losses computed by these surrogates and a group of weights to adjust the effect of different models. In the experiments, they can find the adversarial example for each input within only a few queries, which is 30x faster than previous methods.\n", "questions": "Here are some suggestions which may help improve this paper.\n\n1. In equation (6), the author actually uses the target label y*, which is different from previous expression trying to maximize the loss. It is suggesting that the definition of y* is re-emphasized here for clarification. Similar problem can be found in the expression of PGD optimization.\n\n2. The main idea or the advantage of proposed method is to leverage a variety of existing models to generate adversarial examples that could fool all the surrogate models. Such examples also intend to fool the targeted victim models in large probability. The trick of the method is to transfer the consumed queries in black-box attack to white-box attack performed by a variety of surrogate models. Thus, the number of T consumed by surrogate models should be well-studied and discussed. In experiments, for more comprehensive results, it is suggesting that an appendix column to show the number of T is added in the table to make it clearer or fairer.\n\n3. why use coordinate gradient descent for updating w instead of applying SGD over all elements in w?\n\n4. In the appendix, the authors also study the transferability of their generated adversarial examples in the task of object detection, which is good and convincing. I got another concern that, together the black-box classification task in the first phase, is data used for training surrogates the same as the data used in attack? I mean, for example, can the surrogate be trained in the CIFAR, and the attack is performed in the ImageNet. Can we assume the attacked data is unknown in advance and different from the data used in training surrogate?", "limitations": "See above.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is easy to understand and the structure is clear. Overall, the presentation is also good. \nThe idea of this paper makes sense and is straightforward.\n\nThe novelty of this paper or some analysis seems a little limited and not enough, which i mean, brings limited inspiration for our future studies. To solve this, a comprehensive theoretical analysis or more experimental analysis maybe needed.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept