Random Noise Defense Against Query-Based Black-Box Attacks
Authors: Zeyu Qin
Yanbo Fan
Hongyuan Zha
Baoyuan Wu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Black-Box Defense
Black-Box Attacks
Adversarial Robustness

Abstract: The query-based black-box attacks have raised serious threats to machine learning models in many real applications. In this work, we study a lightweight defense method, dubbed Random Noise Defense (RND), which adds proper Gaussian noise to each query. We conduct the theoretical analysis about the effectiveness of RND against query-based black-box attacks and the corresponding adaptive attacks. Our theoretical results reveal that the defense performance of RND is determined by the magnitude ratio between the noise induced by RND and the noise added by the attackers for gradient estimation or local search.  The large magnitude ratio leads to the stronger defense performance of RND, and it's also critical for mitigating adaptive attacks. Based on our analysis, we further propose to combine RND with a plausible Gaussian augmentation Fine-tuning (RND-GF). It enables RND to add larger noise to each query while maintaining the clean accuracy to obtain a better trade-off between clean accuracy and defense performance. Additionally, RND can be flexibly combined with the existing defense methods to further boost the adversarial robustness, such as adversarial training (AT). Extensive experiments on CIFAR-10 and ImageNet verify our theoretical findings and the effectiveness of RND and RND-GF. 

Random Noise Defense Against Query-Based Black-Box Attacks
Zeyu Qin'
Yanbo Fan
Hongyuan Zha'
Baoyuan Wu
School of Data Science, Shenzhen Research Institute of Big Data,
The Chinese University of Hong Kong. Shenzhen
Tencent AI Lab
zeyuqinelink cuhk edu. cn, Sanyanbo01240gman com, zhahyCcuhk. edu. en, wubaoyuan@cuhk. edu.cn
Abstract
The query-based black-box attacks have raised serious threats to machine learning models in many real applications. In this work, we study a lightweight defense method, dubbed Random Noise Defense (RND), which adds proper Gaussian noise to each query. We conduct the theoretical analysis about the effectiveness of RND against query-based black-box attacks and the corresponding adaptive attacks. Our theoretical results reveal that the defense performance of RND is determined by the magnitude ratio between the noise induced by RND and the noise added the attackers for gradient estimation or local search. The large magnitude ratio leads the stronger defense performance RND, and it's also critical for mitigating adaptive attacks. Based on our analysis, we further propose to combine RND with plausible Gaussian augmentation Fine-tuning (RND-GF). It enables RND to add larger noise to each query while maintaining the clean accuracy to obtain a better trade-off between clean accuracy and defense performance. Additionally, RND can be flexibly combined with the existing defense methods to further boost the adversarial robustness, such as adversarial training (AT). Extensive experiments on CIFAR-10 and ImageNet verify our theoretical findings and the effectiveness of RND and RND-GE.
Introduction
Deep neural networks (DNNs) have been successfully applied in many safety-critical tasks, such as autonomous driving, face recognition and verification, etc. However, it has been shown that DNN models are vulnerable adversarial examples [18] 26 29 48), which are indistinguishable from natural examples but make model produce erroneous predictions. For real-world applications, the DNN model as well as the training dataset, are often hidden from users. Instead, only the model feedback for each query (e.g., labels or confidence scores) is accessible. In this case, the product providers mainly face severe threats from query-based black-bor attacks, which don't require any knowledge about the attacked models.
In this work, we focus on efficient defense techniques against query-based black-box attacks, of which the main challenges are 1) the defender should not significantly influence the model's is feedback to normal queries, but is difficult to know whether query is normal or malicious; the defender has no information about what kinds of black-box attack strategies adopted by the attacker. Considerable
efforts have been devoted to improving the adversarial robustness of DNNs B3
50). Among
them, adversarial training (AT) is considered as the most effective defense techniques
3 49
However, the improved robustness from AT is often accompanied by significant degradation of the
CCoresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

clean accuracy. Besides, the training cost of AT is much higher than that of standard training, and then it also often suffers from poor generalization new samples or adversarial attacks [20, 39 45 46].
Thus, we argue that AT-based defense is not very suitable choice for black-box defense. In contrast, we expect that a good defense technique should satisfy the following requirements: well keeping clean accuracy, lightweight, and felug-and-play.
To this end, we study a lightweight defense strategy, dubbed Random Noise Defense (RND) against query-based black-box attacks. For query-based attacks M 2 7 19] 23
26
27
32 351 the
core to find an attack direction by gradient estimation or random search based on the exact feedback of consecutive queries, which leads to a decrease of the designed objective. RND is realized by adding random noise each query the inference time. Therefore. the returned feedback with randomness results in poor gradient estimation or random search and slows down the attack process.
To better understand the effectiveness of RND, provide theoretical analysis of the defense performance of RND against query-based attacks and adaptive attacks. Our theoretical results reveal that the defense performance of RND determined by the magnitude ratio between the noise induced by RND and the noise added by the attackers for gradient estimation or random search. The attack efficiency is significantly affected by large magnitude ratio. The attackers need more queries to find the adversarial examples or fail to find a successful attack under limited query settings. That is, the larger ratio leads the better defense performance of RND. Apart from standard attacks, the adaptive attack (EOT) [3] has been considered as an effective strategy to mitigate the random effect induced by the defenders. We also conduct theoretical analysis about the defense effect of RND against HOT attacks, and demonstrate that the magnitude ratio is also crucial to mitigate adaptive attacks and the adaptive attacks have the limited impact of evading the RND. On the other hand, large random noises to each query may lead to degradation of clean accuracy. To achieve a better trade-off between the defense effect and the clean accuracy while maintaining training time efficiency, we further propose combining RND with lightweight Gaussian augmentation Fine-tuning (RND-GF). RND-GF enables us to adopt larger noise in inference time to disturb the query process better.
We conduct extensive experiments on CIFAR-10 10 and ImageNet. The experimental results verify our theoretical results and demonstrate the effectiveness of RND-GF. It is worth noting that RND is huu-and--lay and easy to combine with existing defense methods such as AT to boost the defense performance further. To verify these, we evaluate the performance of RND combined with AT [22 and find that the RND can improve the robust accuracy of AT by up to 23.1% against the SOTA black-box attack Square attack 2 with maintaining clean accuracy.
The main contributions of this work are four-fold.
We study lightweight random noise defense (RND) against black-box attacks theoretically and empirically. Our theoretical results reveal that the effectiveness of RND is determined by the magnitude ratio between the noise induced by RND and the noise added by the attackers for gradient estimation and random search.
We theoretically analyze the performance of RND against the adaptive attack (EOT) and demonstrate that EOT has the limited effect of evading the RND.
Leveraging our theoretical analysis, we further propose an efficient and stronger defense strategy RND-GF by combining the Gaussian augmentation Fine-tuning and RND towards a better trade-off between clean and adversarial performance.
Extensive experiments verify our theoretical analysis and show the effectiveness of our defense methods against several state-of- state-of-the-ane the-art qusey-based attacks.
2
Related Work
Query-based Methods. Here we mainly review the query-based black-box attack methods, which can be categorized into two classes, including gradient estimation and search-based methods. Gra- dient estimation methods are based on zero-order (ZO) optimization algorithms. The attacker utilizes a direct search strategy to find the search direction in search- based methods instead of explicitly estimating gradient. Furthermore, for our theoretical analysis, we focus on score-based queries. The continuous score (e.g., the posterior probability or the logit) for each query is returned, in contrast to the decision-bant -baee queries which return hard labels. Specifically, (26 proposed the first limited query-bas based attack method by utilizing the Natural Evolutionary Strategies (NES) to estimate the

---

[32 proposed the ZOsignSGD algorithm, which is similar to NES. Based on Bandit Optimization, 27 proposed to combine the time and data-dependent gradient prior with gradient estimation, which dramatically reduced the number of queries. For the l2 norm constrain, SimBA [23] randomly sampled perturbation from orthonormal basis. SignHunter [ focuses on estimating the sign of gradient and flipped the sign of perturbation to improve query efficiency. Square attack 2 is the state-of-art art query-based attack method that selects localized square-shaped updates at random positions of images.
gradient.
Black-Box Box Defense. Compared with the defense for white-box attacks, the defense specially designed for black-box attacks has not been well studied. Two recent works 8 31 proposed to detect malicious queries based on the comparison with the history queries since the malicious queries more similar with each other compared with normal queries. AdvMind [37 proposed to infer the intent of the adversary, and also needed to store the history queries. However, suppose the attacker adopted the strategy of long-interval malicious queries. In that case, the defender has to store a long history, with a very high cost of storage and comparison. [42] proposed the first black-bon certified defense method, dubbed denoised smoothing. [5] also showed that adding random noise can defend against query-based attacks through experimental evaluations, without theoretical analysis of the defense effect. There also few indomization-based defenses. R&P [51] proposed random input transform-base
method. RSE 33 added large Gaussian noise to both the input and activation of each layer. PNI 25 combined adversarial training with adding Gaussian noise the
input or weight of each layer. However, these methods significantly sacrificed the accuracy of benign examples and also have a huge training cost. PixeIDP 33] and random smoothing [13 proposed to train classifiers with large Gaussian noise to get certified robustness under the & norm. However, they require to obtain a majority prediction of this query. Obviously, that places huge burden on model inference and even helps the attacker get the more accurate gradient estimation. The too large Gaussian noise used in those methods also sacrifices clean accuracy. In contrast, RND maintains good clean accuracy and only perturbs each query once, without any extra burden. [40] showed that the model with Gaussian augmentation training achieves the state-of-art defense against common corruptions, while the defense to black-box attacks is not evaluated.
3 Preliminaries
3.1 Score-based Black-Box Attack
In this work, we mainly focus on the score-based attacks. The analysis and evaluation of decision- based attacks are shown in supplementary materials. We denote the attacked model as M : X + y with X' being the input space and y being the output space. Given benign data (ær, y) E (X) the goal of adversarial attack is to generate an adversarial example Eade that is similar with æ, but to enforce prediction of M to be different with the true label (i.e., untargeted attack) or to be target label (i.e., targeted attack). The optimization of untargeted attack can be formulated as follows,
min f(aadv = min (My max (Z j79
(1)
Lado (x)
Pado (adoeNR(a)))
For targeted attack, the objective function is maxjęth (w.de))) MM (M (Zadddv) denotes the logit or softmax output w.t.. class j. NN(xx) = {o'l x - Ip I R} indicates a lp ball around (p is often specified as 2 or oo), with R > O. In attack evaluation, as long as L(xadv) is less than O, attackers consider the attack to be successful.
The score-based attack algorithms commonly adopt the projected gradient descent algorithms.
1t+1 Proivn (x+ (x;()))).
For white-box attacks, g(x:) is the gradient of f(x) w.t.. Ht. However, for black-box attacks, the gradient (xx)) cannot be directly obtained. So the attackers utilize the gradient estimation random search to conduct the update g(x;).
3.2 Zero-Order Optimization for Black-Box Attack
Zero-order optimization (ZO) has become the mainstream of black-box attacks, dubbed ZO attacks. The general idea of 70 attack methods is to estimate the gradient according to the objective values returned by querying. The gradient estimator widely used in score-based and decision- based attacks
3

---

12 16 26 27
32
36 is
(a) f(x + (u) - f(x)
(3)
f
where u ~ (0 I), and μ 2 0. Based on this gradient estimator, the update of attack becomes
D++1 Proi (x)(())))
3.3 Random Search for Black-Box Attack
The Here, u is a searching direction sampled from some pre-defined distributions, such as gaussian noise in [12), orthogonal basis in [23] and squared perturbations in [2]. The μ is the size of perturbation in
attacks also depend on the value of f( + uu) f(x) to find attack directions.
each search step such as the size square in Square attack [2]. The work of [[] adopt the fixed valuable attack direction. So,
size schedule. If f(x + pm) (x)
the attackers consider
the direction searching becomes
s(x) = [(((z) < 0) -uu where h(x=) f( + pm) f())
(5)
where, is the indicator function. And the updating of search-based attacks becomes
C1+1 ProjNx(w)) (æ+ + s(x))).
(6)
4 RND: Random Noise Defense Against Query-based Attack Methods
4.1 Random Noise Defense
In the gradient estimator (e.g., (3. and searching direction (e.g., (5) the attacker adds one small random perturbation yeu to get the objective difference between two queries. Thus, if the defender can further disturb this random perturbation, the gradient estimation or searching direction is expected to be misled to decrease the attack efficiency. However, it's also hard for defenders to identify whether a query normal or malicious. Inspired by those observations, we study a lightweight defense method, dubbed Random Noise Defense (RND), that simply adds random noise to each query. For RND, the feedback for one query æ is M(x + ve), with v ~ (O, I) is standard Gaussian noise added by the defender, and the factor и 0 controls the magnitude of random noise. Considering the task of defending guery-based black-box attacks, RND should satisfy: 1) the prediction of each query will not be changed significantly; 2) the estimated gradient direction searching should be perturbed as large as possible towards better defense performance. The gradient estimator and searching direction under RND become
f (x + pru + vvi) f (xc + vU2)
u
(xx) -L(hp (x) 0) you where (x) f(x pau + vvi) f(x + v02),
(8)
where 81, 02 N(O,I) are both standard Gaussian noise generated by the defender.
To satisfy the first requirement, one cannot add too large noise to each query. However, to meet the second requirement, v should be large enough to change the gradient estimation searching direction. We need to choose a proper to achieve good trade-off between these two requirements. In the following, we provide the theoretical analysis of RND, which can shed light on the setting of v.
4.2 Theoretical Analysis of Random Noise Defense Against ZO Attacks
In this section, we will present the theoretical analysis of the effect of RND against ZO attacks. Specifically, study the convergence property of ZO attack in Eq.44) with (x) in Eq. being the gradient estimator. Throughout our analysis, the measure of adversarial perturbation is specified &a norm, corresponding to Nre(2)) [a''I][=' - æ1122 < R). To facilitate subsequent analyses, first introduce some assumptions, notations, and definitions.
Assumption 1. f(x) is Lipschitz-con continuous, i.c. (f(x) - f(x) & Lo(f)ly - x/|
Assumption 2. [ax is continuous and differentiable, and V V((x)) lapschite- i.e. [Vf(v) - fz)))] Li(f)]ly - =xl].

---

Notations. We denote the sequence of standard Gaussian noises added by the attacker as Ur {Uo, u....., ur with t being the iteration index in the update Eq.(4). The sequence of standard Gaussian noises added by the defender is denoted as V: = (vo1 1o22.... 121, 1/2). The generated sequential solutions are denoted as {roo æ1..... #Q)) and the benign example x used as the initial solution, Le., Zo = a. d = | denotes the input dimension.
Definition 1. The Gaussian Gaussian-Smoothing Smwothing function corresponding to f() with v > 0, ~ (0, is
(a(x)
1
[(x + vv) 11lv1 de.
(2T)d/2
Due to the noise inserted by the defender, fo becomes the objective function for the attacker. We also define the minimum of fa)) 82 fa(a") min Sy (æ).
4.2.1 Theoretical Analysis under General Non-Convex Case
Here, we report theoretical analysis of RND under general non-convex case satisfying Assumption 1
Theorem 1. Under Assumption for any Q 2 consider a sequence fx generated according to the descent update Eq.14) using the gradient estimator (u.((z) in Eq. 7
with the constant stepsize,
I/2
Re
Then,
Eu,.y (n v 2Lo(f)&Rdd (Q (Q+1) 7(a)
(10)
where"s V. j
and 7(a) (((a=aa a va which is always increasing function of
Remark 1. Due to the non-convexity assumption, we only guarantee the convergence to a stationary point of the function (2), which is smoothing approximation of fv. To bound the gap between (xx)(x) and (at),, i.e., (fx - fx)) S e, x E (N) we could choose S is TI72Lo(f) 366. The minimum of upper bound is denoted as ó, then the upper bound for the expected number of queries is O (7()) 2 L2())
2L)(()
65
The convergence rate of 70 attacks is important since attackers need to find adversarial examples within limited queries effectively. Theorem T shows the convergence rate is positive related to the ratio 2 The larger ratio 2 will lead to the higher upper bound of convergence error and slower x² convergence rate, consequence, the attackers need much more queries to find adversarial examples. Specifically, if << 1, the query complexity is equivalent to the original constant term O edz When
2 1, the query complexity is really improved over O c Therefore, the attack efficiency will be significantly reduced under the queries limited setting, leading to failed attacks much larger number of queries for successful attacks. Therefore, the large ratio leads the effectiveness of RND. In accordance with our intuition, the defenders should insert larger noise (i.e., v) than that added by the attack (i.e., μt) to achieve the satisfied defense effect.
Trade-off of Larger V and Clean Accuracy: If f(x) is Lipschitz-contimnous, then (x) - f(x)] S v (f) (36. The larger v is, the larger the gap between fo (x) and fxx). So the clean accuracy of the model with adding larger noise will also decrease. This is also shown in Figure [ (d), which forms a trade-off between defense performance of RND and clean accuracy.
The attacker may be aware of the defense mechanism. increasing the adopted noise size μ. Our experiment results also verify this point. As shown in Figure 2 (a), for NES attack, the attack failure rate is almost O, when
Larger Noise Size μ Adopted by Attackers:
μ 0.01. This shows adding
small noise can't 't always guarantee an effective defense. Previous work 5 [5] don't 't consider this and overestimate the effect of small noise. However, increasing the noise size μ will also lead to less accurate gradient estimation and random search in (3) and Eq, (5). leading to a significant decrease in attack performance consequentially. Taking the NES (6] and Bandit attack (27) on ImageNet as examples, as shown in Figure 3 when increase from 0.01 to 0.05, the f.x. attack failure rates increase from 5. 1.% %, 5.0% to 22.8%, 7% respectively. For Square attack, 22, when increases from 0.3 to 0.5, the average number of queries has doubled. To guarantee successful
5

---

attack, attackers cannot always increase Our experimental results show that the attack performance decreases significantly when the μ ZO attacks and Square attack is larger than 0.01 and 0.3. Based on the above analysis, we propose the stronger defense method in Section 4.4 It enables us to add larger noise (v > (1) and still maintains good clean accuracy.
4.2.2 Theoretical Analysis of Random Noise Defense Against Adaptive Attacks
As suggested in recent studies of robust defense R 6 49, the defender should take robust evaluation against the corresponding adaptive attack. The attacker aware of the defense mechanism. Here we study the defense effect of RND against adaptive attacks. Since the idea of RND is to insert random noise to each query, an adaptive attacker could utilize Expectation Over Transformation (EOT) [4] to obtain a more accurate estimation, i.e., querying one sample multiple times to get the average value. Then, the gradient estimator used ZO attacks Eq. becomes
1 M
M f(æ + pau + ve,1) - f(x + 10222 4.
(x)(
Σ
(11)
where 1/1, 42 No,,I), I). with j = I....,M M. Note that here the definition of the sequential standard Gaussian noises added by the defender (see Section 4.2.1) should be updated V.
{ (Vol...., VoM ..... U......, VeA J. Vij E V. contains Viy1 and v:j2. The convergence analysis of ZO attack with [T against RND is presented in Theorem 2
Theorem 2. Under Assumption / andd for any Q 2 O, consider a sequence using the gradient estimator (p,x(x) Eq.
< tz= generated we have
according the descent update 14.44)
11
1 SQ t=
Lo(S)
1
Th ((Lo()))2())
2v²
SQ
SQ
p2M
L=
(12)
2² (o() d ( +1) d3)
1
2μ22M
The larger M for EOT: Theorem 2 shows that the upper bound will be reduced with larger M. So, EOT can mitigate the defense effect caused by the randomness of RND. However, with M going to infinity, the upper bound of expected convergence error (i.e., Eq. 12) is still determined by the max term //2 dd. It implies that the attack improvement from EOT is limited, especially with the larger ratio g. Experiments in Section 5.3 verify analysis of EOT attack.
4.3 Theoretical Analysis of Random Noise Defense Against Search-based Attacks
In this section, we will show how RND affects search-based attacks. Based on the Eq. and (q. by adding noise vu, the value of (x:) will be different from that of h(æz). and there is certain probability that Sign(h,(x:) be different from Sign(h(x)), When the random noise 1/v causes inconsistence between Sign(h,(x:) and Sign(h(=)), RND will mislead the attackers to select the incorrect attack directions (i.e., abandoning the descent direction w.r.t. f or selecting the ascent direction), so as to decrease attack performance. We have following theoretical analysis about this.
Theorem 3. Under Assumption 7 considering the direction update Eq. 10 with Eq. ( in search-based attacks, we have,
P(Sign(h(z)) * (i)n(h.( S 2Lo(J)uVd
(13)
|h(x)]
The probability P is intuitively controlled by the relative values of the (x) and h(x)). The proof is shown in Section F of supplementary materials. Theorem shoows the probability of misleading attacker is positive correlated with [n(=)]] Due to the small value μ and local linearity of model,
we have [((=)] = [ + pu) - f()) so Lo(f)p|lull, Therefore, the [h(x)] is also positively correlated with the stepsize μ within the small neighborhoods. So the probability of changing the
sign is positively correlated with i The larger ratio leads to the larger upper bound of the
probability. The attackers are more likely to be misleading and need much more queries to find adversarial examples. So, the defense performance of RND is better. As shown Figure T (a-c), the evaluations Square attack verify our analysis.

---

attack VGG
Scuarn atfact Iroephinn
Sqg
AT moon
dlca difforent models
2:13
362 2.000
FGB 1- 062
FIE ym 045
17%
Vitica
77%
975
1030
45%
40%
wode
300
35%
125
X5
Racterlo
305
3.00
0.6
028
0.5
0.30
0.5
0:12
0.15
425
1:10
06
005
0.15
0315
4.30
4.%
6:10
28
6re
The ctweta
The 120
Theazeet
(a)
(b)
(c)
(d)
Figure 1: (a-c): Attack failure rate (%) of Square foc attacks on VGG-16(CIFAR-100 Inception (ImmaeNet)) and AT model (ImageNet) under different values of μ and v. where p is the square size in Square attacks. (d): clean accuracy for different models on CIFAR- 10 and ImageNet. The
circle lines and triangle lines represent models on CIFAR-10 10 and ImageNet respectively.
4.4 RND with Gaussian Augmentation Fine-tuning
The aforementioned theoretical analyses reveal that RND should choose a proper noise magnitude v to achieve good balance between clean accuracy and defense effectiveness. To achieve high-quality balance, we could reduce the sensitivity of the target model to random noises. The influence of the noise on the accuracy of each query will be reduced. To satisfy the lightweight requirement of black-box defense, we propose to utilize Gaussian Augmentation Fine-tuning (GF), which fine-tuner the deployed model with Gaussian augmentation. We add random Gaussian noise to each training sample as preppeccssingg step in the tine-tuning process. Consequently, the model fine-tuned with GF expected to maintain good accuracy, even though defenders add a relatively large random noise to each query. As shown in Figure (d), GF models still maintain the high clean accuracy under larger noise.
Experiments
5.1 Experimental Settings
Datasets and Classification Models. We conduct experiments two widely used benchmark datasets in adversarial machine learning: CIFAR-10 10 (28) and ImageNet [4]. For classification models, VGG-16 [43] WideResNet-I6-10 and WideResNet-28-11
on CIFAR-10. We conducted standard training, and their clean accuracy on the test set is 92. 76% 95.10, and 96.60%, respectively. For ImageNet, we adopt the pretrained Inception v3 model [47 and ResNet-50 model [24] provided by torchvision package and the clean accuracy are 76.80% and 74.90%, respectively.
Black-box Attack Methods and Compared Defense Methods. We consider several main- streamed query-based black-box attack methods, including NES 126). ZO-signSGD (ZS) (32] Bandit 277, ECO (35), SimBA 233, SignHunter [1] and Square attack (2). Note that the NES, ZS, Bandit are gradient estimation based attack methods, and the other four are search- based methods. SimBA and ECO are only designed for & and fo attack, respectively. Following (26), we evaluate all the attack methods on the whole test set of CIFAR-10 10 and 1 1,000 random sampled images from the validation set of ImageNet. We present the evaluation performance against the untargeted attack under both foo and lg norm settings. The perturbation budget of lxo is set to 0.05 for both datasets. For E, attack, the perturbation budget is set to 1 and on CIFAR-10 10 and ImageNet, respectively. The number of maximal queries is set to 10,000. We adopt the attack failure rate and the average number of query as an evaluation metric. The higher the attack failure rate and the larger the average number of query, the better the adversarial defense performance. We compare our methods with RSE [33] and PNI (25) on CIFAR- 10. We adopt the pre-trained AT model [22) which shows better robustness than other AT models. For ImageNet,
compare pre-trained Feature Denoise (FD) model [52] and adopt pre-trained AT in Robustness Library [7]] The implementation details of those methods are given in the Section B.1 and B.2 of supplementary materials.
5.2 Evaluation of RND Against Query Black-Box Attacks
We first evaluate the defense performance of RND with various settings of j and against query- based black-box attacks and verify the theoretical results in Section 4.2 and 4.3 Specifically, we set
7

---

undort. +0.05
200ggiSc urder
Paidi under
Sqiiuneee (CS, fimMA
55.
960 8:21
160 0:32
BND 200
30%
Wihee 110
75%
3%
10%
55%
-- 180 y- D01
12%
42%
180 822
45%
10%
37%
20%
27%
455
0%
04
4.2
-2.5
2.5
-20
-15
-10 -55
-15
4.0
-25
2000
0.02
0002
0013
0020
(a)
(b)
(c)
(d)
Figure 2: Attack failure rate (%) of query-based attacks on VGG-16 and CIFAR-10 under different values of j and v. We adopt logarithm scale in subplot (a-c) for better illustration. The complete evaluation under lg attack is given in Section B.4 of supplementary materials.
Tandi
Sigel unterr, ECS, SimMA
50%
y-
10%
J5.5
12%
30%
50%
BKD E-0
Mites:
A607--09 2-11
40%
30.
35%
-5
20%
960
13%
IND
0:22
335
E2
Sediis
05
-10
-35
-35
-3.
-1.
0%
-10
-15
-10
-15
2000
5.02
0022
6013
0022
(a)
(b)
(c)
(d)
Figure 3: Attack failure rate (%) of query-based attacks on Inception v3 and ImageNet under different values of and v. We adopt logarithm scale in subplot (a-c) for better illustration. The
complete evaluation under &g attack is given in Section B.4 of supplementary materials.
E (0.0, 0.01, 0.02) and μ {10-4,1000000 5, 10-3 0.005, 0.01, 0.05) for ZO attacks on CIFAR-10
and f E {10-4, 0.001, 0.01, 0.05) on ImageNet. For Square attacks, we set μ E {0.05, 0.1. 0.3). Figure [ (a-c) presents the defense performance of RND against Square attack on CIFAR-10 and ImageNet, respectively. Figure (a-c) 3 (a-c) present the defense performance of RND against NES, ZS and Bandit on CIFAR- 10 and ImageNet, respectively.
The results in Figure | (a-c).
2.c (a-c), and (a-c) show that: the attack failure rate of all attack methods generally increases the value of & increases. Specifically, for fixed value of (e.g., 0.01), the
attack failure rate increases as μ decreases. While for a certain value of ja (e.g., 10-2), the attack failure rate increases as v increases. These collaborate our theoretical analysis that the ratio of 2 P determines the probability of changing sign and the convergence rate of ZO attacks. The larger x the higher the probability of changing the sign and the convergence error of ZO attacks, which results in poor attack performance under the query-limited settings.
We also evaluate the defense performance of RND with various values of against other search- based black-box attacks and the results are shown in Figure (d) for CIFAR-10 10 and Figure 3 (d) for ImageNet. As shown in the plots, the RND can significantly boost the defense performance. The attack failure rate increases as the value of v increases. The complete evaluations, including the Wede on CIFAR-10, the & attack, and the mean and medium number of queries of successful attacks, are reported in Section B.4 of supplementary materials.
5.3 Evaluation of RND Against Adaptive Attacks
We then evaluate the defense effect of RND against the adaptive attack EOT and verify our theoretical analysis in Section 4.2.2 We evaluate EOT with M E {1, 10} CIFAR-10 and ImageNet, and v is set to 0.02 and μ is set to 0.001 and 0.1 for ZO attacks and Square attack, respectively. The evaluation with even larger M and a is presented in Section B.5 of supplementary materials.
We consider two settings of query budget: 1) adaptive query budger that assigns query budget of 10,000 x x. for different value of M; 2) fixed query budget that adopt fixed query budget of 10,000 for all M. We first evaluate the performance with adaptive query budget on NES and ZS, and their numerical results are shown in Table As shown in T the attack failure rate decreases as M increases on both datasets. However, the average number of the query of successful attack also significantly increases as M increases, which demonstrates that the adaptive EOT attack can
8

---

The evaluation of EOT with So attack on CIFAR-10 and ImageNet under the adaptive and fixed query setting. The left part is the results on CIFAR-10 and the right part on ImageNet. The average number of query of successful attack as well as the attack failure rate are reported. The performance of EOT with lg attack is reported in Section B.5 of supplementary materials.
Table I:
settings
Methods
M=1
M=5
M= 10
Methods
M=1
M=5
M= 10
uduptive
NES
1448/04884
4078/0.361
576300342
NES
2532/0.762
5364/0.705
7582/0.691
ZS
1489/0.493
3189/0.374
5912/0.349
ZS
2824/0.825
5735/0.761
7662/0.740
NES
1448/0.484
2528/0.452
3246/0.443
NES
2533/0.762
524000.775
5658/0.781
ZS
1489/0.493
2765/0.448
3123/0,421
ZS
2824/0.825
4022/0,842
4652/0,86|
Bandit
436/0.696
276/0,582
314/0,543
Bandit
305/0.604
759/0.523
946/11.9
fixed
Square
380/0.301
181/0.16: 162
223/0.121
Square
93/0.353
145/0.20
328/0.171
SignHunter
459/0.367
559/0.224
759
SignHunter
173/0.532
336/0.456
659/0,431
ECO
904/0.720
2560/0.793
ECO
1237/0,666
3065/0.678
3091/0.692
SimBA
1353/0.650
3852/0.467
4103/0.396
SimBA
274/0.891
468/0.878
517/0.869
Table 2:
The comparison of RND (v = 0.02). GF, RND-GF (v 0.05), AT, RND-AT (v = 0.05), PNI, RSE, and FD) on CIFAR-10 and Imagenet. The average number of queries of successful attack and the attack failure rates are reported. The best and second best attack failure rate under each attack are highlighted in bold and underlined, respectively. The evaluation under Eo attack is shown in Section B.6 of supplementary materials,
Datasets
Methods
Clean Acc
NESU
7S(
Bandit(
Sign((-..)
Square(f
SimBA(fs)
BCO(F..)
Clean Model
96.60%
165, 5/5.51
581.8/0.05
210.2/0.03
167.6/0.03
137.1/0.02
457.2/0.04
457.8/0.0
GF
91.722 72%
999 0/0, 107
9/9.9/0.544
744.500.116 16
348.340.027
581.000 0/0.061
1146.840.390
883. 683.900.067
RSE
12.1
3/. 396
1927.8/0.422
281.700372
243.7/0.221
413.380 243
498.300.337
578.3/0.334
CIFAR-10
PST
20%
1071,/0
1310.740.88
324.9/0.224
267.0/0.088
295 296612
945.000.857
2342.2/0.623
(WideNat-28)
AT 22
48%
821 64. 802
618.9/0 852
645
766.300477
1135.480.99
1523. 230
1180/4/0/48
RND
93.90%
842.5/0.05
941 1/1./0.11 145
273.1/0.478
971.200.22
462.490.166
3112.600.544
912.8/0.688
RND GF
92. 20%
2805. 1/
2966. 340 730
1223. 541.841
1017.1/0.007
1007-3003775
1220. 2/0
687.2/0.87
RND-AT
87.40%
2499.2/0.842
2625. 720
891.500.991
767.9907377
1170.70077
1787. 4/0.112
687.44 0,911
Clean Model
74.90%
1031 940.0
2013.000.35
22
264.1/0003
76.5810
1234.
347.7/0.0
1685.
140.377
97.28/000
1417.400
362.4/0.0
ImageNet (ResNei-50)
kt
54.207%
1997. 679
1555.500.77 570.75
1579 220
1633, 1/0.332
1092 242
2607
1501
AT 17
61.99%
21133.4033 724
168% 741815
1091 416
1522. 2/00 289
1100 150
2638 2,0 651
1440 6-9.66000 200
RND
73.00%
3011.5/0.210
2266.
330
390.6/0.536
661.0/0.314
$1.5/0.101
825. 025.3/0.612
2435 540 5/0.540
RND-GF RND-AT
71.15%
2489, 3/0 421
2053.590.55 540 563
495 910 603
514,000.3388
1009
146
777.2/ 0.762
994, 8007702
58.15%
2556.6/0.864
2596.600.870
448.0/0.810
724.240.632
1306 300.36
1210.5/0.533
6/1.1/0.66
increase the attacking success rate with a sacrifice of query efficiency. More importantly, we observe that the relative performance improvements induced by EOT generally decrease as M increases. These verify our theoretical analysis Section 4.2.2
The evaluation under the fixed query budget is also reported in Table T On small-scale dataset CIFAR- 10, the attack failure rate of all attacks generally decreases as M increases. Yet. we also observe a similar phenomenon in that the relative performance improvements induced by EOT decrease as MM increases. For the large-scale dataset ImageNet, BOT can increase the attack performance with significant sacrifice of query efficiency under most cases, except for NES and ZS. The attack performance of NES and ZS decreases as M increases. Specifically, for fixed number of queries, the larger the M, the smaller the number of iterations for ZO attack under limited query setting. The poor performance of NES and 7S with larger M may be because the improvements induced by EOT are less than the potential degeneration caused by the decrease of iterations.
5.4 Evaluation of RND-GF and the Combination of RND with AT
In this section, we evaluate the performance of RND combined with Gaussian augmentation Fine- tuning, dubbed RNG-GF In fine-tuning phase, we adopt the cyclic learning rate 44 to achieve superconvergence in 50 epochs. The training detail is shown in Section B.3 of supplementary materials. We adopt the attack failure rate and the average number of queries under the stronger adaptive attack EOT for evaluation. As that in the Section 5.2 we also tune the parameter μ for attackers and tune M E {1, 5, 10) for EOT to achieve the best attack performance for all attack methods.
The comparison with other defense methods is shown in Table 2 where GF refers to standard fine-tuning with Gaussian augmentation. RND denotes the clean model coupled with random noise defense, and RND-GF indicates the GF model coupled with RND. As shown in the Table RND can significantly boost the defense performance of the Clean Model on both datasets. For example, RNG achieves 20% 40% improvement against Bandit and SignHunter on both datasets. The proposed
9

---

GF can protect the clean accuracy from the larger noise induced by RND. Therefore, we adopt relatively larger v 0.05 for RND-GF. According to our theoretical analysis, larger v will lead to better defense performance. Experimental results also show that RNG-GF significantly improves the defense performance under all attack methods while maintaining good clean accuracy compared with RND. For example, RNG-GF achieves 40% 50% and 20% ~ 30% improvements against ZO and search-based attacks on CIFAR- 10. Compared with RSE, PNI, and FD, RND-GF achieves a higher failure rate against Bandit, SimBA, and ECO and leads to much more queries on other attacks. Besides, it also maintains a much better clean accuracy and lower training cost.
Without any extra modification, RND can be easily combined with many existing defense methods. We adopt the pre-trained AT WideNet-28 (fec) model [22] and pre-trained AT ResNet-50 (fxc) model in Robustness Library [17 on CIFAR-10 10 and ImageNet, respectively. As shown in Figure T (d), AT
models are less affected by adding noise like GF models. So we can adopt the larger noise (v 0.05)
in inference time. Combining AT with RND, RND-AT significantly improves the robustness against all attacks and achieves the best performance among all methods. Based on AT models, RND-AT achieves 23. 1 and 72.7% improvements against Square attack on CIFAR-10 and ImageNet.
5.5 Discussion and Limitations
As shown in Section 3.1 we focus on the score-based attacks. The analysis and evaluation of RND shown in Section C of supplementary materials. Besides, as we mainly focus on practical query-based black-box attacks where the models and the training datasets are unknown to the attackers, we don't I cover the attacks utilizing the transferability from surrogate to target models. These methods usually assume that the surrogate models are trained on the same training set with the target model. It is difficult obtain the training set behind the target model in real scenarios. Meanwhile, there have been some defense strategies developed for transfer- based attacks [38 50 [53. It interesting explore the combination of RND and these works towards better defense performance against baaser attacks. We leave it to future works.
against decision- based attacks
Conclusion
In this work, we study a lightweight black-box defense, dubbed Random Noise Defense (RND) for query-based black-box attacks, which is realized by adding proper Gaussian noise to each query. We give the first theoretical analysis of the effectiveness of RND against both standard and adaptive query-based black-box attacks. Based on our analysis, we further propose RND-GF ,which combines RND with Gaussian augmentation Fine-tuning to obtain a better trade-off between clean accuracy and robustness. Extensive experiments verify our theoretical analysis and demonstrate the effectiveness of the RND-GF against several state-of-the-arr query-based attacks. Without any extra modification, RND can easily combine with the existing defense methods, such as AT. We further demonstrate that when combined with RND, RND-AT can significantly boost the adversarial robustness. Given that RND is very simple and effective, we recommend it should become baseline to evaluate new query- based black-bor box attack methods.
Acknowledgment
We want to thank Jiancong Xiao, Ziniu Li, Congliang Chen, Jiawei Zhang, and Yingru Li for helpful discussions. We want to thank the anonymous reviewers for their valuable suggestions and comments. Baoyuan Wu and Zeyu Qin are partially supported by the Natural Science Foundation of China under grant No. 1306220,,,,, the university development fund of the Chinese University of Hong Kong, Shenzhen under grant No.0100181 10, the special project fund of Shenzhen Research Institute of Big Data under grant No.TO.00000 T00120210003, and Tencent AI Lab Rhino-Bird Focused Research Program under grant No. JR202123. Hongyuan Zha is partially supported by a grant from Shenzhen Research Institute of Big Data.
References
[1] Abdullah Al-Dujaili and Una-May O'Reilly. Sign bits are all you need for black-box attacks. In International Conference Learning Representations, 2020.
10

---

[2] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pages 484-501. Springer, 2020.
[3] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In International Conference on Machine Learning, pages 274-283, 2018.
[4] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adver- sarial examples. In International conference machine learning, pages 284-293. PMLR, 2018.
[5] Junyoung Byun, Hyojun Go, and Changick Kim, Small input noise is enough to defend against query- black-box attacks, 2021.
[6] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: /902.0675, 2019.
[7] Jianbo Chen, Michael I Jordan, and Martin Wainwright. Hopskipjumpattack: A query-efficient decision-basee attack. In 2020 ieee symposium on security and privacy (sp), pages 1277-1294. IEEE, 2020.
[8] Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial attacks. In Proceedings of the 1st ACM Workshop Security and Privacy on Artificial Intelligence, pages 30-39, 2020.
[9] Weilun Chen, Zhaoxiang Zhang, Xiaolin Hu, and Baoyuan Wu. Boosting decision-based black-box adversarial attacks with random sign flip. In European Conference Computer Vision, pages 276-293. Springer, 2020.
[10] Minhao Cheng, Thong Le, Pin-Ya Yu Chen, Huan Zhang, JinFeng Yi, and Cho-Jui Hsieh. Query- efficient hard-label black-box attack: An coamiization-asse approach. In International Confer- ence on Learning Representations, 2019.
[11] Minhao Cheng, Simranjit Singh. Patrick Chen, Pin-Yo Chen, Sijia Liu, and Cho-Jui Hsich. Sign-opt: A query-efficient hard-label adversarial attack. arXiv preprint 2019.
[12] Minhao Cheng, Simranjit Singh, Patrick H. Chen, Pin-Yu Chen, Sijia Liu, and Cho-Jui Hsieh. Sign-opt: A query-efficient hard-label adversarial attack. In International Conference on Learning Representations, 2020.
[13] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320, 2019.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE, 2009.
[15] Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. Benchmarking adversarial robustness on image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 321-331, 2020.
[16] John C Duchi, Michael Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, (5):78888-2 2015.
[17] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robust- ness (python library), 2019.
[18] Yanbo Fan, Baoyuan Wu, Tuanhui Li, Yong Zhang, Mingyang Li, Zhifeng Li, and Yujiu Yang. Sparse adversarial attack via perturbation factorization. In Computer VisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pages 35-50. Springer, 2020.

---

[19] Yan Feng, Baoyuan Wu, Yanbo Fan, Li Liu, Zhifeng Li, and Shutao Xia. Boosting black-bor bbx attack with partially transferred conditional adversarial distribution, 2020.
[20] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustmess. In ICLR, 2019.
[21] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint vrXiv:1412.6572. 2014.
[22] Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples, arXiv preprint arXiv:2010.03593, 2020.
[23] Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pages 2484-2492 2019.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[25] Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable random- ness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 588-597, 2019.
[26] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Leurning, pages 1137-2146, 2018.
[27] Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversar- ial attacks with bandits and priors. In International Conference on Learning Representations, 2018.
[28] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[29] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. Artificial Intelligence Safety and Security, page 99-112, Jul 2018.
[30] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pages 656-672. IEEE, 2019.
[31] Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y Zhao Blacklight: Defending black-box adversarial attacks on deep neural networks. arXiv preprint arXiv:2006. 14042, 2020.
[32] Sijia Liu, Pin- Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2018.
[33] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369-385, 2018
[34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[35] Seungyong Moon. Gaon An, and Hyun Oh Song. Parsimonious black-box adversarial attacks via efficient combinatorial optimization. In International Conference on Machine Learning, pages 4636 6635.4645. PMLR, 2019.
[36] Yurii Nesterov and Vladimir Spokoiny, Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):577-56,, 2017.
12

---

[37] Ren Pang, Xinyang Zhang. Shouling Ji, Xiapu Luo, and Ting Wang. Advmind: Inferring adversary intent of black-box attacks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1899-1907, 2020.
[38] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference Machine Learning, pages 4970-4979. PMLR, 2019.
[39] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning, In International Conference on Machine Learning, pages 5093-8104 PMLR, 2020.
[40] Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way make neural networks robust against diverse image corruptions. In European Conference on Computer Vision, pages 53-69. Springer, 2020.
[41] Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems, pages 11292-1 1922-11302 11003, 2019.
[42] Hadi Salman, Mingjie Sun, Greg Yang. Ashish Kapoor, and J Zico Kolter. Denoised smoothing:
A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33, 2020.
[43] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[44] Leslie Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference applications of computer vision (WACV), pages 464-472. IEEE, 2017.
[45] Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Generalization error of invariant classifiers. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, AISTATS, pages 1094-1103, 2017.
[46] David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calibaate adversarial training: Generalizing to unseen attacks. In International Conference Machine Learning, pages PMLR, 2020.
[47] Szegedy, V Vanhoucke, S loffe, J Shlens, and Z Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2818-2826, 2016.
[48] Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199, 2013.
[49] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks adversarial example defenses. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1631., Curran Associates, Inc., 2020.
[50] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: 1705.000000 2017.
[51] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.
[52] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 501-509, 2019.
13

---

[53] Huanrui Yang. Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: Diversifying vulnerabilities for enhanced robust generation of ensembles. Advances in Neural Information Processing Systems, 33, 2020.
[54] Sergey Zagoruyko and Nikos Komodakis. 2t1.:1665.07714,.0166 2016.
Wide residual networks.
arXiv preprint
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See final part of Section [
(b) Did you describe the limitations of your work? [Yes] See Section 5.5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Supplementary Materials
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
4
(b) Did you include complete proofs of all theoretical results? [Yes] See Supplementary
Materials.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] See Section 5.1 and Supplementary Materials
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] We run all experiments times and average all results over 3 random seeds. We find the results of each experiment were almost identical to each other.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Supplementary Materials
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5.1
(b) Did you mention the license of the assets? [Yes] See Supplementary Materials
(c) Did you include any assets either in the supplemental material
URL? [Yes]
Yes, we provide our code in the Supplemental Materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No] No, all the used data are nell-known public benchmark datasets such as CIFAR10, ImageNet.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] No, all the used data are well-known public benchmark datasets such CIFAR10, ImageNet.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Not applicable.
14


Reviews:
[{"review_id": "vPvfNTeP_CH", "reviewer": "Reviewer_CW2N", "summary": "This paper proposes to use random Gaussian noise as defense for black-box attacks. Also, a Gaussian augmentation Fine-tuning is also proposed to enable adding larger noise.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is well motivated. The analysis is straight and well-understood.\n\nIn table 2, can you explain why RND-GF is even better than GF in terms of clean acc?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see my main review", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "v7kDjw-kEw", "reviewer": "Reviewer_opkE", "summary": "This paper proposes a simple but theoretically supported defense method against query-based black-box attack, called as Random Noise Defense. The core idea is to add Gaussian noise to a given input image at each query so it can be considered as quite simple. However, this paper shows theoretical analyses of RND against zero-order optimization, adaptive and search-based attacks. The overall finding through each theoretical analysis is that the number of queries for successful attacks by an attacker changes with the ratio of nu/mu. To overcome trade-off between clean accuracy and larger nu, the authors propose Gaussian augmentation fine-tuning for the target model. From the extensive experiments using CIFAR-10 and ImageNet against various black-box attacks, the authors demonstrated that, not only the results of theoretical analysis are confirmed by experiments but RND achieves the superior performance. Also, they showed RND can be easily combined with other defense methods and it achieves the state-of-the-art defense performance when it is combined with the adversarially-trained model.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\nThe proposed idea, adding Gaussian noise to an input image, has been used for adversarial defense, such as randomized smoothing. \nHowever, This paper has the following differences from it. Firstly, the proposed method is focused on adversarial defense against query-based black-box attack. Second, from a different perspective, this paper theoretically and experimentally shows the superiority of the proposed method. Therefore, this paper has its own contribution.\n\nQuality\n\nI didn't find any flaws in theoretical analysis, but I'm not sure because I am not a theoretical expert.\nThe authors demonstrated the strength and theoretical findings of the proposed method from the extensive and solid experiments well.\nAlso, they stated limitations and future works of their work.\n\nClarity\n\nThis paper is well written and easy to read.\n\nSignificance\n\nThe results of this paper is important because this paper proposes a simple defense method as well as a theoretical analysis of it.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations\n\n1. In my opinion, Assumption 1 is somewhat acceptable for a neural net-based classifier. However, I wonder if Assumption 2 is also generally accepted because it stands for a higher level of smooth functions.\n\n2. Input transformation-based defense methods (e.g. R&P, JPEG and bit-red) are should be considered as a baseline because [1] reported that these methods relatively achieve the superior robustness than other types of defense in case of query-based black-box attacks.\n\n3. I think that there is a methodical connection or similarity between the proposed method and randomized smoothing-based methods [13, 38, 39]. However, this paper seems to lack reference to that content. Therefore, it would be a more solid paper if the authors address differences in theoretical and methodological perspectives from existing similar methods.\n\n\n[1] Benchmarking Adversarial Robustness on Image Classification, Y. Dong et. al., CVPR 2020\n\n\nSocietal Impact\n\nThere is no potential negative social impact in this paper.", "review_text": "", "extra_scores": {"ethical_concerns": "Ethical Concerns\n\nThere is no ethical concerns in this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ei0aM5opb3w", "reviewer": "Reviewer_Q2xR", "summary": "This work proposes addition of Gaussian noise as a defense against black box attacks and analyses the convergence of query attacks with this defense enabled. The work is quite sound. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have few questions wrt experiments\n\n\n- What’s the motivation to choose a different \\nu for RND, RND-GF and RND-AT in Table 2?\n- RND by itself seemed quite poor against NES and Square attack while RND-GF provides quite a boost. Any insights on why this is the case?\n- There is a significant drop in clean accuracy with RND-AT, any suggestions on what hyperparameters could help tune this better?\n- What would you prescribe as a method to choose noise hyperparameters?\n- On a more general note, does the noise need to be Gaussian, or is there an intuition on the suitability of other quantisation noises?\n- Tables with small font size were a bit challenging to parse", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_U6PfMPn-Jh", "reviewer": "Reviewer_RFP8", "summary": "The authors suggest the use of Random Noise Defense (RND) to protect deep learning models from black-box query-based attacks. The authors motivate their proposal via both theoretical and experimental validation -- where RND is a lightweight defense that augments random noise to input queries. The paper further proposes RND-GF that is trained to be robust (smoothened) to random noise -- in order to further improve the adversarial robustness.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Merits\n1. The paper systematically studies decision and score based adversarial attacks, their theoretical implications, and empirical performances\n2. RND-GF helps combine adversarial and noise robustness\n3. This is one of the first works that studies the relation of random noise to black-box query complexity theoretically. \n\n### Qualms\n1. **Adaptive nature of attacks:** In adaptive attacks, EOT is the only attack considered. In my opinion, the knowledge of v and mu (only the magnitudes and not the exact noise) is one of the minimum knowledge that should be provided to any adaptive attacker. Moreover, all the theorems are heavily dependent on the ratio of v/mu. Can you elaborate in greater detail, the motivation behind this choice -- and how do the results fair when the knowledge is available to the attacker.\n\n2. **Scope of work:** The scope of the work, though important, is limited. It does not even consider all the types of black-box query-based attacks -- such as transfer adversaries; and is not applicable in the case of non-query attacks.\n\n3. **Failure Rates:** From the empirical failure rates, it can be seen that at the optimal selection of mu by the attacker, the failure rates can be very high. Given this information (in an empirical setting) -- would you suggest that RND is still beneficial to model robustness? (Given that adding random noise has a small tradeoff on clean accuracy as well)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows that adding a small amount of random noise to the output of a neural network can prevent current black-box attacks. The theory is convincing, the experiments cover a wide range of attacks, and the reviewers are satisfied by the author response. Even though there are still some questions about the generality of the proposed defense and if it could be evaded by stronger attacks, the proposal is interesting and will motivate future attack research to improve on randomized defenses.
Decision: Accept (Poster)