Small Input Noise is Enough to Defend Against Query-based Black-box Attacks
Authors: Junyoung Byun
Hyojun Go
Changick Kim
Venue:  (2021)
Keywords: Gaussian noise
input noise
adversarial defense
black-box attack
adversarial attack
query-based attack

Abstract: While deep neural networks show unprecedented performance in various tasks, the vulnerability to adversarial examples hinders their deployment in safety-critical systems. Many studies have shown that attacks are also possible even in a  black-box setting where an adversary cannot access the target model's internal information. Most black-box attacks are based on queries, each of which obtains the target model's output for an input, and many recent studies focus on reducing the number of required queries. In this paper, we pay attention to an implicit assumption of these attacks that the target model's output exactly corresponds to the query input. If some randomness is introduced into the model to break this assumption, query-based attacks may have tremendous difficulty in both gradient estimation and local search, which are the core of their attack process. From this motivation, we observe even a small additive input noise can neutralize most query-based attacks and name this simple yet effective approach Small Noise Defense (SND). We analyze how SND can defend against query-based black-box attacks and demonstrate its effectiveness against eight different state-of-the-art attacks with CIFAR-10 and ImageNet datasets. Even with strong defense ability, SND almost maintains the original clean accuracy and computational speed. SND is readily applicable to pre-trained models by adding only one line of code at the inference stage, so we hope that it will be used as a baseline of defense against query-based black-box attacks in the future.

Under review as a conference paper at ICLR 2021
SMALL INPUT NOISE IS ENOUGH TO DEFEND AGAINST QUERY-BASED BLACK-BOX ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
While deep neural networks show unprecedented performance in various tasks, the vulnerability to adversarial examples hinders their deployment in safety-critical systems. Many studies have shown that attacks are also possible even in a black- box setting where adversary cannot access the target model's internal infor- mation. Most black-box attacks are based on queries, each which obtains the target model's output for an input, and many recent studies focus reducing the number of required queries. In this paper, we pay attention to an implicit as- sumption of these attacks that the target model's output exactly corresponds the query input. If some randomness is introduced into the model to break this assumption, query- query-based attacks may have tremendous difficulty in both gradi- ent estimation and local search, which are the core of their attack process, From this motivation, we observe even a small additive input noise can neutralize most query- query-based attacks and name this simple yet effective approach Small Noise De- fense (SND). We analyze how SND can defend against query-based black-box attacks and demonstrate its effectiveness against eight different state-of-the art at- tacks with CIFAR-10 and ImageNet datasets. Even with strong defense ability, SND almost maintains the original clean accuracy and computational speed. SND is readily applicable to pre-trained models by adding only one line code at the inference stage, so hope that it will be used baseline of defense against query-based black-box attacks in the future.
INTRODUCTION
Although deep neural networks perform well in various areas, it is now well-known that small and malicious input perturbation can cause them to malfunction (Biggio et al.) 2013 Szegedy et al. 2013). This vulnerability of Al models to adversarial examples hinders their deployment, especially in safety-critical areas. In a white-box setting, where the target model's parameters can be accessed, strong adversarial attacks such Projected Gradient Descent (PGD) (Madry et al.) 2018) gen- erate adversarial examples using the internal information. However, recent studies have shown that adversarial examples can be generated even in black-box setting where the model's interior is hidden to ad adversaries.
These black-box attacks can be largely divided into transfer-based attacks and query-based attacks. Transfer-based attacks take advantage of transferability that adversarial examples generated from network can deceive other networks, Papernot et al. (2017) train a substitute model that mimics the behavior of the target model and show that the adversarial example created from it can successfully disturb different models. However, due differences in training methods and model architectures, the transferabilitty of adversarial examples can be significantly weakened, and thus, transfer-based attacks usually result lower success rates (Chen et al.] 2017). For this reason, most black-box attacks are based on queries, each of which measures the target model's output for input.
Query-based attacks create adversarial examples through an iterative process based on either local search with repetitive small input modifications optimization with estimated gradients of an ad- versary's 's loss with respect to input. However, requesting many queries in their process takes a lot of time and financial loss. Moreover, many similar query images be suspicious to system admin- istrators. For this reason, researchers have focused on reducing the number of queries required to make successful adversarial example.

---

Under review as a conference paper at ICLR 2021
Compared to the increasing number of studies query-basee attacks, the number of defenses against them is still very small (Bhambri et al., 2019). Also, existing defenses developed for white- box attacks may not be effective against svery-based black-box attacks. Dong et al. (2020) find that existing defenses such as ensemble adversarial training (Tramèr et al. 2018) do not effectively defend against decision-haan -aased attacks. Therefore, it is necessary to develop new defense strategies that respond appropriately to the query-based attacks.
To defend against query-based black-box attacks, we pay attention an implicit but important assumption of these attacks that the target model's output exactly corresponds to the query input. If some randomness introduced into the model to break this assumption, they can have tremendous difficulty in both gradient estimation and local search, which are the core of their attack process. This intuition is illustrated Fig. la
high
adversary's objective loss
cleen image
edversary's image
requested query point for estimating grads (perturbed by the adversary)
query point perturbed the model
estimated gradient from estimated gradient from
(a)
(b)
Figure 1: Illustrations of our intuitions. (a) Small noise can effectively disturb gradient estimation of query-based attacks which use finite difference (b) Compared large noise, small noise hardly affects predictions on clean images.
Among previous studies, Dong al. (2020) empirically find that randomization-ble based defenses more effective in defending against query-based black-box attacks than other types of defenses. However, existing randomization-basel defenses introduce much uncertainty into predictions, and thus, they also significantly degrade the accuracy of clean images (clean accuracy).
In this paper, however, we highlight that simply adding small Gaussian noise into image is enough to defeat various query-based attacks by breaking the above core assumption while almost maintain- ing clean accuracy. One may think that additive Gaussian noise cannot defend against most adver- sarial attacks unless we introduce large randomness. This idea is valid for white-box attacks (Gu & Rigazio (2014), but our experimental results show that small noise is surprisingly effective against query-based black-box attacks. Our second intuition regarding the minimization of clean accuracy loss can be seen Fig. [b] Dodge & Karam (017)) find that the classification accuracy decreases in proportion to the variance of Gaussian noise, but for sufficiently small variance, the accuracy drop is negligible. Considering that the robustness against additive Gaussian noise is positively cor- related to the distance to the decision boundary (Gilmer et al. 2019), the above observation implies that clean images have a relatively long distance the decision boundary.
We think an adversarial defense should have the following goals: (1) preventing malfunction of a model against various attacks, (2) minimizing the computational overhead, (3) maintaining the accuracy on clean images, and (4) easily applicable to existing models. The proposed defense against
qussy attacks meets all of the above objectives, and we name this simple yet effective defense Small Noise Defense (SND).
Our contributions can be listed as follows:
We highlight the effectiveness of adding small additive noise to input in defending against query-based black-box attacks. The proposed defense, SND, can be readily applied to pre- trained models by adding only one line of code in the Pytorch framework (Paszke et al.) (019) at the inference stage (x x + sigma * torch. rando, like (x) and almost maintains the performance of the model.
2

---

Under review as a conference paper at ICLR 2021
We analyze how SND can efficiently interfere with gradient estimation and local search, which are the core of query-based attacks.
We explain the difficulty of evading SND. We devise adaptive attack against SND and explain its limitations in terms of query-efficiency.
We have shown that the proposed method can effectively defend against eight different state- of-the-art query-based black-box attacks with the CIFAR-I0 and ImageNet datasets. Specifi- cally, we experimented with four decision-based and three score-based attacks in order show strong defense ability against various attacks, including local search- based and optimization- based methods.
2 BACKGROUND
Adversarial setting, In this paper, we will deal with adversarial attacks on the image classification task. Suppose that neural network f (ax) classifies image among total N classes and returns class-wise probability vector y = [y1.... yN for æ. For notational convenience, we also denote the probability of 7th class (i.e., y.) (z),, and the top-1 class index as h(x:) arg max Vs. where
='= (I, N.
In black-box threat model, an adversary has a clean image 2o whose class index is on and wants to generate an adversarial example â = + to fool a target model f. In the following, we denote the adversarial example at tt step in an iterative attack algorithm as Tt. The adversary should generate an adversarial example within a perturbation norm budget and query budget Q. If we let 9 be the number of queries used to make 8, then we can write the adversary's 's objective as follows:
min E(ro + 8), subject to || pp S and g S Q
(1)
where (())) f()).. maxeffo (() for untargeted attacks and (()) maxcza f()t). f(t) for targeted attacks with target class index Unless otherwise noted, in this paper. we use p = and focus untargeted attacks because it is more challenging for defenders. Besides, we assume that each pixel value is normalized into [0, 1].
Taxonomy of query-based black-box attacks. Query-based attacks can be largely divided into score-based and decision-based according to the available type of the output of the target model (class-wise probabilities for score-based attacks and the top-1 class index for decision-based at- tacks). On the other hand, query-based attacks can be categorized into optimization-base and local search-based attacks. Optimization- based methods optimize an adversary's objective loss with estimated gradients of the loss with respect to ă. In contrast, local search- based attacks repeat- edly update an image according to how the model's output changes after adding a small perturbation.
attacks
In the following, we briefly introduce various query-based attacks used in this paper.
Bandit optimization with priors (Bandit-TD). Ilyas al. 2018) observe that the image gradients in successive steps of an iterative attack have strong correlation. In addition, they find that the gradients of surrounding pixels also have strong correlation. Bandit-TD exploits this information as priors for efficient gradient estimation.
Simple Black-box Attack (SimBA & SimBA-DCT), For each iteration, SimBA (Guo et al. 2019a) samples vector from pre-defined set Q and modify the current image i, with e; q and + and updates the image the direction of decreasing yee. Inspired by the observation that low- frequency components make major contribution to misclassification (Guo et al. 2018). SimBA- DCT exploits DCT basis in low- frequency components for query-efficiency.
Boundary Attack (BA). BA (Brendel et al. 2018) updates á; on the decision-boundant that the perturbation norm gradually decreases via random walks while misclassification maintained.
Sign-OPT..heere et al. (2019a) treat a decision-based attack as a continuous optimization problem of the nearest distance to the decision boundary. They use the randomized gradient-fres frre method (Nesterov & Spokoiny] 2017) for estimating the gradient of the distance. Cheng al. (2019b) propose SIGN-OPT which uses the expectation of the sign of gradient with random directions to efficiently estimate the gradients without exhaustive binary searches.
3

---

Under review as a conference paper at ICLR 2021
Hop Skip Jump Attack [HSJA))Cheneet
Chen al. (2020) improve Boundary Attack with gradient estima- tion. For each iteration HSJA, it finds an image on the boundary with a binary search algorithm, and estimates the gradients, and calculates the step-size towards the decision boundary.
GeoDA, Rahmati et al. (2020) propose a scometry-basee attack that exploits a geometric prior that the decision boundary of the neural network has small curvature average near data samples. linearizing the decision boundary in the vicinity of samples, it can efficiently estimate the normal vector of the boundary, which helps reduce the number of required queries for the adversarial attack.
2.1 ADVERSARIAL DEFENSES
As Dong et al. (2020) observe that randomization is important for effective defense against query- based attacks, we focus on randomization- defenses among various defense methods. In what follows, briefly explain three different randomization-base defenses along with PGD- adversarial training.
Random Self-Ensemble (RSE). RSE (Liu et al. 2018 adds Gaussian noise with Fmner 0.1 the input of each convolutional layer, except for the first convolutional layer where init = 0.2 is used. To stabilize the performance, they use ensemble of multiple predictions for each image.
Parametric Noise Injection (PNT). He et al. (2019) propose a method to increase the robustness of neural networks by adding trainable Gaussian noise to activation weight of each layer. They introduce learnable scale factors of noise and allow them to be learned with adversarial training.
Random Resizing and Padding (R&P). Xie et (2018) propose a random input transform-based method. In front of network inference, it applies random resizing and random padding to its input sequentially, making adversaries obtain noisy gradients. It can be easily applied pre-trained model, but it increases total computational time due to the enlarged input image.
PGD-Adversarial Training (PGD-AT). Madry et al. (2018) propose PGD-adversarial training which trains model with adversarial examples generated by PGD attack. Unlike other defenses that are ineffective against adaptive attacks, well known that PGD-AT provides excellent de- fense against variety of white-box attacks.
ANALYSIS
3.1
OUR APPROACH
To defend against query-based black-box attacks, we add Gaussian noise with a sufficiently small to the input as follows.
fz) = f(x + 1)), where n 0(0,021) and & 1.
(2)
For adversary, since the exact value of v is unknown, there can be multiple output values for any æ, so Sa is a random process. In what follows, we explain how this transform introduces tremendous difficulty in both gradient estimation and local search in query-based black-box attacks.
3.2 DEFENSE AGAINST OPTIMIZATION-
ATTACKS
In this subsection, we will explain how small Gaussian input noise can disturb the gradient estima- tion in optimization-basee attacks. We first look at defense against score-based attacks and then deal with decision- based attacks.
The core of optimization- based attacks is accurate estimation of Vd(x), which needs to be ap- proximated with finite difference because of the black-box setting. For instance, the gradient can be estimated as ğ by Random Gradient-Free method (Nesterov & Spokoiny| 2017) as follows.
B
gi where g; = f(0; + (u) E(wx)) u and u N(0, o
g = B
(3)
B
1=0
4

---

Under review as a conference paper at ICLR 2021
Conceptually, by introducing small Gaussian noise into input, g can differ greatly from the true gradient Vl as shown in Fig. I
To illustrate it more formally, let us represent n by replacing it with n)x) to clarify n depends on both time and æ. Suppose fi(a)) is a sample function of the random process Fnn(x) (x) at some time. Then, this function is very noisy because of m(x)) with regard to æ. We also assume that k" derived from m(a) then unless Var(f"(x + u)] is near zero, f" discontinuous and non- differentible and thus, V does not exist. Therefore, the gradient estimation using finite differences does not converge to the target gradient V..
In decision-basee attacks, 2, is likely to be in the vicinity of the decision boundary. Therefore even the small noise can move tt, across the boundary so that the output is changed. The estimated gradient through frequently erroneous predictions hinders the generation of adversarial examples. We illustrate the above defense mechanism in Appendix [ On the other hand, the error of the binary search algorithm, which is widely used to calculate the distance to the decision boundary, can amplified due to 17. Therefore, algorithms such as HSJA, which assume that is near the decision boundary, are likely to work incorrectly.
3.3 DEFENSE AGAINST LOCAL SEARCH-BASED ATTACKS
Almost all local search-bace based attacks do not take into account the uncertainty of the model output. Suppose an adversary recognizes that the attack objective loss decreases for x + where is perturbation, and updates æ as + T. However, since the actual evaluated input of f is x + T + where 77 ((0, o² I), the attack objective loss might be rather increases at the originally intended input x т. This prediction error makes the attack algorithm stuck in the iterative process and prevents generating adversarial examples.
3.4 DIFFICULTY OF EVADING SMALL NOISE DEFENSE
Although the input of the function, x + 17. is Gaussian random process, but f(x: 17), the result of the nonlinear function f. is no longer Gaussian random process. This makes it very difficult for query-based attacks to bypass SND. One can approximate f(x) as (x,ff((x)) by taking the expectation through multiple queries using the fact that B(n) = O. However, this attempt requires many queries for each iteration and greatly diminishes query efficiency.
addition, even if large amount of queries are used, En, (x=] may be different from f(x) because of the nonlinearity of the deep neural networks. With a simple example, we explain how the expectation value differs from the actual value when Gaussian noise is added to input of nonlinear functions.
Let F(x) = x" x where F Rd + R. and (x) = F(x + n) where n7 ~ NN(0, 02. Suppose we want to estimate F(O) with ()). Then, [[ (0) = B[(O - n0" + 7)] = E[n n = do². Therefore, E (0) d + F(O) and if d is very large (e.g., for an image of size 224x224x3, d=150, 528), then the estimation error would be high.
Additionally, in the case of simple network consisting of an affine layer and ReLU activation, F(x) Rel. + b) and F IR + TR. Since F(x) is max(0, Wx + b) and (x)) is Wx + b - Φ Wath) + [W [WWoo(- W the estimation error would exist (The detailed proof is in Appendix P. From the proof the simple network, we can expect that the average of the output may have an error with the actual output even in a deep neural network.
4 EXPERIMENTS AND DISCUSSION
4.1 EXPERIMENTAL SETTINGS
In this section, we evaluate the defense ability of SND against eight different query-based black-box attacks: BA, OPgT-OPT, HSJA, GeoDA. SimBA, SimBA-DCT, Bandit-TD, Subspace Attack along with other defense methods: PNI. RSE, R&P, PGD-AT. We use the CIFAR-10 (Krizhevsky et al. 2009) and ImageNet (Deng al. 2009 datasets for our experiments and following previous studies (Chen et al. 2020 Guo et al. 2019a He et al., 2019). we use ResNet-20 for CIFAR-10 and ResNet- 50 (He et al. 2016) for ImageNet for target networks. Following Brendel et al. 2018). randomly
5

---

Under review as a conference paper at ICLR 2021
sampled 1,000 and 250 correctly classified images from the CIFAR-10 test set and the ImageNet Validation set for evaluation. We describe a detailed experimental setting in Appendix B
For evaluation metrics, first define successfully attacked image attack can find an adversarial image within the perturbation budget e and query budget Q. With this definition,
image from which an
attack success rate, which is the percentage of the number of successfully attacked images over the total number evaluated images. Note that since we evaluate defense performance, a lower attack success rate is better. We use 1.0 for the CIFAR-IO dataset, and e 5.0 for the ImageNet dataset. In addition, we denote the gth query image as øz, Note that d and 2e can be different.
4.2 EXPERIMENTAL RESULTS
Evaluation of clean accuracy. We first evaluate the clean accuracy of models with defenses on the original test split (10k images) of the CIFAR-10 and validation split (50k images) of the ImageNet dataset. As shown in Table [ SND hardly reduces the clean accuracy compared to other methods. The accuracy drop caused by SND is not significant at a S 0.01 and becomes large at a 0.02, which implies that sufficiently small a is important for maintaining clean accuracy.
Table 1:
Comparison of clean accuracy. For randomization- methods, we denote mean and standard deviation of clean accuracy in repetitive experiments with different random seeds.
ResNet-20 on CIFAR-10
ResNet-50 on ImageNet
Defense
Clean Accuracy (%)
Defense
Clean Accuracy (%)
Baseline
91.34
Baseline
76.13
SND (a 0..00)
91.33 + 0.02
SND 0.001)
76.10 = 0.02
SND (a -0.01)
90.57 + 0.09
SND (a 0.01)
75.47 + 0.03
SND (a 0..2))
87.56 + 0.18
SND (a 0.02)
73.91 + 0.02
RSE
83.40 + 0.15
PGD-AT
57.9
PNI
85.15 + 0.18
R&P
74.26 + 0.07
Evaluation on the CIFAR-10 dataset. We performed four different decision-basee -based attacks against models with defenses, and Table Sable@ shows the evaluated attack success rates. SND shows competitive defense ability despite having more than 5% higher clean accuracy compared to other defenses. Moreover, due significant performance drop, RSE and PNI cannot be applied to the models for large-scale image classification with the ImageNet dataset.
Table Evaluation of attack success rates (%) on the CIFAR-10 dataset.
Attack method
BA
Sign-OPT
HSJA
GeoDA
of queries
2k
5k
10k
2k
5k
10k
2k
5k
10k
2k
5k
10k
Baseline
36.2
69.5
84.6
59.1
88.9
91.2
86.4
89.2
89.2
64.7
71.3
76.5
SND (a = 0.01)
14.0
17.8
20.0
21.7
22.3
22.8
16.5
19.9
22.7
12.0
12.1
12.4
SND (a = 0.001)
33.0
53.0
61.3
20.6
22.2
23.4
48.1
67.6
81.9
11.7
11.8
12.2
RSE
18.2
19.0
19.8
18.7
18.7
18.7
19.9
22.2
23.4
19.7
19.9
20.5
PNI
15.1
15.4
15.8
18.1
18.1
18.1
17.4
19.2
20.6
19.9
20.2
20.4
Evaluation on the ImageNet dataset. We performed six different query-based attacks against mod- els with defenses, and Table shows the evaluated attack success rates. When the query budget Q is 20k, the average of the attack success rates over the attacks against the baseline is 87.9%, whereas SND with a = 0.01 significantly reduces it to 10.0%. SND with a = 0.001 also significantly re- duces the average attack success rate to 29.5%, which is comparable the second-best method, R&P (22.5%). We also calculate the average & norm of adversarial perturbations ||æ:o Φ ||2 at the pre- defined query budget Q to show whether the perturbation norm diverges or not. an attack stops in the middle without requesting all Q queries, we use the last query image instead. In decision-based attacks, it can be seen that randomization- based defenses, SND and R&P, significantly increase the perturbation norm as q increases. In SimBA and SimBA-DCT, the perturbation norm is minimal in SND and R&P, which implies that the attacks have significant difficulty in finding a perturbation which decreases Veo-

---

Under review as a conference paper at ICLR 2021
Table Evaluation of attack success rates against defenses on the ImageNet dataset. We denote the average & norm of perturbations in the parenthesis.
Attack type
Decision- based Attack
Attack method
Sign-OPT
HSJA
GeoDA
of queries
5k
10k
20k
5k
10k
20k
5k
10k
20k
Baseline
36.4%
62.4%
88.0%
64.0%
88.4%
99.6%
50.0%
62.8%
72.0%
(9.71)
(4.72)
(2.38)
(4.43)
(2.34)
(1.28)
(6.38)
(5.04)
(4.12)
SND (o 0.011
6.8%
6.8%
7.2%
6.4%
7.6%
8.4%
7.2%
7.6%
7.6%
(41.37)
(70.27)
(87.94)
(31.03)
(26.81)
(22.93)
(33.48)
(33.14)
(32.61)
SND (o=0.001)
6.8%
7.6%
8.0%
13.6%
20.4%
32.4%
8.4%
8.8%
9.2%
(35.15)
(54.33)
(88.29)
(11.60)
(8.86)
(6.75)
(27.44)
(25,88)
(24.41)
AT
28.8%
30.0%
32.4%
30.4%
33.2%
36.0%
32.4%
34.0%
35.6%
(30.67)
(24.66)
(19.72)
(20.99)
(17.27)
(13.46)
(14.54)
(13.63)
(12.90)
R&P
13.2%
13.2%
13.2%
13.6%
15.2%
16.0%
14.4%
14.4%
15.2%
(51.19)
(82.14)
(85.78)
(33.01)
(31.00) (29.42)
(31.72)
(31.21) (30.45)
Attack type
Score-based Attack
Attack method
SimBA
SimBA-DCT
Bandit-TD
of queries
5k
10k
20k
5k
10k
20k
5k
10k
20k
Baseline
74.0%
74.4%
74.4%
94.8%
95.2%
95.2%
94.0%
97.2%
98.4%
(3.89)
(3.99)
(4.02)
(3.12)
(3.14)
(3.14)
(4.70)
(4.70)
(4.70)
SND 00-0.01)
8.4%
9.2%
10.0%
8.4%
8.8%
10.4%
15.2%
15.2%
16.4%
(0.52)
(0.55)
(0.57)
(0.56)
(0.58)
(0.60)
(4.74)
(4.74)
(4.74)
SND 0o=0.001)
27.2%
35.6%
50.4%
46.4%
60.4%
68.4%
7.2%
7.6%
8.4%
(1.84)
(2.14)
(2.43)
(2.22)
(2.44)
(2.58)
(4.83)
(4.83)
(4.83)
AT
27.6%
27.6%
27.6%
36.0%
36.0%
36.0%
38.8%
45.2%
52.8%
(5.46)
(7.55)
(10.17)
(5.36)
(6.21)
(6.62)
(3.54)
(3.54)
(3.54)
R&P
26.4%
27.6%
28.0%
27.2%
28.4%
29.2%
32.0%
33.2%
33.6%
(0.48)
(0.51)
(0.54)
(0.52)
(0.55)
(0.58)
(4.50)
(4.50)
(4.50)
Empirical evidence for assumptions of SND. To provide the supporting evidence for assumptions
of SND in score-base based attacks, we calculate å = v'Var(e(a + 7)] where ~ N(0, o² With 0.01 and 1,000 test images of the CIFAR- 10, we evaluate f(x + 77) for 100 iterations for each clean image and calculate the & averaged over all images. In our experiment, small but sufficient to make {(rr + 17) to non-differentiable about a.
0.04 which is
In dacision-based attacks, our assumption in Section 3.2 is that if & is near the decision boundary, sufficiently small noise can easily move the image across the boundary. We evaluate Piiie := P(h(æ) h(x + 7)) through experiments. We count the above mismatch case for all queries during the attack process. With o = 0.001, 0.01 and the CIFAR-10 test images, the average Pmis for all attacks is calculated as 0.22 and 0.25, respectively. In constrast, clean images, Pmis is obtained as 0.002 and 0.021 respectively. Therefore, the results shown in Table support our argument.
Evaluation of an adaptive attack against SND. As described in Section 3.4 we devise an adap- tive attack against SND that takes the expectation of predictions for repetitive T queries. In this experiment, we perform HSJA against SND with a = 0.01 on the CIFAR-10 dataset. Since HSJA decision-based attack, we regard the most predicted class in T queries as the expected class. We measure the attack success rate and Pmis according to the query budget, and the adaptive at- tack clearly shows a higher attack success rate than the baseline (T 1), as shown in Table 4 On the same query budget, however, the adaptive attack shows lower attack success rate (e.g.. 22.7% (T=1)> > 18.2% (T=5) at Q=10k and 29.3% (T=5) > 27.3% (7=10) at Q=50k). Therefore, the expectation-basee adaptive attack has limitations due the restricted query budget. Moreover, 7' increases, the Pinie does not decrease and this reinforces our argument in Section 3.4 We also apply the adaptive attack with 1-10 BA, SO, and GeoDA for comparison. The results shown in Table
Varying for each inference. So far, have used fixed for SND. Changing for each query may reduce clean accuracy while maintaining the defense ability. From this motivation, we multiply n with k which is randomly determined between O and I using the beta distribution with three
7

---

Under review as a conference paper at ICLR 2021
Table 4: Evalutation of SND with different T.
Table 5: Evaluation of P(h(2r)) + h(x + m)).
# of queries
2kx7'
5kx
10kx
Piii
Defense
SND
SND
HSJA (7==1)
16.5%
19.9%
22.7%
0.189
Attack
= 0.001) (o 0.01)
HSJA (7=5)
18.2%
23.8%
29.3%
0.321
BA
0.134
0.2277
HSJA (7=10)
21.0%
27.3%
34.9%
0.376
Sign-OPT
0.216
0.215
HSJA (T'=00)
25.2%
34.0%
46.6%
0.410
HSJA
0.255
0.189
# of queries
50k
100k
200k
GeoDA
0.314
0.391
HSJA
29.0%
30.1%
31.0%
0.120
None
0.002
0.021
Table 6: Evalutation of the adaptive attack against SND with T=10.
# of queries
2kxT
5kxT
10kxT
Pmss
BA (7=10)
20.3%
28.5%
32.9%
0.413
SO (7)=0)
20.5%
21.1%
21.9%
0.354
HSJA (T=10)
21.0%
27.3%
34.9%
0.376
GeoDA (7=10)
12.1%
12.2%
12.2%
0.413
different ways: (1) Uniformly random (the same as 0= (2) Sampling from a beta distribution with a=3-2 whose probability density function (PDF) is n-shaped. (3) Sampling from beta distribution with a=5=0.5 whose PDF U-shaped. We calculate clean accuracy and average & norm of perturbations for each method. Among the three ways, a=8-2 is better than the others, but SND with fixed 0-0.01 is better than SND with variable a, which implies large randomness in each query is crucial for a strong defense. Detailed results can be found in Appendix C
Defense against hybrid black-box attack. Recently proposed Subspace attack (Guo et al. 2019b) exploits transferability-bass
priors, gradients from local substitute models trained on small proxy dataset. We use pre-trained ResNet-18 and ResNet-34 (He et al.. 2016) as reference models for gradient priors. For this attack, we perform an attack based on for norm because the authors pro- vide parameter settings only for too norm. Detailed results are described in Appendix D but the result shows that SND alone cannot effectively defend against the hybrid attack with gradient priors. However, when SND is combined with PGD-AT, it effectively protects the model and decreases the attack success rate from 100% to 42.4% at Q=20k and o=0.01. To focus on the defensive abil- ity against gradient estimation, we recalculate the attack success rate without initially misclassified images. Then, the newly obtained attack success rate decreases from 100% to 16.4% at Q=20k. This result implies that SND be combined with other defenses against wransfer-based attacks to achieve strong defense ability against all types of black-box attacks.
5 RELATED WORK
History-based detection methods against query-based black-box attacks. To the best of our knowledge, studies that mainly target defending against query-based black-box attacks have not yet been published. However, history-based detection techniques for query-based attacks have been proposed recently (Chen et al. 2019 Li et al. 2020). Considering that adversary requires many queries of similar images for finding an adversarial example, they store information about past query
images detect the unusual behavior of query-basse based attacks.
Certified defense with additive Gaussian noise. Li et al. (2019) analyze the connection between robustness of models against additive Gaussian noise and adversarial perturbations. They derive the certified bounds on the norm bounded adversarial perturbation and they propose new training strategy to improve the certified robustness. Simlilarly, randomized smoothing (Cohen et al. 2019) creates a smoothed classifier that correctly classifies when Gaussian noise is added to the classifier's input. Cohen et al. (201)) prove that this smoothed classifier can have lg certified robustness for an input. Both SND and the above certified defenses add Gaussian noise to the input. However, the purpose of the addition of noise in the certified defenses is to induce the classifier to gain certified robustness. Whereas, SND adds noise disturb accurate measurement of the output defend
8

---

Under review as a conference paper at ICLR 2021
against query-based black-box attacks at the inference. In addition, the certified defenses much larger σ (2 0.25) than SND (0.01).
6 CONCLUSION
In this paper, we highlight that even small additive input noise can effectively neutralize query- based black-box attacks and name this approach Small Noise Defense (SND). We demonstrate its effectiveness against eight different query-based attacks with CIFAR-IO and ImageNet datasets. Our work suggests that query-based black-box attacks should consider the randomness of the target network well. SND is readily applicable to pre-trate pre-trained models by adding only one code line without any structural changes and fine-tuning. Due to its simplicity and effectiveness, we hope that SND will be used baseline of defense against query based black-box attacks in the future.
We can list various exemplary future work as follows: (1) Finding a better type of noise. Other types of noise, such salt-and-pepper image processing like random contrast, be used instead of Gaussian noise as long as it can disrupt query-based attacks while maintaining clean accuracy; (2) Devising effective black-box attacks against SND while maintaining query efficiency. The uncer- tainty of randomized predictions can be mitigated through expectation, but it requires many queries and hurts guery-efficiency; (3) Rigorous proof of the defense ability of small input noise. We ex- plain how SND can defend against query- busee attacks, but one can establish theorems like the lower bound of mean squared error of estimated gradient with small input noise.
REFERENCES
Siddhant Bhambri, Sumanyu Muku, Avinash Tulasi, and Arun Balaji Buduru. A survey of black-box adversarial attacks on computer vision models. arXiv, pp. a11i 2019.
Battista Biggio, Igino Corona, Davide Maiorca, B Nelson, N Smdic, P Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2013, volume 8190, pp. 387 487-402. Springer, 2013.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018
Jianbo Chen, Michael Jordan, and Martin J Wainwright. Hopskipjumpattack: query-efficient decision-based attack. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1277-1294. IEEE, 2020.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti-
mization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 15-26. 2017.
Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial at- tacks. arXiv preprint
2019.
Minhao Cheng, Thong Le, Pin-Yu Chen, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Query- efficient hard-label black-box attack: An optimization-baset approach. In International Confer- ence on Learning Representation (ICLR), 2019a.
Minhao Cheng, Simranjit Singh, Patrick H Chen, Pin- Yu Chen, Sijia Liu, and Cho-Jui Hsich. Sign- opt: A query-efficient hard-label adversarial attack. In International Conference on Learning Representations, 2019b.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: 1rXiv: 1902.02918, 2019.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.

---

Under review as a conference paper at ICLR 2021
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recogni- tion performance under visual distortions. In 2017 26th international conference on computer communication and networks (ICCCN), pp. 1-7. IEEE, 2017.
Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. Benchmark- ing adversarial robustness on image classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 321-331, 2020.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL
hrt.a//.........aaaaaaa.oaoooaar robusthess
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are natural consequence of test error in noise. In International Conference on Machine Learning, pp. 2280- 2289, 2019.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:14/2.5068, 2014.
Chuan Guo, Jared S Frank, and Kilian Q Weinberger. Low frequency adversarial perturbation. arXiv 2018.
preprint
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pp. 2484-2493, 2019a.
Yiwen Guo, Ziang Yan, and Changshui Zhang. Subspace attack: Exploiting promising subspaces for query-efficient black-box attacks. In Advances in Neural Information Processing Systems, pp- 3825-3834, 2019b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 588-597, 2019.
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black- adversarial attacks with bandits and priors. In International Conference Learning Representations, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. In Advances in Neural Information Processing Systems, pp. 9464-9474, 2019.
Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y Zhao. Black- light: Defending black-box adversarial attacks on deep neural networks. arXiv preprint arXiv:2006. 14042, 2020.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference Computer Vision (ECCV), pp. 369-385, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 1722)5277-566 2017.
Maria- Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wis- tuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018. URL https://arrs ptgg//df///07.00000000
10

---

Under review as a conference paper at ICLR 2021
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference computer and communications security, pp. 5006-119, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance Wallach, Larochelle, A. Beygelzimer, dAlché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
deep learning library.
Ali Rahmati, Seyed-Mohser Mlosavii-ezfooll Pascal Frossard, and Huaiyu Dai. Geoda: a geo- metric framework for black-box adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8446-8455, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference Learning Representations, 2018.
Cihang Xie, Jianyu Wang. Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.

---

Under review as a conference paper at ICLR 2021
A ILLUSTRATION OF DEFENSE AGAINST DECISION-BASED ATTACK WITH SMALL INPUT NOISE
clean image X actual query R, fiu
adversarial example
requested query point + fite for estimating grads
estimated gradient from estimeted gradient from
Figure 2: Illustration of how small noise can defend against decision-based attacks (left) An adver- sary wants to reach the optimal adversarial example from an initial misclassifed point. (middle) To find the direction of the next update, it evaluates ã; + (right) Small noise can disturb this gradient estimation.
B DETAILED EXPERIMENTAL SETTING
B.1 SETTINGS OF ATTACK METHODS
BA: We adopt BA provided by Adversarial Robustness Toolbox (ART) library (Nicolae et al. 2018 with default parameters.
Oign-OPT: We port the code provided by the authors into our framework without changing the special parameters of the attack.
We adopt HSJA provided by Adversarial Robustness Toolbox (ART) library with default parameters except for increasing the maximum number of iterations to 64 to follow the authors' c
HSJA:
GeoDA:
We port the code frovvdde by the authors into our framework without changing the special parameters of the attack. SimBA & SimBA-DCTT We port the code provided by the authors into our framework. Following the authors, we use freq_ dims=28, o order=strided, and stride=) for SimBA-DCT.
Bandit-TD: We port the codo provided by the authors into our framework with default parameters except for batch_si ze=1 and epsi 1on=4.9.
Subspace Attack:
We port the code provided by the authors into our framework with the original setting for foo norm untargeted attack for the ImageNet. We use pre-trained ResNet-18 and ResNet- 34 trained on the imagenetv2 val dataset as reference models that are provided by the authors.
12

---

Under review as a conference paper at ICLR 2021
B.2 SETTINGS OF DEFENSE METHODS
Baseline: We train ResNet-20 model on the CIFAR-10 dataset for 200 epochs and use this model for our experiments. For the ImageNet dataset, we use the pre-trained ResNet-50 model provided by the Pytorch library.
PNI: We use the pre-trained ResNet-20 model with PNI-W (channel-wise) provided by the authors trained on the CIFAR-I0 10 dataset.
PGD-AT: We use the adversarilly trained ResNet-50 model for lg norm with Etrain 3 provided from robustness library (Engstrom et al. 2019) with PGD on the ImageNet dataset for comparison.
RSE: We train RSE-based ResNet-22 20 with intt 0.2 and inner 0.1. Considering computational efficiency, we use 5 ensembles for each prediction of RSE.
R&P applies random resizing and random padding to its input sequentially. It first rescales the input image of size W H 3 with scale factor which is sampled from [8min 8maz], and places it in a random position within an empty image of size Smaz W Smaz Y x 3. Following the respectively.
R&P:
authors, set 8min and smax as 310 and 331
29g
C EVALUATION OF VARYING a FOR EACH INFERENCE
Table 7: Experimental results of varying a with the CIFAR-10 dataset. We evaluate mean and
standard deviation of clean accuracy in repetitive experiments on the original test dataset.
Clean Acc. (%)
E 12
Sign-OPT
HSJA
# of queries
2k
5k
10k
2k
5k
10k
0=0.001
91.33 = 0.02
0.055
20.6%
22.2%
23.4%
48.1%
67.6%
81.9%
0=0.01
90.57 + 0.09
0.550
21.7%
22.3%
22.8%
16.5%
19.9%
22.7%
0=0.01,0=8-1
91.04 + 0.12
0.276
21.6%
22.4%
22.9%
18.1%
23.9%
30.0%
0=0.01,a=6=2
91.15 = 0.04
0.275
20.3%
20.8%
21.7%
19.7%
23.5%
28.5%
0=0.01, a0..01,==8-0.5
91.06 = 0.06
0.275
20.9%
21.4%
22.4%
19.8%
26.3%
32.6%
0-0.2
87.56 + 0.18
1.098
26.4%
26.5%
26.6%
24.2%
26.7%
30.1%
0=0.2 a=
90.17 = 0.09
0.552
22.0%
22.0%
22.1%
19.6%
23.2%
25.4%
0= 0.02
90.44 = 0.08
0.550
21.6%
21.8%
22.1%
18.0%
22.1%
25.0%
0.02, 0.02, 3=5
89.99 + 0.24
0.549
22.5%
22.5%
23.3%
20.3%
24.4%
27.5%
D EVALUATION OF ATTACK SUCCESS RATES OF SUBSPACE ATTACK
Table 8: Evaluation of attack success rates of Subspace Attack against defenses on the ImageNet dataset. We also calculate the attack success rate without initially misclassified images and denote it in the parenthesis.
Attack method
Subspace Attack 10k
of queries
5k
20k
Baseline
99.6% (99.6%)
100.0% (100.0%)
100.0%
(100.0%)
SND (a 0.01)
61.2% (59.6%)
61.2%
(59.6%)
61.2% (59.6%)
SND (a 0.001)
64.4% (64.4%)
66.4% (66.4%)
68.4% (68.4%)
PGD-AT
71.6% (45.6%)
78.4% (52.4%)
81.6% (55.6%)
PGD-AT SND (a 0.01)
40.8% (14.8%)
42.0% (16.0%)
42.4% (16.4%)
PGD-AT SND (o = 0.001)
62.0% (36.0%)
62.4% (36.4%)
63.2% (37.2%)
R&P
73.2% (68.4%)
74.0% (69.2%)
74.0% (69.2%)
13

---

Under review as a conference paper at ICLR 2021
E THE PROOF OF THE EXPECTATION ERROR ON A SIMPLE NETWORK
Let F(z) = Re where F: : - R and (z) = ma x) and F Fx(x) = F(x + 1), where 1) ~ N (0, (0, Suppose we want to estimate F(x) with (x)]. Let W(r + 11) + b be Y,
then Y can be represented with My = Wa and oz = W2 as:
Y N(y oz)-
4))
Then, (2) is max(0, Y) and E (x)) tation.
E mmmax( Y)] can obtained by the law of total expec-
EI E (0, Y)]
(5)
= -| OJPr(Y > 0) + OPr((Y 0)
Using the truncated normal distribution, we recall the fact as follows:
E| a] = My og <((a My)/ow)
(6)
((aa (1aaNy)/op)
where d(x)
exp( 2 2 and is the cumulative distribution function of the standard normal can be represented as:
.2T
distribution. Since Pr(Y 0)) (( E[F, (x)]
(x)) 1y (1- - Φ Φ ))) ogpl
My)
0g
Wx + b ) + +Wool
Wx + 6 W
W + b Φ
| o
14


Reviews:
[{"review_id": "xjR44aFuLSu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adversarial defense method against black-box attack by adding random noise on adversarial examples. The method is simple and empirical results verify that the proposed defense can be used to decrease the attack success rate of both optimization-based and local search-based attacks. \n\nMy main concern of this paper is the simply use of randomness for adversarial defense, rather than directly improving the robustness of the target model itself (e.g. adversarial training). In the discussion about the Obfuscated Gradient [1], the authors criticized the randomness-based defense method: “Stochastic Gradients are caused by randomized defenses, where either the network itself is randomized or the input is randomized before being fed to the classifier.” Although this paper is aimed at defending against black-box attacks, it is still essentially a defense by adding random noise to the input data. In addition, the proposed method is not useful to defense against white-box attacks and transfer-based attacks, since basically it does not change the predictive model.\n\nAnother problem is that while the defense mechanism in this paper is simple, there are equally simple ways to circumvent it. An example is that when the randomness of model output is found by the adversary based on Boundary Attack, the attacker only needs to change the judgment condition of successful attack to “most of the attacks are successful in the same input for multiple queries” to avoid the decrease of attack success rate.\n\n[1] https://arxiv.org/abs/1802.00420\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "u4TPTjkImL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a novel method for defending against\nadversarial attacks on deep neural networks in a black-box setting. The idea is\nsimple as it is effective: to simply add some small Gaussian noise to the input\nprior to passing it through the model. The authors make some heuristic arguments\nfor why this might be effective, including the difficulty of estimating\ngradients (for gradient-based attacks) and robustness against local-search based\nattacks that do not account for variability in the model output. They also\npresent a set of experiments demonstrating the superiority of their defense\nagainst a variety of common attack techniques, and contrast the performance of\nSmall Noise Defense (SND) against other common defense techniques.  The\nexperiments demonstrate that SND maintains a higher clean accuracy (i.e.,\naccuracy on unperturbed, non-adversarial images) compared to other defense\nmethods, while also staying robust to a variety of attack techniques.\n\nStrengths of the paper:\n  + The performance of the approach, given its simplicity (as the authors\n    point out, it only requires one line of code!), is striking and makes\n    SND an attractive option. It will likely be widely adopted given its ease\n    of implementation, in situations where protection against adversarial\n    attacks is critical.\n  + The experimental evaluation was rigorous and supports the paper's main\n    claims.\n  + The paper is well organized, well written and easy to follow.\n\nAreas for improvement/questions for the authors:\n  - The theoretical analysis of the SND model, given its simplicity, is somewhat\n    lacking. The authors themselves acknowledge this in their conclusion, and\n    cite this as a promising future avenue to explore. Currently, the paper's\n    arguments are mostly empirical.\n  - Given the similarity to Random Smoothing (Sec. 5), I was curious as to why\n    that method was not included in the experimental evaluation. Is this because\n    RS uses large variance noise, and is thus unlikely to maintain a sufficient\n    clean accuracy? Some clarification here would be welcome.\n\nOn balance, I think this paper presents some interesting results, and it's rare\nto see such simple ideas have such startling payoffs. That alone makes me\npositive about this paper. While the theoretical work is somewhat light, I don't\nsee that as fatal -- I think this paper as is opens up some interesting avenues\nfor future work, that could strengthen this contribution even further. I thus\nvote for ACCEPTANCE.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "sJaQTN83Duc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#Summary\n\nThis paper introduces a simple but effective method by adding a small Gaussian noise to the input, and shows it can neutralize query-based black-box attacks and achieve strong results on both clean accuracy and attack success rates w.r.t. several SOTA attack/defense methods.\n\n\n#Pros\n- The proposed approach is very simple and seems effective across several query-based black-box attacks. The authors have evaluated their SND defense against both decision-based attacks (Sign-OPT, HSJA, GeoDA) and score-based attacks (SimBA, SimBA-DCT, Bandit-TD), and the results are pretty strong in terms of  lowest attack success rates under a fixed query budget, compared with four defense methods (PGD-AT, R&P, RSE, PNI), over both CIFAR and ImageNet.\n- The proposed approach achieves good clean accuracy as well, which seems to be another plus compared to existing defenses that achieve strong defense ability but at the cost of clean accuracy.\n\n#Cons\n- I think a bit more detailed analysis of how this simple approach works would further strengthen this paper. In Section 3, the authors have provided some example analysis but it would be better to see a more in-depth analysis, e.g., how this method obfuscates the expectation for more general functions in section 3.4; similarly, for section 3.3., the attack objective loss \"might increase\" but there is no further study provided.\n\n- The following paper adopts a similar idea via additive Gaussian noise to the input for certified adversarial robustness, and should be cited and discussed as well: [1] Bai Li, Changyou Chen, Wenlin Wang, Lawrence Carin. Certified Adversarial Robustness with Additive Noise. NeurIPS 2019.\n\n- As the authors have mentioned, one natural way to attack random-noise based defense is through adaptively querying the model and taking the expectation of T queries, at the cost of increased query budget. In experiments, the authors performed a study of HSJA against SND. Could the authors show more study on this, like how is the query efficiency from adaptive attacks based on other attack methods? Also, since $\\sigma$ is fixed, after the first round of queries around a fixed $x$, is there a way to infer what $\\sigma$ is and further improves query efficiency in later rounds?\n\n- I think it would also be interesting to further see how this defense works under transfer-based black-box attacks. Does this defense work in general, or only for query-based black-box attacks?\n\n\n#Overall recommendation\n\nOverall I think the results are strong compared to the baselines and the method is simple and well-explained. The experiments are thorough over various SOTA attacks and defenses, on both CIFAR and ImageNet datasets.  I would vote for accepting the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "a9g_JEX3bfB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new defense against adversarial examples created through query based attacks. The defense is based on adding a small amount of random noise. The authors provide some theoretical arguments and experimental evaluation of their approach.\n\nOverall evaluation\nWhile the presented idea might be promising neither the theoretical nor the experimental evaluation are sufficient to validate this claim. The research on adversarial robustness is very active and many attacks and defenses have been proposed. To ensure good quality of research, proposals have been made on how to correctly evaluate a proposed defense (see below). The paper falls short of these standards. The main shortcoming is the lack of a robust evaluation against an adaptive attack (i.e. an attacker aware of the defense). The majority of the analysis in the paper is restricted to evaluating  the defense against existing attacks, robustness against an attacker that does not take the defense into account is insufficient. \nThe paper provides a short analysis of an adaptive attack, however, this new proposed attack is just a slight variation of an existing attack. This type of defense against a \"patched\" attack is still insufficient. Especially because from the paper it is unclear why this specific attack was chosen, if it is optimal and if it addresses the shortcomings of the original attack. One clear oversight is that the objective of the attacker changes, as it is not looking for a point that gets misclassified, but for a point that gets misclassified even under noise. \nThe shortcoming of the analysis goes beyond the lack described above and I recommend the authors to go through the given guidelines.\n\nReferences\nCarlini, Nicholas, et al. \"On evaluating adversarial robustness.\" arXiv preprint arXiv:1902.06705 (2019).\nAthalye, Anish, Nicholas Carlini, and David Wagner. \"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.\" arXiv preprint arXiv:1802.00420 (2018).\n\nConcrete comments about the unclarity\n\"the above observation implies that clean images have a relatively long distance to the decision boundary\" -- the existence and prevalence of adversarial examples implies exactly the opposite of this statement, if this would be true there would not be any adversarial examples. \n\nSection 3.2 is very unclear. eta which is defined in Section 3.1. a independent of $x$ is suddenly dependent on $x$. The sentence \"this function is very noisy\" is just a claim. It is unclear why the random function becomes discontinuous for large Var. \n\nSection 3.3 is just saying that existing attacks don't take into account the newly proposed defense.\n\nThe analysis in Section 3.4 is for a quadratic function. Most current CNN architectures don't have any quadratic component and in fact the Resnet architecture used in the experiments is a piecewise linear function with a final softmax layer. I fail to see the relevance of the presented argument.\n\n**After Rebuttal**\n\nI thank the authors for their detailed remarks and clarification. While it is encouraging to see that the authors have offered some clarifications and assurances regarding the validity of their approach, my main concern is whether the edit distance between the current work and the proposed modifications is just too high. It appears to me that there simply is too much that the authors need to modify in order to obtain an acceptable manuscript (with the other reviewers' concerns and suggestions as well). I believe that the paper can be significantly improved if the authors incorporate the comments from the current round of reviewing. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a defense against black-box adversarial example attacks based on adding small Gaussian noise to the inputs. Its evaluation is carried out empirically using CIFAR-10 and ImageNet datasets.

Despite a somewhat complete experimental evaluation (on two datasets) the lack of theoretical justification strongly affects the significance of the proposed method. It can be clearly seen from the experimental results that the proposed level of noise is a trade-off between clean accuracy and attack effectiveness. However, this tradeoff neither implies a substantial degree of security (the attack success rate is roughly halved but this does imply robustness against attacks since the initial success rate is rather high) nor is  the impact to clean accuracy negligible. Furthermore, the robustness of the proposed method against an attack which is aware of such defense (similar to the Kerckhoff's principle in cryptography) is not evaluated. The authors mentioned several directions for addressing this issue in their response but implementation of such improvements is impossible within the level of revisions acceptable in a post-review process.

A major revision of the paper taking into account the feedback provided by the current reviews would certainly improve its acceptance chances.  
Decision: Reject