Understanding the Robustness of Randomized Feature Defense Against Query-Based Adversarial Attacks
Authors: Nguyen Hung-Quang
Yingjie Lao
Tung Pham
Kok-Seng Wong
Khoa D Doan
Venue: ICLR 2024 poster (2024)
Keywords: adversarial attacks
adversarial defense
black-box attacks
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We analyze and propose a randomized defense on latent space to query-based attacks.

Abstract: Recent works have shown that deep neural networks are vulnerable to adversarial examples that find samples close to the original image but can make the model misclassify. Even with access only to the model's output, an attacker can employ black-box attacks to generate such adversarial examples. In this work, we propose a simple and lightweight defense against black-box attacks by adding random noise to hidden features at intermediate layers of the model at inference time. Our theoretical analysis confirms that this method effectively enhances the model's resilience against both score-based and decision-based black-box attacks. Importantly, our defense does not necessitate adversarial training and has minimal impact on accuracy, rendering it applicable to any pre-trained model. Our analysis also reveals the significance of selectively adding noise to different parts of the model based on the gradient of the adversarial objective function, which can be varied during the attack. We demonstrate the robustness of our defense against multiple black-box attacks through extensive empirical experiments involving diverse models with various architectures.

Published as a conference paper at ICLR 2024
UNDERSTANDING THE ROBUSTNESS OF RANDOMIZED FEATURE DEFENSE AGAINST CUSRY-BASED ADVER- SARIAL ATTACKS
Quang H. Nguyen', Yingjie Lao", Tung Pham', Kok-Seng Wong', Khoa Doan
Coolleee of Engineering and Computer Science, VinUniversity, Vietnam
TTufts University
VinAA Research
quang. www.nnh@ivnnt edu. vn, yingjie www.tults.eeeeeeoooooo.... v. tunnphh@vvnaii wong. ks @siinnnni....... edu. vn, khoa. dddvinuniinnn.m. edu.
ABSTRACT
Recent works have shown that deep neural networks are vulnerable adversarial examples that find samples close to the original image but can make the model misclassify. Even with access only to the model's output, an attacker can em- ploy black-box attacks to generate such adversarial examples. In this work, we propose a simple and lightweight defense against black-box attacks by adding random noise to hidden features at intermediate layers of the model at infer- ence time. Our theoretical analysis confirms that this method effectively enhances the model's resilience against both score-basee and decision-bai based black-bor box at- tacks. Importantly, our defense does not necessitate adversarial training and has minimal impact accuracy, rendering it applicable Our analysis also reveals the significance of selectively adding noise to different parts of the model based on the gradient of the adversarial objective function, which can be varied during the attack. We demonstrate the robustness of our defense against multiple black-box attacks through extensive empirical experi- ments involving diverse models with various architectures. Code is available at
any pre-trained model.
com mail www.erceeeeeeeeeeeee wwweeearcrrraddiiiic. zed wtw.://anmmmiiiddeeeeeeeeeeeee
INTRODUCTION
Modern deep neural networks have demonstrated remarkable performance in various complex tasks, including image classification and face recognition, among others. However, prior works have pointed out that deep learning models are sensitive to small changes in the input and can be fooled by carefully chosen and imperceptible perturbations Szegedy al. (2014); Goodfellow et al. (2015); Papernot et al. ; Madry et al. (2018). These adversarial attacks can be generally classified into white-box and black-box attacks. In white-box setting, strong attacks such Projected Gra- dient Descent (PGD) Madry et (2018) can generate effective adversarial examples by levering the information inside the model. However, in practical scenarios such machine learning as service (MLaas), the well-trained models and the training datasets are often inaccessible to the users, espe- cially in the of large models. Hence, query-based black-box attacks become the primary threats in most real-world upplications, where the adversary is assumed to have no knowledge of the model architecture and parameters.
This paper proposes lightweight, plug-and-play defensive method that can significantly decrease the success rate query-based black-box attacks, including both score-based and decision- based attacks Illya et al. (2018 2019); (Andriushchenko et al. (2000); Guo et al.] (2019): Al-Dujaili & O' O'Reilly 22000) Liu et al. (2019); ; Chen & Gu (2020); Chen et al. (2020b); Rahmati et al. (2020). Adversarial examples generated through query-based attacks involve iterative procedures that rely on either local search techniques involving small incremental modifications to the imput or optimiza- tion methods leveraging estimated gradients of the adversary's 's loss concerning the input. However, the process of requesting numerous queries is time-consuming and sometimes may raise suspicions with the presence of multiple similar queries. Hence, the objective of defense perplex the adver-

---

Published as a conference paper at ICLR 2024
sary when attempting to generate adversarial examples. Our proposed method accomplishes this by introducing noise into the feature space. While Qin et al. al.] (202I) study the robustness of randomized input, this paper provides both theoretical analysis and empirical evidence demonstrate improved robustness of randomized features. Our analysis also highlights the importance of strategically intro- ducing noise to specific components of the model based on the gradient of the adversarial objective function, which can be dynamically adjusted throughout the attack process.
Our contributions can be summarized as follows:
We investigate the impact of randomized perturbations in the feature space and its connec- tion to the robustness of the model to black-box attacks.
We design a simple yet effective and lightweight defense strategy that hampers the at- tacker's ability approximate the direction toward adversarial samples. As result, the success rate of the attacks is significantly reduced.
We extensively evaluate our approach through experiments on both score-based and decision-basee attacks. The results validate our analysis and demonstrate that our method enhances the robustness of the randomized model against query-based attacks.
RELATED WORKS
2.1 ADVERSARIAL ATTACKS
Extensive research has been conducted white-box attacks, focusing on the generation of adversar- ial examples when the attacker possesses complete access to the target model. Over the years, vari- ous notable methods have emerged representative approaches in this field, including fast gradient sign method (FGSM) Goodfellow et al. 2015). Jacobian-bon based saliency Map Attack (JSMA) Paper- not al. (0166)). and PGD Madry et (2018).
In contrast to white-box attacks, the black-box scenario assumes that the attacker lacks access to the target model, making it more challenging situation. However, this also more realistic setting in real-world applications where the adversary would not have access to the model parameters. One approach in black-box attacks involves utilizing white-box techniques on substitute models to create adversarial examples, which can subsequently be applied to black-box target models Papernot et al. (2077). However, the effectiveness of transfer-based attacks can vary significantly due to several practical factors, such as the initial training conditions, model hyperparameters, and constraints involved in generating adversarial samples Chen et al. (2017). This paper focuses on the defense against query-based attacks instead.
2.2 QUERY-BASH BASED BLACK-BOL ATTACKS
Query-based attacks can be largely divided into score-bace based attacks and decision-based attacks, based on the accessible model output information. Score- based attacks leverage the output proba- bility or logit of the targeted model, allowing the attacker to manipulate the scores associated with different classes. On the other hand, decision-based queries provide the attacker with hard labels, restricting the access to only the final predictions without any probability or confidence values.
We list the query-based attacks used in this paper below:
Natural Evolutionary Strategies (NES) Ilyas et al. (2018) is one of the first query-based attacks that use natural evolutional strategies to estimate the gradient of the model with respect to an image x. By exploring the queries surrounding x, NES effectively gauges the model's gradient, enabling to probe and gain insights into the model's behavior.
SignHunt AI-Dujiili & O'Reilly (020) is another score-based attack, which flips the sign of the perturbation based on the sign of the estimated gradient to improve the query efficiency.
Square attack Andriushchenko et al. (2020) is a type of score-basee attack that differs from gra- dient approximation techniques. Instead. employs random search update square -haped regions located at random positions within the images. This approach avoids relying on gradient information and introduces localized square modification the image.
2

---

Published as a conference paper at ICLR 2024
RayS Chen & Gu (2020) is decision-basel attack that solves discrete problem find the di- rection with the smallest distance to the decision boundary while using a fast check step to avoid unnecessary searches.
SignFlip Chen et al. (2020b) is an e decision based attack that alternately projects the perturbation to smaller 8 ball and flips the sign of some randomly selected entries in the perturbation.
2.3 DEFENSIVE METHODS AGAINST QUERY-BASE BASED ATTACKS
In the recent literature, several defensive solutions have been proposed to counter adversarial ex- amples. One such solution involves the detection of malicious queries by comparing them with previously observed normal queries Chen et al. (2020a): Li et al. (022); Pang et al. (2020). This approach aims to identify anomalous patterns queries and flag them potential adversarial exam- ples. Additionally, adversarial training has also been utilized to enhance the model's 's robustness Co- hen et al. (2019): Wang et al. (2020); Sinha et al. (2017): ZZang et al.] (2020). Adversarial training involves training the model on both regular and adversarial examples to improve its ability to with- stand adversarial attacks. However, it is computationally expensive, especially when dealing with large and complex datasets. In some cases, adversarial training may also inadvertently harm the model's ' overall performance.
In contrast, this paper focuses approaches that involve incorporating noise randomness into the model, thereby providing the adversary with distorted information. The underlying intuition behind these defense mechanisms is to deceive the attacker by introducing perturbations in the model's prediction process. By altering certain signals, the defenses aim to mislead the attacker and divert them from their intended direction. To achieve this, various techniques are employed modify the input data or manipulate the model's internal workings. For instance, some defenses may introduce random noise or distortion the input samples Liu et al. (2017); He et al. (2019); Salman et al. (2019), making them less susceptible to adversarial perturbations. This noise acts as smokescreen, confusing the attacker and making harder for them to generate effective adversarial examples.
We list the defensive methods evaluated in this paper below:
Random Noise Defense (RND) Qin et al. (2021) a lightweight defense that adds Gaussian noise to the input for each query. This work also theoretically shows RND's 's effectiveness against query- based attacks. [yun al. (202T) proposes Small Noise Defense (SND), whose randomization method is identical to RND, thus reporting performance valuation for RND covers both of these works.
Adversarial Attack on Attackers (AAA) Chen et al. (2022) directly optimizes the model's logits to confound the attacker towards incorrect attack directions.
METHOD
3.1 PROBLEM FORMULATIONS
Adversarial attack. Let f: Rd + R be the victim model, where d is the input dimension, K is the number of classes, fx) is the predicted score of class ie for input x. Given an input example (r, y), the goal of adversarial attack is to find sample x' such that
arg max f(x') # s.t d(x, Se,
where d(x, x') is distance between samples x and x'. practice, the distance can be the normorm, || 22112, the frD -orm, ||z- z' oo
This adversarial task can be framed as a constrained optimization problem. More particularly, the attacker tries to solve the following objective
min L(f(x'))v) v), s.t d(z,z') S
(2)
where -(...) is a loss function designed by the attacker. practice, common loss function the max-margin loss, follows:
((f(r),) fz(2) max f((x). i4
33)
3

---

Published as a conference paper at ICLR 2024
Score-based attack. For the query-based attack, attacker can only access the input and output of the model; thus, the attacker cannot compute the gradient of the objective function with respect to the input æ. However, the attacker can upproximate the gradient using the finite difference method:
L(f(x + mu). - [(f(x). u, where pT).
7
(4)
"
Another approach to minimize the objective function is via random search. Specifically, the attacker proposes an update u and computes the value of L of this update to determine if u can help improve the value of the objective function. Formally, the proposed u is selected if [(f(z + u), y) ((f()),)) v) < O, otherwise it is rejected.
Decision-based attack. In contrast score-based attacks, hard-label attacks find the direction that has the shortest distance to the decision boundary. The objective function of an untargeted hard-label attack can be formulated as follows:
ming(d) g(d)
where g(d) = min (r arg max f(x + rd/ [d]la) # v).
(5)
This objective function can be minimized using binary search, in which the attacker queries the model find the distance r for particular direction d. To improve the querying efficiency, binary search can be combined with fine-grained search, in which the radius is iteratively increased until the attacker finds an interval that contains g(d). Hence, the gradient of g(d) can also be approximated by the finite difference method
Vgld)= Σ g(d + m78) g(d)
7
Similar to the of score-based attacks, the attacker can also search for the optimal direction. Given the current best distance rope. a proposed direction d is eliminated if it cannot flip the predic- tion using the current best distance Topt; otherwise the binary search is used compute g(d), which the new best distance.
Randomized model. In this work, consider randomized model frand Rd P(RK) that maps a sample æ E I to a probability distribution on R Given an input x and an attack query, the corresponding output is vector drawn from frana (a:). We assume that the randomized model fraud is 'nice"; that is, the mean and variance of frand (z) exist for every x.
Finally, we define adversarial samples for randomized model. Since the model has stochasticity, the prediction returned by the model of a sample x can be inconsistent at different queries; i.c., the same sample can be correctly predicted at one application of fraud and be incorrectly predicted later in another application of frand. For this reason, adversarial attacks are successful if the ob- tained adversarial example can fool the randomized model in the majority of its applications on the example.
Definition 1 (Attack Success on Randomized Model), Given a data point x with label y and positive real number point x' is called adversarial samples in a closed ball of radius around with respect to the model frand if ||x - ==Ip < e and
arg E + y.
3.2 RANDOMIZED FEATURE DEFENSE
Our method is based the assumption that the attacker relies the model's output to find the update vector toward an adversarial example. Consequently, the attacker receives unreliable feed- back from the model, it will be more challenging for the attacker infer good search directions toward the adversarial sample.
In contrast to the previous inference-time randomization approaches, we introduce stochasticity to the model by perturbing the hidden features of the model. Formally, let hu be the 1-th layer of the model, we sample an independent noise vector ö and forward he(r)) + 8 the next layer. For simplicity, 5 is sampled from Gaussian distribution AO, E), where Σ is a diagonal matrix, or (O vI), v E R. The detailed algorithm is presented in Algorithm
4

---

Published as a conference paper at ICLR 2024
Let frand be the proposed randomized model corre- sponding to the original f. When the variance of in- jected noise is small, diffuses but does not shift the prediction. Particu- larly, make the following assumption.
Algorithm 1 Randomized Feature Defense
assume that small noise
Input: model f. input data
noise statistics Σ, a set perturbed layers H {hy, hi,..... Output: logit vector
Assumption 1. Mean of the randomized model frand with input a is exactly the prediction of the original model for
for layer h; in the model do if h4 H then
E[[sann (z)] = (xx
~ NO, ))
2, h1(2(-1) +
Given Assumption by Definition adversarial samples of the original model are adversarial ples of the randomized model. Therefore, the direc- tion that the attacker seeks is also that of the original model. Recall that the attacker finds this direction by either finite difference or random search.
end if end for
In our method, when the model is injected with an independent noise, the value of objective L affected. If [(ran + ryu). y) [randd oscillates among applications of frand, the attacker is likely misled and selects wrong direction. For random-search attacks, when the sign of [[rrana( + syu), y) - [[frand (1),. 3) and the sign of L(f(x + ru), v) [(f(),, y) are different, the attacker chooses the opposite action to the optimal one. In other words, the attacker can either
accept a bad update or reject a good one in a random search.
3.3 ROBUSTNESS TO SCORE-BASED ATTACKS
In this section, we present the theoretical analysis of the proposed defense against core-based at- tacks.
Theorem 1. Assuming the proposed random vector u is sampled from a Gaussian (0, μl), the model is decomposed into g h, and the defense adds a random noise ö NO vI) to the output of h. At input x, the probability that the attacker chooses opposite action positively correlates with
V N(( (C 9 arclan (-(22
(L ( f
This theorem states that the robustness of the randomized model is controlled by both (i) the ratio between the defense and attack noises and (ii) the ratio of the norm of the gradient with respect to the feature h(x) and the norm of the gradient with respect the input ær Since arctan is monotonically increasing, the model becomes more robust if the ratio (Ceo))l is high. Intuitively, the [V=
perturbations added by the attacker and by the defense induce a corresponding noise in the output; the attack noise is dominated by the defense noise, the attacker cannot perceive how its update affects the model. Note that the arctan function is bounded, which means at some point the robustness saturates when the ratio increases.
While the first ratio is predetermined before an attack, the second ratio varies during the attack when the input a is sequentially perturbed since depends on the gradient of the objective function. To understand this behavior of the randomized model during the attack, we perform the following experiment. First, we compute the ratio of the norms of gradients attacker, we perform single gradient descent step with respect to L. The distributions of the ratios on the raw and perturbed images at different layers are shown in Fi We can observe that these ratios become higher when the data are perturbed toward the adversarial samples. In other words, the randomized model is more robust during the attack. We also illustrate the accuracy under Square attack when adding noise to each layer, verifying analysis.
h(x)) and æ. To simulate an
3.4 ROBUSTNESS TO DECISION BASED ATTACKS
In decision- based attacks, the attacker finds the optimal direction dopt and the corresponding dis- tance Topt the decision boundary such that Topt is minimal. We use the objective function
5

---

Published as a conference paper at ICLR 2024
Block AuA: 28.9
Block 4. AuA: 70.7
Block 8. AuA: 71.7
Block 12. AuA: 58.4
Block 15, AuA: 53.8
2.0
Density
40
TDW nolsy
Demsity
28
Densin
Density
Density
1.5
20
10
1.0
1.0
0.5
0.0
0.0
0.35
0.40
0.2
0.3
0.4
0.25 0.50 0.75
2
Ratio
Ratio
Ratio
Ratio
Ratio
Figure 1: The ratio of the norm of the gradient of L at selected hidden layers and at input of VGG19
on CIFAR1O before and after perturbed. Full results are provided the supplementary material.
Block 1. AuA: 28.9
Block 4. AuA: 70.7
Block 8. AuA: 71.7
Block 12. AuA: 58.4
Block 15, AuA: 53,8
Density
0.6
reature
input
Density
0.6
0.6
Density
0.6
Density
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
2.5
5.0
0.0
2.5 5.0
0.0
2.5
5.0
0.0
2.5
5.0
0.0
2.5
5.0
Rel. robustness
Rel. robustness
Rel. robustness
Rel. robustness
Rel.
robustness
Figure 2: Distributions of the magnitude of the robustness to query-based attacks computed at input and selected hidden layers of VGG19 on CIFAR10.
C(f()) y) to understand how our method affects the decision- based attacks. Indeed, L measures how close the prediction is to the true label: - 0f the prediction is incorrect and otherwise.
To estimate g(d), the attacker can use binary search. Similar to score-based attacks, when noise is injected into the model, the function g(d) becomes stochastic, which eventually affects the attack. Unfortunately, the distribution of g(d) (under binary search with randomness) does not have an analytical form. Nevertheless, we can still use similar analysis to the last section to understand the robustness of our method. To avoid performing binary search on uninformative directions, the attacker relies on best-radius searching. Given the current best distance Topt. for every new direction d, the attacker verifies if the distance along d to the boundary is shorter than Topt by querying x + d/ |d]|2. When adding noise to features h(x) of f = g o h and linearizing the function at the current input x, we have
[[rand (x +
~ L(g(h(x) + Tope Ja (apnn((())//lll + +),8) -(f(x),)) v) + Topt V=L( [1d/ + ((g(())))
(7)
R (Nope
(lan v)o, (9)
where JM(x) is the Jacobian matrix of h evaluated at I, since [(f + L(())=L((()), y))d// R Lif(x + g(d)d/Ild|)a)) ()). = 8 N(O,, the variance of V ((())(((((()) yjō is v M() (igh((()) 1) When this value is large, it can dominate the other terms and increase the chance of flipping the sign of the loss function L. In other words, when L has high variance, the attacker is more likely to misjudge the direction.
3.5
THE EFFECT OF RANDOMIZED FEATURES ON ACCURACY
Let 'D be the data distribution, without any attack or defense, the accuracy of the model Acc(J) E
[1 v)]
(a.))~ [I([([()))) > 0)).
(10)
When injecting noise into the model, it becomes a robust, stochastic model frand : Rdd - ((R
The clean accuracy of the randomized model is Acc(frand) = E
E Adding noise 62 ~ NO, v2 !) to the features at layer h of the model f = g h results in: Acc(frand)
[1(1// -]]
E E
[I(C(V v) > 0)}. (11)
(a.I)>DD y'~frand
(a.))D -' (x)
E ( (0.12)
(1((((()()) + 62). 0))
(12)
00]
(13)
(x.D)D (U,21)
E
E
(LCCf(to.w))/VVnncCCClll))) 0).
(14)
(x.))D 80,2(

---

Published as a conference paper at ICLR 2024
It means that the accuracy of randomized model depends on the objective function and its gradient, which vary for different data points. These ratios C and its gradient computed at the input and hidden layers are different. If L is small samples that have large gradient norm when noise is injected at a layer, these samples will be likely misclassified while the correctly classified samples have low magnitude of robustness (i.e., v M(x g)ll is small, discussed in Theorem and Section 3.4). In contrast, if the gradient norm with respect to the randomized layer is large for samples that have large L, the robustness of the model for the correctly classified samples will be high; thus, adding noise to this layer makes the model more robust against black-box attacks.
We conduct the following experiment to understand how the defense affects the whole dataset. We first compute the ratios of C and its gradient for all samples and keep the top 99% values. Essentially, the standard deviation of defensive noises that makes the accuracy drop by 1% is proportional the value at which 1% of the ratios in the dataset are smaller. The product of this value and the norm of gradient represents the robustness of datasets, which are shown in Figure 2 We also illustrate the accuracy under Square attack when adding noise to each layer.
We can observe that the ratio distributions when randomizing the input and the hidden features are similar at the first few layers of the model; however, these ratios at the deeper layers of the model are higher. This means that randomizing the model at these layers makes more robust than adding noise to the input layer when the defenders desire similar clean accuracy in the randomized models.
4 EXPERIMENTS
In this section, we evaluate the empirical performance of the proposed randomized feature defense.
4.1 EXPERIMENTAL. SETUP
Datasets. We perform our experiments on two widely used benchmark datasets in adversarial ro- bustness: CIFARIO Krizhevsky & Hinton (2009) and ImageNet Russakovsky et al. (2015). We randomly select 1000 images that contain every class from the studied dataset in each experiment.
Defenses. In addition to the proposed defense, also include the related input defenses Qin et al. (001): [yyun et al. (202I) in our evaluation. Note that, the empirical robustness comparison of all adversarial defenses is beyond the scope of the paper since objective is to theoretically and empirically study the effectiveness of the randomized feature defense. We also evaluate AAA de- fense(Chee al. 2022 against decision-based attacks and compare them with randomized defenses.
In all experiments, our defense randomizes only the penultimate layers of the base models, since our theoretical observations in Section 3.3 (Figure [ and empirical results, provided in Table [4] (Supplementary). show that randomizing a deeper layer is consistently more effective.
Attacks. For score-based attacks, consider the gradient-estimation methods, NES Ilyas et al. (2018). and the random-seare methods, Square Andriushchennoo et al. (2020). SignHunt Al- Al-Dujaili & O' O'Reilyy| (2020). For decision-based attacks, we consider RayS Chen & Gu (220 and Sign- Flip Chen et al. (2020b).
Models. We consider victim models on ImageNet, including 2 convolution models that are VGG19 S & Zisserman (2015) and ResNet50 He et al. (2016), 2 transformer models that are Dosoviiskkyy et al. (2021) 1) and DeiT et al. (2021). For the experiments on CIFARIO, 10, we finetuned VGG19, ResNet50, ViT, Dei'T with an input size of 224 x 224.
Evaluation protocol. For a fair comparison, we report each defense's robustness performance re- sults at the corresponding configuration of hyperparameters that achieves a specific drop (i.e., & % or x2%) in clean-data accuracy, In practice, a defender always considers the trade-off between ro- bustness and clean-data performance, with priority satisfactory clean-data performance; thus, achieving higher robustness but a significant drop in clean-data accuracy is usually not acceptable.
4.2 PERFORMANCE AGAINST SCORE-BASED ATTACKS
On ImageNet, report the accuracy under the attack of 6 models and 3 score-based attacks in Ta- As we can observe, while the attacks achieve close to 0% failure rate on the base models (i.e., without any defense), both randomized feature and input defenses significantly improve the models' robustness against score-based attacks. Furthermore, for Square attack and SignHunt, which are strong adversarial attack baselines, randomized feature defense consistently achieves better perfor-
ble I
7

---

Published as a conference paper at ICLR 2024
mance on all 6 models, which supports theoretical analysis in Section For instance, while the base VGG19 models are severely vulnerable, our randomized feature defense achieves 22.2% in ro- bust accuracy after 10000 query, also significantly better than the randomized input defense (17.8% robust accuracy). On the transformer-based Dei'T, our randomized feature defense has 69.1% ro- bust accuracy under Square attack, while the robust accuracy of the randomized input defense is 2% lower. For the NES attack, the randomized-featur VGG19 shows the best robustness. In summary, randomized feature defense consistently achieves high robustness on most models except ResNet50 where the robustness is similar to randomized input defense.
Table I: Defense Performance in ImageNet. The clean-data accuracy of the robust models
Table 2: Defense Performance in CIFARIO 10.
The clean-data accuracy of the robust models is allowed to drop either x 2% or = 4%.
is allowed to drop either x 1% or ~ 2%.
Model
Method Ace
Square
NES
SignHunt
Model
Method Ace
Square
NES
SignHunt
1000
10000 1000
10000
1000 10000
1000
10000 1000
10000
1000 10000
Base 80.37
3.5
36.2
4.3
Base
97.66
0.8
0.1
3.7
0.2
ResNet50
Input
78.46 (Rc
79.18 (Fc 1%)
2%)
40.3
41.1
39.8
39.5
63.8
69.4
41.5
23.9
49.3
47.6
47.2
45.4
ResNer50
Ingut
95.98 (ee 2%)
93.42 (e2
4%)
50.5
56.4
54.8
48.8
93.1
90.0
85.4
85.0
31.1
26.8
29.8
26
Feature
79.70 (PS
78.43 (Ro
2%)
1%)
37.0
42.0
41.5
36.0
65.6
56.7
40.6
16.8
51.3
46.3
43.4
49.3
Feature 93.48 (Rc 4%)
95.95 (F# 2%)
56.7
54.9
52.8
53.4
93.2
89.9
86.2
83.9
37.1
32.5
35.7
30.6
Base
74.21
0.0
Base
96.28
0.6
0.1
3.2
0.3
VGG19
Imput
71.43 (A2
73.24 (Fo 1%)
2%)
18.7
7.7
17.8
6.9
47.4
32.1
11.5
1.5
28.3
18.3
27.1
17.0
VGG19
Ingut
94.92 (Ae 2%)
93.52 (e2 4%))
30.6
42.2
27.1
39.8
90.3
89.5
58.0
68.4
27.5
22.7
26.8
21.8
Feature
71.21 (
72.66 (F= 1%)
2%)
22.4
23.3
22.2
21.6
55.1
50.1
28.4
18.5
34.6
36.5
32.8
32.9
Feature
94.93 (F# 2%)
93.58 (Rc 4%)
61.0
64.2
58
62.8
91.2
92.2
80.1
77.9
43.2
49.2
42.4
46.9
Base
82.00
6.4
46.7
0.8
Base
0.0
7.9
DeiT
Imput
79.60
(0:
1%)
2%)
66.6
67.7
66.0
67.2
75.8
75.7
65.9
67.1
64.9
64.4
63.6
64.3
DeiT
Ingut
96.59 (ee 2%)
94,81 (e2 4%)
70.6
66.9
68.8
67.6
92.6
95.2
90.0
87.7
40.3
40.2
38.5
39.2
Feature
80.80 (F= 1%)
79.76 (Ro
2%)
69.7
69.3
69.0
69.1
75.0
75.1
65.3
59.1
66.4
66
64.3
64.1
Feature
94,91 (R2 4%)
96.29 (F# 2%)
69.1
68.9
67.9
66.1
93.5
94 1
87.6
88.3
45.7
43.6
43,4
40.4
Base
79.15
5.7
7.3
Base
5.1
0.0
84.8
6.1
0.0
WTT
Imput
78.28 (Po 1%)
77.09 (2c 2%)
61.3
58.8
58.1
60.9
70.6
70.8
59.2
51.4
53.7
$3.1
52.2
52.7
VIT
Input
93.40 (e2 4%)
95.80 (es 2%)
63.0
62.6
61.1
61.2
89.7
93.5
87.0
85.5
34,8
33.4
32.2
33.3
Feature
77.18 ( 2%)
78.20 (F=
60.6
63.7
60.2
62,9
69.1
72,2
58.1
47.5
57.0
54.0
52.9
55.3
Feature
95.96 (F# 2%) 63.9
93.39 (ee 4%) 66.2
62.7
65.6
93.7
92.9
85.3
85.6
42.5
44.8
40.7
43.8
We also observe similar robustness results on CIFARI0 experiments with ResNet50, VGG19, DeiT, and ViT for 3 attacks. As we can observe in Table 2 randomized feature and input defenses are effective against score-based attacks. Similar to ImageNet, randomized feature defense achieves significantly better robustness than randomized input defense in most experiments. For Square at- tacks on ResNet50 and Dei'T, while the best robustness is achieved by randomized input defense, randomized feature defense is more robust when the defender sacrifices 2% clean-data accuracy.
Dynamic Analysis of Robustness. As the adversary increases the magnitude of pertur- bation, the attack becomes more effective since the misleading probability decreases as shown in Theorem [ The adversary can vary the square size for Square attack, the exploration step for NES, and the budget for SignHunt (since SignHunt sets the finite- difference probe the perturbation bound).
Table 3: Robustness (higher means more robust) un- der different values of j. Small v corresponds to selected where clean accuracy is allowed to drop by 2%, and Large v corresponds to clean accuracy drop of 4%.
VGG
Attack
Small
Large
Small
I Large 2/
Input Feature Input Feature Input Feature Inpot Feature
Table 3 reports the robustness of the mod- els under stronger attacks from these adver- saries for different values of v. We can ob- serve that increasing the strength of the at- tack leads to lower robustness among all the defenses. However, at the selected defense noise scales corresponding to the same clean accuracy drop, our defense is still more robust than randomized input defense; this improved robustness again can be explained by the analysis in Section 3.3 and [3.5] A larger attack perturbation may also cause the approximation in the attack to be less accurate, which leads to drop in the attack's effectiveness; for example, the robust-
0.05
30.6
61.0
42.2
64.2
63.0
63.9
62.6
66.2
Square
0.1
47.4
65.8
54.6
65.5
09.3
70.2
68.8
69.6
0.2
3
59.7
43.9
64.0
56.
58.0
568
58.6
0.3
27.0
$4.9
38.1
59.7
47.1
51.9
47.7
50.4
0.001
93.4
91.4
93.7
93.5
NES
0.01
93.7
88.0
90.0
89.6
90.6
0.2
93.6
93.0
92.6
91.4
91.0
93.8
87.6
92.0
0.01
91.6
91.0
91.3
88.0
80 1
90.9
85.4
91.3
SignHunt 0.05
22.7
43.2
27.5
29.2
34.8
42.5
314
44.8
0,075
5.6
19.7
8.1
25.6
13.6
22.5
13.7
0.1
1.2
2,4
12.1
12.7
8

---

Published as a conference paper at ICLR 2024
Table
Robustness against decision-baa based attacks (CIFARI0)
Table 6: Robustness in CIFARIO at each layer (fixed v).
Table Robustness with adversarial training.
Model
Method
Acc
RayS SignFlip
Model Layer Square NES SignHunt GradNorm
Base
97 66
0.1
20.5
VGG
56.7
87.8
21.5
1.324
Square NES SignHunt
ResNet50
AAA
97.70
20.4
18.7
Input
12.0
85.5
12
29.4
AT
32,5
67.6
31.7
Feature
92.10
14.4
82.5
15
50.6
37.4
1.710
Ours
37.6
44.1
41.7
Ours+AT
77.8 80.6
67.0
VGG19
Base
AAA
96.28
96.30
0.0
5.7
6.4
VIT
75.3
77.3
94.4
94.8
26.1
28.0
0.462
0.615
Input
93.42
86.0
26.9
93.48
15.4
76.5
86.5
Table 7: Defenses against adaptive attacks on CIFARIO
VGG19
ResNet50
Attacks
Methods
Acc
Acc
M
=
M
QC=1000 QC=1000 QC=5000
1C01000 QC=10000
QC=1000
QC=1000
QC-5000
QC=1000 QC=10000
Square
Input
Feature
94.93
94.92
24.2
10.5
46.7
23.1
52.9
54.5
42.0
35.0
37.3
21.1
13.3
NES
Input 94.92
Feature
94.93
92.2
89.5
94.8
93.4
82.1
88.4
94.5
94.4
86.0
78.8
95.32
95.21
91.8
92.4
93.8
94.0
90.8
91.3
93.9
94.0
90.4
90.7
SignHunt Input 94.92
22.7
15.9
10.4
23.3
95.32
29.9
17.6
13.5
21.1
9.4
Feature
94.93
43.2
27.1
23.0
31.7
17.0
95.21
35.1
17.3
16.4
21.5
11.3
ness increases from 89.6% to 91.4% when the NES's perturbation magnitude increases in VGG19 experiments (similar observations in ViT).
Combined with Adversarial Training (AT). We evaluate the combination of our defense and AT on CIFAR 10/2e model against under score-based attacks with 1000 queries and observe significantly improved robustness, as shown in Table 4
4.3
PERFORMANCE AGAINST SECISION-BASERI ATTACKS
Table 5 reports the performance of VGG19 and ResNet50 against 2 decision-based attacks CI- FAR10. Besides randomized feature and input defenses, we also include AAA defense, which opti- mizes the perturbation that does not change the prediction. While AAA is optimized for score-based attacks directly and thus is successful in fooling these attacks (as seen in Table 3 in Supplementary), the results show that AAA not effective in defending against decision- basod attacks, while ran- domized feature and input defenses improve the robustness. An interesting observation is that RayS attack is more effective than score- based attacks although it only uses hard labels, even when there are defenses.
4.4 RELATIONSHIP BETWEEN THE GRADIENT NORM AND THE ROBUSTNESS TO SCORE-BASER ATTACKS
In Table we provide the corresponding accuracy under attack on CIFARIO with 1000 queries (for when a single layer randomized with a fixed value of v) and the mean of the gradient norm at that layer. As can observe, the gradient Figure [ the robustness also increases, thus verifying our thcoretical results.
increases (also
originally observed in
4.5 PERFORMANCE AGAINST ADAPTIVE ATTACKS
We conduct experiments with adaptive attacks that apply Expectation Over Transformation (EOT)AAthayee et al. (2018) which the attacker queries a sample M times and averages the outputs to cancel the randomness. Table 7 show the robust accuracy of VGGI9 and ResNet50 CIFARIO against EOT attack with M = 5 and M = 10. Note that with EOT, the number of updates in the attack is M times less than that of a normal attack with the same query budget. For this reason, we report the results for adaptive attacks with both 1000 queries and M x 1000 queries. We can observe that EOT can mitigate the effect of randomized defenses even with the same number of queries; however, feature defense still yields better performance.
5 CONCLUSION AND FUTURE WORK
In this work, we study the effectiveness of random feature defense against query-based attacks, in- cluding score-based and decision- based attacks. We provide analysis that connects the robustness to the variance of noise and the local behavior of the model. Our empirical results show that random defense helps improve the performance of the model under query-based attacks with trade-off in clean accuracy. Future works will be directed toward the analysis covering black-box box attacks that transfer adversarial samples from the surrogate model to the target model.

---

Published as a conference paper at ICLR 2024
REFERENCES
Abdullah Al-Dujaili and Una-May O'Reilly. Sign bits are all you need for black-box attacks. In- ternational Conference on Learning Representations, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: query query-efficiiet black-box adversarial attack via random search. In 16th European Confer- ence on Computer Vision, pp. 484-501, 2020.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In the 35th International Conference on Machine Learning, volume 80, pp. 284-293, 10-15 Jul 2018.
Junyoung Byun, Hyojun Go, and Changick Kim. On the effectiveness of small input noise for defending against query-based black-box attacks. In 2022 IEEE/CVE Winter Conference on Ap- plications of Computer Vision (WACV), pp. 3819 1828, 2021.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Jinghui Chen and Quanquan Gu, Rays: A ray searching method for hard-label adversarial attack. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2020.
Pin-Yu Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order opti- mization based black-box attacks to deep neural networks without training substitute models. In the 10th ACM Workshop on Artificial Intelligence and Security, pp. 15-26. ACM, 2017.
Sizhe Chen, Zhehao Huang, Qinghua Tao, Yingwen Wu, Cihang Xie, and Xiaolin Huang. Adversar- ial attack on attackers: Post-process to mitigate black-box ccore-based query attacks. In Advances in Neural Information Processing Systems, volume 35, pp. 14929- 14943, 2022.
Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial tacks. In Proceedings the 1t ACM Workshop on Security and Privacy on Artificial Intelligence, pp. 30-39,2200 2020a.
Weilun Chen, Zhaoxiang Zhang, Xiaolin Hu, and Baoyuan Wu. Boosting decision-based black- box adversarial attacks with random sign flip. In Proceedings of the European Conference on Computer Vision, 2020b.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Proceedings of International Conference on Machine Learning, pp. 1310-1320, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2021.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, 2015.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple blxck-box adversarial attacks. In International Conference on Machine Learning, pp. 2484 2493, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 588-597, 2019.
10

---

Published as a conference paper at ICLR 2024
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pp. 2137- 2146, 2018.
Andrew Iyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In International Conference on Learning Representations, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical Report 0, University of Toronto, Toronto, Ontario, 2009.
Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y Zhao. Blacklight: Scalable defense for neural networks against Query-Based Black-Box attacks. In 31st USENIX Security Symposium (USENIX Security 22), pp. 2117-2134, 2022.
Sijia Liu, Pun-Yu Chen, Xiangyi Chen, and Mingyi Hong. signSGD via zeroth-order oracle. In International Conference on Learning Representations, 2019.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. In European Conference on Computer Vision, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In Proceedings of the 6th Interna- tional Conference on Learning Representations (ICLR), Vancouver, Canada, 2018.
Ren Pang, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. Adymind: Inferring adversary intent of black-box attacks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1899-1907, 2020.
Tianyu Pang. Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pp. 4970-4979, 2019.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. Crafting adversarial input sequences for recurrent neural networks. In IEEE Military Communications Conference, pp. 49-54, 201
Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Anan- thram Swami. The limitations of deep learning in adversarial settings. In Proceedings of the IEEE European Symposium on Security and Privacy (Euro&&P), pp. 372-387, Saarbrücken, Germany, 2016b.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow. Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the ACM on Asia Conference Computer and Communications Security, pp. 506-519. ACM, 2017.
Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against query-based black-box attacks. In Advances in Neural Information Processing Systems, volume 34, 2021.
Ali Rahmati, Seyed- Mored-Mohsen Moosavi-Dezfood Pascal Frossard, and Huaiyu Dai. GeoDA: A geo- metric framework for black-hox adversarial attacks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV,, volume 1 115, pp. 211-252, 2015.
Hadi Salman, Jerry Li, Ilya Razzenshteyn Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems, volume 32, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference Learning Representations (ICLR), 2015.
11

---

Published as a conference paper at ICLR 2024
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable some distributional robustness with principled adversarial training. In International Conference on Learning Representations (ICLR), 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), Banff, Canada, 2014.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representa- tions (ICLR), 2018.
Binghui Wang, Xiaoyu Cao, Neil Zhenqiang Gong, et al. On certifying robustness against backdoor attacks via randomized smoothing. In Proceedings of CVPR Workshop on Adversarial Machine Learning in Computer Vision, 2020.
Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: diversifying vulnerabilities for en- hanced robust generation of ensembles. In Advances in Neural Information Processing Systems, volume 33, pp. 5505-5515, 2020.
Dinghuai Zhang, Mao Ye, Chengyue Gong, Zhanxing Zhu, and Qiang Liu. Black-box certification with randomized smoothing: A functional optimization based framework. In Advances Neural Information Processing Systems, volume 33, pp. 2316-2322 2020.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
This Appendix provides additional details, analysis, and experimental results to support the main paper. We begin by discussing the limitations in Section AAationAAandd societal impacts in Section B Then, we provide the proof for Theorem 1 the main paper in Section C Next, the detailed experimental setup is provided in Section D which is followed by additional robustness experiments to demon- strate the effectiveness of the proposed defense. Finally, we provide additional visualization of the robust behavior of the models with randomized features in Section
A LIMITATIONS
As discussed, we focus studying the effectiveness of the randomized feature defense for DNNs against black-box attacks, including score-based and decision- attacks. We do not include black-box attacks that utilize the transferability from surrogate to target models since our threat model does not make any assumptions about the network architecture and the training dataset.
Our proposed defense adds another layer of protection DNNs against adversarial attacks, and would be interesting to study the adversarial robustness of the combination of our randomized feature defense and existing defense strategies, including those that have been developed for transfer- based attacks Pang et al. (2019;; Tramèr et al. (2018); Yang et al. (2020). We leave these to future works.
Similar to the related works on randomized defenses, our empirical evaluation has limitation for it is designed to assess the relative effectiveness of randomized models. From the adversary perspective, the attack should only need to make the system fail once, during the querying process. Correspond- ingly, it means that the attack can query the model repeatedly, and by chance, the defense fails at some point; however, this failure, equivalently meaning the attack is successful, is not due to the perturbed input being an adversarial example but rather comes from the added randomness of the defense. Here, there are 2 scenarios: (1) if the input (or the perturbed query) is far from the deci- sion boundary, randomization much less likely to shift the input to the other side of the boundary, making this chance very low. On the other hand, (2) for input close to the decision boundary, this "repeated" attack will be more effective, unfortunately; one potential solution is to preemptively stop this attack if the system recognizes the same input is repeatedly forwarded the model. We leave this to future work and urge practitioners to research this inherent problem randomized models.
In practice, attack has cost, and if the attack cost is higher than the potential gain, the attacker will more likely stop; thus a defense should increase this cost as much as possible. A randomized approach like ours increases the attack cost by confusing their optimization trajectory. Note that as explained earlier, if the perturbed query is in scenario (1) (or the randomized radius is small enough), our randomization would likely not make the attack successful due to chance; on the other hand, the defense would make it more costly for the attack to push this query to scenario (2).
B SOCIETAL IMPACTS
Deep neural networks (DNN) rapidly transform our daily lives in various domains and applications. Unfortunately, most well-trained DNNs are vulnerable to adversarial attacks, which decreases con- fidence in their deployment. Among the existing adversarial attacks, query-based attacks pose severe threat to users since these attacks are effective and only require access the model's feed- back; the attackers do not know the trained parameters or model architectures.
Our work tackles this defense challenge against query-based attacks by proposing lightweight ad- versarial defense for existing DNNs. We conduct a detailed theoretical analysis of our defense and show its superior performance compared to other randomized defenses in extensive empirical ex- periments across wide range of DNN architectures, query-based attacks, and benchmark datasets. Most importantly, our method can be directly integrated into any existing off-the-shelf DNN, In summary, the proposed randomized feature defense can boost the adversarial robustness of existing DNNs against most query-based attacks, further improving the users's 's confidence when using them in practice.
13

---

Published as a conference paper at ICLR 2024
C PROOF OF SECTION 3.2
Theorem 1. Assuming the proposed random vector is sampled from Gaussian (00, pT), the model decomposed into f = 9 oo h, and the defense adds a random noise ó NO, vI) to the output of h. At inpua x, the probability that the attacker chooses opposite action positively correlates with
arctan (-(21 \VM() ) (C C)g)l 1
-0.5
(C f)
Proof. For score-based attacks, the attacker finds the direction by computing (if(xtu)-
When applying randomized defense, the direction instead relies L(fraud + u)) [(frand (z)). As discussed in Section 3, the probability that the attacker chooses an opposite action is
[[raudd (x + u)) - [(frand(a)) L(S(x c +u)) L(((x))
<0]
P
(15)
If the defense adds random noise N/O, vI) to the output of layer h of the model g o h, we have f(rrau( = ((g(h(x) + 6). Since and u are small, we can linearly approximate the objection function
= (
fTu, V
(16)
(17)
C(g(h(x + u + M((h))
(18)
= = g) Vm(>(Co g)
(19)
= [[]((())++Vv///) V.Co f) g))
(20)
where (C is the gradient of o evaluated h(x). ( f) is the gradient of L at v, Jn. (ar) the Jacobian matrix of h x.
At a different application of fraud, the randomized model samples a new noise vector. Let 91, 82 be the sampled noises when querying [(frend( u)) and L(frand (z)), the ratio can be approximated by
[(Srana (x + u)) [()rand())) C(f(# u)) C(f(x))
Vz(L TT Maa T) - o
(21)
Vz T
=1+ (Co
(22)
Va(L T)
Since and 82 are independent, we have 81 - 82 ~ (0, 2v1). thus
Va(z)(Co o g) -0)0118))~-000000)01 ~2),0 (0. M() g)) Ve(Co f)
(23)
f)
(24)
The noises 91, d2 added by the defense and the noise added by the attacker are independent, there- fore the ratio of two independent normal variables (Cee)( (81-52) follows Cauchy distribution (Coff)T
with location 0 and scale v 2v [Naes In this case, the probability that the attacker is fooled
[Va(Cof)]
can be approximated by
[(frand (x u)) C( ((fran(())
( g6 52)
RPP
<-1
(25)
C(s(x + uu) - L(f(x))
Va(Co f)Tu
(-(20 [[a f)
(20 VC g)1l3
1
arctan
n
2
(26)
14

---

Published as a conference paper at ICLR 2024
C.I ANALYSIS THE APPROXIMATION ERROR
Due to the high nonlinearity of neural networks, it is difficult to have complete analysis of the convexity of the boundary. However, if the model is -ipLsscchit, the change in the prediction when a noise u is added is bounded by k1 and u.
I + u) - f()) - 11////12
If k is not too large, since ||ul|2 is very small, expectation the prediction of the randomized models stays still. Furthermore, empirically validate whether the majority of the perturbed inputs belong to the original class, as shown in the experiment below.
For each imput, sample 20 pairs of perturbation with opposite directions. It's similar to antithetic sampling employed in Ilyas et al. (2018) for a better estimation of the effect the injected noise. Intuitively, if the majority of the perturbed inputs still in the original class, the average of the predictions is the same as the output of the original model. We also consider the extreme case (or worst case) where input is marked as misclassified if any of those 40 noise vectors misleads the model. The table below shows the original accuracy, the average accuracy, and the extreme accuracy on VGGI9/CIFARIO. As can be observed in Table the decrease in accuracy of the expected case is trivial (0.37%); and even in the extreme case, this decrease is still small (2.3%) even in the extreme case. This supports our assumption.
Table 8: The accuracy of randomized defense
Ace
Expected Acc
Extreme Ace
96.28
95.91
93.95
The perturbation added by the defense induces negligible effect on the prediction of the model; since μ is constrained by the adversarial constraint (i.c., within the Lp ball), it is typically smaller than v, thus inducing even small impact than the prediction. Furthermore, if the gradient of the loss function [(x) with respect to the input is li2 12-Lipsshitt continuous, we can bound the error of the approximation by l2 and the norm of the noise u
L(x +u) < L(x) + VoL ++
2 k2 k2/lu|12
If l and the noise are small, the error is also small. This assumption is also used in Qin et al. (2021) for their analysis. We also provided the error of the first-order approximation in Section F. The histogram of the error implies that the added noise is small enough for close approximation, thus our analysis is valid.
D EXPERIMENTAL SETUP
D.1 DATASET
In this work, we conduct experiments two widely used datasets in adversarial attacks, CIFARIO and ImageNet. We randomly selected 1000 images per dataset such that the test sets cover all classes, each of which has equal size.
CIFARIC consists of 60, 000 images from 10 different classes where the training has 50, 000 images and the test set has 10, 000 images.
ImageNet (ILSVRC) 2012 is a large-scale dataset that consists of 1000 classes. The training set includes 1, 281, 167 images, the validation set includes 50, 000 images, and the test set has 100, 000 images.
For all experiments, we resize the images to 224 224 resolution.
waw. html
15

---

Published as a conference paper at ICLR 2024
D.2 MODELS
As discussed in the main text, we consider 4 models that have various architectures, including ResNet50 He al. (2016). VGG19 [iionnyan & Zisserman (2015), ViT base Dosovitskiy et al. (2021), DeiT base Touvron et al. (2021). We use the pretrained weights from t timm packag for ImageNet, and finetune ResNet50, VGG19, ViT base, DeiT base for CIFARIO.
D.3 IMPLEMENTATION OF THE BLACK-BOX ATTACKS
We perform experiments 3 score-based black box attacks (Square attack Andriushchenko et al. (2020), NES Ilyas et al. (2018), and Signhunter Al-Dujaili & O O'Reilly (2020)) and decision- based attacks (RayS Chen & Gu (220)) and SignFlip Chen et al. al. (2020b)) For 700 attacks, find adversarial samples within the 8 ball of radius 0.05, for e² attacks we set the radius to 5. The detailed hyperparameters of each attack
follows:
Square attack: The initial probability of pixel change is 0.05 for fo attack and 0.1 for l attack.
NES: We estimate the gradient by finite difference with 60 samples for 8 attack and 30 for g² attack. The step size of finite difference 0.01 and 0.005, and the learning rate is set to 0.005 and for 8r and l² attack, respectively.
D.4 EVALUATION
According to Section 3, sample is considered as adversarial if it can fool the model in the majority of its application. Since the randomized model has stochasticity, for any datapoint there is a chance that the prediction is flipped at some application. Therefore, finding the true adversarial sample. To alleviate this issue, when deciding whether to stop the attack, the query is repeated multiple times, and the attack is considered to be successful if the prediction consistently flipped most of the runs. The experiment applies 9 query runs for verification, and these extra not included in the total number of queries.
attacker can stop the attack before
E ADDITIONAL. EXPERIMENTS
E. PERFORMANCE AGAINST & ATTACKS
We provide the results of randomized feature defense against 2 attacks on CIFARIO in Table 9 As we can observe, L attacks are quite successful fooling the model; however, randomized feature defense improves the robustness of the models to these attacks.
Table 9: The robustness against 6 attacks CIFARIO.
Square
NES
SignHunter
Model
Method
Acc
1000
10000
1000
10000
1000
10000
Base
96.28
13.5
3.0
72.9
50.7
44.2
11.2
VGG19
Feature
93.58
85.3
82.8
91.4
87.9
87.1
85.7
Basc
97.86
48.5
28.6
87.8
73.5
60.7
30.3
ViT
Feature
93.38
88.2
88.1
92.7
90.1
86.4
85.7
E.2 PERFORMANCE AGAINST DECISION-BASED ATTACKS
Table 10 shows the performance of the model on ImageNet under decision- based attacks. Similar to CIFARTO, randomized feature defense is effective against decision-based attacks while AAA Chen
16

---

Published as a conference paper at ICLR 2024
et al. (2022) defense is not helpful in this case, since decision-based attacks only rely the label that the model returns and AAA defense keeps the output label the same. We also provide the results for RayS ViT/CIFARIO in Table IT]
Table 10: The robustness against decision-based attacks on ImageNet.
Model
Method
Acc
RayS
SignFlip
Base
74.21
0.1
AAA
74.24
0.4
0.8
VGG19
Input
71.43
6.6
53.3
Feature
71.21
10.0
46.9
Base
79.15
1.7
9.7
AAA
79.14
2.2
9.7
ViT
Input 77.09
41.3
70.3
Feature
77.18
41.0
70.1
AAA's Performance against score-based attacks. We also provide the evaluation of AAA under score-based attacks in Table 12 Since AAA is optimized for score-based attacks directly, it is successful in fooling the attack. However, under general setting where the defender does not know the type of attack is currently performed (a more realistic scenario). AAA failed miserably as shown above, while our defense performs well regardless of the attack.
E.3 PERFORMANCE AGAINST WHITE- BOX ATTACKS
As mentioned in the main paper, similar to previous works Byun et al. (202T); Qin et al. (2021);
Chen et al. (2022). our threat model focuses on defending against black-box, query-based attacks, it is more realistic scenario in practice. Nevertheless, in this section, we provide an additional study about the performance of our defense against white-box attacks, those that require access to the model's architecture and its parameters. We evaluate the performance of method against C& [Carlini & Wagner (2017) and PGD |Madry et al. (2018) with Loo constraint = 0.03. As observed in 13] the proposed defense can boost the robustness against these white-box attacks while having negligible degradation in clean accuracy. We conjecture that adding stochasticity to the model can transform into a smoothed classifier and therefore reduce the adversarial effect.
E.4 ROBUSTNESS CHARACTERISTICS OF LAYERS
As discussed in Section 3.3 in the main paper, the gradient norm varies during the sequence of queries of an attack norm increases during the attack, which is related to the model's robustness as seen in Theorem [ thus the model becomes more resilient to black-box attacks. Here, additionally provide the performance evaluation on CIFARIO with 1000 attack queries when each layer is perturbed alone (with such that the clean accuracy drops within similar threshold), well as the mean rate of change in the gradient norm's 's ratios during the sequence of queries. Table [4 implies that deeper layers induce higher change and lead to better robustness, which confirms our analysis.
input. Figure suggests that, in deeper layers, the ratio of the gradient
Method
RayS
Base
2.0
Input 16.0
Feature
17.5
Table 11: The Accuracy under RayS attack of VIT CIFAR10.
17

---

Published as a conference paper at ICLR 2024
Table 12: Robustness against score-based attacks AAA (CIFARIO).
VGG19
VIT AAA
Attack
Input
AAA
Feature
Input
Feature
Square NES
18.7
27.1
23.3
61.3
66.6
63.7
47.4
58.6
55.1
70.6
73.7
72.2
Table 13: Accuracy under white-box attacks randomized feature defense on CIFARIO
Model
Method
Acc
C&W W
PGD
VGG19
Base
96.28
7.35
3.71
Feature
94.93
34.93
6.06
VIT
Base
97.86
39.23
1.65
Feature
95.96
55.11
26.01
E.5 PERFORMANCE WHEN THE ATTACK IS LUCKY
Our empirical experiments focus on evaluating whether attack can truly find adversarial exam- ples. For non-randomized models, when attack arrives at the decision that is an "adversarial" example, æ unquestionably is on the other side of the decision boundary. However, for a randomized model. could be still on the correct side of the decision boundary, but the added randomization shifts to the other side of the decision boundary; thus, in principle, x is still not an adversarial example, and if we defense could be unlucky. Consequently, for a fair evaluation of the effectiveness of defense, we forward multiple times and decide that it is adversarial example if the majority of the results say so, as seen in our paper.
single application for evaluation,
attack could be lucky
randomized
Nevertheless, we understand that in the case where we ignore fair evaluation, the attack is allowed to be lucky and just has to fool the defense once. Therefore, we also provide the experiments when forwarding only once in Table 15 As we can observe, our defense is still effective against the attacks.
F BEHAVIOR OF MODELS WITH RANDOMIZED FEATURES
F.1 APPROXIMATION ERROR OF OUR ANALYSIS
our theoretical analysis, we apply first-order approximation to study the behavior of the model. To show that the error is negligible, calculate the difference between the loss when the input is shifted by a vector and the value computed by first-order approximation. Figure 3 illustrates the
Table 14: Robustness (CIFARIO) each layer with v corresponding to 4% clean accuracy drop.
Change of Ratio
Model
Layer
Square
NES
SignHunt
VGG
1
41.0
90.7
28.7
0.959
8
68.9
90.7
46.4
1.273
12
63.3
89.4
46.1
1.364
15
55.9
87.4
41.1
1.318
VIT
1
67.5
89.7
17.6
1.089
4
69.0
90.2
24.0
1.692
8
69.5
89.8
37.2
1.751
11
78.0
92.7
43.9
1.728
18

---

Published as a conference paper at ICLR 2024
Table 15: query of N.
The results of Square attack ViT/ImageNet with one forward each iteration and budget
Method
Acc
N=500
N=1000
N=5000
N=10000
Base 79.15
13.4
10.3
0.2
0.0
Input 78.28
48.0
46.6
44.8
44.0
Feature
78.20
48.2
47.0
45.7
44.8
20
Number of samples
15
10
5
0
-7
-6
-5
-4
-3
-2
-1
Approximation error on log scale
Figure 3: The error of first-order approximation of VIT on ImageNet at logio scale.
histogram of the error of ViT on ImageNet at logın scale, showing that our analysis is based on good approximation.
F.2 THE RATIO OF THE NORM OF THE GRADIENT
We report the ratio 2y (ate)
IV (C before and after perturbed at hidden layers of VGG19 and ViT on ImageNeuCIIFAR10 in Figure and 7 The results show that, on both datasets, when the per- turbed sample moves close to adversarial samples, the probability that randomized feature defense can fool the attacker increases while the probability of randomized input defense does change. This explains the effectiveness of randomized feature defense against score-based attacks. Figure 7 shows this ratio on VIT. On ImageNet, the robustness of the defense still increases during the tack; however, CIFAR10, such behaviors between the original and perturbed samples are not significantly different.
We also include the ratio on ResNet50 and DeiT on CIFARIO in Figure 4 observed, the ratios on ResNet50 and DeiT do not increase much during the attack, leading to small improvements.
Fand and Figure 5
As can be
F.3 THE MAGNITUDE OF THE ROBUSTNESS INPUT AND HIDDEN LAYERS
Figure 8 and show the robustness of randomized feature and randomized input defenses by multi- plying the norm of the gradient with the relative magnitude of the defense noise. As we can observe, the robustness when injecting noise to the hidden layers is generally higher than when injecting noise in the input. Such robustness behaviors are more visible the deeper layers.
19

---

Published as a conference paper at ICLR 2024
Block
Block
Block
Dersity
Dinntty &
0.3
DM
50/2)
Figure 4: The ratio of the norm of gradient of at hidden layers and at input of ResNet50 on
CIFARIO 10 before and after perturbed
Block
Block
Block 3
Block 4
Donsity
2.2
Densky
Denekt
1.5
2.0
2.5
0.6
Block
Block
Block 7
Block 8
Donsty
Density
Donsity
5.0
0.5
Black
Block
Block
Blocck 12
Dendity
Density
Densiy
Donsity
2.5
Figure 5: The ratio of the norm of the gradient of L at hidden layers and at input of DeiT on CIFARIO before and after perturbed
deck
Eedi1
Desc2
Block
Doct
hule Bark
hat
Bov
Eco031
Bato
(a) ImageNet
(b) CIFARIO
Figure 6: The ratio of the norm of the gradient of L hidden layers and at input of VGG19 ImageNes/CIFAR10 before and after perturbed
20

---

Published as a conference paper at ICLR 2024
Dieck
Block22
Blokk3
Bleek
Seckk
Blsck
Dbock 3
Bored
6.
0.3
1,0
Block
Block
Bluce
Bocc
bongit
Doop
Deeby
0.0
2.5
5.5
6.3
0.5
0.5
0-5
Block
Stock
Sooc
0.5
5.5
2.00
5:23
1:2
Rudb
naly
raby
(a) ImageNet
(b) CIFARIO
Figure 7: The ratio of the norm of the gradient of L at hidden layers and at input of ViT on Ima- geNeu/CIFAR10 before and after perturbed
BEALE
neaames
noutha
mout1en
25
25
Back
blick 14
mas 1t
2
Tet mouthee
Ne. revutran
bal roverai
nounan
Ball oearnus
fak nearte
No nésthesi
noutvai
(a) ImageNet
(b) CIFARIO
Figure 8: Distributions of the magnitude the robustness to query- query-based attacks computed at input and hidden layers of VGG19 on ImageNet/CIFAR10
Blockk
Block
Block
Block
Block
Black
Block
Brock
Dond
622
Bloce
Dick?
Sin coc
c5
rabustness
rosustress
robustnass
robustriss
rebustrss
rosustiess
robustriss
22
La
230
rotusinnss
moustness
nobudtess
mouuttnees
robustnnes
(a) ImageNet
CIFARIO
Figure 9: Distributions of the magnitude of the robustness to query-based attacks computed at input and selected hidden layers of ViT on ImageNet/CIFARI0
21


Reviews:
[{"review_id": "aOmMX4ZFJt", "reviewer": "Reviewer_pa1M", "summary": "This paper showed that adding noises to some parts of models could protect the models from query-based attacks. The authors derived proofs to show that their method (adding noises) theoretically provided robustness to the models. Besides, they experimented this method with several datasets (Imagenet and CIFAR10) and models' architectures (i.e., ResNet50, VGG19, DeiT and ViT).", "questions": "- In page 5, can you please give a reason for this sentence \"We can observe that these ratios become higher when the data are perturbed toward the adversarial samples. In other words, the randomized model is more robust during the attack.\"?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper has a strong theoretical proof to show that the method can effectively provide robustness.\n- The experiments are strong because the authors used Imagenet and CIFAR10 to show that their method and generalize in small and large datasets. Also, they tried with several models' architectures.", "weaknesses": "- I understand that the paper focuses on black-box attacks, but in the experiment section, the authors may try evaluating models with white-box attacks as well.\n- Please check the parentheses in equation (7).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7qnnh0XRFk", "reviewer": "Reviewer_NLrk", "summary": "This paper proposes to defend against black-box attacks by adding noise to intermediate features at test time. It is empirically validated effective against both score-based and decision-based attacks. The authors also provide theoretical insights on the proposed method.", "questions": "Response to rebuttal: The authors provide a strong rebuttal and a good revision of the paper. My Q1 and Q3 have been well addressed, making me raise my score to 6. Although the method differs from the CVPR 2019 paper in Q2, the novelty is weak, i.e., perturbing feature to defend has been explored for a long time.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea is straightforward, lightweight, and can be plugged into all existing defenses like adversarial training.\n2. It is great to see the theoretical analysis for the defense method.\n3. The paper is well-organized and easy to follow.\n4. The authors do comprehensive experiments to study the effectiveness of the proposed method.", "weaknesses": "1. The motivation to inject feature noise is not clear compared to injecting input noise. \"Unlike previous randomized defense approaches that solely rely on empirical evaluations to showcase effectiveness\" is not correct, since RND also provides lots of theoretical analysis as the authors acknowledged in Sec. 2.3. The results are not significantly better than RND, but injecting feature noise requires a careful choice of the layer.\n\n2. The idea of injecting noise into hidden features is not novel, seeing Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack, CVPR 2019. Although this is for defending against white-box attacks, adopting it for black-box attacks does not seem a significant contribution.\n\n3. Does the proposed method have an advantage against AAA in defending score-based attacks? AAA is not designed for decision-based attacks, where the authors use AAA for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqHYV6dDOf", "reviewer": "Reviewer_s19G", "summary": "This paper investigates a well-known defense against black-box adversarial attacks (both score-based and decision-based) which involves adding a random noise to the input. The paper argues that the robustness-accuracy trade-off of such defense can be improved by adding noise to the intermediate layer instead. Theoretical and empirical analyses are provided to support this claim.\n\n---\n\n## Comment After Rebuttal\n\nOnce again, thank you so much for acknowledging and addressing my concerns! I appreciate your efforts.\n\nBased on the new results (counting one successful attack query as a successful attack), there seems to be a minimal improvement from adding noise at the feature vs at the input. However, the result does show that the defense is effective in this difficult practical setting (~40% of samples are still correctly classified after 10k queries), and this convinces me that there are applications where this type of defense can be successfully applied.\n\nI would really appreciate it if the author(s) could include this type of results and discussion (along with other suggestions during this rebuttal period) in the next revision of the paper. After reading the other reviewers' comments, I have no further concerns and have decided to adjust my rating from 5 to 6.", "questions": "1. The paper mentions that the attack success rate is measured by “majority.” There’s an issue that I have already mentioned above, but I would like to know how many trials are used to compute this majority for the reported robustness in the experiments. If some variance can be reported too, that would be great. \n2. Section 3.3: from my understanding, the ratios plotted in Figure 3 involve both $\\nu$ and $\\nabla_{h(x)}(L \\circ g)$. I wonder how $\\nu$ is picked here. Is it picked like in Section 3.5 where the accuracy drop is fixed at some threshold (e.g., 1%, 2%)?\n3. Section 4.1: it is mentioned that Qin et al. [2021] and Byun et al. [2021] are included in the comparison, but they never show up again. I wonder if the results are really missing or if these two schemes are basically the noise addition in the input.\n4. Generally, I see a larger improvement for VGG and ViT vs ResNet-50 and DeiT. Is there any explanation or intuition the authors can provide to better understand this observation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Quality\n\nThe experiments are thorough, and the metrics are well-designed. Many models, datasets, and attack algorithms are included in the experiments. I like that the other baseline defense like AAA is also included. I also appreciate the comprehensive background section.\n\nThe paper also takes into account the nuance of picking the appropriate noise variance; they nicely solve this issue using the notion of the robustness-accuracy trade-off and pick the variance $\\nu$ that results in a small fixed drop in the clean accuracy.", "weaknesses": "### Disadvantages of randomized models\n\nI understand that the paper focuses on randomized models, but in practice, randomized models can be unattractive for two reasons:\n\n1. Its output is stochastic and unpredictable. For users or practitioners, what a randomized model entails is the fact that all predictions have some unaccounted chance of being wrong. One may argue that it is possible to average the outputs across multiple runs, but doing so would reduce the robustness and just converge back to the deterministic case (and with increased costs).\n2. **Definition of a successful attack**. This has to do with how a successful attack is defined on page 4: “…adversarial attacks are successful if the obtained adversarial example can fool the randomized model in the *majority* of its applications on the example.” I argue that in security-sensitive applications (e.g., authentication, malware detection, etc.), it is enough for the model to be fooled *once*. The randomness enables the attacker to keep submitting the same adversarial examples until by chance, the model misclassifies it.\n\nI believe that these practical disadvantages limit the significance of this line of work.  \n\n### First-order Taylor approximation\n\nAll the analysis in the paper uses the first-order Taylor approximation. First of all, this assumption should be stated more clearly. More importantly, I am not convinced that this approximation is good especially when the added noise or the perturbation is relatively large. Neural networks are generally highly nonlinear so I wonder if there is a way to justify this assumption better. An empirical evaluation of the approximation error would make all the analyses more convincing.\n\n### Method for picking the intermediate layer\n\nFirst, I wonder which intermediate layer is picked for all the results in Section 4. Do you pick the best one empirically or according to some metric? It will also be good to clearly propose a heuristic for picking the intermediate layer and measure if or how much the heuristic is inferior to the best possible choice.\n\n### Loss-maximizing attacks\n\nOne of the big questions for me is whether the attack is related to the fact that most of the black-box attacks try to **minimize the perturbation magnitude**. Because of the nature of these attacks, all the attack iterations stay very close to the decision boundary, and hence, they perform particularly poorly against the noise addition defense. In other words, these attacks are never designed for a stochastic system in the first place so they will inevitably fail.\n\nThe authors have taken some steps to adapt the attacks for the randomized defense, mimicking obvious modifications that the real adversary might do (EoT and Appendix D.4). I really like these initiatives and also wonder if there are other obvious alternatives. One that comes to mind is to use attacks that **maximize loss given a fixed $\\epsilon$ budget**. These attacks should not have to find a precise location near the decision boundary which should, in turn, make it less susceptible to the randomness.\n\nThis actually does NOT mean that the randomness is not beneficial. Suppose that the loss-maximizing attack operates by estimating gradients (via finite difference) and just doing a projected gradient descent. One way to conceptualize the effect of the added noise is a noisy gradient, i.e., turning gradient descent into *stochastic* gradient descent (SGD). SGD convergence rate is slowed down with larger noise variance so the adversary will have to either use more iterations or uses more queries per step to reduce the variance. Either way the attack becomes more costly. I suggest this as an alternative because it directly tests the benefits of the added noise without exploiting the fact that the distance-minimizing attacks assume deterministic target models.\n\n### Additional figures\n\nI have suggestions on additional figures that may help strengthen the paper.\n\n1. **Scatter plot of the robustness vs the ratio in Theorem 1.** The main claim of the paper is that the quantity in Theorem 1 positively correlates with the failure rate of the attack (and so the robustness). There are some figures that show the distribution of this quantity, but the figure that will help empirically verify this message is to plot it against the robustness (perhaps average over the test samples). Then, a linear fit and/or an empirical correlation coefficient can also be shown. Personally, this plot more clearly confirms the theoretical result than the density plot (e.g., Figure 2, etc.) or Table 6. I also think that $\\nu$ should not be fixed across layers/models and should be selected to according to the clean accuracy.\n2. **Scatter plot of the clean accuracy vs the ratio in Eq. (14)**. Similar to the first suggest, I would like to see an empirical confirmation for both of these theoretical analysis.\n3. **Robustness-accuracy trade-off plot**. This has been an important concept for evaluating any adversarial defense. I would like to see this trade-off with varying $\\nu$ as well as varying intermediate layers. The full characterization of the trade-off should also help in choosing the best intermediate layer, instead of just considering a few fixed values of $\\nu$.\n\n### Originality\n\nOne other weakness of the paper is the originality/novelty of the method. The most important contribution of this paper is the analysis on the gradient norm (i.e., sensitivity of the model) of benign and perturbed samples. The proposal to add noise to the intermediate layer instead of the input in itself is relatively incremental. However, the theoretical analysis does seem particularly strong to me, even though it does build up a nice intuition of the scheme. This is a minor weakness to me personally, and I would like to see more empirical results, as suggested earlier, rather than additional theoretical analyses.\n\n### Other minor issues\n\n- Eq. (7): the RHS is missing $L(...,y)$.\n- 2nd paragraph on page 7: I think it was slightly confusing the first time I read it. It was not immediately clear to me that this is about Eq. (14) and what “product of this value and …” refers to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNavznjzgp", "reviewer": "Reviewer_nGVZ", "summary": "This paper studies the defense method against query-based black-box attacks by injecting the noise into the middle layers of models. By theoretically analyzing the impact of both the adversarial perturbation and the noise injection on the prediction of the model, this paper tries to understand the impact of the proposed defense on robustness. Compared to the previous defense works that also inject noise into the model, the novelty of this paper is somehow in the noise injection to the feature space, i.e., the middle layer's outputs. Experimental results generally show the robustness improvement of injecting the noise to the feature rather than the input.", "questions": "1. Please verify if the Eq. (23) and the Eq. (24) are correct.\n\n2. Please verify that assumption 1 is correct and that the theorems and experiments are strictly following this assumption.\n\n3. I am curious about the impact of the ratio on the robustness when the gradients are fixed. Can you present the experimental results if possible?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The proposed method that injects the noise into the features as the defense against the query-based black-box attack is novel and is empirically shown effective. \n\n2. The adaptive attack is well-considered, which makes the evaluation more comprehensive.", "weaknesses": "1. The organization of this paper can be improved. Assumption 1, Figure 1 and 2 are not referred to in the main text. It is confusing on the purpose of presenting the assumption 1 and Figure 1 and 2. \n\n2. The assumption and the theorem are incorrect. Even when the noise is small, the expectation of the randomized model is not necessarily consistent with the original model on the same data point, one simple counter-example is that when the input x is at the decision boundary, a small perturbation can change the prediction, so small noise may change the prediction. Theorem 1 is based on incorrect derivation, Eq. (23) and (24) may be incorrect as the gradient $\\nabla_{h(x)}(L \\cdot g)$ is anisotropic so the multiplication with Gaussian noise should not be an i.i.d. Gaussian noise. In addition, the assumption of the proof is the value of v and $\\mu$ are small, so the approximation holds, but in the experiments, the value of $v$ is not present, and the value of $\\mu$ is as large as 0.3, which is not negligible.\n\n3. The correctness of Theorem 1 is not fully evaluated. The observation based on Theorem 1 is that the ratio $v/\\mu$ is a factor of the robustness, if we fix the input x, then it is the only factor that affects the robustness. In Table 3, it is observed that the robustness is not strictly correlated to the ratio, this is reasonable since the inputs are changing during the multi-step perturbation. The correctness of the influence of the ratio can be verified by trying one-step perturbation so that the input x is kept the same, which is missing in this paper.\n\n4. The evaluation of the decision-based attack is insufficient and the results are not good. It seems the proposed method only works on RayS, and the results on DeiT and ViT are not presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)