Adversarial Robustness through Random Weight Sampling
Authors: Yanxiang Ma
Minjing Dong
Chang Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Adversarial robustness; Randomized defense; Random parameters optimization
TL;DR: Trainable random sampling weights for adversarial defense under the constraint and achieve accuracies over 20% higher than baseline.

Abstract: Deep neural networks have been found to be vulnerable in a variety of tasks. Adversarial attacks can manipulate network outputs, resulting in incorrect predictions. Adversarial defense methods aim to improve the adversarial robustness of networks by countering potential attacks. In addition to traditional defense approaches, randomized defense mechanisms have recently received increasing attention from researchers. These methods introduce different types of perturbations during the inference phase to destabilize adversarial attacks.
Although promising empirical results have been demonstrated by these approaches, the defense performance is quite sensitive to the randomness parameters, which are always manually tuned without further analysis. On the contrary, we propose incorporating random weights into the optimization to fully exploit the potential of randomized defense. To perform better optimization of randomness parameters, we conduct a theoretical analysis of the connections between randomness parameters and gradient similarity as well as natural performance. From these two aspects, we suggest imposing theoretically-guided constraints on random weights during optimizations, as these weights play a critical role in balancing natural performance and adversarial robustness. We derive both the upper and lower bounds of random weight parameters by considering prediction bias and gradient similarity. In this study, we introduce the Constrained Trainable Random Weight (CTRW), which adds random weight parameters to the optimization and includes a constraint guided by the upper and lower bounds to achieve better trade-offs between natural and robust accuracy. We evaluate the effectiveness of CTRW on several datasets and benchmark convolutional neural networks. Our results indicate that our model achieves a robust accuracy approximately 16% to 17% higher than the baseline model under PGD-20 and 22% to 25% higher on Auto Attack.

Adversarial Robustness through Random Weight Sampling
Yanxiang Ma, Minjing Dong, Chang Xu"
School of Computer Science, University of Sydney (yama9404, mdon0736) @uni. momii.syol wyw.sy.ddddd.y..... edu. au xu@sydneyy edu. au
Abstract
Deep neural networks have been found to be vulnerable in variety of tasks. Ad- versarial attacks can manipulate network outputs, resulting in incorrect predictions. Adversarial defense methods aim to improve the adversarial robustness of networks by countering potential attacks. In addition to traditional defense approaches, randomized defense mechanisms have recently received increasing attention from researchers, These methods introduce different types of perturbations during the inference phase to destabilize adversarial attacks. Although promising empirical re- sults have been demonstrated by these approaches, the defense performance is quite sensitive to the randomness parameters, which are always manually tuned without further analysis. On the contrary, we propose incorporating random weights into the optimization to exploit the potential of randomized defense fully. To perform better optimization of randomness parameters, we conduct theoretical analysis of the connections between randomness parameters and gradient similarity as well as natural performance. From these two aspects, we suggest imposing theoretically- guided constraints on random weights during optimizations, as these weights play a critical role in balancing natural performance and adversarial robustness. We derive both the upper and lower bounds of random weight parameters by considering prediction bias and gradient similarity. In this study, introduce the Constrained Trainable Random Weight (CTRW), which adds random weight parameters to the optimization and includes constraint guided by the upper and lower bounds achieve better trade-offs between natural and robust accuracy. We evaluate the effectiveness of CTRW several datasets and benchmark convolutional neural networks. Our results indicate that our model achieves robust accuracy approx- imately 16% to 17% higher than the baseline model under PGD-20 and 22% 25% higher on Auto Attack.
Introduction
Deep neural networks (DNNs) have rapidly become valuable tools for processing complex information [22 3 4 5]. However, their widespread usage has also increased the risk of adversarial attacks [6],
making it essential to design robust learning techniques that can withstand such attacks [7]. Many works have demonstrated that DNNs are highly sensitive to small changes in input, which has led to the development of adversarial attack algorithms such as the Fast Gradient Sign Method (FGSM) [8) Projected Gradient Descent (PGD) [9] based attack, and Automatic Attack (AA) [10]. among others. These white-box attacks assume that the network structure and parameters are accessible at the time of the attack, whereas they are black-box otherwise. Among them, white-box attacks are always regarded as critical threats to DNNs since the gradients are available for attack generation.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Various defense methods, such PGID-ased Adversarial Training (AT) 2 have provided the basic algorithm for training robust models against adversarial attacks. Data preprocessing techniques are also used to enhance model adversarial robustness [А. Besides these defense mechanisms, randomized techniques have gained wide interest in improving adversarial robustness since these white-box attacks are very sensitive gradients to some extent 12. Increasing the randomness of the network can reduce the similarity of gradients after each inference, which ultimately lowers the success rate of attack algorithms. Recent effective adversarial defense efforts have added randomness to specific modules or intermediate results of the network, demonstrating superior performance [6].
7
However, existing randomized defense algorithms always treat randomness parameters such as μ and a hyperparameters 77 1 151, requiring significant tuning experiments. For instance, Li et al. [5] have set the mean value as O and the standard deviation as I, following the standard Gaussian distribution. Although manually tuning the randomness parameters can be effective since the defense performance is sensitive to these hyperparameters, the correlation between randomness parameters and adversarial robustness is still unexplored. Thus, instead of manually tuning the randomness parameters through methods like grid search, we propose discovering theoretically- guided randomness parameters that can achieve better adversarial robustness. Specifically, we focus on the gradients in randomized mechanisms and the prediction bias when randomness is involved. By exploring the constraints on the magnitude of randomness, we establish connections between randomness parameters and gradient similarity as well as natural performance.
Based on previous studies, we propose a defense module called Constrained Trainable Random Weight (CTRW). We update the random parameters, including ju and a, by training them in parallel with the neural network to ensure that the block can adapt DNNs with different structures. Furthermore, we constrain the variance according to the established constraints after each update, ensuring that the variance always remains within the optimal range. In this work, conducted sufficient evaluations to demonstrate the superiority of CTRW on different benchmarks. Moreover, we provide empirical evidence for the theoretical analysis by conducting experiments constraints. In our experiments under Automatic Attack (AA) on ResNet18 [7]. the model with
the location of CTRW and the
CTRW improved the robust accuracy by 25.87% over the baseline model. On the Wide Residual Network (WRN) [8]. the module improved the robustness accuracy by 21 21.99 over the baseline and by 6.47% over the state-of-the-art (SOTA) random algorithm.
2 Randomized Parameter Optimization with Constraints
Let HI be a hypothesis space of classification task and we consider a classifier h E H which maps the input r to the logits h(r). Adversarial attacks are designed to maximize the loss function L with small perturbation A. Given the ground truth label y and maximum perturbation size cd, the adversarial examples i = z + A can be generated via gradient ascent in white-box setting as
T*
argmax C(h(T).v),
8:168-0||
where the perturbation A is constrained by np-nomm. Traditional adversarial training algorithm performs a min-max game as
h argmin Ex (C(()))(C(()....n..) vl.
hell
To further improve the adversarial robustness, randomized mechanisms are introduced to prevent attackers from deriving precise and powerful adversarial examples via the involvement of randomness during attacking and inference phases 77]
[3] 14 19.. Given two classifier hy and h2 sampled from a subset H C H., the defense scheme can be formulated
H argmin Ex,y wtt.xxyyyychhaaa
where 7
C(h (1)))))))) 3| -1
argmax
(3)
hg
Note that sampled classifiers from H share all the weight parameters except for the injected noise. Thus, without additional cost, different classifiers can be randomly sampled from optimized H" during each forwarding to form defense. The white-box attack is transferred to black-box attack when the adversarial example ã generated on h is difficult to generalize to another sampled classifier fog.
In this work, we investigate the incorporation of random weight during the network forwarding process, whereby layers by and h2 share all the weight parameters except for the random weight.

---

Random
x-1
O Wy
DNs
TC
Wi-n W--OOR G
a
(a) Internal structure of CTRW.
(b) Pipeline of the CNNs with CTRW.
Figure 1: CTRW and the whole random adversarial defense framework pipeline.
To be specific, let r be the layer that employs the random weight, and let W be the weight in the r-th layer. We assume that W" follows normal distribution (p, o²), where jr and o² denote the mean and variance, respectively. To ensure that the random weight is adapted to the training data, we involve μ and a in the network optimization process. Although the random sampling process not differentiable, the parameters can be optimized via reparameterization trick as
W μ 8
where 5 is sampled from a standard Gaussian distribution, iee (O, h1 and h2, the only difference lies in 6, which results in different W values. By utilizing the technique mentioned in Equation[4] the training process of the random parameters can be visualized as shown in Figure Ia using the pipeline depicted in Figure 1b] The variance a² plays crucial role in determining the level of randomness as directly controls its magnitude. To ensure the effectiveness of randomized mechanisms, relatively large is required guarantee the difficulty of transferring adversarial examples within the classifier subset H. However, an excessively large can significantly impact the natural performance. While a naive solution for optimizing a could involve adversarial training on both clean examples x and adversarial examples こ, such as Trades MT. it incurs additional training costs clean examples. To achieve unified optimization of a, propose an implicit approach that imposes constraints with upper and lower bounds on a to ensure both the natural performance and defense capability. Let W[] be the weight including W of h2, the objective function can be redefined as
H argmin Ex,gax,y (C(hw where à argmax Aw EH
((hw()(0),)). 2:1l-~2|l:Sed
s.t. Σ EE
where Σ a denotes the sum of every elements in a, A and B represent the lower and upper bounds of respectively. Therefore, when is too low, the associated randomness may be insufficient, making the adversarial example easily transfer between h1 and h2. On the other hand, if is too high, excessive randomness can lead to performance degradation for any h H. To understand the relationship between A, B, and the optimization objective, selecting them through grid search is not sufficient. Thus, to better tune the bounds of how A and impact the gap between h1 and 42, as well as the performance degradation of any h H. The theoretical analysis will guide for tuning the reference. Therefore, have developed objectives for the upper and lower bounds, respectively.
aim to establish, from theoretical perspective,
2.1 Lower Bound for Gradient Similarity
The white-box attacks such as FGSM, PGD, and MIFGSM g 9 20
have high success rates in white- box settings due to the direct derivation of gradients in Eq. However, in randomized mechanisms, the gradients change for the same input due to the random sampling of the classifier h from H during each forwarding. The success of randomized defense can be attributed to the differences in gradients, resulting in poor perturbation generation during attacks and possible ineffectiveness in inference due
3

---

to the changes in output. To quantify and analyze the relationship between the random parameter and gradient difference, we propose measuring their cosine similarity. Formally, For any hı and h22 the cosine similarity between gradient V on x at different weights computed as
COs V (2, ("(1)), (az, (*())).
(6)
The random weights are obtained through element- wwie multiplication (Hadamard product) of the input of the layer in the neural network. To ensure that large gradient difference, require the cosine similarity of the gradients to be always less than a threshold eq. However, since the random weights are sampled from an unbounded distribution, we cannot guarantee that the cosine similarity will always meet this requirement, and can only provide a probabilistic bound. Assume that the random sampled weights are W ['[ and W=[2] respectively, the objective function can be formulated as
14, argmax P(cos((((a [T). (z, WT(D) < <(T). WT,N(
Considering the random weights on the r-th layer, there exists a constant mapping between the gradient of x"~1 and the gradient of xO since the network parameters are all fixed. Thus, we focus on the gradient of x"~1 instead of the one of z." Given random weights [1[1] and (2 we denote the outputs as y1] and v/2) respectively. According to the chain rule, the gradient can be computed as the product of the gradient of the layer after r and the gradient of the random layer. Thus, the angle (V (z,, WT [1]), V (z, (2)))) can be split into the sum of two angles, and then using the summation formula for the cosine function it can be derived that
cos (V (z, [T])). (ax, (2)))) cos( (Vos(V"rri V ( (20) (Vrtoi [1], V (2)
(2])
(8)
sin V [L], V ( (V V ('+))
where ['][ and V [2] are the gradient of x' propagated from x' with [[]] and [ respectively, Assume that y[1] - y/2] < Ey - 0+, i.e. the angle (w[1], y(2)} < <ing - 0+. Since V1,+1.8 and 22,+1.n are only related to y[1] and w/2] in a multiplicative manner, the upper bound of angle [V]+ (1) can be limited as <dng + O. Based on Eq. g the cosine similarity is mainly influenced by and positively correlated with cos(V (V 2 ). Note that the random layers only include the Hadamard product of random weight and the input, the derivatives V*:+t [1] and
Vi.r- [2] are exactly the random weight W ['[1], W [2]. Thus, Eq. 7 can be reformulated
14, a argmax P(cos(W" [1], (T < Er), where W NN,, ((,, P.730
where e, is the upper bound of the cosine similarity between WT[1] and WT[2]. and e, - 0+. Since each element in W obeys Gaussian distribution, we consider using the probability cumulative function of the Gaussian distribution to constrain the probabilities in Eq. [
Specifically, to ensure that P(cos(WTT]) (2] < e,,) is sufficiently large, a lower bound is devised in the form of a cumulative probability function of the standard normal distribution. This lower bound on the probability is specified as F(a). In order for P(cos((" [1*[1) [2] < c) F(a). the condition a < needs to be satisfied. Although a < 0 seems to be a low-confidence guarantee, this bound plays an important role and its effectiveness will be empirically verified in Table 6 The proof of this part is in Eq. (10) in the first part of the supplementary material. Based on this, an upper bound on the a associated with the weights can be defined.
Lemma 1. Consider a fixed W"[1) with size of m, and a constant c. that satisfies = o(1) and an a < O. Lei e, = [[WT[]]] [|WT((2 we have e' = o1). Assume that Ei u4 >> Let F de- note the cumulative function of a standard Gaussian distribution, the constraint cos [1], W [T]2 <
e, holds with the probabiliyy at least F(a),, i.e. P(cossW"(1), Wr(2)) & <r) > F(a),
m
Ea a
0
(10)
2i=1 LI WT
where is the size of Wr, and for any variation a, a; denotes the ith element of a.
Due to Lemma 1. when has high enough lower bound, it is guaranteed that F(a) has a large enough value such that probabilities in Eq. are large enough. During training, a will be checked after each optimization and added by the average bias when Σ a is lower than the lower bound.

---

W
W
comy
comfeconv
Ap1
Sx2
A+2
Ap3
x
X112
(a) Initial random weights.
(b) Projected random weights.
Figure 2: An Illustration of the projection of the random weight through different layers.
2.2
Upper Bound for Natural Performance
Section 2.J highhighss that a greater a leads higher likelihood of low similarity between gradients. However, a network's ss superior robust accuracy requires not only low cosine similarity between sampled networks but also satisfactory accuracy on clean examples. While larger may yield lower similarity between gradients, it may also result in low accuracy due to extreme randomization of network output. Hence, an upper bound needs to be imposed on a to ensure satisfactory accuracy. Additionally, the assumption in Section 2. regarding [y] - y2] requires high natural accuracy to hold. Both convolution and normalization being linear processes, random weights can be mapped to any layer before the Hadamard product calculation, as shown in Figure 2 After convolution, each random weight becomes weighted sum of multiple Independent Identical Distribution (i.i.d) Gaussian distribution, resulting in Gaussian distribution. Let Ha.b denotes layer to b in H. W denotes the i, j.hth element of WT, and W denotes weight layer the mapping W between layer and layer 1 can be can be formulated as
(ka-1)
(WT,) H,411-1
Σ (k--1)
W Wi+k * WR WHijth
(11)
where ks denotes the kernel size of the convolution layer. Let 1iP denote the random weight projected to layer p. the specific form of the random weight after mapped by one convolution layer is
(ke
(ka-1)
H. H.t1r+1(W3) N(
Pitkij+
WP++1 1-k
(attigtt WPth (Ht.)+))) (12)
A= (kx-1))
R= (k=-1)
2
where Pi.j and di.j denotes the (ih element of μ and a respectively. The mapping of W at recursions Eq. 11 and the specific form of K..+1,...77 can be calculated be iterating Eq. 12 for n r times. The sum of each weighted example from the Gaussian distribution still obeys the Gaussian distribution. Denoting the standard representation of the Gaussian distribution in Eq. 12 as o', we can know that a & o'. Thus, a can be expressed by o'.
layer / obtained by performing
Given an n. layer network and assuming RELU to be the activation function, let the parameters of the random weights after mapping n times be pí and o', x' denotes the output feature map of layer I and W denotes the weights in layer I. The output difference Yi 3i-33 Y2 hoped to be smaller than minimal term Ey. The probability of (Y1 (Y1-Y2) Y2) Cy is defined as P(( (2) < Cy).
Lemma 2.
And P((Y - Y2) < Cyy is larger than F( V28 i.e. P((Y Y2) < cg) > F( 228 Em o. is upper bounded as,
m
Ey
(13)
x
W
where is the size of the feature map.
The o' can be obtained through a according to the forward propagation process of the network. For a, the mapping of the upper bound mentioned in [3 is for higher natural accuracy. The upper bound in Eq. can defined as the one in Lemma 2.j which provides a guarantee for natural performance. In contrast, the lower bound A in Eq. 5 can be defined as the one in Lemmal
5

---

Algorithm 1 Adversarial Training under Con-Algorithm 2 Adversarial Defense under White- strain Using Black-Bot Attack
Box Attack
Input:
Training data:X, Y; Network:h((z), with Input: Test data: X, Y; Network:h(x)) with pre- trained weight:W; Attack iterations:f; Size of perturbation:ed
initialized weight:W, including μe and
Lower bound of o:ming; Attack iterations:t;
Size of perturbation:c.
Output: y
Sample 8 from a set of i.i.d. standard Gaussian
1:
while not converged do
distribution;
2:
Sample batch of (Ir, y) (X, Y);
2: while not converged do
3: Re-sample from a set of standas
3: Sample batch of (ær, y) E (X,Y)
Gaussian distribution as 51;
4: Initialize adversarial perturbation Agi
4: Initialize adversarial perturbation Ag;
5: for 1 to do
for 4- 1 to
6: A
8).
6:
Ar = clipea(1 sign( sign(VxL(h(zz +
n))
A,, (1),9))
7: end for
7:
end for
8: Re-sample 5 from set of i.i.dstandard
8: Re-sample from a set of standard
Gaussian distribution;
Gaussian distribution as 82;
9: W W Vw [(h(x Aa, 0),9);
9: y= h1( + 62)
10: Constrain a by ming;
10: end while
11: end while
11: return y;:
which provides a guarantee for low gradient similarity. Since robust accuracy is jointly influenced by gradient similarity and natural accuracy. A and B in Eq. toogethher guarantee high robust accuracy. Thus, is the a that is jointly constrained by and B that keeps the randomness under control in such a way that robust accuracy is sufficiently high.
2.3 Adversarial Training and Defense
Black-box Adversarial Training To train the whole pipeline, we use black-box attack as part of adversarial training as the defense scheme Eq The optimization can be formulated as
H = argmin Ez,gxx,y (() where bw EH
argmax 2: 2
[()w(1)())
m
m 1
(14)
s.t. Σ o,
B). where B x Ey
a
W"3
It can be estimated that the lower bound is with high probability less than Tol simplification of Eq. TO Thus, based on the analysis in Section 2.2 the trade-off between natural accuracy and gradient similarity is controlled by jal. Having the lower bounds selected, a is constrained by it and checked and transformed before each forward propagation period ensure that is in the bound.
Adversarial Defense under White-box When inference, there are two different forward propaga- tion processes. The attackers first conduct set of forward propagation to derive the gradients for adversarial example generation. The second forward propagation will take the generated adversarial example back to the network for inference. However, due to the random sampling strategy in random- ized techniques, random weight W is resampled and thus performs a strong defense. The training and defense schemes of CTRW are demonstrated in Algorithm T and Algorithm 2
3 Related Work
Deep Neural Networks (DNNs) are vulnerable perturbations of numerous adversarial attacks and defense algorithms [7
2
resulting the development
9
10
13
23
24
25
26
14 15]
16] 27
28
20 29 31) One common approach for generating adversarial examples in attack algorithms
is the use of gradient- attacks can used to expand the diversity of the dataset However, adversarial samples also pose problem of model robustness. In recent years, several stochastic adversarial defense algorithms have been proposed that leverage random sampling to alter the gradient of the model each inference
methods 8 9 10 20 Adversarial samples generated by adversarial
15 16). Some of these methods utilize random numbers weights that are multiplied 7 44 33

---

Table 1: The results of robust accuracy on CIFAR- of adversarial defense algorithms.
Model
Method
Natural
PGD)
FGSM
AA
CW12
MIFGSM
DeepFool
baseline
81.84
52.16
56.70
47.69
78.46
54.96
0.35
without constrain
83.64
54.59
58.93
54.99
82.43
57.02
78.39
ResNet18
fixed std
82.62
67.75
65.18
72.03
81.44
65.03
81.25
CTRW (ours)
83.69
69.48
66.49
73.56
82.93
65.47
82.45
baseline
85.27
55.06
60.65
52.24
82.24
58.47
0.40
without constrain fixed std
85.84
56.68
61.29
57.32
85.00
59.37
79.81
WRN34
84.23
70.03
66.47
72.97
83.83
65.74
83.30
CTRW (ours) 85.87
71.34
66.72
74.23
84.29
65.99
83.66
Table 2:
The results of robust accuracy on CIFAR-
100
adversarial defense algorithms.
Model
Method
Natural
2GD
FGSM
AA
CW12
MIFGSM
DeepFool
baseline
55.81
28.71
31.33
24.48
50.94
30.26
0.79
without constrain
57.01
30.05
32.36
30.78
55.38
31.38
51.39
ResNet18
fixed std
56.26
41.44
38.68
44.71
56.12
37.92
55.77
CTRW (ours)
56.63
42.01
38.85
45.58
56.32
38.20
55.56
baseline
60.11
31.69
35,40
28.36
57.11
34.14
0.23
without constrain fixed std
60.59
31.97
34.66
32.16
58.81
33.54
54.41
WRN34
60.12
44.22
40.17
48.18
60.07
39.64
59.54
CTRW (ours)
60.62
44.33
40.33
50.47
59.73
39.90
59.30
with intermediate results of the model 13 Additionally, random noise can be incorporated into the network either as an additive as choice [14, [5]. with the latter method adding a choice to the multiplicative noise [16]. In conclusion, using random weights is a hot research direction in adversarial defense or other tasks. However, among the existing task-heavy, one part is only applied to old or simple machine learning methods, such as SVM and LeNet, and the other part only uses random weights as a tool, instead of investigating the effect of random weights in deep neural networks on the forward and backward processes of the network (29. 30, 31. In addition to this, there is also work in differential privacy that uses random weights 34]. Unfortunately, the design of random weights also does not take into account the optimization process with the network. While in the works above, the parameters of the random numbers, such as mean and variance, typically pre-defined these algorithms, Dredze et al. 355 have demonstrated that the parameters of the random sampling process can also be trained. Building this insight, Kingma al. (36) proposed a re-sampling-bes based method that enables the incorporation of trainable random weights into the neural network training process. Adversarial training the most common method used by adversarial defense algorithms to achieve robust model, and the early-stop algorithm has been proposed by Rice et [37] to prevent overfitting in this process. Adversarial training widely employed by various adversarial defense algorithms [3 14 [5] [6]. In contrast to previous work, the focus of the present study is on training and constraining the parameters of the random sampling process, allowing for better adaptation of the process to the network and identifying an appropriate trade-off between accuracy under clean examples and adversarial attacks.
4 Experiments
4.1 Experiment Setup
We evaluate CTRW on CIFAR 38 and Imagenet 39
We also include various networks for evaluation, including ResNet18, ResNet50 [17), and WRN34 [18].
CIFAR In the CIFAR-IO and CIFAR-I00 datasets, there are 10 and 100 classes respectively. Each CIFAR dataset contains 5.0 x 10s training examples and 1.0 x 10 test examples, with all images scaled 32 x 32 pixels in 3 channels. We implemented CTRW on both ResNet18 [17] and WRN34 [18 for CIFAR. In the experiment, the dataset is split into batches of size 128, and the weight decay set to 5.0 x 10-4, We use an SGD optimizer with a momentum of 0.9. The learning rate is initially set to 0.1 and reduced by multi-step scheduler. The network is trained for 200 epochs, with the learning rate reduced by factor
at epochs 100 and 150. For adversarial training, we set 6, to 2 for a 10-step PGD [9]. The bounds in CTRW are set 1x 10-3 8 and the step length 7) to 255
255
7

---

Table 3: Comparison with SOTA methods.
Table 4: Position Study.
CIFAR- 10
Imagenet
position in one layer
Natural
PGD
Methods
PGDD
AA
PGD PGD
in-channels
83.69
69.27
RP-Enneeble
42
60.30
out-channels
78.06
59.14
RP-Regularizer
42
52.39
both in- and out-channels
72.69
49.52
RNA [14
63.34
67.88
54.61
Additive Noise
15
62.36
58.47
filters
68.11
55.34
Overfit [37
55.06
52.24
39.85
39.19
Table 5: Randomness of results on CIFAR-10.
DWQ 43
52.18
49.70
42.88
42.72
EnsTrans 4-]
67.00
Baseline
Natural
PGD
Robus(WRN 45
59.13
52.48
31.14
ResNet18 83.69 + 0.20 69.27 + 0.28
CTRW(burS)
71.34
74.35
59.29
60.16
WRN34 85.87 0.18 70.87 0.31
and 3= x 10-2, Thus, the lower bound of the sum of a can be derived as shown Eq. 14 and in the experiment, it is estimated x 10 The model is implemented using PyTorch [40 and trained and evaluated on a single NVIDIA GeForce RTX 4090 GPU.
Imagenet The Imagenet dataset consists of 1.2 x 10t training examples and 5.0 10 test examples from 1000 different classes. The images in the dataset pixels. CTRW is implemented on ResNet50 [17]. In the experiment, Imagenet is split into batches containing 1024 examples. The weight decay and optimizer settings are the same as those in the CIFAR dataset, but the learning rate is set to 0.02 initially and managed by a cosine annealing scheduler. During adversarial training, the network is trained for 90 epochs, using a 2-step PGD to generate adversarial examples. For the PGD attack, we set to 255 and the step length 255 The settings of CTRW are the same as those in the CIFAR dataset. The model is implemented using PyTorch 40) and trained and evaluated on eight NVIDIA A100 Tensor Core GPUs.
3-channel images with size of 224 x 224
Adversarial Attacks For CIFAR,
choose multiple adversarial attack algorithms, including black-box and white-box attacks from TorchAttacks 41.. The attack methods include PGD ( FGSM 8 MIFGSM (20), Deep Fool (77. CW 12 (28), and AA (10). For PGD, we set the number of iterations to 20, and the other parameters are the same as in adversarial training. In FGSM, MIFGSM, and AA, Cd is set to 255 and in MIFGSM, the step length is 255 with decay of 0.1 over 5 iterations.
The Deep Fool attack consists of 50 steps with an overshoot of 0.02. The CW12 attack is performed for 1000 steps with 1.0 10-4, learning rate of 0.01, and x O. For Imagenet, we evaluate 10-step and a 50-step PGD attack with ea set to 255
4.2 Results and Discussion
We demonstrate the effectiveness of CTRW against various attacks and compare it with other variants, as shown in Table and Table To assess the effectiveness of trainable μ and as well as the constraints a, added two additional sets of settings. One of them, labeled without constraint, represents the absence of any constraint on a during training. In this setting, observe that for most pixels, a + 0+,, meaning that random weights converge to a fixed value. On the other hand, fixed std refers to fixing a at the lower bound without training, intending maximize the probability that the network has low gradient similarity after each sample. On ResNet18, 8, incorporating CTRW improves the accuracy of the network under PGD attack by 17.32% on CIFAR- compared the baseline, and by 13.3% on CIFAR-100. Under Auto Attack, adding CTRW results in 25.87% higher accuracy than the baseline on CIFAR-10 10 and 21. 10% higher on CIFAR- 100. On WRN34, the improvement of CTRW under PGD is 16.28% CIFAR-10 and 12.64% on CIFAR- 100. Under AA, WRN34 with CTRW is 21.99% more accurate than the baseline on CIFAR-10 and 22.11 1% more accurate on CIFAR- 100. To compare the case without bounds, we plotted the convergence of a as shown in Figure 3al The number of models is divided into two groups: group is number
Results on CIFAR-10/100
which represents the models without constrain, while group 2 is number 5 to 8, which represents the models using CTRW. Specifically, within each group, the four models represent, in order, ResNet18 and WRN34 trained on CIFAR-10, as well as ResNet18 and WRN34 trained on CIFAR- 100. It can be observed that the bounds cause a to converge to a smaller region with bigher value. These bounds prevent unbounded a values from degrading the random weights to fixed weights, resulting in better defense effectiveness of the random weights. When comparing CTRW with the fixed std setting, it can be seen that random weights have greater advantage when is trained. This is mainly attributed to the higher adaptability of trainable o.
8

---

Distribution of g
73
13749
21.06
70
02-
haata
Natural Accurac
St
Robx at Accumicc
uncondraned
13.12
66.90
92
78
Resdwils
0.70
46.0
4.37
66.75
70-
0.00
60.07
52
66
of 70 28
77
Number model
positions CTRW
(a)
(b)
(c)
(d)
Figure 3: Evaluation under different settings: (a) The distribution of the final convergence of the standard deviation o; (b) Comparison of accuracy over different Σ (c) Robust Accuracy under
different PGD strength; (d) Comparison of CTRW in different positions of one convolution layer.
Effectiveness of Bound on Robust Accuracy According to the experimental setup on CIFAR, the lower bound is set to 1000. Based the analysis in Section 2.2 using the network weights, can approximate the upper bound as B x 2400. To verify the feasibility of the proposed bounds in this work, we range of values for a, ranging from 0 to 6 times the image size. Among these values, 0 times the image size was set to to prevent the degradation of random weights into fixed weights. The results are shown in Figure 3b) with the bound highlighted in orange. An interesting observation is that the average value of an optimal _ a falls within the bound at 2048. It can be seen that once Σ a exceeds the lower bound, the robust accuracy experiences a significant drop compared to the natural accuracy. Moving from Σ = 1024 to 1, the natural accuracy drops from 35.05% to 83. 33.64%, while the robust accuracy experiences significant drop from 67. 67.72% 54.59%. Conversely, once Σ exceeds the upper bound, both the robust accuracy and the natural accuracy decrease simultaneously. Beyond the upper bound, the gap between robust accuracy and natural accuracy is always around 14%. These experimental observations demonstrate that the bounds defined in Equation 14 efffecttivy help Σ a take reasonable values for both two factors influencing robust accuracy.
Comparison with Other Randomization Methods on CIFAR and Imagenet To demonstrate the superiority of CTRW, several other methods utilizing full random sampling were used for comparison. RP-Ensemble [S] trains different networks on multiple random mappings of the original input, which are then combined into an ensemble. Reggeaulrrr (13) adds regularization term the original input to confuse adversarial attacks. The experiments on RP-Ensemble and Re-Regularizer were replicated by Carbone et al. 42 producing identical results. RNA 14 proposes method of randomly selecting the normalization layer to map the random sampling process onto the network structure. Additive Noise [15]
introduces an additive to the input data directly confuse adversarial attack algorithms. DWQ 43 integrates random accuracy training and inference, reducing undesirable adversarial transferability through randomness to enhance model accuracy under adversarial attacks. EnsTrans [4 applies randomization to the input de-noising layer and output validation layer to resist internal attacks. RobustWRN 455) proposes a WRN framework for DNNs that is more suitable for adversarial defense. To compare with these models, we used sfate-of-the-arr adversarial training strategies 337 for training. All models are compared on CIFAR-10 with WRN34. The results of the comparison are shown in Table It can be observed that CTRW outperforms other adversarial defense algorithms under all attacks.
4.3 Ablation Study
Effectiveness of Lower Boundaries In lemmal, a is specified to be less than O. To verify the validity of this constraint, count the number of samples that satisfy cos( Wr(2) & e, when joining the constraint versus not joining the constraint, respectively. Samples that satisfy the condition fos(WT]] [1], (T[2)) & 6, called positive samples (PS), and the results
shown in
T Based on the statistical results, it can be seen that without constraints, the number of samples that satisfy the condition is O, that is, (Ccos(W"1p W [222)) < e,) = O. While with constraints, the number of samples that satisfy the condition among le5 samples are all 4e4 or more. That is, with the addition of the constraint, more samples have cosine similarity satisfying the condition of less than e. It can also be seen that 0.5 > ((cos(WT(1) [1), WT < 4) > F(a). That is, the assumption that a 0 is also satisfied. In addition this, can be seen that the constraint in lemmal effectively reduces the cosine similarity based on the minimization of the cosine similarity.
Robust Accuracy under Stronger PGD Attacks To verify the effectiveness of CTRW under PGD attacks of varying strength, we adjusted the number of iteration steps in PGD attacks and tested the model against PGD from 0 steps (i.c., the clear situation) up to 100 steps, with 10 iterations added
9

---

Dataset
Constrain
Number of PS
Ratio PS
Main(cos(( (os[[ W (2))
Acc
CIFARTO
with (ours)
4558
0,4558
-0.0557
69,48
CIFARIO
without
0
0.0000
0.7835
54.59
CIFARIO0
with (ours)
4124
0.4124
-0.0669
42.01
CIFARIOO
without
0
0.0000
0.7361
30.05
Table 6: Effect of adding not adding constraints P(cos( W [1]],, (2) < F(a)
each time. We used CIFAR-10 as the dataset and ResNetl8 8 as the baseline in this ablation study. The results are shown in Figure Bcl According to the curves in the figure, can be observed that CTRW remains stable at 69% Position of CTRW Considering that the upper and lower bounds of may differ different layers, we added random weights to different layers and compared the results. We loaded CTRW at multiple positions in ResNet18 and also added random weights different positions in the first-layer convolution of R. It is important to note that all experiments in this section were conducted on CIFAR-IO. 10. Specifically, we changed the position of CTRW the front of each residual layer and the Fully Connected (FC) layer in ResNet18 8 [7]. In Figure 3d the numbers in the "position" column denote the number of the residual layer adjacent to CTRW, where 0 represents the first convolutional layer and 5 corresponds to the FC layer. We observed that loading CTRW earlier in the network provides more assistance in terms of robust accuracy. Thus, the best position for CTRW in the entire network is at the first convolutional layer. Furthermore, the position of CTRW around the convolutional layer is still question to be considered. We explored various ways of adding the random weights. Apart from adding the weights before the in-channels of the convolutional layer, we also tried the out-channels and both in- and out-channels. Additionally, we placed the random weights into the convolutional layer the filter. The results are presented in Table We tested these methods under PGD-20 and clean examples without altering other settings. We found that placing the random weights in the in-channel yields the best performance. It is observed that more random weights not lead to better performance. Therefore, to ensure the effectiveness of CTRW, the random weights should only influence the input feature map of the selected convolutional layer. Randomness of Evaluation According to the analysis in Section 2.2, the inference results may vary each time. Therefore, to verify the accidental results, we repeated the evaluation on CIFAR-10 10 multiple times. We recorded the means and standard deviations of the natural and PGD20 evaluations, which are shown in Table 5 It can be observed that each network exhibits low standard deviations, indicating that the results of this work are not accidental.
1% for all PGD attacks with more than 10 steps.
Evaluation of fixed a To further explore the significance of the values of a, we fix the values of to the upper bound, the lower bound, and the middle of the upper and lower bounds, respectively, and compare them with CTRW. The results are shown Table
Value of a Fixed to lower bound
PGD20
67.75
Fixed to upper bound
66.06
Fixed to mid value
67.50
CTRW (ours)
69.48
Table 7: Robust accuracy for different values of
5 Conclusion
In this paper, we investigate the role of trainable and restricted random weights in defending against adversarial attacks. We argue that randomness can enhance the variability of gradients between inference processes, but can also reduce the overall performance of the network. We designed CTRW and analyzed the relationship between the bounds of a and its robust accuracy, and proved the validity of the bounds. CTRW found to outperform other randomized methods. The results of this work provide valuable insights for developing strategies to exploit randomization in adversarial defenses.
10

---

Acknowledgments
This work was supported in part by the Australian Research Council under Projects 10210101859 and F1230100549.
References
[1] Krizhevsky, Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," Communications of the ACM, vol. 60, no. 6, pp. 84-90, 2017.
[2] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Advances in neural information processing systems, vol. 28, 2015.
[3] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmenta- tion," in Medical Image Computing and Computer-Assister Intervention -MCCAAI 2015: 18th Internatioaal Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234-241.
[4] Dong, C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks." IEEE transactions on pattern analyxis and machine intelligence, vol. 38, no. 2, pp. 295-307, 2015.
[5] W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, "Deep convolutional neural networks for hyperspectral image classification." " Journal of Sensors, vol. 2015, pp. 1-12, 2015.
[6] I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," arXiv preprint arXiv: 1412.657,, 2014.
A. Chakraborty, M. Alam, V. Dey, Chattopadhyay, and D. Mukhopadhyay, "A survey on adversarial attacks and defences," " CAAI Transactions on Intelligence Technology, vol. 6, no. 1, pp. 25-45, 2021.
[8] C. Szegedy, W. Zaremba, 1. Sutskever, J. Bruna, D. Erhan. 1. Goodfellow, and R. Fergus, "Intriguing properties of neural networks." arXiv preprint 1rXiv:1312.6199 2013.
[9] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," arXiv preprint 1oXXix1706.060000 2017.
[10] F. Croce and M. Hein, "Reliable evaluation of adversarial robustness with ensemble of diverse parameter- free attacks," in International conference on machine learning. PMLR, 2020, pp. 2206-2216.
[11] H. Zhang, Y. Yu, J. Jiao, E. Xing, L.. El Ghaoui, and M. Jordan, "Theoretically principled trade-off between robustness and accuracy," in International conference machine learning. PMLR, 2019, pp. 7472-7482.
[12] Wang, Sun, Li, X. Yuan, W. Ni, E. Hossain, and H. Poor. "Adversarial attacks defenses in machine learning-powen networks: A contemporary survey," arXiv preprint arXiv:2303.063322 2023.
[13] G. Carbone, G. Sanguinetti, and L. Bortolussi, "Random projections for improved adversarial robustness," in 2021 International Joint Conference Neural Networks (IJCNN), 2021. pp. 1-7.
[14] M. Dong, X. Chen, Y. Wang, and C. Xu, "Random normalization aggregation for adversarial defense." Advances in Neural Information Processing Systems, vol. 35, pp. 33 676 33 688, 2022.
[15] B. Li, C. Chen, W. Wang, and L. Carin, "Certified adversarial robustness with additive noise," Advances neural information processing systems, vol. 32, 2019.
[16] K. P. Panousis, S. Chatzis, and S. Theodoridis, "Stochastic local winner-takes-all networks enable profound adversarial robustness," arXiv preprint
2021.
[17] S. Park, C.-H. Yoo, and Y.G. Shin, "Effective shortcut technique for generative adversarial networks." Applied Intelligence. vol. 53, no. 2, pp. 2055-2067, may 2022. [Online]. Available: https://doi.org/ 10.
[18] S. Zagoruyko and N. Komodakis, "Wide residual networks." arXiv preprint 1655:466 2016.
[19] Y. Fu, Q. Yu. M. Li, V. Chandra, and Y. Lin, "Double-wee win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference." in International Conference Machine Learning. PMLR, 2021, pp. 3492- 3504.

---

[20] Y. Dong, E. Liao, T. Pang, X. Hu, and J. Zhu, "Diseovering adversarial examples with momentum," arXiv preprint arXiv:/710.000 vol. 5, 2017.
[21] J. Goodfellow, Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples." " CoRR, vol. abs/1412.6572, 2014.
[22] C. Szegedy, W. Zaremba, 1. Sutskever, J. Bruna, D. Erhan, 1. Goodfellow, and R. Fergus, "Intriguing properties of neural networks." arXiv preprint 1XXiv:3312.6199 2013.
[23] Y. Li and C. Xu, "Trade-off between robustness and accuracy of vision transformers." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2023, pp. 7558-7568.
[24] M. Dong, Y. Li, Y. Wang, and C. Xu, "Adversarially robust neural architectures," arXiv preprint 2XXi:2009.00902 2020.
[25] Z. Cheng, Y. Li, M. Dong, X. Su, S. You, and C. Xu, "Neural architecture search for wide spectrum adversarial robustness," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023, pp. 442 451.
[26] M. Dong. Y. Wang, X. Chen, and C. Xu, "Towards stable and robust addernets," Advances in Neural Information Processing Systems, vol. 34, pp. 13 235-1332222 265, 2021.
[27] S.-M. Moosavi-Dezfooli A. Fawzi, and P. Frossard, "Deepfool: a simple and accurate method to fool deep neural networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2574-2582.
[28] N. Carlini and D. Wagner. "Towards evaluating the robustness of neural networks," in 2017 ieee symposium on security and privacy (sp). leee, 2017, pp. 39-57.
[29] M. Alabdulmohsin, X. Gao, and X. Zhang, "Adding robustness to support vector machines against adversarial reverse engineering." in Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, ser. CIKM '14. New York, NY. USA: Association for Computing Machinery, 2014, p. 231-240. [Online]. Available: https://doi.org//0... 145/2641166661 26622660477
[30] V. Chandrasckaran, K. Chaudhuri, I. Giacomelli, S. Jha, and S. Yan, "Exploring connections between active learning and model extraction." " in 29th USENIX Security Symposium (USENIX Security 20), 2020, pp. 1309-1326.
[31] T. Orekondy. B. Schiele, M. Fritz, "Prediction poisoning: Towards defenses against dnn model stealing attacks," arXiv preprint arXiv:1966. 10908, 2019.
[32] S. Liu, K. Wang, X. Yang, J. Yc, and X. Wang, "Dataset distillation via factorization," in Advances in Neural Information Processing Systems, 2022.
[33] M. Dong and C. Xu, "Adversarial rohustness via random projection filters," in Proceedings the the IEEFICVF Conference Computer Vision and Pattern Recognition, 2023, pp- 4077-4086.
[34] C. Dwork and A. Roth, 2014.
[35] M. Dredze, K. Crammer, and F. Pereira, "Confidence-weighted linear classification," in Proceedings of the 25th international conference Machine learning, 2008 pp. 264-271
[36] D. Kingma and M. Welling, "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 2013.
[37] L. Rice, E. Wong, and Z. Kolter, "Overfitting in adversarially robust deep learning." in Intern ational Conference on Machine Learning. PMLR, 2020, pp. 8093-8104.
[38] A. Krizhevsky, "Leaming multiple layers of features from tiny images," University of Toronto, 05 2012.
[39] K. He, X. Zhang, Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEFF conference on computer vision and pattern recognition, 2016, pp. 770-778.
[40] Paszke. S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, Killeen, Lin, N. Gimelshein, L. Antiga el al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neurul information processing systems, vol. 32, 2019.
[41] H. Kim, "Torchattacks: A pytorch repository for adversarial attacks." arXiv preprint arXiv:2010.01950. 2020.
12

---

[42] G. Carbone et al., "Rohustness and interpretability of neural networks' predictions under adversarial attacks," 2023. [Online]. Available: htttps://rrts.umm units www.u.uiiitttata.ee 1//hmmle//136///0222222 1368/3042163
[43] Y. Fu, Q. Yu, M. Li, V. Chandra, and Y. Lin, "Double-we win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference," in International Conference Machine Learning. PMLR, 2021, pp.
[44] C. Xie. J. Wang. Z. Zhang. Z. Ren, and A. Yuille, "Mitigating adversarial effects through randomization." arXiv preprint urXiv:1711.0199) 2017.
[45] Huang, Y. Wang, S. Erfani, Q. Gu, Bailey, and Ma. "Exploring architectural ingredients of adversarially robust deep neural networks," Advances in Neural Information Processing Systems, vol. 34, pp. 5545-5559, 2021.
13


Reviews:
[{"review_id": "B9q4wNgFJw", "reviewer": "Reviewer_nrzD", "summary": "This paper proposes to incorporate random weights into the optimization to exploit the potential of randomized defense. A theoretical analysis of the connections between randomness parameters and gradient similarity as well as natural performance is also provided. The method is evaluated on several datasets and benchmark CNNs to verify the utility.", "questions": "More analysis should be provided for the curves of the robust accuracy for resnet18 in Figure 3(c).\n\nThe repeated number of the experiments illustrated in Table 5 should be explicitly presented.", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper addresses a novel task and presents a unique method to handle it.\n\nThe paper is well written and easy to follow.\n\nTheoretical analysis is provided in this paper which can provide more insights in the field.\n\nThe reported results are impressive.", "weaknesses": "The method is only evaluated on CNN-based architectures. Since transformer has been widely adopted on both vision and language tasks, it  will be helpful to provide the results on transformer-based architectures.\n\nFrom Figure 3(c), we can see that for resnet18,  with the increasing of the PGD attack steps, the robust accuracy first quickly decreases and then  increases slowly after about 10 steps, can you provide some explanation about this phenomenon?\n\nThe authors state that: to verify the accidental of the results, the authors repeated the evaluation on CIFAR-10 multiple times, and the results are illustrated in Table 5. However, how many times the experiments are repeated should be explicitly presented in the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re Experiments on ViT.\n\nThanks for the suggestions regarding the experimental integrity of this work. Our proposed algorithm can be simply deployed to other neural networks, such as ViTs. For illustration, we deploy CTRW on ViT-S and evaluate the performance on CIFAR-10. The results are shown in the following table:\n\n| Method     | cw20  | $PGD^{20}$ |\n|------------|-------|------------|\n| baseline   | 34.62 | 33.49      |\n| CTRW(ours) | 45.21 | 45.68      |\n\nAs can be seen in these results, our proposed algorithm can be easily adopted by different neural networks. On ViTs, our proposed algorithm achieves better adversarial robustness than the baseline.\n\n### Re Trends in Robust Accuracy.\nThe trend of robust accuracy as shown in Fig. 3 is a constant trend in the method of designing path migration. The method designed in this paper actually makes a difference between the paths at attack time and at inference time. When the number of PGD steps is small, the gradient rise is not sufficient. At this point, as the number of PGD steps increases, the gradient rises along a path that is very close to the direction of the optimization path, which has a greater impact on the network. Therefore the adversarial robustness gradually decreases. However, when the number of PGD steps is high, the gradient rise is already sufficient, and at this time, as the gradient rises, the difference in direction of the gradient at each step is getting larger and larger compared with the gradient at the optimization corresponding to it. So the attack becomes less and less effective.\n\n### Re Times of the Experiments.\nThanks for this nice concern. All experiments in this paper were repeated 10 times and averaged. We will clearly state it in the final version.\n"}, {"review_id": "3brSCC3WmB", "reviewer": "Reviewer_kSR2", "summary": "The authors attempt to analyse the effect of the design of noise in random networks on the network and to improve the network adversarial robustness.\n\nThe authors suggest designing an interval to constrain the intensity of the noise within a desirable range.\n\nThe authors conducted experiments based on CNNs and the results support these conclusions.", "questions": "Please see Weakness part", "limitations": "This is a solid paper that presents a complete analysis and interesting results. The paper provides novel ideas for random network design. As an improvement, the links to the code is needed for more adequate support to the data in the paper.\n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) The paper attempts to answer the question of the relationship between adversarial robustness of stochastic networks and noise. This is one of the topics of the NeurIPS audience.\n\nS2) This paper provides a mathematical analysis that explains, from a theoretical point of view, the effect of random noise strength on network performance. The results of the analysis are also interesting: an interval is used to constrain the noise to achieve relatively desirable results.\n\nS3) This work shows that the optimisation of noise in random networks has a large impact on network adversarial robustness. Their success also provides an interesting way of thinking about the problem of random network design.\n\nS3) The results are interesting and intuitive, and consistent with the theoretical analysis.", "weaknesses": "W1) Lack of code usability statements, although the network structure is relatively simple.\n\nW2) Lack of evaluation under non-adversarial training may be a potential weakness, but existing evaluations are sufficiently representative.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re Code usability.\nThanks for this nice concern about code usability. According to this year's rules, the code about our method has been given to Area Chairs in the form of an anonymous link.\n\n### Re Lack of experiment.\nThanks for this nice concern. It is interesting to see the performance without adversarial training. We conduct evaluation with ResNet-18 on CIFAR-10. All the algorithms are trained with natural training. The results are shown in the following table:\n\n| Method     | Natural | $PGD^{20}$ |\n|------------|---------|------------|\n| baseline   | 84.17   | 0.00       |\n| CTRW(ours) | 84.61   | 3.44       |\n\n\n\nAs shown in the table, our proposed algorithm can achieve better adversarial robustness than the baselines, which demonstrates the effectiveness of our algorithm. Compared with the results after adversarial training, the results without adversarial training cannot be competitive. Thus, adversarial training is still a necessary training strategy in our algorithm."}, {"review_id": "CsiAwHmhrR", "reviewer": "Reviewer_8U3B", "summary": "This paper proposes a random injection approach to improve the adversarial robustness against attacks. Different from previous work, the proposed algorithm includes random weights in the optimization and imposes constraints for better trade-offs. The constraints rely on proving the following Lemma:\n\n1.(Lemma 1). The variance of distribution where the noise is sampled is lower bounded to ensure lower gradient similarity after sampling.\n\n2.(Lemma 2). The variance of distribution is also upper bounded to ensure lower natural classification error.\n\nThe proposed algorithm combines the lower and upper bound to perform adversarial training with random weights under constraints. Several experiments are provided to verify the effectiveness of the proposed algorithm.\n", "questions": "I think more clarification and evaluation are needed for the concerns mentioned in the Weakness section. I would appreciate the authors making amends along those directions.", "limitations": "The evaluation of proposed algorithm under black-box attacks is relatively limited.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The theoretical analysis provides good insight that leads to the proposed constraints, which could benefit randomized defense community.\n\n2.The evaluation is conducted on various datasets and models, and the proposed algorithm achieves reasonable improvement over baselines.\n", "weaknesses": "1.Lack of assumption before or in the lemma, which makes it difficult to verify the scope.\n\n2.Although the proposed algorithm proposes to perform adversarial training in a black-box manner, there is no evaluation of black-box attacks in the experiment section. The authors should conduct more experiments to verify the performance of proposed algorithm under popular black-box attacks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re Lack of assumption.\nThanks for the suggestions on the lack of rigor in our theory. Lemma2 inference requires that the nonlinear layer maintains the conditions mentioned in Equation 14 in the supplementary material. In addition the lower bound of $\\sigma$ should be greater than 0 to ensure the randomness of the random weights. We will add this section to the text in the final version to ensure that the theory is as rigorous as possible.\n\n### Re Lack of black-box attack.\nThanks for the suggestion. We include more evaluation of black-box attacks, such as Square and Pixle. We conduct evaluation ResNet-18 on CIFAR-10. The results are shown in the following Table:\n\n| Method     | Square | Pixle |\n|------------|--------|-------|\n| baseline   | 54.68  | 8.10  |\n| CTRW(ours) | 77.73  | 72.14 |\n\n\nAs shown in the table, our proposed algorithm achieves better performance under black-box attacks, especially under Pixle attack.\n"}, {"review_id": "tJcntpfJBF", "reviewer": "Reviewer_1Bpz", "summary": "The paper studies randomizing the weights in a neural network during inference to improve adversarial robustness. For example, the weights can be sampled from a distribution $\\mathcal{N}(\\mu,\\sigma)$, for some learned parameters $\\mu$ and $\\sigma$. Intuitively, randomizing the weights makes the task of generating adversarial examples harder because it increases the gradient noise during inference.\n\nThe traditional learning setup corresponds to the case where $\\sigma=0$ and we learn $\\mu$ alone (here, I'm using $\\mu$ for the mean of the weights, not the mean of the additive noise). In some prior works, $\\sigma$ is selected based on a grid search. In this paper, on the other hand, the authors propose learning $\\sigma$ using the reparameterization trick $w = \\mu + \\sigma\\delta$, with $\\delta\\sim\\mathcal{N}(0,I)$. This by itself would encourage less variance (i.e. $\\sigma\\approx 0$) (to have a better fit on the training data) so the authors constrain $\\sigma$ in a bounded region $[A, B]$, where $A$ and $B$ are chosen based on some theoretical arguments. \n\nFinally, the authors evaluate this approach on CIFAR10/100 and ImageNet using three ResNet architectures (ResNet18, ResNet50, and WRN34). They report improved robustness compared to three baselines: $\\sigma=0$, fixing $\\sigma$ to its lower bound, and training $\\sigma$ without constraints.  ", "questions": "- Can you please provide a precise statement of the two lemmas? Please see my comments above. \n\n- Have you compared against other fixed values of $\\sigma$ (e.g. as suggested above)? \n\n- In Line 273, you mention that $A=1000$ and $B=2400$. Is this independent of the architecture? ", "limitations": "N/A", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The empirical results are quite positive, showing a big improvement in robustness compared to the baselines.\n- The paper extends prior works by proposing to learn the noise distribution during training.", "weaknesses": "The idea of randomizing weights for robustness is not new and has been studied much earlier dating back to SVM and Differential Privacy (see for example [1, 2, 3, 4]). The primary contribution in this work is to propose a bound on the noise variance. However, the theoretical result do not appear to be sound, and I would appreciate a clarification from the authors on this please. \n\nLet us take Lemma 1 for example. If I choose $\\varepsilon_r'=\\sum_i\\mu_i$ (which is perfectly acceptable), the lower bound says that setting $\\sigma=0$ is enough to guarantee that the cosine similarity is small. But, $\\sigma=0$ means that the cosine similarity is at its maximum.\n\nIn general, the paper contains many claims that are not precisely stated and makes it hard to follow. Again, we can take the main lemmas (Lemma 1 and 2) as examples. In Lemma 1, the authors say that the \"constraint $cos(...)\\le \\epsilon \\to 0$ holds with a probability of at least $F(\\alpha)$. What does  $\\to 0$ in the statement of the constraint mean? Also, in the same lemma, the authors write $\\varepsilon'\\to0$ when defining the symbol in Equation 10 even though it has a precise definition stated later. In Lemma 2, the authors state that \"The probability that [some event] is hoped to be bigger than [some quantity]. The noise parameter $\\sigma$ [some bound]\". I'm not really sure how to read this lemma. Given that these two lemmas are the main contribution in the paper, the authors should state them precisely. In the current form, my understanding is an educated guess. \n\nThe issue with presentation is also there throughout the paper. Here are some examples. \n\n- Line 32: Projected Gradient Descent (PGD) is not an adversarial attack algorithm.\n\n- Line 47: The cited paper [15] does NOT fix $\\sigma$ to 1. In fact, they study what happens as $\\sigma$ is varied and discuss the same tradeoffs mentioned in this paper. \n\n- Throughout the papers, the authors sometimes use $\\mu$ when referring to the mean of the noise (as in Line 45) and sometimes use it as the mean of the parameters (as in Line 85).\n\n- Writing $x^\\star = \\arg\\min_{x^\\star} f(x^\\star)$ is confusing and wrong (see Equations 1 and 3). It should be written with different symbols:   $x^\\star = \\arg\\min_{x} f(x)$.\n\n- The authors sometimes refer to the additive noise as \"random weight\" (see Line 83)\" and later refer to the sampled weights ($\\mu$ plus noise) as random parameters (see Line 91).\n\n- The paper uses non-standard symbols, such as using $G$ for the gradient instead of $\\nabla$. \n\n- Typos: e.g. \"logist\" in Line 69, \"ever element\" in Line 102, \"exceed\" (instead of the opposite) in Line 122, $Wr$ instead of $W^r$ in Line 147 ... \n\n\nIn terms of the experiments, I'm curious to know why the authors choose to compare against a single fixed value of $\\sigma$. If you propose to constrain it to some interval $[A, B]$, it would make sense to compare the method against fixing $\\sigma$ to $A$, $B$, and the middle $(A+B)/2$. The reason I bring this up is because the gap is already small between learning $\\sigma$ and fixing it to the lower bound $A$ in Tables 1 and 2. \n\n[1] Cynthia D, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 2014.\n\n[2] Alabdulmohsin, I, et al. \"Adding robustness to support vector machines against adversarial reverse engineering.\" 2014.\n\n[3] Chandrasekaran, V, et al. \"Exploring connections between active learning and model extraction.\" 2020.\n\n[4] Orekondy, T, et al. \"Prediction poisoning: Towards defenses against dnn model stealing attacks.\" 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re Novelty.\nThanks for providing relevant works of randomness [1-4]. We will discuss these works in the related work in our final version. However, we argue that the contribution of our work is sufficient compared with [1-4]. We will clarify the novelty in two aspects.\n\nFirst, the suggested works [1-4] apply randomized weights to some traditional DL/ML models, such as SVM, LeNet, etc., which are out-of-date in current adversarial robustness evaluation. In contrast, we deploy the randomized weight to ResNet and Wide ResNet so that the randomized defense can achieve superior performance compared with state-of-the-art algorithms. Furthermore, the direct deployment of randomized weights in these popular neural networks cannot achieve satisfactory performance, as discussed in the paper. The robustness performance is related to the noise design (Table 1-3), random weight location (Figure 2), etc. Thus, exploring a better way to incorporate randomized weight into modern complex neural networks can be important. In this work, we focus on the complex neural networks, such as ResNet and  Wide ResNet. Based on the analysis of the influence of randomness parameters on the gradient similarity and output difference (Lemma 1 and 2), we introduce a novel noise design for complex DNNs.\n\nSecond, the major difference between our proposed randomized weights and those in previous works including  [1-4] is that we have a novel way to implement a learnable random weight, while the suggested works often decide the value of $\\sigma$ manually. For example, in [2], while using a simple network like SVM, the tuning method is to manually tune $\\sigma$ as a hyper parameter. In contrast, we maintain a learnable $\\mu$ and $\\sigma$. This means the parameters $\\mu$ and $\\sigma$ will be tuned automatically at the back propagation process while training the network. Furthermore, during the optimization of $\\mu$ and $\\sigma$, we impose the constraints on them according to the theoretical analysis in Lemma 1 and 2. In particular, to maximize the probability of having a relatively high natural accuracy and relatively low cosine similarity, we proposed an upper bound and a lower bound.\n\n### Re $\\sigma$ in Lemma1.\nWe modify Eq. 10 as: $\\sum_{i=1}^{m}\\sigma_i\\geqslant\\frac{\\epsilon_r^\\prime - \\sum_{i=1}^m\\mu_i}{\\alpha\\sum_{i=1}^{m}|W^r[1]_i|}>0$,\n\nwhich does not include the special case of $\\sum_{i = 1}^{m}\\sigma_i=0 $. Because when $\\sum_{i = 1}^{m}\\sigma_i=0$, all the $\\sigma_i$ are $0$. This means $W^r$, whose standard deviation is $\\sigma$, will lose randomness. In this case, $W^r$ will be the same during each inference. However, the motivation of this paper is to use the randomized weight to establish different routes between each inference process, which needs $W^r$ to be different each time. So the situation that $\\sum_{i = 1}^{m}\\sigma_i=0$ does not fit the motivation of this paper.\n\n### Re $\\to$ in Lemma 1. \n$\\epsilon\\to0^+$ denotes that $\\epsilon$ is a constant close to 0 from and is positive. $\\to$ in $\\epsilon\\prime \\to 0^+$ denotes the same. We use this symbol as a simplified version of $\\lim\\limits_{\\epsilon \\to 0^+}$.\n\n### Re Claims in Lemma 2.\nWe provide a more precise statement. We denote the probability that the difference of output is smaller than a minimal term $\\epsilon_y$ as $P((Y_1-Y_2) < epsilon_y)$, and this probability is expected to be larger than $ F(\\sqrt{2}\\beta/2)$. Formally, we have $P((Y_1-Y_2) < \\epsilon_y) \\geqslant F (\\sqrt{2}\\beta/2)$.\n\n### Re Presentation. \nThanks for pointing out these presentation issues.\n* We understand that PGD is an optimization algorithm. But in the field of adversarial attack, it has been used in recent work on adversarial attacks. And in recent research work related to adversarial defensiveness, this method is considered as an important method to verify the effectiveness of adversarial defences.\n\n* We understand that the $\\sigma$ in the altered article is treated as tuned as a hyper-parameter. But what we are trying to convey here is that the $\\sigma$ of that work is not learnable.\n\n* The random weights proposed in this paper are actually a Gaussiannoise. At both line 45 and line 85, $mu$ is the mean value of the Gaussian distribution $N(\\mu, \\sigma)$. At line 85, the random weight $W^r$ introduced is the noise N($\\mu$, $\\sigma$) is mentioned at this point to show that the random weight is a Gaussian noise.\n\n* We will correct the expression errors in those formulas in the final version. We will take a different notation.\n\n* $\\mu$ and $\\delta \\odot \\sigma$ as a whole make up the random weight. This whole is $W^r$ , which is the \"random weight\" in the phrase \"except for the random weight\" at line 83.\n\n* Thank you for the correction to our notation specification. Those notations will be revised in the final version.\n\n* Thank you for pointing this out. We will recheck the grammar, words and symbols and fix the typos in the published manuscript.\n\n### Re Line 273\nThe values taken here for the upper and lower bounds are based on preliminary experiments conducted on networks that did not incorporate random weights. The results about fixing $\\sigma$ on those value mentioned will be\n shown at the rebuttal about question2.\n\n### Re Questions\n\n**Q1.**  We will rewrite Lemma 1 and Lemma 2 according to the reply above.\n\n**Q2.** Related experiments have been added and the results are shown as:\n\n|Value of $\\sigma$|$PGD^{20}$|\n|-|-|\n|Fixed to lower bound| 67.75|\n|Fixed to upper bound| 66.06|\n|Fixed to mid value| 67.50|\n|CTRW (ours)| 69.48 |\n\nOur algorithm achieves the best performance, which demonstrates the necessity of trainable randomness parameters.\n\n**Q3.** \nThe values taken here for the upper and lower bounds are based on preliminary experiments conducted on networks that did not incorporate random weights."}]

Author rebuttal:

Meta-review: This work proposes an efficient Black-box Adversarial Training method called Constrained Trainable Random Weight (CTRW). It incorporates random weight parameters into the optimization process and includes theoretically-guided bounds to ensure better trade-offs. The reviewers acknowledge the valuable insights provided by the theoretical analysis. However, they also raise concerns about experimental limitations (Reviewer nrzD, kSR2, 8U3B), as well as concerns regarding a lemma (Reviewer 8U3B, 1Bpz) and novelty (Reviewer 1Bpz). Following the rebuttal, the concerns raised by Reviewers 8U3B and kSR2 have been addressed, resulting in an increased rating of "Accept" and "Strong Accept," respectively. However, Reviewer 1Bpz still expresses concerns about the correctness of Lemma 1 and Lemma 2 even after the rebuttal and maintains his/her rejection score.

One of the main concerns raised by Reviewer 1Bpz is regarding Lemma 1. The lemma holds when $\epsilon_r'-\sum_i \mu_i+\delta$ for any $\delta>0$. However, as $\delta\rightarrow 0^+$, the condition is satisfied by choosing $\sigma\rightarrow 0^+$, which implies that the cosine similarity approaches 1. The authors argue that in practice, $\delta\rightarrow 0^+$ does not exist since $\epsilon_r'\rightarrow 0$ and $\sum_i \mu_i \nrightarrow 0$. In response, the reviewer points out the contradiction of $\epsilon_r'-\sum_i \mu_i > 0$ since $\sum_i \mu_i$ can always be positive. Then, the authors mention that $\alpha$ is negative, but the reviewer insists that $\alpha$ is freely chosen based on the desired level of confidence. Ultimately, the authors state that their theory can only be applied when $\alpha < 0$, meaning confidence is less than 0.5.

There are two main limitations for the theory: 1) Lemma 1 only studies the probability of $cos(W1, W2) < \epsilon_r$ where $\epsilon_r\rightarrow 0^+$, i.e., $W1$ and $W2$ being completely different. The assumption $\epsilon_r\rightarrow 0$ is too strong; 2) The theory is only valid under the condition of $\alpha < 0$, which is not sufficiently general.

On the other hand, these lemmas also have positive aspects, as noted by the reviewer: 1) This paper attempts to answer the question of the relationship between the adversarial robustness of stochastic networks and noise, which is interesting and warrants further exploration; 2) The results of the theoretical analysis are also interesting: an interval is used to constrain the noise to achieve relatively desirable results. Based on the high praise from Reviewers nrzD, kSR2, and 8U3B regarding the theoretical findings (resulting in weak accept, accept, and strong accept scores), as well as the practical application of these derived results in Adversarial Training, which has led to significant improvements, I lean towards acceptance. I also suggest that the authors revise the paper according to the suggestions of the reviewers and further justify Lemma 1 and Lemma 2 in the final version.
Decision: Accept (poster)