SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {H} ^ {*} = \operatorname * {a r g m i n} _ {h _ {W [ 2 ]} \in \mathcal {H}} \mathbb {E} _ {x, y \sim X, Y} [ \mathcal {L} (h _ {W [ 2 ]} (\tilde {x}), y ], \text {w h e r e} \tilde {x} = \operatorname * {a r g m a x} _ {\tilde {x}: \| \tilde {x} - x \| _ {p} \leq \epsilon_ {d}} \mathcal {L} (h _ {W [ 1 ]} (\tilde {x}), y), \\ \sum_ {i = 1} ^ {m} \sigma_ {i} \in \left[ \frac {\epsilon_ {r} ^ {\prime} - \sum_ {i = 1} ^ {m} \mu_ {i}}{\alpha \sum_ {i = 1} ^ {m} \left| W ^ {r} [ 1 ] _ {i} \right|}, B \right], \text {w h e r e} B \propto \frac {\epsilon_ {y}}{x ^ {n - 1} \cdot W ^ {n} \beta}. \tag {14} \\ \end{array}
$$
It can be estimated that the lower bound is with a high probability less than  $\frac{1}{|\alpha|}$  as a simplification of Eq. 10. Thus, based on the analysis in Section 2.2, the trade-off between natural accuracy and gradient similarity is controlled by  $|\alpha|$ . Having the lower bounds selected,  $\sigma$  is constrained by it and is checked and transformed before each forward propagation period to ensure that  $\sigma$  is in the bound.
Adversarial Defense under White-box When inference, there are two different forward propagation processes. The attackers first conduct a set of forward propagation to derive the gradients for adversarial example generation. The second forward propagation will take the generated adversarial example back to the network for inference. However, due to the random sampling strategy in randomized techniques, random weight  $W$  is resampled and thus performs a strong defense. The training and defense schemes of CTRW are demonstrated in Algorithm 1 and Algorithm 2.
Deep Neural Networks (DNNs) are vulnerable to perturbations [21, 22], resulting in the development of numerous adversarial attacks and defense algorithms [7, 8, 9, 10, 13, 23, 24, 25, 26, 14, 15, 16, 27, 28, 20, 29, 30, 31]. One common approach for generating adversarial examples in attack algorithms is the use of gradient-based methods [8, 9, 10, 20]. Adversarial samples generated by adversarial attacks can be used to expand the diversity of the dataset[32]. However, adversarial samples also pose a problem of model robustness. In recent years, several stochastic adversarial defense algorithms have been proposed that leverage random sampling to alter the gradient of the model at each inference [7, 13, 14, 33, 15, 16]. Some of these methods utilize random numbers as weights that are multiplied
Table 1: The results of robust accuracy on CIFAR-10 of adversarial defense algorithms.
Model	Method	Natural	PGD20	FGSM	AA	CW12	MIFGSM	DeepFool
ResNet18	baseline	81.84	52.16	56.70	47.69	78.46	54.96	0.35
without constrain	83.64	54.59	58.93	54.99	82.43	57.02	78.39
fixed std	82.62	67.75	65.18	72.03	81.44	65.03	81.25
CTRW (ours)	83.69	69.48	66.49	73.56	82.93	65.47	82.45
WRN34	baseline	85.27	55.06	60.65	52.24	82.24	58.47	0.40
without constrain	85.84	56.68	61.29	57.32	85.00	59.37	79.81
fixed std	84.23	70.03	66.47	72.97	83.83	65.74	83.30
CTRW (ours)	85.87	71.34	66.72	74.23	84.29	65.99	83.66
Table 2: The results of robust accuracy on CIFAR-100 of adversarial defense algorithms.
Model	Method	Natural	PGD20	FGSM	AA	CW12	MIFGSM	DeepFool
ResNet18	baseline	55.81	28.71	31.33	24.48	50.94	30.26	0.79
without constrain	57.01	30.05	32.36	30.78	55.38	31.38	51.39
fixed std	56.26	41.44	38.68	44.71	56.12	37.92	55.77
CTRW (ours)	56.63	42.01	38.85	45.58	56.32	38.20	55.56
WRN34	baseline	60.11	31.69	35.40	28.36	57.11	34.14	0.23
without constrain	60.59	31.97	34.66	32.16	58.81	33.54	54.41
fixed std	60.12	44.22	40.17	48.18	60.07	39.64	59.54
CTRW (ours)	60.62	44.33	40.33	50.47	59.73	39.90	59.30
with intermediate results of the model [7, 13]. Additionally, random noise can be incorporated into the network either as an additive or as a choice [14, 15], with the latter method adding a choice to the multiplicative noise [16]. In conclusion, using random weights is a hot research direction in adversarial defense or other tasks. However, among the existing task-heavy, one part is only applied to old or simple machine learning methods, such as SVM and LeNet, and the other part only uses random weights as a tool, instead of investigating the effect of random weights in deep neural networks on the forward and backward processes of the network [29, 30, 31]. In addition to this, there is also work in differential privacy that uses random weights[34]. Unfortunately, the design of random weights also does not take into account the optimization process with the network. While in the works above, the parameters of the random numbers, such as mean and variance, are typically pre-defined in these algorithms, Dredze et al. [35] have demonstrated that the parameters of the random sampling process can also be trained. Building on this insight, Kingma et al. [36] proposed a re-sampling-based method that enables the incorporation of trainable random weights into the neural network training process. Adversarial training is the most common method used by adversarial defense algorithms to achieve a robust model, and the early-stop algorithm has been proposed by Rice et al. [37] to prevent overfitting in this process. Adversarial training is widely employed by various adversarial defense algorithms [7, 13, 14, 15, 16]. In contrast to previous work, the focus of the present study is on training and constraining the parameters of the random sampling process, allowing for better adaptation of the process to the network and identifying an appropriate trade-off between accuracy under clean examples and adversarial attacks.
4.1 Experiment Setup
We evaluate CTRW on CIFAR [38] and Imagenet [39]. We also include various networks for evaluation, including ResNet18, ResNet50 [17], and WRN34 [18].
CIFAR In the CIFAR-10 and CIFAR-100 datasets, there are 10 and 100 classes respectively. Each CIFAR dataset contains  $5.0 \times 10^{4}$  training examples and  $1.0 \times 10^{4}$  test examples, with all images scaled to  $32 \times 32$  pixels in 3 channels. We implemented CTRW on both ResNet18 [17] and WRN34 [18] for CIFAR. In the experiment, the dataset is split into batches of size 128, and the weight decay is set to  $5.0 \times 10^{-4}$ . We use an SGD optimizer with a momentum of 0.9. The learning rate is initially set to 0.1 and reduced by a multi-step scheduler. The network is trained for 200 epochs, with the learning rate reduced by a factor of 10 at epochs 100 and 150. For adversarial training, we set  $\epsilon_{d}$  to  $\frac{8}{255}$  and the step length  $\eta$  to  $\frac{2}{255}$  for a 10-step PGD [9]. The bounds in CTRW are set to  $\alpha = 1 \times 10^{-3}$
Table 3: Comparison with SOTA methods.
Methods	CIFAR-10	Imagenet
PGD20	AA	PGD10	PGD50
RP-Ensemble [42]	60.30	-	-	-
RP-Regularizer [42]	52.39	-	-	-
RNA [14]	63.34	67.88	-	54.61
Additive Noise [15]	62.36	58.47	-	-
Overfit [37]	55.06	52.24	39.85	39.19
DWQ [43]	52.18	49.70	42.88	42.72
EnsTrans [44]	67.00	-	-	-
RobustWRN [45]	59.13	52.48	31.14	-
CTRW(ours)	71.34	74.35	59.29	60.16
Table 4: Position Study.
position in one layer	Natural	PGD20
in-channels	83.69	69.27
out-channels	78.06	59.14
both in- and out-channels	72.69	49.52
filters	68.11	55.34
Table 5: Randomness of results on CIFAR-10.
Baseline	Natural	PGD20
ResNet18	83.69 ± 0.20	69.27 ± 0.28
WRN34	85.87 ± 0.18	70.87 ± 0.31
and  $\beta = 1\times 10^{-2}$ . Thus, the lower bound of the sum of  $\sigma$  can be derived as shown in Eq. 14, and in the experiment, it is estimated as  $1\times 10^{3}$ . The model is implemented using PyTorch [40] and trained and evaluated on a single NVIDIA GeForce RTX 4090 GPU.
Imagenet The Imagenet dataset consists of  $1.2 \times 10^{6}$  training examples and  $5.0 \times 10^{4}$  test examples from 1000 different classes. The images in the dataset are 3-channel images with a size of  $224 \times 224$  pixels. CTRW is implemented on ResNet50 [17]. In the experiment, Imagenet is split into batches containing 1024 examples. The weight decay and optimizer settings are the same as those in the CIFAR dataset, but the learning rate is set to 0.02 initially and managed by a cosine annealing scheduler. During adversarial training, the network is trained for 90 epochs, using a 2-step PGD [9] to generate adversarial examples. For the PGD attack, we set  $\epsilon_{d}$  to  $\frac{4}{255}$  and the step length  $\eta$  to  $\frac{1}{255}$ . The settings of CTRW are the same as those in the CIFAR dataset. The model is implemented using PyTorch [40] and trained and evaluated on eight NVIDIA A100 Tensor Core GPUs.
Adversarial Attacks For CIFAR, we choose multiple adversarial attack algorithms, including black-box and white-box attacks from TorchAttacks [41]. The attack methods include PGD [9], FGSM [8], MIFGSM [20], Deep Fool [27], CW12 [28], and AA [10]. For PGD, we set the number of iterations to 20, and the other parameters are the same as in adversarial training. In FGSM, MIFGSM, and AA,  $\epsilon_d$  is set to  $\frac{8}{255}$ , and in MIFGSM, the step length is  $\frac{2}{255}$  with a decay of 0.1 over 5 iterations. The Deep Fool attack consists of 50 steps with an overshoot of 0.02. The CW12 attack is performed for 1000 steps with  $c = 1.0 \times 10^{-4}$ , a learning rate of 0.01, and  $\kappa = 0$ . ForImagenet, we evaluate a 10-step and a 50-step PGD attack with  $\epsilon_d$  set to  $\frac{4}{255}$
4.2 Results and Discussion
Results on CIFAR-10/100 We demonstrate the effectiveness of CTRW against various attacks and compare it with other variants, as shown in Table 1 and Table 2. To assess the effectiveness of trainable  $\mu$  and  $\sigma$ , as well as the constraints on  $\sigma$ , we added two additional sets of settings. One of them, labeled without constraint, represents the absence of any constraint on  $\sigma$  during training. In this setting, we observe that for most pixels,  $\sigma \rightarrow 0^{+}$ , meaning that random weights converge to a fixed value. On the other hand, fixed std refers to fixing  $\sigma$  at the lower bound without training, intending to maximize the probability that the network has low gradient similarity after each sample. On ResNet18, incorporating CTRW improves the accuracy of the network under PGD attack by  $17.32\%$  on CIFAR-10 compared to the baseline, and by  $13.3\%$  on CIFAR-100. Under Auto Attack, adding CTRW results in  $25.87\%$  higher accuracy than the baseline on CIFAR-10 and  $21.10\%$  higher on CIFAR-100. On WRN34, the improvement of CTRW under PGD is  $16.28\%$  on CIFAR-10 and  $12.64\%$  on CIFAR-100. Under AA, WRN34 with CTRW is  $21.99\%$  more accurate than the baseline on CIFAR-10 and  $22.11\%$  more accurate on CIFAR-100. To compare the case without bounds, we plotted the convergence of  $\sigma$  as shown in Figure 3a. The number of models is divided into two groups: group 1 is number 1 to 4, which represents the models without constrain, while group 2 is number 5 to 8, which represents the models using CTRW. Specifically, within each group, the four models represent, in order, ResNet18 and WRN34 trained on CIFAR-10, as well as ResNet18 and WRN34 trained on CIFAR-100. It can be observed that the bounds cause  $\sigma$  to converge to a smaller region with a higher value. These bounds prevent unbounded  $\sigma$  values from degrading the random weights to fixed weights, resulting in better defense effectiveness of the random weights. When comparing CTRW with the fixed std setting, it can be seen that random weights have a greater advantage when  $\sigma$  is trained. This is mainly attributed to the higher adaptability of trainable  $\sigma$ .
(b)
Figure 3: Evaluation under different settings: (a) The distribution of the final convergence of the standard deviation  $\sigma$ ; (b) Comparison of accuracy over different  $\sum \sigma$ ; (c) Robust Accuracy under different PGD strength; (d) Comparison of CTRW in different positions of one convolution layer.
Effectiveness of Bound on Robust Accuracy According to the experimental setup on CIFAR, the lower bound is set to 1000. Based on the analysis in Section 2.2, using the network weights, we can approximate the upper bound as  $B \approx 2400$ . To verify the feasibility of the proposed bounds in this work, we set a range of values for  $\sigma$ , ranging from 0 to 6 times the image size. Among these values, 0 times the image size was set to 1 to prevent the degradation of random weights into fixed weights. The results are shown in Figure 3b, with the bound highlighted in orange. An interesting observation is that the average value of an optimal  $\sum \sigma$  falls within the bound at 2048. It can be seen that once  $\sum \sigma$  exceeds the lower bound, the robust accuracy experiences a significant drop compared to the natural accuracy. Moving from  $\sum \sigma = 1024$  to 1, the natural accuracy drops from  $85.05\%$  to  $83.64\%$ , while the robust accuracy experiences a significant drop from  $67.72\%$  to  $54.59\%$ . Conversely, once  $\sum \sigma$  exceeds the upper bound, both the robust accuracy and the natural accuracy decrease simultaneously. Beyond the upper bound, the gap between robust accuracy and natural accuracy is always around  $14\%$ . These experimental observations demonstrate that the bounds defined in Equation 14 effectively help  $\sum \sigma$  take reasonable values for both two factors influencing robust accuracy.
Comparison with Other Randomization Methods on CIFAR and Imagenet To demonstrate the superiority of CTRW, several other methods utilizing full random sampling were used for comparison. RP-Ensemble [13] trains different networks on multiple random mappings of the original input, which are then combined into an ensemble. RP-Regularizer [13] adds a regularization term to the original input to confuse adversarial attacks. The experiments on RP-Ensemble and RP-Regularizer were replicated by Carbone et al. [42], producing identical results. RNA [14] proposes a method of randomly selecting the normalization layer to map the random sampling process onto the network structure. Additive Noise [15] introduces an additive to the input data directly to confuse adversarial attack algorithms. DWQ [43] integrates random accuracy training and inference, reducing undesirable adversarial transferability through randomness to enhance model accuracy under adversarial attacks. EnsTrans [44] applies randomization to the input de-noising layer and output validation layer to resist internal attacks. RobustWRN [45] proposes a WRN framework for DNNs that is more suitable for adversarial defense. To compare with these models, we used state-of-the-art adversarial training strategies [37] for training. All models are compared on CIFAR-10 with WRN34. The results of the comparison are shown in Table 3. It can be observed that CTRW outperforms other adversarial defense algorithms under all attacks.
Effectiveness of Lower Boundaries In lemma1,  $\alpha$  is specified to be less than 0. To verify the validity of this constraint, we count the number of samples that satisfy  $\cos(W^r[1], W^r[2]) \leqslant \epsilon_r$  when joining the constraint versus not joining the constraint, respectively. Samples that satisfy the condition  $\cos(W^r[1], W^r[2]) \leqslant \epsilon_r$  are called positive samples (PS), and the results are shown in Table 6. Based on the statistical results, it can be seen that without constraints, the number of samples that satisfy the condition is 0, that is,  $P(\cos(W^r[1], W^r[2]) \leqslant \epsilon_r) = 0$ . While with constraints, the number of samples that satisfy the condition among 1e5 samples are all 4e4 or more. That is, with the addition of the constraint, more samples have cosine similarity satisfying the condition of less than  $\epsilon$ . It can also be seen that  $0.5 > P(\cos(W^r[1], W^r[2]) \leqslant \epsilon_r) > F(\alpha)$ . That is, the assumption that  $\alpha < 0$  is also satisfied. In addition to this, it can be seen that the constraint in lemma1 effectively reduces the cosine similarity based on the minimization of the cosine similarity.
Robust Accuracy under Stronger PGD Attacks To verify the effectiveness of CTRW under PGD attacks of varying strength, we adjusted the number of iteration steps in PGD attacks and tested the model against PGD from 0 steps (i.e., the clear situation) up to 100 steps, with 10 iterations added
Dataset	Constrain	Number of PS	Ratio of PS	min(cos(Wr[1], Wr[2]))	Acc
CIFAR10	with (ours)	4558	0.4558	-0.0557	69.48
CIFAR10	without	0	0.0000	0.7835	54.59
CIFAR100	with (ours)	4124	0.4124	-0.0669	42.01
CIFAR100	without	0	0.0000	0.7361	30.05
Table 6: Effect of adding or not adding constraints on  $P\left( {\cos \left( {{W}^{r}\left\lbrack  1\right\rbrack  ,{W}^{r}\left\lbrack  2\right\rbrack  }\right)  \leq  {\epsilon }_{r}}\right)  > F\left( \alpha \right)$
each time. We used CIFAR-10 as the dataset and ResNet18 as the baseline in this ablation study. The results are shown in Figure 3c. According to the curves in the figure, it can be observed that CTRW remains stable at  $69\% \pm 1\%$  for all PGD attacks with more than 10 steps.
Position of CTRW Considering that the upper and lower bounds of  $\sigma$  may differ on different layers, we added random weights to different layers and compared the results. We loaded CTRW at multiple positions in ResNet18 and also added random weights to different positions in the first-layer convolution of ResNet18. It is important to note that all experiments in this section were conducted on CIFAR-10. Specifically, we changed the position of CTRW to the front of each residual layer and the Fully Connected (FC) layer in ResNet18 [17]. In Figure 3d, the numbers in the "position" column denote the number of the residual layer adjacent to CTRW, where 0 represents the first convolutional layer and 5 corresponds to the FC layer. We observed that loading CTRW earlier in the network provides more assistance in terms of robust accuracy. Thus, the best position for CTRW in the entire network is at the first convolutional layer. Furthermore, the position of CTRW around the convolutional layer is still a question to be considered. We explored various ways of adding the random weights. Apart from adding the weights before the in-channels of the convolutional layer, we also tried the out-channels and both in- and out-channels. Additionally, we placed the random weights into the convolutional layer on the filter. The results are presented in Table 4. We tested these methods under PGD-20 and clean examples without altering other settings. We found that placing the random weights in the in-channel yields the best performance. It is observed that more random weights do not lead to better performance. Therefore, to ensure the effectiveness of CTRW, the random weights should only influence the input feature map of the selected convolutional layer.
Randomness of Evaluation According to the analysis in Section 2.2, the inference results may vary each time. Therefore, to verify the accidental results, we repeated the evaluation on CIFAR-10 multiple times. We recorded the means and standard deviations of the natural and PGD20 evaluations, which are shown in Table 5. It can be observed that each network exhibits low standard deviations, indicating that the results of this work are not accidental.
Evaluation of fixed  $\sigma$  To further explore the significance of the values of  $\sigma$ , we fix the values of  $\sigma$  to the upper bound, the lower bound, and the middle of the upper and lower bounds, respectively, and compare them with CTRW. The results are shown in Table 7.
Value of σ	PGD20
Fixed to lower bound	67.75
Fixed to upper bound	66.06
Fixed to mid value	67.50
CTRW (ours)	69.48
Table 7: Robust accuracy for different values of  $\sigma$  .
In this paper, we investigate the role of trainable and restricted random weights in defending against adversarial attacks. We argue that randomness can enhance the variability of gradients between inference processes, but can also reduce the overall performance of the network. We designed CTRW and analyzed the relationship between the bounds of  $\sigma$  and its robust accuracy, and proved the validity of the bounds. CTRW is found to outperform other randomized methods. The results of this work provide valuable insights for developing strategies to exploit randomization in adversarial defenses.
This work was supported in part by the Australian Research Council under Projects DP210101859 and FT230100549.
[42] G. Carbone et al., "Robustness and interpretability of neural networks' predictions under adversarial attacks," 2023. [Online]. Available: https://arts.units.it/handle/11368/3042163
[43] Y. Fu, Q. Yu, M. Li, V. Chandra, and Y. Lin, "Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference," in International Conference on Machine Learning. PMLR, 2021, pp. 3492-3504.
[44] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. Yuille, “Mitigating adversarial effects through randomization,” arXiv preprint arXiv:1711.01991, 2017.
[45] H. Huang, Y. Wang, S. Erfani, Q. Gu, J. Bailey, and X. Ma, “Exploring architectural ingredients of adversarially robust deep neural networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 5545–5559, 2021.