SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanxiang Ma, Minjing Dong, Chang Xu*  
School of Computer Science, University of Sydney  
{yama9404, mdon0736}@uni.sydney.edu.au  
c.xu@sydney.edu.au
Deep neural networks have been found to be vulnerable in a variety of tasks. Adversarial attacks can manipulate network outputs, resulting in incorrect predictions. Adversarial defense methods aim to improve the adversarial robustness of networks by countering potential attacks. In addition to traditional defense approaches, randomized defense mechanisms have recently received increasing attention from researchers. These methods introduce different types of perturbations during the inference phase to destabilize adversarial attacks. Although promising empirical results have been demonstrated by these approaches, the defense performance is quite sensitive to the randomness parameters, which are always manually tuned without further analysis. On the contrary, we propose incorporating random weights into the optimization to exploit the potential of randomized defense fully. To perform better optimization of randomness parameters, we conduct a theoretical analysis of the connections between randomness parameters and gradient similarity as well as natural performance. From these two aspects, we suggest imposing theoretically-guided constraints on random weights during optimizations, as these weights play a critical role in balancing natural performance and adversarial robustness. We derive both the upper and lower bounds of random weight parameters by considering prediction bias and gradient similarity. In this study, we introduce the Constrained Trainable Random Weight (CTRW), which adds random weight parameters to the optimization and includes a constraint guided by the upper and lower bounds to achieve better trade-offs between natural and robust accuracy. We evaluate the effectiveness of CTRW on several datasets and benchmark convolutional neural networks. Our results indicate that our model achieves a robust accuracy approximately  $16\%$  to  $17\%$  higher than the baseline model under PGD-20 and  $22\%$  to  $25\%$  higher on Auto Attack.
Deep neural networks (DNNs) have rapidly become valuable tools for processing complex information [1, 2, 3, 4, 5]. However, their widespread usage has also increased the risk of adversarial attacks [6], making it essential to design robust learning techniques that can withstand such attacks [7]. Many works have demonstrated that DNNs are highly sensitive to small changes in input, which has led to the development of adversarial attack algorithms such as the Fast Gradient Sign Method (FGSM) [8], Projected Gradient Descent (PGD) [9] based attack, and Automatic Attack (AA) [10], among others. These white-box attacks assume that the network structure and parameters are accessible at the time of the attack, whereas they are black-box otherwise. Among them, white-box attacks are always regarded as critical threats to DNNs since the gradients are available for attack generation.
Various defense methods, such as PGD-based Adversarial Training (AT) [9], have provided the basic algorithm for training robust models against adversarial attacks. Data preprocessing techniques are also used to enhance model adversarial robustness [11]. Besides these defense mechanisms, randomized techniques have gained wide interest in improving adversarial robustness since these white-box attacks are very sensitive to gradients to some extent [12]. Increasing the randomness of the network can reduce the similarity of gradients after each inference, which ultimately lowers the success rate of attack algorithms. Recent effective adversarial defense efforts have added randomness to specific modules or intermediate results of the network, demonstrating superior performance [7, 13, 14, 15, 16].
However, existing randomized defense algorithms always treat randomness parameters such as  $\mu$  and  $\sigma$  as hyperparameters [7, 13, 15], requiring significant tuning experiments. For instance, Li et al. [15] have set the mean value as 0 and the standard deviation as 1, following the standard Gaussian distribution. Although manually tuning the randomness parameters can be effective since the defense performance is sensitive to these hyperparameters, the correlation between randomness parameters and adversarial robustness is still unexplored. Thus, instead of manually tuning the randomness parameters through methods like grid search, we propose discovering theoretically-guided randomness parameters that can achieve better adversarial robustness. Specifically, we focus on the gradients in randomized mechanisms and the prediction bias when randomness is involved. By exploring the constraints on the magnitude of randomness, we establish connections between randomness parameters and gradient similarity as well as natural performance.
Based on previous studies, we propose a defense module called Constrained Trainable Random Weight (CTRw). We update the random parameters, including  $\mu$  and  $\sigma$ , by training them in parallel with the neural network to ensure that the block can adapt to DNNs with different structures. Furthermore, we constrain the variance according to the established constraints after each update, ensuring that the variance always remains within the optimal range. In this work, we conducted sufficient evaluations to demonstrate the superiority of CTRW on different benchmarks. Moreover, we provide empirical evidence for the theoretical analysis by conducting experiments on the location of CTRW and the constraints. In our experiments under Automatic Attack (AA) on ResNet18 [17], the model with CTRW improved the robust accuracy by  $25.87\%$  over the baseline model. On the Wide Residual Network (WRN) [18], the module improved the robustness accuracy by  $21.99\%$  over the baseline and by  $6.47\%$  over the state-of-the-art (SOTA) random algorithm.
2 Randomized Parameter Optimization with Constraints
Let  $\mathbb{H}$  be a hypothesis space of classification task and we consider a classifier  $h\in \mathbb{H}$  which maps the input  $x$  to the logits  $h(x)$ . Adversarial attacks are designed to maximize the loss function  $\mathcal{L}$  with a small perturbation  $\Delta$ . Given the ground truth label  $y$  and maximum perturbation size  $\epsilon_{d}$ , the adversarial examples  $\tilde{x} = x + \Delta$  can be generated via gradient ascent in white-box setting as
$$
\tilde {x} ^ {*} = \underset {\tilde {x}: \| \tilde {x} - x \| _ {p} \leq \epsilon_ {d}} {\operatorname {a r g m a x}} \mathcal {L} (h (\tilde {x}), y), \tag {1}
$$
where the perturbation  $\Delta$  is constrained by  $l_{p}$ -norm. Traditional adversarial training algorithm performs a min-max game as
$$
h ^ {*} = \underset {h \in \mathbb {H}} {\operatorname {a r g m i n}} \mathbb {E} _ {x, y \sim X, Y} [ \mathcal {L} (h (\tilde {x}), y ], \tag {2}
$$
To further improve the adversarial robustness, randomized mechanisms are introduced to prevent attackers from deriving precise and powerful adversarial examples via the involvement of randomness during attacking and inference phases [7, 13, 14, 19]. Given two classifier  $h_1$  and  $h_2$  sampled from a subset  $\mathcal{H} \subset \mathbb{H}$ , the defense scheme can be formulated as
$$
\mathcal {H} ^ {*} = \underset {h _ {2} \in \mathcal {H}} {\operatorname {a r g m i n}} \mathbb {E} _ {x, y \sim X, Y} [ \mathcal {L} \left(h _ {2} \left(\tilde {x} _ {1} ^ {\prime}\right), y \right], \text {w h e r e} \tilde {x} _ {1} ^ {\prime} = \underset {\tilde {x} _ {1} ^ {\prime}: \| \tilde {x} _ {1} ^ {\prime} - x \| _ {p} \leq \epsilon_ {d}} {\operatorname {a r g m a x}} \mathcal {L} \left(h _ {1} \left(\tilde {x} _ {1} ^ {\prime}\right), y\right). \tag {3}
$$
Note that sampled classifiers from  $\mathcal{H}$  share all the weight parameters except for the injected noise. Thus, without additional cost, different classifiers can be randomly sampled from optimized  $\mathcal{H}^*$  during each forwarding to form a defense. The white-box attack is transferred to a black-box attack when the adversarial example  $\tilde{x}$  generated on  $h_1$  is difficult to generalize to another sampled classifier  $h_2$ .
In this work, we investigate the incorporation of a random weight during the network forwarding process, whereby layers  $h_1$  and  $h_2$  share all the weight parameters except for the random weight.
(a) Internal structure of CTRW.
(b) Pipeline of the CNNs with CTRW.
Figure 1: CTRW and the whole random adversarial defense framework pipeline.
To be specific, let  $r$  be the layer that employs the random weight, and let  $W^r$  be the weight in the  $r$ -th layer. We assume that  $W^r$  follows a normal distribution  $N(\mu, \sigma^2)$ , where  $\mu$  and  $\sigma^2$  denote the mean and variance, respectively. To ensure that the random weight is adapted to the training data, we involve  $\mu$  and  $\sigma$  in the network optimization process. Although the random sampling process is not differentiable, the parameters can be optimized via a reparameterization trick as
$$
W ^ {r} = \mu + \delta \odot \sigma , \tag {4}
$$
where  $\delta$  is sampled from a standard Gaussian distribution, i.e.  $N(0,1)$ . In  $h1$  and  $h2$ , the only difference lies in  $\delta$ , which results in different  $W^r$  values. By utilizing the technique mentioned in Equation 4, the training process of the random parameters can be visualized as shown in Figure 1a, using the pipeline depicted in Figure 1b. The variance  $\sigma^2$  plays a crucial role in determining the level of randomness as it directly controls its magnitude. To ensure the effectiveness of randomized mechanisms, a relatively large  $\sigma$  is required to guarantee the difficulty of transferring adversarial examples within the classifier subset  $\mathcal{H}$ . However, an excessively large  $\sigma$  can significantly impact the natural performance. While a naive solution for optimizing  $\sigma$  could involve adversarial training on both clean examples  $x$  and adversarial examples  $\tilde{x}$ , such as Trades [11], it incurs additional training costs on clean examples. To achieve a unified optimization of  $\sigma$ , we propose an implicit approach that imposes constraints with upper and lower bounds on  $\sigma$  to ensure both the natural performance and defense capability. Let  $W[i]$  be the weight including  $W^r$  of  $h_i$ , the objective function can be redefined as
$$
\mathcal {H} ^ {*} = \underset {h _ {W [ 2 ]} \in \mathcal {H}} {\operatorname {a r g m i n}} \mathbb {E} _ {x, y \sim X, Y} \left[ \mathcal {L} \left(h _ {W [ 2 ]} (\tilde {x}), y \right], \text {w h e r e} \tilde {x} = \underset {\tilde {x}: \| \tilde {x} - x \| _ {p} \leq \epsilon_ {d}} {\operatorname {a r g m a x}} \mathcal {L} \left(h _ {W [ 1 ]} (\tilde {x}), y\right), \right. \tag {5}
$$
$$
\begin{array}{l} \text {s . t .} \sum \sigma \in [ A, B ], \end{array}
$$
where  $\sum \sigma$  denotes the sum of every elements in  $\sigma$ ,  $A$  and  $B$  represent the lower and upper bounds of  $\sigma$ , respectively. Therefore, when  $\sigma$  is too low, the associated randomness may be insufficient, making the adversarial example  $\tilde{x}$  easily transfer between  $h1$  and  $h2$ . On the other hand, if  $\sigma$  is too high, excessive randomness can lead to performance degradation for any  $h \in \mathcal{H}$ . To understand the relationship between  $A$ ,  $B$ , and the optimization objective, selecting them through grid search is not sufficient. Thus, to better tune the bounds of  $\sigma$ , we aim to establish, from a theoretical perspective, how  $A$  and  $B$  impact the gap between  $h1$  and  $h2$ , as well as the performance degradation of any  $h \in \mathcal{H}$ . The theoretical analysis will guide for tuning the reference. Therefore, we have developed objectives for the upper and lower bounds, respectively.
2.1 Lower Bound for Gradient Similarity
The white-box attacks such as FGSM, PGD, and MIFGSM [8, 9, 20] have high success rates in white-box settings due to the direct derivation of gradients in Eq. 1. However, in randomized mechanisms, the gradients change for the same input due to the random sampling of the classifier  $h$  from  $\mathcal{H}$  during each forwarding. The success of randomized defense can be attributed to the differences in gradients, resulting in poor perturbation generation during attacks and possible ineffectiveness in inference due
to the changes in output. To quantify and analyze the relationship between the random parameter and gradient difference, we propose measuring their cosine similarity. Formally, For any  $h_1$  and  $h_2$ , the cosine similarity between gradient  $\nabla$  on  $x$  at different weights is computed as
$$
\cos \left(\nabla \left(x, W ^ {r} [ 1 ]\right), \nabla \left(x, W ^ {r} [ 2 ]\right)\right). \tag {6}
$$
The random weights are obtained through element-wise multiplication (Hadamard product) of the input of the layer in the neural network. To ensure that a large gradient difference, we require the cosine similarity of the gradients to be always less than a threshold  $\epsilon_{\nabla}$ . However, since the random weights are sampled from an unbounded distribution, we cannot guarantee that the cosine similarity will always meet this requirement, and can only provide a probabilistic bound. Assume that the random sampled weights are  $W^{r}[1]$  and  $W^{r}[2]$  respectively, the objective function can be formulated as
$$
\mu , \sigma = \underset {W ^ {r} \sim N (\mu , \sigma^ {2})} {\operatorname {a r g m a x}} P \left(\cos \left(\nabla \left(x, W ^ {r} [ 1 ]\right), \nabla \left(x, W ^ {r} [ 2 ]\right)\right) \leqslant \epsilon_ {\nabla}\right). \tag {7}
$$
Considering the random weights on the  $r$ -th layer, there exists a constant mapping between the gradient of  $x^{r-1}$  and the gradient of  $x^0$  since the network parameters are all fixed. Thus, we focus on the gradient of  $x^{r-1}$  instead of the one of  $x^0$ . Given random weights  $W^r[1]$  and  $W^r[2]$ , we denote the outputs as  $y[1]$  and  $y[2]$  respectively. According to the chain rule, the gradient can be computed as the product of the gradient of the layer after  $r$  and the gradient of the random layer. Thus, the angle  $\langle \nabla(x, W^r[1]), \nabla(x, W^r[2]) \rangle$  can be split into the sum of two angles, and then using the summation formula for the cosine function it can be derived that
$$
\begin{array}{l} \cos (\nabla (x, W ^ {r} [ 1 ]), \nabla (x, W ^ {r} [ 2 ])) = \cos (\nabla^ {r, r + 1} [ 1 ], \nabla^ {r, r + 1} [ 2 ]) \cos (\nabla^ {r + 1, n} [ 1 ], \nabla^ {r + 1, n} [ 2 ]) \\ - \sin (\nabla^ {r, r + 1} [ 1 ], \nabla^ {r, r + 1} [ 2 ]) \sin (\nabla^ {r + 1, n} [ 1 ], \nabla^ {r + 1, n} [ 2 ]), \tag {8} \\ \end{array}
$$
where  $\nabla^{i,j}[1]$  and  $\nabla^{i,j}[2]$  are the gradient of  $x^i$  propagated from  $x^j$  with  $W[1]$  and  $W[2]$  respectively. Assume that  $y[1] - y[2] \leqslant \epsilon_y \to 0^+$ , i.e. the angle  $\langle y[1], y[2] \rangle \leqslant \epsilon_{ang}^y \to 0^+$ . Since  $\nabla 1_{r+1,n}$  and  $\nabla 2_{r+1,n}$  are only related to  $y[1]$  and  $y[2]$  in a multiplicative manner, the upper bound of angle  $\langle \nabla^{r+1,n}[1], \nabla^{r+1,n}[2] \rangle$  can be limited as  $\epsilon_{ang}^{\nabla} \to 0$ . Based on Eq. 8, the cosine similarity is mainly influenced by and positively correlated with  $\cos(\nabla 1_{r,r+1}, \nabla 2_{r,r+1})$ . Note that the random layers only include the Hadamard product of random weight and the input, the derivatives  $\nabla^{r,r+1}[1]$  and  $\nabla^{r,r+1}[2]$  are exactly the random weight  $W^r[1]$ ,  $W^r[2]$ . Thus, Eq. 7 can be reformulated as
$$
\mu , \sigma = \underset {\mu , \sigma \geqslant 0} {\operatorname {a r g m a x}} P \left(\cos \left(W ^ {r} [ 1 ], W ^ {r} [ 2 ]\right) \leqslant \epsilon_ {r}\right), \text {w h e r e} W ^ {r} \sim N (\mu , \sigma^ {2}), \tag {9}
$$
where  $\epsilon_r$  is the upper bound of the cosine similarity between  $W^r[1]$  and  $W^r[2]$ , and  $\epsilon_r \to 0^+$ . Since each element in  $W^r$  obeys a Gaussian distribution, we consider using the probability cumulative function of the Gaussian distribution to constrain the probabilities in Eq. 9.
Specifically, to ensure that  $P(cos(W^r[1], W^r[2]) \leqslant \epsilon_r)$  is sufficiently large, a lower bound is devised in the form of a cumulative probability function of the standard normal distribution. This lower bound on the probability is specified as  $F(\alpha)$ . In order for  $P(cos(W^r[1], W^r[2]) \leqslant \epsilon_r) < F(\alpha)$ , the condition  $\alpha < 0$  needs to be satisfied. Although  $\alpha < 0$  seems to be a low-confidence guarantee, this bound plays an important role and its effectiveness will be empirically verified in Table 6. The proof of this part is in Eq. (10) in the first part of the supplementary material. Based on this, an upper bound on the  $\sigma$  associated with the weights can be defined.
Lemma 1. Consider a fixed  $W^r[1]$  with size of  $m$ , and a constant  $\epsilon_r$  that satisfies  $\epsilon_r = o(1)$  and an  $\alpha < 0$ . Let  $\epsilon_r' = \epsilon^r \times \|W^r[1]\|_p \|W^r[2]\|_p$ , we have  $\epsilon_r' = o(1)$ . Assume that  $\sum_{i=1}^{m} \mu_i \gg 0$ . Let  $F$  denote the cumulative function of a standard Gaussian distribution, the constraint  $\cos(W^r[1], W^r[2]) \leqslant \epsilon_r$  holds with the probability at least  $F(\alpha)$ , i.e.  $P(\cos(W^r[1], W^r[2]) \leqslant \epsilon_r) > F(\alpha)$ , if
$$
\sum_ {i = 1} ^ {m} \sigma_ {i} \geqslant \frac {\epsilon_ {r} ^ {\prime} - \sum_ {i = 1} ^ {m} \mu_ {i}}{\alpha \sum_ {i = 1} ^ {m} \left| W ^ {r} [ 1 ] _ {i} \right|} > 0, \tag {10}
$$
where  $m$  is the size of  $W^r$ , and for any variation  $a$ ,  $a_i$  denotes the  $i$ th element of  $a$ .
Due to Lemma1., when  $\sigma$  has a high enough lower bound, it is guaranteed that  $F(\alpha)$  has a large enough value such that probabilities in Eq. 9 are large enough. During training,  $\sigma$  will be checked after each optimization and added by the average bias when  $\sum \sigma$  is lower than the lower bound.
(a) Initial random weights.
(b) Projected random weights.
Figure 2: An Illustration of the projection of the random weight through different layers.
2.2 Upper Bound for Natural Performance
Section 2.1 highlights that a greater  $\sigma$  leads to a higher likelihood of low similarity between gradients. However, a network's superior robust accuracy requires not only low cosine similarity between sampled networks but also satisfactory accuracy on clean examples. While a larger  $\sigma$  may yield lower similarity between gradients, it may also result in low accuracy due to extreme randomization of network output. Hence, an upper bound needs to be imposed on  $\sigma$  to ensure satisfactory accuracy. Additionally, the assumption in Section 2.1 regarding  $|y1 - y2|$  requires high natural accuracy to hold. Both convolution and normalization being linear processes, random weights can be mapped to any layer before the Hadamard product calculation, as shown in Figure 2. After convolution, each random weight becomes a weighted sum of multiple Independent Identical Distribution (i.i.d.) Gaussian distribution, resulting in a Gaussian distribution. Let  $\mathcal{H}_{a,b}$  denotes layer  $a$  to  $b$  in  $\mathcal{H}$ ,  $W_{i,j}^{r}$  denotes the  $i$ ,  $j$ th element of  $W^{r}$ , and  $W^{l}$  denotes weight in layer  $l$ , the mapping of  $W^{r}$  between layer  $l$  and layer  $l-1$  can be can be formulated as
$$
\mathcal {H} _ {r + 1, l} \left(W _ {i, j} ^ {r}\right) = \mathcal {H} _ {r + 1, l - 1} \left(\sum_ {k = - \frac {(k s - 1)}{2}} ^ {\frac {(k s - 1)}{2}} W _ {i + k, j + l} ^ {r} * W _ {i + k, j + l} ^ {n}\right), \tag {11}
$$
where  $k_{S}$  denotes the kernel size of the convolution layer. Let  $\hat{W}^p$  denote the random weight projected to layer  $p$ , the specific form of the random weight after mapped by one convolution layer is
$$
\mathcal {H} _ {r + 1, r + 1} \left(\hat {W} _ {i, j} ^ {p}\right) \sim N \left(\sum_ {k = - \frac {(k s - 1)}{2}} ^ {\frac {(k s - 1)}{2}} \mu_ {i + k, j + l} * \hat {W} _ {i + k, j + l} ^ {p + 1}, \sum_ {k = - \frac {(k s - 1)}{2}} ^ {\frac {(k s - 1)}{2}} \left(\sigma_ {i + k, j + l} * \hat {W} _ {i + k, j + l} ^ {p + 1}\right) ^ {2}\right), \tag {12}
$$
where  $\mu_{i,j}$  and  $\sigma_{i,j}$  denotes the  $(i,j)$ th element of  $\mu$  and  $\sigma$  respectively. The mapping of  $W^r$  at layer  $l$  is obtained by performing  $l - r$  recursions on Eq. 11, and the specific form of  $\mathcal{H}_{r + 1,n}(W_{i,j}^{r})$  can be calculated by iterating Eq. 12 for  $n - r$  times. The sum of each weighted example from the Gaussian distribution still obeys the Gaussian distribution. Denoting the standard representation of the Gaussian distribution in Eq. 12 as  $\sigma'$ , we can know that  $\sigma \propto \sigma'$ . Thus,  $\sigma$  can be expressed by  $\sigma'$ .
Lemma 2. Given an  $n$  layer network and assuming ReLU to be the activation function, let the parameters of the random weights after mapping  $n - r$  times be  $\mu^{\prime}$  and  $\sigma^{\prime}$ ,  $x^{l}$  denotes the output feature map of layer  $l$  and  $W^{l}$  denotes the weights in layer  $l$ . The output difference  $Y_{1} - Y_{2}$  is hoped to be smaller than a minimal term  $\epsilon_{y}$ . The probability of  $(Y_{1} - Y_{2}) < \epsilon_{y}$  is defined as  $P((Y_1 - Y_2) < \epsilon_y)$ . And  $P((Y_{1} - Y_{2}) < \epsilon_{y})$  is larger than  $F(\frac{\sqrt{2}\beta}{2})$ , i.e.  $P((Y_{1} - Y_{2}) < \epsilon_{y}) \geqslant F(\frac{\sqrt{2}\beta}{2})$ , if  $\sum_{i = 1}^{m}\sigma_{i}^{\prime}$  is upper bounded as,
$$
\sum_ {i = 1} ^ {m} \sigma_ {i} ^ {\prime} \leqslant \frac {\epsilon_ {y}}{x ^ {n - 1} \cdot W ^ {n} \beta}, \tag {13}
$$
where  $m$  is the size of the feature map.
The  $\sigma^{\prime}$  can be obtained through  $\sigma$  according to the forward propagation process of the network. For a  $\sigma$ , the mapping of the upper bound mentioned in Eq.13 is for higher natural accuracy.
The upper bound  $B$  in Eq. 5 can be defined as the one in Lemma 2., which provides a guarantee for natural performance. In contrast, the lower bound  $A$  in Eq. 5 can be defined as the one in Lemma1.,
Algorithm 1 Adversarial Training under Con-strain Using Black-Box Attack	Algorithm 2 Adversarial Defense under White-Box Attack
Input:	Training data:X,Y; Network:h(x), with initialized weight:W, including μ and σ; Lower bound of σ:minσ; Attack iterations:t; Size of perturbation:εd	Input: Test data:X,Y; Network:h(x) with pre-trained weight:W; Attack iterations:t; Size of perturbation:εd
1:	Sample δ from a set of i.i.d.standard Gaussian distribution;	1: while not converged do
2:	while not converged do	2: Sample a batch of (x,y) ∈ (X,Y);
3:	Sample a batch of (x,y) ∈ (X,Y);	3: Re-sample δ from a set of i.i.d.standard Gaussian distribution as δ1;
4:	Initialize adversarial perturbation Δx;	4: Initialize adversarial perturbation Δx;
5:	for i←1 to t do	5: for i←1 to t do
6:	Δx = clipεd(sign(∇xL(h(x,δ),y)) · η);	6: Δx = clipεd(η · sign(∇xL(h(x + Δx,δ1),y))))
7:	end for	7: end for
8:	Re-sample δ from a set of i.i.d.standard Gaussian distribution;	8: Re-sample δ from a set of i.i.d.standard Gaussian distribution as δ2;
9:	W = W - ∇W L(h(x + Δx,δ),y);	9: y = h(x + Δx,δ2);
10:	Constrain σ by minσ;	10: end while
11:	end while	11: return y;
which provides a guarantee for a low gradient similarity. Since robust accuracy is jointly influenced by gradient similarity and natural accuracy, A and B in Eq. 5 together guarantee a high robust accuracy. Thus, it is the  $\sigma$  that is jointly constrained by A and B that keeps the randomness under control in such a way that robust accuracy is sufficiently high.
2.3 Adversarial Training and Defense
Black-box Adversarial Training To train the whole pipeline, we use black-box attack as a part of adversarial training as the defense scheme in Eq.3. The optimization can be formulated as