SlowGuess/NeurIPS_2023_OCR huggingface dataset:
5 Analysis of Adversarially Robust Generalization
As mentioned in the introduction, the robust generalization gap is much larger than the standard generalization gap in practical scenarios. What factors contribute to such a significant difference? Previous norm-based bounds might lead to the following hypothesis: The significant disparity could potentially be attributed to the additional terms or assumptions between the standard bound and the robust bound. Our result provides a different perspective: They are solely due to mathematical considerations. The following three factors are (implicitly) different in Theorem 1 and Theorem 2 and possibly contribute to the significant disparity.
Clean Sample and Adversarial Example ( $B$  and  $B + \epsilon$ ). The only difference between the bounds in Theorem 1 and Theorem 2 lies in the factor  $\epsilon$ . In this context,  $B$  represents the magnitude of clean samples, while  $B + \epsilon$  signifies the magnitude of adversarial examples. This factor holds less significance in improving robust generalization, as it is unlikely to be controlled during the training of DNNs.
Standard Margin and Robust Margin  $(\gamma)$ . The margin  $\gamma$  remains consistent in both of these two bounds, but it is implicitly different in the definitions of standard margin loss and robust margin loss. The robust margin is smaller due to the smaller distance between two adversarial examples. As it is discussed in (Neyshabur et al., 2017a),  $\gamma$  is usually considered to normalize the spectral complexity discussed below.
Standard-Trained and Adversarily-Trained Parameters  $(\Phi(f_{\mathbf{w}}))$ . The spectral complexity  $\Phi(f_{\mathbf{w}})$  is implicitly different because the weights  $\mathbf{w}$  of the standard-trained and adversarially-trained models are distinct. The spectral complexity  $\Phi(f_{\mathbf{w}})$  induced by adversarial training is significantly larger. We conducted experiments training MNIST, CIFAR-10, and CIFAR-100 datasets on VGG networks, see Appendix C. See also the work of (Xiao et al., 2022a) for more discussion about the experiments of weights norm of adversarially-trained models. The margin-normalized spectral complexity  $\Phi(f_{\mathbf{w}})$  likely contributes to the huge difference between standard generalization and robust generalization.
6 Main Challenge of Robust Generalization Bound and Proof Sketch
6.1 PAC-Bayesian Framework
The PAC-Bayesian framework (McAllester, 1999) provides generalization guarantees for randomized predictors drawn from a learned distribution  $Q$  (as opposed to a single predictor) that depends on the training data set. In particular, let  $f_{\mathbf{w}}$  be a predictor parameterized by  $\mathbf{w}$ . We consider the distribution  $Q$  over predictors of the form  $f_{\mathbf{w} + \mathbf{u}}$ , where  $\mathbf{u}$  is a random variable and  $\mathbf{w}$  is considered to be fixed. Given a prior distribution  $P$  over the set of predictors that is independent of the training data, the PAC-Bayes theorem states that with probability at least  $1 - \delta$ , the expected loss of  $f_{\mathbf{w} + \mathbf{u}}$  can be bounded as follows
$$
\mathbb {E} _ {\mathbf {u}} \left[ L _ {0} \left(f _ {\mathbf {w} + \mathbf {u}}\right) \right] \leq \mathbb {E} _ {\mathbf {u}} \left[ \widehat {L} _ {0} \left(f _ {\mathbf {w} + \mathbf {u}}\right) \right] + 2 \sqrt {\frac {2 \left(K L \left(\mathbf {w} + \mathbf {u} \| P\right) + \ln \frac {2 m}{\delta}\right)}{m - 1}}. \tag {5}
$$
To get a bound on the margin loss  $L_{0}(f_{\mathbf{w}})$  for a single predictor  $f_{\mathbf{w}}$ , we need to relate the expected loss,  $\mathbb{E}_{\mathbf{u}}[L_0(f_{\mathbf{w} + \mathbf{u}})]$  over a distribution  $Q$ , with the loss  $L_{0}(f_{\mathbf{w}})$  for a single model. The following lemma provides this relation.
Lemma 4 (Neyshabur et al. (2017b)). Let  $f_{\mathbf{w}}(\mathbf{x}) : \mathcal{X} \to \mathbb{R}^k$  be any predictor (not necessarily a neural network) with parameters  $\mathbf{w}$ , and  $P$  be any distribution on the parameters that is independent of the training data. Then, for any  $\gamma, \delta > 0$ , with probability  $\geq 1 - \delta$  over the training set of size  $m$ , for any  $\mathbf{w}$ , and any random perturbation  $\mathbf{u}$  s.t.  $\mathbb{P}_{\mathbf{u}}\left[\max_{\mathbf{x} \in \mathcal{X}} |f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})|_{\infty} < \frac{\gamma}{4}\right] \geq \frac{1}{2}$ , we have:
$$
L _ {0} (f _ {\mathbf {w}}) \leq \widehat {L} _ {\gamma} (f _ {\mathbf {w}}) + 4 \sqrt {\frac {K L (\mathbf {w} + \mathbf {u} \| P) + \ln \frac {6 m}{\delta}}{m - 1}}.
$$
As it is discussed in (Neyshabur et al., 2017a), the KL-divergence is evaluated for a fixed  $\mathbf{w}$  and  $\mathbf{u}$  is random. Lemma 4 is not specific to neural networks and generally holds for any functions. Providing Lemma 4, it is left to provide a bound of  $\| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})\|_2$  to obtain the final generalization bound. $^3$  This framework can be directly extended to adversarially robust settings by replacing  $\| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})\|_2$  by  $\| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x} + \delta_{\mathbf{w} + \mathbf{u}}^{adv}(\mathbf{x})) - f_{\mathbf{w}}(\mathbf{x} + \delta_{\mathbf{w}}^{adv}(\mathbf{x}))\|_2$  (Farnia et al., 2018). For more details, see Appendix B.
6.2 Main Challenge
Based on Lemma 4, to provide an upper bound of robust margin loss is to solve the following problem:
Problem 1. How to provide a bound of
$$
\left\| f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x} + \delta_ {\mathbf {w} + \mathbf {u}} ^ {a d v} (\mathbf {x})) - f _ {\mathbf {w}} (\mathbf {x} + \delta_ {\mathbf {w}} ^ {a d v} (\mathbf {x})) \right\| _ {2}? \tag {6}
$$
We refer to the gap in Eq. (6) as robust weight perturbation. To the best of our knowledge, it remains unclear how to establish a bound for robust weight perturbation. In standard settings, when we perturb the weights from  $\mathbf{w}$  to  $\mathbf{w} + \mathbf{u}$ , the input  $\mathbf{x}$  remains the same. The change in function values is solely attributable to the change in weights. However, the situation becomes much more complex in adversarial settings. If we perturb the weights from  $\mathbf{w}$  to  $\mathbf{w} + \mathbf{u}$ , the adversarial attacks also vary from  $\delta_{\mathbf{w}}^{adv}(\mathbf{x})$  to  $\delta_{\mathbf{w} + \mathbf{u}}^{adv}(\mathbf{x})$ . The combined changes in input  $\mathbf{x}$  and weights  $\mathbf{w}$  may result in a substantial change in function values. The challenge of Problem 1 can be observed in previous studies.
Farnia et al. (2018) introduced additional assumptions to bound Eq. (6). For instance, for FGSM and PGM attacks, they assumed  $|\nabla_{\mathbf{x}}\ell (f_{\mathbf{w}}(\mathbf{x}),y)|\geq \kappa$  for all  $\mathbf{x}\epsilon$ -close to  $\mathcal{X}$ . This parameter  $\kappa$  appears in the bound of Eq. (6) as well as in the final generalization bound. To the best of our knowledge, there has been no attempt at  $\delta_{\mathbf{w}}^{*}(\mathbf{x})$ . It is not because such research is unimportant (as mentioned in Sec. 3), but rather due to the challenge presented by Problem 1. In this case, it remains unclear what assumptions can be made to bound Eq. (6). The related work on Rademacher complexity analysis demonstrates the difficulty, as researchers have found it challenging to bound robust margin loss and have instead resorted to bounding robust loss against soled attack with additional assumptions. Further discussion on this topic can be found in Sec. 2.
3 It is because  $\| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})\|_{\infty}\leq \| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})\|_{2}$
Our solution to this problem consists of two steps. Step 1: We recognize that a general and reasonable bound for Eq. (6) without additional assumptions may not exist. To address this, we establish a bound for a similar expression, namely the weight perturbation of margin operator, without requiring any additional assumptions. To develop this bound, we introduce a generalization framework called "Perturbation Bounds of Robustified Function", which can be further extended to analyze other neural network structures. Step 2: We modify Lemma 4 to incorporate the weight perturbation bound that we have introduced. By combining these two steps, we are able to address the challenges and provide a robust generalization bound.
6.3 Perturbation Bounds of Robustified Function
In this section, we consider functions  $g_{\mathbf{w}}(\mathbf{x})$  parameterized by the weights of a neural network. We mainly consider scalar value functions  $g_{\mathbf{w}}(\mathbf{x}): \mathcal{X} \to \mathbb{R}$ . For example,  $g_{\mathbf{w}}(\mathbf{x})$  can be the  $i^{th}$  output of a neural network  $f_{\mathbf{w}}(\mathbf{x})[i]$ , the margin operator  $f_{\mathbf{w}}(\mathbf{x})[y] - \max_{j \neq y} f_{\mathbf{w}}(\mathbf{x})[j]$ , or the robust margin operator.
Definition 1 (Local Perturbation Bounds). Given  $\mathbf{x} \in \mathcal{X}$ , we say  $g_{\mathbf{w}}(\mathbf{x})$  has a  $(L_1, \dots, L_d)$ -local perturbation bound w.r.t.  $\mathbf{w}$ , if
$$
\left| g _ {\mathbf {w}} (\mathbf {x}) - g _ {\mathbf {w} ^ {\prime}} (\mathbf {x}) \right| \leq \sum_ {i = 1} ^ {d} L _ {i} \| W _ {i} - W _ {i} ^ {\prime} \|, \tag {7}
$$
where  $L_{i}$  can be related to  $\mathbf{w}$ ,  $\mathbf{w}'$  and  $\mathbf{x}$ .
Eq. (7) controls the change of the output of functions  $g_{\mathbf{w}}(\mathbf{x})$  given a slight perturbation on the weights of DNNs. The following Lemma is the key Lemma to estimate perturbation bounds of the robustified function, which is defined as  $\inf_{\| \mathbf{x} - \mathbf{x}'\| \leq \epsilon}g_{\mathbf{w}}(\mathbf{x}')$ . The reason why we require  $g_{\mathbf{w}}(\mathbf{x})$  to be scalar functions is that we can define their corresponding robustified functions.
Lemma 5 (Key Lemma). if  $g_{\mathbf{w}}(\mathbf{x})$  has a  $(A_{1}|\mathbf{x}|,\dots ,A_{d}|\mathbf{x}|)$ -local perturbation bound, i.e.,
$$
\left| g _ {\mathbf {w}} (\mathbf {x}) - g _ {\mathbf {w} ^ {\prime}} (\mathbf {x}) \right| \leq \sum_ {i = 1} ^ {d} A _ {i} | \mathbf {x} | \| W _ {i} - W _ {i} ^ {\prime} \|,
$$
the robustified function  $\inf_{\| \mathbf{x} - \mathbf{x}'\| \leq \epsilon}g_{\mathbf{w}}(\mathbf{x}')$  has a  $(A_{1}(|\mathbf{x}| + \epsilon),\dots ,A_{d}(|\mathbf{x}| + \epsilon))$ -local perturbation bound.
Proof: Let  $\mathbf{x}(\mathbf{w}) = \arg \inf_{\| \mathbf{x} - \mathbf{x}'\| \leq \epsilon}g_{\mathbf{w}}(\mathbf{x}')$  and  $\mathbf{x}(\mathbf{w}') = \arg \inf_{\| \mathbf{x} - \mathbf{x}'\| \leq \epsilon}g_{\mathbf{w}'}(\mathbf{x}')$ , Then,
$$
\left| \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w}} (\mathbf {x} ^ {\prime}) - \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} ^ {\prime}) \right| \leq \max  \left\{\left| g _ {\mathbf {w}} (\mathbf {x} (\mathbf {w})) - g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} (\mathbf {w})) \right|, \left| g _ {\mathbf {w}} (\mathbf {x} (\mathbf {w} ^ {\prime})) - g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} (\mathbf {w} ^ {\prime})) \right| \right\}.
$$
It is because  $g_{\mathbf{w}}(\mathbf{x}(\mathbf{w})) - g_{\mathbf{w}'}(\mathbf{x}(\mathbf{w}')) \leq g_{\mathbf{w}}(\mathbf{x}(\mathbf{w})) - g_{\mathbf{w}'}(\mathbf{x}(\mathbf{w}'))$  and  $g_{\mathbf{w}'}(\mathbf{x}(\mathbf{w})) - g_{\mathbf{w}}(\mathbf{x}(\mathbf{w})) \leq g_{\mathbf{w}'}(\mathbf{x}(\mathbf{w})) - g_{\mathbf{w}}(\mathbf{x}(\mathbf{w}))$ . Therefore,
$$
\left| \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w}} (\mathbf {x} ^ {\prime}) - \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} ^ {\prime}) \right| \leq \sum_ {i = 1} ^ {d} A _ {i} | \mathbf {x} (\mathbf {w}) | \| W _ {i} - W _ {i} ^ {\prime} \| \leq \sum_ {i = 1} ^ {d} A _ {i} (| \mathbf {x} | + \epsilon) \| W _ {i} - W _ {i} ^ {\prime} \|.
$$
Lemma 5 shows that the local perturbation bound of the robustified function  $\inf_{\| \mathbf{x} - \mathbf{x}'\| \leq \epsilon}g_{\mathbf{w}}(\mathbf{x}')$  can be estimated by the local perturbation bound of the function  $g_{\mathbf{w}}(\mathbf{x})$ , which is the key to provide robust generalization bounds.
6.4 Perturbation Bounds of Margin Operator
It should be noted that Lemma 5 is unable to provide a bound for Problem 1. In order to utilize Lemma 5, we shift our focus to the margin operator, which is a scalar function.
Margin Operator. Following the notation of (Bartlett et al., 2017), we define the margin operator of the true label  $y$  given  $\mathbf{x}$  and of a pair of two classes  $(i,j)$  as
$$
M (f _ {\mathbf {w}} (\mathbf {x}), y) = f _ {\mathbf {w}} (\mathbf {x}) [ y ] - \max  _ {j \neq y} f _ {\mathbf {w}} (\mathbf {x}) [ j ], M (f _ {\mathbf {w}} (\mathbf {x}), i, j) = f _ {\mathbf {w}} (\mathbf {x}) [ i ] - f _ {\mathbf {w}} (\mathbf {x}) [ j ].
$$
Robust Margin Operator. Similarly, we define the robust margin operator of the true label  $y$  and of a pair of two classes  $(i,j)$  given  $\mathbf{x}$  as
$$
R M \left(f _ {\mathbf {w}} (\mathbf {x}), y\right) = \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} \left(f _ {\mathbf {w}} \left(\mathbf {x} ^ {\prime}\right) [ y ] - \max  _ {j \neq y} f _ {\mathbf {w}} \left(\mathbf {x} ^ {\prime}\right) [ j ]\right), \quad \text {a n d}
$$
$$
R M (f _ {\mathbf {w}} (\mathbf {x}), i, j) = \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} (f _ {\mathbf {w}} (\mathbf {x} ^ {\prime}) [ i ] - f _ {\mathbf {w}} (\mathbf {x} ^ {\prime}) [ j ]),
$$
respectively. Based on Lemma 5, it is left to provide the form of  $A_{i}$  for the margin operator.
Lemma 6. Let  $f_{\mathbf{w}}$  be a d-layer neural networks with Relu activation. The following local perturbation bounds hold.
1. Given  $\mathbf{x}$  and  $i, j$ , the margin operator  $M(f_{\mathbf{w}}(\mathbf{x}), i, j)$  has a  $(A_1|\mathbf{x}|, \dots, A_d|\mathbf{x}|)$ -local perturbation bound w.r.t.  $w$ , where  $A_i = 2e\prod_{l=1}^d \|W_l\|_2 / \|W_i\|_2$ . And
$$
\left| M \left(f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}), i, j\right) - M \left(f _ {\mathbf {w}} (\mathbf {x}), i, j\right) \right| \leq 2 e B \prod_ {l = 1} ^ {d} \| W _ {l} \| _ {2} \sum_ {i = 1} ^ {d} \frac {\left\| U _ {i} \right\| _ {2}}{\left\| W _ {i} \right\| _ {2}}. \tag {8}
$$
2. Given  $\mathbf{x}$  and  $i,j$ , the robust margin operator  $RM(f_{\mathbf{w}}(\mathbf{x}),i,j)$  has a locally  $(A_{1}(|\mathbf{x}| + \epsilon),\dots ,A_{d}(|\mathbf{x}| + \epsilon))$ -local perturbation bound w.r.t.  $w$ . And
$$
\left| R M \left(f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}), i, j\right) - R M \left(f _ {\mathbf {w}} (\mathbf {x}), i, j\right) \right| \leq 2 e (B + \epsilon) \prod_ {l = 1} ^ {d} \| W _ {l} \| _ {2} \sum_ {i = 1} ^ {d} \frac {\left\| U _ {i} \right\| _ {2}}{\left\| W _ {i} \right\| _ {2}}. \tag {9}
$$
The proof of Lemma 6.1 is adopted from Lemma 2 in (Neyshabur et al., 2017b), and the proof of Lemma 6.2 is a combination of Lemma 5 and Lemma 6.1. It is important to note that Eq. (9) provides a bound for a similar but different form of robust weight perturbation compared to Eq. (6), indicating that Problem 1 has not been fully resolved. However, we are fortunate that the subsequent lemma demonstrates that Eq. (9) is sufficient to yield the final robust generalization bound.
Lemma 7. Let  $f_{\mathbf{w}}(\mathbf{x}) : \mathcal{X} \to \mathbb{R}^k$  be any predictor with parameters  $\mathbf{w}$ , and  $P$  be any distribution on the parameters that is independent of the training data. Then, for any  $\gamma, \delta > 0$ , with probability  $\geq 1 - \delta$  over the training set of size  $m$ , for any  $\mathbf{w}$ , and any random perturbation  $\mathbf{u}$  s.t.
1.  $\mathbb{P}_{\mathbf{u}}[\max_{i,j\in [k],\mathbf{x}\in \mathcal{X}}|M(f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}),i,j) - M(f_{\mathbf{w}}(\mathbf{x}),i,j)| <   \frac{\gamma}{2} ]\geq \frac{1}{2},$  we have:
2.  $\mathbb{P}_{\mathbf{u}}[\max_{i,j\in [k],\mathbf{x}\in \mathcal{X}}|RM(f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}),i,j) - RM(f_{\mathbf{w}}(\mathbf{x}),i,j)| <   \frac{\gamma}{2} ]\geq \frac{1}{2},$  we have:
$$
R _ {0} \left(f _ {\mathbf {w}}\right) \leq \hat {R} _ {\gamma} \left(f _ {\mathbf {w}}\right) + 4 \sqrt {\frac {K L \left(\mathbf {w} + \mathbf {u} \| P\right) + \ln \frac {6 m}{\delta}}{m - 1}}.
$$
Remark: Lemma 7 shows that we can replace the robust weight perturbation (Eq. (6)) by the weight perturbation of the robust margin operator. The proof is deferred to the Appendix.
Now that we have established the complete framework of the perturbation bound of robustified function to derive the robust generalization bound, we are ready to prove Theorem 1. By following the proof of (Neyshabur et al., 2017b), we can replicate the standard generalization bound by combining Lemma 6.1 and 7.1. Similarly, we can obtain the robust generalization bound by combining Lemma 6.2 and 7.2. The flowchart illustrating this process is presented in Figure 2. Additionally, Lemma 5 serves as a crucial link between the robust margin operator and the margin operator, thus establishing the connection between the robust generalization bound and the standard generalization bound.
7 Extension of the Main Result
The provided framework allows us to extend the result to 1) general non- $\ell_p$  adversarial attacks and 2) other neural network structures.
Extension to Non- $\ell_p$  Adversarial Attacks. Even though most of the adversarial robustness studies focused on norm-bounded attacks, real-world attacks are not restricted in the  $\ell_p$ -ball. We consider the following general adversarial attack problem:
$$
\max _ {\mathbf {x} ^ {\prime} \in C (\mathbf {x})} \ell (f _ {\mathbf {w}} (\mathbf {x} ^ {\prime}), y),
$$
where  $C(\mathbf{x})$  can be any reasonable constraint given the original example  $\mathbf{x}$ . Assume that  $\max_{x\in S}\max_{\mathbf{x}'\in C(\mathbf{x})}|\mathbf{x}'| = D$ . In words, the norm of the adversarial examples is bounded by  $D$ .
Theorem 8 (Robust Generalization Bound for non- $\ell_p$  attack.). For any  $D, d, h$ , let  $f_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^k$  be a  $d$ -layer feedforward network with ReLU activations. Then, for any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
R _ {0} ^ {n l} (f _ {\mathbf {w}}) - \hat {R} _ {\gamma} ^ {n l} (f _ {\mathbf {w}}) \leq \mathcal {O} \left(\sqrt {\frac {D ^ {2} d ^ {2} h \ln (d h) \Phi (f _ {\mathbf {w}}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right),
$$
where  $\Phi(f_{\mathbf{w}}) = \Pi_{i=1}^{d} \|W_i\|_2^2 \sum_{i=1}^{d} \frac{\|W_i\|_F^2}{\|W_i\|_2^2}$  and  $nl$  stands for non- $\ell_p$  adversarial attacks.
The proof is based on a slight modification of Lemma 5.
Extension to Other Neural Networks Structure. The framework we have established enables us to extend the PAC-Bayesian generalization bound from standard settings to robust settings, provided that the standard generalization bound is also obtained using this framework. Importantly, this extension is independent of the structure of the neural networks.
ResNet. Consider a neural network:  $f_{\mathbf{w}}^{1}(\mathbf{x}) = W_{1}\mathbf{x}$  and  $f_{\mathbf{w}}^{i}(\mathbf{x}) = W_{i}\phi (f_{\mathbf{w}}^{i - 1}(\mathbf{x})) + f_{\mathbf{w}}^{i - 1}(\mathbf{x})$ . ResNet in practice could be complicated. We use this structure for illustration.
Theorem 9 (Robust Generalization Bound for ResNet). For any  $D, d, h$ , let  $f_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^k$  be a  $d$ -layer ResNet with ReLU activations. Then, for any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
R _ {0} (f _ {R N}) - \hat {R} _ {\gamma} (f _ {R N}) \leq \mathcal {O} \left(\sqrt {\frac {(B + \epsilon) ^ {2} d ^ {2} h \ln (d h) \Phi (f _ {R N}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right),
$$
where  $\Phi (f_{RN}) = \Pi_{i = 1}^{d}(\| W_{i}\|_{2} + 1)^{2}\sum_{i = 1}^{d}\frac{\|W_{i}\|_{F}^{2}}{(\|W_{i}\|_{2} + 1)^{2}}.$
Limitation. The primary limitation lies in the fact that norm-based bounds tend to be excessively large in practical scenarios. As illustrated in Table 1, the bounds for VGG networks surpass  $10^{9}$  in the experiments on CIFAR-10 dataset. The challenge at hand is how to achieve smaller norm-based bounds in practical contexts, not only in adversarial settings but also in standard settings. This remains an open problem.
In this paper, we introduce a PAC-Bayesian spectrally-normalized robust generalization bound. The proof is constructed based on the framework of the perturbation bound of the robustified function. This established framework enables us to extend the generalization bound from standard settings to robust settings, as well as to generalize the results to encompass various adversarial attacks and DNN architectures. The simplicity of this framework makes it a valuable tool for analyzing robust generalization in machine learning.
We would like to thank all the anonymous reviewers for their comments and suggestions. This paper is supported in part by the National Key Research and Development Project under grant 2022YFA1003900; Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project (No.HZQsws-KCCYB-2022046); Guangdong Key Lab on the Mathematical Foundation of Artificial Intelligence, Department of Science and Technology of Guangdong Province. The work of Zhi-Quan Luo is supported by NSFC-617310018 and the Guangdong Provincial Key Laboratory of Big Data Computing.