SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PAC-Bayesian Spectrally-Normalized Bounds for Adversarily Robust Generalization
Jiancong Xiao $^{1,*}$ , Ruoyu Sun $^{2,3,4,\dagger}$ , Zhi-Quan Luo $^{2,4,\dagger}$
<sup>1</sup>University of Pennsylvania, PA, USA
$^{2}$ Scool of Data Science, The Chinese University of Hong Kong, Shenzhen, China
<sup>3</sup>Shenzhen International Center for Industrial and Applied Mathematics
4 Shenzhen Research Institute of Big Data
jcxiao@upenn.edu, {sunruoyu,luozq}@cuhk.edu.cn
Deep neural networks (DNNs) are vulnerable to adversarial attacks. It is found empirically that adversarially robust generalization is crucial in establishing defense algorithms against adversarial attacks. Therefore, it is interesting to study the theoretical guarantee of robust generalization. This paper focuses on norm-based complexity, based on a PAC-Bayes approach (Neyshabur et al., 2017b). The main challenge lies in extending the key ingredient, which is a weight perturbation bound in standard settings, to the robust settings. Existing attempts heavily rely on additional strong assumptions, leading to loose bounds. In this paper, we address this issue and provide a spectrally-normalized robust generalization bound for DNNs. Compared to existing bounds, our bound offers two significant advantages: Firstly, it does not depend on additional assumptions. Secondly, it is considerably tighter, aligning with the bounds of standard generalization. Therefore, our result provides a different perspective on understanding robust generalization: The mismatch terms between standard and robust generalization bounds shown in previous studies do not contribute to the poor robust generalization. Instead, these disparities solely due to mathematical issues. Finally, we extend the main result to adversarial robustness against general non- $\ell_p$  attacks and other neural network architectures.
Even though deep neural networks (DNNs) have impressive performance on many machine learning tasks, they are often highly susceptible to adversarial perturbations imperceptible to the human eye (Goodfellow et al., 2015; Madry et al., 2018). They have received enormous attention in the machine learning literature over recent years and a large number of defense algorithms (Gowal et al., 2020; Rebuffi et al., 2021) are proposed to improve the robustness in practice. Nonetheless, it still fails to deliver satisfactory performance. One major challenge stems from adversarially robust generalization. For example, Madry et al. (2018) demonstrated that the robust generalization gap can extend up to  $50\%$  on CIFAR-10. In contrast, the standard generalization gap is notably small in practical settings. Hence, a theoretical question arises: Why is there a huge difference between standard generalization and robust generalization? This paper focuses on norm-based generalization analysis.
In classical learning theory, one of the most well-known findings is that the generalization bound for neural networks depends on the norms of their layers (Bartlett, 1998). To further explore the generalization of deep learning, a series of work aimed at improving the norm-based bound (Bartlett &
*Work done while at CUHK(SZ).
† Corresponding Authors.
Mendelson, 2002; Neyshabur et al., 2015; Golowich et al., 2018), mainly using tools of Rademacher complexity. The tightest bound is given by Bartlett et al. (2017), using a covering number approach. Neyshabur et al. (2017b) gave a different and simpler proof based on PAC-Bayes analysis, presented an almost equally tight bound. The key step involves bounding the change in output of the predictors in response to slight variations in the predictor parameters. In particular, considering  $f_{\mathbf{w}}(\mathbf{x})$  as the predictor parameterized by  $\mathbf{w}$ , the crucial component for providing the generalization bound lies in bounding the gap  $|f_{\mathbf{w}}(\mathbf{x}) - f_{\mathbf{w}'}(\mathbf{x})|$ , where  $\mathbf{w}$  and  $\mathbf{w}'$  are close. The weight perturbation bound, which addresses this aspect, is presented in Lemma 2 of Neyshabur et al. (2017b).
To comprehend the limited robust generalization capabilities of deep learning, a line of research endeavors to extend the norm-based bounds into robust settings. However, this has proven to be a challenging mathematical problem, as researchers have attempted the mentioned approaches including the Rademacher complexity (Khim & Loh, 2018; Yin et al., 2019; Awasthi et al., 2020), covering number (Gao & Wang, 2021; Xiao et al., 2022a; Mustafa et al., 2022), and the PAC-Bayes analysis (Farnia et al., 2018), yet a satisfactory solution remains elusive. For more details, see Section 2.
We use the PAC-Bayesian approach as an example to illustrate the mathematical challenge. The weight perturbations in adversarial settings differ from those in standard settings. When considering two predictors  $f_{\mathbf{w}}(\cdot)$  and  $f_{\mathbf{w}'}(\cdot)$ , the adversarial examples against these predictors are distinct, leading to a gap referred to as robust weight perturbation (defined later in Problem 1). It remains unclear how to establish a bound for robust weight perturbation. The combined changes in input and weights can potentially cause a significant alteration in the function value. The main challenge is illustrated in Figure 1, the details of which will be provided in Section 6.2. As a result, Farnia et al. (2018) introduced additional assumption to control this
Figure 1: Demonstration of the main challenge of providing robust generalization bound. The weight perturbation bound (Neyshabur et al., 2017b) seems hard to extend to adversarial settings.
gap and provide bounds in adversarial settings. However, the assumption imposed limitations on the effectiveness of the bounds due to two reasons: Firstly, the assumption of sharp gradients throughout the domain is a strong requirement. Secondly, without this assumption, the bounds become unbounded  $(= +\infty)$ . Similarly, other existing norm-based bounds also depend on additional assumptions or involve higher-order terms in certain factors.
Given that the existing robust generalization bounds are much larger than standard generalization bounds, these results suggest a possible hypothesis: The significant disparity between standard and robust generalization in practical scenarios could potentially be attributed to the mismatch terms between the standard bounds and the robust bounds. However, verifying this hypothesis is challenging because it remains unclear whether the existence of these terms or assumptions is due to mathematical issues. Therefore, the current bounds are insufficient to address the main theoretical question.
In this paper, we address this problem and present a PAC-Bayes spectrally-normalized robust generalization bound without additional assumptions. Our robust generalization bound is as tight as the standard generalization bound, with an additional factor representing the perturbation intensity  $\epsilon$ . Furthermore, our bound is strictly smaller than the previous generalization bounds proposed in adversarial robustness settings. To provide an initial overview of the main result, we begin by defining the spectral complexity of a  $d$ -layer neural network  $f_{\mathbf{w}}$  as follows:
$$
\Phi \left(f _ {\mathbf {w}}\right) = \Pi_ {i = 1} ^ {d} \left\| W _ {i} \right\| _ {2} ^ {2} \sum_ {i = 1} ^ {d} \left(\left\| W _ {i} \right\| _ {F} ^ {2} / \left\| W _ {i} \right\| _ {2} ^ {2}\right), \tag {1}
$$
where  $W_{i}$  is the weights of  $f_{\mathbf{w}}$  in each of the  $d$  layers.
Theorem (Informal). Let  $m$  be the number of samples and the training samples  $x$  is bounded by  $B$ .  $\epsilon$  is the attack intensity. Let  $f_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^k$  be a d-layer feedforward network. Then, with high probability, we have
$$
\text {R o b u s t G e n e r a l i z a t i o n} \leq \mathcal {O} (\sqrt {(B + \epsilon) ^ {2} \Phi (f _ {\mathbf {w}}) / m}).
$$
When  $\epsilon = 0$ , the bound reduces to the standard generalization bound presented by Neyshabur et al. (2017b). Our results give a different perspective from existing bounds. The additional factors or assumptions are solely due to mathematical considerations. Our findings suggest that the implicit difference of the spectral complexity  $\Phi(f_{\mathbf{w}})$  likely contributes to the significant disparity between standard and robust generalization.
Technical Proof. It is shown that the robust weight perturbation is not controllable without additional assumptions. Therefore, existing tools are not sufficient to derive the bounds. The main technical tools to derive the bounds are two folds. Firstly, we introduce a crucial inequality to address this problem, which is the preservation of weight perturbation bound under  $\ell_p$  attack. Secondly, we restructure the proof by (Neyshabur et al., 2017b) in terms of the margin operator. This modification enables the application of the aforementioned inequality. To further extend the bound to more general settings, we establish a framework that allows us to derive a robust generalization bound from its corresponding standard generalization bound. The
Figure 2: Demonstration of the framework: perturbation bound of robustified function. Under this framework, a standard generalization bound directly implies a robust generalization bound.
framework's demonstration is presented in Figure 2, and detailed information regarding Figure 2 will be provided in Section 6.3.
Furthermore, we extend the results to encompass general settings. Firstly, although  $\ell_p$  adversarial attacks are widely used, real-world attacks are not always bounded by the  $\ell_p$  norm. Hence, we extend the results to cover general attacks. Secondly, as the current state-of-the-art robust performance is achieved with WideResNet (Rebuffi et al., 2021; Croce et al., 2021), we demonstrate that the results can be extended to other DNN structures, such as ResNet.
The contributions are listed as follows:
1. Main result: We provide a PAC-Bayesian spectrally-normalized robust generalization bound without any additional assumption. The derived bound is as tight as the standard generalization bound and tighter than the existing robust generalization bound.
2. Our results give a different perspective from existing bounds. The significant disparity between standard and robust generalization in practical scenarios is not attributed to the mismatch terms between the standard bound and the robust bound. The implicit difference of the spectral complexity  $\Phi(f_{\mathbf{w}})$  possibly contributes to the significant disparity.
3. We provide a general framework for robust generalization analysis. We show how to obtain a robust generalization bound from a given standard generalization bound.
4. We extend the result to general adversarial attacks and other neural networks architectures.
Adversarial Attack. Adversarial examples were first introduced in (Szegedy et al., 2014). Since then, adversarial attacks have received enormous attention (Papernot et al., 2016; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017). Nowadays, attack algorithms have become sophisticated and powerful. For example, Autoattack (Croce & Hein, 2020) and Adaptive attack (Tramer et al., 2020). Therefore, we consider theoretical analysis on robust margin loss (defined later in Eq. (4)) against any norm-based attacks. Real-world attacks are not always norm-bounded (Kurakin et al., 2018). Therefore, we also consider non- $\ell_p$  attacks (Lin et al., 2020; Xiao et al., 2022c) in Sec. 7.
Adversarially Robust Generalization. Even enormous algorithms were proposed to improve the robustness of DNNs (Madry et al., 2018; Tramér et al., 2018; Gowal et al., 2020; Rebuffi et al., 2021), the performance was far from satisfactory. One major issue is the poor robust generalization, or robust overfitting (Rice et al., 2020). A series of studies (Xing et al., 2021; Xiao et al., 2022b,d; Ozdaglar et al., 2022) have delved into the concept of uniform stability within the context of adversarial training.
However, these analyses focused on general Lipschitz functions, without specific consideration for neural networks.
Rademacher Complexity. Rademacher complexity can provide similar spectral norm generalization bound as PAC-Bayesian bound (Theorem 2). Rademacher complexity was extended to adversarial settings for linear classifier (Khim & Loh, 2018; Yin et al., 2019) and two-layers neural networks (Awasthi et al., 2020). As for DNNs, they found that it was mathematically difficult and provided some discussions on surrogate losses rather than the adversarial loss.
Covering Number. Rademacher complexity can be bounded in terms of the covering number of the function class, as discussed in (Bartlett et al., 2017). Nevertheless, calculating the covering number for an adversarial function class is also shown to be a challenging problem. Gao & Wang (2021) considered adversarial loss against FGSM attacks, employing similar assumptions to those of (Farnia et al., 2018), resulting in a bound similar to Theorem 3. Additionally, Xiao et al. (2022a) and Mustafa et al. (2022) introduced two different methods, respectively, to compute the covering number for adversarial function classes. However, the bounds obtained through these methods remain notably larger when compared to those in standard settings. The related research on Rademacher complexity and covering number help proves the difficulty of the problem we are addressing.
PAC-Bayes Analysis. We mainly compare our results to the previous PAC-Bayesian spectrally-normalized bounds (Neyshabur et al., 2017b; Farnia et al., 2018), which we have already discussed in the introduction. We will provide more details later. The workshop version of this paper is presented in (Xiao et al., 2023). Other PAC-Bayes frameworks for tackling adversarial robustness also exist. Viallard et al. (2021) explored a distinct adversarial attack targeting the loss of the  $Q$ -weighted majority vote over the posterior distribution  $Q$ . Mustafa et al. (2023) introduced a non-vacuous PAC-Bayes bound designed for stochastic neural networks.
We mainly follow the notations of (Neyshabur et al., 2017b). Consider the classification task that maps the input  $\mathbf{x} \in \mathcal{X}$  to the label  $y \in \mathbb{R}^k$ . The output of the model is a score for each of the  $k$  classes. The class with the maximum score will be the prediction of the label of  $\mathbf{x}$ . A sample dataset  $S = \{(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_m, y_m)\}$  with  $m$  training samples is given. The  $l_2$  norm of each of the samples  $x_i$  is bounded by  $B$ , i.e.,  $\| x_i\|_2 \leq B$ ,  $i = 1, \dots, m$ . Let  $\| W\|_F$  and  $\| W\|_2$  denote the Frobenius norm and the spectral norm of the weights  $W$ , respectively.
Fully-Connected Neural Networks. Let  $f_{\mathbf{w}}(\mathbf{x}) : \mathcal{X} \to \mathbb{R}^k$  be the function computed by a  $d$ -layer feed-forward network for the classification task with parameters  $\mathbf{w} = \mathrm{vec}\left(\{W_i\}_{i=1}^d\right)$ ,  $f_{\mathbf{w}}(\mathbf{x}) = W_d\phi(W_{d-1}\phi(\dots\phi(W_1\mathbf{x})))$ , here  $\phi$  is the ReLU activation function. Let  $f_{\mathbf{w}}^i(\mathbf{x})$  denote the output of layer  $i$  before activation and  $h$  be an upper bound on the number of output units in each layer. We can then define fully-connected feed-forward networks recursively:  $f_{\mathbf{w}}^1(\mathbf{x}) = W_1\mathbf{x}$  and  $f_{\mathbf{w}}^i(\mathbf{x}) = W_i\phi(f_{\mathbf{w}}^{i-1}(\mathbf{x}))$ . In Section 7, we extend the results to ResNet (He et al., 2016), since the state-of-the-art robust performance is built on WideResNet (Rebuffi et al., 2021; Croce et al., 2021).
3.2 Standard Margin Loss and Robust Margin Loss
Standard Margin Loss. For any distribution  $\mathcal{D}$  and margin  $\gamma > 0$ , the expected margin loss is defined as follows:
$$
L _ {\gamma} \left(f _ {\mathbf {w}}\right) = \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} \left[ f _ {\mathbf {w}} (\mathbf {x}) [ y ] \leq \gamma + \max  _ {j \neq y} f _ {\mathbf {w}} (\mathbf {x}) [ j ] \right]. \tag {2}
$$
Let  $\widehat{L}_{\gamma}(f_{\mathbf{w}})$  be the empirical estimate of the above expected margin loss. Since setting  $\gamma = 0$  corresponds to the classification loss, we will use  $L_0(f_{\mathbf{w}})$  and  $\widehat{L}_0(f_{\mathbf{w}})$  to refer to the expected loss and the training loss. The loss  $L_{\gamma}$  defined this way is bounded between 0 and 1.
Robust Margin Loss. Adversarial examples are usually crafted by an attack algorithm. Let  $\delta_{\mathbf{w}}^{adv}(\mathbf{x})$  be an algorithm output and  $\delta_{\mathbf{w}}^{*}(\mathbf{x})$  be the maximizer of the following maximization problem
$$
\max  _ {\| \delta \| \leq \epsilon} \ell \left(f _ {\mathbf {w}} (\mathbf {x} + \delta), y\right), \tag {3}
$$
where  $\ell$  is the loss function of the predicted label and true label. Without explicit specification,  $\|\cdot\|$  refers to the  $\ell_2$  norm. The robust margin loss is defined as follows:
$$
\begin{array}{l} R _ {\gamma} \left(f _ {\mathbf {w}}\right) = \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} \left[ \exists \mathbf {x} ^ {\prime} \in \mathbb {B} _ {\mathbf {x}} ^ {p} (\epsilon), f _ {\mathbf {w}} \left(\mathbf {x} ^ {\prime}\right) [ y ] \leq \gamma + \max  _ {j \neq y} f _ {\mathbf {w}} \left(\mathbf {x} ^ {\prime}\right) [ j ] \right] \tag {4} \\ = \mathbb {P} _ {(\mathbf {x}, y) \sim \mathcal {D}} \left[ f _ {\mathbf {w}} (\mathbf {x} + \delta_ {\mathbf {w}} ^ {*} (\mathbf {x})) [ y ] \leq \gamma + \max _ {j \neq y} f _ {\mathbf {w}} (\mathbf {x} + \delta_ {\mathbf {w}} ^ {*} (\mathbf {x})) [ j ] \right]. \\ \end{array}
$$
Let  $\hat{R}_{\gamma}(f_{\mathbf{w}})$  be the empirical estimate of the above expected robust margin loss. The robust margin loss requires the whole norm ball around the original example  $\mathbf{x}$  to be labelled correctly, which is the goal of norm-based adversarial robustness. By replacing  $\delta_{\mathbf{w}}^{*}(\mathbf{x})$  by  $\delta_{\mathbf{w}}^{adv}(\mathbf{x})$  in the above definition, we denote  $R_{\gamma}^{adv}(f_{\mathbf{w}})$  as the margin loss against attacks  $adv$ . The work of (Farnia et al., 2018) consider three attacks: fast gradient sign method (FGSM or FGM), projected gradient method (PGM), and Wasserstein risk minimization (WRM), i.e.,  $adv = \mathrm{FGSM}$ , PGM, and WRM. They provided three different bounds for these adversarial attacks respectively. However, methods for generating these adversarial examples are becoming significantly more sophisticated and powerful. For example, Autoattack (Croce & Hein, 2020) in default settings is a collection of four attacks to find adversarial examples. Therefore, a bound of robust margin loss against a single attack provides a limited robustness guarantee to a machine learning model. In fact, Autoattack collects different attacks to attempt and to provide a close lower estimation of  $R_0(f_{\mathbf{w}})$ . Therefore, this paper focuses on the robust margin loss.
4 Robust Generalization Bound
In this section, we will first provide our main result of robust generalization.
Theorem 1 (Main Result: Robust Generalization Bound). For any  $B, d, h, \epsilon > 0$ , let  $f_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^k$  be a  $d$ -layer feedforward network with ReLU activations. Then, for any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
R _ {0} (f _ {\mathbf {w}}) - \hat {R} _ {\gamma} (f _ {\mathbf {w}}) \leq \mathcal {O} \left(\sqrt {\frac {(B + \epsilon) ^ {2} d ^ {2} h \ln (d h) \Phi (f _ {\mathbf {w}}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right),
$$
where  $\Phi(f_{\mathbf{w}}) = \Pi_{i=1}^{d} \|W_i\|_2^2 \sum_{i=1}^{d} \frac{\|W_i\|_F^2}{\|W_i\|_2^2}$  is the spectral complexity of  $f_{\mathbf{w}}$ .
Remark. Theorem 1 is presented under  $\ell_{2}$  attacks to simplify the notation. For other  $\ell_{p}$  attacks, suppose all the samples  $x_{i}$  have  $\ell_{p}$  norm bounded by  $B$  and  $\| \delta \| _p\leq \epsilon$ , the robust generalization bound is to replace  $(B + \epsilon)$  by  $\max \{1,n^{\frac{1}{2} -\frac{1}{p}}\} (B + \epsilon)$  in Theorem 1, where  $n$  is the dimension of the samples  $x_{i}$ .
Theorem 1 provides the first PAC-Bayesian bound in adversarial robustness settings without introducing new assumptions. Fixing other factors, the generalization gap goes to 0 as  $m \to \infty$ .
Theorem 2 (Standard Generalization Bound (Neyshabur et al., 2017b)). For any  $B, d, h > 0$ , let  $f_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^k$  be a  $d$ -layer feedforward network with ReLU activations. Then, for any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
L _ {0} (f _ {\mathbf {w}}) - \widehat {L} _ {\gamma} (f _ {\mathbf {w}}) \leq \mathcal {O} \left(\sqrt {\frac {B ^ {2} d ^ {2} h \ln (d h) \Phi (f _ {\mathbf {w}}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right),
$$
where  $\Phi (f_{\mathbf{w}}) = \Pi_{i = 1}^{d}\| W_{i}\|_{2}^{2}\sum_{i = 1}^{d}\frac{\|W_{i}\|_{F}^{2}}{\|W_{i}\|_{2}^{2}}.$
Comparison with Existing Standard Generalization Bounds. Comparing the robust generalization bound in Theorem 1 with the standard generalization bound in Theorem 2, the only difference is a factor of the attack intensity  $\epsilon$ , which is unavoidable in adversarial settings. In other words,  $B$  and  $B + \epsilon$  are the magnitudes of the clean and adversarial examples, respectively. Therefore, our main result is as tight as the standard generalization bound in Theorem 2.
Theorem 3 (Robust Generalization Bound (Farnia et al., 2018)). For any  $B, d, h > 0$ , let  $f_{\mathbf{w}}: \mathcal{X} \to \mathbb{R}^k$  be a  $d$ -layer feedforward network with ReLU activations. Consider an FGM attack with noise power  $\epsilon$  according to Euclidean norm  $\| \cdot \|_2$ . Assume that  $\|\nabla_{\mathbf{x}} \ell(f_{\mathbf{w}}(\mathbf{x}), y)\| \geq \kappa$ ,  $\forall \mathbf{x} \in \mathcal{X}$  and  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
R _ {0} ^ {a d v} (f _ {\mathbf {w}}) - \hat {R} _ {\gamma} ^ {a d v} (f _ {\mathbf {w}}) \leq \mathcal {O} \left(\sqrt {\frac {(B + \epsilon) ^ {2} d ^ {2} h \ln (d h) \Phi^ {f g m} (f _ {\mathbf {w}}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right),
$$
$$
\Phi^ {f g m} (f _ {\mathbf {w}}) = \prod_ {i = 1} ^ {d} \| W _ {i} \| _ {2} ^ {2} (1 + C ^ {f g m}) \sum_ {i = 1} ^ {d} \frac {\| W _ {i} \| _ {F} ^ {2}}{\| W _ {i} \| _ {2} ^ {2}}, a n d C ^ {f g m} = \frac {\epsilon}{\kappa} (\prod_ {i = 1} ^ {d} \| W _ {i} \| _ {2}) (\sum_ {i = 1} ^ {d} \prod_ {j = 1} ^ {i} \| W _ {j} \| _ {2}).
$$
Remark: For robust generalization bounds of PGM or WRM adversarial attacks, the bounds have similar forms as in Theorem 3, with different constants  $C^{pgm}$  and  $C^{wrm}$ .
Comparison with Existing Robust Generalization Bounds. Comparing Theorem 1 and Theorem 3, the difference of the upper bounds is the difference of  $\Phi$  and  $\Phi^{fgm}$ , where  $\Phi^{fgm}$  contains an additional term  $C^{fgm}$ . Therefore, our bound is tighter. Moreover, the robust generalization gap is much larger than the FGSM generalization gap based on the observation in practice. We provide a tighter upper bound for a larger generalization gap.
Additionally, the term  $C^{fgm}$  could be very large. Notice that Theorem 3 requires  $\ell(f_{\mathbf{w}}(\mathbf{x}), y)$  to be sharp w.r.t.  $\mathbf{x}$  for all  $\mathbf{x} \in \mathcal{X}$ . It is hard to verify and  $\kappa$  could be small. Therefore, if we remove the additional assumption  $\| \nabla_{\mathbf{x}} \ell(f_{\mathbf{w}}(\mathbf{x}), y) \| \geq \kappa$ , we have  $C^{fgm} \to +\infty$  as  $\kappa \to 0$  and the upper bound in Theorem 3 goes to infinity.
It is also worth noting that our bound is tighter than other norm-based robust generalization bounds derived in Rademacher complexity and covering number approaches, since these bounds are larger than their standard counterpart, the bound given by (Bartlett et al., 2017).