SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Awasthi, P., Frank, N., and Mohri, M. Adversarial learning guarantees for linear hypotheses and neural networks. In International Conference on Machine Learning, pp. 431-441. PMLR, 2020.
Bartlett, P. L. The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE transactions on Information Theory, 44(2):525-536, 1998.
Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Carlini, N. and Wagner, D. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (sp), pp. 39-57. IEEE, 2017.
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Farnia, F., Zhang, J., and Tse, D. Generalizable adversarial training via spectral normalization. In International Conference on Learning Representations, 2018.
Gao, Q. and Wang, X. Theoretical investigation of generalization bounds for adversarial learning of deep neural networks. Journal of Statistical Theory and Practice, 15(2):1-28, 2021.
Golowich, N., Rakhlin, A., and Shamir, O. Size-independent sample complexity of neural networks. In Conference On Learning Theory, pp. 297-299. PMLR, 2018.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. stat, 1050:20, 2015.
Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Khim, J. and Loh, P.-L. Adversarial risk bounds via function transformation. arXiv preprint arXiv:1810.09519, 2018.
Kurakin, A., Goodfellow, I. J., and Bengio, S. Adversarial examples in the physical world. In Artificial intelligence safety and security, pp. 99-112. Chapman and Hall/CRC, 2018.
Lin, W.-A., Lau, C. P., Levine, A., Chellappa, R., and Feizi, S. Dual manifold adversarial robustness: Defense against lp and non-lp adversarial attacks. Advances in Neural Information Processing Systems, 33:3487-3498, 2020.
Xiao, J., Yang, L., Fan, Y., Wang, J., and Luo, Z.-Q. Understanding adversarial robustness against on-manifold adversarial examples. arXiv preprint arXiv:2210.00430, 2022c.
Xiao, J., Zhang, J., Luo, Z.-Q., and Ozdaglar, A. E. Smoothed-sgdmax: A stability-inspired algorithm to improve adversarial generalization. In NeurIPS ML Safety Workshop, 2022d.
Xiao, J., Sun, R., and Luo, Z.-Q. Pac-bayesian adversarially robust generalization bounds for deep neural networks. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023.
Xing, Y., Song, Q., and Cheng, G. On the algorithmic stability of adversarial training. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=xz80iPFIjvG.
Yin, D., Kannan, R., and Bartlett, P. Rademacher complexity for adversarially robust generalization. In International Conference on Machine Learning, pp. 7085-7094. PMLR, 2019.
The proof of the key lemma (Lemma 5), which establishes a connection between the margin operator and the robust margin operator, is presented in the main content.
We still need to demonstrate that the properties in PAC-Bayes analysis hold for both the margin operator and the robust margin operator. The following proofs are adapted from the work of (Neyshabur et al., 2017b), with the steps being kept independent of the (robust) margin operator. We will begin by finishing the proofs of Lemma 6 and Lemma 7. Afterward, we will proceed to complete the proof of Theorem 1, which is our primary result.
A.1 Proof of Lemma 6
Proof of Lemma 6.1:
For any  $i \in [k]$ ,
$$
\left| f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}) [ i ] - f _ {\mathbf {w}} (\mathbf {x}) [ i ] \right| \leq \| f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}) - f _ {\mathbf {w}} (\mathbf {x}) \| _ {2}.
$$
For any  $i,j\in [k]$
$$
\left| M \left(f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}), i, j\right) - M \left(f _ {\mathbf {w}} (\mathbf {x}), i, j\right) \right| \leq 2 \left| f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}) [ i ] - f _ {\mathbf {w}} (\mathbf {x}) [ i ] \right| \leq 2 \| f _ {\mathbf {w} + \mathbf {u}} (\mathbf {x}) - f _ {\mathbf {w}} (\mathbf {x}) \| _ {2}.
$$
Therefore, it is left to bound  $\| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})\|$ . It is provided in (Neyshabur et al., 2017b), we provide the proof here for reference. Let  $\Delta_i = \left|f_{\mathbf{w} + \mathbf{u}}^i (\mathbf{x}) - f_{\mathbf{w}}^i (\mathbf{x})\right|_2$ . We will prove using induction that for any  $i\geq 0$
$$
\Delta_ {i} \leq \left(1 + \frac {1}{d}\right) ^ {i} \left(\prod_ {j = 1} ^ {i} \| W _ {j} \| _ {2}\right) | \mathbf {x} | _ {2} \sum_ {j = 1} ^ {i} \frac {\| U _ {j} \| _ {2}}{\| W _ {j} \| _ {2}}.
$$
The above inequality together with  $\left(1 + \frac{1}{d}\right)^d \leq e$  proves the lemma statement. The induction base clearly holds since  $\Delta_0 = |\mathbf{x} - \mathbf{x}|_2 = 0$ . For any  $i \geq 1$ , we have the following:
$$
\begin{array}{l} \Delta_ {i + 1} = \left| \left(W _ {i + 1} + U _ {i + 1}\right) \phi_ {i} \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x})\right) - W _ {i + 1} \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} \\ = \left| \left(W _ {i + 1} + U _ {i + 1}\right) \left(\phi_ {i} \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x})\right) - \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right)\right) + U _ {i + 1} \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} \\ \leq \left(\left\| W _ {i + 1} \right\| _ {2} + \left\| U _ {i + 1} \right\| _ {2}\right) \left| \phi_ {i} \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x})\right) - \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} + \left\| U _ {i + 1} \right\| _ {2} \left| \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} \\ \leq \left(\| W _ {i + 1} \| _ {2} + \| U _ {i + 1} \| _ {2}\right) \left| f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x}) - f _ {\mathbf {w}} ^ {i} (\mathbf {x}) \right| _ {2} + \| U _ {i + 1} \| _ {2} \left| f _ {\mathbf {w}} ^ {i} (\mathbf {x}) \right| _ {2} \\ = \Delta_ {i} \left(\left\| W _ {i + 1} \right\| _ {2} + \left\| U _ {i + 1} \right\| _ {2}\right) + \left\| U _ {i + 1} \right\| _ {2} \left| f _ {\mathbf {w}} ^ {i} (\mathbf {x}) \right| _ {2}, \\ \end{array}
$$
where the last inequality is by the Lipschitz property of the activation function and using  $\phi(0) = 0$ . The  $\ell_2$  norm of outputs of layer  $i$  is bounded by  $|\mathbf{x}|_2 \Pi_{j=1}^i \|W_j\|_2$  and by the lemma assumption we have  $\|U_{i+1}\|_2 \leq \frac{1}{d} \|W_{i+1}\|_2$ . Therefore, using the induction step, we get the following bound:
$$
\begin{array}{l} \Delta_ {i + 1} \leq \Delta_ {i} \left(1 + \frac {1}{d}\right) \| W _ {i + 1} \| _ {2} + \| U _ {i + 1} \| _ {2} | \mathbf {x} | _ {2} \prod_ {j = 1} ^ {i} \| W _ {j} \| _ {2} \\ \leq \left(1 + \frac {1}{d}\right) ^ {i + 1} \left(\prod_ {j = 1} ^ {i + 1} \| W _ {j} \| _ {2}\right) | \mathbf {x} | _ {2} \sum_ {j = 1} ^ {i} \frac {\| U _ {j} \| _ {2}}{\| W _ {j} \| _ {2}} + \frac {\| U _ {i + 1} \| _ {2}}{\| W _ {i + 1} \| _ {2}} | \mathbf {x} | _ {2} \prod_ {j = 1} ^ {i + 1} \| W _ {i} \| _ {2} \\ \leq \left(1 + \frac {1}{d}\right) ^ {i + 1} \left(\prod_ {j = 1} ^ {i + 1} \| W _ {j} \| _ {2}\right) | \mathbf {x} | _ {2} \sum_ {j = 1} ^ {i + 1} \frac {\| U _ {j} \| _ {2}}{\| W _ {j} \| _ {2}}. \\ \end{array}
$$
Then we complete the proof of Lemma 6.1. By combining Lemma 6.1 and Lemma 5, we directly obtain Lemma 6.2.
A.2 Proof of Lemma 7
The proof of Lemma 7.1 and 7.2 is similar. We provide the proof of Lemma 7.2 below. The proof of Lemma 7.1 follows the proof of Lemma 7.2 by replacing the robust margin operator by the margin operator.
Let  $\mathbf{w}' = \mathbf{w} + \mathbf{u}$ . Let  $S_{\mathbf{w}}$  be the set of perturbations with the following property:
$$
\mathcal{S}_{\mathbf{w}}\subseteq \left\{\mathbf{w}^{\prime}\bigg|  \max_{i,j\in [k],\mathbf{x}\in \mathcal{X}}|RM(f_{\mathbf{w}^{\prime}}(\mathbf{x}),i,j) - RM(f_{\mathbf{w}}(\mathbf{x}),i,j)| <   \frac{\gamma}{2}\right\} .
$$
Let  $q$  be the probability density function over the parameters  $\mathbf{w}'$ . We construct a new distribution  $\tilde{\mathcal{Q}}$  over predictors  $f_{\tilde{\mathbf{w}}}$  where  $\tilde{\mathbf{w}}$  is restricted to  $S_{\mathbf{w}}$  with the probability density function:
$$
\tilde {q} (\tilde {\mathbf {w}}) = \frac {1}{Z} \left\{ \begin{array}{l l} q (\tilde {\mathbf {w}}) & \tilde {\mathbf {w}} \in \mathcal {S} _ {\mathbf {w}} \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
Here  $Z$  is a normalizing constant and by the lemma assumption  $Z = \mathbb{P}[\mathbf{w}' \in S_{\mathbf{w}}] \geq \frac{1}{2}$ . By the definition of  $\tilde{Q}$ , we have:
$$
\max _ {i, j \in [ k ], {\bf x} \in \mathcal {X}} | R M (f _ {\tilde {\bf w}} ({\bf x}), i, j) - R M (f _ {\bf w} ({\bf x}), i, j) | <   \frac {\gamma}{2}.
$$
Since the above bound holds for any  $\mathbf{x}$  in the domain  $\mathcal{X}$ , we can get the following a.s.:
$$
R _ {0} \left(f _ {\mathbf {w}}\right) \leq R _ {\frac {\gamma}{2}} \left(f _ {\tilde {\mathbf {w}}}\right)
$$
$$
\hat {R} _ {\frac {\gamma}{2}} \left(f _ {\bar {w}}\right) \leq \hat {R} _ {\gamma} \left(f _ {w}\right)
$$
Now using the above inequalities together with the equation (5), with probability  $1 - \delta$  over the training set we have:
$$
\begin{array}{l} R _ {0} \left(f _ {\mathbf {w}}\right) \leq \mathbb {E} _ {\tilde {\mathbf {w}}} \left[ R _ {\frac {\gamma}{2}} \left(f _ {\tilde {\mathbf {w}}}\right) \right] \\ \leq \mathbb {E} _ {\tilde {\mathbf {w}}} \left[ \hat {R} _ {\frac {\gamma}{2}} (f _ {\tilde {\mathbf {w}}}) \right] + 2 \sqrt {\frac {2 (K L (\tilde {\mathbf {w}} \| P) + \ln \frac {2 m}{\delta})}{m - 1}} \\ \leq \hat {R} _ {\gamma} (f _ {\mathbf {w}}) + 2 \sqrt {\frac {2 (K L (\tilde {\mathbf {w}} \| P) + \ln \frac {2 m}{\delta})}{m - 1}} \\ \leq \hat {R} _ {\gamma} (f _ {\mathbf {w}}) + 4 \sqrt {\frac {K L (\mathbf {w} ^ {\prime} \| P) + \ln \frac {6 m}{\delta}}{m - 1}}, \\ \end{array}
$$
The last inequality follows from the following calculation.
Let  $S_{\mathbf{w}}^{c}$  denote the complement set of  $S_{\mathbf{w}}$  and  $\tilde{q}^{c}$  denote the density function  $q$  restricted to  $S_{\mathbf{w}}^{c}$  and normalized. Then,
$$
K L (q | | p) = Z K L (\tilde {q} | | p) + (1 - Z) K L (\tilde {q} ^ {c} | | p) - H (Z),
$$
where  $H(Z) = -Z\ln Z - (1 - Z)\ln (1 - Z)\leq 1$  is the binary entropy function. Since KL is always positive, we get,
$$
K L (\tilde {q} | | p) = \frac {1}{Z} \left[ K L (q | | p) + H (Z)\right) - (1 - Z) K L (\tilde {q} ^ {c} | | p) ] \leq 2 (K L (q | | p) + 1).
$$
Given the local perturbation bound of the robust margin operator and Lemma 5, the proof of Theorem 1 follows the procedure of the proof of Theorem 2.
Let  $\beta = \left(\prod_{i=1}^{d}\|W_i\|_2\right)^{1/d}$  and consider a network with the normalized weights  $\widetilde{W_i} = \frac{\beta}{\|W_i\|_2} W_i$ . Due to the homogeneity of the ReLU, we have that for feedforward networks with ReLU activations
$f_{\widetilde{\mathbf{w}}} = f_{\mathbf{w}}$ , and so the (empirical and expected) loss (including margin loss) is the same for  $\mathbf{w}$  and  $\widetilde{\mathbf{w}}$ . We can also verify that  $\left(\prod_{i=1}^{d} \|W_i\|_2\right) = \left(\prod_{i=1}^{d} \left\|\widetilde{W_i}\right\|_2\right)$  and  $\frac{\|W_i\|_F}{\|W_i\|_2} = \frac{\|\tilde{W}_i\|_F}{\|\tilde{W}_i\|_2}$ , and so the excess error in the Theorem statement is also invariant to this transformation. It is therefore sufficient to prove the Theorem only for the normalized weights  $\widetilde{\mathbf{w}}$ , and hence we assume w.l.o.g. that the spectral norm is equal across layers, i.e. for any layer  $i$ ,  $\|W_i\|_2 = \beta$ .
Choose the distribution of the prior  $P$  to be  $\mathcal{N}(0,\sigma^2 I)$ , and consider the random perturbation  $\mathbf{u} \sim \mathcal{N}(0,\sigma^2 I)$ , with the same  $\sigma$ , which we will set later according to  $\beta$ . More precisely, since the prior cannot depend on the learned predictor  $\mathbf{w}$  or its norm, we will set  $\sigma$  based on an approximation  $\tilde{\beta}$ . For each value of  $\tilde{\beta}$  on a pre-determined grid, we will compute the PAC-Bayes bound, establishing the generalization guarantee for all  $\mathbf{w}$  for which  $|\beta - \tilde{\beta}| \leq \frac{1}{d}\beta$ , and ensuring that each relevant value of  $\beta$  is covered by some  $\tilde{\beta}$  on the grid. We will then take a union bound over all  $\tilde{\beta}$  on the grid. For now, we will consider a fixed  $\tilde{\beta}$  and the  $\mathbf{w}$  for which  $|\beta - \tilde{\beta}| \leq \frac{1}{d}\beta$ , and hence  $\frac{1}{e}\beta^{d-1} \leq \tilde{\beta}^{d-1} \leq e\beta^{d-1}$ .
Since  $\mathbf{u} \sim \mathcal{N}(0, \sigma^2 I)$ , we get the following bound for the spectral norm of  $U_i$  (Tropp, 2012):
$$
\mathbb {P} _ {U _ {i} \sim N (0, \sigma^ {2} I)} \left[ \| U _ {i} \| _ {2} > t \right] \leq 2 h e ^ {- t ^ {2} / 2 h \sigma^ {2}}.
$$
Taking a union bond over the layers, we get that, with probability  $\geq \frac{1}{2}$ , the spectral norm of the perturbation  $U_{i}$  in each layer is bounded by  $\sigma \sqrt{2h\ln(4dh)}$ . Plugging this spectral norm bound into the Lipschitz of robust margin operator we have that with probability at least  $\frac{1}{2}$ ,
$$
\begin{array}{l} \max  _ {i, j \in [ k ], \mathbf {x} \in \mathcal {X}} | R M \left(f _ {\mathbf {w} ^ {\prime}} (\mathbf {x}), i, j\right) - R M \left(f _ {\mathbf {w}} (\mathbf {x}), i, j\right) | (10) \\ \leq 2 \epsilon (B + \epsilon) \beta^ {d} \sum_ {i} \frac {\| U _ {i} \| _ {2}}{\beta} \\ = e (B + \epsilon) \beta^ {d - 1} \sum_ {i} \| U _ {i} \| _ {2} \leq e ^ {2} d (B + \epsilon) \tilde {\beta} ^ {d - 1} \sigma \sqrt {2 h \ln (4 d h)} \leq \frac {\gamma}{2}, (11) \\ \end{array}
$$
where we choose  $\sigma = \frac{\gamma}{42d(B + \epsilon)\tilde{\beta}^{d - 1}\sqrt{h\ln(4hd)}}$  to get the last inequality, the first inequality is Lemma 6.2. The second inequality is the tail bound above. Hence, the perturbation  $\mathbf{u}$  with the above value of  $\sigma$  satisfies the assumptions of the Lemma 4.
We now calculate the KL-term in Lemma 4 with the chosen distributions for  $P$  and  $\mathbf{u}$ , for the above value of  $\sigma$ .
$$
\begin{array}{l} K L (\mathbf {w} + \mathbf {u} | | P) \\ \leq \frac {\left| \mathbf {w} \right| ^ {2}}{2 \sigma^ {2}} = \frac {4 2 ^ {2} d ^ {2} (B + \epsilon) ^ {2} \tilde {\beta} ^ {2 d - 2} h \ln (4 h d)}{2 \gamma^ {2}} \sum_ {i = 1} ^ {d} \| W _ {i} \| _ {F} ^ {2} \\ \leq \mathcal {O} \left((B + \epsilon) ^ {2} d ^ {2} h \ln (d h) \frac {\beta^ {2 d}}{\gamma^ {2}} \sum_ {i = 1} ^ {d} \frac {\| W _ {i} \| _ {F} ^ {2}}{\beta^ {2}}\right) \\ \leq \mathcal {O} \left(\left(B + \epsilon\right) ^ {2} d ^ {2} h \ln (d h) \frac {\Pi_ {i = 1} ^ {d} \| W _ {i} \| _ {2} ^ {2}}{\gamma^ {2}} \sum_ {i = 1} ^ {d} \frac {\| W _ {i} \| _ {F} ^ {2}}{\| W _ {i} \| _ {2} ^ {2}}\right). \\ \end{array}
$$
Hence, for any  $\tilde{\beta}$ , with probability  $\geq 1 - \delta$  and for all  $\mathbf{w}$  such that,  $|\beta -\tilde{\beta} |\leq \frac{1}{d}\beta$ , we have:
$$
R _ {0} \left(f _ {\mathbf {w}}\right) \leq \hat {R} _ {\gamma} \left(f _ {\mathbf {w}}\right) + \mathcal {O} \left(\sqrt {\frac {(B + \epsilon) ^ {2} d ^ {2} h \ln (d h) \Pi_ {i = 1} ^ {d} \| W _ {i} \| _ {2} ^ {2} \sum_ {i = 1} ^ {d} \frac {\| W _ {i} \| _ {F} ^ {2}}{\| W _ {i} \| _ {2} ^ {2}} + \ln \frac {m}{\delta}}{\gamma^ {2} m}}\right). \tag {12}
$$
For other  $\ell_p$  attacks, the results are directly obtained by Lemma 4 of (Xiao et al., 2022a).
It is based on a slight modification of the key lemma. if  $g_{\mathbf{w}}(\mathbf{x})$  has a  $(A_{1}|\mathbf{x}|,\dots ,A_{d}|\mathbf{x}|)$ -local perturbation bound, i.e.,
the robustified function  $\inf_{\mathbf{x}' \in C(\mathbf{x})} g_{\mathbf{w}}(\mathbf{x}')$  has a  $(A_1D, \dots, A_dD)$ -local perturbation bound.
Proof: Let
$$
\mathbf{x}(\mathbf{w}) = \arg \inf_{\mathbf{x}^{\prime}\in C(\mathbf{x})}g_{\mathbf{w}}(\mathbf{x}^{\prime}),
$$
$$
\mathbf{x}(\mathbf{w}^{\prime}) = \arg \inf_{\mathbf{x}^{\prime}\in C(\mathbf{x})}g_{\mathbf{w}^{\prime}}(\mathbf{x}^{\prime}),
$$
$$
\left| \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w}} \left(\mathbf {x} ^ {\prime}\right) - \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w} ^ {\prime}} \left(\mathbf {x} ^ {\prime}\right) \right| \leq
$$
$$
\max  \left\{\left| g _ {\mathbf {w}} (\mathbf {x} (\mathbf {w})) - g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} (\mathbf {w})) \right|, \left| g _ {\mathbf {w}} (\mathbf {x} (\mathbf {w} ^ {\prime})) - g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} (\mathbf {w} ^ {\prime})) \right| \right\}.
$$
$$
\begin{array}{l} | \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w}} (\mathbf {x} ^ {\prime}) - \inf  _ {\| \mathbf {x} - \mathbf {x} ^ {\prime} \| \leq \epsilon} g _ {\mathbf {w} ^ {\prime}} (\mathbf {x} ^ {\prime}) | \\ \leq \sum_ {i = 1} ^ {d} A _ {i} | \mathbf {x} (\mathbf {w}) | \| W _ {i} - W _ {i} ^ {\prime} \| \\ \leq \sum_ {i = 1} ^ {d} A _ {i} D \| W _ {i} - W _ {i} ^ {\prime} \|. \\ \end{array}
$$
Therefore, combining the local perturbation bound and Lemma 7.2, we complete the proof.
A.5 Proof of Theorem 9
As shown in the proof of Lemma 6, it is left to bound  $\| f_{\mathbf{w} + \mathbf{u}}(\mathbf{x}) - f_{\mathbf{w}}(\mathbf{x})\|$ . Let  $\Delta_i = \left|f_{\mathbf{w} + \mathbf{u}}^i (\mathbf{x}) - f_{\mathbf{w}}^i (\mathbf{x})\right|_2$ . We will prove using induction that for any  $i\geq 0$ :
$$
\Delta_ {i} \leq \left(1 + \frac {1}{d}\right) ^ {i} \left(\prod_ {j = 1} ^ {i} (\| W _ {j} \| _ {2} + 1)\right) | \mathbf {x} | _ {2} \sum_ {j = 1} ^ {i} \frac {\| U _ {j} \| _ {2}}{(\| W _ {j} \| _ {2} + 1)}.
$$
$$
\begin{array}{l} \Delta_ {i + 1} = \left| \left(W _ {i + 1} + U _ {i + 1}\right) \phi_ {i} \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x})\right) - W _ {i + 1} \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) + \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x}) - f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} \\ = \left| \left(W _ {i + 1} + U _ {i + 1}\right) \left(\phi_ {i} \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x})\right) - \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right)\right) + U _ {i + 1} \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) + \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x}) - f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} \\ \leq \left(\left\| W _ {i + 1} \right\| _ {2} + \left\| U _ {i + 1} \right\| _ {2}\right) \left| \phi_ {i} \left(f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x})\right) - \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} + \left\| U _ {i + 1} \right\| _ {2} \left| \phi_ {i} \left(f _ {\mathbf {w}} ^ {i} (\mathbf {x})\right) \right| _ {2} + \Delta_ {i} \\ \leq \left(\left\| W _ {i + 1} \right\| _ {2} + \left\| U _ {i + 1} \right\| _ {2}\right) \left| f _ {\mathbf {w} + \mathbf {u}} ^ {i} (\mathbf {x}) - f _ {\mathbf {w}} ^ {i} (\mathbf {x}) \right| _ {2} + \left\| U _ {i + 1} \right\| _ {2} \left| f _ {\mathbf {w}} ^ {i} (\mathbf {x}) \right| _ {2} + \Delta_ {i} \\ = \Delta_ {i} \left(\| W _ {i + 1} \| _ {2} + \| U _ {i + 1} \| _ {2} + 1\right) + \| U _ {i + 1} \| _ {2} \left| f _ {\mathbf {w}} ^ {i} (\mathbf {x}) \right| _ {2}, \\ \end{array}
$$
where the last inequality is by the Lipschitz property of the activation function and using  $\phi(0) = 0$ . The  $\ell_2$  norm of outputs of layer  $i$  is bounded by  $|\mathbf{x}|_2 \Pi_{j=1}^i (\|W_j\|_2 + 1)$  and by the lemma assumption we have  $\|U_{i+1}\|_2 \leq \frac{1}{d} \|W_{i+1}\|_2$ . Therefore, using the induction step, we get the following bound:
$$
\begin{array}{l} \Delta_ {i + 1} \leq \Delta_ {i} \left(1 + \frac {1}{d}\right) \left(\| W _ {i + 1} \| _ {2} + 1\right) + \| U _ {i + 1} \| _ {2} | \mathbf {x} | _ {2} \prod_ {j = 1} ^ {i} \left(\| W _ {j} \| _ {2} + 1\right) \\ \leq \left(1 + \frac {1}{d}\right) ^ {i + 1} \left(\prod_ {j = 1} ^ {i + 1} \left(\| W _ {j} \| _ {2} + 1\right)\right) | \mathbf {x} | _ {2} \sum_ {j = 1} ^ {i} \frac {\| U _ {j} \| _ {2}}{\left(\| W _ {j} \| _ {2} + 1\right)} + \frac {\| U _ {i + 1} \| _ {2}}{\left(\| W _ {i + 1} \| _ {2} + 1\right)} | \mathbf {x} | _ {2} \prod_ {j = 1} ^ {i + 1} \left(\| W _ {i} \| _ {2} + 1\right) \\ \leq \left(1 + \frac {1}{d}\right) ^ {i + 1} \left(\prod_ {j = 1} ^ {i + 1} \left(\| W _ {j} \| _ {2} + 1\right)\right) | \mathbf {x} | _ {2} \sum_ {j = 1} ^ {i + 1} \frac {\| U _ {j} \| _ {2}}{\left(\| W _ {j} \| _ {2} + 1\right)}. \\ \end{array}
$$
Therefore, the margin operator of ResNet is locally  $(A_{1}|\mathbf{x}|,\dots ,A_{d}|\mathbf{x}|)$ -Lipschitz w.r.t.  $w$ , where
$$
A _ {i} = 2 e \prod_ {l = 1} ^ {d} \left(\| W _ {l} \| _ {2} + 1\right) / \left(\| W _ {i} \| _ {2} + 1\right).
$$
For any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
\begin{array}{l} L _ {0} (f _ {\mathrm {R N}}) - \hat {L} _ {\gamma} (f _ {\mathrm {R N}}) \\ \leq \mathcal {O} \left(\sqrt {\frac {B ^ {2} d ^ {2} h \ln (d h) \Phi (f _ {\mathrm {R N}}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right); \\ \end{array}
$$
By a combination of Lemma 5 and Lemma 7, for any  $\delta, \gamma > 0$ , with probability  $\geq 1 - \delta$  over a training set of size  $m$ , for any  $\mathbf{w}$ , we have:
$$
\begin{array}{l} R _ {0} \left(f _ {\mathrm {R N}}\right) - \hat {R} _ {\gamma} \left(f _ {\mathrm {R N}}\right) \\ \leq \mathcal {O} \left(\sqrt {\frac {(B + \epsilon) ^ {2} d ^ {2} h \ln (d h) \Phi (f _ {\mathrm {R N}}) + \ln \frac {d m}{\delta}}{\gamma^ {2} m}}\right), \\ \end{array}
$$
where  $\Phi (f_{\mathrm{RN}}) = \Pi_{i = 1}^{d}(\| W_{i}\|_{2} + 1)^{2}\sum_{i = 1}^{d}\frac{\|W_{i}\|_{F}^{2}}{(\|W_{i}\|_{2} + 1)^{2}}.$
B PAC-Bayesian Framework for Robust Generalization
PAC-Bayes analysis (McAllester, 1999) is a framework to provide generalization guarantees for randomized predictors drawn from a learned distribution  $Q$  (as opposed to a single predictor) that depends on the training data set. The expected generalization gap over the posterior distribution  $Q$  can be bounded in terms of the Kullback-Leibler divergence between the prior distribution  $P$  and the posterior distribution  $Q$ ,  $KL(P\|Q)$ .
A direct corollary of Eq. (5) is that, the expected robust error of  $f_{\mathbf{w} + \mathbf{u}}$  can be bounded as follows
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {u}} \left[ R _ {0} ^ {a d v} \left(f _ {\mathbf {w} + \mathbf {u}}\right) \right] \\ \leq \mathbb {E} _ {\mathbf {u}} \left[ \hat {R} _ {0} ^ {a d v} \left(f _ {\mathbf {w} + \mathbf {u}}\right) \right] + 2 \sqrt {\frac {2 \left(K L \left(\mathbf {w} + \mathbf {u} \| P\right) + \ln \frac {2 m}{\delta}\right)}{m - 1}}. \tag {13} \\ \end{array}
$$
By a slight modification of Lemma 4, the following lemma given in the work of (Farnia et al., 2018) shows how to obtain an robust generalization bound.