SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 10 (Farnia et al. (2018)). Let  $f_{\mathbf{w}}(\mathbf{x}) : \mathcal{X} \to \mathbb{R}^k$  be any predictor (not necessarily a neural network) with parameters  $\mathbf{w}$ , and  $P$  be any distribution on the parameters that is independent of the training data. Then, for any  $\gamma, \delta > 0$ , with probability  $\geq 1 - \delta$  over the training set of size  $m$ , for any  $\mathbf{w}$ , and any random perturbation  $\mathbf{u}$  s.t.  $\mathbb{P}_{\mathbf{u}}[\max_{\mathbf{x} \in \mathcal{X}} |f_{\mathbf{w} + \mathbf{u}}(\mathbf{x} + \delta_{\mathbf{w} + \mathbf{u}}^{adv}(\mathbf{x})) - f_{\mathbf{w}}(\mathbf{x} + \delta_{\mathbf{w}}^{adv}(\mathbf{x}))|_{\infty} < \frac{\gamma}{4}] \geq \frac{1}{2}$ , we have:
$$
R _ {0} ^ {a d v} (f _ {\mathbf {w}}) \leq \hat {R} _ {\gamma} ^ {a d v} (f _ {\mathbf {w}}) + 4 \sqrt {\frac {K L (\mathbf {w} + \mathbf {u} \| P) + \ln \frac {6 m}{\delta}}{m - 1}}.
$$
Table 1: Comparison of the empirical results of the standard generalization bound and robust generalization in the experiment of training MNIST, CIFAR-10 and CIFAR-100 on VGG networks.
MNIST	CIFAR-10	CIFAR-100
Standard Generalization Gap	1.13%	9.21%	23.61%
Bound in Theorem 2 (Neyshabur et al., 2017b)	1.33 × 104	1.34 × 109	3.41 × 1011
Robust Generalization Gap	9.67%	51.41%	78.82%
Bound in Theorem 3 (Farnia et al., 2018)	NA	NA	NA
Bound in Theorem 1 (Ours)	3.23 × 104	5.97 × 1010	1.66 × 1013
C Empirical Study of the Generalization Bounds
The spectral complexity  $\Phi(f_{\mathbf{w}})$  induced by adversarial training is significantly larger. We conducted experiments training MNIST, CIFAR-10, and CIFAR-100 datasets using VGG-19 networks, following
the training parameters described in (Neyshabur et al., 2017a). The results are presented in Table 1. It is evident that adversarial training can induce a larger spectral complexity, resulting in a larger generalization bound. We refer the readers to our previous work (Xiao et al., 2022a) for more experiments results about norm-based complexity of adversially-trained models. These experiments align with the findings presented by (Bartlett et al., 2017), indicating: 1) spectral complexity scales with the difficulty of the learning task, and 2) the generalization bound is sensitive to this complexity.
4The settings of standard training follows the experiments in https://github.com/bneyshabur/generalization-bounds.
5The settings of adversarial training follows the experiments in https://github.com/JiancongXiao/Adversarial-Rademacher-Complexity.