SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 5.1 (Upper bounds of adversarial loss). Suppose that the input dimension  $d$ , output dimension  $K$ , and width  $N$  are sufficiently large. Then, for any  $\pmb{x}^{\mathrm{in}} \in \mathbb{R}^{d}$ , the following inequality holds:
$$
\mathcal {L} _ {\mathrm {a d v}} \left(\boldsymbol {x} ^ {\text {i n}}\right) \leq \epsilon \beta_ {p, q} \omega^ {L / 2} = \left\{ \begin{array}{l l} \epsilon \beta_ {p, q} \left(\frac {\alpha}{L N} \sum_ {W \in \mathcal {W}} W ^ {2}\right) ^ {L / 2} & (\text {v a n i l l a}) \\ \epsilon \beta_ {p, q} \left(1 + \frac {\alpha}{L N} \sum_ {W \in \mathcal {W}} W ^ {2}\right) ^ {L / 2} & (\text {r e s i d u a l}) \end{array} , \right. \tag {9}
$$
where  $\mathcal{W} := \{W_{1,1}^{(1)}, W_{1,2}^{(1)}, \ldots, W_{N,N}^{(L)}\}$  denotes the set of all network weights. The constant  $\beta_{p,q}$  for each norm pair  $(p, q)$  is described in Tab. 1.
For some choices of  $(p,q)$ , we cannot derive upper bounds, and thus, Tab. 1 contains blank. Numerical experiments show the tightness of the bounds (cf. Fig. 2). The theorem indicates that (i) the bounds increase linearly with the perturbation budget  $\epsilon$ , (ii) the effects of the input and output dimensions depend on the norms  $(p,q)$  (cf. Tab. 1), and (iii) network depth  $L$  exponentially impacts the bounds, with  $\omega = 1$  as a threshold between order and chaos. Further, the square sum of the weights in Ineq. (9) suggests that adversarial training exhibits a weight regularization effect, which is compared to  $\ell_2$  weight regularization in Appx. I. Besides, we derive equality rather than inequality (upper bound) under specific assumptions (e.g., small  $\epsilon$ ) for some  $(p,q)$  in Appx. K, indicated by  $\dagger$  and  $\diamondsuit$  in Tab. 1.
The input and output dimensions,  $d$  and  $K$ , influence the bounds through the  $(p,q)$ -dependent parameter  $\beta_{p,q}$ . In Tab. 1,  $\beta_{p,q}$  displays a wide range of dependencies on  $d$  and  $K$ . When  $d \to \infty$  and  $q = \infty$ ,  $d$  significantly affects the two phases of the adversarial loss, where  $p = 2$  marks the transition point from order  $(p = 1)$  to chaos  $(p = \infty)$ . In contrast, under realistic assumptions with  $d \gg K$ ,  $K$  affects negligibly. Interestingly, the dimensions do not impact the upper bounds when  $(p,q) = (2,\infty)$ . In practice, we scale the perturbation budget and adversarial loss according to the choice of  $(p,q)$ , respectively, and discuss the scaling effects in Appx. K.
Comparison with other studies. We can consider studies on global Lipschitz of networks in certified adversarial defenses [3, 18, 93] and spectral regularization [60, 108] to analyze Ineq. (8). A key difference is that the proposed probabilistic approach contradicts their deterministic approach. By imposing probabilistic constraints on network parameters, we can obtain exponentially tighter, interpretable, and more theoretically manageable bounds, which facilitates the subsequent section's discussion. The mathematical comparison is described in Appx. H.
Results obtained from [15, 36, 69, 70, 100] can be used to analyze the Jacobian's singular value distribution. Compared to their approaches, which are limited to  $(p,q) = (2,2)$ , the proposed method offers greater generality and flexibility, providing upper bounds for various  $(p,q)$ . Moreover, Thm 4.1 enables the derivation of equality instead of inequality (upper bound), Props G.2 and G.3, which is not achievable using the approaches mentioned in the aforementioned studies because it cannot incorporate perturbation-Jacobian dependence. Moreover, we do not consider their assumption that the variance  $\mathbb{V}[h_i^{(l)}]$  is constant for all  $l\in [L]$ , which is often difficult to satisfy.
Further, we refer to [78], which established a theoretical link between adversarial training and the  $(p,q)$ -operator norm of a Jacobian. Their findings support Ineq. (8) in training scenarios using heuristic attacks, e.g., projected gradient descent [58]. In this study, we derive concrete upper bounds beyond their theoretical link, enabling further investigation of adversarial training properties.
5.2 Time evolution of weight variance
Weight variance plays a critical role in determining deep neural network properties [49, 81]. We substitute the adversarial loss definition (Eq. (3)) with  $\mathcal{L}_{\mathrm{adv}} \coloneqq \epsilon \beta_{p,q} \omega(t)^{L/2}$ , where  $t \geq 0$  denotes the
continuous training step. Considering gradient descent with an infinitely small learning rate (gradient flow), the model parameter  $\theta(t)$  at step  $t$  is updated as:
$$
\frac {\mathrm {d} \theta (t)}{\mathrm {d} t} := - \frac {\partial \mathcal {L} _ {\mathrm {s t d}}}{\partial \theta (t)} - \frac {\partial \mathcal {L} _ {\mathrm {a d v}}}{\partial \theta (t)}. \tag {10}
$$
We make the following assumption.
Assumption 5.2. For  $0 \leq t \leq T \ll N$ , model parameters are independent, and weight and bias follow Gaussian  $\mathcal{N}(0, \sigma_w^2(t)/N)$  and  $\mathcal{N}(0, \sigma_b^2(t))$ , respectively.
This assumption ensures that the properties of the model parameters remain close to their initial values during the early stages of training ( $t \leq T$ ). Under Asm 5.2, the original and proposed mean field theories (Thm 4.1) remain valid during training. For a moderately small value of  $T$ , Asm 5.2 is not strong because the model parameters change minimally and retain their initialized states with sufficiently small learning rates. Recent neural tangent kernel studies partially supported this assumption [4, 46, 55], and it is known that random network theories align well with fully trained networks [25, 84]. For example,  $T = 160$  is reasonable in a specific training setting (cf. Fig. 4).
Now, we summarize other assumptions for reference as follows:
Assumption 5.3. The input dimension  $d$ , output dimension  $K$ , and width  $N$  are sufficiently large. We apply Asm B.1 to the standard loss function  $\mathcal{L}_{\mathrm{std}}$ . The adversarial loss is defined as  $\mathcal{L}_{\mathrm{adv}} := \epsilon \beta_{p,q} \omega(t)^{L/2}$  (cf. Thm 5.1).
Based on the aforementioned settings, we obtain the time evolution of the weight variance.
Theorem 5.4 (Weight time evolution of vanilla network in adversarial training). Suppose that Asms 5.2 and 5.3 hold. Then, the time evolution of  $\sigma_w^2$  of a vanilla network in adversarial training is given by:
$$
\sigma_ {w} ^ {2} (t) = \left(1 - \frac {\epsilon \alpha \beta_ {p , q} \omega_ {\mathrm {v}} (0) ^ {L / 2 - 1}}{N} t\right) \sigma_ {w} ^ {2} (0). \tag {11}
$$
A similar result is obtained for residual networks (Thm G.9). The theorem reveals that the weight variance linearly decreases with  $t$ , which can be attributed to the weight regularization effect of adversarial training (cf. Sec. 5.1). In addition, the norm pair  $(p,q)$  affects only time-invariant constant  $\beta_{p,q}$ , and the dynamics of the weight variance can be represented as a consistent function of  $t$  regardless of  $(p,q)$ . In other words, adversarial training exhibits consistent weight dynamics irrespective of norm selection, with a scale factor varying.
5.3 Vanilla networks are not adversarially trainable under mild conditions
We show that in adversarial training, vanilla networks can fit a training dataset in limited cases (small depth and large width), but residual networks can in most cases. This result suggests that residual networks are better suited for adversarial training. First, we present the trainability condition based on the concept in [81, 105].
Definition 5.5  $((M, m)$ -trainability condition). A network is said to be  $(M, m)$ -trainable if a network satisfies  $m \leq \chi^{(0)} / \chi^{(L)} \leq M$ , where  $0 \leq m \leq 1$  and  $M \geq 1$ .<sup>2</sup>
The value  $\chi^{(l)}$  denotes the squared length of the gradient in the  $l$ -th layer. A near-zero  $\chi^{(0)} / \chi^{(L)}$  suggests gradient vanishing, while a large value implies gradient explosion. Hence, Defn 5.5 is directly linked to successful training. In contrast to the existing definition,  $\chi^{(l-1)} / \chi^{(l)} \approx 1$  [81, 105], Defn 5.5 incorporates  $M$  and  $m$  for the subsequent discussion. Then, we establish specific  $(M, m)$ -trainability conditions for ReLU-like networks.
Lemma 5.6 (Vanilla and residual  $(M,m)$ -trainability condition). Suppose that the width  $N$  is sufficiently large. Then, the  $(M,m)$ -trainability conditions for vanilla and residual networks are respectively given by:
$$
m ^ {1 / L} \leq \alpha \sigma_ {w} ^ {2} \leq M ^ {1 / L} (\text {v a n i l l a}), \quad \alpha \sigma_ {w} ^ {2} \leq M ^ {1 / L} - 1 (\text {r e s i d u a l}). \tag {12}
$$
${}^{2}$  Trainability depends on other factors such as the number of parameters; however, these are not considered herein to maintain simplicity. It is empirically known that this condition represents trainability well [81].
Using the weight time evolution (Thm 5.4) and  $(M, m)$ -trainability condition (Lemma 5.6), the following theorem can be readily derived.
Theorem 5.7 (Vanilla networks are not adversarially trainable). Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, and the  $(M,m)$ -trainability condition holds at  $t = 0$  and  $\alpha \sigma_w^2(0) = 1$ . If
$$
T \geq \frac {\left(1 - m ^ {1 / L}\right) N}{\epsilon \alpha \beta_ {p , q}}, \tag {13}
$$
then there exists  $0 < \tau \leq T$  such that the  $(M,m)$ -trainability condition does not hold for  $\tau \leq t \leq T$ .
This indicates the potential untrainability of vanilla networks in adversarial training, even when satisfying the  $(M,m)$ -trainability condition at initialization, contradicting with standard training where extremely deep networks can be trained if initialized properly [81]. This issue arises from the inconsistency between the trainability condition of a vanilla network, i.e.,  $m^{1 / L} \leq \alpha \sigma_w^2$  (cf. Lemma 5.6) and monotonically decreasing nature of  $\sigma_w^2(t)$  in adversarial training (cf. Thm 5.4). As stated in Asm 5.2, we assume that  $T$  is small. Therefore, if the right-hand term of Eq. (13) becomes large, the assumption and Thm 5.7 are violated. In summary, for large  $L$  (many layers) and  $\epsilon$  (large perturbation constraint), vanilla networks are not adversariably trainable. Moreover, a large  $N$  (a wide network) can mitigate this issue in vanilla networks. For example, the vanilla network with  $L = 20$ ,  $N = 256$ , and  $\epsilon = 0.3$  is not adversarially trainable; however, when the width is increased to 512, it becomes trainable (cf. Fig. 4). In contrast, we can claim the following for residual networks:
Theorem 5.8 (Residual networks are adversarially trainable). Consider a residual network. Suppose that Asms 5.2 and 5.3 hold, and the  $(M,m)$ -trainability condition holds at  $t = 0$  and  $\alpha \sigma_w^2(0) \ll 1$ . Then,  $(M,m)$ -trainability condition always holds for  $0 \leq t \leq T$ .
This occurs due to the  $(M,m)$ -trainability condition for residual networks, which does not have any lower bound (cf. Lemma 5.6), and the monotonically decreasing nature of  $\sigma_w^2$  (cf. Thm G.9). Besides, residual networks are adversarially trainable even without careful weight initialization (Prop G.10). These theorems highlight the robust stability of adversarial training in residual networks.
5.4 Degradation of network capacity
We demonstrate that adversarial training degrades network capacity. We use the Fisher-Rao norm as a metric [56], with alternative metrics discussed in Appx. K. The Fisher-Rao norm is defined as:
$$
\left\| \boldsymbol {w} \right\| _ {\mathrm {F R}} := \boldsymbol {w} ^ {\top} \boldsymbol {F} \left(\boldsymbol {x} ^ {\text {i n}}\right) \boldsymbol {w}, \quad \boldsymbol {F} \left(\boldsymbol {x} ^ {\text {i n}}\right) := \sum_ {i = 1} ^ {K} \left(\frac {\partial f _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial \boldsymbol {w}}\right) ^ {\top} \frac {\partial f _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial \boldsymbol {w}}, \tag {14}
$$
where  $\boldsymbol{w} := (W_{11}^{(1)}, W_{12}^{(1)}, \ldots, W_{NN}^{(L)})^\top$  represents the vector of all the network weights and  $\boldsymbol{F}$  denotes the empirical Fisher information matrix when only one training data point is considered (for simplicity). The Fisher-Rao norm is preferred over other norm-based capacity metrics [8, 65-67] owing to its invariance to node-wise rescaling [56]. We present the following theorem based on [49]:
Theorem 5.9 (Adversarial training degrades network capacity). Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, Asm B.1 is applied to the network output, and the  $(M,m)$ -trainability condition holds at  $t = 0$  and  $\alpha \sigma_w^2(0) = 1$ . Assume  $\left\| \boldsymbol{x}^{\mathrm{in}} \right\|_2 = \sqrt{d}$  and  $\sigma_b^2(t) = 0$ . Then, the expectation of the Fisher-Rao norm is given by:
$$
\mathbb {E} [ \| \boldsymbol {w} (t) \| _ {\mathrm {F R}} ] = L K \left(1 - \frac {\epsilon \alpha \beta_ {p , q} L}{N} t\right). \tag {15}
$$
A related result for residual networks is presented in Thm G.14. As described in [63], adversarial robustness necessitates high capacity. However, Thm 5.9 indicated that capacity decreases linearly with step  $t$  in adversarial training. To address this conflict, large depth (large  $L$ ), which increase the initial capacity with  $\Theta(L)$ , can be considered, but this scenario accelerates the degradation speed with  $\Theta(L^2)$ . To preserve capacity, we must increase the width  $N$  accordingly. Consequently, to achieve high capacity in adversarial training, it is necessary to increase not only the number of layers  $L$  but also the width  $N$  to keep  $L^2 / N$  constant. Although Thms 5.9 and G.14 have been established in the early stages of training, numerical experiments proved that the adversarial robustness following full training is significantly influenced by network width (cf. Tabs. A5 and A6).
Figure 2: Adversarial loss (Eq. (3)) in vanilla networks with  $N = 40,000$ ,  $K = 100$ ,  $L = 3$ , and  $\epsilon = 0.1$ . We generated 100 adversarial examples for each input dimension. The blue curves and bands represent the mean and standard deviation of the adversarial loss, respectively, whereas the orange curves (upper bounds) are predicted based on Thm 5.1. Some samples slightly exceed the upper bounds because we used the finite network width (cf. Appx. L).
Figure 3: Time evolution of the weight variance Figure 4: Heat map of the training accuracy of in the vanilla network with  $L = 10$ ,  $p = \infty$ , vanilla networks with  $p = \infty$ ,  $q = \infty$ , and  $\epsilon = q = \infty$ , and  $\epsilon = 0.3$ . We used  $N = 1,000$  for 0.3. The dashed lines represent the condition of standard and  $\ell_2$  regularized training. The solid  $T$  in Thm 5.7 with  $m = 0.0001$ . In standard lines represent experimental results. The dashed training, high accuracy is obtained across all the lines are predicted by Thm 5.4. depths and widths (cf. Fig. A19).
6 Experimental results
We validate Thms 5.1, 5.4 and 5.7 via numerical experiments. The vanilla ReLU networks were initialized with  $\sigma_w^2 = 2$  and  $\sigma_b^2 = 0.01$  to meet the  $(M,m)$ -trainability conditions (Lemma 5.6). To verify Thms 5.4 and 5.7, we used MNIST [26]. Setup details and more results are given in Appx. L.
Verification of Thm 5.1. We created adversarial examples for initialized networks and computed the adversarial loss (Eq. (3)). As shown in Fig. 2, the upper bounds in Thm 5.1 were considerably tight. Some samples slightly exceeded the upper bounds because we used the finite network width while the infinite width is assumed (cf. Appx. L).
Verification of Thm 5.4. We trained vanilla networks normally (with and without  $\ell_2$  regularization) and adversarially. The time evolution of weight variance is shown in Fig. 3. Adversarial training significantly reduces weight variance, whereas wide width (i.e., large  $N$ ) suppresses it. The validity of Thm 5.4, which forms the basis of our theorems such as Thms 5.7 and 5.9, supports our theorems.
Verification of Thm 5.7. We trained vanilla networks considering various depth and width settings and monitored the training accuracy. As shown in Fig. 4, it was difficult for vanilla networks to fit the training dataset when the depth was large and the width was small; increased width helps in fitting. Although we currently lack a theoretical prediction of the boundary between trainable and untrainable areas determined based on Eq. (13), empirical evidence suggests that  $T = 160$  is relevant.
The mean field theory offers valuable insight into network training. However, its applicability is restricted to the initial stages of training. Although recent studies suggested empirically [25, 84] and theoretically [46] that the analysis of early-stage training extends well to full training, the strict relationship is yet to be explored. Our results have the same limitations. Although some theorems accurately capture the behavior during the initial stages of training and even after full training (cf. Tabs. A5 and A6), as training progresses, some theorems begin to diverge from the actual behavior (cf. Fig. A17). Another caveat is that the mean field theory assumes infinite network width, which is practically infeasible. Empirically, our theorems hold well when the width approximately exceeds 1,000 (cf. Fig. A7), while Thm 5.1 requires larger width approximately exceeds 10,000 for  $(p,q) = (2,2)$  (cf. Fig. A14). These limitations also derive from the mean field theory and are not unique to our study. Despite these limitations, we consider that this study provides a powerful theoretical framework that extends the applicability of the mean field theory to various training methods, and the results obtained for adversarial training are insightful.
8 Conclusions
We proposed a framework based on the mean field theory and conducted a theoretical analysis of adversarial training. The proposed framework addressed the limitations of existing mean field-based approaches, which could not handle the probabilistic properties of an entire network and dependence between network inputs and parameters [71, 81]. Based on this framework, we examined adversarial training from various perspectives, unveiling upper bounds of adversarial loss, relationships between adversarial loss and network input/output dimensions, the time evolution of weight variance, trainability conditions, and the degradation of network capacity. The theorems of this study were validated via numerical experiments. The proposed theoretical framework is highly versatile and can help analyze various training methods, e.g., contrastive learning.
We would like to thank Huishuai Zhang for useful discussions. This work was supported by JSPS KAKENHI Grant Number JP23KJ0789 and JP22K17962, by JST, ACT-X Grant Number JPM-JAX23C7, JAPAN, and by Microsoft Research Asia.
[1] J.-B. Alayrac, J. Uesato, P.-S. Huang, A. Fawzi, R. Stanforth, and P. Kohli. Are labels required for improving adversarial robustness? In NeurIPS, volume 32, 2019.
[2] L. Amsaleg, J. Bailey, D. Barbe, S. Erfani, M. E. Houle, V. Nguyen, and M. Radovanovic. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In WIFS, pages 1-6, 2017.
[3] C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In ICML, pages 291-301, 2019.
[4] S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In NeurIPS, volume 32, 2019.
[5] A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, pages 274-283, 2018.
[6] P. Awasthi, N. Frank, and M. Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In ICML, pages 431-441, 2020.
[7] P. Bartlett, S. Bubeck, and Y. Cherapanamjeri. Adversarial examples in multi-layer random relu networks. In NeurIPS, volume 34, pages 9241-9252, 2021.
[8] P. L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-normalized margin bounds for neural networks. In NeurIPS, volume 30, 2017.
[9] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 3(Nov):463-482, 2002.
[10] Y. Blumenfeld, D. Gilboa, and D. Soudry. A mean field theory of quantized deep networks: The quantization-depth trade-off. In NeurIPS, volume 32, 2019.
[11] S. Bubeck, Y. Cherapanamjeri, G. Gidel, and R. Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. In NeurIPS, volume 34, pages 10081-10091, 2021.
[12] N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In ACM WS, pages 3-14, 2017.
[13] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In SSP, pages 39-57, 2017.
[14] Y. Carmon, A. Raghunathan, L. Schmidt, P. Liang, and J. C. Duchi. Unlabeled data improves adversarial robustness. In NeurIPS, 2019.
[15] M. Chen, J. Pennington, and S. Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. In ICML, pages 873-882, 2018.
[16] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.
[17] Y. Cho and L. Saul. Kernel methods for deep learning. In NeurIPS, volume 22, 2009.
[18] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In ICML, pages 854-863, 2017.
[19] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In ICML, pages 1310-1320, 2019.
[20] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, pages 2206-2216, 2020.
[21] A. Damianou and N. D. Lawrence. Deep gaussian processes. In AISTATS, pages 207-215, 2013.
[22] A. Daniely. SGD learns the conjugate kernel class of the network. In NeurIPS, volume 30, 2017.
[23] A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In NeurIPS, volume 29, 2016.
[24] A. Daniely and H. Schacham. Most reli networks suffer from ell² adversarial perturbations. In NeurIPS, volume 33, pages 6629–6636, 2020.
[25] G. De Palma, B. Kiani, and S. Lloyd. Adversarial robustness guarantees for random deep neural networks. In ICML, pages 2522-2534, 2021.
[26] L. Deng. The MNIST database of handwritten digit images for machine learning research. Signal Process. Mag., 29(6):141-142, 2012.
[27] Z. Deng, L. Zhang, K. Vodrahalli, K. Kawaguchi, and J. Y. Zou. Adversarial training helps transfer learning via better representations. In NeurIPS, volume 34, pages 25179-25191, 2021.
[28] G. W. Ding, Y. Sharma, K. Y. C. Lui, and R. Huang. MMA training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
[29] E. Dobriban, H. Hassani, D. Hong, and A. Robey. Provable tradeoffs in adversarially robust classification. arXiv:2006.05161, 2020.