SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Other metrics of capacity. Here, we consider capacity metrics other than the Fisher-Rao norm. Norm-based metrics, such as the path norm [66], the group norm [67], and spectral norm [8], have a similar definition to the Fisher-Rao norm. They are constructed by the sum and product of the weights of a network. The trajectory length [73] is defined by the arc length of a network output with change of parameters. A similar definition based on curvature was also used in [71]. These metrics concluded that capacity increases with the weight variance of a network, despite differences in speed. Therefore, we can derive similar results as in Sec. 5.4 for metrics other than the Fisher-Rao norm.
Mitigation of adversarial risk. Here, we consider the mitigation of adversarial risk, i.e., the mitigation of the upper bounds of the adversarial loss. There are three solutions: (i) sample each entry of  $P^{\mathrm{in}}$  from  $\mathcal{N}(0,1 / d^2)$  instead of  $\mathcal{N}(0,1 / d)$ ; (ii) sample each entry of  $W^{(l)}$  from  $\mathcal{N}(0,\sigma_w^2 /N^2)$  instead of  $\mathcal{N}(0,\sigma_w^2 /N)$  for  $l\in [L]$ ; (iii) sample each entry of  $P^{\mathrm{out}}$  from  $\mathcal{N}(0,1 / N^2)$  instead of  $\mathcal{N}(0,1 / N)$ .
First, let us examine scenario (i). In this setting, the variance of  $J(\pmb{x}^{\mathrm{in}})$  is transformed into  $\mathbb{V}[J(\pmb{x}^{\mathrm{in}})_{ij}] = \omega^{L} / d^{2}$  from  $\mathbb{V}[J(\pmb{x}^{\mathrm{in}})_{ij}] = \omega^{L} / d$  (cf. the proof of Thm 4.1). In addition,  $\beta_{\infty,\infty}$  changes to  $\Theta(\beta_{\infty,\infty}) = 1$  from  $\Theta(\beta_{\infty,\infty}) = \sqrt{d}$  (cf. the proof of Thm 5.1). These modifications suggest a potential reduction in adversarial risk in scenario (i).
However, this leads to a training failure. Let us consider the variance of a network output under no bias (for simplicity). For sufficiently large input dimension  $d$  and  $\pmb{x}^{\mathrm{in}}\in [0,1]^d$ , it can be calculated as follows:
$$
\mathbb {V} [ \boldsymbol {f} (\boldsymbol {x} ^ {\text {i n}}) ] = \mathbb {V} [ \boldsymbol {J} (\boldsymbol {x} ^ {\text {i n}}) \boldsymbol {x} ^ {\text {i n}} ] = \mathbb {V} [ J (\boldsymbol {x} ^ {\text {i n}}) _ {i j} ] \| \boldsymbol {x} ^ {\text {i n}} \| _ {2} ^ {2} = \frac {\| \boldsymbol {x} ^ {\text {i n}} \| _ {2} ^ {2}}{d ^ {2}} \approx 0. \tag {A255}
$$
This equation implies that the network always outputs a zero vector for any input  $\pmb{x}^{\mathrm{in}}$ . In other words, input information is lost during signal propagation in the network. In this situation, networks cannot learn the data structure well, and thus training does not proceed (cf. [71, 81, 105]).
Situations (2) and (3) are also similar to (1); they can mitigate the adversarial risk, but they break input information during forward and backward. Therefore, we conclude that it is not feasible to mitigate the adversarial effect without compromising the network's trainability.
Extension to Lipschitz continuous activations. In this study, we primarily established our theorems for networks employing ReLU-like activations. Here, we attempt to generalize these theorems to encompass networks with Lipschitz continuous activations. However, this extension involves looser upper bounds or potentially unrealistic assumptions.
First, we examine the upper bounds for  $p = 2$  and  $q = 2$ . Since  $J(\pmb{x}^{\mathrm{in}})$  is a Jacobian at  $\pmb{x}^{\mathrm{in}}$ , Ineq. (8) remains valid for Lipschitz continuous activations. Denote the Lipschitz constant by  $k \geq 0$ . Similar to the discussion in Appx. H, we can derive the following upper bound:
$$
\left\| \boldsymbol {J} \left(\boldsymbol {x} ^ {\text {i n}}\right) \right\| _ {2} \leq k ^ {L} \left(1 + \sqrt {\frac {N}{d}}\right) \left(1 + \sqrt {\frac {K}{N}}\right) 2 ^ {L} \left(\sigma_ {w} ^ {2}\right) ^ {L / 2}. \tag {A256}
$$
An important observation here is that  $\omega^{L/2}$ , which dictates the training properties, is common to networks with both ReLU-like and Lipschitz continuous activations. Consequently, we claim that our theorems also extend to Lipschitz continuous activations, despite differences in the factor. However, this bound is exponentially loose, casting doubt on its ability to accurately represent the properties of adversarial loss.
Second, we use the analysis of a network Jacobian [69]. For a comprehensive comparison between our approach and theirs, see Sec. 5.1. Here, we adopt their assumption where the variance  $\mathbb{V}[h^{(l)}]$  is constant for any  $l\in [L]$ . Drawing from the results expressed in Eq. (22) of [69], we can assert that  $\left\| J(\boldsymbol{x}^{\mathrm{in}})\right\|_2\leq \Theta ((\sigma_w^2)^{L / 2})$  holds. Similar to the first proposition, this implies that our theorems hold to general activations including Lipschitz continuous activations. However, the assumption clearly does not hold in our theorems such as Thm 5.4.
Single-gradient descent attack can find adversarial examples. Here, we demonstrate that a single-gradient descent attack, such as the fast gradient sign method [38], can find adversarial examples. This is not a substantially novel contribution. For example, see [7, 11, 24, 25, 61, 84, 112]. However, to introduce a new approach based on Thm 4.1, we attempt it.
Proposition K.1 (Single-gradient descent attack can find adversarial examples). Suppose that  $\left\| \boldsymbol{x}^{\mathrm{in}}\right\|_2 = \sqrt{d}$ , the perturbation constraint  $\epsilon$  is sufficiently small, and the input dimension  $d$  is sufficiently large. Then, the single-gradient descent attack finds  $\ell_{\infty}$  constrained adversarial examples that flip the prediction of a single-output random deep neural network with high probability
$$
\operatorname {e r f} \left(\epsilon \sqrt {\frac {\omega^ {L} d}{\pi \left(\omega^ {L} + \alpha \sigma_ {b} ^ {2} \sum_ {k = 1} ^ {L} \omega^ {k - 1}\right)}}\right) \xrightarrow {d \rightarrow \infty} 1, \tag {A257}
$$
where erf is the error function.
Proof. Since  $\pmb{\eta}$  is sufficiently small, the same linear regions in a ReLU-like network encompass both  $\pmb{x}^{\mathrm{in}}$  and  $\pmb{x}^{\mathrm{in}} + \pmb{\eta}$ , i.e.,  $J(\pmb{x}^{\mathrm{in}}) = J(\pmb{x}^{\mathrm{in}} + \pmb{\eta})$ . When an  $\ell_{\infty}$  constrained adversarial example flips the prediction of a classifier, the inequality  $|J(\pmb{x}^{\mathrm{in}})\pmb{x}^{\mathrm{in}} + \pmb{a}(\pmb{x}^{\mathrm{in}})| < |J(\pmb{x}^{\mathrm{in}})\pmb{\eta}|$  holds for  $|\pmb{\eta}|_{\infty} \leq \epsilon$ . The perturbation generated by a single-gradient descent is defined as  $\pmb{\eta} \coloneqq \epsilon \operatorname{sgn}(J(\pmb{x}^{\mathrm{in}}))$ . Note that  $J(\pmb{x}^{\mathrm{in}})$  is a one-dimensional vector in single-output networks. Using this perturbation, by Eq. (A87) and Thm 4.1, the right term of the inequality can be transformed into:
$$
| \boldsymbol {J} (\boldsymbol {x} ^ {\text {i n}}) \boldsymbol {\eta} | \leq \epsilon d \frac {1}{d} \sum_ {i = 1} ^ {d} | J (\boldsymbol {x} ^ {\text {i n}}) _ {i} | = \epsilon d \mathbb {E} [ | J (\boldsymbol {x} ^ {\text {i n}}) _ {i} | ] = \epsilon d \sqrt {\frac {2 \omega^ {L}}{\pi d}} = \epsilon \sqrt {\frac {2 \omega^ {L} d}{\pi}}. \tag {A258}
$$
Then, let us consider  $|J(\pmb{x}^{\mathrm{in}})\pmb{x}^{\mathrm{in}} + \pmb{a}(\pmb{x}^{\mathrm{in}})|$ . By the reproductive property of the Gaussian,  $J(\pmb{x}^{\mathrm{in}})\pmb{x}^{\mathrm{in}} + \pmb{a}(\pmb{x}^{\mathrm{in}})$  follows a Gaussian  $\mathcal{N}(0,\| \pmb{x}^{\mathrm{in}}\|_2^2\mathbb{V}[J(\pmb{x}^{\mathrm{in}})] + \mathbb{V}[a(\pmb{x}^{\mathrm{in}})]$ . The variance can be rearranged as follows:
$$
\left\| \boldsymbol {x} ^ {\text {i n}} \right\| _ {2} ^ {2} \mathbb {V} [ J (\boldsymbol {x} ^ {\text {i n}}) _ {i j} ] + \mathbb {V} [ a (\boldsymbol {x} ^ {\text {i n}}) _ {i} ] = \frac {\left\| \boldsymbol {x} ^ {\text {i n}} \right\| _ {2} ^ {2} \omega^ {L}}{d} + \alpha \sigma_ {b} ^ {2} \sum_ {k = 1} ^ {L} \omega^ {k - 1} = \omega^ {L} + \alpha \sigma_ {b} ^ {2} \sum_ {k = 1} ^ {L} \omega^ {k - 1}. \tag {A259}
$$
Note that the c.d.f of the folded normal distribution based on the Gaussian  $\mathcal{N}(0,\sigma^2)$  is given by  $\operatorname{erf}(x / \sqrt{2\sigma^2})$ . Thus, we can compute the probability such that  $|\pmb{J}(\pmb{x}^{\mathrm{in}})\pmb{x}^{\mathrm{in}} + \pmb{a}(\pmb{x}^{\mathrm{in}})|$  is less than
$\epsilon \sqrt{2\omega^L d / \pi}$  as follows:
$$
\begin{array}{l} \operatorname {e r f} \left(\frac {\epsilon \sqrt {2 \omega^ {L} d / \pi}}{\sqrt {2 \left(\omega^ {L} + \alpha \sigma_ {b} ^ {2} \sum_ {k = 1} ^ {L} \omega^ {k - 1}\right)}}\right) = \operatorname {e r f} \left(\epsilon \sqrt {\frac {\omega^ {L} d}{\pi \left(\omega^ {L} + \alpha \sigma_ {b} ^ {2} \sum_ {k = 1} ^ {L} \omega^ {k - 1}\right)}}\right) (A260) \\ \xrightarrow {d \rightarrow \infty} 1. (A261) \\ \end{array}
$$
L Other experimental results
L.1 Setting
We focused on ReLU networks, i.e.,  $u = 1$ ,  $v = 0$ , and  $\alpha = 1/2$ . Adversarial examples were generated using auto projected gradient descent [20] and the loss function defined in Eq. (3). Networks were initialized to satisfy the  $(M, m)$ -trainability condition (Lemma 5.6), i.e.,  $\sigma_w^2 = 2$  for vanilla networks and  $\sigma_w^2 = 0.1$  or  $\sigma_w^2 = 0.01$  for residual networks. The initial bias variance was set to  $\sigma_b^2 = 0.01$ . We employed MNIST [26] and Fashion-MNIST [99] as the training dataset. All experiments are conducted on an NVIDIA A100.
L.2 Verification of Thm 4.1 (new mean field-based framework)
As shown in Fig. 1, the empirical distribution of  $J(\pmb{x}^{\mathrm{in}})$  in the vanilla network aligned well with Thm 4.1. To further verify Thm 4.1, we provide additional results. We sampled 10,000 vanilla networks with  $d = 1,000$  and  $K = 1$ . For vanilla networks, we set  $\sigma_w^2 = 2$ , and for residual networks, we set  $\sigma_w^2 = 0.01$ . The network width  $N$  and depth  $L$  were set to 5,000 and 10, respectively.
The empirical distributions of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(\pmb{x}^{\mathrm{in}})$  for different network depths,  $L = 10$  and 100, and two randomly generated inputs,  $x_{1}^{\mathrm{in}}$  and  $x_{2}^{\mathrm{in}}$ , are shown in Figs. A5 and A6. The accuracy of Thm 4.1 in predicting these distributions remains consistent across varying network depths. Moreover, the distributions of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(\pmb{x}^{\mathrm{in}})$  did not depend on the input  $\pmb{x}^{\mathrm{in}}$ , even though they are defined as functions of  $\pmb{x}^{\mathrm{in}}$ .
We should note that our theory operates under the assumption of infinite network width. To assess the influence of network width on Thm 4.1, please refer to Fig. A7. We found that for smaller values of  $N$ , for instance  $N = 10$ , the empirical distribution of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  deviates from the theoretical prediction as provided by Thm 4.1. As network width increases, the alignment between empirical results and predictions from Thm 4.1 improves.
It is important to clarify that while Thm 4.1 asserts that the distributions of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  do not depend on  $x^{\mathrm{in}}$ , it does not necessarily imply that random variables  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  are independent of  $J(y^{\mathrm{in}})$  and  $a(y^{\mathrm{in}})$ , respectively, when  $x^{\mathrm{in}} \neq y^{\mathrm{in}}$ . For slightly different inputs ( $y^{\mathrm{in}} := x^{\mathrm{in}} \times 0.999$ ,  $x^{\mathrm{in}} \times 0.99$ , and  $x^{\mathrm{in}} \times 0.5$ ), we computed the correlation coefficients between  $J(x^{\mathrm{in}})$ ,  $a(x^{\mathrm{in}})$  and  $J(y^{\mathrm{in}})$ ,  $a(y^{\mathrm{in}})$ , respectively. These findings can be found in Figs. A8 and A9, and clearly demonstrate that more similar inputs result in higher correlation coefficients. However, this does not contravene the claims made in Thm 4.1. Further, for two randomly generated inputs,  $x_1^{\mathrm{in}}$  and  $x_2^{\mathrm{in}}$ , the random variables  $J(x^{\mathrm{in}})$ ,  $a(x^{\mathrm{in}})$  and  $J(x_2^{\mathrm{in}})$ ,  $a(x_2^{\mathrm{in}})$  were found to be relatively uncorrelated, respectively. The theoretical prediction for this correlation is currently unclear, which is a topic for future work.
To validate the independence of different entries of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$ , we computed the correlation coefficient between two distinct entries. As presented in Fig. A10, we found that the two unique entries of  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  were indeed uncorrelated, corroborating the claims in Thm 4.1. Furthermore, Fig. A10 illustrates that the entries of  $J(x^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  were uncorrelated, which also aligned with Thm 4.1.
An examination of the distributions of  $J(\pmb{x}^{\mathrm{in}})$  and  $a(x^{\mathrm{in}})$  after training may yield interesting insights. However, as trained networks have different weight and bias variances, the observed  $J(\pmb{x}^{\mathrm{in}})$  and
4While the correlation coefficient is not necessarily indicative of dependence, we regard it as a conveniently measurable value. For more information, please refer to Remark E.3.
Figure A5: Distributions of  $J(\boldsymbol{x}^{\mathrm{in}})_{1,1}$  and  $a(\boldsymbol{x}^{\mathrm{in}})_1$  in the vanilla ReLU network with  $d = 1,000$ ,  $K = 1$ ,  $N = 5,000$ ,  $\sigma_w^2 = 2$ , and  $\sigma_b^2 = 0.01$ . The description is the same as Fig. 1.
Figure A6: Distributions of  $J(\pmb{x}^{\mathrm{in}})_{1,1}$  and  $a(\pmb{x}^{\mathrm{in}})_1$  in the residual ReLU network with  $d = 1,000$ ,  $K = 1$ ,  $N = 5,000$ ,  $\sigma_w^2 = 0.01$ , and  $\sigma_b^2 = 0.01$ . The description is the same as Fig. 1.
$a(x^{\mathrm{in}})$  in each network do not follow the same distribution. Consequently, empirically evaluating the validity of Thm 4.1 for trained networks presents substantial challenges. Despite this, we consider, based on experimental findings such as those presented in Fig. A18, that Thm 4.1 offers an accurate representation of the early stages of the training process.
L.3 Verification of Thm 5.1 (upper bounds of adversarial loss)
As shown in Fig. 2, the upper bounds in Thm 5.1 indicate the significant tightness in vanilla networks. To further verify Thm 5.1, we provide additional results. We generated 100 adversarial examples and calculated the adversarial losses defined in Eq. (3). For vanilla networks, we set  $\sigma_w^2 = 2$ , and for residual networks, we set  $\sigma_w^2 = 0.01$ . The network width  $N$  was set to 40,000 for vanilla networks and 35,000 for residual networks, and the network depth  $L$  was set to 3. We set the perturbation constraint  $\epsilon$  to 0.1 and iterations of projected gradient descent to 50.
In Fig. 2, we illustrate the tightness of the bounds with varying input dimensions in vanilla networks. We provide further results for residual networks, as shown in Fig. A11. Additionally, Figs. A12 and A13 demonstrate the adversarial loss as a function of varying output dimensions in vanilla and residual networks, respectively. Overall, our findings affirm the tightness of our bounds across both network types. For some  $(p,q)$  combinations, empirically observed adversarial loss samples exceeded the theoretical upper bounds. We consider that this discrepancy can be mitigated by expanding the network width, a topic elaborated upon in the following paragraph.
Figure A7: Distributions of  $J(\pmb{x}^{\mathrm{in}})_{1,1}$  and  $a(\pmb{x}^{\mathrm{in}})_1$  in the vanilla ReLU network with  $d = 1,000$ ,  $K = 1$ ,  $L = 10$ ,  $\sigma_w^2 = 0.01$ , and  $\sigma_b^2 = 0.01$ . The description is the same as Fig. 1.
The derivation of Thm 5.1 assumes a network of infinite width. However, in practical implementation, an infinite network width is unachievable, leading to instances where empirical results do not coincide with Thm 5.1. Specifically, certain samples were observed to exceed the theoretical upper bounds, as shown in Fig. 2. To evaluate the influence of network width, we varied it while sampling the adversarial losses, with the results displayed in Fig. A14. In the case of  $(p,q) = (2,2)$ , it was observed that a wider network width diminished the discrepancy between sampled losses and theoretical bounds. For other  $(p,q)$  pairings, even with a relatively small width (e.g.,  $N = 100$ ), empirical results aligned with Thm 5.1.
The impact of the perturbation constraint  $\epsilon$  can be found in Fig. A15. It was confirmed that larger  $\epsilon$  values corresponded to a greater divergence between empirically sampled adversarial losses and theoretical bounds. This is because for larger  $\epsilon$ , it was harder for the projected gradient descent to tune adversarial examples well.
The effect of the network depth  $L$  can be confirmed in Fig. A16. For  $(p,q) = (2,2)$ , as the number of layers increased, the value exceeded the upper bounds. In contrast, for  $(p,q) = (\infty, \infty)$ , the value fell below the bounds. These differences can be attributed to the varying complexities of optimizing the adversarial loss for each combination of  $p$  and  $q$ . That is, for  $(p,q) = (2,2)$ , the optimization of adversarial examples is relatively straightforward, enabling the generation of adversarial losses that exceed the upper bound, which is imposed by the constraints of finite width. However, for  $(p,q) = (\infty, \infty)$ , the optimization is challenging, and it is difficult to generate high-quality adversarial examples. The underlying reasons for these disparities in optimization complexity remain a topic for future work.
Furthermore, we assessed adversarial loss during training on the MNIST dataset, as shown in Fig. A17. Vanilla networks were trained using stochastic gradient descent with a learning rate of 0.001. While Thm 5.1 broadly holds for  $(p,q) = (1,2)$ , the disparity between theoretical prediction and empirical results expands as training progresses for  $(p,q) = (2,2)$ ,  $(2,\infty)$ , and  $(p,q) = (\infty ,\infty)$ . Determining a theoretical prediction that provides a tight bound on the adversarial loss for fully trained deep neural networks remains an open challenge for future research.
L.4 Verification of Thms 5.4 and G.9 (time evolution of weight variance)
As shown in Fig. 3, the empirical weight variance observed in the vanilla network aligns well with Thm 5.4. To verify Thm G.9, we provide Fig. A18. We trained 10-layers vanilla and residual networks. The vanilla networks were initialized with  $\sigma_w^2(0) = 2$  and  $\sigma_b^2(0) = 0.01$ . Residual networks are initialized with  $\sigma_w^2(0) = 0.1$  and  $\sigma_b^2(0) = 0.01$ . For standard training,  $N$  was set to 1,000. For
Figure A8: Correlation coefficient of  $J(\cdot)_{1,1}$  for two inputs in the vanilla ReLU network with  $d = 1,000$ ,  $K = 1$ ,  $N = 5,000$ ,  $L = 10$ ,  $\sigma_w^2 = 2$ , and  $\sigma_b^2 = 0.01$ . The description of the histogram is the same as Fig. 1.
adversarial training, we designated  $p = \infty$ ,  $q = \infty$ , and  $\epsilon = 0.3$ . In both training, we used stochastic gradient descent with a small learning rate, 0.001. Note that gradient flow assumes an infinitesimal learning rate (cf. Eq. (10)). A theoretically defined training step  $t$  under gradient flow is not equal to a training step in the experiment. We have linked them by  $t := \text{training steps} \times \text{learning rate}$ . In practice, setting the weight variance precisely is not feasible. For example, in a vanilla network,  $\sigma_w^2(0)$  might be set to 1.999 even though we tried to initialize it to satisfy  $\sigma_w^2(0) = 2$ . Thus, for visibility, the experimental results (curves) were shifted parallel to meet  $\sigma_w^2(0) = 2$  in vanilla networks and  $\sigma_w^2(0) = 0.1$  in residual networks.
From Fig. A18, it is evident that adversarial training facilitates weight regularization for both vanilla and residual networks. Compared to the  $\ell_2$  weight regularization discussed in Appx. I, it offers stronger regularization. We can verify that Thm 5.1 can predict these empirical behaviors in the initial stage of training. The validity of Thms 5.4 and G.9, which underpin our theorems such as Thms 5.7 and 5.9, lends credence to our theoretical assertions.
Figure A9: Correlation coefficient of  $a(\cdot)_1$  for two inputs in the vanilla ReLU network with  $d = 1,000$ ,  $K = 1$ ,  $N = 5,000$ ,  $L = 10$ ,  $\sigma_w^2 = 2$ , and  $\sigma_b^2 = 0.01$ . The description of the histogram is the same as Fig. 1.
L.5 Verification of Thms 5.7 and 5.8 (adversarial trainability)
As shown in Fig. 4, vanilla networks with small width and large depth were not adversarially trainable, which aligned well with Thm 5.7. To verify Thms 5.7 and 5.8, we provide Fig. A19. We trained vanilla and residual networks under various width and depth settings, and observed training accuracy. For fast training convergence, we used Adam [51]. The training was stopped if training accuracy was not improved in the last 200 steps. We set the learning rates to 0.001 or 0.0001, adopting the highest training accuracy at the final training step. The vanilla networks were initialized with  $\sigma_w^2(0) = 2$  and  $\sigma_b^2(0) = 0.01$ . The residual networks were initialized with  $\sigma_w^2(0) = 0.01$  and  $\sigma_b^2(0) = 0.01$ . This initialization satisfied the  $(M,m)$ -trainability conditions (Lemma 5.6). In adversarial training, we set  $p = \infty$ ,  $q = \infty$ ,  $\epsilon = 0.1$ , and the iterations of projected gradient descent to 10.
From Fig. A19, we can confirm that vanilla networks with large depth and small width were not adversely trainable and the training difficulty was unique to adversarial training of such vanilla networks. Note that Prop G.10 could not be verified as the condition in which training persists without machine errors and the trainability condition not being satisfied at  $t = 0$  is challenging to implement in practical scenarios.
Figure A10: Correlation coefficient of two different entries of  $J(\pmb{x}^{\mathrm{in}})$  in the vanilla ReLU network with  $d = 1,000$ ,  $K = 1$ ,  $N = 5,000$ ,  $L = 10$ ,  $\sigma_w^2 = 2$ , and  $\sigma_b^2 = 0.01$ . The description of the histogram is the same as Fig. 1.
Figure A11: Adversarial loss (Eq. (3)) in residual networks with  $N = 35,000$ ,  $K = 100$ ,  $L = 3$ ,  $\sigma_w^2 = 0.01$ ,  $\sigma_b^2 = 0.01$ , and  $\epsilon = 0.1$ . The description is the same as Fig. 2.
L.6 Verification of Thms 5.9 and G.14 (capacity degradation)
To verify the degradation of network capacity during adversarial training, we trained vanilla and residual networks on MNIST with  $p = \infty$ ,  $q = \infty$ , and  $\epsilon = 0.1$ , and observed the Fisher-Rao norm as a measure of capacity. The training steps were set to 500. As Thms 5.9 and G.14 are derived under sufficiently small learning rate (cf. Eq. (10)), we employed small learning rate, 0.0001. Vanilla networks were initialized with  $\sigma_w^2(0) = 2$  and  $\sigma_b^2(0) = 0.01$ . Residual networks were initialized with  $\sigma_w^2(0) = 0.01$  and  $\sigma_b^2(0) = 0.01$ . During the derivation of Thms 5.9 and G.14, using Asm B.1, we calculated  $\mathbb{E}[\frac{\partial f}{\partial w_i} \frac{\partial f}{\partial w_j} w_i w_j] = \mathbb{E}[\frac{\partial f}{\partial w_i} \frac{\partial f}{\partial w_j} ] \mathbb{E}[w_i w_j] = 0$ , for  $i \neq j$ . Nevertheless, in practice, this calculation does not always hold. Therefore, for the computation of the Fisher-Rao norm, we employed a diagonal matrix of  $F$  instead of  $F$  itself.
As shown in Figs. A20 and A21, large network depths (large  $L$ ) produced high Fisher-Rao norms but degraded them drastically. Moreover, it was observed that a wider network width could maintain the Fisher-Rao norm at its initial state. The discrepancy between these experimental values and the values predicted by Thms 5.9 and G.14 is considered to originate from Asm B.1, which does not hold in the case where a diagonal matrix of  $\mathbf{F}$  is used instead of  $\mathbf{F}$  itself.