Adversarial Training from Mean Field Perspective
Authors: Soichiro Kumano
Hiroshi Kera
Toshihiko Yamasaki
Venue: NeurIPS 2023 spotlight (2023)
Keywords: adversarial training; mean field theory
TL;DR: We propose a new framework based on mean field theory and analyze adversarial training theoretically.

Abstract: Although adversarial training is known to be effective against adversarial examples, training dynamics are not well understood. In this study, we present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions. We introduce a new theoretical framework based on mean field theory, which addresses the limitations of existing mean field-based approaches. Based on the framework, we derive the (empirically tight) upper bounds of $\ell_q$ norm-based adversarial loss with $\ell_p$ norm-based adversarial examples for various values of $p$ and $q$. Moreover, we prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity. We also show that the network width alleviates these issues. Furthermore, the various impacts of input and output dimensions on the upper bounds and time evolution of weight variance are presented.

Adversarial Training from Mean Field Perspective
Soichiro Kumano
Hiroshi Kera Chiba University kera@chiba-ut
The University of Tokyo
kumano@cvm. www...ttt...aacc.....om ac.
Toshihiko Yamasaki The University of Tokyo
yamasaki@cvm. wet.v....uttet w...tooooooc.. jp
Abstract
Although adversarial training is known to be effective against adversarial examples, training dynamics are not well understood. In this study, we present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions. We introduce a new theoretical framework based mean field theory, which addresses the limitations of existing mean field-based approaches. Based on the framework, we derive the (empirically tight) upper bounds of E, norm-based adversarial loss with l, norm-based adversarial examples for various values of and Moreover, we prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity. We also show that the network width alleviates these issues. Furthermore, the various impacts of input and output dimensions on the upper bounds and time evolution of weight variance are presented.
Introduction
Adversarial training [38, 58] is one of the most effective approaches against adversarial examples [89].
Various studies aimed to improve the performance of adversarial training [28, 94, 113], leading numerous observations. Adversarial training improves accuracy for adversarial examples but decreases it for clean images [58, 88]. Moreover, it requires additional training data [14, 40, 77] training dataset but not a test dataset [58]. To improve the
and achieves high robust accuracy reliability of adversarial training and address the aforementioned challenges, theoretical analysis is essential. Specifically, it crucial to gain insight into network evolution during adversarial training, conditions for adversarial trainability, and differences between adversarial and standard training.
However, the theoretical understanding of network training is challenging, even for standard training, due to the non-convexity of loss surface and optimization stochasticity. Recent studies employed the mean field theory and analyzed the early stage of standard training for randomly initialized deep neural networks (random networks) [71, 81]. Some studies explored network trainability regarding
gradient vanishing/explosion [81, 104, 105] and dynamical isometry [69, 79, 100]. Others examined network representation power [49, 71, 101]. The theoretical results of the early stage of training have been empirically observed to fit well with fully trained networks [25, 84], partially supported
by recent theoretical results [46]. However, existing mean bese- approaches cannot manage the probabilistic properties of an entire network (e.g., the distribution of a network Jacobian) and dependence between network inputs and parameters, which are crucial for analyzing adversarial training.
In this study, we propose a mean field-based framework that addresses the aforementioned limita- tions (Thm 4. 1), and apply to the adversarial training analysis. While previous studies on adversarial
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

training rely on strong assumptions (e.g.. Gaussian data [14, 80] and linear classifiers [76, 107]), the proposed framework includes various scenarios (e.g.. 4p norm-based adversarial examples and deep neural networks with or without shortcuts, i.e., residual or vanilla networks) without any assumptions on data distributions. Our analysis reveals various adversarial training characteristics that have been only experimentally observed or are unknown. The results are summarized as follows.
Upper bounds of adversarial loss. We derive the upper bounds of adversarial loss, quantifying the adverse effect of adversarial examples for various combinations of Ey-adversarial loss with the e--norm e-hall (p, q E (1,2, 2, (o)) (Thm 5.1). Numerical experiments confirm the tightness of these bounds. We also investigate the impacts of input and output dimensions on these bounds, and discover that for the (p. q) = (2, oc) combination, the bound is independent of these dimensions.
Time evolution of weight variance. We present the time (training step) evolution of weight variance in training (Thms 5.4 and G.9). Weight variance has been used to assess training properties [49, 71, 81]. Our analysis indicates that adversarial training significantly regularizes weights and exhibits consistent weight dynamics across different norm choices.
Vanilla networks are not adversarially trainable under mild conditions. We show that gra- dient vanishing occurs in vanilla networks (without shortcuts) with large depths and small widths, making them untrainable via adversarial training, even with careful weight initialization (Thm 5.7). This contradicts standard training, where deep vanilla networks can be trained with proper ini- tialization [81, 100]. However, residual networks are adversarially trainable even without proper initialization (Thm 5.8 and Prop G.10), proving the importance of shortcuts for adversarial training.
Degradation of network capacity and role of network width. As adversarial robustness requires high network capacity [63], deep networks can be used in adversarial training. However, we reveal that network capacity, measured based on the Fisher-Rao norm [56], sharply degrades in deep networks during ad adversarial training (Thms 5.9 and G. G.14). Specifically, we confirm that the capacity training step é is O(L OLL-1L//N) -N,) where L and N denote the network depth and width, respectively, Interestingly, this result contrasts the roles of depth and width, i.e., the depth increases the initial capacity but decays as training proceeds, whereas the width preserves it. While our theory is validated only during the initial stages of training, our experiments confirm that the adversarial robustness after full training is significantly influenced by network width (cf. Tabs. A5 and A6) as demonstrated in our theorems.
Other contributions. Furthermore, we show the followings cases: (a) Equality of the adversarial loss is obtained instead of inequality (upper bound) under several assumptions (Props G.2 and G.3). (b) Adversarial training leads to faster weight decay and less stable gradients compared with & weight regularization. (c) Capacity degradation is discussed for metries other than the Fisher-Rao norm. (d) Adversarial risk cannot be mitigated while maintaining trainability and expressivity. (e) Discussion ReLUUlike activations extends to Lipschitz continuous activations under certain conditions. (1) A single-gradient descent attack can find adversarial examples that flip the prediction of a binary classifier (Prop K. Contributions (b)-(I) are found in Appx. K.
Although this study focuses on adversarial training, our theoretical framework can be applied to other training methods that consider the probabilistic property of an entire network and dependence between network inputs and parameters. Consequently, we believe that this study can potentially contribute to the theoretical understanding of adversarial training and various deep learning methods.
2 Related work
Here, we summarize the full version in Appx. A. A technical discussion follows in Sec. 5.1.
Mean field theory. Mean field theory in machine learning investigates the training dynamics of random networks in chaotic and ordered phases [71]. Networks can trained at the boundary between these phases [81]. The theory has been extended to networks with shortcuts [105, 106), recurrent connections [15, 69], and batch normalization [104]. It has been employed to study dynamical
isometry [15, 69, 70, 79], and subsequent study achieved training of 10,000-layer networks without

---

shortcuts [100]. Moreover, the theory has been applied to analyze network representation power [49, 71, 101]. However, existing mean field-based analysis cannot handle the properties of an entire network and input-parameter dependence, which is a drawback for some deep learning methods, e.g.. adversarial training. Thus, we propose new framework to address these limitations.
Adversarial training. Various questions related to adversarial training have been theoretically addressed by some studies, including the robustness-accon -acuuaacy trade-off [29, 47, 75, 76, 92, 113], generalization gap [6, 50, 102, 107], sample complexity [1, 14, 62, 80, 110], large model require- ment [63], and enhanced transfer learning performance [27]. However, these results are obtained in limited settings (e.g., Gaussian data and linear classifiers) and are not easily extended to deep neural networks or realistic data distributions. To explore more general settings, recent studies used the neural tangent kernel theory [4, 46, 55J. In the kernel regime, adversarial training, even with a heuristic attack, finds a robust network [35, 1 115]. In our study, investigate adversarial training dynamics based on a mean field perspective, covering general multilayered networks with or without shortcuts and without assumptions about data distributions.
Preliminaries
3.1 Setting
Notations are summarized in Tab. A2. For an integer E N, let [n] := (1, (1....,my) n). In this study, we focus on random deep neural networks with ReLU-like activations, called random ReLU-like networks. This formally defined as follows:
Definition 3.1 (ReLU-like network). A network is called a ReLU-like network if all its activation functions are ((=) == uz for 2 2 0 and uz for < O, with E I.
ReLU-like activations [33, 57] are widely used in theoretical and practical applications [42, 45, 53, 85, 109]. In Appx. K, we extend our theorems to networks with Lipschitz continuous activations.
ReI.U-like network, f Rd > R comprises E trainable layers and two non-trainable layers for adjusting input and output dimensions. The input layer projects x!" E Rd to an N-dimensional vector (0) E R using the random matrix pin E IR xd Subsequently, L consecutive affine transformations and activations are applied by RRN + RRV. Then, (o) is multiplied by random matrix pout R
obtain the output vector f(xi"). Finally, the network function is provided by f-= == pou'gl g( (l) We assume that d and K are sufficiently large, and each entry of pin and pout is i.i.d. and sampled from Gaussians (0, 1/d) and A 1/N), respectively.
An L-layer neural network g comprises weights ( = -W RNNN and biases ((9)
(I...... ( T E R--, where E [L] denotes the layer index. The network is assumed to possess a sufficiently large width (i.e., N is sufficiently large). The I-th pre- and post-activation
defined as h0 ( (1-1) + (()) and (0) = h ()), respectively, where ReI.U-like activation & operates wise. The weight wil) and bias o are i.i.d. and sampled from N/ /N//N) and (0, af), respectively. The network function is represented by Eq. (1). For a residual network setting, refer to Appx.
f(x") pout ob (W(())( (1 (( + +b0))... ...) (2))
3.2 Background
Mean field theory. Mean field theory employs probabilistic methods analyze the properties of random deep neural networks. It assumes that h follows Gaussian, justified by the central limit theorem when width N is sufficiently large [71]. Here, we review the mean field- based approach to analyze the forward and backward dynamics of network. Let L Idd R represent the loss function. The mean squared pre-activation E (n, and gradient (!) == )/xx where denotes the neuron index, are calculated as follows [71, 81]:
(((()) -0 E (1 +
x = Blo (1++))
(2)
3

---

These equations represent the dynamics between adjacent layers. We can infer that gradients vanish when Bld (A; (+)))²
]²] 1 and explode when "o (h; ( > indicating that a network is
trainable only if El (1+ {3]=
Adversarial training.
We define adversarial loss as follows:
( = max St
far + n) f f(a!")) II,
pout ob (WW()) 2b ((
(iin (in + n) + b 60) + b())..)...... ....) b())
+..))..)))) (..))......)..
+
max Inll, Se
(4)
-Pout o (W(L) 2(. (W
where < and p.q E (1, oo). The adversarial loss aims to minimize the difference between the network outputs of natural and adversarial inputs. Networks are trained by minimizing the sum of the standard loss Ladd IR x y + R (e.g., cross-entropy loss), where y denotes a label set, and the adversarial loss Ladv. The mean field analysis typically assumes gradient independence for loss functions (Appx. B) J. We use this assumption for Lata, but not Ladv. Although Eq. (3) differs from the standard adversarial loss based on cross-entropy [58], our definition is employed in more robust methods, e.g.. TRADES [113] and is theoretically simpler to analyze. Therefore, herein, the aforementioned loss is analyzed. However, even this simplified definition (Eq. (3)) poses challenge for thcoretical analysis due to the complex nested structure of a deep neural network (cf. Eq, (4)).
4 Theoretical framework
4.1 Limitations of existing mean field-based approaches
We propose a new theoretical framework based on mean field theory to analyze adversarial training. Here, we describe two limitations of existing mean field-based approaches, e.g., Eq. (2).
Layer-wise wise approach. Existing approaches focus on the dynamics between adjacent layers (cf. Eq. (2)). However, analyzing the adversarial loss (Eq. (3)) requires a framework that handles the probabilistic properties of an entire network instead of adjacent layers. This analysis becomes difficult due to the complex nested structure of networks (cf. Eqs. (1) and (4)). For example, there is no clarity on the the probabilistic behavior of f(="),, distribution of f(x + 1)) f(x""), and dependence between f (x and inputs. We need a framework that disentangles the nested structure networks and manages the probabilistic properties of entire network. Recent studies on the mean field theory studies [15, 36, 69, 70, 100] have concepts related to ours; the comparative analysis is given in Sec. 5.1.
Difficulty in analyzing input-parameter dependence. The analysis of adversarial training requires consideration of input-parameter dependence since adversarial perturbations are designed based on network parameters (ef. Eq. (3)). However, this cannot be readily addressed using existing approaches because their frameworks (e.g.. Eq. (2)) do not offer clear view of the dependence between perturbation and network parameters W W and (I.) N.N-
The proposed framework resolves these limitations and provides simple network representation that allows us to capture the entire network property with clear input-parameter dependence.
4.2 Proposed framework
The current mean field-based approaches cannot capture the probabilistic properties of entire net- work due to the complex nested structure of deep neural networks. Moreover, it is difficult to consider the dependence between inputs and numerous number of parameters. To address these limitations, we employ lnear-like representation of ReLU-like network and propose its probabilistic properties. As ReLU-like networks are piecewise linear, vanilla ReLU-like network can be represented as:
f(x"") = J (x²"))"" + ")
(5)
J [xx"") = pout (2 ((o'(h!-1)) ((--1) w(1)pi
(6)

---

Probability density
Table 1: Values of fp.. Under further assump- tions, we can obtain equality of the adversarial loss rather than inequality (upper bound). Val- ues marked with f represent the equality when is sufficiently small. Values marked with & are applicable if is sufficiently small and K
Exp.
10
Theory
-0.10
-0.05
0.00
0.05
0.10
q
q 2
q 00
Figure 1: Distribution of J(x 1.1 in the vanilla
ReLU network with = 1, 000, K = 1, N =
p = 1
2K nd
K
2lm
d
5, 000, L 10, o 2. and of The blue histogram represents the experimental re- sults (10,000-time samplings), and the orange curve is predicted by Thm44 4.1.
0.01.
p = 2
1 +
K
1
to
p = 00
where D(.) denotes diagonal matrix and a(x")) is defined similar to J(x!1) (cf. Eq. (A48)). For residual network, Eq. (A73) can be referred. Importantly, this representation does not rely on approx- imations, e.g., Taylor expansions. As (()) and 6(0) are randomly sampled, (( and a (x²") denote random matrix and vector, respectively. Unlike the original network definition (Eq. (1)), this representation (Eq. (5)) is non-nested and focuses only two parameters, thereby simplifying network analysis. Remarkably, show the following properties of J (x and a(x'"):
Theorem 4.1 (Properties and distributions of J (xx") and a)x Suppose that the width N is sufficiently large. Then, for any x E Rd, (I) (x²") and a(x"") are independent. (II) each entry of J (x and al (x is i.d. and follows the Gaussian below:
Haxl") Day ~ N (0.
wh: )
L a(x² )2 ~ N 0, 0,0 Σ w
(7)
where = (w²+ (ef. Defin 3.1) and w is wy == ao for vanilla networks and w7 == 1 ac for residual networks.
A significance of this theorem lies in that despite being functions of x. the distributions of J Jxx")) and a)x do not depend on x In other words, although J (x and a (xx")) are determined by (a fixed) ai for initialized network, they become different for each sampling of weights and biases, and the selection of these values is independent of x. Besides, J J(x!") and axx" are independent and their distributions are Gaussian, which exhibits convenient properties. A sketch of proof is given in Appx. D and formal one is in Appxs. E and F.
To validate Thm 4, we conducted numerical experiment and present the results in Fig. 1. We randomly sampled 10,000 vanilla ReLU networks and computed J(x ),, for each network using the identical input s. Additional experimental results can be found in Appx. L.
Broader applicability. The proposed framework (Thm 4.1) manages entire network using only two Gaussians. While we focus on adversarial training, Thm 4.1 can be valuable for other deep neural network analyses based on the mean field theory. For example, contrastive learning [16] can be investigated, as it aims to minimize the distance between original and positive samples while maximizing it for negative samples. In this context, instead of considering the adversar- ial loss, [x + n) - f(x"")], we can analyze loss functions, e.g., IIf (x - f( and If (xring) - f(=in)) Il. where in =:uus and sin represent the original, positive, and negative samples, respectively. The complex nested structure of network makes it challenging to consider the difference between two network outputs; however, Thm 4.1 helps theoretically manageable analyses.
5 Analysis of adversarial training
The proof of each theorem is described in Appx. G.
'This does not imply that random variables, (æ) and J (y), are identical for æ A y. In addition, ((x) and (y) are not always independent. Please also refer the empirical results in Appx. L.
5

---

5.1 Upper bounds of adversarial loss
As the ReLU-like network f is locally linear and its imput Jacobian at x is (xx") (cf. Eq. (5)), we can consider a more tractable form of the adversarial loss instead of Eq. (4) as follows:
Ladv (x²
max #cR I
[ max mERd
[x)
(8)
where [I( p. = max/mll 1 (a) denotes the (p. o)-opert norm of ((). Using Thm 4.1, which describes the property of [zx), we transform Ineq. (8) and obtain the following:
Theorem 5.1 (Upper bounds of adversarial loss), Suppose that the input dimension d output dimension K, and width N are sufficiently large. Then, for any x E Rd, the following inequality holds:
Sepp.l EN Ewew W2)2/22 a LN Ewew w2y4/2
(vanilla) (residual)"
Ladv (ad S @pp.uut12 4/22
(9)
epp.(1+
: (W (1) (1 W1....... W(L)) 1,1 1,21....
N,N denotes the set of all network weights. The constant Bp.a for each norm pair (p. q) is described in Tab. 1.
where :
For some choices of (p. q). we cannot derive upper bounds, and thus, Tab. contains blank. Numerical experiments show the tightness of the bounds (cf. Fig. 2). The theorem indicates that (i) the bounds increase linearly with the perturbation budget e, (ii) the effects of the input and output dimensions depend on the norms (p, q) (cf. Tab. 1), and (iii) network depth T. exponentially impacts the bounds, with w = 1 as threshold between order and chaos. Further, the square sum of the weights in Incq. (9) suggests that adversarial training exhibits weight regularization effect, which is compared to & weight regularization in Appx. Besides, we derive equality rather than inequality (upper bound) under specific assumptions (e.g., small <) for some (p.q) in Appx. K, indicated by t and O in Tab. 1.
The input and output dimensions, d and K, influence the bounds through the (p. q) -deeendet parameter Pp.g. In Tab. 1, 3p.g displays wide range of dependencies on d and K. When d + co and g = 20, d significantly affects the two phases of the adversarial loss, where transition point from order (p 1) to chaos (p oc). In contrast, under realistic assumptions with K, K affects negligibly. Interestingly, the dimensions do not impact the upper bounds when (p. (2, oo). In practice, scale the perturbation budget and adversarial loss according to the choice of (p, 4). respectively, and discuss the scaling effects in Appx. K.
marks the
Comparison with other studies. We can consider studies global Lipschitz of networks in certified adversarial defenses [3, 18, 93] and spectral regularization [60, 108] to analyze Ineq. (8). key difference is that the proposed probabilistic approach contradicts their deterministic approach. By imposing probabilistic constraints on network parameters, we can obtain exponentially tighter, interpretable, and more theoretically manageable bounds, which facilitates the subsequent section's discussion. The mathematical comparison is described in Appx. H.
Results obtained from [15, 36, 69, 70, 100] can be used to analyze the Jacobian's 's singular value distribution. Compared to their approaches, which are limited to (p. q) = 2). the proposed method offers greater generality and flexibility, providing upper bounds for various (p, q). Moreover, Thm 4. enables the derivation of equality instead of inequality (upper bound), Props G.2 and G.3, which is not achievable using the approaches mentioned in the aforementioned studies because it cannot incorporate cuturbation-Jacobbee dependence. Moreover, we do not consider their assumption that the variance Vih is constant for all / E [L], which is often difficult to satisfy.
Further, refer [78]. which established theoretical link between adversarial training and the (p, -)-oprrator norm of a Jacobian. Their findings support Ineq. (8) in training scenarios using heuristic attacks, e.g., projected gradient descent [58]. In this study, derive concrete upper bounds beyond their theoretical link, enabling further investigation of adversarial training properties.
5.2 Time evolution of weight variance
Weight variance plays critical role in determining deep neural network properties [49, 81]. We substitute the adversarial loss definition (Eq. (3)) with Ladv := ()) L/2, where t > denotes the

---

continuous training step. Considering gradient descent with an infinitely small learning rate (gradient flow), the model parameter (tt at step is updated as:
do(t)
0hrtd
aLudy a0(1) a0(1)
(10)
dt
We make the following assumption.
Assumption 5.2. For O < L. <T N, model parameters are independent, and weight and bias follow Gaussian (00, o and (00, of (1)), respectively.
This assumption ensures that the properties of the model parameters remain close their initial values during the early stages of training (L < 7). Under Asm 5.2, the original and proposed mean field theories (Thm 4.1) remain valid during training. For a moderately small value of T', Asm 5.2 is not strong because the model parameters change minimally and retain their initialized states with sufficiently small learning rates. Recent neural tangent kernel studies partially supported this assumption [4, 46, 55], and it is known that random network theories align well with fully trained networks [25, 84]. For example, T 160 is reasonable in specific training setting (ef. Fig. 4).
Now, we summarize other assumptions for reference as follows:
Assumption 5.3. The input dimension output dimension K, and width N are sufficiently large. We apply Asm B. to the standard loss function Latd- The adversarial loss is defined as Ladv ( 1/2 (ef. Thm 5.1).
Based on the aforementioned settings, we obtain the time evolution of the weight variance. Theorem 5,4 (Weight time evolution of vanilla network in adversarial training). Suppose that Asms 5.2 and 5.3 hold. Then, the time evolution of o of a vanilla network in adversarial training given by:
(t)( (1 E0I (0)
t (0).
(11)
N
similar result is obtained for residual networks (Thm G.9). The theorem reveals that the weight variance linearly decreases with i, which can be attributed to the weight regularization effect of adversarial training (cf. Sec. 5.1). In addition, the norm pair (p. q) affect only time-invariant constant Sp.q. and the dynamics of the weight variance can be represented as a consistent function of t regardless of (p.g) q). In other words, adversarial training exhibits consistent weight dynamies irrespective of norm selection, with scale factor varying.
(p.q)
5.3 Vanilla networks are not adversarially trainable under mild conditions
We show that in adversarial training, vanilla networks can fit training dataset in limited cases (small depth and large width), but residual networks can in most cases. This result suggests that residual networks are better suited for adversarial training. First, present the trainability condition based on the concept in [81, 105].
Definition 5.5 ((M, m) mmttrainability condition). A network is said to be ttraiaale a network satisfies m S (0) xx(2) < M, where S m S and M 2 1.2
The value (1) denotes the squared length of the gradient in the I-th layer. A near-zero x (0) (1.) suggests gradient vanishing, while a large value implies gradient explosion. Hence, Defn 5.5 is directly linked to successful training. In contrast to the existing definition, (1-1)/x() z [81, 105], Defn 5.5 incorporates M and m for the subsequent discussion. Then, establish specific (M, m)- trainability conditions for ReLU-like networks.
Lemma 5.6 (Vanilla and residual m) sufficiently large. Then, the (M, -rttriibbiiyl conditions for vanilla and residual networks are respectively given by:
(M,.))-trinabbliit -trainability condition). Suppose that the width N is
1/L < 202 S 1// (vanilla),
oo S 1/ - (residual).
(12)
Trainability depends other factors such as the number of parameters; however, these are not considered herein maintain simplicity. It is empirically known that this condition represents trainability well [81].
7

---

Using the weight time evolution (Thm 5.4) and -ratinabbby condition (Lemma 5.6), the following theorem can be readily derived.
Theorem 5.7 (Vanilla networks are not adversarially trainable). Consider a vanilla network. Suppose
that Asms 5.2 and 5.3 hold, and the (M,m))trainnn
m) mia-iiinabblly condition holds at t and ao (0) = 1. If
(1 1-mm/4)
T
(13)
co:3p.4
then there exists < T such that the (M, -tttriiabbbiy condition does not hold for T S S T.
This indicates the potential untrainability of vanilla networks in adversarial training, even when satisfying the (M, m -taainabbyyl condition initialization, contradicting with standard training where extremely deep networks can be trained if initialized properly [81]. This issue arises from the inconsistency between the trainability condition of vanilla network, i... m//" <
a0 (cf. Lemma 5.6) and monotonically decreasing nature of o (t) in adversarial training (cf. Thm 5.4). As stated in Asm 5.2, we assume that T is small. Therefore, if the right-hand term of Eq. (13) becomes large, the assumption and Thm 5.7 are violated. In summary, for large I. (many layers) and 6 (large perturbation constraint), vanilla networks are not adversarially trainable. Moreover, large N (a wide network) can mitigate this issue in vanilla networks. For example, the vanilla network with L = 20, N = 256, and E = 0.3 not adversarially trainable; however, when the width is increased to 512, it becomes trainable (cf. Fig. 4). In contrast, we can claim the following for residual networks:
Theorem 5.8 (Residual networks adversarially trainable). Consider residual network, Suppose that Asms 5.2 and 5.3 hold, and the (M, m) -ttrriibbbilltty condition holds at = 0 and ao m (0) < 1. Then, (M, trainabiityy condition always holds for S.
This occurs due to the (M, mittriibbby condition for residual networks, which does not have any lower bound (cf. Lemma 5.6), and the monotonically decreasing nature of o², (ef. Thm G.9). Besides, residual networks are adversarially trainable even without careful weight initialization (Prop G.10). These theorems highlight the robust stability of adversarial training in residual networks
5.4 Degradation of network capacity
We demonstrate that adversarial training degrades network capacity. We use the Fisher-Ran norm as a metric [56], with alternative metrics discussed in Appx. K. The Fisher-Rao norm is defined as:
(
(ar")
w FR = w F(x:"") W,,
F(win)) Σ
(14)
dw
where w : (W(l), w(2)
) represents the vector of all the network weights and F denotes the empirical Fisher information matrix when only one training data point is considered (for simplicity). The Fisher-Rao norm is preferred over other norm-based capacity metries [8, 65-67] owing to its invariance to node-wise rescaling [56]. We present the following theorem based on [49]:
Theorem 5.9 (Adversarial training degrades network capacity). Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, Asm B.I is applied to the network outpur, and the (M, --trrinbability || vd and (t) Then, the
condition holds expectation of the Fisher-Rao norm is given by:
and ao (0) 1. Assume
((]) Pll T.K ECC N
(15)
A related result for residual networks is presented in Thm G.14. As described in [63], adversarial robustness necessitates high capacity. However, Thm 5.9 indicated that capacity decreases linearly with step in adversarial training. To address this conflict, large depth (large L), which increase the initial capacity with (LL), can be considered, but this scenario accelerates the degradation speed with (L??). To preserve capacity, we must increase the width N accordingly. Consequently, to achieve high capacity adversarial training, it is necessary to increase not only the number of layers but also the width N to keep 1.2/N constant. Although Thms 5.9 and G.14 have been established in the early stages of training, numerical experiments proved that the adversarial robustness following full training is significantly influenced by network width (cf. Tabs. A5 and A6).
8

---

= 1.4..
p 1.g 2
p
1.4
0.75
0.10
0.03
T Theory
Adversarial loss Cuds
0.50
0.05
0.02
Exp.
0.25
0.01
250
500
750
1000
250
750
1000
250
500
750
1000
= 2,
2
= 2,
00
OC,
00
0.20
0.2
2
0.15
1
0.0
250
500
250
1000
750
1000
Input dimension
Figure 2: Adversarial loss (Eq. (3)) in vanilla networks with N = 40, 000, K = 100, = 3, and 6 0.1. We generated 100 adversarial examples for each input dimension. The blue curves and bands represent the mean and standard deviation of the adversarial loss, respectively, whereas the orange curves (upper bounds) are predicted based on Thm 5.1. Some samples slightly exceed the upper bounds because we used the finite network width (cf. Appx. L..
Weight variance d.
2,0000
40
1,0996
91
100
3,6
1.9990
Degth
30-
100
1,908G
250
-3
300
1000
1500
2000
10-
Training
Standard
Adv. (N 26)
Theory (N 2k)
0.0
300
2000
weight
Adv. (N
Theory 1A)
Figure 3: Time evolution the weight variance Figure 4: Heat map of the training accuracy of in the vanilla network with L 10,
vanilla networks with co, q and 0.3. We used N = 1, 000 for 0.3. The dashed lines represent the condition of standard and & regularized training. The solid T in Thm 5.7 with m 0.0001. In standard lines represent experimental results. The dashed training, high accuracy is obtained across all the lines predicted by Thm 5.4.
q = 00, and
depths and widths (cf. Fig. A19).
Experimental results
We validate Thms 5.1, 5.4 and 5.7 via numerical experiments. The vanilla ReLU networks were initialized with o3. and of 0.01 to meet the (M, m) -raiinabbbyy conditions (Lemma 5.6). To verify Thms 5.4 and 5.7, we used MNIST [26]. Setup details and more results are given in Appx. L.
Verification of Thm 5.1. We created adversarial examples for initialized networks and computed the adversarial loss (Eq. (3)). As shown in Fig. 2, the upper bounds in Thm 5.1 were considerably tight. Some samples slightly exceeded the upper bounds because used the finite network width while the infinite width is assumed (cf. Appx. I.).
Verification of Thm 5.4. We trained vanilla networks normally (with and without l regularization) and adversarially. The time evolution of weight variance is shown in Fig. 3. Adversarial training significantly reduces weight variance, whereas wide width (i.e., large N) suppresses it. The validity of Thm 5.4, which forms the basis of our theorems such as Thms 5.7 and 5.9, supports our theorems.
Verification of Thm 5.7. We trained vanilla networks considering various depth and width settings and monitored the training accuracy. As shown in Fig. 4, it was difficult for vanilla networks to training dataset when the depth was large and the width was small; increased width helps in fitting. Although we currently lack theoretical prediction of the boundary between trainable and untrainable areas determined based on Eq. (13), empirical evidence suggests that T 160 is relevant.

---

7 Limitations
The mean field theory offers valuable insight into network training. However, its applicability is restricted to the initial stages of training. Although recent studies suggested empirically [25, 84] and theoretically [46] that the analysis of early-stage training extends well to full training, the strict relationship is yet to be explored. Our results have the same limitations. Although some theorems accurately capture the behavior during the initial stages of training and even after full training (cf. Tabs. A5 and A6), as training progresses. some theorems begin to diverge from the actual behavior (ef. Fig. A17). Another caveat is that the mean field theory assumes infinite network width, which is practically infeasible. Empirically, our theorems hold well when the width approximately exceeds 1,000 (ef. Fig. A7), while Thm 5.1 requires larger width approximately exceeds 10,000 for (p.g) (2, 2) (cf. Fig. A14). These limitations also derive from the mean field theory and are not unique to our study. Despite these limitations, we consider that this study provides a powerful theoretical framework that extends the applicability of the mean field theory to various training methods, and the results obtained for adversarial training are insightful.
Conclusions
We proposed framework based on the mean field theory and conducted a theoretical analysis of adversarial training. The proposed framework addressed the limitations of existing mean field-based approaches, which could not handle the probabilistic properties of entire network and dependence between network inputs and parameters [71, 81]. Based on this framework, we examined adver- sarial training from various perspectives, unveiling upper bounds of adversarial loss, relationships between adversarial loss and network input/output dimensions, the time evolution of weight variance, trainability conditions, and the degradation of network capacity. The theorems of this study were validated via numerical experiments. The proposed theoretical framework is highly versatile and can help analyze various training methods, e.g., contrastive learning.
Acknowledgments and Disclosure of Funding
We would like to thank Huishuai Zhang for useful discussions. This work was supported by JSPS KAKENHI Grant Number JP23KJ0789 and JP22K17962, by JST, ACT-X Grant Number JPM- JAX23C7, JAPAN, and by Microsoft Research Asia.
References
[1] J.-B. Alayrac, J. Uesato, P.-S. Huang, A. Fawzi, R. Stanforth, and P. Kohli. Are labels required for improving adversarial robustness? In NeurlPS, volume 32, 2019.
[2] L. Amsaleg. J. Bailey, D. Barbe, S. Erfani, M. E. Houle, V. Ngugnn,, and M. Radovanović. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In WIFS, pages 1-6, 2017.
[3] Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In ICML, pages 291-301, 2019.
[4] S. Arora, S. S. Du, W. Hu, Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In NewrIPS, volume 32, 2019.
[5] A. Athalye. N. Carlini, and D. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, pages 274-283, 2018.
[6] P. Awasthi, N. Frank, and M. Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In /CML, pages 43 131-441, 2020.
[7] P. Bartlett, S. Bubeck, and Y. Cherapanamjeri. Adversarial examples in multi-layer random relu networks. In NeurIPS, volume 34, pages 9241-9252, 2021.
[8] P. L.. Bartlett, D. J. Foster, and M. Telgarsky. neural networks. In NeurIPS, volume 30, 2017.
margin bounds for
10

---

[9] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR. 3(Nov):463-45) 482, 2002.
[10] Y. Blumenfeld, D. Gilboa, and D. Soudry. A mean field theory of quantized deep networks: The quantization- trade-off. In NeurIPS, volume 32, 2019.
[11] S. Bubeck, Y. Cherapanamjeri. G. Gidel, and R. Tachet des Combes. A single gradient step finds adversarial examples random two-layers neural networks. In NeurIPS, volume 34, pages 10081-10091, 2021.
[12] N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing detection methods. In ACM WS, pages 3-14, 2017.
[13] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In SSP, pages 2017.
[14] Y. Carmon, A. Raghunathan, L. Schmidt, P. Liang, and J. C. Duchi. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
[15] M. Chen, J. Pennington, and S. Schoenholz. Dynamical isometry and mean field theory RNNs: Gating enables signal propagation in recurrent neural networks. In ICML, pages 873 873-882, 2018
[16] T. Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.
[17] Cho and Saul. Kernel methods for deep learning. In NeurlPS, volume 22, 2009.
[18] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness adversarial examples. In ICML, pages 854-863, 2017.
[19] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In ICML, pages 1310-1320, 2019.
[20] F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, pages 2206-2216, 2020.
[21] A. Damianou and N. D. Lawrence. Deep gaussian processes. AISTATS, pages 207-215, 2013.
[22] A. Danicly. SGD learns the conjugate kernel class of the network. In NeurIPS, volume 30, 2017.
[23] Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and dual view expressivity. In NeurIPS, volume 29, 2016.
[24] A. Daniely and H. Schacham. Most relu networks suffer from ell2 adversarial perturbations. In NeurIPS, volume 33, pages 6629-6636, 2020.
[25] G. De Palma, B. Kiani, and S. Lloyd. Adversarial robustness guarantees for random deep neural networks. In ICML, pages 2522-2534, 2021.
[26] L. Deng. The MNIST database of handwritten digit images for machine learning research. Signal Process. Mag., 29(6):111-144 2012.
[27] Z. Deng, L. Zhang. K. Vodrahalli, K. Kawaguchi, and Y. Zou. Adversarial training helps transfer learning via better representations. In NeurlPS, volume 34, pages 25179-25191, 2021.
[28] G. W. Ding, Y. Sharma, K. Y. C. Lui, and R. Huang. MMA training: Direct input space margin maximization through adversarial training, In ICLR, 2020.
[29] Dobriban, H. Hassani, D. Hong, and Robey. Provable tradeoff' in adversarially robust classification, vrXiiv2206605/66 2020.

---

[30] A. Fawzi, H. Fawzi, and O. Fawzi, Adversarial vulnerability for any classifier. In NeurlPS, volume 31, 2018.
[31] A. Fawzi, O. Fawzi, and P. Frossard. Analysis of classifiers' robustness to adversarial pertur- bations. ML, 107(3)4881-508 2018.
[32] A. Fawzi, S.-M. Moosavi-Dezfooli, and P. Frossard. Robustness classifiers: from adversarial to random noise. In NewrIPS, volume 29, 2016.
[33] K. Fukushima. Cognitron: A self-organizing multilayered neural network. Biol. Cybern.,
20(3):121-136, 1975.
[34] A. Galloway, A. Golubeva, T. Tanay, M. Moussa, and G. W. Taylor. Batch normalization is cause of adversarial vulnerability. In ICML WS, 2019.
[35] R. Gao, T. Cai, H. Li, Hsieh, L. Wang, and J. D. Lee. Convergence of adversarial training in overparametrized neural networks. NeurlPS, volume 32, 2019.
[36] D. Gilboa, B. Chang, M. Chen, G. Yang, S. S. Schoenholz. E. H. Chi, and J. Pennington. Dynamical isometry and mean field theory LSTMs and GRUs. arXiv:1901.00 1901.08987. 2019.
[37] J. Gilmer, L. Metz, F. Faghri. S. S. Schoenholz, M. Raghu, M. Wattenberg, and Goodfellow. Adversarial spheres. In ICLR WS, 2018.
[38] I. J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples. In ICLR, 2015.
[39] S. Gowal, K. D. Dvijotham, R. Stanforth, Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli. Scalable verified training for provably robust image classification. In ICCV, pages 4842-4851, 2019.
[40] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann. Improving robustness using generated data. In NeurIPS, 2021.
[41] S. Hayou, A. Doucet, and J. Rousseau, On the selection of initialization and activation function for deep neural networks. arXiv:/805.08266, 2018.
[42] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[43] M. Hein and M. Andriushchenko. Formal guarantees on the robustness of classifier against adversarial manipulation. In NeurIPS, volume 30, 2017.
[44] J. Hron, Y. Bahri, J. Sohl-Dickstein, and R. Novak. Infinite attention: NNGP and NTK for deep attention networks. In ICML, pages 4376-4388 2020.
[45] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 4700-4708, 2017.
[46] A. Jacot, Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NearlPS, volume 31, 2018.
[47] A. Javanmard, M. Soltanolkotabi, and H. Hassani. Precise tradeoff's in adversarial training for linear regression. In COLT, pages 2034-2078, 2020.
[48] H. Kannan, A. Kurakin, and I. Goodfellow. Adversarial logit pairing. arXiv: 2018.
[49] Karakida, S. Akaho, and S.-i. Amari. Universal statistics of Fisher information in deep neural networks: Mean field approach. In AISTATS, pages 1032-1041, 2019.
[50] J. Khim and P.-L. Loh. Adversarial risk bounds via function transformation. //X0v:/8/00095//. 2018.
[51] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In CVPR, 2015.
12

---

[52] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[53] Krizhevsky, Sutskever, and G. Hinton, Imagenet classification with deep convolutional neural networks. In Neur/PS, pages 1097-1105, 2012.
[54] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz. J. Pennington, and J. Sohl-Dickstein. Deep neural networks as gaussian processes. In ICLR, 2018.
[55] Lee, L. Xiao, Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS, volume 32, 2019.
[56] T. Liang, Poggio, Rakhlin, and J. Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In AISTAT, pages 888 896,9,62225 2019.
[57] A. L. Maas, A. Y. Hannun, A. Y. Ng, et al. Rectifier nonlincarities improve neural network acoustic models. In ICML, volume 30, page 3, 2013.
[58] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[59] A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR, 2018.
[60] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks, In ICLR, 2018.
[61] A. Montanari and Y. Wu, Adversarial examples in random neural networks with general activations. arXiv:2203. 10003.772000 /7209, 2022.
[62] A. Najafi, S.-i. Maeda, M. Koyama, and T. Miyato. Robustness adversarial perturbations in learning from incomplete data. In NeurlPS, volume 32, 2019.
[63] P. Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv:1901.00532 2019.
[64] R. M. Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
[65] B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring generalization in deep learning. In NeurlPS, volume 30, 2017.
[66] B. Neyshabur, R. R. Salakhutdinov, and N. Srebro. Path-SGD: Path-normalized optimization in deep neural networks. In NeurlPS, volume 28, 2015.
[67] B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In COLT, pages 1376-1401, 2015.
[68] R. Novak, L. Xiao, Lee, Y. Bahri, G. Yang, J. Hron, D. A. Abolafia, J. Pennington, and J. Sohh-Dicksteen. Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR, 2019.
[69] J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NeurlPS, volume 30, 2017.
[70] J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In AISTATS, pages 1924-1932, 2018.
[71] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. In NewrIPS, volume 29, 2016.
[72] R. Rade and S.-M. Moosavi-Dezfooli. Helper-based adversarial training: Reducing excessive margin to achieve better accuracy vs. robustness trade-off. In ICML, 2021.
[73] M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and Sohl-Diekstein. On the expressive power of deep neural networks. In ICML, pages 2847-2854, 2017.
13

---

[74] A. Raghunathan, J. Steinhardt, and P. Liang. Certified defenses against adversarial examples. In ICLR, 2018.
[75] A. Raghunathan, S. M. Xie, F. Yang, J. Duchi, and P. Liang. Understanding and mitigating the tradeoff' between robustness and accuracy. In ICML, 2020.
[76] A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang. Adversarial training can hurt generalization. In ICML WS, 2019.
[77] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann. Fixing data augmentation improve adversarial robustness. arXiv:2103.0 01946 2021.
[78] K. Roth, Y. Kilcher, and T. Hofmann. Adversarial training is form of data-dependent operator norm regularization. In NeurlPS, volume 33, pages 14973-14985, 2020.
[79] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR, 2014.
[80] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust general- ization requires more data. In NeurIPS, 2018.
[81] S. Schoenholz, J. Gilmer, Ganguli, and J. Sohl-Dickstein. Deep information propagation. In ICLR, 2017.
[82] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? In ICLR, 2019.
[83] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! In NeurIPS, 2019.
[84] C.-J. Simon-Gabriel, Y. Ollivier, L. Bottou, B. Schölkopf, and D. Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In ICML, pages 5809-5817, 2019.
[85] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2014.
[86] A. Sinha, H. Namkoong, R. Volpi, and J. Duchi. Certifying some distributional robustness with principled adversarial training. ICLR, 2018.
[87] H. Sompolinsky, A. Crisanti, and H.-J. Sommers. Chaos in random neural networks. Phys. Rev. Lett., 63())225, 1988.
[88] Su, H. Zhang, H. Chen, Yi, P.-Y. Chen, and Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In ECCV, pages 631-648, 2018.
[89] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[90] F. Tramer, N. Carlini, W. Brendel, and Madry. On adaptive attacks adversarial example defenses. In NeuriPS, volume 33, pages 1633-1645, 2020.
[91] J. A. Tropp. Topics in sparse approximation. The University of Texas at Austin, 2004.
[92] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be odds with accuracy. In /CLR, 2019.
[93] Y. Tsuzuku, Sato, and M. Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In NeurlPS, volume 31, 2018
[94] Y. Wang, Zou, Yi, J. Bailey, X. Ma, and Q. Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[95] C. Williams. Computing with infinite networks. In NeurIPS, volume 9, 1996.
14

---

[96] E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. ICML, pages 5286-5295 2018.
[97] E. Wong, L. Rice, and J. Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[98] B. Wu, J. Chen, D. Cai, X. He, and Q. Gu. Do wider neural networks really help adversarial robustness? In NeurlPS, volume 34, pages 7054-7067, 2021.
[99] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.077477 2017.
[100] L. Xiao, Y. Bahri, J. Sohl-Dickstein. S. Schoenholz, and Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In ICMI., pages 5393-5402, 2018.
[101] L. Xiao, Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. In ICML, pages 10462-10472, 2020.
[102] Y. Xing, Q Song, and G. Cheng. On the generalization properties of adversarial training. In AISTATS, pages 505-513, 2021.
[103] G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv:1902.04766. 2019.
[104] G. Yang, J. Pennington, V. R.o, J. Sohl-Dickstein, and S. S. Schoenholz. A mean field theory of batch normalization. ICLR, 2019.
[105] Yang and S. Schoenholz. Mean field residual networks: On the edge of chaos. In NeurIPS, volume 30, 2017.
[106] G. Yang and S. S. Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. OpenReview, 2018.
[107] D. Yin, R. Kannan, and P. Bartlett. Rademacher complexity for adversarially robust general- ization. In ICML, pages 7085-7094, 2019.
[108] Yoshida and Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv: 10941, 2017.
[109] S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, pages 1-12, 2016.
[110] R. Zhai, T. Cai, D. He, C. Dan, K. He, J. Hoperoft, and L. Wang. Adversarially robust generalization just requires more unlabeled data. arXiv: 2019.
[111] D. Zhang, T. Zhang, Y. Lu, Z. Zhu, and B. Dong. You only propagate once: Accelerating adversarial training via maximal principle. In NeurIPS, 2019.
[112] H. Zhang, D. Yu, Y. Lu, and D. He. Adversarial noises are linearly separable for (nearly) random neural networks. arXiv:2206.0433 2022.
[113] H. Zhang, Y. Yu, J. Jiao, E. Xing. L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, pages 7472-7482, 2019.
[114] J. Zhang, X. Xu, B. Han, G. Niu, L. Cui, M. Sugiyama, and M. Kankanhalli, Attacks which do not kill training make adversarial learning stronger. In ICML, pages 1 1278-11287, 2020.
[115] Y. Zhang, O. Plevrakis, S. Du, Li, Z. Song, and Arora. Over-parameterized adversarial training: An analysis overcoming the curse of dimensionality, In NewrIPS, volume 33, pages 679-688, 2020.
15

---

Table A2: Notation. While h is a function that takes æ!t as input, we omit the argument as
h = h0 for notational simplicity. Other symbols sometimes follow this.
Notation
Description
Cf.
deN Kel N
Input dimension of the network
Sec. 3.1
Output dimension of the network
Sec. 3.1
LEN
Number layers in the network, i.e., network depth
Sec. 3.1
NEN
Number of neurons in layer, i.e., network width
Sec. 3.1
U, V R P (1,2,
Slope of a ReLU-like function
Defn 3.1
Perturbation constraint norm, ||n|| Se
Eq. (3)
g (1,2,
Output difference norm, f + n) f f(x!")l,
Eq. (3)
me 00 0
(M, m) trainabbiyl condition Perturbation constraint, ||17|| S
Defn 5.5
Eq. (3)
0
0: (u² //2
Thm 4.1
3n 0
Time-invariant constant determined by (p.g)
Thm 5.1
1 12 0
Continuous training step Upper limit of training steps Weight variance, Wi, -V o
Eq. (10)
72 0
Asm 5.2
02
Sec. 3.1
of 0
Bias variance, b, ~N,, of
Sec. 3.1
W W 0
Wy == ao (vanilla) and w/ == + ao (residual) (M, m) -raainabiiy condition
Thm 4.1
M
Defn 5.5
i Rd
Input vector
Sec. 3.1
E Rd
Perturbation, ||m]| S
Eq. (3)
( E NNN
1-th weight, W ~ NN (N/N
Sec. 3.1
Set of all network weights, (W(1) w w(l)
W WCR
WNN
Thm 5.1
w E RANZ
Vector of all the weights, (W(l1) w(l))
Eq. (14)
11
((9) R
1-th bias, 6 (0,
Sec. 3.1
piu E RR
Rand. mat. for input adjustment, Pin N/0, 2)
Sec. 3.1
pout E RR
Rand. mat. for output adjustment, Pout ~N(0, k)
Sec. 3.1
p() RNXN
Rand, mat. in the 1-th shorteuts, p(() ~NO,
Eq. (A16)
R
ReLU-like activation, = u2(2 2 0); e2(=
Defn 3.1
( F. + R
Mean squared /-th gradient, (()(()))/Ox)))
Sec. 3.1
(1) Rd + N
1-th pre-activation, h == W (1) (1-1) (x.") +6
Sec. 3.1
(1) Rd + N
1-th post-activation, ( == o (2"")))))
Sec. 3.1
F Rd + RAN LNN
Empirical Fisher information matrix
Eq. (14)
f Rd + R
Overall network, f(x"") == pout g (Pin
Sec. 3.1
9 R. + RN
L-trainable layer ReL.Uliike network Slope of a piecewise linear region at a Bias of piecewise linear region at in
See.
Rd Rxxd
Thm 4.1
Rd R
Thm 4.1
D RN + RNXN
Diagonal matrix
Eq, (5)
Cal Rd x ) + F
Standard loss function, where y represents label set Adversarial loss function
Sec. 3.2
Ladv
Rd R20
Eq. (3)
A Additional related work
Please also refer to Secs. 2 and 5.
A.1 Random networks and mean field theory
Understanding general deep neural networks is challenging due to the non-convexity of loss surfaces and stochastic nature of optimization. Researchers have explored random networks, which have random parameters instead of trained parameters. Random networks have been primarily studied in three areas: compositional kernels [17, 22, 23], neural network Gaussian processes [21, 44, 54, 59,
16

---

64, 68, 95, 103], and mean field theory. These fields share close relationships, particularly between neural network Gaussian processes and mean field theory. For example, the forwarding dynamics in mean field theory (Eq. (2)) is equivalent to the kernel representation of Gaussian process [64, 71]. Furthermore, the accuracy of Gaussian process is significantly influenced by the edge of chaos, which has been studied in mean field theory [68]. We provide a more detailed review of the mean field theory.
Mean field theory for neural networks was first introduced in [87] and later extended in [71, 81]. This theory examines the signal propagation, dynamies, and trainability of random networks in two phases: ordered and chaotic. The ordered phase is characterized by decreasing layer output variance and vanishing gradients during backpropagation. while the chaotic phase is characterized by expanding variance and exploding gradients. Effective training of deep neural networks occurs near the boundary between these two phases [71, 81]. Researchers have applied mean field theory to study various network architectures, including dropout [81], batch normalization [104], residual network [105, 106), recurrent network [15], quantized network [10], and Swish [41]. Recent research [15, 36, 69, 70, 100] has also utilized the mean field theory to analyze dynamical isometry, where all singular values of the Jacobian are one. Some of these findings can be applied to analyze Ineq. (8) in this study, with detailed comparisons provided in Sec. 5.1. Other studies have explored network representation power [49, 71, 101]. This work utilized the proof idea of [49] in the derivation of Thms 5.9 and G.14.
In this study, we employed mean field theory to analyze adversarial training behavior. However, the original theory has two limitations that make it unsuitable for this purpose (cf. Sec. 4.1). To address these limitations, we introduced new mean field-based framework (ef. Sec. 4.2). Our theory is also applicable to mean field-based analyses of other deep learning methods beyond adversarial training.
A.2 Adversarial examples
A.2.1 Adversarial examples in random neural networks
We summarize the literature adversarial examples in random neural networks [7, 11, 24, 25, 61, 84, 112]. It has been reported that adversarial perturbations can be found through gradient flow in most random ReLU networks with decreasing layer widths [24]. This result extended to two-layer random networks with greater width than input dimension [11] and generalized to random ReLU networks with constant depth and wide width [7]. Recently, similar results were presented without width restrictions and for local Lipschitz continuous activation [61]. Additionally, it has been demonstrated that adversarial noises generated by single-step attack are linearly separable in two-layer random network and the neural tangent kernel regime [112]. More information on [25, 84] can be found in Appx. A.2.2.
Although the context differs somewhat, we mention [34], which provides empirical evidence that batch normalization leads to adversarial vulnerability. This observation aligns with previous results from mean field theory suggesting that batch normalization causes gradient explosion [104].
In this study, we focused on early stage adversarial training properties rather than adversarial examples. key difference that we primarily investigated the maximum difference between network outputs for standard and adversarial inputs rather than misclassification which was the main focus of previous studies [7, 11, 24, 61]. Nonetheless, some findings in Sec. 5.1 can be applied to the understanding of adversarial examples in random networks. In addition, in Appx. K, we proved the existence of adversarial examples in random networks to demonstrate the effectiveness of Thm 4.1.
A.2.2 Adversarial examples and input dimension
We present research investigating the relationship between adversarial examples and input dimen- sions [2, 25, 30-32, 37, 38, 82, 84]. Apart from [38],' most studies have indicated that adversarial example threats increase with the square root of the input dimension [25, 30-32, 37, 82, 84]. Some studies have focused on specific data distributions or simple classifiers [30-32, 37, 82), while others targeted random networks [25, 84]. The study in [25] has examined the distance from the decision boundary in random networks, and concluded that ad adversarial examples deceive classifiers more easily with the square root of the input dimension. Another study has utilized the first-order Taylor
'Indeed, they omitted weight scaling for simplicity, and their results were essentially consistent with those of other studies when the scaling considered.
17

---

expansion of a loss function to amalyze the loss gradient with respect to an imput [84]. While direct comparison between our work and [84] is challenging due to differing loss functions, our theorems offer two advantages. First, we considered both the input and output dimensions, whereas they addressed only input dimensions. Second, our theorems are applicable to residual networks, whereas their assumptions are not.
In this study, we examined the relationship between an input dimension and adversarial risk in Sec. 5.1. Our analysis did not depend on specific data distributions or architectures, except for ReLUUlliee activations and random parameters, which is advantageous. In addition, we assessed it for a wide variety of norms, demonstrating that the impact of adversarial examples is not limited to the square root of the imput dimension alone (cf. Tab. 1). Moreover, our bound considers not only the input dimension but also the number of classes.
A.3 Adversarial training
Numerous empirical adversarial defenses have been proposed; however, most are ineffective against stronger attacks [5. 12, 13, 20, 90]. Some studies have focused on theoretically certified defenses [19, 39, 43, 74, 96], but these are often only applicable to specific or small networks, or are weaker than empirical methods. Adversarial training [38, 58] is considered the most effective empirical defense against various attacks [5, 20, 90]. This involves training a classifier using a dataset that contains natural images and adversarial examples [38] or solely adversarial examples [58]. Various forms of adversarial training exist, including more effective loss functions [28, 48, 58, 94, 113], time-efficient frameworks [83, 97, 111], and procedures that preserve the clean accuracy [72, 113, 114]. Recent studies have demonstrated that combining adversarial training and data augmentation with unlabeled or generated data results in high robustness [40, 77]. Despite significant progress in empirical methods, theoretical understanding of adversarial training remains incomplete. We provide summary of theoretical studies on adversarial training, including robust generalization research.
Several studies have investigated the trade-off between robust and clean accuracy [29, 75, 76, 92, 113], initially observed empirically [58, 88]. The trade-off has been proven inevitable even in the infinite data limit, assuming data is constructed from moderately correlated single feature and weakly correlated many features [92]. Similar claims were found in [113] for different data distributions. It has been reported that the trade-off in finite data settings for linear and slightly more complex predictors can be mitigated with additional unlabeled data [76]. While comparable results were reported in [75], contrasting study also exists [47]. Moreover, the trade-off has been shown to depend on class imbalance in a dataset using Gaussian classification models [29].
Various studies have examined the generalization gap of adversarial robust models [6, 50, 102, 107]. For example, ResNet [42] trained adversarially on CIFAR-10 [52] achieved 96% robust training accuracy but only 47% robust test accuracy [107]. The lower bound of adversarial Rademacher complcxity [9] has been shown to increase with the square root of the input dimension for lincar classifiers trained with for adversarial examples [107], which was later extended in [6]. Similar results using a tree transformation approach have been reported in [50]. However, the influence of network width and depth, or other training settings, such training with & adversarial examples, on the bound remains unclear. The generalization gap has been investigated for linear regression models and two-layer neural networks with lazy training in data interpolation contexts [102].
Numerous studies have explored the sample complexity of robust generalization [1, 14, 62, 80, 110]. Robust learning may require a larger sample size than standard learning [80]. Subsequent research
has indicated that unlabeled data can be sufficient to achieve robust generalization [1. 14, 62, 110]. The majority of these studies have focused on data sampled from Gaussian mixture models [1, 14, 110]. Moreover, the sample complexity of distributionally robust learning with perturbations in the Wasserstein ball has been examined [62], and some studies have suggested that the trade-off between robustness and accuracy can be mitigated using additional unlabeled data [75, 76].
Recent findings have suggested that robust classification requires complex classifiers [63]. which is supported by the results in the neural tangent kernel regime [35]. In the context of transfer learning, robust classifiers have been shown to perform better [27]. Furthermore, a certifiable adversarial training procedure has been established, constraining perturbation by the distributional Wasserstein distance [86].
18

---

The aforementioned outcomes rely on specific data distributions, such as Gaussian or heuristic-tuned distributions, or simplistic models, such as linear classifiers or two-layer neural networks. The lack of theoretical research on adversarial training in deep neural networks stems from the complexity of training these models, including non-convex loss surfaces and stochastic optimization. Recent research has employed the neural tangent kernel regime to address these challenges, demonstrating that adversarial training can yield robust network with near-zero robust loss [35]. This result was later extended in two-layer neural networks [115], eliminating the assumption in [35] that requires exponentially large width and runtime.
In this study, we conducted a theoretical analysis of adversarial training, focusing on the time evolution of network parameters during training, the conditions promoting adversarial training, and the differences between adversarial and standard training. Our analysis targeted deep neural networks without relying on any assumptions about data distribution and employed lp norms practically used as perturbation constraints, rather than more impractical metrics such as the distributional Wasserstein distance. To address the training difficulty of deep neural networks, we utilized mean field theory.
Finally, we mention the report from a perturbation instability perspective that increasing network width does necessarily improve robustness in adversarial training [98]. This may seem to contradict our results, which suggest that a wider network can help the model maintain capacity during adversarial training, implying greater robustness in wider networks. However, these two claims are compatible. Robustness is determined by both perturbation instability (negative effect) and network capacity (positive effect). While the negative effect of width appears dominant in [98]'s experiments on CIFAR-10 10 and WideResNet, the positive effect appeared more prevalent in our experiments on MNIST,
and fully connected networks with or without shortcuts. The dominant factor may depend on the dataset and model architectures.
B Gradient independence assumption
The gradient independence assumption was first introduced in [81] for backward dynamics in mean field theory and later refined in [105]. We provide a definition based on [105] as follows:
Assumption B.1 (Gradient independence assumption [105]). (a) We use different set of weights for backpropagation than those used to compute the network outputs, but sampled i.i.d. from the same distributions. (b) For any loss L, the gradient at layer OC/Ox((), is independent of h() and (1--1)
Although not strictly accurate, this assumption has been empirically found to hold well [81, 105]. has been rigorously justified for specific architectures, including vanilla, residual, and convolutional networks [103]. In this study, we applied the assumption the standard loss function but not to the adversarial loss function. In addition, we regard a network output as loss and apply Asm B. to the network output in Sec. 5.4.
C Setting of residual networks
The mean field theory for residual networks studied in [105]. Although they employed trainable weights in shortcuts, we employ the untrainable matrix. Formally, the pre- and post-activations in the 1-th layer of residual network are defined as follows:
h ( = ( xz (1-1) (0)
x ( = !1-1 + p000pph)p)) ()hh)))
(A16)
where P() E IR NxN is an untrained random matrix. Each entry of P()) is i.idd sampled from a Gaussian (O 1/N) at initialization. The random matrix p()) is introduced for simplifying probabilistic calculations and is applied in accordance with Asm B. (a)) The definition of x( given in Eq. (A16), is slightly different from the original definition in [105]. Therefore, we will derive the basic probabilistic properties pre- and post-activation again, based on Eq, (A16). Following a similar approach to [71, 105], the mean squared pre- and post-activation can be calculated follows:
[( (2)
= - [< -1-1))1-1)11- +00
E[(a
E[(a (1 1))) E((()
(A17)
19

---

Additionally, following [81, 105], the mean squared gradient with respect to pre- and post-activation can be calculated as follows:
27
0c
Oc
E
(A18)
!
an!
x
(1 + o1+ Blo (1+ (1)x
(A19)
We can derive Eq. (A18) under Asm B.1 as follows:
oc
ac
0h (++) du; !1)
E
=E
(A20)
0h!
(1)
@h(++1)
du,
Oli (8)
N
0C
E
Dh
(1+1)
jt
1)
o (1(
(A21)
27
OL
= E (h,
(1)
(A22)
((t))
We derive Eq. (A19) under Asm B.1 as follows.
27
OC
X
=
E
(A23)
0z1
E
OC
27
2E
N
N
OC
OC
p(t+))
sh p'(n ((1(tt))
Ox ((+T) (L+T)
dx
(1+1)
Oxy
[41)
j=1
E
N ΣΣΣΣ
N
Oc
dc
p(l+1)
p((+1) (1((t)))d (oh (1(111) jk jk
w ki
(A24)
Ory
On
1)+1)
=(1 Elo (h
(++1)
(A25)
From Eq. (A23) to Eq. (A24), we used the following equation:
Oc
N
OC
dx
((+1)
(A26)
0x
11)
0x
(1+1)
dr (1)
3
1)+ ao(h (1+1))
0c
8ig
(A27)
j-1 Ox (1+1
Ox!!!)
N
N
Oc
ΣΣ
p((+))
o' d'(h
(1+1) w(++)) )
(A28)
dr
(1+1)
dx
1+
j=1
k=
The second term of Eq. (A24) is rearranged as follows:
N
Oc
Oc
p((+1)
Ø (ni (11((41)) ((
2
(1+1)
Dn
jk (+1)
ki
/=1
N
11{+1)11
OL
OC
a+ o' d'(n (h(+t))wwl
(A29)
Or (++1) dx
=0.
(A30)
20

---

The third term of Eq. (A24) is rearranged as follows:
E
X (+1)
N
N
Oc
Oc
p((+)))
p((t))
(11++=)))) (h' a+
Oryy
141)) jk jk
UC
20'h (1+1) ( (p(ltT) )
E
(A31)
Dxf (+))
OC
E
B ( (T)
(A32)
=o Bl (1x
(A33)
D Sketch of proof for Thm 4.1
In this section, we provide plain but informal proof of Thm 4.1 for the stepping stone to the formal proof. We present two levels of explanation: the most straightforward one and the other that is closer to formal proof.
First, we introduce the simplest proof of counterintuitiiv independence of the distribution of J (x from ii As indicated in Eq. (6), the definition of J (a:1)) includes Ø (h( (x²"); thus, the distri- bution appears to depend x Here, we consider the distribution of ø' (h(x)), where h(x) == we with w (O, ( and E RR From the following proposition, although of (((x))) is defined as function of æ, its distribution is independent of z. This characteristic holds even for J [xx"), which encompasses multiple ø' (h() (z.").
Proposition D.1. Let (=)) == uz (z 2 0); Uz (z < 0) be ReI.U-like function, w NN0, o be Gaussian variable, and be a fixed real number. Then, the distribution of of (wx) is independent of x.
Proof. Consider the derivative of 6, given by d'(2) := u (2 (22 0); u (2 0). The input to ø', i... wa, follows Gaussian (00 =0))))))) The probability of zero-mean Gaussian being greater than or equal to zero is the same as it being less than zero, regardless of the value of æ. Therefore, for any given æ, the probabilities of ((wx) = u and o (wwx) = u are invariant to x. Consequently, the claim established.
Then, let us consider Thm 4.1 in a neural network with one activation and one weight layer, re- spectively, as f(x)= := pout of (1()) pin "") In this setting, the network Jacobian is represented by [xx") == pout ( (w() ( "") ) pin. Moreover, we assume that the uncorrelated Gaussian variables are independent. This assumption is incorrect. Uncorrelated Gaussian variables are not necessarily independent (cf. Remark E.3). However, for the intuition about the formal proof, we assume this. The simplified Thm 4.1 and its proof are as follows:
Proposition D.2. Consider a neural network f(x) == pout (ii Suppose that the width N is sufficiently large. Assume that uncorrelated Gaussian variables are independent. Then, for any xt E Rd, each entry of J(x")) pont ( ((p((W(1) pinx"")) (1)) pin is i.i.d. and follows the Gaussian (0, /
Proof. First, we prove that each entry of [xx is i.i.d. and follows Gaussian. To this end, we consider the probabilistic properties in the order of in (( pin W(1) pin and poat ( (1) pin (1)) pin, Then, we derive the concrete distribution, i.c. its mean and variance. Comparing the distributions between ) and
we achieve this.
We first consider pin =: (0). Recall that zn is a deterministic vector and pin is a random matrix where each entry is i..d. and follows Gaussian. Since the weighted sum of independent Gaussian variables follows Gaussian, each entry of x (0) is i..d. and follows Gaussian.
21

---

Then, let us consider ())) The i-th entry of (()) is expressed ΣØ= (1) x.)) Since
W is random matrix with ii.d. entries, W (1) (0) 2 ii.d. respect to j. By the central limit theorem with infinite N, the i-th entry of a()) follows a Gaussian. In addition, the entries of (()) follow the same Gaussian. Moreover, since B[(_)= W 4;
(0)) (0) {_) W 2;
holds for i * i', different entries of (())(()) are uncorrelated and thus independent (this originally incorrect but guaranteed by the assumption here). conclusion, (()) is a random vector with i..d. Gaussian entries. Naturally, D(Ø' (W()))ia is diagonal matrix with i.id. entries.
Similar to the discussion above, ( pin A is random matrix with ii.. entries.
Finally, consider pout ( (W(()) (in (1)) pin, For notational simplicity, we denote == ((v((W() pinn)))))))) Thus, Pout ((o((W()) pin (()) pin pout DA. The entry of i-th row and j-th column of this matrix is expressed as Σx=1 Pot ARj- Note that pout is inde- pendent of DA and its entries are i.i.d. Moreover, based network symmetry, the dependence between D2 and Akj is invariant to k. Thus, Pot Asj is i.idd with respect to and each entry of this matrix follows the same Gaussian by the central limit theorem. Moreover, since holds for (i,j) * (i', different entries of pout D are uncorrelated and thus independent. Therefore, each entry of [xx"") is i.i.d. and follows the Gaussian.
BIOE== ==1 Pout Dj (x)) k= PV De
Next, derive the mean and variance of (a="). Trivially, ([J(""") since the mean of an entry of pout is zero. We derive the variance by comparing (x) and E{ ((x"" (yjaj) )
Recall f(x); == Σÿ= (((="") As preliminary, with a Gaussian variable NN(0, o²) and its probabilistic density function g(z). we derive the following equation:
d(z)² o²
(A34)
0 1 + [ g²2²g((;; o 2)
u² 222g(2; )d
(A35)
u²
0 + 2
(A36)
2
2
a
(A37)
In addition, if pout and are independent, then
N Σ
N
Ee~A10,02)
pout ((2)g
pout
[[}{ 2
-xo2.
(A38)
(0) follows the Gaussian. Moreover, we can As shown in the discussion above, each entry of (1 x simply derive its variance o || /d. Thus,
)
N
B[(())]) E
pout Φ 20
= ao 1|/1/21111//11
(A39)
Since f(x); Ej= J(xx )=yz))n [[f()) is also expanded as:
2
d Σ
E[f(x)}1= =E
in J( )(yjj
= Σ E[ "| E[J(<"") 3|l (A40)
/=1
Comparing the two equations above, we obtain E[J(a.")33] = 2d. Therefore, the claim is established.
22

---

E Derivation of Thm 4.1 for vanilla networks
E.1
Preliminary
First, we introduce some lemmas. These results are more general and not restricted to the context of neural networks. The notation in this section is independent of that in other sections.
We refer to a random matrix where the mean of an entry is zero as a zero-mean matrix. Formally, this is defined as follows:
Definition E.1 (zero-mean matrix/vector). A random matrix/vector is called a zero-mean ma- trix/vector if B[A = 0 for any and j.
For zero-mean matrices, the following properties hold: Remark E.2 (Properties of zero-mean matrix). Let be a zero-mean matrix.
Let B be zero-mean matrix. Then, their sum, A + B, is zero-mean matrix.
Let be random matrix. A and B are independent, then their products, AB and BA, are zero-mean matrices.
Then, review several properties of Gaussian variables. Remark E.3 (Properties of Gaussian variables). The following statements hold:
Any linear combination of multiple Gaussian variables follows Gaussian if and only if they are jointly distributed Gaussian.
A weighted sum of independent Gaussian variables is Gaussian distributed.
Gaussian variables are jointly distributed and uncorrelated if and only if they are independent.
Next, examine the properties random matrices/vectors with i..d. Gaussian entries, called Gaussian matrix/vector. Their formal definitions are as follows:
Definition E.4 (Gaussian matrix/vector). matrix/vector called Gaussian matrix/vector if all the entries are i.i.d. and follow Gaussian.
Gaussian matrices and vectors satisfy the following lemmas.
Lemma E.S. If A and B are independent Gaussian matrices, then A + B is Gaussian matrix. In purticular, when A and B are zero-mean, A + is zero-mean Gaussian mairix.
Proof. By Remarks E.2 and E.3, this is trivial.
The following lemmas are derived for the proof of Thm 4.1.
Lemma E.6. Suppose that random matrices A E Rmxn and B Rax! satisfy the following properties: (a) and B are independent.
(b) A is a zero-mean matrix. (c) All the entries of A i.i.d. (d) All the entries of B are identically distributed. (e) Any two entries of B from different rows are independent.
(f) For any E [n],j, k E [4], & k, the dependence between Bij and Ba is invariant to i.j, i k. (g) For any i [m],j, k E [].j j F k. E(B,, 0 holds. Then, for sufficiently large n,, A BBA. is a zero-mean Gaussian matrix.
Proof. By (a), (b), and Remark E.2, AB is zero-mean matrix. The linear combination of all the entries of AB is expressed as
Sij B.
8ij
A Bkj
Sig Al Bkj
(A41)
where Sij E is constant. By (a), (c), (d), (e), and (f), En E/= respect to k. By the central limit theorem, e follows a Gaussian. By Remark E.3, all the entries of AB are jointly Gaussian distributed. Moreover, by (c) and (d), the entries of AB follow the same Gaussian. By (a), (b), (c), (e), and (g), the covariance between any two entries of AB zero. Thus, by Remark E.3, the claim is established.
An i.i.d. with
D
23

---

Lemma E.7. Suppose that random matrices A E RR B E Buxi,, and C' E Ruxiz satisfy the following properties: (a) A, B. and are independent. (b) A a zero-mean matrix. (c) All the entries of A are i.i.d (d) All the entries of B are identically distributed. (e) All the entries of C are identically distributed. (f) Any two entries of B from different rows are independent. (g) Any two entries of C' from different rows are independent. (h) For any i E [n], E [1),5 # k, the dependence between Bug and Ba is invariant to j. (i) For any E [n], E [12 * k, the dependence between Osj and Ca is invariant to i, j.j.k, j.k. (j) For any i E [n],j E [4], k E [2]] ElB,, = O holds. Then, for a sufficiently large n, AB and AC are independent.
Proof. By (a), (c), (d), (e), (1,, (g). (h), and (i), similar to Lemma E.6, all the entries of AB and AC jointly Gaussian distributed. By (a), (b), (c), and (j., the covariance between any two entries from AB and AC' is zero. Thus, by Remark E.3, the claim is established.
Lemma E.8. Let w. E T be Gaussian vectors. Let d' (z) := (z 2 0); (2 < 0). Then, for sufficiently large m, o' d'ww" x)w and x are independent.
Proof. We prove Pio'(u x)w a x x== P(o' ToxT)))) z)w a). For any z, w a/uu /m or = a/v a/ v/vm m satisfy s'ww" x)w a. Since m is sufficiently large, w a/u + a /m and w = a/v 2/ /mm asymptotically approach w a/u and ur a/v, respectively. Thus, the claim is established.
E.2 Definitions of J "x and axx"") for vanilla networks
Here, we define (derive) J (x² and axx")) for vanilla networks. The (-th pre- and post-activation defined as follows:
(x W (1) (1 (x²") + b(),
x (x (5 (1x")))x (x)
(A42)
Using (x1") = D( (z'i) (zin), the I-th pre-activation can be rearranged as follows:
h h (ax)) w() D (h (1-1) (a (- (x."") b()).
(A43)
Because ReLU-like networks are piecewise linear, the I-th pre-activation is also represented as follows:
(0) (xxi)) = J (2'") (x + (1) (z²").
(A44)
Substituting Eq. (A44) for Eq. (A43), the equation can be rearranged as follows:
h0 (x = w D( (h (1
)z(0)))))))))))))))))) (m
+ ( ( (h (-"")( )))" (ax"") + (()..
(A45)
Comparing Eq. (A44) and Eq. (A45), the following equations be derived:
(xii)) == ( D(O (a
(A46)
(1-1) a (xon ( D (( (1-1 (2")) (2x + (1)
(A47)
where (20()))) == ( and a (zxin) := (1) Finally, J in and a axx")) are defined as follows:
f( J ) )az + aa
"xx = pout D (()"))
(6)
a "m = pout D (( (L) )) (4)
(A48)
E.3
Main derivation
First, we remark several fundamental properties of weights, biases, random projections, and pre- activations in a network. Remark E.9. By definition in Sec. 3.1, the following statements hold for any E [L]:
24

---

(Gaussian matrix/vector) W Pin, and pout are zero-mean Gaussian matrices and b( is zero-mean Gaussian vector.
(Variance) The variance of W Pia and pout is oN/N, 1/d, and 1 I/N, respectively.
(Independence) W and b are independent of each other, as any random variable in the layer before the /-th. In addition, pout is independent of any random variable without network output.
(9) and D(0)
(Dependence between pre-activation and bias) For i = j. the dependence between is invariant to i. For i + j. hg) and bg) are independent.
Lemma E.10. Consider a vanilla network. For any E [L]. the following hold:
(a) Jo (x!") is zero-mean Gaussian matrix. (b) al (xr") is zero-mean Gaussian vector.
(c) (x!")) and (a))))))))))) (xx are independent.
Proof. We prove the claim by induction. As preliminary, Remark E.9. The case with / is trivial. Suppose that all the claims hold for 1 - 1 E 11-
(a) By Lemma E.6, the claim is established. Note that Eld' (2 (1 = since N sufficiently large.
E|d' ("d'hh"))
(b) By Lemma E.6, the claim is established.
(c) By Lemma E.7, the claim is established.
Lemma E.111 The mean squared network output is given by:
wh po oo
I
(f(()) d
wi-
(A49)
Proof. By Remark E.9, the mean squared pre- and post- activation calculated follows (cf. Eq. (A34)): [( (1) = 0? E(1)) 21] of
(A.50)
(h, )
(vanilla) (residual)
(A51)
E[(x (1- 1 + (E(()))))
Recursively calculating Eqs. (A50) and (A51), the mean squared post-activation in vanilla network calculated as follows:
(=; = (E((h()) w. () + aoل = [(E((x ("))) ao!
w
(A52)
The mean squared post-activation in residual network is calculated as follows:
[<x = [a (1)(((1)))11 + (( (1)(((==))))++
(A53)
w. (( 1) + ao²
(A54)
(- (0) + ao wi-1
(A55)
Using z (0) = in the above results are expanded as follows:
wt
[x (0))))))) w Σ B((Pin)) + ao 1-1
||
ao Σ wi
(A56)
d
25

---

The mean squared network output is rearranged to the mean squared L-th post-activation as follows:
N
Σ
El( P (1
[F( (}.
(A57)
j=1
By Eqs. (A56) and (A57), the claim established.
Theorem 4.1 (Properties and distributions of J (x²")) and a(xl")), Suppose that the width N is sufficiently large. Then, for any x²" & Rd, (I) (xx") and axx"") are independent. (II) each entry of J ( and a(x!") is i... and follows the Gaussian below:
J( N (0. d
)
wt
a(x");; N (o Σ )
(7)
where = (u² + 12 (cf. Defn 3.1) and w is wy == ao for vanilla networks and wi :- 1 ac for residual networks.
Proof. Similar to Lemma E.8, pin and a a(x independent. Similar to Lemmas E. and F.2, Jx and a (x are a zero-mean Gaussian matrix and vector, respectively, and J (xx and a(x²)) independent. The mean squared network output can be expanded as follows:
27
=E
Σ J(x Jaja + =al""),
(A58)
d d
E (x: 2m )را [ (se [ xx + 2 >
B[J( ((a(")) + B[a(
(A59)
2=1
Σ [[J(x ( 22) 1(a(x>")))
(A60)
Using the symmetry of entries of J ( and a ") the equation can be simplified as follows:
B[f(= B[J(""") - _/ [[a()"))))
(A61)
Comparing Lemma E.11, Eq. (7) is obtained. Thus, the claim is established.
F Derivation of Thm 4.1 for residual networks
F1 Definitions of J (x"") and a(x"") for residual networks
Similar Appx. E.2, define (derive) J [sxin) and aax: for residual networks. For notational simplicity, we omit the argument in First, we represent (!() as follows:
x (I + (z) + c(),
(A62)
where V (0) and (0)) 0. With (1) and (!)) we can represent h (1) as follows:
(1)) (() + (())
(A63)
(1- + ((ا- + (()) (T+ + ( (1-1) )) + (0)
(A64)
(A65)
Then, we derive recurrence representations of ( and (ا as follows:
( (1-11) ( ((h()))
(A66)
=(1 + ((1-1) x (0) + (l--))
+ ( D( (w (I (z ( + (11) +b()))
(A67)
26

---

For reference, we denote
U
d D(d'b
(A68)
With the notations above,
xx ()) = (I + V((-1)) + ((-1) + (I (((-1) x (0) + c11-) +d
(A69)
(I+ ) (I+ ( (0) (I U |[ + d
(A70)
(I+U0 + v U v(1-)) (1-1)) + U c d
(A71)
Thus,
4 = U + v(i-1) + ( ((-1)
c + U c (1-1) + d
(A72)
Finally, (a and a are defined as follows:
J == pout (I + p(i))
a(x²") = pout pL)
(A73)
F.2
Main derivation
First, we note the properties of random projections in shortcuts. Remark F.1. By definition (cf. Appx. C), the following statements hold for any E [Z]:
(Gaussian matrix/vector) p()) is a zero-mean Gaussian matrix.
(Variance) The variance of p(!) is 1/N.
(Independence) p()) is independent of W((,, b(), and h(",, as is any random variable in the layer before the 1-th.
Finally, we introduce a lemma similar to Lemma E.10. A subsequent discussion to derive Thm 4.1 is the same as the proof in Appx. E.3.
Lemma F2. Consider residual network. For any 1 E [Z], the following statements hold:
(a) J (xit) is a zero-mean Gaussian matrix.
(b) (ar")) is zero-mean Gaussian vector.
(c) J and (!)
(1)(()))))) (xiin) are independent.
Proof. As a preliminary, please refer to Remarks E.9 and F.1 and Lemma E.10.
Note that effects of all the random variables in the layer before the 1-th, such as p(-1), (1-1) ((I+1), and (f-1), are aggregated (!-1)) By Lemma E.8, U()) and d( are independent of U(') and d(')) for any + Similarly, as V(I) and (! consist of U()), ,..., U d(1), d(),, U(++1)
and (!!-1) are independent of (1) and c(!), In addition, similar to Lemma E. U and d() are independent by Lemma E.7.
We prove the claim by induction. Assume that (1-1) a Gaussian matrix. This holds for = 1 because of ((1) UU)) Similar to Lemma E.8, U.y independent of U. V.j. Since (T)) and ((l-1) are independent Gaussian matrices and N is sufficiently large, ( is a Gaussian matrices. Similarly, (!) is a Gaussian vector. By Lemma E.7, ( and ( are independent. Thus, similar to Lemma F. 10, !ax!") and a(x!") are independent Gaussian matrices.
G Derivation of the theorems in Sec. 5
G.1 Derivation of the theorems in Sec. 5.1
First, we introduce the following lemma, which is required to derive the maximum Eoo norm of column of (xx
Lemma G.1. The maximum absolute value of n E N i...d. Gaussian variables with zero mean and variance o² > 0 is smaller than 2 Inn.
27

---

Table A3: Computable combination of (p. operaaor norm [91].
q 1
g 2
g
p
max. L norm of column
max. Eg norm of a column
max. lo norm of column
2
NP-hard
max. singular value NP-hard
max. l norm of a row
p= 00
NP-hard
max. & norm of row
Proof. Let z E IR be a Gaussian variable with zero mean and variance o² and a > 0 be positive value. First, compute the upper bound of probability of follows:
PIz > a)
g(z; o 0
(A74)
((; 0 dd
(A75)
1
-0² exp
2² 00
(A76)
aV
2TT2
20
1
a²
exp
(A77)
a
2TT
20
Then, we compute the lower bound of probability of ||| < a as follows:
a
P||2| < a) =
g(2; 0²)
(A78)
g(2; 0² ddz + 2 f g(z; o2
(A79)
=
2
giz 0 )2 dd -
(A80)
20
= 2
1
g(2; (12;02)dz) -1
(A81)
=
1
g(z;
ddz
(A82)
1
20
a²
21
exp
(A83)
20
Let {= be i.i.d. Gaussian variables. Finally, we compute the upper bound of probability of maxic(1] |2i > follows:
1
20
2
P max
|2
(1 P||2|| < a))"
exp
(A84)
iE(n]
20
In particular, when
20 Inn,
n
I max
2//
20 Inn
n=000
(A85)
iE|n]
n Inn
Thus, the claim is established.
Theorem 5.1 (Upper bounds of adversarial loss). Suppose that the input dimension d, output dimension K, and width are sufficiently large. Then, for any xit E Rd., the following inequality holds:
(Gppl LiN Ewew W2)L/2 (pp.g1+ LN Ewew w2yr/2 (residual)
(vanilla)
Ladv (x0 2pp 4/2
where W = (W()), 1,1 W(22 ....... w( denotes the set of all network weights. The constant Sp for each norm pair (p, is described in Tab. 1.
28

---

Proof. We note the following:
By Thm 4.1, each entry of [( is i..d. and follows Gaussian with zero mean and variance v[J(a"));) = wh/d. /d.
The input and output dimensions, d and K, are sufficiently large (cf. 3 3.1). For some combinations of (p, q), (p, o)-opeato norms are computable (cf. Tab. A3) [91].
In addition, we note the following upper bound (cf. Sec. 5.1):
Ladv (a
max wERd
||J (z)
mlm.
(Ineq. (8)
As preliminary, compute the mean absolute value of J (x
follows:
2w
BI
(A86)
nd
In the above derivation, we use the following equation
2
E o=0
(A87)
Based on Tab. A3, we compute [ la as follows: Maximum & norm of column, [IJ(
1 K
max JEId
J(x "0 )ayl max K
(A88)
jEld K
max [ [T ())lll 5Eld]
(A89)
2i0h
max K .E[4
(A90)
2 V rd
K/2
(A91)
Maximum & norm of a column,
|| (11.2
K (xin)
max JELd
Σ [(xin)
= max JEld]
K
(A92)
K 1=1
max
KE (xcin
(A93)
SE[4]
K w..
max JE[d]
(A94)
d
K
v WL/2
(A95)
Maximum lx norm of a column, [> )|, By Lemma G.I,
2 ln K
max
[ Disk S (zi)) In
L/2
(A96)
KK[K
d
Maximum singular value, [ [[](=i")||22> ||2.2- By the Marchenko
Pastur law,
K V
| <
1
+
(A97)
d
29

---

where || ll2 denotes the spectral norm (largest singular value).
Maximum l norm of row, (a)"")))))
d
max EE[K]
J(
max KE[K]
d
(A98)
max kE[K]
[TE] )
(A99)
dwu.
max KE[K] w/2
(A100)
(A101)
Maximum & norm of a row, || (ax"))
10,00
1 d Σ
max VE
J in )ayl = max d
J(x )ssl
(A102)
kE[K] d
max dEll (z))))))))sll [E[K]
(A103)
20
max dv (E(KK
(A 104)
2d
w1/2
(A105)
In addition, we try to derive equalities rather than inequalities (upper bounds). First, we note the following equation:
Ladv(x"") max [llll,
f + +n)
- fx="")
(Eq. (3))
max |xx + (x)" + m) + al + +n) J ) alx"")|-- ml
(A106)
Because J( ) and axx"") are the slope and bias of a piecewise linear region, respectively, for sufficiently small n, J [xx + n) and a(zin + n) are identical to J(x!") and a(x"",, respectively. Therefore, for sufficiently small v,, we can derive the following equation:
Ladv ()) = max St
Jxx "n mnl = J(
(A107)
Proposition G.2. Suppose that the perturbation constraint is sufficiently small. For any x: Rdd and (p. q) = (1, (1,2), (2, oo). and (oo, oo), the following equality holds:
(alv( = =Bppqu 1/22
(A (A108)
Proof. As shown in Eq. (A107), the adversarial loss (Eq. (3)) is equal to the (p, -)-peratoor norm of J (x under this assumption. For (p.q) 1). (1, 2), (2, co), and (co, oc), equalities of the (p. -)porrato norms (cf. the proof of Thm 5. Thus, the claim is established. Note that, for (p. g) (1, oo) and (2,2), can derive only upper bounds (cf. the proof of Thm 5.1). D
obtain the
Proposition G.3. Suppose that the perturbation constraint e is sufficiently small and the output dimension one. For any "! E Rdd and (p, q) = (2, 00) and (oc, oo), the following equality holds:
Ladv = 2.p.
(A109)
In addition, for any i E Rdd and (p, 9) = 2), the following equality holds:
Ladv ) ewr2
(A110)
30

---

Proof. Similar to Prop G.2, the upper bounds for (p.g) (2, and (oo, oo) become equalities.
The upper bounds for (p. q) (1, 1) and (1, require sufficiently large K, and thus, they do not become equalities. In addition, as Tab. A3 shows, the (2, 20-oprato norm is maximum singular value. When K = 1, the maximum singular value is identical to the Frobenius norm (&z norm), which is calculated as follows:
J (x: (a="))
(A111)
j=1
d d
{ami))
(A112)
==1
v dBE (x 3g
(A113)
4/2
(A114)
Thus, in contrast to the proof Thm 5.1, we can obtain the equality of the (2-eraorr norm instead of the inequality (upper bound).
G.2 Derivation of the theorems in Sec. 5.2
In the following, we set Lady == @/p. As a preliminaryy we state the following two lemmas. These are used to solve the differential equation derived from gradient flow.
Lemma G.4. For L, a, b IR, x: R + I,, and x0)) : Io, the following holds:
dx((() = ar(t)) + xx()) = (a(b 1)t + xo (b-1) --101)()1 1/(6b11)
(A115)
dt
Proof.
dx(t) ar(())
(A116)
dt
x dr = -a / dt
(A117)
1
(b-1) -at + C1
(A118)
- 1)
z(t) (a(b 1)t + C2) 11(0-1()
(A119)
By the initial value,
x(0) = x0 C₂ 1-1((b11
(A120)
- (h-1)
C2 = xo
(A121)
Thus,
x(1) (a(b 1)l + x (b-11)) (b--1)-11(0-1))
(A (A122)
Lemma G.5. For E RR a, 0, 0.::: R. x00)= = To. and < bzx(0) < 1, the following
holds:
dr(t)
= aa( bx)) e sxt)
2
(A123)
dt
(1 + bc exp(at) - bcxo
Proof. By -a(1 + (1) < 0, (t) does not increase with t. That is, bar(t) < bo((0) < Therefore, we can approximate (1 + bor as 1 + bex(t) using the binomial theorem. Thus, we try to solve the following differential equation:
dx(t))
= a(1 +
(A124)
dt
31

---

whose solution is as follows:
C1 xxt) be(exp(at) C1)
(A125)
By the initial value,
C1
z(0) Ib bc( C1)
(A126)
bc(1 - Ci) C1
(A127)
berp = (1 + bcoro C1
(A (A128)
bcro
C1 + bczu
(A129)
Thus,
bctt T4-bemn
x(t) = ba(exp
(A130)
bezo
T-tbbeag
bczo
(A131)
be((1 + berr)) exp(al) bcxio)
(A132)
(1+ + bexr) exp(at) bewg
L
Then, using the update equation of parameters (Eq. (10)), we consider the differential equation of weight variance as follows:
be the loss functions and W: = (Wil), W ..... w(1 NN be the set of all network weights. A network is trained by minimizing the sum of loss functions, Diz Li(x""). The network parameters are updated similarly to Eq. (10). The differential equation of o w (1) is given by:
Lemma G.6. Suppose that Asm 5.2 holds. Let Li,..., C,, : R
do (t) df
ac, (n))) aw (t)
-NEweW
W(t)
(A133)
Proof. Since Asm 5.2 holds and the number of weights, LN2, is sufficiently large, (t + dt) can be represented follows:
2tt + dt) N - at dt) NVwew(W d()] NEw (W dt)
(A134)
N
We can expand W(t dt) using Eq. (10) and rearrange the above equation as follows:
27
" aCi(()t))
2 + d0) NEwew
di W(I) OW(()
(A135)
"
(2,(
NEwew W()) w(t) Σ
dt + Odttt)
(A136)
(W
(xr
(t) NEWeW W(t)
dt
+ +Odtt
(A137)
(t)
1
Rs (1) NEW W(1)
de.
(A (A138)
OW (t)
Thus.
do (1)
CC.
-NEwew ((t)
(A139)
dt
aw
32

---

Lemma G.7. For Ladv = pp.g 1/2, , the following equality holds:
GLddy
ca:3p.que 1/2-1
W.
(A140)
N
Proof. If the number of network weights, i.e., LN2, is sufficiently large, we can represent the weight variance as follows:
Σ Wº.
(A141)
LN
WEW
The derivative of w with respect to W E W can be calculated as follows:
Ow,, M(a))) i(aN
N
0
LN Evew V2)
2a
aN
W,
(A142)
OW OW
OW
OW
LN
Ow, 0(1 + a
2a
W.
(A143)
aw OW
LN
The derivative of 4 with respect to W E W can be calculated as follows:
Ow
w 1/2-1 Ow
L
4/2-1
2a
aw-l2-1
W
W.
(A144)
OW
OW
LN
N
Thus.
dLadv
Ow
Ea pa W.
4/2-1
=.p.
(A145)
OW
OW
Lemma G.8. Suppose that Asm 5.2 holds. Let Lard Rd - E be the standard loss function and Ladv : Rd - R be the adversarial loss function. Suppose that Asm B.1 applies to Las-- The adversarial loss function is defined Cadv(x"": == =Bp.w(())= 1// network is trained by minimizing fisdd + Ladv- Network parameters are updated by Eq. (10). Then, the differential equation of a² (t) is given by:
2o dt
ECI p.g
((1)2/2-103))
(A146)
Proof. By Lemma G.6,
do (1)
2Lad OW(t)
((t) dLadv @W(()
NE ((t)
NE
(A147)
dt
Since the standard loss satisfies Asm B.1, E (t) 8C.s
= 0 and
aw
do (t)
dLadv (W())
dt = - NE
W(1)
(A148)
By Lemma G.7,
do (1)
NE (t)) Ea (() w(t)
(A149)
dt
N
(t)
(A150)
capp
w(t)2/2-100())))
(A151)
N
Theorem 5.4 (Weight time evolution of vanilla network in adversarial training). Suppose that Asms 5.2 and 5.3 hold. Then, the time evolution of o of a vanilla network in adversarial training given by:
E w (0) 4/2-1
(t,()) (1-
{ - (0).
(11)
N
33

---

Proof.
By Lemma G.8, the time evolution of weight variance in vanilla network is represented as follows:
do (t)
ca.5p
w. (t())//2-1022(1)
(A152)
dt
N
ea.pa 4/2-1 1 (2)1))//-))
(A153)
N
ea
3 p. o2 (1)
(A154)
N
Denote L' == L/2 - 1. By Lemma G.4, we can obtain
-1/(2/2-1)
(1)=
(L/2 1)1 (0)
(A155)
N
-1/2
t (0)-
(A156)
-1/1'
Eat L' +100 (0) Bonll' Li 1+
(0)
(A157)
1111!
Eac,, (0) Bpr 1
(0)-
(A158)
N
&N N,
1 p.gh, (0)
(0) = -(1 (a)
(t)+)= ~
2 (0).
(A159)
L'
N
N
Theorem G.9 (Weight time evolution of a residual network in adversarial training). Suppose that Asm 5.2 holds and ao? w (0)
< 1. The time evolution of o of residual network in adversarial
training is given by:
a (t) - a1'
(A160)
Proof. By Lemma G.8, the time evolution of weight variance in residual network is represented as follows: do (t)
capp (a)pee(())) (1)_/22)2
EC.
+ (0)
(A161)
dt
N
N
By Lemma G.5, we can obtain
の (1)
(A162)
(1 + 1 exp
caspar a -L/2
=
L/N) aL''-1
(A163)
By the Maclaurin expansion of the exponential function and < T
< N,
(t) ( [1 ca:3pa
-1
- al'
(A164)
N
+62))))++ Ecr
(0) ( +aL)
(A165)
N
Eapp.tt (0).
1 (1+ aL' (0)
(A166)
N
By < T N,
cappat
22) 1 (1-(1 (1 a.
02 (0))
o (0)
(A167)
N
34

---

Then, we consider the time evolution of weight variance, Thms 5.4 and G.9, at initialization satisfying (M, m-)triinability condition (Lemma 5.6). Here, we assume w()) = 1 satisfying Lemma 5.6. Under this assumption, the time evolution of weight variance in vanilla networks, Eq. (11), can be rearranged as follows:
(t) (1 ca.
(0) - (1 // (0).
(A168)
In addition, the time evolution of weight variance residual networks, Eq. (A (A160), can be rearranged as follows:
(1) - (1 a (1-
(,) o (0) (+ - pat-aap..//Nool (0).
(A169)
Thus, the time evolution of weight variance is consistent in vanilla and residual networks at initializa- tion satisfying Lemma 5.6.
G.3 Derivation of the theorems in Sec. 5.3
Lemma 5.6 (Vanilla and residual (M, t-aaniabbbill condition). Suppose that the width is sufficiently large. Then, the (M,m)--rrannbbiiii conditions for vanilla and residual networks are respectively given by:
m // S 2o S M (vanilla),
oo S M - (residual).
(12)
Proof. Applying Eq. (A38) to Eqs. (2) and (A19), obtain the following equations:
( Eld 11 11 Bl ( [+] = 1 ao wi (residual)
ao Why
(vanilla)
(A170)
[++
Then, recursively computing the above equation,
derive the following equations:
(0)
( (a0 (1+ 20 (residual)
(vanilla)
wt
(A171)
[T)
Applying this equation to Defn 5.5,
x ()) (
m
SM + 1/4 sw M//4
(A172)
For a vanilla network, For a residual network,
m²/4 2002- s M
(A173)
1/4 - 1 5 a0² < 1/ 1. In particular, for a residual network, by m/ /L 15 and ao 2 0,
(A174)
(0 (0 < M 1
(A175)
Theorem 5.7 (Vanilla networks are not adversarially trainable). Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, and the (M, m)
(M,m))-raimatieee -rainabbliiy condition holds at L 0 and ag (0) 1. If
T> (1- --
(13)
co.Bp.g
then there exists 0 T S T such that the (M, m))traiiabilliyy condition does not hold for T S
Proof. Let us consider T breaking the (M, m) (1,.m)))triiaiaa -trainability condition. As Thm 5,4 claims, o² (1) decreases monotonically with Thus, consider T such that o (T)) is less than the lower bound.
m 1/t 2 20 (T) (1 ca.3p. T
20 n0 (0) 1 cc.Sp.g T
(A176)
N
N
cc.$p..T
T
2 1 1/L
(A177)
N
(1 LLN
T T>
(A178)
EaxBp.
35

---

Theorem 5.8 (Residual networks are adversarially trainable). Consider a residual network, Suppose
that Asms 5.2 and 5.3 hold, and the (M, m -raiiniibby condition holds at 0 and ao (0) < Then, (M, m) -tarrnabbbyl condition always holds for 0 S.ST.
Proof. By Thm G.9, (t) decreases monotonically. Thus, the following inequality holds, and, by Lemma 5.6, the (M, m) -tttinnbbiy condition always holds in 0 S & T:
ao w (t) I ad (0) S M
(A179)
Proposition G.10 (Residual networks are adversarially trainable without careful weight initialization). Suppose that Asm 5.2 holds, the network is residual, and the (M).mtarraiiabili condition is not satisfied at If
(ao (1 (M 1))
T
(A180)
Eat/2+1 0z (0)
there exists < such that the (M, mt)-trainnabll condition hold for T S ST.
Proof. Denote the time evolution of the weight variance on a vanilla network by o and the variance on residual network by oZ By Eq. (A161),
d mm (t)
ca.pa (1 ao (t))
ca. 122.133p. 0 (t)
(A181)
dt
N
N
Recall that the following differential equation represents the time evolution weight variance on vanilla network (cf. Thm 5.4):
(l) 4/2 ppg o (1)
(A182)
dt
N
From the above, (1) S (t) if a (0) is the same. Here, consider T such that o2 ( is less than the upper bound of the (M, traiinbbbyyy condition. By Thm 5.4,
ao u. (T) S 11/L-11
(A183)
1 (1-
ea:Bp.g 4T) LT
a0 2i (0) S M//L_
(A 1844)
N
Ea Ea 1'' 0z t0 1
1 -1
(A185)
N
(02
-1
ca@pga"
0 TT
S
(A186)
a
N
(oo (0) 1/4 1/ + 1)N
ao (0))c. o' (0) ST
(A187)
(ao (0) - M 1. + 1)N
c/21121 221 pp (0) ST.
(A188)
G.4 Derivation of the theorems in Sec. 5.4
First, we state the following two lemmas as a preliminary.
Lemma G.11. Consider vanilla network. Suppose that Asm B. is applied to the network output. For any e [L], the mean squared gradient of fs with respect to h is given by:
27
= ait Li (L).
(A189)
!
36

---

Proof.
27
Of
Oh 5
E
(A190)
(1+1))
Ouy
)hg (1
0f:
W(+11
W (+11)
E
(A191)
0 ((+1)
By Asm B.1 and Eq. (A38),
0f:
af
2 L (1 )
(A192)
27
a5,
F.
(A193)
ahk (1+1)
0f:
(A194)
01k (L)
Moreover, by Asm B.I,
(
0f E
E
Of
Oug
)
EE'
(
Of: (T) 0he
o' d'(h
= (1)
(A195)
5h (T)
an. (1)
Thus,
2
0fi
E
aw! 1-11 x
(A196)
oh!
Lemma G.12. Suppose that Asm B.1 is applied to the network output. For any 1 [L], the mean squared gradient of fü with respect to weight is given by:
aʃ @w!! jk
2
H
E
ow -
(1 2 w
1-1
10)
(L)
Σ
m-1
(A197)
Proof. Vanilla. By Asm B.1,
0f oh 1
27 )
Of
0f:
E
=
B[(at (1)))
(A198)
aw
th jk
Oh a
@
By Lemmas E.1l and G.11,
Of
1-1
E
owN L- (2) wtt ([([(((z))) + ao
Σ
why
m1
(A199)
awe 3k
m=1
(0) )1 a2 -owt (
Σ w m-1
(A200)
m=
37

---

Residual. By Asm B.I.
27
27
of:
Of. dr
(A201)
@w ,jR
(
yry 1
Th
(1)
aw!l (1
jk
2
a
& (1(1) (1 22
E
(A202)
0x (1)
OS
El Blo (E((()))))>>
E
(A203)
0u (1)
ElO (h) (1)
(1 12.
(A204)
By Eq. (A38),
Of
ax E( 11)
E
(A205)
aw
By Eqs. (A19) and (A38),
x²) = w (1)
(A206)
Thus, by Lemma E.
Of: E
2
1-1
= ow, (1)) w/ " (0))) + aog Σ
Wr
m-1
(A207)
aw jk
m=1
1-1 Σ
aw,
1- (0)( + azw L- (1) o (1))
w
m-1
(A208)
m=1
Now, we can simply represent the mean of the Fisher-Bo Rao norm.
Lemma G.13. Suppose that Asm B.1 is applied to the network output. Suppose ||=||| ||2 vd and
of 0. The mean of the Fisher-Rao norm given by:
|[[[||
(A209)
Proof. The following discussion is based on [49]. We rearrange and expand their discussion for ReL.Ulikee network. We also consider it for residual network. Note that w == (W(1) "........ w(() NN) T. As preliminary, see Remark E.9. The Fisher-Rao norm is calculated as follows:
LN LN² Σ
LN2 LN K
f 0f
w
FR
w
Fw
F.FFyu
ow; Ow,
wwwy.
(A210)
By Asm 1.
LNZ LN Σ
0fx Ofk Bwu Ow,
E 10
[F]]
wawj
(A211)
1=1 #=1
I.2² Lx² K
0.5x 05k aws Ow,,
Σ
Σ
E [[gwy]]
(A212)
1=1 グ=1
2 LNV2 K
27
Σ
E
(A213)
N
i=1
38

---

By Lemma G.12, ([ ) is invariant to l if o = 0. In other words, El(OSx/ does depend on i of 0. Thus,
Ofk 27 0ro
|||l w FR = LNKO
LN 10
wa 1-1
(A214)
Moreover,
0fi
0 D k=
Pik It
(2)
(1) E
=E
El(P ) N
(A215)
Quly
gxy (L)
[((x = E
pin in 1j f
Σ
B[( (x
(A216)
Thus,
||||2|||R a
(A217)
Theorem 5.9 (Adversarial training degrades network capacity). Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, Asm B.I is applied to the network output, and the (M, m))triinability
condition holds at = 0 and oo (0) 1. Assume ||ac"||| = vd and of 0. Then, the
expectation of the Fisher-Rao norm given by:
E|] [[]( LK( LX(1- (1-)))))L)))
(15)
Proof. By Lemma G.13,
E [|]lw||p = (t) LKatou(1)
(A218)
By Thm 5.4,
B Bi]lw||nd LKa (0)² (1- earrooa LK (1 1- Bar
(A219)
N
By 1/N 11
Elllw|ra L. (1 LeaBpat
(A220)
N
Theorem G.14 (Adversarial training degrades network capacity). Consider a residual network. Suppose that Asm 5.2 holds, Asm B.l is applied to the network output, and the (M, m) m)-triinabilliy condition holds at I = 0, e.g,, uo (0) < 1. Suppose || vd and of = 0. Then, the mean
of the Fisher-Ran norm is given by:
- -(2(L- (-
1 + (L - 1)a02
E B[]lw|| =LKao
( L -
coppiit N
+ 1)
1+
Proof. By Lemma G.13,
E [| = LKao (t))
(A221)
LKao? (t)(1 + (1) 4-1
(A222)
~ LKa02 (t) + (L 10)00 (t)).
(A223)
39

---

Temporally, denote A := (1+ (L/2 (1 By Thm G.9,
B[[ = LKa(1 - - + (L.
() (0)
(A224)
(akao"(0)( A) + (L-1)ao (L 1)Aao
(A225)
11 A (L- (1
LKao
(A226)
((. 1) 0
By (11NN3 = 0. 0. = 0. Thus,
E|l w |rl =LKao (0) A+ (L 1) A)ao (L 1) 1)Aao
(A227)
=KKoo ( (0) - (L-1100 2( - 1) Auo (0))
(A228)
= =KKao (0) + - -0)a0 (2(L 1) (0) + 1)4)
(A229)
1 (L 1)00
ea.Sp.tt
(A230)
-(2(L - 1 (0) 1) 1+ - ao (0)
N
H
Comparison with matrix decomposition approaches
Here, we analyze Ineq. (8) with another approach, which decomposes [xx") into products of matrices, and consider norm of each matrix. As related studies based on this approach, we cite the literature on certified adversarial defense [3, 18, 93] and spectral regularization [60, 108]. We consider the case of (p. q) = (2,2) for simplicity. In comparison to them, our approach different in two ways. First, their bound is not tractable due to deterministic approach. For example, they consider the spectral norm of J (xi") by decomposing J (x"") and computing the norm of each matrix consisting J (x as follows:
J (x: 112 = pont D (h W()) ... (6
(A231)
pout 2
D
/
w
D
/
pin
(A232)
L |1
< maxx(uu) [u))
pin
pout
II
(A233)
Because Eqs. (A232) and (A233) are hard theoretically manage, cannot derive theorems in this study such as Thms 5.4, 5.7 and 5.9. Second, because we calculate the spectral norm of J [a.!") directly in Thm 5.1 instead of considering the norm of each matrix and the spectral norm is submultiplicative, our upper bounds are always tighter than those derived from Eq. (A232).
Here, proceed with the discussion of tightness of their bound using probabilistic theory. One approach is the rearrangement of Eq. (A232) using the Marchenko-Pastur law. As a preliminary, we note the following:
pin
N
pout S
K
|w S 2
(A234)
2
N'
Using the above equations, Eq. (A232) can be rearranged follows:
|| ||2 I max([ul, 101)
1
N
[+
K N
(0))
(A235)
d
This is exponentially looser than Thm 5.1.
40

---

Another approach is the rearrangement of Eq. (A233) using the assumption that each matrix is sufficiently large. As a preliminary, we note the following:
N
piu
Nd Nd
v NN = V Nd Ndz =VN
(A236)
K
N
pout
KN
KNE pout
KN VK,
(A237)
KN
1=
N
N2
Σ
W
12
(W,
N2
VNog.
(A238)
N
=1
Using the above equations, Eq. (A233) can be rearranged as follows:
||(a"))|lz
max([u],/|]) N max(|u]. mxx( (A239)
This is also exponentially looser than Thm 5.1.
I Comparison with le weight regularization
As stated in Secs. 5.1 and 5.2, adversarial training serves the role of weight regularizer. Here, compare adversarial training with & weight regularization. First, we define &g weight regularization follows:
L ( == Σ w².
(A240)
WEW
where À > 0 is a scaling factor. Then, we determine the scaling factor À based on the concept of gradient vanishing and explosion. The mean of L. (WV A) is calculated as follows:
ALN o2
B|Cw (WW; A Σ WEW
ALNO
(A241)
To prevent (gradient) vanishing and explosion of E[Cw A)] under sufficiently large L and N, should be 1/(LN). Moreover, for simplicity of the derivation, regularized training tries to minimize the following loss function:
X 1/(2LN). Finally, &
:: Lard 2LN
Σ WEW
W
(A242)
Next, we consider the time evolution o2 with E weight regularization. Similar to Thms 5.4 and G.9, we can derive the following proposition: Proposition I.1 (Weight time evolution with & weight regularization). Suppose that Asm 5.2 holds. Let Lad be the standard loss function and Lu (W) (=))=11((2L)) Wwew W be the &g regularization loss function. Suppose that Asm B.1 applies to Letas but not to Lw network is trained by minimizing Lata + L (W). Then, the time evolution of o² is given by:
0 2²
LN (0).
(A243)
Proof. Similar Lemma G.8,
do (t) dt
OCw (W)
NE
W(I)
(A244)
(t
W())
.I..
W(I)
(A245)
LN
-EW (t)
(A246)
LN (t.
(A247)
41

---

Thus,
(1 - exp
(0).
(A248)
T.N
By S T N, using Maclaurin expansion of the exponential function,
a (1) (1 L. a (0)-
(A249)
Here, we consider the adversarial loss defined as (x:") == 1//2 (cf. Thm 5.1). The difference between adversarial training and &g regularization lies in the scale: ca:Sp.// PN in adversarial training (cf. Thm 5.4) and 1/ ILL & regularized training. Adversarial training with strong adversarial examples (e.g.. foo constrained ones) reduces ay (l) more drastically than E2 regularized training. For example, with = 100, N 1, 000, d 3 224 x 224, 6 = 0.1, = 1/2. oc, 10-2 and @(1((LN)) 10-5.
q xx, then p
Finally, we compare adversarial training and l2 regularization in terms of stability of gradient descent. The derivations of both losses with respect to the weight W E W are given by (cf. Lemma G.7);
dLadv OW
co:.p.guu co02p.uu//2-11
dh W
(A250)
LN This indicates that the derivation of é2 regularizer is smooth across the entire weight space, whereas the derivation of adversarial loss changes significantly at the w = 1 boundary. Note that w evolves during training (cf. the definition in Thm 4.1). The steep derivation in adversarial training prevents the gradient descent from reaching a minimum.
N
oW
Revisiting trainability condition in adversarial training
Here, we revisit Defn 5.5. Recall that this condition can be derived exclusively under Asm B.1. This is because ( can only be computed under Asm B.1 [81]. In this study, we assume that Asm B. applies the standard loss Land, but not to the adversarial loss Ladv- Consequently, the composite loss : Lad + Ladv does not satisfy Asm B.1, implying that strictly speaking, Defn 5.5 is not valid for L and considering Defn 5.5 in adversarial training might not be entirely accurate.
However, we must emphasize that Defn 5.5 continues to be a determining factor in the success of adversarial training, in other words, training with L. This is primarily due to the fact that if Defn 5.5 is not met, networks will be unable to adequately fit the training dataset owing to gradient vanishing or explosion.
Examining this from a mathematical standpoint, the necessity of Defn 5.5 underlined by the linearity of the differential operator. The update equation for the parameter ((t) is formulated as follows:
dC dCstd 0Ladv d0(t) = a0(t) a0(t) a0(t)
(A251)
dt
As demonstrated by the equation above, updates with Lord and Ladv are conducted independently. Although Defn 5.5 may not be considered for Ladv, it can be for Latd- In order to forestall gradient vanishing or explosion in the standard loss and to ensure classification ability for clean images, Defn 5.5 remains necessity. Therefore, it can be concluded that Defn 5.5 continues to dictate the success of training, even within the context of adversarial training.
K
Other theoretical results
Scaled effect of input and output dimensions. In Tab. 1, we showed a wide effect of the input dimension d and the output dimension K on the upper bounds the adversarial loss. In practice, the perturbation budget and the adversarial loss are scaled based on the chosen and For example, we can rearrange the definition of the adversarial loss (Eq. (3)) as follows:
Ladv (1 == max
fall + n7) (1,
(A252)
Inll,5App
42

---

Table A4: Values of 3
The description the same as Tab.
P.q'
=
q
==
00
vd
p 1
2d
In
Vd
p= = 2
+
K
to
p = 00
2d
where Ap and X denote the scaling factors. In this context, the upper bound (Thm 5.1) can be rearranged as follows:
Ladv (x S e. >p>a2p. w//2 E 2
(A253)
where S/.0 = Ap For example, we consider the following Ao and Agi
(p = 1) Ap = Vd (p = 2) pp co)
1/K
(q =
A
1/VK
(q 2)
(A254)
(q (q=03) 20)
Under these conditions, S corresponds to Tab. A4. This table reveals that (i) the input dimension impacts the upper bounds with O @(vi), but the output dimension has little no effects; (ii) although the dimensions generally exert similar influence on the bounds, the specific factors vary widely; (iii) for (p, q) = (1, oo) and (2, 2). the output dimension behaves a distinctive manner. In conclusion, our upper bounds, even inclusive of scaling effects, provide several insights into the relationship between the adversarial loss and input/output dimensions.
Other metries of capacity. Here, we consider capacity metrics other than the Fisher-Rao norm. Norm-based metries, such as the path norm [66], the group norm [67], and spectral norm [8], have similar definition to the Fisher-Rao norm. They are constructed by the sum and product of the weights of a network. The trajectory length [73] is defined by the are length of network output with change of parameters. A similar definition based on curvature was also used in [71]. These metrics concluded that capacity increases with the weight variance of network, despite differences in speed. Therefore, we can derive similar results as in Sec. 5.4 for metrics other than the Fisher-Rao norm.
Mitigation of adversarial risk. Here, we consider the mitigation of adversarial risk, i.e., the mitigation of the upper bounds of the adversarial loss. There are three solutions: (i) sample each entry of pia from ((0, 1/d²) instead of (0, 1/d); (ii) sample each entry of (()) from (0, /N22) instead of (O, /N/ for 1/N).
[L]; (iii) sample each entry of pout from (0 1/N2) instead of
First, let us examine scenario (i). In this setting, the variance of J(x"") is transformed into V[J(x.") )yp = / from V[J(z!)))) )g] = wh/a (cf. the proof of Thm 4.1). In addition, 330,0 changes to 20) = from @(B = vd (cf. the proof of Thm 5.1). These modifications suggest potential reduction adversarial risk in scenario (i).
However, this leads to a training failure. Let us consider the variance of a network output under no bias (for simplicity). For sufficiently large input dimension and in E [0, 1]d, it can be calculated as follows:
= V[J(x"") 10
=
(A255)
d2
This equation implies that the network always outputs a zero vector for any input in In other words, input information is lost during signal propagation in the network. In this situation, networks cannot learn the data structure well, and thus training does not proceed (cf. [71, 81, 105]).
Situations (2) and (3) are also similar to (1); they can mitigate the adversarial risk, but they break input information during forward and backward. Therefore, we conclude that it not feasible to mitigate the adversarial effect without compromising the network's 's trainability.
43

---

Extension to Lipschitz continuous activations. In this study, we primarily established our theo- rems for networks employing ReLU-like activations. Here, we attempt to generalize these theorems to encompass networks with Lipschitz continuous activations. However, this extension involve looser upper bounds or potentially unrealistic assumptions.
First, we examine the upper bounds for and 2. Since J( ) is a Jacobian at x Ineqq (8) remains valid for Lipschitz continuous activations. Denote the Lipschitz constant by 0. Similar to the discussion in Appx. H, we can derive the following upper bound:
K
|3a2"))12 S E-
1+
2 (o N
(A256)
An important observation here is that w// which dictates the training properties, is common to networks with both ReLU-like and Lipschitz continuous activations. Consequently, claim that our theorems also extend to Lipschitz continuous activations, despite differences in the factor. However, this bound is exponentially loose, casting doubt on its ability accurately represent the properties of adversarial loss.
Second, we use the analysis of network Jacobian [69]. For a comprehensive comparison between our approach and theirs, see Sec. 5.1. Here, we adopt their assumption where the variance V[h(9] is constant for any E [Z]. Drawing from the results expressed in Eq. (22) of [69], we can assert that [J(x')) ||, S (((o).))))) holds. Similar to the first proposition, this implies that our theorems hold to general activations including Lipschitz continuous activations. However, the assumption clearly does not hold in our theorems such as Thm 5.4.
Single-gradient descent attack can find adversarial examples. Here, we demonstrate that single-gradient descent attack, such as the fast gradient sign method [38], can find adversarial examples. This is not a substantially novel contribution. For example, see [7, 11, 24, 25, 61, 84, 112]. However, to introduce new approach based on Thm 4.1, we attempt it.
Proposition K.1 "Single-gradient descent attack find adversarial examples). Suppose that | Vd, the perturbation constraint e is sufficiently small. and the input dimension d is sufficiently large. Then, the single-gradien descent attack finds to constrained adversarial examples that flip the prediction of a single-output random deep neural nefwork with high probability
whid T(wk u ao? E w wh-1)
erf
d-000 1,
(A257)
where erf is the error function.
Proof. Since v is sufficiently small, the same linear regions in a ReI.U-like network encompass both
i and S + 77. i.c., J [xx")) J (x + 17). When an foo constrained adversarial example flips the prediction of classifier, the inequality [ (x( ) + a(x"") | < [ [xx "m) holds for |n < e. The perturbation generated by single-gradient descent defined as n == (og("(( Note that J (x.")) is a one mindimmnsional vector in single-output networks. Using this perturbation, by Eq. (A87)
and Thm 4.1, the right term of the inequality can be transformed into:
2. vd
2w
[( ml S ed
)al [a
edy
EV
(A258)
m
Then, let us consider iax ta (I.
By the reproductive property of the Gaussian, J(x )ar" (I follows Gaussian (0, | [[= + V(a(z="))). The variance can be rearranged as follows:
|| | || (2
I
in 1 ( V[a(="))
Σ wh wk-l= wh + oo Σ wk k- (A259)
oo
1=1
Note that the c. of the folded normal distribution based on the Gaussian NN 0²) is given by erf( 2202). Thus, compute the probability such that | [xxl") + al a(xl")) is less than
44

---

2v/2 d/ as follows:
ev 2w22td//
wid
erf
= erf
(A260)
V2(ww + aot Ek=r
a aof Σ w
d+
(A261)
L
Other experimental results
L.1 Setting
We focused on ReLU networks, i.e., u = == and a = 1/2.. Adversarial examples were generated using projected gradient descent [20] and the loss function defined in Eq. (3). Networks were initialized to satisfy the (M, -taiinabbby condition (Lemma 5.6), i.e., ae = 2 for vanilla networks and o 0.1 or o 0.01 for residual networks. The initial bias variance was set to of = 0.01. We employed MNIST [26] and Fashion-MNIST [99] as the training dataset. All experiments are conducted an NVIDIA A100.
L.2 Verification of Thm 4.1 (new mean field-based framework)
As shown Fig. 1, the empirical distribution of J [(x!") in the vanilla network aligned well with Thm 4.1. To further verify Thm 4.1, we provide additional results. We sampled 10, 000 vanilla networks with d 1, 000 and K 1. For vanilla networks, we set o = 2, and for residual networks, we set o 0.01. The network width N and depth L were set to 5, 000 and 10, respectively.
The empirical distributions of (wx and a(xi") for different network depths, L 10 and 100, and two randomly generated inputs, æ/ and zb are shown in Figs. A5 and A6. The accuracy of Thm 4. in predicting these distributions remains consistent across varying network depths. Moreover, the distributions of J(x"") and a( "( did not depend on the input x. even though they are defined as functions of in
We should note that our theory operates under the assumption of infinite network width. To assess the influence of network width Thm 4.1, please refer to Fig. A7. We found that for smaller values of N, for instance N = 10, the empirical distribution of (x and a(xi") deviates from the theoretical prediction as provided by Thm 4. As network width increases, the alignment between empirical results and predictions from Thm 4.1 improves.
It is important to clarify that while Thm 4.1 asserts that the distributions of J J(x")) and a(x")) do not depend on in it does not necessarily imply that random variables J "x=") and a (z)") are independent of J (z" and a(y ((y")) respectively, when 't & For For slightly different inputs (y
x x 0.999, a. x 0.99, and "h x 0.5), we computed the correlation coefficients between [xx a a(x!") and J() ")y"",, respectively.* These findings can be found in Figs. A8 and A9, and clearly demonstrate that more similar inputs result in higher correlation coefficients. However, this does not contravene the claims made in Thm 4.1. Further, for randomly generated inputs, a and a the random variables J ", a( a(x"") and J (x ), a (xxd)) were found to be relatively uncorrelated, respectively. The theoretical prediction for this correlation is currently unclear, which is topic for future work.
To validate the independence of different entries of J (x:") and a(xi"), we computed the correlation coefficient between two distinct entries.* 4 As presented in Fig. A10, we found that the two unique entries of J (2:1n) and ax="") were indeed uncorrelated, corroborating the claims in Thm 4.1. Fur- thermore, Fig. 11 illustrates that the entries J (x and a (x) were uncorrelated, which also aligned with Thm 4.1.
An examination the distributions of J (xx")) and a (x after training may yield interesting insights. However, as trained networks have different weight and bias variances, the observed J ( and
'While the correlation coefficient is not necessarily indicative of dependence, we regard it as conveniently measurable value. For more information, please refer to Remark E.3.
45

---

J(x"" 1.1. L=10
[(x 11... L=100
a(a"),, h. L-10
a(x² /1. L=100
0,50
5
5
0.25
Exp.
Theory
0.00
0.1
0.0
0.1
-0.1
0.0
0.0
0.5
-2
2
0.50
10
1
5
5
0.25
0
0
0
0.00
-0.1
0.0
0.1
-0.1
0.0
0.1
0.5
0.0
0.5
-2
0
Figure A5: Distributions of J (xx") " 1. and a(xx")) in the vanilla ReLU network with = 1, 000,
K 1, N 5, 000, o 2. and of 0.01. The description is the same as Fig.
J(x -=10
J(x 11. L=100
a(x"))) L-10
a a(x ), 1=100
10
0.4
Exp.
Theory
0
0.0
0.1
0.0
0.1
0.1
0.0
0.1
0.5
0.0
0.5
-2
10
0.4
5
0.2
0.1 0.0
-0.1
0.0
0.1
-0.5
0.0
0.5
-2
Figure A6: Distributions of J(x 1,1 and u(x!"); in the residual ReLU network with d = 1, 000,
K = 1, N = 5, 000, o² = 0.01, and of = 0.01. The description is the same as Fig. 1.
a(x in each network do not follow the same distribution. Consequently, empirically evaluating the validity of Thm 4.1 for trained networks presents substantial challenges. Despite this, consider, based on experimental findings such as those presented in Fig. A18, that Thm 4. offers an accurate representation of the early stages of the training process.
L.3 Verification Thm 5.1 (upper bounds of adversarial loss)
As shown in Fig. 2, the upper bounds in Thm 5.1 indicates the significant tightness in vanilla networks. To further verify Thm 5.1, we provide additional results. We generated 100 adversarial examples and calculated the adversarial losses defined in Eq. (3). For vanilla networks, we set o = 2, and for residual networks, we set oZ = 0.01. The network width was set to 40, 000 for vanilla networks and 35, 000 for residual networks, and the network depth L was set to 3. We set the perturbation constraint e to 0.1 and iterations of projected gradient descent to 50.
In Fig. 2, we illustrate the tightness of the bounds with varying input dimensions in vanilla networks. We provide further results for residual networks, as shown Fig. A11. Additionally, Figs. A12 and A13 demonstrate the adversarial loss as a function of varying output dimensions in vanilla and residual networks, respectively. Overall, our findings affirm the tightness our bounds across both network types. For some (p. g) combinations, empirically observed adversarial loss samples exceeded the theoretical upper bounds. We consider that this discrepancy can mitigated by expanding the network width, topic claborated upon in the following paragraph.
46

---

N=10
N=100
N=1,000
N=10,000
10
1001
Exp.
Theory
5
5
5
0
0
-0.1
0.0
0.1
-0.1
0.0
0.1
-0.1
0.0
0.1
-0.1
0.0
0.1
2.0
3
1.5
1.5
1.0
1.0
0.5
0.5
0.5
0
0.0
0.0
0.0
-0.5
0.0
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
00.
0.0
0.5
Figure A7: Distributions of (( 1.1 and a(xin)) 1 in the vanilla ReLU network with 1, 000,
K 1, L= 10, of 0.01, and of 0.01. The description is the same as Fig. 1.
The derivation of Thm 5.1 assumes a network of infinite width. However, in practical implementation, an infinite network width is unachievable, leading to instances where empirical results do not coincide with Thm 5. Specifically, certain samples were observed to exceed the theoretical upper bounds, as shown in Fig. To evaluate the influence of network width, we varied it while sampling the adversarial losses, with the results displayed in Fig. A14. In the case of (p, 4) (2, 2). it was observed that wider network width diminished the discrepancy between sampled losses and theoretical bounds. For other (p, q) pairings, even with a relatively small width (e.g., N 100), empirical results aligned with Thm 5.
The impact of the perturbation constraint e can be found in Fig. A15, It was confirmed that larger e values corresponded greater divergence between empirically sampled adversarial losses and theoretical bounds. This is because for larger it was harder for the projected gradient descent to tune adversarial examples well.
The effect of the network depth L can be confirmed Fig. A16. For (p.g) = (2, 2). as the number of layers increased, the value exceeded the upper bounds. In contrast, for (p. (00,o0)
the value fell below the bounds. These differences can be attributed to the varying complexities of optimizing the adversarial loss for each combination of p and q. That is, for (p,g) (2, 2). the optimization of adversarial examples is relatively straightforward, enabling the generation of adversarial losses that exceed the upper bound, which is imposed by the constraints of finite width. However, for (p, q) = (oo, oc), the optimization is challenging, and it is difficult to generate high- quality adversarial examples. The underlying reasons for these disparities optimization complexity remain a topic for future work.
Furthermore, assessed adversarial loss during training the MNIST dataset, shown in Fig. A17. Vanilla networks were trained using stochastic gradient descent with learning rate of 0.001. While Thm 5.1 broadly holds for (p, q) (1, 2), the disparity between theoretical prediction and empirical results expands as training progresses for (p, q) (2, 2). (2, 20), and (p,4) (oc, oo). Determining theoretical prediction that provides tight bound on the adversarial loss for fully trained deep neural networks remains an open challenge for future research.
L.4 Verification of Thms 5.4 and G.9 (time evolution of weight variance)
As shown in Fig. 3, the empirical weight variance observed in the vanilla network aligns well with Thm 5.4. To verify Thm G.9, we provide Fig. A18. We trained 10-layers vanilla and residual networks. The vanilla networks were initialized with o (0) 2 and o0 0.01. Residual networks are initialized with og (0) 0.1 and (0) 0.01. For standard training, N was set to 1, 000. For
47

---

Correlation coefficient: 0.99
Correlation coefficient: 0.91
0.15
0.15
J(a x (..99)
Jiw x 1.00011
0.00
0.00
Experiment Theory
0.15
0015
0.155
0.00
0.15
-0.15
0.00
0.15
J(x:") h.
J(xe") h11
Correlation coefficient: 0.5
Correlation coefficient: 0.03
0.15
0.15
J( X (0.5).1
J(
0.00
0.00
-0.15
-0.15
00.15
0.00
0.15
0.15
0.00
0.13
J(ae") 11
J(x]))
Figure A8: Correlation coefficient of J(.)..1 for two inputs in the vanilla ReLU network with 1, 000, K = 1, 5, 000, L 10, o 2, and of = 0.01. The description of the histogram is the same as Fig. 1.
adversarial training, we designated p= co, == 0o, and 0.3. In both training, used stochastic gradient descent with small learning rate, 0.001. Note that gradient flow assumes an infinitesimal learning rate (cf. Eq. (10)). A theoretically defined training step under gradient flow is not equal to a training step in the experiment. We have linked them by t = training steps X learning rate. In practice, setting the weight variance precisely is not feasible. For example, in vanilla network. o² (0) might be set 1.999 even though we tried to initialize to satisfy o² (0) = Thus, for visibility, the experimental results (curves) were shifted parallel to meet o² (0) = 2 in vanilla networks and o (0) = 0.1 in residual networks.
From Fig. A18, it is evident that adversarial training facilitates weight regularization for both vanilla and residual networks. Compared to the la weight regularization discussed Appx. I, it offers stronger regularization. We can verify that Thm 5.1 can predict these empirical behaviors in the initial stage of training. The validity of Thms 5.4 and G.9, which underpin our theorems such as Thms 5.7 and 5.9, lends credence to our theoretical assertions.
48

---

Correlation coefficient: 0.99
Correlation coefficient: 0.9
01341 X 199991 11
1,990 x 1,230
Experiment Theory
1
1
a(=T") 1
a(æ1
Correlation coefficient: 0.53
Correlation coefficient: 0.42
deait x 0.511
al 1la 1
1
1
a aa=:"))
a(x)")
Figure A9: Correlation coefficient of a(-)1 for two inputs in the vanilla ReLU network with d 1, 000, = N= 5, 000, I, = 10, o? 2, and of 0.01. The description of the histogram is the same as Fig. 1.
L.5 Verification of Thms 5.7 and 5.8 (adversarial trainability)
As shown in Fig. 4, vanilla networks with small width and large depth were not adversarially trainable, which aligned well with Thm 5.7. To verify Thms 5.7 and 5.8, we provide Fig. A19. We trained vanilla and residual networks under various width and depth settings, and observed training accuracy. For fast training convergence, we used Adam [51]. The training was stopped if training accuracy was not improved the last 200 steps. We set the learning rates to 0.001 or 0,0001, adopting the highest training accuracy at the final training step. The vanilla networks were initialized with o (0) 2 and (0) = 0.01. The residual networks were initialized with o (0) 0.01 and (0) = 0.01. This initialization satisfied the (M, m) traiinabbbyl conditions (Lemma 5.6). In adversarial training, we set
0 20, = 0.1, and the iterations projected gradient descent to 10.
From Fig. A19, we can confirm that vanilla networks with large depth and small width were not adversarially trainable and the training difficulty was unique to adversarial training of such vanilla networks. Note that Prop G. could not be verified as the condition in which training persists without machine errors and the trainability condition not being satisfied at is challenging to implement in practical scenarios.
49

---

Correlation confiicint: -0.01
Correlation coefficient: 0.0
Correlation coefficient: 0.0
Experiment
-0.15
Theory
0
J(e haa
Figure A10: Correlation coefficient of two different entries of ((x the vanilla ReLU network with d = 1. 000, K = N 5, 000, I. = 10, oh = 2, and og 0.01. The description of the
histogram is the same as Fig. 1.
1.9
p 1.4
p
1,9
0c
0.75
0.10
0.03
T Theory Exp.
Adversarial loss Calv
0.50
0.02
0.05
0.25
0.01
250
500
750
1000
250
500
750
1000
250
500
750
1000
g
/
OC. y
0.20
0.2
2
0.15
1
0.0
250
500
750
1000
250
500
750
1000
250
500
750
1000
Input dimension d
Figure A11: Adversarial loss (Eq. (3)) in residual networks with N 35, 000, K 100, 0 0.01, of 0.01, and c = 0.1. The description is the same as Fig. 2.
L.6 Verification Thms 5.9 and G.14 (capacity degradation)
To verify the degradation of network capacity during adversarial training, we trained vanilla and residual networks MNIST with p co, = co, and = 0.1, and observed the Fisher-Rao norm as a measure of capacity. The training steps were set to 500. Thms 5.9 and G. are derived employed small learning rate, 0.0001. Vanilla networks were initialized with o (0) = 2 and of (0) = 0.01. Residual networks were initialized with o (0) 0.01 and (0) 0.01. During the derivation of Thms 5.9 and G.14, using Asm B.1, we calculated E| Jhee Owy w,wy] = E Dws, dee
under sufficiently small learning rate (cf. Eq. (10)),
0f af [[ = 0, for + j. Nevertheless, in practice, this calculation does not always hold. Therefore, for the computation of the Fisher- Rao norm, we employed diagonal matrix of F instead of F itself.
As shown in Figs. A20 and A21, large network depths (large L) produced high Fisher-Rao norms but degraded them drastically. Moreover, was observed that wider network width could maintain the Fisher-Rao norm at its initial state. The discrepancy between these experimental values and the values predicted by Thms 5.9 and G.14 considered originate from Asm B.I, which does not hold in the case where a diagonal matrix of F is used instead of F itself.
Additionally, assessed the influence of network width the robust test accuracy of fully-trained networks. The robust test accuracy was determined using AutooAttttcck with c = 0.1. The adversarial training was conducted under p do, q xc, 0.1, and 10 iterations. We employed Adam with a learning rate of 0.001 and epochs set to 200.
From Tabs. A5 and A6, it is evident that the robust test accuracy depends network width rather than depth. The same superscripts in the tables indicate results from networks with an identical number of parameters. For example, the accuracy for (N, L) (250, significantly surpasses that
50

---

= 1,9
p 1
0.04
1,
00
0.3
5.0
0.03
T heory
Adversarial loss Cads
2.5
Exp.
250
500
750
1000
250
500
750
1000
250
500
750
1000
= 2.
2
p
2,
00
00, 9
00
0.4
0.2
0.9
0.8
0.0
0.7
250
500
750
1000
750
1000
250
500
750
1000
Output dimension K
Figure A12: Adversarial loss (Eg, (3)) in vanilla networks with d = 100, N = 000, L = 3.
0u 2, o
0.01, and e = 0.1. The description is the same as Fig.
p p=
p 1.9=2
p 1, 00
0.04
7.5
0.3
Adversarial loss Cute
5.0
0.2
0.03
T heory
2.5
Exp.
0.1
0.02
250
500
750
1000
250
500
750
1000
250
500
750
1000
2, g
2
p
2.
4
p 0C, 9
0.4
0.2
0.8
0.2
0.0
0.7
250
500
750
1000
250
500
750
1000
250
500
750
1000
Output dimension K
Figure A13: Adversarial loss (Eq. (3)) in residual networks with d 100, N 35, 000, - 0.01, of 0.01, and = 0.1. The description is the same as Fig. 2.
for (N, L) = (125, 20). Although Thms 5.9 and 14 is primarily applicable to the early stages of training, these theorems, emphasizing the importance of network width, hold true for fully- trained networks.
51

---

p 1
p
1
p= 1,9
1.0
0.15
0.025
Adversarial loss Cuiv
0.10
Theory
Exp.
0.000
0.6
0.05
0
10000
20000
30000
40000
10000
20000
30000
40000
10000
20000
30000
40000
p
2,
=
0.15
p
2,
0.9
OC,
0.225
0.10
0.8
0.200
0.05
0.7
10000 20000 30000 40000
0
10000 20000
30000 40000
10000)
20000)
30000
40000
Width
Figure A14: Adversarial loss (Eq. (3)) in vanilla networks with d 500, K 100, 3, o of = 0.01,
2, = 2, and = 0.1. The description is similar to Fig. 2.
p= ==11==1
p
=1,0=2
p= 1,9
0.3
0.075
Theary
0.2
Adversarial loss Catv
0.050
Exp.
0.1
0.025
0.02
0.10
0.20
0.30
0.02
0.10
0.20
0.30
0.02
0.10
0.20
0.30
2, q
p 2,
00,
0.50
0.3
2
0.2
0.25
1
0.1
0.02
0.10
0.20
0.30
0.02
0.10
0.20
0.30
0.02
0.10
0.20
0.30
Epsilon
Figure A15: Adversarial loss (Eq. (3)) in vanilla networks with 500, N 40, 000, K 100,
L = 3, oi = 2, of = 0.011, 2, and q = 2. The description is similar to Fig. 2.
p= -=1-==
p == 11122
p=
0.15
0.025
Adversarial loss Cuds
0.10
Theory Exp.
0.6
0.000
0.05
3 20
40
60
80
100
3 10
20
40
60
80
100
3 10 20
40
60
80
100
2, g
2
0.15
p
2,
P
0.75
0.25
0.10
0.50
0.20
0.05
3
20
40
60
80
100
3 10 20
40
60
80
100
310 20
40
60
80
100
Depth L
Figure A16: Adversarial loss (Eq. (3)) in vanilla networks with 500, 40, 000. K 100.
02 2, of 0.01, 2, 2, and = 0.1. The description is similar to Fig. 2.
Table A5: Test accuracy (%) on MNIST [26] under AtttAot with the perturbation constraint
0.1. We used residual networks with o 0.01 and of 0.01. For adversarial attack in training, we used p= XC, g oc, c = 0.1, and 10 iterations. We set an optimizer to Adam, a learning rate to 0.001, and epochs 200. Values marked with the superscript denote results from networks with the same number of parameters L
L ==
L
10
L
15
20
125
13.8
13.1
9.79
8.69
N 250
45.8
49.8
47.0
48.1t
N = 500
79.2t
76.5
75.3
73.49
1,000
89.80
88.4
87.3
87.9
52

---

1.9
p=1.==2
p 1,4 114=00
0.04
0.015
0.01
Theory Experiment
Adversarial loss Cudv
0.010
0.00
0.02
1000
500
1000
= 2.9.9
= 2, 80
20, 9
0.12
0.11
0.11
0.10
2.2
0
500
1000
500
1000
500
1000
Training step
Figure A17: Adversarial loss (Eq. (3)) calculated with = 0.1. We used vanilla networks with 28 28, N 10, 000, K = 10. 3, a (0) and of 0.01. We trained the vanilla
networks normally (not adversarially) with the learning rate 0.001. The description is similar to Fig.
Vanilla
Residual
Weight variance d'
2.00000
0.10000
1.9995
1.9890
0.09996
1.9985
1000
1500
2000
500
1000
1500
2000
Training step
Standard
Adversarial (N 2k)
Theory 2k)
6 weight
Adversarial (N - I8)
Theory (N 18)
Figure A18: Time evolution of the weight variance in the vanilla and residual network with L = 10 during adversarial training with = oc, 0O, and = 0.3. In standard and adversarial training, set the learning rate to 0.0001. In standard training with without & weight regularization, we set
000. In adversarial training, we set p = oo, q = 0o, and e = 0.3. The vanilla network was initialized with 2 (0) 2 and (0 0.01. The residual network was initialized with o (0) 0.1 and (0) 0.01. The orange dashed lines are predicted by Thms 5.4 and G.9. To derive the theoretical predictions, we calculated
training steps x learning rate, For visibility, the experimental results were shifted parallel to satisfy o (0) 2 on the vanilla networks and o (0) = 0.1 on the residual networks.
Standard
Adversarial
10
40
1.0
T = 100
Vanilla
20-
20
-.88
=: 250
Training accuracy
0.6
200
300
400
500
600
700
900
100
200
300
500
600
700
800
40
-04
Residual
20
20
0.0
300
100
500
700
800
100
200
300
600
700
800
Figure A19: Heat map of the training accuracy on the vanilla and residual networks trained on MNIST. The description is the same Fig. 4. The vanilla network was initialized with o² (0) and of (0) 0.01. The residual network was initialized with of (0) = 0.1 and (0) 0.01. The horizontal axis represents the width of the network and the vertical axis represents the depth.
53

---

Vanilla
Residual
60
Fisher-Rao norm ||x]||Fs
1.5
200
0.5
100
300
400
100
100
Training
20,
20,
Figure A20: Fisher-Rao norm vanilla and residual networks adversarially trained on MNIST. We set p 3C, q
0.1, and the learning rate 0.0001. The vanilla network was initialized with z (0) = 2 and of = 0.01. The residual network was initialized with a (0) = 0.1 and (0) = 0.01.
Vanilla
Residual
Fisher-Rao norm ( FR:
0.00
-0.05
0.30
Training
20,
Figure A21: Fig. A20 with the origin shifted parallel to zero.
Table A6: Test accuracy (%) on Fashion-MNIST [99] under for -AutoAttack. The description the same as Tab. A5.
I.
L 10
L
15
20
N 125
7.78
9.22
10.4
11.0*
N 250
33.1
33.6
32.6
34.0t
500
51.71
51.7
51.6
50.8
N 1,000
67.90
67.3
66.7
66.8
54


Reviews:
[{"review_id": "JN0oDu2Myi", "reviewer": "Reviewer_e8jY", "summary": "The theoretical understanding of adversarial training is an important and valuable topic. This work proposes a new theoretical framework for this based on mean field theory. With the proposed framework, the authors analyze the properties of adversarial training from multiple aspects, including the upper bounds of adversarial loss, the time evolution of weight variance, the adversarial trainable conditions, and the degradation of network capacity. These results could be helpful for the understanding of adversarial training and inspire more efforts on this topic. ", "questions": "I am not an expert on mean field theory and didn't check all the proofs. I give my rating by considering that 1) the theoretical understanding of adversarial training is important and the progress on this is very helpful for the community and 2) the proposed framework seems to be generic and may inspire more work in the future. ", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. It proposes a new framework to analyze adversarial training theoretically based on mean field theory. \n2. Based on the proposed framework, it presents several theoretical results for adversarial training.\n3. The proposed framework and the presented theoretical results are non-trivial and helpful for the understanding of adversarial training.", "weaknesses": "It studies several different adversarial training characteristics in the main paper. Is there any correlation between these different characteristics？ Why do we choose these aspects for analysis? Further, is it possible to provide a global diagram to better see which properties can be analyzed and which cannot be analyzed at present based on the proposed framework?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank your insightful comments.\n\n> It studies several different adversarial training characteristics in the main paper. Is there any correlation between these different characteristics？ Why do we choose these aspects for analysis?\n\nOur theoretical results on adversarial training can be divided into two categories: those related to training dynamics (Theorems 5.6 and 5.7) and those related to test time performance (Theorem 5.8). Both are fundamental aspects of machine learning. More specifically, Theorems 5.6 and 5.7 answer why vanilla networks cannot achieve high training accuracy in some situations, while residual networks always can. Theorem 5.8 addresses the degradation of network capacity, which is tied to the generalization performance of networks, i.e., test accuracy. Similar studies have been conducted on standard training [1,2]. However, the untrainability of vanilla networks and the loss of capacity are unique to adversarial training. Thus, our choice of analysis allows us to highlight the significant differences between adversarial and standard training.\n\nTheorems 5.1 and 5.3 present the upper bounds of the adversarial loss and the time evolution of weight variance, respectively. While they were originally formulated to derive Theorems 5.6-5.8, they are also interesting results in their own right.\n\n[1] Deep information propagation, ICLR17.  \n[2] Universal statistics of Fisher information in deep neural networks: Mean field approach, AISTATS19.\n\n> Further, is it possible to provide a global diagram to better see which properties can be analyzed and which cannot be analyzed at present based on the proposed framework?\n\nThe proposed framework can analyze the following properties of adversarial training: (i) the upper bounds of the adversarial loss, (ii) time evolution of weight variance, (iii) adversarial trainability, and (iv) degradation of network capacity. Moreover, we believe that our approach can be applied to other deep learning methods besides adversarial training (cf. Section 4.2).\n\nA major limitation of our analysis, which is common to all mean field-based analyses, is its theoretical restriction to the early stages of training, making it not directly applicable to full training process. Nevertheless, mean field-based analyses offer superior flexibility compared to other approaches; they can handle multi-layer perceptrons (including shortcuts) with nonlinear activations without imposing any assumptions on the data distribution. We have discussed this in Section 7, but will strive to make it clearer in the updated version.\n\nNote that it has been observed empirically that the theoretical results from the early stages of training align well with fully trained networks [3,4]. In this study, for example, our theoretical prediction in Theorems 5.8 and F.16 - network width strongly influences generalization performance in adversarial training - holds even for fully trained networks (cf. Tables A7 and A8). Therefore, even though our framework is not strictly applicable to fully trained networks, we believe that it can still provide important insights for fully trained networks.\n\n[3] First-order adversarial vulnerability of neural networks and input dimension, ICML19.  \n[4] Adversarial robustness guarantees for random deep neural, ICML21."}, {"review_id": "nTSxGLTc3p", "reviewer": "Reviewer_55dm", "summary": "The paper provides a mean field analysis on relu networks for adversarial training. The main insight is that networks without residual connections are not most likely inevitably suffer from gradient explosion or vanishing and thus are not adversarially trainable, unlike vanilla network training.", "questions": "Could the authors elaborate a bit on why residual connections could make a bigger difference in adversarial training than vanilla training?", "limitations": "Limitations are well discussed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Analyzing adversarial training performance is an important problem both theoretically.\n\nThe insight from the analysis that adversarial training vanilla relu networks is more likely to suffer from gradient explosion/vanishing is an interesting insight.", "weaknesses": "The verification of the theorems might need more effort. E.g., Figure 4 is showing accuracy of vanilla network, it would be helpful to also show the curves for residual networks.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to appreciate your fruitful suggestion and question.\n\n> The verification of the theorems might need more effort. E.g., Figure 4 is showing accuracy of vanilla network, it would be helpful to also show the curves for residual networks.\n\nThe training accuracy of adversarial training in residual networks is presented in Figure A19. Due to the complexity of the heatmap design, we believe that combining them into a single image would be challenging. In the updated version, we plan to clarify in the caption of Figure 4 that the results for residual networks can be found in Figure A19.\n\n> Could the authors elaborate a bit on why residual connections could make a bigger difference in adversarial training than vanilla training?\n\nThis is because adversarial training has a strong weight regularization effect (cf. Theorem 5.3), while vanilla training (standard training) does not. As adversarial training progresses, the network weights become smaller. Therefore, without residual connections, the small weights lead to gradient vanishing as the input signals diminish during the forward pass. Note that network weights also decay during standard training with L2 regularization. However, adversarial training causes the weights to decay at a significantly faster rate than L2 regularization (cf. Section H in Appendix)."}, {"review_id": "Sx7r6V97Ff", "reviewer": "Reviewer_CfQ9", "summary": "The authors propose a mean field based framework for theoretically analyzing the training dynamics of adversarial training for MLP and residual networks with ReLU non-linearities. Based on this framework, the authors provide tight bounds on the adversarial loss (squared loss between clean and adversarial example output) and investigate trainability of MLP and residual networks.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- While I am not very familiar with related work on mean field theory for (adversarially) training networks, the proposed framework seems to address limitations of prior work quite elegantly.\n- The paper includes quite a number of interesting results using the proposed framework (loss bounds, training dynamics, trainability, impact of width).\n- Formulation includes MLP and residual network with a class of “ReLU-like” non-linearities; is also includes adversarial training with clean loss (as e.g. done by TRADES).\n- I feel the framework can be quite insightful and helpful for future work in understanding or improving adversarial training.\n- Many claims are empirically supported on MNIST.", "weaknesses": "My main concern about the paper is that it is incredibly dense (due to the number of included claims) and its structure does not make checking the proofs and claims easy. Even though I invested significantly more time in this review compared to other reviews, I was unable to fully follow all derivations and proofs. I believe this is mainly due to the extremely convoluted way of presenting the theoretical results across main paper and appendix. Unfortunately, I feel that this also makes me less enthusiastic about the results. Here are some more detailed comments and suggestions:\n- Starting with 4.2, the reader is somewhat forced to jump to the appendix at least once just to see that the authors simply reformulate the ReLU using its derivative and D for J and a. This trend continues throughout the paper and appendix. Especially in the appendix, one is forced to jump around a lot just to follow 2-3 sentences.\n- The proofs are structured starting with simple lemmata and building up to the actual theorems. This is generally fine, but again, the referencing is overdone. For Thm 4.1 in Appendix E, there are 9 pages of derivation with >20 individual lemmata. So if I want to follow the proof of Thm 4.1, I am forced to go through many but not all of these lemmeta. Many have proofs of only 1-2 lines, but reference 2+ other lemmata or remarks and I need to remember the numbers or jump back and forth 2+ times just to read a single sentence. I feel the root cause for this is that many of the results are over-generalized and compartmentalized too much. I appreciate the thorough job of the authors in establishing many of the independence results, but as a reviewer and reader my #1 interest is following Thm 4.1, nothing more and nothing less. Everything that complicates this job is – in my opinion – bad for the paper. For me, the ideal solution would be a separate, easier to follow section for the proof of Thm 4.1 - even if it restates many of the lemmata and remarks, and moving the other results to a separate section for the (very very) interested reader.\n- The main paper includes so many results that there is basically no discussion of each individual result. Often it feels that every sentence refers to some additional result in the appendix. I think for me, and many readers, actually discussing the results informally, in words, and taking more time and space to introduce the required notation would be more beneficial than including the current amount of results. I would prefer to have fewer results well-described and the remaining ones being in the appendix.\n- Empirical results are discussed twice – after the corresponding theorems as well as in Section 6 – the space of the latter could be used to address one of the points above.\n\nComments and questions unrelated to structure and writing:\n- In the introduction, contribution (d) is unclear to me – what theoretical result does it refer to, the trainability?\n- The use of “probabilistic properties” is a bit unclear until the discussion of training dynamics. It would be helpful if the meaning would be detailed earlier in the paper.\n- In 3.1, why is having $P^{in}$ and $P^{out}$ important, i.e., why do we need these fixed layers?\n- Usually, the adversarial loss is also cross-entropy or something similar. While I saw papers arguing for a squared loss, TRADES does not use it AFAIK. Instead, the common setting is adversarial cross-entropy loss only, combined with clean cross-entropy loss or clean cross-entropy + KL as in TRADES. This makes me ask how assuming an adversarial cross-entropy loss would impact the results? Can similar results be derived?\n- In 4, the assumption of independence is also unclear. I feel making this more explicit, e.g., by informally providing a short result on independence from the appendix, could be useful.\n- The authors also highlight broader applicability; I am wondering if the authors derived similar results for standard training as reference? Or has this been done in previous work with other frameworks? This also related to the statement in l280.\n- In 5.3 l261, I can’t follow how transfer attacks are relevant here? Transfer attacks should be weaker than the general attack modeled in the paper …\n\nConclusion:\nI think that the paper has many interesting contributions and will be valued by the NeurIPS community. However, as I was not able to follow all derivations in a reasonable time, I will closely follow what the other reviewers have to say about the theoretical results. Also, I believe that the paper in its current form would have better fitted a long-format journal. For NeurIPS, I hope the authors invest some time in simplifying the main paper and restructuring the appendix. I think the current format will limit the audience of the paper to those very familiar with related work or willing to invest hours jumping back and forth between main paper and appendix. Some restructuring could really make this paper more accessible to the broad audience at NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Structure and writing\nWe extend our deepest appreciation to you for the considerable time and effort you have devoted to reviewing our paper. Taking the reviewer’s comments into account, we plan to improve the manuscript as follows.\n\n**Simplified version of Theorem 4.1.**\n\nWe plan to present a simplified claim and its proof of Theorem 4.1 for one-hidden-layer vanilla networks. If readers can understand this, we believe that they can more easily follow the claim for multi-layer networks (i.e., Theorem 4.1). This is because its proof relies on mathematical induction with respect to the layers.\n\nIn addition, one factor that complicates the proof arises from a well-known but sometimes overlooked fact: uncorrelated Gaussian variables are not necessarily independent. We have prepared many lemmas to discuss this rigorously in our context. Considering the reviewer’s feedback, as a stepping stone, we decided to prove Theorem 4.1 with an incorrect claim (i.e., uncorrelated Gaussian variables are independent) as well as the formal proof.\n\nNote that Proposition A.1 was provided to highlight the essence of Theorem 4.1, i.e., the independence of the distributions of $J$ and $a$ from the input $x$. However, we acknowledge that this has not been sufficiently emphasized.\n\n**Separation of proofs for vanilla and residual networks.**\n\nCurrently, some of our lemmas apply only to residual networks, which may not be relevant to readers solely interested in the results for vanilla networks. We intend to separate the formal proofs for vanilla and residual networks. \n\n**Generality and over-compartmentalization of lemmas.**\n\nWe plan to reconsider the generality of all claims to improve readability. However, maintaining a rigorous discussion while doing so is not easy. Therefore, we will initially focus on the previously mentioned step-by-step presentations. We acknowledge the over-compartmentalization and try to group lemmas with similar claims (e.g., Lemma E20--23).\n\n**Simplification of main text.**\n\nAlthough we have struggled with this, we believe that all claims in the main text are essential. We are not considering removing an entire theorem. Instead, we intend to increase intuitive explanations and informal presentations to improve readability. Following the reviewer’s suggestion, we also plan to remove some unnecessary references to experimental results to create space.\n\n## Contents\n> In the introduction, contribution (d) is unclear to me – what theoretical result does it refer to, trainability?\n\nThe content in \"Other contributions\" can be found in Section J. Contribution (d) corresponds to the paragraph \"Mitigation of adversarial risk\" in Section J. We will explicitly mention this in the updated version.\n\n> The use of \"probabilistic properties\" is a bit unclear until the discussion of training dynamics.\n\nIn the updated version, we will improve the clarity of this part as follows (e.g., l30):\n\nBefore: existing mean field-based approaches cannot manage the probabilistic properties of an entire network ...\n\nAfter: existing mean field-based approaches cannot manage the probabilistic properties of an entire network (e.g., the distribution of a network Jacobian) ...\n\n> Why do we need $P^{in}$ and $P^{out}$?\n\nThey are introduced to discuss networks with general input and output sizes. They also slightly simplify the calculations as it allows us to focus on trainable layers of fixed size $N\\times N$. For example, without these layers, $\\chi^{(0)}/\\chi^{(L)}=\\omega^L$ would become $\\chi^{(0)}/\\chi^{(L)}=N\\omega^L/d$. Note that they do not harm training.\n\n> Can similar results be derived for adversarial cross-entropy loss?\n\nBased on the reviewer’s suggestion, we have considered the case with cross-entropy loss, and found that similar results can be obtained. We appreciate the comment and share some of the results. Suppose that the class label of $x$ is one and that the network has no biases. The adversarial loss is defined as:\n$$\\mathcal{L}(x):=\\max\\_{\\|\\eta\\|\\_\\infty\\leq\\epsilon}\\left(-\\ln\\frac{\\exp(J(x+\\eta)\\_{1\\cdot}(x+\\eta))}{\\sum^K\\_{k=1}\\exp(J(x+\\eta)\\_{k\\cdot}(x+\\eta))}\\right).$$\nFor simplicity, we assume that (I) $x=0$. (II) $\\epsilon$ is sufficiently small, and thus $J(x+\\eta)=J(x)$ (cf. l931). (III) the input dimension is sufficiently large. Then,\n$$\\mathcal{L}(0)\\leq\\ln\\left(\\sum^K\\_{k=1}\\exp\\left(\\max\\_{\\|\\eta\\|\\_\\infty\\leq\\epsilon}J(0)\\_{k\\cdot}\\eta\\right)\\right)+\\max\\_{\\|\\eta\\|\\_\\infty\\leq\\epsilon}J(0)\\_{1\\cdot}\\eta.$$\nFor any $k\\in\\\\{1,\\ldots,K\\\\}$ (cf. (A105)),\n$$\\max\\_{\\|\\eta\\|\\_\\infty\\leq\\epsilon}J(0)\\_{k\\cdot}\\eta=\\mathcal{O}(\\omega^{L/2}).$$\nThus,\n$$\\mathcal{L}(0)\\leq\\mathcal{O}(\\omega^{L/2}).$$\nThis is consistent with Theorem 5.1, which also gives $\\mathcal{O}(\\omega^{L/2})$. Thus, we consider that a similar discussion can be applied to Theorem 5.3, as well as those based on it (i.e., Theorems 5.6-5.8).\n\n> In 4, the assumption of independence is unclear.\n\nIn this paper, \"independence\" was used in the probabilistic sense: two random variables are called independent if their joint probability equals the product of their probabilities.\n\n> The authors also highlight broader applicability; the authors derived similar results for standard training? Or has this been done in previous work with other frameworks?\n\nWe did not apply the proposed framework to standard training because this has been done in previous studies using other mean field-based approaches. However, previous approaches are not applicable to adversarial training, which is why we developed a new framework.\n\nBy \"broader applicability\" we mean that the utility of Theorem 4.1 is not limited to adversarial training. Theorem 4.1 provides a simple view of a ReLU-like network that should be useful for analyzing other training methods (e.g., contrastive learning).\n\n> In l261, how transfer attacks are relevant?\n\nA similar question was raised by Reviewer QtQE. Please kindly refer to our reply to Revewer QtQE (due to character limitation)."}, {"review_id": "m9tuaMViSp", "reviewer": "Reviewer_QtQE", "summary": "The authors proposed a new theoretical framework based on the mean field theory to analyse adversarial training from several perspectives, such as the upper bounds of adversarial loss, the time evolution of weight variance and adversarially trainable conditions. Besides the theoretical analysis, the authors conducted several experiments verifying the proposed theoretical framework. \nGenerally speaking, the proposed theoretical framework provides a new perspective to analyse adversarial training and is highly versatile and even can extend to other training methods.\n", "questions": "See Weakness above.\n\n", "limitations": "The authors have clearly discussed the limitations of the proposed framework, such as, ‘’some theorems begin to diverge from the actual behaviour‘’ and ’’ the mean field theory assumes infinite network width, which is practically infeasible’’.\nFrom my point of view, this article does not involve any potential negative social impact. \n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is organised well and easy to follow. \nThe proposed theoretical framework seems inspiring and intriguing and gives a new perspective to analyse adversarial training from several aspects, which may serve as a good guidance for future work. Besides the proposed theory, verification experiments were also conducted to prove its effectiveness further.\n\n", "weaknesses": "1.\tThe verification experiments were only conducted on the easy dataset (MNIST); it may strengthen the findings if additional experiments are conducted on more challenging datasets.\n2.\t Including a more systematic evaluation of the results may be beneficial, such as adversarial loss in residual networks vs. training steps for normally or adversarially training. \n3.\tSeveral conclusions match the previous works; it could be more convincing if the reference could be given in the main content, such as ‘’the square sum of the weights in Ineq. (9) suggests that adversarial training exhibits a weight regularisation effect,’’ is consistent with [1], ‘’to achieve high capacity in adversarial training, it is necessary to increase not only the number of layers L but also the width N to keep L^2/N constant’’ is somehow consistent with [2].\n4.\tThe authors mentioned in Line 260 that ‘’This result suggests that residual networks are better suited for adversarial training. However, one of the previous studies indicated that residual networks are more vulnerable to transfer attacks [54] than vanilla networks. ’’ However, the proposed theoretical framework did not explain such a transfer attack phenomenon. Could the authors explain or give more comments about this?\n5. in Equation 3 and 4, it seems that the minimize part is not shown, perhaps it would be more reasonable to change the form of min-max in equation 3 and 4.\n\n[1] A unified gradient regularization family for adversarial examples, in: IEEE International Conference on Data Mining (ICDM), 2015.\n[2] Do wider neural networks really help adversarial robustness? Advances in Neural Information Processing Systems, 34.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank your careful reading.\n\n> The verification experiments were only conducted on the easy dataset (MNIST); it may strengthen the findings if additional experiments are conducted on more challenging datasets.\n\nWe have only tested our theory on simple datasets. This is because fully connected networks struggle to achieve high training accuracy and generalization performance on more complex datasets, such as CIFAR-10 and ImageNet, even in standard training. This complicates a fair comparison between standard and adversarial training. To perform both theoretical and experimental analysis of adversarial training on complex datasets, our theoretical results would need to be extended to more practical architectures, such as convolutional networks. Note that Table A8 shows results for Fashion-MNIST that are consistent with our theoretical findings. We also highlight that using MNIST for verification is a common practice in several theoretical approaches to adversarial defense, such as certified adversarial defense [3,4].\n\n[3] Certified defenses against adversarial examples, ICLR18.  \n[4] Semidefinite relaxations for certifying robustness to adversarial examples, NeurIPS18.  \n\n> Including a more systematic evaluation of the results may be beneficial, such as adversarial loss in residual networks vs. training steps for normally or adversarially training. \n\nOur extensive experiments cover each of our theoretical claims for vanilla and residual networks. Some results were sent to Appendix due to the page limitation. However, thanks to the reviewer’s comments, we found that some results for residual networks were not included, such as how the adversarial loss in residual networks changes during adversarial training. We will include these results in the updated version, along with more visually clear illustrations and comparisons.\n\n> Several conclusions match the previous works; it could be more convincing if the reference could be given in the main content, such as \"the square sum of the weights in Ineq. (9) suggests that adversarial training exhibits a weight regularisation effect,\" is consistent with [1], \"to achieve high capacity in adversarial training, it is necessary to increase not only the number of layers $L$ but also the width $N$ to keep $L^2/N$ constant\" is somehow consistent with [2].\n\nWe appreciate the reviewer’s introduction to related work and will include comparisons with these studies in the main text.\n\n[1] suggested that adversarial training with the fast gradient sign method regularizes network weights around training instances, which aligns with Theorems 5.3 and F.9. However, their research relies on a strong assumption about the network's Jacobian, which does not strictly hold for realistic data distributions. Our study does not require such an assumption. Furthermore, we have provided explicit details regarding the time evolution of weight variance and the impacts of network width, depth, and other structures that were not provided in [1].\n\nFrom a perturbation instability perspective, [2] demonstrated that increasing network width does not necessarily improve robustness. This may seem to contradict our result, which suggests that a wider network can help the model maintain capacity during adversarial training, implying greater robustness in wider networks. However, these two claims are compatible. Robustness is determined by both perturbation instability (negative effect) and network capacity (positive effect). While the negative effect of width appears dominant in [2]'s experiments on CIFAR-10 and WideResNet, the positive effect appeared more prevalent in our experiments on MNIST, Fashion-MNIST, and fully connected networks with or without shortcuts. The dominant factor may depend on the dataset and model architectures.\n\n> The authors mentioned in Line 260 that \"This result suggests that residual networks are better suited for adversarial training. However, one of the previous studies indicated that residual networks are more vulnerable to transfer attacks [54] than vanilla networks.\" However, the proposed theoretical framework did not explain such a transfer attack phenomenon. Could the authors explain or give more comments about this?\n\nIn practice, adversarial training of residual networks is more common than that of vanilla networks. We mentioned transfer attacks as a motivation for analyzing adversarial training of vanilla networks. However, as several reviewers have pointed out, our study does not address transfer attacks. To avoid confusion, we have decided to remove this section in the updated version.\n\n> in Equation 3 and 4, it seems that the minimize part is not shown, perhaps it would be more reasonable to change the form of min-max in equation 3 and 4.\n\nEquations 3 and 4 do not include minimization because they define adversarial loss. The minimization of this loss with the standard loss defines adversarial training, which is described below Equations 3 and 4 (perhaps we should highlight this point more clearly). Many of our results and proofs refer to the adversarial loss, and we would like to maintain its current form without minimization."}]

Author rebuttal:
We appreciate the reviewers' critical reading, constructive comments, and overall positive scores. We have carefully taken into account all the comments and questions. Please kindly refer to the response to each reviewer. If our answers require further explanation or clarification, we are more than willing to provide them.

Meta-review: All reviewers agree that the paper has made a good theoretical contribution on understanding adversarial training and keep positive opinions on the paper. I therefore recommend acceptance of the paper. Please revise the paper accordingly according to the reviewers' comments in the camera-ready version.
Decision: Accept (spotlight)