Title: Adversarial Training from Mean Field Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial training; mean field theory
Abstract: Although adversarial training is known to be effective against adversarial examples, training dynamics are not well understood. In this study, we present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions. We introduce a new theoretical framework based on mean field theory, which addresses the limitations of existing mean field-based approaches. Based on the framework, we derive the (empirically tight) upper bounds of $\ell_q$ norm-based adversarial loss with $\ell_p$ norm-based adversarial examples for various values of $p$ and $q$. Moreover, we prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity. We also show that the network width alleviates these issues. Furthermore, the various impacts of input and output dimensions on the upper bounds and time evolution of weight variance are presented.

Adversarial Training from Mean Field Perspective

Abstract

Although adversarial training is known to be effective against adversarial examples, training dynamics are not well understood. In this study, we present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions. We introduce a new theoretical framework based on mean field theory, which addresses the limitations of existing mean field-based approaches. Based on the framework, we derive the (empirically tight) upper bounds of ℓ q norm-based adversarial loss with ℓ p norm-based adversarial examples for various values of p and q . Moreover, we prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity. We also show that the network width alleviates these issues. Furthermore, the various impacts of input and output dimensions on the upper bounds and time evolution of weight variance are presented.

1 Introduction

Adversarial training [38, 58] is one of the most effective approaches against adversarial examples [89]. Various studies aimed to improve the performance of adversarial training [28, 94, 113], leading to numerous observations. Adversarial training improves accuracy for adversarial examples but decreases it for clean images [58, 88]. Moreover, it requires additional training data [14, 40, 77] and achieves high robust accuracy in a training dataset but not in a test dataset [58]. To improve the reliability of adversarial training and address the aforementioned challenges, a theoretical analysis is essential. Specifically, it is crucial to gain insight into network evolution during adversarial training, conditions for adversarial trainability, and differences between adversarial and standard training.

However, the theoretical understanding of network training is challenging, even for standard training, due to the non-convexity of loss surface and optimization stochasticity. Recent studies employed the mean field theory and analyzed the early stage of standard training for randomly initialized deep neural networks (random networks) [71, 81]. Some studies explored network trainability regarding gradient vanishing/explosion [81, 104, 105] and dynamical isometry [69, 79, 100]. Others examined network representation power [49, 71, 101]. The theoretical results of the early stage of training have been empirically observed to fit well with fully trained networks [25, 84], partially supported by recent theoretical results [46]. However, existing mean field-based approaches cannot manage the probabilistic properties of an entire network (e.g., the distribution of a network Jacobian) and dependence between network inputs and parameters, which are crucial for analyzing adversarial training.

In this study, we propose a mean field-based framework that addresses the aforementioned limitations (Thm 4.1), and apply it to the adversarial training analysis. While previous studies on adversarial

Hiroshi Kera

training rely on strong assumptions (e.g., Gaussian data [14, 80] and linear classifiers [76, 107]), the proposed framework includes various scenarios (e.g., ℓ p norm-based adversarial examples and deep neural networks with or without shortcuts, i.e., residual or vanilla networks) without any assumptions on data distributions. Our analysis reveals various adversarial training characteristics that have been only experimentally observed or are unknown. The results are summarized as follows.

Upper bounds of adversarial loss. We derive the upper bounds of adversarial loss, quantifying the adverse effect of adversarial examples for various combinations of ℓ q -adversarial loss with the ℓ p -norm ϵ -ball ( p, q ∈ { 1 , 2 , ∞} ) (Thm 5.1). Numerical experiments confirm the tightness of these bounds. We also investigate the impacts of input and output dimensions on these bounds, and discover that for the ( p, q ) = (2 , ∞ ) combination, the bound is independent of these dimensions.

Time evolution of weight variance. Wepresent the time (training step) evolution of weight variance in training (Thms 5.4 and G.9). Weight variance has been used to assess training properties [49, 71, 81]. Our analysis indicates that adversarial training significantly regularizes weights and exhibits consistent weight dynamics across different norm choices.

Vanilla networks are not adversarially trainable under mild conditions. We show that gradient vanishing occurs in vanilla networks (without shortcuts) with large depths and small widths, making them untrainable via adversarial training, even with careful weight initialization (Thm 5.7). This contradicts standard training, where deep vanilla networks can be trained with proper initialization [81, 100]. However, residual networks are adversarially trainable even without proper initialization (Thm 5.8 and Prop G.10), proving the importance of shortcuts for adversarial training.

Degradation of network capacity and role of network width. As adversarial robustness requires high network capacity [63], deep networks can be used in adversarial training. However, we reveal that network capacity, measured based on the Fisher-Rao norm [56], sharply degrades in deep networks during adversarial training (Thms 5.9 and G.14). Specifically, we confirm that the capacity at training step t is Θ( L -tL 2 /N ) , where L and N denote the network depth and width, respectively. Interestingly, this result contrasts the roles of depth and width, i.e., the depth increases the initial capacity but decays it as training proceeds, whereas the width preserves it. While our theory is validated only during the initial stages of training, our experiments confirm that the adversarial robustness after full training is significantly influenced by network width (cf. Tabs. A5 and A6) as demonstrated in our theorems.

Other contributions. Furthermore, we show the followings cases: (a) Equality of the adversarial loss is obtained instead of inequality (upper bound) under several assumptions (Props G.2 and G.3). (b) Adversarial training leads to faster weight decay and less stable gradients compared with ℓ 2 weight regularization. (c) Capacity degradation is discussed for metrics other than the Fisher-Rao norm. (d) Adversarial risk cannot be mitigated while maintaining trainability and expressivity. (e) Discussion on ReLU-like activations extends to Lipschitz continuous activations under certain conditions. (f) A single-gradient descent attack can find adversarial examples that flip the prediction of a binary classifier (Prop K.1). Contributions (b)-(f) are found in Appx. K.

Although this study focuses on adversarial training, our theoretical framework can be applied to other training methods that consider the probabilistic property of an entire network and dependence between network inputs and parameters. Consequently, we believe that this study can potentially contribute to the theoretical understanding of adversarial training and various deep learning methods.

2 Related work

Here, we summarize the full version in Appx. A. A technical discussion follows in Sec. 5.1.

Mean field theory. Mean field theory in machine learning investigates the training dynamics of random networks in chaotic and ordered phases [71]. Networks can be trained at the boundary between these phases [81]. The theory has been extended to networks with shortcuts [105, 106], recurrent connections [15, 69], and batch normalization [104]. It has been employed to study dynamical isometry [15, 69, 70, 79], and a subsequent study achieved training of 10,000-layer networks without

shortcuts [100]. Moreover, the theory has been applied to analyze network representation power [49, 71, 101]. However, existing mean field-based analysis cannot handle the properties of an entire network and input-parameter dependence, which is a drawback for some deep learning methods, e.g., adversarial training. Thus, we propose a new framework to address these limitations.

Adversarial training. Various questions related to adversarial training have been theoretically addressed by some studies, including the robustness-accuracy trade-off [29, 47, 75, 76, 92, 113], generalization gap [6, 50, 102, 107], sample complexity [1, 14, 62, 80, 110], large model requirement [63], and enhanced transfer learning performance [27]. However, these results are obtained in limited settings (e.g., Gaussian data and linear classifiers) and are not easily extended to deep neural networks or realistic data distributions. To explore more general settings, recent studies used the neural tangent kernel theory [4, 46, 55]. In the kernel regime, adversarial training, even with a heuristic attack, finds a robust network [35, 115]. In our study, we investigate adversarial training dynamics based on a mean field perspective, covering general multilayered networks with or without shortcuts and without assumptions about data distributions.

3 Preliminaries

3.1 Setting

Notations are summarized in Tab. A2. For an integer n ∈ N , let [ n ] := { 1 , . . . , n } . In this study, we focus on random deep neural networks with ReLU-like activations, called random ReLU-like networks. This is formally defined as follows:

Definition 3.1 (ReLU-like network) . A network is called a ReLU-like network if all its activation functions are ϕ ( z ) := uz for z ≥ 0 and vz for z < 0 , with u, v ∈ R .

ReLU-like activations [33, 57] are widely used in theoretical and practical applications [42, 45, 53, 85, 109]. In Appx. K, we extend our theorems to networks with Lipschitz continuous activations.

A ReLU-like network, f : R d → R K , comprises L ∈ N trainable layers and two non-trainable layers for adjusting input and output dimensions. The input layer projects x in ∈ R d to an N -dimensional vector x (0) ∈ R N using the random matrix P in ∈ R N × d . Subsequently, L consecutive affine transformations and activations are applied by g : R N → R N . Then, g ( x (0) ) is multiplied by a random matrix P out ∈ R K × N to obtain the output vector f ( x in ) . Finally, the network function is provided by f ( x in ) := P out g ( P in x in ) . We assume that d and K are sufficiently large, and each entry of P in and P out is i.i.d. and sampled from Gaussians N (0 , 1 /d ) and N (0 , 1 /N ) , respectively.

An L -layer neural network g comprises weights W ( l ) = ( W ( l ) ij ) ∈ R N × N and biases b ( l ) = ( b ( l ) 1 , . . . , b ( l ) N ) ⊤ ∈ R N , where l ∈ [ L ] denotes the layer index. The network is assumed to possess a sufficiently large width (i.e., N is sufficiently large). The l -th pre- and post-activation are defined as h ( l ) := W ( l ) x ( l -1) + b ( l ) and x ( l ) := ϕ ( h ( l ) ) , respectively, where ReLU-like activation ϕ operates entry-wise. The weight W ( l ) ij and bias b ( l ) i are i.i.d. and sampled from N (0 , σ 2 w /N ) and N (0 , σ 2 b ) , respectively. The network function is represented by Eq. (1). For a residual network setting, refer to Appx. C.

3.2 Background

Mean field theory. Mean field theory employs probabilistic methods to analyze the properties of random deep neural networks. It assumes that h ( l ) follows a Gaussian, justified by the central limit theorem when width N is sufficiently large [71]. Here, we review the mean field-based approach to analyze the forward and backward dynamics of a network. Let L : R d → R represent the loss function. The mean squared pre-activation E [( h ( l ) i ) 2 ] and gradient χ ( l ) := E [( ∂ L ( x in ) /∂x ( l ) i ) 2 ] , where i denotes the neuron index, are calculated as follows [71, 81]:

These equations represent the dynamics between adjacent layers. We can infer that gradients vanish when σ 2 w E [ ϕ ′ ( h ( l +1) i ) 2 ] < 1 and explode when σ 2 w E [ ϕ ′ ( h ( l +1) i ) 2 ] > 1 , indicating that a network is trainable only if σ 2 w E [ ϕ ′ ( h ( l +1) i ) 2 ] ≈ 1 .

Adversarial training. We define adversarial loss as follows:

where ϵ > 0 and p, q ∈ { 1 , 2 , ∞} . The adversarial loss aims to minimize the difference between the network outputs of natural and adversarial inputs. Networks are trained by minimizing the sum of the standard loss L std : R d ×Y → R (e.g., cross-entropy loss), where Y denotes a label set, and the adversarial loss L adv . The mean field analysis typically assumes gradient independence for loss functions (Appx. B) [81]. We use this assumption for L std , but not L adv . Although Eq. (3) differs from the standard adversarial loss based on cross-entropy [58], our definition is employed in more robust methods, e.g., TRADES [113] and is theoretically simpler to analyze. Therefore, herein, the aforementioned loss is analyzed. However, even this simplified definition (Eq. (3)) poses a challenge for theoretical analysis due to the complex nested structure of a deep neural network (cf. Eq. (4)).

4 Theoretical framework

4.1 Limitations of existing mean field-based approaches

We propose a new theoretical framework based on mean field theory to analyze adversarial training. Here, we describe two limitations of existing mean field-based approaches, e.g., Eq. (2).

Layer-wise approach. Existing approaches focus on the dynamics between adjacent layers (cf. Eq. (2)). However, analyzing the adversarial loss (Eq. (3)) requires a framework that handles the probabilistic properties of an entire network instead of adjacent layers. This analysis becomes difficult due to the complex nested structure of networks (cf. Eqs. (1) and (4)). For example, there is no clarity on the the probabilistic behavior of f ( x in ) , distribution of f ( x in + η ) -f ( x in ) , and dependence between f ( x in ) and inputs. We need a framework that disentangles the nested structure of networks and manages the probabilistic properties of an entire network. Recent studies on the mean field theory studies [15, 36, 69, 70, 100] have concepts related to ours; the comparative analysis is given in Sec. 5.1.

Difficulty in analyzing input-parameter dependence. The analysis of adversarial training requires consideration of input-parameter dependence since adversarial perturbations are designed based on network parameters (cf. Eq. (3)). However, this cannot be readily addressed using existing approaches because their frameworks (e.g., Eq. (2)) do not offer a clear view of the dependence between perturbation η and network parameters W (1) 1 , 1 , W (1) 1 , 2 , . . . , and W ( L ) N,N .

The proposed framework resolves these limitations and provides a simple network representation that allows us to capture the entire network property with clear input-parameter dependence.

4.2 Proposed framework

The current mean field-based approaches cannot capture the probabilistic properties of an entire network due to the complex nested structure of deep neural networks. Moreover, it is difficult to consider the dependence between inputs and numerous number of parameters. To address these limitations, we employ a linear-like representation of a ReLU-like network and propose its probabilistic properties. As ReLU-like networks are piecewise linear, a vanilla ReLU-like network can be represented as:

0

.

10

0

.

05

0

.

00

0

.

05

0

.

10

0

5

10

Probability density

Exp.

Theory

-

-

where D ( · ) denotes a diagonal matrix and a ( x in ) is defined similar to J ( x in ) (cf. Eq. (A48)). For a residual network, Eq. (A73) can be referred. Importantly, this representation does not rely on approximations, e.g., Taylor expansions. As W ( l ) and b ( l ) are randomly sampled, J ( l ) ( x in ) and a ( l ) ( x in ) denote a random matrix and vector, respectively. Unlike the original network definition (Eq. (1)), this representation (Eq. (5)) is non-nested and focuses only two parameters, thereby simplifying network analysis. Remarkably, we show the following properties of J ( x in ) and a ( x in ) :

Theorem 4.1 (Properties and distributions of J ( x in ) and a ( x in ) ) . Suppose that the width N is sufficiently large. Then, for any x in ∈ R d , (I) J ( x in ) and a ( x in ) are independent. (II) each entry of J ( x in ) and a ( x in ) is i.i.d. and follows the Gaussian below:

where α := ( u 2 + v 2 ) / 2 (cf. Defn 3.1) and ω is ω v := ασ 2 w for vanilla networks and ω r := 1 + ασ 2 w for residual networks.

Asignificance of this theorem lies in that despite being functions of x in , the distributions of J ( x in ) and a ( x in ) do not depend on x in . 1 In other words, although J ( x in ) and a ( x in ) are determined by (a fixed) x in for an initialized network, they become different for each sampling of weights and biases, and the selection of these values is independent of x in . Besides, J ( x in ) and a ( x in ) are independent and their distributions are Gaussian, which exhibits convenient properties. A sketch of proof is given in Appx. D and formal one is in Appxs. E and F.

To validate Thm 4.1, we conducted a numerical experiment and present the results in Fig. 1. We randomly sampled 10,000 vanilla ReLU networks and computed J ( x in ) 1 , 1 for each network using the identical input x in . Additional experimental results can be found in Appx. L.

Broader applicability. The proposed framework (Thm 4.1) manages an entire network using only two Gaussians. While we focus on adversarial training, Thm 4.1 can be valuable for other deep neural network analyses based on the mean field theory. For example, contrastive learning [16] can be investigated, as it aims to minimize the distance between original and positive samples while maximizing it for negative samples. In this context, instead of considering the adversarial loss, ∥ f ( x in + η ) -f ( x in ) ∥ , we can analyze loss functions, e.g., ∥ f ( x in pos ) -f ( x in ori ) ∥ and ∥ f ( x in neg ) -f ( x in ori ) ∥ , where x in ori , x in pos , and x in neg represent the original, positive, and negative samples, respectively. The complex nested structure of a network makes it challenging to consider the difference between two network outputs; however, Thm 4.1 helps theoretically manageable analyses.

5 Analysis of adversarial training

The proof of each theorem is described in Appx. G.

̸

5.1 Upper bounds of adversarial loss

As the ReLU-like network f is locally linear and its input Jacobian at x in is J ( x in ) (cf. Eq. (5)), we can consider a more tractable form of the adversarial loss instead of Eq. (4) as follows:

where ∥ J ( x ) ∥ p,q := max ∥ η ∥ p =1 ∥ J ( x ) η ∥ q denotes the ( p, q ) -operator norm of J ( x ) . Using Thm 4.1, which describes the property of J ( x ) , we transform Ineq. (8) and obtain the following:

Theorem 5.1 (Upper bounds of adversarial loss) . Suppose that the input dimension d , output dimension K , and width N are sufficiently large. Then, for any x in ∈ R d , the following inequality holds:

where W := { W (1) 1 , 1 , W (1) 1 , 2 , . . . , W ( L ) N,N } denotes the set of all network weights. The constant β p,q for each norm pair ( p, q ) is described in Tab. 1.

For some choices of ( p, q ) , we cannot derive upper bounds, and thus, Tab. 1 contains blank. Numerical experiments show the tightness of the bounds (cf. Fig. 2). The theorem indicates that (i) the bounds increase linearly with the perturbation budget ϵ , (ii) the effects of the input and output dimensions depend on the norms ( p, q ) (cf. Tab. 1), and (iii) network depth L exponentially impacts the bounds, with ω = 1 as a threshold between order and chaos. Further, the square sum of the weights in Ineq. (9) suggests that adversarial training exhibits a weight regularization effect, which is compared to ℓ 2 weight regularization in Appx. I. Besides, we derive equality rather than inequality (upper bound) under specific assumptions (e.g., small ϵ ) for some ( p, q ) in Appx. K, indicated by † and ♢ in Tab. 1.

The input and output dimensions, d and K , influence the bounds through the ( p, q ) -dependent parameter β p,q . In Tab. 1, β p,q displays a wide range of dependencies on d and K . When d →∞ and q = ∞ , d significantly affects the two phases of the adversarial loss, where p = 2 marks the transition point from order ( p = 1 ) to chaos ( p = ∞ ) . In contrast, under realistic assumptions with d ≫ K , K affects negligibly. Interestingly, the dimensions do not impact the upper bounds when ( p, q ) = (2 , ∞ ) . In practice, we scale the perturbation budget and adversarial loss according to the choice of ( p, q ) , respectively, and discuss the scaling effects in Appx. K.

Comparison with other studies. We can consider studies on global Lipschitz of networks in certified adversarial defenses [3, 18, 93] and spectral regularization [60, 108] to analyze Ineq. (8). A key difference is that the proposed probabilistic approach contradicts their deterministic approach. By imposing probabilistic constraints on network parameters, we can obtain exponentially tighter, interpretable, and more theoretically manageable bounds, which facilitates the subsequent section's discussion. The mathematical comparison is described in Appx. H.

Results obtained from [15, 36, 69, 70, 100] can be used to analyze the Jacobian's singular value distribution. Compared to their approaches, which are limited to ( p, q ) = (2 , 2) , the proposed method offers greater generality and flexibility, providing upper bounds for various ( p, q ) . Moreover, Thm 4.1 enables the derivation of equality instead of inequality (upper bound), Props G.2 and G.3, which is not achievable using the approaches mentioned in the aforementioned studies because it cannot incorporate perturbation-Jacobian dependence. Moreover, we do not consider their assumption that the variance V [ h ( l ) i ] is constant for all l ∈ [ L ] , which is often difficult to satisfy.

Further, we refer to [78], which established a theoretical link between adversarial training and the ( p, q ) -operator norm of a Jacobian. Their findings support Ineq. (8) in training scenarios using heuristic attacks, e.g., projected gradient descent [58]. In this study, we derive concrete upper bounds beyond their theoretical link, enabling further investigation of adversarial training properties.

5.2 Time evolution of weight variance

Weight variance plays a critical role in determining deep neural network properties [49, 81]. We substitute the adversarial loss definition (Eq. (3)) with L adv := ϵβ p,q ω ( t ) L/ 2 , where t ≥ 0 denotes the

continuous training step. Considering gradient descent with an infinitely small learning rate (gradient flow), the model parameter θ ( t ) at step t is updated as:

We make the following assumption.

Assumption 5.2. For 0 ≤ t ≤ T ≪ N , model parameters are independent, and weight and bias follow Gaussian N (0 , σ 2 w ( t ) /N ) and N (0 , σ 2 b ( t )) , respectively.

This assumption ensures that the properties of the model parameters remain close to their initial values during the early stages of training ( t ≤ T ). Under Asm 5.2, the original and proposed mean field theories (Thm 4.1) remain valid during training. For a moderately small value of T , Asm 5.2 is not strong because the model parameters change minimally and retain their initialized states with sufficiently small learning rates. Recent neural tangent kernel studies partially supported this assumption [4, 46, 55], and it is known that random network theories align well with fully trained networks [25, 84]. For example, T = 160 is reasonable in a specific training setting (cf. Fig. 4).

Now, we summarize other assumptions for reference as follows:

Assumption 5.3. The input dimension d , output dimension K , and width N are sufficiently large. We apply Asm B.1 to the standard loss function L std . The adversarial loss is defined as L adv := ϵβ p,q ω ( t ) L/ 2 (cf. Thm 5.1).

Based on the aforementioned settings, we obtain the time evolution of the weight variance.

Theorem 5.4 (Weight time evolution of vanilla network in adversarial training) . Suppose that Asms 5.2 and 5.3 hold. Then, the time evolution of σ 2 w of a vanilla network in adversarial training is given by:

A similar result is obtained for residual networks (Thm G.9). The theorem reveals that the weight variance linearly decreases with t , which can be attributed to the weight regularization effect of adversarial training (cf. Sec. 5.1). In addition, the norm pair ( p, q ) affect only time-invariant constant β p,q , and the dynamics of the weight variance can be represented as a consistent function of t regardless of ( p, q ) . In other words, adversarial training exhibits consistent weight dynamics irrespective of norm selection , with a scale factor varying.

5.3 Vanilla networks are not adversarially trainable under mild conditions

We show that in adversarial training, vanilla networks can fit a training dataset in limited cases (small depth and large width), but residual networks can in most cases. This result suggests that residual networks are better suited for adversarial training. First, we present the trainability condition based on the concept in [81, 105].

Definition 5.5 ( ( M,m ) -trainability condition) . Anetwork is said to be ( M,m ) -trainable if a network satisfies m ≤ χ (0) /χ ( L ) ≤ M , where 0 ≤ m ≤ 1 and M ≥ 1 . 2

The value χ ( l ) denotes the squared length of the gradient in the l -th layer. A near-zero χ (0) /χ ( L ) suggests gradient vanishing, while a large value implies gradient explosion. Hence, Defn 5.5 is directly linked to successful training. In contrast to the existing definition, χ ( l -1) /χ ( l ) ≈ 1 [81, 105], Defn 5.5 incorporates M and m for the subsequent discussion. Then, we establish specific ( M,m ) -trainability conditions for ReLU-like networks.

Lemma 5.6 (Vanilla and residual ( M,m ) -trainability condition) . Suppose that the width N is sufficiently large. Then, the ( M,m ) -trainability conditions for vanilla and residual networks are respectively given by:

Using the weight time evolution (Thm 5.4) and ( M,m ) -trainability condition (Lemma 5.6), the following theorem can be readily derived.

Theorem 5.7 (Vanilla networks are not adversarially trainable) . Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, and the ( M,m ) -trainability condition holds at t = 0 and ασ 2 w (0) = 1 . If

then there exists 0 < τ ≤ T such that the ( M,m ) -trainability condition does not hold for τ ≤ t ≤ T .

This indicates the potential untrainability of vanilla networks in adversarial training, even when satisfying the ( M,m ) -trainability condition at initialization, contradicting with standard training where extremely deep networks can be trained if initialized properly [81] . This issue arises from the inconsistency between the trainability condition of a vanilla network, i.e., m 1 /L ≤ ασ 2 w (cf. Lemma 5.6) and monotonically decreasing nature of σ 2 w ( t ) in adversarial training (cf. Thm 5.4). As stated in Asm 5.2, we assume that T is small. Therefore, if the right-hand term of Eq. (13) becomes large, the assumption and Thm 5.7 are violated. In summary, for large L (many layers) and ϵ (large perturbation constraint), vanilla networks are not adversarially trainable . Moreover, a large N (a wide network) can mitigate this issue in vanilla networks . For example, the vanilla network with L = 20 , N = 256 , and ϵ = 0 . 3 is not adversarially trainable; however, when the width is increased to 512 , it becomes trainable (cf. Fig. 4). In contrast, we can claim the following for residual networks:

Theorem 5.8 (Residual networks are adversarially trainable) . Consider a residual network. Suppose that Asms 5.2 and 5.3 hold, and the ( M,m ) -trainability condition holds at t = 0 and ασ 2 w (0) ≪ 1 . Then, ( M,m ) -trainability condition always holds for 0 ≤ t ≤ T .

This occurs due to the ( M,m ) -trainability condition for residual networks, which does not have any lower bound (cf. Lemma 5.6), and the monotonically decreasing nature of σ 2 w (cf. Thm G.9). Besides, residual networks are adversarially trainable even without careful weight initialization (Prop G.10). These theorems highlight the robust stability of adversarial training in residual networks.

5.4 Degradation of network capacity

We demonstrate that adversarial training degrades network capacity. We use the Fisher-Rao norm as a metric [56], with alternative metrics discussed in Appx. K. The Fisher-Rao norm is defined as:

where w := ( W (1) 11 , W (1) 12 , . . . , W ( L ) NN ) ⊤ represents the vector of all the network weights and F denotes the empirical Fisher information matrix when only one training data point is considered (for simplicity). The Fisher-Rao norm is preferred over other norm-based capacity metrics [8, 65-67] owing to its invariance to node-wise rescaling [56]. We present the following theorem based on [49]:

Theorem 5.9 (Adversarial training degrades network capacity) . Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, Asm B.1 is applied to the network output, and the ( M,m ) -trainability condition holds at t = 0 and ασ 2 w (0) = 1 . Assume ∥ ∥ x in ∥ ∥ 2 = √ d and σ 2 b ( t ) = 0 . Then, the expectation of the Fisher-Rao norm is given by:

A related result for residual networks is presented in Thm G.14. As described in [63], adversarial robustness necessitates high capacity. However, Thm 5.9 indicated that capacity decreases linearly with step t in adversarial training. To address this conflict, large depth (large L ), which increase the initial capacity with Θ( L ) , can be considered, but this scenario accelerates the degradation speed with Θ( L 2 ) . To preserve capacity, we must increase the width N accordingly. Consequently, to achieve high capacity in adversarial training, it is necessary to increase not only the number of layers L but also the width N to keep L 2 /N constant . Although Thms 5.9 and G.14 have been established in the early stages of training, numerical experiments proved that the adversarial robustness following full training is significantly influenced by network width (cf. Tabs. A5 and A6).

250

500

750

1000

0

.

25

0

.

50

0

.

75

p

= 1

, q

= 1

250

500

750

1000

0

.

05

0

.

10

p

= 1

, q

= 2

250

500

750

1000

0

.

01

0

.

02

0

.

03

p

= 1

, q

=

∞

Theory

Exp.

250

500

750

1000

0

.

15

0

.

20

p

= 2

, q

= 2

250

500

750

1000

0

.

0

0

.

2

p

= 2

, q

=

∞

250

500

750

1000

1

2

p

=

∞

, q

=

∞

Input dimension

d

Adversarial loss

L

adv

0

500

1000

1500

2000

Training step

1

.

9985

1

.

9990

1

.

9995

2

.

0000

Weight variance

σ

2

w

Standard

/lscript

2

weight

Adv. (

N

= 2

k

)

Adv. (

N

= 1

k

)

Theory (

N

= 2

k

)

Theory (

N

= 1

k

)

100

200

300

400

500

600

700

800

N

10

20

30

40

Depth

L

T

=100

T

=160

T

=250

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Training accuracy

6 Experimental results

We validate Thms 5.1, 5.4 and 5.7 via numerical experiments. The vanilla ReLU networks were initialized with σ 2 w = 2 and σ 2 b = 0 . 01 to meet the ( M,m ) -trainability conditions (Lemma 5.6). To verify Thms 5.4 and 5.7, we used MNIST [26]. Setup details and more results are given in Appx. L.

Verification of Thm 5.1. We created adversarial examples for initialized networks and computed the adversarial loss (Eq. (3)). As shown in Fig. 2, the upper bounds in Thm 5.1 were considerably tight. Some samples slightly exceeded the upper bounds because we used the finite network width while the infinite width is assumed (cf. Appx. L).

Verification of Thm 5.4. We trained vanilla networks normally (with and without ℓ 2 regularization) and adversarially. The time evolution of weight variance is shown in Fig. 3. Adversarial training significantly reduces weight variance, whereas wide width (i.e., large N ) suppresses it. The validity of Thm 5.4, which forms the basis of our theorems such as Thms 5.7 and 5.9, supports our theorems.

Verification of Thm 5.7. We trained vanilla networks considering various depth and width settings and monitored the training accuracy. As shown in Fig. 4, it was difficult for vanilla networks to fit the training dataset when the depth was large and the width was small; increased width helps in fitting. Although we currently lack a theoretical prediction of the boundary between trainable and untrainable areas determined based on Eq. (13), empirical evidence suggests that T = 160 is relevant.

7 Limitations

The mean field theory offers valuable insight into network training. However, its applicability is restricted to the initial stages of training. Although recent studies suggested empirically [25, 84] and theoretically [46] that the analysis of early-stage training extends well to full training, the strict relationship is yet to be explored. Our results have the same limitations. Although some theorems accurately capture the behavior during the initial stages of training and even after full training (cf. Tabs. A5 and A6), as training progresses, some theorems begin to diverge from the actual behavior (cf. Fig. A17). Another caveat is that the mean field theory assumes infinite network width, which is practically infeasible. Empirically, our theorems hold well when the width approximately exceeds 1,000 (cf. Fig. A7), while Thm 5.1 requires larger width approximately exceeds 10,000 for ( p, q ) = (2 , 2) (cf. Fig. A14). These limitations also derive from the mean field theory and are not unique to our study. Despite these limitations, we consider that this study provides a powerful theoretical framework that extends the applicability of the mean field theory to various training methods, and the results obtained for adversarial training are insightful.

8 Conclusions

We proposed a framework based on the mean field theory and conducted a theoretical analysis of adversarial training. The proposed framework addressed the limitations of existing mean field-based approaches, which could not handle the probabilistic properties of an entire network and dependence between network inputs and parameters [71, 81]. Based on this framework, we examined adversarial training from various perspectives, unveiling upper bounds of adversarial loss, relationships between adversarial loss and network input/output dimensions, the time evolution of weight variance, trainability conditions, and the degradation of network capacity. The theorems of this study were validated via numerical experiments. The proposed theoretical framework is highly versatile and can help analyze various training methods, e.g., contrastive learning.

Acknowledgments and Disclosure of Funding

We would like to thank Huishuai Zhang for useful discussions. This work was supported by JSPS KAKENHI Grant Number JP23KJ0789 and JP22K17962, by JST, ACT-X Grant Number JPMJAX23C7, JAPAN, and by Microsoft Research Asia.

References

J.-B. Alayrac, J. Uesato, P.-S. Huang, A. Fawzi, R. Stanforth, and P. Kohli. Are labels required for improving adversarial robustness? In NeurIPS , volume 32, 2019.

L. Amsaleg, J. Bailey, D. Barbe, S. Erfani, M. E. Houle, V. Nguyen, and M. Radovanovi´ c. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In WIFS , pages 1-6, 2017.

C. Anil, J. Lucas, and R. Grosse. Sorting out lipschitz function approximation. In ICML , pages 291-301, 2019.

S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In NeurIPS , volume 32, 2019.

A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML , pages 274-283, 2018.

P. Awasthi, N. Frank, and M. Mohri. Adversarial learning guarantees for linear hypotheses and neural networks. In ICML , pages 431-441, 2020.

P. Bartlett, S. Bubeck, and Y. Cherapanamjeri. Adversarial examples in multi-layer random relu networks. In NeurIPS , volume 34, pages 9241-9252, 2021.

P. L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-normalized margin bounds for neural networks. In NeurIPS , volume 30, 2017.

P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR , 3(Nov):463-482, 2002.

Y. Blumenfeld, D. Gilboa, and D. Soudry. A mean field theory of quantized deep networks: The quantization-depth trade-off. In NeurIPS , volume 32, 2019.

S. Bubeck, Y. Cherapanamjeri, G. Gidel, and R. Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. In NeurIPS , volume 34, pages 10081-10091, 2021.

N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In ACM WS , pages 3-14, 2017.

N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In SSP , pages 39-57, 2017.

Y. Carmon, A. Raghunathan, L. Schmidt, P. Liang, and J. C. Duchi. Unlabeled data improves adversarial robustness. In NeurIPS , 2019.

M. Chen, J. Pennington, and S. Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. In ICML , pages 873-882, 2018.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML , pages 1597-1607, 2020.

Y. Cho and L. Saul. Kernel methods for deep learning. In NeurIPS , volume 22, 2009.

M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval networks: Improving robustness to adversarial examples. In ICML , pages 854-863, 2017.

J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In ICML , pages 1310-1320, 2019.

F. Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML , pages 2206-2216, 2020.

A. Damianou and N. D. Lawrence. Deep gaussian processes. In AISTATS , pages 207-215, 2013.

A. Daniely. SGD learns the conjugate kernel class of the network. In NeurIPS , volume 30, 2017.

A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In NeurIPS , volume 29, 2016.

A. Daniely and H. Schacham. Most relu networks suffer from ell ˆ 2 adversarial perturbations. In NeurIPS , volume 33, pages 6629-6636, 2020.

G. De Palma, B. Kiani, and S. Lloyd. Adversarial robustness guarantees for random deep neural networks. In ICML , pages 2522-2534, 2021.

L. Deng. The MNIST database of handwritten digit images for machine learning research. Signal Process. Mag. , 29(6):141-142, 2012.

Z. Deng, L. Zhang, K. Vodrahalli, K. Kawaguchi, and J. Y. Zou. Adversarial training helps transfer learning via better representations. In NeurIPS , volume 34, pages 25179-25191, 2021.

G. W. Ding, Y. Sharma, K. Y. C. Lui, and R. Huang. MMA training: Direct input space margin maximization through adversarial training. In ICLR , 2020.

E. Dobriban, H. Hassani, D. Hong, and A. Robey. Provable tradeoffs in adversarially robust classification. arXiv:2006.05161 , 2020.

A. Fawzi, H. Fawzi, and O. Fawzi. Adversarial vulnerability for any classifier. In NeurIPS , volume 31, 2018.

A. Fawzi, O. Fawzi, and P. Frossard. Analysis of classifiers' robustness to adversarial perturbations. ML , 107(3):481-508, 2018.

A. Fawzi, S.-M. Moosavi-Dezfooli, and P. Frossard. Robustness of classifiers: from adversarial to random noise. In NeurIPS , volume 29, 2016.

K. Fukushima. Cognitron: A self-organizing multilayered neural network. Biol. Cybern. , 20(3):121-136, 1975.

A. Galloway, A. Golubeva, T. Tanay, M. Moussa, and G. W. Taylor. Batch normalization is a cause of adversarial vulnerability. In ICML WS , 2019.

R. Gao, T. Cai, H. Li, C.-J. Hsieh, L. Wang, and J. D. Lee. Convergence of adversarial training in overparametrized neural networks. In NeurIPS , volume 32, 2019.

D. Gilboa, B. Chang, M. Chen, G. Yang, S. S. Schoenholz, E. H. Chi, and J. Pennington. Dynamical isometry and a mean field theory of LSTMs and GRUs. arXiv:1901.08987 , 2019.

J. Gilmer, L. Metz, F. Faghri, S. S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow. Adversarial spheres. In ICLR WS , 2018.

I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR , 2015.

S. Gowal, K. D. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli. Scalable verified training for provably robust image classification. In ICCV , pages 4842-4851, 2019.

S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann. Improving robustness using generated data. In NeurIPS , 2021.

S. Hayou, A. Doucet, and J. Rousseau. On the selection of initialization and activation function for deep neural networks. arXiv:1805.08266 , 2018.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

M. Hein and M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In NeurIPS , volume 30, 2017.

J. Hron, Y. Bahri, J. Sohl-Dickstein, and R. Novak. Infinite attention: NNGP and NTK for deep attention networks. In ICML , pages 4376-4386, 2020.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR , pages 4700-4708, 2017.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS , volume 31, 2018.

A. Javanmard, M. Soltanolkotabi, and H. Hassani. Precise tradeoffs in adversarial training for linear regression. In COLT , pages 2034-2078, 2020.

H. Kannan, A. Kurakin, and I. Goodfellow. Adversarial logit pairing. arXiv:1803.06373 , 2018.

R. Karakida, S. Akaho, and S.-i. Amari. Universal statistics of Fisher information in deep neural networks: Mean field approach. In AISTATS , pages 1032-1041, 2019.

J. Khim and P.-L. Loh. Adversarial risk bounds via function transformation. arXiv:1810.09519 , 2018.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In CVPR , 2015.

A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS , pages 1097-1105, 2012.

J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as gaussian processes. In ICLR , 2018.

J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS , volume 32, 2019.

T. Liang, T. Poggio, A. Rakhlin, and J. Stokes. Fisher-rao metric, geometry, and complexity of neural networks. In AISTAT , pages 888-896, 2019.

A. L. Maas, A. Y. Hannun, A. Y. Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In ICML , volume 30, page 3, 2013.

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR , 2018.

T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In ICLR , 2018.

A. Montanari and Y. Wu. Adversarial examples in random neural networks with general activations. arXiv:2203.17209 , 2022.

A. Najafi, S.-i. Maeda, M. Koyama, and T. Miyato. Robustness to adversarial perturbations in learning from incomplete data. In NeurIPS , volume 32, 2019.

P. Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv:1901.00532 , 2019.

R. M. Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks , pages 29-53. Springer, 1996.

B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring generalization in deep learning. In NeurIPS , volume 30, 2017.

B. Neyshabur, R. R. Salakhutdinov, and N. Srebro. Path-SGD: Path-normalized optimization in deep neural networks. In NeurIPS , volume 28, 2015.

B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In COLT , pages 1376-1401, 2015.

R. Novak, L. Xiao, J. Lee, Y. Bahri, G. Yang, J. Hron, D. A. Abolafia, J. Pennington, and J. Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR , 2019.

J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In NeurIPS , volume 30, 2017.

J. Pennington, S. Schoenholz, and S. Ganguli. The emergence of spectral universality in deep networks. In AISTATS , pages 1924-1932, 2018.

B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. In NeurIPS , volume 29, 2016.

R. Rade and S.-M. Moosavi-Dezfooli. Helper-based adversarial training: Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In ICML , 2021.

M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein. On the expressive power of deep neural networks. In ICML , pages 2847-2854, 2017.

A. Raghunathan, J. Steinhardt, and P. Liang. Certified defenses against adversarial examples. In ICLR , 2018.

A. Raghunathan, S. M. Xie, F. Yang, J. Duchi, and P. Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In ICML , 2020.

A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang. Adversarial training can hurt generalization. In ICML WS , 2019.

S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann. Fixing data augmentation to improve adversarial robustness. arXiv:2103.01946 , 2021.

K. Roth, Y. Kilcher, and T. Hofmann. Adversarial training is a form of data-dependent operator norm regularization. In NeurIPS , volume 33, pages 14973-14985, 2020.

A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR , 2014.

L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust generalization requires more data. In NeurIPS , 2018.

S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. In ICLR , 2017.

A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? In ICLR , 2019.

A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! In NeurIPS , 2019.

C.-J. Simon-Gabriel, Y. Ollivier, L. Bottou, B. Schölkopf, and D. Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In ICML , pages 5809-5817, 2019.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR , 2014.

A. Sinha, H. Namkoong, R. Volpi, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR , 2018.

H. Sompolinsky, A. Crisanti, and H.-J. Sommers. Chaos in random neural networks. Phys. Rev. Lett. , 61(3):259, 1988.

D. Su, H. Zhang, H. Chen, J. Yi, P.-Y. Chen, and Y. Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In ECCV , pages 631-648, 2018.

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In ICLR , 2014.

F. Tramer, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses. In NeurIPS , volume 33, pages 1633-1645, 2020.

J. A. Tropp. Topics in sparse approximation . The University of Texas at Austin, 2004.

D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In ICLR , 2019.

Y. Tsuzuku, I. Sato, and M. Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In NeurIPS , volume 31, 2018.

Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR , 2020.

C. Williams. Computing with infinite networks. In NeurIPS , volume 9, 1996.

E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML , pages 5286-5295, 2018.

E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. In ICLR , 2020.

B. Wu, J. Chen, D. Cai, X. He, and Q. Gu. Do wider neural networks really help adversarial robustness? In NeurIPS , volume 34, pages 7054-7067, 2021.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 , 2017.

L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In ICML , pages 5393-5402, 2018.

L. Xiao, J. Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. In ICML , pages 10462-10472, 2020.

Y. Xing, Q. Song, and G. Cheng. On the generalization properties of adversarial training. In AISTATS , pages 505-513, 2021.

G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv:1902.04760 , 2019.

G. Yang, J. Pennington, V. Rao, J. Sohl-Dickstein, and S. S. Schoenholz. A mean field theory of batch normalization. In ICLR , 2019.

G. Yang and S. Schoenholz. Mean field residual networks: On the edge of chaos. In NeurIPS , volume 30, 2017.

G. Yang and S. S. Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. OpenReview , 2018.

D. Yin, R. Kannan, and P. Bartlett. Rademacher complexity for adversarially robust generalization. In ICML , pages 7085-7094, 2019.

Y. Yoshida and T. Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv:1705.10941 , 2017.

S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC , pages 1-12, 2016.

R. Zhai, T. Cai, D. He, C. Dan, K. He, J. Hopcroft, and L. Wang. Adversarially robust generalization just requires more unlabeled data. arXiv:1906.00555 , 2019.

D. Zhang, T. Zhang, Y. Lu, Z. Zhu, and B. Dong. You only propagate once: Accelerating adversarial training via maximal principle. In NeurIPS , 2019.

H. Zhang, D. Yu, Y. Lu, and D. He. Adversarial noises are linearly separable for (nearly) random neural networks. arXiv:2206.04316 , 2022.

H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML , pages 7472-7482, 2019.

J. Zhang, X. Xu, B. Han, G. Niu, L. Cui, M. Sugiyama, and M. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML , pages 11278-11287, 2020.

Y. Zhang, O. Plevrakis, S. S. Du, X. Li, Z. Song, and S. Arora. Over-parameterized adversarial training: An analysis overcoming the curse of dimensionality. In NeurIPS , volume 33, pages 679-688, 2020.

L

→

A Additional related work

Please also refer to Secs. 2 and 5.1.

A.1 Random networks and mean field theory

Understanding general deep neural networks is challenging due to the non-convexity of loss surfaces and stochastic nature of optimization. Researchers have explored random networks, which have random parameters instead of trained parameters. Random networks have been primarily studied in three areas: compositional kernels [17, 22, 23], neural network Gaussian processes [21, 44, 54, 59,

64, 68, 95, 103], and mean field theory. These fields share close relationships, particularly between neural network Gaussian processes and mean field theory. For example, the forwarding dynamics in mean field theory (Eq. (2)) is equivalent to the kernel representation of a Gaussian process [64, 71]. Furthermore, the accuracy of a Gaussian process is significantly influenced by the edge of chaos, which has been studied in mean field theory [68]. We provide a more detailed review of the mean field theory.

Mean field theory for neural networks was first introduced in [87] and later extended in [71, 81]. This theory examines the signal propagation, dynamics, and trainability of random networks in two phases: ordered and chaotic. The ordered phase is characterized by decreasing layer output variance and vanishing gradients during backpropagation, while the chaotic phase is characterized by expanding variance and exploding gradients. Effective training of deep neural networks occurs near the boundary between these two phases [71, 81]. Researchers have applied mean field theory to study various network architectures, including dropout [81], batch normalization [104], residual network [105, 106], recurrent network [15], quantized network [10], and Swish [41]. Recent research [15, 36, 69, 70, 100] has also utilized the mean field theory to analyze dynamical isometry, where all singular values of the Jacobian are one. Some of these findings can be applied to analyze Ineq. (8) in this study, with detailed comparisons provided in Sec. 5.1. Other studies have explored network representation power [49, 71, 101]. This work utilized the proof idea of [49] in the derivation of Thms 5.9 and G.14.

In this study, we employed mean field theory to analyze adversarial training behavior. However, the original theory has two limitations that make it unsuitable for this purpose (cf. Sec. 4.1). To address these limitations, we introduced a new mean field-based framework (cf. Sec. 4.2). Our theory is also applicable to mean field-based analyses of other deep learning methods beyond adversarial training.

A.2 Adversarial examples

A.2.1 Adversarial examples in random neural networks

We summarize the literature on adversarial examples in random neural networks [7, 11, 24, 25, 61, 84, 112]. It has been reported that adversarial perturbations can be found through gradient flow in most random ReLU networks with decreasing layer widths [24]. This result was extended to two-layer random networks with greater width than input dimension [11] and generalized to random ReLU networks with constant depth and wide width [7]. Recently, similar results were presented without width restrictions and for local Lipschitz continuous activation [61]. Additionally, it has been demonstrated that adversarial noises generated by a single-step attack are linearly separable in a two-layer random network and the neural tangent kernel regime [112]. More information on [25, 84] can be found in Appx. A.2.2.

Although the context differs somewhat, we mention [34], which provides empirical evidence that batch normalization leads to adversarial vulnerability. This observation aligns with previous results from mean field theory suggesting that batch normalization causes gradient explosion [104].

In this study, we focused on early stage adversarial training properties rather than adversarial examples. A key difference is that we primarily investigated the maximum difference between network outputs for standard and adversarial inputs rather than misclassification which was the main focus of previous studies [7, 11, 24, 61]. Nonetheless, some findings in Sec. 5.1 can be applied to the understanding of adversarial examples in random networks. In addition, in Appx. K, we proved the existence of adversarial examples in random networks to demonstrate the effectiveness of Thm 4.1.

A.2.2 Adversarial examples and input dimension

We present research investigating the relationship between adversarial examples and input dimensions [2, 25, 30-32, 37, 38, 82, 84]. Apart from [38], 3 most studies have indicated that adversarial example threats increase with the square root of the input dimension [25, 30-32, 37, 82, 84]. Some studies have focused on specific data distributions or simple classifiers [30-32, 37, 82], while others targeted random networks [25, 84]. The study in [25] has examined the distance from the decision boundary in random networks, and concluded that adversarial examples deceive classifiers more easily with the square root of the input dimension. Another study has utilized the first-order Taylor

expansion of a loss function to analyze the loss gradient with respect to an input [84]. While a direct comparison between our work and [84] is challenging due to differing loss functions, our theorems offer two advantages. First, we considered both the input and output dimensions, whereas they addressed only input dimensions. Second, our theorems are applicable to residual networks, whereas their assumptions are not.

In this study, we examined the relationship between an input dimension and adversarial risk in Sec. 5.1. Our analysis did not depend on specific data distributions or architectures, except for ReLU-like activations and random parameters, which is advantageous. In addition, we assessed it for a wide variety of norms, demonstrating that the impact of adversarial examples is not limited to the square root of the input dimension alone (cf. Tab. 1). Moreover, our bound considers not only the input dimension but also the number of classes.

A.3 Adversarial training

Numerous empirical adversarial defenses have been proposed; however, most are ineffective against stronger attacks [5, 12, 13, 20, 90]. Some studies have focused on theoretically certified defenses [19, 39, 43, 74, 96], but these are often only applicable to specific or small networks, or are weaker than empirical methods. Adversarial training [38, 58] is considered the most effective empirical defense against various attacks [5, 20, 90]. This involves training a classifier using a dataset that contains natural images and adversarial examples [38] or solely adversarial examples [58]. Various forms of adversarial training exist, including more effective loss functions [28, 48, 58, 94, 113], time-efficient frameworks [83, 97, 111], and procedures that preserve the clean accuracy [72, 113, 114]. Recent studies have demonstrated that combining adversarial training and data augmentation with unlabeled or generated data results in high robustness [40, 77]. Despite significant progress in empirical methods, a theoretical understanding of adversarial training remains incomplete. We provide a summary of theoretical studies on adversarial training, including robust generalization research.

Several studies have investigated the trade-off between robust and clean accuracy [29, 47, 75, 76, 92, 113], initially observed empirically [58, 88]. The trade-off has been proven inevitable even in the infinite data limit, assuming data is constructed from a moderately correlated single feature and weakly correlated many features [92]. Similar claims were found in [113] for different data distributions. It has been reported that the trade-off in finite data settings for linear and slightly more complex predictors can be mitigated with additional unlabeled data [76]. While comparable results were reported in [75], a contrasting study also exists [47]. Moreover, the trade-off has been shown to depend on class imbalance in a dataset using Gaussian classification models [29].

Various studies have examined the generalization gap of adversarial robust models [6, 50, 102, 107]. For example, ResNet [42] trained adversarially on CIFAR-10 [52] achieved 96% robust training accuracy but only 47% robust test accuracy [107]. The lower bound of adversarial Rademacher complexity [9] has been shown to increase with the square root of the input dimension for linear classifiers trained with ℓ ∞ adversarial examples [107], which was later extended in [6]. Similar results using a tree transformation approach have been reported in [50]. However, the influence of network width and depth, or other training settings, such as training with ℓ 1 adversarial examples, on the bound remains unclear. The generalization gap has been investigated for linear regression models and two-layer neural networks with lazy training in data interpolation contexts [102].

Numerous studies have explored the sample complexity of robust generalization [1, 14, 62, 80, 110]. Robust learning may require a larger sample size than standard learning [80]. Subsequent research has indicated that unlabeled data can be sufficient to achieve robust generalization [1, 14, 62, 110]. The majority of these studies have focused on data sampled from Gaussian mixture models [1, 14, 110]. Moreover, the sample complexity of distributionally robust learning with perturbations in the Wasserstein ball has been examined [62], and some studies have suggested that the trade-off between robustness and accuracy can be mitigated using additional unlabeled data [75, 76].

Recent findings have suggested that robust classification requires complex classifiers [63], which is supported by the results in the neural tangent kernel regime [35]. In the context of transfer learning, robust classifiers have been shown to perform better [27]. Furthermore, a certifiable adversarial training procedure has been established, constraining perturbation by the distributional Wasserstein distance [86].

The aforementioned outcomes rely on specific data distributions, such as Gaussian or heuristic-tuned distributions, or simplistic models, such as linear classifiers or two-layer neural networks. The lack of theoretical research on adversarial training in deep neural networks stems from the complexity of training these models, including non-convex loss surfaces and stochastic optimization. Recent research has employed the neural tangent kernel regime to address these challenges, demonstrating that adversarial training can yield a robust network with near-zero robust loss [35]. This result was later extended in two-layer neural networks [115], eliminating the assumption in [35] that requires exponentially large width and runtime.

In this study, we conducted a theoretical analysis of adversarial training, focusing on the time evolution of network parameters during training, the conditions promoting adversarial training, and the differences between adversarial and standard training. Our analysis targeted deep neural networks without relying on any assumptions about data distribution and employed ℓ p norms practically used as perturbation constraints, rather than more impractical metrics such as the distributional Wasserstein distance. To address the training difficulty of deep neural networks, we utilized mean field theory.

Finally, we mention the report from a perturbation instability perspective that increasing network width does not necessarily improve robustness in adversarial training [98]. This may seem to contradict our results, which suggest that a wider network can help the model maintain capacity during adversarial training, implying greater robustness in wider networks. However, these two claims are compatible. Robustness is determined by both perturbation instability (negative effect) and network capacity (positive effect). While the negative effect of width appears dominant in [98]'s experiments on CIFAR-10 and WideResNet, the positive effect appeared more prevalent in our experiments on MNIST, Fashion-MNIST, and fully connected networks with or without shortcuts. The dominant factor may depend on the dataset and model architectures.

B Gradient independence assumption

The gradient independence assumption was first introduced in [81] for backward dynamics in mean field theory and later refined in [105]. We provide a definition based on [105] as follows:

Assumption B.1 (Gradient independence assumption [105]) . (a) We use a different set of weights for backpropagation than those used to compute the network outputs, but sampled i.i.d. from the same distributions. (b) For any loss L , the gradient at layer l , ∂ L /∂ x ( l ) , is independent of h ( l ) and x ( l -1) .

Although not strictly accurate, this assumption has been empirically found to hold well [81, 105]. It has been rigorously justified for specific architectures, including vanilla, residual, and convolutional networks [103]. In this study, we applied the assumption to the standard loss function but not to the adversarial loss function. In addition, we regard a network output as a loss and apply Asm B.1 to the network output in Sec. 5.4.

C Setting of residual networks

The mean field theory for residual networks is studied in [105]. Although they employed trainable weights in shortcuts, we employ the untrainable matrix. Formally, the pre- and post-activations in the l -th layer of a residual network are defined as follows:

where P ( l ) ∈ R N × N is an untrained random matrix. Each entry of P ( l ) is i.i.d. sampled from a Gaussian N (0 , 1 /N ) at initialization. The random matrix P ( l ) is introduced for simplifying probabilistic calculations and is applied in accordance with Asm B.1(a). The definition of x ( l ) , given in Eq. (A16), is slightly different from the original definition in [105]. Therefore, we will derive the basic probabilistic properties of pre- and post-activation again, based on Eq. (A16). Following a similar approach to [71, 105], the mean squared pre- and post-activation can be calculated as follows:

Additionally, following [81, 105], the mean squared gradient with respect to pre- and post-activation can be calculated as follows:

We can derive Eq. (A18) under Asm B.1 as follows:

We can derive Eq. (A19) under Asm B.1 as follows.

From Eq. (A23) to Eq. (A24), we used the following equation:

The second term of Eq. (A24) is rearranged as follows:

The third term of Eq. (A24) is rearranged as follows:

D Sketch of proof for Thm 4.1

In this section, we provide a plain but informal proof of Thm 4.1 for the stepping stone to the formal proof. We present two levels of explanation: the most straightforward one and the other that is closer to a formal proof.

First, we introduce the simplest proof of counterintuitive independence of the distribution of J ( x in ) from x in . As indicated in Eq. (6), the definition of J ( x in ) includes ϕ ′ ( h ( l ) ( x in )) ; thus, the distribution appears to depend x in . Here, we consider the distribution of ϕ ′ ( h ( x )) , where h ( x ) := wx with w ∼ N (0 , σ 2 w ) and x ∈ R . From the following proposition, although ϕ ′ ( h ( x )) is defined as a function of x , its distribution is independent of x . This characteristic holds even for J ( x in ) , which encompasses multiple ϕ ′ ( h ( l ) ( x in )) .

Proposition D.1. Let ϕ ( z ) := uz ( z ≥ 0); vz ( z < 0) be a ReLU-like function, w ∼ N (0 , σ 2 w ) be a Gaussian variable, and x ∈ R be a fixed real number. Then, the distribution of ϕ ′ ( wx ) is independent of x .

Proof. Consider the derivative of ϕ , given by ϕ ′ ( z ) := u ( z ≥ 0); v ( z < 0) . The input to ϕ ′ , i.e., wx , follows a Gaussian N (0 , x 2 σ 2 w ) . The probability of a zero-mean Gaussian being greater than or equal to zero is the same as it being less than zero, regardless of the value of x . Therefore, for any given x , the probabilities of ϕ ′ ( wx ) = u and ϕ ′ ( wx ) = v are invariant to x . Consequently, the claim is established.

Then, let us consider Thm 4.1 in a neural network with one activation and one weight layer, respectively, as f ( x ) := P out ϕ ( W (1) P in x in ) . In this setting, the network Jacobian is represented by J ( x in ) := P out D ( ϕ ′ ( W (1) P in x in )) W (1) P in . Moreover, we assume that the uncorrelated Gaussian variables are independent. This assumption is incorrect . Uncorrelated Gaussian variables are not necessarily independent (cf. Remark E.3). However, for the intuition about the formal proof, we assume this. The simplified Thm 4.1 and its proof are as follows:

Proposition D.2. Consider a neural network f ( x ) := P out ϕ ( W (1) P in x in ) . Suppose that the width N is sufficiently large. Assume that uncorrelated Gaussian variables are independent. Then, for any x in ∈ R d , each entry of J ( x in ) := P out D ( ϕ ′ ( W (1) P in x in )) W (1) P in is i.i.d. and follows the Gaussian N (0 , ασ 2 w /d ) .

Proof. First, we prove that each entry of J ( x in ) is i.i.d. and follows a Gaussian. To this end, we consider the probabilistic properties in the order of P in x in , D ( ϕ ′ ( W (1) P in x in )) , W (1) P in , and P out D ( ϕ ′ ( W (1) P in x in )) W (1) P in . Then, we derive the concrete distribution, i.e., its mean and variance. Comparing the distributions between f ( x in ) 2 and ( J ( x in ) x in ) 2 , we achieve this.

We first consider P in x in =: x (0) . Recall that x in is a deterministic vector and P in is a random matrix where each entry is i.i.d. and follows a Gaussian. Since the weighted sum of independent Gaussian variables follows a Gaussian, each entry of x (0) is i.i.d. and follows a Gaussian.

̸

Then, let us consider W (1) x (0) . The i -th entry of W (1) x (0) is expressed as ∑ N j =1 W (1) ij x (0) j . Since W (1) is a random matrix with i.i.d. entries, W (1) ij x (0) j is i.i.d. with respect to j . By the central limit theorem with infinite N , the i -th entry of W (1) x (0) follows a Gaussian. In addition, the entries of W (1) x (0) follow the same Gaussian. Moreover, since E [( ∑ N j =1 W (1) ij x (0) j )( ∑ N j =1 W (1) i ′ j x (0) j )] = 0 holds for i = i ′ , different entries of W (1) x (0) are uncorrelated and thus independent (this is originally incorrect but guaranteed by the assumption here). In conclusion, W (1) x (0) is a random vector with i.i.d. Gaussian entries. Naturally, D ( ϕ ′ ( W (1) P in x in )) is a diagonal matrix with i.i.d. entries.

Similar to the discussion above, W (1) P in =: A is a random matrix with i.i.d. entries.

̸

Finally, consider P out D ( ϕ ′ ( W (1) P in x in )) W (1) P in . For notational simplicity, we denote D := D ( ϕ ′ ( W (1) P in x in )) . Thus, P out D ( ϕ ′ ( W (1) P in x in )) W (1) P in = P out DA . The entry of i -th row and j -th column of this matrix is expressed as ∑ N k =1 P out ik D k A kj . Note that P out is independent of DA and its entries are i.i.d. Moreover, based on network symmetry, the dependence between D k and A kj is invariant to k . Thus, P out ik D k A kj is i.i.d. with respect to k and each entry of this matrix follows the same Gaussian by the central limit theorem. Moreover, since E [( ∑ N k =1 P out ik D k A kj )( ∑ N k =1 P out i ′ k D k A kj ′ )] = 0 holds for ( i, j ) = ( i ′ , j ′ ) , different entries of P out DA are uncorrelated and thus independent. Therefore, each entry of J ( x in ) is i.i.d. and follows the Gaussian.

Next, we derive the mean and variance of J ( x in ) . Trivially, E [ J ( x in ) i ] = 0 since the mean of an entry of P out is zero. We derive the variance by comparing E [ f ( x ) 2 i ] and E [( ∑ d j =1 J ( x in ) ij x in j ) 2 ] . Recall f ( x ) i := ∑ d j =1 J ( x in ) ij x in j . As a preliminary, with a Gaussian variable z ∼ N (0 , σ 2 ) and its probabilistic density function g ( z ) , we derive the following equation:

In addition, if P out and z are independent, then

As shown in the discussion above, each entry of W (1) x (0) follows the Gaussian. Moreover, we can simply derive its variance as σ 2 w ∥ x in ∥ 2 /d . Thus,

Since f ( x ) i := ∑ d j =1 J ( x in ) ij x in j , E [ f ( x ) 2 i ] is also expanded as:

Comparing the two equations above, we obtain E [ J ( x in ) 2 ij ] = ασ 2 w /d . Therefore, the claim is established.

E Derivation of Thm 4.1 for vanilla networks

E.1 Preliminary

First, we introduce some lemmas. These results are more general and not restricted to the context of neural networks. The notation in this section is independent of that in other sections.

We refer to a random matrix where the mean of an entry is zero as a zero-mean matrix. Formally, this is defined as follows:

Definition E.1 (zero-mean matrix/vector) . A random matrix/vector A is called a zero-mean matrix/vector if E [ A ij ] = 0 for any i and j .

For zero-mean matrices, the following properties hold:

Remark E.2 (Properties of zero-mean matrix) . Let A be a zero-mean matrix.

Let B be a zero-mean matrix. Then, their sum, A + B , is a zero-mean matrix.

Let B be a random matrix. If A and B are independent, then their products, AB and BA , are zero-mean matrices.

Then, we review several properties of Gaussian variables.

Remark E.3 (Properties of Gaussian variables) . The following statements hold:

Any linear combination of multiple Gaussian variables follows a Gaussian if and only if they are jointly distributed Gaussian.

A weighted sum of independent Gaussian variables is Gaussian distributed.

Gaussian variables are jointly distributed and uncorrelated if and only if they are independent.

Next, we examine the properties of random matrices/vectors with i.i.d. Gaussian entries, called a Gaussian matrix/vector. Their formal definitions are as follows:

Definition E.4 (Gaussian matrix/vector) . A matrix/vector is called a Gaussian matrix/vector if all the entries are i.i.d. and follow a Gaussian.

Gaussian matrices and vectors satisfy the following lemmas.

Lemma E.5. If A and B are independent Gaussian matrices, then A + B is a Gaussian matrix. In particular, when A and B are zero-mean, A + B is a zero-mean Gaussian matrix.

Proof.

The following lemmas are derived for the proof of Thm 4.1.

̸

Lemma E.6. Suppose that random matrices A ∈ R m × n and B ∈ R n × l satisfy the following properties: (a) A and B are independent. (b) A is a zero-mean matrix. (c) All the entries of A are i.i.d. (d) All the entries of B are identically distributed. (e) Any two entries of B from different rows are independent. (f) For any i ∈ [ n ] , j, k ∈ [ l ] , j = k , the dependence between B ij and B ik is invariant to i, j, k . (g) For any i ∈ [ n ] , j, k ∈ [ l ] , j = k , E [ B ij B ik ] = 0 holds. Then, for a sufficiently large n , AB is a zero-mean Gaussian matrix.

̸

Proof. By (a), (b), and Remark E.2, AB is a zero-mean matrix. The linear combination of all the entries of AB is expressed as

where s ij ∈ R is a constant. By (a), (c), (d), (e), and (f), ∑ m i =1 ∑ l j =1 s ij A ik B kj is i.i.d. with respect to k . By the central limit theorem, c follows a Gaussian. By Remark E.3, all the entries of AB are jointly Gaussian distributed. Moreover, by (c) and (d), the entries of AB follow the same Gaussian. By (a), (b), (c), (e), and (g), the covariance between any two entries of AB is zero. Thus, by Remark E.3, the claim is established.

E.3 Main derivation

First, we remark several fundamental properties of weights, biases, random projections, and preactivations in a network.

Remark E.9 . By definition in Sec. 3.1, the following statements hold for any l ∈ [ L ] :

̸

Lemma E.7. Suppose that random matrices A ∈ R m × n , B ∈ R n × l 1 , and C ∈ R n × l 2 satisfy the following properties: (a) A , B , and C are independent. (b) A is a zero-mean matrix. (c) All the entries of A are i.i.d. (d) All the entries of B are identically distributed. (e) All the entries of C are identically distributed. (f) Any two entries of B from different rows are independent. (g) Any two entries of C from different rows are independent. (h) For any i ∈ [ n ] , j, k ∈ [ l 1 ] , j = k , the dependence between B ij and B ik is invariant to i, j, k . (i) For any i ∈ [ n ] , j, k ∈ [ l 2 ] , j = k , the dependence between C ij and C ik is invariant to i, j, k . (j) For any i ∈ [ n ] , j ∈ [ l 1 ] , k ∈ [ l 2 ] , E [ B ij C ik ] = 0 holds. Then, for a sufficiently large n , AB and AC are independent.

̸

Proof. By (a), (c), (d), (e), (f), (g), (h), and (i), similar to Lemma E.6, all the entries of AB and AC are jointly Gaussian distributed. By (a), (b), (c), and (j), the covariance between any two entries from AB and AC is zero. Thus, by Remark E.3, the claim is established.

Lemma E.8. Let w , x ∈ R m be Gaussian vectors. Let ϕ ′ ( z ) := u ( z ≥ 0); v ( z < 0) . Then, for a sufficiently large m , ϕ ′ ( w ⊤ x ) w and x are independent.

Proof. Weprove P [ ϕ ′ ( w ⊤ x ) w = a | x = b ] = P [ ϕ ′ ( w ⊤ x ) w = a ] . For any x , w = a /u + x / √ m or w = a /v -x / √ m satisfy ϕ ′ ( w ⊤ x ) w = a . Since m is sufficiently large, w = a /u + x / √ m and w = a /v -x / √ m asymptotically approach w = a /u and w = a /v , respectively. Thus, the claim is established.

E.2 Definitions of J ( x in ) and a ( x in ) for vanilla networks

Here, we define (derive) J ( x in ) and a ( x in ) for vanilla networks. The l -th pre- and post-activation are defined as follows:

Using ϕ ( h ( l ) ( x in )) = D ( ϕ ′ ( h ( l ) ( x in ))) h ( l ) ( x in ) , the l -th pre-activation can be rearranged as follows:

Because ReLU-like networks are piecewise linear, the l -th pre-activation is also represented as follows:

Substituting Eq. (A44) for Eq. (A43), the equation can be rearranged as follows:

Comparing Eq. (A44) and Eq. (A45), the following equations can be derived:

where J (1) ( x in ) := W (1) and a (1) ( x in ) := b (1) . Finally, J ( x in ) and a ( x in ) are defined as follows:

(Gaussian matrix/vector) W ( l ) , P in , and P out are zero-mean Gaussian matrices and b ( l ) is a zero-mean Gaussian vector.

(Variance) The variance of W ( l ) , P in , and P out is σ 2 w /N , 1 /d , and 1 /N , respectively.

(Independence) W ( l ) and b ( l ) are independent of each other, as is any random variable in the layer before the l -th. In addition, P out is independent of any random variable without a network output.

(Dependence between pre-activation and bias) For i = j , the dependence between h ( l ) i and b ( l ) i is invariant to i . For i = j , h ( l ) i and b ( l ) j are independent.

Lemma E.10. Consider a vanilla network. For any l ∈ [ L ] , the following hold:

̸

(a) J ( l ) ( x in ) is a zero-mean Gaussian matrix.

(b) a ( l ) ( x in ) is a zero-mean Gaussian vector.

(c) J ( l ) ( x in ) and a ( l ) ( x in ) are independent.

Proof. We prove the claim by induction. As a preliminary, see Remark E.9. The case with l = 1 is trivial. Suppose that all the claims hold for l -1 ∈ [ L -1] .

(a) By Lemma E.6, the claim is established. Note that E [ ϕ ′ ( h ( l ) i ) 2 J ( l ) ( x in ) ij J ( l ) ( x in ) ik ] = E [ ϕ ′ ( h ( l ) i ) 2 ] E [ J ( l ) ( x in ) ij ] E [ J ( l ) ( x in ) ik ] = 0 since N is sufficiently large.

(b) By Lemma E.6, the claim is established.

(c) By Lemma E.7, the claim is established.

Lemma E.11. The mean squared network output is given by:

Proof. By Remark E.9, the mean squared pre- and post- activation are calculated as follows (cf. Eq. (A34)):

Recursively calculating Eqs. (A50) and (A51), the mean squared post-activation in a vanilla network is calculated as follows:

The mean squared post-activation in a residual network is calculated as follows:

Using x (0) := P in x in , the above results are expanded as follows:

The mean squared network output is rearranged to the mean squared L -th post-activation as follows:

By Eqs. (A56) and (A57), the claim is established.

Theorem 4.1 (Properties and distributions of J ( x in ) and a ( x in ) ) . Suppose that the width N is sufficiently large. Then, for any x in ∈ R d , (I) J ( x in ) and a ( x in ) are independent. (II) each entry of J ( x in ) and a ( x in ) is i.i.d. and follows the Gaussian below:

where α := ( u 2 + v 2 ) / 2 (cf. Defn 3.1) and ω is ω v := ασ 2 w for vanilla networks and ω r := 1 + ασ 2 w for residual networks.

Proof. Similar to Lemma E.8, P in and a ( x in ) are independent. Similar to Lemmas E.10 and F.2, J ( x in ) and a ( x in ) are a zero-mean Gaussian matrix and vector, respectively, and J ( x in ) and a ( x in ) are independent. The mean squared network output can be expanded as follows:

Using the symmetry of entries of J ( x in ) and a ( x in ) , the equation can be simplified as follows:

Comparing Lemma E.11, Eq. (7) is obtained. Thus, the claim is established.

F Derivation of Thm 4.1 for residual networks

F.1 Definitions of J ( x in ) and a ( x in ) for residual networks

Similar to Appx. E.2, we define (derive) J ( x in ) and a ( x in ) for residual networks. For notational simplicity, we omit the argument x in . First, we represent x ( l ) as follows:

where V (0) := 0 and c (0) := 0 . With V ( l ) and c ( l ) , we can represent h ( l ) as follows:

Then, we derive recurrence representations of V ( l ) and c ( l ) as follows:

For reference, we denote

With the notations above,

Thus,

Finally, J ( x in ) and a ( x in ) are defined as follows:

F.2 Main derivation

First, we note the properties of random projections in shortcuts.

Remark F.1 . By definition (cf. Appx. C), the following statements hold for any l ∈ [ L ] :

(Gaussian matrix/vector) P ( l ) is a zero-mean Gaussian matrix.

(Variance) The variance of P ( l ) is 1 /N .

(Independence) P ( l ) is independent of W ( l ) , b ( l ) , and h ( l ) , as is any random variable in the layer before the l -th.

Finally, we introduce a lemma similar to Lemma E.10. A subsequent discussion to derive Thm 4.1 is the same as the proof in Appx. E.3.

Lemma F.2. Consider a residual network. For any l ∈ [ L ] , the following statements hold:

(a) J ( l ) ( x in ) is a zero-mean Gaussian matrix.

(b) a ( l ) ( x in ) is a zero-mean Gaussian vector.

(c) J ( l ) ( x in ) and a ( l ) ( x in ) are independent.

Proof. As a preliminary, please refer to Remarks E.9 and F.1 and Lemma E.10.

̸

Note that effects of all the random variables in the layer before the l -th, such as P ( l -1) , h ( l -1) , W ( l -1) , and b ( l -1) , are aggregated to x ( l -1) . By Lemma E.8, U ( l ) and d ( l ) are independent of U ( l ′ ) and d ( l ′ ) for any l ′ = l . Similarly, as V ( l ) and c ( l ) consist of U (1) , . . . , U ( l ) , d (1) , . . . , d ( l ) , U ( l +1) and d ( l +1) are independent of V ( l ) and c ( l ) . In addition, similar to Lemma E.10, U ( l ) and d ( l ) are independent by Lemma E.7.

We prove the claim by induction. Assume that V ( l -1) is a Gaussian matrix. This holds for l ′ = 1 because of V (1) = U (1) . Similar to Lemma E.8, U ij is independent of U i · V · j . Since U ( l ) and V ( l -1) are independent Gaussian matrices and N is sufficiently large, V ( l ) is a Gaussian matrices. Similarly, c ( l ) is a Gaussian vector. By Lemma E.7, V ( l ) and c ( l ) are independent. Thus, similar to Lemma E.10, J ( x in ) and a ( x in ) are independent Gaussian matrices.

G Derivation of the theorems in Sec. 5

G.1 Derivation of the theorems in Sec. 5.1

First, we introduce the following lemma, which is required to derive the maximum ℓ ∞ norm of a column of J ( x in ) .

Lemma G.1. The maximum absolute value of n ∈ N i.i.d. Gaussian variables with zero mean and variance σ 2 > 0 is smaller than √ 2 σ 2 ln n .

∞

Proof. Let z ∈ R be a Gaussian variable with zero mean and variance σ 2 and a > 0 be a positive value. First, we compute the upper bound of probability of z > a as follows:

Then, we compute the lower bound of probability of | z | < a as follows:

Let { z i } n i =1 be n i.i.d. Gaussian variables. Finally, we compute the upper bound of probability of max i ∈ [ n ] | z i | > a as follows:

In particular, when a = √ 2 σ 2 ln n ,

Thus, the claim is established.

Theorem 5.1 (Upper bounds of adversarial loss) . Suppose that the input dimension d , output dimension K , and width N are sufficiently large. Then, for any x in ∈ R d , the following inequality holds:

where W := { W (1) 1 , 1 , W (1) 1 , 2 , . . . , W ( L ) N,N } denotes the set of all network weights. The constant β p,q for each norm pair ( p, q ) is described in Tab. 1.

Proof. We note the following:

By Thm 4.1, each entry of J ( x in ) is i.i.d. and follows a Gaussian with zero mean and variance V [ J ( x in ) ij ] = ω L /d .

The input and output dimensions, d and K , are sufficiently large (cf. Sec. 3.1).

For some combinations of ( p, q ) , ( p, q ) -operator norms are computable (cf. Tab. A3) [91].

In addition, we note the following upper bound (cf. Sec. 5.1):

As a preliminary, we compute the mean absolute value of J ( x in ) ij as follows:

In the above derivation, we use the following equation

Based on Tab. A3, we compute ∥ J ( x in ) ∥ p,q as follows:

Maximum ℓ 1 norm of a column, ∥ J ( x in ) ∥ 1 , 1 .

Maximum ℓ 2 norm of a column, J ( x ) 1 , 2

Maximum ℓ ∞ norm of a column, ∥ J ( x in ) ∥ 1 , ∞ . By Lemma G.1,

Maximum singular value, ∥ J ( x in ) ∥ 2 , 2 . By the Marchenko-Pastur law,

where ∥ · ∥ 2 denotes the spectral norm (largest singular value).

Maximum ℓ 2 norm of a row, ∥ J ( x in ) ∥ 2 , ∞ .

Maximum ℓ 1 norm of a row, ∥ J ( x in ) ∥ ∞ , ∞ .

In addition, we try to derive equalities rather than inequalities (upper bounds). First, we note the following equation:

Because J ( x in ) and a ( x in ) are the slope and bias of a piecewise linear region, respectively, for sufficiently small η , J ( x in + η ) and a ( x in + η ) are identical to J ( x in ) and a ( x in ) , respectively. Therefore, for sufficiently small η , we can derive the following equation:

Proposition G.2. Suppose that the perturbation constraint ϵ is sufficiently small. For any x in ∈ R d and ( p, q ) = (1 , 1) , (1 , 2) , (2 , ∞ ) , and ( ∞ , ∞ ) , the following equality holds:

Proof. As shown in Eq. (A107), the adversarial loss (Eq. (3)) is equal to the ( p, q ) -operator norm of J ( x in ) under this assumption. For ( p, q ) = (1 , 1) , (1 , 2) , (2 , ∞ ) , and ( ∞ , ∞ ) , we can obtain the equalities of the ( p, q ) -operator norms (cf. the proof of Thm 5.1). Thus, the claim is established. Note that, for ( p, q ) = (1 , ∞ ) and (2 , 2) , we can derive only upper bounds (cf. the proof of Thm 5.1).

Proposition G.3. Suppose that the perturbation constraint ϵ is sufficiently small and the output dimension is one. For any x in ∈ R d , and ( p, q ) = (2 , ∞ ) and ( ∞ , ∞ ) , the following equality holds:

In addition, for any x in ∈ R d and ( p, q ) = (2 , 2) , the following equality holds:

Proof. Similar to Prop G.2, the upper bounds for ( p, q ) = (2 , ∞ ) and ( ∞ , ∞ ) become equalities. The upper bounds for ( p, q ) = (1 , 1) and (1 , 2) require sufficiently large K , and thus, they do not become equalities. In addition, as Tab. A3 shows, the (2 , 2) -operator norm is a maximum singular value. When K = 1 , the maximum singular value is identical to the Frobenius norm ( ℓ 2 norm), which is calculated as follows:

Thus, in contrast to the proof of Thm 5.1, we can obtain the equality of the (2 , 2) -operator norm instead of the inequality (upper bound).

G.2 Derivation of the theorems in Sec. 5.2

In the following, we set L adv := ϵβ p,q ω L/ 2 . As a preliminary, we state the following two lemmas. These are used to solve the differential equation derived from gradient flow.

Lemma G.4. For t, a, b ∈ R , x : R → R , and x (0) := x 0 , the following holds:

Proof.

By the initial value,

Thus,

Lemma G.5. For t, c ∈ R , a, b ≥ 0 , x : R → R , x (0) := x 0 , and 0 ≤ bx (0) ≪ 1 , the following holds:

Proof. By -a (1 + bx ( t )) c x ( t ) ≤ 0 , x ( t ) does not increase with t . That is, bx ( t ) ≤ bx (0) ≪ 1 . Therefore, we can approximate (1 + bx ( t )) c as 1 + bcx ( t ) using the binomial theorem. Thus, we try to solve the following differential equation:

whose solution is as follows:

By the initial value,

Thus,

Thus,

Then, using the update equation of parameters (Eq. (10)), we consider the differential equation of weight variance as follows:

Lemma G.6. Suppose that Asm 5.2 holds. Let L 1 , . . . , L n : R → R be the n loss functions and W := { W (1) 11 , W (1) 12 , . . . , W ( L ) NN } be the set of all network weights. A network is trained by minimizing the sum of loss functions, ∑ n i =1 L i ( x in ) . The network parameters are updated similarly to Eq. (10) . The differential equation of σ 2 w ( t ) is given by:

Proof. Since Asm 5.2 holds and the number of weights, LN 2 , is sufficiently large, σ 2 w ( t +d t ) can be represented as follows:

We can expand W ( t +d t ) using Eq. (10) and rearrange the above equation as follows:

Lemma G.7. For L adv := ϵβ p,q ω L/ 2 , the following equality holds:

Proof. If the number of network weights, i.e., LN 2 , is sufficiently large, we can represent the weight variance as follows:

The derivative of ω with respect to W ∈ W can be calculated as follows:

The derivative of ω L/ 2 with respect to W ∈ W can be calculated as follows:

Thus,

Lemma G.8. Suppose that Asm 5.2 holds. Let L std : R d → R be the standard loss function and L adv : R d → R be the adversarial loss function. Suppose that Asm B.1 applies to L std . The adversarial loss function is defined as L adv ( x in ; t ) := ϵβ p,q ω ( t ) L/ 2 . A network is trained by minimizing L std + L adv . Network parameters are updated by Eq. (10) . Then, the differential equation of σ 2 w ( t ) is given by:

Since the standard loss satisfies Asm B.1, E [ W ( t ) ∂ L std ∂W ( t ) ] = 0 and

Theorem 5.4 (Weight time evolution of vanilla network in adversarial training) . Suppose that Asms 5.2 and 5.3 hold. Then, the time evolution of σ 2 w of a vanilla network in adversarial training is given by:

Proof. By Lemma G.8, the time evolution of weight variance in a vanilla network is represented as follows:

Denote L ′ := L/ 2 -1 . By Lemma G.4, we can obtain

By t ≤ T ≪ N ,

Theorem G.9 (Weight time evolution of a residual network in adversarial training) . Suppose that Asm 5.2 holds and ασ 2 w (0) ≪ 1 . The time evolution of σ 2 w of a residual network in adversarial training is given by:

Proof. By Lemma G.8, the time evolution of weight variance in a residual network is represented as follows:

By the Maclaurin expansion of the exponential function and t ≤ T ≪ N ,

By t ≤ T ≪ N ,

Then, we consider the time evolution of weight variance, Thms 5.4 and G.9, at initialization satisfying ( M,m ) -trainability condition (Lemma 5.6). Here, we assume ω (0) ≈ 1 satisfying Lemma 5.6. Under this assumption, the time evolution of weight variance in vanilla networks, Eq. (11), can be rearranged as follows:

In addition, the time evolution of weight variance in residual networks, Eq. (A160), can be rearranged as follows:

Thus, the time evolution of weight variance is consistent in vanilla and residual networks at initialization satisfying Lemma 5.6.

G.3 Derivation of the theorems in Sec. 5.3

Lemma 5.6 (Vanilla and residual ( M,m ) -trainability condition) . Suppose that the width N is sufficiently large. Then, the ( M,m ) -trainability conditions for vanilla and residual networks are respectively given by:

Proof. Applying Eq. (A38) to Eqs. (2) and (A19), we can obtain the following equations:

Then, recursively computing the above equation, we can derive the following equations:

Applying this equation to Defn 5.5,

For a vanilla network,

For a residual network,

In particular, for a residual network, by m 1 /L -1 ≤ 0 and ασ 2 w ≥ 0 ,

Theorem 5.7 (Vanilla networks are not adversarially trainable) . Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, and the ( M,m ) -trainability condition holds at t = 0 and ασ 2 w (0) = 1 . If

then there exists 0 < τ ≤ T such that the ( M,m ) -trainability condition does not hold for τ ≤ t ≤ T .

Proof. Let us consider T breaking the ( M,m ) -trainability condition. As Thm 5.4 claims, σ 2 w ( t ) decreases monotonically with t . Thus, we consider T such that σ 2 w ( T ) is less than the lower bound.

Theorem 5.8 (Residual networks are adversarially trainable) . Consider a residual network. Suppose that Asms 5.2 and 5.3 hold, and the ( M,m ) -trainability condition holds at t = 0 and ασ 2 w (0) ≪ 1 . Then, ( M,m ) -trainability condition always holds for 0 ≤ t ≤ T .

Proof. By Thm G.9, σ 2 w ( t ) decreases monotonically. Thus, the following inequality holds, and, by Lemma 5.6, the ( M,m ) -trainability condition always holds in 0 ≤ t ≤ T :

Proposition G.10 (Residual networks are adversarially trainable without careful weight initialization) . Suppose that Asm 5.2 holds, the network is residual, and the ( M,m ) -trainability condition is not satisfied at t = 0 . If

there exists 0 < τ ≤ T such that the ( M,m ) -trainability condition hold for τ ≤ t ≤ T .

Proof. Denote the time evolution of the weight variance on a vanilla network by σ 2 w, v and the variance on a residual network by σ 2 w, r . By Eq. (A161),

Recall that the following differential equation represents the time evolution of weight variance on a vanilla network (cf. Thm 5.4):

From the above, σ 2 w, r ( t ) ≤ σ 2 w, v ( t ) if σ 2 w (0) is the same. Here, we consider T such that σ 2 w, v ( T ) is less than the upper bound of the ( M,m ) -trainability condition. By Thm 5.4,

G.4 Derivation of the theorems in Sec. 5.4

First, we state the following two lemmas as a preliminary.

Lemma G.11. Consider a vanilla network. Suppose that Asm B.1 is applied to the network output. For any l ∈ [ L ] , the mean squared gradient of f i with respect to h ( l ) j is given by:

Proof.

By Asm B.1 and Eq. (A38),

Moreover, by Asm B.1,

Thus,

Lemma G.12. Suppose that Asm B.1 is applied to the network output. For any l ∈ [ L ] , the mean squared gradient of f i with respect to a weight is given by:

Proof. Vanilla. By Asm B.1,

By Eq. (A38),

By Eqs. (A19) and (A38),

Now, we can simply represent the mean of the Fisher-Rao norm.

Lemma G.13. Suppose that Asm B.1 is applied to the network output. Suppose ∥ ∥ x in ∥ ∥ 2 = √ d and σ 2 b = 0 . The mean of the Fisher-Rao norm is given by:

Proof. The following discussion is based on [49]. We rearrange and expand their discussion for a ReLU-like network. We also consider it for a residual network. Note that w := ( W (1) 11 , . . . , W ( L ) NN ) ⊤ . As a preliminary, see Remark E.9. The Fisher-Rao norm is calculated as follows:

By Lemma G.12, E [( ∂f i /∂W ( l ) jk ) 2 ] is invariant to l if σ 2 b = 0 . In other words, E [( ∂f k /∂w i ) 2 ] does not depend on i if σ 2 b = 0 . Thus,

Moreover,

Thus,

Theorem G.14 (Adversarial training degrades network capacity) . Consider a residual network. Suppose that Asm 5.2 holds, Asm B.1 is applied to the network output, and the ( M,m ) -trainability condition holds at t = 0 , e.g., ασ 2 w (0) ≪ 1 . Suppose ∥ ∥ x in ∥ ∥ 2 = √ d and σ 2 b ( t ) = 0 . Then, the mean of the Fisher-Rao norm is given by:

≪

Theorem 5.9 (Adversarial training degrades network capacity) . Consider a vanilla network. Suppose that Asms 5.2 and 5.3 hold, Asm B.1 is applied to the network output, and the ( M,m ) -trainability condition holds at t = 0 and ασ 2 w (0) = 1 . Assume ∥ ∥ x in ∥ ∥ 2 = √ d and σ 2 b ( t ) = 0 . Then, the expectation of the Fisher-Rao norm is given by:

Temporally, denote A := (1 + ( L/ 2 -1) ασ 2 w (0)) ϵαβ p,q t/N . By Thm G.9,

By O (1 /N 2 ) ≈ 0 , O ( A 2 ) ≈ 0 . Thus,

H Comparison with matrix decomposition approaches

Here, we analyze Ineq. (8) with another approach, which decomposes J ( x in ) into products of matrices, and consider norm of each matrix. As related studies based on this approach, we cite the literature on certified adversarial defense [3, 18, 93] and spectral regularization [60, 108]. We consider the case of ( p, q ) = (2 , 2) for simplicity. In comparison to them, our approach is different in two ways. First, their bound is not tractable due to a deterministic approach. For example, they consider the spectral norm of J ( x in ) by decomposing J ( x in ) and computing the norm of each matrix consisting J ( x in ) as follows:

Because Eqs. (A232) and (A233) are hard to theoretically manage, we cannot derive some theorems in this study such as Thms 5.4, 5.7 and 5.9. Second, because we calculate the spectral norm of J ( x in ) directly in Thm 5.1 instead of considering the norm of each matrix and the spectral norm is submultiplicative, our upper bounds are always tighter than those derived from Eq. (A232).

Here, we proceed with the discussion of tightness of their bound using probabilistic theory. One approach is the rearrangement of Eq. (A232) using the Marchenko-Pastur law. As a preliminary, we note the following:

Using the above equations, Eq. (A232) can be rearranged as follows:

This is exponentially looser than Thm 5.1.

Another approach is the rearrangement of Eq. (A233) using the assumption that each matrix is sufficiently large. As a preliminary, we note the following:

Using the above equations, Eq. (A233) can be rearranged as follows:

This is also exponentially looser than Thm 5.1.

∥ ∥ 2 ≤ | | | | √ √ L | | | |

I Comparison with ℓ 2 weight regularization

As stated in Secs. 5.1 and 5.2, adversarial training serves the role of a weight regularizer. Here, we compare adversarial training with ℓ 2 weight regularization. First, we define ℓ 2 weight regularization as follows:

where λ > 0 is a scaling factor. Then, we determine the scaling factor λ based on the concept of gradient vanishing and explosion. The mean of L w ( W ; λ ) is calculated as follows:

To prevent (gradient) vanishing and explosion of E [ L w ( W ; λ )] under sufficiently large L and N , λ should be 1 / ( LN ) . Moreover, for simplicity of the derivation, we set λ := 1 / (2 LN ) . Finally, ℓ 2 regularized training tries to minimize the following loss function:

Next, we consider the time evolution of σ 2 w with ℓ 2 weight regularization. Similar to Thms 5.4 and G.9, we can derive the following proposition:

Proposition I.1 (Weight time evolution with ℓ 2 weight regularization) . Suppose that Asm 5.2 holds. Let L std be the standard loss function and L w ( W ) := 1 / (2 LN ) ∑ W ∈W W 2 be the ℓ 2 regularization loss function. Suppose that Asm B.1 applies to L std , but not to L w . A network is trained by minimizing L std + L w ( W ) . Then, the time evolution of σ 2 w is given by:

Proof. Similar to Lemma G.8,

Thus,

By t ≤ T ≪ N , using Maclaurin expansion of the exponential function,

Here, we consider the adversarial loss defined as L adv ( x in ) := ϵβ p,q ω L/ 2 (cf. Thm 5.1). The difference between adversarial training and ℓ 2 regularization lies in the scale: ϵαβ p,q /N in adversarial training (cf. Thm 5.4) and 1 / ( LN ) in ℓ 2 regularized training. Adversarial training with strong adversarial examples (e.g., ℓ ∞ constrained ones) reduces σ 2 w ( t ) more drastically than ℓ 2 regularized training. For example, with L = 100 , N = 1 , 000 , d = 3 × 224 × 224 , ϵ = 0 . 1 , α = 1 / 2 , p = ∞ , q = ∞ , then Θ( ϵαβ p,q /N ) = 10 -2 and Θ(1 / ( LN )) = 10 -5 .

Finally, we compare adversarial training and ℓ 2 regularization in terms of stability of gradient descent. The derivations of both losses with respect to the weight W ∈ W are given by (cf. Lemma G.7):

This indicates that the derivation of ℓ 2 regularizer is smooth across the entire weight space, whereas the derivation of adversarial loss changes significantly at the ω = 1 boundary. Note that ω evolves during training (cf. the definition in Thm 4.1). The steep derivation in adversarial training prevents the gradient descent from reaching a minimum.

J Revisiting trainability condition in adversarial training

Here, we revisit Defn 5.5. Recall that this condition can be derived exclusively under Asm B.1. This is because χ ( l ) can only be computed under Asm B.1 [81]. In this study, we assume that Asm B.1 applies to the standard loss L std , but not to the adversarial loss L adv . Consequently, the composite loss L := L std + L adv does not satisfy Asm B.1, implying that strictly speaking, Defn 5.5 is not valid for L and considering Defn 5.5 in adversarial training might not be entirely accurate.

However, we must emphasize that Defn 5.5 continues to be a determining factor in the success of adversarial training, or in other words, training with L . This is primarily due to the fact that if Defn 5.5 is not met, networks will be unable to adequately fit the training dataset owing to gradient vanishing or explosion.

Examining this from a mathematical standpoint, the necessity of Defn 5.5 is underlined by the linearity of the differential operator. The update equation for the parameter θ ( t ) is formulated as follows:

As demonstrated by the equation above, updates with L std and L adv are conducted independently. Although Defn 5.5 may not be considered for L adv , it can be for L std . In order to forestall gradient vanishing or explosion in the standard loss and to ensure classification ability for clean images, Defn 5.5 remains a necessity. Therefore, it can be concluded that Defn 5.5 continues to dictate the success of training, even within the context of adversarial training.

K Other theoretical results

Scaled effect of input and output dimensions. In Tab. 1, we showed a wide effect of the input dimension d and the output dimension K on the upper bounds of the adversarial loss. In practice, the perturbation budget ϵ and the adversarial loss are scaled based on the chosen p and q . For example, we can rearrange the definition of the adversarial loss (Eq. (3)) as follows:

where λ p and λ q denote the scaling factors. In this context, the upper bound (Thm 5.1) can be rearranged as follows:

where β ′ p,q := λ p λ q β p,q . For example, we consider the following λ p and λ q :

Under these conditions, β ′ p,q corresponds to Tab. A4. This table reveals that (i) the input dimension impacts the upper bounds with Θ( √ d ) , but the output dimension has little or no effects; (ii) although the dimensions generally exert a similar influence on the bounds, the specific factors vary widely; (iii) for ( p, q ) = (1 , ∞ ) and (2 , 2) , the output dimension behaves in a distinctive manner. In conclusion, our upper bounds, even inclusive of scaling effects, provide several insights into the relationship between the adversarial loss and input/output dimensions.

Other metrics of capacity. Here, we consider capacity metrics other than the Fisher-Rao norm. Norm-based metrics, such as the path norm [66], the group norm [67], and spectral norm [8], have a similar definition to the Fisher-Rao norm. They are constructed by the sum and product of the weights of a network. The trajectory length [73] is defined by the arc length of a network output with change of parameters. A similar definition based on curvature was also used in [71]. These metrics concluded that capacity increases with the weight variance of a network, despite differences in speed. Therefore, we can derive similar results as in Sec. 5.4 for metrics other than the Fisher-Rao norm.

Mitigation of adversarial risk. Here, we consider the mitigation of adversarial risk, i.e., the mitigation of the upper bounds of the adversarial loss. There are three solutions: (i) sample each entry of P in from N (0 , 1 /d 2 ) instead of N (0 , 1 /d ) ; (ii) sample each entry of W ( l ) from N (0 , σ 2 w /N 2 ) instead of N (0 , σ 2 w /N ) for l ∈ [ L ] ; (iii) sample each entry of P out from N (0 , 1 /N 2 ) instead of N (0 , 1 /N ) .

First, let us examine scenario (i). In this setting, the variance of J ( x in ) is transformed into V [ J ( x in ) ij ] = ω L /d 2 from V [ J ( x in ) ij ] = ω L /d (cf. the proof of Thm 4.1). In addition, β ∞ , ∞ changes to Θ( β ∞ , ∞ ) = 1 from Θ( β ∞ , ∞ ) = √ d (cf. the proof of Thm 5.1). These modifications suggest a potential reduction in adversarial risk in scenario (i).

However, this leads to a training failure. Let us consider the variance of a network output under no bias (for simplicity). For sufficiently large input dimension d and x in ∈ [0 , 1] d , it can be calculated as follows:

This equation implies that the network always outputs a zero vector for any input x in . In other words, input information is lost during signal propagation in the network. In this situation, networks cannot learn the data structure well, and thus training does not proceed (cf. [71, 81, 105]).

Situations (2) and (3) are also similar to (1); they can mitigate the adversarial risk, but they break input information during forward and backward. Therefore, we conclude that it is not feasible to mitigate the adversarial effect without compromising the network's trainability.

Extension to Lipschitz continuous activations. In this study, we primarily established our theorems for networks employing ReLU-like activations. Here, we attempt to generalize these theorems to encompass networks with Lipschitz continuous activations. However, this extension involve looser upper bounds or potentially unrealistic assumptions.

First, we examine the upper bounds for p = 2 and q = 2 . Since J ( x in ) is a Jacobian at x in , Ineq. (8) remains valid for Lipschitz continuous activations. Denote the Lipschitz constant by k ≥ 0 . Similar to the discussion in Appx. H, we can derive the following upper bound:

An important observation here is that ω L/ 2 , which dictates the training properties, is common to networks with both ReLU-like and Lipschitz continuous activations. Consequently, we claim that our theorems also extend to Lipschitz continuous activations, despite differences in the factor. However, this bound is exponentially loose, casting doubt on its ability to accurately represent the properties of adversarial loss.

Second, we use the analysis of a network Jacobian [69]. For a comprehensive comparison between our approach and theirs, see Sec. 5.1. Here, we adopt their assumption where the variance V [ h ( l ) ] is constant for any l ∈ [ L ] . Drawing from the results expressed in Eq. (22) of [69], we can assert that ∥ ∥ J ( x in ) ∥ ∥ 2 ≤ Θ(( σ 2 w ) L/ 2 ) holds. Similar to the first proposition, this implies that our theorems hold to general activations including Lipschitz continuous activations. However, the assumption clearly does not hold in our theorems such as Thm 5.4.

Single-gradient descent attack can find adversarial examples. Here, we demonstrate that a single-gradient descent attack, such as the fast gradient sign method [38], can find adversarial examples. This is not a substantially novel contribution. For example, see [7, 11, 24, 25, 61, 84, 112]. However, to introduce a new approach based on Thm 4.1, we attempt it.

Proposition K.1 (Single-gradient descent attack can find adversarial examples) . Suppose that ∥ ∥ x in ∥ ∥ 2 = √ d , the perturbation constraint ϵ is sufficiently small, and the input dimension d is sufficiently large. Then, the single-gradient descent attack finds ℓ ∞ constrained adversarial examples that flip the prediction of a single-output random deep neural network with high probability

where erf is the error function.

Proof. Since η is sufficiently small, the same linear regions in a ReLU-like network encompass both x in and x in + η , i.e., J ( x in ) = J ( x in + η ) . When an ℓ ∞ constrained adversarial example flips the prediction of a classifier, the inequality | J ( x in ) x in + a ( x in ) | < | J ( x in ) η | holds for | η | ∞ ≤ ϵ . The perturbation generated by a single-gradient descent is defined as η := ϵ sgn( J ( x in )) . Note that J ( x in ) is a one-dimensional vector in single-output networks. Using this perturbation, by Eq. (A87) and Thm 4.1, the right term of the inequality can be transformed into:

Then, let us consider | J ( x in ) x in + a ( x in ) | . By the reproductive property of the Gaussian, J ( x in ) x in + a ( x in ) follows a Gaussian N (0 , ∥ ∥ x in ∥ ∥ 2 2 V [ J ( x in )] + V [ a ( x in )]) . The variance can be rearranged as follows:

Note that the c.d.f of the folded normal distribution based on the Gaussian N (0 , σ 2 ) is given by erf( x/ √ 2 σ 2 ) . Thus, we can compute the probability such that | J ( x in ) x in + a ( x in ) | is less than

L Other experimental results

L.1 Setting

We focused on ReLU networks, i.e., u = 1 , v = 0 , and α = 1 / 2 . Adversarial examples were generated using auto projected gradient descent [20] and the loss function defined in Eq. (3). Networks were initialized to satisfy the ( M,m ) -trainability condition (Lemma 5.6), i.e., σ 2 w = 2 for vanilla networks and σ 2 w = 0 . 1 or σ 2 w = 0 . 01 for residual networks. The initial bias variance was set to σ 2 b = 0 . 01 . We employed MNIST [26] and Fashion-MNIST [99] as the training dataset. All experiments are conducted on an NVIDIA A100.

L.2 Verification of Thm 4.1 (new mean field-based framework)

As shown in Fig. 1, the empirical distribution of J ( x in ) in the vanilla network aligned well with Thm 4.1. To further verify Thm 4.1, we provide additional results. We sampled 10 , 000 vanilla networks with d = 1 , 000 and K = 1 . For vanilla networks, we set σ 2 w = 2 , and for residual networks, we set σ 2 w = 0 . 01 . The network width N and depth L were set to 5 , 000 and 10 , respectively.

The empirical distributions of J ( x in ) and a ( x in ) for different network depths, L = 10 and 100 , and two randomly generated inputs, x in 1 and x in 2 , are shown in Figs. A5 and A6. The accuracy of Thm 4.1 in predicting these distributions remains consistent across varying network depths. Moreover, the distributions of J ( x in ) and a ( x in ) did not depend on the input x in , even though they are defined as functions of x in .

We should note that our theory operates under the assumption of infinite network width. To assess the influence of network width on Thm 4.1, please refer to Fig. A7. We found that for smaller values of N , for instance N = 10 , the empirical distribution of J ( x in ) and a ( x in ) deviates from the theoretical prediction as provided by Thm 4.1. As network width increases, the alignment between empirical results and predictions from Thm 4.1 improves.

̸

It is important to clarify that while Thm 4.1 asserts that the distributions of J ( x in ) and a ( x in ) do not depend on x in , it does not necessarily imply that random variables J ( x in ) and a ( x in ) are independent of J ( y in ) and a ( y in ) , respectively, when x in = y in . For slightly different inputs ( y in := x in × 0 . 999 , x in × 0 . 99 , and x in × 0 . 5 ), we computed the correlation coefficients between J ( x in ) , a ( x in ) and J ( y in ) , a ( y in ) , respectively. 4 These findings can be found in Figs. A8 and A9, and clearly demonstrate that more similar inputs result in higher correlation coefficients. However, this does not contravene the claims made in Thm 4.1. Further, for two randomly generated inputs, x in 1 and x in 2 , the random variables J ( x in ) , a ( x in ) and J ( x in 2 ) , a ( x in 2 ) were found to be relatively uncorrelated, respectively. The theoretical prediction for this correlation is currently unclear, which is a topic for future work.

To validate the independence of different entries of J ( x in ) and a ( x in ) , we computed the correlation coefficient between two distinct entries. 4 As presented in Fig. A10, we found that the two unique entries of J ( x in ) and a ( x in ) were indeed uncorrelated, corroborating the claims in Thm 4.1. Furthermore, Fig. A10 illustrates that the entries of J ( x in ) and a ( x in ) were uncorrelated, which also aligned with Thm 4.1.

An examination of the distributions of J ( x in ) and a ( x in ) after training may yield interesting insights. However, as trained networks have different weight and bias variances, the observed J ( x in ) and

-

0

.

1

0

.

0

0

.

1

0

5

10

x

in

1

J

(

x

in

)

1

,

1

,

L

=10

-

0

.

1

0

.

0

0

.

1

0

5

10

J

(

x

in

)

1

,

1

,

L

=100

-

0

.

5

0

.

0

0

.

5

0

1

2

a

(

x

in

)

1

,

L

=10

-

2

0

2

0

.

00

0

.

25

0

.

50

a

(

x

in

)

1

,

L

=100

Exp.

Theory

0

.

1

0

.

0

0

.

1

0

5

10

x

in

2

0

.

1

0

.

0

0

.

1

0

5

10

0

.

5

0

.

0

0

.

5

0

1

2

2

0

2

0

.

00

0

.

25

0

.

50

-

-

-

-

-

0

.

1

0

.

0

0

.

1

0

5

10

x

in

1

J

(

x

in

)

1

,

1

,

L

=10

-

0

.

1

0

.

0

0

.

1

0

5

10

J

(

x

in

)

1

,

1

,

L

=100

-

0

.

5

0

.

0

0

.

5

0

1

2

a

(

x

in

)

1

,

L

=10

-

2

0

2

0

.

0

0

.

2

0

.

4

a

(

x

in

)

1

,

L

=100

Exp.

Theory

0

.

1

0

.

0

0

.

1

0

5

10

x

in

2

0

.

1

0

.

0

0

.

1

0

5

10

0

.

5

0

.

0

0

.

5

0

1

2

2

0

2

0

.

0

0

.

2

0

.

4

-

-

-

-

a ( x in ) in each network do not follow the same distribution. Consequently, empirically evaluating the validity of Thm 4.1 for trained networks presents substantial challenges. Despite this, we consider, based on experimental findings such as those presented in Fig. A18, that Thm 4.1 offers an accurate representation of the early stages of the training process.

L.3 Verification of Thm 5.1 (upper bounds of adversarial loss)

As shown in Fig. 2, the upper bounds in Thm 5.1 indicates the significant tightness in vanilla networks. To further verify Thm 5.1, we provide additional results. We generated 100 adversarial examples and calculated the adversarial losses defined in Eq. (3). For vanilla networks, we set σ 2 w = 2 , and for residual networks, we set σ 2 w = 0 . 01 . The network width N was set to 40 , 000 for vanilla networks and 35 , 000 for residual networks, and the network depth L was set to 3 . We set the perturbation constraint ϵ to 0 . 1 and iterations of projected gradient descent to 50 .

In Fig. 2, we illustrate the tightness of the bounds with varying input dimensions in vanilla networks. We provide further results for residual networks, as shown in Fig. A11. Additionally, Figs. A12 and A13 demonstrate the adversarial loss as a function of varying output dimensions in vanilla and residual networks, respectively. Overall, our findings affirm the tightness of our bounds across both network types. For some ( p, q ) combinations, empirically observed adversarial loss samples exceeded the theoretical upper bounds. We consider that this discrepancy can be mitigated by expanding the network width, a topic elaborated upon in the following paragraph.

-

0

.

1

0

.

0

0

.

1

0

50

100

150

J

(

x

in

)

1

,

1

N

=10

-

0

.

1

0

.

0

0

.

1

0

5

10

15

N

=100

-

0

.

1

0

.

0

0

.

1

0

5

10

N

=1,000

-

0

.

1

0

.

0

0

.

1

0

5

10

N

=10,000

Exp.

Theory

0

.

5

0

.

0

0

.

5

0

1

2

3

a

(

x

in

)

1

0

.

5

0

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

0

.

5

0

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

0

.

5

0

.

0

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

-

-

-

-

The derivation of Thm 5.1 assumes a network of infinite width. However, in practical implementation, an infinite network width is unachievable, leading to instances where empirical results do not coincide with Thm 5.1. Specifically, certain samples were observed to exceed the theoretical upper bounds, as shown in Fig. 2. To evaluate the influence of network width, we varied it while sampling the adversarial losses, with the results displayed in Fig. A14. In the case of ( p, q ) = (2 , 2) , it was observed that a wider network width diminished the discrepancy between sampled losses and theoretical bounds. For other ( p, q ) pairings, even with a relatively small width (e.g., N = 100 ), empirical results aligned with Thm 5.1.

The impact of the perturbation constraint ϵ can be found in Fig. A15. It was confirmed that larger ϵ values corresponded to a greater divergence between empirically sampled adversarial losses and theoretical bounds. This is because for larger ϵ , it was harder for the projected gradient descent to tune adversarial examples well.

The effect of the network depth L can be confirmed in Fig. A16. For ( p, q ) = (2 , 2) , as the number of layers increased, the value exceeded the upper bounds. In contrast, for ( p, q ) = ( ∞ , ∞ ) , the value fell below the bounds. These differences can be attributed to the varying complexities of optimizing the adversarial loss for each combination of p and q . That is, for ( p, q ) = (2 , 2) , the optimization of adversarial examples is relatively straightforward, enabling the generation of adversarial losses that exceed the upper bound, which is imposed by the constraints of finite width. However, for ( p, q ) = ( ∞ , ∞ ) , the optimization is challenging, and it is difficult to generate highquality adversarial examples. The underlying reasons for these disparities in optimization complexity remain a topic for future work.

Furthermore, we assessed adversarial loss during training on the MNIST dataset, as shown in Fig. A17. Vanilla networks were trained using stochastic gradient descent with a learning rate of 0 . 001 . While Thm 5.1 broadly holds for ( p, q ) = (1 , 2) , the disparity between theoretical prediction and empirical results expands as training progresses for ( p, q ) = (2 , 2) , (2 , ∞ ) , and ( p, q ) = ( ∞ , ∞ ) . Determining a theoretical prediction that provides a tight bound on the adversarial loss for fully trained deep neural networks remains an open challenge for future research.

L.4 Verification of Thms 5.4 and G.9 (time evolution of weight variance)

As shown in Fig. 3, the empirical weight variance observed in the vanilla network aligns well with Thm 5.4. To verify Thm G.9, we provide Fig. A18. We trained 10 -layers vanilla and residual networks. The vanilla networks were initialized with σ 2 w (0) = 2 and σ 2 b (0) = 0 . 01 . Residual networks are initialized with σ 2 w (0) = 0 . 1 and σ 2 b (0) = 0 . 01 . For standard training, N was set to 1 , 000 . For

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

)

1

,

1

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

×

0

.

999)

1

,

1

Correlation coefficient:

0.99

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

)

1

,

1

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

×

0

.

99)

1

,

1

Correlation coefficient:

0.91

Experiment

Theory

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

)

1

,

1

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

×

0

.

5)

1

,

1

Correlation coefficient:

0.5

-

0

.

15

0

.

00

0

.

15

J

(

x

in

1

)

1

,

1

-

0

.

15

0

.

00

0

.

15

J

(

x

in

2

)

1

,

1

Correlation coefficient:

0.03

adversarial training, we designated p = ∞ , q = ∞ , and ϵ = 0 . 3 . In both training, we used stochastic gradient descent with a small learning rate, 0 . 001 . Note that gradient flow assumes an infinitesimal learning rate (cf. Eq. (10)). A theoretically defined training step t under gradient flow is not equal to a training step in the experiment. We have linked them by t := training steps × learning rate . In practice, setting the weight variance precisely is not feasible. For example, in a vanilla network, σ 2 w (0) might be set to 1 . 999 even though we tried to initialize it to satisfy σ 2 w (0) = 2 . Thus, for visibility, the experimental results (curves) were shifted parallel to meet σ 2 w (0) = 2 in vanilla networks and σ 2 w (0) = 0 . 1 in residual networks.

From Fig. A18, it is evident that adversarial training facilitates weight regularization for both vanilla and residual networks. Compared to the ℓ 2 weight regularization discussed in Appx. I, it offers stronger regularization. We can verify that Thm 5.1 can predict these empirical behaviors in the initial stage of training. The validity of Thms 5.4 and G.9, which underpin our theorems such as Thms 5.7 and 5.9, lends credence to our theoretical assertions.

-

1

0

1

a

(

x

in

1

)

1

-

1

0

1

a

(

x

in

1

×

0

.

999)

1

Correlation coefficient:

0.99

-

1

0

1

a

(

x

in

1

)

1

-

1

0

1

a

(

x

in

1

×

0

.

99)

1

Correlation coefficient:

0.9

Experiment

Theory

-

1

0

1

a

(

x

in

1

)

1

-

1

0

1

a

(

x

in

1

×

0

.

5)

1

Correlation coefficient:

0.53

-

1

0

1

a

(

x

in

1

)

1

-

1

0

1

a

(

x

in

2

)

1

Correlation coefficient:

0.42

L.5 Verification of Thms 5.7 and 5.8 (adversarial trainability)

As shown in Fig. 4, vanilla networks with small width and large depth were not adversarially trainable, which aligned well with Thm 5.7. To verify Thms 5.7 and 5.8, we provide Fig. A19. We trained vanilla and residual networks under various width and depth settings, and observed training accuracy. For fast training convergence, we used Adam [51]. The training was stopped if training accuracy was not improved in the last 200 steps. We set the learning rates to 0 . 001 or 0 . 0001 , adopting the highest training accuracy at the final training step. The vanilla networks were initialized with σ 2 w (0) = 2 and σ 2 b (0) = 0 . 01 . The residual networks were initialized with σ 2 w (0) = 0 . 01 and σ 2 b (0) = 0 . 01 . This initialization satisfied the ( M,m ) -trainability conditions (Lemma 5.6). In adversarial training, we set p = ∞ , q = ∞ , ϵ = 0 . 1 , and the iterations of projected gradient descent to 10 .

From Fig. A19, we can confirm that vanilla networks with large depth and small width were not adversarially trainable and the training difficulty was unique to adversarial training of such vanilla networks. Note that Prop G.10 could not be verified as the condition in which training persists without machine errors and the trainability condition not being satisfied at t = 0 is challenging to implement in practical scenarios.

-

0

.

15

0

.

00

0

.

15

J

(

x

in

)

1

,

1

-

0

.

15

0

.

00

0

.

15

J

(

x

in

)

1

,

2

Correlation coefficient:

-0.01

-

1

0

1

a

(

x

in

)

1

-

1

0

1

a

(

x

in

)

2

Correlation coefficient:

0.0

-

0

.

15

0

.

00

0

.

15

J

(

x

in

)

1

,

1

-

1

0

1

a

(

x

in

)

1

Correlation coefficient:

0.0

Experiment

Theory

250

500

750

1000

0

.

25

0

.

50

0

.

75

p

= 1

, q

= 1

250

500

750

1000

0

.

05

0

.

10

p

= 1

, q

= 2

250

500

750

1000

0

.

01

0

.

02

0

.

03

p

= 1

, q

=

∞

Theory

Exp.

250

500

750

1000

0

.

15

0

.

20

p

= 2

, q

= 2

250

500

750

1000

0

.

0

0

.

2

p

= 2

, q

=

∞

250

500

750

1000

1

2

p

=

∞

, q

=

∞

Input dimension

d

Adversarial loss

L

adv

L.6 Verification of Thms 5.9 and G.14 (capacity degradation)

To verify the degradation of network capacity during adversarial training, we trained vanilla and residual networks on MNIST with p = ∞ , q = ∞ , and ϵ = 0 . 1 , and observed the Fisher-Rao norm as a measure of capacity. The training steps were set to 500 . As Thms 5.9 and G.14 are derived under sufficiently small learning rate (cf. Eq. (10)), we employed small learning rate, 0 . 0001 . Vanilla networks were initialized with σ 2 w (0) = 2 and σ 2 b (0) = 0 . 01 . Residual networks were initialized with σ 2 w (0) = 0 . 01 and σ 2 b (0) = 0 . 01 . During the derivation of Thms 5.9 and G.14, using Asm B.1, we calculated E [ ∂f ∂w i ∂f ∂w j w i w j ] = E [ ∂f ∂w i ∂f ∂w j ] E [ w i w j ] = 0 , for i = j . Nevertheless, in practice, this calculation does not always hold. Therefore, for the computation of the Fisher-Rao norm, we employed a diagonal matrix of F instead of F itself.

̸

As shown in Figs. A20 and A21, large network depths (large L ) produced high Fisher-Rao norms but degraded them drastically. Moreover, it was observed that a wider network width could maintain the Fisher-Rao norm at its initial state. The discrepancy between these experimental values and the values predicted by Thms 5.9 and G.14 is considered to originate from Asm B.1, which does not hold in the case where a diagonal matrix of F is used instead of F itself.

Additionally, we assessed the influence of network width on the robust test accuracy of fully-trained networks. The robust test accuracy was determined using ℓ ∞ -AutoAttack with ϵ = 0 . 1 . The adversarial training was conducted under p = ∞ , q = ∞ , ϵ = 0 . 1 , and 10 iterations. We employed Adam with a learning rate of 0 . 001 and epochs set to 200 .

From Tabs. A5 and A6, it is evident that the robust test accuracy depends on network width rather than depth. The same superscripts in the tables indicate results from networks with an identical number of parameters. For example, the accuracy for ( N,L ) = (250 , 5) significantly surpasses that

250

500

750

1000

2

.

5

5

.

0

7

.

5

p

= 1

, q

= 1

250

500

750

1000

0

.

1

0

.

2

0

.

3

p

= 1

, q

= 2

250

500

750

1000

0

.

02

0

.

03

0

.

04

p

= 1

, q

=

∞

Theory

Exp.

250

500

750

1000

0

.

2

0

.

4

p

= 2

, q

= 2

250

500

750

1000

0

.

0

0

.

2

p

= 2

, q

=

∞

250

500

750

1000

0

.

7

0

.

8

0

.

9

p

=

∞

, q

=

∞

Output dimension

K

Adversarial loss

L

adv

250

500

750

1000

2

.

5

5

.

0

7

.

5

p

= 1

, q

= 1

250

500

750

1000

0

.

1

0

.

2

0

.

3

p

= 1

, q

= 2

250

500

750

1000

0

.

02

0

.

03

0

.

04

p

= 1

, q

=

∞

Theory

Exp.

250

500

750

1000

0

.

2

0

.

4

p

= 2

, q

= 2

250

500

750

1000

0

.

0

0

.

2

p

= 2

, q

=

∞

250

500

750

1000

0

.

7

0

.

8

0

.

9

p

=

∞

, q

=

∞

Output dimension

K

Adversarial loss

L

adv

for ( N,L ) = (125 , 20) . Although Thms 5.9 and G.14 is primarily applicable to the early stages of training, these theorems, emphasizing the importance of network width, hold true for fully-trained networks.

0

10000

20000

30000

40000

0

.

6

0

.

8

1

.

0

p

= 1

, q

= 1

0

10000

20000

30000

40000

0

.

05

0

.

10

0

.

15

p

= 1

, q

= 2

0

10000

20000

30000

40000

0

.

000

0

.

025

p

= 1

, q

=

∞

Theory

Exp.

0

10000

20000

30000

40000

0

.

200

0

.

225

p

= 2

, q

= 2

0

10000

20000

30000

40000

0

.

05

0

.

10

0

.

15

p

= 2

, q

=

∞

0

10000

20000

30000

40000

0

.

7

0

.

8

0

.

9

p

=

∞

, q

=

∞

N

Adversarial loss

L

adv

0

.

02

0

.

10

0

.

20

0

.

30

1

2

p

= 1

, q

= 1

0

.

02

0

.

10

0

.

20

0

.

30

0

.

1

0

.

2

0

.

3

p

= 1

, q

= 2

0

.

02

0

.

10

0

.

20

0

.

30

0

.

025

0

.

050

0

.

075

p

= 1

, q

=

∞

Theory

Exp.

0

.

02

0

.

10

0

.

20

0

.

30

0

.

25

0

.

50

p

= 2

, q

= 2

0

.

02

0

.

10

0

.

20

0

.

30

0

.

1

0

.

2

0

.

3

p

= 2

, q

=

∞

0

.

02

0

.

10

0

.

20

0

.

30

1

2

p

=

∞

, q

=

∞

Epsilon

/epsilon1

Adversarial loss

L

adv

3 10 20

40

60

80

100

0

.

6

0

.

8

p

= 1

, q

= 1

3 10 20

40

60

80

100

0

.

05

0

.

10

0

.

15

p

= 1

, q

= 2

3 10 20

40

60

80

100

0

.

000

0

.

025

p

= 1

, q

=

∞

Theory

Exp.

3 10 20

40

60

80

100

0

.

20

0

.

25

p

= 2

, q

= 2

3 10 20

40

60

80

100

0

.

05

0

.

10

0

.

15

p

= 2

, q

=

∞

3 10 20

40

60

80

100

0

.

50

0

.

75

p

=

∞

, q

=

∞

Depth

L

Adversarial loss

L

adv

0

500

1000

0

.

02

0

.

04

p

= 1

, q

= 1

0

500

1000

0

.

010

0

.

015

p

= 1

, q

= 2

0

500

1000

0

.

00

0

.

01

p

= 1

, q

=

∞

Theory

Experiment

0

500

1000

0

.

11

0

.

12

p

= 2

, q

= 2

0

500

1000

0

.

10

0

.

11

p

= 2

, q

=

∞

0

500

1000

2

.

2

2

.

4

p

=

∞

, q

=

∞

Training step

Adversarial loss

L

adv

0

500

1000

1500

2000

1

.

9985

1

.

9990

1

.

9995

2

.

0000

Vanilla

0

500

1000

1500

2000

0

.

10000

0

.

09995

Residual

Training step

Weight variance

σ

2

w

Standard

/lscript

2

weight

Adversarial (

N

= 2

k

)

Adversarial (

N

= 1

k

)

Theory (

N

= 2

k

)

Theory (

N

= 1

k

)

100

200

300

400

500

600

700

800

20

40

Vanilla

Standard

100

200

300

400

500

600

700

800

20

40

Adversarial

T

=100

T

=160

T

=250

100

200

300

400

500

600

700

800

20

40

Residual

100

200

300

400

500

600

700

800

20

40

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Training accuracy

0

100

200

300

400

500

20

40

60

Vanilla

0

100

200

300

400

500

0

.

5

1

.

0

1

.

5

Residual

Training step

Fisher-Rao norm

‖

w

‖

FR

L

= 10

,

N

= 1250

L

= 20

,

N

= 1250

L

= 20

,

N

= 2500

L

= 20

,

N

= 5000

L

= 20

,

N

= 10000

0

100

200

300

400

500

-

40

-

20

0

Vanilla

0

100

200

300

400

500

-

0

.

15

-

0

.

10

-

0

.

05

0

.

00

Residual

Training step

Fisher-Rao norm

‖

w

‖

FR

L

= 10

,

N

= 1250

L

= 20

,

N

= 1250

L

= 20

,

N

= 2500

L

= 20

,

N

= 5000

L

= 20

,

N

= 10000

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a new theoretical framework based on the mean field theory to analyse adversarial training from several perspectives, such as the upper bounds of adversarial loss, the time evolution of weight variance and adversarially trainable conditions. Besides the theoretical analysis, the authors conducted several experiments verifying the proposed theoretical framework. \nGenerally speaking, the proposed theoretical framework provides a new perspective to analyse adversarial training and is highly versatile and even can extend to other training methods.\n\nSTRENGTHS:\nThe paper is organised well and easy to follow. \nThe proposed theoretical framework seems inspiring and intriguing and gives a new perspective to analyse adversarial training from several aspects, which may serve as a good guidance for future work. Besides the proposed theory, verification experiments were also conducted to prove its effectiveness further.\n\nWEAKNESSES:\n1.\tThe verification experiments were only conducted on the easy dataset (MNIST); it may strengthen the findings if additional experiments are conducted on more challenging datasets.\n2.\t Including a more systematic evaluation of the results may be beneficial, such as adversarial loss in residual networks vs. training steps for normally or adversarially training. \n3.\tSeveral conclusions match the previous works; it could be more convincing if the reference could be given in the main content, such as ‘’the square sum of the weights in Ineq. (9) suggests that adversarial training exhibits a weight regularisation effect,’’ is consistent with [1], ‘’to achieve high capacity in adversarial training, it is necessary to increase not only the number of layers L but also the width N to keep L^2/N constant’’ is somehow consistent with [2].\n4.\tThe authors mentioned in Line 260 that ‘’This result suggests that residual networks are better suited for adversarial training. However, one of the previous studies indicated that residual networks are more vulnerable to transfer attacks [54] than vanilla networks. ’’ However, the proposed theoretical framework did not explain such a transfer attack phenomenon. Could the authors explain or give more comments about this?\n5. in Equation 3 and 4, it seems that the minimize part is not shown, perhaps it would be more reasonable to change the form of min-max in equation 3 and 4.\n\n[1] A unified gradient regularization family for adversarial examples, in: IEEE International Conference on Data Mining (ICDM), 2015.\n[2] Do wider neural networks really help adversarial robustness? Advances in Neural Information Processing Systems, 34.\n\nQUESTIONS:\nSee Weakness above.\n\nLIMITATIONS:\nThe authors have clearly discussed the limitations of the proposed framework, such as, ‘’some theorems begin to diverge from the actual behaviour‘’ and ’’ the mean field theory assumes infinite network width, which is practically infeasible’’.\nFrom my point of view, this article does not involve any potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a mean field based framework for theoretically analyzing the training dynamics of adversarial training for MLP and residual networks with ReLU non-linearities. Based on this framework, the authors provide tight bounds on the adversarial loss (squared loss between clean and adversarial example output) and investigate trainability of MLP and residual networks.\n\nSTRENGTHS:\n- While I am not very familiar with related work on mean field theory for (adversarially) training networks, the proposed framework seems to address limitations of prior work quite elegantly.\n- The paper includes quite a number of interesting results using the proposed framework (loss bounds, training dynamics, trainability, impact of width).\n- Formulation includes MLP and residual network with a class of “ReLU-like” non-linearities; is also includes adversarial training with clean loss (as e.g. done by TRADES).\n- I feel the framework can be quite insightful and helpful for future work in understanding or improving adversarial training.\n- Many claims are empirically supported on MNIST.\n\nWEAKNESSES:\nMy main concern about the paper is that it is incredibly dense (due to the number of included claims) and its structure does not make checking the proofs and claims easy. Even though I invested significantly more time in this review compared to other reviews, I was unable to fully follow all derivations and proofs. I believe this is mainly due to the extremely convoluted way of presenting the theoretical results across main paper and appendix. Unfortunately, I feel that this also makes me less enthusiastic about the results. Here are some more detailed comments and suggestions:\n- Starting with 4.2, the reader is somewhat forced to jump to the appendix at least once just to see that the authors simply reformulate the ReLU using its derivative and D for J and a. This trend continues throughout the paper and appendix. Especially in the appendix, one is forced to jump around a lot just to follow 2-3 sentences.\n- The proofs are structured starting with simple lemmata and building up to the actual theorems. This is generally fine, but again, the referencing is overdone. For Thm 4.1 in Appendix E, there are 9 pages of derivation with >20 individual lemmata. So if I want to follow the proof of Thm 4.1, I am forced to go through many but not all of these lemmeta. Many have proofs of only 1-2 lines, but reference 2+ other lemmata or remarks and I need to remember the numbers or jump back and forth 2+ times just to read a single sentence. I feel the root cause for this is that many of the results are over-generalized and compartmentalized too much. I appreciate the thorough job of the authors in establishing many of the independence results, but as a reviewer and reader my #1 interest is following Thm 4.1, nothing more and nothing less. Everything that complicates this job is – in my opinion – bad for the paper. For me, the ideal solution would be a separate, easier to follow section for the proof of Thm 4.1 - even if it restates many of the lemmata and remarks, and moving the other results to a separate section for the (very very) interested reader.\n- The main paper includes so many results that there is basically no discussion of each individual result. Often it feels that every sentence refers to some additional result in the appendix. I think for me, and many readers, actually discussing the results informally, in words, and taking more time and space to introduce the required notation would be more beneficial than including the current amount of results. I would prefer to have fewer results well-described and the remaining ones being in the appendix.\n- Empirical results are discussed twice – after the corresponding theorems as well as in Section 6 – the space of the latter could be used to address one of the points above.\n\nComments and questions unrelated to structure and writing:\n- In the introduction, contribution (d) is unclear to me – what theoretical result does it refer to, the trainability?\n- The use of “probabilistic properties” is a bit unclear until the discussion of training dynamics. It would be helpful if the meaning would be detailed earlier in the paper.\n- In 3.1, why is having $P^{in}$ and $P^{out}$ important, i.e., why do we need these fixed layers?\n- Usually, the adversarial loss is also cross-entropy or something similar. While I saw papers arguing for a squared loss, TRADES does not use it AFAIK. Instead, the common setting is adversarial cross-entropy loss only, combined with clean cross-entropy loss or clean cross-entropy + KL as in TRADES. This makes me ask how assuming an adversarial cross-entropy loss would impact the results? Can similar results be derived?\n- In 4, the assumption of independence is also unclear. I feel making this more explicit, e.g., by informally providing a short result on independence from the appendix, could be useful.\n- The authors also highlight broader applicability; I am wondering if the authors derived similar results for standard training as reference? Or has this been done in previous work with other frameworks? This also related to the statement in l280.\n- In 5.3 l261, I can’t follow how transfer attacks are relevant here? Transfer attacks should be weaker than the general attack modeled in the paper …\n\nConclusion:\nI think that the paper has many interesting contributions and will be valued by the NeurIPS community. However, as I was not able to follow all derivations in a reasonable time, I will closely follow what the other reviewers have to say about the theoretical results. Also, I believe that the paper in its current form would have better fitted a long-format journal. For NeurIPS, I hope the authors invest some time in simplifying the main paper and restructuring the appendix. I think the current format will limit the audience of the paper to those very familiar with related work or willing to invest hours jumping back and forth between main paper and appendix. Some restructuring could really make this paper more accessible to the broad audience at NeurIPS.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

comments: SUMMARY:
The paper provides a mean field analysis on relu networks for adversarial training. The main insight is that networks without residual connections are not most likely inevitably suffer from gradient explosion or vanishing and thus are not adversarially trainable, unlike vanilla network training.

STRENGTHS:
Analyzing adversarial training performance is an important problem both theoretically.

The insight from the analysis that adversarial training vanilla relu networks is more likely to suffer from gradient explosion/vanishing is an interesting insight.

WEAKNESSES:
The verification of the theorems might need more effort. E.g., Figure 4 is showing accuracy of vanilla network, it would be helpful to also show the curves for residual networks.

QUESTIONS:
Could the authors elaborate a bit on why residual connections could make a bigger difference in adversarial training than vanilla training?

LIMITATIONS:
Limitations are well discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

comments: SUMMARY:
The theoretical understanding of adversarial training is an important and valuable topic. This work proposes a new theoretical framework for this based on mean field theory. With the proposed framework, the authors analyze the properties of adversarial training from multiple aspects, including the upper bounds of adversarial loss, the time evolution of weight variance, the adversarial trainable conditions, and the degradation of network capacity. These results could be helpful for the understanding of adversarial training and inspire more efforts on this topic.

STRENGTHS:
1. It proposes a new framework to analyze adversarial training theoretically based on mean field theory. 
2. Based on the proposed framework, it presents several theoretical results for adversarial training.
3. The proposed framework and the presented theoretical results are non-trivial and helpful for the understanding of adversarial training.

WEAKNESSES:
It studies several different adversarial training characteristics in the main paper. Is there any correlation between these different characteristics？ Why do we choose these aspects for analysis? Further, is it possible to provide a global diagram to better see which properties can be analyzed and which cannot be analyzed at present based on the proposed framework?

QUESTIONS:
I am not an expert on mean field theory and didn't check all the proofs. I give my rating by considering that 1) the theoretical understanding of adversarial training is important and the progress on this is very helpful for the community and 2) the proposed framework seems to be generic and may inspire more work in the future.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-26