SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[30] A. Fawzi, H. Fawzi, and O. Fawzi. Adversarial vulnerability for any classifier. In NeurIPS, volume 31, 2018.
[31] A. Fawzi, O. Fawzi, and P. Frossard. Analysis of classifiers' robustness to adversarial perturbations. ML, 107(3):481-508, 2018.
[32] A. Fawzi, S.-M. Moosavi-Dezfooli, and P. Frossard. Robustness of classifiers: from adversarial to random noise. In NeurIPS, volume 29, 2016.
[33] K. Fukushima. Cognitron: A self-organizing multilayered neural network. Biol. Cybern., 20(3):121-136, 1975.
[34] A. Galloway, A. Golubeva, T. Tanay, M. Moussa, and G. W. Taylor. Batch normalization is a cause of adversarial vulnerability. In ICML WS, 2019.
[35] R. Gao, T. Cai, H. Li, C.-J. Hsieh, L. Wang, and J. D. Lee. Convergence of adversarial training in overparametrized neural networks. In NeurIPS, volume 32, 2019.
[36] D. Gilboa, B. Chang, M. Chen, G. Yang, S. S. Schoenholz, E. H. Chi, and J. Pennington. Dynamical isometry and a mean field theory of LSTMs and GRUs. arXiv:1901.08987, 2019.
[37] J. Gilmer, L. Metz, F. Faghri, S. S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow. Adversarial spheres. In *ICLR* WS, 2018.
[38] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[39] S. Gowal, K. D. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli. Scalable verified training for provably robust image classification. In ICCV, pages 4842-4851, 2019.
[40] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann. Improving robustness using generated data. In NeurIPS, 2021.
[41] S. Hayou, A. Doucet, and J. Rousseau. On the selection of initialization and activation function for deep neural networks. arXiv:1805.08266, 2018.
[42] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[43] M. Hein and M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In NeurIPS, volume 30, 2017.
[44] J. Hron, Y. Bahri, J. Sohl-Dickstein, and R. Novak. Infinite attention: NNGP and NTK for deep attention networks. In ICML, pages 4376-4386, 2020.
[45] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, pages 4700-4708, 2017.
[46] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, volume 31, 2018.
[47] A. Javanmard, M. Soltanolkotabi, and H. Hassani. Precise tradeoffs in adversarial training for linear regression. In  $COLT$ , pages 2034-2078, 2020.
[48] H. Kannan, A. Kurakin, and I. Goodfellow. Adversarial logit pairing. arXiv:1803.06373, 2018.
[49] R. Karakida, S. Akaho, and S.-i. Amari. Universal statistics of Fisher information in deep neural networks: Mean field approach. In AISTATS, pages 1032-1041, 2019.
[50] J. Khim and P.-L. Loh. Adversarial risk bounds via function transformation. arXiv:1810.09519, 2018.
[51] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In CVPR, 2015.
[74] A. Raghunathan, J. Steinhardt, and P. Liang. Certified defenses against adversarial examples. In ICLR, 2018.
[75] A. Raghunathan, S. M. Xie, F. Yang, J. Duchi, and P. Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In ICML, 2020.
[76] A. Raghunathan, S. M. Xie, F. Yang, J. C. Duchi, and P. Liang. Adversarial training can hurt generalization. In ICML WS, 2019.
[77] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. Mann. Fixing data augmentation to improve adversarial robustness. arXiv:2103.01946, 2021.
[78] K. Roth, Y. Kilcher, and T. Hofmann. Adversarial training is a form of data-dependent operator norm regularization. In NeurIPS, volume 33, pages 14973-14985, 2020.
[79] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR, 2014.
[80] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust generalization requires more data. In NeurIPS, 2018.
[81] S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. In ICLR, 2017.
[82] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein. Are adversarial examples inevitable? In ICLR, 2019.
[83] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! In NeurIPS, 2019.
[84] C.-J. Simon-Gabriel, Y. Ollivier, L. Bottou, B. Scholkopf, and D. Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In ICML, pages 5809-5817, 2019.
[85] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2014.
[86] A. Sinha, H. Namkoong, R. Volpi, and J. Duchi. Certifying some distributional robustness with principled adversarial training. In ICLR, 2018.
[87] H. Sompolinsky, A. Crisanti, and H.-J. Sommers. Chaos in random neural networks. Phys. Rev. Lett., 61(3):259, 1988.
[88] D. Su, H. Zhang, H. Chen, J. Yi, P.-Y. Chen, and Y. Gao. Is robustness the cost of accuracy? a comprehensive study on the robustness of 18 deep image classification models. In ECCV, pages 631-648, 2018.
[89] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In *ICLR*, 2014.
[90] F. Tramer, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses. In NeurIPS, volume 33, pages 1633-1645, 2020.
[91] J. A. Tropp. Topics in sparse approximation. The University of Texas at Austin, 2004.
[92] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
[93] Y. Tsuzuki, I. Sato, and M. Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In NeurIPS, volume 31, 2018.
[94] Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[95] C. Williams. Computing with infinite networks. In NeurIPS, volume 9, 1996.
[96] E. Wong and Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, pages 5286-5295, 2018.
[97] E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[98] B. Wu, J. Chen, D. Cai, X. He, and Q. Gu. Do wider neural networks really help adversarial robustness? In NeurIPS, volume 34, pages 7054-7067, 2021.
[99] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747, 2017.
[100] L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In ICML, pages 5393-5402, 2018.
[101] L. Xiao, J. Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. In ICML, pages 10462-10472, 2020.
[102] Y. Xing, Q. Song, and G. Cheng. On the generalization properties of adversarial training. In AISTATS, pages 505-513, 2021.
[103] G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv:1902.04760, 2019.
[104] G. Yang, J. Pennington, V. Rao, J. Sohl-Dickstein, and S. S. Schoenholz. A mean field theory of batch normalization. In ICLR, 2019.
[105] G. Yang and S. Schoenholz. Mean field residual networks: On the edge of chaos. In NeurIPS, volume 30, 2017.
[106] G. Yang and S. S. Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. OpenReview, 2018.
[107] D. Yin, R. Kannan, and P. Bartlett. Rademacher complexity for adversarially robust generalization. In ICML, pages 7085-7094, 2019.
[108] Y. Yoshida and T. Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv:1705.10941, 2017.
[109] S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, pages 1-12, 2016.
[110] R. Zhai, T. Cai, D. He, C. Dan, K. He, J. Hopcroft, and L. Wang. Adversarily robust generalization just requires more unlabeled data. arXiv:1906.00555, 2019.
[111] D. Zhang, T. Zhang, Y. Lu, Z. Zhu, and B. Dong. You only propagate once: Accelerating adversarial training via maximal principle. In NeurIPS, 2019.
[112] H. Zhang, D. Yu, Y. Lu, and D. He. Adversarial noises are linearly separable for (nearly) random neural networks. arXiv:2206.04316, 2022.
[113] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, pages 7472-7482, 2019.
[114] J. Zhang, X. Xu, B. Han, G. Niu, L. Cui, M. Sugiyama, and M. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML, pages 11278-11287, 2020.
[115] Y. Zhang, O. Plevrakis, S. S. Du, X. Li, Z. Song, and S. Arora. Over-parameterized adversarial training: An analysis overcoming the curse of dimensionality. In NeurIPS, volume 33, pages 679-688, 2020.
Table A2: Notation. While  $\pmb{h}^{(l)}$  is a function that takes  $\pmb{x}^{\mathrm{in}}$  as input, we omit the argument as  $\pmb{h}^{(l)} \coloneqq \pmb{h}^{(l)}(\pmb{x}^{\mathrm{in}})$  for notational simplicity. Other symbols sometimes follow this.
Notation	Description	Cf.
d ∈ N	Input dimension of the network	Sec. 3.1
K ∈ N	Output dimension of the network	Sec. 3.1
L ∈ N	Number of layers in the network, i.e., network depth	Sec. 3.1
N ∈ N	Number of neurons in a layer, i.e., network width	Sec. 3.1
u, v ∈ R	Slope of a ReLU-like function	Defn 3.1
p ∈ {1,2,∞}	Perturbation constraint norm, ||η||p ≤ ε	Eq. (3)
q ∈ {1,2,∞}	Output difference norm, ||f(xin + η) - f(xin)||q	Eq. (3)
m ∈ [0,1]	(M, m)-trainability condition	Defn 5.5
ε ≥ 0	Perturbation constraint, ||η||p ≤ ε	Eq. (3)
α ≥ 0	α := (u2 + v2)/2	Thm 4.1
βp,q ≥ 0	Time-invariant constant determined by (p, q)	Thm 5.1
t ≥ 0	Continuous training step	Eq. (10)
T ≥ 0	Upper limit of training steps	Asm 5.2
σw2≥ 0	Weight variance, Wij ~ N(0, σw2/N)	Sec. 3.1
σb2≥ 0	Bias variance, bi ~ N(0, σb2)	Sec. 3.1
ω ≥ 0	ωv := ασw2(vanilla) and ωr := 1 + ασw2(residual)	Thm 4.1
M ≥ 1	(M, m)-trainability condition	Defn 5.5
xin ∈ Rd	Input vector	Sec. 3.1
η ∈ Rd	Perturbation, ||η||p ≤ ε	Eq. (3)
W(l) ∈ RN×N	l-th weight, Wl(j) ~ N(0, σw2/N)	Sec. 3.1
W ⊂ R	Set of all network weights, {Wl(1), Wl(2), ..., WLn}	Thm 5.1
w ∈ RLN2	Vector of all the weights, (Wl(1), ..., Wl(L))T	Eq. (14)
b(l) ∈ RN	l-th bias, bi(l) ~ N(0, σb2)	Sec. 3.1
Pin ∈ RN×d	Rand. mat. for input adjustment, PIn ij ~ N(0, 1/d)	Sec. 3.1
Pout ∈ RK×N	Rand. mat. for output adjustment, Pout ij ~ N(0, 1/N)	Sec. 3.1
P(l) ∈ RN×N	Rand. mat. in the l-th shortcuts, P(l) ~ N(0, 1/N)	Eq. (A16)
φ: R → R	ReLU-like activation, φ := uz(z ≥ 0); vz(z ≤ 0)	Defn 3.1
χ(l): Rd → R	Mean squared l-th gradient, E[(∂L(xin)/∂x(i))2]	Sec. 3.1
h(l): Rd → RN	l-th pre-activation, h(l) := W(l)x(l-1)(xin) + b(l)	Sec. 3.1
x(l): Rd → RN	l-th post-activation, x(l) := φ(h(l)x(in))	Sec. 3.1
F: Rd → RLN2×LN2	Empirical Fisher information matrix	Eq. (14)
f: Rd → RK	Overall network, f(xin) := Pout g(Pin xin)	Sec. 3.1
g: RN → RN	L-trainable layer ReLU-like network	Sec. 3.1
J: Rd → RK×d	Slope of a piecewise linear region at xin	Thm 4.1
a: Rd → RK	Bias of a piecewise linear region at xin	Thm 4.1
D: RN → RN×N	Diagonal matrix	Eq. (5)
Lstd: Rd × Y → R	Standard loss function, where Y represents a label set	Sec. 3.2
Ladv: Rd → R≥0	Adversarial loss function	Eq. (3)
A Additional related work
Please also refer to Secs. 2 and 5.1.
A.1 Random networks and mean field theory
Understanding general deep neural networks is challenging due to the non-convexity of loss surfaces and stochastic nature of optimization. Researchers have explored random networks, which have random parameters instead of trained parameters. Random networks have been primarily studied in three areas: compositional kernels [17, 22, 23], neural network Gaussian processes [21, 44, 54, 59,
64, 68, 95, 103], and mean field theory. These fields share close relationships, particularly between neural network Gaussian processes and mean field theory. For example, the forwarding dynamics in mean field theory (Eq. (2)) is equivalent to the kernel representation of a Gaussian process [64, 71]. Furthermore, the accuracy of a Gaussian process is significantly influenced by the edge of chaos, which has been studied in mean field theory [68]. We provide a more detailed review of the mean field theory.
Mean field theory for neural networks was first introduced in [87] and later extended in [71, 81]. This theory examines the signal propagation, dynamics, and trainability of random networks in two phases: ordered and chaotic. The ordered phase is characterized by decreasing layer output variance and vanishing gradients during backpropagation, while the chaotic phase is characterized by expanding variance and exploding gradients. Effective training of deep neural networks occurs near the boundary between these two phases [71, 81]. Researchers have applied mean field theory to study various network architectures, including dropout [81], batch normalization [104], residual network [105, 106], recurrent network [15], quantized network [10], and Swish [41]. Recent research [15, 36, 69, 70, 100] has also utilized the mean field theory to analyze dynamical isometry, where all singular values of the Jacobian are one. Some of these findings can be applied to analyze Ineq. (8) in this study, with detailed comparisons provided in Sec. 5.1. Other studies have explored network representation power [49, 71, 101]. This work utilized the proof idea of [49] in the derivation of Thms 5.9 and G.14.
In this study, we employed mean field theory to analyze adversarial training behavior. However, the original theory has two limitations that make it unsuitable for this purpose (cf. Sec. 4.1). To address these limitations, we introduced a new mean field-based framework (cf. Sec. 4.2). Our theory is also applicable to mean field-based analyses of other deep learning methods beyond adversarial training.
A.2 Adversarial examples
A.2.1 Adversarial examples in random neural networks
We summarize the literature on adversarial examples in random neural networks [7, 11, 24, 25, 61, 84, 112]. It has been reported that adversarial perturbations can be found through gradient flow in most random ReLU networks with decreasing layer widths [24]. This result was extended to two-layer random networks with greater width than input dimension [11] and generalized to random ReLU networks with constant depth and wide width [7]. Recently, similar results were presented without width restrictions and for local Lipschitz continuous activation [61]. Additionally, it has been demonstrated that adversarial noises generated by a single-step attack are linearly separable in a two-layer random network and the neural tangent kernel regime [112]. More information on [25, 84] can be found in Appx. A.2.2.
Although the context differs somewhat, we mention [34], which provides empirical evidence that batch normalization leads to adversarial vulnerability. This observation aligns with previous results from mean field theory suggesting that batch normalization causes gradient explosion [104].
In this study, we focused on early stage adversarial training properties rather than adversarial examples. A key difference is that we primarily investigated the maximum difference between network outputs for standard and adversarial inputs rather than misclassification which was the main focus of previous studies [7, 11, 24, 61]. Nonetheless, some findings in Sec. 5.1 can be applied to the understanding of adversarial examples in random networks. In addition, in Appx. K, we proved the existence of adversarial examples in random networks to demonstrate the effectiveness of Thm 4.1.
A.2.2 Adversarial examples and input dimension
We present research investigating the relationship between adversarial examples and input dimensions [2, 25, 30–32, 37, 38, 82, 84]. Apart from [38], most studies have indicated that adversarial example threats increase with the square root of the input dimension [25, 30–32, 37, 82, 84]. Some studies have focused on specific data distributions or simple classifiers [30–32, 37, 82], while others targeted random networks [25, 84]. The study in [25] has examined the distance from the decision boundary in random networks, and concluded that adversarial examples deceive classifiers more easily with the square root of the input dimension. Another study has utilized the first-order Taylor
3Indeed, they omitted weight scaling for simplicity, and their results were essentially consistent with those of other studies when the scaling was considered.
expansion of a loss function to analyze the loss gradient with respect to an input [84]. While a direct comparison between our work and [84] is challenging due to differing loss functions, our theorems offer two advantages. First, we considered both the input and output dimensions, whereas they addressed only input dimensions. Second, our theorems are applicable to residual networks, whereas their assumptions are not.
In this study, we examined the relationship between an input dimension and adversarial risk in Sec. 5.1. Our analysis did not depend on specific data distributions or architectures, except for ReLU-like activations and random parameters, which is advantageous. In addition, we assessed it for a wide variety of norms, demonstrating that the impact of adversarial examples is not limited to the square root of the input dimension alone (cf. Tab. 1). Moreover, our bound considers not only the input dimension but also the number of classes.
A.3 Adversarial training
Numerous empirical adversarial defenses have been proposed; however, most are ineffective against stronger attacks [5, 12, 13, 20, 90]. Some studies have focused on theoretically certified defenses [19, 39, 43, 74, 96], but these are often only applicable to specific or small networks, or are weaker than empirical methods. Adversarial training [38, 58] is considered the most effective empirical defense against various attacks [5, 20, 90]. This involves training a classifier using a dataset that contains natural images and adversarial examples [38] or solely adversarial examples [58]. Various forms of adversarial training exist, including more effective loss functions [28, 48, 58, 94, 113], time-efficient frameworks [83, 97, 111], and procedures that preserve the clean accuracy [72, 113, 114]. Recent studies have demonstrated that combining adversarial training and data augmentation with unlabeled or generated data results in high robustness [40, 77]. Despite significant progress in empirical methods, a theoretical understanding of adversarial training remains incomplete. We provide a summary of theoretical studies on adversarial training, including robust generalization research.
Several studies have investigated the trade-off between robust and clean accuracy [29, 47, 75, 76, 92, 113], initially observed empirically [58, 88]. The trade-off has been proven inevitable even in the infinite data limit, assuming data is constructed from a moderately correlated single feature and weakly correlated many features [92]. Similar claims were found in [113] for different data distributions. It has been reported that the trade-off in finite data settings for linear and slightly more complex predictors can be mitigated with additional unlabeled data [76]. While comparable results were reported in [75], a contrasting study also exists [47]. Moreover, the trade-off has been shown to depend on class imbalance in a dataset using Gaussian classification models [29].
Various studies have examined the generalization gap of adversarial robust models [6, 50, 102, 107]. For example, ResNet [42] trained adversariably on CIFAR-10 [52] achieved  $96\%$  robust training accuracy but only  $47\%$  robust test accuracy [107]. The lower bound of adversarial Rademacher complexity [9] has been shown to increase with the square root of the input dimension for linear classifiers trained with  $\ell_{\infty}$  adversarial examples [107], which was later extended in [6]. Similar results using a tree transformation approach have been reported in [50]. However, the influence of network width and depth, or other training settings, such as training with  $\ell_{1}$  adversarial examples, on the bound remains unclear. The generalization gap has been investigated for linear regression models and two-layer neural networks with lazy training in data interpolation contexts [102].
Numerous studies have explored the sample complexity of robust generalization [1, 14, 62, 80, 110]. Robust learning may require a larger sample size than standard learning [80]. Subsequent research has indicated that unlabeled data can be sufficient to achieve robust generalization [1, 14, 62, 110]. The majority of these studies have focused on data sampled from Gaussian mixture models [1, 14, 110]. Moreover, the sample complexity of distributionally robust learning with perturbations in the Wasserstein ball has been examined [62], and some studies have suggested that the trade-off between robustness and accuracy can be mitigated using additional unlabeled data [75, 76].
Recent findings have suggested that robust classification requires complex classifiers [63], which is supported by the results in the neural tangent kernel regime [35]. In the context of transfer learning, robust classifiers have been shown to perform better [27]. Furthermore, a certifiable adversarial training procedure has been established, constraining perturbation by the distributional Wasserstein distance [86].
The aforementioned outcomes rely on specific data distributions, such as Gaussian or heuristic-tuned distributions, or simplistic models, such as linear classifiers or two-layer neural networks. The lack of theoretical research on adversarial training in deep neural networks stems from the complexity of training these models, including non-convex loss surfaces and stochastic optimization. Recent research has employed the neural tangent kernel regime to address these challenges, demonstrating that adversarial training can yield a robust network with near-zero robust loss [35]. This result was later extended in two-layer neural networks [115], eliminating the assumption in [35] that requires exponentially large width and runtime.
In this study, we conducted a theoretical analysis of adversarial training, focusing on the time evolution of network parameters during training, the conditions promoting adversarial training, and the differences between adversarial and standard training. Our analysis targeted deep neural networks without relying on any assumptions about data distribution and employed  $\ell_p$  norms practically used as perturbation constraints, rather than more impractical metrics such as the distributional Wasserstein distance. To address the training difficulty of deep neural networks, we utilized mean field theory.
Finally, we mention the report from a perturbation instability perspective that increasing network width does not necessarily improve robustness in adversarial training [98]. This may seem to contradict our results, which suggest that a wider network can help the model maintain capacity during adversarial training, implying greater robustness in wider networks. However, these two claims are compatible. Robustness is determined by both perturbation instability (negative effect) and network capacity (positive effect). While the negative effect of width appears dominant in [98]'s experiments on CIFAR-10 and WideResNet, the positive effect appeared more prevalent in our experiments on MNIST, Fashion-MNIST, and fully connected networks with or without shortcuts. The dominant factor may depend on the dataset and model architectures.
B Gradient independence assumption
The gradient independence assumption was first introduced in [81] for backward dynamics in mean field theory and later refined in [105]. We provide a definition based on [105] as follows: