Large-width asymptotics and training dynamics of $\alpha$-Stable ReLU neural networks
Authors: Stefano Favaro
Sandra Fortini
Stefano Peluchetti
Venue: Accepted by TMLR ()

Abstract: Large-width asymptotic properties of neural networks (NNs) with Gaussian distributed weights have been extensively investigated in the literature, with major results characterizing their large-width asymptotic behavior in terms of Gaussian processes and their large-width training dynamics in terms of the neural tangent kernel (NTK). In this paper, we study large-width asymptotics and training dynamics of $\alpha$-Stable ReLU-NNs, namely NNs with ReLU activation function and $\alpha$-Stable distributed weights, with $\alpha\in(0,2)$. For $\alpha\in(0,2]$, $\alpha$-Stable  distributions form a broad class of heavy tails distributions, with the special case $\alpha=2$ corresponding to the Gaussian distribution. Firstly, we show that if the NN's width goes to infinity, then a rescaled $\alpha$-Stable ReLU-NN converges weakly (in distribution) to an $\alpha$-Stable process, which generalizes the Gaussian process. As a difference with respect to the Gaussian setting, our result shows that the activation function affects the scaling of the  $\alpha$-Stable NN; more precisely, in order to achieve the infinite-width $\alpha$-Stable process, the ReLU activation requires an additional logarithmic term in the scaling with respect to sub-linear activations. Secondly, we characterize the large-width training dynamics of $\alpha$-Stable ReLU-NNs in terms an infinite-width random kernel, which is referred to as the $\alpha$-Stable NTK, and we show that the gradient descent achieves zero training error at linear rate, for a sufficiently large width, with high probability. Differently from the NTK arising in the Gaussian setting, the $\alpha$-Stable NTK is a random kernel; more precisely, the randomness of the $\alpha$-Stable ReLU-NN at initialization does not vanish in the large-width training dynamics.

Published in Transactions on Machine Learning Research (11/2024)
Large-width asymptotics and training dynamics of a- a-Stable ReLU neural networks
Stefano Favaro
reffono.aavrrouumitt.
Department of Economics and Statistics University of Torino and Collegio Carlo Alberto
Sandra Fortini
sundn..forttn@@hnboroon.
Department Decision Sciences
Bocconi University
Stefano Peluchetti Cogent Labs, Tokyo
pldd P.ggmallll..mmmmmmmmmmmmmmmm com
Reviewed on OpenReview: https: // openreview. net/ forum? id= AEAAAEmRbA
Abstract
Large-width asymptotic properties of neural networks (NNs) with Gaussian distributed weights have been extensively investigated in the literature, with major results characterizing their large-width asymptotic behavior in terms of Gaussian processes and their large-width training dynamics in terms of the neural tangent kernel (NTK), In this paper, we study large-width asymptotics and training dynamics of a-Stable ReLU-NNs, namely NNs with ReLU activation function and a-Stable distributed weights, with o E (0, 2). For (0,2), a-Stable distributions form broad class of heavy tails distributions, with the special case = 2 corresponding to the Gaussian distribution. Firstly, we show that if the NN's width goes to infinity, then rescaled a-Stable ReLU-NN converges weakly (in distribution) to an o-Stable process, which generalizes the Gaussian process. As a difference with respect to the Gaussian setting, our result shows that the activation function affects the scaling of the a-Stable NN; more preciscly, in order to achieve the infinitewidtth a-Stable process, the ReLU activation requires additional logarithmie term in the scaling with respect to sub-linear activations. Secondly, a-Stable ReLU-NNs in terms an infinite-width random kernel, which is referred to as the a-Stable NTK, and we show that the gradient descent achieves zero training error at linear rate, for a sufficiently large width, with high probability. Differently from the NTK arising in the Gaussian setting, the a-Stable NTK a random kernel; more precisely, the random- ness of the a-Stable ReLU-NN at initialization does not vanish in the large-iidth training dynamics.
characterize the large-width training dynamics of
1 Introduction
There exists a vast. literature on the interplay ween Gaussian processes and the large-width asymptotic
behaviour of Gaussian neural networks (NNs), namely NNs with Gaussian distributed weights NNea
1996:
Der and Lee
2006
Garriga-Alonse et al.
2018 Lee et al. 2018
Matthews et al., 2018 Novak et al.
2018
Yang
2019 a
Bracale et al. 2021 Eldan al. 2021
Klukowski 2022
Yang and Hu 2021 Basteri and
Trevisan 2022
Favaro al.
2023 Hanin 2023
Trevisan 2023 Hanin 2024). To define Gaussian NN.
consider the fo following elements: for d, k
let
X be d k NN's input, such that x; (I,I.... Fja)² is
the j-th input (column vector); ii) let & be an activation function; iii) for m 2 let W = (w (0) wm w)
(0)
be the NN's weights, such that (o(") (wi ...., ((()) and (wi,..., wm) with the w
(0)
(0) and the wy's

---

Published in Transactions on Machine Learning Research (11/2024)
being i.... according to Gaussian distribution with mean 0 and variance o². A Gaussian d-NN of width m is
= (W, æ1, Φ), (w, Ik, o)),
(1)
where
m fm, Σ wn xz))) j= I..,...,
(0)
Neal 1996)) first investigated the large-width behaviour of (m X, Φ), which follows by straightforward application of the central limit theorem (CLT). In particular, it is well-known that if + +00, then the rescaled Gaussian 6-NN m m/12fmm fm (W, Φ) converges weakly (or in distribution) to a Gaussian process
with covariance function Σx,e such that Ex.s|r,s] = o'E(o((w)
(w) .x,)). Some extensions of this
infinite-width limit available for deep NNs (Matthews et al.)
2018 more general NN's architectures
[Yang] 2019a bj, and infiniteddimensional inputs (Braaale et al.
2021 Eldan et al. 2021 Favaro et al.
2023
The large-width training dynamics of Gaussian NNs has been also extensively investigated in the literature. with the training being performed through the gradient descent (Jacot et al. 2018 Arora et al. 2019 Du
et al. 2019 Lee al. 2019). In particular, consider the Gaussian ReLU-NN and set
fm = fm (W, X, ReLU),
(W. == m/ fm XW.
Let (X,Y) be the training set, where Y = (y1,...,, is the (training) output such that corresponds to the j-th input Tz. By considering a random initialization W(0) for the NN's weights, and assuming a squared-error loss, the gradient flow of W(!) leads to the training dynamics of (W),, X), that is for any
df (W)), -fm (W(),)), Y) Hm, dt
(W(),,)) (W))
where Mmn is the learning rate, and Hm (W),,, X) is &o x k random matrix whose (i.j") entry is (W a1),, large-width training dynamics of fm(W(t),. (t), showing that: i) if m + too then Hm (W) converges in probability to a deterministic matrix H' (X, X); ii) the gradient descent achieves zero training error at linear rate, i.c.
By assuming T/m = 1, Jacot et al. (2018) first. characterized the
| - i x) S exp(->ot)) - (W(),,,
for m sufficiently large, with high probability. The limiting matrix (X, refereed to the neural tangent kernel (NTK). See Yang (2019) and Yang and Littwin/ (2021) for extensions to deep NNs and general architectures.
1.1 Our contributions
In this paper, study large-width asymptotics and training dynamics of a-Stable ReLU-NNs, namely NNs with ReLU activation function and a-Stable distributed weights. For (0,2), a-Stable distributions form broad class of heavy tails discributions, with the special case a 2 corresponding to the Gaussian distribution; see Samoradnitsky and Taqqu (1994) and references therein for an overview on a-Stable dis- tributions. According to the definition (1), we denote by fm (W,X, a;) the a-Stable ø-NN, namely a NN of the form (T] with the weighs W distributed according to the a-Stable distribution with a E (0, 2), thus excluding the Gaussian case a = 2. In particular, fm, fn, ReLU; denotes the a-Stable ReLU-NN.
1.1.1 Related work
considered a-Stable distributions to initialize NNs' weights, showing that while all Gaussian weights vanish the infinito-width limit, some o-Stable weights retain a non-negligible contribution. Such different behaviour may be attribute the diversity of the NN's path properties as (0, 2] varies, which makes o-Stable NNs more flexible than Gaussian NNs; Figure 1. Further works demonstrating practical
Neal
1996
2

---

Published in Transactions on Machine Learning Research (11/2024)
applications of a-Stable NNs, with respect to Gaussian NN's, are Der and Lee (2006 Fortuin et al. (2019)
Fortuin [2022), Lee et al. (2022) and Li et al. (2022); ; the empirical analyses developed in Fortuin et al. (2019) shows that wide a-Stable NNs trained with gradient descent lead to a higher classification accuracy than Gaussian NNs. Motivated by these works, Favaro et al. (2020 2021 first investigated the large with asymplotic behavior of fm(W; X, a)). particular, assuming a E (0, 2) and a sub-linear activation function Φ it proved that if m > +00, then the rescaled a-Stable @-NN m m-1/a (m, X, o; a) converges weakly to a-Stable process, that is a stochastic process with a-Stable fmite-dimensional distributions. See also Jung 2023
Figure 1: Sample paths of a-Stable NNs, random function mapping input in [0, E, with ReLU activation function and width 1024: top-left panel a = 2.0 (Gaussian distribution); ii) top-right panel = 1.;; iii) bottom-left panel = 1.0 (Cauchy distribution); iv) bottom-right panel 0.5 (Lévy distribution).
1.1.2 width asymptotics
We extend the main results of Favaro et al. one of the most popular activation function in the field of NNs. In particular, we show that if m > +00, then the rescaled a-Stable ReLU-NN (m log m0gmm fm (W, X;a) converges weakly to an a-Stable process. For NNs with a single input, i.e 1, the large-width limit follows by a direct application the generalized CLT for heavy tails distributions (Uchaïkin and Zolotarey] 2011 Bordino et al., 2022), whereas for > it requires develop an alternative strategy that may be of independent interest in the context of multidi- mensional o-Stable distributions (Samoradnitsky and Taqqui 1994 Chapter and Chapter 2). Differently from the Gaussian setting, the large-width asymptotic behaviour of a-Stable NNs shows how the choice of the activation function ø affects the scaling of the NN. More preciscly, in order to achieve the ffmite-width a-Stable process, the use of the ReLU aclivation in place of a sub-linear activation results in change of the scaling m- m1aa of the NN through the additional (log m))m//aa term. See also Bordino et al. (2022) for a detailed discussion of this peculiar phenomenon the context of a-Stable ReLU-NN with single input (k= = 1).
2020
2021 to the ReLU activation function, which is arguably
1.1.3 Large-width training dynamics
We investigate the large-width training dynamics of a-Stable ReLU-NNs, with the training being performed gradient descent. under the squared-error loss. In particular, consider the Staba ReLU-NN fm (W, X;
3

---

Published in Transactions on Machine Learning Research (11/2024)
and set
(W; (m // fm (W,X;a)
(3)
In analogy with (2), we define the training dynamics of fm ( (W (t)),;a),, with learning rate mm and a k k random matrix (tt,,, whose (i,j") entry is fm assuming the learning rate 7/m- = [logm)3/e converges weakly to an (e/2)--table (almost surely) positive defimite random matrix H XX; fi) and for every 8 > 0 the gradient descent achieves zero training error at linear rate, for m sufficiently large, with probability --6 6. The limiting random matrix (x,,, a;) is refereed to as the a-Stable NTK. Differently from the NTK that arises from the Gaussian selting, the a-Stable NTK is a random kernel. More precisely, the randomness of the a-Stable ReLU-NN initialization does not vanish in the large- width training dynamics.
(f(((())) (y); ())))))))))))) //W)). By show that: i) if m + too then (log m) m))/a Hm (W (0), a;
1.2 Organization of the paper
The paper is organized as follows. In Section we characterize its large-width asymptotic behaviour of a-Stable ReLU-NNs in terms of the infinite-width a-Stable process. In Section X we characterize the large- width training dynamics of o-Stable ReI.U-Nss in terms of the o-Stable NTK, and we show that the gradient descent achieves zero training error at linear rate, for sufficiently large width, with high probability. Section contains a discussion of our results with respect to some directions of future work. Proofs are deferred lo the appendix.
2 Large-width asymptotics of a-Stable ReLU-NNs
We study the large-width asymptotic behaviour of a-Stable ReLU-NNs. The section is organized as follows: i) we recall the definition of multidimensional a-Stable distribution (Section 2.1); ii) we define the a-Stable ReLU-NN and characterize its large-width asymptotic behaviour in terms of the infinite-width a-Stable process (Section 2 iii) we present some numerical illustrations of the large-width behaviour of a-Stable ReLU-NNs (Section ??). The main result of this section is Theorem 2.1 whose proof is deferred to Appendix
2.1
Multidimensional a-Stable distribution
For a E 20, a random variable S E R is distributed according to a symmetric and centered 1-dimensional a-Stable distribution with scale o > O its characteristic function is E(exp(izsS) = exp (2) and we write ~ St a). If the stability parameter a 2 then S distributed as Gaussian distribution with mean 0 and variance d². Let g be the unit sphere in R&, with k 2 1, and let I' be symmetric finite measure S For a 2), we say that random variable S E R is distributed according to symmetric and centered k-dimensional a-Stable distribution with spectral measure I if its characteristic function is
(Texpfi(2,5)) exp (t ()) sx, T(dss
34-1
and we write S ~ It, Let 1, be the r-dimensional (column) vector with 1 in the 7-th entry and 0 elsewhere, for any = 1, 1,...,ke h. Then, the r-th element of S, that is S1, is distributed as an a-Stable distribution with scale
1/a
o=
We deal mostly with k-dimensional a-Stable distributions with discrete spectral measure, that is T(.) Eision (-) with n E N, % E R and E gk-1, for i= 1,..., (Samoradnitsky and Taqqu 1994 Chapter 2). All the random variables are defined on a common probability space, say (52, (12,-,P)) IP). unless otherwise stated.
We make use of the following characterization of the spectral measure of a-stable distributions (Samoradnit- sky and Taqqu 1994 Chapter 2): if S (t, T), then for every Borel set B of g such that T(aB) 0.
4

---

Published in Transactions on Machine Learning Research (11/2024)
it holds
S
lim P
|| >
E
B Car(B),
||||
where
1-a C. [[2-a) c0s(5/222
a ##1
(4)
a=1. 1.
The proof of this result is reported in Appendix to the domain of attraction of the (a, I') distribution, with E (0,2) and I simmetric fimite measure on gk-1, if and only if
B Moreover, the distribution of random vector belongs
[IEIl Va E IEl
EA) Car(A)
lim nP n-00
(5)
for every Borel set A of S such that I(0A) = We refer to Appendix B for more details the derivation of See also Samoradnitsky and Taqqu 1994 Chapter 1 and Chapter 2) for further details on the constant
C...
2.2
The infinite-wine width a-Stable process
To define a generic ReLU NN, let us consider the following elements: i) for d, 2 let X be the d x k
NN's input, such that 2z = (Zii...., is the j-th input (column vector); ii) for m 2 1 let W
(w()
(0) (0) w) be the NN's weights, such that wi"h (w ...., (0(0) and w (wi,...: Wm.. A ReLU-
NN of width m
fm (W, = (W,, (W
(6)
where
fm (W, = Σ wr(w; (0) 2r,[[(u5 (0) xz)) > 0) j = 1, I.....k,
(0) (0) ..)...., w (O), (()) the NN's weights with I(-) being the indicator function. We denote by W(0) (w)
(0) 's and the wy's 's are initialized as i.i.d a-Stable random at random initialization. If the NN's weights wiy variables, with a E 20, and a > 0, then (m(W(() (0 a) defines an a-Stable ReLU-NN of width m.
Theorem 2.1. For any E (0,2), let fm (0), X;) be an a-Stable ReLU-NN of width m. If m + to0 then
1
Imlogm) 1/a m/ fm fm (W(O), X:a) - S(X),
where f(X) Sta with
() C. Σ (11 (),1))) [[ ([ {)))))) (X) Tx
such that
(x) 8 [2nl [z (zzz > 0)}
[x 0],
+
| (zzz 0)lil
| 0)
and
DJ (X) = [Zz][(z < <0], 0],
(1 < 0)1,l + (-1n < 0)],
(
[2
[T < 0)],
where, for any E gk-1 8(s) is the probability measure degenerate in 8, and Ca is in 4 The stochastic process = (f(x),,, ((xx)), as a process indexed by X, un atStaale process with spectral measure Tx.
Sketch of the proof of Theorem The a-stable ReLU-NN of width m is a sum of m independent and identically distributed random vectors. The proof relies on the analysis of the tail behavior of these sum- mands, and it exploits characterization of the multivariate o-Stable distribution the limiting distribution
5

---

Published in Transactions on Machine Learning Research (11/2024)
of sums of independent random vectors that exhibit specific tail properties. We refer to Appendix A.1 for details.
For a broad class of bounded or sub-linear activation functions,
Favaro et al. (2021) characterizes the large-
width distribution of deep o-Stable NNs. See also Bordino et al. (0022 and references therein. In particular, let
(n (,, 03 =
Σ wi (
(0) zz}
be the a-Stable @-NN of width m for the input 2jz for j function. Let fm (X(;a) ai) = (fm(=1;))..... .... (m a)). From Favaro al. 2021 Theorem 1.2), if m +00 then
1,....,,, k with being bounded activation
m Jm (W, X, 0z
W. f(X),
(7)
with f(X) being an o-Stable process with spectral measure Tx xx Theorem 2.1 extends Favaro al. (2021
Theorem 1.2) to the ReLU activation function. Theorem 2.1 shows that the use of the ReLU activation in place of a bounded activation results in change of the scaling m m/ in 77. through the inclusion of the (log 3) term. This is a critical difference between the a-Stable setting and Gaussian setting, as in the latter the choice of the activation function & does not affect the scaling m m-1/2 required to achiove the infinite-width Gaussian process. For 1, we refer to Bordino et al. (2022) for a detailed analysis of infinitely wide limits of a-Stable NNS with general classes of sub-linear, linear and super-linear activation functions.
Remark 2.1. The need of the additional log(m) can be clarified by considering the a-Stable ReLU-NN with a single input, i.e. k =, where the proof of Theorem 2.1 reduces to straightforward application of the generalized CLT for heavy tails distributions (Uchaškin and Zolotaren 2011 Bordino et al. 2022).. In particular, refer to Theorem 2.1. and Theorem 2.6 of Bordino et al. (202%) which show how the log term arises from the tail behaviour of the product of o-Stable random variable w. w.w, "s, which defines the NN; see Cline (1866) and references therein. The log term expected to hold for any activation that has a linear growth.
To demonstrate numerically Theorem 2.1 we sample random neural nelworks according to for various values of width m and stability index a. We evaluate these networks on a fine uniform grid of points in jo, 113. Figure 2 displays the results, which show that the function samples remain well-behaved grows larger.
3 Large-width training dynamics of a-Stable ReLU-NNs
We study the largo-width training dynamics of a-Stable ReLU-NNs. The section organized as follows: we define the training dynamics of the a-Stable ReLU-NN and characterize its trge-width asymptotic behaviour in terms of the a-Stable NTK (Section 3.1] ii) show that the gradient descent achieves zero training error at linear rate, for sufficiently large width, with high probability (Section 3.2). The main results of
this section are Theorem 3.1 and Theorem 3.2 whose proofs are deferred to Appendix A.2 and Appendix respectively.
A.4
3.1
The a-Stable NTK
Let fm (W, be the a-Stable ReLU-NN defined in G with a (0.2), and let (X, be the training set, where Y (V1,..., Uk) w is the (training) output such that y, corresponds to the j-th input 2z. Then, we set.
1
Fm (;a) = (m log m/a fm (W, x:a),
such that (w, a)) = (m m) f fm, (W, a)
the (model) output of the j-th input z,, for j ....., k. Assuming the squared-error loss function E(y,, f' (W, 2-11 Um n,) by

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 2: Sample paths of a-Stable NNs, as random function mapping an input in [0, 1] to R, with a ReLU activation function; width values (left to right): m = 64, 256, 1024, 4096; a = 1.5 (top panel), a == 1.0 (bottom panel).
a direct application of the chain rule we obtain the training dynamics of f... (; In particular, for any 2 0
df (t), (fm, (m
(8)
where the kernel Hm, (W), X) in the NN's training dynamics is h: x h random matrix whose (j.j") entry is
(w)) X)b.," afm
( (W(t), (t))))yy;)) Zy;a))
aw
aw
and Tm is the learning rate. The training dynamics for fm (W, x:a) is standard, and if follows training dynamics presented in Section therein for details.
for the Gaussian setting. See (Arora et al. 2019 Section 3) and references
For the training dynamics we study the large-width behaviour of Hm (0), x) in 9 In particular, we set
(W), (0), (log m) 2/a Hm Hm (O))
(10)
and show that if m + +00, then A.. (W) converges weakly to a positive definite random matrix ( X, a) distributed according to an (a:2))Sttable distribution. To prove this result useful to decompose Hm (W)),, as
Am (W X) A (W(O), ( (W()),
(11)
where
m
(W()
w (() (0) , 0)7((1( (0) (xy)
0),
(12)
m2/a
and
0 XIb.5") = m/a
(w) (0)
IT ((w:
(0) ,xz) > ()( II ((w, (0) ,2,) 0),
(13)

---

Published in Transactions on Machine Learning Research (11/2024)
respectively. The next theorem characterizes the large-width asymptotic behaviour of (W((), X) in terms of the a-Stable NTK.
Theorem 3.1. For any a E (0, (), let Hm (W( matrices whose (j.j') entries are defined in 10 12 and u E {0, let
(0),0))..)))))))) X).
Am (W)) X) and A (W(0), X) be the k k random and 13 respectively. Moreover, for every 2 1
Bu = {o E EURRR
> = +
50 uf = 0.j kk
and for every i= 1,, ,,..,.,,,,, d, lel be a d-dimensional vector such that esg = for jorj = and eij = 0 for j i. As m + +00,
(1 (W X), ( " (ii(a), (a))))
where (o) and (a) are k x random matrices that are stochastically independent, posilive semi-definile, and distributed according to (a:2)) distributions with spectral measures I and F22, respectively, such that
( bigr (u
r = Caf2
Σ
P(u, (0) (0) E B.)
(14)
uE(0.1)*
EEgrr (2) Izg. )
and
[
8
E, 2j.us
r = Caf2
Σ
Σ
(15)
-a/2
wE{0,1)* (i:(es-ee)nBure)
with Caf2 in 4 Furthermore, as m + 00,
Hn (W(),, X) " (X;))
where (X;));;) is a k x le random matrir that is positive semi-definee and distributed according to an (a/2)-Stabll distribution with spectral measure D + +.. H (X, a;a) is refereed to as the a-Stable NTK.
Sketch of the proof of Theorem 3.1 We can see (H (W()), Ar (W()), as a random vector of dimension 2k2, with k whose elements are sums of independent and identically distributed random vectors. The proof relies the analysis of the tail behavior of these summands, and it exploits charac- terization of the multivariate o-Stable distribution limiting distribution of the sum of independent and identically distributed random vectors that exhibit specific tail properties. We refer to Appendix A.2 for the details.
It turns out that the (a/2)-Stabr) distributions of the limiting random matrices (if(aa and (a) are absolutely continuous in suitable subspaces of the space of symmetric and positive scmi-definite matrices; see Lemma A.4 and Lemma A.5 for details on the distribution of the random matrix Hī (a), and Lemma A.6 and Lemma A.7 for details on the distribution of the random matrix Hz(a). This applied in the next theorem to show that the minimum eigenvalues of A (W(O), X) and of H (X) are bounded away from zero, uniformly in for sufficiently large, with arbitrarily high probability. Accordingly, the minimum eigenvalue of Hm (W(,), X) 77 (W (O) + H (W (0), X) bounded away from zero, uniformly in m, for sufficiently large, with arbitrarily high probability. We denote by Amin(.) the minimum eigenvalue.
Proposition 3.1. For any E (0, let (W, A ( () and f. (W, be the random matrices in Theorem 3.1 For every 8 > 0 there exist strictly positive numbers du, A1 and Xg such that, for sufficiently large,
Ami (1 (W))), A i= 1,2,
and
Amin Hm (W X)) An-
with probability at least 1 8.. 8.
8

---

Published in Transactions on Machine Learning Research (11/2024)
See Appendix A.3 for the proof of Proposition 3.1 of some of the main results of Jacot et al. (2018) to the setting of o-Stable ReLU NN, for a E (0, 2). See also Du et al. 2019 Arora et al. (2019), Lee et al. (2019) and references therein. In particular, our results show that
Theorem 3.1 and Proposition 3.1 provide an extension
as m + t00, the random matrix
X) converges weakly to the a-Stable NTK H (X,)x;;),, such that (X, a;a) is (a/2)-Stable (almost surely) positive definite random ma- trix;
ii) at random initialization for the a-Stable ReLU-NN, for every d > 0 the minimum eigenvalue of the random matrix (W(),, remains bounded away from zero, for m sufficiently large, with probability 8.
Differently from the NTK that arises from the Gaussian setting, the a-Stable NTK is a random kernel. That is, the randomness of the a-Stable ReLU-NN at initialization does not vanish in the large-width training dynamics. Such a randomness makes more challenging the study of the corresponding large-with training dynamics.
3.2 Zero training error at linear rate
Under the training dynamics S we show that for every d > 0 the gradient descent achieves zero training error at linear rate, for sulficiently large, with probability 1 - 6. In order to prove this result combine Proposition [3.1 with the next proposition, which shows that, if m is sufficiently large, then with high probability the minumum eigenvalue of the random matrix (W (Wt,,, X) remains bounded away from zero. We denote by I| || and || ||2 the Frobenius and operator norms symmetric and positive semii-definie matrices, respectively.
Proposition 3.2. Let y E (0,1) and c O. For l 2 1 let the NN's inputs Ti....,22 be linearly independent and such that ||,د||| 1. For any E (0,2), let Hm HW, X) and H (W,X) be the random matrices in Theorem 3.1 For every 5 > 0 the following properties hold for every t 2 0, with probabilily at least 5, for sufficiently large:
(i) for every j lk....k,
05 (W(t),
afm dw
1/ga (log 2/a
(w (W(0),z;:))
< <om 22//
Ow
(i)) there erists do such that
< Aam 7//
and
Ao
(H, X)) 2
Sketch of the proof of Proposition 9.2 The inequality displayed in (1) holds as long as W(L) stays within a neighborhood of W (0) with radius on the order of (log m)2 and viceversa. This implies that the fluctuations during the training of the a-Stable ReLU-NN vanish as m - oc. Consequently, the first (ii) also holds throughout training if m is large enough. Together with Proposition 3.1 this ensures that the minimum eigenvalue of the random matrix f (W(),, remains bounded away from zero during training. The same argument applies to the random matrix Hm (t),,, which is the sum of (x(3) (W(t), ) X) and of the non-negative definite matrix A (t), X). We refer to Appendix A.4 for the details.
of (t), inequality displayed in

---

Published in Transactions on Machine Learning Research (11/2024)
Now, we are in the position to show that the gradient descent achieves zero training error at lincar rate, for sufficiently large width, with high probability. From Proposition 3.2 for a fixed d >> 0, let m and Ao > 0 be such that
Ao
Amin (M (s),
2
for every 8 S on a set N E F with P[N] > 1-8. Accordingly, for any random initialization W (O)),,, with w E N,
d ds IY - (W)) - - - (M))))) X;a))
and hence
- -x (x) S.
ds
Therefore, by observing that exp(Ans)]IY (W)) x;a) 2 decreasing function of 8 >> 0, then we write
||Y - (W()(()) expl- Soo(-Aos)lly- (M(((()))
In the next theorem we summarize the main finding on the large-width training dynamics of o-Stable ReLU NNs.
Theorem 3.2. For k 2 1 let the NN's inputs It... N1s...,2 Iu be linearly independent and such that || = 1. For any a E (0,2), under the training dynamics ()) if the learning rate 7m = (log m)2/a then for every there erists d > such that, for m sufficiently large and any t 0, with probability at least 6 it holds true that
||Y (m(())) exp((1ot)) (M))))
4 Discussion
In this paper, we investigated large-width asymptotics and training dynamics of a-Stable ReLU-NNs, namely NNs with ReLU activation function and a-Stable distributed weights. With regards to the large-width asymptotics, our result (Theorem 2.1 extends the main result of Favaro et al. (2020 2021 to the ReLU activation function, showing the need of an additional logarithmic term in the scaling of the NN to achieve the infhh a-Stable process. With regards to the large-width training dynamics, our results (Thcorem 3.1
and Theorem 3.2 extends some of the main results of Jacot et al. (2018) to a-Stable ReLU-NNs, showing that randomness of the a-Stable ReLU-NN at initialization does not vanish in the large-width training dynamics.
It remains open establish width equivalence between training an a-Stable ReLU-NN and performing kernel regression with the a-Stable NTK. For Gaussian NN, Jacot et al. (2018) showed that during training O, if is sufficiently large then the fluctuations of the squared Frobenious norm || (W (t), X) Hm (W)), X are vanishing. This suggested to replace nm Hm Hm (W()) with the NTK H (X, in the
>>
dynamics
2
and write
df - -f(x di
This the dynamics of a kernel regression under gradient flow, for which at + t00 the prediction for generic test point Rd is of the form fx( = YH* HX of the Gaussian NN fm (W),,, x:) at +- +00, for m sufficiently large, is equivalent to the kernel regression prediction (x) Arroa et al. 2019). Within the a-Stable setting, it is not clear whether the fluctuations of Hm (t), H!!!! (t) X) (2 (t),, during the training vanish, as m - co. Proposition 3.2 shows that the fluctuations of ( (W Xt) vanish, as m + so. Such a result is based on the fact that for every 5 > 0 it holds that
In particular, the prediction
@J (W, (W,zza)
afm (W x2
2
(logm)2/e 2/a
< cm 27/0
dw
Dw
10

---

Published in Transactions on Machine Learning Research (11/2024)
for every j = I...., k and for every W such that | - (logm)2/o, with probability at least 1 1-66 6, if m is sufficiently large. In particular, refer to Lemma A.8 for details. The same large-width property is not true if the partial derivatives with respect to w are replaced by the partial derivatives with respect to w()) Accordingly, it is not clear whether the fluctuations of H (W),, during training also vanish, m 00.
Another interesting avenue for future research would be to extend our results to the more general setting of deep NNs, with 2 2 being the depth. Let us consider the following setting: i) for d, k 2 1 let X be the d k NN's imput, with zj (.51...., Faa) being the j-th input (column vector); ii) for D, m 2 and n 2
let: (W(),, ....., W(D) be the NN's weights such that W(() (wf.).... .......... m and W (wi,1
( w (I)
for 2 <
D, where the w i.i.d. a-Stable distribution with scale 0, e-g- we assume o= 1. Then,
d Σ wij
1 (X;a)
(1)
Dj
5=1
and
1 (X;a)= = Σ 1
( (X. (X) > 0)
j=1
with f (X;a) (x; is a deep a-Stable ReLU-NN of depth D and width If the NN's width grows scquentially over the NN's layers, i.c. too layer at time, it is easy to extend Theorem 2.1 f! (X: Under the same assumption on the growth of m, we expect the analysis of the large-width training dynamics to follow along lines similar to that of Theoree 3nd Theorem 3.2, though computations may be more involved. A more challenging task would to extend our results to deep a-Stable ReLU-NNs under the assumptions that the NN's width grows jointly over the NN's layers, i.e. m - +00 simultaneously over the layers
Acknowledgments
The authors wish to thank the Action Editor, Professor Murat A. Erdogdu, and three anonymous Referees for their helpful suggestions. Stefano Favaro funded by the European Research Council under the Horizon 2020 research and innovation programme,, grant 817257. Stefano Favaro also gratefully acknowledge support from the Italian Ministry of Education, Universily and Research, "Dipartimenti di Eccellenza" grant 2023-2027.
References
ARORA, S., DU, S.S., HU, W., LI, Z., SALAKHUTDINOV, R.R. AND WANG, R. (2019). On exact compu- (ation with infinitely wide neural net. In Advances in Neural Informalion Processing Systems.
BASTERI, A. AND TREVISAN, D. (2022). Quantitative Gaussian approximation of randomly initialized deep neural networks. Machine Learning 113, 6373 639.
BORDINO, A., FAVARO, S. AND FORTINI (2022). Infinite-wide limits for Stable deep neural networks: sub- linear, linear and super-linear activation functions. Transactions of Machine Learning Research.
BRACALE, D., FAVARO, S., FORTINI AND PELUCHETTI, S. (2021). Large-width functional asymptotics for deep Gaussian neural networks. In International Conference on Learning Representations.
CLINE, D. (1986). Convolution tails, product tails and domains of attraction. Probability Theory and Related Fields 72, 525-557.
DER, R. AND LEE, D. (2006). Beyond Gaussian processes: on the distributions of infinite networks. In Advances in Neural Information Processing Systems.
11

---

Published in Transactions on Machine Learning Research (11/2024)
DU, S.S., ZHAI, X., Poczos, B. AND SINGH, A. (2019). Gradient descent provably optimizes parameterized neural networks. In International Conference on Learning Represenlations.
ELDAN, R., MIKULINCER, D. AND SCHRAMM, T. (2021), aon-asymptotic approsimations of neural networks by Gaussian processes. In Conference on Learning Theory.
FAVARO, S., FORTINI, S. AND PELUCHETTI, S. (2020). Stable behaviour of infinitely wide deep neural networks. In International Conference on Artificial Intelligence and Statistics.
FAVARO, S., FORTINI, S. AND PRIUCHETTI, S. (2020). Deep Stable neural networks: large-width asymp- totics and convergence rates. Bernoulli 29, 2574-2597.
FAVARO, S., HANIN, B., MARINUCCI, D., NOURDIN, AND PECCATI, G. (2023). Quantitative central limit theorems in deep neural networks. Preprint arXto:2307.0009
FORTUIN, V. (2022). Priors in Bayesian deep learning: a review. International Statistical Review 90, 563- 591.
FORTUIN, V., GARRIGA-ALONSO, A., WENZEL, F., RATSCH, G, TURNER, R.E., VAN DER WILK, M. AND ArrCHISON, L. (2020). Bayesian neural network priors revisited. In Advances in Neural Information Processing Systems.
GARRIGA-ALONSO, A., RASMUSSEN, C.E. AND AITCHISON, L. (2018), Deep convolutional networks as shallow Gaussian processcs. International Conference on Learning Representation.
HANIN, B. (2023). Random neural networks in the infinite width limit as Gaussian processes. The Annals of Applied Probability 33, 1798-4814
HANIN, B. (2024). Random fully connected neural networks perturbatively solvable hierarchies. Journal of Machine Learning Research 25, 58.
JACOT, A., GABRIEL, F, AND HONGLER, C. (2018). Neural tangent kernel: convergence and generalization in neural networks. In Advances in Neural Information Processing Systems.
JUNG, P., LEE, H., LEE, J. AND YANG, H. (2023). a-Stable convergence of heavy-/light-tail infinitely wide noural networks. Journal of Applied Probability 55, 1415-1441.
KLUKOWSKI, A. (2022). Rate convergence of polynomial networks to Gaussian processes. In Conference on Learning Theory.
LEF, H., AYED, F., JUNG, P., LEE, J., YANG, H., CARON, F. (2022). Deep neural networks with dependent weights: Gaussian process mixture limit, heavy tails, sparsity and compressibility, Journal of Machine Learning Research 24, 1-78.
LEE, J., SOHLDICKSTEN, J., PENNINCTON, J., NOVAK, R.,, SCHOENHOLZ, S. AND BAHRI, Y. (2018).
Deep neural networks as Gaussian processes. In International Conference on Learning Representation.
LEE, J., XIAO, L., SCHOENHOLZ, S., BAHRI, Y., SOHL-DICKSTEIN, J. AND PENNINGTON, J. (2019). Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Newral Information Processing Systems.
LI, C., DUNLOP, M. AND STADLER, G. (2022). Bayesian neural network priors for sige-preseeving inversion. Inverse Problems and Imaging 16, 1229-1254.
MATTHEWS, A.G., ROWLAND, M., HRON, J., TURNER, R.E. AND GHAHRAMANI, Z. (2018). Gaussian process behaviour in wide deep neural networks. In International Conference Learning Representations.
NEAL R.M. (1996). Bayesian learning for neural networks. Springer.
12

---

Published in Transactions on Machine Learning Research (11/2024)
NOVAK, R.,, XIAO, L., BAHRI, Y., LEE, J., YANG, G., HRON, J., ABOLAFIA, D., PENNINGTON, J. AND SOHLDICKSTRIN, J. (2018). Bayesian deep convolutional networks with many channels are Gaussian processes. In International Conference on Learning Representation.
SAMORADNIISKY, G. AND TAQQU, M.S (1994). Stable non- Gan-Gausiaan rondom processes: stochastic models with infinite variance. Chapman and Hall/CRC.
TREVISAN, D. (2023). Wide deep neural networks with Gaussian weights are very close to Gaussian processes. Preprint arxi::3312.11777
UCHAIKII, V.V. AND ZOLOTAREV, V.M. (2011). Chance and stability: stable distributions and their appli- calions. Walter de Gruyler.
YANG, G. (2019). Tensor programs II: neural tangent kernel for any architecture. Preprint or 0rXiv20006.45545
YANG, G. (2019). Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural langent kernel derivation. Preprinl: urXiv 760.
YANG, G. (2019). Tensor programs I: wide feedforward or recurrent neural networks of any architecture are Gaussian processes. In Advances in Neural Information Processing Systems.
YANG, G. AND HU, E.J. (2021). Tensor programs IV: feature learning in infinite-width neural networks. In International Conference on Machine Learning.
YANG, G. AND LITTWIN, (2021). Tensor programs llb:
architectural universality of neural tangent kernel training dynamics. In International Conference Machine Learning.
13

---

Published in Transactions on Machine Learning Research (11/2024)
A
A.1 Proof of Theorem 2.1
To simplify the notation, we set in this section: w == w(0), w := (0), and W := (0). First, we will prove that / ((w) mcasure
(0) xz)) > O)l5 belongs the domain of attraction of an a-stable law with spectral
11 Calburto (Ill,))))((x>>>>> 0111118
[u, ((a,)) [0), [[[,, (7)) xg)) > 0)151l
where To is the spectral measure of w For this, it is sufficient to show that
p ( 2,, .a,) 0)l E B, I1 ( xz)) 0]1i|>
(2) (0) .ry) ,x,) 0)1ر
C.Ti(B).
for every Borel set B of g such that (0B)- = (see Appendix B Let T g + [0, 1 and C R {0} + gh-1 be defined as Twu) = (y), zy) > Oly and C(v) = l/|lui], respectively. Fix Borel
set B of gk-1 such that (OB) = 0. This condition implics that
To (fu E st-1 |[T(w)] # (0, E (BB)))
EBB) To (fmm (L [ |T([i]] # 0, [T(x)]] E 0B }) = 0.
T(u)
Hence
To (T (f [0, 11 || * 0, E (C-)(B))))
= =To (T ((2 E [o. 1] ||||| + 0, = 0.
Now, let 2 = =Tww; /|| (0) ())
: 1 + 0). We can write that
Kw
(0)
,,,) 0), E B, ( (0) xz)) > 0]][>> >r)
[1( (0) ,ry) (110
.. > 0)1
Z
=r
||||| + 0, |
E B, |00 (0) ||| > r)
P (0)
> -||=||- 2 E dz)
(0)
Jo Jo-10mynta1)
(=(")( ( r||
W
E
T
(dz))
Since To (T-1 (f2 e [0, 11f (0.2 = 0, then the points of discontinuity of the function (CI (B) have zero ()(T--()) follows that
Jovenmouu (=10((1 r||2||-1 .w (dz))
Ca J
So-1 (1((TT((~'(dd))
= C. I T(uE B)
( T(u)
((T(w))(T((du)) (
Bt-
= CB),
14

---

Published in Transactions on Machine Learning Research (11/2024)
oo, which completes the proof that [w, zij ) ((m of an a-stable law with spectral measure I1. Then, for every k-dimensional vector s,
(0)
(0) 2.3))
O]]; belongs to the domain of attraction
m
k
Σ sy(w, 10)
Σ
(y) " ,x)) > 0),
m/aa
as a. sequence of random variables in m, converges distribution, as m - +oc, a random variable with a-stable distribution and characteristic function
exp (-11) E1t (1 Σ
sy (1)(,,)) > 0)9)).
j=1
Thus, the distribution of شッ 84 (w: ( (0)
(0)
xz)) 0) belongs the domain of attraction of an a-stable
law. In particular, this implics that m + to0
&
85 (wj
(0)
((m > 0)1 0))rr)
k + CoTTo
((1,))) 0)1)
We now study the tail behaviour of |u: Ej= 83 (w (0) x.y [( (10 (0) ..y) > 0) By Cline (1986 Section 5),
P [x
8y ww . (( (0) .2)) > 0)1 et) = F= G(()),
where
F(t) = P | >
[1ss)>e) )
10 x2) ((w:) , > 0)1 > e
G(t) = P
sy (w;
We now prove that F' and satisfy the assumptions of Cline 1986 Theorem 4) with 3 = 7 = 0. The distribution functions and have exponential tails with rate a. Indeed, for all real u,
lim F(L- u) lim F(t) t-000
P([w:] > el-u) ea((-- P( et) e-at ccu
Analogously for G. Moreover the functions b(t) F(t and c(t) eat are regularly varying with exponent zero: for all y > 0,
lim b(y() lim t
a P(wu) P(tu(
gayle -ayt lim 4000
= =1= y0
b(t)
(1000
e²)
eate-
The same property holds for e(t). By Cline 1986
Theorem 4 (v)), as t - 00,
IP |wal
83 (w:
,r T( xn) > 0) > e¹ = G(1((
1=1
C Eunnoo (1
Σ
((u, > 0) atte)
1=1
15

---

Published in Transactions on Machine Learning Research (11/2024)
co. Thus, for oc,
r P
way 12 sy ,
((w) .2)) 0)1 rr)
02 L
>
Let [(r) = CoEuroo (BEnrro(EEe 3y) 83 (mc [(wu,
0) logr,
Since the distribution of
(0) (0)
0z) > 0) is symmetric, then write that
(w) T(10 xz)) > 0),
Oym
sequence of random variables in m, converges in distribution, +o0, to random variable with symmetric a-stable law with scale provided (2m)m21 satisfies
mL((mm) Ca
+ The condition is satisfied if
1/a
Om Ca CuBuuoo
2, > 0)( ) log j=1
It follows that
8j (w.
..) 2z) ((1u (0) ,r)) ,,,) > 0),
(m log 2/a
1/a
1=1
as a sequence of random variables in m, converges in distribution, as m + toc, a random variable with symmetric a-stable distribution with scale of the form
1/a
C. C. (1 Σ
(() (()())))))00))))
Ø=1
Since this holds for every vector s, then
Σ
wxl(w; (0) 22y) (0) .zz) > 0)1E
(m log 1/a
as a sequence of random variables in m, converges in distribution, as m + +00, to a random vector with symmetric a-stable law with the spectral measure
F'x Cn [Il(u,,s)((uu >)
[u, X1, 24)) [0){)>>)]]-1
0)1
5 ,,, ( 0]1 x; 0)]>
[[( 2.y) x.)) > 0)];1|
Since To 1 Dd (6(e,) + 8(-e;)), where eij = 1 if j = and otherwise, then
Co Tx
(
(5( [ 0}]
8 [Zrj [1rs 0),
[<55 > 0)1,11
1|2 > 0),11 || 0)l,
+|| +1||2g: Txiss < <0]b11 (5( [0 0)} + +
[[ (2rgs < 0 )
[ < 0),
1225 0)],
16

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Proof of Theorem 3.1
To simplify the notation, set in this section: w == w(0), w (0)) w W Am in (W))), X) and A/ = Hm (O))),); with f (W, x) and Am (W, defined
12 and
13
The proof of Theorem 3.1 is split into several steps.
Lemma A.1. If m too then
# - ())()
where (a) is on (a/2)-Stable positive semi-definite random matrix with spectral measure
[ Tgz Sackkyuuuupp /u
1 = Caf2
Σ
P(w (0) E BB)()
<B)
(1 (y
Σ
ji (1is 2gg Fuz )2/2
3
where, for every u E {0, B, E Rdd (1, > uf = 1, (1, S if u, = o,J 1,..., k). and Ca/2 is the constant defined in
Proof. Since ! is symmetric, is sufficient to show that, for every k-dimensional vector s,
T TH 5 S Hj
We first prove that the functions defined, for t E -00), by F(t) = P w > ) and
G(t) P
sjsy 21)))) (0) ,z,)) 0) (0) 224) 0) e¹
5,'==1
P
8y ( ((w (0) ,Iz) 0)11² > et)
satisfy the assumptions of Cline the properties of the stable law,
1986
Lemma 1). Indeed, F has exponentail tails with rate a/2, since by
F(1 - u) lim
P(we: e(l-u)/2) ca/22
lim
F(t)
P( [w et12)
Moreover, for any 7,
mc(1) {o ed"G( F(I) Σ
8,J 850, I(w, (0) .z)) 0)11) xo.
#=1
By Cline 1986..
Section 5 and Lemma 1, as t - 00,
&
8388 (r,, 2. ((m) ,,,,)) >
(0) ;,((0,)))))))))
JJ'=1
F G(t mG ma
(e')
a ( Σ
> ()((( ,x,) > 0))
=--=1
By the properties of the stable law,
T (1)) g
W Σ
2 .2,) 0)(((u(),2))
(2)) 0)
m2
2/a
=1
17

---

Published in Transactions on Machine Learning Research (11/2024)
converges weakly, as + co, to totally skewed to the right, a/2-stable random variable, with scale parameter E( Syy xz)) > 0)(((u} ( ,z))) > 0) a//3 2//a/a ) Hence, for every & E R, as m + 00,
E
explits" (iiits"H(111) H())ss)
exp
-1110/2B
Σ
(.))
.x)) "
..y) > 0) (1) sign tan(mra/44)
=5'=1
= exp
Σ
1888 10/2 10/2(11 sign(t Σ
)ررد Lan(Ta//4)T 3
1.3
where
ri Ca/2B (1kks, ,,y) ((, () (0)
mry) 0)lis 1///2
( (0) .zn) ((1( (0) .zz²) > 0)
(((0),zz))(((( x.)) 0)7((" (0) ..y) 0) 11r llm)) )
It follows that, - +00,
A " (a).
where (a) is an (a/2)-Stable random matrix with spectral measure r of the form
P(w; (0) E Ba) (I, (xn 2yz ( ( [ ay bg B is
Caf2
Σ
(Ejsr (1 Tg Ru
(0,11
We will prove that (a)) is positive semi-definite. By definition, 1 (w) is positive semi-definite for every and every m. By Portmanteau Theorem, for every vector E Sk-1
P (u T (j)a) 2 0) 2 lim (u" ( 2 0) =1
Let A be a countable dense subset of S*-1, Then, with probability one, a HT 2 0 for every E A. By
continuity, this implies that the same property holds true with probability one for every St-1, which proves that (a is almost surely positive semi-definite. By eventually modifying H; (a) mull set, we obtain positive semi definiefnite random matrix.
Lemma A.2. If m + to0 then
A " Ha)a),
where (a) is an (a/2)- positive semi-definite random matrix with spectral measure
[ Σ3 23
r = Ca/2
Σ
Σ
Σ
we " (elea.ee)nnB.kkk #0)
where By {u E Fd (v, > 0 if 11 < 0. k), ea is a d- d-dimensional vector
salisfying eij = 1 ifj j i, and egj ( 1..., d), and Caf2 is the constant defined in
18

---

Published in Transactions on Machine Learning Research (11/2024)
By the properties of the multivariate stable distribution (sce Appendix B it is sufficient to show that
Proof.
[1ot .zz) (0) M. [((1]) (0) . () (0) ..y) > 0) j.j E
P
(wof . 10) u
(0) xz)) ((m]
0) 00 ..g)) > 0) 13
||F
|| [taf (z) 2z)) .z)) 2z) 0) ,xx) 0) ju
|F >
a/))())
88 r - +00. We can write that
(wof" ( (0) Fyg . [((1)) 10) .ay) ()7(((]
(0) ..y)) > 0)
IP
1.j"
E
(wof") 2x3)) (m) 0 .z)) 0) xzy) 0) ju
||F
|| [(ve (w) [ .zz) ()(((1] (0) ,z)) > 01
|F r)
(mn
)
Σ P (0.0.
E,,
|1 (1ot (w) uyraz))
||F
1.t'
|| [(vo] (0)
(w my is
||F
> r, w
(0)
Bu
For every u E 10, let X, be the d x k matrix, defined
X, [ kim
Then we can write that
(wm u, ))r
18 E,,
P
118
(m1) (yي (m) my 1.3'
11 > w] ( E By
X
)=x,
[ (0) 11) 1/2) E
(Ir(X* (w1 (0) y w X
X T u0 X,)) > 12, "( (0) E Bu)
(0)
X (w (0) S (0)
X
(0) X [m) (0) > r, 1 E Bu
E s w
w1 X, X (w) 10)
Notice that the maximum eigenvalue of the matrix X,,X is smaller than or equal to k, since the norm of each column of X, is smaller than equal to one. Then wi(" Xx (xxx((o0 (T > We can therefore write that
implies that [ || (r/k)1/2.
XT [w]
(0) (0)) W (0) X
P
E. 11
(0)
x, X (1) r, w (0) B.
X,, XI (mo ('lyr
w1
=P
X (wi (0) u
(T (0)
X
E
w
(0)
X (w) (0))"" r, 110 (r/k)2/2 w (0) E Bu
(0) X,, XT (0)
w
19

---

Published in Transactions on Machine Learning Research (11/2024)
Since By cone and the spectral measure of wi given by + by the properties of the multivariate stable distribution, can write that
(0)
X (w) (0) )z w X2 a
P
E, w
(0) X x (m) (10 > |u > (r/k))1/2 wi E Bu
X2 X7 (w) (w) 1T
Ca/2r 0//2
Σ
2 2022 ② j=1
+00. The proof that (a) is positive semi-definite can be done by following the same line of reasoning as in the proof of Lemma A.1
Lemma A.3. As m + too, the probability distribution of (H Hm))) converges weakly to the law of independent slable random malrices, with spectral measures Ií and 1 as 14 and 75 respeclively.
Proof. Since H and H converge marginally to a/2-stable random matrices, by the properties of the multivariate stable distributions it is sufficient show that they converge stochastically independent random matrices. By Theorem B.T
know that
nPP [[ .ry)) > 0)(((W; xz.) > 0)1,>| > n²/a
[|[,, wi Mar w; (0) M(w, xzح) > ((w; :0 . > 0)هi || > 2/2a"
and
nP (1 . .z)) > 0)[((1( .zy) ),))) > 0) 11 > nz/aa
converge finite limits, oo. Hence, again by Theorem B.1 it is sufficient to show that
lim nP (1 2z) > () .xy) > 0) > n2aa
10000
10) [|[ w (2y, (0) (( .z,)) > () =イ)
> 0) > n²/a 0,
which ensures that the Lévy measure of the limit infinitely divisible distribution of (H H is the sum of measure Vn concentrated on the space spanned by the first. k² coordinates and measure u2 the space
20

---

Published in Transactions on Machine Learning Research (11/2024)
spanned by the last k² coordinates. We can write that
nP
(g) (0) .zy > 0)I(( (0) . > 0)1ssrl1e > n2/aa
|1[e ui (0 Kay, w (0) ( > 0)/((W;)) .zy) > 0)1 > n2/a
Σ
P( E B.)
(0)
=
uE(U,1p
IP
(Ilu
> 12/0 ,,2,,,1[(,;, w 00) (xy. ii 10 yu 22aa w! Bu
=n Σ
p( ( E B,)P [[[(2] wi May, u u 2//a :0 E Bu
(E(0,1)
P (1),
> n2/0
Σ aP [I[(z], " (z u n2/a w E Bu
=E{0,17
P ((1lm((o, n²/a 0,
-
j
Now, in the position of proving Theorem 3.1
By Lemma A.1 Lemma A.1
Lemma A.3 and the
properties of stable distributions, (W),,, X) converges in distribution to a positive semi-definite random matrix, with (a/2)-stable distribution, and spectral measure I + r5. This completes the proof of Theorem
A.3 Proof of Proposition 3.1
To simplify the notation, we set in this section: w = w(0), ( == (0)) W W(O). A = A(l) (W(O), X) and H: == ( (W(0), X), with (W, and H H (W.X) defined in 12 and 13
the sum of two positive semi-definite random matrices, H and 1 The following results show that for every > 0, there exist A > O and da such that, for m sufficiently large, with probability at least 1 8
From 11 Hm (W),),
Amin ( Ai-
with the largo-width behaviour of H(() being characterizod in Lemma A.1 and Lemma A.2 through an (a/2))Stable limiting random matrix H2 (a) with spectral measure r of the form [14] and 115 To prove
that the minumum eigenvales of i7(!)) and (2) are bounded away from zero, we first need to inspect the characteristics of the distributions of (a) and (a)... This is the content Lemma A.4 and Lemma
(2) are given in Lemma A.5 and A.6 Then, the results concerning the minumum eigenvalues of (1) and (2) Leunaa A.7
Lemma A.4. Under the assumptions of Theorem 3.2 the distribution of the random matriz (i) is absolutely continuous in the subspace of the symmetric positive semi-definite matrices with zero entries in the positions (j.j') such that (x,,0z) = 0, with j.j E k), with the topology of Frobenius norm.
Proof. From Nolan 2010 it is sufficient to show that
inf / ((ss + #0,
21

---

Published in Transactions on Machine Learning Research (11/2024)
sk--1 is the unit sphere in the space of the &s x & symmetric matrices where rí is the spectral measure (14 S0 such that S. 0 if Ez) 0, with the Frobenius metric. Now, since
i (1s, u) (dda)
= Ca,BB
[ 3.s
(0) ,.y) 0))0/2 $53 (y), (0) xx,)) )2))= > 0)1((u;
is a continuous function of that takes value in compact set, then the minimum attained. Thus is sufficient to show that for every E sp -h²-1
E
(2 xzn) > ()d((w/),))))) .xy) > 0)ja/2 # 0.
j.j
For every j and every u, {0, 1), let A be the event ((w, 2z) 0) if uj and its complement if " =. Then
E
Σ
ر (2,, (2 rz)) > () .zr) > 03|a/2
Σ P(A" n... ... O A
Σ 1.
Ujug (
Since Ti....,22 Tk are linearly independent, then for every ui...., Uk: P(A n A..., > 0. To prove it, assume, without loss of generality, that u, for every i. Since F1,... Ih are lincarly independent, then we can complete the matrix X = [2] ... æx] by adding k- columns in such way that the completed matrix X is non-singular. For every d-dimensional vector such that 1 0, ...... Uk 0 there exists vector u such that (X²))-t Thus,
{u RR (u, 21) > O,....(u,, (u, > 0) = ((8T)-'0 0:01 > 0......4
uh 112000 >0}
is open non-empty set. Since (0) has independent and identically distributed components, with stable distribution, then
P w {(8))11 uu u1 O, uK >m)>00) >
(0)
This concludes the proof that PPA n ...... Alt > O for every (u1...., ue) {0, {0,1)*}. It follows that f' (s, (ddu) is zero if and only if, for every (ui...., un) (0, 1}*, it holds
Σ 3J
,,, o.
The only solution of the above system of equations in the space of symmetric matrices such that 835 0
if (Iy, = 0 is 0, which is not consistent with || = 1.
We observe that the space of the symmetric positive semi-definite matrices with zeros in the entries (j.j') such that (z,,4y)> = 0 contains all the matrices with non-zero diagonal element since (z,, * 0 for every index j.
Lemma A.5. Under the assumptions of Theorem 3.2 for every > O there enists X > 0 such that. with probability at least - 5
Amin (T)())) X1-
22

---

Published in Transactions on Machine Learning Research (11/2024)
Proof. Since the distribution of H[)a) is absolutely continuous in the space of symmetric positive semi- definite matrices with wero entries in the positions j.j' such that (I, = 0, and since this space contains all the symmetric positive semi-definite matrices with non-zero diagonal entries, then we can write that (a)) = -) = 0. Morcover, since (a) is positive semi-definite, then P(Amin (H) > 0) = 1. Thus, for every å > O, the exists \ such that P(Ami((M > >
Lemma A.6. Under the assumptions of Theorem g. the distribution of the random matrie Hī(aa is absolutely continuous the subspace of the symmettri positive semi-defonite matrices, with the topology of Frobenius norm.
Proof. From Nolan 2010), is sufficient to show that
inf / us)1/ + 0, <EB42
where 12 is the spectral measure (15
Sh is the unit sphere in the space of the k x symmetric positive semi-definite matrices, with the Frobenius norm. For every u E 13*, let B, = (u E Rd my) uj = 1, x1) I if u = 0}. Moreover, for every = I....,K, k; let e, be a d-dimensional random vector satisfying Egj for j i and eij- for j Hi. Finally, let Ca/2 be the constant defined in (4. Then
/ ((s, = Caf2
Σ 33 3.3
E
E
2
(0,0,1)
Since ΣnS' $3 EnEu ΣE Eu is continuous as function of and takes values in a compact set, then the minimum is altained. Thus it is sufficient to show that for every E ShP-1,
Σ
Σ
E
T #
We ( fide.. -, JBB 35'
Since ||>|| = 1, then is not the null matrix. Hence there exist >> 0, a vector a with ||a|| = 1 and positive semi-definite, symmetric matrix s' such that
caa. + 8'.
Since By n Bu 0, when F u', then, for every 1.... d and j 1,...,
...,. k there exists one and only one {0, 1 such that uj and tei, -ei) n Bu z 0. Then we write that
Σ
Σ
Σ
2,
"E (0,1))) (ir(es - Buydd id
Σ
L
Σ
)
xE(0.1)
(c(ex_))nB_zu))
) 2
Σ
Uj
(refer -e,) Buz0)
)
which is strictly positive, since the x are linearly independent, and ||a]| = 1. This concludes the proof. r
Lemma A.7. Under the assumptions of Theorem 9.2 for every 8 > O there exists d2 0 such that with probability at least 1 - 5
Amin (a)) X2-
23

---

Published in Transactions on Machine Learning Research (11/2024)
Proof. Since the distribution of Hg)a) is absolutely continuous in the space of symmetric positive semi- definite matrices then we can write that P(det(F()()) 0) = O. Moreover, since H2 (a) is positive semi- definite, then P( (l)) > 0) = 1. Thus, for every ô > 0, the exists X2 > 0 such that P P(Amin (M5 > X2) 1-
]
Now, we are the position of proving Proposition 3.1 Let 8 O be a fixed number. By Lemmas A.5 and A.7 there exist d > 0 and X2 > such that, for i = 1,2, P(Amin (a)) > A²) 2 6/2. Since the minimum eigenvalue map is continuous with respect to Frobenius norm then, by Portmanteau theorem, for 1,2,
lim inf P( ( (W(O), A.) 2 P(Amin (a)) > A.) 2 1- 8/22.
m
Let Ao = A + X2. Since the minimum eigenvalue of a sum of symmetric, positive semi-definie matrices is greater than or equal to the sum of the eigenvalues of the two matrices (see Horn and Johnson 19855 Theorem 4.3.1), then we can write that
lim inf PPmmi (m (W) X)) > 2o)
lim inf IP (Amin (H (0) X)) + Amin (W()), > 20)
m
lim inf ((0==1,2(Amins ( ( (0). x)) X;))
lim sup
Σ P (Amiiu P(Amin ( s x1)
8,
thus completing the proof of Proposition 3.1
A.4 Proof of Proposition 3.2
Before proving Proposition 3.2 we give some preliminary results.
Lemma A.8. Let y E (0, 1) and >>> 0 be fizee numbers. For every ó > 0 the following property holds true, for sufficienlly large, with probability at least 1-
@5m Ow
0fm (W (W))), < cm -21/a
(logm) 2/a
(W,zjjo)
Ow
for every W such that I|W W(O)]l== < (logm)2/a and every NN's inpul xjj with j I,...,k.
Proof. For fixed W(O), let W be such that I|W -W--wi W(O)l S 2//a Then holds (0 (0))) < ||W (()) < (log m) 4/a, Accordingly, we can write the following
0fm (W, (W a) dw
jfm
(log (/ 2/a
(W()), -y: a)) (0)).
Ow
(w)) xz | (wm x > 0) - (1) (0), > 0))²
< m2/a iml
m
(ww xz) (n) (0) (0), I((w!)) (0) ,2,) > 0)
m2
2
(w: (0) (0), x3) [((w,))) (0) 2.) 0) 1((w,) 1 (0),z,) > =0)²
22aa
24

---

Published in Transactions on Machine Learning Research (11/2024)
We will bound the two terms of the sum separately. First, we define ri 1(w) (0) -w; (0), Then, we can write that
(0), (1حس for ,,..., m.
(0)
1,,,11 1 |2w - w'0) < (log mlgmm)//o (0)1I
w;
Since 7 < 1,
2
Σ
(w: 10:::9 xz) - (w0; (0). I((u; (0) . > 0)
(w,"
(0)
2
1=1
< 2m 2/a (logm) 4/a < m 21/0
for sufficiently large. In order to bound the second term, observe that the following set
{w ()())))))))))))))))) Eu (0) s. (0) wi (0), ra, I((w!)) .g)) 0) o T((m0 (0), > 0))
(0) is included in the set {w: (0) Iw: (0) (0), -لحر r.). Therefore, we write that
22) ,2)) (0) (0), 2,)2 (I((u])) (0)
0) (0), 0))
sup
sup
m2/o
Σ. Slog m (o m (0)
(r)
i=1
w 0 (0), 2, ( (0), x,) < rd)
< sup
sup
m//a
Σ, mlogo (0)
(9) [0)
i=l
< sup
sup (0) Σ, Alogg ww! -w (0)Ssr
2/aa
i=1
S m2/a (logm) 1/a
22//a
7
for m sufficiently large.
Lemma A.9. For every 5 > 0 there crist À O such that the following two properties hold true, for sufficiently large, with probabilily at least 1 - 8:
i)
( ( X) A ((0),x)|| < 2m-7/a;
(1)
A Amsia (W,
2
for every W such that || - W(0)]lF < (logm)225 m 2//a
Proof. By Lemma A.7 for every > 0 there exists À such that
Xmin (H)
25

---

Published in Transactions on Machine Learning Research (11/2024)
with probability at least 1 1 8822. For every vector W, we can write that
( (W, j
H (W(0),
pm (W, (W,z;;a) a).
0fm
Sf (W(0), 24; a),
ofm
-log m)3/a
(W. (W.zj:a))
a)),)
(W (W)))
dw
dw
dw
Ow
@fm
0f
0f
- (log 2//a
(W, WW,a;a)
(W,z3;0)
(W (W(O),
Sw
Sw
F
05m (O)).
05m (W, (W,z;a)=
alm (W (0), x2:aa
+ (log (logm)2/a 2//
dw
Ow
dw
F
(1
@m (W (W(O), 20),
mm (W()).
0ī (W, (W.
S (log (logm)2/a 2//
+
dw
Ow
dw
F
05m dw
ofm (W(0), (0),
(W, W,zzjaa
Ov
0fm 20))
0f dw
0fm dw
+ (log ())gm)2// 2/a
(W, a))
(W (W))),;;0))
F
For every i == 1, k,...,,, k,
jfm (W(0), x.;
(logm) 2/a
(w (0)
xz) ([) ( (0). > 0)
dw
22/a
F
m
(0),
m2/a
i=1
which converges in distribution, as m + 00. Thus there exist M > and my such that for every m 2 mo and every i I,...,k, k,
O5 (W(0), 25; > >M)
(logm) 1/o
Ow
8K22
F
By Lemma A.8 for sufficiently large, with probability at least 1 5/(222
(1 [ (W (W()), (0), Fw
0fm (0),
0fm
)) < 2M
(logm) 1/a
+
(W, 2;;a)
Ow
Ow
F)
whenever ||W - W < (log m) m/a/o. Lemma A.8 also implics that, for every & E (0, and = 1..... k.
with probability at least 8/(22 0i (W, dw
ajm (W(0),
(log (logm) m) 1/a
7/a
dw
4M
whenever IW - W()) < (logm)4/a provided m is sufficiently large, Thus, with probability at least 1 - 6, if m is sufficiently large
A
A m (W, # (W(0), < $2 m -//0
max
whenever ||W - = (logm)22/ 2/a Thus
( (W, X) i (2) (W))),
X 2
1 (W, X) # (W()), < Am 1//
whenever ||W - W( < (logm)2/e 2/a provided m is sufficiently large. The last inequality and Lemma A.6 imply that, with probability at least 1 - 6, if m is sufficiently large, then
i (W, X) 2/2.
26

---

Published in Transactions on Machine Learning Research (11/2024)
for every W such that NW--(OF (0 (logm)2/o. Since Hm (W, is the sum of two positive semi-definite
matrices in X) and i (W,X) then
| (W, x)ll 2 | (I, > X/2,
for every W such that || - [O)) < (log m)2/a, if m is sufficiently large.
Lemma A.10. For every ó > O the following property holds true, for m sufficiently large, with probabillity at least 8: there erists M > O such that
0m Owi()) (W, 2z;))
0fm (W (0) < <M,
mlogm)// 1/a
Owil))
for every j .,...,k, and for every W such that |W W(0))l S (log m)2/a
Proof. Let us define r; = (uxe w. 10) (0), for = 1....,m. m. Now, since || = 1 by assumption, for j l....,k. k. then we can write
Σ s || 3 |100 w (0)) = VIW w()) s (log m/
It holds
Ofm
afm
(log m) 2/a m)2/0
Ow (W, ay;za
Ow (W(0), zy: =y:aa) (W())
1
0) w. (0)( (0). (1) o)) .2z) (w (0)
m.
2/a
2
(uy w; (;)) ((w
(0) xxح)
> 0)
2//a
1=1
2
m
W/ (0) (0) ..)) 0) I((w (((w;) (0) 0)).
22aa
We will bound the two terms separately. First,
2
(w-- (x,(0) (0) .2z)
.2)) > 0)
m
2/a
m
[2 w. (0))
m 2/a
2
|w w(0)ll?
1)/a
2
m) (logm)
(log
/4/00 < _ fm -27/a
m2/a
if m is sufficiently large. To bound the second term, we write that
2 m
wc (O) (0) (xz)) > 0) ((tu()) > 0)
m/aa
2
(0)
m²/a
which converges in distribution to stable random variable, with probability at least 1- 8/4,
Hence there exists M1 such that,
2 m2
23) > 0) < M?
Σ [wv - (w
22aa
21-2
27

---

Published in Transactions on Machine Learning Research (11/2024)
and
M2
(0) (z))) > 0) - (0), > 0)1 < (1x
m
21.2
for m sufficiently large, which entail
0f ()) Ow
ff (W(0)(w), Ow(()
M1
(log (logm)'/ 1/a m)
<
k
F
On the other hand, there exist N3 E F and M2 with P(Ng) > - -6/4 such that, for every w E N3 and for m sufficiently large,
(W()))) X;a) Y|IF < M2,
and
a
ow fm (W() -; < M2(logm))
max ISEC
The above inequalities follow from the convergence in distribution of (0)),, a;:a) and of
θ b ow (m(()))
(logm)2/4 2/a
(a)l() (-......,k)
88 m + 00.
Lemma A.11. Let y E (0,1) and > O be fixed numbers. For every 8 > 0 the following property holds true, for m sufficiently large, with probability at least 1- 8:
[[((t) W(O) < (logm)2/a.
if
0im (0()) Dw
2/ax (log m) m)2/a
Ow 25)))
cm 22/a
for every NN's input 2j, with j
1....,k, k, and for every tt.
Proof. By Lemmas A.8 and A.9 there exists N1 E F with probability at least 1 5/2 such that, for every
w E N1.
0f
ofm Dw
(log m) 2/a
(W(0)();
< cn 22/0
Ow
F
for arbitrarily fixed c > and y E (0, 1/2), and
A 2
Amiu (H,(W,,WW.... ())))
for some A > O, for every W such that (W) S (logm)22a and every j j= 1,...,k, provided m is sufficiently large. Moreover, by Lemma A.10 there exist, for m sufficiently large, M1 > 0 and N₂ with PPN2) -- 8, such that
05m Ore (W,
aim Ow (W(O)(w), (3;)
Mi k
(log 1/a
<
for every j I,...,k, and for every W such that I|W W(O) S (logm)2/a We will prove, by contradiction, that for every w N1 n N₂ n N3. (t W( (log (logm)2/a for every In the following we will write W(s) in the place of (s))(w) and always assume that w belongs to N₁ N,2 0 Ng. Suppose that there exists t such that [t](t) W(O)] 2 (log m/2/0 and let
argmin (t: [W(() W())] > (log (o)m)//a3.
28

---

Published in Transactions on Machine Learning Research (11/2024)
Since [W(s) W(0))l & (logm)2/ 2/ for every to, then, for every to.
Amin (M 2'
05 Ow
0f (W (0))), (O))), 23;00
(W(6),
< <an <an (log 1/a 1//
dw
0mm
0f (W())w)), Ow(
Mi
Owl (W (s), (z;
<
#
(logm) (0ogm)-11
(j = I=....,,h)
F
11 (W())(W)) x;o) - 1r M2,
aw fm (W) 2i;a)
< Mg(logm)-1/a
max 1sik
Let us now consider the gradient descent dynamic, with continuous learning rate 7 (logm)33a;
(s)
-log ml 2/ V
1
Σ
- v:)²
w
ds
2
-v) 0fm
-log m) m)3m) 2/a Σ
az)
ow (W(s),
This expression allows to write
(ta) W(O)l
$ 1.1% to d W(s)ds)
ds
s 2// | to
Jf (;;)s
I5m
(m)) - -v:)
Oc
0fm
1/' (W)), - Yllds.
< 1logm)2/a 2/a
USo5to
max
Σ
aw
(W (s))
To bound the term (W)) Y|l we will exploit the dynamics of the NN output
dfm (s),
aw (W()), X; dsF(s)
ds
ds
x; = (logm)2 2/a (Jm (W X:)) - - (W),, x)
that gives
de (l x; 21 = -2 2 -V) A.. (V) X;a) x;a))x; 7)-
d
Since (Hin X/2 for every - to, then
ds (m()),, X:a) v1 s (W),,, X;a) -Y12
which implies that
d
ds (exp(As)] (W(s), 21 0o.
It. follows that exp()>))) (W),,, f13 is a decreasing function of 8, and therefore
(s), Y|I2 exp(=N/2)JJ X;a) Y|
29

---

Published in Transactions on Machine Learning Research (11/2024)
for every S to. Substituting in the integral, we write that
(Wo) F
k
/
$ (log m)) 2/o m
max 0Sesto
OW (W (8), 215 (1
exp(-As/2)es
[15m((())) X;a) YIl
F
2(1og m)logm)) m)2/a
Of aw
0fm
ofm aW
X
max USoS
(W (0),
+
OW
(W(s),
(W(0), (0)) -y a0
(W (0),
F
(W(),, X; Yll
2(1og m)2//a
(1)
jfm ow
8fm (W)))
ofm (W()).2;00)
Σ
(W(0), (00))
A
max OSasto
+
Ewil
Ow(0)
10fm Ow
0fm
(W(0), zzi ad) (1)))
(W(s),
-Yl
dw
2(1og m)!/a
(M2 + M1 + kcm kcm-n/a M2.
A
which, for m large, contradicts ([W)))) W(0) 2 1lggm)2/a
Now, we are in the position of proving Proposition and the properties mentioned in Lemma A.S Lemma
3.2
Let E N and N E E be such that P(N) 1 A.9 Lemma A. 10 and Lemma A.11 hold true for every it is sufficient to show that
w E N. Therefore, by means of Lemma
A.8 and of Lemma A.9
[[((t) (w))
(logm)2/a
for every 1 >> and E N. By contradiction, suppose that there exists, for some E N, lo(w) finite with
to(w) = inf [[(()) - (())]la(w)) 2 (log m)a/a)
t20
Since (w))(w) is continuous function of t, then [[((to(w) W(O) (w) (log (logm) 2/0 Then, by Lemma A.8
@m (W (s), aw aw
0fm
2 (W(0), a) (w) < cm 22/0
2/x (log m/2
for every S to and every j. Therefore, by Lemma A.11 it holds true that || (to(w) ((0))le(w) < (logm) (/g/) which contradicts the definition of to. This completes the proof of Proposition 3.2
B
The distribution of random vector & is said to be infinitely divisible if, for every n, there exist some i.i.d.
random vectors Ext,..., Son such that Σ Enk d g. A k-dimensional random vector & is infinitely divisible if and only if its characteristic function admits the representation elP()), where
JuT au f (cunt= - 11 - 1--iu" x1<[<=||| s 1)) v(dx)
(w)) = iu 2
(16)
where v is a measure on R {0} satislying [[ A 1)v(dx) < ac, a is a k x k positive semi-definite, symmetric matrix and b is a vector. The measure called the Lévy measure of & and (a, b, v) are called i.d.(a,b,v). Other kinds of
the charactoristics of the infinitely divisible distribution. We will write truncation can be used for the term Tu' This affects only the vector of centering constants b. An i.i.d. array of random vectors is collection of random vectors {Enjij I ma,n 1} such that, for every n.
30

---

Published in Transactions on Machine Learning Research (11/2024)
Sni..... SSm,, are ii.d. The class of infinitely divisible distributions coincides with the class of limits of sums of Li.d. arrays (Kallenberg) 2002 Theorem 13.12).
To state a general criterion of convergence, we first introduce some notations. Let & ~ i.d.(a, (a, b, v) Define, for each h > 0,
(h)) =a / /
xx xFFv(dd)
v(dx):
60h] = b - /
xv(d)),
where Ja<l|el|sı = Jieleliss if h > 1. Denote by - vague convergence, that is convergence of measures
with respect to the topology induced by bounded, measurable functions with compact support. Moreover, let R& be the one- one-point compactilication of RR The following criterion for convergence holds (Kallenberg, 2002 Corollary 13.16).
Theorem B.1. Consider in Re on i.i.d. array (Ens))=l....,mnnt and let & be i.d. i.d. b, v). Let h > 0 be such that |||k=|| = h) 0. Then Ej Eng 4 & if and only if the following conditions hold:
(0) m.P (E.1 E.) - v()) on R$ (0)
(6)) m., /(1lmn1) < h)) (h)
(m8) m. ((.. B(Sn < h)) + ((())
Inside the class of infinitely divisible distribution, we can distinguish the subclass of stable distributions. A k-dimensional random vector g has stable distribution if, for every independent random vectors & and &2
with d & á & and every a, e R, there exists E and d R such that at + 6ε of + d. This is equivalent to the condition: for every 2 1,
& + + = n/ + dn
(17)
where a E (0, 2). E,,..., are i.i.d. copies of and dn vector. The random vector said be strictly stable if [17) holds with d, 0. A stable vector & is strictly stable if and only if all its components are strictly stable. The coefficient a is called the index of stability of & and the law of & is called o-stable. A stable vector & is symmetric stable if IF(E A) = P A) for every Borel set A. A symmetric stable vector is strictly stable. The class of stable distributions coincides with the class of limit laws of sequences - b,)/an), where (X,)) are iid. random variables.
X
A stable distribution infinitely divisible. Thus its characteristic function admits the Lévy representation (16). If a then the Lévy measure is the null measure and, therefore, the stable distribution coincides with the multivariate normal distribution with covariance matrix and mean vector b. If a 2, then 0 (the zero matrix) and the a-stability implics that there exists a mcasure o on the unit sphere gk-1 such that v(dz)= = (a+1) dro((s)) where = =||=|| and = 2/||=|1. Substituting in (16 we obtain
(u) = iu b+ / 1 ( - 1- iru sl S 11) fIta (ds)
For a 1, the centering iru sI(r 51) 1) is not needed, since the function (of r) is integrable, and we can write
1 ((u) = iu 1 Jo ( (dru"' 114a dra (ds),
for some vector B. After evaluating the inner integrals as in Feller 1968 Example XVII.3), we obtain
(u)) = in lu" T a:) (cos(ITa/2) sign(u" ( (ogn(u")ssin(Tea)) s)sin(Ta/2)) (ds))
31

---

Published in Transactions on Machine Learning Research (11/2024)
iu |u" s1"h (1 sign(u" s) tan(Ta/2)) r(1 a) cos(Ta//)o((ds)
For 1, using the centering iru"s, s, we can write
((u) = - i b" / 1.000 (ctru"e iru
y dra dra(ds) (ds);
for some 8" After evaluating the inner integrals as in Feller 1968
Example XVII.3,, we obtain
[ a)
(1))= bu"
s|
(cos(TTa 22 isign(u' s) sin /2) odss)
58
a
=iu / 1u= 8xx iignn(u" tan(To/2)) [(2 - a) cos(maa/2)
3
Since, for < 1, 1(2 a a) (1 aa) for a # 1,
encompass the above results in one equation, and write,
T(2 - a))
((u)= iu" lu" s| isign(" 8) tan(
1 - cos(za/2)oo(ds),
for some 8" Finally, for a = 1, using the centering sin ru 8, we can write
(u) iu 6" + $ (eiru' -1 -1 sin
-i ru ruf's s) ddo(
p2 for some 8" Evaluating the inner integral as in Feller (1968 Example XVII.3,, we obtain
π (u) in i 6" - . |uTs] s|
(m + isign(u" s) log luu s)) odss)
iu 6" 6"
T (dss). |ui i- 8) log lu s)
2
Considering the spectral representation el(()) of the multivariate stable characteristic function - Ss Iu= s/0x (1 sign(u" s) tan(Ta/2)) l'(ds) iu" (0)
x
(w) we can establish the following relationship between the Lévy mcasure and the spectral mcasure r:
Ss IuT s| (1 + & sign s) log uT r(ds) iu (0)
1,
1 v(dx) = Ca pa+t r(ds),
where = ||x||, = =>||<>|| and
1
a
I(2 - cos( (COS))T//2)
Ca
2/T
a=1
A Stable random vector & is strictly stable if and only if
{
n (0)
a a
Sg (ds) for every
a =
(see e.g. Samoradnilsky and Taqqu 1994 Theorem 2.4.1)). By Theorem
B.1 the spectral measure of
symmetric stable random vector & satisfies
lim nP
[|E]] > n 1/ x.
E TEE
E =
(18)
n n-000
for every Borel set A of S such that T(@A)= = Morcover, the distribution of a random vector belongs the domain of attraction of the St, I') distribution, with a (0,2) and I' simmetric finite measure on Theorem 4.3)).
gk-1, if and only if (18) holds (see e.g. Davydov al. 12008
32

---

Published in Transactions on Machine Learning Research (11/2024)
References
CLINE, D.B.H. (1986). Convolution tails, product tails and domains of attraction. Probability Theory and Related Fields 72, 525-557.
DAVYDOV, Y., MOLCHANOV, I. AND ZUYEV, S. (2008). Strictly stable distributions on convex cones. Electronic Journal of Probability 13, 259-321.
FELLER, W. (1968). An introduction to probability theory and its applications. Wiley.
HORN, R.A. AND JOUNSON, C.R. (1985). Matrix Analysis. Cambridge University Press.
KALLENBERG, O. (2002). Foundations of modern probability. Springer.
NOLAN, J.P. (2010). Metrics for multivariate stable distributions. Banach Center Publications 90, 83-102.
33


Reviews:
[{"review_id": "cr9IrfjoM3", "reviewer": "Reviewer_Szs9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n1. This paper provides a solid theoretical framework. \n2. A detailed description of related works, especially for NTK.\n\nWeakness:\n\n1. It is unclear to me what the new insights from this paper are compared with existing NTK frameworks.\n\n2. The assumptions still follow those from NTK, which are neither practical nor novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the gradient dynamics of $\\alpha$-stable ReLU neural networks when the neuron neural width reaches infinity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5BFcTvwGc1", "reviewer": "Reviewer_yfNq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper extends the characterization of the large-width distribution of $\\alpha-$Stable $\\phi-$NNs to the ReLU activation function and generalizes previous results to the $\\alpha-$Stable setting.\n\n2. The authors analyze the dynamics of training such networks using gradient descent. They first show in the infinite width setting the kernel converges to the so-called $\\alpha-$Stable Neural Tangent Kernel (NTK) a positive definite random kernel.\n\n3. The paper provides convergence guarantees for the gradient descent algorithm, showing that it achieves zero training loss at a linear rate, with high probability, even in the presence of randomness in the $\\alpha-$Stable NTK.\n\n\n### Weaknesses\n1. It would make a stronger case with some simulations validating the theoretical results. \n\n2. The paper focuses only on the math and the technical aspects of the proofs (some of which could be deferred to the appendix), without providing any walkthroughs, intuitions, and discussions, especially in sections 3 and 4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes large-width asymptotics and training dynamics of $\\alpha$-stable ReLU neural networks (NNs). The main contributions are as follows,\n\ni)  The prior results showed Gaussian NNs, and their generalization $\\alpha$-stable NNs converge weakly to the Gaussian and $\\alpha$-stable process respectively. On this front, the contribution of the paper is an extension of the results on $\\alpha$-stable NNs to $\\alpha$-stable ReLU NNs. The results suggest the choice of activation function affects the scaling of the NN to achieve the infinitely wide $\\alpha$-stable process. Authors show that ReLU activation scaling by $(m \\log m)^{-1/\\alpha}$ is required,  in contrast to prior works on other activation functions where $m^{-1/\\alpha}$ scaling was required.\n\nii)  Authors characterize the training dynamics of $\\alpha$-stable ReLU NNs in terms of a random kernel they refer to as the $\\alpha$-stable NTK and show when such a network with sufficiently large width is training using gradient descent then zero training error is achieved at a linear rate. These results generalize prior results on understanding the gradient descent’s training dynamics on Gaussian NNs (Jacot et al. 2018, Arora et al. 2019, Du et al. 2019, Lee et al. 2019).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please walk through the reader in sections 3 and 4. In 3, can you explain in words why the factor of  $(m \\log m)^{-1/\\alpha}$ is required? The previous results have shown the convergence of $\\alpha$-stable NNs with other activations, what makes results with ReLU interesting and what is the main challenge that is introduced due to ReLU and how do the authors tackle it? Please try to answer these questions in section 3 without going into too much notation and algebra. I’d like to see similar changes in section 4. Please start the section with an overview of what you want to show and a rough overview of how you plan to prove those results.  \n\n2. It might be helpful to have informal theorem statements for the main results in the contributions section.\n\n3. Please include simulations on the $\\alpha-$Stable ReLU NNs with some choices of $alpha$ (distributions), i) to show the convergence to $\\alpha-$Stable process with infinite width ii) to validate the claims made on the training dynamics of these NNs using gradient descent.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "chukcjYWTm", "reviewer": "Reviewer_KFEa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The main result is a clear hole in the literature, and is a \"straightforward\" extension of several important results to this new setting. (I use straightforward not in the sense that the proofs are straightforward, but in the sense it is clear how these results could be extending previous results).\n\n- The contributions are clearly laid out early, and connections to the literature are clearly made throughout the paper.\n\n## Weaknesses\n\nThe primary weakness for this paper is in the presentation. I often found it difficult to piece together exactly how the different theorems were fitting together to provide evidence for the final claim (i.e. theorem 4.4). This is not to say all the ideas were not presented, but instead the ideas were not presented clearly enough without considerable work from the reader to parse the different sections. \n\nI will go through some examples on how this could be improved.\n\n- E1: The introduction dives right into the mathematical background and setting of prior works. While I appreciate the background, you delay the motivation on why this work is important until later.\n\n- E2: The introduction has several in-line lists with the assumptions/constraints of prior work. These lists are often hard to parse and relate to the result you are considering. It might be worth splitting these lists out and using more space to go through these ideas (you have plenty of room still).\n\n- E3: The motivation behind using $\\alpha$-stable distributions is buried behind awkward dense language. It isn't until part way through the first paragraph of section 1.1 did I begin to understand why we want to use these distributions (i.e. the long-tail effects allow for potentially better representational qualities). \n\n- E4: You use the ability represent \"hidden features\" as main motivation for the $\\alpha$-stable distributions, but what this means is unclear without going through the prior literature. You should contextualize this in more straight forward language. Lee et al 2019 do a better job by providing the full quote from Neal 1995. This likely comes from me not understanding what is meant by $\\alpha$-stable NNs being \"more flexible\" than Gaussian NNs.\n\n- E5: Your contributions section is too dense and does not just highlight your contributions. Here you should precisely enumerate your contributions and link to the appropriate sections/theorems. While they don't enumerate, Arora et al, 2019 does an excellent job at clearly stating their contributions and the impact of these contributions in plain language.\n\nWhile I provide only 5 examples, the style throughout is overly dense in my opinion. The technical content is unavoidable (and shouldn't be removed), but taking inspiration from [Lee et al, 2019], [Arora et al, 2019], and [Du et al, 2019] would improve the quality of the paper and improve its readability. Some notable properties of their style:\n- Taking room for lists (not in-lining lists multiple times)\n- Giving more plain language descriptions throughout, and saving the precise mathematics for where they are needed.\n- Paragraphs with clear purpose. This paper has many large paragraphs that often meandered through several topics.\n\n\nQuestions:\n\n- Q1: What are the consequences of fluctuations of H_m^(1) during training?\n- Q2: What interesting phenomena do you mean in the last sentence of the first paragraph of your discussion? Are these phenomena your results or do you mean more generally? Some insights here would be great, because I appreciate the theoretical consequences, I'm struggling to understand how the use of $\\alpha$-stable distributions relate back to practice.\n- Q3: Can we compare the convergence rates of $\\alpha$-stable networks to their gaussian counterparts? I think both are linear, but maybe there isn't much we can say about the absolute values given how this was proved.  Maybe this is in the paper, but I missed it.\n- Q4: Why is $\\alpha=2$ excluded from your analysis? How difficult would it be to include? What results are valid for $\\alpha=2$?\n\nThis paper seems relevant and is not cited:\n- \"alpha-Stable convergence of heavy-tailed infinitely-wide neural networks\", Paul Jung et al 2021\n\nSome minor edits I saw:\n- Sentence 1 of the introduction \"namely NNs with weights *that* are Gaussian distributed\".\n- In the first equation of the introduction should it be $w_{ij} instead of just $w_i$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the properties of large width neural networks using ReLU activation functions and initial weight distribution sampled from an $\\alpha$-stable distribution. The paper shows that the (re-scaled) weights kernel defined by the initial distribution (synonymous with Du et al's neural tagent kernel) converges to another $\\alpha$-stable distribution. From this result, the paper then is able to show that such a network almost-surely converges to zero training error when optimizing a squared loss using gradient descent.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I highly suggest re-writing this paper for clarity, and thinking about how to address the examples provided above.\n\nOne critical aspect which needs fixing for this paper to be publishable is the references section. It is inconsistent in what names it uses for authors, and whether the first or last name appears first in the entry making searching through it difficult. Also some of the papers listed as arxiv publications have full publications one example is Lee et al 2022 (which has now been published at JMLR). I did not check all entries.\n\nWhile I did spot check some of the proofs, I do not have the expertise to validate their accuracy. The results seem reasonable, but I will hold judgment until I see other reviews.", "broader_impact_concerns": "I have no concerns about this paper's ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received positive reviews however it did not receive strong support from the reviewers.

Reviewers have requested certain edits; the authors should carefully go over those and address them in their final version of the paper. Note that any questions asked by reviewers should be clarified in the main text as well.
Decision: Accept (poster)