Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions
Authors: Alberto Bordino
Stefano Favaro
Sandra Fortini
Venue: Accepted by TMLR ()

Abstract: There is a growing literature on the study of large-width properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed parameters or weights, and Gaussian stochastic processes. Motivated by some empirical and theoretical studies showing the potential of replacing Gaussian distributions with Stable distributions, namely distributions with heavy tails, in this paper we investigate large-width properties of deep Stable NNs, i.e. deep NNs with Stable-distributed parameters. For sub-linear activation functions, a recent work has characterized the infinitely wide limit of a suitable rescaled deep Stable NN in terms of a Stable stochastic process, both under the assumption of a  ``joint growth" and under the assumption of a ``sequential growth" of the width over the NN's layers. Here, assuming a ``sequential growth" of the width, we extend such a characterization to a general class of activation functions, which includes sub-linear, asymptotically linear and super-linear functions. As a novelty with respect to previous works, our results rely on the use of  a generalized central limit theorem for heavy tails distributions, which allows for an interesting unified treatment of infinitely wide limits for deep Stable NNs. Our study shows that the scaling of Stable NNs and the stability of their infinitely wide limits may depend on the choice of the activation function, bringing out a critical difference with respect to the Gaussian setting.


Published in Transactions on Machine Learning Research (11/2022)
Infinitely wide limits for deep Stable neural networks: sub- linear, linear and super-linear activation functions
Alberto Bordino University of Warwick
alberto. www.:/bbbbonoowwwwui.....c
Stefano Favaro
stefano.fuuvrrooeduuu unito.
University of Torino and Collegio Carlo Alberto
Sandra Fortini
aondm.forttni@@mmbooon....
Bocconi University
Reviewed on OpenReview: https: // openreview. net/ ftr/f id= A5tIluADD
Abstract
There is growing literature the study of large- wigth properties of deep Gaussian neural networks (NNs), i.c. deep NNs with Gaussian-distibbute sian stochastic processes. Motivated by some empirical and theoretical studies showing the potential of replacing Gaussian distributions with Stable distributions, namely distributions with heavy tails, in this paper we investigate large-width properties of deep Stable NNs, i.e. deep NNs with distbe-dist parameters. For sub-linear activation functions, a recent work has characterized the infinitely wide limit of suitable rescaled deep Stable NN in terms of Stable stochastic process, both under the assumption of "joint growth' and under the assumption of "sequential growth" of the width over the NN's layers. Here, assuming a "sequential growth' of the width, we extend such a characterization to general class of activation functions, which includes sub-linear, asymptotically linear and super- linear functions. As a novelty with respect to previous works, our results rely on the use of a generalized central limit theorem for heavy tails distributions, which allows for an in- teresting unified treatment of infinitely wide limits for deep Stable NNs. Our study shows that the scaling of Stable NNs and the stability of their infinitely wide limits may depend on the choice of the avaiviion function, bringing out critical difference with respect to the Gaussian setting.
parameters weights, and Gaus-
1 Introduction
Deep (feed-forward) neural networks (NNs) play a critical role many domains of practical interest, and nowadays they are the subject of numerous studies. Of special interest is the study of prior distributions over the NN's parameters or weights, namely random initializations of NNs. In such a context, there is a growing interest on large-width properties of deep NNs with Caussian-distribuue the interplay between infinitely wide limits of such NNs and Gaussian stochastic processes. Neal (1996) characterized the infinitely wide limit of shallow Gaussian NN. In particular, let: i) x E Rd be the input of the NN; ii) T : R + IR be an activation function; iii) = {w]" w, (0) .bb21 be the collection NN's parameters such that (.)))) d wi äid N oz) and (()) d b sid N(0 of) for o of 0, with N(,, being the Gaussian distribution with mean i and variance o², Then, consider a rescaled shallow Gaussian NN defined as
parameters, with emphasis
fn(n] -1/2 b n1/2 Σ WHT (w0,) (0) x + 6,0)
(1)

---

Published in Transactions on Machine Learning Research (11/2022)
with n-1 being the scaling factor. Neal (1996) showed that, as too the NN output fa(n)[r,n=1/ converges in distribution to a Gaussian random variable (RV) with mean zero and a suitable variance. The proof follows by an application of the Central Limit Theorem (CLT), thus relying on minimal assumptions on T, as is sufficient to ensure that (x())()))) (x)) is finite, where gy gy(x) = wyr((u) z)R4 + The result of Neal (1966) has been extended to general input matrix, i.c. k > 1 imputs of dimension d, and deep Gaussian NNs, assuming both "sequential growth' (Der & Lee 2005 and a "joint growth' (de G. Matthews et al., 2018) of the width over the NN's layers. See Theorem A.1 In general, all these harge-widter asymptotic results rely on some minimal assumptions for the function T, thus allowing to cover the most popular activation functions.
Neal 1996) first discussed the problem of replacing the Gaussian distribution of the NN's parameters with a Stable distribution, namely a distribution with heavy tails (Samorodnitsky & Taqqu 1994), leaving to future research the study of infinitely wide limits of Stable NNs. In recent work, Favaro et al. 22020) 2022a) characterized the infinitely wide limit of deep Stable NNs in terms of Stable stochastic process, assuming both a "joint growth" and a "sequential growth' of the width over the NN's layers. Critical to achieve the infinitely wide Stable process is the assumption of a sub-lincar activation function т, i.c. |T(x)| I a + b|z|", with a, b > 0 and 0 0< < 1. In particular, for a shallow Stable NN, let: æ E Rd be the input of the NN; ii) R- + R be the sub-linear activation function of the NN; iii) 0 {w, w. 6,%) .b22 be the NN's parameters such that w!] _ wy iid (ow) and 6(0) L b üdd (os) for E (0, and ow > 0, with (a) being the symmetric Stable distribution with stability a and scale a. Then, consider the rescaled shallow Stable NN
(0)
fa(n)] i0li,,n-/1 11/1= = =b+ n²/a Σ
a)) + )b,
(0) x)
(2)
with n 1/a -1/a being the scaling factor. The NN [ is recovered from 2 by setting = 2. Favaro et al. (0020) showed that, as n + +o0 the NN output fa [n)(TT n 1//a] converges in distribution to a Stable RV with stability a and a suitable scale. See Theorem A.2 in the Appendix. Differently from the Gaussian setting of (de G. Matthews et al., 2018),, the result of Favaro et al. (2020) relies on the assumption of a sub-linear activation function. This is strong assumption, as does not allow to cover some popular activation functions.
The use of Stable distribution for the NN's parameters, in place of Gaussian distributions, was first motivated through empirical analyses in Neal 1996), which show that while all Gaussian weights vanish in the infinitely wide limit, some Stable weights retain a non negllggligible contribution, allowing to represent "hidden features'. See also Der & Lee (2005) and Lee et al. (2022), and references therein, for an up-to-date discussion on random initializations of NNs with classes of distributions beyond the Gaussian distribution. In particular, in the context. of heavy tails distributions, Fortuin et al. (2022) showed that wide Stable (convolutional) NNs trained with gradient descent lead to higher classification accuracy than Gaussian NNs. Still in such a context, Favaro et al. (002b)) considered a Stable NN with ReLU activation function, showing that the large-width training dynamics of the NN is characterized in terms of kernel regression with a Stable random kernel, in contrast with the well-known (deterministic) neural langent kernel in the Gaussian setting Jacot et al. [2018); Arora et al. (2019). In general, the different behaviours between the Gaussian setting and the Stable settings arise from the large-width sample path propertics of the NNs, as shown in Favaro et al.) (2022a b), which make a-Stable NNs more flexible than Gaussian NNs. See Figure [ and Figure in the Appendix.
1.1 Our contributions
In this paper, we investigate the large-width asymptoli behaviour of deep Stable NNs with a general activa- tion function. Given f F + IR and g R + R we write f(x) ~ g(z) for 2 + +oo if lim f=))/g(2) = 1, and f2) (g(()) for z + too if if there exists C> 0 and 2g such that f(z)/g(>) S C for every 2 2 ag- Analogously for 2 - -0. We omit the explicit reference to the limit of 2 when there is no ambiguity or when the relations hold both for + to0 and for + -00. Now, let 7: IR - R be continuous functions
2

---

for Published in Transactions on Machine Learning Research (11/2022)
and define:
E1 E C(R: : [7(=] = 0(=1)) =(=1)) with 0 <11;
E2 (TEC(R::) R): [r()] |z| and T strictly increasing for |2| > a, for some 7 > 0};
Fg E r(=) = 2² for = +00, 0( with , < for - -00
and strictly increasing for a, for some 7, 0}.
We characterize the infinitely wide limits of shallow Stable NNs with activation functions in E1, E2 and Es: assumiin d-dimensional input. Such characterization is then applied recursively to derive the behaviour of deep Stable NNs under the simplified setting of "sequential growth'. i.e. when the hidden layers grow wide one at the time. Our results extends the work of Favaro et al. (2020 2022a) to a general asymptotically linear function, i.c. E2 U Es choosing 7 = 1, and super-lincar functions, i.c. Eg U Es choosing 7 > 1. As a novelty with respect to previous works, our results rely on the use of generalized CLT (Cchaikin & Zolotarev 2011 Otiniano & Gonçalves) (010)), which reduces the characterization of the infinitely wide limit of a deep Stable NNs to the study of the tail behaviour of a suitable transformation of Stable random variables. This allows for unified treatment of infinitely wide limits for deep Stable NNs, providing an alternative proof of the result of Favaro et al. (2020 2022a) under the class Fj. Our results show that the scaling of a Stable NN and the stability of its infinitely wide limits depend on the choice of the activation function, thus bringing out a critical difference with respect to the Gaussian setting. While in the Gaussian setting the choice of r does not affect the scaling 1-/2 required to achieve the Gaussian process, in the Stable sctting the use of an asymptotically linear function results change of the scaling n 1/o, through an additional (log -11/ term, to achieve the Stable process. Such phenomenon was first observed in Favaro et al. (022bb) for shallow Stable NN with ReLU activation function, which is indeed asymptotically linear activation function.
1.2 Organization of the paper
Section 2 contains the main results of the paper: i) the weak convergence of a shallow Stable NN with an activation function 7 in the classes E1, E2 and E,, for an imput x = 1 and no biases; ii) the weak convergence of deep Stable NN with an activation function 7 in the classes F,, Eg and Fg, for an input E R and we discuss some natural extensions of our work, as well as some directions for future
biases. In Section 3
rescarch.
2 Main results
Let (9, P) be a generic probability space on which all the RVs are assumed to be defined. Given RV Z, we define its cumulative distribution function (CDF) as Pz(=) = P(Z s =), its survival function as Pz(z)= = 1 and its the density function with respect to the Lebesgue measure pz(2) dPz(a) using the notation (z(z) to indicate p2(2)dz. A RV Z is symmetric Z 4 Z, i.e if Z and -Z have the same distribution, that is Pz(z) P2(-2) for all as +oo, if for every point of continuity E IR of Pz it holds Pz, (z) + Pz(z) as + +o0, in which R + I we write f(z) o(g(2)) for 2 + +oo if
R. We say that Zn converges to Z in distribution,
case we write Zn 4 Z. Given f R + R and limz-++00 [x))/g(>) = O. Analogously for z + -00. As before, we omit the reference to the limit of = when there is no ambiguity or when the relation holds for both 2 + too and for 2 + 00. Recall that a mcasurable function L: (0, +00) >> (0, +00) is called slowly varying at 400 if lim L(ax)/(()) all > O.
Definition 2.1. A R-valued RV X has Stable distribution with slability a E (0,2), skeuness 3 E [-1, 1, 1), scale о 0 and shift p R, and we write X Sa B, n), if its characteristic function is px(() Exe e(t) for E RR, where
o [1| + i,3 tan sign(()) + ipst a +
(tt)
a A[[1 + i sign(t) log (14)) ipct 1
3

---

Published in Transactions on Machine Learning Research (11/2022)
Property 1.2.16), if X (, with 0 a < then [[[X]]] < +o0 for 0 < < a, and E. [|"] = +o for any r 2 a. A R-valued RV X is distributed the symmetric a-Stable distribution with scale parameter and we write X ~ (o)). if X ~ Sa(a,0,0 which implics that (x = [[e"<] = e="114" t R. This allows to prove that X ~ Sa(a), then aX ~ (lala)) see Samorodnitsky & Tagqu [1994 Property 1.2.3). Furthermore, one has the following complete characterization of the tail behaviour of the CDF and PDF of Stable RVs: for symmetric a- Stable distribution, Samorodnitsky & Taqqu (1994 Proposition 1.2.15) states that, if X Sa( with
By means of SSamorodnitsyy & Taqqu 1994
0 a<a<>
Px(x) = Px(-x) 2 c.00
where
Ca = (f x==ssin(())) sin(x)dx)
2 ((a sin ( T
I-a I(2-a) co)(x))
a
π
a=
As before, if X (a) with a 2, then px(x) px(-x) (a/2)C,o"|o/ for - +o0 holds truc.
For an activation function belonging to the classes F1, F2 and Fa, we characterize the infinitely wide limit of a deep Stable NN, assuming a d-dimensional input and a "sequential growth" of the width over the NN's layers. Critical the use of the following generalized CLT (Uchaikin & Zolotarev 2011 Otiniano &
Gonçalves 2010).
Theorem 2.1 (Generalized CLT). Let 7 be a RV such that Pz(=) = cT=PT(z) and Pz(-2) = da dL() for some c, d > 0, < < 2 and with being a slow varying function. Moreover, let (Zn)a21 be a sequence of RVs üdd as Z. If
0<p<1 p=1
On
(c d) log
E|Z]
1 <<p<2,
then, as n - n +00
[ [c+d> d c-d
Cp 'c+ d' 0
Σ (2: an) $ Sp 1=1
{[
(3)
(nI(n))
For < 1, no centering turns out to be necessary in (3.. due to the "large' normalizing constants n 1/p which smooth out the differences between the right and the left tail of Z. For > 1, the centering in 3 is the common one, namely the expectation. The case p = 1 is a special case: the expectation does not exist, so it cannot be used as a centering in 3 on the other hand, centering is necessary for convergence because the normalizing constant n does not grow sufficiently fast to smooth the differences between the right and the left tail of Z. In particular, the term inchuding log comes from the asymptotic behaviour of truncated moments.
2.1 Shallow Stable NNs: largh-width asymptotics for an input x = 1 and no biases
(0) We start by considering a shallow Stable NN, for an input = and no biases. Let w(()) [w] wu2 .... J and w = [Wi: u2,... independent sequences of RVs such that w/)) idd (au and wy iid Sa. (a1). Then, we set Z, = u (0)), where T T
IR- + R is a continuous non-decreasing function, and define the shallow
Stable NN
f(n)
Zz n-/p J=1
(4)
with p > 0. From the definition of the shallow Stable NN (4. being Z1. Zz.... iid according to certain
RV 7, it is sufficient. to study the tail behaviour of Pz(z) and Pz(-2) in order to obtain the convergence
4

---

Published in Transactions on Machine Learning Research (11/2022)
in distribution of [[])[..pl. As a general strategy, we proceed follows: i) we study the tail behaviour of X x+(Y where X ~ Sa, (aa), Y Sa, (oy), X Il Y and T E F E Eg and E Fg; ii) we make use of the generalized CLT, ie. Theorem 2.1 in order to characterize the infinitely wide limit of the shallow Stable NN
Note that to find the tail behaviour of X xV) it is sufficient find the tail behaviour of |X 7(V),, and then use the fact that IP[ > x] = (1/2)-P[L -(Y)] > 2] for every 2 0, since X- -Y)) is symmetric X is so. Then, to find the asymptotic behaviour of the survival function of [X -X)) we make use of some results in the theory of convolution tails and domain of attraction of Stable distributions. Hereafter, some basic facts. Given two CDFs F and G, the convolution F G is defined F G(t( J y)d(y) which inherits the linearity of the convolution and the commutativity of the convolution from properties of the integral operator. Recall that a function F on [o, +ool has exponential tails with rate a (F E Ca) if and only if
recall
F(t F(t-v), eay, for all real y E R. F(t)
lim
Then,
F(t) exp exp - L'' a(o)du)) where a(t) + 0, + a, as t t00.
A complimentary definition is the following: function U on [0, +o] regularly varying with exponent p (U RV,) if and only if
U(yl) ++ U(t)
lim
= ye, for all > 0.
Then,
U(t) = a(t) exp [ p(t) v
du where a(t) + a > 0,p(t) - p. as + to0,
iee the Karamata's representation of U. Clearly F E La if and only if F(Int) E RV-a. The next lemma provides the tail behaviour of the convolution of F and G, assuming that they have exponential tails with the same rates.
Lemma 2.2 (Theorem 4 of Cline
1986 Let F,G G La for some > O, f RVa and g E RV,, where f(t) e F(t) and g(t) = e G(t) and 3 > -1 and 7 > -1. Then
T(1+ ()+(++))
G()(1)
+(1+ ate af (())((1,,, as f - +00.
We make use of Lemma 2.2 to find the tail behaviour of |X (Y)) when [X] and [r(Y)] have regularly varying truncated CDFs with same rates, If [X] and [r(Y)] have regularly varying truncated CDFs with different rates, then we make use of the next lemma, which describes the tail behaviour of U W, where U and W are two independent non-negative RVs such that PPU u] is regularly varying of index -a S 0 and B[Wo]] +00.
Lemma 2.3. Suppose and W are two independent non-negative RVs such that E[W*] < too and P[W u| = oP > ul). If P(U > u] x cu with e 0, then PUWW u| E[W]] PiU > u].
Lemma 2.3 was stated in Breiman 1965 for a E [0, 1], and then extended by Cline & Samorodnitsky 1994) for all values of a, still under the hypothesis that B[Wat]] < +oo for some e > O. Lemma 2.3 provides a further extension in the case P(U > x] cr co=a, with
O, and has been proved in Denisov & Zwart (2005). Based Lemma 2.2 and Lemma 2.3 it remains to find the tail behaviour of |X and [(Y)). For the former, it is easy to show that P (tx next lemma.
P|X| t ~ Ca.o2.t aa.ttan while, for the latter, we have the
Lemma 2.4 (Tail behaviour of T(Y), T E Eg Ez). Assuming Y Sa(a), then: i) P[]T(Y)) > t -a/ if Fa; ú) P(I+(Y) > 1] a is Fa.
5

---

Published in Transactions on Machine Learning Research (11/2022)
Proof. If T2) is strictly increasing for > and T(2) 22 for + too with y 0, then (v) ~ yl/T.
Analogously at -co. We refer to Theorem 5.1 of Olver (1774) for the case 7 = 1. Now, starting with T E F2 and defining the inverse of 7 where the activation is strictly increasing, we can write for a sufficiently large t:
P(I(()) P)()))))))>>>>>)))) > 1) -((())))))((()))))) < <t) = 1 (0)) (-00) 2
1 2 2.0 1t Ca
Instead, if E F3, then there exits > 0 and yo < 0 such that [T(Y] < blyl for S yo. Then, for sufficiently large,
P(T(()) <0) P( 4, <0) 1 (ao0(bl)-
Furthermore,
1 Cao
P(I(()) > ,,Y > 0) = > (t)) -
2 (1))
~
2
hence, since 3 < <;,, it holds that PV] > t) ~ aa which concludes the proof.
Based on the previous results, it is easy derive the tail behaviour of |X- -YY)),), which is stated in the next theorem.
Theorem 2.5 (Tail behaviour of |X- =(Y))) Let | = [(V) where X and Y are independent and distributed respectively as Sa. (ax) and So,, (oy)) If t E E1 and Baz ag. then
PIz(()) = Ca. of B(|(())
For T E E2 U Es, define a = min(az, /n/") and e = 1 if 7 E Es and c, = otherwise. Then
c CC2y "
if ? > ag/as
P12)(2) =
C = log if 7 aul /
C
if 7 < ag /
Proof. We start from the proof of the first case, ie. T E ET. Here, [r()) < b for certain 8 E (0, 1) and > 0, when YY| is larger than some Yo > 0, hence there exists c > 0 such that [[] S c+ [+ ( < +00, being Sag < ay by hypothesis. The thesis then follows from Lemma 2.3 An analogous strategy can be used the case Our * ag/77 Indeed, [E[[|]//1) < +o0 if or > 0y/y and E [V) < +o0 if ag < ag/7. Hence Lemma [.3 allows conclude. different situation arises when aa 0g/7 In this case, consider the RVs log X and log (r()) and observe that, for t >> 0,
(x) = PPoo X = P[| > et) Ca, of ast Las
i.c. Plog X t] has exponential tail with index az2 and the same has P Ping ((() == P[log [T(Y) t] since a₂ 0 Furthermore, eaat Plog(x RVo and cast Plorlr(M)) E RVo, hence we apply Lemma 2.2 with 7 0, and obtain that
Pllog [X , 7(Y) > t] = P(log |X| + log ((Y) > t Pog |x| (7))))())
= agte P lox |x| (t) P log (t)
Ca, Ca a te
It sufficient to evaluate this expression in logt obtain the thesis. As for the case Es, the proof is the same except for an extra in the tail behaviour of P (t)

---

Published in Transactions on Machine Learning Research (11/2022)
Based on Theorem 2.5 the next theorem is application of the generalized CLT, i.c Theorem 2.1 that provides the infinitely wide limit of the shallow Stable NN with the activation function belonging to the classes E1. E2, Ez.
Theorem 2.6 (Shallow Stable NN, T Ei, E2, Ez). Consider f(n)[T,P) pl defined in I E Ei and Ba1 cog then
M()]T, -s San (1 (B2=s.,, 1/a1
If r E U Ea, define a = min 0n/7), eT = if E E3 and cr = 1 otherwise, and mn (7) = log n if
= 00/01 and m,) = otherwise. Then
m/(())1//(m()TT - Sz(o);
where
Car B.-s,, 01112199) 1/2 (o
(ct
if > a
0 (c,2Crg) 2//2
if 7 an/a1
01 (Znss (10 1/2
if y an/a1.
Proof. Observe that, since wy -(w is symmetric, then
Piwy ((w) (0)) t = P(w, r(u,)) < -t] P(lw, r(u, s).
Hence, the proof of this theorem follows by combining Theorem 2.5 and the generalized CLT, Le. Theorem 2.1 after observing that due to the symmetry of wy [wu)
The term (logn)
(0gg)-1/2 1/2 in the scaling in the case 7 E Eg U Eg and an = au/7, is a novelty with respect to parameters are not allected by the presence
the Gaussian setting. That is, NNs with Gaussian-disttibutted of one activation place of another as the scaling is always n 11/2, while this not true for Stable NNs shown above.
2.2 Deep Stable NNs: large-width asymptotics for input Rd and biases
The above results can be extended to deep Stable NNs, assuming "sequential growth" of the width over the NN's layers, for input
(.., Ta) Rd and biases. Differently from the "joint growth', under which the widths of the layers growth simultaneously, the "sequential growth" implies that the widths of the layers growth one at a time. Because of the assumption of "sequential growth', the study of the large
width bchaviour of deep Stable NN roduces to recursive application of Theorem 2.6 In particular, let
(w: w/ w. b3 Df. (1. 1b)221 the set of all parameters and R be the input. Define Vi 2 1
(0) (I-1) (0)
and V1 1,....
m
(0)
[ W w.
(0) (0)
3.d
(0)
R
(1)
(1)
= [w]1] W
Win
IR
[Wi, W2y w..]
I
(5)
w ". wi b 5
RR
w w 1. wy (1) -
sid
~ (1).

---

Published in Transactions on Machine Learning Research (11/2022)
Then, we define the deep Stable NN
9j (z) ow (w( (0) 2/ + orbb (0)
9j (z) (m)
obhi
1111 onbl)-)) (1-1) (n)(()) Ei= Wiit " (x)) (x)). 14 = 2..,. T.
(6)
(n(n)(T,,)) a] = = (1+1) (x)) = =rbb+ (n) Σラ= wyr w
(2) (x))
where v(n) = n----oop log(n)) (n) if E Eg Es with y = and v(n) = n otherwise, and (::)g/ denotes the Euclidean inner product in Rd. Note that the definition (6) coincides with the definition 4 provided that L = 1, o = 0, = and x = 1. For the sake of simplicity and readability of the results, we have restricted ourselves to the case where all the parameters are Stable-distributed with same index a, but this setting can be further generalized.
The next theorem provides the infinitely wide limit of the deep Stable NN 66 assuming a "sequential growth" of the width over the NN's layers. In particular, if we expand the width of the hidden layers to infinity one at the time, from 1 to L, then it is sufficient to apply Theorem 2.6 recursively through the NN's layers.
1) Theorem 2.7 (Deep Stable NN, 7 E Eg and E Fa U E3 with y S 1). Consider 9j (x) for fived j = 1,..., n and ( = 2,...., L + 1 as defined in 0 Then, as the width goes to infinity sequentially over the NN's layers, it holds
9j (x)) 4 Sa(o
where
of (o Eo [zy1" + ogg) and, for 1 22...,, L.
1/a
of (i 20. u- (1) +08) 1/a (caCao (0 - oz)
if T TE Eg or Eg U Eg, with y < 1,
if TE F Ea, with 7 =1,
with c, = 1/2 is r E Eg and olherwise.
Proof. The case == 1 deals again with shallow Stable NN but considering non-null Stable biases and a more complex type of input. The result follows from Theorem 2.6 by replacing w/( with g (x) and do with 02 = (of + oh Ej_ (=,))) thanks to the fact that gj (x) std (az)) for j == 1,..,, n. This can be casily proved using the following properties of the Stable distribution (Samorodnitsky & Taqqu 1994 Chapter 1): i) if X1 11 X2 and X; ~ (a,) then X1 + X₂ ~ Sa( + ag]=); ii) if c + and X (a))) then X-X1 ~ ((clo).. The proof for the case L > 1 is based on the fact that the g, (x)'s are independent and identically distributed as (of since they inherit these properties from the iid initialization of weights and biases: the thesis then follows applying the result for I. layer after layer and substituting of (1-1) in place of oz.
Theorem 2.7 includes the limiting behaviour of fa (T))TT, a] in the case / - + It is possible to write an explicit form of the scale parameter by recursively expanding the scale parameters of the hidden layers. See Subsection 2.3 for an example in the case the ReLU activation function. Before concluding, we point out that when using a sub-linear activation, i.c. r E1 or T E E3 with y E 10, or a asymptotically lincar with 7 1, the index a of the limiting Stable distribution does not change as the
activation, i.e. 7 E depth of node increases so that, even for very deep NN, the limiting output is distributed as a-Stable distribution. Such behaviour is not preserved for super-linear activation functions, i.e. E Eg U E's with > 1. When an < Qu/y, the convergence result of Theorem 2.6 involves a Stable RV with index equal to 00/7; and not ag- In case Og = 0y = a, this is the case when 7 > 1, which corresponds to a super- linear activation in Eg U Ez. The fact that the limiting RV takes a factor 1/r prevent us from writing a thcorem in the setting of Definition (6) because we would not be able to apply the property i) above as it describes the distribution of the sum of independent Stable RVs with different scales but same index. We are then forced to adjust the initialization of the biases and to this purpose we define a new setting. Tet
8

---

Published in Transactions on Machine Learning Research (11/2022)
(0) {w, ;., Wi ww
(1-1)
(0)
0/ 1
,08221 the set of all parameters and I&d be the input. Define Vi 2
and 1A 0, ....
m (0)
[ 1
(0) (0)
W
R
w
[ w 1220
(1)
(0)
R"
i,a
[W]. W2 wn)
R"
wag wi, 0/
E R
(7)
w wj Rd So (1)
61 S4(1)
sid S (1).
Then, we define the deep Stable NN as
9 n (x) = aw( (0) 2)gs + ogbb (0)
0j (z) anbl-11) own D jjs T(hi
(1)
(1-1), (1)1) (x)),
(x))) 4I = 2,...
(8)
(1+1) fa(n]]T, a) = =6 =1 (x) = o,b + own Ej= W
(4) (x))
The next theorem provides the counterpart of Theorem 2.7 for the deep Stable NN G It. provides the infinitely wide limit of the deep Stable NN assuming "soquential growth' of the width over the NN's layers.
Theorem 2.8 (Deep Stable NN, T E Eg Us with & > 1). Consider 93 (x) for fized j = 1...., n and 2, ...., + 1 as defined As the width goes to infinity sequentially over the NN's layers,
(x)) 4 Saft-11 (0!0)
where oe -o Eg 12,1% "/ and
06 a/t 2 xt
Cr 2 a/7' "dl-:))aat B2- (1)_s((11120))) "22
1 =
Ca/a!-1 (Capppi-
- -
with e, = 1/2 if r E Fi and C =1 otherwise.
Proof. The proof along lines similar to the proof of Theorem 2.7 Notice that the fact that ((-11) std (1)) critical to conclude the proof.
As a corollary of Theorem 2.8 the limiting distribution of fa (T))([TT a), as n > +oc, follows a (a/74)-Stabll distribution with scale parameter that can be computed recursively. That is, for a large number of layers, the stability parameter of the limiting distribution close to zero. As we have pointed out for a shallow Stable NN, this is a peculiar feature of the class T E Eg U Eg with 7 >> 1, and can be object of further analysis.
2.3 Some examples
As Theorem 2.6 is quite abstract, present concrete examples using well-known activation functions. First consider the case when 7 tanh E E1, since it is bounded. Then, the output of a shallow Stable NN is such that
f(n)[tanh, on] d San (a, (lonss 1/a1

---

Published in Transactions on Machine Learning Research (11/2022)
See also Favaro et al. 2020). As for the new classes of activations introduced here, we can start considering the super-linear activation r(2) 23 E B2 with 3, in the case of shallow NN with on = oo = and obtain that
0800 ( Ca B2..22sscc1117711 Ca/3
3/a
[()) aa/3)
- Sa/3
with the novelty here lying in the fact that the index of the limiting output is a/3 instead of a. As for asymptotically linear activations, if you take T = id, i.e. the identity function, again under the hypothesis of a shallow NN with a1 = ou = a, you obtain that
(logn)-*4 1/a (in)ldd a] +s Sa ((aca) 1/a
oo01 )
which shows the presence of an extra logarithmic factor of (logn)-1/an in the scaling for the first time. Beware that this behaviour, which is critical difference with the Gaussian case, does not show up only with asymptotically linear activations: for example, if you take a₁ 1, i.e. Cauchy distribution, do 3/2, ie. Holtsmark distribution, and r(2) = 2//2 then
(log m.) - 1] ب Si (C10001).
Finally, we consider the ReLU activation function, which is one of the most popular activation functions. The following theorems deal with shallow Stable NNs and deep Stable NNs, respectively, with a ReLU activation function.
Theorem 2.9 (Shallow Stable NN, ReLU). Consider Z X ReLU(Y) where X ~ Sa(ax), Y ~ So X Y. Then
Pz(z) = PP((=2) ~ a C2ogozz log 4
Furthermore, if [(L)(RLLU = n E Wj WuyReLUUuu ReL where ReLU(t) max {0, t and w ud
sid (o0) IL wj ~ Sa(a1), then
a/b (logn) - f(n)(T, + So [a 000l )
Proof. The theorem follows easily from Theorem 2.5 and Theorem 2.6 after noticing that ReLU e Eg with 3 = 0 and y 1. In addition to that, we also provide an alternative proof which can be useful in other applications. First, the distribution of Q = ReLU(Y) is TPQ & 4] = -P(max (0, Y) S 4) = PPY < 1 2 0), from which we observe that Q is neither discrete nor absolutely continuous with respect to the Lebesgue while the remaining 1 of the mass is concentrated on R,,
measure as it has a point mass of 1 at x accordingly the Stable law of on (0, oo). Hence, having in mind the shape of PQ(4) P[Q s g). we derive the approximation for the tails of the distribution of X ReLU(()) and, usual, we make use of the generalized CLT to prove the following theorem. We prove the tail behaviour of Pz(z) first. For any > 0 we can write that
px px(x)dd== = /
400
P2(2) =
P Q x
P
px px(x)dr,
since Y and Q have the same distribution on (0, +00). Now, observe that
P[XY 2| /PLOY 2 =) P
P
px px(x)
x
where the second equality holds by splitting the integral on R into the sum of the integrals on (-o0, 0) and (0, +0o) and using the fact that Y is symmetric. It follows that, for every 2 > 0, Pz(z) +P(XY 2 2]. Applying the results for id, we find that
P2(2)= = P 2 ~
a " C
0 loge.
4
10

---

Published in Transactions on Machine Learning Research (11/2022)
The proof for the asymptotic behaviour of P2(2) works in the same way after fixing < 0 and using change of variable while the convergence in distribution of (ogn)== # [[(n)[r, of follows by direct application of the generalized CLT.
Theorem 2.9 can be extended to deep Stable NN with imput a = (#1. E R and considering the biases.
Theorem 2.10 (Deep Stable NN, ReLU). Consider the deep Stable NN with ReLU activation defined follows
(x) ow (w x2 + tob) (0) 5f onby
11 + o (n log n) Ei= Wii (1-1) [a(())ReLU,aa] = gf ( (ax) =oab ow log 3 Ej= wy (
ReLUig (x( VI = 2..., L
(9)
Then, under the hypothessi of Stable initialization for weights and biases as as the width of the previous layers goes to infinity sequentially,
11) (z)) (xx) - Sa (141
0j
where dg -o E/= 11,1a + og) and, for 1 = -2..., 1,
df = (2acalo5--)) tob
1/a
Proof.
The proof is along lines similar to the proof of Theorem 2.7 with 7 = and r E Eg.
Then, as a corollary of Theorem 2.10 the limiting distribution of (n) al, as n + too, is the distribution of a-Stable RV whose scale can be computed recursively. particular, we can wrile the following statement.
Corollary 2.10.1. Under the setting of Theorem with a generic depth L.,
1 f(n)[ReLU,a) 4 Sa (((2aco0)) of E
L-1
Proof. The claim is true for L which can be proved using the standard two properties of the Stable distribution. Moreover, for NN with depth of L + I, using Theorem 2.10 the scale is
(bac.oz)²
1-1
2 aCaoo
Σ
CaCo
oh
+ to
2
k=0
[(
[(aaCoot)
4+1
0f
Σ
[aCoo
08
2
which concludes the proof.
3 Discussion
In a recent work, Favaro et al. 2020 2022a) has characterized the infinitely wide limit of deop Stable NNs under the assumption of a sub-linear activation function. Here, we made use of a generalized CLT to characterize the infinitely wide limit deep Stable NNs with a general activation function belonging to the classes E,, Eg and Eg. For a an/7, and in particular for the choices 7 id and ReLU with au a1, Theorem 2.6sshows that the right scaling of the NN is (n log logn)-1/o, thus including the extra factor (logn)-1/o with respect sub-linear activation functions. For on 00/7: and in particular for the choice
11

---

Published in Transactions on Machine Learning Research (11/2022)
of super-linear activations with ou a1, Theorem 2.8 shows that the distribution of the limiting output is (ooab)-Stable, with 7 > 1, and this may have undesirable consequences for posterior estimates in case of a very deep NN. In general, our work brings out the critical role of the generalized CLT, which is not as popular as the classical CLT, in the study of the large width behaviour of deep Stable NNs. As the classical CLT plays a critical role in the study of the large-width behaviour of deep Gaussian NNs (Neal 1996), our work shows how the generalized CLT plays the same critical role in the study of the large-widtil behaviour of deep Stable NNs.
A natural direction for future research consists in extending results to deep Stable NNs with k 1 inputs of dimension d, i.c. d x k input matrix X. A unified treatment of such a problem, would require a multidimensional versions of the generalized CLT, i.e. a CLT dealing with A > 1 dimensional Stable distributions, which is not available in the probabilistic/sstatiiiii literature. For a shallow Stable NN with a ReLU activation function, this problem has been considered in Favaro et al. (2022b), where the infinitely wide limit of the NN is characterized through a careful analysis of the large-width behaviour of the characteristic function of the NN. A further natural problem consists in exlending our results to the case of a "joint growth" of the width over the NN's layers, i.e. the widths of the layers growth simultaneously (Favaro) ct al. 2022a). In general, under the setting specified in definition (6) or definition
one may consider a
deep NN defined as follows:
11 (X. n) = Σ
w
W + b16 1TT
j=1
and
n
1 (X.n) f(n)
Σ j=1
(1)
- + 6y 1T.
Wij
with (X, m) = 1 (X), where denotes the k-dimensional unit (columm) vector, o is the element-wise application and f(n = logn if T E E2 and f(n) = otherwise. Then, the goal consists in extending our results to ((()) (X, n2,, assuming a "joint growth' of the width over the NN's layers. The case T E E₁ was
already tackled by Favaro et al. (020 2022ab but the other two cases are missing. In particular, Favaro et al. (2022a) showed that the assumptions of a "joint growth' and of a "sequential growth' lead to the same infinitely wide limit for a deep Stable NN with a sub-linear activation function. Instead, a critical difference between the assumption of a "joint growth' and the assumption of a "sequential growth" arises in the study of rate of convergence of the NN to its infinitely wide limit. In particular, Favaro et al. (2022a) investigated rates of convergence, in the sup-norm distance, for deep Stable NNs with sub-linear activation function, showing that the assumption of "joint growth" leads to a rate that depends on the depth, whereas the assumption of a "sequential growth' leads to a rate that is independent of the depth. We conjecture that an analogous phenomenon holds true for deep Stable NNs with lincar and super-lincar activation functions. In particular, we expect that the infinitely wide limits presented in our work hold true under the assumption that the width grows jointly over the layers, suggesting that a difference between the "joint growth" and the "scquential growth' may require the study of convergence rates. To study the large width asymptotic behaviour under the assumption of "joint growth', it might be useful to use Theorem 1 of Fortini et al. (1997), which gives sufficient conditions for the convergence to mixture of infinitely divisible laws: to apply this theorem, one should prove the convergence of a certain sequence of random measures Vn to the Levy measure of the infinitely divisible law, and then show that this limiting measure is the Lévy measure of a Stable law.
Another interesting research direction consists in studying the training dynamics of Stable NNs. For Gaussian NNs, Jacot et al. (2018) and Arora et al. (019)) established the equivalence between a specific training setting of deep Gaussian NNs and kernel regression. In particular, they considered deep Gaussian NN where the hidden layers are trained jointly under quadratic loss and gradient flow, i.c. gradient descent with infuitesimal learning rate, and it was shown that, as the width of the NN goes to infinity simultaneously, the point predictions are arbitrarily close to those given by a kernel regression with respect to the so-called neural tangent kernel. Such an analysis typically referred to as the neural tangent kernel analysis of the NN (Arora) et al. 2019). The largo-width training dynamics of shallow Stable NNs with ReLU activation function, and input X, has been considerd in Favaro et al. 2022b). In particular, they proved linear convergence of the
12

---

Published in Transactions on Machine Learning Research (11/2022)
squared error loss for suitable choice of the learning rate. The equivalence between gradient flow and kernel regression is connected to the so-called Slazy training' phenomenon, which is one of the hottest topics in the field of machine learning since it is phenomenon which can affect any model, not only NNs. More prociscly, Chizat et al. (2019) showed that lazy training is caused by an implicit choice of the scaling and that every parametrii model can be trained in the lazy regime provided that its output is initialized close to zero. Furthermore, coming back to NNs, they considered a two layers NN with Gaussian weights and proved a sufficient condition for achieving lazy training, provided that E[w; (0) T(wu; =))] = 0. Clearly, the theorem applies also in the case of symmetric Stable weights and biases when a > 1, but not when a S 1 as the expectation of such RVs undefined. It would be then intcresting to study what happens in that case in order to find a new theoretical result which leads suitable scaling for which we have the lazy training regime.
Acknowledgements
The authors are very grateful to the Action Editor (Jeffrey Penningtou) and three Referees for their comments and suggestions that improved remarkably the paper. Stefano Favaro received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement No 817257. Stefano Favaro is also affiliated to IMATI-CNR "Enrico Magenes' (Milan, Italy).
References
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, 2019.
Leo Breiman. On some limit theorems similar to the arc-sin law. Theory of Probability & Its Applications, 10:323-331 1965.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Newral Information Processing Systems, 2019.
Daren B.H. Cline. Convolution tails, product tails and domains of attraction. Probubility Theory and Related Fields, 72:529-551 -57, 1986.
Daren B.H. Cline and Gennady Samorodnitsky. Subexponentialityy of the product of independent, random variables. Stochastic Processes and their Applications, 49:75-99 98, 1994.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Rep- resentations, 2018.
Denis Denisov and Bert Zwart. On a theorem of breiman and a class of random difference equations. Journal of Applied Probability, 44:1031-1046, 2005.
Ricky Der and Daniel Lec. Beyond gaussian processes: On the distributions of infinite networks. In Advances in Neural Information Processing Systems, 2005.
Stefano Favaro, Sandra Fortini, and Stelano Peluchetti. Stable behaviour of infinitely wide deep neural networks. In International Conference on Artificial Intelligence and Statistics, 2020.
Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Deep stable neural networks: largo-width asymp- totics and convergence rates. Bernoulli, to appear, 2022a.
Stefano Favaro, Sandra. Fortini, and Stefano Peluchetti. Neural tangent kernel analysis of shallow alpha-stable relu neural networks. Preprint or arXi::2206.0800 2022b.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Sandra Fortini, Lucia Ladelli, and Eugenio Regazzini. A central limit problem for partially exchangeableran- dom variables. Theory of Probability & Tts Applications, 41:224-246 242244246, 1997.
Vincent Fortuin, Adrià Garriga-Alonse Sebastian Ober, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Baycsian ncural network priors revisited. In International Conference on Learning Representations, 2022.
Arthur Jacot, Franck Gabriel, and Clément. Hongler. Neural langent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Hoil Lee, Fadhel Ayed, Paul Jung, Juho Lee, Hongseok Yang, and Francois Caron. Deep neural networks with dependent weights: Gaussian process mixture limit, heavy tails, sparsity and compressibility. Preprint arXiv:220508818 2022.
Radford M. Neal. Bayesian Learning for Neural Networks. Peringer-Verlage 1996.
Frank W. J. Olver. Asymptotics and special functions. Academic Press, 1974.
Cira. E.G. Otiniano and Catia R. Gonçalves. Domain of attraction of alpha-stable distributions under finite mixture models. TEMA- Tendências em Matemática Aplicada e Computacional, 11:69-76, 2010.
Gennady Samorodnitsky and Murad Taqqu. Stable Non- Man-Gaussian Random Processes: Stochastic Models with Infinite Voriance. Routledge, 1994.
Vladimir V. Uchaikin and Vladimir M. Zolotarev. Chance and stability: stable distributions and their applications. Walter de Gruyter, 2011.
A Complementary statements and proofs
Definition A.1 (Multivariate Stable distribution). Let & be the unit sphere in I : {u I2 |ul =1}. A random vector, X, has a multivariate Stable distribuution, denoted as X ~ Sty Tt, (,,), 6), if the joint char- acteristic function of X is
E Exex (iv (iu*x)= = exp {-/ {|uFs) +
i (u² I(ds) + iu
SES
where 0 < a < 2, and for E IR
- sign(y) tan(Ta/2))lf # (2/T)y Iyl
a(y,a)=
a=1
The case with d = 0 is denoted by Sty(a,I).
The next. two theorems characterize the infinitely wide limits of deep NN with Gaussian and Stable parameters respectively, under the setting of joint growth and taking matrix input.
Theorem A.1 (fde G. Matthews et al. 2018)). For any d 2 and matrix, with Xj being the j-th (input signal) row, and for any D 2 1 and 2 1 let: i) (W()) W(D)
let X denote d k. (input signal)
isisn.sssd and ( (w be i. random (weight) matrices, such that W(1; ((w() (1)
(1)
1siSn.1siSm for 2 2 where the w 's are ii.dd as N (0, for = 1..... D; ū) (b())...,,bl ((D)) be i.dd random (bias) vectors, such that (()) = (D (o""...,bb)) b where the (!1) 's are i.i.d. as N (0, of) for I - I,...,.. Now, let R + IR be continuous activation function (nonlinearty) such that
(wis
[o(s)] S a bisl
14

---

Published in Transactions on Machine Learning Research (11/2022)
for every E R and for any a, 2 0, and consider (fully connected) feed-forward NN (F (X, IsisnIsieD of depth D and width n defined as follows
A (X)
Σ w w x, 6
and
12 (X, n)
1
1
!I) w
(f0 (X. + 1
vn
with (x = f.a)())) where is the t-dimensional unit (columnn) vector, and o denotes the element-wise application. Por any I,..., D, if (1. (X)) i21 is the sequence oblained by exlending (W(1),.. ,.... W(D)
and (b())...,,bl)) b(o1) to infinite i.i.d. arrays, then as +00 jointly over the first 1 NN 's layers
(x,)) i21
(x) i21
where (1 (x)) 121
is distributed as the product measure Ni21 (0, ("), and the covariance matrix E()
has the (u, -tth entry defined recursicely follows:
Σ(1) oî a (Xu; X,)
and
Σ( n.v
of {
(!))
(1-11)
x((-1)
where (f.9) ~ N₂
(1-1)
(1-1))
Ee.a
20,,
Theorem A.2 (Theorem 2 of Favaro et al. (2022a) ). For any d 2 and 2 1 let X denote a d x
&o (input signal) matrix, with Xj being the j-th (input signal) row, and for any D 2 and n 2 let:
i) (W()),... w(D) be i.i.d. random (weight) matrices, such that W w issnnssid and W
(1) wis) Isisn,Isisn for 2 S D, where the w are i..dd as Salow) for l,... D; úi (ba) ,..., b(D))
be i.t.d. random (bias) vectors, such that b(0) (b9) (011... (.....,bb)) blt) where the ( 's are i.ddd
Sa(or)
for I....,.. D. Now, let o: IR be continuous activation function (nontinearty) such that [o(s)] S (a + ))
for every IR and for any a. > 0,7 a and 3 < ,-,, and consider (fully connected) feed forward NN (x/(X,)) ISiSH,1SIDD of depth D and width n defined as follows
(x)
Σ w X, + +,,0 11
and
(X, n) nz/a j=l
Σ
(1) ih (o
(x, (x,) + +6.0 1
15

---

Published in Transactions on Machine Learning Research (11/2022)
with (X, - 1 ( (X), where 1 is the k-dimensional unit (column) veclor, and denoles the elemenl-wise application. For any ,....,,, D, if (1 (0) (X,((,,) n)) :21
is the sequence obtained by extending (WO) ..... W(D)
and (b(),. (b(),....b!?)) to infinite i.i.d. arrays, then as + too jointly over the first INN layers
(J (x.)
w
1100 (x)
i21
121
is distributed as the product measure 0:21 Stz (a, (0)),)) with E (0, 2), and the spectral measure (t] being defined recursively as follows:
where (x) i21
r = -1001111 < STT
|x, 11
C
|
j=1
and
T |1o01711
17
+
/ I
f)1
sos
bos 8" (df)
12-
10071
where
1
8
h
1/
lh
2
0
otherwise,
with d being the Dirac measure, and q1- is the distribution of fi (X). The limiting SP (x1) referred to as the Stable SP with parameter (a, I).
is
121
16

---

Published in Transactions on Machine Learning Research (11/2022)
Figure [Figure 1 of Favaro al. (2022b) ). Samples of a shallow Stable NN mapping [0,1]2 to R, with a tanh activation function and witdth n 1024, for different values of the stability parameter a: i) a 2.0 (Gaussian distribution) top-left; ii) = 1.5 top-right; iii) a 1.0 (Cauchy distribution) bottom-left; iv) = 0.5 (Lévy distribution) boghom-right.
17

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 2: Fiiuuee of Favaro (002a) ]. Samples of a deep Stable NN mapping [0, 1]2 to R, with tanh activation function and 2 hidden layers of width n 1024, for different values of the stability parameter os i) 2.0 (Gaussian distribution) top-left; fi) a 1.5 top-right; iii) 1.0 (Cauchy distribution) bottom-lcft; iv) = 0.5 (Lévy distribution) bottom-right.
18


Reviews:
[{"review_id": "wShJkKueXW", "reviewer": "Reviewer_X1ij", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* Extension of stable behaviour to a class of non-linearities which includes ReLU-like, polynomial, and linear FCNs.\n* Interesting results on tail behaviour of products $X \\cdot \\tau (Y)$ where $X$ and $Y$ are independent symmetric stable random variables.\n\n### Weaknesses\n* Use of \"sequential limit\" to make statements about behaviour of FCNs with more than one hidden layer.\n* Little discussion the \"joint limit\", and the difficulties of deriving it for stable init/why it is not included in the current paper.\n\nHere:\n- \"Sequential limit\" characterises large width behaviour of **1-hidden layer** NNs, optionally with random inputs drawn from a stable distribution.\n- \"Joint limit\" characterises large width behaviour of arbitrary fixed depth NN, where all layers are finite and grow wide together.\n\nThis distinction is not sufficiently emphasised, and the theorems claiming to characterise behaviour of **deep** stable NNs are thus misleading.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines large width function space behaviour of fully connected neural networks (FCNs) with weights drawn i.i.d. from a **stable**—instead of Gaussian—distribution. Some results of this kind have already been obtained in the cited papers by Favaro et al. (2020, 2021). One limitation of the Favaro et al. papers is that they assume that the  activation function $\\tau$ is bounded by a  **strictly** sub-linear function. This among else excludes linear and ReLU-like non-linearities. (The sublinearity in Favaro et al. is a simple way to ensure that $\\tau(X)$ has well-behaved tail behaviour for $X$ drawn from a symmetric stable distribution $S_\\alpha(\\sigma)$.) The paper presented here extends the class of activation functions to those bounded by certain linear and super-linear functions. This extends the class of non-linearities for which we understand large width behaviour to, e.g., ReLU and polynomial activations, but—unlike in the Favaro et al. (2020, 2021)—restricts focus onto **1-hidden layer** FCNs. (Only _sequential limits_ considered for deeper FCNs; more on this in \"Strengths and Weakness\"). \n\nTo me, the main contribution of the paper is in characterising the tail behaviour of $X \\cdot \\tau(Y)$ where $X \\sim S_{\\alpha} (\\sigma)$ and $Y \\sim S_{\\alpha} (\\sigma)$, for the various activation functions $\\tau$, and subsequent application of these results to characterising the large width behaviour of 1-hidden layer FCNs at initialisation.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Joint limit\nI am concerned that the extension to linearly and super-linearly bounded activations, _while reverting to 1-hidden layer NNs_, is too thin to warrant an independent publication (esp. given the mentioned work of Favaro et al., which does prove the joint limit for strictly sub-linear activations).\n\nA change that would convince me to recommend accept is proving the joint limit for the activations studied here.\n\nI am potentially willing to reconsider the above requirement if the authors can provide a convincing argument for why proving the joint limit is beyond the current capabilities of the research community. Few (seemingly?) relevant papers:\n\n* https://www.jstor.org/stable/2243946\n* https://link.springer.com/chapter/10.1007/978-3-0348-8829-5_8\n* https://arxiv.org/abs/1204.4357\n* https://epubs.siam.org/doi/abs/10.1137/S0040585X97975459\n\n\n### Relevance of stable limits\n\nWhile I appreciate there is some theoretical interest in understanding the stable limits, there is little discussion of their practical importance (beyond a short mention of Radford Neal's original work). \n\nIt is not necessary to cover this in detail, but the work would be strengthened if the authors could provide a convincing argument (or even better, empirical evidence) for why stable limits could provide modelling benefits (as opposed to, e.g., the NNGP/NTK limits).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yGy1SIXz8Y", "reviewer": "Reviewer_WgWA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the results are quite general, supporting a broad class of activations, as well as large choices of stable distributions for the weights.\n\nWeaknesses:\n- The improvements over previous works of Favaro et al. are somewhat minor, mainly regarding extending the generality of activations. For instance, the presence of the log-factor in the scaling seemed to already be present in [this previous paper](https://arxiv.org/abs/2206.08065) for the special case of the ReLU.\n- The results only study convergence of the network output for a single fixed input, as opposed to previous works which study additional properties of the stochastic process, namely the dependency structure across different inputs, e.g. through a covariance function (in the GP case), or spectral measure on a finite set of datapoints (in the Stable case).\n- The motivation for studying Stable weights instead of Gaussian is somewhat limited. It would be helpful to provide more references that highlight their relevance. (looking at some of the papers by Favaro et al., it seems that an additional, interesting motivation is that the weights *after training* often exhibit heavy tails, even though the initialization is Gaussian)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies infinite-width limits of (deep) neural networks with weights sampled from $\\alpha$-Stable distributions (which may have different tail behaviors than the usual Gaussian initialization, for which $\\alpha = 2$), and actions with different asymptotic growth conditions.\nThe authors provide convergence results to stable distributions for shallow and deep networks with different stable weight distributions and activations, along with appropriate scaling factors, when looking at the network output on a single input example. In particular, for asymptotically linear activations such as the ReLU, it is shown that the scaling factor needs an additional logarithmic term, and for activations with super-linear growth, the output distribution may have a scale that shrinks exponentially with depth, when all layers have an equal scale.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Despite the limited significance, the results in the present paper may be useful for the community. Here are some suggestions for how the paper could be improved:\n* more discussion of the motivation for studying Stable networks, as explained in the previous section.\n* a more detailed comparison with related works, in particular Favaro et al., 2022, which considers the specific case of the ReLU activation.\n* in terms of presentation, perhaps the examples in section 2.3 could be incorporated in the corresponding prior sections?\n\nSome additional minor comments/typos:\n* the term \"linear activation\" e.g. in the abstract sounds like you're only considering linear activation functions, although this is only referring to the asymptotic behavior. Perhaps you can change this to \"asymptotically linear\" for more generality?\n* typo at end of abstract: \"infinitely with\"?\n* section 1.2: \"shallow Stable with\" -> \"shallow Stable NN with\"?\n* beginning of proof of Lemma 2.4: $z$ should be $x$?\n* before Thm 2.6: \"CLT 2\" refers to Theorem 2.1?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XtHB268UBO", "reviewer": "Reviewer_zH9t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The flow of the presentation of theoretical results seems logical and sensible.\n- This is a direction of infinite-width models that has not been explored a lot (in contrast with showing that another architecture converges to a GP, or finding the covariance function of another activation function, or finding good approximation algorithms, or etc.)\n- The proofs appear to be a straight-forward application of the GCLT. The construction is where the innovation happens.\n\nWeaknesses:\n- No quantification of the rate of convergence\n- There are no experiments in this paper. While this is not a requirement per se, it would be nice to see. For example, you could evaluate the MMD between a finite width process and an infinite width process. Even more compelling would be if you could do inference using one of these infinite width models.\n\nQuestions:\n- In the GCLT, I am trying to understand the centering in the case $p =1$. When $p \\neq 1$, the centering term does not depend on $n$. But when $p=1$, it seems to get larger in magnitude with $n$. Is this correct? Why does this happen for $p=1$ but not the others?\n- (An interesting question IMO, though not necessarily one that needs to be answered in this paper. Maybe discussed though.) Is the additional logarithmic scaling encountered in certain settings \"meaningful\"? Maybe around the Chizat and Bach reference in the conclusion. Can finite-width models be trained lazily in this regime?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "It is well-established that, under mild assumptions, infinite width neural networks with well-behaved i.i.d. parameters converge in law to Gaussian processes as the width goes to infinity. This work studies a relaxation where the network's parameters belong to an alpha-stable distribution and the limiting stochastic process is a so-called Stable stochastic process. \n- The authors study certain classes of sub-linear, linear and super-linear activations (E1, E2, E3, section 1.1). \n- Unlike the Gaussian case where scaling is $1/\\sqrt{n}$ for a width $n$, here the scaling may also involve an activation-dependent logarithmic factor.\n- Theorem 2.6 gives the result for shallow networks.\n- Theorem 2.7 gives the result for deep networks, E1, E2 and E3, sub-linear (roughly),\n- Theorem 2.8 gives E2, E3 and super-linear", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Minor points:\n- the notation for the distributions in the first paragraph. $w \\sim w \\sim \\mathcal{N}$. I think what we mean here is actually more like $w \\stackrel{d}{=} w \\sim \\mathcal{N}$. The first notion of equality is between a random variable and a random variable, and the second is between a random variable and a distribution. Same for page 2.\n- It is a little bit confusing to use $\\sim$ to denote equal limits and distributed according to, e.g. in Theorem 2.1.\n- It would be nice to have a definition of a slow varying function mentioned in Theorem 2.1, perhaps around the beginning of section 2?\n- I am not familiar with the terminology \"truncated CDF\". Is this maybe supposed to be \"complementary CDF\"? Is truncated CDF also terminology in use?\n- Top of page 6. \"stated in the next theorem.\"\n- \"and it is one of the hottest topic in the field of machine learning since \". Change \"topic\" to \"topics\", or maybe even rejig this sentence so it is a bit less casual.\n- Theorem 2.7 as a period . and comma , for $\\gamma = 1$.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the infinite width limit of neural networks with heavy-tailed stable distributions, as opposed to Gaussian distributions. It generalizes prior work to examine this limit for a broader class of activation functions, including functions with linear and super-linear growth. The proof requires taking the layers of the network to infinity in a sequential manner, rather than jointly, and at least one reviewer found this restriction to be a fairly significant limitation of the results. I would agree with the reviewer that there may be a real distinction between the two limits and more emphasis should be devoted to this question. Otherwise, the reviewers and I agree that the paper is of good quality with relevant results and merits publication at TMLR.

To more thoroughly address the question of sequential vs joint limits, there are two options. First, the authors could prove their perspective (put forward in the discussion) that the two limits yield results that are qualitatively equivalent. Of course, this proof may be well out of scope, as the authors suggest in their comments. Alternatively, the authors could revise some of the their claims so that the sequential nature of the result is prominent, with caveats added to make clear that other conclusions might be possible if the limit were taken jointly. This latter approach would involve minor edits and is the minimum revision necessary to more fully satisfy the TMLR standards for acceptance.
Decision: Accept (poster)