On the infinite-depth limit of finite-width neural networks
Authors: Soufiane Hayou
Venue: Accepted by TMLR ()

Abstract: In this paper, we study the infinite-depth limit of finite-width residual neural networks with random Gaussian weights. With proper scaling, we show that by fixing the width and taking the depth to infinity, the pre-activations converge in distribution to a zero-drift diffusion process. Unlike the infinite-width limit where the pre-activation converge weakly to a Gaussian random variable, we show that the infinite-depth limit yields different distributions depending on the choice of the activation function. We document two cases where these distributions have closed-form (different) expressions. We further show an intriguing change-of-regime phenomenon of the post-activation norms when the width increases from 3 to 4. Lastly, we study the sequential limit infinite-depth-then-infinite-width, and compare it with the more commonly studied infinite-width-then-infinite-depth limit.

Published in Transactions on Machine Learning Research (01/2023)
On the infinite-depth limit of finite-width neural networks
Soufianc Hayou Department of Mathematics
Mayou@russ edu.s..dd..g...........
National University of Singapore
Reviewed on OpenReview: https: // openreview. net/ formm? id Be-.RRLsssssa
Abstract
In this paper, we study the infinite-eptth limit of finite-width residual neural networks with random Gaussian weights. With proper scaling, show that by fixing the width and tak- ing the depth to infinity, the pre-activations converge in distribution to a zero-drift diffusion process. Unlike the infinite- witth limit where the pre-activation converge weakly to Gans- sian random variable, we show that the infinite-eppth limit yields different distributions depending on the choice of the activation function. We document two cases where these distributions have closod-form (different) expressions. We further show an intriguing change of regime phenomenon of the post-aclivation norms when the width increases from 3 to 4. Lastly, we study the sequential limit himniii--dppt--hhe--iiinieeeeee width and compare it with the more commonly studied infthite-width-then whw.eeeni-niiieeeedddll limit.
1 Introduction
The empirical success of over-parameelerized neural networks has sparked growing interest in the theoretical understanding of these models. The large number of parameters millions not billions - and the complex (non-lincar) nature of the neural computations (presence of non-linearities) make this hypothesis space highly non-trivial. However, in certain situations, increasing the number of parameters has the effect of 'placing' the network in some 'average' regime that simplifies the theoretical analysis. This is the case with the infinite- width asymptotics of random neural networks. The infinite-width limit of neural network architectures has been extensively studied in the literature (e.g. Neal (1995); Schoenholz et al. (2017); Yang (2020); Poole et al. (2016); Arora et al. (2019); Hayou et al. (2019a)), and has led to many interesting theoretical and algorithricc innovations (see Appendix for a comprehensive discussion). However, most works on this limit consider a fixed depth network. What about infmiit-depth?? Existing works on the infinile-depth limit can generally be divided into three categories:
limit: in this case, the width is taken to infinity first, then the depth is take to infinity. This is the infinite-depth limit of infinite-width neural networks. This limit was particularly used to derive the Edge of Chaos initialization scheme (Schoenholz et al., 2017; Poole et al., 2016), study the impact of the activation function (Hayou et al., 2019a), the behaviour the NTK (Hayou et al., 2020; Xiao al., 2020) etc.
The joint winiitt--iidh--add--a the width and depth are jointly taken to infinity at the same time. There are few works that study the joint width-depth limit. For instance, in (Li et al., 2021), the authors showed that for special form of residual neural networks (ResNet), the network output exhibits (scaled) log-normal behaviour in this joint limit. This is different from the sequential limit where width is taken to infinity first, followed by the depth, in which case the distribution of the network output is asymptotically normal (Schoenholz et al., 2017; Hayou et al., 2019a). In (Li et al., 2022), the authors studied the covariance kernel of an MLP the joint limit, and showed that it converges weakly to the solution of Stochastic Differential Equation (SDE). In Hanin & Nica (2020), the authors showed that in the joint limit case, the NTK of an MLP
limit: in this case, the depth-to-width ratio is fixed, and therefore,

---

Published in Transactions on Machine Learning Research (01/2023)
remains random when the width and depth jointly go to infinity. This is different from the deterministic limit of the NTK where the width is taken to infinity before depth (Hayou et al., 2020). More recently, Hanin (2022) explored the impact of the depth-to-width ratio on the correlation kernel and the gradient norms in the case of an MLP architecture, and showed that this ratio can be interpreted as an effective
network depth.
Inftnite-depth limit of winite-width neural networks: in both previous limits finfnite-widtt--hhen inniniiiiii depth limit, and the joint infiite-width-deppt limit), the width goes to infinity. Naturally, one might ask what happens if width is fixed and depth goes to infinity? What is the limiting distribution of the network output at initialization? In (Hanin, 2019), the author showed that neural networks with bounded width are still universal approximators, which motivates the study of width large depth neural networks. In (Peluchetti & Favaro, 2020), the authors showed that the pre-activations of particular ResNet architecture converge weakly to diffusion process in the infinite-depth limit. This is the result of the fact that ResNet can be seen as discretizations of SDEs (see Section 2).
In the present paper; we study the infinile-depth limit of finite-width ResNet with random Gaussian weights (an architecture that is different from the one studied in (Peluchetti & Favaro, 2020)). We are particularly interested in the asymptotic behaviour of the pre/post-acee pptpp-ttttiitiiion values. Our contributions are four-fold:
1. Unlike the width limit, we show that the distribution of the pre-activations in the infinite-dppth limit is not necessarily Gaussian. In the simple case of networks of width 1, study two cases where we obtain known but completely different distributions by carefully choosing the activation function.
2. For ReLU activation function, we introduce and discuss the phenomenon of network collapse. This phenomenon occurs when the pre-activations in some hidden layer have all non-positive values which results in zero post-activations. This leads to stagnant network where increasing the depth beyond certain level has no elfect on the network output. For any fixed width, we show that in the infinite-eppth limit, network collapse is zero-probability event, meaning that almost surely, all post-activations in the network are non-zero.
3.
For general width networks, the distribution of the pre-activations is generally intractable. We focus on the norm of the post-activations with ReLU activation function, and show that this norm has approximatcly a Geometric Bronwian Motion (GBM) dynamics. We call this Quasi-GBM. We also shed light on change of regime phenomenon that occurs when the width n increases from 3 to 4. For width n S 3, resp. n 2 4, the logarithmic growth factor of the post-activations is resp. positive.
4. We study the sequential limit seapii-ddept--hhh----iiitteeeee limit, and show some key similarities between these two
which is the converse of the more com-
monly studied limits. We particularly show that the pre-activations converge to the solution of a Mckean- Mckean-Vlasov process, which has marginal Gaussian distributions, and thus we recover the Gaussian behaviour in this limit.
All the proofs are provided in the appendix and referenced in the main text. Empirical evaluations of these thcoretical findings are also provided.
2 The fefinite-deptle limit
Hereafter, we denote the width, resp. depth, of the network by n, resp. L. We also denote the input dimension by d. Let d, n, L 2 1, and consider the following ResNet architecture of width n and depth L
Yo = Win x EER"
1 VI
(1)
Y Y-it
Wio(Y-11, 1 1=,,..,, L.
where o: R-- is the activation function, L 2 the network depth, W., E Raxd and Wi E Raxx is the weight matrix in the tth layer. We assume that the weights are randomly initialized with üd Gaussian
2

---

Published in Transactions on Machine Learning Research (01/2023)
variables W/S ~o, 2), W ~V(0,, 2). For the sake of simplification, we only consider networks with no bias, and we omit the dependence of Y1 on n in the notation. While the activation function is only defined for real numbers, we will abuse the notation and write ((2) = (ø(="),.... o(²*)) for any k-dimensional vector 2 == ==(='.... x*) E R$ for any k 2 1. We refer to the vectors (Y, = O..... } by the pre-activations and the vectors {o()), o,..., L by the post-aclivations. Hereafter, x E R2d is fixed, and we assume that x + 0.
The 1/vL scaling in Eq. (1) is not arbitrary. This specific scaling was shown to stabilize the norm of Y well as gradient norms in the large depth limit (e.g. Hayou et al. (2021); Marion et al. (2022)). In the next result (which has been shown for the single imput case in Peluchetti & Favaro (2020)), we show that the infinite depth limit of Eq. (1) (in the weak sense) exists and has the same distribution of the solution of a stochastic differential equation. For the sake of simplicity, we only state the result for the single input case; the multiple inputs case is provided in in Appendix B.
Proposition 1. Assume that the activation function & is Lipschilz on R Then, in the limit I, + 0o, the process X = Y(LL), f [0, 1], converges in distribution to the solution of the following SDE
1 dx; J/1 vn
- (Xx))ll Xo -
(2)
where (Bi)220 is Brownian motion (Wiener process), independent from Win- Moreover, if the activation function & is only locally Lipschitz, then X commerges locally to X,- More precisely, for any fixed > 0, we consider the stopping times -" = (t 2 0: |X/| 2 r), and T inf 2 0 : [X,] 2 r), then the stopped process t converges in distribution to the slopped solution XeAT of the above SDE.
The proof of Proposition 1 is provided in Appendix B.6. We use classical results on the numerical approx- imations of SDEs. Proposition shows that the infinite-depth limit of finite-width ResNet (Eq. (1)) has a similar behaviour to the solution of the SDE given in Eq. (8). In this limit, YIuI converges in distribution to Xx. Hence, properties of the solutions of Eq. (8) should theoretically be 'shared' by the pre-activations YILZ when the depth is large. For the rest of the paper, we study some properties of the solutions of Eq. (8). This requires the definition of filtered probability spaces which we omit here. All the technical details are provided in Appendix B. We compare the theoretical findings with empirical results oblained by simulating the hee/post-accivatinn of the original network Eq. (1). We refer to Xt, the solution of Eq. (8), by the infinite-depth network.
The distribution of X1 (the last layer in the infinite-dppth limit) is generally intractable, unlike in the limit (Gaussian, Hayou et al. (2021)) or joint nufnite-depth--nd-veee limit (involves a log-normal distribution in the case of an MLP architecture, Li et al. (2021)). Intuitively, one should not expect universal behaviour (e.g. the Gaussian behaviour in the infinite-width case) of the solution of Eq. (8) as this latter is highly sensitive to the choice of the activation function, and different activation functions might yield completely different distributions of X1. We demonstrate this in the next section by showing that we can recover closed-forn form distributions by carefully choosing the activation function. The main ingredient is the use of Ttô 's lemma. See Appendix B for more details.
3 Different behaviours depending on the activation function
In this section, we restrict our analysis to a width-1 ResNet with one-dimensional inputs, where each layer consists of single neuron, i.c. d = n = 1. In this case, the process (Xt)ustsi is one-dimensional and is solution of the following SDE
dx, = 1ox=)oXx))dBB. Xo = Win..
We can get rid of the absolute value in the equation above since the process X, has the same distribution as X,, the solution of the SDE dx, = o (X,)dB, The intuition behind this is that the infinitesimal random variable 'dB,' is Gaussian distributed with zero mean and variance dt. Hence, it is a symmetric random variable and can absorb the sign of Φ(X)). The rigorous justification of this fact is provided in Theorem 7 in the Appendix. Hereafter in this section, we consider the process X, solution of the SDE
dx, :X,=6(X,)dBBB ((,))d,, Xo Win..
(3)
3

---

Published in Transactions on Machine Learning Research (01/2023)
Given function g E (R)),,, use Itō 's lemma (Lemma 4 in the appendix) to derive the dynamics of the process ((x)). We obtain,
dg(X,) = di ((X))g dB. + 1 2 (X)) dt.
(4)
((x))
m(X,)
In financial mathematics nomenclature, the function μ is called the drift and a is called the volatility of the diffusion process. Itō 's lemma is a valuable tool stochastic calculus and is often used transform and simplify SDEs to better understand their properties. It can also be used to find candidate functions g and activation functions & such that the SDE Eq. (4) admits solutions with known distributions, which yields closed-form distribution for Xt. We consecrate the rest of this section to this purpose.
3.1 ReLU activation
ReLU is pieco-wise linear activation function. Let first deal with the simpler case of linear activation functions. In the next result, we show that linear activation functions yield log-normal distributions. In this case, the process X, follows the Geometric Brownian motion dynamics. Later in this section, we show that this result can be adapted to the case of the ReLU activation function given by Φ(x) = max(x,0) 0).
Proposition 2. Let x TR such that F O. Consider a lincar activation function Φly) ay + B, where a > 0 E R are constants. Let o > O and define the function by g(y) (ox + B)", where = oa Consider the slochaslic process Xt, solution of Eq. (3). Then, the process g(x,) is a solution of the SDE
dg(X.) = ag( (d)) dt. + (g(X.),B
where a = =00 - 1). As a result, we have that for all t [0,1],
g(Xi) g(Xo) exp ((a- t oB.).
The proof of Proposition 2 is provided in Appendix E, and consists of using ltô lemma and solving a differen- tial equation. When the activation function is ReLU, we still obtain a log-normal distribution conditionally on the event that the initial value Xo is positive.
Proposition 3. Let x. E R such that x + O, and let & be the ReLU activation function given by 6(z) = max(2, 0) for all = E R. Consider the stochastic process Xt, the solution of Eq. (3). Then, the process X is a misture of a Geometric Brownion motion and a constant process. More precisely, we have for all t [0, 1]
(-21+ + 1xx50)
X, ~ 1 [xx200 Xo exp exexp
Hence, given fixed No O, the process X is a Geometric Brounian motion.
The proof of Proposition 3 is provided in Appendix F. We show that conditionally on Xo > O, with probability 1, the process X, is positive for all t [o, 1Ja. When X, > 0, the ReLU activation is just the identity function, which justifies the similarity between this result and the one obtained with linear activations (Proposition 2). Conversoly, if Xo < O, the process is constant equal to Xo since the updates 'dX,'' are equal to zero in this case. A rigorous justification of this is given for general width n later in the paper (Lemına 1). An empirical verification of Proposition 2 is provided in Fig. where we compare the theoretical results to simulations of the neural paths (Yi) and (log())))ssizz from the original (finite-depth) ResNet given by Eq. (1). We observe an excellent match with theoretical predictions for depths L 50 and 100. In the case of a small depth (T. 5), the theoretical distribution does not fit well the empirical one (obtained by simulations).
HHer (R) refers to the vector space of fuuctions g R B that twice diffcrentiable and their second derivatives continuous.
2In Appendix F, we show that the stopping inf(t. 2 s.t. X2 S 03 infinite almost surely, which stronger that what we need. This classic result. stochastic calculus.
4

---

Published in Transactions on Machine Learning Research (01/2023)
Grimass dess Hor civensaiees
Extrsied sirtyy 3t
Ertinstd Seats
(a) Hist of log(Yz) with L = 5
(b) Hist of lYg(Yz) with L = 50
(c) Hist of log(Yz) with 100
Themetice ment
Deme Sotmmate covevrtrrr
64-
No0 pab
Kours path
Treersizal
Laper
Layer
(d) (log(Y))sSEE with L = 100
(e) (Xi)ssss with L 100
(1) Hist of YL with L = 100
Figure 1: Empirical verification of Proposition 2. (a), (b), (d) Histograms of log(Yt) and based on N 5000 simulations for depths L E 50,50, 100} with Yo 1. Estimated density (Gaussian kernel estimate) and theoretical density (Gaussian) are illustrated on the same graphs. (c), (e) 30 Simulations of the sequence (log(Yi))ss (c) and the sequence (Y))Ez (e). We call such sequences Neural paths. The results are reported for depth L = 100, with Yo = 1, 6 being the ReLU activation. The thcoretical mcan of log(Y)) is given by 1 m(l) 2z and that of Y; is equal to Yo 1. 1. We also illustrate the 99% confidence intervals, based on the theoretical prediction for log(Yi) (Proposition 2), and the empirical Quantiles for Yi. (f) Histogram of YL based on N = 5000 simulations for depth L = 100.
which expected since the dynamics of X describe (only) the infinite-dppth limit of the ResNet. More figures are provided in Appendix L.
Remark: notice that the log-normal behaviour is result of the fact that we only consider the case n 1 (width one). Indeed, the single neuron case forces ReLU to act like linear activation when Xo 0, and like 'zero' activation when Xo I O. For general width n 2 1, such behaviour does not hold in general, and usually some coordinates of X, will be negative while others are non-negative, which implies that the volatility term [|x(X.)] has non-trivial dependence on X,, We discuss this in more details in Section 4. In the next section, we illustrate a case of an exotic (non-standard) activation function that yields a completely different closcd-form distribution of X,.
3.2 Exotic activation
With a particular choice of the activation function & and mapping 9. the stochastic process (X,, is the solu- tion of well-known type of SDEs known as the Ornstein- Uhllebbee SDEs. In this case, the activation function is non-standard and involves the inverse of the imaginary error function, a variant of the error function.
Proposition 4 (Ornsein-Uhlleneeck Uhlenbeck neural networks). Let x E R such that x f O. Consider the following activation function ob
Ml m (1p()
5
(ly) = maxty, 01
p(y) = exp(h -a + 8)²),
2
where ax, B E IR are constants and h1 is the inverse function of the imag- inary error function given by h(z) = - So e dt. Let g be the function defined by
1
((y) a -
(ay 3)
Figure 2: Exotic activation
5

---

Published in Transactions on Machine Learning Research (01/2023)
Let the process Xt be the solution of Eq. (3)². Then, the stochastic process g(Xi) follows the Ornstein- Uhlenbeck dynamies on (0, given by
dg(X,) ag(X,)dt 2adBe g(Xo) = g(Wn.
where za². As result, conditionally on Xo (fized Xo), we have that for all E [0. 1},
s(X.) ~ N (9(X))e "2 ", (1---=2))))
and the process X, is distributed as X, a 1h(aa11aa111111 (g(Xo)e 2 c c catc))t)) B).
and B = O. With this choice of the activation function, the infinito-depth network output X1 has the distribution g- 1 (N (g(o))e at 2(1 + =(a))) (conditionally on Xo), where g is given in the statement of the proposition. This distribution, although easy to simulate, is different from both the Gaussian distribution that we obtain in the infinite-width limit and the log-normal distribution associated with ReLU activation. This confirms that not only do neural networks exhibit completely different behaviours when the ratio depth-to-width is large, but in this case, that their behaviour very sensitive the choice of the activation function. The results of Proposition are empirically confirmed in Fig. 3. The original ResNet given by Eq. (8) with depth L 100 exhibit very similar behaviour to that of the SDE.
Fig. shows the graph of the function o(y) (g)) mentioned in Proposition 4 with
Oeimatcr dena dusily
Estruted Emsty OEET
Eximated demils darary
(a) Hist of g(z)) with L =
(b) Hist of g(\2) with L = 50
(c) Hist of 9(z)) with L 100
Themetic
Same Deratl Criinse
Nour pat
Laper robos
Loyer IndeR
(d) (o(Y)ssisL with L = 100
(e) (Yi)agisz with L = 100
(f) Hist of Y2 with = 100
Figure 3: Empirical verification Proposition 4. (a), (b), (d) Histograms of g(Y)) based on N = 5000
simulations for depths L E (5, 50, 100) with Yo = Estimated density (Gaussian kernel estimate) and theoretical density (Gaussian) are illustrated on the same graphs. (c), (e) 30 Simulations of the neural paths (g(Y)))ssz (c) and (Y))EL (e). The results are reported for depth Proposition 4. The theoretical mean of g(Y)) (conditionally on Yo) is approximated by m(!) = g(Y)e 전 and that Yú is equal to Yo = 1. We also illustrate the 99% confidence intervals, based on the theoretical prediction for g(Vi) (Proposition 2), and the empirical Quantiles for Vi. (f) Histogram of Y, based on N 5000 simulations for depth L 100.
100, with Yo 1, é is given un
Pin Appendix D, we show that the activation function is only locally Lipechita. Hence, the solution of this SDE exists only in the local sense and the convergence in distribution of Yiu to X is also in the local sense (Proposition 1). However, by continuity of the Brownian path, the stopping times 74 and diverge almost surely when goes infinity. Therefore, the conclusion of Proposition remains true for all E [0, 1). Technical details provided Appendix D.

---

case, which states that Published in Transactions on Machine Learning Research (01/2023)
4 General width n 2
Let 2 and IRd such that + 0. Consider the process X given by the SDE
1 dx, vn nz [lo(X)) Xu W.
(5)
where Φ is the activation function, and B is an n-dimensional Brownian motion, independent from W in For 2 2, the coordinates of X₁ dependent which makes the distribution of X2 generally intractable. This intractability is purely due to this dependence between the coordinates and not to the non-linearity itself. To understand this, we have added in Appendix K comprehensive analysis of the case of the identity activation and other piece-wist wise activation functions.
Intuitively, if for some 8, if [lx(X,)]l = 0 in Eq. (5), then for all 2 8, X, X, since the increments 'dX,'' are all zero for 2 s. This holds for any choice of the activation function 6, provided that the process X exists, Le. the SDE has unique solution. We summarize this in the next lemma.
Lemma 1 (Collapse). Let æ E Hd such that # O, and solution of the SDE given by Eq. (5). Assume that for some 0, <(X,) = 0. Then, for all 2 8, X X almost surely.
R be a Lipschitz function. Let X be the
Lemma 1 is particular case of Lemma 7 in the Appendix. The proof consists of using the uniqueness of the solution of Eq. (5) when the volatility term is Lipschitz. This result is trivial in the finite depth case (Eq. (1)). When there exists s such that o(X,) = O, the process X becomes constant (equal to X,) for all /. 2 8 (almost surely). We call this phenomenon process collapse. In the case finite-depth networks (Eq. (1)), we call the same phenomenon network collapse. Understanding when, and whether, such event occurs is useful since it has significant implications on the the large depth bchaviour of neural networks. Indeed, if such event occurs, it would mean that increasing depth has no effect on the network output after some time 8 (or approximately, after layer index [sT])- In the next result, show that under mild conditions on the activation function, process collapse is a zero-probability event.
4.1 Network collapse
The next result gives (mild) sufficient conditions on the activation function that the process X almost surely does not collapse. In the proof, use Itô 's lemma in the for any function IR" R that is C² (TTT), we have that
dg((,) = Vg(X,) ax, 2n IIxxM)II [v'((x))]
Lemma 2. Let x E ddd such that + O, and consider the stochastic process X given by the following SDE
dx2 = J Vn
1 [o(x:)]|dBe 1 (0),, Xo = Wm,
where d(z) : is Lipschilz, injective, (RR and satisfies Φ(0) = O, and ø' and 6² are bounded on K, and (Br)t20 is an n--dimensional Brownian motion independent from Win ~ NN0, ((0,d-11). Let T be the stopping time given by r min(t 2 0: ((Xi) = =0.. Then, we have that
P = 00) =1
The proof Lemma is provided in Appendix G. Many standard activation functions satisfy the conditions of Lemma 2. Examples include Hyperbolic Tangent Tanh(z) = _ and smooth versions of ReLU activation such as GeLU given by (2) = =x)(z) where is the cumulative distribution function of the standard Gaussian variable, and Swish (or SiLU) given by @swish (2) = 2h(2) where h(z) (1+e-3)-11 is the Sigmoid function. The result of Lemma can be extended to the case when @ the ReLU function with miner changes.

---

Published in Transactions on Machine Learning Research (01/2023)
Lemma 3. Consider the stochastic process (8) given by the SDE
dxX, = Vn [l((i))lddB I E (0,00) Xo = Win.
where & is the ReLU activation function, and (Be)izo is an n-dimensional Brownian motion independent from Win N(0,d 1). Let be the slopping time given by
T min 2 0: [xx(x|1| 0} min 2 20: ViE [n], X S 50.
Them, we have that
= 1 > 0) 1.
As resull, we have thal.
= =0)= -1-
The proof of Lemma relics on a particular choice of a sequence of functions (0m)m21 that approximate the ReLU activation é. Details are provided in Appendix G.
The result of Lemma 3 shows that for all T > O, with probability 1, if there exists j [n] such that X > O, then for all t E [0, there exists coordinate i such that X; 0, which implics that the volatility of the process X given by / (xx))] does not vanish in finite time t. Notably, this implies that for any E [0, the norm of post-activations given by [|ø(X))]l does not vanish (with proba- bility 1). This important as it ensures that the vector d(X2), which represents the post-activations in the infinite-depth network, does not vanish, and therefore the process X, does not get stuck in absorbent point. The dependence between the ordinates of the process X, is crucial in this result. In the opposite case where X₁ are independent, the event
n= 1
[Il((X.)] = 0} has probability 2-x Notice also that this result holds only in the infinite-depth limit. With
0.150
Collapse probability
n
0.125
finite-depth ResNet (Eq. (1)) with ReLU activation, is not hard to show that the network collapse event (31 [L], s.t. [1 = 0} has non-zero probability. However, the depth increases, the probability of net- work collapse goes to zero. Fig. 4 shows the probability of network collapse for a finite-width and depth ResNet (Eq. (1)). As the depth L increases, becomes unlikely that the wetw collapses. This is in agreement with our theoretical prediction that the infinite-depth network rep- resented by the process X, has zero-probabiln pro-probbbbiliy collapse event, conditionally probability of neural collapse also decreases with width, which is expected, since it becomes less likely to have all pre-activations non-poses non-positive as the width increases.
0.100
0.025
0.000
3
11 Depth
13
15
17
19
Probability of the event [31 e [T. such that e(Y) 0} (collapse) for varying widths and depths. The probability and the 95% confidence intervals are estinated using N 5000 samples.
Figure 4:
the fact that [|o(Xo)] > 0. The
4.2 Post-activation norm
As result of Lemma 3, conditionally on [[o(X)) > O, we can safely consider manipulating functions that require positivencss such as the logarithm of the norm of the post-activations. In the next result, we show that the norm of the post-activations has distrribution that resembles the log-normal distribution. We call this Quasi Geometric Brownian Motion distribution (Quasi-GBM)
Thcorem (Quasi-GBM behaviour of the post-activations norm). We have that for all E [0, 1),
1 Vn
||x (I) = || (X)) exp
B,
vads) almost surely,
8

---

Published in Transactions on Machine Learning Research (01/2023)
Noma buraly Catrait Hos inserved
Rame Denriy Ererata
Gehel Dern GETTAO esservaties
Hor af wicheeneiion
2115
(a) 72 = 2
(b) 1 3
(c) n=
Kome Dorals Eatreite Hin observed
Komel Catrin Iist obsoratton
kome Deraty Calimtt Hit dtseraato
413
(d) n n 6
(e) n = 20
(f) n = 100
Figure 5: Histogram of ( for depth L = 100 and different widths n E {2,3, 4, 20, 100} based on N 5000 simulations. Gaussian density estimate and (Gaussian) kernel density estimates shown. We observe a great match between the best Gaussian estimate and the empirical distribution, which confirms the quasi-log- normal theoretical predictions from Theorem 1.
nori-ll
where pa = (x.)1 1, and (B)120 is a one-dimensional Brownian motion. As a result, for all 0 S $ 1
||o(xxo)) of
(1--2-))
1 12
E log
(t s).
4
Moreover, for n 2 2, we have
(11((x))1) [|0((xo)1 >0 0 S 01=1(1112 (n-1/2 + +1,22 (t ss- s),
Var log
[|x(x.)]|
where I'oo. = ((E) - (1-2-7)² + +n
(- (1-2 -
Es' (x) du.
Different tools from stochastic calculus and probability theory are used in the proof of Theorem 1. Technical details are provided in Appendix H. The first result in the thcorem suggests that the norm of the post- activations has a quasi-log-normaa distribution (conditionally Gaussian (n 1//2B, and the second term depends n 'm Mag which involves an average over (d' (d(())) 1<isn. In the large width limit, this average concentrates around its mean we will see in Theorem 2. In Fig. 5, we show the histogram of Voloo( (()L))llll((Xxx)))) for depth L = 100 and varying widths Surprisingly, the log-normal approximation fits the empirical distribution very well even for small widths n E (2,3, 4, 6} for which the term n n-1,4 is not necessarily close to its mean'. More interestingly, the result Theorem 1 shods light on an intriguing change of regime that occurs between widths n 3 and n 4. Indeed, for n S 3, the logarithmic growth factor of the norm of the post-activations [|ø(X.)]l tends to decrease with depth on average, while it increases for 2 4. When small. This change of regime phenomenon suggests that for n < 3, the random variable [0(X))l//ll(X) has significant probability mass in the region 1). This probability mass tends to 0 as increases since (((x))ll/16( converges to a deterministic constant. (we will see this in the next theorem), and the variance upperbound in Theorem converges to when n goes to infinity, which can be explained by the fact that Eø' (X "2 1/4 (the coordinates become independent in the large width limit, see next
Xo). The first term in the exponential is
4, the average growth is positive although very
We currently do not have rigorous explanation for this effect. possible explanation for this empirical result that the integral over Ms some 'averaging' effect.
9

---

Published in Transactions on Machine Learning Research (01/2023)
Empricolmssn
Emprrcalmenn
10g (BANN
-42
-15
-18
-15
Layer
Layer
(a) n= 2
(b) 3
Thecretical meen Emprrtcalmasnn
Theorelical meen Empiiicalmenn
20
10g (1eo
-18
Layer Index
Layer Index
(c) n n=
(d) 71 6
Empinical
Emprricalmenn
100 GERNNE
log
Layer
Layer
20
(I) 100
Figure 6: 30 simulations of the sequence n. E [2,3, 6, 20, 100). Theoretical means from Theorem are shown in red dashed lines and compared to their empirical counterparts. We observe that when the ratio L/n incrcases (especially for n = 100), the empirical mean also increases and becomes significantly different from the theoretical prediction.
for depth L = 100 and different widths
theorem). Experiments showing this concentration are provided in Appendix L.5. In Fig. 6, we simulate 30 neural paths (i.e. (Yi) for depth L 100 and compute the logarithmic factor log( (()))((())///l(()()))))) An excellent match with the theoretical results observed for widths n E {2, 3, 4, 6, 20}. A mismatch between theory and empirical results appears when 50, which expected, since the theoretical results of Theorem 1 yield good approximations only when < L.
Notice that the case of ((X)) > 0, we have log(ø(X ((()(()))((()))) = log( X./ Xx)Xo) ~ -t -t/2+ B, where B is one-dimensional Brownian motion, and where we have used the fact that X, > 0 for all t. This result can be readily obtained from Theorem 1 by setting n =
matches the result of Proposition 3. Indeed, the latter implies that conditionally
An interesting question that of the indtnito-width limit of the process X,, which corresponds the sequential limit infiniteddept--heet section.
of the ResNet Yu..j (Eq. (1)). We discuss this in the next
10

---

Published in Transactions on Machine Learning Research (01/2023)
4.3 lnfihite-width limit of infinite-eppth networks
In the next result, we show that when the width goes infinity, the ratio ( concentrates around a layer dependent (t-dependent) constant. In this limit, the coordinates of X, converge in L2 to a Mckcan-Vlasor process, which allows us to recover the Gaussian behaviour of the pro-activations of the ResNet. We later compare this with the converse sequential limit wwwthwiiewhhhhihiiieeee the pre-activations are also normally distributed, and show a key difference the variance of the Gaussian distribution.
where
Theorem
limit). For 0 ssstss we have
log (1o((x))) (11((X))1>00) - and, [
||x(X,)]
t-8
| 1 (11((X0)1>0) - exp
1-000
where the convergence holds in LA- Moreover, we have that
sup E (E(m]
sup [E(0.1]
X x;12 =O(n
where X3 is the solution of the following (Mekean- Vlasov) SDE
dx: = (Eo(X)))) 1/2
dB X = Xó-
As a result, the pre-activations YiLL (Eq. (1)) converge in distribution to a Gaussian distribution in the limit infinite-depth deattt--dpth-theeee ceeen--ifiiiteeeee
Vi E [n], [(r
then n-100 NO (10 exp(t/2)).
The proof of Theorem 2 requires the use of special variant of the Law of large numbers for non sid random variables, and a convergence result of particle systems from the theory of Mckean Vlasov processes. Details are provided in Appendix 1. In neural network terms, Theorem 2 shows that the logarithmic growth factor of the norm of the post-activations, given by log (o.((()))l///l((( II),, converges (t (t-s)/4 in the soquential limit --+ 0o; then + co. More importantly, the pre-activations YiL converge in distribution to zero- mean Gaussian distribution in this limit, with layer-dependent variance. In the converse sequential limit, i.e. 00, then L + co, the limiting distribution of the pre-activations Yiu is also Gaussian with the same variance. We show this in the following result, which uses Lemma in (Hayou et al., 2021).
Theorem seghiitt-width--te---nnnnie
limit). Let [0. Then, the limit lim lim (infinite width, then infinite depth), we have that
1 IIl [(+())
[1l0())110) - exp (
where the convergence holds in probability.
Moreover, the pre-activations Yiul (Eq. (1)) converge in distribution to Gaussian distribution in the limit hithee--iii-----e----iinnneee
Vi [r], MiL n then L-+20 d(0,d=~1|211 exp(t/2))
The proof of Theorem 3 is provided in Appendix J. We use existing results from Hayou et al. (2021) on the infinitedepth asymptotics of the neural network Gaussian process (NNGP). It turns out that the order to the sequential limit (taking the width to infinity first, then taking the depth to infinity, or the converse) does not affect the limiting distribution, which a Gaussian with variance exp(t/2)). Intuitively, by taking the width infinity first, we make the coordinates independent from each other, and the processes (Yf) become idd Markov chains. Taking the infinite-depth limit after the infinite-widtt limit consists of taking the
11

---

Published in Transactions on Machine Learning Research (01/2023)
infinite-depth limit of one-dimensional Markov chains. On the other hand, when take depth to infinity first, the coordinates (Xi)ısign remain dependent (through the volatility term n=11/2||0b (Xx)))) which results in the moasi-llg-normal behaviour of the norm of the post-activations (Theorem 1). Taking the width to infinity then yiclds an asymptotic norm of the post-activations equal to [[x(X))] exp(t/2) (Theorem 2) which is the same norm in the converse limit (Theorem 3). It remains to take the width to infinity to decouple the coordinates and obtain the Gaussian distribution (through the Mckean- Meksan-Vlasov dynamics). Knowing that the variance of the pre-activations is mainly determined by the norm of the post-activations (Eq. (5)), we can see why the variance is similar in both sequential limits.
5 Discussion on the case of multiple inputs
The result of Proposition 1 can be easily generalized to the multiple input case, and the resulting dynamics is still an SDE. The generalization to the multiple imputs case is given by Proposition 5 in the Appendix.
An important question in the literature on mhimt neural networks is the behaviour of the cor- relation of the pre-activations (or the post-activations) for different. inputs a and b, which is given by (Y (a), Yuz, (6)) This correlation can be as a geometric measure of the information as it propagates through ea) (22 the worwork. In the infinit-wiittb---ee--dee
limil, this correlation (generally) converges to a degenerate limit (a constant value) which results in either a constant or a sharp landscape of the network output and causes gradient exploding/vanishing issues (Schoenholz et al., 2017; Yang & Schoenholz, 2017; Hayou et al., 2019a). Techniques such block scaling (Hayou et al., 2021), or kernel shaping (Zhang et al., 2022; Martens et al., 2021) solve this problem and ensure that the correlation is well-behaved in the large depth limit.. In our case, when the width n finite and the depth L is taken to infinity, can define the correlation for two inputs b and time t [0,1] by
ca(a, b) del (Xa(a), (.)O)) [
Correlation
Using Itō 's lemma, e has dynamics of the form dc b) = D (()))))
-0.50
(6)
0.75
for some non-trivial mapping V. Unfortumately, this kind of dynamics (which is not an SDE) is generally in- tractable, and we are currently investigating these dy- namics for future work. However, since scale the ResNet blocks with the factor 1/VL (Eq. (1)), which is the same scaling that solves the degeneracy issue in the sofnite-widlu--hen--eeee
-1.00
100 125
150
200
Layer Index
Figure 10 Simulations of the correlation path (uuu (1)))
[1 (a)lll (()z) ISIST. for depth L = 200,
width n = 20, and different (a, (different initial correlations co). The color code depends only the initial correlation value co (red for the largest correlations values)
limit (Hayou et al., 2021), it should be expected that the correlation kernel e does not converge to a degenerate limit. In Fig. 7, we simulate 200 and
the correlation path in ResNet of depth width n 20. The paths exhibits some level of stochas- ticity but degeneracy can be observed. Understanding the correlation dynamics (Eq. (6)) in the infinite-depth limit of finito-width networks is an interesting open question. The infinite-width limit of these dynamics is also an interesting open question. We leave this for future work.
6 Practical implications
Our thcoretical analysis has many interesting implications from a practical standpoint. Here we summarize some key insights form our results.
5Te widhh limit of dnfih correlations
12

---

Published in Transactions on Machine Learning Research (01/2023)
Initialization and stability in the large depth limit. An important factor pertaining to the trainability of neural networks is the behaviour of the neurons (prr/post-activations). Ensur- ing that the neurons are well-behaved at initialization is crucial for training since the first step of any gradient- based training algorithm depends on the values of the neurons at initialization.
This has led to interesting developments in initialization schemes for MLPs such as the Edge of Chaos (Poole et al., 2016; Schoenholz et al., 2017) which ensures that the variance of the pre-activation does not (exponentially) vanish explode in the large depth limit. In the case of ResNet, we know from the existing theory on the infinitewwidth limit of neural networks that scaling the residual blocks with 1/VL stabilizes the pre/post-actiiations in the large depth limit (Hayou et al., 2021). Hence, we do not need a special initialization scheme with this scaling. However, one could argue that this (approximately) ensures stability only when the width is much larger than the depth. What about the other cases when Re L or < L? the last case can be studied by fixing the width and taking the depth to infinity. In our paper, we not only show that the neurons remain stable in fixed-width large-depth notworks,
Scaled
Gradient norm (normalized)
1.6
1.4
1.2
1.0
0.8
20
40
60
80
100
Layer index
3.0 1eB
Unscaled
norm (normalized)
but we fully characterize their behaviour when the depth is infi- nite and show that it follows an SDE in this limit. To summarize, we show that initializing ResNet Eq. (1) with standard Gaussian random variables and scaling the blocks with 1/vL. ensures stability inside the network in large-depth (fixed-width) networks (notice that this is actually equivalent to scaling the variance of the initialization weights with 1/L, which can be seen as an initialization scheme). Intuitively, by stabilizing the pro-activations, we also stabilize the gradients. To confirm this intuition, we show in Fig. the evolu- tion of gradient norms as they back-propagate through the network. This experiment was conducted by fixing the last layer's gradient to constant value and back- back-propagattin the gradient from there. The result shows that the 1/VT. scaling, along with standard Gaus- sian initialization, ensure well-behaved gradients which is a desirable property for gradicnt- based training. Another interesting property of the Edge of Chaos initialization scheme for MLPs is that it en- sures that correlation kernel (correlation between the pre-activations for different inputs) does not exponentially converge to a degenerate value (constant value)". We discussed some aspects of the correla- tion kernel in Section 5 and showed empirically that with the 1 1/VT. scaling, the correlation is well-behaved and does not converge to degenerale values (Fig. 7).
2.5
2.0
Gradient norm
1.5
1.0
0.5
0.0
20
40
60
80
100
Layer Index
Figure 8: 10 Simulations of the gradi-
ent norm for scaled ResNet. (Eq. (1)) and non-scaled ResNet (Yi = Y--1 + Wio(Y-1)) for depth L = 100 and width 10. We normalize the gra- dients norms by the gradient norm of the last layer. The color code depends only on the ratio of the graident norm at the first layer to that of the last layer (dark red for the largest values). Without scaling, the gradient norm explodes (highly likely). The 1/VL stabilizes the gradients as they back- propagate through the network.
Network collapse. Another issuc that could occur in finito-width networks is that of network collapse, ie. when the pre-activations a hidden layer are all negative, which causes the post-activations to be all zero. In ResNet (Eq. (1)), this implies that increasing depth beyond some level has no effect on the network output. This is problematic since the weights in those 'inactive' layers have zero gradient and thus will not be updated when such event occurs. A simple way to understand network collapse is to see what happens at initialization. When the width n is sufficiently large, one can expect that such event is unlikely to occur. What about small-width neural networks? we offer a simple answer to this question: for finite-width neural networks, increasing the depth L ensures that such event unlikely to happen. This is true even for extremely small widths, c.g. n 2,3, which is counter-intuilive. Empirical results in Fig. support this theoretical prediction.
TTe correlation still converges 1 with an EOC initialization. The benefit of the EOC lies the fact that the convergence rate much slower (polynomial Vs exponential) (Schoenholz et al., 2017; Hayou al., 2019a)
13

---

Published in Transactions on Machine Learning Research (01/2023)
No universal kernel regime. An interesting application of fixed-depth infinite-width neural network is the so-called Neural Network Gaussian Process (NNGP). This is the Gaussian process limit of neural networks, that can be used to perform posterior inference and obtain uncertainty estimates (Lee et al., 2018). The converse case, i.c. fixed-width infinite-depth, has been however poorly understood, and the question of whether the infinite-eppth limit of finite-width networks has some universal behaviour has been an open question since. We addressed this question this work and showed that the limit (in the case of the Nes architecture Eq. (1)) does not admit universal distribution (e.g. Gaussian process in the infinitewwidtt limit). More precisely, this limit is highly sensitive to the choice of the activation function.
the infinite-depth limit of infinite-width neural networks has been studied in the literature (Hayou et al., 2019a; 2020). It is known that in this limit, the network behaves as a Gaussian process with a well-defined kernel. What about the converse limit, Le.. infinite-width limit of infinite-eppth networks? this has been so far an open question, and our work addresses one part of it. We show that the marginal distributions are zero-mean Gaussians with the same variance as in the infinite-widthh width-then--epth. limit. Characterizing the full covariance kernel is still however an open question (sce Section for a discussion this topic).
What about infhiite-depth--hhn--iihhh
7 Conclusion and limitations
Understanding the limiting laws of randomly initialized neural networks is important on many levels. Primar- ily, understanding these limiting laws allows us to derive new designs that are immune to seploding/vanisshinn cecsactivations/rradiene phenomena. Next, they also enable a deeper understanding of overparameterized neural networks, and (often) yield many interesting (and simple) justifications to the apparent advantage of overparameterizatio. So far, the focus has been mainly on the infniit-width limit (and infnnit--witth-hhen infinite-depth limit) with few developments on the joint limit. Our work adds to this stream of papers by studying the infinitedepphh limit of finito-width neural networks. We showed that unlike the infinite- width limit, where we always obtain (under some mild conditions on the activation function) Gaussian distribu- tion, the infinite-epphh limit is highly sensitive to the choice of the activation function; using the Ttô 's lemma, we showed how we can obtain certain known distributions by carefully tuning the activation function. In the general width limit, we showed an important characteristic of infinite-depth neural networks with general activation functions (including ReLU, conditionally on (l > 0): the probability of process collapse is zero, meaning that with probability one, the process X, does not get stuck at any absorbent point. This is not true for finite-dpthh ResNets as we can see in Fig. 4, which highlights the fact that as we increase depth, the collapse probability tends to decrease, and eventually converges to zero in the infinite-eppth limit, which is in agreement with our results.
This work, although novel in many aspects, is still far from depicting complete picture of the infinite-eppth limit of finite-width networks. There are still numerous interesting open questions in this research direction. Indeed, one of these is the dynamics of the gradient, and more specifically the behaviour of the NTK in the infinite-eptth limit of finite-width neural networks. For instance, we already know that in the joint infintt-wiitt-deey limit of ML.Ps, the NTK is random (Hanin, 2019); but what happens when the width is fixed and the depth goes to infinity? In the MLP case, a degenerate NTK should be expected. Henceforth, questions remain as to whether suitable scaling leads to interesting (non-degenerate) infinitedepth limit of the NTK as is the case of the infinitedepthh limit of infinile-width NTK (Hayou et al., 2021).
14

---

Published in Transactions on Machine Learning Research (01/2023)
References
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine--gained analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedinggs of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 322 332. PMLR, 2019.
Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 7(10), 2019.
Boris Hanin. Correlation functions in random fully connected neural networks at finite width. 2022.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In International Conference Leurning Represenlations, 2020.
S. Hayou, A. Doucet, and J. Rousseau. On the impact of the activation function on deep neural networks training. In International Conference Machine Learning, 2019a.
S. Hayou, A. Doucet, and J. Rousscau. Mean-field behaviour of neural tangent kernel for deep neural networks, arXiv preprint (tfe::999511354. 2020.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. Etraining dynamics of deep networks using stochastic gradient descent via neural tangent kernel. 2019b.
Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. Stable resnet. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Re- search, pp. 1324 1332. PMLR, 13 15 Apr 2021.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. The curse of depth in kernel regime. In Melanie F. Pradier, Aaron Schein, Stephanie Hyland, Francisco J. R. Ruiz, and Jessica Z. Forde (eds.), ), Proceedings on 'I (Still) Can't Believe It's Not Better!" at NeurIPS 2021 Workshops, volume 163 of Proceedings of Machine Learning Research, pp. 41-47. PMLR, 2022.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: NNGP and NTK for deep attention networks. In Hal Dauné III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Reseorch, pp. 4386- PMLR, 2020.
Jonathan Ingersoll. Theory of Financial Decision Making. 1987.
Arthur Jacot. Theory of Deep Learning: Neural Tangent Kernel and Beyond. 2022. URL https: TtF.___H982ppppp
//infoscience. epfl. wwwpe/eeeeeee...cceeeeeeeeeeeeee wwt.c//eeeor/////////eeeee/////e
Arthur Jacot, Franck Gabriel, Francois Ged, and Clement Hongler. Freeze and chaos: Ntk views on dun nor- malization, checkorboard and boundary artifacts. In Bin Dong, Qianxiao Li, Lei Wang, and Zhi-Qin John Xu (eds.), Proceedings of Mathematical and Scientifie Machine Learning, volume 190 of Proceedings of Machine Learning Research, pp. 257-270. PMLR, 2022.
Benjamin Jourdain, Sylvie Meleard, and Wojbor Woyczynski. Noulinear sdes driven by lévy processes and related pdes. Lalin American journal of probabilily and mathematical slalistics, 4, 08 2007.
Peter Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Springer Berlin, Heidelberg, 1995.
J. Lee, Y. Bahri, R. Novak, S.S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks Gaussian processes. In International Conference on Learning Representations, 2018.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Mufan Li, Mihai Nica, and Dan Roy. The future is nog-gaussian: Resnets and their inf-itte-deppt--nd--iiie -iith limit at initialization. In M. Ranzato, A. Beygekzime, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan
(eds.), Advances in Neural Information Processing Systems, volume 34, pp. 7852 7864. Curran Associates, Inc., 2021.
Mufan Bill Li, Mihai Nica, and Danicl M. Roy. The neural covariance sde: Shaped infinite depth-and networks at initialization. orXiv, 2022.
Fusheng Liu, Haizhao Yang, Soufiane Hayou, and Qianxiao Li. Connecting optimization and gencralization via gradient flow path length. 2022.
Pierre Marion, Adeline Fermanian, Gérard Biau, and Jean-Philippe Vert. Scaling resnets in the large-depth regime. arXiv, 2022.
James Martens, Andy Ballard, Guillaume Desjardins, Grzegorz Swirszez, Valentin Dalibard, Jascha Sohl- Dickstein, and Samuel S. Schoenholz. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv, preprint 2110.01763 2021.
A.G. Matthews, J. Hron, M. Rowland, R.E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
R.M. Neal. Bayesian Learning for Newral Networks, volume 118. Springer Science & Business Media, 1995.
Stefano Peluchetti and Stefano Favaro. Infinitely deep neural networks as diffusion processes. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenly Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1126- 1136. PMLR, 2020.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. 30th Conference on Neural Information Processing Systems, 2016.
S.S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein Deep information propagation. In International Conference Learning Representations, 2017.
Mariia Seleznova and Gitta Kutyniok. Analyzing finite neural networks: Can we trust neural tangent kernel theory? In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova (eds.), Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pp. 868-895 PMLR, 2022.
Soo Sung, Supranee Lisawadi, and Andrei Volodin. Weak laws of large numbers for arrays under a condition of uniform integrability. The Korean Mathematical Society J. Korean Math. Soc, 45:280-300 300, 2008
Peter Tankov and Nizar Touzi. CALCUL STOCHASTIQUE ET FINANCE. 2018. URL http://wwe. cmap. polytechnique.. wttp:/iuu....ttuui/ieee /to/ttoou//PPlly-MMP5....
Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generalization in deep neural networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10462 10472. PMLR, 2020.
G. Yang. Tensor programs iif: Neural matrix laws. arXiv preprint arXi::2009.1068 2020.
G. Yang and S. Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in neural information processing systems, pp. 7103 7114, 2017.
Guodong Zhang, Aleksandar Botev, and James Martens. Deep lcarning without shortcuts: Shaping the kernel with tailored rectifiers. In International Conference on Leurning Representations, 2022.
Bernt Oksendal. Stochastic Differential Equations. 2003.
16

---

Published in Transactions on Machine Learning Research (01/2023)
A Discussion on the infinite-widee limit
The infinite-widd witth limit of neural network architectures has been extensively studied in the literature, and has led to many interesting theoretical and algorithmic innovations. We summarize these results below.
Initialization schemes: the widhhh limit neural networks has been extensively studied in the literature. In particular, for multi-layer perceptrons (MLP), a new initialization scheme that stabilizes forward and backward propagation (in the infinite-width limit) was derived in (Poole al., 2016; Schoen- holz et al., 2017). This initialization scheme is known the Edge of Chaos, and empirical results show that it significantly improves performance. In Yang & Schoenholz (2017); Hayou et al. (2021), the authors derived similar results for the ResNet architecture, and showed that this architecture is placed by-default on the Edge of Chaos for any choice of the variances of the initialization weights.
Gaussian process behaviour: Multiple papers (e.g. Neal (1995); Lee et al. (2018); Yang (2020); Matthews et al. (2018); Hron et al. (2020)) studied the weak limit of neural networks when the width goes to infinity. The results show that a randomly initialized neural network (with Gaussian weights) has a similar behaviour to that of a Gaussian process, for a wide range neural architectures, and under mild conditions on the activation function. In Lee et al. (2018), the authors leveraged this result and introduced the neural network Gaussian process (NNGP), which is Gaussian process model with neural kernel that depends on the architecture and the activation function. Bayesian regression with the NNGP showed that NNGP surprisingly achieves performance close to the one achieved by an SGD-trained finite-width neural network. The large depth limit of this Gaussian process was studied in Hayou et al. (2021), where the authors showed that with proper scaling, the infinite-depth (weak) limit a Gaussian process with a universal kernel?.
Neural Tangent Kernel (NTK): the infinitewwidte limit of the NTK is the so-called NTK regime or Lazy-training regime. This topic has been extensively studied the literature. The optimization and generalization properties (and some other aspocts) of the NTK have been studied in Liu et al. (2022); Arora et al. (2019); Seleznova & Kutyniok (2022); Hayou et al. (2019b). The large depth asymptotics of the NTK have been studied in (Hayon et al,, 2020; 2022; Jacot et al., 2022; Xiao et al., 2020). We refer the reader to Jacot (2022) for comprehensive discussion on the NTK.
B Review of Stochastic Calculus
In this section, we introduce the required mathematical framework and tools to handle stochastic differential equations (SDEs). We suppose that we have a probability space (Ω, F, IP), where Ω is the event space, P is the probability measure, and F is the sigmaa-lgeeba associated with Q. For n 2 1, we denote by B the standard n-dimensional Brownian motion, and Fr natural filtration. Equipped with (Fr))20, we say that the probability space (,, (Fr)roo IP) is a filtered probability space. F. is the collection of events that are measurable up to time t, i.e. can be verified if we have knowledge of the Brownian motion B (and potentially some other independent source such as the initial condition of process X defined by B-driven stochastic differential equation) up to time t. We are now ready to define a special type of stochastic processes known as Itå processes.
B.1 Existence and uniqueness
Definition 1 (Itō diffusion process). A stochastic process (X1)eejo valued in IR" is called an Itò diffusion process if it can be expressed as
X + f Mods + f ogdBs,
where B is n-dimennsoonl Brownion motion and ae E TR / TR are predictable processes salisfying 1 (llpalla + ||a.a] I12)ds < oo almost surely.
A kernel called universal when any continuous function on some compact set can be approximated arbitrarily well with kernel features.
17

---

Published in Transactions on Machine Learning Research (01/2023)
The following result gives conditions under which strong solution of given SDE exists, and is unique. Theorem (Thm 8.3 in Tankov & Touzi (2018)). Let 2 1, and consider the following SDE dX, p(t, X,)dl (1, X,)d Xo 12.
where B is a m-dimensional Brownion process for some m 1, and μ R+ R RR and IR+ x R gaxm are measurable functions satisfying
there erists constant > 0 such that for all 2 0, 2, x' E RR"
(l)(t, p(t, x1)|| (lo(t,,)) (t,, x')l S k kl1l 2'1.
the functions [|v(,, 0>ll and [I)(),))| are (R(R)) with respect to the Lebesgue measure on R++,
Then, for all T 2 O, there erists unique strong solution of the SDE above.
B.2 Itō 's lemma
The following result, known Itô 's lemma, is classic result in stochastic calculus. We state version of this result from Tankov & Touzi (2018). Other versions and extensions exist in the literature (e.g. Ingersoll (1987); Øksendal (2003); Kloeden & Platen (1995)).
Lemma 4 (Itō 's lemma, Thm 6.7 in Tankov & Touzi (2018)). Let X be an Itô diffusion process (Defini- tion 1) of the form
dX, = pidl + o.dBi, No
where v some given distribution. Let g : R R + TR be C1,2 TJ. R") (t.e. C¹ in the first variable and C² in the second variable x). Then, with probability 1, we have that
I Ir ( VoJ] ds, -(.X))= + f[' Vef(s.))) dx, + 1 (o.fle.x.)
where Vaf and Vaf refer to the gradient and the Hessian, respectively. This can also expressed as an SDE
df V.J(I, dx, + (a.S(, X1) + 7 lor V1,
dl.
B.3 Convergence of Euler's scheme to the SDE solution
The following result gives a convergence rate of the Fuler discretization scheme to the solution of the SDE. Theorem Corollary Thm 10.2.2 in Kloeden Platen (1995)). Let 2 and consider the Rd valued ito process X (Definition 1) given by
( ((s,x.)ds p(s..x))ds + S a(s, X.)dB.
X, X,
where B is m- mn-dimensional Brownian motion for some m 2 1, Xo satisfies E|| < co, and μ IR xIRd Rd are o R Rd Rdxm are measurable functions satisfying the following conditions:
1. There erists constant K > O such that for all t R, Rd,
[n((1,)) p((, (|o(1, ol1, a')|| S K x
x'1l-
There erists conslant K' > 0 such that for all E R, Rd
[I((1 [1)(1 K*(1 + ||-1))
18

---

Published in Transactions on Machine Learning Research (01/2023)
3. There crists constant K" >> 0 such that for all t, R E R"
[lu(t u(ss + a(s,2)) S K1 + || -s/1/2
Let. 5 E (0,1) such that 8-1 E N (integer), and consider the times ts = kd for k E (1.....8-1). Consider the Euler scheme given by
m 2 (1x,
Yeu1 1% (1k, 16
Y Yo X².
where Yi, ,1, oh denote the coordinates of these vectors for i E [d], E [m], and AB ~ NO0, Then, we have that
E sup |X, Y 116-11 = tE(o.]]
We can extend the result of Theorem 5 to the case of locally Lipschitz drift and volatility functions and a. For this purpose, let us first define local convergence.
Definition 2. Let (x2 be sequence of processes and X be stochastic process. For 0, define the sollowing slopping times
(x
We say that X converges locally to X is for any 0, XL EnT converge to XeNT. This definition is general for any type of convergence, we will specify clearly the type of convergence when we use this notion of local convergence.
Lemma 5 (Locally-Lipschitz coefficients). Consider the same setting of Theorem 5 with the following con- ditions instead
For any > O, there crists a constant K such that for all t R. a, Rd with || S
|| σ) - p(t, + (lo(t,)) - ott, S K | x'|l.
For any 0, there erists a constant K' such that for all t R, E Rd satisfying ||>||| S
[u((t, + (1, S K'1 + ||=|]).
3. For any 0, there crists constant K" 0 such that for all t,s E R. Rd satisfying ||~||| sr,
[l((t -aae))-((ss000) x(s,,))l++ (|o(t,,
(ss 2)) < K" || 3/1/2
Then, for any > 0, we have that
| Y(ttn))6-111==000) 0(0), (E[0.1]
sup
where inf(t 0: [118-11) -ll r), and inf(t 0: |X > r).
We omit the proof here it consists of the techniques used in Kloeden & Platen (1995), with the only difference consisting of considering the stopped process X By stopping the process, force the process to stay in a region where the cocfficients are Lipschitz.
B.4 Convergence of Particles the solution of Mckean- -lasov process
The next result gives sufficient conditions for the system of particles converge to its mean-field limit, known the Mckean-Vlasee process.
19

---

Published in Transactions on Machine Learning Research (01/2023)
Theorem ( Mckean- Vlasov process, Corollary of Thm 3 in Jourdain et al. (2007)). et d 2 and consider the Rd valued ilo process X (Definition. 1) given by
=x== V dBs,
Xo has iid components,
where B is a d-dimensional Brownian molion, us def à Ei= 6xx; the empirical distribulion of the coordi- nates of Xt, and is real-valued given by (I, = f <(a, (drr)) for all x E R2 and distribution v. Assume that the function Ç is Lipschitz continuous. Then, we have that for all T E IR,,
sup E
sup xx - x:p) = O((n)) 1ST
iE(n)
where X the solution of the following Mckean- Vlasov equation
dx; (X) X = =x=
where ot is the distribution of 8
B.5 Other results from probability and stochastic calculus
The next trivial lemma has been opportunely used in Li al. (2021) to derive the limiting distribution of the network output (multi-layer perceptron) in the joint infinite width-depth limit. This simple result will also prove useful in our case of the Lemma 6. Let W Rax" be metriz of standard Gaussian random variables Wy ~NO0, 1). Let v E R" be random vector independent from W and salisfies ||u||2 =1 Then, We ~ N T).
limit.
Proof. The proof follows simple characteristic function argument. Indeed, by conditioning on e, we observe that Wo ~ N0,, I). Let u I2,, we have that
Ew.ole (ww.WW) = Egy Ew [ (e [o]]
=By 1412
This concludes the proof as the latter is the characteristic function of a random Gaussian vector with Identity covariance matrix.
The next thcorem shows when a stochastic process (ito)
Theorem (Variation of Thm 8.4.3 in Øksendal (2003)). Let (Xt)se(o.r) and (Yi) be two stochastic processes given by
axx = b(X))dt + (X.)dBB X, = r dY; = b,dt ri =oo
where o :R + RLink (b:)220 and (ve)22 are real valued adapted slochastic processes, and v is adapled to the filtration of the Brownian motion (Bi)20,, (Bi)c20 is an k-dimensional Brownian motion and (B.) is a 1-dimensional Brownian motion. Assume that B[b. b(Y;) where N₁ = =o(())ssss is the o-Algebra generated by (Y. s < 1), and v² = (Y.) almost surely (in lerms of dt x dP measure where di is the natural Borel measure on [0, I' and dP is the probability measure associated with the probability space). Then, X, and Y, have the same distribution for all [O.
Proof. The proof of this theorem is the same that of Thm 8.4.3 in Øksendal (2003) with small differences. Indeed, our result slightly different from that of Oksendal (2003) in the sense that here we consider Brownian motions with different dimensions, while in their theorem, the author considers the where the
20

---

Published in Transactions on Machine Learning Research (01/2023)
Brownian motions involved in (Xi) and (Y)) of the same dimension. However, both results make use of the so-called Martingale problem, which characterizes the weak uniqueness and hence the distribution of Tto processes". The generator of X₁ is given for f E (R) by
as dr + (z) 05 T85
(())))= b(x)
Ox.²
Now define the process H(f) for (2R by
0s
(Yi) + 2 S
H()(1) bi Ox
Ox (Vi).
Let N₁ o((G))sst) be the o-Algebra generated by (Y, 1 S St.. Using Itō lemma,, we have that for 8 < t,
[ = S(Y) + E L." (())())))))))
f(Y) + E J. ETHLNNNANGNNN
f(() + E J. (U)
where we have used the fact that [[,, b(Y.). Now define the process M by
Mr f(Y)) - S sU)(x.)dr.
For > 1, we have that
B[M,JM]] = f(Y)) + J. (()()))+( - E L. GCN0X.)AMNN) (by Itō lemma),
=((Y) E I/ G(F)()))+ Me
Hence, M, is a martingale (w.r.t to M,). We conclude that Y, has the same law as X₂ by the uniqueness of the solution of the martingale problem (see 8.3.6 in Ōksendal (2003)).
The next result simple corollary of the existence and uniqueness of the strong solution of an SDE under the Lipschitz conditions on the drift and the volatility. It basically shows that zero-drift process collapses (becomes constant) once the volatility is zero.
Lemma 7. Let g R be Lipschitz function. Let Z be the solution of the stochastic differential equation
dZ, = d Zo E R
If gZ))) =0, 0, then Z, Zo almost surely.
Proof.
This follows for the uniqueness of the strong solution of an SDE(Theorem 4).
We omit the details the Martingale problem here. invite the curious reader check Chapter Oksendal (2003) further details.
21

---

Published in Transactions on Machine Learning Research (01/2023)
B.6 Proof of Proposition 1
We are now ready to prove the following result.
Proposition 1. Assume that the activation function & is Lipschitz on RR. Then, in the limit - 0o, the process X = ((L), [0, II, converges in distribulion to the solution of the following SDE
dX, 1 Tri -lo((x)lddB, Xu Win
(7)
where (Be)t20 is a Brownian motion (Wiener process). Moreover, we have that for any t E [0, Lipschitz function R_ + I,
(VL(( = + O(L
where the constant
does not depend on L.
Moreover, if the activation function & is only locally Lipschitz, then X!¹ converyes locally Xt. More precisely, for any fized r > 0, we consider the stopping times
Th = in 2 0: r), in 2 0: |X r
then the stopped process X LxTL converges in distribution to the stopped solution XINT of the above SDE.
Proof. The proof is based on Theorem in the appendix. It remains to express Eq. (1) in the required form and make sure all the conditions are satisfied for the result to hold. Using Lemma 6, we can write Eq. (1) as
Yi = Yı- + VL (V-1)
where (yy) def J le(v)ll for all IR" and &f are iid random Gaussian vectors with distribution (N0,,) Z). This is equal in distribution to the Euler scheme of SDE Eq. (8). Since o trivially inherits the Lipschitz local Lipschitz propertics of o, we conclude for the convergence using Theorem 5 and Lemma 5.
Now let V be K-Lipschitz for some constant K > 0. We have that EY - (X)) & KE sup [YI - x, rE[o.1]
where Y is the Euler scheme as in Theorem 5, and where we have used the fact that Y(u and y [L have the same distribution.
The result of Proposition 1 can be generalized to the case with multiple inputs with minimal changes in the proof. We summarize this result in the next proposition.
Proposition 5. Let 22,......, E Rdd be non-zero inpuls, and denote by Y((xs) the pre-activation vector in layer for the input z,, Consider the vector Y = (Yi(x1)" Yi(x2) ,..., T)T E g consisting of the concatenation of the pre-aclivations vectors for all impuls Te. Assume that the activation function is Lipschitz on R" Then, in the limit L + oc, the process X X Y ELL t [0, 1}, converges in distribution to the solution of the following SDE
1 -E(x Vn
dx.
x ((WmI1)) ..... W1)T))
(8)
where (Bi)20 is kn-dimensional Brownian motion (Wiener process), independent from W Win, and E E(X)) is the covariance matria given by
01,1 Jn
01
Tn In
0221 In
0:2,2 In
-(X,)=
0 Jn
22

---

Published in Transactions on Machine Learning Research (01/2023)
where a43 = (6(x; ") ((xt))) (), with (x: (X. ................................ Xt T def X Moreover, if the activation function & is only locally Lipschitz, then X. converges locally to X,. More precisely, for any fized r > 0, consider the stopping times -f = 2 0 : | 2 r}, and T inf 2 0: | 2 r), then the stopped process xl.k converges in distribution to the stopped solution X of the above SDE. thrt
Proof. The proof is similar to that of Proposition 1. The only difference lies the definition of the Gaussian vector G/" In this case, we have for all z;
1 (V--((()
(a()))) = (zi) +
vL vn
where (M-()--( def (=Wi)(Y--(zi)). Concatenating these identities yield
1 St
===YE +
VL vn
where - is the concatenation of the vector // (x+1 for i = 1,...,.. k. It is straightforward that the covariance matrix of the Gaussian vector // is given by the matrix E above (with X replaced by Y). We conclude using Theorem 5.
C Some technical results for the proofs
C.1 Approximation of X
In the next lemma, we provide approximate stochastic process X to X, that differs from X by the volatility term. The upper-bound on the L2 norm of the difference between X and X will prove useful in the proofs of other results. The proof of this lemma requires the use of Gronwall's lemma, a tool that is often used in stochastic calculus.
Lemma 8. Let x Id such that + 0, m 2 be an integer, and consider the two stochastic processes X and X given by
dxp"m Ja (x."))
t 10,0, Xx Win
dx, 1 (oll E (0), Xo Win
where dm((2) = Sod h(mu)du where h is the Sigmoid function given by h(u) (1 e- is the ReLU activation function, and (Br)e20 is n-dimensional Brownian motion. We have the following
2nd m²
Vt O, E XX.T"
xel1 <
21
Proof. Let t 2 0. We have that
1/'' (Ilom(()))) [[(( (lxx))))
E EI 2 -
n
Using Itô isometry and the fact that (Ilem (XT")] )| (((((x))1) < 1@ (X3") o(X.)ll, we obtain
E| x,1² <
1l - Φ6 (xxx)) (,)) )| ds
E Ell@m (X.T) - ((X.)))) 2 ds d 1 EllØ(x")) -(x,)) ds
< 2
2nl
+2 ( E 1xTmm xx,1 ds.
m²
23

---

Published in Transactions on Machine Learning Research (01/2023)
where we have used Lemma 9 and the fact that ReLU is I-Lipschitz. We concldue using Gronwall's lemma.
C.2
Approximation of ø
The next lemma provides simple upper-bound the distance between the ReLU activation ø and an approximate function 6m that converges to ø in the limit of large m.
Lemma 9. Consider the function @m (2) = Jo h(mu)du where 2 I where m 2 1. We have that
1 sup (2) - (=)=1(=))1 S m sER
Proof. Let m and IR Assume that > 0. We have that
(2m - -0(2]
du
+ mu
<
e mu du
1
(1
E
mz
m
m
For the case where S O, the proof is the same. We have that
ma
(mm(2) - 0(2)
+ ema du
e mu du
(1 m=²) <
which concludes the proof.
C.3
Other lemmas
The next lemma shows that the logarithmic growth factor log
| (xT')
[X converges to log
| To(Xa)l when
m goes to infinity, where the convergence holds in L₁- The key ingredient in the use of uniform integrability coupled with convergence in probability, which is sufficient to conclude on the Lı convergence, This result will help us conclude the proof of Theorem
Lemma 10. Let x Rd such that x + O, m 2 1 be an integer, and consider the two stochastic processes x and X given by
dx.:" = (d.") t (0,,,0,, X Win dX, = (d. tE E (0, Xo = Winz,
where (=) = Je h(mu)dd where h is the Sigmoid function given by h(u)) (1+e-1)-1, is the ReLU activation function, and (Bi)zo is an n-dimensional Brownian motion. Then, conditionally the fact that [[)(()))| > 0, have that
1.. 2 0, log [l0m TommX (Xm²)]
1. log
[x(X))]
[xx)))]]
24

---

Published in Transactions on Machine Learning Research (01/2023)
Proof. Let > O. From Lemma 8, we know that x converges in L to X. Using Lemma 9 and the fact that ReLU is 1-Lipschitz, we obtain
2n m²
E| 0m (XT") - (xx))11
2 X Xell
which implics that om 6m(X7' (X converges L to o(x,). In particular, the convergence holds in probability. Using this fact with the Continuous mapping theorem, we oblain that
Vt 2 0, log (ll@m (X?")I) _ log (l)((X))I))
(9)
Let us show the following,
v2 2 0, log
11 (Xxx)) (xxm)
[X²m(
P log
[|o(Xo)||
[lx(X,)11
Let > 0 and t >> O. We have
(X,")]] (Xx")]]
[xxx)) [1o(X))]
P log
logg
26) I P (llog [x?" log (11 c/2)
P (log | (Xx") log (1lo(X)))l)) 2 e/2).
where the first term converges to zero by Eq. (9), and the second converges zero by Lemma 9, Hence, the convergence in probability holds.
To conclude, it suffices to show that the sequence of random variables (ym log formly integrable.
[ is uni- [x,[)]]]]]]]]
m21
Let K > 0. From the proof of Lemma 2, with Ç = Øm, we have that
Y
Vn I p(X"")dss ds +
1
2n Jo Σ or (x('
1
/ T
where (X
ld (x (m)))) and u(X²)
E
(o' (ommmi)) + d', (XMr))22)
1m (X
I (X Jomm (o
Therefore,
(x," "1
1
(f ((XT)dd)
m
E y
E
a (X )
4n
(10)
2
E 4n²
(
Jn
where we have used the Itô isometry and Cauchy-Schwartz inequality. Using the conditions on Φn. it is straightforward that term 2 E Σm (xx')))) ? is uniformly bounded. It remains to bound the first term. Similarly to the proof of Theorem 1, we condition on the regions of |Xm* and obtain that the terms Ex(X?")² are uniformly bounded over m (we omit the proof here as it is just a repetition the techniques used in the proof of Thoorem 1). Therefore, we have that supmz1 Y oc, which implics uniform integrability. This concludes the proof.
Lemma 11. Let x E Rd such that x + 0, m 2 be an integer, and consider the stochastic processes X given by
1 dX, = _ ld t t 20) Xo = WII,,
25

---

Published in Transactions on Machine Learning Research (01/2023)
where & is the ReLU activation function, and (Bi) is an n-dimensional Brownian motion independent from No. Then, condilionally the fact that | Xo)]l > O, we have that for all E [0, i E [n]
P(X S 8) O8-)((0);
where the bound holds uniformly over E [0,1].
Proof. We have that
Vn / lo(X))lld
= X
Since (xx,) > for all u 0 almost surely, and by the independence and Xo, we can casily see that X has no Dirac mass and is the sum of two continuous random variables (not independent) X and Va So (lx(Xx) IdB; that have bounded density functions, and thus X3 has bounded density function h,,
Hence, writing P(|X: & 6) = S_s (t)dtt (d)) concludes the proof. The bound can be taken uniformly over E [0, 1] by taking supse(0,
D The Ornstein-Uhlenbeck (OU) process
The OU process is the (unique) strong solution to the following diffusion
dX, = a(b Xt)dt + adBr,
(11)
where a, b, E I constants, and B is dimensional Brownian motion. Tn financial mathematics, the OU process is used model of short-term interest rate under the name of the Vasicek model. The OU process has closod-form expression and its marginal distribution is Gaussian. The next lemma gives full characterization of the marginal distributions of an OU process. Lemma 12. Eq. (11) admits the following solution
X, = ot + b(1 - --a)) + L' co((-s)
As result, we have the following
X, is Gaussian.
B[X,] -ot b(1 e all). 2² 2n
Cov(X,, 2 (e-alt-ol e- -a((tss))
Proof. Consider the process Z₁ att X2, using Itō lemma, we have that
dZ. aZ,dl + dX, abe dt + oe dWe
Integrating between and yields
Z, Zn + bleat - 1) + o J' ewedW..
We conclude by multiplying both sides with at
26

---

Published in Transactions on Machine Learning Research (01/2023)
The result for [| is straightforward since [o e e dW. = 0 by the properties of Itō integral and the Brownian motion. For the covariance, without loss of generality assume that / 0. We have that
a(t-a) (-l / --als-- dW,
Cov(X,, X,) = O²E
a dWn {o e -alo-woWW,
2²F
d²
20l + -a( -w) du
( aa(-s))) c -aa(tts))
2a
which completes the proof.
We would like to find sufficient. conditions the activation function & and a function g such that the process g(Xe) (Eq. (8)) follows the OU dynamics. For this purpose, proceed by reverse-engineering the problem; Using Itō is lemma (Eq. (4)), this is satisfied when there exist constants a, b, a such that
(u()) = 1 oly) (v) = a(b s(v))
2n
o(y) (()) (y)
This implies that g"(y) gol 200 2b ((v)). Letting G I be the primitive function of g, we obtain that
G satisfies a differential equation of the form
G" ay + (G(y) + +
where a, S.5 E R are constants.
Let consider the case where a < and 8 + 0, i.. लमं 3G(y). Equivalently, we solve the differential equation Gy) & where E R. Multiplying both sides by G' and integrating we obtain (y) Blg[(G])) + y. A sufficient condition for this to hold is to have G and G satisfies
Gvv 5
Vlo(GG)+
for some constants Ç, Integrating the left-hand side yields
Vlg(GG(u) du G(v)
(u)
Tog(u) +7 du Erfi( Erriir/llll/llll + +
where Erfi the imaginary error function given by
Erfi(z) 2 f E dt.
VT
To alleviate the notation, we denote h == Erfi in the rest of this section. From the above, G should have the form
G(y) = exp (c + (ay + +)))
where 3,C are all constants, and h is the inverse function of the imaginary error function. We conclude that the activation function & should have the form
20 a exp(-s+ h (aa + 8)3).
9 Althhough the name might he misleading, imaginary error function real when the input real.
27

---

Published in Transactions on Machine Learning Research (01/2023)
In this case, the coefficients and are given by
b= o²
a² exp( (xp)-
Letting g = G", the process g( (x) has the following dynamics
dg(X,) -ug( )dl + adB,,
Hence g(x)) an OU process, and we can conclude that the network output in the infiniteddepth limit X1 satisfies
o² 9(X1) - (((Xo)e o² (1 (1-
We can then infer the distribution of X, by simple change of variable. Note that this distribution is non-trivial, and unlike the infinite-width limit of the same ResNet (Hayou et al. (2021)) where the distribution is Gaussian, here the distribution of the pre-activations is directly impacted by the choice of the activation function 6.
However, with this particular choice of the activation function Φ, the existence of the process can only be proven in the local sense, because & only locally Lipschitz. Let us first show this the next lemma. We will see how we can mitigate this issue later.
Lemma 13. Let Φ IR + R defined by
Qky) = exp((= 3)²),
where B E Th are lwo constants.
We have that & is locally Lipschitz, meaning that for any compact set K C I,, there exists Ck such that Vx e KKlo(x') <(x) S Ckl='= xx.
Proof. It suffices to show that the derivative of Φ is locally bounded to conclude. We have that
(v) 2a (aay -) (ay 3) exp(h (ay 8)2)
= = ( 3).
Since -1 is continuous on R, then o' is bounded on any compact set of R which concludes the proof.
Now we can rigorously prove the following result.
Proposition 6. Let æ R such that x + 0. Consider the following activation function &
2a a² xxp(-S+h-10 ( + 3))),
(ly)
where a, B and a. <> O are constants. Let g be the function defined by
glv) = (xt=)))))x)(
Consider the stochastic process X, defined by
dx, = lo(X.)ldB,. Xo Win..
Then, we have that for all E [0,
o² o(Xi) ~ N (o(Xo)e=
(1 -222)
2a
x² where -5 exp(-2()
28

---

Published in Transactions on Machine Learning Research (01/2023)
Proof. For N > 0, consider the stopping time TN defined by
TN inf 20: 0: |x, N).
Using the continuity of paths of X, it is straightforward that lim TN almost surely. Let N >> 0 be large enough. The SDE satisfied by the process X has unique strong solution for t E [0, Tx) since the activation function & is Lipschitz on the interval (-N,N). By applying Itō lemma for & E Tw), we have that
= -ag(;) (x) + adB,
(from previous results). Using the fact that lim TN almost surely, and taking N large enough, we obtain that for all t (0, 1], we have that
dg(X,) -ag( + odBi,
conclude using Lemma 12.
E
The Geometric Brownian Motion (GBM)
The GBM dynamics refers to stochastic differential equations of the form
dX, = aX,dt X.dBB,
(12)
where are constants and B is a one dimensional Brownian motion. This SDE played crucial role in financial mathematics and is often used as model of stock prices. It admits a closed-form solution given in the next lemma.
Lemma 14. Eq. (12) admits the following solution
X, = =xx exp (a=2o²))+
The distribution of X, is known log- Gaussaas distribution. Moreover, the solution is unique.
Proof. The existence and uniqueness of the solution follows from Theorem 4. Indeed, it suffices to have the drift and the volatility both Lipschitz to obtain the result. This is satisfied in the case of GBM. Now consider the process Z. = IX,( Using Itô lemma 10 it is casy to verify that
dz, = a 1 00² dl adB
we conclude by integrating both sides.
Now let us find sufficient conditions under which the infinite-depth network represented by the process has GBM behaviour. In order for this hold, it suffices have
(u(v) d(v))) ag(v) (ol)) (u)))) =g(().
This implics 2" x 1. equivalently d" o x gh which in turn yields log(lg'l) = log(1gl) + 3, and therefore ggll oc lg(S. Assuming that g' g > O, we can easily verify that functions of the form g(y) a(y + 5) where a, B.y > 0 satisfy the requirements. Hence, the activation function should satisfy Φ(y) = =o oo (v 3), i.c.
Notiie that here, should positive in order consider (x). This easy show and the proof similar that. of Lemma
29

---

Published in Transactions on Machine Learning Research (01/2023)
the activation should be linear. In this case, have 2 (r 1) and the process g(X,) has the following GBM dynamics
dg(Xi) = ag(X))dt + og(X))dBb.
From Lemma 14, we conclude that
g(X))~ ~ ((Xo) exp ((p0(()00) oB
Observe that in the special case of 1, = 0,a L, we have g(v) y and 0. In this case, we obtain Yi exp (-3o²t + BB).
We summarize the previous results in following proposition. Proposition 7. Let æ R such that x + Consider the following activation function
s(y) ary + 3,
where a > 3 E are constants. Let o > 0 and define the function by
g (ay + 8)7.
where = -1 Consider the stochastic process X, defined by
dX, = (d Xo WinT.
Then, the process g(X:) satisfies the following GBM dynamics
dg(X:) ag(X;) + ogg
where a == = 1). As result, we have that for all t [0.
g(Xi) ~ g(xo) exp ((a-)0) t oB.)
F ReLU in the case d =
Consider the process X given by the SDE
dx, = dY) EE [0, >
where d(2) m) for E R is the ReLU activation function. Note that assume Xo in this case. We will deal with the general case later in this section.
It is straightforward that if X, S 0 for some s E 1), then for all 2 8, s,X, = X,, This is because dx, = 0 x dB, whenever X, I O. rigorous justification is provided in Lemma 1. Hence, the event (X, < 0} constitutes stopping event where the process becomes constant. We also say that is an absorbent point of the process X. classic tool in stochastic caleulus deal with such situations is the notion of stopping time which is random variable that depend on the trajectory of X (or equivalently the natural filtration F, associated with the Brownian motion B). Consider the following stopping time
= e [0, 1]. s.t. X, 03.
(13)
Observe that we have for all t [0,
dx, X,d.
which implics that Y, a Geometric Brownian motion in the interval [0, r]. Hence, > (a.s.), the network output has also a log-normal distribution in the infinite-depth limit. In the next lemma, we show that with probability 1 which confirms the above.
30

---

Published in Transactions on Machine Learning Research (01/2023)
Lemma 15. Let be the stopping time defined by Eq. (13). We have that P(T 00) = 1.
Proof. By contimuity of the Brownian path and the ReLU function 6, the paths of the process X are also continuous"! 11, we have that > 0 almost surely. From the observation above, taking the limit t + T- and using the continuity, we obtain
X, X6 exp (-ZT+ (T++Br) Br).
For some w E fr < oo}, we have that (w) 0 (by contimuity). Hence - (w + X-(w) -0. This happens with probability zero, which means that the event {T < oo) has probability zero. This concludes the proof.
Hence, with the ReLU activation function, given Xo O, the network output is distributed as
X1 Xxexp (-2 + Bi)
Now let us go back to the original setup for Xo- Recall that Xo = Win for some x + and Win N0 1). By conditioning on Xg and observing that is an absorbent point of the process X, we obtain that
X1 1 (N exp (-2 B1) + (xooo))
We summarize these results in the next proposition.
Proposition 8. Let x E R such that x + O, and let & be the ReLU activation function given by d(z) max(2,0) 0) for all 2 E R. Consider the stochastic process X, defined by
=x =(X.dd: Xo = Win
Then, the process X is a miature of Geometric Brownian motion and a constant process. More precisely, we have for all E []
X, ~ 1 [xx200 No exp exp (-21+ (X050)
Hence, conditionally on Xo 0, the process X is a Geometric Bronwian motion.
G Proof of Lemma 2 and Lemma 3
Lemma 2. Let x E Rdd such that + 0, and consider the stochastic process X given by the following SDE
dX, - [1xx))ldBB I (0, Xo = Win
where (bz) : R + R is Lipschilz, injective, (²(R and satisfies ((0) = O, and ø' and 6²6 are bounded on R, and (Bi)e20 is an n-dimensional Brownian motion independent from Wi ~ No, d ' I). Let T be the stopping time given by
min (({(()) =0.
Then, we have that
P(T= 00) 1
11This is classic result stochastic calculus. More rigorously, X chosen have continuous paths with probability
31

---

Published in Transactions on Machine Learning Research (01/2023)
Proof. It straightforward that with probability we have ||o 0, which implies that with probability 1, J > O. Let & <T. Using Itô 's lemma with the function g(z) log(II((x)]13) we obtain
dg(X,) dX, 2n [((( (XX,))
dx,
Therefore,
((xi) 9((x) = vn / ((X,)ds+ +
1 Σ (X,)d,
2n /
i=1
where (X,) (x()xx)( and x(X,)= 2 E. (x)))))(X)) + (x:)²)
(X. (X.)X. IIl" and refers
[o(X,)]
to the Hadamard product of vectors, i.e. coordinate-wise product.
For some w E {T < oo), using the path continuity of the process X and the continuity g, we have that lim)->(()- g(X (WX+(w)(W) -0. Therefore, we should also have
T(w)
1 (w) 2n 1. Σ
Vn u(X,(w))ds +
[o((X((w)))BB()) -CC
1=1
Hence, we have that
1 ( m(X.)ds +
1 n
(0)< S P
Vn Jo
2n 1 Σ
(X,)ddB -00)
Vn (' m(Xx)ds J
1
lim P A-000
oi (xx))dB3s S s-1)
(' p(X,)ds+ + 2n l o(X.)dBB S S-A)
lim P A
V
where B is a one-dimensional Brownian motion, and where we use Theorem 7, and oXx)) E (X,)3321//2 (o(XX))o(X.)) Using the conditions on o, there exists constant K such
[xXx))|
10' & K, ol & K. With this we obtain for all Z E R
1 (Z) ((Z) [a(2)|
< K,
[Ø(Z)]
and
1
Σ (Z" (Z'))(Z)) ) +Z((2)))) ( o øi 22)1)
1 nK (nn+1) K².
[m(Z)
$ 2
2
|x(2)
Hence, the random variable J 50 1 ((x))ds + //2 fo (X,)dB, is finite with probability 1. We conclude that
P o0) 1.
Lemma 3. Consider the stochastic process (8) given by the SDE
dx, = Vn lXXx)) t te (0,00, Xo = Win
where o is the ReLU activation function, and (Bi)z20 is an n-dimensional Brownian motion. Let T be the stopping time given by
min fln(l22 0: (|x((xx))) 0) min 0: Vi [n], x x 0).
32

---

Published in Transactions on Machine Learning Research (01/2023)
Then, we have that As a result, we have that
P = 001=ooll > 0) = 1.
P(T 2o) -1
Proof. Let to > O. Using Lemmn 7, we know that if for some t1, (,, ) = 0, then for all t 2 t1, we have that X, = Xe, and [Xx())) = 0. Hence, we have that
IP ( S to to 0)
=
0).
Let m 2 1 and consider the function dm(2) Ser hmm uudu and h(() (1+ is the Sigmoid function It is straightforward that on satisfies the conditions of Lemma 2. Let X be the solution of the following SDE (the solution exists and is unique since om is trivially Lipschitz)
dxpT va mm )|d. I E (0, Xo = WinT.
We know from Lemma 8 that X converges in I to X, (uniformly over E [0, T) for any > 0). particular, this implies convergence in distribution. Moreover, observe that for all
E pm - -xx))) 2n
+ XT - Xell..
m²
where we used triangular inequality and the upperbound from Lemma 9. Thus, we have that om (X:") converges T,2 (and distribution) to ((x)).
Let & [1k + 1), 1/k) for k 1, and define on (1, For m 1, using Lemma 9, have that
20
TP (Ilo((X Ill = n (Xx)l > >0 S
(X 1/m n [1x((x))ll e 6z).
Given & 2 0, we have that for m > n ( 1),
P (X s 1/m n [xx()) 6x) S P (l (XT S 1/m log(m)/m n ((x(X))ll E 5k) (I@m (Xx)) - ((x.))l > log(m)/m n [1o(X))) E 6x)
(14)
Let us deal with the first term. Using Lemma 9, we have that
P (Ilmm (x,mm((!)) || S 1/m + log(m)/m n [[x(X)) E 5x)
< [10m (x²x || < 1/m log(m)/m I (X,') 2 1/(k + 1) 1/ [[ 6:)
IP
1m (X%) ||
< IP
log
< log(mm/1++og/////n log(m)) + log((k + - 112 [|o(Xo)]l dx
dm (X
[| (X ||l
converges in L' and thus it is bounded in L norm (over m). Therefore, simple application of Markov's inequality yields that the probability above goes to O when goes to co.
From Lemma 10, know that the random variable log | (Xgx)
The second term in Eq. (14) also converges to 0 using the L convergence of @m (X()) to o(Xi,) coupled with a simple application of Markov's inequality. We therefore obtain that for all & 2 O,
12tote that dm has closed-form formula given by dmn(2) m- m-- (log(1 m log (2)): which be seen shifted and scaled version of the Softplus function. However, we do not need the closed-form formula analysis.
33

---

Published in Transactions on Machine Learning Research (01/2023)
mimm- P (1 (Xt,)) 1/m n [ox(X))] E 6k) = 0. Using the Dominated convergence theorem, ob- tain that for all to,
PP (1 = 0 n llo(X))]l > 0
which implies that E [1 (Xx(()))) > 0] for all &> 0. Another application of the Dominated conver- gence theorem yields the result. The second part is straightforward by observing that P (Il((X))] 0) 2n
H Proof of Theorem 1
Theorem 1.
We have that for all E [0, 1},
Vn ( B.
11 (
[ods)
||x(x)) - [1o(XX)) exp
almost surely,
where μs (X.)) - 1, and (B)r20 is one-dimensional Brownian motion. As resull, we have that
for all s
[[x((.)]l [11o(x0)] =0
-(--
(1-2
1
log
(t - 5),
[[o(X.)]
4 11
Moreover, for n 2 2, we have
|[x(x))1 |[x(X.)l| ||x((6o]] >0/ s (n-1/2 +
11/2) (1 8),
Var
log
where Te, = = 1' (EEd(X))) [1-2 -1 + n-
1--2- (x( du.
Proof. Let E 00. Let us firs consider the case where ||o(X))l = For all we have [lo(X))) = and the result trivial.
We consider the case where [[xo)l 0 (happens with probability 1- 2",,, and all the expectations in this proof are conditionally on this event. Consider the function g IR" + IR given by
( = log((l>())1)) = log(Il((z)13))
Ideally, we would like to use Itô 's lemma and Lemma 3, which ensures that [|ø(X.)l remains positive on [0, 1], and obtain for all E [0
dg) dist 1 di, + _uddt vn
n
where dt is some well defined quantity. This would let us conclude. However, Itō lemma requires that the function be (R)), which violated by our choice of g. To mitigate this issue, we consider a sequence of function (gm)m2m that approximates the function g when goes to infinily. For 2 let Jm be defined by
1
Im (x)
2 (2)
where @m (1) = - [mm uddu and 1((1) (1 + --1 is the Sigmoid function. We have that
dgm dz
(x)
A(ma, John (2z)
dim [2]]]
0 pm
(x)
mh(man [11
h(mm, Do.., (a.) (R.) 2 Afmar; Po (2 1o [x]]
1 [2]
Let X be the solution of the following SDE
d. = 1
f11m (XP))
t (0, Xo = Win,,
34

---

Published in Transactions on Machine Learning Research (01/2023)
Using Itō 's lemma, we have that
1 (X,")
1 " vn
M1 ds
Σ 0 ==1
where
1 n
[lh(mx Φm (I!"))12
Σ Ps 2
(mh(mxX, (mm)) [1 (1mx)))))) (X."...) h(mx?m.1)))
10m (X
By Lemma 10, we know that 9m (X!") Im (X0') converges in L1 g(x.) - s(Xo). Let us now compute the limit of gm (X.")) Im Im(X (Xi")) from the equation above to conclude.
and am 1m ")og""")))oooa 10 !)
1m (X
More preciscly, let us show that for all 8 (0.1)
lim I9 (Xx") 9m 9m (X0")) (Z. - Zo)1 = 0, m=
where Z is the process given by
Vn Σ %/¹ dB + J
{ ugds,
Z, = 9(xo +
o(x)) where μa 2 (x:) 1, and 0 = 1|o(X,)11*
Let E (0, 1]. Using triangular inequality, ltō isometry, and Cauchy-Schware inequality, we have that
1
[ E l Mal
E 1m (X:"") 9m( - (Ze %o)l
ds
E
0
ddB
0
Vn
0
i=1
(15)
1/2
1
<
E 11TTT Mal
ds
Vn
E
8
))
0
0
We first deal with the the term E 1. uol ds. Let us show that this term converges to 0. Let show the following.
Vs > 0, limm Elp Mol = O.
Let E [0, 1]. We have that
11th
J -m
where J.mh = m with the term T Observe that G < almost surely. We have that
(X...)) + hmmX.mi.3 and GTD
[l(m Jonn 110m (X")
Do (X"m))
Let us start
[[11 h( (mX.))2) om (X
E
GD
E
(Xm))) (Xm)
[I(1 MmmXm om (Xm))12
E
Il )1:
(min, (x (0mgmmmmm)
[1 h(mx:) 2)1/2)2)121111 6m (Xm) )|
+ E
@m (1m)
{min: X (llogm))mmmmm/mm
35

---

Published in Transactions on Machine Learning Research (01/2023)
When min; |z*| 2 log(m)/ /m, we have that for all i e [n]]
(1 h(mx²))) s 2(1 h(mar)) s exp(-m x log(m)/m) 2m 2m-1
Therefore,
(1 h om (x
E
I (Xg"
[mine (mmg(m//m)
I 2m 22m
For the remaining term, using the fact that 1 - 122 I 1, we have that
[1 h(mX.")))14 2m (12
E
$ P min X llog(m)mm)
1m (xm")
{min, [x," "Ioo"""/llommm////
P (1xmi < <log(m)/m)
Now using Lemma 8, we have that
On
P(II X.Il 2 log(m)/m) S log(m))²
for some constant C₂ that depends on n. Therefore, for all i, we have
P(| < log(m)/m) s P(XX)) 2 llg(mm)mm) + P[ x 2 log(m)/m)
Cn
s |[ 2log(m)/m) + log (m))
Recall that X; = X + An So (1(((x)))ddBB By Lemma 11, we know that
P([X) < 2lg(mm)mm) Ollg(m))/m)
We conclude that limm-soo B|1 - GmD = 1.
Now let us show that for all i, limm-soo F. (x(x)) Let [m] and Ama mh(mx!..)) (omm We have that
E E|AT/
E [A 12000 + B ( logg(mm)
m 2 log(m)/m m x PP < 2 (m)og//m//m// /m) + El (Xm..)
log(m) (P([ < 3 llg(mmmmm)
P( X 2 (0g(m)mm)) + m "' (X
2C,
2log(m) | 3og(m)) /m)
+ E'
log(m)
where we have used Lemma and Markov's inequality. Using Lemma 9 and the fact that the absolule value function is Lipschitz, we know that limm-+o0 E Jmm (Xm=))) = E x(x)) < oc. Therefore, the third term vanishes in the limit m + oc. The second term 2C,/ log(m) also vanishes. The first also vanished using Lemma 11. Therefore, oom El =
36

---

Published in Transactions on Machine Learning Research (01/2023)
Let us now deal with the last term in J.". We have that
E hhmmX - 6 o'())]
= E o (Xx)) (1xx""12lo (2l0g(m)/m))
-
E hmmx o' (x:) (1x;7 <log(mm/mm
< 3m 1 + 2P( log(m)) mm)
On
$ 3m + 2P((X 2llg(mmmm /m) + log(m)²
Using Lemma 11, we obtain that lim [m - xxx))) = 0. Hence, we obtain that limm-toc J ('x)))l = 0. We conclude that limm- Blp Mel 0. Morcover, from the anal- ysis above, it is easy to see that supm21, (E(01] Elug" pal < 00.
We now deal with the second term (E Ei= - :d 1/2
from Eq. (15). For this part only, we define
the stopping time Te for (0. (X)((())) Xo) (1x(())11-- (Recall that the analysis is conducted conditionally on the fact that [[o(X))] > 0) by
T, inf 2 0, s.t. ||o(x,)) [0,c)ue=)),o))
Notice that T. > 0 almost surely since ||o(Xo)] E (c,c
Let s E (0, 1]. We have that
IAT
thr (((;)
om (Xmii)
E
0
)
2E
ds
[lx(X.)
1 (X.m)
1/T
[I(1 hmmX (
2221/2 o oie (xx))
+ 2E
ds. ||1(x))11
The second term can be upperbounded the following fashion
dAT
[[(1 h(mX:)))) om (I
/ E [11 (mX:)22)11
om (X."))
E
ds 1
ds
|
E
Gma.
where Gm is defined above. We know that 1 B|1- Gm converges to 0 in the limit by the Dominated convergence theorem (the integrand is bounded). Let us show that the first term also vanishes. We have that
IAT n
o(X))
6m (X 1m0m (X:'))l
)2
MAT
x(X:) Φm (X
E
ds 2E
ds
[[o(X.)
|Ø(X,)]
i=1
AAT.
+ 2E
Om (Xm.
ds
Jo
[x((x)]
[
1=1
AAT.
2c 2e-2p 2E
1Ø(X)) - Om (xmm
MAT.
+ 2E
Om (x...)²²
(1o(xx)) |[x(x²)1| ds.
37

---

Published in Transactions on Machine Learning Research (01/2023)
The first term 2c 2c So [lx(X;) 6m Il converges to 0 in the limit m - by Lemma 9 and Lemma 8. Let deal with the second term. Define the event. F. = (suPae(o.1) || - X,ll I log(m)/m) for m large enough such that log(m)/m < e. Observe that on the event E, we have that for all a E (0, 1]. (l,") - ((X,)ll & (vn + log(m))m- Honce,
LAT
)
Elg
pm (x
ds c + log(m)) m0 0.
|xx(x))
[X(')))
Moreover, letting Ee be the complementary event of E, we have
E gg
pm :2
ds
[[((x)]]
AAT
2
s E g=
Om (X.m.))
+
ds
[(x(X,)
| [o((x')1'
2c 2c-2EE EE
11 (X?")) + 21.(
< y + [|o((x)] - -|X-- -xell (by Lemma 9 and the fact that ReLU is Using the fact that (X!")) Lipschitz), we obtain
E gg / IAT
l (XM() 3nm ssp E -x.1² + 36 PP-E). 151
The term sup,ç1 E xT 12 converges to by Lemma 8. Using Doob's martingale inequality the submartingale ||x X,ll (with respect to the natural filtration generated by the Brownian motion 1 we oblain that
BI - X1112 Cn m lm) log(m)2*
P(E))
where we have used Lemmaa 8, We conclude that limm E ESo pLATe Em
omm = 0.
By observing that I. AAA 11th 1u3--Haldd S E J - Halds, ds, a simple application of the Dominated convergence theorem yields lim E feAT 1 ua lds O. Hence, proved that
lim E 1m (X - Im (X.) - (ZeAT. - -2o)l =0. m==0c
From Lemma 10, know that 9m (X Im (Xg") converges in L1 to g(Xent.) - 9(Xo)14, therefore Elg(())))) -g(X)) - (Zun. 7o)l = O which implies that almost surely,
log (11o(Xen)))) = =ZEAT. Zo-
[To(x))l1
Recall that this holds for any c small enough. Obscrve that Te is almost surely non-docrcasing as we decrease e. Hence Tr has limit almost surely. Using Lemina 3 and the continuity of the paths of X, have that lim,-ot Te Taking the limit - conclude that almost surely we have
[[(x))]| log T [oxX))]
2, Zo-
The submartingale behaviour a result the convexity the norm function the fact that X martingale since d(X X.) = Va (XT") ((x)))dR.. simple application of Jensen's inequality yields the result. 14ehe result Lemma holds when replaced T. and the proof exactly the same. We omit the proof avoid redundancies
38

---

Published in Transactions on Machine Learning Research (01/2023)
Now observe that the coordinates of X, are identically distributed (not independent since condition on ||)(X > 0). Thus, for all i E [n], E (x; = [x((X;) where X is the first coordinate of the vector X.- Another key observation is that the event {X} > 0} is included in the event {llo(Xo)]l > 0} (Lemma 1). Hence, P(X. > on (X)( > 0) = P(X: > 0), where the last term P(X) > 0) is free from any conditioning on [xx)))]l > O. By observing that the random variable X X + vम So [l has symmetric distribution around zero (by properties of the Xg and the Brownian motion B), we have that P(X! 0]]lo(Xo]l > 0) P(X ())((1((Xo)]] 0)-1 (1-2
We conclude that
T Ep
4
which yields the desired result for the conditional mean by substraction.
Now let us deal with the variance. To alleviate the notation, we omit the conditioning on the event [Ilo(x))l > 0}. All the expectations below are taken conditionally on this event. Let S s S S Let À 0. We have that
[[o(X,)]1 27 [lx(,)]|
Var
log
Var(Ze Zz)²
(1+ (1+
1 X
(f. (mu
E
pe pm)ddu
n
n²
1+
1 X
St (t - s)
Varpu du
n²
where we have used the inequality (a + b)2 < (1 + x- + (1 + xi6² and the Cauchy-Schwartz inequality.
It remains to simplify Varu Let p = (X.)) = 2- 2 (1-2-)) and D = (x (xx3)) We have that
1
Ep² 7 Bllo's Bloo'(X.)]1 (X.))1=-E11e E-E'''XX.00 (x.))|1 +1
n - 1)
3n
P2
4 PI
+1,
where we have used the exchangeability property of the family (d'(X.); 1,...n). n). Thus, for the variance Varaa, we obtain
n²² Varyă ((p) (pf)²) +n - (pp -)))).
Therefore,
[[o(X,)]1 [Tx(x,I| ) 27
[1+ + (1+ A)]
Var
log
S (1 -) 8)
where Fe. def fe2 ] ((p) (pt)²) + I - pb)) du. Optimizing over À yields
[o(X.)] 27 [|o(x.)]]
|]
< (t 5) (n-1/2 r 1/2)
Var
log
The term Fae can be shown to have O(n -1/2) asymptotic behaviour using tools from Mckean- Vlasov theory. Thus, the variance term has (atmost)) (nn- behaviour.
39

---

As result, the pre-accivations Y(L) (Eq. (1) converge in distribution to Gaussian distribution in the limit whweeedeetttttt--nnteeeee Published in Transactions on Machine Learning Research (01/2023)
Proof of Theorem 2
In this section, we provide the proof of Theorem 2. We use the following Law of Large numbers that does not require independence.
Theorem 8 (Corollary 3.1 in Sung et al. (2008)). Let (Nilsisan2) be triangular array of random variables. Assume that the following holds
1 suPa21 n E E| < 00.
oo supn21 1 Σ. E
(1Y
Then, we have that
1
O.
n2
i=1
where the convergence is in L1 and Ç E[
Fno with = , $1 i.e. the sigma algebra
generated by the variables (Y,,, 1 S l S j), and Fn,o~ = (0 by definition.
Let us now prove our result.
Theorem 2. For
have
[ log (To(x.)])
1-8 and,
||)(X.)]
(-8
1110(X0)110) n-000
_ exp
n-000
[x(x)]]
where the convergence holds in L1. Moreover, we have that
sup E sup [E(0,1] ml]
X Xt:1/ -OO
where X; is the solution of the following (Mackean-Vie Vlasou) SDE
ax; = (Eo(X)))) 1/2 aB X = X
Vi [n]. Kiul L-xo then no000 N d || exp(t/2)).
Proof. Let 128555 From Theorem 1, have that almost surely
| exp |o(X.)]|
(22 [ B.)
1 f pudu
log
We know that (B. B.)
B.) converges to zero almost surely (by continuity of Brownian paths) and in L1- Let
us now deal with the sccond term n- J dudu. We have that 3, 11 Σ" Ei ('(x:)) 1, Fix u E [s, and 2 let. Zí, =X(X)) (recall that X has implicit dependence on n.) Since Z is uniformly bounded across and n, it is straightforward that the conditions of Theorem are satisfied. Therefore, have the following convergence in L
Σ
St
0,
where 5: E[Z2,[nntta) Recall from the proof of Theorem that the event {X) 0} is included in the event (l]o(Xn)] 0}. Another key observation that will allow us to conclude is that the distribution of X
40

---

Published in Transactions on Machine Learning Research (01/2023)
given Fn,t-1 is symmetric around since the dependence is reflected only in the variance of the Brownian motion. Hence, S = 2 2--)= almost surely. Since n- Σ ii=1 Ch = (1 -1---)--- _ 2: (in I1), then n E Z converges to 1/2 in L1. Using the Dominated convergence theorem, we obtain the first result.
Let us now deal with the second result on the absolute growth factor. Let N > and define the event
[[x(X))] [[x()))
S exp(N)).
EN
and let ENy be its complementary event. For N large enough, we have that
|[(())]|
[xx)) [[x(X.))
[x())] [[x(X.)
[xx))
exp(t/4)
=
exp((t-))) -
LEN + EE
exp((t - s)/4)
LEZ
log [10(xx)11
< exp(NN x
- (1 s)/4
[[x(X))]
+ [x(X,)I
exp((t 8)/4) 1x
< exp(N) x log [x(())11
- - ((-55). -)/5 + P(EE))
[T((x))]
where K is (t dependent) constant and where we have used Theorem 1 obtain that E Taking to infinity in the inequality above, we obtain that for N large enough
[e(x.)] [x(X)]
is finite.
[xx(,)]
lim sumsupT E
exp(1/4)
S KP((E))
[x(())]
22+00
[(x())]
S N KE log
[[x(x.I]]
where we have used Markov's incquality. Since this is true for all N large enough, we conclude that exp((t- () 5)/ 0. [xxI]]
lim E
[o(X))l
The convergence to Mckean-Vlason dynamics is straightforward from Theorem 6, and the Gaussian distri- bution is given by Lemma 16.
1.1 Some technical lemmas
Lemma 16. Let x E Rdd such that + 0, m 2 1 be an integer, and consider the real-valued (Mekean- Vlasov) stochastic process X given by
dx, = (16(Xx)2) dB,, tE (0, Xo = WT,
where is the ReLU activation function, (B:)12o is one-dimensional Brownian motion, and W.n d()) We have the following
Vt 2 ~
d' exp(t/2)).
Proof. Let t > 0. From the SDE, it is clear that X, is Gaussian with zero mean and variance f/r (l)((x))la. 1e, (by Ito isometry). Therefore, since ReL.U is homogeneous, it staaghttffrr that for all > || x,)
41

---

Published in Transactions on Machine Learning Research (01/2023)
3||Xall2. Using Itô is lemma, we obtain
= + 1 211
Taking the expectation 15 yields the following ordinary differential equation
d X1 12 1111 dl
which has closed-for form solution given by
|| ||xo 12, exp(t/2).
We conclude by observing that 1| EX d ||x|?
Lemma 17. Let a E Rd such that x + 0, m 2 1 be an integer, and consider the two real-valued (Mckean- Vlasov) stochastic processes Xm and X given by
dX = (Bóm (X 1/2 dB,, LE (0),, X = WTT,.
dX, (Eσ(X)))) dB, tE 10,,,, Xu WTTz.
where @m((2) = Jet h(mu)dd where the Sigmoid function given by h(u) = (1 + e- is the ReLU activation function, (Bt)e2o is a one-dimensional Brownian molion, and Win ~ ((0,d-11) We have the following
21
Vt 2 O, E| 8112= S m²
2
Proof. The proof of Lemma 17 is similar to that of Lemma 8 with the only difference of replacing the euclidean norm with the T2 norm in probability space. Let t 2 O, we have that
E|x m X,P = (f
2
-
(X - (x))) )
((mm(""") (1
2t
2 f 1|8TT X.11i,
m?
where we have used the triangular incquality and Lemma 9. We conclude using Gronwall's lemma.
J Proof of Theorem 3
Theorem 3. Let E [0, Then, in the limit lim lim (infinite width, then infinite depth), we have that
10(( Ill [|o(%) ((lо(rrii - exp (
where the convergence holds in probability.
18rThs should be understood integrating the SDF, then taking expectation, then differentiating once again.
42

---

Published in Transactions on Machine Learning Research (01/2023)
Moreover, the pre-activations Yiul (Eq. (1)) converge in distribution to Gaussian distribution in the limát
Vi [n], Mar n- then L-x0, NO d || ||]]| exp(1/2)).
Proof. Let E [0, 1]. It is straightforward that lima-+o 1 (1 = almost surely. Moreover, we have that for all & E [0, 1], n "==))l(((= converges in distribution to ( ) when n goes to infinity (Yang, 2020; Hayou et al., 2021; Matthews et al., 2018). Since the limiting value is constant, then the convergence holds also in probability. Now let > O. We have that
[az,]
P (
1 (1l((1))110)) - exp |oo()))]
>C) S
1 [[ exp 4 -c
[|o(Y))]
[
P
1 (1loC)) exp
[lo(Y))]
Let us show that the first term in the right-hand side converges to 0 in the sequential limit 'infinite width then infinite depth'. The proof is similar for the second term. We have that
v EBN(Y (er)))
[1(())))) IIl [[и0(~>1>))) - >0) ||o(Y))]
[|0(Y))))) (Yo())
P
P
1 [(00001110000
c/2
VE
(22))
+1
exp
> c/2
where 1(2 > c/2) def (222//2) (to alleviate the notation). n (1)())1 to (er)) )², we obtain for all L
Using the convergence in probability of
V
1 [100())))000-- exp 1
exp (1 20/2
lim P 100000
51
VED(Y?)
Using Lemma 5 in Hayou et al. (2021), and the homegenous property of ReLU, have that
lim Eo L000
=0
where de [0, + + is the solution of the ordinary differential equation de 39., which has unique solution given by q: 4o exp(1/2). Dividing by 4o and taking the square root, and taking L to infinity, we obtain the desired result.
Regarding the convergence in distribution of the pre-activations, in the limit + co, the pre-activations become Gaussian with zero mean and variance E(Y [et, )2. This variance converges to given above in the limit + oo. The conclusion straightforward using Slutsky's lemma.
43

---

Published in Transactions on Machine Learning Research (01/2023)
K Piece-wite wise linear activation functions
We have seen in Section 4 that the distribution of X2 is generally intractable for n 2 2. This is purely due to sinile width n 2 2 and not to the non-linearity of the activation function. To understand this, let us see what happens when the activation function the identity function. In this case the process X₃ is solution of the following SDE
dx,- = - |XilldB..
(16)
When = 1, the SDE Eq. (16) has closed-form solution given by the (conditional) GBM distribution (Proposition 2). For general n 2 2, the entries of X, are dependent and the resulting dynamics (generally) do not admit closed-form solutions. However, we can obtain flosed-form solutions for the norm ||eell Indeed, simple application of Itô 's lemma yields the following results.
Theorem (Norms with the identity activation).
With the linear activation, we have that for all t [0,1],
(J Br
1 vn
2 ( ))) almost surely,
| = X exp
+
where (B)ez0 is a one-dimensional Broumian motion. As resull, we have that for all 0 < 1
[
(1 2 n1 (t s).
E log
The proof of Theorem 9 is straightforward using Ttô 's lemma. We omit the proof here.
Role of the non-linearity. By comparing the result of Theorem I and Theorem 9, observe some differences between the case of ReLU and that of the identity activation function. With ReLU, the drift term
in log((l@((.)IIIIIl))) (X)))) is given by 1 So ugds which is a stochastic term with mean given by ( - #)
With the identity activation, this drift term is deterministic and is equal to (7 - +)) t. This allows to conclude the following:
Non-linearits induces stochastic drift: the non- non-lincarity of ReLU induces stochasticity in the drift term of log(()X,ll|||xoll, which results in the Quasi-GBM dynamics given by Theorem 1.
Non-linearity induces change of regime: with ReLU, the mean drift of log(llo(xx)]///](XXX))] is given by ("-p"- which negative for n 1,2,3. This induces the change of regime we discussed after Theorem 1 (having a negative mean drift implies that there is significant mass of the distribution of || Xoll in the regime 1)). With the identity activation function, the drift term is always non- nen-nveaaive for n 2 2, and negative for n = 1. Thus, the change of regime cover some values n 2 only when there a non-lincarity. We give more details about this observation the next result.
To capture the effect of non-linearity in the regime change phenomenon discussed above, we study the dynamics of the post-norm activation for special class of piece-wise linear activations that include both ReLU and the identity function. The result of Theorem 1 can be easily extended to the case of general picco-wise lincar activation functions using the same proof techniques. We obtain the following result which generalizes that of Theorem 1 and Theorem 9.
Theorem 10 (Post-activations norm for piece-wise linear activations). Let a E R,, and let dna be the activation function given by (2) = aReLU((z) + (B)LU( We have that for all t e [0, 1],
1 (X (Xx)) exp (11 B, + n po (g almost surely.
where p. - 1 (a² 12, 1;11,,c) 1, and (B)120 is one-dimensional Brownion motion. As result, have that for all 0 < 8 $15
44

---

Published in Transactions on Machine Learning Research (01/2023)
a 0,8 #0
1 (Xi))l 1 (Xa)] 11100.0 >
=8801227)
E
log
(t (t-s).
4
a + 0,8
[ [1
aaa(1- 2 -
(a02112-")
E
log
(t s).
4
a #0
[
a + 8 1 s).
E log
1 (x,)
Theorem 10 generalizes that of ReLU (Theorem 1, = 1, = 0) and the identity activation (Theorem 9, a = -- The discontinuity of the mean of log ((x))) at the poles a = 0 (and 3 * 0) and , = 0 [
x,)Il
(and a + 0) is due the fact that the event a ( (x)] > 0f has non-zero probability in these cases and zero probability when + and B + 0.
Perturbation analysis around the identity function.
Consider the case when and
for some < 1. The mean logarithmic growth factor is given by
G" ( 4 --)---))
1 (1 =))
1) (1 (1- 8)
2 ( (---- (1- m
1-E
Observe that for O, recover the result of Theorem 9 (identity activation). Hence, small perturbation of the identity function has the effect of decreasing the factor G which results in having negative values for G for certain values of n. Indeed, by fixing a = 1, notice that the minimum values of G is obtained when B xs 0, for which Φ1.0 = ReLU. Notice that we can also control the change of regime by tuning the parameter a. This allows us to control the sign G for any n by tuning the parameter a. We leave the analysis of the practical implications of tuning for future work.
45

---

Published in Transactions on Machine Learning Research (01/2023)
L Additional Experiments
L.1 Geometric Brownian motion
Additional histograms of Yz and log(Yi) (Proposition 2) are shown in Fig. 9 and Fig. 10.
Estimated Thepretical density
Kamel Estimate
Theoretical mean
035
Nist of observations
Hist observations
0.6
Densty
0.10
0.-
0.05
(a) Distribution of log(Y2) with 5
(b) Distribution of Yz with =
0.43
Density Throratical maan observations
Theoretical density observations
0.35
Density
Denstry
0.6
0.4
0.32
0.05
(c) Distribution of log(Yr.) with 10
(d) Distribution Y, with
density
Density Theoretical mean
Theoretical density
Density
Donsity
0.6
0.2
62-
(e) Distribution of log(YL) with 50
(f) Distribution of Yz with 50
Figure 9: Empirical verification of Proposition 2. (a), (c), (c) Histograms of log(Yz) and based on N 5000 simulations for depths E (5, 10, 50} with Y, theoretical density (Gaussian) illustrated
Estimated density (Gaussian kernel estimate) and the same graphs. (b), (d), (f) Histograms of Yz, based
N 5000 simulations for depths L E (5, 10, 50) with Yo =1
46

---

Published in Transactions on Machine Learning Research (01/2023)
0.0
Estimated density
Kernel Densily Estimate
Theoretical density
Thearetical mean
observations
Densty
013
(a) Distribution of log(Yr) with L 100
(b) Distribution Y,, with L 100
Estimated density
Kernel Density Estimate
0.42
Theoretical density
10-
Theoretical
0.35
Hist observations
observations
0.51
Density
0.25
Density
(c) Distribution of log(Yz) with L 200
(d) Distribution of Y, with I. = 200
Figure 10: Empirical verification Proposition (a), (c) Histograms of log(Y2) and based on 5000 simulations for depths E {100, 200} with Yo Estimated density (Gaussian kernel estimate) and theoretical density (Gaussian) are illustrated on the same graphs. (b), (d) Histograms of Yz based on
N 5000 simulations for depths L E {100, 2003 with Yo 1.
47

---

Published in Transactions on Machine Learning Research (01/2023)
L.2
IinsteinUUhlebedd process
Additional histograms of Yz and g(Y) (Proposition 4) are shown in Fig. 9 and Fig. 10.
Estimated
Estimate Theoretical mean
Thepretical density Nist of observations
035
Hist observations
0.6
Densty
0.10
0.-
0.05
(a) Distribution of log(Y2) with 5
(b) Distribution of Yz with =
0.43
Density Throratical maan observations
Theoretical density observations
0.35
Density
Densty
0.6
0.4
0.32
0.05
(c) Distribution of log(Yr.) with 10
(d) Distribution Y, with
density
Density Theoretical mean
Theoretical density
Donsity
0.6
0.2
62-
(e) Distribution of log(YL) with 50
(f) Distribution of Yz with 50
(a), (c), (c) Histograms of log(Yz) and basod on N 5000 simulations for depths L E (5, 10, 50} with Yo 1. Estimated density (Gaussian kernel estimate) and theoretical density (Gaussian) are illustrated the same graphs. (b), (d), (f) Histograms of Y,. based N 5000 simulations for depths L (5, 10,50) 50} with Yo 1.
Figure 11: Empirical verification of Proposition
48

---

Published in Transactions on Machine Learning Research (01/2023)
0.0
Estimated density
Kernel Densily Estimate
Theoretical density
Thearetical mean
observations
Densty
013
(a) Distribution of log(Yr) with L 100
(b) Distribution Y,, with L 100
Estimated density
Kernel Density Estimate
0.42
Theoretical density
10-
Theoretical
0.35
Hist observations
observations
0.51
Density
0.25
Density
(c) Distribution of log(Yz) with L 200
(d) Distribution of Y, with I. = 200
Figure 12: Empirical verification Proposition (a), (c) Histograms of log(Y2) and based on 5000 simulations for depths E {100, 200} with Yo Estimated density (Gaussian kernel estimate) and theoretical density (Gaussian) are illustrated on the same graphs. (b), (d) Histograms of Yz based on
N 5000 simulations for depths L E {100, 2003 with Yo 1.
49

---

Published in Transactions on Machine Learning Research (01/2023)
L.3 Histograms of non-scaled log- norm of post-activations
In Fig. 13, we show the histogram of log(|l@(Y.)ll//)((()) based on N 5000 simulations. We observe that as the width n increases, the Gaussian approximate is no longer accurate, which is due to the fact that [ converges to a deterministic value (Theorem 2).
Keme Density Estimatee
Kamei Dendity Estimate
Gauss an estimaae Hist of observations
Gaussion extimate Hist abservat ons
66-
68
63+
611
(a) 2
(b)
Keme Dersity ustimate
Karmi Deraity Extimate eatimate Hat
osservations
Deniige
6A
02-
(c) n
(d) n
1.2
Nemel Density Estimate obornadions
Gaussion estimate
Gamel Denally Entimaze
1.00
(8s) sboenorians
Density
0.7%
0.00
20
(l) 100
Figure 13: Histogram of log(
for depth L 100 and different widths {2, 2, 6, 20, 100).
Gaussian density estimate and (Gaussian) kernel density estimate are shown. As the width increases, we observe deterioration of the match between the best Gaussian estimate and the empirical distribution. This is due to the fact that the norm of the post-activations concentrates around deterministic value when n goes to infinity (Theorem 2).
50

---

Published in Transactions on Machine Learning Research (01/2023)
L.4 Evolution of V=llog(lo(03)///0XXX000
In Fig. 14, Fig. 15, Fig. 16, and Fig. 17, we show the histograms of vn ( for depth L. = 100, hidden layers I E (10, 30, 40, 60, 70,90 70,90), and widths E (2, 20, 100). We observe that Gaussian distribution fits better the last layers. This was expected since the limiting distribution (Quasi-GBM) given in Theorem is only valid for layer indices [tL] when goes to infinity. Thus, for small 1, it should be expected that the Gaussian distribution would not be good approximation.
Layer
Layer
1.4
0.8
Density
Gaussian estimate
0.7
Hist of observations
1.0
0.6
0.5
0.8
0.2
0.2
0.1
0.0
0.0
2
Layer
Layer
0.6
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.0
0.0
-2
-3
Layer 70
Layer 90
0.5
0.4
0.3
0.3
0.2
0.2
0.1
Figure 14: Distribution of /n log([1 lo (Eq. (1)) for different layer indices, with depth L 100 and width n = 2
51

---

Published in Transactions on Machine Learning Research (01/2023)
0.8
Kornel Density Estimate
12
ussian estimate
0.7
observations
1.0
0.6
0.5
0.8
0.4
0.4
0.2
Layer
Layer
0.6
0.5
0.5
0.4
0.4
0.3
0.2
0.1
0.1
0.0
0.0
2
Layer 70
Layer 90
0.4
0.4
0.3
0.2
0.1
0.1
0.0
0.0
-4
-3
-2
-4
-3
-2
-1
Figure 15: Distribution of vn (X)))//)()))) (Eq. (1)) for different layer indices, with depth L 100 and width n = 3
52

---

Published in Transactions on Machine Learning Research (01/2023)
Layer
12
Komel Estimate
Gaussian estimate
0.7
1.0
observations
0.6
0.5
0.3
0.2
Layer
Layer
0.5
0.5
0.4
0.4
0.3
0.2
0.1
0.1
0.0
0.0
Layer 70
Layer 90
0.5
0.4
0.35
0.30
0.2
0.15
0.10
0.0
0.00
-4
-3
-2
-1
-4
-2
Figure 16: Distribution of vn log( ( (Eq. (1)) for different layer indices, with depth L 100 and width n 20
53

---

Published in Transactions on Machine Learning Research (01/2023)
Komel Estimate
12
Gaussian estimate
0.7
observations
0.6
1.0
0.8
0.4
0.3
0.4
0.2
0.2
Layer
Layer
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
2
Layer 70
Layer 90
0.5
0.4
0.3
0.2
0.2
0.1
0.0
0.0
-4
-2
-4
-2
6
Figure 17: Distribution of vn (Eq. (1)) for different layer indices, with depth L 100 and width n 100
54

---

Published in Transactions on Machine Learning Research (01/2023)
L.5 Evolution of log([|[(X)1//]]((60]] (non-scaled).
In Fig. 18, Fig. 19, Fig. 20, and Fig. 21, we show the non-scaled versions of the histograms from the previous section. We observe that the histogram concentrates around single value (the distribution converges to Dirac mass) as increases. This is result of the asymptotic behaviour of the ResNet in the infinite-depth- theu-infiitt-wwide width limit shown in Theorem 2.
Layer
Layer
Density
Gaussian estimate
1.0
150
Hist of observations
0.8
1.25
1.00
0.4
0.2
0.00
0.0
Layer
Layer
0.8
0.8
0.7
0.6
0.6
0.5
0.4
0.4
0.3
0.2
02
0.1
0.0
0.0
-2
Layer 70
Layer 90
0.6
0.6
0.5
0.4
0.4
0.3
0.3
011
Figure 18: Distribution of (og(()0)))//100(()))) (Eq. (1)) for different layer indices, with depth L 100 and
width =
55

---

Published in Transactions on Machine Learning Research (01/2023)
Kornel Density Estimate
Gaussian estimate
observations
10:
1.5
0.8
10-
0.4
0.5
0.2
Layer
Layer
1.0
0.8
0.8
0.6
0.2
0.2
0.0
0.0
Layer 70
Layer 90
0.8
0.7
0.7
0.6
0.5
0.5
0.4
0.4
0.3
0.2
0.2
0.1
0.0
0.0
-4
-3
-2
-1
-4
-2
-1
Figure 19: Distribution of loggl (Eq. (1)) for different layer indices, with depth L = 100 and width n =
56

---

Published in Transactions on Machine Learning Research (01/2023)
5
3.0
2.5
Korne Density Estimate
2.0
Gaussian estimate
Hist of observations
2-
1.0
1-
0.5
Layer
Layer
3.0
2.0
1.5
1.0
0.5
0.0
0.0
Layer 70
Layer 90
2.00
2.0
1.75
1.50
15-
0.5
0.50
0.25
0.0
0.00
-4
-2
-4
Figure 20: Distribution of log() (Eq. (1)) for different layer indices, with depth L = 100 and
width n = 20
57

---

Published in Transactions on Machine Learning Research (01/2023)
Kornel Density Estimate
12
Gssiin estimate
7-
observations
6-
10-
8-
3-
2
Layer
Layer
6-
5-
5
4-
4-
3-
3-
2-
2-
1-
2
Layer 70
Layer 90
5
4-
3
2-
1-
-3
-2
-4
-2
-1
Figure 21: Distribution of (Eq. (1)) for different layer indices, with depth L = 100 and width n = 100.
58


Reviews:
[{"review_id": "W1XxU9xVT4", "reviewer": "Reviewer_2TWB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: the results are novel and have some fundamental importance in understanding deep learning, especially at initialization\n\nWeakness:\n1. the authors should probably spend at least a little effort in demonstrating the possible usefulness of the theory, say, suggesting a new initialization trick\n\n2. there are a few other problems that I directly outline in the requested change section", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the distribution of the infinite-depth limit of activations of a fully connected resnet\n\nThe authors showed that the distribution is, in general, not tractable and identified a few special cases where the distribution can actually be identified", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. There are two theoretical cases I would like to see, and I think would improve the comprehensibility of the theory\n- The case when the activation is linear. It seems like the distribution is analytically tractable here\n- The case when the activation is only perturbatively away from linearity; this would allow us to understand the leading effect of being nonlinear\n- having these results, the authors should be able to make much more insightful discussions about the distribution -- for example, what part of the derived results can be attributed to having \"depth\" and what part can be attributed to being \"nonlinear\"\n\n2. The authors have indeed demonstrated the correctness of the theory, but I think the authors should probably spend at least a little effort in demonstrating the possible usefulness of the theory, say, suggesting a new initialization trick\n- this does not have to be a very large-scale experiment, just suggesting a minor trick that works on a toy example is satisfactory to me\n\n3. I am against using the word \"phase transition.\" The word phase transition comes from statistical physics and has a quite precise definition. To talk about a phase transition, one first needs to define/identify the free energy, and one then needs to identify where it becomes nonanalytic, but this is not what the authors have achieved. \n- I would suggest using the word \"regime change,\" \"qualitative change in behavior,\" \"regime crossover,\" or similar wording\n\n\nAfter these changes, I would be happy to recommend acceptance\n", "broader_impact_concerns": "Fine ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PkRKF6NKiV", "reviewer": "Reviewer_iyP8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Contains several interesting statistical observations. \n- Simulations seem to support the authors' insights. \n\n## Weaknesses \nThe biggest concern is the relevance of the paper to ML. The major connection I can tell is: residual networks can be viewed as an Euler discretion scheme of SDEs. Follow-up results derived from this observation provide limited insights into ML, e.g., insights into\n- What is the distribution of the outputs (for multiple inputs), and how does this distribution affect the training and performance of networks? Note that the paper only handles single input.  \n- How is this limit ($L\\to\\infty$ first) related to training / initializing of neural networks? My understanding is that this scaling limit is not popular in practice and practitioners often scale the width and depth simultaneously. Moreover, the depth (in tens) << width (in thousands) in practice.  \n- What do the backward gradients (NTKs) look like, and what is their connection to network training? \n\nI am not convinced that this paper is of great interest to the ML community without such or similar findings. I think the statistics avenue will be more suitable for this paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies random `residual` neural networks in the following limits: depth $L\\to \\infty$ and the width $n$ are either fixed and finite or go to infinity. Based on the observation that random `residual` neural networks can be viewed as an Euler discretion scheme of SDEs, the authors convert the studies of random residual networks to the studies of continuous time SDEs. As such, tools from stochastic calculus (e.g., Ito's formula) can be applied directly. The authors obtained several results, including \n- Single input, $d=n=1$ ($d$: input dimension, $n$ width of the network) case. Characterization of the distribution of the output for some special activations. \n-  Single input, $n>1$. The distribution is hard to get. Instead, the authors compute the distribution of the norm of the output, $\\|X_t\\|$. \n- The authors also point out that the distributions of the norm $\\|X_t\\|$ are not the same under a different order of limits $d\\to \\infty$ then $n\\to\\infty$ v.s. $n\\to\\infty$ then $d\\to\\infty$. \n\nOverall, the paper contains several interesting statistical insights, and the approaches of the paper seem rigorous. However, I find the results not very relevant, useful, or interesting from an ML perspective. It is unclear to me what this paper's main take-home `ML` message is. In my opinion, this paper may be more suitable for the stat community.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I don't have requested changes for this paper. The major concern I have is the lack of relevance to the ML community. I would like to see more insights that could strengthen our theoretical understanding of neural networks. ", "broader_impact_concerns": "NA. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sia9RRYNBh", "reviewer": "Reviewer_aoN3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1. The infinite depth limit results are novel, as this regime is under explored. \n2. The characterization of the limit is fairly detailed, despite a somewhat intractable limiting SDE. \n\nWeaknesses \n1. I would personally prefer some more discussion towards how this work relates to other regimes, and what the authors suspect happens the kernel for multiple inputs. More details in the requested changed section. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studied ResNets at initialization with a particular scaling of $1/\\sqrt{L}$ weighting on the fully connected component, and study the limit as depth $L\\to\\infty$ for some fixed finite width $n$. The main contributions of the paper revolve around the limiting SDE in Proposition 1, and the consequences that follows from this SDE. In particular, the authors showed that the network does not collapse under some basic conditions, the norm of post activations behave approximately like a geometric Brownian motion, and characterized the sequent depth then width limit. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "On the infinite-depth-then-width limit \n\nAs the authors described in the introduction, the order of the limits matter for neural networks. In particular, the width-then-depth limit typically loses some information and therefore is not equal to the joint limit. I'm curious if the authors have any thoughts towards whether or not the depth-then-width limit behaves similarly and loses some information compared to the joint limit. \n\nOn the degeneracy of C-maps \n\nOne common issue of infinite-depth analysis is that the C-maps becomes degenerate in the limit, and this usually leads to poor training stability unless a normalization method or deep kernel shaping is used [1]. In the case of the scaling in this paper, do the authors have any thought on whether or not it leads to degenerate C-maps? As far as I can tell, the contribution of the activation function should be bounded in the limit. \n\nOn joint distribution over multiple inputs \n\nRelated to the earlier point, can the authors comment on any foreseeable technical challenges for extending towards the multiple input setting? In particular, can techniques from [2] be used towards deriving an SDE for the covariance kernel? \n\nMinor typo in equation 4, I believe the drift is missing a factor of $1/2$. \n\nReferences\n\n1. Martens, James, Andy Ballard, Guillaume Desjardins, Grzegorz Swirszcz, Valentin Dalibard, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. \"Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping.\" arXiv preprint arXiv:2110.01765 (2021).\n\n2. Li, Mufan Bill, Mihai Nica, and Daniel M. Roy. \"The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization.\" arXiv preprint arXiv:2206.02768 (2022).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the infinite-depth limit of finite-width residual neural networks. The paper discusses several interesting results including characterization of the output distribution for some special cases. The reviewers noted several positive aspects of the paper including: 
- The novelty of the infinite depth limit as this regime is relatively under-explored
- The simulations support the theoretical insights.
- Detailed characterization of special cases
- Comparisons of infinite-depth-then-infinite-width with the more popular infinite-width-then-infinite-depth limit.

The authors have also satisfactorily addressed suggested changes proposed by reviewers including:
- linear activation and perturbation analysis
- discussion of practical implications
- discussion of joint distribution over multiple inputs

During the discussion, the consensus decision of the reviewers leaned towards acceptance. There was some discussion around relevance to TMLR and if the paper would be a better fit for stats journal, but I think that this paper is very relevant to TMLR audience (see my justification under "Audience" above). 

I recommend accept as is. Congrats!  
Decision: Accept (poster)