Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training
Authors: Huy Tuan Pham
Phan-Minh Nguyen
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning theory
mean field limit
finite-width correction
fluctuation
implicit bias

Abstract: The mean field theory of multilayer neural networks centers around a particular infinite-width scaling, in which the learning dynamics is shown to be closely tracked by the mean field limit. A random fluctuation around this infinite-width limit is expected from a large-width expansion to the next order. This fluctuation has been studied only in the case of shallow networks, where previous works employ heavily technical notions or additional formulation ideas amenable only to that case. Treatment of the multilayer case has been missing, with the chief difficulty in finding a formulation that must capture the stochastic dependency across not only time but also depth.

In this work, we initiate the study of the fluctuation in the case of multilayer networks, at any network depth. Leveraging on the neuronal embedding framework recently introduced by Nguyen and Pham, we systematically derive a system of dynamical equations, called the second-order mean field limit, that captures the limiting fluctuation distribution. We demonstrate through the framework the complex interaction among neurons in this second-order mean field limit, the stochasticity with cross-layer dependency and the nonlinear time evolution inherent in the limiting fluctuation. A limit theorem is proven to relate quantitatively this limit to the fluctuation realized by large-width networks.

We apply the result to show a stability property of gradient descent mean field training: in the large-width regime, along the training trajectory, it progressively biases towards a solution with "minimal fluctuation" (in fact, vanishing fluctuation) in the learned output function, even after the network has been initialized at or has converged (sufficiently fast) to a global optimum. This extends a similar phenomenon previously shown only for shallow networks with a squared loss in the empirical risk minimization setting, to multilayer networks with a loss function that is not necessarily convex in a more general setting.

Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training
Huy Tuan Pham
Phan-Minh Nguyen The Voleon Group
Department of Mathematics, Stanford University
Abstract
The mean field theory of multilayer neural networks centers around a particu- lar wifdh scaling, in which the learning dynamics is shown to be closely tracked by the mean field limit. A random fluctuation around this infinite-widhh limit is expected from a large-width expansion to the next order. This fluctuation has been studied only in the case of shallow networks, where previous works em- play heavily technical notions or additional formulation ideas amenable only to that case. Treatment the multilayer case has been missing, with the chief diffi- culty in finding formulation that must capture the stochastic dependency across not only time but also depth.
In this work, we initiate the study of the fluctuation in the case multilayer net- works, any network depth. Leveraging on the neuronal embedding framework recently introduced by Nguyen and Pham [17], we systematically derive a system of dynamical equations, called the second-order mean field limit, that captures the limiting fluctuation distribution. We demonstrate through the framework the com- plex interaction among neurons in this second-order mean field limit, the stochas- ticity with cross-layer dependency and the nonlinear time evolution inherent in the limiting fluctuation. A limit theorem is proven relate quantitatively this limit the fluctuation realized by large-width networks.
We apply the result to show stability property of gradient descent mean field training: the large-wide width regime, along the training trajectory, progressively biases towards a solution with "minimal fluctuation" (in fact, vanishing fluctua- tion) in the learned output function, even after network has been initialized has converged (sufficiently fast) to a global optimum. This extends a similar phenomenon previously shown only for shallow networks with a squared loss the empirical risk minimization setting, to multilayer networks with loss func- tion that not necessarily convex a more general setting.
Introduction
Recent literature has witnessed much interest and progresses in the mean field theory of neural networks. In particular, it is shown that under a suitable scaling, as the widths tend to infinity, the neural network's learning dynamics converges to nonlinear deterministic limit, known the mean field (MF) limit [14, 17]. This line of works starts with analyses the shallow case under various settings and has led to number of nontrivial exciting results [18, 14, 5, 23, 25, 9, 22, 19, 29, 24, 30, 12, 1, 16]. The generalization to multilayer neural networks, already much more conceptually and technically challenging, has also been met with serious efforts from different groups of authors, with various novel ideas and insights [15, 17, 20, 2, 26, 6].
Since the MF limit is basically first-order infinite-wide approximation of the neural network, it is natural to consider the next order term in the expansion for a more faithful wiith-width description.
"The author ordering is randomized.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This leads to the consideration of the fluctuation, up-scaled appropriately with the widths, around the MF limit. On one hand, this fluctuation should display MF interactions among neurons. On the other hand, it is random due to the inherent stochasticity in the finite-width network which, for instance, is randomly initialized and hence induces randomness in the interactions among neurons.
For shallow networks, [27, 23,3] has identified the limiting fluctuation in the form of a time-evolving signed measure over weights, leading to a central limit theorem (CLT) in the space of measures. In particular, [27] pinpoints the fluctuation via a compactness argument in an appropriate measure space. Avoiding the heavy technicality, [3] realizes neat trick, specific to shallow networks, where quantities of interest are described via push-forwane of the initialized values of the weights.
There has been no similar attempt for the multilayer case, which faces major obstacles in formu- lating the limiting fluctuation. Firstly, existing formulations of the MF limit already move away from working with measures over weights [15, 17, 26, 6], unless restricted conditions assumed [2]. This is due to the complexity of MF interactions: the presence of middle layers brings about simultaneous actions of multiple symmetry groups. We face the same complexity when formulating the fluctuation. Secondly, unlike shallow networks, the fluctuation in multilayer networks displays stochastic dependency not only through time but also across layers. Specifically the cross-layer dependency is propagated by both forward and backward propagation signals, at any time instance.
Contributions. We tackle the challenge by leveraging on the neuronal embedding framework, recently proposed by [17]. An important concept that supported by the neuronal embedding the sampling of neurons. We use this concept to formulate the limiting fluctuation via a decomposition into two components with different roles in Section 3. One component is a random process, which encodes CLT-like stochasticity in the fluctuation of the sampled neurons around the MF ensemble. The other component, named the second-order mean field limit, is a system of ordinary differential equations (ODEs), which displays MF interactions in the deviation of the neural network under gradient descent (GD) training from the sampled neurons.
This decomposition is an innovation of the work. Our formulation enjoys the generality brought about by the neuronal embedding framework without restrictive assumptions (e.g. i.i.d. initialization assumption made in [2, 26)) and faithfully describes the expected stochastic dependency across time and depth. We prove a limit theorem that establishes the connection with the fluctuation in finite- sized networks (Theorems 5 and 6). Unlike [27, 3], this result is quantitative.
Using this formulation, in Section 4, we show trajectorial stability property in a large-width mul- tilayer network, particularly a variance reduction effect: GD training traverses a solution trajectory that reduces and eventually eliminates the (width-correeeed)) variance of the learned output function. That is, a bias towards "minimal fluctuation". This holds even if the network is initialized at, or if it is trained to converge sufficiently fast to, a global optimum (Theorems 9 and 10). The same phenomenon has been shown in the shallow case [3], which requires a squared loss in the empirical risk minimization setup. As demonstrated by our result, it not an isolated phenomenon and can indeed hold for a neural architecture where MF interactions are more complex, loss function that is not necessarily convex and training setup unrestricted to finitely many training data points.
Limitations and potential extensions. Let us finally mention a few limitations. Our work consid- ers fully-com colle-coneeeeed multilayer networks trained with GD, setup much less general than [17]. We expect certain extensions in this direction are doable. We also do not treat stochastic GD and hence disregard the stochasticity of data sampling, unlike [27]. Given the broader literature this subject (e.g. [1IJ, this extension is foreseeable. Here we focus instead on the stochasticity that is inherent in the interactions among neurons, which is the more interesting aspect of neural networks.
Our result on the output variance is specific to unregularized training, unlike [3], and also requires a sufficient global convergence rate. While there have been several proven global convergence guar- antees for multilayer networks [17, 21, 20], understanding of the convergence rate is still lacking. Even in the shallow case, global convergence has been studied only for type of -ansity-innuucin regularization [5, 4]. Unless the convergence rate for multilayer networks is generally perilous (an unlikely scenario in light of the experiments in [15]),
result is expected to be relevant.
Our development is specific to the MF scaling. This scaling allows for nonlinear feature learning. unlike the NTK scaling [8]. While there are other scalings that also admit a certain sense of feature learning [7, 31], the standard parameterization to degenerate into NIK-like behaviors, which are not expected of practical butteebbtt-argge--ee width
practice - in the infinite-width limit - known
2

---

neural networks [13, 31]. In other words, all infinite-width scalings that display feature learning are only proxies of practical networks. This limitation motivates finite-width studies as we pursue here.
Notations. We use K to denote generic constant that may change from line to line, and similarly K2 for finite constant that may depend on some constant u. For simplicity, for L the network depth, we write K in place of Kr. For E (resp. E) being the expectation, we use V and C (resp. V and C) for the variance and covariance. We write Ez for the expectation w.r.t. data Z (X, Y) P. We write f f to denote the partial derivative w.r.t. the i-th variable of
2 Background on the MF limit for multilayer networks and assumptions
We describe the necessary backgrounds based on (and simplifying) the work [17]. This section also introduces several important notations, well as the assumptions for our study of the fluctuation.
Multilayer network. Following [17], we consider I.-layer fully-connected neural network with widths (Noligz. (for N = 1):
y (1, yr. (Hr (t, 1,,;),
H. (t, = E, [w; (t. 11 (:) P: (H,-) 1 (+,,,1))))) Js E [Ni), == L,..,,,.
H1 (f, (W1 (t,, 1, (c)),,, 2) f1 [Mi]
in which X Rd is the input, the weights are Wi (t, W, Rd and Wi (t, W, = R. ps: R + IR is the activation and E T R>0 the continuous time. Here we reserve the notation Jj for the random variable Jj ~ Unif ([NV.) and we write E, denote the expectation w.r.t these random variables. For convenience, we take jo {1} and No = 1.
Given initialization w, ..,, train the network with gradient descent (GD) w.r.t. the loss L: x R + R20 and the data Z (X,Y) E X x R drawn from a training distribution P:
By (I. ]. i [4],
Oew; (i, = Ez (2 (t,
Ow, (ii-1,5))
in which define
1 H (t, 1, x)):
i= L, Wi+1 ((.1) (H, ((..))))) i< L,
09 (1)
0y (t,
0H, (ja)
E,
ÖH;+1 (J+t1)
Łÿ (t,
fi-1 (H₂-1 (1, (1)),)),)),
i>1,
0ÿ (t,
HH. (ii)
Ow: (fi-1.
0ÿ (t,x)
F,
1=1.
HH, (i1)
One may recognize that these "derivative" quantities are detined in a perturbative fashion; for in- stance. ()(w.) represents how (t, 2) changes (rescaled by widths) one perturbs H, xt)
aH.(1)
Mean field limit. The MF limit is defined upon a given neuronal ensemble F),
I (2,, F,, P.) (in which 2g = 22 = (11)), which is product probability space. We reserve the notation C, for the random variable C; P. and we write Eg denote the expectation w.r.t. these random variables. The MF limit that is associated with the network (1) is described by the evolution of (w; (1, "eeuu given by the following MF ODEs:
OG (t. X)
Oaw, (t, c.1.)))) = EEz 02L (t,
Ow; (Q_1,))
i [4], E 9. G-1 E Q4-1.
where w. T x 0, Q - W, and & T. Here we define the forward quantities:
1 (1,2x) = Pr. (Hr, (1, 1, 1,2)), H; (t, G,,z) = EEc wwj (t, C,-1, ci) p (Hi-1 (t, Ci-1, i= L...,, 2,
H1 ())) = (wi (t, (1),,,,))
3

---

and the backward quantities:
77 1 (t. 1,
i= L, Wit1 (t, e. C4+1) (H) (t, (t,c,,))
0g (t,
dÿ (t,z)
0H; (ci)
Bc
ic L.
0HH+1 (C(+1)
Oi (1,
Pi-1 (H;- (1, ()), i>1, 1
Og (t,z)
0H; (c:
Ow, (G)
0ÿ (t,x)
0H1 (c1)
The evolution with time & the MF limit is generally complex due to the nonlinear activations.
Sampling of neurons, neuronal embedding, and the neural net-MF limit connection. A pri- ori there is no connection between the MF limit and the neural network (I); they are two separate self-contained time-evolving systems, despite the similarity in their definitions. To formalize their connection as in [17], we describe the neuron sampling procedure. In particular, we independently sample (C; (.e)), Miidd P. for ,.... L.. The samples C, (jx) should be thought of as "sampled neurons", " Let use E denote the expectation w.r.t. the sampling randomness. Al- though [17] considers more general sampling rule, this suffices for our purposes.
Now suppose that we are given functions w. Ω4-1 x Sr - W, and set the initializations for the neural network and the MF limit:
W; (0,
ua (C:-1 (i-1) C. (i:)), uy (0,-,)) u (.,-) e [4].
Then lets them evolve according to their own respective dynamics. The sampling of neurons hence allows to couple them on the basis of the tuple (2, P, ( [w]] <<(L) known as the neuronal embedding, the key idea in [17]. Note that this does not pose major limitation on the type of initializations for the neural network (I); indeed it allows both i.i.d. and non-i.id. initializations
since we have freedom to choose the neuronal embedding, as studied in [17, 20).
Once the coupling is done, [17] obtains the following result, which realizes the connection in math- ematical terms. For any finite terminal time T T, we have 2:
max E. sup w, (t, J4) wy (t, C. (Jr-1) G. (J.)) 1/2 4
1ST
with high probability, where C, 0 is universal constant, Nuin min No, Nuax maxie()] N2 and hides the dependency on T. L and log Nmax. This result is akin to the law of large numbers (LLN). In words, it says that for large (ideally infinite) widths, the evolution of the network (1) can be tracked closely by the MF limit. This result is fundamental and gives a useful suggestion that is to study the neural network via analyzing the MF limit. For example, [17] uses it to study the optimization efficiency of the neural network.
The sampling of neurons is inspired by the propagation of chaos argument [28], but is not mere proof device. Indeed, in this work, we again make a crucial use of this sampling in the formulation of the limiting fluctuation (Section 3.3).
Set of assumptions. We consider the following assumptions for the rest of the paper:
Assumption 1. We make the following assumptions:
(Regularity) yps K -boundedd for i E [L -1]; v/ and yi are K --bounded and K K-Lipschitz for i E [T]]; 0zL and 8.f L are K K-bounded and K K-Lipschitz in the second variable; [X| S K almost surely (a.s.).
"Sub-Gaussian initialization) supm21 -1/2BBB Ioe (C.-1, [n]
1m/1//m 1/m
K for any i [L].
(Measurabiliry) (P:) is separable for any i E [Z]. (Constant hidden widths) N1 = Nr-1 = N (with ideally N + oo).
An example for the regularity assumption is & (u) u the identity, s; tanh for [L 1] and L (u), (2) is smooth version of the Huber loss. It is noteworthy that L does not need to be
While [17] treats stochastic GD in discrete time, this result implicit in the proof.

---

convex. The initialization assumption is also mild; it allows most common i..d. initializations as well as non-i.id. schemes in [17. 20]. These assumptions satisfy the conditions these works. Measurability assumption is a technical condition needed for well-defined-ness of the fluctuation; it not conceptually restrictive and can accommodate results in [17. 20].
The constant width assumption aligns with our interest in the infinite-width regime. Our develop- ment can be extended easily to the proportional scaling, where N, [o,N] for some constants Pl,..., PL-1 > 0 with N + oc. These scalings are relevant to practical setups and also to yield interesting results. Specifically once some widths are much larger than others, we expect the fluctu- ation to be dominated by a subset of layers.
Limit system for the fluctuation around the MF limit
We describe our formulation for the fluctuation. As introduced, it is composed of two components: stochastic process that induces Gaussian CLT-like behavior, and another process - called the second- order MF limit that signifies the MF interactions at the fluctuation level. This is done on the basis of the sampling of neurons, on top of the neuronal embedding, introduced Section 2.
We introduce the first component in Section 3.1 and then the second-oder MF limit in 3.2. Together the two components are connected with the fluctuation around the MF limit realized by a large- width network in Section 3.3, to give firstly the analog of (2) and secondly a limit theorem on the fluctuation distribution. We end with discussion in Section 3.4. Proofs are deferred the appendix.
3.1 Gaussian component G
Recall the sampling of neurons described in Section 2. Given the MF limit, we define the following random quantities:
(t, (HL 1t),
H. (t, E, wi (t, (1)_)
i= L....,,
in (f, = (wi (1),
(ltr (,1,
i i= L,
05 (t) OH, (c))
Oy (4xx)
F2,
Wi+1 (1.6, Cin (f
i i<l,
|fli+1 (Ci+1 (J++1))
aī (t)
Pp--1 (fti-1 (1)))))) i>1. 1,
0ÿ (1,2x)
0H, (ci)
ow; (c:-1.
θÿ (t,z) Ir
1=
0H, (c1)
These quantities are analogues of the forward and backward MF quantities, but with Ec; being replaced by an empirical average over {Ca (i))
Now we define
(1,2)) V (4, - (1,x)),
TFe (1, (1,0)1,,6,))== = ( Oī (1,
Oy (t,
i i= 1=
Ouc (G) Ow; (G-1,
and let & denote the collection of these functions. Notice that the randomness of Ô is induced by the samples C.(ji); thus we write the expectation w.r.t. G via E. Intuitively & is the fluctuation of Gaussian process. We show
the sampled neurons around the MF limit and thus should converge that this is indeed the case in the following mode of convergence.
Definition 1. Given joint distributions over a sequence (in N) of functions G.,N : T x Ω x Ω4 x X + W. for i E [L], we say that GN = (Gr.N converges G-polynomially in moment (as N + oo) to G if for any finite collection in l of square-interrable fe T Q4(0)-1 Q4(0)) x X
W that is continuous in time and integers aee 3e 2 1,
II, (fe. Tre No
E II, (fe. Goe 9r 1| =0 suup max 1 1-//8 100 100)
sup (0) ve
5

---

for D Ee ae and constant terminal time T. Here we define:
(S,)), = Ez,c [f (t, C,, for any two such functions f and h, and |1.f122 () As convenient convention, if x Og x Qt + W is tested against GI,N, we define (S, U,Gi,,), and ||/ using Euclidean inner product and the Euclidean norm.
Ci-1, Cx,
Note that this mode of convergence gives a quantitative rate in N.
Theorem 2 (Gaussian component G). For any terminal time T 2 0, G converges G-polynomially in moment to {G", G. i [L]]. which a collection of centered Gaussian processes.
The covariance structure of G in Theorem 2 is explicit but lengthy; we refer to the appendix for full details. Here let us examine the variance of G², given recursively by:
U2 (1, 02, 47) Ve [w2 (t, Ci, 02) pi (II1 (1, (1))),
us (z))) = Be wu (1, Ci-1, 4/11 (Hi-1 (1, Ci-1, 0-1 (1, C,)1, + +C [w, (t, C,) Pi-1 (H (t, C)),, i= ....., L.
G (1,x)1 = le' (Hz (t, 1,)), un (t, 1,x).
The recursive structure shows that G compounds the stochasticity propagated forwardly through the depth of the multilayer network. From the appendix, a similar observation can be made in the backward propagation direction.
3.2 Second-order MF limit
We introduce system ODE's for a dynamics that represents MF interactions at the fluctuation level. As shall see later, this system gives limiting description of the the deviation of the neural network from the sampled neurons under GD training.
Let us denote by G collection of functions GV T R. G. i (L.) For p 2 1, let
TxQ:-1 X
W, for S X
G - (G: ||G|| 8 VT 2),
|| sup Σz,c tT
| (t,X)
Σ i=1
G (t, C x)
with ||G] = || We define processes R; : ::g9 T x Ω Ω2 W, for i [L] as follows.
The processes R2 are initialized at R₃ (,,0,--) = for all i, and for each fixed G G, solve the differential equations:
0, (G,4,
Oy (t. X)
06 (t,, X) Ow, (Cr- C+) 1)
-Ez
(t,))
Σ Ec Prr (G,t,Cr-1,C)
Owi (ci-1,
0Φ (t,
Ez 02C (Y, (t, X)) Ec Σ
R2 (,)
ow, (a, b) Ow; (Gi-1, ci) a= -
Ez 02 (Y, (t, X)) G (t,0--1 c, x)
(3)
8g(, Here Tw, (o)0 is defined in a perturbative fashion similar to those Section and is hence self-explanatory: we give the full explicit definitions in the appendix. We shall write R, (G,t) R. (G,t,-,) for brevity. Let It denote the collection { (eet)
Theorem 3. Under Assumption 1, for any > O and G G with ||G < <0, there exists unique solution i. H R, (G E T.2 (P:-1 x P:) which is continuous in time. Furthermore, for each f, R R t) a continuous linear functional G.
The process R is called the second-order MF limit for reason shall see in Section 3.4.

---

Remark 4. Care should be taken in the derivation of the "second derivative" quantity in Eq. (3), To illustrate, consider a simplified problem of taking perturbation w.r.t. wo (a, b) of the quantity
g (wy (G-1, Gi) BC [fi (wy (C--,,))) IEc [Ja (m, C,)),, [fs (wy (C--1, C.)II) (c)
For instance, when i, we have:
0g Ci)
R₁ (G, b)
Sw, (), a==1-1, bc
R, (G, 01 (c-1. R2 (G,1, C--,,,) 02g (,)-,,,)0) 0f (my, (C)-,,
+ Ru (G,4, 0gg (G-1,4)005 (w; (G-1, C.)) + R. (G, Ci-1, Ct) 0ag (c- c,)s (C-1,, C.))
3.3 Fluctuation around the MF limit via sampling of neurons
We are interested in a characterization of the following quantity, which represents the deviation of the weights of the neural network under GD training from the sampled neurons:
R3 (1,)i = VN (ws (t,Ji-1,5k) - wi (1, C--1 (ii-1), C, (i:))).
Let R denote the collection [Ri)ce[a) We now state our first main result of the section. Theorem 5 (Second-order MF). Under Assumption 1, we have, for any t TT
EE, R3 RRC.
SKT/N.
KT/N.
Upon the correct choice G, one deduces from the theorem the second-order expansion:
w, (t, x wy (t, C.-1 (ii-1), C, (i))
(i--1,
where the first-order term is directly from Eq. (2). Importantly this relation is the basis of the sampled neurons {Ci (j:) ja E [N,], i E [/]]- The dynamics of R2 at the i-th layer is depen- dent on all (EL) and GV, exemplifying the cross-layer interaction of fluctuations in multilayer networks.
Although Theorem involves G. as shown in the derivation of Section 3.4, we actually do not utilize the specific structure of G. Incidentally we study the limiting structure of G separately in Theorem 2. other words, our decomposition of the fluctuation via G and allows relatively independent treatments of the two components (which have different natures, to be discussed in Section 3.4).
Combining the two theorems, we obtain the following CLT for the limiting output fluctuation.
Theorem (CLT for output function). Under Assumption 1, the fluctuation VN (1, ((1,))) converges weakly to the Gaussian process & indexed by T x X.
Oi(t, x)) Σ Ec R.(.C,,CC-II.C)) Sw, CC)
G (t,
GG,,t,
where G is the Gaussian process described in Theorem 2. Specifically, for any integer m 2 1, t < T, and 1-bounded h + R,
EEz [XX) - O(t, xI)) EE (XX)((()))) S Kr - N118400(1)))
We note in passing the technicality we devise to prove Theorem 6. As noted, Theorems 5 and separate treatments of R and G. but recall we are interested in t(G,t). This necessitates a way to describe them jointly. We do so via the following notion of convergence, extending Definition
Definition 7. Suppose we are given sequence (in N) of functions G, Tx X2; x + W; for E [L] such that GN (G.,N hie(r) converges G-polynomially in moment to G. Given further RAN T x 94-1 x Qt x + W, defined jointly with G and Ry = (i.. heez), we say that (Rx, Gx) converges G-polynomially in moment and R-linearly in moment (as N o0) (R,
7

---

if for any fe, he that are T x P1(0)-1 x Pice x X - Way mappings and continuous in time and any inteeers ae,, 3r 2 L.
IE II.
(Se, Goe 9r No
II. Reco).m)cc 10: [(e))
-E II. Woyul Ihe RuCs)
sup (e)) Ve
=00( sup max [[fell sup max | (
1ST 8
tST
for D Ee Bear BeB
Proposition 8. Assume that G converges G-polynomially in moment G {G², G,,, i [4]] Then (R(G.))G)) converges G-polynomially in moment and R-linearly in moment to (R(G, G)
3.4 A heuristic derivation and discussion
heuristic argument. We wish to give a heuristic to derive the relation between R, and R. R.(G, (G, t), ic. Theorem 5. Suppose we look at the i-th layer:
O,R, (t,ji-1,j()
VNO.w, (t, - VNOW., (t, C. (
06 Xt)
0y Xt, ow: (i)
VNEZ O2C (Y, (t) Ow; (C.- (Ji-1) C, (f:))
-NEE (r, (t,
VN E2
02f (3 (Y, (1, - 02h (Y,9 (1, ())
O5 X)
Bryy (Ci-1 (fi-1), C. (j+))
dÿ (t)
0ÿ (t,, X)
Ez 0zL 02 (t, VN
(4)
ow; (C,-1 (i-1)) C, (ji)
Ow, (ji-1.
Let us zoom into A₁, with some care not to eliminate the fluctuation of interest:
A =
(0 (t,,))
= CY (t. (-6x (t, VN (f y (t,x))
Now observe that ÿ (t) y (t, X)) is difference between the sampled neurons and the neural network, and hence one should be able express it in terms of R:
VN (Ū X) & X))
Oi (1, X)
E,
(U- - w, (t,J
Ow, (C+-1 (Jr-1), (J+))
0ī (1,
Σ
E Ow, (C,-1 (Jr-1), Cr (J+)) R, (t, Jr,,,,1,
Furthermore the limit N - co, we expect from the LIN, for a test function f.
(5) As such, one identify the term that involves A₁ in Eq. (4) for O,R, (1,ji-1, with the first term in Eq. (3) for O R₁ (.). One can derive similarly for the rest of the terms. With these, one arrives at R,R, (1)) Re O, t, C.-1 (ji-1), C, (i-)). Recalling that R, (0,-,:) = R- 0, -). -). one arrives the conclusion of Theorem 5.
E, [f (C C, (J+)) Ec [f C,)],,
We make two comments. Firstly, the sampling of neurons makes transparent the derivation of R as the limit of R: it allows compare one-to-one "neuron" of the former neuron of the latter. This demonstrates an advantage of the neuronal embedding framework, which easily accommodates the sampling of neurons in the multilayer setup. We also see that the only involvement of & this derivation is via its definition.
Secondly, arises (by replacing R) at steps that invoke (5). This LLN-type nature signifies MF interactions among neurons at the fluctuation level, and hence the name second-order MF limit. In contrast, the component & displays Gaussian CLT-type nature, evident from its definition.

---

Technical difficulties. So far we have seen R captures the difference between the neural network and the sampled neurons. Recall the other component in the fluctuation is the process G, which cap- tures the v scslee difference between the sampled neurons and the MF limit. Taking into account the whole system of fluctuation, we note major technical subtlety: the aforementioned two differ- ences captured by R. and G share the same source of randomness {C, (j+)= j: E [M.] i i [4]].
We mention two particular complications that arise from this subtlety. The first complication is how one should define the process R in relation with G. given that R is meant to be the infinite-width limit of R while R and G are stochastically coupled. Our solution is to let t + R(G, t) defined for any G E G, not restricted to only G. This streamlines the definition of the second-order MF limit and allows to separate our treatments of R and G, as evident from Theorems and 2.
The second complication lies with Eq. (5). In fact, to arrive at the desired conclusion, we require the function f to depend on G. This is because the main object of interest is R[G, t), even though we have treated R and G separately. In a nutshell, f shares randomness with C, (jr) and Cr-1 (ir-1), jr E [[Vr]. Jr-1 E [NV,-1]. The random variable on the left-hand side of Eq. (5) is therefore complex, and it is highly questionable whether this equation should hold. The analysis becomes delicate; without taking into account this shared randomness, the derivation would he a mere heuristic. As present in the proof, we verify this equation for relevant set of functions f.
4 Asymptotic variance of the output function
We study the following width asymptotic variance quantity:
V (t) lim EEZ || (y X) x)) P lim EE [E2[|VN((((t)) - Ety X))
-000
N
The second equality holds and the limits in NN exist by Theorems and 6. We would like to under- stand (t) in the long-tim horizon, and specifically in situation of considerable interest where the MF limit converges to a global optimum as t + co. To that end, we assume the following.
Assumption 2. We assume Ez [ f (x)) x] = K a positive constant a.s. for any f in which Ez [0. (Y, (X)) = 0 a.s.
Note that convexity of the loss is not required. For example, the assumption holds for L (B. v/) i (y - y²) for any quasi-convex smooth function when Y = x() deterministic function of X. is driven by our interest in what happens at global convergence: [17, 20] show convergence to a global optimum attainable (with suitable initialization strategies) if either (case 1) L is convex in the second variable, or (case 2) 8z5 (y. v/) = O implies L [v. v/) = 0 (recalling L 2 0) and Y = y (X). both cases, one can find reasonable loss functions that satisfy Assumption 2. We also note in both cases, any ʃ with Ez 102[ (Y, [X]) = 0 a.s. is global optimizer.
Our first result indicates that even if one initializes the network at a global optimum and hence there is no evolution with time at the MF limit level, GD training still helps by reducing the output variance V (t) with time the fluctuation level, in the large-width regime.
Theorem 9. Suppose at initialization Ez [ (Y, ú (0, X)) = 0 a.s. Under Assumptions 1 and V* (t) is non non-increasing and V* (t) - as - Dc.
This variance reduction effect continues to hold in the long-time horizon if, instead of initializing at global optimum, we assume to have global convergence at sufficiently fast rate.
1/2 dt < oo for some 8 0. Under
Theorem 10. Assume S g |E j(t, x))x1]] Assumptions and 2, V (t) + 0 as + oc.
These results suggest that the variance reduction effect of GD in MF training is a phenomenon more general than the case of shallow networks with squared loss and finitely many training points in [3]. Specifically it is shown for multilayer networks with a loss that is not necessarily convex with arbitrary training data distribution P. Though we discuss in the context of global optimum as main interest, the theorems apply to any stationary point where Assumption holds.
Let us briefly discuss the proof. For simplicity, consider the context of Theorem 9. Recall from
Theorem 6 the Gaussian process & that the output fluctuation converges to N + oo; we thus

---

Training loss
Fluctuation histogram
Output variance
MC output variance
Outpur variance
10
200
100
10-3
10"1
10
0.2
10-4
10-2
10
10-7
10
10
10a
10*
-0.25
0.00
0.25
10a
10
10'
10*
10'
10
nooration
Itoration
Itaration
Itaration
(a)
(b)
(c)
(d)
(e)
Figure MNIST classification of digits 0 and versus and 9, with full-batch GD training on 100 images. The network has 3 layers, tanh activations and Huber loss. We vary the width N E {50, 100, 200, 400). See Appendix for details. (a): The evolution of the training loss, at a random initialization. (b): Histogram of ( x) ÿ(t, x)) at iteration 10 for a test image x. (c): Output variance E (V) X) (t, x)) (d): Monte-Carlo (MC) output variance, a.k.a. EEzI|G" (t, X) Ps) (e): Similar to (c), but the network is initialized at global optimum.
would like to show E [ (t, x) (P) (P) linear operator defined by
0 as - cc. In this case, one can show that for 인
(z) = -c _ Bz,c
(w Ci) Ow; f(x)
05(0 x)
ai(0, X)
with a constant c > O, we have O,G (t,-) = 21 (t, In particular, 2 is self-adjoint and has non-positive eigenvalues and as such, if at initialization G (0. .) lies in the range of 21, we reach the desired conclusion. This is simple in the shallow case. Indeed note that & (0.-) -) = G" (0.-) by the assumption on initialization of Theorem 9. Assuming 6r (u) u for simplicity, in the shallow case where ğ (0, x) Ec, [w> (0, C1 1) 1i ((w) (0, 1, Ci), x))],
have GV (0,-) is zero-mean
Gaussian with the covariance E (0, x) Go (0,x')l equal to
Cc, [w2 (0, C1, 1) sn (ww (0, (,),,C) ()),,)) Immediately from this simple covariance structure, see that x GD (0, lies in the span 09(0.z)
(0, C1, 1) 9 ((wn (0, ,x))))
of {x + PP (ww1 (0, (1) .)) c1 E supp (Pi)), i.e. the desired conclusion since ( = i (ww1 (0, 1, C1). ..)). In the multilayer case, this is no longer obvious due to the complex covari- ance structure as pointed out in Section 3, and yet interestingly we show that it still indeed holds.
The proof of Theorem 10 leverages on Theorem 9. In particular, implicit in this proof is the fact that the variance reduction effect takes place mostly after global optimum is reached. Here the technical bulk lies in uniform control on weight movements, which is delicate in the multilayer case.
5 Numerical illustration
We give several illustrations in Fig. I via simple experimental setup with MNIST [10] (see the cap- tion and also Appendix G for more details). Fig. 1(a) shows that the network converges to global optimum around iteration 10$ and displays a nonlinear dynamics (which takes the shape of a super- position of two sigmoids). The agreement of the different histogram plots of the output fluctuation for varying widths N in Fig (bb) verifies the existence of a limiting Gaussian-like like behavior, predicted
by Theorems and 6. Fig 1(c) shows that the output variance EEZ [ NNV/SSS (1, X) i(t,, X)) 2| decreases with time quickly after iteration 104, which is predicted by Theorem 10. Fig 1(d) plots the Gaussian component G" in the output variance. Note that after iteration 10 this component no longer moves since a global optimum is reached. Contrasting this plot with Fig ((c), we see the central role of GD training (in particular, the second-order MF limit component R) in reducing the variance. As shown Fig I(e), congruent with Theorem when the network is instead initialized at a global optimum, the output variance is decreasing on the entire period. These plots also highlight an interesting fact (previously mentioned in Section 4): the variance reduction effect takes place mostly after global optimum is reached.
10

---

Acknowledgement
The work of H. T. Pham is partially supported by a Two Sigma Fellowship.
References
[1] Andrea Agazzi and Jianfeng Lu, Global optimality of softmax policy gradient with single hid- den layer neural networks in the mean-field regime, arXiv preprint 118. (2020).
[2] Dyego Araújo, Roberto Oliveira, and Daniel Yukimura, A mean-field limit for certain deep neural networks, arXiv preprint arXiv:19066 1906.00193 (2019).
[3] Zhengdao Chen, Grant M Rotskoff, Joan Bruna, and Eric Vanden-Eijnden, A dynamical central limit theorem for shallow neural networks, arXiv preprint (xXi:2008.00622) (2020).
[4] Lenaic Chizat, Sparse optimization on measures with over-parameterized gradient descent, arXiv preprint arXiv: 1907. 10300 (2019).
[5] Lénaic Chizat and Francis Bach, On the global convergence of gradient descent for over- parameterized models using optimal transport, Advances Neural Information Processing Systems, 2018, pp. 3040-3050.
[6] Cong Fang, Jason D Lee. Pengkun Yang, and Tong Zhang, Modeling from features: a mean-field framework for over-parameterized deep neural networks, arXiv preprint Siiv:2007.014555 (2020).
[7] Eugene A Golikov, Dynamically stable infinite-width limits of neural classifiers, arXiv preprint arXiv:2006.06574 (2020).
[8] Arthur Jacot, Franck Gabriel, and Clement Hongler, Neural tangent kernel: Convergence and generalization in neural networks, Advances in Neural Information Processing Systems 31, 2018, pp. 8580-8589.
[9] Adel Javanmard, Marco Mondelli, and Andrea Montanari, Analysis of a two-layer neural net- work via displacement convexity, arXiv preprint
(2019).
[10] Yann LeCun, Corinna Cortes, and CJ Burges, Mnist handwritten digit database, ATT Labs [Online]. Available:
2 (2010).
[11] Qianxiao Li, Cheng Tai, and E Weinan, Stochastic modified equations and dynamics of stochas- tic gradient algorithms i: Mathematical foundations.. J. Mach. Learn. Res. 20 (2019), 40-1.
[12] Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, and Lexing Ying, A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth, International Conference on Machine Learning, PMLR, 2020, pp. 6426-6436.
[13] Song Mei, Theodor Misiakiewiez, and Andrea Montanari, Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, arXiv preprint 10iv:1902.06011 (2019).
[14] Song Mei, Andrea Montanari, and Phan-Minh Nguyen, A mean field view of the landscape of two-layers neural networks, Proceedings of the National Academy of Sciences, vol. 115, 2018, pp. 7665-7671.
[15] Phan-Minh Nguyen, Mean field limit of the learning dynamics of multilayer newral networks, arXiv preprint 1eXXi:19922.2220 (2019).
[16]
Analysis of feature learning weight-tied autoencoders via the mean meanfield lens, arXiv preprint arXiv:2102.0833 (2021).
[17] Phan-Minh Nguyen and Huy Tuan Pham, A rigorous framework for the mean field limit of multilayer neural networks, arXiv preprint arXiv:2001. (2020).
[18] Atsushi Nitanda and Taiji Suzuki, Stochastic partiele gradient descent for infinite ensembles, arXiv preprint arXiv: 1ww::1712.04444 2.05438 (2017).
[19] Atsushi Nitanda, Denny Wu, and Taiji Suzuki, Particle dual averaging: Optimization of mean field neural networks with global convergence rate analysis, arXiv preprint arXiv:2012. 1547 (2020).
11

---

[20] Huy Tuan Pham and Phan-Minh Nguyen, A note on the global convergence of multilayer neu- ral networks in the mean field regime, arXiv preprint 2XXi:2006.09355 (2020).
[21] Huy Tuan Pham and Phan-Minh Nguyen, Global convergence of three-layer neural networks in the mean field regime, International Conference on Learning Representations, 2021.
[22] Grant Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden- Eijnden,,, Global convergence of neuron birth- dynamics, arXiv preprint arXiv:1902.018:: (2019).
[23] Grant M Rotskoff and Eric Vanden-Eijnden Neural networks interacting particle systems: Asymptotic convexity of the loss landscupe and universal scaling of the approximation error, arXiv preprint arXiv: 5005.00915 (2018).
[24] Alexander Shevchenko and Marco Mondelli, Landscape connectivity and dropout stability of sgd solutions for pver-parameterize neural networks, arXiv preprint arXiv:1912.10095 (2019).
[25] Justin Sirignano and Konstantinos Spiliopoulos, Mean field analysis of neural networks, arXiv preprint PrXiv:1805.011 (2018).
[26]
Mean field analysis of deep neural networks, arXiv preprint arXiv: (2019))
[27]
Mean field analysis of neural networks: A central limit theorem, Stochastic Processes and their Applications 130 (2020), no. 3, 1820-1852.
[28]
Alain-Sol Sznitman, Topics propagation of chaos, Ecole d' été de probabilités de Saint-Flour XIX-1993 Springer, 1991, pp. 165-251.
[29] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma, Regularization matters: Generaliza-
fion and optimization of neural nets v.s. their induced kernel, Advances in Neural Information Processing Systems 32, 2019, pp. 9712-9724.
[30] Stephan Wojtowytsch, On the convergence of gradient descent training for two-layer relu- networks in the mean field regime, arXiv preprint arXiv:2005.13530 (2020).
[31] Greg Yang and Edward J Hu, Feature learning in infinite-width neural networks, arXiv preprint 1.XY::0011.14522 (2021).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The two main contributions described the abstract and the introduction are in Section and Section 4.
(b) Did you describe the limitations of your work? [Yes] We do this in Section 1.
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not expect any immediate negative societal impacts from this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We present them in Assumptions and 2.
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We include the details and link to the code in Appendix G.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide these details in Appendix G.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] We do not report enror bars. Firstly this reduces computational burden for this set of simple experiments. Secondly we instead show
12

---

agreements among plots of different widths N, which convey certain sense of vari- ation that appropriate in the context of (infinite-N) limit laws.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide these details in Appendix G.
4. If you are using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite [10] for our use of MNIST.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "XZSlXbliGrq", "reviewer": "Reviewer_Zo8z", "summary": "This paper studies the mean-field limit of deep fully connected neural networks. The approach is based on extending the analysis of previous work on “neuronal embedding”. Several results are obtained in the considered setting, including 1) the derivation of an ODE system that describes the the asymptotic fluctuations around the mean field limit when there are finite samples; 2) a result showing that the these fluctuations can be characterized by a Gaussian process via a CLT. This extends the analysis of Ref [13] in the paper, which establishes the validity of the mean-field limit, without explicit characterization of the fluctuations around this limit. As an application of these results, the authors studies the evolution of the limiting variance process, where the authors show that under assumptions on the initial condition or the convergence rate, the variance is non-increasing under gradient descent, alluding to a variance reduction effect.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic under investigation is an interesting one, since most existing analysis (be it NTK or MF) usually focus on the limiting process itself, without explicit characterization of the fluctuations and their effects. In this sense, the paper addresses a relevant problem. The theoretical results of the paper appear sound, but I think there is some lost opportunity here, in that the consequences of these results with respect to deep learning are insufficiently explored. As far as I can see, the only consequence of these results explored is Theorem 9 and 10 on the long-time limit of the scaled variance.\n\nBelow I list some explicit questions that the authors may wish to address:\n\n1. Pertaining to theorems 9 and 10: the limit here studied is $N\\rightarrow\\infty$, and then $t\\rightarrow\\infty$, yielding 0 asymptotic variance. However, in practice, we hold $N$ constant, and training gives $t\\rightarrow\\infty$, thus it appears the correct limit to consider when discussing variance reduction is $t\\rightarrow\\infty$ at finite $N$, or at least $N\\rightarrow\\infty$ after taking $t\\rightarrow \\infty$. Does your results apply to the exchanged limits? \n2. One of the main results here is the derivation of the limiting ODEs in Section 3.1,3.2 and the corresponding characterizations of the finite-sample process as a Gaussian perturbation of this. Hence, it may be useful to discuss any insights from these results. For example, are there anything different about the modes of convergence involved with the deep neural network that is different from its shallow counterparts? What do the results in 3.2 tell us about the nature of the fluctuations (R_i)? As written, the paper reads like a collection of technical results with little summary of its implications.\n3. Other issues\n   1. Eq (2) may require more explanation without the need to read ref [13], how is it fundamental? It appears to be a L^2 law of large numbers type of result (with a convergence rate). What does the universal constant $c_*$ capture?\n   2. What is $O_{D,T}$? in definition 1?\n   3. Theorem 9: at initialization, why would one expect the derivative to have zero expectation? Does it only apply when one initialize it at the global optimum. Also, doesn’t Theorem 10 includes Theorem 9 as a special case? Is there a need to present 9 then?\n\n---\n\nI thank the authors for addressing the comments - most of the minor questions have been addressed. While the issue with the exchanged limits cannot yet to addressed, there is some experimental evidence that it holds - if these new results can be considered for review then I am happy to recommend acceptance. The main limitation is, as rightly identified by the other reviewers, is limited motivation for the practical readers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "RS2IoWwtFtl", "reviewer": "Reviewer_SSUu", "summary": "Based on the infinite-width mean-field (MF) limit of multi-layer neural networks (NNs) under the neuronal embedding framework proposed previously, this paper studies the fluctuation of a finite-width multi-layer NN (under random sampling and gradient flow training) around its mean-field limit. The authors show that the fluctuation depends on two components - one being a Gaussian process resulting from the Central Limit Theorem, and the other following a second-order MF equation. The authors also show that under certain conditions, the fluctuations tend to vanish as we take the long-time limit.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: The novelty of this paper is extending the analysis of the fluctuations of NNs from their infinite-width limits from shallow models to multi-layer ones, based on the neuronal embedding mean-field limit.\n\n2. Clarity: The amount of efforts that the authors spend in formalizing all the theory is quite impressive. However, perhaps for most of the audience at NeurIPS, many of the technical details that allow the framework to be rigorous are not of huge interest, but instead make the paper difficult to penetrate. This is partly due to the abstract nature of the subject, though perhaps there is also some room for the authors to improve the readability of the paper.\n\n3. It would help to comment on the connection between the theory proposed in this paper and the one in ref. [3] when restricting to shallow NNs - is there a similarity in their high-level proof ideas as well as a correspondence between the various objects being defined? For example, to argue for the decay of the fluctuations over time, it seems that both papers rely on showing that a fluctuation term lying in the range of a negative semi-definite linear operator. Are there deeper connections?\n\n4. For the audience at NeurIPS, a main take-home message of this paper seems to be that gradient flow has a variance reduction effect for not only shallow but also multi-layer NNs. It would be more relevant and convincing if this theoretical result can be supported by numerical experiments. And overall, I am not very sure if NeurIPS is the best venue for the mathematical rigor of this paper to be fully appreciated.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "More could be said about the limitations and certain assumptions of the theoretical results. For example, how does the neuron sampling method adopted in this paper compare with the common initialization methods in practice?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "IY8QKvWiZep", "reviewer": "Reviewer_dTFu", "summary": "Building upon on the neural embedding framework for multi-layer mean field limit of neural networks [13], the authors derive a system of ODEs that characterized the second order fluctuation of large width network to its MF limit. Interestingly, the authors also show that in the large depth large time limit, variance of the learned output vanishes. The paper contains no experimental results and it is purely theoretical. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Theoretical understanding of neural networks is a central topic in machine learning. The line of research that applies mean field approximation has attracted considerable amount of interest recently. The MF approximation in shallow networks is quite successful. However, the multiple layer setting seems to be quite different and challenging. Building upon on the formulation in [13], the current work derived the second order fluctuation to the mean field limit. Roughly speaking, the authors obtained approximation of several quantities of interest (e.g. outputs, parameters, derivatives of parameters), which are of the following general form \n\\begin{align}\nN^{1/2 }(f_n -f_{\\infty})\\approx  R_{\\infty}\n\\end{align} \nWhere $N^{1/2 }(f_n -f_{\\infty})$ is the scaled correction term and $R_\\infty$ is the residual, which is usually characterized by a system of ODE. \n\nOverall, I think the theoretical results of the paper is quite interesting, in particularly, for readers who are interested in MF approximation of neural networks. However, the current work relies heavily on [13] which I am not familiar with and I could not justify the correctness of its proof. If other reviewers couldn't do so either, I would recommend the authors to submit the paper to a journal of which the reviewers have the interest and bandwidth to go through all the technical details, regardless whether the paper will be accepted by NeurIPS or not. Finally, the MF approach to (multilayer) neural networks has become more theoretical/technical and less practicable, what can be achieved by this direction? Will it lead us to a mathematical foundation for deep learning that explains the fundamental principle behind its great success? A more principle-guided approach for architecture design/ optimization? Or is it just a kind of well-defined mathematical formulation of pure theoretical interest?  \n\n\nOther comments/ suggestions.\n\nMy biggest question is: how relevant is the current formulation to practice? In addition, what audiences are you targeting? People with familiarity of multiple layer mean field or just shallow networks or none? It seems to me that the authors assume the readers have some familiarity with [13]. \n\nSince this is a heavy theory paper, I would appreciate the authors to provide empirical support to major claims made in their paper.  \n\n\nNotation in (1) is a bit dense for readers who are not familiar with [13].  What is the connection to practical networks, what parameterization is used? Standard (e.g. He initialization), NTK style parameterization, or the recent feature parameterization claimed in Yang&Hu. From the expectation operator defined on the second line of (1), it seems like there is a $1/ N_i$ normalization factor (rather than the standard $1/\\sqrt {N_i}$) which is different from all parameterization approaches mentioned above. Authors should connect (1) to practically used networks in order for readers to understand if this is closely related to practical networks or it is just a mathematical formulation(not necessary relevant to practice).  \n\nBelow 101: lowercase $n_i$ should be uppercase $N_i$? \nSecond Equation below 102: update for $\\omega_i$, should there be a $1/N_{i-1}$ factor in the updates? Or the updates are rescaled by a layer wise learning rate which is suggested in line 104? \n\nEq 2; what is $J$ under the $\\mathbb E$ ? \n\nYang&Hu Feature Learning in Infinite-Width Neural Networks\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations of the work. As far as I know, this is a pure theory paper and has no negative societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the fluctuation of a finite-width multi-layer neural network around its mean-field limit, based on the infinite-width mean-field limit under the neuronal embedding framework. While referees have concerns on the practicality of the parametrization, they find the result interesting and novel, especially with the newly added numerical experiments. The meta-reviewer recommends acceptance of the paper as a poster. 
Decision: Accept (Poster)