Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training
Authors: Huy Tuan Pham
Phan-Minh Nguyen
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning theory
mean field limit
finite-width correction
fluctuation
implicit bias

Abstract: The mean field theory of multilayer neural networks centers around a particular infinite-width scaling, in which the learning dynamics is shown to be closely tracked by the mean field limit. A random fluctuation around this infinite-width limit is expected from a large-width expansion to the next order. This fluctuation has been studied only in the case of shallow networks, where previous works employ heavily technical notions or additional formulation ideas amenable only to that case. Treatment of the multilayer case has been missing, with the chief difficulty in finding a formulation that must capture the stochastic dependency across not only time but also depth.

In this work, we initiate the study of the fluctuation in the case of multilayer networks, at any network depth. Leveraging on the neuronal embedding framework recently introduced by Nguyen and Pham, we systematically derive a system of dynamical equations, called the second-order mean field limit, that captures the limiting fluctuation distribution. We demonstrate through the framework the complex interaction among neurons in this second-order mean field limit, the stochasticity with cross-layer dependency and the nonlinear time evolution inherent in the limiting fluctuation. A limit theorem is proven to relate quantitatively this limit to the fluctuation realized by large-width networks.

We apply the result to show a stability property of gradient descent mean field training: in the large-width regime, along the training trajectory, it progressively biases towards a solution with "minimal fluctuation" (in fact, vanishing fluctuation) in the learned output function, even after the network has been initialized at or has converged (sufficiently fast) to a global optimum. This extends a similar phenomenon previously shown only for shallow networks with a squared loss in the empirical risk minimization setting, to multilayer networks with a loss function that is not necessarily convex in a more general setting.

Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training
Huy Tuan Pham
Phan-Minh Nguyen The Voleon Group
Department of Mathematics, Stanford University
Abstract
The mean field theory of multilayer neural networks centers around a particu- lar wifdh scaling, in which the learning dynamics is shown to be closely tracked by the mean field limit. A random fluctuation around this infinite-widhh limit is expected from a large-width expansion to the next order. This fluctuation has been studied only in the case of shallow networks, where previous works em- play heavily technical notions or additional formulation ideas amenable only to that case. Treatment the multilayer case has been missing, with the chief diffi- culty in finding formulation that must capture the stochastic dependency across not only time but also depth.
In this work, we initiate the study of the fluctuation in the case multilayer net- works, any network depth. Leveraging on the neuronal embedding framework recently introduced by Nguyen and Pham [17], we systematically derive a system of dynamical equations, called the second-order mean field limit, that captures the limiting fluctuation distribution. We demonstrate through the framework the com- plex interaction among neurons in this second-order mean field limit, the stochas- ticity with cross-layer dependency and the nonlinear time evolution inherent in the limiting fluctuation. A limit theorem is proven relate quantitatively this limit the fluctuation realized by large-width networks.
We apply the result to show stability property of gradient descent mean field training: the large-wide width regime, along the training trajectory, progressively biases towards a solution with "minimal fluctuation" (in fact, vanishing fluctua- tion) in the learned output function, even after network has been initialized has converged (sufficiently fast) to a global optimum. This extends a similar phenomenon previously shown only for shallow networks with a squared loss the empirical risk minimization setting, to multilayer networks with loss func- tion that not necessarily convex a more general setting.
Introduction
Recent literature has witnessed much interest and progresses in the mean field theory of neural networks. In particular, it is shown that under a suitable scaling, as the widths tend to infinity, the neural network's learning dynamics converges to nonlinear deterministic limit, known the mean field (MF) limit [14, 17]. This line of works starts with analyses the shallow case under various settings and has led to number of nontrivial exciting results [18, 14, 5, 23, 25, 9, 22, 19, 29, 24, 30, 12, 1, 16]. The generalization to multilayer neural networks, already much more conceptually and technically challenging, has also been met with serious efforts from different groups of authors, with various novel ideas and insights [15, 17, 20, 2, 26, 6].
Since the MF limit is basically first-order infinite-wide approximation of the neural network, it is natural to consider the next order term in the expansion for a more faithful wiith-width description.
"The author ordering is randomized.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

This leads to the consideration of the fluctuation, up-scaled appropriately with the widths, around the MF limit. On one hand, this fluctuation should display MF interactions among neurons. On the other hand, it is random due to the inherent stochasticity in the finite-width network which, for instance, is randomly initialized and hence induces randomness in the interactions among neurons.
For shallow networks, [27, 23,3] has identified the limiting fluctuation in the form of a time-evolving signed measure over weights, leading to a central limit theorem (CLT) in the space of measures. In particular, [27] pinpoints the fluctuation via a compactness argument in an appropriate measure space. Avoiding the heavy technicality, [3] realizes neat trick, specific to shallow networks, where quantities of interest are described via push-forwane of the initialized values of the weights.
There has been no similar attempt for the multilayer case, which faces major obstacles in formu- lating the limiting fluctuation. Firstly, existing formulations of the MF limit already move away from working with measures over weights [15, 17, 26, 6], unless restricted conditions assumed [2]. This is due to the complexity of MF interactions: the presence of middle layers brings about simultaneous actions of multiple symmetry groups. We face the same complexity when formulating the fluctuation. Secondly, unlike shallow networks, the fluctuation in multilayer networks displays stochastic dependency not only through time but also across layers. Specifically the cross-layer dependency is propagated by both forward and backward propagation signals, at any time instance.
Contributions. We tackle the challenge by leveraging on the neuronal embedding framework, recently proposed by [17]. An important concept that supported by the neuronal embedding the sampling of neurons. We use this concept to formulate the limiting fluctuation via a decomposition into two components with different roles in Section 3. One component is a random process, which encodes CLT-like stochasticity in the fluctuation of the sampled neurons around the MF ensemble. The other component, named the second-order mean field limit, is a system of ordinary differential equations (ODEs), which displays MF interactions in the deviation of the neural network under gradient descent (GD) training from the sampled neurons.
This decomposition is an innovation of the work. Our formulation enjoys the generality brought about by the neuronal embedding framework without restrictive assumptions (e.g. i.i.d. initialization assumption made in [2, 26)) and faithfully describes the expected stochastic dependency across time and depth. We prove a limit theorem that establishes the connection with the fluctuation in finite- sized networks (Theorems 5 and 6). Unlike [27, 3], this result is quantitative.
Using this formulation, in Section 4, we show trajectorial stability property in a large-width mul- tilayer network, particularly a variance reduction effect: GD training traverses a solution trajectory that reduces and eventually eliminates the (width-correeeed)) variance of the learned output function. That is, a bias towards "minimal fluctuation". This holds even if the network is initialized at, or if it is trained to converge sufficiently fast to, a global optimum (Theorems 9 and 10). The same phenomenon has been shown in the shallow case [3], which requires a squared loss in the empirical risk minimization setup. As demonstrated by our result, it not an isolated phenomenon and can indeed hold for a neural architecture where MF interactions are more complex, loss function that is not necessarily convex and training setup unrestricted to finitely many training data points.
Limitations and potential extensions. Let us finally mention a few limitations. Our work consid- ers fully-com colle-coneeeeed multilayer networks trained with GD, setup much less general than [17]. We expect certain extensions in this direction are doable. We also do not treat stochastic GD and hence disregard the stochasticity of data sampling, unlike [27]. Given the broader literature this subject (e.g. [1IJ, this extension is foreseeable. Here we focus instead on the stochasticity that is inherent in the interactions among neurons, which is the more interesting aspect of neural networks.
Our result on the output variance is specific to unregularized training, unlike [3], and also requires a sufficient global convergence rate. While there have been several proven global convergence guar- antees for multilayer networks [17, 21, 20], understanding of the convergence rate is still lacking. Even in the shallow case, global convergence has been studied only for type of -ansity-innuucin regularization [5, 4]. Unless the convergence rate for multilayer networks is generally perilous (an unlikely scenario in light of the experiments in [15]),
result is expected to be relevant.
Our development is specific to the MF scaling. This scaling allows for nonlinear feature learning. unlike the NTK scaling [8]. While there are other scalings that also admit a certain sense of feature learning [7, 31], the standard parameterization to degenerate into NIK-like behaviors, which are not expected of practical butteebbtt-argge--ee width
practice - in the infinite-width limit - known
2

---

neural networks [13, 31]. In other words, all infinite-width scalings that display feature learning are only proxies of practical networks. This limitation motivates finite-width studies as we pursue here.
Notations. We use K to denote generic constant that may change from line to line, and similarly K2 for finite constant that may depend on some constant u. For simplicity, for L the network depth, we write K in place of Kr. For E (resp. E) being the expectation, we use V and C (resp. V and C) for the variance and covariance. We write Ez for the expectation w.r.t. data Z (X, Y) P. We write f f to denote the partial derivative w.r.t. the i-th variable of
2 Background on the MF limit for multilayer networks and assumptions
We describe the necessary backgrounds based on (and simplifying) the work [17]. This section also introduces several important notations, well as the assumptions for our study of the fluctuation.
Multilayer network. Following [17], we consider I.-layer fully-connected neural network with widths (Noligz. (for N = 1):
y (1, yr. (Hr (t, 1,,;),
H. (t, = E, [w; (t. 11 (:) P: (H,-) 1 (+,,,1))))) Js E [Ni), == L,..,,,.
H1 (f, (W1 (t,, 1, (c)),,, 2) f1 [Mi]
in which X Rd is the input, the weights are Wi (t, W, Rd and Wi (t, W, = R. ps: R + IR is the activation and E T R>0 the continuous time. Here we reserve the notation Jj for the random variable Jj ~ Unif ([NV.) and we write E, denote the expectation w.r.t these random variables. For convenience, we take jo {1} and No = 1.
Given initialization w, ..,, train the network with gradient descent (GD) w.r.t. the loss L: x R + R20 and the data Z (X,Y) E X x R drawn from a training distribution P:
By (I. ]. i [4],
Oew; (i, = Ez (2 (t,
Ow, (ii-1,5))
in which define
1 H (t, 1, x)):
i= L, Wi+1 ((.1) (H, ((.)))))) i< L,
09 (1)
0y (t,
0H, (ja)
E,
ÖH;+1 (J+t1)
Łÿ (t,
fi-1 (H₂-1 (1, (1)),)),)),
i>1,
0ÿ (t,
HH. (ii)
Ow: (fi-1.
0ÿ (t,x)
F,
1=1.
HH, (i1)
One may recognize that these "derivative" quantities are detined in a perturbative fashion; for in- stance. ()(w.) represents how (t, 2) changes (rescaled by widths) one perturbs H, xt)
aH.(1)
Mean field limit. The MF limit is defined upon a given neuronal ensemble F),
I (2,, F,, P.) (in which 2g = 22 = (11)), which is product probability space. We reserve the notation C, for the random variable C; P. and we write Eg denote the expectation w.r.t. these random variables. The MF limit that is associated with the network (1) is described by the evolution of (w; (1, "eeuu given by the following MF ODEs:
OG (t. X)
Oaw, (t, c.1.)))) = EEz 02L (t,
Ow; (Q_1,))
i [4], E 9. G-1 E Q4-1.
where w. T x 0, Q - W, and & T. Here we define the forward quantities:
1 (1,2x) = Pr. (Hr, (1, 1, 1,2)), H; (t, G,,z) = EEc wwj (t, C,-1, ci) p (Hi-1 (t, Ci-1, i= L...,, 2,
H1 ())) = (wi (t, (1),,,,))
3

---

and the backward quantities:
77 1 (t. 1,
i= L, Wit1 (t, e. C4+1) (H) (t, (t,c,,))
0g (t,
dÿ (t,z)
0H; (ci)
Bc
ic L.
0HH+1 (C(+1)
Oi (1,
Pi-1 (H;- (1, ()), i>1, 1
Og (t,z)
0H; (c:
Ow, (G)
0ÿ (t,x)
0H1 (c1)
The evolution with time & the MF limit is generally complex due to the nonlinear activations.
Sampling of neurons, neuronal embedding, and the neural net-MF limit connection. A pri- ori there is no connection between the MF limit and the neural network (I); they are two separate self-contained time-evolving systems, despite the similarity in their definitions. To formalize their connection as in [17], we describe the neuron sampling procedure. In particular, we independently sample (C; (.e)), Miidd P. for ,.... L.. The samples C, (jx) should be thought of as "sampled neurons", " Let use E denote the expectation w.r.t. the sampling randomness. Al- though [17] considers more general sampling rule, this suffices for our purposes.
Now suppose that we are given functions w. Ω4-1 x Sr - W, and set the initializations for the neural network and the MF limit:
W; (0,
ua (C:-1 (i-1) C. (i:)), uy (0,-,)) u (.,-) e [4].
Then lets them evolve according to their own respective dynamics. The sampling of neurons hence allows to couple them on the basis of the tuple (2, P, ( [w]] <<(L) known as the neuronal embedding, the key idea in [17]. Note that this does not pose major limitation on the type of initializations for the neural network (I); indeed it allows both i.i.d. and non-i.id. initializations
since we have freedom to choose the neuronal embedding, as studied in [17, 20).
Once the coupling is done, [17] obtains the following result, which realizes the connection in math- ematical terms. For any finite terminal time T T, we have 2:
max E. sup w, (t, J4) wy (t, C. (Jr-1) G. (J.)) 1/2 4
1ST
with high probability, where C, 0 is universal constant, Nuin min No, Nuax maxie()] N2 and hides the dependency on T. L and log Nmax. This result is akin to the law of large numbers (LLN). In words, it says that for large (ideally infinite) widths, the evolution of the network (1) can be tracked closely by the MF limit. This result is fundamental and gives a useful suggestion that is to study the neural network via analyzing the MF limit. For example, [17] uses it to study the optimization efficiency of the neural network.
The sampling of neurons is inspired by the propagation of chaos argument [28], but is not mere proof device. Indeed, in this work, we again make a crucial use of this sampling in the formulation of the limiting fluctuation (Section 3.3).
Set of assumptions. We consider the following assumptions for the rest of the paper:
Assumption 1. We make the following assumptions:
(Regularity) yps K -boundedd for i E [L -1]; v/ and yi are K --bounded and K K-Lipschitz for i E [T]]; 0zL and 8.f L are K K-bounded and K K-Lipschitz in the second variable; [X| S K almost surely (a.s.).
"Sub-Gaussian initialization) supm21 -1/2BBB Ioe (C.-1, [n]
1m/1//m 1/m
K for any i [L].
(Measurabiliry) (P:) is separable for any i E [Z]. (Constant hidden widths) N1 = Nr-1 = N (with ideally N + oo).
An example for the regularity assumption is & (u) u the identity, s; tanh for [L 1] and L (u), (2) is smooth version of the Huber loss. It is noteworthy that L does not need to be
While [17] treats stochastic GD in discrete time, this result implicit in the proof.

---

convex. The initialization assumption is also mild; it allows most common i..d. initializations as well as non-i.id. schemes in [17. 20]. These assumptions satisfy the conditions these works. Measurability assumption is a technical condition needed for well-defined-ness of the fluctuation; it not conceptually restrictive and can accommodate results in [17. 20].
The constant width assumption aligns with our interest in the infinite-width regime. Our develop- ment can be extended easily to the proportional scaling, where N, [o,N] for some constants Pl,..., PL-1 > 0 with N + oc. These scalings are relevant to practical setups and also to yield interesting results. Specifically once some widths are much larger than others, we expect the fluctu- ation to be dominated by a subset of layers.
Limit system for the fluctuation around the MF limit
We describe our formulation for the fluctuation. As introduced, it is composed of two components: stochastic process that induces Gaussian CLT-like behavior, and another process - called the second- order MF limit that signifies the MF interactions at the fluctuation level. This is done on the basis of the sampling of neurons, on top of the neuronal embedding, introduced Section 2.
We introduce the first component in Section 3.1 and then the second-oder MF limit in 3.2. Together the two components are connected with the fluctuation around the MF limit realized by a large- width network in Section 3.3, to give firstly the analog of (2) and secondly a limit theorem on the fluctuation distribution. We end with discussion in Section 3.4. Proofs are deferred the appendix.
3.1 Gaussian component G
Recall the sampling of neurons described in Section 2. Given the MF limit, we define the following random quantities:
(t, (HL 1t),
H. (t, E, wi (t, (1)_)
i= L,...,,
in (f, = (wi (1),
(ltr (,1,
i i= L,
05 (t) OH, (c))
Oy (4xx)
F2,
Wi+1 (1.6, Cin (f
i i<l,
|fli+1 (Ci+1 (J++1))
aī (t)
Pp--1 (fti-1 (1)))))) i>1. 1,
0ÿ (1,2x)
0H, (ci)
ow; (c:-1.
θÿ (t,z) Ir
1=
0H, (c1)
These quantities are analogues of the forward and backward MF quantities, but with Ec; being replaced by an empirical average over {Ca (i))
Now we define
(1,2)) V (4, - (1,x)),
TFe (1, (1,0)1,,6,))== = ( Oī (1,
Oy (t,
i i= 1=
Ouc (G) Ow; (G-1,
and let & denote the collection of these functions. Notice that the randomness of Ô is induced by the samples C.(ji); thus we write the expectation w.r.t. G via E. Intuitively & is the fluctuation of Gaussian process. We show
the sampled neurons around the MF limit and thus should converge that this is indeed the case in the following mode of convergence.
Definition 1. Given joint distributions over a sequence (in N) of functions G.,N : T x Ω x Ω4 x X + W. for i E [L], we say that GN = [Gr.N converges G-polynomially in moment (as N + oo) to G if for any finite collection in l of square-interrable fe T Q4(0)-1 Q4(0)) x X
W that is continuous in time and integers ae, 3e 2 1,
II, (fe. Tre No
E II, (fe. Goe 9r 1| =0 suup max 1 1-//8 100 100)
sup (0) ve
5

---

for D Ee ae and constant terminal time T. Here we define:
(S,)), = Ez,c [f (t, C,, for any two such functions f and h, and |1.f122 () As convenient convention, if x Og x Qt + W is tested against GI,N, we define (S, U,Gi,,), and ||/ using Euclidean inner product and the Euclidean norm.
Ci-1, Cx,
Note that this mode of convergence gives a quantitative rate in N.
Theorem 2 (Gaussian component G). For any terminal time T 2 0, G converges G-polynomially in moment to {G", G. i [L]]. which a collection of centered Gaussian processes.
The covariance structure of G in Theorem 2 is explicit but lengthy; we refer to the appendix for full details. Here let us examine the variance of G², given recursively by:
U2 (1, 02, 47) Ve [w2 (t, Ci, 02) pi (II1 (1, (1))),
us ()))) = Be wu (1, Ci-1, 4/11 (Hi-1 (1, Ci-1, 0-1 (1, C,)1, + +C [w, (t, C,) Pi-1 (H (t, C)),, i= ....., L.
G (1,x)1 = le' (Hz (t, 1,)), un (t, 1,x).
The recursive structure shows that G compounds the stochasticity propagated forwardly through the depth of the multilayer network. From the appendix, a similar observation can be made in the backward propagation direction.
3.2 Second-order MF limit
We introduce system ODE's for a dynamics that represents MF interactions at the fluctuation level. As shall see later, this system gives limiting description of the the deviation of the neural network from the sampled neurons under GD training.
Let us denote by G collection of functions GV T R. G. i (L.) For p 2 1, let
TxQ:-1 X
W, for S X
G - (G: ||G|| 8 VT 2),
|| sup Σz,c tT
| (t,X)
Σ i=1
G (t, C x)
with ||G] = || We define processes R; : ::g9 T x Ω Ω2 W, for i [L] as follows.
The processes R2 are initialized at R₃ (,,0,--) = for all i, and for each fixed G G, solve the differential equations:
0, (G,4,
-Ez
Oy (t. X)
C2 (C*())))))))))
(t,))
Σ Ec Prr (G,t,Cr-1,C)
06 (t,, X) Ow, (Cr- C+) 1)
Owi (ci-1,
0Φ (t,
Ez 02C (Y, (t, X)) Ec Σ
R2 (,)
ow, (a, b) Ow; (Gi-1, ci) a= -
Ez 02 (Y, (t, X)) G (t,0--1 c, x)
(3)
8g(, Here Tw, (o)0 is defined in a perturbative fashion similar to those Section and is hence self-explanatory: we give the full explicit definitions in the appendix. We shall write R, (G,t) R. (G,t,-,) for brevity. Let It denote the collection { (eet)
Theorem 3. Under Assumption 1, for any > O and G G with ||G < <0, there exists unique solution i. H R, (G E T.2 (P:-1 x P:) which is continuous in time. Furthermore, for each f, R R t) a continuous linear functional G.
The process R is called the second-order MF limit for reason shall see in Section 3.4.

---

Remark 4. Care should be taken in the derivation of the "second derivative" quantity in Eq. (3). To illustrate, consider a simplified problem of taking perturbation w.r.t. wo (a, b) of the quantity
g (wy (G-1, Gi) BC [fi (wy (C--,,))) IEc [Ja (m, C,)),, [fs (wy (C--1, C.)II) (c)
For instance, when i, we have:
0g Ci)
R₁ (G, b)
Sw, (), a==1-1, bc
R, (G, 01 (c-1. R2 (G,1, C--,,,) 02g (,) 0f (my, (C)-,,
+ Ru (G,4, 0gg (G-1,4)005 (w; (G-1, C.)) + R. (G, Ci-1, Ct) 0ag (c- c,)s (C-1,, C.))
3.3 Fluctuation around the MF limit via sampling of neurons
We are interested in a characterization of the following quantity, which represents the deviation of the weights of the neural network under GD training from the sampled neurons:
R3 (1,)i = VN (ws (t,Ji-1,5k) - wi (1, C--1 (ii-1), C, (i:))).
Let R denote the collection [Ri)ce[a) We now state our first main result of the section. Theorem 5 (Second-order MF). Under Assumption 1, we have, for any t TT
EE, R3 RRC.
SKT/N.
KT/N.
Upon the correct choice G, one deduces from the theorem the second-order expansion:
w, (t, x wy (t, C.-1 (ii-1), C, (i))
(i--1,
where the first-order term is directly from Eq. (2). Importantly this relation is the basis of the sampled neurons {Ci (j:) ja E [N,], i E [/]]- The dynamics of R2 at the i-th layer is depen- dent on all (EL) and GV, exemplifying the cross-layer interaction of fluctuations in multilayer networks.
Although Theorem involves G. as shown in the derivation of Section 3.4, we actually do not utilize the specific structure of G. Incidentally we study the limiting structure of G separately in Theorem 2. other words, our decomposition of the fluctuation via G and allows relatively independent treatments of the two components (which have different natures, to be discussed in Section 3.4).
Combining the two theorems, we obtain the following CLT for the limiting output fluctuation.
Theorem (CLT for output function). Under Assumption 1, the fluctuation (y(((1, ((1,))) converges weakly to the Gaussian process & indexed by T x X.
Oi(t, x)) Σ Ec R.I.C,,CC-II.C) Sw, CC)
G (t,
GG,,t,
where G is the Gaussian process described in Theorem 2. Specifically, for any integer m 2 1, t < T, and 1-bounded h + R,
EEz [XX) - O(t, xI)) EE (XX)((()))) S Kr - N118400(1)))
We note in passing the technicality we devise to prove Theorem 6. As noted, Theorems 5 and separate treatments of R and G. but recall we are interested in t(G,t). This necessitates a way to describe them jointly. We do so via the following notion of convergence, extending Definition
Definition 7. Suppose we are given sequence (in N) of functions G, Tx X2; x + W; for E [L] such that GN (G.,N hie(r) converges G-polynomially in moment to G. Given further RAN T x 94-1 x Qt x + W, defined jointly with G and Ry = (i.. heez), we say that (Rx, Gx) converges G-polynomially in moment and R-linearly in moment (as N o0) (R,
7

---

if for any fe, he that are T x P1(0)-1 x Pice x X - Way mappings and continuous in time and any inteeers ae,, 3r 2 L.
IE II.
(Se, Goe 9r No
II. Reco).m)cc 10: [(e))
-E II. Woyul Ihe RuCs)
sup (e)) Ve
=00( sup max [[fell sup max | (
1ST 8
tST
for D Ee Bear Be.
Proposition 8. Assume that G converges G-polynomially in moment G {G², G,,, i [4]] Then (R(G.))G)) converges G-polynomially in moment and R-linearly in moment to (R(G, G)
3.4 A heuristic derivation and discussion
heuristic argument. We wish to give a heuristic to derive the relation between R, and R. R.(G, (G, t), ic. Theorem 5. Suppose we look at the i-th layer:
O,R, (t,ji-1,j()
VNO.w, (t, - VNOW., (t, C. (
06 Xt)
0y Xt, ow: (i)
VNEZ O2C (Y, (t) Ow; (C.- (Ji-1) C, (f:))
-NEE (r, (t,
VN E2
02f (3 (Y, (1, - 02h (Y,9 (1, ())
O5 X)
Bryy (Ci-1 (fi-1), C. (j+))
dÿ (t)
0ÿ (t,, X)
Ez 0zL 02 (t, VN
(4)
ow; (C,-1 (i-1)) C, (ji)
Ow, (ji-1.
Let us zoom into A₁, with some care not to eliminate the fluctuation of interest:
A =
(0 (t,,))
= CY (t. (-6x (t, VN (f y (t,x))
Now observe that ÿ (t) y (t, X)) is difference between the sampled neurons and the neural network, and hence one should be able express it in terms of R:
VN (Ū X) & X))
Oi (1, X)
E,
(1 - w, (t,J
Ow, (C+-1 (Jr-1), (J+))
0ī (1,
Σ
E Ow, (C,-1 (Jr-1), Cr (J+)) R, (t, Jr,,,,1,
Furthermore the limit N - co, we expect from the LIN, for a test function f.
(5) As such, one identify the term that involves A₁ in Eq. (4) for O,R, (1,ji-1, with the first term in Eq. (3) for O R₁ (.). One can derive similarly for the rest of the terms. With these, one arrives at R,R, (1)) Re O, t, C.-1 (ji-1), C, (i-)). Recalling that R, (0,-,:) = R- 0, -). -). one arrives the conclusion of Theorem 5.
E, [f (C C, (J+)) Ec [f (C,-1,
We make two comments. Firstly, the sampling of neurons makes transparent the derivation of R as the limit of R: it allows compare one-to-one "neuron" of the former neuron of the latter. This demonstrates an advantage of the neuronal embedding framework, which easily accommodates the sampling of neurons in the multilayer setup. We also see that the only involvement of & this derivation is via its definition.
Secondly, arises (by replacing R) at steps that invoke (5). This LLN-type nature signifies MF interactions among neurons at the fluctuation level, and hence the name second-order MF limit. In contrast, the component & displays Gaussian CLT-type nature, evident from its definition.

---

Technical difficulties. So far we have seen R captures the difference between the neural network and the sampled neurons. Recall the other component in the fluctuation is the process G, which cap- tures the v scslee difference between the sampled neurons and the MF limit. Taking into account the whole system of fluctuation, we note major technical subtlety: the aforementioned two differ- ences captured by R. and G share the same source of randomness {C, (j+)= j: E [M.] i i [4]].
We mention two particular complications that arise from this subtlety. The first complication is how one should define the process R in relation with G. given that R is meant to be the infinite-width limit of R while R and G are stochastically coupled. Our solution is to let t + R(G, t) defined for any G E G, not restricted to only G. This streamlines the definition of the second-order MF limit and allows to separate our treatments of R and G, as evident from Theorems and 2.
The second complication lies with Eq. (5). In fact, to arrive at the desired conclusion, we require the function f to depend on G. This is because the main object of interest is R[G, t), even though we have treated R and G separately. In a nutshell, f shares randomness with C, (jr) and Cr-1 (ir-1), jr E [[Vr]. Jr-1 E [NV,-1]. The random variable on the left-hand side of Eq. (5) is therefore complex, and it is highly questionable whether this equation should hold. The analysis becomes delicate; without taking into account this shared randomness, the derivation would he a mere heuristic. As present in the proof, we verify this equation for relevant set of functions f.
4 Asymptotic variance of the output function
We study the following width asymptotic variance quantity:
V (t) lim EEZ || (y X) x)) P lim EE [E2[|VN((((t)) - Ets X))
-000
N
The second equality holds and the limits in NN exist by Theorems and 6. We would like to under- stand (t) in the long-tim horizon, and specifically in situation of considerable interest where the MF limit converges to a global optimum as t + co. To that end, we assume the following.
Assumption 2. We assume Ez [ f (x)) x] = K a positive constant a.s. for any f in which Ez [0. (Y, (X)) = 0 a.s.
Note that convexity of the loss is not required. For example, the assumption holds for L (B. v/) i (y - y²) for any quasi-convex smooth function when Y = x() deterministic function of X. is driven by our interest in what happens at global convergence: [17, 20] show convergence to a global optimum attainable (with suitable initialization strategies) if either (case 1) L is convex in the second variable, or (case 2) 8z5 (y. v/) = O implies L [v. v/) = 0 (recalling L 2 0) and Y = y (X). both cases, one can find reasonable loss functions that satisfy Assumption 2. We also note in both cases, any ʃ with Ez 102[ (Y, [X]) = 0 a.s. is global optimizer.
Our first result indicates that even if one initializes the network at a global optimum and hence there is no evolution with time at the MF limit level, GD training still helps by reducing the output variance V (t) with time the fluctuation level, in the large-width regime.
Theorem 9. Suppose at initialization Ez [ (Y, ú (0, X)) = 0 a.s. Under Assumptions 1 and V* (t) is non non-increasing and V* (t) - as - Dc.
This variance reduction effect continues to hold in the long-time horizon if, instead of initializing at global optimum, we assume to have global convergence at sufficiently fast rate.
1/2 dt < oo for some 8 0. Under
Theorem 10. Assume S g |E j(t, x))x1]] Assumptions and 2, V (t) + 0 as + oc.
These results suggest that the variance reduction effect of GD in MF training is a phenomenon more general than the case of shallow networks with squared loss and finitely many training points in [3]. Specifically it is shown for multilayer networks with a loss that is not necessarily convex with arbitrary training data distribution P. Though we discuss in the context of global optimum as main interest, the theorems apply to any stationary point where Assumption holds.
Let us briefly discuss the proof. For simplicity, consider the context of Theorem 9. Recall from
Theorem 6 the Gaussian process & that the output fluctuation converges to N + oo; we thus

---

Training loss
Fluctuation histogram
Output variance
MC output variance
Outpur variance
10
200
100
10-3
10"1
10
0.2
10-4
10-2
10
10-7
10
10
10a
10*
-0.25
0.00
0.25
10a
10
10'
10*
10'
10
nooration
Itoration
Itaration
Itaration
(a)
(b)
(c)
(d)
(e)
Figure MNIST classification of digits 0 and versus and 9, with full-batch GD training on 100 images. The network has 3 layers, tanh activations and Huber loss. We vary the width N E {50, 100, 200, 400). See Appendix for details. (a): The evolution of the training loss, at a random initialization. (b): Histogram of ( x) ÿ(t, x)) at iteration 10 for a test image x. (c): Output variance E (V) X) (t, x)) (d): Monte-Carlo (MC) output variance, a.k.a. EEzI|G" (t, X) Ps) (e): Similar to (c), but the network is initialized at global optimum.
would like to show E [ (t, x) (P) (P) linear operator defined by
0 as - cc. In this case, one can show that for 인
(z) = -c _ Bz,c
(w Ci) Ow; f(x)
05(0 x)
Oi(0, X)
with a constant c > O, we have O,G (t,-) = 21 (t, In particular, 2 is self-adjoint and has non-positive eigenvalues and as such, if at initialization G (0. .) lies in the range of 21, we reach the desired conclusion. This is simple in the shallow case. Indeed note that & (0.-) -) = G" (0.-) by the assumption on initialization of Theorem 9. Assuming 6r (u) u for simplicity, in the shallow case where ğ (0, x) Ec, [w> (0, C1 1) 1i ((w) (0, 1, x))],
have GV (0,-) is zero-mean
Gaussian with the covariance E (0, x) Go (0,x')l equal to
Cc, [w2 (0, C1, 1) sn (ww (0, (,),,C) ()),,)) Immediately from this simple covariance structure, see that x GD (0, lies in the span 09(0.z)
(0, C1, 1) 9 ((wn (0, ,x))))
of {x + PP (ww1 (0, (1) .)) c1 E supp (Pi)), i.e. the desired conclusion since ( = i (ww1 (0, 1, C1). ..)). In the multilayer case, this is no longer obvious due to the complex covari- ance structure as pointed out in Section 3, and yet interestingly we show that it still indeed holds.
The proof of Theorem 10 leverages on Theorem 9. In particular, implicit in this proof is the fact that the variance reduction effect takes place mostly after global optimum is reached. Here the technical bulk lies in uniform control on weight movements, which is delicate in the multilayer case.
5 Numerical illustration
We give several illustrations in Fig. I via simple experimental setup with MNIST [10] (see the cap- tion and also Appendix G for more details). Fig. 1(a) shows that the network converges to global optimum around iteration 10$ and displays a nonlinear dynamics (which takes the shape of a super- position of two sigmoids). The agreement of the different histogram plots of the output fluctuation for varying widths N in Fig (bb) verifies the existence of a limiting Gaussian-like like behavior, predicted
by Theorems and 6. Fig 1(c) shows that the output variance EEZ [ NNV/SSS (1, X) i(t,, X)) 2| decreases with time quickly after iteration 104, which is predicted by Theorem 10. Fig 1(d) plots the Gaussian component G" in the output variance. Note that after iteration 10 this component no longer moves since a global optimum is reached. Contrasting this plot with Fig ((c), we see the central role of GD training (in particular, the second-order MF limit component R) in reducing the variance. As shown Fig I(e), congruent with Theorem when the network is instead initialized at a global optimum, the output variance is decreasing on the entire period. These plots also highlight an interesting fact (previously mentioned in Section 4): the variance reduction effect takes place mostly after global optimum is reached.
10

---

Acknowledgement
The work of H. T. Pham is partially supported by a Two Sigma Fellowship.
References
[1] Andrea Agazzi and Jianfeng Lu, Global optimality of softmax policy gradient with single hid- den layer neural networks in the mean-field regime, arXiv preprint 118. (2020).
[2] Dyego Araújo, Roberto Oliveira, and Daniel Yukimura, A mean-field limit for certain deep neural networks, arXiv preprint arXiv:19066 1906.00193 (2019).
[3] Zhengdao Chen, Grant M Rotskoff, Joan Bruna, and Eric Vanden-Eijnden, A dynamical central limit theorem for shallow neural networks, arXiv preprint (xXi:2008.00622) (2020).
[4] Lenaic Chizat, Sparse optimization on measures with over-parameterized gradient descent, arXiv preprint arXiv: 1907. 10300 (2019).
[5] Lénaic Chizat and Francis Bach, On the global convergence of gradient descent for over- parameterized models using optimal transport, Advances Neural Information Processing Systems, 2018, pp. 3040-3050.
[6] Cong Fang, Jason D Lee. Pengkun Yang, and Tong Zhang, Modeling from features: a mean-field framework for over-parameterized deep neural networks, arXiv preprint Siiv:2007.014555 (2020).
[7] Eugene A Golikov, Dynamically stable infinite-width limits of neural classifiers, arXiv preprint arXiv:2006.06574 (2020).
[8] Arthur Jacot, Franck Gabriel, and Clement Hongler, Neural tangent kernel: Convergence and generalization in neural networks, Advances in Neural Information Processing Systems 31, 2018, pp. 8580-8589.
[9] Adel Javanmard, Marco Mondelli, and Andrea Montanari, Analysis of a two-layer neural net- work via displacement convexity, arXiv preprint
(2019).
[10] Yann LeCun, Corinna Cortes, and CJ Burges, Mnist handwritten digit database, ATT Labs [Online]. Available:
2 (2010).
[11] Qianxiao Li, Cheng Tai, and E Weinan, Stochastic modified equations and dynamics of stochas- tic gradient algorithms i: Mathematical foundations.. J. Mach. Learn. Res. 20 (2019), 40-1.
[12] Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, and Lexing Ying, A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth, International Conference on Machine Learning, PMLR, 2020, pp. 6426-6436.
[13] Song Mei, Theodor Misiakiewiez, and Andrea Montanari, Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, arXiv preprint 10iv:1902.06011 (2019).
[14] Song Mei, Andrea Montanari, and Phan-Minh Nguyen, A mean field view of the landscape of two-layers neural networks, Proceedings of the National Academy of Sciences, vol. 115, 2018, pp. 7665-7671.
[15] Phan-Minh Nguyen, Mean field limit of the learning dynamics of multilayer newral networks, arXiv preprint 1eXXi:19922.2220 (2019).
[16]
Analysis of feature learning weight-tied autoencoders via the mean meanfield lens, arXiv preprint arXiv:2102.0833 (2021).
[17] Phan-Minh Nguyen and Huy Tuan Pham, A rigorous framework for the mean field limit of multilayer neural networks, arXiv preprint arXiv:2001. (2020).
[18] Atsushi Nitanda and Taiji Suzuki, Stochastic partiele gradient descent for infinite ensembles, arXiv preprint arXiv: 1ww::1712.04444 2.05438 (2017).
[19] Atsushi Nitanda, Denny Wu, and Taiji Suzuki, Particle dual averaging: Optimization of mean field neural networks with global convergence rate analysis, arXiv preprint arXiv:2012. 1547 (2020).
11

---

[20] Huy Tuan Pham and Phan-Minh Nguyen, A note on the global convergence of multilayer neu- ral networks in the mean field regime, arXiv preprint 2XXi:2006.09355 (2020).
[21] Huy Tuan Pham and Phan-Minh Nguyen, Global convergence of three-layer neural networks in the mean field regime, International Conference on Learning Representations, 2021.
[22] Grant Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden- Eijnden,,, Global convergence of neuron birth- dynamics, arXiv preprint arXiv:1902.018:: (2019).
[23] Grant M Rotskoff and Eric Vanden-Eijnden Neural networks interacting particle systems: Asymptotic convexity of the loss landscupe and universal scaling of the approximation error, arXiv preprint arXiv: 5005.00915 (2018).
[24] Alexander Shevchenko and Marco Mondelli, Landscape connectivity and dropout stability of sgd solutions for pver-parameterize neural networks, arXiv preprint arXiv:1912.10095 (2019).
[25] Justin Sirignano and Konstantinos Spiliopoulos, Mean field analysis of neural networks, arXiv preprint PrXiv:1805.011 (2018).
[26]
Mean field analysis of deep neural networks, arXiv preprint arXiv: (2019))
[27]
Mean field analysis of neural networks: A central limit theorem, Stochastic Processes and their Applications 130 (2020), no. 3, 1820-1852.
[28]
Alain-Sol Sznitman, Topics propagation of chaos, Ecole d' été de probabilités de Saint-Flour XIX-1993 Springer, 1991, pp. 165-251.
[29] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma, Regularization matters: Generaliza-
fion and optimization of neural nets v.s. their induced kernel, Advances in Neural Information Processing Systems 32, 2019, pp. 9712-9724.
[30] Stephan Wojtowytsch, On the convergence of gradient descent training for two-layer relu- networks in the mean field regime, arXiv preprint arXiv:2005.13530 (2020).
[31] Greg Yang and Edward J Hu, Feature learning in infinite-width neural networks, arXiv preprint 1.XY::0011.14522 (2021).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The two main contributions described the abstract and the introduction are in Section and Section 4.
(b) Did you describe the limitations of your work? [Yes] We do this in Section 1.
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not expect any immediate negative societal impacts from this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We present them in Assumptions and 2.
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs are in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We include the details and link to the code in Appendix G.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide these details in Appendix G.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] We do not report enror bars. Firstly this reduces computational burden for this set of simple experiments. Secondly we instead show
12

---

agreements among plots of different widths N, which convey certain sense of vari- ation that appropriate in the context of (infinite-N) limit laws.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide these details in Appendix G.
4. If you are using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite [10] for our use of MNIST.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "kOeFmEHg1DF", "reviewer": "Reviewer_EQhg", "summary": "The paper studies the fluctuations of a wide but finite multilayer network about the mean field theory limit. The first main result of the paper establishes the form of these fluctuations as a sum of two components, one given by a Gaussian process and another by what the authors call the second-order mean field limit. Additionally, the paper shows that in the large-width limit and near the global optimum, the gradient descent reduces the variance of the network output.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses the important topic of network fluctuation about the infinite-width limit. The paper proposes a novel solution in the multilayer case and also shows that the (previously observed) output variance reduction effect extends to this case. The paper is well written.  \n\nThe two (mutually connected) main issues with the paper are a rather unwieldy form of the main result and the absence of its independent validation. The obtained structure of the fluctuations is quite complex and is only fully described in the appendix. Because of this complexity, it is not obvious how useful is this theorem. The paper is purely theoretical and does not include any numerical experiments or applications. Moreover, the main text contains only a couple of very brief sketches of the arguments, with 50 pages of detailed proofs deferred to the appendix. I have not checked these detailed proofs, so it's hard for me to judge if the results are correct. Accordingly, I think that a journal publication with a comprehensive review process would be more appropriate for this paper. Also, the paper would benefit substantially from some experimental confirmation of its findings - at least the variance reduction effect that should be easy to demonstrate. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "hf_ikuXKyt", "reviewer": "Reviewer_D7BN", "summary": "This work focuses on the random fluctuation of the multilayer network in the mean-field limit. Leveraging on the neuronal embedding framework introduced in previous work, the authors derived a second-order mean-field limit to captures the limiting fluctuation distribution. In the large-width regime, it is shown that the gradient descent helps to reduce and eventually eliminating the variance when the learning dynamics converge sufficiently fast to a global optimum.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: It seems that the technique introduced in this work is new and makes a novel combination of previous techniques. The related work is adequately cited and discussed. For example, the authors make a detailed comparison to the related work in terms of networks and optimization settings [1,2].   \n\nQuality: This submission is technically sound. It seems a complete piece of work in progress. Besides, the authors careful and honest about evaluating both the strengths and weaknesses of their work.\n\nClarity: This submission is clearly written with good organization.\n\nSignificance: My main concern is the significance of the result. It extends a similar phenomenon (gradient descent helps to reduce and eventually eliminating the variance) previously shown for shallow networks [3]. Compared with the previous work [3], the loss function setting of this work is more general. However, how the result could be used in the practice or understanding deep learning is unclear to me.\n\nEven though this work focuses on the theoretical perspective, it would be better if the authors could provide some simple (perhaps on toy model/dataset) but insightful numerical verifications. For example, [3] complemented their results with numerical experiments.\n\n[1] Phan-Minh Nguyen and Huy Tuan Pham, A rigorous framework for the mean field limit of multilayer neural networks, arXiv preprint arXiv:2001.11443 (2020). \n\n[2] Justin Sirignano and Konstantinos Spiliopoulos, Mean field analysis of neural networks: A central limit theorem, Stochastic Processes and their Applications 130 (2020), no. 3, 1820–1852. \n\n[3] Zhengdao Chen, GrantMRotskoff, Joan Bruna, and Eric Vanden-Eijnden, A dynamical central limit theorem for shallow neural networks, arXiv preprint arXiv:2008.09623 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "WZDU7MgSDDZ", "reviewer": "Reviewer_PZpb", "summary": "The paper presents theoretical results on the fluctuations in the mean-field limit. Building on the framework proposed by Nguyen and Pham, the paper goes beyond the strict mean-field limit and derives next-order results about the fluctuations of weights and network output. Specifically, the paper shows that weight fluctuations scale as `1/\\sqrt{N}` (for width `N`), and derives a differential equation that describes the fluctuation. It further shows that the network output obeys a CLT-like result, placing an upper bound on the error in terms of an inverse power of the width. Finally, the paper shows that gradient descent close to a global optimum tends to send the variance in the network output to zero.\n\nUPDATE FOLLOWING AUTHOR RESPONSE:\n\nI would like to thank the authors for sharing the results of numerical experiments verifying key results from the paper, and for addressing my and other reviewers' concerns. In light of these results I am updating my score and recommend accepting the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes an interesting and valuable contribution to the study of large width networks in the mean-field regime. Making the connection between infinite-width and finite-width networks is worthwhile, and the paper makes a step in this direction by studying the fluctuations around the large width limit in detail. The results are highly technical: They rely on explicit expressions for the weight fluctuations in the mean-field limit, which are expressed in terms of a complicated ODE. This level of complexity is to be expected from second-order analysis. \n\nThe exposition is as clear as can be given the significant technical complexity. The notation is consistent, there is a section devoted to reviewing the existing framework on which these results build, and heuristic explanations are provided in several instances which aid understanding. I do suggest numbering all equations and referring to these numbers in the text to help the reader navigate the text more easily. I also suggest referring to the main theorems in the introduction by number, to help guide the reader toward the main results when reading the later sections.\n\nMy main concern is that it is not possible to verify the results of the paper from the main text alone. There are no experimental results to back up the theoretical claims, and the proofs are not in the main text. The authors include useful heuristic arguments for several results, but given the technical complexity these are not enough to be convincing arguments. (See for example equation 3, a 4-line equation that describes the gradient descent evolution of the mean-field fluctuations.) I do believe these results are valuable. I will vote to accept the paper if the authors provide empirical evidence supporting Theorem 5 (including equation 3), and the results of Section 4.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "This paper does not present any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "6AlZXU7GJx3", "reviewer": "Reviewer_ErCU", "summary": "This manuscript proposes a framework for analyzing multilayer neural nets based on the mean field theory, and prooven that gradient descent (GD) in a large-width multilayer network has a variance reduction effect.\nThis research is important and highly novel because mean-field analysis of \"multilayer\" neural networks is considered to be very difficult.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I had a strong interest in the purpose of the manuscript, but unfortunately I did not understand it.\nThis is because the manuscript was confusing to me. \nFrom my view point, I think there are the lack of explanation and numerous careless mistakes.\nThe following is a list of the relevant points.\n\nLack of explanation:\n1.(Line 29) I cannot understand the word \"deterministic limit\" means what kind of limit, because there are no explanation.\n\n2. (Line 94) What is the meaning of $ j_i $?\n\n3. (Line 97) Is $ J_i \\sim Uinf([N_i]) $ means $ J_i \\sim N_i $ ?\n\n4. (Line101) What is the meaning of index 2 in $ \\partial_2 $?\n\n5. (Line106) There might be no definition and explanation of $ \\mathcal{F} $?\n\n6. (Line 142) There seem to be some sentences which have no information, such as the following one.\nThe initialization assumption is also mild.\n\nCareless mistakes:\n1. In Reference, there are no author name for reference [12], [21], and [22].\nI don't think this is a response for double blind either, since other papers written by the same author have the names listed.\nIs it suggested that the paper is incomplete?\n\n2. There is \"Theorem 2\" in Line 179. But there is not \"Theorem 1\" in the manuscript.\nIs it also suggested that the paper is incomplete?\n\n3. The words \"GD\" and \"ODEs\" are suddenly omitted without formal names.\n\n4. There are some elementary errors in English grammar, such as the following\n(Line 20) \"helps reducing and\" -> \"helps to reduce, and\"\n\n\nIn addition, since the theoretical results are not verified numerically, the validity of the derivation process cannot be verified by me.\nIn case other reviewers have not verified the validity of the derivation process, the authors should add numerical experiments.\n\n\nFor your information, here are my attributes.\nI have no experience in theoretical analysis of DNNs using mean field theory.\nAlso, since I was educated in the Department of Physics, I am not familiar with mathematical paper structure.\nSince NeuRIPS has participants with various attributes, I think that the influence of the manuscript will be greater if the authors take care to make it readable by people from a wide range of fields.\nI am happy if I can understand your nice work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It was noted around Line 74 in the manuscript.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents theoretical results on neural network training for finite width networks, which extends prior work which assumes an infinite width limit. The theory proves (among other results) bounds on the output variance of standard feed forward networks as a function of network width when trained with GD. Reviewers agreed that the technical contribution is significant enough to warrant publication, however no reviewer had time to go over the 50 pages of proofs provided in the Appendix. Unfortunately, the ML conference system is ill equipped to handle these kinds of submissions (a reviewer for a mathematics journal may have over a year to review a highly technical submission). 
During the rebuttal period, authors provided experimental confirmation of the theoretical predictions for a simple network trained on MNIST which was enough to satisfy reviewers. Although I am still hesitant to accept an almost purely theoretical work where no reviewer had time to go over the proofs, I will side with the reviewers in light of the experimental confirmation provided during the rebuttal period and recommend acceptance. 

Decision: Accept (Poster)