Mean Field Models for Neural Networks in Teacher-student Setting
Authors: Lexing Ying
Yuandong Tian
Venue:  (2020)
Keywords: mean field model
optimal transport
ResNet

Abstract: Mean field models have provided a convenient framework for understanding the training dynamics for certain neural networks in the infinite width limit. The resulting mean field equation characterizes the evolution of the time-dependent empirical distribution of the network parameters. Following this line of work, this paper first focuses on the teacher-student setting. For the two-layer networks, we derive the necessary condition of the stationary distributions of the mean field equation and explain an empirical phenomenon concerning training speed differences using the Wasserstein flow description. Second, we apply this approach to two extended ResNet models and characterize the necessary condition of stationary distributions in the teacher-student setting.

Under review as a conference paper at ICLR 2020
MEAN FIELD MODELS FOR NEURAL NETWORKS IN TEACHER-STUDENT SETTING
Anonymous authors Paper under double-blind review
ABSTRACT
Mean field models have provided convenient framework for understanding the training dynamics for certain neural networks in the infinite width limit. The resulting mean field equation characterizes the evolution of the time-dependent empirical distribution of the network parameters. Following this line of work, this paper first focuses on the teacher-student setting. For the two-layer networks. we derive the necessary condition of the stationary distributions of the mean field equation and explain an empirical phenomenon concerning training speed differ- ences using the Wasserstein flow description. Second, we apply this approach to two extended ResNet models and characterize the necessary condition of station- ary distributions in the teacher-student setting.
INTRODUCTION
In the past several years, neural networks have achieved tremendous successes in many areas of machine learning and artificial intelligence (Devlin al. 2018 He al. 2016 Silver et al. 2016
Vaswani et al. (017), based on remarkable developments in algorithms (Robbins & Monro 1951
Duchi et | Rumelhart et al.| 1988), software (Paszke et al. 2017 Jia al. 2014 Abadi al. 2016), and hardware (Lindholm et al. 22008] Jouppi et al. 2018). In spite of the empirical successes, the theoretical understanding of neural networks is rather incomplete. To bridge this gap, tools from different fields, such as statistical mechanics, partial differential equations, dynamics systems, have been brought into use. Recently, a series of remarkable papers (Chizat & Bach 2018 Mei et al.) 2018 Rotskoff & Vanden-Eijnden 2018 Sirignano & Spiliopoulos) 2018) analyze the two-layer neural networks using mean field models and nonlinear partial differential equations (PDEs). The main result is that, under the large width limit and sufficient small learning rates, stochastic gradient descent (SGD) dynamics of a two-layer fully connected neural network with a convex loss can be modeled gradient flow of a convex functional of probability distributions under the Wasserstein metric (Peyré et al. 2019 (Santabbogioo 2015) Villani 2008). Though only for two-layer networks, this result significantly simplifies our understanding of the training dynamics. In two recent papers [gguyen 2019 Sirignano & Spiliopoulos 2019), this approach has been extended to multi-layer fully-connected networks by taking the widths infinity successively from higher lower layers.
Contributions. This paper follows this line of research, focusing on the teacher-student setting. The teacher network is typically narrow network that computes the function to be learned, while the student network is a vastly werrparameterrzed wide network. The goal is to optimize the parame- ters of the student in order to produce the function represented by the teacher. Though somewhat restricted, the teacher-studed setting allows for assessing performance by comparing the parameters of the networks. The main contributions this paper are listed as follows.
First, for the two-layer networks in the teacher-student setting, we characterize for the first time the necessary condition for a stationary distribution of the mean field PDE and show how relates the parameters of the teacher network. We also discuss how different activation functions influence the results. Using the Wasserstein flow description, we also provide a simple explanation for why the convergence to teacher parameters with larger weight is much faster than to the ones with smaller weight.
Second, extend the mean field analysis to two extended ResNet models: the ensemble- ResNett (an average of a large number of independent ResNets) and the wide-ResNee (a ResNet with wide residual blocks). We derive mean field PDEs for the training dynamics of the parameters in both

---

Under review as a conference paper at ICLR 2020
cases and also characterize the necessary condition for stationary distribution for the ensemble- ResNet model as well as the first and second order approximations of the wide-ResNet model. Though our derivation remains formal, this is the first time that the mean field model of the ResNet has been considered.
Related work. Besides the aforementioned papers on the mean field analysis for neural networks,
tools from stochastic analysis (Mandt et al.] [2115] Li et al. 2017 2019 Yaida [0188] An et al. (0188)
and Wasserstein geometry (Arjovsky et al. 2017 Cuturi 2013 Frogner et al., 2019: Peyré al.
2019 Solomon et al.) 2014) have been playing an increasingly prominent role in machine learning
and artificial intelligence.
Another direction that considers the infinite width neural networks is the work (Jacot et al. 2018) of the neural tangent kernel (NTK), which describes under an appropriate scaling the evolution of the function represented by the neural network. At the infinite width limit, the NTK converges to a deterministic limit that depends on the nonlinear activation function and remains unchanged during the training process. Though it is not yet clear how this limit relates to the actual training dynamics of neural networks used practice (Chizat et al. 2019), mathematically it is an interesting regime that has inspired a lot of recent work (Arora et al. 2019a b, Cao & Gu 2019).
In recent years, particle-based methods have also been widely applied to many machine learning problems, such as in Bayesian inference. One such recent example is the Stein variational gradient descent (SVGD) method (Liu & Wang 2016 Liu 2017) that approximates a target distribution with group of interacting particles using iterative gradient updates. It has been shown that, in the large particle number limit, the empirical distribution of the parameters can be modeled by nonlinear PDE, which can be viewed as gradient flow of a convex loss function under Wasserstein-type but non-local metric (Liu 2017 | Lu et al., 2019).
Organization. The rest of the paper organized follows. Section 2 concerns the mean field description of two-layer networks and provides necessary conditions for stationary distributions. Section 3 extends the analysis to two extended ResNet models. Finally, Section 4 discusses some future directions.
2 TWO-LAYER NETWORK IN TEACHER-STUDEN SETTING
2.1 TWO-LAYER NETWORKS
Setup. Given a function f R2 - IR of the independent variable E Rd considec its approximation by two-layer network model N Σ wyo(x 0z). where N is the number of hidden units, {0₂} are the parameters in a domain e C Rdd {wi} are the weights, and o : R - R is an activation function such as rectified linear unit (ReLU), softplus, etc. The following discussion focuses on the special case where the weights wy are fixed to be one, i.e.
Σ a 0,).
The more general case of trainable W/ values are discussed in Appendix 5.1 is sampled from measure uzz) independently, the learning problem of this two-layer network
Assuming that the input
is to minimize the loss E(O)...., = So e ( Σnn a(x 0,) v(x)) w(x)dx, where E(-) is loss function. The rest of this paper focuses for example the quadratic loss case, i.e., {(2) 1
1 / E(O ... 0N) 2
N Σ alx 0,) -(() u(x)dr== =
1 2
Σ a( - 0,) y(x))
N
When optimized with a standard algorithm such as gradient descent (GD), each parameter 0, for 1 S j < N is updated with
aE [[k+1] = -
- 00,
[k]
OE 00,
[k] = S.
N axx 0ylk)) - v(x))
o (x(x=0011)))
I
N
N
1=1
2

---

Under review as a conference paper at ICLR 2020
where is the learning rate of the iterative algorithm. For the SGD, the right hand side is supple- mented with a mean-zero noise term. When the learning rate a is sufficiently small, the iterates {0, [K]) can be modeled by a continuous-time ordinary differential equation (ODE) system (0,(t))
by identifying & /o/N
(1,(1)
dt --
E F=1
N ox 02(1)) v(x) d'(x (0(x=0;00 0.(t)aap(a)dz.
(2)
N
Mean field equation. Viewing (t( as interacting particles and considering the empirical distribu- tion p(t, 8) t EØ= (1)())()), we can rewrite ES a(x 02) y(x) as o(x ()p(O)dd and
..... 0N) as
E(p) 2/ f ol (otel00)p000000000 v(x)
1
The mean field evolution equation for p(t. 0) associated with (2) is
(0, = dive p(0.0) t) /. (J. o(x o(xx 0')((0'' t)d0' u(x) d' (ece)))x(()dd
Noticing that the functional derivative of E(p) with respect to is given by
8F())
[.(V/. olx 00) ('0')) v(x) o(x ())(x)dz,
8p
write the mean field PDE for p compactly as
8E dive pV óp
Pt
(3)
which can be interpreted as the gradient flow of the functional E(p) under the the Wasserstein metric
(V)DVo)+ (Jordan et al. 1998 Otto 2001
Otto & Villani 2000).
Though the above derivation is mostly formal, one can show that the dynamics of 0; (f) converges to the evolution of p(0, t) for any fixed period of time [0, 7] when the number of hidden units N goes to infinity. This is formalized for example in the following theorem.
Theorem (Reformulated from (Chizat & Bach, 2018 ((0) are sampled independently from density po(x:) at time 1. = 0, then for any fixed T > O the measure > (0,(0)(0) converges weakly the solution p(t, 0) of 3) any E [0, N goes infinity.
For the case of bounded parameter domain O such ball in Rd with a fixed radius, one needs to specify the appropriate boundary conditions both for the ODE system (2) and the mean field PDE
3). First, the dynamics of (t()) should follow normal reflection when (f) hits the boundary de of
O Second, for the mean field equation of p(t, f). the Neumann boundary condition @p(t.0) (t. 0)
On
should be used at 0 E 20. The bounded case is particularly relevant to the discussion in Section 3.2
Stationary distribution in teacher-student setting. Given the PDE formulation the parameter evolution, one natural question is to characterize the stationary distribution of (3). In what follows, we address this question the teacher-student setting, i.c., there exists a parameter distribution p. (0) such that the target function y(z) satisfies
2(x) ffoc--0pp_00)1 ()).
With this teacher representation, the loss E(p) can be written equivalently as
ElP) = 1 / (f ()0 0) p.)())))) -
2
p(x)da.
The following theorem characterizes the stationary distribution in this teacher-student setting.
3

---

Under review as a conference paper at ICLR 2020
Theorem 2. Suppose that plz) for E Rd. Then any stationary distribution p(0) that satisfies p(0) > for any è has zero loss and satisfies the condition
/ [(cx0)(((0)) 0)(p()) (f))dd = 0.
(4)
Proof. By introducing the operator E with kernel given by ox f), one can write the loss as E(p) E1 p.)ll and the functional derivative as
E 8p
E2 pElp- p.);
where μ is considered to the operator of multiplying with ((z) at each æ. At a stationary distribution P. since Pt dive (Vo
5E
0 any 0 O. Taking inner product with 6B 0p gives
óp
PE 6p
dE 8p
diva
PV
0.
Using the fact that div, is the adjoint of Vo. this states v BE 0, which further implies Ve 6E
as p(0) > O. This means that 5E Σ pD p.) as function of is a constant.
6p
Taking inner-product of 5E with (p - p.) and using the fact that f(p p. (0) gives 8p
(p- p.) ( p.) 0.
This implies zero loss and / [pp --) 0, which further implies E(p p=) 0 since u(x) > O. Writing E(p - p.) = 0 in the integral form gives 4
Below consider the implication of Theorem for few commonly-used activation functions. Example 1. Consider the ReLU activation (2) = max(z,0). It is convenient to use the polar coordinates - (2, (æ:)) and 0 - (8.100 (8), where and denote the radial coordinate and the angular variables, respectively. Then 44) reads
Jal ( ( (01) - p= (0)) =0,
where the factor 10 comes from the homogeneity of ReLU well as the Jacobian from the polar coordinate transfornation. If the operator with kernel (a(a 0)),,083-- has trivial null space, then one can conclude that the stationary distribution p(O, 10)) satisfies for any E Sd-1
f. (00 - (p, ( =
Example Consider for simplicity the ID case d When a22) is an even function such as the Gaussian or quadratic activations, (4) states that p(0) is stationary distribution if and only if the difference (p - (8) is an odd function.
When (2)) is an odd function such as tanh or arctan, 44 states that p is stationary distribution if and only if the difference (p- p+)( is an even function. In addition, when (22) is the sum of an odd function and constant (for example, like the Sigmoid function), is equivalent (p-px)(0)
being even, since the integral of (p (0) is zero.
Finally, the softplus function (2)) In(1 e²), which smooth approximation of ReLU. states that
[In(t
In(1 + ( p=)(0)d0=0) =.
Let us assume for simplicity that both p(0) and (0) decay faster than 0-2 large 0 and define (() as the anti-derivative of (p (0). Integrating by parts for (4) results in
1 f(0)d0 = 0.
4

---

Under review as a conference paper at ICLR 2020
Noticing that the Sigmoid function 14e-3z is sum of odd function and constant, conclude that is equivalent to f() being and mean-zero. In terms of the difference (p))))(0))))) these two conditions means that (p (0) is odd and has vanishing first moment
J po) O.
2.2 CONVERGENCE SPEED ISSUE
One interesting empirical observation from training neural networks in the teacher-student setting is that the convergence teacher parameter with large weight much faster than the speed teacher parameter with small weight. More specifically, consider teacher network given by
(+) = a19., (0) + (0),,
where e; are the parameter values in R and the weights a satisfy 01 > 02 and 01 + a2 = 1. This can either be realized as general two-layer network of two hidden units with weights on and
02 or an equal-weight network (T) of o N units with parameter e₁ and 022 units with parameter c2. After an initial short period, the evolution of the student parameters decouple into two nearly independent problems: an 0i fraction of the N student parameters converges to C₁, while an aq fraction of the student parameters converges to cg. The observed behavior is that the convergence to e1 is much faster than the convergence to C2.
The mean field PDE allows us understand this phenomenon using scale analysis. After the initial pp - p.))] is effectively decoupled into two equations
stage, the PDE P. = divo [o (Σ
p: dive V Σ pE(p' pb))], where p._ a 6c (0),
p = dive V (Σ EE - -p?))] where p² = a2d (0).
The key observation is that the quadratic dependence on p on the right hand side is the reason of different convergence speeds. To see this, let us divide these two equations by an and 02, respec- tively.
) a1
where e
aj divo
Vo
Σ
6,y (0),
01
01
11
a1
p²
p² = 6es (0).
= 22 diva
Σ
where
These two are the same equations for normalized densities p¹ and e² except the coefficients
as and 02 at the beginning of the right hand sides. The difference in these two coefficients clearly demonstrates that the first equation converges a /1/22 times faster than the second one.
RESNET MODELS IN TEACHER SEREER- SETTING
Consider simple ResNet model with input E Rd (see Figure Ileft)
= X = x + 7 7 (0)))) 056<L,
where X E Rdd the matrix parameter 0f E C Rdxd, and the activation is applied entry-wise to each component. By introducing 0 (0" (0"...., 0l-1), ) we denote the final result at level by X in order to emphasize the dependence on O. Given function y : Rdd + Rd the training problem searches for the parameters 0 (0"...., 01-1) E O in order to minimize, say, the quadratic loss
E
x (x) x(x)) p(z)dr = 2 [x(x) -0(0)><>))) y(=)
2 J..
When optimized with GD, the parameter vector updated at each step with aE
0 + 1| 0 [k]
(0k))
00
(or with extra noise term for SGD). Instead of directly working with this vanilla ResNet, introduce two extended ResNet models in the rest of this section and derive a mean field equation for each of them.
5

---

Under review as a conference paper at ICLR 2020
50
0
0
0
0
0
0
40
0
0
0
0
00
Figure 1: ResNet models. Left: vanilla ResNet. Middle: ensemble-ResNet. Right: Wide-ResNet.
Mean field models for the ensemble-ResNer and Wide-ResNet models are discussed in Section
3.1 ENSEMBLE-RESNEE
Setup. The first model, called ensemble-ResNet, averages over a group of N ResNets with inde- pendent parameters (see Figure [ middle). Its j-th ResNet has the same structure as the vanilla ResNet described above and contains parameters 0j (0) j...... 0 E O4. The output of the
ensemble-ResNetiss N Σj= X (z)
and the loss function is
E 0N) = 2
Σ j=1
x (z) - y(z)
The GD/SGD algorithm with learning rate a takes the following step for each i
N
i + 1| 1 - a f. Vo. X
(x)
x
(1)
y(x) u(z)dr.
N
When a/N is sufficiently small, the dynamics of 0,]k] can be approximated by an ODE system (t) (t) by identifying ka/N
d0,
dt - -
1 N
Σ
xbb(z)) (x) v(x) p(z)dr.
Viewing 0:(1) = (0) (.. as particles in 00 and introducing the distribution p(1, 0) N E (0) we rewrite E(O,..... θN) as E(p) = I1.oo x (x)p(0)d0 y(x) 1| The
mean field equation of p(t, 0) takes the same form
8E (0) = dive p
8p
which is gradient flow of quadratic loss E(p) under the Wasserstein metric on the densities p(0) on the product space O4,
Stationary distribution in teacher-student setting. In the teacher-student setting, the function y(x) from 2d to d assumed to take the form y(æ) = So x (x)p, (0)dd, for a certain distribution Pa on the space 0 This include the special case that vzx) can be represented by a vanilla ResNet described above. More precisely, assuming that
x x,
x = x 1
[0(0'') 7(z) (x),
then y()) corresponds p. (0) 50. (0) with O. .0...... 05-11 100....,0--1 04.

---

Under review as a conference paper at ICLR 2020
Theorem 3. Suppose that ja(x) frror E BR. Then any stationary distribution p(0) that satisfies p(t, 0) > for any > 0 and E @ has zero loss and satisfies the condition. each a,
(x) (O)))d 0.
(5)
The proof of the theorem follows the same argument as Theorem 2 One potential criticism of Mesemble-ResNee model is that the space @² (Raxd)² can be very nigh-dimensional even for moderate values of L. Therefore, the number of branches N required in order for the mean field model be accurate can be quite large.
3.2 WIDE-RESNET
In order to overcome the exponential grow of the space 6 in the Besemble-ResNee model, we consider a different ResNet model in this subsection. Instead of allowing for multiple independent branches, the wide-ResNet model sticks to single path but increases the width of each residual block (see Figure right). For an input x E Rdd the Wide-ResNet evaluates
X = x = x
LN Σ o( OSE<L,
(6)
where the activation function oxz) is assumed to be smooth and vanishes at 2 = 0. By introduc- ing the parameter group 8¹ (ff.....66) .by for each level E, we write the final result of as X or. (z) and the quadratic loss function
E(0". E( ......,.--11 ---1)-
= E
80 .
1 O%-
X (x..,, v(oo
2
The GD/SGD algorithm with learning rate takes the form
[h 1| (9(k]] oot X Xo (n-)))()))
v(x)) p(x)dx,
where Vo X dynamics of 0 can approximated with ODE system
(z) contains a factor Z.x. When the stepsize ar sufficiently small, the
dog
gl- / (o..x........ (x) (x (X
g (x- -(()))(())) p(z)dx.
By viewing (1)..... ...,, (1) as interacting particles at each level and introducing (1,
N E, 80g10) (0') for each l, one can rewrite the wide-ResNet computation as
x x + + 010 (0"))d0 OSE<L.
By taking the limit N to infinity, the loss becomes E(p", ,.... 11- 1 po...p (xx) y(x))
(with the subscripts denoting the dependence p ,,... xb11) and the mean field equation for p' (t, 88) for each level e reads
8E(p p(00) divgr (
(8)
óp
As a(2) is assumed to be smooth, expanding recursively and applying the Taylor expansion (see Appendix 5.2 for a derivation) gives the following approximation
% x 1 L
Σ
/ ((0X)))00000
1 1
Σ
||
Vo((" x20 (0 a6 +...

---

Under review as a conference paper at ICLR 2020
Furthermore, since (22 vanishes at zero by assumption, this expansion has L terms of order O(0) each with weight 1 I/L, terms of order O(f²) each with weight 1/T.2, etc. When the domain O of the matrix parameter 9 is bounded by a sufficiently small radius, the contribution from the k-linear terms of order (11/k!) and hence decays rapidly with increasing values of k. Therefore, in the same parameter regime, it is sufficient to consider approximations of the first few orders.
Linear approximation. The linear approximation for X in (9) leads to the following first-order approximation to the loss
F(p"..... p)- = PA )0-))) x +
E (10))) v(z)
Now take the limit of I. going infinity and define p(f, a rescaled continuous variable associated with the depth of the ResNet. In this formal limit, the loss function becomes (p) $ || + (0x)) =)dddde y(=))l;
=a p² (0) -5./I(e) where E 0, 1] is
The mean field equation for
p(t, 8, takes the form
(c(0,e) dive p(0, E Vo 8B óp
(10)
Notice that this is degenerate gradient flow with mobility only in the 0 variable. In the teacher- student setting, i.c y(z) r+ Jf (0x)), e)ddde for some teacher distribution p. (0, E). we can
write compactly (p) = 1 (p p.) where Σj is the operator with kernel o(f)). Notice that E does not appear in the kernel explicitly.
Theorem 4. Suppose that p(x)= 0 for x Rd. Then any stationary distribution p(8. E) that satisfies p(0,E) > 0 for any and E [0, 1] has zero loss and satisfies, at each æ E Rdd
[[o(02) (fmle - p_de do = O.
(11)
The proof follows the argument of Theorem 2 is not surprising that the constraint [I] is formu- lated in terms the marginal of p, since the parameters at different levels become almost commu- tative in the small 0 regime.
Quadratic approximation. The quadratic approximation for x second-order approximation to the loss
9 leads to the following
(p .P.... 2
-
r
+
(())) v(x)
x0 0f0 x ) ( (o vta)
Taking the large L limit and introducing p(0,E) E., (f) gives rise to the loss function
(e"))) 1 x (000))((0,))00 + /| Vo (0') ,'))((,, e)00 dode'de de - s(x) (p) =
terms of p(0,s) s) and the mean field equation takes the same form as (10) except with E(1 replaced with E(g)- In the teacher-studens setting, there exists some (0, such that
y(x) = x (0)) + +/[ (0(0'))0''((z)) (0' e')p, e) e)do dode'ds de.
By introducing new operator Σ2(p @ p) J. we can write more compactly as
s')) (0, dOde's
1 E(2)(p) |E p.) + Ex P. p.)
Unfortunately, since Ee2) (p) is quartic in p. stationary distribution might be local minimum even with the extra assumption p(9,e) On the other hand, the explicit and simple form of E (p) might allow multiple runs with different initial condition p(0, 0) to identify the p. more efficiently.
8

---

Under review as a conference paper at ICLR 2020
4 DISCUSSIONS
This paper discusses necessary conditions for the stationary distributions of mean field models for the two-layer neural networks as well as two extended ResNet models. lot of interesting questions remain to be addressed concerning the ResNet. The first, and probably most important, question that whether it possible to obtaín a mean field model for the vanilla ResNet by treating depth L as the large parameter going to infinity. This in fact has been the original motivation of this paper and still remains unresolved. Second, in the wide-ResNet model, the resulting PDE degenerate there is no diffusion in the variable. Would it be possible to include the diffusion in by allowing parameters to jump between adjacent layers? If this is possible, the next step is to formulate mean field model with densities some "architecture" space for neural architecture search Zopp & Le 2016 Zoph et al. 2018 Elsken et al. 2018 Wistuba et al. 2019
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Jing An, Jianfeng Lu, and Lexing Ying. Stochastic modified equations for the asynchronous stochas- 2018.
tic gradient descent. arXiv preprint
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214 224,2223,222 2017.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint 1rXiv:1904.11993 2019a.
Sanjeev Arora, Simon S Du Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08884 2019b.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. arXiv preprint arXiv: 1905.13210 2019.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Advances in neural information processing systems, pp. 3036-3046, 2018
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.08800055 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(111)2121--2159 2011.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. arXiv preprint 1808.08808.05
2018.
Charlie Frogner, Farzaneh Mirzazadeh, and Justin Solomon. Learning embeddings into entropic wasserstein spaces. arXiv preprint (:X.v:/905.03322 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.

---

Under review as a conference paper at ICLR 2020
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 675 678. ACM, 2014.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker- planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
Norman Jouppi, Cliff Young, Nishant Patil, and David Patterson. Motivation for and evaluation of the first tensor processing unit. IEEE Micro, 38-3):10-19, 2018.
Qianxiao Li, Cheng Tai, et al. Stochastic modified equations and adaptive stochastic gradient algo- rithms. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 2101-2110. JMLR. org, 2017.
Qianxiao Li, Cheng Tai, and Weinan. Stochastic modified equations and dynamics stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40): 1-40, 2019.
Erik Lindholm, John Nickolls. Stuart Oberman, and John Montrym. Nvidia tesla: A unified graphics and computing architecture. IEEE micro, 28(2):39-55 2008
Qiang Liu. Stein variational gradient descent as gradient flow. In Advances neural information processing systems, pp. 3115-3123, 2017.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. In Advances in neural information processing systems, pp. 2378-2386, 2016.
Jianfeng Lu, Yulong Lu, and James Nolen. Scaling limit of the stein variational gradient descent: The mean field regime. SIAM Journal on Mathematical Analysis, (1()):648 671, 2019.
Stephan Mandt, Matthew Hoffman, and David M Blei. Continuous limit stochastic gra- dient descent revisited. NIPS-2015, 2015.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, (1etmeess,1(13335)) 115(33):7766-E7777, 2018.
Phan-Minh Nguyen. Mean field limit the learning dynamics of multilayer neural networks. arXiv preprint 1r2.00190..02888, 2019.
Felix Otto. The geometry of dissipative evolution equations: the porous medium equation. Comm. Partial Differential Equations, 26(1-2):101-174, 2001.
Felix Otto and Cédric Villani. Generalization of an inequality by talagrand and links with the loga-
rithmic soboley inequality. Journal of Functional Analysis, 173(2):361-00) 2000.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 11(566):355-607 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Grant Rotskoff and Eric Vanden- Eijnden. Parameters interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in neural information processing systems, pp. 7146-7155, 2018
10

---

Under review as a conference paper at ICLR 2020
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, al. Learning representations by prck-propagating errors. Cognitive modeling, 5(3):1, 1988.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55:58-63, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis neural networks. arXiv preprint 1Xxv:/805.00053 2018.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of deep neural networks. arXiv preprint arXiv:1903.0444 2019.
Justin Solomon, Raif Rustamov, Leonidas Guibas, and Adrian Butscher. Wasserstein propagation for semi-supervised learning. In International Conference on Machine Learning, pp. 306-314, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp.
2017.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati. A survey neural architecture search. arXiv preprint arXiv:1905.013990 2019.
Sho Yaida, Fluctuation-dissipatio relations for stochastic gradient descent. trXiv:1810.0000 2018.
arXiv preprint
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv.1611.01578, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 8697-8710, 2018.

---

Under review as a conference paper at ICLR 2020
5 APPENDIX
5.1 TWO-LAYER NETWORK WITH VARYING WEIGHTS
In this subsection, we discuss the mean field equation for the general two-layer network model with trainable weights
wjola 0j),
where wy E W C IR for each j. The quadratic loss E(O,, wi,..., ,N, WN) is given by
N
1
E(O,, Wi.... ON WN)
(z 03) - y(z)
2
N
GD or SGD with sufficiently small step-size can be modeled by the ODE system
d1)(1)
/
wjola 0,(t)) 3(x)
u,(t)o'(x ((())=p(z)dd
dt N
5=1
dt:(t) /
1
Σ
wa 0,(()) y(x)
olx (l.-0:(())())da
dt
N
Viewing (0)(t), w;(()) (t)) interacting particles in W and considering the distribution
N
(tt,,.w= = N
Σ
(1),
one can write N E3 wz A2) - y(z) as wo 0)p(0, w) wddddw and E(O,, wi,..., .N, wN) as
E(()) = / wa 0)p(0, w dd0dw y(x))
11
The mean field equation for p(t, 0, w) takes the form
SE
= divow pv ow 8p
(12)
which is the gradient flow of E(p) under the Wasserstein metric on the densities p(0, w). In the scacher-student setting, i.c., y(z) Sew wa (0,,, w)dddw for some teacher parameter distri- bution pa (0, w).
1 / Elpl -
( worx ()(p(, p.(0, ) dx.
Ow
The following theorem characterizes the stationary distribution of 122
Theorem 5. Suppose that u(x)) > for E I.d. Then any stationary distribution p(0, w) that satisfies p(0, w) > 0 for any and w W has zero loss and satisfies the condition
f wal (p(0, w) - (= ())ddd))dd =0.
(13)
Proof. By introducing the operator Σ with kernel given by worx 0), one can write the loss E(p) [|[(pp P.) p]] and
8E
ΣT pTp(pp - pa):
8p
12

---

Under review as a conference paper at ICLR 2020
where y is considered to the operator of multiplying with pız) at each æ. At stationary distribution
since Ps 0, divów [Vo ow 8p
áE
0 at any 0 E θ and w W. Taking inner product with &E
gives
8F
SE
divow p(0, w)
Vow ow 8p
= 0.
8p
Using the fact that diva is the adjoint of Vew. this means pp w) V ow áp 0, which further implies Vew Ap 0 as p(0, w) is constant. Taking inner-product of JE with (p po)(O, w). one arrives 8p
O. This means that 6E Σ pΣ(p pa) as function of 0 and 6p
(o P.) E. p p+)= 0,
using the fact that [(p-ppp p-) w dddw = 0. This implies zero loss and
v up=)><pp - p.) = 0,
which further implies E(p p.) 0 since μ(x) > Writing the last statement explicitly in the integral form gives (I3).
Example 3. Let us consider the Rel.U activation (z) = max(2, 0). Using the polar coordinates
- (2, xl) and 0 - (0) 13
reads
J 6.100 |0|10 [ wo ()(((, 0), w) - 00, w) ) (l =
If the operator with the kernel (o ( (0)) has trivial null space, then one can conclude that that for each
J 10 (000,10),0) (.(0.10),v) dlo|dw
that the stationary distribution p.
5.2
DERIVATION OF
The computation follows the definition in and uses Taylor expansion. At the first level.
1 J
x = 0 T. 00
x000
At the second level,
x²= = x2 + 1
/ x00'x2)p'' (o- (do)
L
= x I /
1
1 0¹
1 gt
x + T, //1 a x (0) (0%)00))))))) (0) )
a000 x0) ) ( )
T,
Recalling that o is assumed to be smooth near the origin, a Taylor expansion for a in the last term
1
/ o
1 " 00 (0')))"
x x
x00xx))
(O")00"
L go
/ (0
(f a x )0 (09) dd" (² )do h.o.t.
2 gr
Iterating this until the L-th level gives rise to
X X
+-
1
Σ
(
1 17 b>a
Σ //
th..tt
L
a=0
13


Reviews:
[{"review_id": "ryx47deptH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the mean fields limit of neural networks and extends the mean field limit treatment to resnets. The paper seems to be presenting non-trivial formal results but I could not really understand what are their specific consequences of interest. What are the concrete insights that the results of this paper bring? Even as a theoretician, I did not really understand what should I take from this paper. As such I cannot recommend its acceptance. But it is well plausible that with better explanations I will understand its value and change my mind. Some more concrete questions/issues follow: \n\nThe introduction several times states: \"First, for the two-layer networks in the teacher-student setting, we characterize for the first time the\nnecessary condition for a stationary distribution of the mean field PDE and show how it relates to the parameters of the teacher network. We also discuss how different activation functions influence the results.\" The outline states: \"Section 2 concerns the mean field description of two-layer networks and provides necessary conditions for stationary distributions.\"\n\nBut where is this discussion of necessary condition? I cannot find a single mention of \"necessary\" in section 2 that only includes formal derivations and statements. Can the result of Section 2 be translated to somewhat more general audience? What is the take-home message there?  \n\nWhere is the discussion on the role of the activation function? I see the two examples stating the formal result for two different activations, but again I am unable to understand what should one take from this?\n\nI could understand the result of section 2.2. about stronger weight-teachers being learned first. But this is completely intuitive in the same way as independent components corresponding to larger eigenvalues would get learned first in PCA. See also\nsimilar conclusions in works on so-called \"INCREMENTAL LEARNING\" in Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013. And subsequent results. So this alone does not seem very novel result. \n\nAs I had hard time understanding the section two on the simpler feedforward case, Section 3 was even less clear to me. How does what is done here compare to what is known on resnets and what should somebody interested in resnets (but not specifically in the mean field equations) take out from this. Some non-technical summary of the findings is seriously missing in this paper.\n\nComments that are less fundamental to the overall understanding:\n\n** The mean field treatment was also extended to the multi-layer case in: https://arxiv.org/abs/1906.00193\n\n** The paper present nice account on previous results involving the mean field limit. The manuscript should also discuss the long line of papers analyzing the teacher student setting on one-hidden-layer neural networks. After-all this seems to be the main object of study here so the results only make sence presented against what was previously known. Notably the case of eq. (1) where the 2nd layer weights are fixed to one is a model called the committee machine widely considered in previous literature (in the case of finitely wide one-hidden-layer network).\n\nThe (non-exhaustive) list of paper on the teacher-student setting is (more references are in those papers):\n\n-- The teacher-student setting was introduced in Garder, Derrida'83 (model B, https://iopscience.iop.org/article/10.1088/0305-4470/22/12/004/pdf).\n\n-- In the classical textbook on neural networks Engel, Andreas, and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001, there is a rather detailed account of many results on the setting from 80s and 90s.\n\n-- Notably the plain SGD was analyzed via ODEs for the teacher-student setting in classical papers: David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995.\nDavid Saad and Sara A Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. In Advances in neural information processing systems, pp. 302–308, 1996.\n\n-- This line of work was recently extended with a focus on the overparametrized regime (but not infinitely wide) in: Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup, Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová, NeurIPS'19.\n\n-- There even in analysis with more than one hidden layer in Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized\nneural networks, going beyond two layers. NeurIPS, 2019a.\n\nIt would be really interesting to see a discussion of what is similar and different between these works and results and the present work. And what is the added value of the present one about understanding the teacher-student setting.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eM1dARYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the mean field models of learning neural networks in the teacher-student scenario. The main contributions are summarized in Theorems 2-4, which characterize the stationary distribution of gradient-descent learning \nfor what are commonly called committee machines. Theorem 2 is for a committee of simple perceptrons, whereas Theorem 3 is for what the authors call the ensemble-ResNet, which is in fact a committee of ResNets, and Theorem 4 is for what the authors call the wide-ResNet, which is actually a stacked committees of simple perceptrons.\n\nThese three theorems are straightforward to derive from differentiation of the expected squared loss E[\\rho] with respect to \\rho and equating the result with zero. The argument in Example 2 in Section 2.1 has flaws. The argument on the wide-ResNet is based on the linear approximation, which effectively reduces the stacked committee network to a large committee of simple perceptrons, so that its significance should be quite limited. Because of these reasons, I would not be able to recommend acceptance of this paper.\n\nThe authors do not seem to know the existing literature on analysis of learning committee machines. See e.g. [R1] and numerous subsequent papers that cite it.\n\nIn Example 2, the authors claim that when \\sigma is an odd function \\rho is a stationary distribution if and only if the difference is an even function. This claim is not true in general. Consider the case where \\sigma is a sign function. Then for any function f(\\theta) satisfying \\int_0^\\infty f(\\theta) d\\theta=0 (such functions are straightforward to construct), let g(\\theta)=f(\\theta) if \\theta\\ge0, and g(\\theta)=-f(-\\theta) if \\theta<0. Then \\int \\sigma(x\\cdot\\theta)g(\\theta) d\\theta=0 holds, whereas g is an odd function. This is a counterexample of the authors' claim here. Many more such counterexamples are quite easily constructed on the basis of orthogonal polynomials, demonstrating that the stated stationary condition in Theorem 2 may not be so strong for characterizing the stationary distribution.\n\nPage 2, line 30: Given a function (f -> y)\nPage 2, line 38: I do not understand what the subscript $x$ of the integral sign means.\nPage 2, line 40: The last term should be squared.\nPage 3, line 7: Should \\theta_j be \\theta_j(t)? Should \\rho(\\theta) be \\rho(t,\\theta)?\nPage 3: Both \\rho(t,\\theta) and \\rho(\\theta,t) appear, which are inconsistent.\nPage 3, line 16: under (the the) Wasserstein metric\nPage 3, line 22: \\rho_0(x) should probably read \\rho(0,\\theta).\nPage 3, line 26: \"should follow normal reflection\" I do not understand the reason for it. How \\theta_i(t) should behave when it hits the boundary depends on how the gradient-descent algorithm is defined in such cases.\nPage 6, line 7, page 7, line 18: \"The GD/SGD algorithm\": What is shown here is the GD algorithm and is not the SGD.\nPage 7, line 14: and (vanishes -> to vanish)\n\n[R1] David Saad and Sara A. Solla, \"On-line learning in soft committee machines,\" Physical Review E, volume 52, number 4, pages 4225-4243, October 1995.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xnlKs3KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the dynamics of neural network in the Teacher-Student setting, using the approach pionnered in the last few years.\n\nConcerning the presentation and the review of other works, I am a bit surprise that the \"teacher-student\" setting appears out of the blue, without any references to any paper. This is not a new area! Especially when it concerns learning with SGD, these were the subject of intense studies, especially in the 80s, see for instance:\n* Saad & Sola On-line learning in soft committee machines, '95\n* Exact solution for on-line learning in multilayer neural networks '95\nor the book \"On-Line Learning in Neural Networks\", By David Saad, with contribution of Botou, etc...\n* Many of these results were proven rigorously recently in \"Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup\" by Goldt et al,\n\nThere is a difference with the current formulation: in those papers, both the teacher AND student had a finite-size second (or more) layer, while here, one is working in the mean-field regime where the student is infinity wide. This is indeed a different situation, where the system can be much more \"over-parametrized\". But this does not mean that the subject is \"terra incognita\".\n\nThere are three main sections in the paper, discussing the results.\n\n* The first result is a theorem that, if I understand correctly, says that the stationary distribution of gradient descent on the population loss (i.e. the test error) a necessary condition for the stationary distribution is that it has zero generalisation error (Eq. 4). That seems like an incremental-step compared to previous results that write down the PDE (the four mean-field papers from last year) and it looks very similar to results that show gradient descent provably converges to global optima etc. I am not sure I see the importance of the result. Also, it is not clear \"how long\" SGD should run to reach such a point and this regime might be entirely out of interest.\n\n* Sec. 2.2 also discusses the difference in learning to teacher nodes with large or small weights, resp. Again, this is well-known in the asymptotic limit and rather unsurprising. This is also discussed in, e.g.  in Saxe et al \"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\" at least in the linear case.\n\n* The extension to ResNets is definitely more interesting. The authors write down mean-field equations for two models, and prove, if I understand correctly, that it is a necessary condition to recover the teacher weights to generalise perfectly, which, as I said above, seems unsurprising.\n\nIn the end, given the paper is not discussing the relevant literature in teacher-student setting, and that I (perphaps wrongly)\n do not find the results surprising enough, I would not support acceptance in ICLR.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the evolution of the mean field dynamics of a two layer-fully connected and Resnet model. The focus is in a realizable or student/teacher setting where the labels are created according to a planted network. The authors study the stationary distribution of the mean-field method and use this to explain various observations. I think this is an interesting problem to study. However, the reviewers and I concur that the paper falls short in terms of clearly putting the results in the context of existing literature and demonstrating clear novel ideas. With the current writing of the paper is very difficult to surmise what is novel or new. I do agree with the authors' response that clearly they are looking at some novel aspects not studied by the previous work but this was not revised during the discussion period. Therefore, I do not think this paper is ready for publication. I suggest a substantial revision by the authors and recommend submission to future ML venues. 
Decision: Reject