On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
Authors: Zhiyan Ding
Shi Chen
Qin Li
Stephen Wright
Venue: ICLR 2022 Submitted (2022)

Abstract: Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first-order methods nevertheless find the global optimum in the overparameterized regime. We study this phenomenon with mean-field analysis, by translating the training process of ResNet to a gradient-flow partial differential equation (PDE) and examining the convergence properties of this limiting process. 
The activation function is assumed to be $2$-homogeneous or partially $1$-homogeneous; the regularized ReLU satisfies the latter condition. We show that if the ResNet is sufficiently large, with depth and width depending algebraically on the accuracy and confidence levels, first-order optimization methods can find global minimizers that fit the training data.

Under review as a conference paper at ICLR 2022
ON THE GLOBAL CONVERGENCE OF GRADIENT DE- SCENT FOR MULTI-LAYER RESNETS IN THE MEAN- FIELD REGIME
Anonymous authors Paper under double-blind review
ABSTRACT
Finding the optimal configuration of parameters in ResNet is nonconvex mini- mization problem, but first order methods nevertheless find the global optimum in the overparameterized regime. We study this phenomenon with mean-field analysis, by translating the training process of ResNer to a gradient-flow partial differential equation (PDE) and examining the convergence properties of this limit- ing process. The activation function is assumed to be 2- 2-homogencous or partially 1 t-homogeneous; the regularized ReL.U satisfies the latter condition. We show that if the ResNet is sufficiently large, with depth and width depending algebraically on the accuracy and confidence levels, first order optimization methods can find global minimizers that fit the training data.
INTRODUCTION
Training of multi-layer neural networks (NN) requires us to find weights in the network such that its outputs perfectly match the prescribed outputs for given set of training data. The usual approach is to formulate this problem as a nonconvex minimization problem and solve it with a first-order optimization method based gradient descent (GD). Extensive computational experience shows that in the overparametrized regime (where the total number of parameters in the NN far exceeds the minimum number required fit the training data), GD methods run for sufficiently many iterations consistently find global minimum achieving the zero-loss property, that is, perfect fit to the training data.
What is the mechanism that allows GD to perform so well on this large-scale nonconvex problem?
Part of the explanation is that the overparametrized case, the parameter space contains many global minima, and some evidence suggests that they are distributed throughout the space, making easier for the optimization process find one such solution. Many approaches have been taken to characterize this phenomenon more rigorously, including landscape analysis, the neural tangent kernel approach, and mean-field analysis. All such viewpoints aim to give an idea of the structure and size of the NN required to ensure global convergence.
Our approach this paper is based on mean-field analysis and gradient-flor flow analysis, the latter being the continuous and mean-field limit of GD. We will examine residual neural networks (ResNets), and study how deep and wide a ResNet needs to be to match the data with high accuracy and high confidence. To relax the assumptions
the activation function as far possible, we follow the setup in (Chizat& & Bach] 2018). which requires this function to be either 2-homogeneous or partially l-homogeneous. We show that both depth and width of the NN depend algebraically on e and n, which are the accuracy and confidence levels, respectively.
Mean-field analysis translates the training process of the ResNet gradient-flow partial differential equation (PDE). The training process evolves weights on connections between neurons. When dealing with wide neural networks, instead of tracing the evolution of each weight individually, one can record the evolution of the full distribution of the weight configuration. This perspective translates the coupled ordinary differential equation system (ODE) that characterizes evolution of individual weights into PDE (the gradient-flow equation) characterizing the evolution of the distribution. The parameters the PDE naturally depend on the properties the activation functions. Gradient-flow

---

Under review as a conference paper at ICLR 2022
analysis is used show that the PDE drives the solution to point where the loss function becomes zero. We obtain our results on zero-loss training of ResNet with GD by translating the zero-loss
property of the gradient-flow fllow PDE back to the discrete-step setting.
This strategy of the proof was taken in an carlier paper (Ding et al. 2021) where multi-layer ResNets were also analyzed. The main difference in this current paper is that the assumptions the activation function and the initial training state for obtaining the global convergence are both much relaxed. This paper adopts the setup from (Chizat & Bach 2018) of minimal Lipschitz continuity requirements on the activation function. Furthermore, the paper (Ding et al. (02T)) required a dense support condition be satisfied on the final parameter configuration has support condition. This condition hard to justify in any realistic setting, and is discared from current paper. Further details on these issues appear in Section
We discuss the setup of the problem and formally derive the continuous and mean-field limits in Section [ Section 3 we discuss related work, identify our contribution and present the main theorem in its general terms. After precise definitions and assumptions are specified in Section ] we present the two main ingredients the proof strategy. The mean-field limit is obtained by connecting the training process of the ResNet to gradient-flow PDE in Section S and the zero-loss property of the limiting PDE is verified Section 6
The main theorem is a direct corollary of Theorem 5.1 and
Theorem (or Theorem 6.2
RESNET AND GRADIENT DESCENT
The ResNet can be specified as follows:
2441 (I) (x) MT. (aat))) 1= 1=1,,...,L.
where M and I are the width and depth, respectively; =x(x) = x E IR. is the input data; and (x) is the output from the last layer. The configuration of the NN is encoded in parameters OI,,M nm ) where each parameter 0r,m a vector T&& and ʃ Rdd x T Rd the [ convesscconventionaa" ResNets, which have the specific form
activation function. The formulation
M
2+1(0)) = (x)
Σ ML m=m
1
(W.m + bl,)), 1= ....1....1111 1,
where Wi,m, Ul.m E Rdd bt.m R, and is the ReLU activation function. In this example, we have 0pm (Wi,,: Ul.m, bt,m) E RR*, with k 2d +1
Denote by Ze,,,r (1; x) the output of the ResNet defined by ( (This quantity the same as 21. (a)) defined above, but we use this alternative notation to emphasize the dependece parameters LL.M The goal of training ResNet is to seek parameters OL.M that minimize the following mismatch or
loss function:
E(OL,M) = Exzny [ (o(Zor,,, (L;x)) v(x))
(2)
where g(x): : IRd is the probability from which the data is drawn.
a given measuring funetion, y(æ) E the label corresponding to æ, and Jr
Classical gradient descent updates the parameters according to the formula
1 L.M O7, A E(",))
where h is the step length. In the limit as - O, the updating process can be characterized by the following ODE (Chizat & Bach 2018 Def 2.2) (rescaled by L, M);
d0 (s)
E(L. for 2 0,
3
ds
where represents pseudo-time, the continuous analog of the discrete stepping process.
2

---

Under review as a conference paper at ICLR 2022
2.1 THE CONTINUOUS LIMIT AND THE MEAN-FIEL LIMIT
The continuous limil of I is obtained when the ResNet is infinitely deep, with L + oc. By
reparametrizing the indices 1 [0..... L-1] with the continuous variable / E [0, 1]. we can view in ① as a function in that satisfies a coupled ODE, with 1/L being the stepsize in L. Accordingly, ft.m can be recast 0m (t 1/ L). and denoting O {6m (t) (t)m))m we can write the continuous limit of (T as
dz(t; σ
M Σ
(m))))
tE [0, with z(0; = 2.
(4)
dt
M
m=
Extending
we define the cost functional E as
2 (g(Ze(1:=)) - v(x))²
((O)
Ezzp
(5)
where (t; 2) solves 4 for given collection ( of the M functions (0m (t)). Similar to (3)
we can use GD to find the configuration of ((t) that minimizes (5) by making ( flow in the
descending direction of E(@). Denote the pseudo-time of the training process, and ((s;)) the
collection of functions at the training time
ae M
SE
8 8> 0,
E [0.
(6)
Ös
80
((s))
where 6E is the functional derivative of E with respect to θ, and thus a list of M functions of t for de every fixed
The mean-field limit is obtained by making the ResNet infinitely wide, that is, M + co. Considering that the right hand side of (4) has the form of an expectation, approaches an integral in the limit, with respect to a certain probability density. Denoting this PDF by p(0,t) t) e C((o, 1}: and assuming that the 6m are drawn from the ODE for 2 translates to the following:
dz(l;a) dt
s(ttt)))))) Op(0( I E [0, 1]
with =(0; x)) x.
Mimicking (5. we define the following cost function in the mean-field setting:
E(p) = Enep 2 (g(Zx(1:z) - v(x))2
(8)
where (p; .)) is the solution to for a given p. Then, similar to the gradient flow for θ L.M and @(t), the probability distribution P that encodes the configuration of 9 flows in the descending direction of E(p) in pseudo-time s. Since p(0, f, needs to be probability density for all 8 and f. its evolution in 8 is characterized by gradient flow in the Wasserstein metric (Chizat & Bach 2018
Lu et al. 2020 Ding et al.) 2021):
0p Vo. Os
SE
(VB 5p ol.ris))
8 > O, E [0.1] with p(0,4,) - (0,
SE is the functional derivative with respect to p. and thus function of (0, 2) for every fixed s. where 8p Using the classical calculus-ot of variations method, this functional derivative can be computed as:
8E
(0,t) = Ev-n (t:
(10)
8p
where po(:2), parameterized by x, maps [0, 1] + Rd. and is a vector solution to the following ODE:
dp
Pp / ()((Z,,0)p(0,))) do
(11)
dt
JRk
AA collection of probability distribution that is continuous in and has bounded second moment in 9 for all t.
The definition made rigorous in Def
3

---

Under review as a conference paper at ICLR 2022
with Pp(t = 1;x) (I(Zz(1:z)) y(z)) dependence, we use (F(e(e)) and {F(r(a)) to denote
Vg(Zz(1: In the later sections, to emphasize the SE
4E 36 (
and op p(r))))
respectively, As
8p
summary, to update p(0. t, s) to p( t, s+ + ds) with an infinitesimal ós, we solve 7 for Z (t: x). [E(p(s)) (0, t)
using the given p(0, s)) and compute Pp using (T... This then allows us to compute op which, in turn, yields p(0, t, s + os) from In 0zf d x d matrix that stands for the Jacobian of f with respect to its 2 argument.
3 RELATED WORK AND CONTRIBUTION
There is a vast literature addressing the overparameterization of DNN. Many perspectives have been taken to justify the success of the application of the first order (gradient descent) optimization methods, in this overparameterized regime. We briefly review related works, and identify our contribution.
The earliest approach to understanding overparametrization was landscape analysis, in which the countours of the nonconvex objective function were studied to find which properties make it possible for a first order method to converge to the optimizer. Different NN structures are then analyzed to see which have these properties (Jin et al., 20177; Ge et al. 20155 Du al., 2017 Ge et al. [0018] Nguyen & Hein 2018 Du & Lee 2018)) Soltanolkotal et al. 2019 Nguyen & Hein 2017 Kawaguchi 2016)
Yun et al. 2018). This approach naturally limits the types of DNN that can be "explained." since most DNN structures do not satisfy the required properties.
Another approach taken in the literature is related to the Neural Tangent Kernel (NTK) regime, which is the regime in which the nonlinear problem is reduced to nearly linear model due to the confinement of the iterates to small region around the initial values. Insensitivity of the so-called
Gram matrix is evaluated in the limit of the number of weights (Allen-Zhu et al. 2019: Du et al.
2019a Zhang et al. 2019; Chatterji et al.) 2021; Du al. 2019b Jacot et al. 20188] Liu al. 2020 Frei et al., 2019). The argument is that zero-loss solutions are close to every point in the space, and one can find an optimal point within a small region of the initial guess. The NTK arguments
are shown to work well in several real application problems, such as the classification problem (Li
|Liinng 2018: Zou et al. 2019). However, as pointed out by (Ba et al. 2020 Wei et al. 2019) 9 Fang et al. 2019), 9) NTK approximately views nonlinear DNN as linear kernel model, a rather
limited description so the estimates obtained through NTK might not be sharp. Indeed, the empirical
observation in AlllenZZhu & Li 2019 Arora et al. 2019) have suggested that the kernel models are not as general as NN, and certain
features of NN are not captured.
Finally, there is the mean-field limit perspective that we adopt in this paper. The term "mean-field indicates that in a system with a large ensemble of particles, the field formed by averaging across all samples exerts a force each sample. Instead of tracing the trajectory of each sample, one can characterize the evolution of the full distribution function that represents the field. This idea originated in statistical physics, and is made rigorous under the framework of kinetic theory. In training an overparametrized ResNet context, a large number of weights evolve to decrease the cost function. In the mean-field limit, the training process evolves the distribution function of these weights. A significant advantage of the mean-fiel field approach is that once we derive formula for the gradient flow, standard PDE techniques can be adopted describe the convergence behavior. This approach was taken in (Araújo et al. 2019, Fang et al. [2019 Nguyen| 2019: Du et al. [019a] Chatterji et al. 2022 Chizat& & Bach 2018] Mei et al. 2018: Wojtowytsch 2020 Lu et al. 2020 Sirignano & Spiliopoulos 2021; The case of a single hidden layer NN in the regime M oo is studied by Chizat & Bach (2018) Mei et al. (2018); Wojtowytsch (2020), who justified the mean-field approach and demonstrated convergence of the gradient flow process to a zero objective. In the multi-layer case, Lu et al. (2020) showed the convergence of PDE that can be viewed as a modified version of the true gradient flow, hinting at convergence of the real mean-field limit. Nguyen & Pham (021) also gave the global convergence of the mean-field limit of DNN for certain class of NN structures, but their work excludes such important practical NN structures as ResNet. The work most closely related to ours is (Ding et al., 2. but this paper makes technical assumptions on P.o and f that restrict the usefulness of the results, as we discuss below following the statement of Theorem 3.1
We note that in certain parameter regimes, the mean-field and NTK perspectives can sometimes be unified; see (Chen et al. 2020).
4

---

Under review as a conference paper at ICLR 2022
We follow the roadmap of Chizat & Bach 2018): : Ding et al. (2021), which shows that the PDE (9) achieves the global minimum for which E ( f. 8 = x)) ) = 0, and that the gradient flow in the dis- crete setting (3) can be closely approximated by the PDE, so that E,
E(p(...s) These
two results together show that L. (8)) ) as 0 for pseudo-time 8 sufficiently large. Specifically, the two main tasks of the paper
follows.
Task We need to give a rigorous proof of the continuous and mean-field limit. This will be stated in Theorem 5.1 to justify that for every fixed oo, when M, L + oo, (((L.M(s)) R E()(s:)) R s(p(..,,)) The dependence of these approximations on L and M are made precise.
Task 2: We need to demonstrate the convergence to global minimum. This is stated in Theorem 6.. and 6.2 for two different cases. In both theorems, we obtain the global convergence for the gradient flow, assuming certain homogeneity and the Sard-type regularity for f. weak assumption of the initialization of Pini is also imposed.
By combining these two, we obtain the main result of the paper.
Theorem 3.1 Let the conditions in Theorem 5.7 and 6. (or
(or 6.2) hold. Then for any positive and there exist positive constants Co depending on Pini (0,,0, 1), and C depending on Pinil (0, (), 8 such that when
Co, M> C(pini (0, (0,
C(rri (0, 1),
L>
2n
we have
P ((E(OL (s)) S 1-7.
where defined in 2 and θ L.M solves
This theorem gives quantitative bounds for M and L. The number of weights required to reduce the cost function below is O(ML) = OL)ee). The theorem also suggests that L and M are independent parameters.
The results resonate with those obtained in (Chizat & Bach 2018) for the 2-layer NN, and extend those in (Ding et al. 202T) greatly. Specifically, compared with the results in (Chizat & Bach 2018). where p(0,,) s) follows typical gradient flow in the probability space on in time (Ambrosio et al.. 2008), we have, at each training time 8, a "list" of probability measures p(0, () on 0, for all 7. The members of this list are coupled, flowing together in the descending direction of the cost function F. New analytical estimates are developed to deal with this non-traditional gradient flow.
Ding et al. (202)) takes similar approach to ours, but their assumptions on the support of p(0,t, t, 00) are quite strong: The limiting probability measure p(0, t, oo) is assumed to have the full support over 0. The assumption greatly reduces the technical difficulty of the proof, but is impractical and hard to justify, thus preventing the results from being of practical use. In the current paper, this support condition is replaced by the well-accepted homogeneity condition adopted by (Chizat & Bach 2018). As consequence, the structure of the gradient flow must be examined closely to demonstrate convergence, requiring considerable technical complications, Lu et al.) (2020) also investigates gradient flow for training multi-layer neural network, but the gradient flow structure is modified for mathematical convenience. All blocks are integrated together, making p a probability measure over the full (0, spscee This design is inconsistent with the structure of the ResNet design that we investigate in this paper.
4 NOTATIONS, ASSUMPTIONS, AND DEFINITIONS
Throughout the paper we denote the collection of probability distributions with bounded second moments as pa (R*), that is, p2 (R*) {p fg. 10 dp(() < oo). We assume certain regularity properties for the activation function f. the measuring function g, the data y, and the input measure μ, as follows.
Assumption 4.1 (Assumptions on f) Let f Rd I2t + Rd be C² function.
1. (linear growth) For all a e Rdd 0 E IRA, there a constant C₁ such that
|/| (10 1)(l) [ 1).
(12)
5

---

Under review as a conference paper at ICLR 2022
2. (locally Lipschitz) For all r O, and |x] <,,0 R², we have for C2(r) monotonically increasing with respect to r that the following bounds hold:
|0,fl - C2(r)(10)) + +1, |0af1 C2(r)(10)+ +1)
(13)
3. (local smoothness) There exists by E (0, k] with the following property: Denoting
011)), = max{|=] |-| [0n1]. where Onj E Rh,, 0/2) E Rk-k, we have for Cy(r) monotonically increasing with respect to r that the following bounds hold:
0200(1x (xx, 0) S CC((r),
i+j i,j 0.
(14)
When h1 <k, we have in addition that max S C3(r) (191g) + +1), 1 fl s C2((xl) 11m]++1) (15)
4. (universal kernel) The function set { = fox f(2,0) PE (²()tt} is dense in C (|æ) R, for all R > O.
Assumption 4.2 (Assumptions data) Let g. probability distribution of x. We assume the following.
R R be C² functions, and let μ be the
5. p(x) is compactly supported, meaning that there is R, > 0 such that the support of is within ball of size Ry around the origin, that is, supp(μ) C Bnn (Ö).
6. y(z) E Line (R)), that is, sup [=()) < co.
7. g(x) and Vg(z) are Lipschitz continuous. Moreover, [V(()) has a positive lower bound, that is, inf [g(x)]
We note that Assumption 4.1 admits many commonly used activation functions (E et 2020 One example of a function satisfying this assumption is f fxr 0) = f(z 111, 02)) 0(s)) = Oa] +12)))))))
where 011 E Rddd 022 E Rd, O(3) IR and is component-wise regularized ReLU activation function, see Remark H.2
We now build the metric the function space for solutions. Note that the solution p(0, t, s) to (9) is expected to be a continuous function in (t, s), and a distribution of for each (t, s). For this non-standard probability space, first introduce the following metric.
Definition 4.1 F Ac P2) is collection of continuous paths of probability distribution v(0, t) (0 E RR E 1.1)) where 1. t) E (R*) for every fixed t E [0, 1]; 2. For any to E [0. 1), lim W2 (,,,
= 0, where W2 is the classical Wasserstein-2 distance. The space C((O, P2) is equipped with the following metric di (2), = sup, W2(u (-,(), v2(:,)))
Accordingly, C((O, oo); C(O, P²)) is a collection of continuous paths of probability distribution (0,oo)), where 1. v((;,,)) E C((O, 1}; P2)
) for every fixed s E [0, oo). 2. any 80 [0. oo), lim di (u(s,, 8),u(( ((,)(()))))))))))))))) (0)) = (where d2 is defined above). The metric in C(JO, oo); C(10, 1]; P²)) is defined by d2 (V1, = sup,,, W2(21(-. (t),,)) s), s))..
v(0, 1, (with O E RR, L E [0, 1J.
Since 22 is complete in W₂ distance, C(JO. 1]: P2) and C(1O, oo); C((O, 1}: are complete metric spaces under di and d2 respectively also. To give a rigorous justification of the mean-field limit, we use the particle representation of p(0, f, s). Thus, at least we need to assume that we can find stochastic process that drawn from the initial condition (0, We call such initial conditions adnissible.
Definition 4.2 We call continuous path of probability distribution v(0,t) C(1O, 11: P²)) ad- missible if it has a particle representation, namely there exists a continuous stochastic process ((t) [0, 1] + & and r >> 0 such that for any to [0. 1), we have
0(to) to),
lim E (10(1) (( 0, t-
10(10) <r.
(16)
Pc{[0, T]; .) is defined to be the set of functions f(a t) such that for any LE [0, f(, E A and f(,t) is continuous with respect to f under the metric defined on A. In Definition 4.1 we set T = 1 and A = P2 where the natural metric on A is W2.

---

Under review as a conference paper at ICLR 2022
Furthermore, v(0,t) t) is called unnit-admsssible if its M M-averaged trajectory is bounded and Lipschitz with high probability. (See the rigorous definition in Definition E.1
We note that without the dependence I, probability distributions "admissible", in the sense that one can draw sample from given distribution. In Appendix [ we show that if the imitial condition Plni(0, is admissible, has unique solution p(0, i, s) that admissible for each s.
The global convergence result depends on Sard-type regularity, defined as follows.
Definition 4.3 Civen a metric space θ, a differentiable function h e - R. and subset e θ say h satisfies Sard-rype regularity in @ if the set regular values of hlg is dense in its range, where hl@ ② + IR is a confinement of h in ②.
Remark 4.1 This regularity assumption is not common one; we have adopted it from Chizat & Bach (2018). This property essentially that most of the points in the range of lie an open set and h is locally monotonic. The assumption is rather mild and can be satisfied by most commonly seen regular functions, unless the function oscillates wildly.
MEAN-FIELD AND CONTINUOUS LIMIT
In this section, focus on the justification of mean-field and continuous-limit result. This is prove that E(p(:.,,)) s)), ({O(s;) -,))), and E( (s)) are asymptotically close to each other for every In the next section, we prove convergence of Epp(, s))
cc.
To show the asymptotic equivalence of the three quantities, we need to compare (3. (6 and and take the measurement E. according to (5) and (8)
Theorem 5.1 Suppose that Assumptions 4. and 4.2 are satisfied. Assume that Pini (0, t) is limit- admissible and (Pmii( (.f.t)) t)) C [6) [1] I
RJ with some R > 0 for all t E [0, 1]. Let
( (0; 1) JM in (6) be i.i.d. drawn from (0n t). Let
(s,)s(s) = (+)) be the solution to with initial condition (s = 0) = Om (0; t.);
(s; be the solution 6 with the initial condition 0n to;
((0,t,s) be the solution to with initial condition Pini (m.
Then for any positive n, and S, there exists constamt C > -0 that depends Pimi (0, 1) and S such that when
(()im(0,t).S)
I. C(Pma(0,t)..
M
8 SSS,
2n
we have:
- E(O(s; min(F (
e/2) (IE(O((::)) Elp(.,,, ())1 & <e/2)) 2 1- 1n
follows that
P P(IE(Or.M(s)) - E(p( (8)()))))))) <6) 1- Vs < 5. Here (s),,, ((O(s;) )), and E(p(s.,8) 8)) are defined 2 and g respectively.
The proof of this result appears in Appendix This theorem suggests that for every fixed S> the gradient descent of O is approximately the same as the gradient flow of p(0, l), in the sense that the two costs are close to each other with high probability, when L and M are sufficiently large. The size of the ResNet depends aggatie-algebrrically on (the desired accuracy) and 1/ (the confidence of success). The result translates the evolution (gradient descent) of ÓL,M to the evolution (gradient flow) of p(0, t), and thus matches the zero-loss property of the parameter configuration of a finite sized ResNet to its limiting PDE, whose analysis can be performed with standard PDE tools.
For function h θ + R, regular value is real number in the range of h such that h A-a(a) is included open where h diffcrentiable and where dh does vanish.
7

---

Under review as a conference paper at ICLR 2022
The proof of Theorem 5.1 divides naturally into two components. We show that for all S, E(p(....)) x ((: )) and (()(s:)) (((L.a((s) with high probability. The former is obtained from mean-field limit theory, justifying that the particle trajectory 6.. (t. follows p(0, t, s) for all in pseudo-time & E S). The latter makes use of continuity / and traces the differences between pm (i) and ft. These two components of the proof are summarized in Theorems E. and E.2 respectively. According to the formula of the Fréchet derivatives (TO) and (IT), the estimates in these theorems naturally route through the boundedness of Pp. Pe. P and similarly Zp.0.er.uu L. It is technically demanding to derive these bounds, but they portion of the appendix to addressing the well-posessess of these systems. See Appendices [ where we show these equations have unique solutions with proper initial conditions, along with the required bounds. Naturally, these estimates depend on regularity of f' y, and
surprising. We dedicate large
To gain some intuition for the equivalence between 66) and (9), we test them on the same smooth function h(0). To test ( we multiply both sides by h and perform integration by parts to ob- de Jg. h (p(0)) = Jg. VahV, &E(p(s))
tain 4
dp. This is to say 1b) = E (VhVV 6B(p(s)) op
Testing h on 6 also gives the same formula. Supposing that p 1 Em= 6gm. we have 1
da B() =
Σ m== Voh(fm) da fm -- -EM= Voh(fm) àE 50... The right hand side is also 6p (fm,t), This will be shown hold
M
V SE(p(s))
if and only if M (E(0((:)) Vo 8E(p)
6p
60m.
true in Appendix
6 CONVERGENCE TO GLOBAL MINIMIZER
After translating the study of L. to the study of p(0, f), this section presents results on when and how Ep)) converges to zero loss by examining the conditions for the global convergence ( We first identify the property of global minimum.
Proposition 6.1 Suppose that p C((0, ([00,]];P] has E(p) 0. Then for any to E [0, 1). there exists a measure v(0) of TR such that fEx (f) = and
[
8E 8p
(0, dv(0) < <0.
(17)
See Appendix H.1 for the proof of this result. At the stationary point of the cost function, AE
äp (0,4))
then there no satisfying [7]. so E(p) is necessarily trivial. Our task now becomes check under what conditions on f requiring a separation initialization and certain homogeneities. the first case, we require f to be 2-homogeneous, and the result is collected in Section 6.1 In the second case, we require f to be partially 1 -homogeneous; see Section 6.2
have 5E
6p becoming 0 co. We give two possibilities, both
6.1 THE 2-HOMOGENEOUS CASE
The results in this section obtained under the following assumption on (Functions can be designed to satisfy this assumption easily; see Remark HH.l)
Assumption 6.1 The function f(x, 0) fa, 0) for all (x, 0, E Rdd x R x R.
Rdd x RR* is 2-homogeneous, meaning that fax
Theorem 6.1 Let Assumption 6.1 and conditions of Theorem
5.1 hold true with ky k. Let p (0, t) be the long-time time limit of (9), that is, p(8, t, converges to pac (6, t) in C((0, 1]; p2) as + co. Then El E(px) 0 if the following hold:
Separation initialization: There exists to E j0, 1] such that Pinil Pmmi(0, to) separate the spheres S. and r,, for some 0 < Ta <. Tb-
*We that set C' separates the sets A and B if any continuous path in with endpoints in A and B intersects C.
8

---

Under review as a conference paper at ICLR 2022
Sard-type regularity:
(0, satisfies the Sard-type regularity condition in S
6p Pro
proof appears Appendix H.2
corollary of this theorem, when combined with Theorem 5.1 the main result of our paper.
with L = k. gives Theorem 3.1
6.2
THE PARTIALLY 1 1-HOMOGENEOUS CASE
The following assumption is used in this section. (Functions that satisfy this assumption include regularized ReLU; see Remark H.2
Assumption 6.2 Let 0 = (0,) 12)) with 111 R and 022 E 1R=11
1. (partially 1-homogeneous in 0) can be written as f(x,0) f(x, (1,,001.01)) fn 0z)))
2. (locally bounded and smooth) For any r >> 0, f(x, 022) is bounded and Lipschitz with Lipschitz continuous differential for (x, 022)) E (0) 12-1
When Assumption 6.2 holds true, Assumption 4.1 part can be satisfied with k1 1. Then,
introduce the main result this section.:
Theorem 6.2 Let Assumption 6.2 and conditions of Theorem 5.1 hold true with h Let (0, be the limit of (9)
Then Eppoo) = 0 if the following hold:
Separation initialization: There exists E [0, 1] such that pini (0mi 022 (n) separates the spheres { R- and {ro} x IR. for some ro O, where U[]) 8(2) are defined by item 3 with ki = 1.
Assumption 4.1
6E Sard-type regudarity:
((100p)))1)) RR-1 TR satisfies the Sard-type regularity
5p
Ps
condition.
1], P2), define Hr.p (f1)) For any p E C[
SE(el (1,r01z) to) where 0/2) r0(2)
op
with 18zl and ([2) Sk-2, Suppose that Hr,p converges in (SK as function H. Furthermore. assume that Ho satisfies the Sard-type regularity condi-
tion in k and that the intersection of regular values of Hoo,p and (E(e) is also dense in the intersection of their range.
(1, (1,0 to)
6p
Poo
The proof is found in Appendix H.3 This result, combined with Theorem 5.1 in the case of k 1. gives the main result Theorem seemingly tedious. However, we note that only the first two assumptions the separation assumption - are crucial. The third and fourth assumptions concern the regularity of the Fréchet derivative of p; they are rather mild and serve to exclude wildly oscillating functions; see Remark [4.T Most commonly seen functions are regular enough that these two assumptions are satisfied.
The assumptions in Theorem 6.2 are rather technical and 1-homogeneous and
ETHICS STATEMENT
This work does not present any foresecable societal consequence.
REPRODUCIBILITY STATEMENT
The notations, assumptions and definitions clarified in Section 4 And all proofs appear in the Appendix.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Z. Allen-Zhu and Y. Li. What can ResNet learn efficiently, going beyond kernels? In Advances in Neural Information Processing Systems, volume 32, 2019.
Allen-Zhu, Li, and Z. Song. convergence theory for deep learning via over-paramettrriation In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 242-252. PMLR, 09-15 Jun 2019.
L. Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science and Business Media. Birkhäuser Basel, 2008
D. Araújo, R. Oliveira, and D. Yukimura. A mean-field limit for certain deep neural networks. arXiv/1906.00193, 2019.
S. Arora, S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In NeurlPS, 2019.
J. Ba, M. Erdogdu, T. Suzuki, D. Wu, and T. Zhang. Generalization two-layer neural networks: An asymptotic viewpoint. In ICLR, 2020.
N. Chatterji, P. Long, and P. Bartletr. When does gradient descent with logistic loss interpolate using deep networks with smoothed relu activations? UrXiv/212.00499. 2021.
Z. Chen, Y. Cao, Q. Gu, and T. Zhang. A generalized neural tangent kereel analysis for two- layer neural networks. In Advances in Neural Information Processing Systems, volume 33, pp. 13363-13373, 2020.
L. Chizat and F. Bach. On the global convergence of gradient descent for over- wer-paramettrized models using optimal transport. In Advances in Neural Information Processing Systems, volume 31, 2018.
Z. Ding, S. Chen, Q. Li, and S. Wright. Overparameterization of deep resnet: zero loss and mean-field analysis. arXiv/2105.1 14417, 2021.
and J. Lee. On the power of over- paeemerrametinnnooonn in neural networks with quadratic activation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1329-1338. PMLR, 10-15 Jul 2018.
S. Du, C. Jin, J. Lee, M. Jordan, A. Singh, and B. Póczos. Gradient descent can take exponential time to escape saddle points. In Advances in Neural Information Processing Systems, volume 30, 2017.
S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1675-1685, 09-15 Jun 2019a.
Du, X. Zhai, B. Póezos, and A. Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference Learning Representations, 2019b.
W. E, C. Ma, S. Wojtowytsch, and L. Wu. Towards mathematical understanding of neural network- based machine learning: what we know and what we don't. wwXiv/2009.0071: 2020.
C. Fang, Y. Gu, W. Zhang, and T. Zhang. Convex formulation of overparameterized deep neural networks. arXiv/1911/07626, 2019.
Frei, Y. Cao, and Q Gu. Algorithm-dependent generalization bounds for overparameterized deep residual networks. In NeurlPS, 2019.
R. Ge, F. Huang, C. Jin, and Y. Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pp. 797-842. PMLR, 03-06 Jul 2015.
R. Ge, J. Lee, and T. Ma. Learning one-hidden-layer neural networks with landscape design. In International Conference on Learning Representations, 2018
10

---

Under review as a conference paper at ICLR 2022
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
C. Jin, R. Ge, P. Netrapalli, S. Kakade, and M. Jordan. How to escape saddle points efficiently, In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1724-1732. PMLR, 06-11 Aug 2017.
K. Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information Processing Systems, volume 29, 2016.
Y. Li and Y. Liang. Learning overparameterized neural networks via stochastic gradient descent structured data. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, pp. 8168-8177, 8177,2015 2018
C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kernel constant. In Advances in Neural Information Processing Systems, volume 33, pp. 15954-15964, 2020.
Y. Lu, C. Ma. Y. Lu, J. Lu, and L. Ying. A mean field analysis of deep ResNet and beyond: Towards provably optimization via overparameterization from depth. Proceedings of the 37th International Conference Machine Learning, volume 119, pp. 6426-6436, 13-18 Jul 2020.
S. Mei, A. Montanari, and P. M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 11((3)):77665-E777) 2018.
P. M. Nguyen. Mean field limit of the learning dynamics of multilayer neural networks. arXiv/1902.02280 902.0/800.,.8 2019.
M. Nguyen and H. Pham. A rigorous framework for the mean field limit multilayer neural networks. arxiv/2001.11445 2021.
Q Nguyen and M. Hein. The loss surface of deep and wide neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.
PMLR, 06-11 Aug 2017.
Q. Nguyen and M. Hein. Optimization landscape and expressivity of deep cnns. In ICML, 2018.
Sirignano and K. Spiliopoulos. Mean field analysis neural networks: law of large numbers. SIAM Journal on Applied Mathemalics, 80(2):725-75) 2020.
J. Sirignano and K. Spiliopoulos. Mean field analysis of deep neural networks. Mathematics of Operations Research, 2021. doi: 10. (0.1287/mor.2000.111
M. Soltanolkotabi, A. Javanmard, and J. Lee. Theoretical insights into the optimization landscape of pwer-parameterized shallow neural networks. IEEE Trans. Inf. Theor, 65(2):742-769, February 2019. ISSN 0018-9448.
C. Wei, J. Lee, Q. Liu, and T. Ma. Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. In NeurIPS, 2019.
S. Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime. arXiv/2005/135500 2020.
C. Yun, S. Sra, and A. Jadbabaie. Global optimality conditions for deep neural networks. In International Conference on Learning Representations, 2018.
H. Zhang, D. Yu, M. Yi, W. Chen, and T. Liu. Convergence theory of learning over parameteeeee resnet: full characterization. arXiv/1903.00 12000, 2019.
D. Zou, Y. Cao, D. Zhou, and Q Gu. Gradient descent optimizes wer-parameterized deep relu networks. Machine Learning, 109:467-492. 2019.
11

---

Under review as a conference paper at ICLR 2022
APPENDIX: INTRODUCTION
The appendix contains proofs and supporting analysis for the theorems in the main text. We start by proving well-posedness of the ResNet ODE and the gradient flow. We then justify the continuous and mean-field limit. Finally, we prove the global convergence of the gradient-flow PDE. The sections are organized as follows.
Appendix A Well-posedness. We summarize the well-posedness of the continuous limit ODE ( the mean field limit ODE (7). and the associated gradient flows collected in Theorem A.1
and 99. The results are
Appendix B
This section collects the detailed proof for the well-posedness of the ResNer ODE in
its continuous limit (4) and the mean-fied -ield limit (7), and finalizes the proof of Theorem A.2
This section prepares some a-priori estimates for showing the well-poseddness of the gradient flow equations.
Appendix
C
Appendix D
This section contains a detailed proof for the well-posedness of gradient flow equa-
tions and and finalizes the proofs of Theorem A.3 and A.4
Appendix Proof of Theorem 5.1 the continuous and mean- feean-field limit. Section E lays out the structure of the proof, Section shows the continuous limit, and Section G shows the mean-fieh
limit.
Appendix H Proof of Theorem 6. and 6.2 Global convergence of the gradient flow.
The analytical core of the paper lies in appendicesHHandd which describe properties of the gradient flow PDEs and explain why the gradient descent method for ResNet can be explained by these equations. The technical results of Appendix B can be skipped by readers who are interested to proofs of the main results.
Throughout, we denote by C(-) generic constant whose value depends only on its arguments. The precise value this constant may change each time it invoked. Throughout Appendices we assume that Assumption 4.1 holds for some O < kn S k.
A WELL-POSEDNESS RESULT
In this section, we show the well-posedness of ODEs 44, and gradient flows
3
6 and 99.
Theorem A.1 The following claims hold.
Well-posedness of ODE:
IIf (fm((t MM m= is contimuous, then (4 has a unique C³ solution. pf E C([O, 1}; then 7 has a unique C² solution.
.Well Well-posedness of gradient flow:
has unique solution.
If (0 (0; {) is continuous, then 66 has unique solution (8m(s; that is continuous in (s, t).
If Puni (0, t) is admissible and suppo(Pmm
C (0 < R} with some R > for all t e [0, 1), then (9) has a unique solution p(0, t, 8) in C( oo); C((o, 1]; with initial condition (0, t). Furthermore, for each s, p(0, t, s) is admissible.
Note that the well-posedness of and are direct corollaries of that of 7 and ( (the corresponding continuous versions), according to Remark posedness on the continuous level, in Theorems
A.2 2 Thus, we merely prove well- Specifically,
Theorem A.2
Appendix A.1 shows the well-posedness of the dynamical system for 2;
Theorems A.3 and A.4 (Appendix A.2 justify the well-posedness of the gradient flow of the parameter configuration.
12

---

Under review as a conference paper at ICLR 2022
A.1 WELL-POSEDNESS OF THE OID
As L and M approach bo, satisfies the ordinary-intes equation 7 We justify that this differential equation is well-posed, in the sense that the solution unique and stable.
Theorem A.2 Suppose that Assumption 4.1 holds with and that is in the support of p., then has unique C solution. Specifically, for P2 E C(O0, 1]; P²), we have pp (f:x:) S C(C1),
(18)
and for all E [0.
P2 (1; - Pz (t: S (2)
(19)
where Li are the second moments of Pt, that is,
C = [.¹ / dp? dt, = 1,2.
(20)
We leave the proof to Appendix B Besides the well-posedness result, the theorem also suggests that small perturbation to p is reflected linearly in Zp. the solution to (7). This means that small perturbation in the parameterization of the ResNet leads to only small perturbation to the ResNet output.
Remark A.1 Although we do not directly show the well-posede poseepossss of (4), follow immediately from Theorem A.2
One way make this connection is reformulate the discrete probability distribution
as
M
1
pdis
Σ
50m.
(f)
M
m=m
where (t)) {0 1 is the list of trajectories. Since Om (1) continuous t, we have pdii (0, E C([0, 1]: P Since
M
Σ f(e(t:z) (m) =
( - f(e(t;z), do
M
m=1
using Theorem A.2
has a unique C solution when @(t) is continuous.
A.2
WELL-POSHDNESS OF THE GRADIENT FLOW
The gradient flow of the parameterization is also well-posed. both in the discrete setting and the continuous mean-field limit. In the discrete setting, we have the following result.
Theorem A.3 has a unique solution.
Further, 9 characterizes the dynamics of the continuous mean-field limit of the parameter configura- tion, and is also well-posed.
Theorem A.4 If Pini (0,t) is admissible and (uppo (0, (0,t)) C (m] S R3 with some R 0 all E [0, 1). Then (9) has a unique solution p(0, t, s) in C((O, oo): C((,, 1]: P2)) that admissible for each 8. Further, we have
dE(p((.,,s) ds The proofs these two theorems can be found in Appendix
<
(21)
Remark A.2 Using the same argument as in Remark A.1 calling
dis (,tts))
1
Σ
80
(stt(sst)( (8),
(22)
M
the well-posedness of T6 follows immediately from Theorem A.4 definition (5) and have
Furthermore, according to the
(O(s:))) As a consequence, if O satisfies (6). then prii satisfies (9). and vice versa. The well well-posedness
E.
result in Theorem A.4 for then can be extended justify well-posedness of (6).
13

---

Under review as a conference paper at ICLR 2022
B PROOF OF THEOREM A.2
This section contains the proof of Theorem A.2
We rewrite 7
as follows:
dZp(t:z) F(Zp, t),
Vt [0, 1] with 2(0;z) = x,
(23)
dt
where for given E C(O0, 1]; p2) we use the notation:
F(2,1) = J ((2,0)
The proof of Theorem A.2 relies on the classical Lipschitz condition for the well-posedness of an ODE.
Proof [Proof of Theorem A.2
Since p E C((0, 1]; P²),, we have a constant C such that
10 dp(0,t) C c0.
supp 05151 JR
For any E [0, 1]. using 12) from Assumption 4.1 equation, we have
[f(z,0) do(0, <1(x+++ / gt 1))(0))
FF(z,)) J
(24)
To show the boundedness result (88) we multiply 23 by Zo (t: x) and use (24 obtain
Zp xt;
S2C (lZp,1 2 + 1Zp 1) (1012 (p,
dt
S4C₁ R
(10)2 + (p, (1Z,, (t: +1)
Using Grönwall's inequality, we have
1Z, (t: s exp 20, (f,) / 1012 dp dt +
(|x) + +1)
S exp (201 (f1 + 1)) (xx++)).
where L1 = So f 1 dei dt. Since a E supp (so that |x| < R), we have
18
This gives us an
a-priori estimate of 23
Next, using 13) from Assumption we have
(s 0) f(22, 0) s C2([=1 + (22)) (1012 + 1) 22 - 22l
(25)
Then, using boundedness of the second moment 6, we have
| (f(21, 0) f(22, 0)) dp((,t)
[F(,,, t) (22,,t)
C2(1211 + (22))
(1012 + dp(( t) 22l
R*
< C2(12 122 (C + 1)[21 - -221.
which implies that F(2,t) is locally Lipschitz in 2 for all E [0. 1]. Combining this with the a-priori estimate in (18), classical ODE theory implies that (33 has unique C¹ solution.
To prove the stability result (19). we define Zp. as 23 parameterized by P: E C((O, 1]: and denote
A (t; x) = Z,, (t: Zp (t; xt;
14

---

Under review as a conference paper at ICLR 2022
Then by subtracting the two equations, we obtain
d 21)(1; Al1:2));
dt
( Jg
S (()p)((t=),))dpp2((11
=2
A(liz),
[ R2
((xx) [ f(Z (dp)
(26)
(1)
Jec
+ A(t:z),
We now bound (I) and (IT). For (I), we have using 13 and 25 that
I(D) $2C² (1 (t:z) + Z (t:x))) (t: x> a
(10)2 + 1p
(27)
<C(L), / (1012 d),
where the second inequality comes from (88 For (II), denote the particle representation
0 (t) PI (0, t) and (2(t) ~ P2(0, t) such that (E (101 - 02l3)) 1/2 W2(p1(, pa(:,)) Then
((D) < (1])
S (() + |0] + 1)|0 - 021)
s ((0)+))(((
021)
s C(C2) (E (10)) +1)=++)))))) (B (10) -212)) 1/2
(28)
C((2) (B (10,12 |0012
+ 1)) 1/2 (V(t)((.
< (((2) (L 10 (0, / 10 (0,
+1) ()++)
1/2
di (p1,p2);
g²
R
where we use mean-value theorem and 13 in the first inequality to obtain
1((22(15xx)00 J(Zp, (1; xx)) 02)l 510a (4: x), 01 + 00) - 02l
(Z (t: (1; 10al 1)|0: 021.
for some À E [0, 1]. We use (88 in the second inequality of (88
Plugging
27
28
into 26 and using Hölder's inequality, we obtain that
d x) dt
(2)
dp) (L dp)) + / g+
[0(dp2(0,t)+11 1/2 d2(p).22)
+ +((C)))((t:))
SC(C1, (//, (0) f 10Pdp2(0. +1) (IA(t:)))2 di P2))
where we used Young's 's inequality the last line. Since | x) = 0, we complete the proof of 19 using Grönwall's inequality.
C A-PRIORI ESTIMATION OF THE COST FUNCTION
Some priori estimates necessary in the proof for the main theorems. We first consider the case when if satisfies only Assumption 4.1 with 0 < k < k. (Better a a-priori estimates can be obtained when f also satisfies the homogeneity properties of Sections 6.1 or 6.2
15

---

Under review as a conference paper at ICLR 2022
C.I
A-PRIORI ESTIMATE FOR GENERAL
According to (T)). the Fréchet derivative can be computed, similarly to (Lu et al. 2020). as follows:
(E(P)
öp (0,t) Ezza (t2)
(29)
where 2)( is the solution to the following ODE:
Opp -p/ { ())
Ot g Pp = (o(Z,(1;z)) v(z)) (1;))(
(30)
We now show that Pp is Lipschitz continuous with respect to p.
Lemma C.1 Suppose that is in the support of p. Suppose that p1. P2 C((0, 1); and Per Pez
are the corresponding solutions of (30). Denote Ci and L2 as in 20 and
= min (r(spp(())) Usupp(p2) C 0 Oul <r}]
Then the following two bounds are satisfied:
(31)
and
Pp (1;x) - (t:(t:)) C(R. () (p))
(32)
where C max C2).
Proof From
13
in Assumption 4.1
0. (z, dpr dp ((;2m((;=))) / (1012 + d),
(33)
sC((,) / (101 + 1)
gt
where we use (18) in the second inequality. It follows from the initial condition of 30 that
Ppr (1;x)) s (1, 1) C(C.),
where Assumption 4.1 in the first inequality and (18) in the second inequality. Noting that (30
a linear equation, (31) follows naturally when we combine 333) with the inequality above.
To prove
32..
define
A(;:)) = Poi (t: Pps (t:
For = 1, with æ E supp μ. we have
[A(1:))
Ip,, (1; Po. (1;xr]
= (g(Z,, (1;z)) - v(x)) (1,,,,);))) - (=(2,) - v(x)) (4;z))
(34)
< Zp, (1;;) (1:z) Zp:
s C((d) (21,
where we use Assumption 4. 18). and [x| < R2 in the first inequality and 19 in the second inequality. The following ODE satisfied by A:
AA DAT(LE (1;#)) = - (t; /
(p,) + (;) (t
(35)
Øt
gt
where
D (p:0o = O ()) (t))
(36)
16

---

Under review as a conference paper at ICLR 2022
To show the boundedness of D pa,,pa(tt we follow the same strategy that for (2)) By splitting into two terms, we obtain
p11102 (1;#))
S
0.5(
(
(37)
| R2
- /
+
(tz
0. (2:(2
12*
(II)
The bound of (II) relies on part 3 of Assumption 4.1 {0) [u] < R] and Z is bounded by (T8). we have
Because the supports p1, P2 are contained in
[(I)] < C(R, (),, (4;z) Zra (1; S (d11.)))))
(38)
where we use 19 in the second inequality.
To bound (I),
the particle representation 0, pi (0, and O2 p2(0,t) such that (E (10, 02 3) 1/2 (2(p)(-1),p2(: ()). We then have
(I) - E (10,. (t: (t; 01) (z (t:2),
< C(R, L) (10, - 02l)
(39)
< C(R, () (E (101 0213)) 1/2
s C(R, L)d (p1,
where we use the value theorem, part 3 of Assumption 4.1 with |0 [01, < R. and [02,11| < R. 18 in the second inequality. Substituting (38) and (39) into
we obtain
S C(R, C)d
By substituting this bound into
335 and using (33). we have
d A (t; 1
S C(R, (lA(t:))) di (p1,
(40)
dt
The result 32 follows from the initial condition 34 and Grönwall's inequality.
The second lemma concerns the continuity of V
6E(p)
8p
Lemma C.2 Suppose that p. Pl: P2 E C([O, 1]; P²). Define
[
/
L max 18153
dp, dt,
LuP max sup 1si53 (e Jzx
|0 (p,(t,t),
JRz
and
R = min {r/supp(p)
U Usupp(pi) U supp(p2) c [011mm < <r})
Then for any (0, f), (01,t)) (02, E &R= x [0, and > 0, the following properties hold.
Boundedness:
(E(p) (0.1) <C(E)(10 + 1),
SE(p)
V
Oom
(0,1) s (C())(0)) (41)
5p
8p
Lipschitz continuity in 0 and t; There exists Q1 182 + Rt that depends increasingly on both arguments such that
8E(p) (01.t1)
8E(p)
V öp
V
(82, tz)
8p
(42)
<QI (axx((((0.
10, - 02l Q1
1 m_a (02z + 1) t2),
CouP,
17

---

Under review as a conference paper at ICLR 2022
Lipschitz continuity in p. There exists Q IR? RR that increasingly depends on both arguments such that
(F(p)) 0 6p
(F((2)
(0,8)
(0, - Q(1,
(P).)(p1. (43)
6p
where d1 is defined in Definition
Proof To prove the first bound of (41 we restate
29 as follows
5b(()) (0, = Ex-p (20
V
6p
from which it follows that
(E(P)
V
t)
S Exwn (z(;)(Z((t;
C(C)(1)) +),
0p
where we use (13). the bound of 10 (x,
(8) and
31 in the second inequality. To prove the second bound in (11) we
0)
according Assumption 4.1 part 3 to obtain
(p)
(0, t)
< Er-n (ZZ((t;)))))) (p, C(E)( 1),
6p
To prove 442
we assume L 12 without loss of generality, and use the triangle inequality obtain
8E(p)
8b(p)
(01, V
(02, t2)
öp
8p
Earwy (n) (11;)) 01) - -1)p: -12(01120011(12(11100))0
(1)
+ Bzva (21(tt:2))0f(((2(111)).00 11tp://0....11100000000000111111 (1((1=1000/(222112))00
(44)
(1)
+ Brza (1 - (1:
(II)
To bound (I), the mean-value theorem, Assumption
and I8 obtain
(0ef(Zo(t1:z), 01) - az (z 02)1 f (1 A)01 + x62) - 02l
< C
(. 11,2 (ax((1m1)))
10: - 1421-
For (II), we note first that
Z)(ti;z)- (1) S 1/ [
< ((C) 1/) [ (101² +1) ()(
< (CC)(C + 1)111 tal
< C((T)))) - 12|,
where we use Assumption 4.1
12 together with 18 and L S Luup, This bound implies that
(II)
02) - fz (z) (t) s C(uup, 1 E2 (1Zp(11;2) (1 -Zo(tz:z))
Exe
< C()) (()))(2,((11:)
S C( 10,,1104411 (2l,
where
31 in the first inequality, Assumption 4.1 14p for and 18 in the second inequality.
18

---

Under review as a conference paper at ICLR 2022
To bound term (IIII, we again boundedness of Zp, Pp and the Lipschitz condition of f to obtain
1/- Po ((1:
(po(b1;z)) pp(12;00) =
- c(0) 1/2 / (101 1))
s (C + 1)111 -t2
1 ttll
so that
(III) < Eewp (10))(2(()2;2)),))) (lppp( (b( S (((="))(0)+++++ 1)/41 122,,
where we use Assumption 4.1 11 in the second inequality. By substituting these bounds into (44).
we complete the proof of 422
Finally, to prove
we recall the definition of the Fréchet derivative, to obtain
(E(p))
(0,4) Vo (E(22)
(0,
6p
5p
(10of(Zp, (tzz) 2z (t))) (t) (ttt
(t:
E anp
Pp (4;=))
Eenny (t:4), Pp. (t;2) Po (t:
S0(2 p1) B (1 (t:x) Z,, (t:z)) ((C) + 1)E (Ipp. (t;x) Ppu (t;
<C(c, 10). (1
where we Assumption 4.1 for f with 18 and 331 in the third inequality. In the last inequality.
we use 32
C.2 A-PRIORI ESTIMATE UNDER THE HOMOGENEOUS ASSUMPTION
In Theorem 6 and 6.2
2-homogeneity or partial 1 I-homogeneity are assumed. When these properties hold, we can sharpen the estimates obtained in the previous section. We summarize our results here.
Lemma C.3 Suppose that Assumption 4.1 holds, then there exists constant Cz(r) depending increasingly on r such that for any > with |x| < r and 0 E Rt we have the following.
f is 2- 2-homogeneous (Assumption 6. then
|ohs| S Ca(r) (102 100001 S Ca(r) (10) +1),
|obs| s Cs(r).
(45)
If f is partially 1 I-homogeneous (Assumption 6.2 then
[ < Ca(r)(10)+ +1) 10.0afl S Ca(r)((0) +1) |o0f| < Cs(r)( + +1, (46)
where
denotes the Frobenius norm.
Proof When f satisfies Assumption 4. and Assumption calculation. When f is 2-homogeneous in 0, we have
6.2 46 can be obtained from direct
f(x, 0) 20| (a. 0/10)) (0) 0)
where f1) 0 fz 8) = f (z. 0/10 Naturally, with product rule, the derivatives of f
becomes the products of derivatives of S1 and f2. We then obtain (45). noting that when [0] > 1, have
|og - 0, 10,0afi 0,
10, = 0, 10.fil 2/0),
|0 S 2k,
and
C(r) 10
C(r)
C(r)
|0 C(r), 10.0efel <
10,fel = CC(), 10xfal <
|08.22 <
10
10
19

---

Under review as a conference paper at ICLR 2022
The estimates above allow to improve the stability results in Lemmas and
Lemma C.4 Suppose that Assumption 4.1 holds and let P1: E C((0, 11 P2). Define Li and L2 as in 20 and
sup dpr tE|o.1] JR
Then we have the following results.
If either satisfies Assumption 6. 1
or Assumption 6.2
we have the following properties.
1. Stability of Pp:
(1;x)) - Pez Pea (t:+)) s C(L,, (2)d
(47)
2. Lipschitz continuity in p.: (0,t) Ve 8E(p2) (0,t) $ (2)d,, (10) + 1), (48)
8E(pi)
V 8p
öp
where di is defined in
Lipschitz continuity in and t: If satisfies Assumption 6.1
then for any (01,t1), (02, t2)
R x [0, 1.. we have
(F1) 8p (01,t2) Vo
(F(()) (02, s CC(G)) -8 Ccl (102) +1)[t
öp
(49)
Lipschitz continuity in and t: f satisfies Assumption 6.2
then for any (01,t)), (02, (2)
k x [0,1], we have
sE(P1)
SE(PI) (02, 12)
(01,4)--V
Vo
12
8p
5p
(50)
Remark C.1 We note that in comparing 32 with 47 and with (48 50) the main
differences are the dependence of the bounds on O. The new estimates have explicil (and mild) dependence on O.
Proof First, to prove 47 we let A A(::)) = Pp: (l;x)) Pps (1; r), and recall (35 and (36) Using the boundedness of Z, in 18 and (19), and calling (45) (or (46)). have from 36 that
Pp p2 (t:z) C(L1, C2) (fr
0p (0, + dp22(()) + +1 di (21.
RR
By substituting into 35 and using (31) 13 and Hölder's inequality, we have
d dl=t;x)) dt
<O C2) (fLe dpp)))().)) + / (Pdpp((. + [A(22)))
gt
+CC1, K2) (/ 10 (p +1) (p))
The result 47
follows from the initial condition
34p together with Grönwall's 's inequality.
Next, prove 48 have
(p))
8F((2)
Vo 8p
(0,t) Vo
(0,t)
8p
Sa (10 x), 0)p,, (1; x) 0g (2 (1;z), (pzz(())))
Ba (10gf (t; 0) 005 (t: 8)] 1p (1;
+ Exwg (10s (1;4), 0) 1pl (1;z) Pos (1;
<C(Li, L2) + (Em-p (lZ,, (1;x)) Zoo (1;x))) + Evey (1ppy (1; Poa (1; x)))
<C(L,, C2) (p1,
20

---

Under review as a conference paper at ICLR 2022
where also
(33
18
(or (46) in the second inequality and 19 and 47) in the
final inequality.
Finally, to prove
49
and
50
we have in 44 that
(p()
5E(p1) (02. 12) öp
V
(01,
V
6p
Exnn (t, (11;z), 01) (1t
(51)
Exwn (t):)000 (1::))/(111)))))1 (11:2), 82) Pp (t1: (t) 82))
(1))
+ Brzp (t:3))00) (12; 02)
Pp. (1: (1) 02))
(II)
The boundedness of the three terms above relies on Assumption To bound (1), if f is 2-homogencous, we have
and (45) (or
I(I) & C(C)) - -02,
where we use 18 31
and 0ß.5 S C(|=])- If f satisfies Assumption 6.2 we have
[(D] ((C)) |02] + 1) 02), and f2, C((=))(10) + 1).
where we use 18
The bounds of (II) and (III) are same for both homogeneity assumptions and similar to the proof of Lemma C.2 For (II), we have
(t1:2), 02) 0af( (t2; 02) CCC,)(1) 1) (f1:2) Zp, (t2;
and
1p, (t1: Zp, (t2; S CCC:) 1) tzl
For (III), we also use
((23),))) C(C)) +1)
and
Pp (11; Ppr (f2; s ((C)) + 1) - tzl
These estimates, together with L, S L and (1). prove the claims (49 and (50).
D WELL POSEDNESS GRADIENT FLOW
Theorem A.4 is about well- posedness of the gradient flow equation 9 in the mean-field limit, while
Theorem A.3 shows the corresponding result in the discrete setting. The proof for the two are similar.
We first show the mean-field limit well-posednng posednssss,,, Theorem A.4
D.
PROOF OF THEOREM A. 44
We use the fixed-poine argument. To do so, we first define subset of C((O,S] 1]; P?)) with compact support measures, follows:
Ug (0(0,, s) E (((0,,))(((001;;))))) 0, supp(ø) 10110 r), V(t E [0. 1 [0. S1}
n ( ((((0,)):C(O),1))))))
Pmn (0, (8,t)) t)}
For any d(0 t. E Us with d0,,t (0, we define a map
80 (Φ))) Ug - Ωg
(52)
21

---

Under review as a conference paper at ICLR 2022
where yp solves
@.((0,(,8) Vo (00,,,,)V 8E(((})) 8p (0,1)
0s
(53)
(00, 0, = Pmml
The strategy is to show this map contraction map, that there is fixed point in the set Ug which is then the solution to equation (
The proof of Theorem A.4 divided into three steps:
Step 1: We show Ts is well-defined map from Ωs to fls.
Step 2: We give bound of d2(Ts (21))( Ts()2)) in terms of d2 -2) One can then tune $ to ensure Ts is contraction map, meaning there is unique fixed point Φ² that ø" (T))")) and thus o² solves (33) for < S.
Step 3: We extend the local solution to global solution.
Step 1.
According to the definition of (53), for fixed Φ(0, t, s), let
sE(d(a)) ( V (0:t)) (0,
(0,(s; t)
sp
(54)
Then 0g ~ = Ts()). To show the existence of yp amounts to showing the wellposedness of (54).
According to Lemma C.2 (41 and
(422. the force Ve 6o(s) (.,t) has linear growth and locally
Lipschitz. Classical ODE theory then suggests there is unique solution for E [0, S]. which depends continuously on the initial value 0(0; t).
Denoting
(') / 1012 do(0, dl,
Cs. sup 05$58
sup Cmm
10 (0,
0gis1 Rd
(55)
S.
sup
10 ddo(0 1,
05151,05
Bkk
Rini = inf (00 c 1010mm] < <r}, Vt E [0.1]] r>0
e
Rs. inf {supp(o(0,t) t, s)) c [0]10n] < < v(t,s) [0. [0.
r50
have the following proposition.
Proposition D.1 Suppose that (g(s; f) solves 54 and Φ E Ωg. Then for any (1,, 812,
[0, 1] x [0, S), we have
(100(1):t))))) exp(SS((Lss)) (C + +1). (0.n(()1)tii
I exp(SC((ss.)) (Rai +1)
(56)
and
(10((s):ti) (82:t2))
(57)
s C S.0r Rini, S) (B (0000:))-0000:11)) + 111 - 1212 + 181 82
Proof To prove the first bound in (56), we multiply (54) on both sides by 0g and use boundedness of the forcing term 41 obtain
(s;
I C(Css.) +1).
ds
22

---

Under review as a conference paper at ICLR 2022
Using Grönwall's inquality together with E (1:t(0;t)))3) Lini obtain the first bound in 56)
sE(p) To prove the second bound in (56). the bound of 0b,; op
(0,1) according to Lemma C.2
0 (s) S C(CCs,)) + +1).
ds
Using Grönwall's inquality together with [1 & Rinis we obtain the second bound in (56).
To show
(57 we first write
E (10p()1:t) - (s())tt))(²)
0x (s2; $ 2B ( (0()))t2)) +2 (
(58)
(I)
then bound the two terms (l) and (II) separately. For (I, we use
(4). the second bound of (56). and
Lemma C.2 42
(Btot) (2)
ds
SC(Ls,p, Rmid) (s: (s;ttt)) C(CS.) (10 (81: 1)(ti ttll
Using the first bound in (56) and Grönwall's inequality, this implies that
E (10((11it)) (0(s1;t2))²) < SC (C Rini, 8) (E (0:()) 0s (0:(6)))))) |11 -621²)
(59)
For (II), obtain estimate by integrating 54) from 81 to $2 and using the boundedness of
5Ele) in (TT.. From the Grönwall inequality, we have V 6p
(0((1))2)) 0p(32; t2)l s C (Cs.o) (0(sstttt))) ds + Is1 22)
Using first bound in (6) and Hölder's inequality, this implies
E (s1 (2): C (Cs.s) Li S) |s
s2l
(60)
and thus combining with (59) and substituting in 588 we complete the proof.
An immediate corollary of Proposition D.1 is that the map Ts(-) is well defined.
Corollary D.1 For every fixed S > O, the map Ts is well defined. That is, for any Φ E Ωg. one can
find ip Ts()) E Qg the unique solution of 133 In particular, for any (t [0, 1] x [0, S). we have
J 10 dpp(0, 1,80 exp(SS (C + +1),
(61)
sup s)) C [0|10n] (xp(SQ1((s,g)) (Riui + 1)}
where Q1 : R4 RR is depends only Ls.s and is an increasing function its argument.
Proof For fixed s) E [0. 1] [0. ST. define c00, t, s) as the distribution of (s: Using classical stochastic theory (Ambrosio et al. 2008 Prop 8.1.8), (0, 0, t, s) is solution to (53). The estimate of the support is consequence (56). Finally, using (16) and (57). we obtain that
lim W2(v p(: su)) lim E (10)(s: (0(sutto))?)) 1/2 =0, (t.s)
(to.s))
which proves the continuity in and I, so that &p E C(1O, S):C(( 1}: P2)). By combining all the factors above, we conclude that yp E Ug.
23

---

Under review as a conference paper at ICLR 2022
Step 2.
We show now that Ts is contraction map for S sufficiently small.
Proposition D.2 For any Φ1, 62 E Ωg, we have
(T((()))) Ts(Ø2)) S SQ2(Ls, Rs, Sdla(tt
(62)
= where Q2 R2 + R_ is an increasing function and Ls = , Rs
max (Rs R..02h with Ls Rs, defined in (55).
Proof Denote by 0a (s; t) the solutions to
54 with = oi using the same initial data, that is,
0₂ (0:4) 022 (0;
As in the previous subsection, we translate the study of yo to the study of 0g.- Defining
(s(s) = 1p (s: t) oe (s; t)l,
we have according to the definition of Wasserstein distance that
d2 (s(s(()) S
sup (t.))e[0.]x((0.5]
(A]
Using 54
we obtain
d()r(s))
V (E()1(8) (0), V 5E(o2(s))
S 4(A((s))
(8 (80..t) t)
ds
öp
6p
V (E)))($))
(0)1 - V (
2
< 4()xs))) +8 8
(0 1)
(63)
óp
8p
(s))
(00,, Vo (F(()(())
+ Vo
(0 t)
6p
5p
The second term on the right hand side involves the continuity addressed in Lemma C.2 42
(B() (00r, 8p
(E()1 (0oss s CC(ss ((,
Vo
8p
where we use the second bound in (56 to substitute the constant in involves the continuity discussed in 43). particular, have
(42) Then the last term of 63
8
(E(01(s) (00z () V 5E(o2(s))
(0sz:t) 1))
V
5p
5p
<C(Rs, 10 (1 + Dd (01, 22) s C(Ss, Rs.
62),
where d2 is defined in Definition 4.1 and in the second inequality we use the second bound (56) to
substitute the constant in 433
By substituting in 63 we obtain
S (s))
(1)
ds
which implies
(E(A)())))
Rs, Rs, (E((A + de x2))
ds
where we use the first inequality in (56). From the Grönwall inequality, there exists Q2 R - R.. an increasing function such that
IE (s))))) <
completing the proof.
To apply the contraction mapping theorem, we need to verify two conditions order to show that there exists fixed point o" (T(("))
24

---

Under review as a conference paper at ICLR 2022
There is a closed subset in f2g such that Ts maps this set to itself.
Ts a contraction map this subset.
For the closed subset, we define
Bp { G2g supp(o(t, s)) [0110T]] 4(imi + 1)}, v(t, [0, x [0.
(64)
10 do(0, (, = 4 (miil +1) v((,8) E [0, [0.5]]
Ωs
We now claim that for small enough S. Ts is a contraction map in Bp
Proposition D.3 Suppose that S is small enough that (0o(QQ14((Ea))) + 1]]) ani +1 4( 4( 1,
exp(SQ1(4(2m)) + +))) (Rii + 1) < (Riai + 1). SQ2(4( + 1), 4(Rini + 1),
2
where Qı and Q2 defined in Corollary D.7 and Proposition D.2 respectively. Then we have the following.
If de E B Boo then Ts(Ø) E Bpo that is, for any (t, 8) E [0, 1] x [0, S}, we have supp(Ts(0)(L 8)) (0110m] s 4(Rini + 1)},
(65)
and
J [Ts S < 4 + +
(66)
Ts is contraction map in this subset, meaning that for any Φn, ,2 E Bpg have
(TTs dd((01.
(67)
Proof First, using Corollary D.1 61 and noticing Ls. 4 (Cimi + 1), prove 65
(66 Then,
using 622 with (55) and (66). we have
1
(Ts Ts(o)) < SQ2(4( + 1), 4 + 1), (2)))
which proves 67
Using the contraction mapping theorem, obtain directly that Ts( has fixed point in Bpd when S is small enough.
Corollary D.2 If S satisfies conditions in Proposition D.3 then there exists a unique o" (0, t, s) e Bo Ug such that o² (0, s) is a solution to (9) with initial condition Pini(0, t).
This is direct consequence of the application of contraction mapping theorem. Finally, we prove that the cost function decreases along the flow.
Lemma D.I Suppose that (0, 1, C((O, 1]; P2)) solves with initial condition Pfni(0, t). Then for for 0 < <. S, we have
dE(o" (0,t,s) dE(o"(O,ts)) ds
- L. JR
(E(O*(s) (0,t)
V
10= dt 0.
(68)
8p
Proof Denote by (s; t) the associated path, meaning that 0 (s; solves 54) with の o², then 8" ~ o*, meaning the distribution of 8* is d² According to 99. we obtain using change of variable that
(E
8E(d
V
(0,t) do s0,t,)) 50.
(69)
RR
öp
which proves the result. We note that the derivation in in (Ding et al. 2021 Appendix I).
69 formal. A rigorous proof can be found
25

---

Under review as a conference paper at ICLR 2022
Step 3. In this final step of the proof, extend the local solution from Corollary D.2 to global solution. Lemma D.1 shows that the formula dE de so we can then use this formula to improve the bound for the support of the solution (1). This improvement will be shown in the following corollary. This improved estimate helps in extending the local solution to the global solution.
Corollary D.3 For fixed S satisfying the condition in Proposition D.3 denote by o" (0, t, C((O, S):C((0 P))) the solution to 9 [0, 1] x [0, S), we have
with initial condition Pini 0, b). Then for any (t,
1 10 (0,00,,0,,000000 S C(S, Rini Lini) BA
(70)
supp( (1, () c [0 S C(S, Rini, finl)
where the quantity C depends only on S, Rimi and P
Proof According to 61 it suffices to prove
1/' /.x0 do s) 2 C(S, Rini Lini
L
sup
05
Denote by (s; the particle representation of or, meaning that 0 (s; solves [4 with á = o²
Since l (s; t) 60 (s;
( J do st sup / E (10))s))) de. 05a58
C
sup
0S$5$
Using 54
we obtain that
d (2;
Vo (s))
S 10 (s:
(0" (s:
ds
6p
which gives
d a E (10 dt ds
1/2
(f E
(E)
2 (0(sstt,
E (10()))) dt)
V
dt
öp
-( E ((00 (100(s;
1/2
dE dE(
where we use the Hölder inequality and (68) from Lemma D. the last equality. Since
E (10" t)1²) dt < Caup
dE (0, (0),,)))
ds S ((pmi((0,)) (0,,5), C ((C))))
ds
we obtain
Cs.0 sup 0S
E dt < C(S, C
by Grönwall's inequality. This proves (70).
By contrast with (61) this estimate removes the dependence of the bound on Ls This improvement important because relaxes the fixed-point argument from the dependence on the initial guess é.
We now ready prove Theorem A.4
Proof [Proof of Theorem A.4 From Corollary
D.2 let Sj be a constant satisfying the conditions in
Proposition D.3 Then there local solution & E C( 1); P2)) to (9)
26

---

Under review as a conference paper at ICLR 2022
We now denote by S the largest time within which the solution exists, where we denote this solution by 6² E C (I0, C([o 1]; PR)). We aim to show that S = co. According to Corollary D.3 (70).
for any 8 < S" and E [0. 1], we have
0 do (0,4,s S C(S" Rinis Lim
R
supp(ø" (1, 8)) C [0110 < C(S", Rini, Lini )
Define R' = h L.s- 0- according to Since R f oo, let us choose As- small enough to satisfy
Q (C + +1 +1(*+))),
(R +1) s +1)
and
s.. (2( + +1) 4(R* + 1)) 2'
S" finite, then, using Proposition D.3 and Corollary D.2
we can further extend o² to be supported
on C' (10 As- C(, 1]: p²)), giving a contradiction. If follows that S = co, desired.
Finally, 21 is direct result of Lemma D.I
D.2 PROOF OF THEOREM
This section is dedicated to Theorem A.3 we show the well posedness of the gradient flow in the finite-layer case. We rewrite the gradient of (2) as follows
0E(0L.M) 1
Eara (Zaf(Zer, (l;x), 0m Iper, (4:x))
(71)
10
ML
where Pe (1;x) solves:
T Pa Per.ar (l:x) = Pe (1 1;z)
1+ ML
1
Σ Ozf (Z (1 1;2). (72)
M (L 1;x) (g(Zer, (L;z)) y(x)) Vg(Zer,
for 0 S --2. We unify the space in similar fashion to Definition 4.1
2.
Definition D.1 Or,M (0 (01mm))=0mm
1-11M
LE. if and only if
sup 10tml| 00. 1,
The metric in Lf.M is defined as
1/2
di,t, (OL,M, 0r.M) max
01,m mm,m! 12)
M
Definition D.2 For 2 0, we have O2, {f), (s)
(s) 1-1M E E((O, o0); LF,M) and only if
1. For fixed s [0, oo), (s. L..M-
2. For any 80 E [0, oo),
lim d1, (LL,M(s), 0,
8-150
where di, L, is defined in Definition D.
The metric in C((O, oo); LF,M) is defined by
d2. (OL.M @e.M))
s. (L.,.
27

---

Under review as a conference paper at ICLR 2022
Theorem A.3 is say that the solution B is unique in C((O, oo); LT,M) if Q L. (0) Lf.
Before proving the theorem, prepare some a-priori estimates of Zer., and P
Lemma D.2 Suppose that Assumption 4.1 holds and that x is in the support of p. Let
{ {0rm 1-1.M
Or.M
{0tm
and 1r.M-
mm0m=
and denote
L-1 M
1
Le
01,m 1
LM
1=0
m=1
1
Σ
1t.m
LM
RL,M = sup
100mm 1l.m
1,m
Then for 0 SISL-1 1, we have the following properties:
Boundedness in Zer.
Ze (1 +1+ 11x) S O(Lor.M)
(73)
Lipschitz in Z
ZZe (1 + 1;x) Zor.. 1;xx s SC (Coouu Lor,,s) dr.L, (OL,Ns, Gz,r))
(74)
Boundedness in Pe
(0: < C(Lor,))
(75)
Lipschite in pe
pe (1:x) Po (1:x) S C(L,, dI PLL.M.UL..
(76)
Proof From 1 and 12 we obtain
1
Zo (1 + 1;z) + 1) sc
1+
Σ (10 +1) ( Me (1:x) + +1)
LM
1
exp
C1 exp
(10mmll +1) 1) Zo (1x2)/ 1),
LM
which proves 73
by iteration on l.
From
13 and
73
we obtain
M4
C(Cer.r)
M
Σ
10.f (Zer, (1+ 1;x)),..m) 11.
Σ m0) + 1),
ML
ML
m=
m=1
which by 72
implies
1 C(
M
Σ (10 + +1) pe (1 1;x)].
pe m (1:x)) S
ML
m=1
From this bound, together with |pe,. (x, 1)1 < Ze (I:x)) S C(Lor. ). we prove 75 by iteration on l.
28

---

Under review as a conference paper at ICLR 2022
To prove (4). subtract the two updating formulas and split the estimate to obtain
Zer.ne (1 1; - ZZn (1 1;x]
C(COr.M)
M
Σ
(10rm ++ +1)
Ze (1:xx - Zon, (1:x)
ML
m=1
C( Sor
M
(10t,ml + 10 + +) di (OL,M, BL.M)
M
m==
where we use together with the bounds
13 and
(73.. Noting that ZZe,, (0; ( (0;z)] =
O, we prove 744 by iteration on I.
Finally, for 76 we subtract two equations in the form of
72 and use 72
together with
Lipschitz continuity to obtain
pe (1;x) P (1:z)
M
$
(per, (1 1; - PêL. (1+ 1;1)) (T+
Σ 0.5 (Zer, ar (1 + 1;20), Bl..m) m==
1121),0+1.,m)
ML
M Σ ML
T (1 1;x)
0z (70..M 1; 01 11 - 0.S (Z (1 1; t+t1m))
+
PêL.
m==1
C(RL,)
1 1+
Σ (10).m 12 + +1)
pe (1 1;x) Po (1 + 1;x)
ML
C(RLMM)
dI, (OL.M, GL,M)
L
(77)
The initial data is also controlled, follows:
|per, (L - 1; Pr (L 1;x)] S C| Ze (I;z) - ZB (I;x0)
S (((RLMM)d (OL.M 22.1z)
By combining this with (77 we prove (76) by iteration on l.
Lemma D.2 resembles Theorem A.2 and Lemma C.1 These estimates allow us to prove Theorem A.3 Since the proof strategy is exactly the same, omit details. Essentially we define map
(s) Ts (e)(s)) c()) +, +x)
T,,
where (ss) solves:
( - MLV for 2 0,
ds
where & defines the forcing term. The estimates above provide all the ingredients to show the map is well-defined, and for small enough S, the map is also contracting, leading to the uniqueness of the solution to 33). Similar to Lemma D.I one can also show dE de - L the estimates and removing the constants' dependence on the initial guess. This extends the local solution to the global one, as done in Step 3 for the continuous case.
improving
E PROOF OF THEOREM 5.1
Theorem 5.1 links the cost defined by OL.M (s) with that defined by p(0, f, s) for all s. The continuous and mean-held limits obtained, with both L and M sent to infinity. We decompose this result into two parts, discussing mean-field and continuous limits separately.
We start with the full definition of ""imit-admissible" for distribution p.
29

---

Under review as a conference paper at ICLR 2022
Definition E.1 For admissible p(8,t), say p(0,t) .) is limit- if the average of large number of particle presentations is bounded and Lipschitz with high probabiliry. That is, for an admissible p(8, t), there two constants C3 and Ca. both greater than supre(o.1] Jg+ dp t) such that, for any M stochastic process presentation {0m(t) JM that are i..d. from p(0, t). the following properties are satisfied for any dj > and M Ca:
1. Second moment boundedness in time:
IP
sup 6e|0.1]
Σ (m)(1))
sca
21-1
(78)
M
2. For all L >> 0, we have
L-1
M
1 I
C2 I
P
(l))) - Om
dt S
2 1 n
(79)
M
1=0 m==
We now state the two theorems that play complementary parts in Theorem 5.1 addresses the limit in M under the assumption that L oc. This is the mean-field part of the analysis.
The first theorem
Theorem E.1 Let Assumptions 4.1 and 4.2 hold with some 0 < k2 S k. Assume that (., is limit-admissible and suppo (Pimi (,0) c To & R} with some R > 0 for all t E [0, 1]. Suppose that 10m (0; t) {) are i.i.d drawn from (0, (0,6 t). Suppose in addition that
p(o,t, t, 8) solves 9
with the initial condition Pini (0, t), and
0m,(s;) 1) solves 16 with the initial condition Om (0; t).
Then for any 6, TJ, O, there exists a constant (ptin (6, t), > 0 depending on Pini t), S such that when M ((.m)(0.)).S) we have
city
(IE(O(s:)) )) E(p(.,,)) S 17 vs <.
Proof See Appendix
The conclusion this result suggests that for a - r) confidence of an accuracy, grows polynomially with respect to 1/c and 1/n.
The second result considers the convergence of the parameter configuration for the discrete ResNet I to that for the continuous ResNet as L cc. This is the continuous- limit part of the analysis.
Theorem E.2 Let Assumptions 4.1 and 4.2 hold with some 0 < ka S k. Assume that Pini (n,()) is T8n I R} with some R > 0 for all t E [0, 1). Suppose
Smnit-ddmissbble and suppo (Pini ( (,D) that {0m (0; {
m= are i.i.d drawn from (mn(0, t). Suppose in addition that
(m; solves (6 with initial condition 6m (0;
0.,m(s) (s) solves 3 with initial condition 9... (0: +
Then for any e, > O, there exists constant C(pri (0, t); > 0 depending on Pini (0,tt) such that when M (9.))) and 2 Clem Clem.(9.t).S) (9.t).S) we have for all S that
P ((EE(O(s:))) E(OL.((S)) $ <) 21
Proof See Appendix G
This theorem shows that when the width is large enough, then with high probability, in the whole training process with < S, the difference between the loss functions defined by the discrete ResNet and its continuous counterpart decreases L oo.
30

---

Under review as a conference paper at ICLR 2022
F CONVERGENCE TO THE MEAN-FIELD PDE
This section is dedicated to mean-field analysis and the proof of Theorem E.1 The intuition of this theorem is largely aligned with many other mean-field results, demonstrated in (Ding et 2021).
As argued in Section 5 to show "equivalence" between (6 and ( we can test them on the same smooth function (()). Testing ( on amounts multiplying h on both sides of the equation by h. From integration by parts we have
d
/ h hdp(0) = R
VohVo 8E(p(s)) dp, óp
ds
that is,
d
[h) = E VahVa SF(p(s))
ds
8p
To test 66 on h, we let
1
Em=1 50.. and obtain
M
d B(h)
d
5E
M Σ Vol(Om)
0m
voh(Om)) 50m
ds
We see that 9 and O equivalent when tested by h, if and only if the right hand sides of the two equations above are the same, that is,
8E
8E(P)
M 50,
V
(0m, 1).
(80)
8p
This claim can be established from the definitions of the Fréchet derivatives for 6B(p) 6p
and 6E
89
(Ding et al. 2021 Lemma 33).
To give quantitative estimate on how quickly 6 converges to of (9), we utilize the particle method, a classical strategy for the mean-field limit. We sketch the proof here and will it more rigorous in the following subsections. We make use of two particle systems. In one system, the particles evolve themselves, while in the second, the particles are moved forward according the underlying field constructed by the limit. In our situation, the former particle system consists of the M stochastic processes (s; t)} that descend according to E(O(s::)) The latter particle system will be termed (s;; 1) = {0m (s; })) it descends according to F'p(s, 8)), the limiting cost function. Essentially, prove
E(O(s;))) E (6(s:t)) x E(p(.;,8))
The latter approximation arises roughly from the law of large numbers, but the former needs to be proved rigorously by tracing the two different evolving ODEs.
To be more specific, let p(0, s) be the solution to 9 with admissible initial conditions Pini (0, t). and let 6. (s: be the solution to (6) with initial conditions (fm 0) M that are i.idd drawn from Pini(0, t). Using the definition of E. in (). have
[E(p(. 8)) E(O(s:)))
1
1 (1()) - )()))
<Enwy
- (1])))
2
(81)
E (1((Z(() - (( (19( ((0()(()(1:=))) ((2o) (22 + lb(z)))
((C.) 9(2,(s) - g(Zo(s) (1:+))
< <C(L.) (1;z) - Z (1:
where L. max { Jgx. 1012 dp(O, t.s) M So EM (s) at). In this derivation, we used the Lipschitz property of g. the boundedness of y (required in Assumptions 4.1 4.22.. and the boundedness of Z, and Ze(a). Boundedness of Zp was shown in Theorem A.2 while the bound for Zo will be addressed in Lemma F3 The constant C' depends on the support Pinis as well as on
31

---

Under review as a conference paper at ICLR 2022
and the Lipschitz constant of g. It follows from 88 that to control E(p(.,,,)) s)) E(O(s; -),
need to control
p(s) (1: ((())():))
(82)
To do so, we employ the particle method and invent new particle system.
According to (8), we can reformulate the original particle system as
d0m(s;;) Vo
ò (((dde()))
(mm(s)t))
v(t,s) E [0,1] x [0, 00),
(83)
ds
óp
where we denote
M
dis (0, 0,t,8)
1
Σ 8p (e:t) (e:t) (0). m=1
(84)
M
We invent new system that follows the underlying flow governed by the limit. Define O(s)
{0m (s; ) where 6m solves
d0
(E(p())
(om(sitt)
V0
V(t, [0.1] x (0,00),
(85)
ds
6p
with initial condition
0m (0: () = Om (0:1)-
(86)
As consequence, we have (s; p(),t,s) for all s). The corresponding ensemble distribu- tion is
M
p 0,t,s) = M
Σ m=1
@mm (s:t) (s:)) (0).
(87)
Now we have available particle system ((s) = (s)). a newly invented particle system θ(s)
(8m(s) and the mean-field flow Accordingly, there are three versions of Z: 7p(=) that solves using p(s) and Ze(s) and Zō(s) that solve using (ss) and θ(s). respectively. We use the following relabelling for convenience:
Z, Zpls) Zella Zo(s): Z Z
(88)
Similarly, there are three sets of p. Pp(s), (s). and (a) that solve using p(s), (s) and (s), respectively. We relabel similarly to (88 and write
= Po(s):
Ps (p) dil = (s)
(89)
Since (ss) serves as bridge, we translate the control of (2 to:
[t;(; - - (t; x) s (z(t:)) 2 (t; x)) + Z (t; (t:((tt (t:
(90)
Bounding [t((t))= (t;)) can be done using the law of large numbers.
Bounding
7dll (1; 2 (1; translates to controlling Σm Su' (s; 0. (s; 1) dt, for which we will evaluate the difference between equations (83 and (85) Since these two equations have the same initial data, the difference between 0m, and 0n, can then be controlled when the right-hand side forcing terms are close.
This approach divides the proof naturally into two components. In Section we give the rigorous bound of (90). while in Section F22 On (s; 0) dt in 8, thus finalizing the proof for Theorem E.
trace the evolution of the differrnnce Σm. 10m (s;
32

---

Under review as a conference paper at ICLR 2022
F.1 STABILITY IN THE MEAN-FIELD REGIME
Here we discuss control of the two terms on the right-hand side of (90). Recall that ( and (8)
satisfy (83) and (85), and the two corresponding ensemble distribution defined in 884 and (87)
respectively. For any 5 > O, define
L
sup USIS100
( 10 dp t,,,
Σ M i=1
1 M
mm (s; t) M
1
Σ mm (s: i=1
(91)
Rs inf (supp(p(0,t, U (0 (8: r20
[0110n] <r), v(t, E [0, (0, S]]
have the following lemma:
Lemma F.1 For every fixed 8, let Za and Zaia be as defined in 88
Then there exists constant
C(C such that for all E [0, 1], E [0, oo), we have
(t;txx))
Zadi (1:2r)
1/2
f/ (m((::))) [m)(s:) dr
C(
Σ
M m=
(92)
(L,
T f(Z.(T:2).0)
2
1/2
C(CSuP)
d(p(0,T,s) (0l
dr
Proof Since the statement holds for fixed s, we eliminate all s dependence all calculations the proof, for conciseness.
Recalling definitions in (8), we denote
Í(t:x) = (t:tz) Z (t;x), A(t;::) 7eie at; Zulla (t:
follows from the triangle inequality that
(x(t; - - (4:z)) < (1;; x)) + [A(+=))]
We now bound these two terms. We first apply the same argument as the proof of Theorem A.2
(see (26 in Appendix B to obtain
(1;z))
+ 1 I (Za(1;;)), 0) d(p(0,1,8) (0,
I C(CTTP)
dt
Using the Grönwall inequality and the fact that [ x:) = we have
1/2
A(t:z)) & C(CSuP) Ve (Z. d(p(0, t, - 1t dr
(93)
for E [0, 1]. Similarly, to bound A (t; we have
d d|A(t:)) dt
(((=))) 1 / (Z."((1)))) (0, - -0,
2
From Assumption 4.1 and the fact that Zais is bounded in Theorem A.2
we have
(t:)), 0) (
1 f
(0, (, (0,
s ((
1
M
(m(s:) (s:
2
M
similar to the proof of Theorem A.2 (see 28 in Appendix B).
33

---

Under review as a conference paper at ICLR 2022
Using 20 0, apply Grönwall's inequality to obtain
1/2
[A(+)) C(C
1 M
Σ
(mn(s; - -0 (8i7) dr
(94)
The result is obtained from adding (93 and 94
The difference in and p:ls:
Lemma F.2 For every fixed E [0. 20), let ps and pai be defined in 89) There exists a constant C(R.) with Ra, Lgup defined in (91) such that for all [0, 1]:
(t.(t;))= - Ps dis (t;z))
M
/'' (mn(5)))0mm1::7))) (8:7)] dr
1/2
$ CR,,, LzuP)
Σ
M
m=1
+ CR,,, () (fo Uee ((Z.(T;)). d(p(0, (p
2
1/2
(95)
dr
(f. V 0.S (Z.((:))). a(p(0, - (0,
2 dr
1/2
C(Ris, CBup)
Proof As in the previous proof, we eliminate dependence on in some notation, for conciseness. Denoting (tttr) ps (t; ddin (t; x), we recall IT] to have:
|| (1;x))]
dt
< [ 0,5( (21 Ap(1;)))
/ 0 (p) 0zf(Zzlie (t;()) (0, t,
+ 2/1 x) 1
< C(C (pp;)))))
/ 0. 0.1(2 / (.((Z(i((;;))) do (0, s)
+2
<C(C
(t:
I 0.f () dp((,,t, Sfc 0zf(Z,(t:2), 0) dp (0,
+6
/ 0.f (.f(Z.(ttz). dp (0,,,s)
+6
JR d. (z:(z.(tzz), 0) do dis (0,t,s)
R*
@
+6
(0,
azf(Z (t:z). dp dis (0, t,t,s)
RR
(D)
where we use 13 from Assumption 4.1 together with bound (I) on the righ side, recall the definition
18 and 31
in the second inequality. To
84
and 87 and Assumption 4.1
34

---

Under review as a conference paper at ICLR 2022
along with the boundedness of Z (as shown in (8) to obtain
M Σ
2
(1) -
6m (s: 02 (2) (t)
M
m==1
M
(R( P)
(m; 1) - 0m
(96)
M
m=1
M
<R L
Om (s:t) (s:t)
m=1
where we use Hölder's inequality in the last inequality. For (II), we have
M Σ
2
(II <
@.f(Z,(t: (m, 02 fz (t; (t; (m(:
M m=1
(97)
I C(R,: D
Z. (t: (t: - Z x)
By substituting
(96) and 97 into the bound above, we obtain
d [1;p(t;))
dt
-C (t;:)))))
(m( (msts + (1aa(t;)) (t:
| Rt
+ 6
0zf ())
0z (z:(Z.(tzz).0) dptis dpdis s)
(98)
The "initial condition" for pa and pftn yields
(p:(1)) S (Z(:7)) (1;z)]
The result is obtained when substitute 22 into (98 and use the Grönwall's inequality.
F.2 PROOF OF THEOREM
With the quantitative deseription of 90
presented in Lemma F.
complete the proof for Theo-
rem E. in this section. Recall from 55 that
Rini inf (supp(pma(t)) [0 < <r). Vt E [0,1]]
sup
1012
dpini(0,
r20
05:51
3*
and note that Rimi 2 (0; for all m, f. Define
dus
Σ (0:1))
ini
sup 05151
M
We note that when M is large, adis is close to C (which has no randomness) with high probability, We have the following lemma:
Lemma E. For given S there exists a constant C (S, Rui) such that for any E [0. 1]. s [0, S]. we have
M (x: < C(S, Riui. Edls,sup).
Σ
M
(99)
m=1
(0.m (s; t)) M
10110 S C(S, Rini, Li cdlsssup))
35

---

Under review as a conference paper at ICLR 2022
Furthermore, for any with [x: < Ry (as in Assumption 4.2 t E [0, 11. the ODE solution is bounded as follows:
item 4), and any [0, S] and
- (t: - Rimi, shsss
(100)
while the following bound holds on ps
|0 (1;x)] < C(S, Riais Cdissup)
(101)
The bound (99) is result of Corollary D.3 The bounds (1O0) and [01] are obtained using the same arguments as in Theorem A.2 and Lemma
The next lemma bounds the support and second moment of 7 (0,4,8).
Lemma F.4 Under conditions of Theorem E.1 for any S there a constant C(S, Rini. Lini sup) depending only on Rini, ma sup S such that for any t E [0, 1], s E [0, we have
M
(s: - C(S, Raui a
M
(102)
f6m (s: M)
C (8110]] & C(S, Rini Lin (dinsup))
Proof Similar to the proof of Proposition D.I we multiply 85
by 0m (8; t) on both sides and utilize where C in (41 is replaced by C(S, Rinis Edis.uu) according to
the bound (11 from Lemma
C.2
Corollary D.3
We thus obtain
@0 (s; C(S, Rini
Lin
Edis.sup)
(0:) (0:t))+1)
and
mm. (s; f) S C(S, Rimis L
Edls.sup) (0:,(1(0::)1+1]
which implies (102 by 86).
Denote L8 max Lii C1 with high probability. We now ready for the main proof of this section.
Cdis.sup Csup
}. According to Definition E.
L should be bound by
Proof [Proof of Theorem depending only on Rinis
E. rap
First, using Lemmas F.3
F.4
there is a constant C(S, Rinis 2)
Lo
S such that for any t [0,
1],
E []
max |
0 dp(e,,))
1
Σ
(s;
2
1
Σ 10 (s:
Rini D)
M
M
supp(p(0,tt,)) U (0 M) U { (8m()))))
[0110 C(S, Rinis ((up))
m=1
(103)
Recalling (1. we have C. < C(S, Rinis L and E(pls,, ;)) - ((O(s;))) S C(S, Rinis (H [2(1;2)) - 2::ii (104)
Furthermore, according to Lemma (92.. from (003
(2,1;;)
1/2
< C(S, Rini, Coupy
1
Σ
[ (m((::)) - (s] dr
M
(105)
m=1
(f.
2
1/2
+ +CSS Rumi Coup)
f (Z.((;=)),) d(p(0, - s)))
dr
The second term in this bound can be treated using the law large numbers. We focus on controlling the first term.
36

---

Under review as a conference paper at ICLR 2022
Step 1: Estimating 1 Σ M
Lo' O0 ts: dt. Defining
(s) = (m:(m)) -m)):))))0m((:)) (s)
we note that A (0) 0. By taking the difference (83) and 885 and multiplying both sides by
A a.m (5), obtain
d A (s)
ds
A (s), Esw fm) 02, (t; 6m )p. (t:
2
(s), Euwny (Zo (t; -af.
(106)
- 2 Ar. (8), Euryy (Z(())() 0m )r (1;z) - 0o. SZ (1;x;,, 0m Dr (t;
For (I), we have from the bounds of Z in 18 respectively, that
(0) < C(S, Runi Co
@mml + 1) Evnp (Ip((t:)) - Ps dis (t:"(t))))
which can be controlled using Lemma F2 955
For (II), we have
(ID) SC(S, Rini, Bee ( (0of(Z,(t;z), Om)) - -0. (Z (t; 0m)
<C(S, Rii. ( [Bewa [. Z (t: (t:z))) + 18m 8ml
where the first term can be controlled using Lemma 922 In both estimates, we used the property of f in Assumption 4.1 and bounds on Z, m1][ and m,, By substituting these estimates into (166, we obtain
d (s)
C(S, Rini, 10 A1, (s)12
ds
+ C(S, Rini: LO ) A.. (m) ()z,(t:x) - Z (t: x)1)
+ C(S, Rini: L. D (+)) (mml +1) Exna (t))();
(t) - -p (t: x)
which implies that
M
A mm
(2)
M
ds
m=1
1 M
<
CCS, Rinis Lour)
Asm()))
M
m=l
1
+ +C(S, Rinis c)up)
[+)l Eneay ()Zo(t;z)) - 2 (t: 1)
M
1
D (s)l (10ml +1) Bawn ((pa(t:z) pdli (t: (t;z))
CCSS, Rinis 1)
M
< CSS, Rinis Loup)
Aa,(()))
+ C(S, Rinis La Benn (leal1:)) - 1 (1:2)) 12)
M
m=1
+ +C(S, Rinis Exen
(IZx(t:z) - 2. (t; (t:))))))
(107)
37

---

Under review as a conference paper at ICLR 2022
where in the last inequality we use Hölder's inequality
A mm (s)l (18m) +11) Brnn (Ipa(t:x)) (t:
M
M
1/2
1/2
10ml + +11))
A A
Σ
Bawn (ppo(t:x) - (t: x)
M
M
1/2
<C(S, Rini, )
Anm(())
Errn (1pa(1:)) - -p (1:
M
<C(S, Rini, ))
A mm (s)
- Eown (Ipaltizt) D- (t; x)) x))²
M
m=1
Noting the estimate in Lemmal F.1
F2
we obtain
d 1/4 Σ M
m=1 So A
(s)
ds
1 M
L' [s dt
C(S, Riuis ))
Σ
M mmm
(fo [(Z.(T:))) a a(-pole(0, T, 8)) dr
C(S, Rini, Lo P)
+ C(S, Rini E (1'
(,, Lee Oz 0.f(Zo(T:)))0) a(- d(- T, 8))
2
dr
which implies, using Grönwall's incquality, that
M
(s)
M m=1
< Rin, L0 1 (fo S.' (/
5 ((Za(t:)) U(p(0,Tss) - -ti (0, s)) dr ds
+ C(S, Rimi, Baow (1² /1 Ozl (zs(2(())))) 0) d(p(0, T, - - (0, dr ds
(108)
where we use ,m (0) = 0.
38

---

Under review as a conference paper at ICLR 2022
Step 2: Completing the proof. By substituting (08 into 92
noticing R2 I C(Rmns S), we obtain
Z, (t;+))
Zaii (t)
1/2
Bxnn (fs [.' 1/1e (Za(T::)), 0) d(p(0,7,8) (0, dr ds
<C(S, Rini, P)
1/2
Beup (1o 1. O.1 (T; a(p(0, T, - 0 (,),,,))
+ CSS, Rini, D)
dr ds
(1)
1/2
! Vee Ozf (Z. 0) d(p(0, r, s) (0,
2
+ C(S, Rini, D)
-
(109)
All three terms in (109 can be controlled in expectation. Here we take the expectation with respect the randomness initial drawing of {0. (0; ()) For (I), we have
E(I- E Eznp (1 J (Z,(T:2). 0) d(p(0, T.S) - 7dl T. s))
dr ds
ds)
Exny (o (1 d(p(0, T, 20l (.5)
dr ds
Bown (fo (1(111((( (dd C(S, Rinis Limp) M
C(S, Rinis Laup)
M
where we use 0m (s;t) ~ p(0,t t, s) the first inequality. second inequality, kı k, we use first
inequality of Assumption 4.1 15 with [()]] S C(S, Rini, L and Zzl S C(S, Rinis (amP). If
kj ks we use (12) and | ppl S C(S, Rinis 2 and [Z, S C(S, Rinis Li the second
inequality. By similar reasoning, we obtain
C(S, Rinis Ch
BAIII) < C(S,Rai csup
B(I) <
M
M
From Markov's inequality, these bounds imply that when M C
we have
et1)
((() < <20 {an) 2 < <)]) 11/2.
(110)
Finally, using Definition E.1 (78). when M > 2Ca
P < C4) 2- //
(111)
By substituting 110 and I1T) into 109), we see that there exists constant C(Rini Cy. C4 such that for any 6, when M > C(Rim Ca.Cm..
S)
obtain that
e"
IP ([Z,(1:=)) - (1:2) < <) 11 n7. By using this result and (111) in conjunction with (044, we complete the proof.
39

---

Under review as a conference paper at ICLR 2022
G CONVERGENCE TO THE CONTINUOUS LIMIT
This section is dedicated to the continuous limit and, in particular, the proof of Theorem F.2
G.1 STABILITY WITH DISCRETIZATION
Before proving Theorem and similarly to Appendix F.1 we first consider the stability of Z and under discretization, Defining the path of parameters ( {tm( (t)] m=1 and the set of parameters
1L,M- (0 1 we have the following lemma.
Lemma G.1 Suppose that Assumption
4.7 holds and that is in the support of p. Denoting
M
M
Cuup sup 051511
0
2
10 10m(t) (t)1
M
M
1=1
(112)
MLL R= inf { (0,m m= U f (t) C {0110m) rr, WE [0. 10. 130
there exists constant Cou)) depending only on R, Loup such that for any 0 s 1 S L - we
have
sup 5154 ((z)((;a) Zer.ua (1;x) [0((;=)) Ze a1 111)
1-1 M
1/2
C(R, (oup)
(113)
$ CR,, (up)
10mm - 0ma())/2 )
+
M
L
m=1
and
sup 1 $1
pe(1;a) - pe (1:2:)
1 1-11
1/2 18 (m dr dr)
C(R, (sup)
(114)
s C
Proof Define
Z(t:x) Ze(t;x), p(t;x)=pest pe((:a))
and
Z(::)) Ze M (1:2) + p(t:)) Σ pe (lix) (lix) 4
(115)
with
21;1)) Zer (L;x), 7(0; P02. (0;x)
(116)
Using
( 72. Assumption 4.1 and Lemma D.2 (73) and (75). obtain for all = 0, 0....... L --1
that
Zer.ar (1 + 1;x) Zer (1;z) < C(Cuup)
(117)
pe (1+ 1;x) - P0 (1:z)) < C([sup)
Now define A, by
A Z(1;x)) - -((;))
Fort t [t, 411, we have from (4) that
(()
M
C(uuup)
M
(T) + +1))) dr
(118)
M
C(Esup)
L
40

---

Under review as a conference paper at ICLR 2022
where use that
12
18
and
112 in the last two inequalities. From ① and 4 obtain further
M
(()()) - (2)1)))))))))))))))))))) dr f(Z(() (Z(T; 01m)) dr
A
M
M
A
M
(T (Z(T:20) dr
M
M
4
+ C(C
(10m(()] + 10mmll + m)) 0m,ll dd
M
m=1
M ML
+ C(coup))Ael
Σ (10,mml 12 + +1)
m=1
1+ C(CCUP)
| + C(C
1
(10m((T) + 10t,ml + 1)) 0r,ml
M
C(C
L2
where & E 1, and we used the mean-value theorem with 13
18
73
in (I) and (122
118
(II). By applying this bound iteratively, we obtain
MΣΣ
1 M
C(C
A < C((Sup) (CCp)
(10m(T)] + 10
1)(0m((T) 0m, dr +
L
where
Agl By combining this bound with (118) and using Hölder's inequality with (112). we
obtain that
1 L-1 M
1/2
C([su))
A < C( ((
10 (m) dr
(119)
M
I.
1=0
mmm
By combining 1199 with (177) prove 113
To prove [114 we define
Ap(4;) p(1;x) ((t;z).
Similarly to 34 obtain
[1;a1= < C(C
7(1;x) 2(1;z))
1
1/2
+ C(CuP)
(120)
C(Cuup)
10 (m) dr
M
Fort t E (t, L+1] and using
we obtain that
M
/+1
P (t:z)
A
12
L
[+1
C(
M
(10m(T)²²+))
(121)
120
L
M
m=1
[++ L
C(Lsup)
4
1200
L
41

---

Under review as a conference paper at ICLR 2022
where we use 13
18 and
31
in the second inequality and the definition of L from 112 in
the last line.
From we obtain that
C+ p
{1=
1 x TT_
1+1 (
p²
x
+
(r::))
L
M
m=
while 72 implies that
1++1
M
11
1+1
T
7:00 = pT
T
0.f (zf(Z (l:x),, (m,) dr,
L
where p defined in (15)).
116
By bounding differences of these two expressions, we have
P
x M
< A
;
L
(1+1;4) ( (7+,11))
(T:x) 02 (m (T ot
(T) dr
M 172
①
M
++1
[+1
L ; 0. - (1 (+1 (1 i
T
0. (1:))) dr
M
m=
(I)
M
[++ 1
(1 (1++ ; 0.5(Ze, ,, (1; 0mm) dr
0.f (. N (1:z), (m)(T) - -p"
L ;
L
m=
(III)
(122)
We bound (I), (II), and (III) in turn.
Using and [122
we obtain that
C(C
(I)
Ap
(f:z))
(II): Using Assumption 4.1
14
together with 18
73
75
and
112
we obtain that
(II) C(R,
Z(Ti2)
Zer. m (l:z)) dr
M
m=1
1/2
C(R, (oup)
1--1 M
MΣΣJ
10 - 0m dr
+ +
L.
M
where make of (113) in the last inequality.
(III): Using Assumption 4.1 item 3 together with 73
75 and 112
obtain that
(II) C(R, Coup) Σ
10m(7) Ormll dr
1/2
((RR,Sun)
10m(T) 2r
2
dr
M
42

---

Under review as a conference paper at ICLR 2022
By substituting these three bounds and [21
into 122
obtain
C(R,
1
++1
A
7i
< 1+
A
L
1/2
(aup)
L-1
Σ
Σ
10
Pm (n((T)
+
L
1/2
C(R, Laup)
(m(()
0r,ml
dr +
M
I
By applying this bound iteratively, and using
120) and 121
obtain
L-1
+
1/2
1
(t(t=)) S C(R, (onD)
10.m 0m (m(
+
(123)
where we also use Hölder's inequality write
1/2
CRR,
(T. - 0m,ml
dr
M
L-1
1/2
[((R,
10,m (m) dr
M
+
We obtain (114) by combining 123 with 121
G.2 PROOF OF THEOREM
We denote by (s; t) the solution (0, t). Further, Oz (s) is a solution
with initial {9m (O;tt) that are i.i.d drawn from
3 with initial value Or,m (0) Om (0; f.) for <
- 1 and < S M. Define
M Σ
dis Lin
0m (0; 1)12
sup 0.151
M
and recall Rini defined in (55). According to Definition E.1 diss.sup bounded with high probability. Then, have the following lemma:
Lemma G.2 For fixed S > and any E S), there exists constant C(S, Rini L depending only on S, Rinis ed
adis,aup such that
Σ M
1
(s)
IC(S Rini, Lini Ldisssup)
(124)
(0 (5)
M.L Ma)
{0110 s C(S, Rini, edis.suy)
Lin
Furthermore, the ODE solution and (s) are bounded follows, for any in the support of and 0, 1....,.. L
S SC (s, disssup Rini)
(125)
and
(1: SC (s,, L
dis,sup Rini)
(126)
43

---

Under review as a conference paper at ICLR 2022
The proof is quite similar to that of Lemma
H3
omit the details.
We are now ready to prove Theorem E.2
Proof [Proof of Theorem E.2
From 99 and 124
we obtain for all t E [0, 11, [0.
1 M
1 m (s)1?
1
Om (s:01)) <C(S, Rini; Lini Cdisssu)
max
M
(127)
(0m (s;
{0, (5)
C 00110111 S C(S, Riai, Lini Sipsssu)
where C(S, Riai, Lini (lisssup) is constant depending on S, Rini and Cat
From a similar derivation to (8T. we obtain
((O(s:))) )) E(OL,M (s)) s C(S, Runs -int-sup) (1(s) - Zo (()((())
(128)
Thus, prove the theorem, it suffices to prove that Zer (s) (L;x) Zo(s) (1:z) is small. For this purpose, according to (133) we need to bound the quantity
MEE
(s) dt,
(129)
M
1=0
I
where Ar,m (8) 01 On (s; (). The next part of the proof contains the required bound.
First, using and (6), we obtain that
(m
ds
(m 2 (Aem Ex-n (1)
2
(). otl.
(1)
(130)
To bound (I), we use 126 to obtain
((D] < C(S, Rinis Ldis.sup Erwya (er. (t0 (17)).)(()();)) (3,4) - Ze(e) (1; xr) + 10 (8) - On 1)]
CS, Rinis C suis,sup)
(131)
where we use Assumption 4.1
14
100
125 and 127 in the second inequality. To bound (II),
we use 13) 100), and (27) to obtain
(II) < C(S, Rini- ini
adis,sup
(16m (s; + 1) E) (1);))))))) Pe(o)(tiz))
(132)
By substituting 132 and 131
into 130g we obtain
d (2)
ds
< C(S, Rinis Edis.sup) ) Ar (s)
+ C(S, Rini, L sups D (s)
- (t(s)(t:z))
+ C(S, Rini, Liu
cdisssu) | (s)) (s; t)1 + 1) E (Ipe).r(s)(::))) (1; pe())(;;)))
44

---

Under review as a conference paper at ICLR 2022
Using Hölder's inequality similar (107
we obtain
1 > 2m
2
ds
1 M
CSS, Rinis
A (s)
(133)
m=1
C(S, Rinis C
(pee (e(s)(t:))) )))
C(S, Rini, ini e Berw
(7 (t;
By substituting 113 and (144 into 133 we obtain
d
Et=
Em
1 m (5)
dt
(1
( 1/
S C(S, Rimis Ciiissup)
ds
M
12
1=0 m=1
which implies, from Grönwall's inequality, that
1-1 M
1 1-1 M
ΣΣЛЛ
(s) dt s C(S, Rinis Lui
A Amm (0) dt
M m=1
M
+
L
1=0
1=0 m=1
(134)
We have thus established the bound 129
We also have
MEΣ
4- M
(0)
L-1
pm 0: 1
(
Om (0:t)
dt. (135)
M
M
1=0
m=1
t
To complete the proof, use
79
and take M
8Ca
to obtain
"
M
P
A (0) S
C4
2
1
1
M
1
8
1=0
m=1
According to 78 when M
8Cas
2 IP dis..su sup < <C,)
(136)
8
By using these expressions to substitute we find that there exists constant C²(C4, Rini 8) depending on C4, Rinis and S such that
M El- Em= S
a and cdie.sup into (34).
8C I Z C²(Ca, Rini S)
M > r
then we have
L-1 M
IP
(s) S
2
7 4
(137)
M
I=0
m=1
127) bound the right hand side of [13). constant C" Rini, S) depending on Ca, Rini, and S such that if
Using (37))
136 and
we find that there exists another
8C
LZ (" Rini, S)
M 2
then we have
P (1:0)) (s) 2 2 21 2
By using this result and (166) in conjunction with (28). we complete the proof.
45

---

Under review as a conference paper at ICLR 2022
H
PROOF OF GLOBAL CONVERGENCE RESULT
Intuitively, if the equation
converges to a stationary point, denote by Poo, so that Ospoc = 0, then
SE V
(0,t) = 0, (so 0 E I a.e. t [0,1].
6p (..)
The rest of the analysis shows that E(px)) = when this happens. However, is not direct because the condition above only suggests the fact that 8p P. is piecewise constant function. We need a stronger result that shows this constant has to be zero. This is achieved by Proposition 6.1] To show this proposition, of (a, 0), particularly the universal kernel property of Assumption 4.1 is this proposition that identifies stationary points with the global minimizer.
follow the proof in (Lu et al. 20200) that explores the expressive power
We should mention that the zero loss was demonstrated by Chizat & Bach (2018) for the 2-layer problem where the stability equates to zero-loss due convexity. The extension to the multi-layer case is more difficult since convexity is not present.
H. PROOF OF PROPOSITION 6.
We first prove a lower bound for Pp in the following lemma.
Lemma H.1 Suppose that C(O0, 1]: P2) and that Pp is a solution
Denoting
Sp S/' 1 10)2d((0.t,
then for any E [0, we have that
Enwn (1p ( (()))
(138)
where Q R. + R4 is decreasing function.
Proof Recall that the initial condition for Pp in is:
(l; (9((,(1:z)) v(x)) (I(Z)();z)),
so from Assumption 4.1 we have
(Vo(aa)) F(p).
Exwn (Ipp(1:))) 2
inf wERd
Further, since the equation linear, we have
Öp
Pp
T
/
0. (2)(2p:))dp(0.
at
gt
According to equation
13 in Assumption
4.1
we obtain
Vga ())
/ (1012 + dp((,tt R/
s <C(z)(t;)))
<C(C,) / (1012 d),
where we use obtain
18
in the second inequality. By combining the last two bounds in the usual way,
(1;)((;))1 S (20(C,) ( (1012 + 1)dp(0,t)) (t:
dt
By solving the equation, we have
(1)()=))12 2 (2;1) exp -22((cp) S. / (102 12400.1) (((p))oo(110)1
46

---

Under review as a conference paper at ICLR 2022
The proof is finalized by taking expectation both sides, and note that monotonicity comes from the format of the exponential term.
We are now ready prove Proposition 6.1
Proof [Proof of Proposition 6.1 Denote
c, 4-= L. J dp(0
According to existence and uniqueness of the solution to 7 for any E [0, 1]. we can construct map 2, such that
(x)
= 2p (1;4)
Since the trajectory can be computed backwards in time, 2, is well defined. Further, we denote M = (Z))zu to be the pushforward of under map Z, and let
p- (1;x) = Pp (1: (x))
By Assumption 4.1 and classical ODE theory, 2, and 2, are both continuous maps in æ, and
Polt; x) and p" ()) With the change of variables, we have for all E [0, 1] that
(E(p) (0, | P (t,)))()))))((()))
(v' (1" (1;x)) (),00)pp) dej
(139)
6p
For fixed to [0, 1], have boundedness of the Jacobian from Lemma C.I meaning that
dut, (z) dy(x)
sup Esupp(p)
CCCp))
As a consequence, Mf, (x) has a compact support since p(x)) does. We denote the size of the support by Rt defined to be a real number such that supp (Mi, (x)) c f |æ| < }.
We now derive a general formula for kpp (6, t) dv. Recalling (13).. we have 8E(p) 8p
J
/ (p" (t): (f f(z, 0) dv(()) dp (x)
(0, to) du(0)
| (x" (to:))))))
(fee dv(()) (x(,, tu)) dpto (140)
f (p' (to: Tp (x, dp).
Noticing that according to Lemma H.l if E(p) O. the second term above is strictly negative (less than -CC the goal then is to find for which f dv = 0 that makes the first term small, so that the right-hand side in (140) is negative. Defining the continuous function h to be
((z) pp" x))
f(z, ((z,0) dp(0,t))
then according Assumption 4.1 for arbitrarily small e, there is i so that f f and
Mxx) [ f(x, f(x,0) dū))))
ce.
Setting p & and substituting into the first term of (140), we obtain
[ (v" (to: (aa ((x,0) du(()) + (x, duea (x)
$ J Ip" (to: h(x)) I Jor f(a, () (x)
(141)
h(x)) - 1 f(z,0) (1),0)d((0)
< [1pp" (1=(to)))lla
47

---

Under review as a conference paper at ICLR 2022
By choosing small enough that (141) is less than {(())))(p), have from ( (0, to) dv(0) < 0, completing the proof.
140
H.2 2-HOMOGENEOUS CASE: PROOF OF THEOREM 6.
We first give an example of 2-homogeneous activation function that satisfy Assumption 4.1 and 6.1
Remark H.1 A finction that satisfies Assumption 4.7 and the 2-homogeneous property of Assump- tion 6.1 is f(x, 0) fxx B (22) = a(O p2l [x]²), where 11] E Rdxdd B/) E Rd.
and a ( max (2, 0}1² applied componentwise.
we first introduce the following lemma, which shows that the separation property is preserved in the training process. Our proof of this result is adapted from (Chizat & Bach 2018).
Before proving the Theorem 6.1.
Let Assumptions 4.1 ladd 4.2 hold, and suppose that Pini (0, 1) is admissible with compact support. Let p(0,1, 8) E C(1O, oc); CCO (0, 1}; P2)) solve (9). If there exists to [0, 11. so that the initial condition Pini (0, to) separates the spheres r ra and ruSk 1 for some O < Ta < Tb. then for
Lemma H.2
any So E oo), p(0, to, so) separates the spheres p and r for some 0 r' To
Proof For every fixed < 80 < S < oc, we note that the particle representation 0,(8; (0) of p(0, to s) updates the following equation:
(0 to) V
8E(p(s))
6p (o) to), 8 8E(05))
ds
(142)
(0; 0.
Define the map (0) to be the solution map that solves the equation above for given initial condition
9 up to time Our proof amounts to showing that this map preserves the separation property.
According to (Chizat & Bach, 2018 Proposition C.11), we need only show that the inverse map of
(0) is stable near 0 for any fixed ( < S. That is, for any O, we need to identify 0 such that
(0) B. (0), v0 B, (0)
(143)
where (0) the k-dimennional ball around original with radius 17
Since f is 2-homogencous in 0, we have that f 0) = 0 for all 2. Thus, from (10)
{F(((()) (0, 0.
8p
Using estimate 49 from Lemma
obtain
ò ((p(s)
(0, (0. C(
8p
where L. = stpoo Jgx 2 dp(0, 1, 8) dt. This upper bound on the velocity implies in
particular that
p. (0) < 10| exp(C(c )s)
which establishes
143
when choose 7 satisfy 77 eexp(- (CCpp( concluding the proof.
We are now ready to prove Theorem 6.1
Proof [Proof of Theorem 6.1 Since p(0, 1, s) converges poo C(O0, 1}: P2), we have any ln that
sup $20
2 dp(O, to, 8) 00.
(144)
According to Proposition 6.1
it suffices to prove that
(E(2))
(0,ta) = O. VB E R.K
(145)
8p
48

---

Under review as a conference paper at ICLR 2022
We contradiction argument: We will assume that (45 is not satisfied and show that Jox [0]2 dp(0, (f, to: s) blows up to infinity as s - 0c, contradicting (144). In particular, we will use homogeneity to construct set in which the second moment blows up.
Define the functions ho and h, as follows:
n0 (0) (pc)
h.(0) 8E(p(s)) (0,10) 6p
(0, (0),
Öp
Recall from 10p that
pp(P)
(0, to) = =Ep (p7 (t),
(146)
óp
Since (155) is not satisfied, there exists 0 such that sF(p~) inequality,
6p (0" to) + 0. From (16)). by Hölder's
(E(p (0", to) s (Bewa (Ip (to; ()o;)))3)) 1/2 (Eeup ())
0<
öp
which implies
Exzy (Ipp> (to: x)) x) > 0.
Then, Since is universal kernel according Assumption [(((oo;; 0) de approximates -Pp. (7). leading to
we can find such that
l heo (z) Rt
1 [ 8E(px) (0, dv()) - Eurn (Ippm (to: x))]) 0.
gx öp
As a consequence, there exists at least one point Gg F such that hoo (Bu) 0. Since f is 2- homoognneus, by (TO). is also 2-homogeneous, so that
1o (00/(00) <..
Because of continuity, there is a small neighborhood around 00/ in S where h is strictly negative. Moreover, since is Sard-type regular, there exist and 77 0 such that
A 0
S o0 los
0 -
V Vol
lat-
0 n > VÕ E OA,
where hoo ig--1 is the confinement of hoc 1 and ny is the outer normal vector 04. This statement of hoo can be extended to h, for sufficiently larger as well. Using estimate (48 from Lemma C.4 we obtain that
h0) hoo (0) in Gbo (TR*),
0o.
meaning there exists S > 0 such that for any 2 S, we have
Nta Malst-1 (0) <
Võ A,
V vahelor 0 127 1
Vē OA.
Extending this patch on the unit sphere to the whole domain, we define the cone:
A= (0 E | 0, 0/|0| E A}
Using the 2-homogeneous property of fr. we have for S that
h (0) < </0/ (0) n > 0,
V0 E A,
(147)
VB E aAn
n (10) > 0},
where ni is the outer normal vector to 0.4.
49

---

Under review as a conference paper at ICLR 2022
We now define new system that follows the gradient flow corresponding to Ps. Denote by 0 (s:a) the solution to the following ODE:
d0 (s; a)
(E(p(s))
(0(s;a),to) - Vohts (ō(s;a)), 8>8
Ve
ds
8p
(148)
a) =
where a E RR According to 147 when ÖAN (10) > 0), Voh,(0) points outwards, away from
A. We also notice that 0 s; 0
ö. Thus if the ODE starts with from some a E A, then for any
2 S, the particle stays within A, that is,
(s;a) EA.
(149)
As consequence, we have
d ((s:a)) - (0(s:)), Voh, (0(s:a))) -h,, (0():a)) > 2c 0 (s: (150)
ds
where the 2-homogeneous property of h, in the second equality and (147) in the final inequality.
According to Lemma H.2 there exist two spheres separated by p(0. to S), meaning that there exist 3 > and >> relatively small (for example, with 3 < rry) such that
f SamBBat) dp(0, S) >
(151)
By tracing the trajectory of (148), we have
f Sancerro) dp(0, lo 8)
f ((osa)eeAn dp(a, l0
(T)) dp(a, (0, 8) > 7, s2 8,
d(o()a))
where in the first inequality we also use
2 0 when a E A. Further, we have
de
d (()(( (les))ean ((::a) 2
dp(a, to, S)
ds
(Snn(s,(T))) O (s:0a) lp
ds
2e / (An(Ba(())
2 0 ((s:a) dp(a,to. $)
2 20y
where we use 150) in the second inequality and [151 in the final inequality. It follows that
lim [ (Ar(BA(O))= 000
(s: a)) dp(a, tu S) = 0o.
(B))))
follows from this result that
lim 1012 dpt,, to. = 00.
contradicting 144
Therefore, we must have
(pxw)) (0, (0) =0, v0 v0 E ER
6p
which completes the proof.
50

---

Under review as a conference paper at ICLR 2022
H.3 PARTIALLY 1-HOMOGENEOUS CASE: PROOF OF THEOREM 6.2
We first give an example of partially 1-homogeneous activation function that satisfy Assumption 4.1
and 6.2
Remark H.2 The following function satisfies Assumptions 4.7 and 6.2
Let 0 (0n)- 12) (3))
with fij Rd. 01] Rdxd 0/] E Rdd Define f(z,
f(x; 012 0p))
18/2 x (13)02(1011) ) where (x) is a regularized ReLU activation function, and 02 (2()/// R R.+ are bounded, Lipschitz, and differentiable with Lipschitz differential. One
Pija (11102(10011)
191s
way (of many) to define a regularized ReLU activation function is o(x) = (x+ +1) n1² (4n)) xe(-n.ol
al co( for some small r).
As in the previous theorem, prove lemma, adapted from (Chizat & Bach 2018 Lemma C.13), that asserts preservation the separation property.
Lemma H.3 Let Assumptions
and 4.2 with 1 1. Suppose that Piul (0.t))) t) is admissible
and suppo (Pmn (0, t)) C {0]
RJ with some R > 0 for all t E [0. Let p(0, t, E
c((o, oo); C(10, 1); P2)) solve Suppose in addition that
f satisfies the partial 1 I-homogeneous condition (see Assumption 6.2
The initial conditions satisfy the separation condition, that is, there exists to E E 1] such that Pmni (0 0/)) separates the spheres {-ro} R- and {ro} R for some
> O.
Then for any su E [0, oo), p(0 (2)) to, 8o) separates {-r'} x k-1 and {r'} x TR--1 for some
Proof Note that the particle representation (p(s; to) of p(0, s, to) satisfies
de (ssta)
= 5E(pia)) 6p (0, (s:
de 0 (0:
(152)
= 0.
Define a continuous map P : R* x 0, oc) + R& as the solution to (52).. that is, P(0,s) s) is the solution (152) with initial condition (0( tu) 0, where to fixed. Define diffeomorphism di: R IR x B1 (0) as follows:
(0 = { (0p)
tanh (1)))),
0 + 1 = 0,
{ ())
where 011 E is the first component of and 02 E k-1 contains the remaining components. This map keeps the first component of Oμ1 intact and shrinks O1t to push its amplitude below 1. This diffeomorphism preserves the commection//eppraaion property.
Define the continuous map Q follows:
2(0, s) vi P(u (0) : B1 (0) x [0, 00) + - x Bi
Since vb preserves the connection property, the lemma proved if we can show sb P(supp(Pmmi) s)
separates {-r') } Bi (0) and (r') Bi (0) for some r' > 0. Since
((supp((mi)).)
O vi (itsspppppa )), s). we trace the evolution Q(0, 8) for E R x (1))). According to (Chizat &
Bach 2018. Proposition C.14), this translates to showing @ s) can be continuously extended to
B (0) x 00, S] R B, (0), with the extension satisfying
2 E O 0
vO
0B
0
8 SE 00)
(153)
meaning that the extension Q(: s) maps R (3:(0) to itself for all j0)
51

---

Under review as a conference paper at ICLR 2022
Denoting
(0) Q s), consider the velocity field of this flow (similar to the proof of (Chizat Bach 2018 Lemma C.13)):
dQ.
dPs o
(Vor) (Pa o の (000~((0) (0)))
ds
ds
= (Vor (Ps s/ (0)))
Vi 8B(p(s)) 8p
(Ps (P, to))
V öB(p(s))
= (Vop 4 (2.)))
(v (2)) = V(2., 8).
6p
From the fourth condition of Theorem 0B, (0) as follows:
6.2 the velocity field V(O, 8) can be continuously extended to
(v (v.
(E(p(a)) 8p (1b (0),
|0/1 <
V( s) = V (0]) 02), ( (0))) (0)2))
|0
where Hoc,p(s) the limit of 5B(p(e))
(1, 10 20 to) as r + co. Within this velocity field, Q. can be
6p
continuously extended to IR x B1 (0) x [0, oo) - R x B1 (0) with the extension satisfying (153)
This completes the proof.
We are now ready to prove Theorem 6.2
Proof [Proof of Theorem 6.2 The technique of proof is similar to the 2-homogeneous case. Since p(0, f, s) converges to Poo (8, t) in C( 1]; P2), we have
sup dp) to 8) < 0o. $20 gz
(154)
According to Proposition 6.1 it suffices to prove that
(poo) 8p
(p(o)) (01)) 012 to) = 0111 5p
(p(())) (1, 012). =0, 40 E R
(0, (0) = 5p
(155)
In the following, we will show that Jgo 1p]3 s) blows up as + oo if 155 fails to hold, in contradiction (154).
Denote
No0 (0)) (((o)) (1, 11,022,,40 (0), (0p))))) 8E(p(s)) (1, 012))4))
(156)
8p
5p
Then if (155) is not satisfied, without loss of generality, we can assume that there exists 0/) such that
h (2) < 0. Since ho satisfies Sard-type regularity, there exist and such that
A = [0(2 E IR- hhc (012)) < -c * 0; Very (o0(1))))))) ne > n. A/ 4022 E 0A,
where n is the outer normal vector on AA.
Using the definition of 5E &p in I0 we have
8E(p(s))
oE((so)
V Mia
(1, 8p). (1) - Vo óp
öp
Evwn (2pts) 0 (s) (1;x1) 0up f(z (1:x)), O2z )r (t:x))
Euwp
(() x:); (t), 8 f(Zp- (t: 222)) (p:cs)(:z))
Exnnye ( |00p)) f( (t: 021) (pcc) - Pp- (t: (t:z))
C (Eeny (1:(o) Z, (1;x)) + Enay (ppoco)(t; Paa (1;
< (p(s)
52

---

Under review as a conference paper at ICLR 2022
where we have used the Lipschitz continuity of f and its derivatives in Assumption 6.2 and the estimates 18). 31I. and (47). We thus have
h.(012)) + (012)) in C²(R+11), as + 00.
(157)
As consequence, there exists S such that for any 2
hb (0(2) < cc/2, 1 ho (0(2) pe 2
l012 E A, 60 0A.
(158)
Extending this set to the whole space, we define
A = (0 022)) E (0) x A}
Since 0.A = {(01)) (2)) E o0) x dA} U (0[ = 0, 0,0 E A), from (58 and the definition of hg, we have
{E(s(s) (0, (o) 7gg = 011 (Vora (02)) n > 0, 40 E (0, x)) x 0A,
(159)
5p
where 7g the outer normal direction on 0.A, and 10 is the outer normal vector on 0A. When
0 E (0 = -0, 021 E A].
(F(((s)) (0,to) (0(2)) <
8F(p(s)) (0, (0)
V
0, i= 2......, k, (160)
0p
8p
where [:]s means the i-component of the vector. This implies that Vo (E(a(a)) downward when 0 E {0p] 02 E A}.
6p (0, to) points strictly
As in the 2-homogeneous case, we consider the gradient flow corresponding to Ps. Denote by 0 (s; the solution the following ODE:
s((o))))) ds = = &E(p(s)) (0(:;a) = -o (Buuh, (f1)) (0():)) 8> s 6p
(161)
0 (S:
0
where a E RR. Since the minus gradient is pointing inward to A, as stated in [159 and 160), 0 (s;
stays in A if 0 (s';a) E A for some s' E (S, s). Moreover, using (T6T. if (s; E A, we have
dl0m (s; 20 (s; h, (0r2) (0; > e0 (8)
(162)
ds
where the last inequality uses 158
Similar (Chizat & Bach 2018 Proposition C.4), we claim that there exists S 2 S, > and
>>> 0 such that (see detailed proof in Appendix H.3.1
[ Lesmnna di( >7
(163)
Then
fg. (( 011 (1;
dp(a, to S)
ds
20 / gg 1 6 (s; a) dp(a, to S)
(164)
2 eB [ ( dp(a, la S)
= e, /
((xo)x
dp(0. to S1) 2 eBy.
Since
10111 dp(O, to. 3) lim 1 [01] (s: to, = 00.
lim 40000
(B,00)xA
10000
53

---

Under review as a conference paper at ICLR 2022
finally obtain, using (164). that
lim 8-000
10/2 dp(0, 10, 2 lim
10t1] 2 dp(0, 40, 8) 20.
R
5-000 (8,00)
This limit contradicts
154). implying that (155) must hold, as claimed.
H.3.1 CLAIM IN THE PROOF OF THEOREM 6.2
In this section, prove the statement in 163 meaning that need find Si 2 S, > and > such that
Jescoya dp(0, S1) 7
(165)
Supposing that SA dp(0, to, S) > O, then by making 3 and small enough, (165 is satisfied naturally.
JA dp(0, fo, S) = O, then it suffices to show that there exists S > S such that
S dp(0, to: S1) > 0.
(166)
Define h, (0(2) and h0x as in (156). Because of the fourth condition in Theorem 6.2 there exists a function h'(0) on C (S*-2) such that
hoce (rŌ) - 1((0), in C² (s
Combining this with (157 there exists h² > e/2 such that |1 h². Further, for any & > 0, we can find 02) E A and S" large enough such that
Vely ha (0( <E, 2'
(167)
According to Lemma H.3 there exists Ts 0 such that p(0 012) (o, separates {-rs } x
R- and [rs x R Considering the set [-rs"rrs x (0%2) ). it must intersect the support of
p(0p Op2) fo, S*) due to the separation property. Thus, any open set that contains [-rs], rT x 10
must have a positive measure in p(0 012,, , to, S²). Because SA dp(0, to. S*) = and (-rs: 1, oo) is a open set that covers (-s x (0))). there exists open set C (-rss 1, 0] x
such that
J dp(0 0g.. to. S*) > 0.
Thus, can find point r S rs + 1 and arbitrary small 0 such that
Ba(-r", 022) C (-rs - - 0, x A, and |
dp(0, l0, 8¹) > 0.
(.((r)))))
Recalling the system 161 we claim the following:
When E, a are small enough, there exists S > St such that 0 (Sija) E Afor any a E B,(-r" (g))
If this claim is true, then
f
dp(0, dp(0, 2
dp(0,to, 0.
B.(-rr)
which proves (166) and the lemma.
54

---

Under review as a conference paper at ICLR 2022
Now, prove the claim. Because satisfies Assumption 6.2 and the second moment of p is uniformly bounded in 8, for all s > O, we have
(E(p(s))
(1, (2,0 to) (E(p(s))
(1, (1,82)- (2) to)
$ 100 8/21
öp
öp
(168)
8 {E()(s)
(1, 1,0 (0) Ve (E(p(s))
(1, 0 (2) to)
< 1/ 0/2
öp
5p
for some constant L. According to 166 we have d0 [0)()a))) = -h (1)((sa))
ds
1 (s; 0/2 12
8 82 (169)
2 01]((s;)) - -0/2 [0n(s:a))] Va hs ((1a(5:aa))
ds
where
o ( s, and (S': 0 0
(170)
To prove the claim, it suffices to show that there exists S > S' such that
رl (S); > and 001 (Si: a)) E A.
(171)
We first show that Un increases and the right hand-side of (169) is bounded. Since A is a open set, there exists an arbitrary small such that (8 A. We first choose a < (2, 2}. When
Pu (s: a) S
2r + ht + 2,
(s) - 072) - Σ,
(172)
have from 158).
167 and (168) that
d0 (s; a))
(1; -hs (8/)(6:0)) > 6/2
hs (8)((5:))) < <h²,
(173)
ds
ds
and
d
1p1 (s; 022
ds
2r + 2
+2) (s: 0// (L (s; -s; -0p] +5)
(174)
S
(2r + 2 h1- 2 [0p](s;;)) - 0/2 2 +
2rs: + 112 +2 Σ
$ 2L
When S" and E Bo(-r" 0(z)) we have from 170) that
(St S rss + 1 to 2rs + h-- + 2, aS') 0/// 5o<E.
Thus, for slightly larger than S², we still have that
(uy(s; a) <
2rs + 2 ht + 2 and
[]][(s;a- E.
Denote by S* the first time that
(n((s;:)) 2 2rs + 2
h" +2 or
(S a) - 07 > Σ.
Then we show that there exists S₁ S S*1 such that (7I) is satisfied when o, and & are small
enough. From (17)). we have for E (S", S*)
that
(1](s;a) I a S
s')h"
(1)(5;a)= -rs + (s S'')e/2,
(s; -0/) <exp 1
2rs ( +2 11 +2) (s (s-)") (0² (22+2(s- 5') (2rss) +2 h² + EL
1/2
55

---

Under review as a conference paper at ICLR 2022
where the first two inequlaties come from [73 and last inequality comes from (744 via Grönwall's inequality. Defining
S1 2r + 2 + S"
we can choose the positive values 0, E, and & small enough that
(1)(S,, 05 -rs + + -2'ce2=
and for s E [S" Si]
pnn (s: a)
2rs +2 h² +
[p)(s::) a) -8(2) E.
(175)
According (175). the bounds 172) are satisfied for [ S1], which implies that S S and 8/2 (S1, g8') E A. Further, we have (Sn > 1. By combining these two results, conclude that [77 is satisfied with the chosen values of a, Σ, and Si. Thus, we have 0 (Staa) E (L) A forr any Bo(-r",0p which proves the claim.
56


Reviews:
[{"review_id": "vlM4VjPgVu1", "reviewer": "Reviewer_pGat", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work reports a new mean-field explanation for an infinitely deep and infinitely wide neural network with skip connection and provides global convergence results with the framework of (Chizat & Bach, 2018) under standard assumptions and the universal kernel property exploited in (Lu et al., 2020). My main comments are as follows:\n\n* In P4, the literature review paper list for mean-field limit perspective is not correct, e.g., (Allen-Zhu et al., 2019), (Zhang et al., 2019) are not in that regime.\n\n* In Assumption 4.1, the activation f is required to be C^2. In P6, just below Assumption 4.2, they claim \\sigma can be regularized ReLU, which is seems not introduced. Please explain.\n\n* In Theorem 5.1, the final bound seems not uniform for 0<=s<S, which is a much weaker nonasymptotic result than that in (Mei et al., 2018) for two-layer NN.\n\n* The sequential limit, as shown in Theorem E.1 and E.2, is taking limit for L before increasing M, which is similar to the layer-wise scaling used in (Sirignano & Spiliopoulos, 2018). Thus, the M and L would have to grow up at specific speeds simultaneously. So, for curiosity, can we exchange the order of limits of M and L in an asymptotic version of Theorem 3.1?\n", "summary_of_the_paper": "This paper studies the gradient flow training dynamics for deep and wide ResNet with the mean-field scaling. For a similar setting as (Lu et al., 2020), they report a limiting model with finite-dimensional approximation, and its global convergence result. The authors also justify the well-posedness of the resulting distributional dynamics, which seems only heuristically defined in prior work. ", "summary_of_the_review": "Overall, I think this paper has useful ideas for understanding the training dynamics of ResNet in the nonlinear regime, and it complements the existing research in the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "oSAoOV62AnG", "reviewer": "Reviewer_Q3Rw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work analyzes infinitely deep and infinitely wide residual networks. Overall, I found the writing to be quite good, and these types of results are of interest to the community (at least they interest me). I have not been able to check the details of the proofs, but they seem plausible.\n\nThe limit of viewing infinitely deep ResNets as an ODE is due to the Neural ODE paper, and further taking the infinite width limit to consider the mean-field regime was considered in [1]. Therefore, the main contribution of this work is in analyzing the training dynamics of regular gradient flow on this network ([1] considered a different, unusual training mechanism).\n\nHowever, I would like to get more clarity on the incremental novelty. As the authors acknowledge, the paper is very similar to [2]. Since this paper is meant to be a standalone paper distinct from [2], I would like to get some clarification as to what are the specific contributions not present in [2]. \n\nThe results of section 6 seem to be the global convergence of the continuous limit, is the core contribution. The technical challenges that had to be overcome appear to be quite similar to those considered in [3]. I would like to hear from the authors about the novelty of the proof technique compared to that presented and used in [3].\n\n\n[1] Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, Lexing Ying. A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth, ICML, 2020.\n[2] Zhiyan Ding, Shi Chen, Qin Li, Stephen Wright, Overparameterization of deep ResNet: zero loss and mean-field analysis, arXiv, 2021.\n[3] Huy Tuan Pham, Phan-Minh Nguyen, Global Convergence of Three-layer Neural Networks in the Mean Field Regime, ICLR, 2021.\n", "summary_of_the_paper": "This work analyzes infinitely deep and infinitely wide residual networks. The infinite depth limit is the neural ODE. The infinite width limit is the mean-field limit. The main contribution establishing global convergence of the infinitely wide and deep ResNet. Finally, the global convergence of the infinite limit is translated to the finite but large setup.", "summary_of_the_review": "Interesting results, but contribution should be clarified.\n\nI am willing to raise the score if the issue of novelty is clarified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Ujlwl5fgfwl", "reviewer": "Reviewer_YcBa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Many thanks to the authors for addressing my concerns and correcting the mistake.\n\nAs said in my review, there are reasons to believe why the contribution of this paper is a clearly meaningful one on top of prior works in similar resnet mean field setups, namely the removal of the full support assumption on $\\rho_{t=\\infty}$. Though again the technicality is not necessarily surprising, this is hard work. My final recommendation would have been 7 if possible; in support of this paper, I'm making 8.\n\n==============================\n\nThere is a lot of interests in the mean field analysis for neural networks, so the subject of the paper is timely. Before this work, Lu et al 2020 attempted to analyze a very similar problem setup, but had to make the full support assumption of $\\rho_{t=\\infty}$ for their global convergence result. The current paper fills in this gap via Theorems 6.1 and 6.2, along with Theorem 5.1 which shows that the wide and deep resnet can be approximated well by the corresponding continuous MF model. \n\nIt should be stressed that this assumption on $\\rho_{t=\\infty}$ is a very strong one (as pointed out in Pham and Nguyen 2021) and is unrealistic in certain situations. Removal of this assumption tends to be highly technical; one can just look at the length of the proof of this paper, compared to that in Lu et al 2020. As such, the paper has good and meaningful technical contents.\n\nIn that sense, my overall evaluation is that the paper gives completion to Lu et al 2020, though I am not surprised by the result nor the techniques. In particular, while I cannot check the lengthy proof details, I do not doubt the results, and the arguments in the presented proof are within expectation. Let me go into further details of the global convergence proof of Theorems 6.1 and 6.2:\n- Compared to Lu et al 2020, perhaps the one piece that differs is Proposition 6.1, which recognizes that the condition of global optimum can be checked at any residual block, unlike what was done in Lu et al 2020 which sort of integrates over all blocks. The key is the appearance of $g(Z_\\rho) - y$ term in the gradient update of every residual block, and any place where this term appears can be exploited. I must say, though, this observation is not new and has been used elsewhere in the mean field literature.\n- Since the problem comes down to any of the residual blocks, which are just two-layer MF neural nets, the entire stability argument from Chizat and Bach 2018 can be replicated. Another piece of technical work is in proving the support property of $\\rho(s)$ for any time $s$, and this is again a direct application of the topological degree lemma in Chizat and Bach 2018.\n\nTo add to this point, there are results which are proven with similar analyses (with the Sard-type regularity and the stability argument) on setups other than two-layer nets; examples include Theorem 50 in Nguyen and Pham 2020 for deep feedforward nets and Theorem 11 in Agazzi and Lu 2020 for temporal difference dynamics. In other words, this analysis can be carried out (with some technical confidence, admittedly) for the right assumptions.\n\nOne major issue is that Theorem 5.1 and Theorem 6.2 cannot be merged to claim Theorem 3.1 the main result. As far as I understand, Theorem 5.1 assumes compactly supported initialization distributions. This compact support assumption makes it possible to prove well-posedness and eases various parts of the proof of Theorem 5.1. However it is important NOT to have initializations with a finite support for the global convergence results (Theorem 6.2) to hold. Due to this mismatch, one cannot claim Theorem 3.1. To be fair, the development in Chizat and Bach 2018 also goes along the same line; the issue can be avoided by either:\n\n- assuming a compact domain for the first layer weight with an appropriate version of GD (which Chizat and Bach 2018 covers, but the current paper does not),\n\n- working directly with projection onto the sphere in the 2-homogenous case (which Chizat and Bach 2018 does, but the current paper does not).\n\nThis mismatch in the conditions is technical and should be an interesting mathematical problem to fill in, but does not seem to pose any conceptual issue in machine learning. As such, in light of the established literature, I would not evaluate strongly against the paper, but I ask that the paper remove Theorem 3.1 and make a clear acknowledgment of the mismatch.\n\nAnother request is that the paper should also make more efforts in discussing the last Sard-type regularity condition in Theorem 6.2. While the paper claims this is a very mild assumption and refers to Remark 4.1, this remark only concerns with the third Sard-type condition and not the last one. In my opinion, it is this last condition that is extremely difficult to verify (the third one can be satisfied for sufficiently smooth activation functions). This is because it involves dealing with points that are at infinity. Again since this assumption is adopted from Chizat and Bach 2018, I would not evaluate strongly against the paper.\n\nMinor comments:\n- $\\epsilon$ and $\\eta$ are not explained in the first page.\n- The equation before (3) seems to be missing a factor of $ML$.\n- The paper should clarify what “regularized ReLU” is; this is not a standard term.\n- The example activations in Remarks 6.1 and 6.2 are somewhat contrived; can the paper give more familiar examples?\n\nReferences:\n\nL. Chizat, F. Bach, On the global convergence of gradient descent for over-parameterized models using optimal transport, NeurIPS 2018.\n\nY. Lu, C. Ma, Y. Lu, J. Lu, L. Ying, A mean field analysis of deep resnet and beyond: Towards provable optimization via overparameterization from depth, ICML 2020.\n\nH. T. Pham, P.-M. Nguyen. Global convergence of three-layer neural networks in the mean field regime, ICLR 2021.\n\nP.-M. Nguyen, H. T. Pham. A rigorous framework for the mean field limit of multilayer neural networks, 2020\n\nA. Agazzi, J. Lu, Temporal-difference learning with nonlinear function approximation: lazy training and mean field regime, 2020.\n", "summary_of_the_paper": "The paper proves a global convergence result for GD in resnets, whose residual blocks are mean field two-layer neural nets. This is taken with a double limit, where the depth limit is the neural ODE and the width limit is the two-layer MF limit. To prove global convergence of the infinite-depth/width limit, the paper employs previous techniques that assume Sard-type regularities and the accompanying stability argument from Chizat and Bach 2018. ", "summary_of_the_review": "This is a technical paper that removes a strong assumption from the analysis of Lu et al 2020, though the techniques are not surprising. A number of edits are necessary. Though giving a confirmation to a result that is expected, the paper is still a good theoretical contribution, in light of the heavy technical works that are required.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "MdS6uApMzkj", "reviewer": "Reviewer_jaQt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe proof of this paper seems reasonable and the assumptions and theorems are well-presented. It contributes to an interesting and important research direction, which is to study the convergence of neural ODE/PDEs. However, I have several concerns about this paper.\n\n1. The contributions of this paper are kind of incremental given existing works [1,2]. The main theorems in these papers and this submission are quite similar. In fact, the contributions given in Section 3 are almost identical to the contributions listed in [2], which is concerning. Below Theorem 3.1, it is only discussed how it is nontrivial to extend existing results in the mean-field regime to multilayer networks, but a detailed explanation of how this paper (in terms of the convergence results) differs from [1,2] is missing. \n\n[1] Lu et. al., A mean field analysis of deep ResNet and beyond: Towards provably optimization via overparameterization from depth. ICML 2020.\n\n[2] Ding et. al., Overparameterization of deep ResNet: zero loss and mean-field analysis. arXiv preprint arXiv:2105.14417.\n\n2. It is also not very clear what is the advantage of this paper compared with prior works on mean-field or NTK. For mean-field, while the mean-field works on two-layer networks are not applicable to multi-layer networks, this paper does not seem to be applicable to any practical multi-layer network architectures either, as this paper requires the network depth to go to infinity. So the authors may need to comment on why studying this infinite limit is interesting. Besides, in (1), is it standard to consider a scaling parameter $1/L$ in each layer? In practice, the residual block is directly added without using this scaling. \n\n3. My next concern is regarding the comparison to the NTK results. In particular, [3] shows that for two-layer NN, when considering updating the “distribution” of model parameters, the obtained solution still exhibits a “kernel-like” behavior. So I would like the authors to discuss more regarding the advantage of the developed results compared to the kernel models, rather than simply mention “NTK views DNN as a kernel model, a rather limited description, so the estimates obtained through NTK can be far from sharp”. One way to show this is to provide some examples to demonstrate the gap between these two regimes.\n\n[3] Chen et. al.,. A generalized neural tangent kernel analysis for two-layer neural networks. NeurIPS 2020.\n\n4. Following my previous comment, the discussion on the advantages of ResNet should also be included, since I believe the main goal of studying ResNet is because of its superior performance compared to fully-connected NN. In the NTK setting, [4] has shown that adding residual structure can improve the dependency on NN depth compared to FC-NN. Then one question is what can we gain by studying the convergence of ResNet in the mean-field regime?\n\n[4] Frei et. al., Algorithm-dependent generalization bounds for overparameterized deep residual networks. NeurIPS 2019.\n\n5. My last question is whether the mean-field analysis can be adapted to classification problems? It has been widely known this can be done in the NTK regime [5, 6] since the convergence analysis in the NTK regime does not require the loss function to be MSE loss but can be applied to any convex loss function. The authors may also need to discuss this point in the mean-field regime.\n\n[5] Li and Liang, Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data. NeurIPS 2018\n\n[6] Zou et. al., Gradient descent optimizes over-parameterized deep ReLU networks. Machine Learning 2019.\n\n\nMinor comments\n\nThe notations in Section 2, Assumption 6.2, and Remark 6.2 are not very consistent.\n\n\n", "summary_of_the_paper": "This paper studies the training of a multi-layer ResNet using mean-field tools, where the training procedure of the ResNet is given in its continuous limit as a partial differential equation. A rigorous proof is given, and the convergence to a global minimum of the cost functional is given.", "summary_of_the_review": "While this paper provides some interesting insights into the training of infinitely deep and wide ResNets, it is not well-demonstrated whether the results are novel and significant enough compared to various existing works. Some detailed discussion and comparison may be very helpful to improve the quality of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an improved mean-field analysis for multi-player residual networks. Compared with prior works, the proposed analysis removes a full support assumption needed in prior works. The authors have addressed some of the reviewers’ concerns by adding comparisons with the existing analysis of ResNet in the NTK regime, and a more detailed comparison with Ding et al. 2021. While this paper gathers some support from a reviewer, there is still concern that the novelty of this paper is not significant, especially given that the analysis is heavily built upon prior works. I think this paper can benefit from providing a proof sketch to highlight the key difference between the new analysis and existing analyses,  or explicitly demonstrating the key proof technique/technical lemmas that enable the removal of the full support assumption. This paper might be a strong work after careful revision.
Decision: Reject