Year: 2021

Abstract:
Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first order methods nevertheless find the global optimum in the overparameterized regime. We study this phenomenon with mean-field analysis, by translating the training process of ResNet to a gradient-flow partial differential equation (PDE) and examining the convergence properties of this limiting process. The activation function is assumed to be 2-homogeneous or partially 1-homogeneous; the regularized ReLU satisfies the latter condition. We show that if the ResNet is sufficiently large, with depth and width depending algebraically on the accuracy and confidence levels, first order optimization methods can find global minimizers that fit the training data.

heading: 1 INTRODUCTION
text: Training of multi-layer neural networks (NN) requires us to find weights in the network such that its outputs perfectly match the prescribed outputs for a given set of training data. The usual approach is to formulate this problem as a nonconvex minimization problem and solve it with a first-order optimization method based on gradient descent (GD). Extensive computational experience shows that in the overparametrized regime (where the total number of parameters in the NN far exceeds the minimum number required to fit the training data), GD methods run for sufficiently many iterations consistently find a global minimum achieving the zero-loss property, that is, a perfect fit to the training data.
What is the mechanism that allows GD to perform so well on this large-scale nonconvex problem?
Part of the explanation is that in the overparametrized case, the parameter space contains many global minima, and some evidence suggests that they are distributed throughout the space, making it easier for the optimization process to find one such solution. Many approaches have been taken to characterize this phenomenon more rigorously, including landscape analysis, the neural tangent kernel approach, and mean-field analysis. All such viewpoints aim to give an idea of the structure and size of the NN required to ensure global convergence.
Our approach in this paper is based on mean-field analysis and gradient-flow analysis, the latter being the continuous and mean-field limit of GD. We will examine residual neural networks (ResNets), and study how deep and wide a ResNet needs to be to match the data with high accuracy and high confidence. To relax the assumptions on the activation function as far as possible, we follow the setup in (Chizat & Bach, 2018), which requires this function to be either 2-homogeneous or partially 1-homogeneous. We show that both depth and width of the NN depend algebraically on and η, which are the accuracy and confidence levels, respectively.
Mean-field analysis translates the training process of the ResNet to a gradient-flow partial differential equation (PDE). The training process evolves weights on connections between neurons. When dealing with wide neural networks, instead of tracing the evolution of each weight individually, one can record the evolution of the full distribution of the weight configuration. This perspective translates the coupled ordinary differential equation system (ODE) that characterizes evolution of individual weights into a PDE (the gradient-flow equation) characterizing the evolution of the distribution. The parameters in the PDE naturally depend on the properties of the activation functions. Gradient-flow
analysis is used to show that the PDE drives the solution to a point where the loss function becomes zero. We obtain our results on zero-loss training of ResNet with GD by translating the zero-loss property of the gradient-flow PDE back to the discrete-step setting.
This strategy of the proof was taken in an earlier paper (Ding et al., 2021) where multi-layer ResNets were also analyzed. The main difference in this current paper is that the assumptions on the activation function and the initial training state for obtaining the global convergence are both much relaxed. This paper adopts the setup from (Chizat & Bach, 2018) of minimal Lipschitz continuity requirements on the activation function. Furthermore, the paper (Ding et al., 2021) required a dense support condition to be satisfied on the final parameter configuration has a support condition. This condition is hard to justify in any realistic setting, and is discared from current paper. Further details on these issues appear in Section 3.
We discuss the setup of the problem and formally derive the continuous and mean-field limits in Section 2. In Section 3, we discuss related work, identify our contribution and present the main theorem in its general terms. After precise definitions and assumptions are specified in Section 4, we present the two main ingredients in the proof strategy. The mean-field limit is obtained by connecting the training process of the ResNet to a gradient-flow PDE in Section 5, and the zero-loss property of the limiting PDE is verified in Section 6. The main theorem is a direct corollary of Theorem 5.1 and Theorem 6.1 (or Theorem 6.2).

heading: 2 RESNET AND GRADIENT DESCENT
text: The ResNet can be specified as follows:
zl+1(x) = zl(x) + 1
ML M∑ m=1 f(zl(x), θl,m) , l = 0, 1, . . . , L− 1 , (1)
where M and L are the width and depth, respectively; z0(x) = x ∈ Rd is the input data; and zL(x) is the output from the last layer. The configuration of the NN is encoded in parameters ΘL,M = {θl,m}L−1,Ml=0,m=1, where each parameter θl,m is a vector in R
k and f : Rd × Rk → Rd is the activation function. The formulation (1) covers“conventional” ResNets, which have the specific form
zl+1(x) = zl(x) + 1
ML M∑ m=1 Ul,mσ(W > l,mzl(x) + bl,m) , l = 0, 1, . . . , L− 1 ,
where Wl,m, Ul,m ∈ Rd, bl.m ∈ R, and σ is the ReLU activation function. In this example, we have θl,m = (Wl,m, Ul,m, bl,m) ∈ Rk, with k = 2d+ 1. Denote by ZΘL,M (l;x) the output of the ResNet defined by (1). (This quantity is the same as zL(x) defined above, but we use this alternative notation to emphasize the dependece on parameters ΘL,M .) The goal of training ResNet is to seek parameters ΘL,M that minimize the following mismatch or loss function:
E(ΘL,M ) = Ex∼µ [ 1
2
( g(ZΘL,M (L;x))− y(x) )2] , (2)
where g(x) : Rd → R is a given measuring function, y(x) ∈ R is the label corresponding to x, and µ is the probability from which the data x is drawn.
Classical gradient descent updates the parameters according to the formula
Θn+1L,M = Θ n L,M − h∇ΘE(ΘnL,M ) ,
where h is the step length. In the limit as h→ 0, the updating process can be characterized by the following ODE (Chizat & Bach, 2018, Def 2.2) (rescaled by L,M ):
dΘL,M (s)
ds = −ML∇ΘE(ΘL,M ) , for s ≥ 0 , (3)
where s represents pseudo-time, the continuous analog of the discrete stepping process.

heading: 2.1 THE CONTINUOUS LIMIT AND THE MEAN-FIELD LIMIT
text: The continuous limit of (1) is obtained when the ResNet is infinitely deep, with L → ∞. By reparametrizing the indices l = [0, · · · , L− 1] with the continuous variable t ∈ [0, 1], we can view z in (1) as a function in t that satisfies a coupled ODE, with 1/L being the stepsize in t. Accordingly, θl,m can be recast as θm(t = l/L), and denoting Θ(t) = {θm(t)}Mm=1, we can write the continuous limit of (1) as
dz(t;x)
dt =
1
M M∑ m=1 f(z(t;x), θm(t)) , t ∈ [0, 1], with z(0;x) = x . (4)
Extending (2), we define the cost functional E as E(Θ) = Ex∼µ [ 1
2 (g(ZΘ(1;x))− y(x))2
] , (5)
where ZΘ(t;x) solves (4) for a given collection Θ(t) of the M functions {θm(t)}. Similar to (3), we can use GD to find the configuration of Θ(t) that minimizes (5) by making Θ(t) flow in the descending direction of E(Θ). Denote s the pseudo-time of the training process, and Θ(s; t) the collection of functions at the training time s:
∂Θ ∂s = −M δE δΘ ∣∣∣∣ Θ(s;·) , s > 0, t ∈ [0, 1] (6)
where δEδΘ is the functional derivative of E with respect to Θ, and thus a list of M functions of t for every fixed s.
The mean-field limit is obtained by making the ResNet infinitely wide, that is, M →∞. Considering that the right hand side of (4) has the form of an expectation, it approaches an integral in the limit, with respect to a certain probability density. Denoting this PDF by ρ(θ, t) ∈ C([0, 1];P2)1, and assuming that the θm are drawn from it, the ODE for z translates to the following:
dz(t;x)
dt = ∫ Rk f(z(t;x), θ) dρ(θ, t) , t ∈ [0, 1] with z(0;x) = x . (7)
Mimicking (5), we define the following cost function in the mean-field setting: E(ρ) = Ex∼µ [ 1
2 (g(Zρ(1;x))− y(x))2
] , (8)
where Zρ(t;x) is the solution to (7) for a given ρ. Then, similar to the gradient flow for ΘL,M and Θ(t), the probability distribution ρ that encodes the configuration of θ flows in the descending direction of E(ρ) in pseudo-time s. Since ρ(θ, t, s) needs to be a probability density for all s and t, its evolution in s is characterized by a gradient flow in the Wasserstein metric (Chizat & Bach, 2018; Lu et al., 2020; Ding et al., 2021):
∂ρ ∂s = ∇θ ·
( ρ∇θ δE
δρ ∣∣∣∣ ρ(·,·,s) ) , s > 0, t ∈ [0, 1] with ρ(θ, t, 0) = ρini(θ, t) , (9)
where δEδρ is the functional derivative with respect to ρ, and thus a function of (θ, t) for every fixed s. Using the classical calculus-of-variations method, this functional derivative can be computed as:
δE
δρ ∣∣∣∣ ρ (θ, t) = Ex∼µ ( p>ρ (t;x)f(Zρ(t;x), θ) ) , (10)
where pρ(·;x), parameterized by x, maps [0, 1]→ Rd, and is a vector solution to the following ODE:
dp>ρ dt = −p>ρ ∫ Rk ∂zf(Zρ, θ)ρ(θ, t) dθ . (11)
1A collection of probability distribution that is continuous in t and has bounded second moment in θ for all t. The definition is to be made rigorous in Def 4.1.
with pρ(t = 1;x) = (g(Zρ(1;x))− y(x))∇g(Zρ(1;x)). In the later sections, to emphasize the s dependence, we use δE(Θ(s))δΘ and δE(ρ(s)) δρ to denote δE δΘ ∣∣ Θ(s;·) and δE δρ ∣∣∣ ρ(·,·,s) respectively. As a summary, to update ρ(θ, t, s) to ρ(θ, t, s + δs) with an infinitesimal δs, we solve (7) for Zρ(t;x), using the given ρ(θ, t, s), and compute pρ using (11). This then allows us to compute δE(ρ(s)) δρ (θ, t) which, in turn, yields ρ(θ, t, s + δs) from (9). In (11), ∂zf is a d × d matrix that stands for the Jacobian of f with respect to its z argument.

heading: 3 RELATED WORK AND CONTRIBUTION
text: There is a vast literature addressing the overparameterization of DNN. Many perspectives have been taken to justify the success of the application of the first order (gradient descent) optimization methods, in this overparameterized regime. We briefly review related works, and identify our contribution.
The earliest approach to understanding overparametrization was landscape analysis, in which the countours of the nonconvex objective function were studied to find which properties make it possible for a first order method to converge to the optimizer. Different NN structures are then analyzed to see which have these properties (Jin et al., 2017; Ge et al., 2015; Du et al., 2017; Ge et al., 2018; Nguyen & Hein, 2018; Du & Lee, 2018; Soltanolkotabi et al., 2019; Nguyen & Hein, 2017; Kawaguchi, 2016; Yun et al., 2018). This approach naturally limits the types of DNN that can be “explained,” since most DNN structures do not satisfy the required properties.
Another approach taken in the literature is related to the Neural Tangent Kernel (NTK) regime, which is the regime in which the nonlinear problem is reduced to a nearly linear model due to the confinement of the iterates to a small region around the initial values. Insensitivity of the so-called Gram matrix is evaluated in the limit of the number of weights (Allen-Zhu et al., 2019; Du et al., 2019a; Zhang et al., 2019; Chatterji et al., 2021; Du et al., 2019b; Jacot et al., 2018; Liu et al., 2020; Frei et al., 2019). The argument is that zero-loss solutions are close to every point in the space, and one can find an optimal point within a small region of the initial guess. The NTK arguments are shown to work well in several real application problems, such as the classification problem (Li & Liang, 2018; Zou et al., 2019). However, as pointed out by (Ba et al., 2020; Wei et al., 2019; Fang et al., 2019), NTK approximately views nonlinear DNN as a linear kernel model, a rather limited description, so the estimates obtained through NTK might not be sharp. Indeed, the empirical observation in (Allen-Zhu & Li, 2019; Arora et al., 2019) have suggested that the kernel models are not as general as NN, and certain (nonlinear) features of NN are not captured.
Finally, there is the mean-field limit perspective that we adopt in this paper. The term “mean-field" indicates that in a system with a large ensemble of particles, the field formed by averaging across all samples exerts a force on each sample. Instead of tracing the trajectory of each sample, one can characterize the evolution of the full distribution function that represents the field. This idea originated in statistical physics, and is made rigorous under the framework of kinetic theory. In training an overparametrized ResNet context, a large number of weights evolve to decrease the cost function. In the mean-field limit, the training process evolves the distribution function of these weights. A significant advantage of the mean-field approach is that once we derive a formula for the gradient flow, standard PDE techniques can be adopted to describe the convergence behavior. This approach was taken in (Araújo et al., 2019; Fang et al., 2019; Nguyen, 2019; Du et al., 2019a; Chatterji et al., 2021; Chizat & Bach, 2018; Mei et al., 2018; Wojtowytsch, 2020; Lu et al., 2020; Sirignano & Spiliopoulos, 2021; 2020). The case of a single hidden layer NN in the regime as M →∞ is studied by Chizat & Bach (2018); Mei et al. (2018); Wojtowytsch (2020), who justified the mean-field approach and demonstrated convergence of the gradient flow process to a zero objective. In the multi-layer case, Lu et al. (2020) showed the convergence of a PDE that can be viewed as a modified version of the true gradient flow, hinting at convergence of the real mean-field limit. Nguyen & Pham (2021) also gave the global convergence of the mean-field limit of DNN for a certain class of NN structures, but their work excludes such important practical NN structures as ResNet. The work most closely related to ours is (Ding et al., 2021), but this paper makes technical assumptions on ρ∞ and f that restrict the usefulness of the results, as we discuss below following the statement of Theorem 3.1.
We note that in certain parameter regimes, the mean-field and NTK perspectives can sometimes be unified; see (Chen et al., 2020).
We follow the roadmap of Chizat & Bach (2018); Ding et al. (2021), which shows that the PDE (9) achieves the global minimum for which E(ρ(θ, t, s =∞)) = 0, and that the gradient flow in the discrete setting (3) can be closely approximated by the PDE, so that E(ΘL,M (s)) ≈ E(ρ(·, ·, s)). These two results together show that E(ΘL,M (s)) ≈ 0 for pseudo-time s sufficiently large. Specifically, the two main tasks of the paper are as follows.
– Task 1: We need to give a rigorous proof of the continuous and mean-field limit. This will be stated in Theorem 5.1, to justify that for every fixed s < ∞, when M,L → ∞, E(ΘL,M (s)) ≈ E(Θ(s; ·)) ≈ E(ρ(·, ·, s)). The dependence of these approximations on L and M are made precise. – Task 2: We need to demonstrate the convergence to global minimum. This is stated in Theorem 6.1 and 6.2, for two different cases. In both theorems, we obtain the global convergence for the gradient flow, assuming certain homogeneity and the Sard-type regularity for f . A weak assumption of the initialization of ρini is also imposed.
By combining these two, we obtain the main result of the paper.
Theorem 3.1 Let the conditions in Theorem 5.1 and 6.1 (or 6.2) hold. Then for any positive and η, there exist positive constants C0 depending on ρini(θ, t), and C depending on ρini(θ, t), s such that when
s > C0(ρini(θ, t), ) , M > C(ρini(θ, t), s)
2η , L >
C(ρini(θ, t), s)
,
we have P (|E(ΘL,M (s))| ≤ ) ≥ 1− η , where E is defined in (2) and ΘL,M solves (3).
This theorem gives quantitative bounds for M and L. The number of weights required to reduce the cost function below is O(ML) = O(1/ 3). The theorem also suggests that L and M are independent parameters.
The results resonate with those obtained in (Chizat & Bach, 2018) for the 2-layer NN, and extend those in (Ding et al., 2021) greatly. Specifically, compared with the results in (Chizat & Bach, 2018), where ρ(θ, s) follows a typical gradient flow in the probability space on θ in time s (Ambrosio et al., 2008), we have, at each training time s, a “list” of probability measures ρ(θ, t) on θ, for all t. The members of this list are coupled, flowing together in s in the descending direction of the cost function E. New analytical estimates are developed to deal with this non-traditional gradient flow.
Ding et al. (2021) takes a similar approach to ours, but their assumptions on the support of ρ(θ, t, s = ∞) are quite strong: The limiting probability measure ρ(θ, t, s = ∞) is assumed to have the full support over θ. The assumption greatly reduces the technical difficulty of the proof, but it is impractical and hard to justify, thus preventing the results from being of practical use. In the current paper, this support condition is replaced by the well-accepted homogeneity condition adopted by (Chizat & Bach, 2018). As a consequence, the structure of the gradient flow must be examined closely to demonstrate convergence, requiring considerable technical complications. Lu et al. (2020) also investigates gradient flow for training multi-layer neural network, but the gradient flow structure is modified for mathematical convenience. All blocks are integrated together, making ρ a probability measure over the full (θ, t)-space. This design is inconsistent with the structure of the ResNet design that we investigate in this paper.

heading: 4 NOTATIONS, ASSUMPTIONS, AND DEFINITIONS
text: Throughout the paper we denote the collection of probability distributions with bounded second moments as P2(Rk), that is, P2(Rk) = {ρ : ∫ Rk |θ|
2 dρ(θ) < ∞}. We assume certain regularity properties for the activation function f , the measuring function g, the data y, and the input measure µ, as follows.
Assumption 4.1 (Assumptions on f ) Let f : Rd × Rk → Rd be a C2 function.
1. (linear growth) For all x ∈ Rd, θ ∈ Rk, there is a constant C1 such that |f | ≤ C1(|θ|2 + 1)(|x|+ 1) . (12)
2. (locally Lipschitz) For all r > 0, and |x| < r, θ ∈ Rk, we have for C2(r) monotonically increasing with respect to r that the following bounds hold:
|∂xf | ≤ C2(r)(|θ|2 + 1), |∂θf | ≤ C2(r)(|θ|+ 1) . (13)
3. (local smoothness) There exists k1 ∈ (0, k] with the following property: Denoting θ = (θ[1], θ[2]), r = max{|x|, |θ[1]|}, where θ[1] ∈ Rk1 , θ[2] ∈ Rk−k1 , we have for C3(r) monotonically increasing with respect to r that the following bounds hold:∣∣∣∂ix∂jθf(x, θ)∣∣∣ ≤ C3(r), i+ j = 2, i, j ≥ 0 . (14) When k1 < k, we have in addition that
max{|∂xf |, |f |} ≤ C3(r) ( |θ[2]|+ 1 ) , ∣∣∂θ[1]f ∣∣ ≤ C3(|x|) (|θ[1]|+ 1) . (15)
4. (universal kernel) The function set { h ∣∣h = ∫Rk f(x, θ) dρ(θ), ρ ∈ P2(Rk)} is dense in
C ( |x| < R;Rd ) for all R > 0.
Assumption 4.2 (Assumptions on data) Let g, y : Rd → R be C2 functions, and let µ be the probability distribution of x. We assume the following.
5. µ(x) is compactly supported, meaning that there is Rµ > 0 such that the support of µ is within a ball of size Rµ around the origin, that is, supp(µ) ⊂ BRµ(~0).
6. y(x) ∈ L∞loc(Rd), that is, sup|x|≤R |y(x)| <∞.
7. g(x) and ∇g(x) are Lipschitz continuous. Moreover, |∇g(x)| has a positive lower bound, that is, infx∈Rd |∇g(x)| > 0.
We note that Assumption 4.1 admits many commonly used activation functions (E et al., 2020). One example of a function satisfying this assumption is f(x, θ) = f(x, θ[1], θ[2], θ[3]) = θ[3]σ(θ[1]x+θ[2]), where θ[1] ∈ Rd×d, θ[2] ∈ Rd, θ[3] ∈ R and σ is a component-wise regularized ReLU activation function, see Remark H.2.
We now build the metric on the function space for our solutions. Note that the solution ρ(θ, t, s) to (9) is expected to be a continuous function in (t, s), and a distribution of θ for each (t, s). For this non-standard probability space, we first introduce the following metric.
Definition 4.1 2 C([0, 1];P2) is a collection of continuous paths of probability distribution ν(θ, t) (θ ∈ Rk, t ∈ [0, 1]) where 1. ν(·, t) ∈ P2(Rk) for every fixed t ∈ [0, 1]; 2. For any t0 ∈ [0, 1], limt→t0 W2 (ν(·, t), ν(·, t0)) = 0, where W2 is the classical Wasserstein-2 distance. The space C([0, 1];P2) is equipped with the following metric d1 (ν1, ν2) = suptW2(ν1(·, t), ν2(·, t)). Accordingly, C([0,∞); C([0, 1];P2)) is a collection of continuous paths of probability distribution ν(θ, t, s) (with θ ∈ Rk, t ∈ [0, 1], s ∈ [0,∞)), where 1. ν(·, ·, s) ∈ C([0, 1];P2) for every fixed s ∈ [0,∞). 2. For any s0 ∈ [0,∞), lims→s0 d1 (ν(·, ·, s), ν(·, ·, s0)) = 0 (where d1 is defined above). The metric in C([0,∞); C([0, 1];P2)) is defined by d2 (ν1, ν2) = supt,sW2(ν1(·, t, s), ν2(·, t, s)).
Since P2 is complete in W2 distance, C([0, 1];P2) and C([0,∞); C([0, 1];P2)) are complete metric spaces under d1 and d2 respectively also. To give a rigorous justification of the mean-field limit, we use the particle representation of ρ(θ, t, s). Thus, at least we need to assume that we can find a stochastic process that is drawn from the initial condition ρini(θ, t). We call such initial conditions admissible.
Definition 4.2 We call a continuous path of probability distribution ν(θ, t) ∈ C([0, 1];P2) admissible if it has a particle representation, namely there exists a continuous stochastic process θ(t) : [0, 1]→ Rk and r > 0 such that for any t0 ∈ [0, 1], we have
θ(t0) ∼ ν(θ, t0) , lim t→t0
E ( |θ(t)− θ(t0)|2 ) = 0, |θ(t0)| < r . (16)
2C([0, T ];A) is defined to be the set of functions f(a, t) such that for any t ∈ [0, T ], f(·, t) ∈ A and f(·, t) is continuous with respect to t under the metric defined on A. In Definition 4.1, we set T = 1 and A = P2, where the natural metric on A is W2.
Furthermore, ν(θ, t) is called limit-admissible if its M -averaged trajectory is bounded and Lipschitz with high probability. (See the rigorous definition in Definition E.1.)
We note that without the dependence on t, probability distributions are “admissible", in the sense that one can draw a sample from a given distribution. In Appendix A, we show that if the initial condition ρini(θ, t) is admissible, (9) has a unique solution ρ(θ, t, s) that is admissible for each s.
The global convergence result depends on Sard-type regularity, defined as follows.
Definition 4.3 Given a metric space Θ, a differentiable function h : Θ→ R, and a subset Θ̃ ⊂ Θ, we say h satisfies Sard-type regularity in Θ̃ if the set of regular values3 of h|Θ̃ is dense in its range, where h|Θ̃ : Θ̃→ R is a confinement of h in Θ̃.
Remark 4.1 This regularity assumption is not a common one; we have adopted it from Chizat & Bach (2018). This property is essentially that most of the points in the range of h lie in an open set and h is locally monotonic. The assumption is rather mild and can be satisfied by most commonly seen regular functions, unless the function oscillates wildly.

heading: 5 MEAN-FIELD AND CONTINUOUS LIMIT
text: In this section, we focus on the justification of mean-field and continuous-limit result. This is to prove that E(ρ(·, ·, s)), E(Θ(s; ·)), and E(ΘL,M (s)) are asymptotically close to each other for every s. In the next section, we prove convergence of E(ρ(·, ·, s)) as s→∞. To show the asymptotic equivalence of the three quantities, we need to compare (3), (6), and (9), and take the measurement in E according to (2), (5) and (8).
Theorem 5.1 Suppose that Assumptions 4.1 and 4.2 are satisfied. Assume that ρini(θ, t) is limitadmissible and suppθ(ρini(θ, t)) ⊂ {θ||θ[1]| ≤ R} with some R > 0 for all t ∈ [0, 1]. Let {θm(0; t)}Mm=1 in (6) be i.i.d. drawn from ρini(θ, t). Let
• ΘL,M (s) = {θl,m(s)} be the solution to (3) with initial condition θl,m(s = 0) = θm ( 0; lL ) ;
• θm(s; t) be the solution to (6) with the initial condition θm(0; t);
• ρ(θ, t, s) be the solution to (9) with initial condition ρini(θ, t).
Then for any positive , η, and S, there exists a constant C > 0 that depends on ρini(θ, t) and S such that when
M > C(ρini(θ, t), S)
2η , L >
C(ρini(θ, t), S)
, s < S ,
we have:
min{P (|E(ΘL,M (s))− E(Θ(s; ·))| ≤ /2) ,P (|E(Θ(s; ·))− E(ρ(·, ·, s))| ≤ /2)} ≥ 1− 12η .
It follows that P (|E(ΘL,M (s))− E(ρ(·, ·, s))| ≤ ) ≥ 1− η , ∀s < S . Here E(ΘL,M (s)), E(Θ(s; ·)), and E(ρ(·, ·, s)) are defined in (2), (5), and (8), respectively.
The proof of this result appears in Appendix E. This theorem suggests that for every fixed S > 0, the gradient descent of ΘL,M is approximately the same as the gradient flow of ρ(θ, t), in the sense that the two costs are close to each other with high probability, when L and M are sufficiently large. The size of the ResNet depends negative-algebraically on (the desired accuracy) and η (the confidence of success). The result translates the evolution (gradient descent) of ΘL,M to the evolution (gradient flow) of ρ(θ, t), and thus matches the zero-loss property of the parameter configuration of a finite sized ResNet to its limiting PDE, whose analysis can be performed with standard PDE tools.
3For a function h : Θ̃→ R, a regular value is a real number α in the range of h such that h−1(α) is included in an open set where h is differentiable and where dh does not vanish.
The proof of Theorem 5.1 divides naturally into two components. We show that for all s < S, E(ρ(·, ·, s)) ≈ E(Θ(s; ·)) and E(Θ(s; ·)) ≈ E(ΘL,M (s)) with high probability. The former is obtained from mean-field limit theory, justifying that the particle trajectory θm(t, s) follows ρ(θ, t, s) for all t in pseudo-time s ∈ [0, S]. The latter makes use of continuity in t and traces the differences between θm( lL ) and θl,m. These two components of the proof are summarized in Theorems E.1 and E.2, respectively. According to the formula of the Fréchet derivatives (10) and (11), the estimates in these theorems naturally route through the boundedness of pρ, pΘ, pΘL,M , and similarly Zρ,Θ,ΘL,M . It is technically demanding to derive these bounds, but they are not surprising. We dedicate a large portion of the appendix to addressing the well-posedness of these systems. See Appendices A-D, where we show these equations have unique solutions with proper initial conditions, along with the required bounds. Naturally, these estimates depend on regularity of f , g, y, and µ.
To gain some intuition for the equivalence between (6) and (9), we test them on the same smooth function h(θ). To test (9), we multiply both sides by h and perform integration by parts to obtain dds ∫ Rk hdρ(θ) = − ∫ Rk ∇θh∇θ δE(ρ(s)) δρ dρ. This is to say d dsE(h) = E ( ∇θh∇θ δE(ρ(s))δρ ) .
Testing h on (6) also gives the same formula. Supposing that ρ = 1M ∑M m=1 δθm , we have
d dsE(h) = 1 M ∑M m=1∇θh(θm) d dsθm = − ∑M m=1∇θh(θm) δE δθm . The right hand side is also
E ( ∇θh∇θ δE(ρ(s))δρ ) if and only if M δE(Θ(s;·))δθm = ∇θ δE(ρ) δρ (θm, t). This will be shown to hold true in Appendix F.

heading: 6 CONVERGENCE TO GLOBAL MINIMIZER
text: After translating the study of ΘL,M to the study of ρ(θ, t), this section presents results on when and how E(ρ) converges to zero loss by examining the conditions for the global convergence of (9). We first identify the property of global minimum.
Proposition 6.1 Suppose that ρ ∈ C([0, 1];P2) has E(ρ) > 0. Then for any t0 ∈ [0, 1], there exists a measure ν(θ) of Rk such that ∫ Rk dν(θ) = 0 and∫
Rk
δE
δρ ∣∣∣∣ ρ (θ, t0) dν(θ) < 0 . (17)
See Appendix H.1 for the proof of this result. At the stationary point of the cost function, δEδρ (θ, t0) = 0, then there is no ν satisfying (17), so E(ρ) is necessarily trivial. Our task now becomes to check under what conditions on f we can have δEδρ becoming 0 as s→∞. We give two possibilities, both requiring a separation initialization and certain homogeneities. In the first case, we require f to be 2-homogeneous, and the result is collected in Section 6.1. In the second case, we require f to be partially 1-homogeneous; see Section 6.2.

heading: 6.1 THE 2-HOMOGENEOUS CASE
text: The results in this section are obtained under the following assumption on f . (Functions can be designed to satisfy this assumption easily; see Remark H.1.)
Assumption 6.1 The function f(x, θ) : Rd × Rk is 2-homogeneous, meaning that f(x, λθ) = λ2f(x, θ) for all (x, θ, λ) ∈ Rd × Rk × R.
Theorem 6.1 Let Assumption 6.1 and conditions of Theorem 5.1 hold true with k1 = k. Let ρ∞(θ, t) be the long-time limit of (9), that is, ρ(θ, t, s) converges to ρ∞(θ, t) in C([0, 1];P2) as s→∞. Then E(ρ∞) = 0 if the following hold:
• Separation initialization: There exists t0 ∈ [0, 1] such that ρini(θ, t0) separates4 the spheres raSk−1 and rbSk−1, for some 0 < ra < rb.
4We say that a set C separates the sets A and B if any continuous path in with endpoints in A and B intersects C.
• Sard-type regularity: δEδρ ∣∣∣ ρ∞ (θ, t0) satisfies the Sard-type regularity condition in Sk−1.
A proof appears in Appendix H.2. A corollary of this theorem, when combined with Theorem 5.1 with k1 = k, gives Theorem 3.1, the main result of our paper.

heading: 6.2 THE PARTIALLY 1-HOMOGENEOUS CASE
text: The following assumption is used in this section. (Functions that satisfy this assumption include regularized ReLU; see Remark H.2.)
Assumption 6.2 Let θ = (θ[1], θ[2]) with θ[1] ∈ R and θ[2] ∈ Rk−1:
1. (partially 1-homogeneous in θ) f can be written as f(x, θ) = f(x, θ[1], θ[2]) = θ[1]f̂(x, θ[2]),
2. (locally bounded and smooth) For any r > 0, f̂(x, θ[2]) is bounded and Lipschitz with Lipschitz continuous differential for (x, θ[2]) ∈ Br(~0)× Rk−1.
When Assumption 6.2 holds true, Assumption 4.1 part 3 can be satisfied with k1 = 1. Then, we introduce the main result in this section.:
Theorem 6.2 Let Assumption 6.2 and conditions of Theorem 5.1 hold true with k1 = 1. Let ρ∞(θ, t) be the limit of (9) as s→∞. Then E(ρ∞) = 0 if the following hold:
• Separation initialization: There exists t0 ∈ [0, 1] such that ρini(θ[1], θ[2], t0) separates the spheres {−r0} × Rk−1 and {r0} × Rk−1 for some r0 > 0, where θ[1], θ[2] are defined by Assumption 4.1 item 3 with k1 = 1.
• Sard-type regularity: δEδρ ∣∣∣ ρ∞
((1, θ[2]), t0) : Rk−1 → R satisfies the Sard-type regularity condition.
• For any ρ̃ ∈ C([0, 1],P2), define Hr,ρ̃ ( θ̃[2] ) = δE(ρ)δρ ∣∣∣ ρ̃ ( 1, rθ̃[2], t0 ) where θ[2] = rθ̃[2]
with r = |θ2| and θ̃[2] ∈ Sk−2. Suppose that Hr,ρ̃ converges in C1(Sk−2) as r → ∞ to a function H∞,ρ̃. Furthermore, assume that H∞,ρ∞ satisfies the Sard-type regularity condition in Sk−2 and that the intersection of regular values of H∞,ρ∞ and δE(ρ) δρ ∣∣∣ ρ∞ (1, θ[2], t0) is also dense in the intersection of their range.
The proof is found in Appendix H.3. This result, combined with Theorem 5.1 in the case of k1 = 1, gives the main result in Theorem 3.1. The assumptions in Theorem 6.2 are rather technical and seemingly tedious. However, we note that only the first two assumptions — 1-homogeneous and the separation assumption — are crucial. The third and fourth assumptions concern the regularity of the Fréchet derivative of ρ; they are rather mild and serve to exclude wildly oscillating functions; see Remark 4.1. Most commonly seen functions are regular enough that these two assumptions are satisfied.

heading: 7 ETHICS STATEMENT
text: This work does not present any foreseeable societal consequence.

heading: 8 REPRODUCIBILITY STATEMENT
text: The notations, assumptions and definitions are clarified in Section 4. And all proofs appear in the Appendix.

heading: A WELL-POSEDNESS RESULT
text: In this section, we show the well-posedness of ODEs (4), (7) and gradient flows (3), (6), and (9).
Theorem A.1 The following claims hold.
– Well-posedness of ODE:
• If {θm(t)}Mm=1 is continuous, then (4) has a unique C1solution. • If ρ ∈ C([0, 1];P2), then (7) has a unique C1 solution.
– Well-posedness of gradient flow:
• (3) has a unique solution. • If {θm(0; t)}Mm=1 is continuous, then (6) has a unique solution {θm(s; t)}Mm=1 that is continuous
in (s, t). • If ρini(θ, t) is admissible and suppθ(ρini(θ, t)) ⊂ {θ||θ[1]| ≤ R} with some R > 0 for all t ∈
[0, 1], then (9) has a unique solution ρ(θ, t, s) in C([0,∞); C([0, 1];P2)) with initial condition ρini(θ, t). Furthermore, for each s, ρ(θ, t, s) is admissible.
Note that the well-posedness of (4) and (6) are direct corollaries of that of (7), (3), and (9) (the corresponding continuous versions), according to Remark A.1, A.2. Thus, we merely prove wellposedness on the continuous level, in Theorems A.2, A.3, and A.4. Specifically,
• Theorem A.2 (Appendix A.1) shows the well-posedness of the dynamical system for z; • Theorems A.3 and A.4 (Appendix A.2) justify the well-posedness of the gradient flow of
the parameter configuration.

heading: A.1 WELL-POSEDNESS OF THE OID (7)
text: As L andM approach∞, z satisfies the ordinary-integral equation (7). We justify that this differential equation is well-posed, in the sense that the solution is unique and stable.
Theorem A.2 Suppose that Assumption 4.1 holds with and that x is in the support of µ, then (7) has a unique C1 solution. Specifically, for ρ1, ρ2 ∈ C([0, 1];P2), we have |Zρ1(t;x)| ≤ C(L1) , (18) and for all t ∈ [0, 1]: |Zρ1(t;x)− Zρ2(t;x)| ≤ C(L1,L2)d1(ρ1, ρ2) , (19) where Li are the second moments of ρi, that is,
Li = ∫ 1
0 ∫ Rk |θ|2dρi(θ, t) dt, i = 1, 2. (20)
We leave the proof to Appendix B. Besides the well-posedness result, the theorem also suggests that a small perturbation to ρ is reflected linearly in Zρ, the solution to (7). This means that a small perturbation in the parameterization of the ResNet leads to only a small perturbation to the ResNet output.
Remark A.1 Although we do not directly show the well-posedness of (4), it follow immediately from Theorem A.2. One way to make this connection is to reformulate the discrete probability distribution as
ρdis(θ, t) = 1
M M∑ m=1 δθm(t)(θ) ,
where Θ(t) = {θm(t)}Mm=1 is the list of trajectories. Since θm(t) is continuous in t, we have ρdis(θ, t) ∈ C([0, 1];P2). Since
1
M M∑ m=1 f(z(t;x), θm(t)) = ∫ Rk f(z(t;x), θ) dρdis(θ, t) ,
using Theorem A.2, (4) has a unique C1 solution when Θ(t) is continuous.

heading: A.2 WELL-POSEDNESS OF THE GRADIENT FLOW
text: The gradient flow of the parameterization is also well-posed, both in the discrete setting and the continuous mean-field limit. In the discrete setting, we have the following result.
Theorem A.3 (3) has a unique solution.
Further, (9) characterizes the dynamics of the continuous mean-field limit of the parameter configuration, and is also well-posed.
Theorem A.4 If ρini(θ, t) is admissible and suppθ(ρini(θ, t)) ⊂ {θ||θ[1]| ≤ R} with someR > 0 for all t ∈ [0, 1]. Then (9) has a unique solution ρ(θ, t, s) in C([0,∞); C([0, 1];P2)) that is admissible for each s. Further, we have
dE(ρ(·, ·, s)) ds ≤ 0 . (21)
The proofs of these two theorems can be found in Appendix D.
Remark A.2 Using the same argument as in Remark A.1, calling
ρdis(θ, t, s) = 1
M M∑ m=1 δθm(s;t)(θ) , (22)
the well-posedness of (6) follows immediately from Theorem A.4. Furthermore, according to the definition (5) and (8), we have
E ( ρdis(·, ·, s) ) = E (Θ(s; ·)) .
As a consequence, if Θ(s; t) satisfies (6), then ρdis satisfies (9), and vice versa. The well-posedness result in Theorem A.4 for (9) then can be extended to justify well-posedness of (6).

heading: B PROOF OF THEOREM A.2
text: This section contains the proof of Theorem A.2. We rewrite (7) as follows:
dZρ(t;x)
dt = F (Zρ, t) , ∀t ∈ [0, 1] with z(0;x) = x , (23)
where for a given ρ ∈ C([0, 1];P2) we use the notation:
F (z, t) = ∫ Rk f(z, θ) dρ(θ, t) .
The proof of Theorem A.2 relies on the classical Lipschitz condition for the well-posedness of an ODE.
Proof [Proof of Theorem A.2] Since ρ ∈ C([0, 1];P2), we have a constant C such that
sup 0≤t≤1 ∫ Rk |θ|2 dρ(θ, t) < C <∞ .
For any t ∈ [0, 1], using (12) from Assumption 4.1 equation, we have |F (z, t)| ≤ ∫ Rk |f(z1, θ)| dρ(θ, t) ≤ C1(|z|+ 1) ∫ Rk (|θ|2 + 1) dρ(θ, t) . (24)
To show the boundedness result (18), we multiply (23) by Zρ1(t;x) and use (24) to obtain
d|Zρ1(t;x)|2
dt ≤2C1
( |Zρ1 |2 + |Zρ1 | ) ∫ Rk (|θ|2 + 1) dρ1(θ, t)
≤4C1 ∫ Rk (|θ|2 + 1) dρ1(θ, t) ( |Zρ1(t;x)|2 + 1 ) .
Using Grönwall’s inequality, we have |Zρ1(t;x)| ≤ exp ( 2C1 (∫ 1 0 ∫ Rk |θ|2 dρ1 dt+ 1 )) (|x|+ 1)
≤ exp (2C1 (L1 + 1)) (|x|+ 1) , where L1 = ∫ 1
0 ∫ Rk |θ|
2 dρ1 dt. Since x ∈ suppµ (so that |x| < R), we have (18). This gives us an a-priori estimate of (23).
Next, using (13) from Assumption 4.1, we have |f(z1, θ)− f(z2, θ)| ≤ C2(|z1|+ |z2|) ( |θ|2 + 1 ) |z1 − z2| . (25)
Then, using boundedness of the second moment of θ, we have |F (z1, t)− F (z2, t)| ≤ ∣∣∣∣∫
Rk (f(z1, θ)− f(z2, θ)) dρ(θ, t) ∣∣∣∣ ≤ C2(|z1|+ |z2|) ∫ Rk (|θ|2 + 1) dρ(θ, t)|z1 − z2| < C2(|z1|+ |z2|)(C + 1)|z1 − z2| ,
which implies that F (z, t) is locally Lipschitz in z for all t ∈ [0, 1]. Combining this with the a-priori estimate in (18), classical ODE theory implies that (23) has a unique C1 solution. To prove the stability result (19), we define Zρi as in (23) parameterized by ρi ∈ C([0, 1];P2), and denote
∆(t;x) = Zρ1(t;x)− Zρ2(t;x) .
Then by subtracting the two equations, we obtain
d|∆(t;x)|2
dt
= 2 〈 ∆(t;x), ∫ Rk f(Zρ1(t;x), θ) dρ1(θ, t)− ∫ Rk f(Zρ2(t;x), θ) dρ2(θ, t) 〉 = 2 〈 ∆(t;x),
∫ Rk f(Zρ1(t;x), θ) dρ1(θ, t)− ∫ Rk f(Zρ2(t;x), θ) dρ1(θ, t)︸ ︷︷ ︸
(I)
〉
+ 2 〈 ∆(t;x), ∫ Rk f(Zρ2(t;x), θ) dρ1(θ, t)− ∫ Rk f(Zρ2(t;x), θ) dρ2(θ, t)︸ ︷︷ ︸
(II)
〉 .
(26)
We now bound (I) and (II). For (I), we have using (13) and (25) that |(I)| ≤2C2 (|Zρ1(t;x)|+ |Zρ2(t;x)|) |∆(t;x)| ∫ Rk (|θ|2 + 1) dρ1(θ, t)
≤C(L1,L2)|∆(t;x)| ∫ Rk (|θ|2 + 1) dρ1(θ, t) , (27)
where the second inequality comes from (18). For (II), we denote the particle representation θ1(t) ∼ ρ1(θ, t) and θ2(t) ∼ ρ2(θ, t) such that ( E ( |θ1 − θ2|2 ))1/2 = W2(ρ1(·, t), ρ2(·, t)). Then
|(II)| ≤ E (|f(Zρ2(t;x), θ1)− f(Zρ2(t;x), θ2)|) ≤ C2(Zρ2(t;x))E ((|θ1|+ |θ2|+ 1)|θ1 − θ2|) ≤ C(L2)E ((|θ1|+ |θ2|+ 1)|θ1 − θ2|)
≤ C(L2) ( E ( |θ1|2 + |θ2|2 + 1 ))1/2 (E (|θ1 − θ2|2))1/2 ≤ C(L2) ( E ( |θ1|2 + |θ2|2 + 1 ))1/2 W2(ρ1(·, t), ρ2(·, t))
≤ C(L2) (∫
Rk |θ|2dρ1(θ, t) + ∫ Rk |θ|2dρ2(θ, t) + 1 )1/2 d1(ρ1, ρ2) ,
(28)
where we use mean-value theorem and (13) in the first inequality to obtain |f(Zρ2(t;x), θ1)− f(Zρ2(t;x), θ2)| ≤|∂θf(Zρ2(t;x), θ1 + λθ2)||θ1 − θ2| ≤C2(Zρ2(t;x))(|θ1|+ |θ2|+ 1)|θ1 − θ2|, for some λ ∈ [0, 1]. We use (18) in the second inequality of (28). Plugging (27)-(28) into (26) and using Hölder’s inequality, we obtain that
d|∆(t;x)|2
dt ≤C(L1,L2)|∆(t;x)|2 ∫ Rk |θ|2 dρ1(θ, t)
+ C(L2)|∆(t;x)| (∫
Rk |θ|2dρ1(θ, t) + ∫ Rk |θ|2dρ2(θ, t) + 1 )1/2 d1(ρ1, ρ2)
≤C(L1,L2) (∫
Rk |θ|2dρ1(θ, t) + ∫ Rk |θ|2dρ2(θ, t) + 1 )( |∆(t;x)|2 + d21(ρ1, ρ2) ) ,
where we used Young’s inequality in the last line. Since |∆(0;x)| = 0, we complete the proof of (19) using Grönwall’s inequality.

heading: C A-PRIORI ESTIMATION OF THE COST FUNCTION
text: Some a-priori estimates are necessary in the proof for the main theorems. We first consider the case when f satisfies only Assumption 4.1 with 0 < k1 ≤ k. (Better a-priori estimates can be obtained when f also satisfies the homogeneity properties of Sections 6.1 or 6.2.)
C.1 A-PRIORI ESTIMATE FOR GENERAL f
According to (10), the Fréchet derivative can be computed, similarly to (Lu et al., 2020), as follows:
δE(ρ)
δρ (θ, t) = Ex∼µ
( p>ρ (t;x)f(Zρ(t;x), θ) ) , (29)
where pρ(t;x) is the solution to the following ODE: ∂p>ρ ∂t = −p>ρ ∫ Rk ∂zf(Zρ(t;x), θ) dρ(θ, t) ,
pρ(1;x) = (g(Zρ(1;x))− y(x))∇g(Zρ(1;x)) . (30)
We now show that pρ is Lipschitz continuous with respect to ρ.
Lemma C.1 Suppose that x is in the support of µ. Suppose that ρ1, ρ2 ∈ C([0, 1];P2) and pρ1 , pρ2 are the corresponding solutions of (30). Denote L1 and L2 as in (20) and
R = min r
{ r ∣∣supp(ρ1) ∪ supp(ρ2) ⊂ {θ∣∣∣∣θ[1]∣∣ < r}} .
Then the following two bounds are satisfied:
|pρ1(t;x)| ≤ C(L1) , (31)
and |pρ1(t;x)− pρ2(t;x)| ≤ C(R,L) d1(ρ1, ρ2) , (32)
where L = max{L1,L2}.
Proof From (13) in Assumption 4.1,∣∣∣∣∫ Rk ∂zf(Zρ1(t;x), θ) dρ1(θ, t) ∣∣∣∣ ≤C(Zρ1(t;x))∫ Rk (|θ|2 + 1) dρ1(θ, t)
≤C(L1) ∫ Rk (|θ|2 + 1) dρ1(θ, t) , (33)
where we use (18) in the second inequality. It follows from the initial condition of (30) that
pρ1(1;x) ≤ C(|Zρ1(1, x)|+ 1) ≤ C(L1) ,
where we use Assumption 4.1 in the first inequality and (18) in the second inequality. Noting that (30) is a linear equation, (31) follows naturally when we combine (33) with the inequality above.
To prove (32), we define ∆(t;x) = pρ1(t;x)− pρ2(t;x) . For t = 1, with x ∈ suppµ, we have
|∆(1;x)| = |pρ1(1;x)− pρ2(1;x)| = |(g(Zρ1(1;x))− y(x))∇g(Zρ1(1;x))− (g(Zρ2(1;x))− y(x))∇g(Zρ2(1;x))| ≤ C(L)|Zρ1(1;x)− Zρ2(1;x)| ≤ C(L)d1(ρ1, ρ2) , (34)
where we use Assumption 4.1, (18), and |x| < Rµ in the first inequality and (19) in the second inequality. The following ODE is satisfied by ∆:
∂∆>(t;x)
∂t = −∆>(t;x) ∫ Rk ∂zf(Zρ1(t;x), θ) dρ1(θ, t) + p > ρ2(t;x)Dρ1,ρ2(t;x) , (35)
where
Dρ1,ρ2(t;x) = ∫ Rk ∂zf(Zρ2(t;x), θ) dρ2(θ, t)− ∫ Rk ∂zf(Zρ1(t;x), θ) dρ1(θ, t) . (36)
To show the boundedness of Dρ1,ρ2(t;x), we follow the same strategy as that for (26). By splitting into two terms, we obtain
|Dρ1,ρ2(t;x)| ≤ ∣∣∣∣∫
Rk ∂zf(Zρ2(t;x), θ) dρ2(θ, t)− ∫ Rk ∂zf(Zρ2(t;x), θ) dρ1(θ, t) ∣∣∣∣︸ ︷︷ ︸ (I)
+ ∣∣∣∣∫ Rk ∂zf(Zρ2(t;x), θ) dρ1(θ, t)− ∫ Rk ∂zf(Zρ1(t;x), θ) dρ1(θ, t) ∣∣∣∣︸ ︷︷ ︸ (II) . (37)
The bound of (II) relies on part 3 of Assumption 4.1: Because the supports of ρ1, ρ2 are contained in{ θ ∣∣∣∣θ[1]∣∣ < R} and Z is bounded by (18), we have |(II)| ≤ C(R,L) |Zρ1(t;x)− Zρ2(t;x)| ≤ C(R,L)d1(ρ1, ρ2) , (38) where we use (19) in the second inequality. To bound (I), we use the particle representation θ1 ∼ ρ1(θ, t) and θ2 ∼ ρ2(θ, t) such that( E ( |θ1 − θ2|2 ))1/2 = W2(ρ1(·, t), ρ2(·, t)). We then have
(I) ≤ E (|∂zf(Zρ2(t;x), θ1)− ∂zf(Zρ2(t;x), θ2)|) ≤ C(R,L)E (|θ1 − θ2|)
≤ C(R,L) ( E ( |θ1 − θ2|2 ))1/2 ≤ C(R,L)d1(ρ1, ρ2),
(39)
where we use the mean-value theorem, part 3 of Assumption 4.1 with |θ1,[1]| < R and |θ2,[1]| < R, (18) in the second inequality. Substituting (38) and (39) into (37), we obtain
|Dρ1,ρ2(t;x)| ≤ C(R,L)d1(ρ1, ρ2) . By substituting this bound into (35) and using (33), we have
d|∆(t;x)|2
dt ≤ C(R,L)
( |∆(t;x)|2 + d21(ρ1, ρ2) ) . (40)
The result (32) follows from the initial condition (34) and Grönwall’s inequality.
The second lemma concerns the continuity of∇θ δE(ρ)δρ .
Lemma C.2 Suppose that ρ, ρ1, ρ2 ∈ C([0, 1];P2). Define
L = max 1≤i≤3 ∫ 1 0 ∫ Rk |θ|2 dρi(θ, t) dt, Lsup = max 1≤i≤3 sup t∈[0,1] ∫ Rk |θ|2 dρi(θ, t) ,
and R = min
r
{ r ∣∣supp(ρ) ∪ supp(ρ1) ∪ supp(ρ2) ⊂ {θ||θ[1]| < r}} .
Then for any (θ, t), (θ1, t1), (θ2, t2) ∈ Rk × [0, 1] and s > 0, the following properties hold.
• Boundedness:∣∣∣∣∇θ δE(ρ)δρ (θ, t) ∣∣∣∣ ≤ C(L)(|θ|+ 1), ∣∣∣∣∂θ[1] δE(ρ)δρ (θ, t) ∣∣∣∣ ≤ C(L)(|θ[1]|+ 1) (41) • Lipschitz continuity in θ and t: There exists Q1 : R2 → R+ that depends increasingly on
both arguments such that∣∣∣∣∇θ δE(ρ)δρ (θ1, t1)−∇θ δE(ρ)δρ (θ2, t2) ∣∣∣∣
≤Q1 ( L,max
i=1,2 (|θi,[1]|)
) |θ1 − θ2|+Q1 ( Lsup,max
i=1,2 (|θi,[1]|)
) (|θ2|+ 1)|t1 − t2| , (42)
• Lipschitz continuity in ρ. There exists Q : R2 → R+ that increasingly depends on both arguments such that∣∣∣∣∇θ δE(ρ1)δρ (θ, t)−∇θ δE(ρ2)δρ (θ, t) ∣∣∣∣ ≤ Q(L, |θ[1]|,R)(1 + |θ|)d1(ρ1, ρ2) , (43) where d1 is defined in Definition 4.1.
Proof To prove the first bound of (41), we restate (29) as follows
∇θ δE(ρ)
δρ (θ, t) = Ex∼µ
( p>ρ (t;x)∂θf(Zρ(t;x), θ) ) ,
from which it follows that∣∣∣∣∇θ δE(ρ)δρ (θ, t) ∣∣∣∣ ≤ Ex∼µ (|∂θf(Zρ(t;x), θ)| |pρ(t;x)|) ≤ C(L)(|θ|+ 1) ,
where we use (13), (18), and (31) in the second inequality. To prove the second bound in (41), we use the bound of ∣∣∂θ[1]f(x, θ)∣∣ according to Assumption 4.1 part 3 to obtain∣∣∣∣∂θ[1] δE(ρ)δρ (θ, t) ∣∣∣∣ ≤ Ex∼µ (∣∣∂θ[1]f(Zρ(t;x), θ)∣∣ |pρ(t;x)|) ≤ C(L)(|θ[1]|+ 1) ,
To prove (42), we assume t1 > t2 without loss of generality, and use the triangle inequality to obtain∣∣∣∣∇θ δE(ρ)δρ (θ1, t1)−∇θ δE(ρ)δρ (θ2, t2) ∣∣∣∣
≤ ∣∣Ex∼µ (p>ρ (t1;x)∂θf(Zρ(t1;x), θ1)− p>ρ (t1;x)∂θf(Zρ(t1;x), θ2))∣∣︸ ︷︷ ︸
(I) + ∣∣Ex∼µ (p>ρ (t1;x)∂θf(Zρ(t1;x), θ2)− p>ρ (t1;x)∂θf(Zρ(t2;x), θ2))∣∣︸ ︷︷ ︸
(II) + ∣∣Ex∼µ (p>ρ (t1;x)∂θf(Zρ(t2;x), θ2)− p>(t2;x)∂θf(Zρ(t2;x), θ2))∣∣︸ ︷︷ ︸
(III)
.
(44)
To bound (I), we use the mean-value theorem, Assumption 4.1 (14), and (18) to obtain
|∂θf(Zρ(t1;x), θ1)− ∂θf(Zρ(t1;x), θ2)| ≤ |∂2θf(Zρ(t1;x), (1− λ)θ1 + λθ2)||θ1 − θ2| ≤ C ( L,max
i=1,2 (|θi,[1]|)
) |θ1 − θ2| .
For (II), we note first that |Zρ(t1;x)− Zρ(t2;x)| ≤ ∣∣∣∣∫ t1 t2 ∫ Rk f(Zρ(t;x), θ) dρ(θ, t) dt ∣∣∣∣ ≤ C(L)
∣∣∣∣∫ t1 t2 ∫ Rk ( |θ|2 + 1 ) dρ(θ, t) dt ∣∣∣∣ ≤ C(L)(Lsup + 1)|t1 − t2| ≤ C(Lsup)|t1 − t2| ,
where we use Assumption 4.1 (12) together with (18) and L ≤ Lsup. This bound implies that (II) ≤ Ex∼µ ( |p>ρ (t1;x)| |∂θf(Zρ(t1;x), θ2)− ∂θf(Zρ(t2;x), θ2)| ) ≤ C(L)Ex∼µ (|∂θf(Zρ(t1;x), θ2)− ∂θf(Zρ(t2;x), θ2)|) ≤ C(Lsup, |θ2,[1]|)Ex∼µ (|Zρ(t1;x)− Zρ(t2;x)|) ≤ C(Lsup, |θ2,[1]|)|t1 − t2| ,
where we use (31) in the first inequality, Assumption 4.1 (14) for f , and (18) in the second inequality.
To bound term (III), we again use boundedness of Zρ, pρ and the Lipschitz condition of f to obtain |pρ(t1;x)− pρ(t2;x)| ≤ ∣∣∣∣∫ t1 t2 ∫ Rk p>ρ (t;x)∂zf(Zρ(t;x), θ) dρ(θ, t) dt ∣∣∣∣ ≤ C(L)
∣∣∣∣∫ t1 t2 ∫ Rk (|θ|2 + 1) dρ(θ, t) dt ∣∣∣∣
≤ C(L)(Lsup + 1)|t1 − t2| ≤ C(Lsup)|t1 − t2| ,
so that
(III) ≤ Ex∼µ (|∂θf(Zρ(t2;x), θ2)||pρ(t1;x)− pρ(t2;x)|) ≤ C(Lsup)(|θ2|+ 1)|t1 − t2| ,
where we use Assumption 4.1 (13) in the second inequality. By substituting these bounds into (44), we complete the proof of (42). Finally, to prove (43), we recall the definition of the Fréchet derivative, to obtain∣∣∣∣∇θ δE(ρ1)δρ (θ, t)−∇θ δE(ρ2)δρ (θ, t) ∣∣∣∣
≤Ex∼µ (|∂θf(Zρ1(t;x), θ)pρ1(t;x)− ∂θf(Zρ2(t;x), θ)pρ2(t;x)|) ≤Ex∼µ (|∂θf(Zρ1(t;x), θ)− ∂θf(Zρ2(t;x), θ)| |pρ1(t;x)|)
+ Ex∼µ (|∂θf(Zρ2(t;x), θ)| |pρ1(t;x)− pρ2(t;x)|) ≤C(L, |θ[1]|)Ex∼µ (|Zρ1(t;x)− Zρ2(t;x)|) + C(L)(|θ|+ 1)Ex∼µ (|pρ1(t;x)− pρ2(t;x)|) ≤C(L, |θ[1]|,R)(1 + |θ|)d1(ρ1, ρ2) ,
where we use Assumption 4.1 for f with (18) and (31) in the third inequality. In the last inequality, we use (19), (32).

heading: C.2 A-PRIORI ESTIMATE UNDER THE HOMOGENEOUS ASSUMPTION
text: In Theorem 6.1 and 6.2, 2-homogeneity or partial 1-homogeneity are assumed. When these properties hold, we can sharpen the estimates obtained in the previous section. We summarize our results here.
Lemma C.3 Suppose that Assumption 4.1 holds, then there exists a constant C3(r) depending increasingly on r such that for any r > 0 with |x| < r and θ ∈ Rk, we have the following.
• If f is 2-homogeneous (Assumption 6.1), then∣∣∂2xf ∣∣ ≤ C3(r) (|θ|2 + 1) , |∂x∂θf | ≤ C3(r) (|θ|+ 1) , ∣∣∂2θf ∣∣ ≤ C3(r) . (45) • If f is partially 1-homogeneous (Assumption 6.2), then∣∣∂2xf ∣∣ ≤ C3(r)(|θ|+ 1), |∂x∂θf | ≤ C3(r)(|θ|+ 1), ∣∣∂2θf ∣∣ ≤ C3(r)(|θ|+ 1) , (46)
where | · | denotes the Frobenius norm.
Proof When f satisfies Assumption 4.1 and Assumption 6.2, (46) can be obtained from direct calculation. When f is 2-homogeneous in θ, we have
f(x, θ) = |θ|2f (x, θ/|θ|) = f1(θ)f2(x, θ) ,
where f1(θ) = |θ|2, f2(x, θ) = f (x, θ/|θ|). Naturally, with product rule, the derivatives of f becomes the products of derivatives of f1 and f2. We then obtain (45), noting that when |θ| > 1, we have ∣∣∂2xf1∣∣ = 0, |∂x∂θf1| ≤ 0, |∂xf1| = 0, |∂θf1| ≤ 2|θ|, ∣∣∂2θf1∣∣ ≤ 2k, and∣∣∂2xf2∣∣ = C(r), |∂x∂θf2| ≤ C(r)|θ| , |∂xf2| = C(r), |∂θf2| ≤ C(r)|θ| , ∣∣∂2θf2∣∣ ≤ C(r)|θ|2 .
The estimates above allow us to improve the stability results in Lemmas C.1 and C.2.
Lemma C.4 Suppose that Assumption 4.1 holds and let ρ1, ρ2 ∈ C([0, 1];P2). Define L1 and L2 as in (20) and
Lsup1 = sup t∈[0,1] ∫ Rk |θ|2dρ1(θ, t) .
Then we have the following results.
• If f either satisfies Assumption 6.1 or Assumption 6.2, we have the following properties.
1. Stability of pρ: |pρ1(t;x)− pρ2(t;x)| ≤ C(L1,L2)d1(ρ1, ρ2) . (47) 2. Lipschitz continuity in ρ:∣∣∣∣∇θ δE(ρ1)δρ (θ, t)−∇θ δE(ρ2)δρ (θ, t) ∣∣∣∣ ≤ C(L1,L2)d1(ρ1, ρ2) (|θ|+ 1) , (48)
where d1 is defined in (4.1).
• Lipschitz continuity in θ and t: If f satisfies Assumption 6.1, then for any (θ1, t1), (θ2, t2) ∈ Rk × [0, 1], we have∣∣∣∣∇θ δE(ρ1)δρ (θ1, t1)−∇θ δE(ρ1)δρ (θ2, t2) ∣∣∣∣ ≤ C(L1)|θ1−θ2|+C(Lsup1 )(|θ2|+1)|t1−t2| . (49)
• Lipschitz continuity in θ and t: If f satisfies Assumption 6.2, then for any (θ1, t1), (θ2, t2) ∈ Rk × [0, 1], we have∣∣∣∣∇θ δE(ρ1)δρ (θ1, t1)−∇θ δE(ρ1)δρ (θ2, t2) ∣∣∣∣ ≤ C(Lsup1 )(|θ1|+|θ2|+1)(|θ1−θ2|+|t1−t2|) , (50)
Remark C.1 We note that in comparing (32) with (47) and (42)-(43) with (48)-(50), the main differences are the dependence of the bounds on θ. The new estimates have explicit (and mild) dependence on θ.
Proof First, to prove (47), we let ∆(t;x) = pρ1(t;x) − pρ2(t;x), and recall (35) and (36). Using the boundedness of Zρ in (18) and (19), and calling (45) (or (46)), we have from (36) that
|Dρ1,ρ2(t;x)| ≤ C(L1,L2) (∫
Rk |θ|2dρ1(θ, t) + ∫ Rk |θ|2dρ2(θ, t) + 1 ) d1(ρ1, ρ2) .
By substituting into (35), and using (31), (33), and Hölder’s inequality, we have d|∆(t;x)|2
dt ≤C(L1,L2) (∫ Rk |θ|2dρ1(θ, t) + ∫ Rk |θ|2dρ2(θ, t) + 1 ) |∆(t;x)|2
+ C(L1,L2) (∫
Rk |θ|2dρ1(θ, t) + ∫ Rk |θ|2dρ2(θ, t) + 1 ) d21(ρ1, ρ2) .
The result (47) follows from the initial condition (34) together with Grönwall’s inequality. Next, to prove (48), we have∣∣∣∣∇θ δE(ρ1)δρ (θ, t)−∇θ δE(ρ2)δρ (θ, t) ∣∣∣∣
≤Ex∼µ (|∂θf(Zρ1(t;x), θ)pρ1(t;x)− ∂θf(Zρ2(t;x), θ)pρ2(t;x)|) ≤Ex∼µ (|∂θf(Zρ1(t;x), θ)− ∂θf(Zρ2(t;x), θ)| |pρ1(t;x)|)
+ Ex∼µ (|∂θf(Zρ2(t;x), θ)| |pρ1(t;x)− pρ2(t;x)|) ≤C(L1,L2)(|θ|+ 1) (Ex∼µ (|Zρ1(t;x)− Zρ2(t;x)|) + Ex∼µ (|pρ1(t;x)− pρ2(t;x)|)) ≤C(L1,L2)d1(ρ1, ρ2) (|θ|+ 1) ,
where we also use (13), (18), (31), (45) (or (46)) in the second inequality and (19) and (47) in the final inequality. Finally, to prove (49) and (50), we have as in (44) that∣∣∣∣∇θ δE(ρ1)δρ (θ1, t1)−∇θ δE(ρ1)δρ (θ2, t2) ∣∣∣∣
≤ ∣∣Ex∼µ (p>ρ1(t1;x)∂θf(Zρ1(t1;x), θ1)− p>ρ1(t1;x)∂θf(Zρ1(t1;x), θ2))∣∣︸ ︷︷ ︸
(I) + ∣∣Ex∼µ (p>ρ1(t1;x)∂θf(Zρ1(t1;x), θ2)− p>ρ1(t1;x)∂θf(Zρ1(t2;x), θ2))∣∣︸ ︷︷ ︸
(II) + ∣∣Ex∼µ (p>ρ1(t1;x)∂θf(Zρ1(t2;x), θ2)− p>ρ1(t2;x)∂θf(Zρ1(t2;x), θ2))∣∣︸ ︷︷ ︸
(III)
.
(51)
The boundedness of the three terms above relies on Assumption 4.1 and (45) (or (46)).
To bound (I), if f is 2-homogeneous, we have
|(I)| ≤ C(L1)|θ1 − θ2| ,
where we use (18), (31), and ∂2θf ≤ C(|z|). If f satisfies Assumption 6.2, we have
|(I)| ≤ C(L1)(|θ1|+ |θ2|+ 1)|θ1 − θ2| ,
where we use (18), (31), and ∂2θf(z, θ) ≤ C(|z|)(|θ|+ 1). The bounds of (II) and (III) are same for both homogeneity assumptions and similar to the proof of Lemma C.2. For (II), we have
|∂θf(Zρ1(t1;x), θ2)− ∂θf(Zρ1(t2;x), θ2)| ≤ C(L1)(|θ2|+ 1)|Zρ1(t1;x)− Zρ1(t2;x)|
and |Zρ1(t1;x)− Zρ1(t2;x)| ≤ C(L1)(L sup 1 + 1)|t1 − t2| . For (III), we also use |∂θf(Zρ1(t2;x), θ2)| ≤ C(L1)(|θ2|+ 1) and |pρ1(t1;x)− pρ1(t2;x)| ≤ C(L1)(L sup 1 + 1)|t1 − t2| . These estimates, together with L1 ≤ Lsup1 and (51), prove the claims (49) and (50).

heading: D WELL POSEDNESS OF GRADIENT FLOW
text: Theorem A.4 is about well-posedness of the gradient flow equation (9) in the mean-field limit, while Theorem A.3 shows the corresponding result in the discrete setting. The proof for the two are similar. We first show the mean-field limit well-posedness, Theorem A.4.

heading: D.1 PROOF OF THEOREM A.4
text: We use the fixed-point argument. To do so, we first define a subset of C([0, S]; C([0, 1];P2)) with compact support measures, as follows:
ΩS = { φ(θ, t, s) ∈ C([0, S]; C([0, 1];P2)) ∣∣∃r > 0, supp(φ) ⊂ {θ||θ[1]| < r}, ∀(t, s) ∈ [0, 1]× [0, S]} ∩ { φ(θ, t, s) ∈ C([0, S]; C([0, 1];P2))
∣∣φ(θ, t, 0) = ρini(θ, t)} For any φ(θ, t, s) ∈ ΩS with φ(θ, t, 0) = ρini(θ, t), we define a map
ϕ = TS(φ) : ΩS → ΩS (52)
where ϕ solves  ∂ϕ(θ, t, s) ∂s = ∇θ · ( ϕ(θ, t, s)∇θ δE(φ(s)) δρ (θ, t) ) ,
ϕ(θ, t, 0) = ρini(θ, t) .
(53)
The strategy is to show this map is a contraction map, so that there is a fixed point in the set ΩS , which is then the solution to equation (9).
The proof of Theorem A.4 is divided into three steps:
Step 1: We show TS is well-defined map from ΩS to ΩS . Step 2: We give a bound of d2(TS(φ1), TS(φ2)) in terms of d2(φ1 , φ2). One can then tune S to
ensure TS is a contraction map, meaning there is a unique fixed point φ∗ so that φ∗ = TS(φ∗), and thus φ∗ solves (53) for s < S.
Step 3: We extend the local solution to a global solution.
Step 1. According to the definition of (53), for a fixed φ(θ, t, s), let{ dθφ(s;t)
ds = −∇θ δE(φ(s)) δρ (θφ(s; t), t) ,
θφ(0; t) ∼ ρini(θ, t) . (54)
Then θφ ∼ ϕ = TS(φ). To show the existence of ϕ amounts to showing the wellposedness of (54).
According to Lemma C.2 (41) and (42), the force ∇θ δEδφ(s) (·, t) has a linear growth and is locally Lipschitz. Classical ODE theory then suggests there is a unique solution for s ∈ [0, S], which depends continuously on the initial value θ(0; t).
Denoting
LS,φ = sup 0≤s≤S ∫ 1 0 ∫ Rk |θ|2 dφ(θ, t, s) dt,
Lsupini = sup 0≤t≤1 ∫ Rk |θ|2 dρini(θ, t),
LsupS,φ = sup 0≤t≤1,0≤s≤S ∫ Rk |θ|2 dφ(θ, t, s) Rini = inf r>0 { supp(ρini(θ, t)) ⊂ {θ||θ[1]| < r}, ∀t ∈ [0, 1] } , RS,φ = inf r>0 { supp(φ(θ, t, s)) ⊂ {θ||θ[1]| < r}, ∀(t, s) ∈ [0, 1]× [0, S] } ,
(55)
we have the following proposition.
Proposition D.1 Suppose that θφ(s; t) solves (54) and φ ∈ ΩS . Then for any (t1, s1), (t2, s2) ∈ [0, 1]× [0, S], we have
E ( |θφ(s1; t1)|2 ) ≤ exp(SC(LS,φ)) (Lsupini + 1) , |θφ,[1](s1; t1)|
≤ exp(SC(LS,φ)) (Rini + 1) (56)
and
E ( |θφ(s1; t1)− θφ(s2; t2)|2 ) ≤ C ( LsupS,φ,Rini, S )( E ( |θφ(0; t1)− θφ(0; t2)|2 ) + |t1 − t2|2 + |s1 − s2|2 ) .
(57)
Proof To prove the first bound in (56), we multiply (54) on both sides by θφ and use boundedness of the forcing term (41) to obtain
d |θφ(s; t1)|2
ds ≤ C(LS,φ)(|θφ(s; t1)|2 + 1) .
Using Grönwall’s inquality together with E ( |θφ(0; t1)|2 ) ≤ Lsupini , we obtain the first bound in (56).
To prove the second bound in (56), we use the bound of ∣∣∣∂θ[1] δE(ρ)δρ (θ, t)∣∣∣ according to Lemma C.2 (41): d ∣∣θφ,[1](s; t1)∣∣2
ds ≤ C(LS,φ)( ∣∣θφ,[1](s; t1)∣∣2 + 1) . Using Grönwall’s inquality together with |θφ,[1](0; t1)| ≤ Rini, we obtain the second bound in (56). To show (57), we first write
E ( |θφ(s1; t1)− θφ(s2; t2)|2 ) ≤ 2E ( |θφ(s1; t1)− θφ(s1; t2)|2
) ︸ ︷︷ ︸
(I)
+2E ( |θφ(s1; t2)− θφ(s2; t2)|2 ) ︸ ︷︷ ︸
(II)
, (58)
then bound the two terms (I) and (II) separately. For (I), we use (54), the second bound of (56), and Lemma C.2 (42)
d |θφ(s; t1)− θφ(s; t2)|2
ds
≤C(LS,φ,Rini) |θφ(s; t1)− θφ(s; t2)|2 + C(LsupS,φ,Rini)(|θφ(s1; t1)| 2 + |θφ(s1; t2)|2 + 1)|t1 − t2|2 ,
Using the first bound in (56) and Grönwall’s inequality, this implies that E ( |θφ(s1; t1)− θφ(s1; t2)|2 ) ≤ C ( LsupS,φ,Rini, S )( E ( |θφ(0; t1)− θφ(0; t2)|2 ) + |t1 − t2|2 ) . (59) For (II), we obtain an estimate by integrating (54) from s1 to s2 and using the boundedness of ∇θ δE(ρ)δρ in (41). From the Grönwall inequality, we have
|θφ(s1; t2)− θφ(s2; t2)| ≤ C (LS,φ) (∫ s2
s1
|θφ(s; t2)|ds+ |s1 − s2| ) .
Using first bound in (56) and Hölder’s inequality, this implies
E |θφ(s1; t2)− θφ(s2; t2)|2 ≤ C (LS,φ,Lsupini , S) |s1 − s2| 2 (60)
and thus combining with (59) and substituting in (58), we complete the proof.
An immediate corollary of Proposition D.1 is that the map TS(·) is well defined.
Corollary D.1 For every fixed S > 0, the map TS is well defined. That is, for any φ ∈ ΩS , one can find ϕ = TS(φ) ∈ ΩS as the unique solution of (53). In particular, for any (t, s) ∈ [0, 1] × [0, S], we have ∫
Rk |θ|2 dϕ(θ, t, s) ≤ exp(SQ1(LS,φ)) (Lsupini + 1) , suppθ(ϕ(θ, t, s)) ⊂ { θ ∣∣|θ[1]| ≤ exp(SQ1(LS,φ)) (Rini + 1)} , (61)
where Q1 : R+ → R+ is depends only on LS,φ and is an increasing function of its argument.
Proof For fixed (t, s) ∈ [0, 1]× [0, S], define ϕ(θ, t, s) as the distribution of θφ(s; t). Using classical stochastic theory (Ambrosio et al., 2008, Prop 8.1.8), ϕ(θ, t, s) is a solution to (53). The estimate of the support is a consequence of (56). Finally, using (16) and (57), we obtain that
lim (t,s)→(t0,s0) W2(ϕ(·, t, s), ϕ(·, t0, s0)) ≤ lim (t,s)
( E ( |θφ(s; t)− θφ(s0; t0)|2 ))1/2 = 0 ,
which proves the continuity in s and t, so that ϕ ∈ C([0, S]; C([0, 1];P2)). By combining all the factors above, we conclude that ϕ ∈ ΩS .
Step 2. We show now that TS is a contraction map for S sufficiently small.
Proposition D.2 For any φ1, φ2 ∈ ΩS , we have
d2(TS(φ1), TS(φ2)) ≤ SQ2(LS ,RS , S)d2(φ1, φ2), (62)
where Q2 : R3 → R+ is an increasing function and LS = max{LS,φ1 ,LS,φ2}, RS = max{RS,φ1 ,RS,φ2}, with LS,φ,RS,φ defined in (55).
Proof Denote by θφi(s; t) the solutions to (54) with φ = φi using the same initial data, that is,
θφ1(0; t) = θφ2(0; t) .
As in the previous subsection, we translate the study of ϕi to the study of θφi . Defining
∆t(s) = |θφ1(s; t)− θφ2(s; t)| ,
we have according to the definition of Wasserstein distance that
d2(TS(φ1), TS(φ2)) ≤ sup (t,s)∈[0,1]×[0,S]
E ( ∆2t (s) ) .
Using (54), we obtain
d(∆t(s)) 2
ds ≤ 4(∆t(s))2 + 4 ∣∣∣∣∇θ δE(φ1(s))δρ (θφ1 , t)−∇θ δE(φ2(s))δρ (θφ2 , t) ∣∣∣∣2
≤ 4(∆t(s))2 + 8 ∣∣∣∣∇θ δE(φ1(s))δρ (θφ1 , t)−∇θ δE(φ1(s))δρ (θφ2 , t) ∣∣∣∣2 + 8 ∣∣∣∣∇θ δE(φ1(s))δρ (θφ2 , t)−∇θ δE(φ2(s))δρ (θφ2 , t) ∣∣∣∣2 .
(63)
The second term on the right hand side involves the continuity addressed in Lemma C.2 (42)∣∣∣∣∇θ δE(φ1(s))δρ (θφ1 , t)−∇θ δE(φ1(s))δρ (θφ2 , t) ∣∣∣∣2 ≤ C(LS ,RS , S)(∆t(s))2 ,
where we use the second bound in (56) to substitute the constant in (42). Then the last term of (63) involves the continuity discussed in (43). In particular, we have∣∣∣∣∇θ δE(φ1(s))δρ (θφ2 , t)−∇θ δE(φ2(s))δρ (θφ2 , t) ∣∣∣∣ ≤C(RS , |θφ2,[1]|)(1 + |θφ2 |)d1(φ1, φ2) ≤ C(LS ,RS , S)(1 + |θφ2 |)d2(φ1, φ2) ,
where d2 is defined in Definition 4.1 and in the second inequality we use the second bound in (56) to substitute the constant in (43).
By substituting in (63), we obtain
d(∆t(s)) 2
ds ≤ C(LS ,RS , S)
[ (∆t(s)) 2 + (1 + |θφ2 |2)d22(φ1, φ2) ] ,
which implies
d ( E(∆t(s))2 ) ds ≤ C(LS ,RS , S) [( E(∆t(s))2 ) + d22(φ1, φ2) ] ,
where we use the first inequality in (56). From the Grönwall inequality, there exists Q2 : R3 → R+ is an increasing function such that
E ( (∆t(s)) 2 ) ≤ SQ2(LS ,RS , S)d22(φ1, φ2) ,
completing the proof.
To apply the contraction mapping theorem, we need to verify two conditions in order to show that there exists a fixed point φ∗ = TS(φ∗):
• There is a closed subset in ΩS such that TS maps this set to itself. • TS is a contraction map in this subset.
For the closed subset, we define Bρ0 = { φ ∈ ΩS ∣∣supp(φ(t, s)) ⊂ {θ||θ[1]| ≤ 4(Rini + 1)}, ∀(t, s) ∈ [0, 1]× [0, S]} ∩ { φ ∈ ΩS ∣∣∣∣∫ Rk |θ|2 dφ(θ, t, s) ≤ 4 (Lsupini + 1) , ∀(t, s) ∈ [0, 1]× [0, S] } .
(64)
We now claim that for small enough S, TS is a contraction map in Bρ0 .
Proposition D.3 Suppose that S is small enough that exp(SQ1(4(Lsupini + 1))) (L sup ini + 1) ≤ 4(L sup ini + 1 ,
exp(SQ1(4(Lsupini + 1))) (Rini + 1) ≤ 4(Rini + 1) ,
SQ2(4(Lsupini + 1), 4(Rini + 1), S) < 1
2 ,
where Q1 and Q2 are defined in Corollary D.1 and Proposition D.2, respectively. Then we have the following.
• If φ ∈ Bρ0 , then TS(φ) ∈ Bρ0 , that is, for any (t, s) ∈ [0, 1]× [0, S], we have supp(TS(φ)(t, s)) ⊂ {θ||θ[1]| ≤ 4(Rini + 1)} , (65)
and ∫ Rk |θ|2 dTS(φ)(θ, t, s) ≤ 4 (Lsupini + 1) . (66)
• TS is a contraction map in this subset, meaning that for any φ1, φ2 ∈ Bρ0 , we have
d2(TS(φ1), TS(φ2)) < 1
2 d2(φ1, φ2) . (67)
Proof First, using Corollary D.1 (61) and noticing LS,φ ≤ 4 (Lsupini + 1), we prove (65), (66). Then, using (62) with (65) and (66), we have
d2(TS(φ1), TS(φ2)) ≤ SQ2(4(Lsupini + 1), 4(Rini + 1), S) < 1
2 d2(φ1, φ2),
which proves (67).
Using the contraction mapping theorem, we can obtain directly that TS(φ) has a fixed point in Bρ0 when S is small enough.
Corollary D.2 If S satisfies conditions in Proposition D.3, then there exists a unique φ∗(θ, t, s) ∈ Bρ0 ⊂ ΩS such that φ∗(θ, t, s) is a solution to (9) with initial condition ρini(θ, t).
This is a direct consequence of the application of contraction mapping theorem.
Finally, we prove that the cost function decreases along the flow.
Lemma D.1 Suppose that φ∗(θ, t, s) ∈ C([0, S]; C([0, 1];P2)) solves (9) with initial condition ρini(θ, t). Then for 0 < s < S, we have
dE(φ∗(θ, t, s))
ds = − ∫ 1 0 ∫ Rk ∣∣∣∣∇θ δE(φ∗(s))δρ (θ, t) ∣∣∣∣2 dφ∗(θ, t, s) dt ≤ 0 . (68)
Proof Denote by θ∗(s; t) the associated path, meaning that θ∗(s; t) solves (54) with φ = φ∗, then θ∗ ∼ φ∗, meaning the distribution of θ∗ is φ∗. According to (9), we obtain using a change of variable that
dE(φ∗(θ, t, s))
ds = − ∫ 1 0 ∫ Rk ∣∣∣∣∇θ δE(φ∗(s))δρ (θ, t) ∣∣∣∣2 dφ∗(θ, t, s) dt ≤ 0 , (69)
which proves the result. We note that the derivation in (69) is formal. A rigorous proof can be found in (Ding et al., 2021, Appendix I).
Step 3. In this final step of the proof, we extend the local solution from Corollary D.2 to a global solution. Lemma D.1 shows that the formula of dEds , so we can then use this formula to improve the bound for the support of the solution (61). This improvement will be shown in the following corollary. This improved estimate helps in extending the local solution to the global solution.
Corollary D.3 For fixed S satisfying the condition in Proposition D.3, denote by φ∗(θ, t, s) ∈ C([0, S]; C([0, 1];P2)) the solution to (9) with initial condition ρini(θ, t). Then for any (t, s) ∈ [0, 1]× [0, S], we have ∫
Rk |θ|2 dφ∗(θ, t, s) ≤ C(S,Rini,Lsupini ) , supp(φ∗(t, s)) ⊂ { θ ∣∣|θ[1]| ≤ C(S,Rini,Lsupini )} , (70)
where the quantity C depends only on S,Rini, and Lsupini ).
Proof According to (61), it suffices to prove
LS,φ∗ = sup 0≤s≤S ∫ 1 0 ∫ Rk |θ|2 dφ∗(θ, t, s) dt ≤ C(S,Rini,Lsupini ) .
Denote by θ∗(s; t) the particle representation of φ∗, meaning that θ∗(s; t) solves (54) with φ = φ∗. Since θ∗(s; t) ∼ φ∗(s; t),
LS,φ∗ = sup 0≤s≤S ∫ 1 0 ∫ Rk |θ|2 dφ∗(θ, t, s) dt = sup 0≤s≤S ∫ 1 0 E ( |θ∗(s; t)|2 ) dt .
Using (54), we obtain that
d|θ∗(s; t)|2
ds ≤ |θ∗(s; t)| ∣∣∣∣∇θ δE(φ∗(s))δρ (θ∗(s; t), t) ∣∣∣∣ ,
which gives d ∫ 1 0 E ( |θ∗(s; t)|2 ) dt
ds ≤ (∫ 1
0
E ( |θ∗(s; t)|2 ) dt )1/2(∫ 1 0 E (∣∣∣∣∇θ δE(φ∗(s))δρ (θ∗(s; t), t) ∣∣∣∣2 ) dt )
= (∫ 1 0 E ( |θ∗(s; t)|2 ) dt )1/2 ∣∣∣∣ dE(φ∗(θ, t, s))ds ∣∣∣∣ ,
where we use the Hölder inequality and (68) from Lemma D.1 in the last equality. Since∫ 1 0 E ( |θ∗(0; t)|2 ) dt ≤ Lsupini ,∫ S
0
∣∣∣∣ dE(φ∗(θ, t, s))ds ∣∣∣∣ ds ≤ E(ρini(θ, t))− E(φ∗(θ, t, S)) ≤ C(Lsupini ) ,
we obtain
LS,φ∗ = sup 0≤u≤s ∫ 1 0 E ( |θ∗(u; t)|2 ) dt ≤ C(S,Lsupini )
by Grönwall’s inequality. This proves (70).
By contrast with (61), this estimate removes the dependence of the bound on LS,φ. This improvement is important because it relaxes the fixed-point argument from the dependence on the initial guess φ.
We are now ready to prove Theorem A.4.
Proof [Proof of Theorem A.4] From Corollary D.2, let S1 be a constant satisfying the conditions in Proposition D.3. Then there is a local solution φ∗ ∈ C([0, S1]; C([0, 1];P2)) to (9).
We now denote by S∗ the largest time within which the solution exists, where we denote this solution by φ∗ ∈ C ( [0, S∗); C([0, 1];P2) ) . We aim to show that S∗ =∞. According to Corollary D.3 (70), for any s < S∗ and t ∈ [0, 1], we have∫ Rk |θ|2 dφ∗(θ, t, s) ≤ C(S∗,Rini,Lsupini ) ,
supp(φ∗(t, s)) ⊂ { θ ∣∣|θ[1]| ≤ C(S∗,Rini,Lsupini )} ,
DefineR∗ = RS∗,φ∗ , L∗ = LsupS∗,φ∗ according to (55). SinceR∗,L∗ <∞, let us choose ∆S∗ small enough to satisfy exp(∆S∗Q1(4(L∗+1))) (L∗ + 1) ≤ 4(L∗+1), exp(∆S∗Q1(4(L∗+1))) (R∗ + 1) ≤ 4(R∗+1) , and
∆S∗Q2(4(L∗ + 1), 4(R∗ + 1),∆S∗) ≤ 1
2 ,
If S∗ is finite, then, using Proposition D.3 and Corollary D.2, we can further extend φ∗ to be supported on C ( [0, S∗ + ∆S∗); C([0, 1];P2) ) , giving a contradiction. If follows that S∗ =∞, as desired.
Finally, (21) is a direct result of Lemma D.1.

heading: D.2 PROOF OF THEOREM A.3
text: This section is dedicated to Theorem A.3 — we show the well posedness of the gradient flow in the finite-layer case. We rewrite the gradient of (2) as follows
∂E(ΘL,M )
∂θl,m =
1
ML Ex∼µ
( ∂θf(ZΘL,M (l;x), θl,m)pΘL,M (l;x) ) , (71)
where pΘL,M (l;x) solves: p>ΘL,M (l;x) = p > ΘL,M (l + 1;x) ( I + 1 ML M∑ m=1 ∂zf ( ZΘL,M (l + 1;x), θl+1,i )) pΘL,M (L− 1;x) = ( g(ZΘL,M (L;x))− y(x) ) ∇g(ZΘL,M (L;x)), , (72) for 0 ≤ l ≤ L− 2. We unify the space in a similar fashion to Definition 4.1.
Definition D.1 ΘL,M = {θl,m}L−1,Ml=0,m=1 ∈ L ∞ L,M if and only if
sup l,m |θl,m| <∞ .
The metric in L∞L,M is defined as
d1,L,M ( ΘL,M , Θ̃L,M ) = max
l
( 1
M M∑ m=1
|θl,m − θ̃l,m|2 )1/2 .
Definition D.2 For s ≥ 0, we have ΘL,M (s) = {θl,m(s)}L−1,Ml=0,m=1 ∈ C([0,∞);L ∞ L,M ) if and only if
1. For fixed s ∈ [0,∞), ΘL,M (s) ∈ L∞L,M .
2. For any s0 ∈ [0,∞), lim s→s0 d1,L,M (ΘL,M (s),ΘL,M (s0)) = 0 ,
where d1,L,M is defined in Definition D.1.
The metric in C([0,∞);L∞L,M ) is defined by
d2,L,M ( ΘL,M , Θ̃L,M ) = sup
s d1,L,M (ΘL,M (s), Θ̃L,M (s)) .
Theorem A.3 is to say that the solution to (3) is unique in C([0,∞);L∞L,M ) if ΘL,M (0) ∈ L∞L,M . Before proving the theorem, prepare some a-priori estimates of ZΘL,M and pΘL,M .
Lemma D.2 Suppose that Assumption 4.1 holds and that x is in the support of µ. Let ΘL,M = {θl,m}L−1,Ml=0,m=1 , and Θ̃L,M = { θ̃l,m }L−1,M l=0,m=1 ,
and denote
LΘL,M = 1
LM L−1∑ l=0 M∑ m=1 |θl,m|2,
LΘ̃L,M = 1
LM L−1∑ l=0 M∑ m=1 |θ̃l,m|2,
RL,M = sup l,m
{ |θl,m|, ∣∣∣θ̃l,m∣∣∣} . Then for 0 ≤ l ≤ L− 1, we have the following properties:
• Boundedness in ZΘL,M : ∣∣ZΘL,M (l + 1;x)∣∣ ≤ C(LΘL,M ) , (73) • Lipschitz in ZΘL,M :∣∣∣ZΘL,M (l + 1;x)− ZΘ̃L,M (l + 1;x)∣∣∣ ≤ C (LΘL,M ,LΘ̃L,M) d1,L,M (ΘL,M , Θ̃L,M) ,
(74)
• Boundedness in pΘL,M : ∣∣pΘL,M (l;x)∣∣ ≤ C(LΘL,M ) , (75) • Lipschitz in pΘL,M :∣∣∣pΘL,M (l;x)− pΘ̃L,M (l;x)∣∣∣ ≤ C (RL,M ) d1,L,M (ΘL,M , Θ̃L,M) . (76)
Proof From (1) and (12) we obtain (∣∣ZΘL,M (l + 1;x)∣∣+ 1) ≤ C1 ( 1 + 1
LM M∑ m=1 (|θl,m|2 + 1) ) ( ∣∣ZΘL,M (l;x)∣∣+ 1)
≤ C1 exp
( 1
LM M∑ m=1 (|θl,m|2 + 1) ) ( ∣∣ZΘL,M (l;x)∣∣+ 1) ,
which proves (73) by iteration on l.
From (13) and (73) we obtain
1
ML M∑ m=1 |∂zf ( ZΘL,M (l + 1;x), θl,m ) | ≤ C(LΘL,M ) ML M∑ m=1 (|θl,m|2 + 1) ,
which by (72) implies
|pΘL,M (l;x)| ≤
( 1 +
C(LΘL,M ) ML M∑ m=1 (|θl,m|2 + 1) ) |pΘL,M (l + 1;x)| .
From this bound, together with |pΘL,M (x, L− 1)| ≤ C|ZΘL,M (L;x)| ≤ C(LΘL,M ), we prove (75) by iteration on l.
To prove (74), we subtract the two updating formulas and split the estimate to obtain∣∣∣ZΘL,M (l + 1;x)− ZΘ̃L,M (l + 1;x)∣∣∣ ≤ ( 1 +
C(LΘL,M ) ML M∑ m=1 (|θl,m|2 + 1) )∣∣∣ZΘL,M (l;x)− ZΘ̃L,M (l;x)∣∣∣ + C(LΘL,M ,LΘ̃L,M )
L
( 1
M M∑ m=1 (|θl,m|2 + |θ̃l,m|2 + 1)
) d1,L,M ( ΘL,M , Θ̃L,M ) ,
where we use (1) together with the bounds (13), and (73). Noting that |ZΘL,M (0;x)−ZΘ̃L,M (0;x)| = 0, we prove (74) by iteration on l.
Finally, for (76), we subtract two equations in the form of (72), and use (72)-(75) together with Lipschitz continuity to obtain∣∣∣pΘL,M (l;x)− pΘ̃L,M (l;x)∣∣∣ ≤ ∣∣∣∣∣(pΘL,M (l + 1;x)− pΘ̃L,M (l + 1;x))> ( I + 1 ML M∑ m=1 ∂zf ( ZΘL,M (l + 1;x), θl+1,m
))∣∣∣∣∣ + ∣∣∣∣∣p>Θ̃L,M (l + 1;x) ( 1 ML M∑ m=1 ( ∂zf ( ZΘL,M (l + 1;x), θl+1,i ) − ∂zf ( ZΘ̃L,M (l + 1;x), θ̃l+1,m
)))∣∣∣∣∣ ≤ ( 1 +
C(RL,M ) ML M∑ m=1 (|θl,m|2 + 1) )∣∣∣pΘL,M (l + 1;x)− pΘ̃L,M (l + 1;x)∣∣∣ + C(RL,M )
L d1,L,M
( ΘL,M , Θ̃L,M ) .
(77) The initial data is also controlled, as follows:
|pΘL,M (L− 1;x)− pΘ̃L,M (L− 1;x)| ≤ C|ZΘL,M (L;x)− ZΘ̃L,M (L;x)| ≤ C(RL,M )d1,L,M ( ΘL,M , Θ̃L,M ) .
By combining this with (77), we prove (76) by iteration on l.
Lemma D.2 resembles Theorem A.2 and Lemma C.1. These estimates allow us to prove Theorem A.3. Since the proof strategy is exactly the same, we omit details. Essentially we define a map
Θ̃(s) = T L,MS (Θ ′(s)) : C([0,∞);L∞L,M )→ , C([0,∞);L∞L,M ) ,
where Θ̃(s) solves: dΘ̃(s)
ds = −ML∇ΘE(Θ′(s)) , for s ≥ 0 ,
where Θ defines the forcing term. The estimates above provide all the ingredients to show the map is well-defined, and for a small enough S, the map is also contracting, leading to the uniqueness of the solution to (3). Similar to Lemma D.1, one can also show dEds = −ML|∇ΘE(ΘL,M )|
2, improving the estimates and removing the constants’ dependence on the initial guess. This extends the local solution to the global one, as done in Step 3 for the continuous case.

heading: E PROOF OF THEOREM 5.1
text: Theorem 5.1 links the cost defined by ΘL,M (s) with that defined by ρ(θ, t, s) for all s. The continuous and mean-field limits are obtained, with both L and M sent to infinity. We decompose this result into two parts, discussing mean-field and continuous limits separately.
We start with the full definition of "limit-admissible" for a distribution ρ.
Definition E.1 For an admissible ρ(θ, t), we say ρ(θ, t) is limit-admissible if the average of a large number of particle presentations is bounded and Lipschitz with high probability. That is, for an admissible ρ(θ, t), there are two constants C3 and C4, both greater than supt∈[0,1] ∫ Rk |θ| 2dρ(θ, t) such that, for any M stochastic process presentation {θm(t)}Mm=1 that are i.i.d. drawn from ρ(θ, t), the following properties are satisfied for any η > 0 and M > C3η :
1. Second moment boundedness in time:
P ( sup t∈[0,1] 1 M M∑ m=1 |θm(t)|2 ≤ C4 ) ≥ 1− η . (78)
2. For all L > 0, we have
P
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L ∣∣∣∣θm(t)− θm( lL )∣∣∣∣2 dt ≤ C4L2 ) ≥ 1− η . (79)
We now state the two theorems that play complementary parts in Theorem 5.1. The first theorem addresses the limit in M under the assumption that L =∞. This is the mean-field part of the analysis.
Theorem E.1 Let Assumptions 4.1 and 4.2 hold with some 0 < k1 ≤ k. Assume that ρini(θ, t) is limit-admissible and suppθ(ρini(θ, t)) ⊂ {θ||θ[1]| ≤ R} with some R > 0 for all t ∈ [0, 1]. Suppose that {θm(0; t)}Mm=1 are i.i.d drawn from ρini(θ, t). Suppose in addition that
• ρ(θ, t, s) solves (9) with the initial condition ρini(θ, t), and
• θm(s; t) solves (6) with the initial condition θm(0; t).
Then for any , η, S > 0, there exists a constant C(ρini(θ, t), S) > 0 depending on ρini(θ, t), S such that when M > C(ρini(θ,t),S) 2η , we have
P (|E(Θ(s; ·))− E(ρ(·, ·, s))| ≤ ) ≥ 1− η , ∀s < S .
Proof See Appendix F.
The conclusion of this result suggests that for a 1 − η confidence of an accuracy, M grows polynomially with respect to 1/ and 1/η.
The second result considers the convergence of the parameter configuration for the discrete ResNet (1) to that for the continuous ResNet (4) as L→∞. This is the continuous-limit part of the analysis.
Theorem E.2 Let Assumptions 4.1 and 4.2 hold with some 0 < k1 ≤ k. Assume that ρini(θ, t) is limit-admissible and suppθ(ρini(θ, t)) ⊂ {θ||θ[1]| ≤ R} with some R > 0 for all t ∈ [0, 1]. Suppose that {θm(0; t)}Mm=1 are i.i.d drawn from ρini(θ, t). Suppose in addition that
• θm(s; t) solves (6) with initial condition θm(0; t), • θl,m(s) solves (3) with initial condition θm ( 0; lL ) .
Then for any , η, S > 0, there exists a constant C(ρini(θ, t), S) > 0 depending on ρini(θ, t), S such that when M ≥ C(ρini(θ,t),S)η and L ≥ C(ρini(θ,t),S) , we have for all s < S that
P (|E(Θ(s; ·))− E(ΘL,M (s))| ≤ ) ≥ 1− η .
Proof See Appendix G.
This theorem shows that when the width is large enough, then with high probability, in the whole training process with s < S, the difference between the loss functions defined by the discrete ResNet and its continuous counterpart decreases to 0 as L→∞.

heading: F CONVERGENCE TO THE MEAN-FIELD PDE
text: This section is dedicated to mean-field analysis and the proof of Theorem E.1. The intuition of this theorem is largely aligned with many other mean-field results, as demonstrated in (Ding et al., 2021).
As argued in Section 5, to show “equivalence" between (6) and (9), we can test them on the same smooth function h(θ). Testing (9) on amounts to multiplying h on both sides of the equation by h. From integration by parts we have
d
ds ∫ Rk hdρ(θ) = − ∫ Rk ∇θh∇θ δE(ρ(s)) δρ dρ ,
that is, d
ds E(h) = E
( ∇θh∇θ δE(ρ(s))
δρ
) .
To test (6) on h, we let ρ = 1M ∑M m=1 δθm and obtain
d
ds E(h) =
1
M M∑ m=1 ∇θh(θm) d ds θm = − M∑ m=1 ∇θh(θm) δE δθm .
We see that (9) and (6) are equivalent when tested by h, if and only if the right hand sides of the two equations above are the same, that is,
M δE
δθm = ∇θ
δE(ρ)
δρ (θm, t) . (80)
This claim can be established from the definitions of the Fréchet derivatives for δE(ρ)δρ and δE δθm ; see (Ding et al., 2021, Lemma 33).
To give a quantitative estimate on how quickly (6) converges to of (9), we utilize the particle method, a classical strategy for the mean-field limit. We sketch the proof here and will it more rigorous in the following subsections. We make use of two particle systems. In one system, the particles evolve themselves, while in the second, the particles are moved forward according to the underlying field constructed by the limit. In our situation, the former particle system consists of the M stochastic processes {θm(s; t)} that descend according to E(Θ(s; ·)). The latter particle system will be termed Θ̃(s; t) = {θ̃m(s; t)}Mm=1; it descends according to E(ρ(·, ·, s)), the limiting cost function. Essentially, we prove
E(Θ(s; ·)) ≈ E ( Θ̃(s; t) ) ≈ E(ρ(·, ·, s)) .
The latter approximation arises roughly from the law of large numbers, but the former needs to be proved rigorously by tracing the two different evolving ODEs.
To be more specific, let ρ(θ, t, s) be the solution to (9) with admissible initial conditions ρini(θ, t), and let θm(s; t) be the solution to (6) with initial conditions {θm(t, 0)}Mm=1 that are i.i.d drawn from ρini(θ, t). Using the definition of E in (8), we have
|E(ρ(·, ·, s))− E(Θ(s; ·))| ≤Ex∼µ [∣∣∣∣12 (g(Zρ(s)(1;x))− y(x))2 − 12 (g(ZΘ(s)(1;x))− y(x))2 ∣∣∣∣] ≤Ex∼µ
(∣∣g(Zρ(s)(1;x))− g(ZΘ(s)(1;x))∣∣ (∣∣g(Zρ(s)(1;x)) + g(ZΘ(s)(1;x))∣∣+ |y(x)|)) ≤C(Ls)
∣∣g(Zρ(s)(1;x))− g(ZΘ(s)(1;x))∣∣ ≤C(Ls) ∣∣Zρ(s)(1;x)− ZΘ(s)(1;x)∣∣ , (81)
where Ls = max {∫ 1
0 ∫ Rk |θ| 2 dρ(θ, t, s) dt, 1M ∫ 1 0 ∑M m=1 |θm(s; t)| 2 dt }
. In this derivation, we used the Lipschitz property of g, the boundedness of y (required in Assumptions 4.1-4.2), and the boundedness of Zρ and ZΘ(s). Boundedness of Zρ was shown in Theorem A.2, while the bound for ZΘ(s) will be addressed in Lemma F.3. The constant C depends on the support of ρini, as well as on
s and the Lipschitz constant of g. It follows from (81) that to control E(ρ(·, ·, s))−E(Θ(s; ·)), we need to control ∣∣Zρ(s)(1;x)− ZΘ(s)(1;x)∣∣ . (82) To do so, we employ the particle method and invent a new particle system.
According to (80), we can reformulate the original particle system as
dθm(s; t)
ds = −∇θ
δE(ρdis(s))
δρ (θm(s; t)) , ∀(t, s) ∈ [0, 1]× [0,∞) , (83)
where we denote
ρdis(θ, t, s) = 1
M M∑ m=1 δθm(s;t)(θ) . (84)
We invent a new system that follows the underlying flow governed by the limit. Define Θ̃(s) = {θ̃m(s; t)}Mm=1, where θ̃m solves
dθ̃m(s; t)
ds = −∇θ
δE(ρ(s))
δρ
( θ̃m(s; t) ) , ∀(t, s) ∈ [0, 1]× [0,∞) , (85)
with initial condition θ̃m(0; t) = θm(0; t) . (86)
As a consequence, we have θ̃m(s; t) ∼ ρ(θ, t, s) for all (t, s). The corresponding ensemble distribution is
ρ̃dis(θ, t, s) = 1
M M∑ m=1 δθ̃m(s;t)(θ) . (87)
Now we have available particle system Θ(s) = {θm(s)}, a newly invented particle system Θ̃(s) = {θ̃m(s)} and the mean-field flow ρ. Accordingly, there are three versions of Z: Zρ(s) that solves (7) using ρ(s) and ZΘ(s) and ZΘ̃(s) that solve (4) using Θ(s) and Θ̃(s), respectively. We use the following relabelling for convenience:
Zs = Zρ(s) , Z dis s = ZΘ(s) , Z̃ dis s = ZΘ̃(s) . (88)
Similarly, there are three sets of p: pρ(s), pρdis(s), and pρ̃dis(s) that solve (11) using ρ(s), ρdis(s), and ρ̃dis(s), respectively. We relabel similarly to (88) and write
ps = pρ(s) , p dis s = pρdis(s) , p̃ dis s = pρ̃dis(s) . (89)
Since Θ̃(s) serves as a bridge, we translate the control of (82) to:∣∣Zs(t;x)− Zdiss (t;x)∣∣ ≤ ∣∣∣Zs(t;x)− Z̃diss (t;x)∣∣∣+ ∣∣∣Z̃diss (t;x)− Zdiss (t;x)∣∣∣ . (90) Bounding
∣∣∣Zs(t;x)− Z̃diss (t;x)∣∣∣ can be done using the law of large numbers. Bounding∣∣∣Z̃diss (t;x)− Zdiss (t;x)∣∣∣ translates to controlling∑m ∫ 10 |θm(s; t)− θ̃m(s; t)|dt, for which we will evaluate the difference between equations (83) and (85). Since these two equations have the same initial data, the difference between θm and θ̃m can then be controlled when the right-hand side forcing terms are close.
This approach divides the proof naturally into two components. In Section F.1, we give the rigorous bound of (90), while in Section F.2, we trace the evolution of the difference ∑ m ∫ 1 0 |θm(s; t) − θ̃m(s; t)|dt in s, thus finalizing the proof for Theorem E.1.

heading: F.1 STABILITY IN THE MEAN-FIELD REGIME
text: Here we discuss control of the two terms on the right-hand side of (90). Recall that Θ(s) and Θ̃(s) satisfy (83) and (85), and the two corresponding ensemble distribution are defined in (84) and (87), respectively. For any S > 0, define
LsupS = sup 0≤t≤1,0≤s≤S {∫ Rk |θ|2 dρ(θ, t, s), 1 M M∑ i=1 |θm(s; t)|2, 1 M M∑ i=1 |θ̃m(s; t)|2 }
RS = inf r>0
{ supp(ρ(θ, t, s)) ∪ {θm(s; t)}Mm=1 ⊂ {θ||θ[1]| < r}, ∀(t, s) ∈ [0, 1]× [0, S] } , (91)
We have the following lemma:
Lemma F.1 For every fixed s, let Zs and Zdiss be as defined in (88). Then there exists a constant C(Lsups ) such that for all t ∈ [0, 1], s ∈ [0,∞), we have∣∣Zs(t;x)− Zdiss (t;x)∣∣
≤ C(Lsups )
( 1
M M∑ m=1 ∫ 1 0 ∣∣∣θm(s; τ)− θ̃m(s; τ)∣∣∣2 dτ)1/2
+ C(Lsups ) (∫ 1 0 ∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ )1/2 .
(92)
Proof Since the statement holds for a fixed s, we eliminate all s dependence in all calculations in the proof, for conciseness.
Recalling the definitions in (88), we denote
∆̃(t;x) = Zs(t;x)− Z̃diss (t;x), ∆(t;x) = Z̃diss (t;x)− Zdiss (t;x) .
It follows from the triangle inequality that∣∣Zs(t;x)− Zdiss (t;x)∣∣ ≤ ∣∣∣∆̃(t;x)∣∣∣+ |∆(t;x)| . We now bound these two terms. We first apply the same argument as in the proof of Theorem A.2 (see (26) in Appendix B) to obtain
d ∣∣∣∆̃(t;x)∣∣∣2
dt ≤ C(Lsups ) ∣∣∣∆̃(t;x)∣∣∣2 + ∣∣∣∣∫ Rk f (Zs(t;x), θ) d(ρ(θ, t, s)− ρ̃dis(θ, t, s)) ∣∣∣∣2 . Using the Grönwall inequality and the fact that
∣∣∣∆̃(0;x)∣∣∣ = 0, we have ∣∣∣∆̃(t;x)∣∣∣ ≤ C(Lsups ) (∫ 1 0 ∣∣∣∣∫ Rk f (Zs(t;x), θ) d(ρ(θ, t, s)− ρ̃dis(θ, t, s)) ∣∣∣∣2 dτ )1/2 . (93)
for all t ∈ [0, 1]. Similarly, to bound ∆(t;x), we have
d |∆(t;x)|2
dt ≤C(Lsups ) |∆(t;x)| 2 + ∣∣∣∣∫ Rk f ( Z̃diss (t;x), θ ) d(ρdis(θ, t, s)− ρ̃dis(θ, t, s)) ∣∣∣∣2 . From Assumption 4.1 and the fact that Z̃dis is bounded in Theorem A.2, we have∣∣∣∣∫ Rk f ( Z̃diss (t;x), θ ) d(ρdis(θ, t, s)− ρ̃dis(θ, t, s)) ∣∣∣∣2 ≤ C(Lsups ) ( 1 M M∑ m=1
∣∣∣θm(s; t)− θ̃m(s; t)∣∣∣)2 . similar to the proof of Theorem A.2 (see (28) in Appendix B).
Using ∆(0;x) = 0, we apply Grönwall’s inequality to obtain
|∆(t;x)| ≤ C(Lsups )
( 1
M M∑ m=1 ∫ 1 0 ∣∣∣θm(s; τ)− θ̃m(s; τ)∣∣∣2 dτ)1/2 . (94) The result is obtained from adding (93) and (94).
The difference in ps and pdiss :
Lemma F.2 For every fixed s ∈ [0,∞), let ps and pdiss be defined in (89). There exists a constant C(Rs) withRs,Lsups defined in (91) such that for all t ∈ [0, 1]:∣∣ps(t;x)− pdiss (t;x)∣∣ ≤ C(Rs,Lsups ) ( 1
M M∑ m=1 ∫ 1 0 ∣∣∣θm(s; τ)− θ̃m(s; τ)∣∣∣2 dτ)1/2
+ C(Rs,Lsups ) (∫ 1 0 ∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ )1/2
+ C(Rs,Lsups ) (∫ 1 0 ∣∣∣∣∫ Rk ∂zf (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ )1/2 .
(95)
Proof As in the previous proof, we eliminate dependence on s in some notation, for conciseness. Denoting ∆p(t;x) = ps(t;x)− pdiss (t;x), we recall (11) to have:
d|∆p(t;x)|2
dt ≤ 2 ∣∣∣∣∫
Rk ∂zf(Zs(t;x), θ) dρ(θ, t, s) ∣∣∣∣ |∆p(t;x)|2 + 2|∆p(t;x)| ∣∣pdiss ∣∣ ∣∣∣∣∫ Rk ∂zf(Zs(t;x), θ) dρ(θ, t, s)− ∫ Rk ∂zf(Z dis s (t;x), θ) dρ dis(θ, t, s)
∣∣∣∣ ≤ C(Lsups )|∆p(t;x)|2
+ 2 ∣∣∣∣∫ Rk ∂zf(Zs(t;x), θ) dρ(θ, t, s)− ∫ Rk ∂zf(Z dis s (t;x), θ) dρ dis(θ, t, s) ∣∣∣∣2 ≤C(Lsups )|∆p(t;x)|2
+ 6 ∣∣∣∣∫ Rk ∂zf(Zs(t;x), θ) dρ(θ, t, s)− ∫ Rk ∂zf(Zs(t;x), θ) dρ̃ dis(θ, t, s) ∣∣∣∣2 + 6
∣∣∣∣∫ Rk ∂zf(Zs(t;x), θ) dρ̃ dis(θ, t, s)− ∫ Rk ∂zf(Zs(t;x), θ) dρ dis(θ, t, s) ∣∣∣∣2︸ ︷︷ ︸ (I)
+ 6 ∣∣∣∣∫ Rk ∂zf(Zs(t;x), θ) dρ dis(θ, t, s)− ∫ Rk ∂zf(Z dis s (t;x), θ) dρ dis(θ, t, s) ∣∣∣∣2︸ ︷︷ ︸ (II) ,
where we use (13) from Assumption 4.1 together with (18) and (31) in the second inequality. To bound (I) on the right-hand side, we recall the definition (84) and (87), and use Assumption 4.1 (14)
along with the boundedness of Z (as shown in (18)) to obtain
(I) ≤
( 1
M M∑ m=1
∣∣∣∂zf(Zs(t;x), θm(s; t))− ∂zf(Zs(t;x), θ̃m(s; t))∣∣∣)2
≤C(Rs,Lsups )
( 1
M M∑ m=1 |θm(s; t)− θ̃m(s; t)|
)2
≤C(Rs,Lsups )
( 1
M M∑ m=1
∣∣∣θm(s; t)− θ̃m(s; t)∣∣∣2) , (96)
where we use Hölder’s inequality in the last inequality. For (II), we have
(II) ≤
( 1
M M∑ m=1 ∣∣∂zf(Zs(t;x), θm(s; t))− ∂zf(Zdiss (t;x), θm(s; t))∣∣ )2
≤ C(Rs,Lsups )|Zs(t;x)− Zdiss (t;x)|2 .
(97)
By substituting (96) and (97) into the bound above, we obtain
d|∆p(t;x)|2
dt
=C(Lsups )|∆p(t;x)|2 + C(Rs,Lsups )
( 1
M M∑ m=1 ∣∣∣θm(s; t)− θ̃m(s; t)∣∣∣2 + |Zs(t;x)− Zdiss (t;x)|2 )
+ 6 ∣∣∣∣∫ Rk ∂zf(Zs(t;x), θ) dρ(θ, t, s)− ∫ Rk ∂zf(Zs(t;x), θ) dρ̃ dis(θ, t, s) ∣∣∣∣2 . (98)
The “initial condition" for ps and pdiss yields
|∆p(1;x)| ≤ C(Lsups )|Zs(1;x)− Zdiss (1;x)| .
The result is obtained when we substitute (92) into (98) and use the Grönwall’s inequality.

heading: F.2 PROOF OF THEOREM E.1
text: With the quantitative description of (90) presented in Lemma F.1, we complete the proof for Theorem E.1 in this section. Recall from (55) that
Rini = inf r>0
{ supp(ρini(t)) ⊂ { θ ∣∣|θ[1]| < r} , ∀t ∈ [0, 1]} , Lsupini = sup
0≤t≤1 ∫ Rk |θ|2 dρini(θ, t)
and note thatRini ≥ |θm,[1](0; t)| for all m, t. Define
Ldis,supini = sup 0≤t≤1
1
M M∑ m=1 |θm(0; t)|2 ,
We note that when M is large, Ldis,supini is close to L sup ini (which has no randomness) with high probability. We have the following lemma:
Lemma F.3 For a given S > 0, there exists a constant C (S,Rini) such that for any t ∈ [0, 1], s ∈ [0, S], we have
1
M M∑ m=1 |θm(s; t)|2 ≤ C(S,Rini,Ldis,supini ) ,
{θm(s; t)}Mm=1 ⊂ {θ||θ[1]| ≤ C(S,Rini,L dis,sup ini )} ,
(99)
Furthermore, for any x with |x| < Rµ (as in Assumption 4.2, item 4), and any s ∈ [0, S] and t ∈ [0, 1], the ODE solution is bounded as follows:∣∣Zdiss (t;x)∣∣ ≤ C(S,Rini,Ldis,supini ) , (100) while the following bound holds on pdiss :∣∣pdiss (t;x)∣∣ ≤ C(S,Rini,Ldis,supini ) . (101) The bound (99) is a result of Corollary D.3. The bounds (100) and (101) are obtained using the same arguments as in Theorem A.2 and Lemma C.1.
The next lemma bounds the support and second moment of ρ̃dis(θ, t, s).
Lemma F.4 Under conditions of Theorem E.1, for any S > 0, there is a constantC(S,Rini,Ldis,supini ) depending only onRini,Ldis,supini , S such that for any t ∈ [0, 1], s ∈ [0, S], we have
1
M M∑ m=1 |θ̃m(s; t)|2 ≤ C(S,Rini,Ldis,supini ) ,
{θ̃m(s; t)}Mm=1 ⊂ {θ||θ[1]| ≤ C(S,Rini,L dis,sup ini )} ,
(102)
Proof Similar to the proof of Proposition D.1, we multiply (85) by θ̃m(s; t) on both sides and utilize the bound (41) from Lemma C.2, where L in (41) is replaced by C(S,Rini,Ldis,supini ) according to Corollary D.3. We thus obtain
|θ̃m(s; t)| ≤ C(S,Rini,Ldis,supini ) ( |θ̃m(0; t)|+ 1 ) ,
and |θ̃m,[1](s; t)| ≤ C(S,Rini,Ldis,supini ) ( |θ̃m,[1](0; t)|+ 1 ) which implies (102) by (86).
Denote Lsup0 = max{L dis,sup ini ,L sup ini }. According to Definition E.1 (78), L sup 0 should be bound by C4 with high probability. We are now ready for the main proof of this section. Proof [Proof of Theorem E.1] First, using Lemmas F.3, F.4, there is a constant C(S,Rini,Lsup0 ) depending only onRini,Lsup0 , S such that for any t ∈ [0, 1], s ∈ [0, S]
max {∫ Rk |θ|2 dρ(θ, t, s), 1 M M∑ m=1 ∣∣∣θ̃m(s; t)∣∣∣2 , 1 M M∑ m=1 |θm(s; t)|2 } ≤ C(S,Rini,Lsup0 ) ,
supp(ρ(θ, t, s)) ∪ {θm(s; t)}Mm=1 ∪ { θ̃m(s; t) }M m=1 ⊂ { θ ∣∣|θ[1]| ≤ C(S,Rini,Lsup0 )} ,
(103)
Recalling (81), we have Ls ≤ C(S,Rini,Lsup0 ) and |E(ρ(·, ·, s))− E(Θ(s; ·))| ≤ C(S,Rini,Lsup0 ) ∣∣Zs(1;x)− Zdiss (1;x)∣∣ . (104) Furthermore, according to Lemma F.1 (92), from (103)∣∣Zs(t;x)− Zdiss (t;x)∣∣ ≤ C(S,Rini,Lsup0 ) ( 1
M M∑ m=1 ∫ 1 0 ∣∣∣θm(s; τ)− θ̃m(s; τ)∣∣∣2 dτ)1/2
+ C(S,Rini,Lsup0 ) (∫ 1 0 ∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ )1/2 . (105)
The second term in this bound can be treated using the law of large numbers. We focus on controlling the first term.
Step 1: Estimating 1M ∑M m=1 ∫ 1 0 ∣∣∣θm(s; t)− θ̃m(s; t)∣∣∣2 dt. Defining ∆t,m(s) = θm(s; t)− θ̃m(s; t) ,
we note that |∆t,m(0)| = 0. By taking the difference of (83) and (85) and multiplying both sides by ∆t,m(s), we obtain
d|∆t,m(s)|2
ds =− 2 〈 ∆t,m(s),Ex∼µ ( ∂θf(Zs(t;x), θ̃m)ps(t;x)− ∂θf(Zdiss (t;x), θm)pdiss (t;x) )〉 =− 2 〈 ∆t,m(s),Ex∼µ ( ∂θf(Zs(t;x), θ̃m)ps(t;x)− ∂θf(Zs(t;x), θ̃m)pdiss (t;x)
) ︸ ︷︷ ︸
(I)
〉
− 2 〈 ∆t,m(s),Ex∼µ ( ∂θf(Zs(t;x), θ̃m)p dis s (t;x)− ∂θf(Zdiss (t;x), θm)pdiss (t;x) ) ︸ ︷︷ ︸
(II)
〉 .
(106)
For (I), we have from the bounds of Zs in (18), respectively, that |(I)| ≤ C(S,Rini,Lsup0 ) ( |θ̃m|+ 1 ) Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣) , which can be controlled using Lemma F.2 (95). For (II), we have
|(II)| ≤C(S,Rini,Lsup0 )Ex∼µ (∣∣∣∂θf(Zs(t;x), θ̃m)− ∂θf(Zdiss (t;x), θm)∣∣∣)
≤C(S,Rini,Lsup0 ) [ Ex∼µ (∣∣Zs(t;x)− Zdiss (t;x)∣∣)+ |θ̃m − θm|] , where the first term can be controlled using Lemma F.1 (92). In both estimates, we used the property of f in Assumption 4.1 and bounds on Z, p, θm,[1], and θ̃m,[1]. By substituting these estimates into (106), we obtain
d|∆t,m(s)|2
ds ≤ C(S,Rini,Lsup0 )|∆t,m(s)|2 + C(S,Rini,Lsup0 )|∆t,m(s)|Ex∼µ (∣∣Zs(t;x)− Zdiss (t;x)∣∣)
+ C(S,Rini,Lsup0 )|∆t,m(s)| ( |θ̃m|+ 1 ) Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣) , which implies that
1
M M∑ m=1 d|∆t,m(s)|2 ds
≤ C(S,Rini,Lsup0 )
( 1
M M∑ m=1
|∆t,m(s)|2 )
+ C(S,Rini,Lsup0 )
( 1
M M∑ m=1 |∆t,m(s)|
) Ex∼µ
(∣∣Zs(t;x)− Zdiss (t;x)∣∣) + C(S,Rini,Lsup0 ) ( 1
M M∑ m=1 |∆t,m(s)| ( |θ̃m|+ 1 )) Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣) ≤ C(S,Rini,Lsup0 ) ( 1
M M∑ m=1
|∆t,m(s)|2 ) + C(S,Rini,Lsup0 )Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣2)
+ C(S,Rini,Lsup0 )Ex∼µ (∣∣Zs(t;x)− Zdiss (t;x)∣∣2) ,
(107)
where in the last inequality we use Hölder’s inequality( 1
M M∑ m=1 |∆t,m(s)| ( |θ̃m|+ 1 )) Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣) ≤ ( 1
M M∑ m=1
|∆t,m(s)|2 )1/2( 1
M M∑ m=1 ( |θ̃m|+ 1 )2)1/2 Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣) ≤C(S,Rini,Lsup0 ) ( 1
M M∑ m=1
|∆t,m(s)|2 )1/2 Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣)
≤C(S,Rini,Lsup0 )
[( 1
M M∑ m=1
|∆t,m(s)|2 ) + Ex∼µ (∣∣ps(t;x)− pdiss (t;x)∣∣2)
] .
Noting the estimate in Lemma F.1-F.2, we obtain d 1M ∑M m=1 ∫ 1 0 |∆t,m(s)|2 dt
ds
≤ C(S,Rini,Lsup0 )
( 1
M M∑ m=1 ∫ 1 0 |∆t,m(s)|2 dt )
+ C(S,Rini,Lsup0 )Ex∼µ (∫ 1 0 ∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(−ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ )
+ C(S,Rini,Lsup0 )Ex∼µ (∫ 1 0 ∣∣∣∣∫ Rk ∂zf (Zs(τ ;x), θ) d(−ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ ) ,
which implies, using Grönwall’s inequality, that
1
M M∑ m=1 ∫ 1 0 |∆t,m(s)|2 dt
≤ C(S,Rini,Lsup0 )Ex∼µ (∫ S 0 ∫ 1 0 ∣∣∣∣∫ Rk f (Zs(t;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ ds )
+ C(S,Rini,Lsup0 )Ex∼µ (∫ S 0 ∫ 1 0 ∣∣∣∣∫ Rk ∂zf (Zs(t;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ ds ) ,
(108) where we use |∆t,m(0)| = 0.
Step 2: Completing the proof. By substituting (108) into (92), noticingRs ≤ C(Rini, S), we obtain∣∣Zs(t;x)− Zdiss (t;x)∣∣ ≤C(S,Rini,Lsup0 ) Ex∼µ (∫ S 0 ∫ 1 0 ∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ ds )
︸ ︷︷ ︸ (I)
 1/2
+ C(S,Rini,Lsup0 ) Ex∼µ (∫ S 0 ∫ 1 0 ∣∣∣∣∫ Rk ∂zf (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ ds )
︸ ︷︷ ︸ (II)
 1/2
+ C(S,Rini,Lsup0 )  ∫ 1
0 ∣∣∣∣∫ Rk ∂zf (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ︸ ︷︷ ︸ (III)
 1/2 ,
(109) All three terms in (109) can be controlled in expectation. Here we take the expectation with respect to the randomness initial drawing of {θm(0; t)}Mm=1. For (I), we have
E(I) = E ( Ex∼µ (∫ S 0 ∫ 1 0 ∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 dτ ds ))
= Ex∼µ (∫ S 0 ∫ 1 0 E (∣∣∣∣∫ Rk f (Zs(τ ;x), θ) d(ρ(θ, τ, s)− ρ̃dis(θ, τ, s)) ∣∣∣∣2 ) dτ ds )
≤ C(S,Rini,L sup ini )
M Ex∼µ (∫ S 0 ∫ 1 0 ∫ Rk |f (Zs(τ ;x), θ) |2 dρ(θ, τ, s) dτ ds )
≤ C(S,Rini,L sup ini )
M ,
where we use θ̃m(s; t) ∼ ρ(θ, t, s) in the first inequality. In second inequality, if k1 < k, we use first inequality of Assumption 4.1 (15) with ∣∣θ[1]∣∣ ≤ C(S,Rini,Lsupini ) and |Zs| ≤ C(S,Rini,Lsupini ). If k1 = k, we use (12) and |θ| =
∣∣θ[1]∣∣ ≤ C(S,Rini,Lsupini ) and |Zs| ≤ C(S,Rini,Lsupini ) in the second inequality. By similar reasoning, we obtain
E(II) ≤ C(S,Rini,L sup ini ) M , E(III) ≤ C(S,Rini,L sup ini ) M .
From Markov’s inequality, these bounds imply that when M > C(Rini,S,L sup ini )
2η , we have P ({ (I) < 2 } ∩ { (II) < 2 } ∩ { (III) < 2 }) > 1− η/2 . (110)
Finally, using Definition E.1 (78), when M > 2C3η , P (Lsup0 ≤ C4) ≥ 1− η/2 . (111) By substituting (110) and (111) into (109), we see that there exists a constant C(Rini, C3, C4, S) such that for any , η > 0, when M > C(Rini,C3,C4,S) 2η we obtain that
P (∣∣Zs(1;x)− Zdiss (1;x)∣∣ < ) > 1− η .
By using this result and (111) in conjunction with (104), we complete the proof.

heading: G CONVERGENCE TO THE CONTINUOUS LIMIT
text: This section is dedicated to the continuous limit and, in particular, the proof of Theorem E.2.

heading: G.1 STABILITY WITH DISCRETIZATION
text: Before proving Theorem E.2, and similarly to Appendix F.1, we first consider the stability of Z and p under discretization. Defining the path of parameters Θ(t) = {θm(t)}Mm=1 and the set of parameters ΘL,M = {θl,m}L−1,Ml=0,m=1, we have the following lemma.
Lemma G.1 Suppose that Assumption 4.1 holds and that x is in the support of µ. Denoting
Lsup = sup 0≤t≤1,l
{ 1
M M∑ i=1 |θl,m|2, 1 M M∑ i=1
|θm(t)|2 }
R = inf r>0
{ {θl,m}M,Lm=1,l=1 ∪ {θm(t)} M m=1 ⊂ {θ||θ[1]| < r}, ∀t ∈ [0, 1] } ,
(112)
there exists a constant C(R,Lsup) depending only onR,Lsup such that for any 0 ≤ l ≤ L− 1, we have
sup l L≤t≤ l+1 L
{∣∣ZΘ(t;x)− ZΘL,M (l;x)∣∣ , ∣∣ZΘ(t;x)− ZΘL,M (l + 1;x)∣∣}
≤ C(R,Lsup)
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |θl,m − θm(τ)|2 dτ )1/2 + C(R,Lsup) L ,
(113)
and sup
l L≤t≤ l+1 L ∣∣pΘ(t;x)− pΘL,M (l;x)∣∣ ≤ C(R,Lsup) ( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |θl,m − θm(τ)|2 dτ )1/2 + C(R,Lsup) L .
(114)
Proof Define Z(t;x) = ZΘ(t;x), p(t;x) = pΘ(t;x) , and
Z̃(t;x) = L−1∑ l=0 ZΘL,M (l;x)1 l L≤t< l+1 L , p̃(t;x) = L−1∑ l=0 pΘL,M (l;x)1 l L<t≤ l+1 L , (115)
with Z̃(1;x) = ZΘL,M (L;x), p̃(0;x) = pΘL,M (0;x) . (116)
Using (1), (72), Assumption 4.1, and Lemma D.2 (73) and (75), we obtain for all l = 0, 1, . . . , L− 1 that ∣∣ZΘL,M (l + 1;x)− ZΘL,M (l;x)∣∣ < C(Lsup)L ,∣∣pΘL,M (l + 1;x)− pΘL,M (l;x)∣∣ < C(Lsup)L . (117) Now define ∆t by ∆t = Z(t;x)− Z̃(t;x) . For t ∈ [ lL , l+1 L ], we have from (4) that
|∆t| ≤ ∣∣∣∆ l
L ∣∣∣+ 1 M M∑ m=1 ∫ t l L |f(Z(τ ;x), θm(τ))|dτ
≤ ∣∣∣∆ l
L ∣∣∣+ C(Lsup) M M∑ m=1 ∫ l+1 L l L (|θm(τ)|2 + 1) dτ
≤ ∣∣∣∆ l
L ∣∣∣+ C(Lsup) L ,
(118)
where we use (12), (18), and (112) in the last two inequalities. From (1) and (4), we obtain further that∣∣∣∆ l+1
L ∣∣∣ = ∣∣∣∆ l L ∣∣∣+ ∣∣∣∣∣ 1M M∑ m=1 ∫ l+1 L l L f(Z(τ ;x), θm(τ))− f ( Z̃(τ ;x), θl,m ) dτ ∣∣∣∣∣ ≤ ∣∣∣∆ l
L ∣∣∣+ ∣∣∣∣∣ 1M M∑ m=1 ∫ l+1 L l L f(Z(τ ;x), θm(τ))− f (Z(τ ;x), θl,m) dτ ∣∣∣∣∣ +
∣∣∣∣∣ 1M M∑ m=1 ∫ l+1 L l L f(Z(τ ;x), θl,m)− f ( Z̃(τ ;x), θl,m ) dτ ∣∣∣∣∣ (I)
≤ ∣∣∣∆ l
L ∣∣∣+ C(Lsup)( 1 M M∑ m=1 ∫ l+1 L l L (|θm(τ)|+ |θl,m|+ 1)|θm(τ)− θl,m|dτ )
+ C(Lsup)|∆ξ|
( 1
ML M∑ m=1 (|θl,m|2 + 1) ) (II) ≤ ( 1 + C(Lsup)
L
) ∣∣∣∆ l L ∣∣∣+ C(Lsup)( 1 M M∑ m=1 ∫ l+1 L l L (|θm(τ)|+ |θl,m|+ 1)|θm(τ)− θl,m|dτ )
+ C(Lsup) L2 ,
where ξ ∈ [ lL , l+1 L ], and we used the mean-value theorem with (13), (18), (73) in (I) and (112), (118) in (II). By applying this bound iteratively, we obtain ∣∣∣∆ l L ∣∣∣ ≤ C(Lsup) |∆0|+C(Lsup)  1 M l−1∑ j=0 M∑ m=1 ∫ j+1 L j L (|θm(τ)|+ |θl,m|+ 1)|θm(τ)− θl,m|dτ +C(Lsup) L ,
where |∆0| = 0. By combining this bound with (118) and using Hölder’s inequality with (112), we obtain that
|∆t| ≤ C(Lsup)
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |θl,m − θm(τ)|2 dτ )1/2 + C(Lsup) L . (119)
By combining (119) with (117), we prove (113).
To prove (114), we define ∆p(t;x) = p(t;x)− p̃(t;x) . Similarly to (34), we obtain
|∆p(1;x)| ≤ C(Lsup) ∣∣∣Z̃(1;x)− Z(1;x)∣∣∣
≤ C(Lsup)
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |θl,m − θm(τ)|2 dτ )1/2 + C(Lsup) L .
(120)
For t ∈ ( l L , l+1 L ] and using (11), we obtain that
|∆p (t;x) | ≤ ∣∣∣∣∆p( l + 1L ;x )∣∣∣∣+ 1M M∑ m=1 ∫ l+1 L t |∂zf(Z(τ ;x), θm(τ))| |p(τ ;x)|dτ
≤ ∣∣∣∣∆p( l + 1L ;x )∣∣∣∣+ C(Lsup)M M∑ m=1 ∫ l+1 L l L (|θm(τ)|2 + 1) dτ
≤ ∣∣∣∣∆p( l + 1L ;x )∣∣∣∣+ C(Lsup)L , (121)
where we use (13), (18), and (31) in the second inequality and the definition of Lsup from (112) in the last line.
From (11), we obtain that
p> ( l
L ;x
) = p> ( l + 1
L ;x
) + 1
M M∑ m=1 ∫ l+1 L l L p> (τ ;x) ∂zf(Z(τ ;x), θm(τ)) dτ ,
while (72) implies that
p̃> ( l
L ;x
) = p̃> ( l + 1
L ;x
) + 1
M M∑ m=1 ∫ l+1 L l L p̃> ( l + 1 L ;x ) ∂zf(ZΘL,M (l;x), θl,m) dτ ,
where p̃ is defined in (115), (116). By bounding differences of these two expressions, we have∣∣∣∣∆p( lL ;x )∣∣∣∣ ≤ ∣∣∣∣∆p( l + 1L ;x
)∣∣∣∣ + 1
M M∑ m=1 ∫ l+1 L l L ∣∣∣∣p> (τ ;x) ∂zf(Z(τ ;x), θm(τ)) dτ − p̃>( l + 1L ;x ) ∂zf(Z(τ ;x), θm(τ)) ∣∣∣∣ dτ︸ ︷︷ ︸ (I)
+ 1
M M∑ m=1 ∫ l+1 L l L ∣∣∣∣p̃>( l + 1L ;x ) ∂zf(Z(τ ;x), θm(τ))− p̃> ( l + 1 L ;x ) ∂zf(ZΘL,M (l;x), θm(τ)) ∣∣∣∣ dτ︸ ︷︷ ︸ (II)
+ 1
M M∑ m=1 ∫ l+1 L l L ∣∣∣∣p̃>( l + 1L ;x ) ∂zf(ZΘL,M (l;x), θm(τ))− p̃> ( l + 1 L ;x ) ∂zf(ZΘL,M (l;x), θl,m) ∣∣∣∣ dτ︸ ︷︷ ︸ (III) .
(122) We bound (I), (II), and (III) in turn.
(I): Using (13), (18), and (112), we obtain that
(I) ≤ C(L sup)
L |∆p(t;x)|
(II): Using Assumption 4.1 (14) together with (18), (73), (75), and (112), we obtain that
(II) ≤ C(R,L sup)
M
M∑ m=1 ∫ l+1 L l L |Z(τ ;x)− ZΘL,M (l;x)|dτ
≤ C(R,L sup)
L
( 1 M L−1∑ l′=0 M∑ m=1 ∫ l′+1 L l′ L |θl′,m − θm(τ)|2 dτ )1/2 + 1 L  , where we make use of (113) in the last inequality. (III): Using Assumption 4.1 item 3 together with (73), (75), and (112), we obtain that
(III) ≤ C(R,L sup)
M
M∑ m=1 ∫ l+1 L l L |θm(τ)− θl,m|dτ
≤ C(R,Lsup) ∫ l+1 L
l L
( 1
M M∑ m=1
|θm(τ)− θl,m|2 )1/2 dτ
By substituting these three bounds and (121) into (122), we obtain∣∣∣∣∆p( lL ;x )∣∣∣∣ ≤ (1 + C(R,Lsup)L ) ∣∣∣∣∆p( l + 1L ;x )∣∣∣∣
+ C(R,Lsup)
L
( 1 M L−1∑ l′=0 M∑ m=1 ∫ l′+1 L l′ L |θl′,m − θm(τ)|2 dτ )1/2 + 1 L  + C(R,Lsup) ∫ l+1 L
l L
( 1
M M∑ m=1
|θm(τ)− θl,m|2 )1/2 dτ + C(R,Lsup)
L2 .
By applying this bound iteratively, and using (120) and (121), we obtain
|∆p(t;x)| ≤ C(R,Lsup) ( 1 M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |θl,m − θm(τ)|2 dτ )1/2 + 1 L  , (123) where we also use Hölder’s inequality to write
C(R,Lsup) L−1∑ l=0 ∫ l+1 L l L ( 1 M M∑ m=1 |θm(τ)− θl,m|2 )1/2 dτ
≤ C(R,Lsup)
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |θl,m − θm(τ)|2 dτ )1/2 .
We obtain (114) by combining (123) with (121).

heading: G.2 PROOF OF THEOREM E.2
text: We denote by θm(s; t) the solution to (6) with initial {θm(0; t)}Mm=1 that are i.i.d drawn from ρini(θ, t) . Further, θl,m(s) is a solution to (3) with initial value θl,m(0) = θm ( 0; lL ) for 0 ≤ l ≤ L− 1 and 1 ≤ i ≤M . Define
Ldis,supini = sup 0≤t≤1
{ 1
M M∑ i=1
|θm(0; t)|2 }
and recallRini defined in (55). According to Definition E.1, Ldis,supini is bounded with high probability. Then, we have the following lemma:
Lemma G.2 For fixed S > 0 and any s ∈ [0, S], there exists a constant C(S,Rini,Ldis,supini ) depending only on S,Rini,Ldis,supini such that
1
M M∑ m=1 |θl,m(s)|2 ≤ C(S,Rini,Ldis,supini ) ,
{θl,m(s)}M,Lm=1,l=1 ⊂ { θ ∣∣∣|θ[1]| ≤ C(S,Rini,Ldis,supini )} ,
(124)
Furthermore, the ODE solution and pΘL,M (s) are bounded as follows, for any x in the support of µ and l = 0, 1, . . . , L− 1: ∣∣ZΘL,M (s)(l + 1;x)∣∣ ≤ C (S,Ldis,supini ,Rini) , (125) and ∣∣pΘL,M (s)(l;x)∣∣ ≤ C (S,Ldis,supini ,Rini) . (126)
The proof is quite similar to that of Lemma F.3, so we omit the details.
We are now ready to prove Theorem E.2.
Proof [Proof of Theorem E.2] From (99) and (124) we obtain for all t ∈ [0, 1], s ∈ [0, S],
max
{ 1
M M∑ m=1 |θl,m(s)|2, 1 M M∑ m=1 |θm(s; t)|2 } ≤ C(S,Rini,Ldis,supini ) ,
{θm(s; t)}Mm=1 ∪ {θl,m(s)} M,L m=1,l=1 ⊂ { θ ∣∣∣|θ[1]| ≤ C(S,Rini,Ldis,supini )} ,
(127)
where C(S,Rini,Ldis,supini ) is a constant depending on S,Rini, and L dis,sup ini .
From a similar derivation to (81), we obtain |E(Θ(s; ·))− E(ΘL,M (s))| ≤ C(S,Rini,Ldis,supini ) ∣∣ZΘ(s)(1;x)− ZΘL,M (s)(L;x)∣∣ . (128)
Thus, to prove the theorem, it suffices to prove that ∣∣ZΘL,M (s)(L;x)− ZΘ(s)(1;x)∣∣ is small. For this purpose, according to (113), we need to bound the quantity
1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(s)|2 dt , (129)
where ∆t,m(s) = θl,m(s)− θm(s; t). The next part of the proof contains the required bound. First, using (3) and (6), we obtain that
d|∆t,m(s)|2
ds = −2 〈 ∆t,m(s),Ex∼µ ( ∂θf(ZΘL,M (s)(l;x), θl,m(s))pΘL,M (s)(l;x)− ∂θf(ZΘ(s)(t;x), θm(s; t))pΘ(s)(t;x) )〉 = −2 〈 ∆t,m(s),Ex∼µ ( ∂θf(ZΘL,M (s)(l;x), θl,m(s))pΘL,M (s)(l;x)− ∂θf(ZΘ(s)(t;x), θm(s; t))pΘL,M (s)(l;x)
)︸ ︷︷ ︸ (I)
〉
− 2 〈 ∆t,m(s),Ex∼µ ( ∂θf(ZΘ(s)(t;x), θm(s; t))pΘL,M (s)(l;x)− ∂θf(ZΘ(s)(t;x), θm(s; t))pΘ(s)(t;x)
)︸ ︷︷ ︸ (II)
〉 .
(130) To bound (I), we use (126) to obtain
|(I)| ≤ C(S,Rini,Ldis,supini )Ex∼µ (∣∣∂θf(ZΘL,M (s)(l;x), θl,m(s))− ∂θf(ZΘ(s)(t;x), θm(s; t))∣∣)
≤ C(S,Rini,Ldis,supini ) [ Ex∼µ (∣∣ZΘL,M (s)(l;x)− ZΘ(s)(t;x)∣∣+ |θl,m(s)− θm(s; t)|)] , (131)
where we use Assumption 4.1 (14), (100), (125), and (127) in the second inequality. To bound (II), we use (13), (100), and (127) to obtain
|(II)| ≤ C(S,Rini,Ldis,supini )(|θm(s; t)|+ 1)Ex∼µ (∣∣pΘL,M (s)(l;x)− pΘ(s)(t;x)∣∣) . (132)
By substituting (132) and (131) into (130), we obtain
d|∆t,m(s)|2
ds
≤ C(S,Rini,Ldis,supini )|∆t,m(s)| 2 + C(S,Rini,Ldis,supini )|∆t,m(s)|Ex∼µ (∣∣ZΘL,M (s)(l;x)− ZΘ(s)(t;x)∣∣)
+ C(S,Rini,Ldis,supini )|∆t,m(s)|(|θm(s; t)|+ 1)Ex∼µ (∣∣pΘL,M (s)(l;x)− pΘ(s)(t;x)∣∣) .
Using Hölder’s inequality similar to (107), we obtain d 1M ∑M m=1 |∆t,m(s)|2
ds
≤ C(S,Rini,Ldis,supini )
( 1
M M∑ m=1
|∆t,m(s)|2 )
+ C(S,Rini,Ldis,supini )Ex∼µ (∣∣pΘL,M (s)(t;x)− pΘ(s)(t;x)∣∣2)
+ C(S,Rini,Ldis,supini )Ex∼µ (∣∣ZΘL,M (s)(t;x)− ZΘ(s)(t;x)∣∣2) ,
(133)
By substituting (113) and (114) into (133), we obtain
d 1M ∑L−1 l=0 ∑M m=1 ∫ l+1 L l L |∆t,m(s)|2 dt
ds ≤ C(S,Rini,Ldis,supini )
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(s)|2 dt+ 1 L2 ) ,
which implies, from Grönwall’s inequality, that
1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(s)|2 dt ≤ C(S,Rini,Ldis,supini ) ( 1 M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(0)|2 dt+ 1 L2 ) . (134) We have thus established the bound (129). We also have
1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(0)|2 dt = 1 M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L ∣∣∣∣θm(0; lL ) − θm(0; t) ∣∣∣∣2 dt . (135) To complete the proof, we use (79) and take M ≥ 8C3η to obtain
P
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(0)|2 dt ≤ C4 L2 ) ≥ 1− η 8 .
According to (78), when M > 8C3η , P ( Ldis,supini ≤ C4 ) ≥ 1− η
8 . (136)
By using these expressions to substitute 1M ∑L−1 l=0 ∑M m=1 ∫ l+1 L l L
|∆t,m(0)|2 and Ldis,supini into (134), we find that there exists a constant C ′(C4,Rini, S) depending on C4,Rini, and S such that if
M ≥ 8C3 η , L ≥ C
′(C4,Rini, S)
,
then we have
P
( 1
M L−1∑ l=0 M∑ m=1 ∫ l+1 L l L |∆t,m(s)|2 dt ≤ 2 ) ≥ 1− η 4 . (137)
Using (137), (136), and (127) to bound the right hand side of (113), we find that there exists another constant C ′′(C4,Rini, S) depending on C4,Rini, and S such that if
M ≥ 8C3 η , L ≥ C
′′(C4,Rini, S)
,
then we have P (∣∣ZΘ(s)(1;x)− ZΘL,M (s)(L;x)∣∣ ≤ ) ≥ 1− η2 , By using this result and (136) in conjunction with (128), we complete the proof.

heading: H PROOF OF GLOBAL CONVERGENCE RESULT
text: Intuitively, if the equation (9) converges to a stationary point, denote by ρ∞, so that ∂sρ∞ = 0, then
∇θ δE
δρ ∣∣∣∣ ρ∞(·,·) (θ, t) = 0, ρ∞(θ, t)-a.e. θ ∈ Rk , a.e. t ∈ [0, 1] .
The rest of the analysis shows that E(ρ∞) = 0 when this happens. However, it is not direct because the condition above only suggests the fact that δEδρ |ρ∞(,̇)̇ is a piecewise constant function. We need a stronger result that shows this constant has to be zero. This is achieved by Proposition 6.1. To show this proposition, we follow the proof in (Lu et al., 2020) that explores the expressive power of f(x, θ), particularly the universal kernel property of Assumption 4.1. It is this proposition that identifies stationary points with the global minimizer.
We should mention that the zero loss was demonstrated by Chizat & Bach (2018) for the 2-layer problem where the stability equates to zero-loss due to convexity. The extension to the multi-layer case is more difficult since convexity is not present.

heading: H.1 PROOF OF PROPOSITION 6.1
text: We first prove a lower bound for pρ in the following lemma.
Lemma H.1 Suppose that ρ ∈ C([0, 1];P2) and that pρ is a solution to (11). Denoting Lρ = ∫ 1
0 ∫ Rk |θ|2dρ(θ, t) ,
then for any t ∈ [0, 1] we have that Ex∼µ ( |pρ(t;x)|2 ) ≥ Q(Lρ)E(ρ) , (138)
where Q : R+ → R+ is a decreasing function.
Proof Recall that the initial condition for pρ in (11) is:
pρ(1;x) = (g(Zρ(1;x))− y(x))∇g(Zρ(1;x)) ,
so from Assumption 4.1, we have
Ex∼µ ( |pρ(1;x)|2 ) ≥ (
inf x∈Rd
|∇g(x)| )2 E(ρ) .
Further, since the equation is linear, we have
∂p>ρ ∂t = −p>ρ ∫ Rk ∂zf(Zρ, θ)dρ(θ, t) .
According to equation (13) in Assumption 4.1, we obtain∣∣∣∣∫ Rk ∂zf(Zρ(t;x), θ) dρ1(θ, t) ∣∣∣∣ ≤ C(Zρ(t;x))∫ Rk (|θ|2 + 1) dρ(θ, t)
≤ C(Lρ) ∫ Rk (|θ|2 + 1) dρ1(θ, t) ,
where we use (18) in the second inequality. By combining the last two bounds in the usual way, we obtain
d|pρ(t;x)|2 dt ≤ ( 2C(Lρ) ∫ Rk (|θ|2 + 1)dρ(θ, t) ) |pρ(t;x)|2 ,
By solving the equation, we have |pρ(t;x)|2 ≥ |pρ(1;x)|2 exp ( −2C(Lρ) ∫ 1 t ∫ Rk (|θ|2 + 1)dρ(θ, t) ) ≥ C(Lρ)|pρ(1;x)|2 .
The proof is finalized by taking expectation on both sides, and note that monotonicity comes from the format of the exponential term.
We are now ready to prove Proposition 6.1.
Proof [Proof of Proposition 6.1] Denote Lρ = ∫ 1
0 ∫ Rk |θ|2 dρ(θ, t) dt .
According to existence and uniqueness of the solution to (7), for any t ∈ [0, 1], we can construct a map Zt such that Zt(x) = Zρ (t;x) . Since the trajectory can be computed backwards in time, Z−1t is well defined. Further, we denote µ∗t = (Zt)]µ to be the pushforward of µ under map Zt and let
p∗(t;x) = pρ ( t;Z−1t (x) ) .
By Assumption 4.1 and classical ODE theory, Zt and Z−1t are both continuous maps in x, and so are pρ(t;x) and p∗(t;x). With the change of variables, we have for all t ∈ [0, 1] that
δE(ρ)
δρ (θ, t) = ∫ Rd p>ρ (t;x)f(Zt(x), θ) dµ = ∫ Rd (p∗(t;x))>f(x, θ) dµ∗t . (139)
For a fixed t0 ∈ [0, 1], we have boundedness of the Jacobian from Lemma C.1, meaning that
sup x∈supp(µ) ∥∥∥∥∥ dµ∗t0(Z−1t (x))dµ(x) ∥∥∥∥∥
2
≤ C(Lρ).
As a consequence, µ∗t0(x) has a compact support since µ(x) does. We denote the size of the support by R∗, defined to be a real number such that supp ( µ∗t0(x) ) ⊂ {x : |x| < R∗}.
We now derive a general formula for ∫ δE(ρ)
δρ (θ, t) dν. Recalling (139), we have∫ Rk δE(ρ) δρ (θ, t0) dν(θ) = ∫ Rd (p∗(t0;x)) > (∫ Rk f(x, θ) dν(θ) ) dµ∗t0(x)
= ∫ Rd (p∗(t0;x)) > (∫ Rk f(x, θ) dν(θ) + p∗(x, t0) ) dµt0(x)
− ∫ Rd (p∗(t0;x)) >p∗(x, t0) dµ ∗ t0(x) .
(140)
Noticing that according to Lemma H.1, if E(ρ) 6= 0, the second term above is strictly negative (less than −Q(Lρ)E(ρ)), the goal then is to find ν for which ∫ dν = 0 that makes the first term small, so that the right-hand side in (140) is negative. Defining the continuous function h to be
h(x) = p∗ (t0;x) + ∫ Rk f(x, θ) dρ(θ, t0) ,
then according to Assumption 4.1, for arbitrarily small , there is a ν̂ so that ∫ ν̂ = 0 and∥∥∥∥h(x)− ∫
Rk f(x, θ) dν̂(θ) ∥∥∥∥ L∞|x|<R∗ ≤ .
Setting ν = ρ− ν̂ and substituting into the first term of (140), we obtain∫ Rd (p∗(t0;x)) > (∫ Rk f(x, θ) dν(θ) + p∗(x, t0) ) dµt0(x)
≤ ∫ Rd |p∗(t0;x)| ∣∣∣∣h(x)− ∫ Rk f(x, θ) dν̂(θ) ∣∣∣∣ dµ∗t0(x) ≤ ‖p∗(t0;x)‖L∞|x|<R∗ ∥∥∥∥h(x)− ∫ Rk f(x, θ) dν̂(θ) ∥∥∥∥ L∞|x|<R∗ .
(141)
By choosing small enough that (141) is less than 12Q(Lρ)E(ρ), we have from (140)∫ δE(ρ) δρ (θ, t0) dν(θ) < 0, completing the proof.
H.2 2-HOMOGENEOUS CASE: PROOF OF THEOREM 6.1
We first give an example of 2-homogeneous activation function that satisfy Assumption 4.1, and 6.1.
Remark H.1 A function that satisfies Assumption 4.1 and the 2-homogeneous property of Assumption 6.1 is f(x, θ) = f(x, θ[1], θ[2]) = σ(θ[1]x + θ[2]) exp(−|x|2), where θ[1] ∈ Rd×d, θ[2] ∈ Rd, and σ(x) = |max{x, 0}|2 applied componentwise.
Before proving the Theorem 6.1, we first introduce the following lemma, which shows that the separation property is preserved in the training process. Our proof of this result is adapted from (Chizat & Bach, 2018).
Lemma H.2 Let Assumptions 4.1 and 4.2 hold, and suppose that ρini(θ, t) is admissible with compact support. Let ρ(θ, t, s) ∈ C([0,∞); C([0, 1];P2)) solve (9). If there exists t0 ∈ [0, 1], so that the initial condition ρini(θ, t0) separates the spheres raSk−1 and rbSk−1 for some 0 < ra < rb, then for any s0 ∈ [0,∞), ρ(θ, t0, s0) separates the spheres r′aSk−1 and r′bSk−1 for some 0 < r′a < r′b.
Proof For every fixed 0 < s0 < S < ∞, we note that the particle representation θρ(s; t0) of ρ(θ, t0, s) updates the following equation: dθρ(s; t0) ds = −∇θ δE(ρ(s)) δρ (θρ(s; t0), t0) , s ∈ (0, S)
θρ(0; t0) = θ .
(142)
Define the map Ps(θ) to be the solution map that solves the equation above for given initial condition θ up to time s. Our proof amounts to showing that this map preserves the separation property. According to (Chizat & Bach, 2018, Proposition C.11), we need only show that the inverse map of Ps(θ) is stable near 0 for any fixed 0 < s < S. That is, for any > 0, we need to identify η > 0 such that P−1s (θ) ⊂ B (0) , ∀θ ∈ Bη (0) , (143) where Bη(0) is the k-dimensional ball around original 0 with radius η. Since f is 2-homogeneous in θ, we have that |∂θf(z, 0)| = 0 for all z. Thus, from (10),∣∣∣∣∇θ δE(ρ(s))δρ (0, t0)
∣∣∣∣ = 0. Using estimate (49) from Lemma C.4, we obtain∣∣∣∣∇θ δE(ρ(s))δρ (θ, t0)
∣∣∣∣ ≤ C(LsupS )|θ| , where LsupS = sup0≤s≤S,t∈[0,1] ∫ Rk |θ|
2 dρ(θ, t, s) dt. This upper bound on the velocity implies in particular that
|P−1s (θ)| ≤ |θ| exp(C(L sup S )s) ,
which establishes (143) when we choose η to satisfy η < exp(−C(LsupS )s), concluding the proof.
We are now ready to prove Theorem 6.1.
Proof [Proof of Theorem 6.1] Since ρ(θ, t, s) converges to ρ∞(θ, t) in C([0, 1];P2), we have for any t0 that
sup s≥0 ∫ Rk |θ|2 dρ(θ, t0, s) <∞ . (144)
According to Proposition 6.1, it suffices to prove that
δE(ρ∞)
δρ (θ, t0) = 0, ∀θ ∈ Rk . (145)
We use a contradiction argument: We will assume that (145) is not satisfied and show that∫ Rk |θ|
2 dρ(θ, t0, s) blows up to infinity as s → ∞, contradicting (144). In particular, we will use homogeneity to construct a set in which the second moment blows up.
Define the functions h∞ and hs as follows:
h∞(θ) = δE(ρ∞)
δρ (θ, t0), hs(θ) =
δE(ρ(s))
δρ (θ, t0) .
Recall from (10) that
δE(ρ)
δρ (θ, t0) = Eµ
( p>ρ (t0, x)f(Z(t0;x), θ) ) . (146)
Since (145) is not satisfied, there exists a θ∗ such that δE(ρ∞)δρ (θ ∗, t0) 6= 0. From (146), by Hölder’s inequality,
0 < ∣∣∣∣δE(ρ∞)δρ (θ∗, t0) ∣∣∣∣ ≤ (Ex∼µ (|pρ∞(t0;x)|2))1/2 (Ex∼µ (|f(Z(t0;x), θ∗)|2))1/2 ,
which implies Ex∼µ ( |pρ∞(t0;x)|2 ) > 0 . Then, Since f is a universal kernel according to Assumption 4.1, we can find ν such that∫ f(Z(t0;x), θ) dν approximates −pρ∞(t0, x). leading to∫
Rk h∞(θ) dν(θ) = ∫ Rk δE(ρ∞) δρ (θ, t0) dν(θ) < − 1 2 Ex∼µ ( |pρ∞(t0;x)|2 ) < 0 .
As a consequence, there exists at least one point θ0 ∈ Rk such that h∞(θ0) < 0. Since f is 2-homogeneous, by (10), h is also 2-homogeneous, so that
h∞(θ0/|θ0|) < 0 .
Because of continuity, there is a small neighborhood around θ0/|θ0| in Sk−1 where h is strictly negative. Moreover, since h is Sard-type regular, there exist > 0 and η > 0 such thatA = { θ̃ ∈ Sk−1 ∣∣∣h∞|Sk−1 (θ̃) < − } 6= ∅ , ∇θ̃h∞|Sk−1 ( θ̃ ) · nθ̃ > η , ∀ θ̃ ∈ ∂A ,
where h∞|Sk−1 is the confinement of h∞ on Sk−1, and nθ̃ is the outer normal vector to ∂A. This statement of h∞ can be extended to hs for sufficiently larger s as well. Using estimate (48) from Lemma C.4, we obtain that
hs(θ)→ h∞(θ) in C1loc(Rk), as s→∞ ,
meaning there exists S > 0 such that for any s ≥ S, we havehs|Sk−1 ( θ̃ ) < − /2, ∀ θ̃ ∈ A , ∇θ̃hs|Sk−1 ( θ̃ ) · nθ̃ > 1 2η , ∀ θ̃ ∈ ∂A .
Extending this patch on the unit sphere to the whole domain, we define the cone: A = { θ ∈ Rk ∣∣|θ| > 0, θ/|θ| ∈ A} . Using the 2-homogeneous property of hs, we have for s ≥ S that{
hs(θ) < − |θ| 2
2 , ∀θ ∈ A , ∇θhs(θ) · ~nθ > 0 , ∀θ ∈ ∂A ∩ {|θ| > 0} ,
(147)
where ~nθ is the outer normal vector to ∂A.
We now define a new system that follows the gradient flow corresponding to ρs. Denote by θ̂ (s;α) the solution to the following ODE: dθ̂ (s;α) ds = −∇θ δE(ρ(s)) δρ ( θ̂ (s;α) , t0 ) = −∇θhs ( θ̂ (s;α) ) , s > S
θ̂ (S;α) = α,
(148)
where α ∈ Rk. According to (147), when θ ∈ ∂A∩{|θ| > 0},∇θhs(θ) points outwards, away from A. We also notice that θ̂ ( s;~0 )
= ~0. Thus if the ODE starts with from some α ∈ A, then for any s ≥ S, the particle stays within A, that is,
θ̂ (s;α) ∈ A . (149) As a consequence, we have
d ∣∣∣θ̂ (s;α)∣∣∣2
ds = −2
〈 θ̂ (s;α) ,∇θhs ( θ̂ (s;α) )〉 = −4hs ( θ̂ (s;α) ) > 2 ∣∣∣θ̂ (s;α)∣∣∣2 , (150) where we use the 2-homogeneous property of hs in the second equality and (147) in the final inequality.
According to Lemma H.2, there exist two spheres separated by ρ(θ, t0, S), meaning that there exist β > 0 and γ > 0 relatively small (for example, with β < r′a) such that∫
A∩(Bβ(~0)) c
dρ(θ, t0, S) > γ . (151)
By tracing the trajectory of (148), we have∫ A∩(Bβ(~0)) c dρ(θ, t0, s) = ∫ Rk 1θ̂(s;α)∈A∩(Bβ(~0))c dρ(α, t0, S)
≥ ∫ A∩(Bβ(~0)) c dρ(α, t0, S) > γ, s ≥ S ,
where in the first inequality we also use d|θ̂(s;α)|2
ds ≥ 0 when α ∈ A. Further, we have
d ∫ A∩(Bβ(~0)) c 1θ̂(s;α)∈A∩(Bβ(~0))c ∣∣∣θ̂ (s;α)∣∣∣2 dρ(α, t0, S)
ds = d ∫ A∩(Bβ(~0)) c ∣∣∣θ̂ (s;α)∣∣∣2 dρ(α, t0, S) ds
≥ 2 ∫ A∩(Bβ(~0)) c ∣∣∣θ̂ (s;α)∣∣∣2 dρ(α, t0, S) ≥ 2 γβ2
where we use (150) in the second inequality and (151) in the final inequality. It follows that
lim s→∞ ∫ A∩(Bβ(~0)) c 1θ̂(s;α)∈A∩(Bβ(~0))c ∣∣∣θ̂ (s;α)∣∣∣2 dρ(α, t0, S) =∞ . It follows from this result that
lim s→∞ ∫ Rk |θ|2 dρ(θ, t0, s) =∞ ,
contradicting (144). Therefore, we must have
δE(ρ∞)
δρ (θ, t0) = 0, ∀θ ∈ Rk ,
which completes the proof.

heading: H.3 PARTIALLY 1-HOMOGENEOUS CASE: PROOF OF THEOREM 6.2
text: We first give an example of partially 1-homogeneous activation function that satisfy Assumption 4.1, and 6.2.
Remark H.2 The following function satisfies Assumptions 4.1 and 6.2: Let θ = (θ[1], θ[2], θ[3]) with θ[1] ∈ Rd, θ[2] ∈ Rd×d, θ[3] ∈ Rd. Define f(x, θ) = f(x, θ[1], θ[2], θ[3]) = θ[1]σ ( θ[2]σ2(|θ[2]|) |θ[2]| x+ θ[3]σ2(|θ[3]|) |θ[3]| ) , where σ(x) is a regularized ReLU activation function, and σ2, σ2(x)/x : R+ → R+ are bounded, Lipschitz, and differentiable with Lipschitz differential. One way (of many) to define a regularized ReLU activation function is σ(x) = (x+ η)2/(4η)1x∈[−η,η] + x1x∈(η,∞), for some small η.
As in the previous theorem, we prove a lemma, adapted from (Chizat & Bach, 2018, Lemma C.13), that asserts preservation of the separation property.
Lemma H.3 Let Assumptions 4.1 and 4.2 with k1 = 1. Suppose that ρini(θ, t) is admissible and suppθ(ρini(θ, t)) ⊂ {θ||θ[1]| ≤ R} with some R > 0 for all t ∈ [0, 1]. Let ρ(θ, t, s) ∈ C([0,∞); C([0, 1];P2)) solve (9). Suppose in addition that
• f satisfies the partial 1-homogeneous condition (see Assumption 6.2),
• The initial conditions satisfy the separation condition, that is, there exists t0 ∈ [0, 1] such that ρini(θ[1], θ[2], t0) separates the spheres {−r0} × Rk−1 and {r0} × Rk−1 for some r0 > 0.
Then for any s0 ∈ [0,∞), ρ(θ[1], θ[2], t0, s0) separates {−r′} × Rk−1 and {r′} × Rk−1 for some r′ > 0.
Proof Note that the particle representation θρ(s; t0) of ρ(θ, s, t0) satisfies{ dθρ(s;t0)
ds = −∇θ δE(ρ(s)) δρ (θρ(s; t0), t0) ,
θρ(0; t0) = θ . (152)
Define a continuous map P : Rk × [0,∞) → Rk as the solution to (152), that is, P(θ, s) is the solution to (152) with initial condition θρ(0; t0) = θ, where t0 is fixed. Define a diffeomorphism ψ : R× Rk−1 → R× B1 (0) as follows:
ψ(θ[1], θ[2]) =
{( θ[1], ( θ[2]/|θ[2]| ) · tanh ( |θ[2]| )) , θ[2] 6= 0
(θ[1], 0), θ[2] = 0 ,
where θ[1] ∈ R is the first component of θ and θ[2] ∈ Rk−1 contains the remaining components. This map keeps the first component of θ[1] intact and shrinks θ[2] to push its amplitude below 1. This diffeomorphism preserves the connection/separation property.
Define the continuous map Q as follows:
Q(θ, s) = ψ ◦ P(ψ−1(θ), s) : R× B1 (0)× [0,∞)→ R× B1 (0) .
Since ψ preserves the connection property, the lemma is proved if we can show ψ ◦ P(supp(ρini), s) separates {−r′} × B1 (0) and {r′} × B1 (0) for some r′ > 0. Since ψ ◦ P(supp(ρini), s) = Q(ψ(supp(ρini)), s), we trace the evolution of Q(θ, s) for θ ∈ R× B1(0). According to (Chizat & Bach, 2018, Proposition C.14), this translates to showing Q(θ, s) can be continuously extended to R× B1 (0)× [0, S]→ R× B1 (0), with the extension satisfying
Q(θ, s) ∈ R× ∂B1 ( ~0 ) , ∀θ ∈ R× ∂B1 ( ~0 ) , s ∈ [0,∞) , (153)
meaning that the extension Q(·, s) maps R× ∂B1(~0) to itself for all s ∈ [0,∞).
DenotingQs(θ) = Q(θ, s), we consider the velocity field of this flow (similar to the proof of (Chizat & Bach, 2018, Lemma C.13)):
dQs ds
= ( ∇θψ ( Ps ◦ ψ−1 (θ) )) dPs ◦ ψ−1 ds
= − ( ∇θψ ( Ps ◦ ψ−1 (θ) ))( ∇θ δE(ρ(s))
δρ
( Ps ◦ ψ−1 (θ) , t0 )) = − ( ∇θψ ( ψ−1(Qs) ))( ∇θ δE(ρ(s))
δρ
( ψ−1(Qs), t0 )) = V (Qs, s) .
From the fourth condition of Theorem 6.2, the velocity field V (θ, s) can be continuously extended to R× ∂B1 (0) as follows:
V (θ, s) = V (θ[1], θ[2], s) =
{ − ( ∇θψ ( ψ−1(θ) )) ( ∇θ δE(ρ(s))δρ ( ψ−1(θ), t0 )) , |θ[2]| < 1(
−H∞,ρ(s)(θ[2]), 0 ) , |θ[2]| = 1 ,
where H∞,ρ(s) is the limit of δE(ρ(s))
δρ
( 1, rθ[2], t0 ) as r →∞. Within this velocity field, Qs can be
continuously extended to R × B1 (0) × [0,∞) → R × B1 (0) with the extension satisfying (153). This completes the proof.
We are now ready to prove Theorem 6.2.
Proof [Proof of Theorem 6.2] The technique of proof is similar to the 2-homogeneous case. Since ρ(θ, t, s) converges to ρ∞(θ, t) in C([0, 1];P2), we have
sup s≥0 ∫ Rk |θ|2dρ(θ, t0, s) <∞ . (154)
According to Proposition 6.1, it suffices to prove that
δE(ρ∞)
δρ (θ, t0) =
δE(ρ∞)
δρ (θ[1], θ[2], t0) = θ[1]
δE(ρ∞)
δρ (1, θ[2], t0) = 0, ∀θ ∈ Rk . (155)
In the following, we will show that ∫ Rk |θ|
2dρ(θ, t0, s) blows up as s→∞ if (155) fails to hold, in contradiction to (154).
Denote
h∞(θ[2]) = δE(ρ∞)
δρ (1, θ[2], t0), hs(θ[2]) =
δE(ρ(s))
δρ (1, θ[2], t0) . (156)
Then if (155) is not satisfied, without loss of generality, we can assume that there exists θ[2] such that h∞(θ[2]) < 0. Since h∞ satisfies Sard-type regularity, there exist > 0 and η > 0 such that
A = { θ[2] ∈ Rk−1 ∣∣h∞ (θ[2]) < − } 6= ∅; ∇θ[2]h∞(θ[2]) · ~nθ[2] > η , ∀ θ[2] ∈ ∂A , where nθ[2] is the outer normal vector on ∂A.
Using the definition of δEδρ as in (10), we have∣∣∣∣∇θ[2] δE(ρ(s))δρ (1, θ[2], t)−∇θ[2] δE(ρ∞)δρ (1, θ[2], t) ∣∣∣∣
= Ex∼µ ( ∂θ[2] f̂(Zρ(s)(t;x), θ[2])pρ(s)(t;x)− ∂θ[2] f̂(Zρ∞(t;x), θ[2])pρ∞(t;x) ) ≤ Ex∼µ
(∣∣∣∂θ[2] f̂(Zρ(s)(t;x), θ[2])− ∂θ[2] f̂(Zρ∞(t;x), θ[2])∣∣∣ |pρ(s)(t;x)|) + Ex∼µ
(∣∣∣∂θ[2] f̂(Zρ∞(t;x), θ[2])∣∣∣ ∣∣pρ(s)(t;x)− pρ∞(t;x)∣∣) ≤ C ( Ex∼µ
(∣∣Zρ(s)(t;x)− Zρ∞(t;x)∣∣)+ Ex∼µ (∣∣pρ(s)(t;x)− pρ∞(t;x)∣∣)) ≤ Cd1(ρ(s), ρ∞) ,
where we have used the Lipschitz continuity of f and its derivatives as in Assumption 6.2 and the estimates (18), (31), and (47). We thus have hs(θ[2])→ h∞(θ[2]) in C1(Rk−1), as s→∞ . (157) As a consequence, there exists S > 0 such that for any s ≥ S{
hs ( θ[2] ) < − /2, ∀ θ[2] ∈ A , ∇θ[2]hs(θ[2]) · ~nθ[2] > 12η , ∀ θ[2] ∈ ∂A . (158)
Extending this set to the whole space, we define A = { (θ[1], θ[2]) ∈ (0,∞)×A } .
Since ∂A = { (θ[1], θ[2]) ∈ (0,∞)× ∂A } ∪ {θ[1] = 0, θ[2] ∈ A}, from (158) and the definition of hs, we have
∇θ δE(ρ(s))
δρ (θ, t0) · ~nθ = θ[1]
( ∇θ[2]hs(θ[2]) · ~nθ[2] ) > 0 , ∀θ ∈ (0,∞)× ∂A , (159)
where ~nθ is the outer normal direction on ∂A, and ~nθ[2] is the outer normal vector on ∂A. When θ ∈ {θ[1] = 0, θ[2] ∈ A},[ ∇θ δE(ρ(s))
δρ (θ, t0) ] 1 = hs(θ[2]) < 0, [ ∇θ δE(ρ(s)) δρ (θ, t0) ] i = 0, i = 2, . . . , k , (160)
where [·]i means the i-component of the vector. This implies that ∇θ δE(ρ(s))δρ (θ, t0) points strictly downward when θ ∈ {θ[1] = 0, θ[2] ∈ A}.
As in the 2-homogeneous case, we consider the gradient flow corresponding to ρs. Denote by θ̂ (s;α) the solution to the following ODE:{
dθ̂(s;α) ds = −∇θ δE(ρ(s)) δρ ( θ̂ (s;α) , t0 ) = −∇θ ( θ[1]hs ( θ[2] )) ( θ̂ (s;α) ) , s > S
θ̂ (S;α) = α , (161)
where α ∈ Rk. Since the minus gradient is pointing inward toA, as stated in (159) and (160), θ̂ (s;α) stays in A if θ̂ (s′;α) ∈ A for some s′ ∈ [S, s]. Moreover, using (161), if θ̂ (s;α) ∈ A, we have
d|θ̂[1] (s;α) |2
ds = −2θ̂[1] (s;α)hs
( θ̂[2] (s;α) ) > θ̂[1] (s;α) , (162)
where the last inequality uses (158).
Similar to (Chizat & Bach, 2018, Proposition C.4), we claim that there exists S1 ≥ S, β > 0, and γ > 0 such that (see detailed proof in Appendix H.3.1)∫
(β,∞)×A dρ(θ, t0, S1) > γ . (163)
Then d ∫ Rk 1θ̂(S1;α)∈(β,∞)×A ∣∣∣θ̂[1] (s;α)∣∣∣2 dρ(α, t0, S) ds
≥ ∫ Rk 1θ̂(S1;α)∈(β,∞)×Aθ̂[1] (s;α) dρ(α, t0, S)
≥ β ∫ Rk 1θ̂(S1;α)∈(β,∞)×A dρ(α, t0, S)
= β ∫ (β,∞)×A dρ(θ, t0, S1) ≥ βγ .
(164)
Since
lim s→∞ ∫ (β,∞)×A |θ[1]|2 dρ(θ, t0, s) ≥ lim s→∞ ∫ Rk 1θ̂(S1;α)∈(β,∞)×A ∣∣∣θ̂[1] (s;α)∣∣∣2 dρ(α, t0, S) =∞ ,
we finally obtain, using (164), that
lim s→∞ ∫ Rk |θ|2 dρ(θ, t0, s) ≥ lim s→∞ ∫ (β,∞)×A |θ[1]|2 dρ(θ, t0, s) =∞ .
This limit contradicts (154), implying that (155) must hold, as claimed.

heading: H.3.1 CLAIM IN THE PROOF OF THEOREM 6.2
text: In this section, we prove the statement in (163), meaning that we need to find S1 ≥ S, β > 0, and γ > 0 such that ∫
(β,∞)×A dρ(θ, t0, S1) > γ . (165)
Supposing that ∫ A dρ(θ, t0, S) > 0, then by making β and γ small enough, (165) is satisfied naturally.
If ∫ A dρ(θ, t0, S) = 0, then it suffices to show that there exists S1 > S such that∫
A dρ(θ, t0, S1) > 0 . (166)
Define hs(θ[2]) and h∞(θ[2]) as in (156). Because of the fourth condition in Theorem 6.2, there exists a function h′(θ̃) on C1 ( Sk−2 ) such that
h∞(rθ̃) r→∞−−−→ h′(θ̃), in C1(Sk−2 .
Combining this with (157), there exists h∗ > /2 such that ‖hs‖L∞A ≤ h ∗. Further, for any ξ > 0, we can find θ∗[2] ∈ A and S ′ large enough such that
|∇θ∗ [2] hs(θ ∗ [2])| < ξ , ∀s ≥ S ′ . (167)
According to Lemma H.3, there exists rS′ > 0 such that ρ(θ[1], θ[2], t0, S′) separates {−rS′} × Rk−1 and {rS′} × Rk−1. Considering the set [−rS′ , rS′ ]× {θ∗[2]}, it must intersect the support of ρ(θ[1], θ[2], t0, S
′) due to the separation property. Thus, any open set that contains [−rS′ , rS′ ]×{θ∗[2]} must have a positive measure in ρ(θ[1], θ[2], t0, S′). Because ∫ A dρ(θ, t0, S
′) = 0 and (−rS′ − 1,∞)×A is a open set that covers [−rS′ , rS′ ]×{θ∗[2]}, there exists a open set U ⊂ (−rS′−1, 0]×A such that ∫
U
dρ(θ[1], θ[2], t0, S ′) > 0 .
Thus, we can find a point 0 < r∗ ≤ rS′ + 1 and an arbitrary small σ > 0 such that Bσ(−r∗, θ∗[2]) ⊂ (−rS′ − 1− σ, σ)×A, and ∫ Bσ(−r∗,θ∗[2]) dρ(θ, t0, S ′) > 0 .
Recalling the system (161), we claim the following:
When ξ, σ are small enough, there exists S1 > S′ such that θ̂ (S1;α) ∈ A for any α ∈ Bσ(−r∗, θ∗[2]).
If this claim is true, then∫ A dρ(θ, t0, S1) ≥ ∫ Bσ(−r∗,θ∗[2]) dρ(θ, t0, S ′) > 0 .
which proves (166) and the lemma.
Now, we prove the claim. Because f satisfies Assumption 6.2 and the second moment of ρ is uniformly bounded in s, for all s > 0, we have∣∣∣∣δE(ρ(s))δρ (1, θ[2], t0)− δE(ρ(s))δρ (1, θ′[2], t0) ∣∣∣∣ ≤ L|θ[2] − θ′[2]| ,∣∣∣∣∇θ δE(ρ(s))δρ (1, θ[2], t0)−∇θ δE(ρ(s))δρ (1, θ′[2], t0) ∣∣∣∣ ≤ L|θ[2] − θ′[2]| , (168) for some constant L. According to (161), we have dθ̂[1](s;α) ds = −hs ( θ̂[2](s;α) ) d ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣2
ds ≤ 2
∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣ ∣∣∣θ̂[1](s;α)∣∣∣ ∣∣∣∇θ[2]hs (θ̂[2](s;α))∣∣∣ , s ≥ S′ (169)
where −r∗ − σ ≤ θ̂[1](S′;α) ≤ σ, and ∣∣∣θ̂[2](S′;α)− θ∗[2]∣∣∣ ≤ σ . (170) To prove the claim, it suffices to show that there exists S1 > S′ such that
θ̂[1] (S1;α) > 1 and θ̂[2] (S1;α) ∈ A . (171)
We first show that θ̂[1] increases and the right hand-side of (169) is bounded. Since A is a open set, there exists an arbitrary small Σ > 0 such that BΣ(θ∗[2]) ⊂ A. We first choose σ < min{Σ, 2}. When∣∣∣θ̂[1](s;α)∣∣∣ ≤ 2rS′ + 2 h∗ + 2,
∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣ ≤ Σ , (172) we have from (158), (167), and (168) that
dθ̂[1](s;α)
ds = −hs
( θ̂[2](s;α) ) < h∗, dθ̂[1](s;α)
ds = −hs
( θ̂[2](s;α) ) > /2 (173)
and d ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣2
ds ≤ 2 ( 2rS′ + 2 h∗ + 2 ) ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣ (L ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣+ ξ) ≤ 2L ( 2rS′ + 2 h∗ + 2 ) ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣2 + 2(2rS′ + 2 h∗ + 2 ) Σξ .
(174)
When s = S′ and α ∈ Bσ(−r∗, θ∗[2]), we have from (170) that
|θ̂[1](S′;α)| ≤ rS′ + 1 + σ < 2rS′ + 2 h∗ + 2, |θ̂[2](S′;α)− θ∗[2]| ≤ σ < Σ .
Thus, for s slightly larger than S′, we still have that∣∣∣θ̂[1](s;α)∣∣∣ < 2rS′ + 2 h∗ + 2 and ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣ < Σ. Denote by S∗ the first time that∣∣∣θ̂[1](S∗;α)∣∣∣ ≥ 2rS′ + 2 h∗ + 2 or ∣∣∣θ̂[2](S∗;α)− θ∗[2]∣∣∣ ≥ Σ.
Then we show that there exists S1 ∈ [S′, S∗] such that (171) is satisfied when σ, Σ, and ξ are small enough. From (170), we have for s ∈ (S′, S∗) that
θ̂[1](s;α) ≤ σ + (s− S′)h∗
θ̂[1](s;α) > −rS′ − σ + (s− S′) /2,∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣ < exp(L(2rS′ + 2 h∗ + 2 ) (s− S′) )( σ2 + 2(s− S′) ( 2rS′ + 2 h∗ + 2 ) Σξ )1/2 ,
where the first two inequlaties come from (173) and last inequality comes from (174) via Grönwall’s inequality. Defining
S1 = 2rS′ + 2 + S′,
we can choose the positive values σ, Σ, and ξ small enough that
θ̂[1](S1, θ S′) > −rS′ + (S1 − S′) /2 = 1 ,
and for s ∈ [S′, S1] ∣∣∣θ̂[1](s;α)∣∣∣ < 2rS′ + 2 h∗ + 2, ∣∣∣θ̂[2](s;α)− θ∗[2]∣∣∣ < Σ . (175)
According to (175), the bounds (172) are satisfied for s ∈ [S′, S1], which implies that S1 < S∗ and θ̂[2](S1, θS ′ ) ∈ A. Further, we have θ̂[1](S1, θS ′ ) > 1. By combining these two results, we conclude that (171) is satisfied with the chosen values of σ, Σ, ξ, and S1. Thus, we have θ̂ (S1;α) ∈ (1,∞)×A ⊂ A for any α ∈ Bσ(−r∗, θ∗[2]), which proves the claim.

Review summaries:
The paper proves a global convergence result for GD in resnets, whose residual blocks are mean field two-layer neural nets. This is taken with a double limit, where the depth limit is the neural ODE and the width limit is the two-layer MF limit. To prove global convergence of the infinite-depth/width limit, the paper employs previous techniques that assume Sard-type regularities and the accompanying stability argument from Chizat and Bach 2018. 

This work analyzes infinitely deep and infinitely wide residual networks. The infinite depth limit is the neural ODE. The infinite width limit is the mean-field limit. The main contribution establishing global convergence of the infinitely wide and deep ResNet. Finally, the global convergence of the infinite limit is translated to the finite but large setup.

This paper studies the gradient flow training dynamics for deep and wide ResNet with the mean-field scaling. For a similar setting as (Lu et al., 2020), they report a limiting model with finite-dimensional approximation, and its global convergence result. The authors also justify the well-posedness of the resulting distributional dynamics, which seems only heuristically defined in prior work.