Title: Entropy-dissipation Informed Neural Network for McKean-Vlasov Type PDEs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Entropy-dissipation

McKean-Vlasov

Navier-Stokes

PDE

Coulomb

singular interaction
Abstract: The McKean-Vlasov equation (MVE) describes the collective behavior of particles subject to drift, diffusion, and mean-field interaction. In physical systems, the interaction term can be singular, i.e. it diverges when two particles collide. Notable examples of such interactions include the Coulomb interaction, fundamental in plasma physics, and the Biot-Savart interaction, present in the vorticity formulation of the 2D Navier-Stokes equation (NSE) in fluid dynamics. Solving MVEs that involve singular interaction kernels presents a significant challenge, especially when aiming to provide rigorous theoretical guarantees. In this work, we propose a novel approach based on the concept of entropy dissipation in the underlying system. We derive a potential function that effectively controls the KL divergence between a hypothesis solution and the ground truth. Building upon this theoretical foundation, we introduce the Entropy-dissipation Informed Neural Network (EINN) framework for solving MVEs. In EINN, we utilize neural networks (NN) to approximate the underlying velocity field and minimize the proposed potential function. By leveraging the expressive power of NNs, our approach offers a promising avenue for tackling the complexities associated with singular interactions. To assess the empirical performance of our method, we compare EINN with SOTA NN-based MVE solvers. The results demonstrate the effectiveness of our approach in solving MVEs across various example problems.

Entropy-dissipation Informed Neural Network for McKean-Vlasov Type PDEs

Abstract

The McKean-Vlasov equation (MVE) describes the collective behavior of particles subject to drift, diffusion, and mean-field interaction. In physical systems, the interaction term can be singular, i.e. it diverges when two particles collide. Notable examples of such interactions include the Coulomb interaction, fundamental in plasma physics, and the Biot-Savart interaction, present in the vorticity formulation of the 2D Navier-Stokes equation (NSE) in fluid dynamics. Solving MVEs that involve singular interaction kernels presents a significant challenge, especially when aiming to provide rigorous theoretical guarantees. In this work, we propose a novel approach based on the concept of entropy dissipation in the underlying system. We derive a potential function that effectively controls the KL divergence between a hypothesis solution and the ground truth. Building upon this theoretical foundation, we introduce the Entropy-dissipation Informed Neural Network ( EINN ) framework for solving MVEs. In EINN , we utilize neural networks (NN) to approximate the underlying velocity field and minimize the proposed potential function. By leveraging the expressive power of NNs, our approach offers a promising avenue for tackling the complexities associated with singular interactions. To assess the empirical performance of our method, we compare EINN with SOTA NN-based MVE solvers. The results demonstrate the effectiveness of our approach in solving MVEs across various example problems.

1 Introduction

Scientists use Partial Differential Equations (PDEs) to describe natural laws and predict the dynamics of real-world systems. As PDEs are of fundamental importance, a growing area in machine learning is the use of neural networks (NN) to solve these equations [Han et al., 2018, Zhang et al., 2018, Raissi et al., 2020, Cai et al., 2021, Karniadakis et al., 2021, Cuomo et al., 2022]. An important category of PDEs is the McKean-Vlasov equation (MVE), which models the dynamics of a stochastic particle system with mean-field interactions

Here X t ∈ X denotes a random particle' position, X is either R d or the torus Π d (a cube [ -L,L ] d with periodic boundary condition), V ∶ R d → R denotes a known potential, K ∶ R d → R d denotes some interaction kernel and the convolution operation is defined as h ∗ φ = ∫ X h ( x -y ) φ ( y ) d y , { B t } t ≥ 0 is the standard d -dimensional Wiener process with ν ≥ 0 being the diffusion coefficient, and ¯ ρ t ∶ X → R is the law or the probability density function of the random variable X t and the initial data ¯ ρ 0 is given. Under mild regularity conditions, the density function ¯ ρ t satisfies the MVE

where div denotes the divergence operator, div h ( x ) = ∑ d i = 1 ∂ h i / ∂ x i for a velocity field h ∶ R d → R d , ∆ denotes the Laplacian operator defined as ∆ φ = div ( ∇ φ ) , where ∇ φ denotes the gradient of a scalar function φ ∶ R d → R . Note that all these operators are applied only on the spatial variable x .

In order to describe dynamics in real-world phenomena such as electromagnetism [Golse, 2016] and fluid mechanics [Majda et al., 2002], the interaction kernels K in the MVE can be highly singular , i.e. ∥ K ( x ) ∥ → ∞ when ∥ x ∥ → 0 . Two of the most notable examples are the Coulomb interactions with S d -1 ( 1 ) denoting the surface area of the unit sphere in R d , and the vorticity formulation of the 2D Navier-Stokes equation (NSE) where the interaction kernel K is given by the Biot-Savart law

where x = ( x 1 , x 2 ) and ∥ x ∥ denotes the Euclidean norm of a vector. Classical methods for solving MVEs, including finite difference, finite volume, finite element, spectral methods, and particle methods, have been developed over time. A common drawback of these methods lies in the constraints of their solution representation: Sparse representations, such as less granular grids, cells, meshes, fewer basis functions, or particles, may lead to an inferior solution accuracy; On the other hand, dense representations incur higher computational and memory costs.

As a potent tool for function approximation, NNs are anticipated to overcome these hurdles and handle higher-dimensional, less regular, and more complex systems efficiently [Weinan et al., 2021]. The most renowned NN-based algorithm is the Physics Informed Neural Network (PINN) [Raissi et al., 2019]. The philosophy behind the PINN method is that solving a PDE system is equivalent to finding the root of the corresponding differential operators. PINN tackles the latter problem by directly parameterizing the hypothesis solution with an NN and training it to minimize the L 2 functional residual of the operators. As a versatile PDE solver, PINN may fail to exploit the underlying dynamics of the PDE, which possibly leads to inferior performance on task-specific solvers. For example, on the 2D NSE problem, a recent NN-based development Zhang et al. [2022] surpasses PINN and sets a new SOTA empirical performance, which however lacks rigorous theoretical substantiation. Despite the widespread applications of PINN, rigorous error estimation guarantees are scarce in the literature. While we could not find results on the MVE with the Coulomb interaction, only in a very recent paper [De Ryck et al., 2023], the authors establish for NSE that the PINN loss controls the discrepancy between a candidate solution and the ground truth. We highlight that their result holds average-in-time , meaning that at a particular timestamp t ∈ [ 0 , T ] , a candidate solution with small PINN loss may still significantly differ from the true solution. In contrast, all guarantees in this paper are uniform-in-time . Moreover, there is a factor in the aforementioned guarantee that exponentially depends on the total evolving time T , while the factor in our guarantee for the NSE is independent of T . We highlight that these novel improvements are achieved for the proposed EINN framework since we take a completely different route from PINN: Our approach is explicitly designed to exploit the underlying dynamics of the system, as elaborated below.

Our approach Define the operator

By noting ∆ ¯ ρ t = div ( ¯ ρ t ∇ log ¯ ρ t , we can rewrite the MVE in the form of a continuity equation

For simplicity, we will refer to A [ ¯ ρ t ] as the underlying velocity . Consider another time-varying hypothesis velocity field f ∶ R × R d → R and let ρ f t be the solution to the continuity equation

for t ∈ [ 0 , T ] , where we recall that the initial law ¯ ρ 0 is known. We will refer to ρ f t as the hypothesis solution and use the superscript to emphasize its dependence on the hypothesis velocity field f . We propose an Entropy-dissipation Informed Neural Network framework ( EINN ), which trains an NN parameterized hypothesis velocity field f θ by minimizing the following EINN loss

The objective (8) is obtained by studying the stability of carefully constructed Lyapunov functions. These Lyapunov functions draw inspiration from the concept of entropy dissipation in the system, leading to the name of our framework. We highlight that we provide a rigorous error estimation guarantee for our framework for MVEs with singular kernels (3) and (4), showing that when R ( f θ ) is sufficiently small, ρ f θ t recovers the ground truth ¯ ρ t in the KL sense, uniform in time.

Theorem 1 (Informal) . Suppose that the initial density function ¯ ρ 0 is sufficiently regular and the hypothesis velocity field f t ( ⋅ ) = f ( t, ⋅ ) is at least three times continuously differentiable both in t and x . We have for the MVE with a bounded interaction kernel K or with the singular Coulomb (3) or Biot-Savart (4) interaction, the KL divergence between the hypothesis solution ρ f t and the ground truth ¯ ρ t is controlled by the EINN loss for any t ∈ [ 0 , T ] , i.e. there exists some constant C > 0 ,

Having stated our main result, we elaborate on the difference between EINN and PINN in terms of information flow over time, which explains why EINN achieves better theoretical guarantees: In PINN, the residuals at different time stamps are independent of each other and hence there is no information flow from the residual at time t 1 to the one at time t 2 ( > t 1 ) . In contrast, in the EINN loss (8), incorrect estimation made in t 1 will also affect the error at t 2 through the hypothesis solution ρ f t . Such an information flow gives a stronger gradient signal when we are trying to minimize the EINN loss, compared to the PINN loss. It partially explains why we can obtain the novel uniform-in-time estimation as opposed to the average-in-time estimation for PINN and why the constant C in the NSE case is independent of T for EINN (Theorem 2), but exponential in T for PINN.

Contributions. In summary, we present a novel NN-based framework for solving the MVEs. Our method capitalizes on the entropy dissipation property of the underlying system, ensuring robust theoretical guarantees even when dealing with singular interaction kernels. We elaborate on the contributions of our work from theory, algorithm, and empirical perspectives as follows.

1. (Theory-wise) By studying the stability of the MVEs with bounded interaction kernels or with singular interaction kernels in the Coulomb (3) and the Biot-Savart case (4) (the 2D NSE) via entropy dissipation, we establish the error estimation guarantee for the EINN loss on these equations. Specifically, we design a potential function R ( f ) of a hypothesis velocity f such that R ( f ) controls the KL divergence between the hypothesis solution ρ f t (defined in equation (7)) and the ground truth solution ¯ ρ t for any time stamp within a given time interval [ 0 , T ] . A direct consequence of this result is that R ( f ) can be used to assess the quality of a generic hypothesis solution to the above MVEs and ρ f t exactly recovers ¯ ρ t in the KL sense given that R f 0 .

( ) = 2. (Algorithm-wise) When the hypothesis velocity field is parameterized by an NN, i.e. f = f θ with θ being some finite-dimensional parameters, the EINN loss R ( f θ ) can be used as the loss function of the NN parameters θ . We discuss in detail how an estimator of the gradient ∇ θ R ( f θ ) can be computed so that stochastic gradient-based optimizers can be utilized to train the NN. In particular, for the 2D NSE (the Biot-Savart case (4)), we show that the singularity in the gradient computation can be removed by exploiting the anti-derivative of the Biot-Savart kernel.

3. (Empirical-wise) We compare the proposed approach, derived from our novel theoretical guarantees, with SOTA NN-based algorithms for solving the MVE with the Coulomb interaction and the 2D NSE (the Biot-Savart interaction). We pick specific instances of the initial density ¯ ρ 0 , under which explicit solutions are known and can be used as the ground truth to test the quality of the hypothesis ones. Using NNs with the same complexity (depth, width, and structure), we observe that the proposed method significantly outperforms the included baselines.

2 Entropy-dissipation Informed Neural Network

In this section, we present the proposed EINN framework for the MVE. To understand the intuition behind our design, we first write the continuity equation (7) in a similar form as the MVE (6):

where f is the hypothesis velocity (recall that f t f t, ) and can be regarded as a perturbation to the original MVE system. Consequently, it is natural to study the deviation of the hypothesis solution ρ f t from the true solution ¯ ρ t using an appropriate Lyapunov

function L ( ρ f t , ¯ ρ t ) . The functional relation between this deviation and the perturbation is termed as the stability of the underlying dynamical system, which allows us to derive the EINN loss (8). Following this idea, the design of the EINN loss can be determined by the choice of the Lyapunov function L used in the stability analysis. In the following, we describe the Lyapunov function used for the MVE with the Coulomb interaction and the 2D NSE (MVE with Biot-Savart interaction). The proof of the following results are the major theoretical contributions of this paper and will be elaborated in the analysis section 3.

For the MVE with the Coulomb interaction, we choose L to be the modulated free energy E (defined in equation (23)) which is originally proposed in [Bresch et al., 2019a] to establish the mean-field limit of a corresponding interacting particle system. We have (setting L = E )

where C is a universal constant depending on ν and ( ¯ ρ t ) t ∈ [ 0 ,T ] . · For the 2D NSE (MVE with the Biot-Savart interaction), we choose L as the KL divergence. Our analysis is inspired by [Jabin and Wang, 2018] which for the first time establishes the quantitative mean-field limit of the stochastic interacting particle systems where the interaction kernel can be in some negative Sobolev space. We have

After applying Gr¨ onwall's inequality on the above results, we can see that the EINN loss (8) is precisely the term derived by stability analysis of the MVE system with an appropriate Lyapunov function. In the next section, we elaborate on how a stochastic approximation of ∇ θ R ( f θ ) can be efficiently computed for a parameterized hypothesis velocity field f = f θ so that stochastic optimization methods can be utilized to minimize R ( f θ ) .

where again C is a universal constant depending on ν and ( ¯ ρ t ) t ∈ [ 0 ,T ] .

2.1 Stochastic Gradient Computation with Neural Network Parameterization

While the choice of the EINN loss (8) is theoretically justified through the above stability study, in this section, we show that it admits an estimator which can be efficiently computed. Define the flow map X t via the ODE d x ( t ) = f t ( x ( t ) ; θ ) d t with x ( 0 ) = x 0 such that x ( t ) = X t ( x 0 ) . From the definition of the push-forward measure, one has ρ f t = X t ♯ ¯ ρ 0 . Recall the definitions of the EINN loss R ( f ) in equation (8) and the perturbation δ t in equation (11). Use the change of variable formula of the push-forward measure in (a) and Fubini's theorem in (b). We have

we can write the potential function (8) as an expectation R ( f ) = E x 0 ∼ ¯ ρ 0 [ R ( f ; x 0 )] . Similarly, when f is parameterized as f = f θ , we obtain the expectation form ∇ θ R ( f θ ) = E x 0 ∼ ¯ ρ 0 [ ∇ θ R ( f θ ; x 0 )] .

We show ∇ θ R ( f θ ; x 0 ) can be computed accurately, via the adjoint method (for completeness see the derivation of the adjoint method in appendix D). As a recap, suppose that we can write R ( f θ ; x 0 ) in a standard ODE-constrained form R ( f θ ; x 0 ) = /lscript ( θ ) = ∫ T 0 g ( t, s ( t ) , θ ) dt , where { s ( t )} t ∈ [ 0 ,T ] is the solution to the ODE d dt s ( t ) = ψ ( t, s ( t ) ; θ ) with s ( 0 ) = s 0 , and ψ is a known transition function. The adjoint method states that the gradient d d θ /lscript ( θ ) can be computed as where a ( t ) solves the final value problems d dt a ( t ) ⊺ + a ( t ) ⊺ ∂ψ ∂ s ( t, s ( t ) ; θ ) + ∂ g ∂ s ( t, s ( t ) ; θ ) = 0 , a ( T ) = 0 . In the following, we focus on how R ( f θ ; x 0 ) can be written in the above ODE-constrained form.

Write R ( f θ ; x 0 ) in ODE-constrained form Expanding the definition of δ t in equation (11) gives δ t ( x ( t )) = f t ( x ( t )) -( -∇ V ( x ( t )) + K ∗ ρ f t ( x ( t )) -ν ∇ log ρ f t ( x ( t )) ) . (17)

Note that in the above quantity, f and V are known functions. Moreover, it is known that ∇ log ρ f t ( x ( t )) admits a closed form dynamics (e.g. see Proposition 2 in [Shen et al., 2022])

which allows it to be explicitly computed by starting from ∇ log ¯ ρ 0 ( x 0 ) and integrating over time (recall that ¯ ρ 0 is known). Here J f t denotes the Jacobian matrix of f t . Consequently, all we need to handle is the convolution term K ∗ ρ f t ( x ( t )) .

1. For the Coulomb type kernel (3), we first approximate K ρ f t with K c ρ f t , where

A common choice to approximate the convolution operation is via Monte-Carlo integration: Let y i ( t ) iid ∼ ρ f t for i = 1 , . . . , N and denote an empirical approximation of ρ f t by µ ρ f t N = 1 N ∑ N i = 1 δ y i ( t ) , where δ y i ( t ) denotes the Dirac measure at y i ( t ) . We approximate the convolution term in equation (17) in different ways for the Coulomb and the Biot-Savart interactions:

If ρ f t is bounded in X , we have sup x ∈ X ∥ ( K -K c ) ∗ ρ f t ( x ) ∥ = sup x ∈ X ∥ ∫ ∥ x -y ∥ ≤ c x -y ∥ x -y ∥ d ρ f t ( y ) d y ∥ ≤ ∥ ρ f t ∥ L ∞ ( X ) ∫ ∥ y ∥ ≤ c 1 ∥ y ∥ d -1 d y . To compute the integral on the right-hand side, we will switch to polar coordinates r, ψ :

2. For Biot-Savart interaction (2D Navier-Stokes equation), there are more structures to exploit and we can completely avoid the singularity: As noted by Jabin and Wang [2018], the convolution kernel K can be written in a divergence form:

Here, J ( r, ψ ) denotes the determinant of the Jacobian matrix resulting from the transformation from the Cartesian system to the polar coordinate system. In inequality (20), we utilize the fact that J ( r, ψ ) ≤ r d -1 , which allows us to cancel out the factor 1 / r d -1 . Now that K c is bounded by c -d + 1 , we can further approximate K c ∗ ρ f t using K c ∗ µ ρ f t N with error of the order O ( c -d + 1 / √ N ) . Altogether, we have sup x ∈ X ∥ K ∗ ρ f t ( x ) -K c ∗ µ ρ f t N ( x ) ∥ = O ( c + c -d + 1 / √ N ) which can be made arbitrarily small for a sufficiently small c and a sufficiently large N .

where the divergence of a matrix function is applied row-wise, i.e. [ K ( x )] i = div U i ( x ) . Using integration by parts and noticing that the boundary integration vanishes on the torus, one has

If the score function ∇ log ρ f t is bounded, then the integrand in the expectation is also bounded. Therefore, we can avoid integrating singular functions and the Monte Carlo-type estimation 1 N ∑ N i = 1 U ( x -y i ( t )) ∇ log ρ f t ( y i ( t )) is accurate for a sufficiently large value of N.

Remark1. Let /lscript N ( θ ) be the function we obtained using the above approximation of the convolution, where N is the number of Monte-Carlo samples. The above discussion shows that /lscript N ( θ ) and R ( f θ ; x 0 ) are close in the L ∞ sense, which is hence sufficient when the EINN loss is used as error quantification since only function value matters. When both /lscript N ( θ ) and R ( f θ ; x 0 ) are in C 2 , one can translate the closeness in function value to the closeness of their gradients. In our experiments, using ∇ /lscript N ( θ ) as an approximation of ∇ θ R ( f θ ; x 0 ) gives very good empirical performance already.

With the above discussion, we can write R ( f θ ; x 0 ) in an ODE-constrained form in a standard way, which due to space limitation is deferred to Appendix D.1.

3 Analysis

In this section, we focus on the torus case, i.e. X = Π d is a box with the periodic boundary condition. This is a typical setting considered in the literature as the universal function approximation of NNs only holds over a compact set. Moreover, the boundary integral resulting from integration by parts vanishes in this setting, making it amenable for analysis purposes. For completeness, we provide a discussion on the unbounded case, i.e. X = R d in the Appendix G, which requires additional regularity assumptions. Given the MVE (2), if K is bounded, it is sufficient to choose the Lyapunov functional L ( ρ f t , ¯ ρ t ) as the KL divergence (please see Theorem 6 in the appendix). But for the singular Coulomb kernel, we need also to consider the modulated energy as in [Serfaty, 2020]

where g is the fundamental solution to the Laplacian equation in R d , i.e. -∆ g = δ 0 , and the Coulomb interaction reads K = -∇ g (see its closed form expression in equation (3)). If we are only interested in the deterministic dynamics with Coulomb interactions, i.e. ν = 0 in equation (2), it suffices to choose L ( ρ f t , ¯ ρ ) as F ( ρ f t , ¯ ρ t ) (please see Theorem 3). But if we consider the system with Coulomb interactions and diffusions, i.e. ν > 0 , we shall combine the KL divergence and the modulated energy to form the modulated free energy as in Bresch et al. [2019b], which reads

(Modulated Free Energy) E ( ρ , ¯ ρ ) def = ν KL ( ρ , ¯ ρ ) + F ( ρ , ¯ ρ ) . (23) This definition agrees with the physical meaning that 'Free Energy = Temperature × Entropy + Energy', and we note that the temperature is proportional to the diffusion coefficient ν . We remark also for two probability densities ρ and ¯ ρ , F ( ρ , ¯ ρ ) ≥ 0 since by looking in the Fourier domain F ( ρ , ¯ ρ ) = ∫ ˆ g ( ξ ) ∣ ̂ ρ -¯ ρ ( ξ ) ∣ 2 d ξ ≥ 0 as ˆ g ( ξ ) ≥ 0 . Moreover, F ( ρ , ¯ ρ ) can be regarded as a negative Sobolev norm for ρ -¯ ρ , which metricizes weak convergence. To obtain our main stability estimate, we first obtain the time evolution of the KL divergence.

d d t ∫ X ρ f t log ρ f t ¯ ρ t = -ν ∫ X ρ f t ∣ ∇ log ρ f t ¯ ρ t ∣ 2 + ∫ X ρ f t K ∗ ( ρ f t -¯ ρ t ) ⋅ ∇ log ρ f t ¯ ρ t + ∫ X ρ f t δ t ⋅ ∇ log ρ f t ¯ ρ t , where X is the tours Π d . All the integrands are evaluated at x .

Lemma 1 (Time Evolution of the KL divergence) . Given the hypothesis velocity field f = f ( t, x ) ∈ C 1 t,x . Assume that ( ρ f t ) t ∈ [ 0 ,T ] and ( ¯ ρ t ) t ∈ [ 0 ,T ] are classical solutions to equation (7) and equation (6) respectively. It holds that (recall the definition of δ t in equation (11))

We refer the proof of this lemma and all other lemmas and theorems in this section to the appendix E. We remark that to have the existence of classical solution ( ¯ ρ t ) t ∈ [ 0 ,T ] , we definitely need the regularity assumptions on -∇ V and on K . But the linear term -∇ V will not contribute to the evolution of the relative entropy. See [Jabin and Wang, 2018] for detailed discussions.

Similarly, we have the time evolution of the modulated energy as follows.

Lemma 2 (Time evolution of the modulated energy) . Under the same assumptions as in Lemma 1, given the diffusion coefficient ν ≥ 0 , it holds that (recall the definition of δ t in equation (11))

Theorem 2 (Stability estimate of the 2D NSE) . Notice that when K is the Biot-Savart kernel, div K = 0 . Assume that the initial data ¯ ρ 0 ∈ C 3 ( Π d ) and there exists c > 1 such that 1 c ≤ ¯ ρ 0 ≤ c . Assume further the hypothesis velocity field f ( t, x ) ∈ C 1 t,x . Then it holds that where C = ∫ ∞ 0 M ( t ) d t < ∞ with M ( t ) def = ∥ ∇ log ¯ ρ t ∥ 2 L ∞ / 2 ν + 2 ∥ ∇ 2 ¯ ρ t / ¯ ρ t ∥ L ∞ .

We remark that given ¯ ρ 0 is smooth enough and fully supported on X , one can propagate the regularity to finally show the finiteness of C . See detailed computations as in Guillin et al. [2021]. We give the complete proof in the appendix E. This theorem tells us that as long as R ( f ) is small, the KL divergence between ρ f t and ¯ ρ t is small and the control is uniform in time t ∈ [ 0 , T ] for any T . Moreover, we highlight that C is independent of T , and our result on the NSE is significantly better than the average-in-time and exponential-inT results from [De Ryck et al., 2023].

To treat the MVE (2) with Coulomb interactions, we exploit the time evolution of the modulated free energy E ( ρ f t , ¯ ρ t ) . Indeed, combining Lemma 1 and Lemma 2, we arrive at the following identity.

Lemma3 (Time evolution of the modulated free energy) . Under the same assumptions as in Lemma 1, one has (recall the definitions of δ t and A in (11) and (5) respectively)

Inspired by the mean-field convergence results as in Serfaty [2020] and Bresch et al. [2019b], we finally can control the growth of E ( ρ f t , ¯ ρ t ) in the case when ν > 0 , and F ( ρ f t , ¯ ρ t ) in the case when ν 0 . Note also that E ρ f t , ¯ ρ t can also control the KL divergence when ν 0 .

= ( ) > Theorem 3 (Stability estimate of MVE with Coulomb interactions) . Assume that for t ∈ [ 0 , T ] , the underlying velocity field A [ ¯ ρ t ]( x ) is Lipschitz in x and sup t ∈ [ 0 ,T ] ∥ ∇ A [ ¯ ρ t ]( ⋅ ) ∥ L ∞ = C 1 < ∞ . Then there exists C > 0 such that

In the deterministic case when ν 0 , under the same assumptions, it holds that

Recall the definition of the operator A in equation 5. Given that X = Π d , and ¯ ρ 0 is smooth enough and bounded from below, one can propagate regularity to obtain the Lipschitz condition for A [ ¯ ρ t ] . See the proof and the discussion on the Lipschitz assumptions on A [ ¯ ρ t ]( ⋅ ) in the appendix E.

Approximation Error of Neural Network Theorems 2 and 3 provide the error estimation guarantee for the proposed EINN loss (8). Suppose that we parameterize the velocity field f = f θ with an NN parameterized by θ , as we did in Section 2.1 and let ˜ f be the output of an optimization procedure when R ( f θ ) is used as objective. In order the explicitly quantify the mismatch between ρ ˜ f t and ¯ ρ t , we need to quantify two errors: (i) Approximation error, reflecting how well the ground truth solution can be approximated among the NN function class of choice; (ii) Optimization error, involving minimization of a highly nonlinear non-convex objective. In the following, we show that for a function class F with sufficient capacity, there exists at least one element ˆ f ∈ F that can reduce the loss function R ( ˆ f ) as much as desired. We will not discuss how to identify such an element in the function class F as it is independent of our research and remains possibly the largest open problem in modern AI research. To establish our result, we make the following assumptions.

We here again need to propagate the regularity for f t at least for a time interval [ 0 , T ] . It is easy to do so for the torus case, but for the unbounded domain, there are some technical issues to be overcome. Similar assumptions are also needed in some mathematical works for instance in Jabin and Wang [2018]. We also make the following assumption on the capacity of the function class F , which is satisfied for example by NNs with tanh activation function [De Ryck et al., 2021].

Assumption 1. ρ 0 is sufficiently regular, such that ∇ log ρ 0 ∈ L ∞ ( X ) and ¯ f t = A [ ¯ ρ t ] ∈ W 2 , ∞ ( X ) . ∇ V is Lipschitz continuous. Here W 2 , ∞ ( X ) stands for the Sobolev norm of order ( 2 , ∞ ) over X .

Assumption 2. The function class is sufficiently large, such that there exists ˆ f ∈ F satisfying ˆ f t ∈ C 3 ( X ) and ∥ ˆ f t -¯ f t ∥ W 2 , ∞ ( X ) ≤ /epsilon1 for all t ∈ [ 0 , T ] .

Theorem 4. Consider the case where the domain is the torus. Suppose that Assumptions 1 and 2 hold. For both the Coulomb and the Biot-Savart cases, there exists ˆ f ∈ F such that R ( ˆ f ) ≤ C ( T ) ⋅ ( /epsilon1 ⋅ ln 1 / /epsilon1 ) 2 , where C ( T ) is some constant independent of /epsilon1 . Here R is the EINN loss (8).

The major difficulty to overcome is the lack of Lipschitz continuity due to the singular interaction. We successfully address this challenge by establishing that the contribution of the singular region ( ∥ x ∥ ≤ /epsilon1 ) to R ( ˆ f ) can be bounded by O (( /epsilon1 log 1 /epsilon1 ) 2 ) . Please see the detailed proof in Appendix F.

4 Related Works on NN-based PDE solvers

Solving PDEs is a key aspect of scientific research, with a wealth of literature [Evans, 2022]. Due to space limitations, a detailed discussion about the classical PDE solvers is deferred to Appendix A. In this section, we focus on the NN-based approaches as they are more related to our research.

As previously mentioned, PINN is possibly the most well-known method of this type. PINN regards the solution to a PDE system as the root of the corresponding operators { D i ( g )} n i = 1 , and expresses the time and space boundary conditions as B ( g ) = 0 , where g is a candidate solution and D i and B are operators acting on g . Parameterizing g = g θ using an NN, PINN optimizes its parameters θ by minimizing the residual L ( θ ) def = ∑ n i = 1 λ i ∥ D i ( g θ ) ∥ 2 L 2 ( X ) + λ 0 ∥ B ( g θ ) ∥ 2 L 2 ( X ) . The hyperparameters λ i balance the validity of PDEs and boundary conditions under consideration and must be adjusted for optimal performance. In contrast, EINN requires no hyperparameter tuning. PINN is versatile and can be applied to a wide range of PDEs, but its performance may not be as good as other NNbased solvers tailored for a particular class of PDEs, as it does not take into account other in-depth properties of the system, a phenomenon observed in the literature [Krishnapriyan et al., 2021, Wang et al., 2022]. [Shin and Em Karniadakis, 2020] initiates the work of theoretically establishing the consistency of PINN by considering the linear elliptic and parabolic PDEs, for which they prove that a vanishing PINN loss L ( θ ) asymptotically implies g θ recovers the true solution. A similar result is extended to the linear advection equations in [Shin et al., 2020]. Leveraging the stability of the operators D i (corresponding to PDEs of interest), non-asymptotic error estimations are established for linear Kolmogorov equations in [De Ryck and Mishra, 2022], for semi-linear and quasi-linear parabolic equations and the incompressible Euler in [Mishra and Molinaro, 2022], and for the NSE in [De Ryck et al., 2023]. We highlight these non-asymptotic results are all average-in-time, meaning that even when the PINN loss is small the deviation of the candidate solution to the true solution may be significant at a particular timestamp t ∈ [ 0 , T ] . In comparison, our results are uniformin-time, i.e. the supremum of the deviation is strictly bounded by the EINN loss. Moreover, we show in Theorem 2, for the NSE our error estimation holds for any T uniformly, while the results in [De Ryck et al., 2023] have an exponential dependence on T .

Recent work from Zhang et al. [2022] proposes the Random Deep Vortex Network (RDVN) method for solving the 2D NSE and achieves SOTA performance for this task. Let u θ t be an estimation of the interaction term K ∗ ρ t in the SDE (1) and use ρ θ t to denote the law of the particle driven by the SDE d X t = u θ t ( X t ) d t + √ 2 ν d B t . To train u θ t , RDVN minimizes the loss L ( θ ) = ∫ T 0 ∫ X ∥ u θ t ( x ) -K ∗ ρ θ t ( x ) ∥ 2 L 2 d x d t . Note that in order to simulate the SDE, one needs to discretize the time variable in loss function L . After training θ , RDVN outputs ρ θ t as a solution. However, no error estimation guarantee is provided that controls the discrepancy between ρ θ t and ρ t using L ( θ ) .

Shen et al. [2022] propose the concept of self-consistency for the FPE. However, unlike our work where the EINN loss is derived via the stability analysis, they construct the potential R ( f ) for the hypothesis velocity field f by observing that the underlying velocity field f ∗ is the fixed point of some velocity-consistent transformation A and they construct R ( f ) to be a more complicated Sobolev norm of the residual f -A ( f ) . In their result, they bound the Wasserstein distance between ρ f and ρ by R ( f ) , which is weaker than our KL type control. The improved KL type control for the Fokker-Planck equation has also been discussed in [Boffi and Vanden-Eijnden, 2023]. A very recent work [Li et al., 2023] extends the self-consistency approach to compute the general Wasserstein gradient flow numerically, without providing further theoretical justification.

5 Experiments

To show the efficacy and efficiency of the proposed approach, we conduct numerical studies on example problems that admit explicit solutions and compare the results with SOTA NN-based PDE solvers. The included baselines are PINN [Raissi et al., 2019] and DRVN [Zhang et al., 2022]. Note that these baselines only considered the 2D NSE. We extend them to solve the MVE with the Coulomb interaction for comparison, and the details are discussed in Appendix C.1.

We consider the following two instances that admit explicit

Lamb-Oseen Vortex (2D NSE) [Oseen, 1912]: Consider the whole domain case where X = R 2 and the Biot-Savart kernel (4). Let N ( µ , Σ ) be the Gaussian distribution with mean µ and covariance Σ . If ρ 0 = N ( 0 , √ 2 ν t 0 I 2 ) for some t 0 ≥ 0 , then we have ρ t ( x ) = N ( 0 , √ 2 ν ( t + t 0 ) I 2 ) .

Equations with an explicit solution solutions. We verify these solutions in Appendix C.2.

Barenblatt solutions (MVE) [Serfaty and V´ azquez, 2014]: Consider the 3D MVE with the Coulomb interaction kernel (3) with the diffusion coefficient set to zero, i.e. d = 3 and ν = 0 . Let Uniform [ A ] be the uniform distribution over a set A . Consider the whole domain case where X = R 3 . If ρ 0 = Uniform [ ∥ x ∥ ≤ ( 3 4 π t 0 ) 1 / 3 ] for some t 0 ≥ 0 , then we have ρ t = Uniform [ ∥ x ∥ ≤ ( 3 4 π ( t + t 0 )) 1 / 3 ] .

= = = we use is an MLP with 7 hidden layers, each of which has 20 neurons.

Numerical results We present the results of our experiments in Figure 1, where the first row contains the result for the Lamb-Oseen vortex (2D NSE) and the second row contains the result for the Barenblatt model (3D MVE). The explicit solutions of these models allow us to assess the quality of the outputs of the included methods. Specifically, given a hypothesis solution ρ f t , the ground truth ¯ ρ t and the interaction kernel K , define the relative /lscript 2 error at timestamp t as Q ( t ) def = ∫ Ω ∥ K ∗ ( ρ f t -¯ ρ t )( x ) ∥/∥ K ∗ ¯ ρ t ( x ) ∥ d x , where Ω is some domain where ρ t has non-zero density. We are particularly interested in the quality of the convolution term K ∗ ρ f t since it has physical meanings. In the Biot-Savart kernel case, it is the velocity of the fluid, while in the Coulomb case, it is the Coulomb field. We set Ω to be [ -2 , 2 ] 2 for the Lamb-Oseen vortex and to [ -0 . 1 , 0 . 1 ] 3 for the Barenblatt model. For both models, we take ν 0 . 1 , t 0 0 . 1 , and T 1 . The neural network that

From the first column of Figure 1, we see that the objective loss of all methods has substantially reduced over a training period of 10000 iterations. This excludes the possibility that a baseline has worse performance because the NN is not well-trained, and hence the quality of the solution now solely depends on the efficacy of the method. From the second and third columns, we see that the proposed EINN method significantly outperforms the other two methods in terms of the timeaverage relative /lscript 2 error, i.e. 1 T ∫ T 0 Q ( t ) dt and the relative /lscript 2 error at the last time stamp Q ( T ) . This shows the advantage of our method.

Conclusion By employing entropy dissipation of the MVE, we design a potential function for a hypothesis velocity field such that it controls the KL divergence between the corresponding hypothesis solution and the ground truth, for any time stamp within the period of interest. Built on this potential, we proposed the EINN method for MVEs with NN and derived the detailed computation method of the stochastic gradient, using the classic adjoint method. Through empirical studies on examples of the 2D NSE and the 3D MVE with Coulomb interactions, we show the significant advantage of the proposed method, when compared with two SOTA NN based MVE solvers.

Acknowledgments and Disclosure of Funding

Zhenfu Wang is supported by the National Key R&D Program of China, Project Number 2021YFA1002800, NSFC grant No.12171009, Young Elite Scientist Sponsorship Program by China Association for Science and Technology (CAST) No. YESS20200028 and the Fundamental Research Funds for the Central Universities (the start-up fund), Peking University. Zebang Shen's work is supported by ETH research grant and Swiss National Science Foundation (SNSF) Project Funding No. 200021-207343.

References

M. Boffi and E. Vanden-Eijnden. Probability flow solution of the fokker-planck equation. Machine Learning: Science and Technology , 4(3):035012, 2023.

Bresch, P.-E. Jabin, and Z. Wang. On mean-field limits and quantitative estimates with a large class of singular kernels: application to the patlak-keller-segel model. Comptes Rendus Mathematique , 357(9):708-720, 2019a.

Bresch, P.-E. Jabin, and Z. Wang. Modulated free energy and mean field limit. S´ eminaire Laurent Schwartz-EDP et applications , pages 1-22, 2019b.

Cai, Z. Mao, Z. Wang, M. Yin, and G. E. Karniadakis. Physics-informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica , 37(12):1727-1738, 2021.

A. Carrillo, K. Craig, and F. S. Patacchini. A blob method for diffusion. Calculus of Variations and Partial Differential Equations , 58:1-53, 2019.

A. Carrillo, K. Craig, L. Wang, and C. Wei. Primal dual methods for wasserstein gradient flows. Foundations of Computational Mathematics , pages 1-55, 2022.

Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-informed neural networks: where we are and what's next. Journal of Scientific Computing , 92(3):88, 2022.

De Ryck and S. Mishra. Error analysis for physics-informed neural networks (pinns) approximating kolmogorov pdes. Advances in Computational Mathematics , 48(6):1-40, 2022.

De Ryck, S. Lanthaler, and S. Mishra. On the approximation of functions by tanh neural networks. Neural Networks , 143:732-750, 2021. ISSN 0893-6080. doi: https://doi.org/10.1016/j. neunet.2021.08.015. URL https://www.sciencedirect.com/science/article/ pii/S0893608021003208 .

De Ryck, A. Jagtap, and S. Mishra. Error estimates for physics-informed neural networks approximating the navier-stokes equations. IMA Journal of Numerical Analysis , 2023.

Erd˝ os and H.-T. Yau. A dynamical approach to random matrix theory , volume 28. American Mathematical Soc., 2017.

C. Evans. Partial differential equations , volume 19. American Mathematical Society, 2022.

Golse. On the dynamics of large particle systems in the mean field limit. Macroscopic and large scale phenomena: coarse graining, mean field limits and ergodicity , pages 1-144, 2016.

Guillin, P. L. Bris, and P. Monmarch´ e. Uniform in time propagation of chaos for the 2d vortex model and other singular stochastic systems. arXiv preprint arXiv:2108.08675 , 2021.

Gupta, X. Xiao, and P. Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems , 34:24048-24062, 2021.

Han, A. Jentzen, and W. E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences , 115(34):8505-8510, 2018.

P.-E. Jabin and Z. Wang. Quantitative estimates of propagation of chaos for stochastic systems with W -1 , ∞ kernels. Inventiones mathematicae , 214(1):523-591, 2018.

Johnson. Numerical solution of partial differential equations by the finite element method . Courier Corporation, 2012.

Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the fokker-planck equation. SIAM journal on mathematical analysis , 29(1):1-17, 1998.

E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481 , 2021.

Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems , 34:26548-26560, 2021.

Li, S. Hurault, and J. Solomon. Self-consistent velocity matching of probability flows. arXiv preprint arXiv:2301.13737 , 2023.

Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 , 2020a.

Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485 , 2020b.

D.-G. Long. Convergence of the random vortex method in two dimensions. Journal of the American Mathematical Society , 1(4):779-804, 1988.

J. Majda, A. L. Bertozzi, and A. Ogawa. Vorticity and incompressible flow. cambridge texts in applied mathematics. Appl. Mech. Rev. , 55(4):B77-B78, 2002.

Mishra and R. Molinaro. Estimates on the generalization error of physics-informed neural networks for approximating pdes. IMA Journal of Numerical Analysis , 43(1):1-43, 01 2022. ISSN 0272-4979. doi: 10.1093/imanum/drab093. URL https://doi.org/10.1093/imanum/ drab093 .

Moukalled, L. Mangani, M. Darwish, F. Moukalled, L. Mangani, and M. Darwish. The finite volume method . Springer, 2016.

Oseen. Uber die wirbelbewegung in einer reibenden flussigkeit. Ark. Mat. Astro. Fys. , 7, 1912.

Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Raissi, A. Yazdani, and G. E. Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science , 367(6481):1026-1030, 2020.

Rosenzweig. Mean-field convergence of point vortices to the incompressible euler equation with vorticity in L ∞ . Archive for Rational Mechanics and Analysis , 243(3):1361-1431, 2022. S. Serfaty. Mean field limit for coulomb-type flows. Duke Math. J. , 169(15):2887-2935, 2020.

Serfaty and J. L. V´ azquez. A mean field equation as limit of nonlinear diffusions with fractional laplacian operators. Calculus of Variations and Partial Differential Equations , 49:1091-1120, 2014.

Shen, T. Tang, and L.-L. Wang. Spectral methods: algorithms, analysis and applications , volume 41. Springer Science & Business Media, 2011.

Shen, Z. Wang, S. Kale, A. Ribeiro, A. Karbasi, and H. Hassani. Self-consistency of the fokker planck equation. In P.-L. Loh and M. Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 817-841. PMLR, 02-05 Jul 2022. URL https://proceedings.mlr.press/v178/shen22a. html .

Shin, YeonjongDarbon and G. Em Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. Communications in Computational Physics , 28(5):2042-2074, 2020.

Shin, Z. Zhang, and G. E. Karniadakis. Error estimates of residual minimization using neural networks for linear pdes. arXiv preprint arXiv:2010.08019 , 2020.

D. Smith, G. D. Smith, and G. D. S. Smith. Numerical solution of partial differential equations: finite difference methods . Oxford university press, 1985.

Villani et al. Optimal transport: old and new , volume 338. Springer, 2009.

Wang, S. Li, D. He, and L. Wang. Is L 2 physics informed loss always suitable for training physics informed neural network? In Advances in Neural Information Processing Systems , 2022.

Weinan, J. Han, and A. Jentzen. Algorithms for solving high dimensional pdes: from nonlinear monte carlo to machine learning. Nonlinearity , 35(1):278, 2021.

Xiao, D. Cao, R. Yang, G. Gupta, G. Liu, C. Yin, R. Balan, and P. Bogdan. Coupled multiwavelet neural operator learning for coupled partial differential equations. arXiv preprint arXiv:2303.02304 , 2023.

Zhang, J. Han, H. Wang, and R. Car. Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Physical review letters , 120(14):143001, 2018.

Zhang, P. Hu, Q. Meng, Y. Wang, R. Zhu, B. Chen, Z.-M. Ma, and T.-Y. Liu. Drvn (deep random vortex network): A new physics-informed machine learning method for simulating and inferring incompressible fluid flows. Physics of Fluids , 34(10):107112, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on deriving a uniform-in-time bound to the KL divergence between the NN-derived solution $\\rho_t^f$ and the true solution $\\bar{\\rho}_t$, where $\\rho_t^f$ is obtained via a push forward of samples of $\\bar{\\rho}_0$ with velocity parameterized by the neural network $f$. The neural network $f$ is trained to match the operator $\\mathcal{A}[\\rho]$ via the the Entropy-dissipation Informed Neural Network (EINN) loss given as an expectation over $\\bar{\\rho}_0$ of the integral from time $0$ to $T$ of the difference between $f$ and $\\mathcal{A}[\\rho^f]$. The method is then demonstrated to perform better than SOTA on McKean-Vlasov equations with singular interaction kernels: Coulomb interaction and Biot-Savart interaction.\n\nSTRENGTHS:\n1. Uniform-in-time bound to the KL divergence for finite duration $T$.\n2. Method was shown to work on singular interaction kernels: Coulomb interaction,\\ and Biot-Savart interaction.\n3. Singularity for Biot-Savart kernel was removed in gradient computation during training.\n\nWEAKNESSES:\n1. Addressed in the last paragraph of section 4, the similarity in methodology to Shen et al. [2022]. Though the implementation is similar, the theoretical contribution is different. Perhaps include a comparison of their result in Figure 1 with and without generalization to MVE. And perhaps discuss the generalization to MVE.\n2. Also addressed in the second last paragraph of section 4, the similar in loss function to Random Deep Vortex Network (RDVN) of Zhang et al. [2022]. Though the losses are similar, the authors mention a lack of discrepancy control in Zhang et al. [2022]. Perhaps provide more discussion.\n\nQUESTIONS:\n1. Maybe I am misunderstanding, how is using a NN to parameterize $\\mathcal{A}[\\rho^f]$ helping in computation? $\\rho_t$ is still computed via push forward of $\\rho_0$. Would using the true $\\mathcal{A}[\\rho]$ be better? Or is the goal to first compute $f$ as a surrogate to $\\mathcal{A}[\\rho]$ then compute $\\rho^f$ from the PDE with $f$ as a surrogate for $\\mathcal{A}[\\rho]$?\n2. Why is $K_c$ set to $0$ instead of clipping some large value (line 173, equation (19))?\n\nLIMITATIONS:\n1. Addressed in Section 3, approximation error of neural network that if a neural network can represent the true $\\mathcal{A}[\\rho]$, EINN loss is zero. Perhaps reference some papers on neural networks designed to represent $f$ for MVE.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for learning solutions to partial differential equations of the McKean-Vlasov type, and presents a theoretical analysis of the proposed method. In particular, for a certain class of equations, it is shown that the error bound does not increase exponentially in time. This is a significant result that takes advantage of the characteristics of the equations.\n\nSTRENGTHS:\nThe uniform bound presented by the authors is a very strong theoretical result. Although I do not understand the details of the proofs, the proofs seem to be reliable because several references are provided. In my opinion, this paper is an excellent paper as a paper in the research area of numerical analysis.\n\nWEAKNESSES:\nThe most important weakness is that it is not clear what practical applications the McKean-Vlasov type of partial differential equations have.  Unlike numerical analysis, which is a research area in mathematics, there are many people who are interested in applications in the machine learning community. So these results may not be of much interest to them. \n\nAnother significant weakness is that the adjoint method and the Monte Carlo method are required in the proposed approach, which are computationally expensive. Considering the computational cost, the proposed method is not considered superior to classical numerical methods, such as the finite difference method.\n\nIn addition, the uniform bound seems to hold only for a class of equations, but this is not clearly written in Introduction.\n\nQUESTIONS:\n(1) What are the practical applications of the McKean-Vlasov-type partial differential equations?\n(2) Because computationally expensive methods such as the adjoint method and the Monte Carlo method are required, the computational cost of the proposed method may be larger than the classical approaches such as the finite difference method. So, what are the advantages of the proposed neural-network solver compared to the classical numerical methods (e.g, the finite difference method)?\n\nLIMITATIONS:\nNo potential negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel method for solving the McKean-Vlasov Equation (MVE), a Partial Differential Equation (PDE) that models the dynamics of stochastic particle systems with mean-field interactions. The new approach, named Entropy-dissipation Informed Neural Network (EINN), leverages neural networks to estimate the velocity field underlying the MVE and minimize a proposed potential function. This method is particularly designed to tackle singular interaction kernels, where traditional methods struggle. The authors provide empirical comparisons with state-of-the-art (SOTA) neural network-based MVE solvers and demonstrate the superior performance of the EINN framework. Despite the advancements, a noticeable weakness in this work is the omission of a discussion on the neural operator framework for solving PDEs, which represents a significant progression in the field. This lack of consideration limits the comprehensiveness and depth of the study.\n\nSTRENGTHS:\n1. The paper provides a novel approach to solving MVEs. The idea of using entropy dissipation in the underlying system is inventive and beneficial to the community. \n2. The research is solid and thorough. The authors have established theoretical guarantees for their EINN framework and provided clear derivations and proofs. \n3. The empirical comparison of EINN with state-of-the-art methods and the positive results strengthen the claim of the proposed method's effectiveness. The authors ensured a fair comparison by using NNs with the same complexity for all methods tested.\n\nWEAKNESSES:\n1. The lack of discussion on the neural operator framework for solving PDEs. This approach has been a considerable advancement in the field, and the authors' failure to address or refer to it is an oversight. Further exploration or comparison of the neural operator approach could enhance the comprehensiveness and relevance of the paper.\n2. While the authors did a commendable job providing rigorous mathematical proofs, the paper could be difficult for those without a strong background in the relevant mathematical fields. Providing more intuitive explanations or graphical illustrations might have made the methodology more accessible. \n3. The empirical testing seems to be limited in scope. More extensive experimentation, including larger-scale problems and more varied examples, would have strengthened the empirical evidence.\n\nQUESTIONS:\n1. While the authors aim to utilize the Lyapunov function to study the deviation of the hypothesis solution from the true solution, they might have overlooked the possibility of overfitting. There seems to be no clear mention of measures taken to avoid overfitting in the model. How does the model ensure that it is generalizable and not just fitting the data it was trained on?\n2. How can the EINN framework be adapted or expanded to deal with other types of PDEs? \n3. How does the performance of EINN scale with the complexity of the problem, particularly in high dimensional cases? \n4. Given the reliance on NNs, how sensitive is the performance of the EINN approach to the choice of NN architecture and parameters?\n5. The authors should refer to and discuss the neural operator work for solving PDEs as related work. Here are some references: [Li, Zongyi, et al. \"Fourier neural operator for parametric partial differential equations.\"]; [Kovachki, Nikola, et al. \"Neural operator: Learning maps between function spaces.\"]; [Li, Zongyi, et al. \"Neural operator: Graph kernel network for partial differential equations.\"]; [Gupta, Gaurav, Xiongye Xiao, and Paul Bogdan. \"Multiwavelet-based operator learning for differential equations.\"]; [Xiao, Xiongye, et al. \"Coupled Multiwavelet Neural Operator Learning for Coupled Partial Differential Equations.\"].\n\nLIMITATIONS:\nThe authors have specifically designed this approach for solving MVEs, which might limit its general applicability for solving various types of PDEs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper proposed an entropy-dissipation informed neural network for solving certain PDE systems. Using energy dissipation, the paper designed a special structure that propagates information from previous dynamics. They provided a theoretical guarantee for this framework, including the control of the KL divergence between the target and the estimator and a discussion of the approximation error of the NN. They also did some numerical experiments to verify their claims.

STRENGTHS:
1. The idea of using energy dissipation to design a framework is very interesting. EINN also takes the previous dynamics precisely and provides a uniform-in-time solution. 

2. Mathematically speaking, the proofs are solid and well-written.

3. McKean-Vlasov type PDEs is an important class of PDEs for people from both math and physics communities. The presence of the singular kernel does bring challenges, which are handled by EINN.

WEAKNESSES:
1. The authors specifically take advantage of the special structure of this type of PDEs. On one hand, it performs well for those PDEs; on the other hand, this limits the application of this method to other PDEs. 
2. Numerical experiments are done with only comparison with PINN and DRVN.

QUESTIONS:
1. In Figure 1, first row,  at iteration 0, the value of the objective functions are not the same. 
2. In Figure 1, it seems DRVN and EINN perform similarly. Please comment on this.
3. Please comment on how EINN compares with more recent neural-net-based PDE solvers, such as Fourier neural networks.
4. I wonder if more experiments can be done, other than 2D NSE and 3D MVE.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12