SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here,  $J_{(r,\psi)}$  denotes the determinant of the Jacobian matrix resulting from the transformation from the Cartesian system to the polar coordinate system. In inequality (20), we utilize the fact that  $J_{(r,\psi)} \leq r^{d-1}$ , which allows us to cancel out the factor  $1/r^{d-1}$ . Now that  $K_c$  is bounded by  $c^{-d+1}$ , we can further approximate  $K_c * \rho_t^f$  using  $K_c * \mu_N^{\rho_t^f}$  with error of the order  $O(c^{-d+1}/\sqrt{N})$ . Altogether, we have  $\sup_{\boldsymbol{x} \in \mathcal{X}} \| K * \rho_t^f(\boldsymbol{x}) - K_c * \mu_N^{\rho_t^f}(\boldsymbol{x}) \| = O(c + c^{-d+1}/\sqrt{N})$  which can be made arbitrarily small for a sufficiently small  $c$  and a sufficiently large  $N$ .
2. For Biot-Savart interaction (2D Navier-Stokes equation), there are more structures to exploit and we can completely avoid the singularity: As noted by Jabin and Wang [2018], the convolution kernel  $K$  can be written in a divergence form:
$$
K = \nabla \cdot U, \text {w i t h} U (\boldsymbol {x}) = \frac {1}{2 \pi} \left[ \begin{array}{c c} - \arctan \left(\frac {\boldsymbol {x} _ {1}}{\boldsymbol {x} _ {2}}\right), & 0 \\ 0, & \arctan \left(\frac {\boldsymbol {x} _ {2}}{\boldsymbol {x} _ {1}}\right) \end{array} \right], \tag {21}
$$
where the divergence of a matrix function is applied row-wise, i.e.  $[K(\pmb{x})]_i = \operatorname{div} U_i(\pmb{x})$ . Using integration by parts and noticing that the boundary integration vanishes on the torus, one has
$$
\begin{array}{l} K * \rho_ {t} ^ {f} (\boldsymbol {x}) = \int K (\boldsymbol {y}) \rho_ {t} ^ {f} (\boldsymbol {x} - \boldsymbol {y}) d \boldsymbol {y} = \int \nabla \cdot U (\boldsymbol {y}) \rho_ {t} ^ {f} (\boldsymbol {x} - \boldsymbol {y}) d \boldsymbol {y} = \int U (\boldsymbol {y}) \nabla \rho_ {t} ^ {f} (\boldsymbol {x} - \boldsymbol {y}) d \boldsymbol {y} \\ = \int U (\boldsymbol {x} - \boldsymbol {y}) \rho_ {t} ^ {f} (\boldsymbol {y}) \nabla \log \rho_ {t} ^ {f} (\boldsymbol {y}) d \boldsymbol {y} = \mathbb {E} _ {\boldsymbol {y} \sim \rho_ {t} ^ {f} (\boldsymbol {y})} [ U (\boldsymbol {x} - \boldsymbol {y}) \nabla \log \rho_ {t} ^ {f} (\boldsymbol {y}) ]. \\ \end{array}
$$
If the score function  $\nabla \log \rho_t^f$  is bounded, then the integrand in the expectation is also bounded. Therefore, we can avoid integrating singular functions and the Monte Carlo-type estimation  $\frac{1}{N}\sum_{i=1}^{N}U(\pmb{x} - \pmb{y}_i(t))\nabla \log \rho_t^f(\pmb{y}_i(t))$  is accurate for a sufficiently large value of N.
With the above discussion, we can write  $R(f_{\theta}; x_0)$  in an ODE-constrained form in a standard way, which due to space limitation is deferred to Appendix D.1
Remark 1. Let  $\ell_N(\theta)$  be the function we obtained using the above approximation of the convolution, where  $N$  is the number of Monte-Carlo samples. The above discussion shows that  $\ell_N(\theta)$  and  $R(f_{\theta};\pmb{x}_0)$  are close in the  $\mathcal{L}^{\infty}$  sense, which is hence sufficient when the EINN loss is used as error quantification since only function value matters. When both  $\ell_N(\theta)$  and  $R(f_{\theta};\pmb{x}_0)$  are in  $C^2$ , one can translate the closeness in function value to the closeness of their gradients. In our experiments, using  $\nabla \ell_N(\theta)$  as an approximation of  $\nabla_{\theta}R(f_{\theta};\pmb{x}_{0})$  gives very good empirical performance already.
In this section, we focus on the torus case, i.e.  $\mathcal{X} = \Pi^d$  is a box with the periodic boundary condition. This is a typical setting considered in the literature as the universal function approximation of NNs only holds over a compact set. Moreover, the boundary integral resulting from integration by parts vanishes in this setting, making it amenable for analysis purposes. For completeness, we provide a discussion on the unbounded case, i.e.  $\mathcal{X} = \mathbb{R}^d$  in the Appendix G, which requires additional regularity assumptions. Given the MVE (2), if  $K$  is bounded, it is sufficient to choose the Lyapunov functional  $L(\rho_t^f,\bar{\rho}_t)$  as the KL divergence (please see Theorem 6 in the appendix). But for the singular Coulomb kernel, we need also to consider the modulated energy as in [Serfaty, 2020]
$$
\text {(M o d u l a t e d E n e r g y)} \quad F (\rho , \bar {\rho}) \stackrel {\mathrm {d e f}} {=} \frac {1}{2} \int_ {\mathcal {X} ^ {2}} g (x - y) \mathrm {d} (\rho - \bar {\rho}) (x) \mathrm {d} (\rho - \bar {\rho}) (y), \tag {22}
$$
where  $g$  is the fundamental solution to the Laplacian equation in  $\mathbb{R}^d$ , i.e.  $-\Delta g = \delta_0$ , and the Coulomb interaction reads  $K = -\nabla g$  (see its closed form expression in equation (3)). If we are only interested in the deterministic dynamics with Coulomb interactions, i.e.  $\nu = 0$  in equation (2), it suffices to choose  $L(\rho_t^f, \bar{\rho})$  as  $F(\rho_t^f, \bar{\rho}_t)$  (please see Theorem 3). But if we consider the system with Coulomb interactions and diffusions, i.e.  $\nu > 0$ , we shall combine the KL divergence and the modulated energy to form the modulated free energy as in Bresch et al. [2019b], which reads
$$
\text {(M o d u l a t e d F r e e E n e r g y)} \quad E (\rho , \bar {\rho}) \stackrel {\text {d e f}} {=} \nu \mathbf {K L} (\rho , \bar {\rho}) + F (\rho , \bar {\rho}). \tag {23}
$$
This definition agrees with the physical meaning that "Free Energy = Temperature × Entropy + Energy", and we note that the temperature is proportional to the diffusion coefficient  $\nu$ . We remark also for two probability densities  $\rho$  and  $\bar{\rho}$ ,  $F(\rho, \bar{\rho}) \geq 0$  since by looking in the Fourier domain  $F(\rho, \bar{\rho}) = \int \hat{g}(\xi) |\overline{\rho - \bar{\rho}}(\xi)|^2 \mathrm{d}\xi \geq 0$  as  $\hat{g}(\xi) \geq 0$ . Moreover,  $F(\rho, \bar{\rho})$  can be regarded as a negative Sobolev norm for  $\rho - \bar{\rho}$ , which metricizes weak convergence.
To obtain our main stability estimate, we first obtain the time evolution of the KL divergence.
Lemma 1 (Time Evolution of the KL divergence). Given the hypothesis velocity field  $f = f(t,x) \in C_{t,x}^{1}$ . Assume that  $(\rho_t^f)_{t \in [0,T]}$  and  $(\bar{\rho}_t)_{t \in [0,T]}$  are classical solutions to equation (7) and equation (6) respectively. It holds that (recall the definition of  $\delta_t$  in equation (11))
$$
\frac {\mathrm {d}}{\mathrm {d} t} \int_ {\mathcal {X}} \rho_ {t} ^ {f} \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}} = - \nu \int_ {\mathcal {X}} \rho_ {t} ^ {f} | \nabla \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}} | ^ {2} + \int_ {\mathcal {X}} \rho_ {t} ^ {f} K * (\rho_ {t} ^ {f} - \bar {\rho} _ {t}) \cdot \nabla \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}} + \int_ {\mathcal {X}} \rho_ {t} ^ {f} \delta_ {t} \cdot \nabla \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}},
$$
where  $\mathcal{X}$  is the tours  $\Pi^d$ . All the integrands are evaluated at  $x$ .
We refer the proof of this lemma and all other lemmas and theorems in this section to the appendix E. We remark that to have the existence of classical solution  $(\bar{\rho}_t)_{t\in [0,T]}$ , we definitely need the regularity assumptions on  $-\nabla V$  and on  $K$ . But the linear term  $-\nabla V$  will not contribute to the evolution of the relative entropy. See [Jabin and Wang, 2018] for detailed discussions.
Similarly, we have the time evolution of the modulated energy as follows.
Lemma 2 (Time evolution of the modulated energy). Under the same assumptions as in Lemma [7] given the diffusion coefficient  $\nu \geq 0$ , it holds that (recall the definition of  $\delta_t$  in equation ([7])
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} F \big (\rho_ {t} ^ {f}, \bar {\rho} _ {t} \big) = - \int_ {\mathcal {X}} \rho_ {t} ^ {f} \| K * \big (\rho_ {t} ^ {f} - \bar {\rho} _ {t} \big) \| ^ {2} - \int_ {\mathcal {X}} \rho_ {t} ^ {f} \delta_ {t} \cdot K * \big (\rho_ {t} ^ {f} - \bar {\rho} _ {t} \big) + \nu \int_ {\mathcal {X}} \rho_ {t} ^ {f} K * \big (\rho_ {t} ^ {f} - \bar {\rho} _ {t} \big) \cdot \nabla \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}} \\ - \frac {1}{2} \int_ {\mathcal {X} ^ {2}} K (x - y) \cdot \left(\mathcal {A} [ \bar {\rho} _ {t} ] (x) - \mathcal {A} [ \bar {\rho} _ {t} ] (y)\right) \mathrm {d} \left(\rho_ {t} ^ {f} - \bar {\rho} _ {t}\right) ^ {\otimes 2} (x, y) \\ \end{array}
$$
where we recall that the operator  $\mathcal{A}$  is defined in equation (5).
By Lemma 1 and careful analysis, in particular by rewriting the Biot-Savart law in the divergence of a bounded matrix-valued function (21), we obtain the following estimate for the 2D NSE.
Theorem 2 (Stability estimate of the 2D NSE). Notice that when  $K$  is the Biot-Savart kernel,  $\mathrm{div}K = 0$ . Assume that the initial data  $\bar{\rho}_0\in C^3 (\Pi^d)$  and there exists  $c > 1$  such that  $\frac{1}{c}\leq \bar{\rho}_0\leq c$ . Assume further the hypothesis velocity field  $f(t,x)\in C_{t,x}^{1}$ . Then it holds that
$$
\sup _ {t \in [ 0, T ]} \int_ {\Pi^ {d}} \rho_ {t} ^ {f} \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}} \mathrm {d} x \leq \frac {e ^ {C}}{\nu} R (f),
$$
where  $C = \int_0^\infty M(t)\mathrm{d}t <   \infty$  with  $M(t)\stackrel {\mathrm{def}}{=}\| \nabla \log \bar{\rho}_t\|_{L^\infty}^2 /2\nu +2\bigg\| \nabla^2\bar{\rho}_t / \bar{\rho}_t\bigg\|_{L^\infty}.$
We remark that given  $\bar{\rho}_0$  is smooth enough and fully supported on  $\mathcal{X}$ , one can propagate the regularity to finally show the finiteness of  $C$ . See detailed computations as in [Guillin et al., 2021]. We give the complete proof in the appendix E. This theorem tells us that as long as  $R(f)$  is small, the KL divergence between  $\rho_t^f$  and  $\bar{\rho}_t$  is small and the control is uniform in time  $t \in [0,T]$  for any  $T$ . Moreover, we highlight that  $C$  is independent of  $T$ , and our result on the NSE is significantly better than the average-in-time and exponential-in- $T$  results from [De Ryck et al., 2023].
To treat the MVE (2) with Coulomb interactions, we exploit the time evolution of the modulated free energy  $E(\rho_t^f,\bar{\rho}_t)$ . Indeed, combining Lemma 1 and Lemma 2, we arrive at the following identity.
Lemma 3 (Time evolution of the modulated free energy). Under the same assumptions as in Lemma 1 one has (recall the definitions of  $\delta_t$  and  $\mathcal{A}$  in (11) and (5) respectively)
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} t} E \left(\rho_ {t} ^ {f}, \bar {\rho} _ {t}\right) = - \int_ {\mathcal {X}} \rho_ {t} ^ {f} \left| K * \left(\rho_ {t} ^ {f} - \bar {\rho} _ {t}\right) - \nu \nabla \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}} \right| ^ {2} - \int_ {\mathcal {X}} \rho_ {t} ^ {f} \delta_ {t} \cdot \left(K * \left(\rho_ {t} ^ {f} - \bar {\rho} _ {t}\right) - \nu \nabla \log \frac {\rho_ {t} ^ {f}}{\bar {\rho} _ {t}}\right) \\ - \frac {1}{2} \int_ {\mathcal {X} ^ {2}} K (x - y) \cdot \left(\mathcal {A} [ \bar {\rho} _ {t} ] (x) - \mathcal {A} [ \bar {\rho} _ {t} ] (y)\right) \mathrm {d} \left(\rho_ {t} ^ {f} - \bar {\rho} _ {t}\right) ^ {\otimes 2} (x, y). \\ \end{array}
$$
Inspired by the mean-field convergence results as in [Serfaty [2020] and Bresch et al. [2019b], we finally can control the growth of  $E(\rho_t^f, \bar{\rho}_t)$  in the case when  $\nu > 0$ , and  $F(\rho_t^f, \bar{\rho}_t)$  in the case when  $\nu = 0$ . Note also that  $E(\rho_t^f, \bar{\rho}_t)$  can also control the KL divergence when  $\nu > 0$ .
Theorem 3 (Stability estimate of MVE with Coulomb interactions). Assume that for  $t \in [0, T]$ , the underlying velocity field  $\mathcal{A}[\bar{\rho}_t](x)$  is Lipschitz in  $x$  and  $\sup_{t \in [0, T]} \| \nabla \mathcal{A}[\bar{\rho}_t](\cdot) \|_{L^\infty} = C_1 < \infty$ . Then there exists  $C > 0$  such that
$$
\sup  _ {t \in [ 0, T ]} \nu \mathbf {K L} (\rho_ {t} ^ {f}, \bar {\rho} _ {t}) \leq \sup  _ {t \in [ 0, T ]} E (\rho_ {t} ^ {f}, \bar {\rho} _ {t}) \leq \exp (C C _ {1} T) R (f).
$$
In the deterministic case when  $\nu = 0$ , under the same assumptions, it holds that
$$
\sup  _ {t \in [ 0, T ]} F \left(\rho_ {t} ^ {f}, \bar {\rho} _ {t}\right) \leq \exp \left(C C _ {1} T\right) R (f).
$$
Recall the definition of the operator  $\mathcal{A}$  in equation 5. Given that  $\mathcal{X} = \Pi^d$ , and  $\bar{\rho}_0$  is smooth enough and bounded from below, one can propagate regularity to obtain the Lipschitz condition for  $\mathcal{A}[\bar{\rho}_t]$ . See the proof and the discussion on the Lipschitz assumptions on  $\mathcal{A}[\bar{\rho}_t](\cdot)$  in the appendix E.
Approximation Error of Neural Network Theorems [2] and [3] provide the error estimation guarantee for the proposed EINN loss [8]. Suppose that we parameterize the velocity field  $f = f_{\theta}$  with an NN parameterized by  $\theta$ , as we did in Section [2.1] and let  $\tilde{f}$  be the output of an optimization procedure when  $R(f_{\theta})$  is used as objective. In order the explicitly quantify the mismatch between  $\rho_t^{\tilde{f}}$  and  $\bar{\rho}_t$ , we need to quantify two errors: (i) Approximation error, reflecting how well the ground truth solution can be approximated among the NN function class of choice; (ii) Optimization error, involving minimization of a highly nonlinear non-convex objective. In the following, we show that for a function class  $\mathcal{F}$  with sufficient capacity, there exists at least one element  $\hat{f} \in \mathcal{F}$  that can reduce the loss function  $R(\hat{f})$  as much as desired. We will not discuss how to identify such an element in the function class  $\mathcal{F}$  as it is independent of our research and remains possibly the largest open problem in modern AI research. To establish our result, we make the following assumptions.
Assumption 1.  $\rho_0$  is sufficiently regular, such that  $\nabla \log \rho_0\in \mathcal{L}^{\infty}(\mathcal{X})$  and  $\bar{f}_t = \mathcal{A}[\bar{\rho}_t]\in W^{2,\infty}(\mathcal{X})$ .  $\nabla V$  is Lipschitz continuous. Here  $W^{2,\infty}(\mathcal{X})$  stands for the Sobolev norm of order  $(2,\infty)$  over  $\mathcal{X}$ .
We here again need to propagate the regularity for  $f_{t}$  at least for a time interval  $[0,T]$ . It is easy to do so for the torus case, but for the unbounded domain, there are some technical issues to be overcome. Similar assumptions are also needed in some mathematical works for instance in Jabin and Wang [2018]. We also make the following assumption on the capacity of the function class  $\mathcal{F}$ , which is satisfied for example by NNs with tanh activation function [De Ryck et al., 2021].
Assumption 2. The function class is sufficiently large, such that there exists  $\hat{f} \in \mathcal{F}$  satisfying  $\hat{f}_t \in \mathcal{C}^3(\mathcal{X})$  and  $\| \hat{f}_t - \bar{f}_t \|_{W^{2,\infty}(\mathcal{X})} \leq \epsilon$  for all  $t \in [0,T]$ .
Theorem 4. Consider the case where the domain is the torus. Suppose that Assumptions  $\square$  and  $\boxed{2}$  hold. For both the Coulomb and the Biot-Savart cases, there exists  $\hat{f} \in \mathcal{F}$  such that  $R(\hat{f}) \leq C(T) \cdot (\epsilon \cdot \ln 1 / \epsilon)^2$ , where  $C(T)$  is some constant independent of  $\epsilon$ . Here  $R$  is the EINN loss (8).
The major difficulty to overcome is the lack of Lipschitz continuity due to the singular interaction. We successfully address this challenge by establishing that the contribution of the singular region  $(\| x \| \leq \epsilon)$  to  $R(\hat{f})$  can be bounded by  $O\left((\epsilon \log \frac{1}{\epsilon})^{2}\right)$ . Please see the detailed proof in Appendix F.
4 Related Works on NN-based PDE solvers
Solving PDEs is a key aspect of scientific research, with a wealth of literature [Evans, 2022]. Due to space limitations, a detailed discussion about the classical PDE solvers is deferred to Appendix A. In this section, we focus on the NN-based approaches as they are more related to our research.
As previously mentioned, PINN is possibly the most well-known method of this type. PINN regards the solution to a PDE system as the root of the corresponding operators  $\{\mathcal{D}_i(\pmb {g})\}_{i = 1}^n$ , and expresses the time and space boundary conditions as  $\mathcal{B}(\pmb {g}) = 0$ , where  $\pmb{g}$  is a candidate solution and  $\mathcal{D}_i$  and  $\mathcal{B}$  are operators acting on  $\pmb{g}$ . Parameterizing  $\pmb {g} = \pmb{g}_{\theta}$  using an NN, PINN optimizes its parameters  $\theta$  by minimizing the residual  $L(\theta)\stackrel {\mathrm{def}}{=}\sum_{i = 1}^{n}\lambda_{i}\| \mathcal{D}_{i}(\pmb{g}_{\theta})\|_{\mathcal{L}^{2}(\mathcal{X})}^{2} + \lambda_{0}\| \mathcal{B}(\pmb{g}_{\theta})\|_{\mathcal{L}^{2}(\mathcal{X})}^{2}$ . The hyperparameters  $\lambda_{i}$  balance the validity of PDEs and boundary conditions under consideration and must be adjusted for optimal performance. In contrast, EINN requires no hyperparameter tuning. PINN is versatile and can be applied to a wide range of PDEs, but its performance may not be as good as other NN-based solvers tailored for a particular class of PDEs, as it does not take into account other in-depth properties of the system, a phenomenon observed in the literature [Krishnapriyan et al., 2021, Wang et al., 2022]. [Shin and Em Karniadakis, 2020] initiates the work of theoretically establishing the consistency of PINN by considering the linear elliptic and parabolic PDEs, for which they prove that a vanishing PINN loss  $L(\theta)$  asymptotically implies  $\pmb{g}_{\theta}$  recovers the true solution. A similar result is extended to the linear advection equations in [Shin et al., 2020]. Leveraging the stability of the operators  $\mathcal{D}_i$  (corresponding to PDEs of interest), non-asymptotic error estimations are established for linear Kolmogorov equations in [De Ryck and Mishra, 2022], for semi-linear and quasi-linear parabolic equations and the incompressible Euler in [Mishra and Molinaro, 2022], and for the NSE in [De Ryck et al., 2023]. We highlight these non-asymptotic results are all average-in-time, meaning that even when the PINN loss is small the deviation of the candidate solution to the true solution may be significant at a particular timestamp  $t\in [0,T]$ . In comparison, our results are uniform-in-time, i.e. the supremum of the deviation is strictly bounded by the EINN loss. Moreover, we show in Theorem 2 for the NSE our error estimation holds for any  $T$  uniformly, while the results in [De Ryck et al., 2023] have an exponential dependence on  $T$ .
Recent work from Zhang et al. [2022] proposes the Random Deep Vortex Network (RDVN) method for solving the 2D NSE and achieves SOTA performance for this task. Let  $\pmb{u}_t^\theta$  be an estimation of the interaction term  $K * \rho_t$  in the SDE [1] and use  $\rho_t^\theta$  to denote the law of the particle driven by the SDE  $\mathrm{d}\mathbf{X}_t = \pmb{u}_t^\theta (\mathbf{X}_t)\mathrm{d}t + \sqrt{2\nu}\mathrm{d}\mathbf{B}_t$ . To train  $\pmb{u}_t^\theta$ , RDVN minimizes the loss  $L(\theta) = \int_0^T\int_{\mathcal{X}}\| \pmb{u}_t^\theta (\pmb {x}) - K*\rho_t^\theta (\pmb {x})\|_{\mathcal{L}^2}^2\mathrm{d}\pmb {x}\mathrm{d}t$ . Note that in order to simulate the SDE, one needs to discretize the time variable in loss function  $L$ . After training  $\theta$ , RDVN outputs  $\rho_t^\theta$  as a solution. However, no error estimation guarantee is provided that controls the discrepancy between  $\rho_t^\theta$  and  $\rho_t$  using  $L(\theta)$ .
Shen et al. [2022] propose the concept of self-consistency for the FPE. However, unlike our work where the EINN loss is derived via the stability analysis, they construct the potential  $R(f)$  for the hypothesis velocity field  $f$  by observing that the underlying velocity field  $f^*$  is the fixed point of some velocity-consistent transformation  $\mathcal{A}$  and they construct  $R(f)$  to be a more complicated Sobolev norm of the residual  $f - \mathcal{A}(f)$ . In their result, they bound the Wasserstein distance between  $\rho^f$  and  $\rho$  by  $R(f)$ , which is weaker than our KL type control. The improved KL type control for the Fokker-Planck equation has also been discussed in [Boffi and Vanden-Eijnden, 2023]. A very recent work [Li et al., 2023] extends the self-consistency approach to compute the general Wasserstein gradient flow numerically, without providing further theoretical justification.
To show the efficacy and efficiency of the proposed approach, we conduct numerical studies on example problems that admit explicit solutions and compare the results with SOTA NN-based PDE solvers. The included baselines are PINN [Raisi et al., 2019] and DRVN [Zhang et al., 2022]. Note that these baselines only considered the 2D NSE. We extend them to solve the MVE with the Coulomb interaction for comparison, and the details are discussed in Appendix C.1.
Figure 1: The first row contains results for the 2D NSE and the second row contains the results for the 3D MVE with Coulomb interaction. The first column reports the objective losses, while the second and third columns report the average and last-time-stamp relative  $\ell_2$  error.
Equations with an explicit solution We consider the following two instances that admit explicit solutions. We verify these solutions in Appendix C.2. Lamb-Oseen Vortex (2D NSE) [Oseen, 1912]: Consider the whole domain case where  $\mathcal{X} = \mathbb{R}^2$  and the Biot-Savart kernel 4. Let  $\mathcal{N}(\pmb{\mu},\pmb{\Sigma})$  be the Gaussian distribution with mean  $\pmb{\mu}$  and covariance  $\pmb{\Sigma}$ . If  $\rho_0 = \mathcal{N}(0,\sqrt{2\nu t_0}\pmb{I}_2)$  for some  $t_0\geq 0$ , then we have  $\rho_t(\pmb {x}) = \mathcal{N}(0,\sqrt{2\nu(t + t_0)}\pmb{I}_2)$ .
Barenblatt solutions (MVE) [Serfaty and Vázquez, 2014]: Consider the 3D MVE with the Coulomb interaction kernel (3) with the diffusion coefficient set to zero, i.e.  $d = 3$  and  $\nu = 0$ . Let  $\mathrm{Uniform}[\mathbb{A}]$  be the uniform distribution over a set  $\mathbb{A}$ . Consider the whole domain case where  $\mathcal{X} = \mathbb{R}^3$ . If  $\rho_0 = \mathrm{Uniform}\big[\| \pmb{x}\| \leq \left(\frac{3}{4\pi} t_0\right)^{1 / 3}\big]$  for some  $t_0\geq 0$ , then we have  $\rho_{t} = \mathrm{Uniform}\big[\| \pmb {x}\| \leq \left(\frac{3}{4\pi} (t + t_{0})\right)^{1 / 3}\big]$ .
Numerical results We present the results of our experiments in Figure  $\mathbb{I}$ , where the first row contains the result for the Lamb-Oseen vortex (2D NSE) and the second row contains the result for the Barenblatt model (3D MVE). The explicit solutions of these models allow us to assess the quality of the outputs of the included methods. Specifically, given a hypothesis solution  $\rho_t^f$ , the ground truth  $\bar{\rho}_t$  and the interaction kernel  $K$ , define the relative  $\ell_2$  error at timestamp  $t$  as  $Q(t)\stackrel {\mathrm{def}}{=}\int_{\Omega}\| K*(\rho_t^f -\bar{\rho}_t)(\pmb {x})\| /\| K*\bar{\rho}_t(\pmb {x})\| dx$ , where  $\Omega$  is some domain where  $\rho_t$  has non-zero density. We are particularly interested in the quality of the convolution term  $K*\rho_t^f$  since it has physical meanings. In the Biot-Savart kernel case, it is the velocity of the fluid, while in the Coulomb case, it is the Coulomb field. We set  $\Omega$  to be  $[-2,2]^2$  for the Lamb-Oseen vortex and to  $[-0.1,0.1]^3$  for the Barenblatt model. For both models, we take  $\nu = 0.1$ ,  $t_0 = 0.1$ , and  $T = 1$ . The neural network that we use is an MLP with 7 hidden layers, each of which has 20 neurons.
From the first column of Figure I, we see that the objective loss of all methods has substantially reduced over a training period of 10000 iterations. This excludes the possibility that a baseline has worse performance because the NN is not well-trained, and hence the quality of the solution now solely depends on the efficacy of the method. From the second and third columns, we see that the proposed EINN method significantly outperforms the other two methods in terms of the time-average relative  $\ell_2$  error, i.e.  $\frac{1}{T}\int_0^T Q(t)dt$  and the relative  $\ell_2$  error at the last time stamp  $Q(T)$ . This shows the advantage of our method.