Mesh-free Eulerian Physics-Informed Neural Networks
Authors: Fabricio Arend Torres
Marcello Massimo Negri
Monika Nagy-Huber
Maxim Samarin
Volker Roth
Venue: Submitted to ICLR 2023 (2023)
Keywords: Physics-informed Neural Network
PINN
SIREN
fluid dynamics
implicit neural representations
PDEs

Abstract: Physics-informed Neural Networks (PINNs) have recently emerged as a principled way to include prior physical knowledge in form of partial differential equations (PDEs) into neural networks. Although PINNs are generally viewed as mesh-free, current approaches still rely on collocation points within a bounded region, even in settings with spatially sparse signals. Furthermore, if the boundaries are not known, the selection of such a region is difficult and often results in a large proportion of collocation points being selected in areas of low relevance. To resolve this severe drawback of current methods, we present a mesh-free and adaptive approach termed particle-density PINN (pdPINN), which is inspired by the microscopic viewpoint of fluid dynamics. The method is based on the Eulerian formulation and, different from classical mesh-free method, does not require the introduction of Lagrangian updates. We propose to sample directly from the distribution over the particle positions, eliminating the need to introduce boundaries while adaptively focusing on the most relevant regions. This is achieved by interpreting a non-negative physical quantity (such as the density or temperature) as an unnormalized probability distribution from which we sample with dynamic Monte Carlo methods. The proposed method leads to higher sample efficiency and improved performance of PINNs. These advantages are demonstrated on various experiments based on the continuity equations, Fokker-Planck equations, and the heat equation.

Under review as a conference paper at ICLR 2023
MESH-FREE EULERIAN PHYSICS-INFORMER NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Physics-informed Neural Networks (PINNs) have recently emerged principled way to include prior physical knowledge in form of partial differential equations (PDEs) into neural networks. Although PINNs are generally viewed mesh-free, current approaches still rely on collocation points within a bounded region, even in settings with spatially sparse signals. Furthermore, if the boundaries are not known, the selection of such a region is difficult and often results in a large proportion of collocation points being selected in areas of low relevance. To resolve this severe drawback of current methods, we present a mesh-free and adaptive approach termed particle-densit PINN (pdPINN), which is inspired by the microscopic viewpoint of fluid dynamics. The method is based on the Eulerian formulation and, different from classical mesh-free method, does not require the introduction Lagrangian updates. We propose to sample directly from the distribution over the particle positions, eliminating the need to introduce boundaries while adaptively foeusing on the most relevant regions. This is achieved by interpreting non- negative physical quantity (such as the density or temperature) as an unnormalized probability distribution from which we sample with dynamic Monte Carlo methods. The proposed method leads to higher sample efficiency and improved performance of PINNs. These advantages are demonstrated on various experiments based on the continuity equations, Fokker-P Pokker-Planck equations, and the heat equation.
INTRODUCTION
Many phenomena in physics are commonly described by partial differential equations (PDEs) which give rise to complex dynamical systems but often lack tractable analytical solutions. Important examples can be found for instance in fluid dynamics with typical applications the design of gas and steam turbines (Oosthuizen & Carscallen] 2013), as well as modeling the collective motion of self-driven particles (Marchetti et al., 2013) such as flocks of birds or bacteria colonies (Szabó et al., 2006, Nussbaumer al. 2021). Despite the relevant progress in establishing numerical PDE solvers, such as finite element and finite volume methods, the seamless incorporation of data remains an open problem (Freitag) 2020). To fill this gap, Physics-informed Neural Networks (PINNs) have emerged as an attractive alternative to classical methods for data- based forward and inverse solving of PDEs.
The general idea of PINNs
use the expressive power of modern neural architectures for solving partial differential equations (PDEs) in a data-driven way by minimizing PDE-based loss, cf. Raissi al. 2019). Consider parameterized PDEs of the general form
ftt, x: a² = a, .)) P(u[A) =0 0,
where P is a non-linear operator parameterized by X, and 0 is the partial time derivative w.r.t. t [". The position condition 9o
Ω defined on spatial domain Ω Rd The PDE is subject to initial
(0, x)) (o
for x E Ω2 and boundary conditions gan
u(t, x
(an
(3)
for æ E 82 and E |0, TJ. The main idea of PINNs consists in approximating unt, æ)) (and hence f(,, æ)) with a neural network given a small set of noisy observations Ugbs
u (1)) x (1) + (1) 2 abs
(4)
(4)

---

Under review as a conference paper at ICLR 2023
with noise e( < O Vi E 10. ... This allows us to consider the following two important problem settings: If A is known, the PDE is fully specified, and we aim to find a solution u in a data-driven manner by training a neural network. The PDE takes the role of a regularizer, where the particular physical laws provide our prior information. A second setting considers the inverse learning of the parameters À by including them into the optimization process in order to infer physical properties such as the viscosity coefficient of fluid (Jagtap et al. 2020). Initial work on solving time-independeet PDEs with neural networks with such PDE-based penalties was pioneered by Dissanayake & Phan- Then 1994) and van Milligen et al. (1955), with later adoptions such as Parisi et al. (2003) extending it to non-steady and time-dependent settings.
Loss functions. Typically, PINNS approximate f(t, x) by the network (t x) in which the pa- rameters & are adjusted by minimizing the combined loss of (i) reconstructing available observations (Lob)), (ii) softly enforcing the PDE constraints on the domain (L. and (iii) fulfilling the boundary (Lo) and initial conditions (Lim), i.e.
θ = arg min [w) Lobs X, tt Uobs: θ) + w2 (O) + wgLx(0) + we Timi (+)]
(5)
with loss weights wi approximated via the average L.² loss over the observations and via sampled boundary and initial conditions, respectively. should be noted that the formulation of the forward and inverse problem are identical in this setting, observations and initial conditions are implemented in a similar manner.
R20- A common choice for Lobs, La. and Linit is the expected L loss,
Enforcing the PDE. the soft enforcement of Eq. [ requires a similar discretization step for approximating an integral over the continuous signal domain,
Although PINNs are by nature mesh-free, the PDE loss Ly in Eq 5 used for
/ [I,f((,,))) di = Ep)e. [1e
1 Ly( 10,, 7 X 2
I. 21),1 (6)
n
with p(t, æ) being supported on [] x Ω. The points {(t(), 0 C [0, x S which the PDE loss is evaluated are commonly referred to as collocation points. This formulation of PINNs for solving Eq. is an Eulerian one, as the function fe is updated by evaluating the PDE with respect to collocation points fixed in space. Initial approaches for selecting the collocation points in PINNs relied on a fixed grid (Lagaris et al., 1998 Rudd 2013 Lagaris al., 2000), followed up by work proposing stochastic estimates of the integral via (Quasi-) Monte Carlo methods (Sirignano & Spiliopoulos/ 2018 Lu et al.] [202]] Chen et al., 2019) or Latin Hypercube sampling (Raissi et al. 2019). However, these approaches to Eulerian PINNs cannot be directly applied if there are no known boundaries or boundary conditions, e.g. for s2 = Rd. Additionally, problems can arise if the constrained region is large compared to the area of interest. Considering for example the shock wave (of a compressible gas) in a comparably large space, most collocation points would fall into areas of low density. We argue that due to the locality of particle interactions, the regions with higher density are more relevant for regularizing the network.
To address these shortcomings of previous methods, we propose mesh-free and adaptive approach for sampling collocation points, illustrated on the example of compressible fluids. By changing p(t, x) to the distribution over the particle positions in the fluid we effectively change the loss functional in [6 We then generalize to other settings, such as thermodynamics, by interpreting a positive, scalar quantity of interest with finite integral on PDES that can be derived based on local particle interactions or can be shown to be equivalent to such a view, as for example is the case for the heat equation with its connection to particle diffusion. Notably, we do not require the introduction of Lagrangian updates, as classical mesh- free methods do, which would be based on evaluating the PDE with respect to moving particles (see also section
particle density. Within this work specifically focus
Main contributions. The main contributions of this paper are as follows:
We demonstrate that PINNs with uniform sampling strategies (and refinement methods based on uniform proposals) fail in settings with spatially sparse signals as well as in unbounded signal domains; these problems can severely degrade the network's predictive performance.
2

---

Under review as a conference paper at ICLR 2023
In order to overcome these limitations of existing approaches, we propose truly mree--free version of Eulerian PINNs, in which the collocation points are sampled using physics- motivated MCMC methods. By staying within the Eulerian framework, avoid conceptual challenges of classical mesh-free methods based on Lagrangian updates such as the enforce- ment of boundary conditions.
The proposed model is applicable to a huge range of dynamical systems governed by PDEs that share an underlying microscopic particle description, such as several hydrodynamic, electro- and thermo-dynamic problems.
We rigorously evaluate and compare our proposed method with existing approaches in high-dimensional settings. Compared to existing mesh refinement methods, significantly fewer collocation points are required to achieve similar or better predictive performances, while still being more flexible.
RELATED WORK
Mesh-Free Fluid Dynamies. Classical mesh-free approaches in computational fluid dynamics based on non-pann paramarramtcc function representations, with Smoothed Particle Hydrodynamics (SPH) (Lind et al. 2020; Gingold & Monaghan 1977) being the most prominent example, In SPH, fluid properties such as the density and pressure are represented by discrete set of particles and interpolated using a smoothing kernel function. For updating the function forward in time, the particles have to be propagated according to the Lagrangian formulation of the PDE, relying on the kernel for computing spatial derivatives. One of the benefits of such a representation is that mass is conserved by construction. However, Lagrangian updates become challenging when enforeing boundary conditions, requiring the introduction of ad-hoc "dummy" or "mirror" particles (Lind et al. 2020). Instead, we present mesh-free, particle-based, PINN that does not require Lagrangian updates, and is already applicable in the Eulerian formulation. It should be noted that the proposed PdPINNs can in principle be combined with Lagrangian updates such as proposed by Raissi et al.) (2019) and later by Wessels et al. (2020). But as the intention of this work is to improve upon current Eulerian PINNs, we refer to future work for the comparison and extension to the Lagrangian formalism.
Alternative Meshes and Losses for PINNs. Recent work proposes local refinement methods
for PINNs by adding more samples within regions of high error (Lu et al. 2021 Tadiparthi & Bhattacharya 2021). Residual adaptive refinement (RAR) is suggested by Lu et al.
2021). which
is based on regularly evaluating the PDE loss on a set of uniformly drawn samples. The locations corresponding to the highest PDE loss are then added the set of collocation points used in training.
Tadiparthi & Bhattacharyal (202I, preprint) further enhance RAR by learning a linear map between the uniform distribution and the distribution
the PDE loss by optimizing optimal transport objective. By sampling uniformly and subsequently transforming these samples, it is attempted to focus on regions of higher error. Due the conceptual similarity RAR, we will denote this method as "OT-RAR". The work of Nabian et al. (202T) explores Importance Sampling based on the (unnormalized) proposal distribution fl (t. x) for more sample efficient evaluation of Eq. Samples are drawn using a variation of Inverse Transform sampling (Steele 1987).
However, in all these cases the underlying mechanism for exploring regions of high error is based on (quasi-) uniform sampling within the boundaries. As such, they do not resolve the issues of unknown boundaries and will furthermore be infeasible in higher dimensions.
Kinetic Theory:
From particles PDEs. Kinetic theory shows that essential conservation laws of
fluids can be derived from microscopic (or molecular) viewpoint (Born & Green 1946). Interactions describing the dynamics of fluid are described starting from set of individual particles. The basis of this approach is the so-called molecular distribution function vi over phase space, i.e. (tt, such that
/S 11(1, 2,, v)dddda
is the probability that a molecule with a velocity within Av = Au, AuzAvs occupies the volume Ar An Azz. Based on this distribution function, it is possible to define common quantities
3

---

Under review as a conference paper at ICLR 2023
as the (mass particle) density, (local mean) velocity, and macroscopic PDEs by considering the local interactions of individual particles. The one- one-particle phase space is commonly known from its application in the Boltzmann equation for modelling two-body interactions describing gases (Green 1956) and active matter (e.g. flocks of birds) (Bertin et al. 2006). The more general form including higher interaction terms is necessary for deriving conservation laws of liquids (Born & Green 1946).
PARTICLE-DENSE PINNS
In this section we introduce the concept of mesh-free particle-density PINNS (pdPINNs). Firstly, we examine limitations of the common PDE loss in Eq. [6 and, secondly, we present a solution by integrating over the position of particles instead of the full support of the signal domain.
The underlying assumption of our approach is that the dynamics described by the PDE can be explained in terms of local interactions of particles. This is the case, for instance, for commonly considered dynamics of gases, liquids or active particles (Hoover & Hoover 2003 Toner & Tu 1995).
Existing limitations of Eulerian PINNs. Consider the problem of modeling a (possibly non-steady) compressible fluid, i.c. a fluid with a spatially and temporally evolving density p(t, x) and velocity v(t, x). For the sake of notational brevity, we will denote these by p and w. Given noisy observations, our particular interest lies in the prediction of particle movements, hence in the approximation of the density (and potentially other physical quantities) with neural network pe- Additional quantities such as the velocity or pressure might also be observed and modeled.
Commonly, the PDE then serves as a physics-based regularizer of the network by enforcing the PDE loss L, in Eq. 6 during standard PINN training. For this, L5 is evaluated on a set of collocation points that are, Tor example, uniformly distributed on a bounded region. However, the limitations of this approach already become apparent when considering a simple advection problem defined by the following PDE:
ppt v. (Vp) = 0.
(8)
Figure iilluttaats a one-dimensional case on the domain [0, 7] x 2 with S2 = IR, and a known constant velocity u о 1. We measure the density ( at different (spatially fixed) points in time and space {(t(), a ())) }} on which a neural network (t, x) is trained. For optimizing the standard PDE loss L given in Eq. 6 we would require bounded region f2g b] C Q with < b and a, E R. This, in turn, leads to two issues:
1. Since the moving density occupies a small subset of f2 uniformly distributed collocation points within Gg will enforce Eq. 8 in areas with low-density. This results in insufficient regularization of pe.
2. Defining suitable bounded region Qg requires a priori knowledge about the solution the PDE, which is generally not available. Choosing too tight boundaries would lead to large parts of the density moving out of the considered area Qg. Too large boundaries would instead lead to poor regularization as this would worsen the sparsity problem in issue (1.).
In practice, most Eulerian PINNs approaches opt for naively defining a sufficiently wide region Ug. resulting in poor reconstruction. In the context of our advection problem, this is showcased in Figure [b] To properly resolve the aforementioned issues, one should (i) focus on areas that have relevant regularizing effect on the prediction of pe and (ii) adapt to the fluid movements without being restricted to predefined mesh.
4

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 1: Advection experiment in ID: (a) ground truth at time t1 and (c) time tg. (b) density prediction with uniform collocation points and (d) particle-denstt-baale collocation points for t E [11, 12), with crosses indicating sampled points.
Mesh-Free Eulerian PINNs.
We thus propose to reformulate the PDE loss Eq. 6 as the expecta- tion of ff 1)1 with respect to the molecular distribution (t, x) introduced in the related work section
/ (tt, (11f(((.. x)
Oe) = /r=0//
dæ dt.
This completely removes the need of defining ad-hoc boundaries while providing the ability to flexibly focus highly relevant regions, i.e. those that are more densely populated. the particle density corresponds directly to the occupation probability of molecule (t, x) with changed normalization constant, we can estimate Lya via samples drawn from the normalized particle density, which is denoted as pv. For homogeneous fluids, this coincides with the normalized mass density.
In summary, we propose to draw collocation points from the normalized density:
(1,, ~ (t((,x)) pt),
(10)
The true particle positions and the density PN however unknown in practice. Instead, have to rely on the learned density (t x) as a proxy provided by the neural network. We denote the associated normalized PDF by go (t, a:) z pe (t, æ)) with support on [0, 7) Ω. The PDE loss is
then defined as the expectation w.r.t. (t,,, 2:):
[1sfot,. m1]]3] = - // (t)
Ipa (0) (t.
(lfetzz (.)
dx dt.
(11)
In order to approximate this integral, samples need be drawn from (t, x). This can be done in a principled way by using dynamic Monte Carlo methods, despite the fact that the normalization constant Z is unknown. We highlight that, in contrast the mesh-based loss in Eq. 6 the loss in Eq. is also suitable for problems on unbounded domains such as S2 = Rd
Applicability of pdPINNs. Although motivated in the context of an advection problem, the proposed approach is generally applicable to wide range of PDEs. The advection equation X can be seen as special case of mass conservation (assuming physical principles expressed continuity equation. This continuity equation relates temporal changes of the fluid density to spatial changes of the flux density pv through
v 0), which one the fundamental
a.p+ V (pv) 0.
(12)
Another common physical process that is suited for approach is diffusion, such in the Heat Equation, where local interactions partieles give rise to the following PDE (as established by Fick's second law):
0,7 v 2T =0
(13)
where T denotes the temperature interpreted as density, a the thermal (or mass) diffusivity, and 2 the Laplacian operator. By introducing additional constraints to the diffusion and mass-conee muss-conservation. one can describe viscous fluids with the Havier-Stokes equations or even self propeeeo,,,,,, active particles, for which Toner and Tu TTnnr & Tu 1995 Tu et al. 1998 Toner & Tu 1998) introduced
5

---

Under review as a conference paper at ICLR 2023
hydrodynamic equations. Other possible applications involve Maxwell'' 's equations for conservation of charge in electrodynamics, as well as the distribution of Brownian particles with drift described by the Fokker-Planee equations. In general, scalar field (with a finite integral) of interest can be interpreted as a particle density, and (ii) the local interactions of these particles give rise to the considered PDEs.
method is applicable in settings where (i) non non-negative
MODEL AND IMPLEMENTATION
wide range of different network architectures and optimization strategies for PINNs have emerged. They emphasize well-behaved derivatives with respect to the input domain (Sitzmann et al. 2020), allow higher expressivity for modelling high frequency data (Tancik et al., 2020; Wang et al. 2021b),
or resolve gradient pathologies within PINNs (Wang et al., 2021a). As our method does not rely on a specific architecture, any such improvement can be easily combined with the proposed pdPINNs. For the experiments in this submission will use simple fully-connected networks with sinusoidal (Sitzmann et al. 2020) or tanh activations (see section
Finite total density. For reformulating the predicted density pe as a probability, we have to ensure non-negativity well as finite integral over the input domain Q. Non negaaivvtty can for example be achieved via a squared activation function after the last layer. An additional bounded activation function g is then added, which guarantees the output to be within a pre-specified range [0, Cmaz]- The integral IRd can then be enforced to be finite by multiplying the bounded output with a Gaussian kernel. Summarizing these three steps, let ee denote the output of the last layer of our fully connected neural network and Peamss (x) = N N2; μ1 ),, then we predict the density as
(14) In practice, the choice of Cmax does not affect the model as long as it is sufficiently large. The used mean and covariance E' are maximum likelihood estimates based on the observations æ, i.e. the sample mean and covariance Σ of the sensor locations. To allow more flexibility in the network, we add a scaled identity matrix to the covariance V = E + c L, which can be set to a large value for solving PDEs when only initial conditions, but no observations, are available.
pe (t x)) Peauss (x) g( )²)) Craxpeauss (z)).
Markov chain Monte Carlo (MCMC) sampling. Finally, MCMC methods allow us draw samples from the unnormalized density (t, 2). We consider several MCMC samplers and empha- size that the wide range of well-established methods offer the ability to use a specialized sampler for the considered problem, if the need may arise, Gradient-based samplers such as Hamiltonian Monte Carlo (Duane et al., 1987: Betancourt 2017) are particularly suited for our setting, as the gradients of pe with respect to the input space are readily available. For problems where boundaries are known and we have to sample from constrained region, a bijective transformation is used so that the Markov chain may operate in an unconstrained space (Parno & Marzouk (008).. In our experience, both Mctropolis Hastings and Hamiltonian Monte Carlo already worked sufficiently well for a wide range of PDEs without requiring much fine-tuning. We highlight that PdPINNs do not directly depend on MCMC as a sampler, and alternative sampling methods such as modern variational inference schemes (Rezende & Mohamed] 2015) can also be directly used as substitute.
For details regarding the samplers used and implementation we refer to the Experiments section 5 and Appendix section A.1
EXPERIMENTS
In this section we demonstrate the advantages of pdPINNs compared to uniform sampling, importance sampling (Nabian et al.] (021) as well as the adaptive refinement methods RAR (Lu al. (202)) and OT-RAR (Tadiparthi & Bhattacharya 2021). Despite the term uniform sampling, we rely in all our experiments quasi-random Sobol sequences for more stable behavior in the low samples regime. To guarantee a fair comparison, we considered slight variations of the proposed implementations of RAR and OT-RAR, that only a limited number of collocation points are used. For the PdPINNs consider multiple MCMC schemes, including inverse transform sampling (IT (IT-pdPINN), Metropolis- Hastings (MH-pdPINNN, and Hamiltonian Monte Carlo (HMC-pdPINN) methods.

---

Under review as a conference paper at ICLR 2023
The models in sections 5.1 and 5.2 are implemented in PyTorch (Paszke al. 2019), with custom Python implementation of the MH and Inverse Transform samplers. For the Fokker-Plamn experiment in section 5.3| we make use of the efficient MCMC implementations provided by TensorFlow probability (Abadi et al. 2016 Lao et al. 2020) and the utilities of the DeepXDE library (Lu et al., 202Tp. More details, well further experiments comparing the wall-time of the various samplers, are provided in the Appendix with the code being provided in the supplementary material.
5.1 MASS CONSERVATION FOR SIMULATED PARTICLES
As a challenging prediction task we consider a setting motivated by the real world problem of modelling bird densities and velocities measured from a set of weather radars (Dokter al,, 2011 Nussbaumer et al. 2 2021 or more generally the area of radar aeroecology. A non- non-steady compressible fluid three dimensions is simulated by propagating fluid parcels through pre-defined velocity field, i.e. the fluid is simulated using the conservation of mass as the underlying PDE (see To provide the network with training observations, we introduce a set of spatially fixed sensors (comparable to radars) which count over time the number of fluid parcels within radius and over 21 contiguous altitude layers. Another disjoint set of sensors is provided for the validation set while the test performance is evaluated
Eq. 12
grid. The birds-eye view of the setting is shown in Figure 2a where circles indicate the area covered by the radars. Figure 2b additionally shows the 3D simulated data projected along the z-axis and over time. In the Appendix section A.33 we describe the data generation and training setting in detail and provide the corresponding code in the supplementary.
2000
17500
1000
66
1540
04
1000
6
O+
0
400
5000
D
240
00 05
10
15
2.0
25
30
(b)
(a)
Figure 2: Visualization of the 2D compressible fluid experiment. (a) Bird-eye view of the ground truth particle density. (b) z-projection of the density over time, obtained summing over the xy grid cells.
For modeling the density and velocity, two sinusoidal representation networks (SIREN) (Sitzmann et al. 2020) Pe, (1, æ)) and Ve, (1, x) are used, which are then regularized by enforcing the continuity equation for the conservation of mass (see Eq. 12). To showcase the sample efficiency of pdPINNs, experiments are performed over a wide range of collocation points (256 to 65536). In each setting the PDEEweighhts u2 (see Eq. 5) were selected with grid search based on the highest st quartile R² in a validation set. The resulting box-plots of the test R² are provided in Figure 3 where the "Baseline" corresponds to training without any PDE loss. The proposed PdPINN approach clearly outperforms alternative (re-)sampling methods across all numbers of collocation points, Already with very few collocation points (512) PdPINNs achieve results that require orders of magnitude more points (32768) for uniform sampling. Finally, we observe that the performance gap shrinks as the number of collocation points increases, eventually converging the same limiting value. Even when getting close to the memory limit of NVIDIA Titan X GPU, other sampling strategies at best achieve comparable results with PdPINNs. the Appendix (Figure A.6) we provide an additional qualitative comparison of the mass conservation between OT-RAR and MH-pdPINN 2048 samples.
As an additional experiment we simplified the setting by projecting the data onto the xy-axis, i.e. the birds-eye view, which is common setting for geostatistical data (e.g. in Nussbaumer al. (2019)). The results in this 2D setting, which are provided in the Appendix (Figure (A.) and described in details in section A.3] are very similar in nature the 3D setting, although with smaller performance gap with respect to alternative sampling methods. This decrease of the gap is to be expected, as the lower dimensional space is much easier explore with uniform proposals.
7

---

Under review as a conference paper at ICLR 2023
3,750
Method
2.675
Baseline
2.500
RAR
importance sarr pling
IT PTPPIIN
3,000
2045
0192
32700
00336
Number Collocation Points
Figure 3: Explained variance of vp evaluated on the test set. for different number of collocation points for the 3D mass conservation experiment and for 10 different seeds.
5.2 HEAT EQUATION
We further consider 2D diffusion problem, namely the heat equation introduced in section 3 where randomly distributed sensors provide measurements of the temperature. We focus on general setting with the initial conditions being zero temperature everywhere except for specified region, as shown in Figure 4a and we let the system evolve for t E [0, 0.2]. The networks are only provided sensor measurements of the temperature; for further details the Appendix section A.4
Temperature predictions for PINNs with uniform sampling and pdPINNs are illustrated in Figure 4b 4c] respectively, with the ground truth in Figure 4a We can observe that the uniform sampling strategy does not allow to focus on the relevant parts of the domain, i.e. regions with high temperature, and that it visibly fails reconstruct the temperature profile. In contrast, the PdPINN promotes sampling in regions of higher density and predicts the true temperature more reliably. We also evaluate quantitatively the performance of the two approaches in terms of the R² test error over the predicted temperature and illustrate the results in the Appendix section A.4 where we again observe the same convergence between uniform sampling and PdPINNs for high numbers of collocation points.
(c)
Figure 4: Temperature predictions of the heat equation experiment (trained with 128 collocation points) at time 0.044. (a) Ground truth (b) uniform sampling, and (c) pdPINN.
5.3 FOKKER-PLANCK PLANCK EQUATION
For demonstration of a forward problem, i.e. a setting without any observed data but only initial conditions, we solve the Fokker-Planck (FP) equations in setting where an analytical solution is available (cf. Särkkä & Solin (2019). The FP equations describe the evolution of the probability density of the movement of Brownian particles under drift. More specifically, assume we given particles at time to. which are distributed according to p(to. 2). Let the movements of these particles be described by the following stochastic differential equation, where W, denotes the standard Wiener process:
dXX, p(l, X,) dt (1, X,) dW.
(15)
8

---

Under review as a conference paper at ICLR 2023
RAR
IT-pdPINN
20
2
KL (
pogplt, x)
sampler uniforr
p(t, as sampler
Analytical Solution
RAR
1FPdPINN
MI pdPINN
HMC-pdPINN
pt sampler
W
5000
10000
15000 steps
20000 25000
30000
(a)
(b)
equation in ID. (a) KL divergence between the true target distribution and approximation during training, (b) predicted log (1, after training, cropped to x E [-0.5, 0.5].
Figure 5:
with known drift ((x,, 1) and diffusion coefficient D(X,, t) = (X,, The FP equation for the probability density 2(t,x)) of the random variable X, then given by
0 p(t, x) dr x)(t,=)p(t,x)) +
a
8² [T (t(t,)) Ox.2
(16)
We train a network to predict the (probabilityy density (t x) given a known sinusoidal drift and constant diffusion, which are discussed in detail in the Appendix. Data is only provided for the initial condition, and the PDE loss is based on Eq. [6 within the space = [-.15, 1.5] and time estimate
I E [1,, 1]. As the analytical solution is available in form of probability density, the KI. divergence K Lipl (pe) to evaluate the performance. Furthermore, we can sample collocation points from the true particle distribution p(t, .x) (referred to as "p(t, æ) as sampler"), offering "best case scenario" of pdPINNs. A total of 5000 collocation points were used, and weights were manually tuned based on the error on a validation set. Figure (alure/aassa shows the evolution of KL. divergence during training, highlighting that PdPINN based methods require fewer steps to achieve low divergence. In addition, sampling from the true particle distribution leads to the fastest improvement and the lowest divergence after 30000 training steps. A qualitative comparison of the results is given in Figure 5b showing that RAR and uniform sampling fail to propagate the sine wave forward. The ground truth of the problem and wall-times for different methods are given in the Appendix section A.5
6 CONCLUSION
In this work, we introduced general extension to PINNs applicable to great variety of problem settings involving physics-based regularization of neural networks. In order to overcome the limita- tions of classical mesh-based Eulerian PINNs, we introduce novel PDE loss that is defined with respect to the particle density in rather general types of PDEs. By employing MCMC methods to sample collocation points from the density approximated by the network, we derive an efficient and casy-to-implement improvement for providing a more appropriate regularization objective in PINNs. In particular, our new pdPINNs are completely mesh-free, thereby overcoming severe efficiency problems of classical PiNNs in high- siggh-imensional and sparse settings. Further, the absence of a mesh allows us elegantly handle settings with uncertain or unknown domain boundaries.
As we have demonstrated, our method is applicable to a wide spectrum of PDEs, ranging from hydrodynamic flow problems electro- and -hermo-dynamic problems, as well as more general applications of the Fokker- -lanckk equations.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abadi, M., Barham, P., Chen, J., Chen, Z.,, Davis, A., Dean, Devin, M., Ghemawat, S., Irving, G., Isard, M., al. [TensorFlow): system for [Large-Scale] machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp. 265-283, 2016.
Bertin, E., Droz, M., and Grégoire, G. Boltzmann and hydrodynamic description for self-propellee particles. Physical Review E, 74(2)022101, 2006.
Betancourt, M. conceptual introduction to hamiltonian monte carlo.
arXiv preprint
arXiv:1701.02434, 2017.
Born, M. and Green, H. S. A general kinetic theory of liquids i. the molecular distribution functions. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 188 (1012):10-18, 1946.
Chen, J., Du, R., Li, P., and Lyu, L. Quasi-monte carlo sampling for machine-leaminng partial differential equations. arXiv preprint arXiv:1911.01612, 2019.
Dissanayake, M. and Phan-Thien, N. Neural-network--asse approximations for solving partial differential equations. communications in Numerical Methods in Engineering, 10(3):195-201, 1994.
Dokter, A. M., Liechti, F., Stark, H., Delobbe, L., Tabary, P., and Holleman, Bird migration flight altitudes studied by network of operational weather radars. Journal of the Royal Society Interface, 8(54):30-43, 2011.
Duane, S., Kennedy, A. D., Pendleton, B. J., and Roweth, D. Hybrid monte carlo. Physics letters B, 195(2):216-22) 222, 1987.
Earl, D. J. and Deem, M. W. Parallel tempering: Theory, applications, and new perspectives. Physical Chemistry Chemical Physics, 7(23):3910-3) 2005.
Flamary, R., Courty, N., Gramfort, A., Alaya, M. 7., Boishunon, A., Chambon, S., Chapel, I..,
Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N. T., Janati, H., Rakotomamoniy. A., Redko, I., Rolet, A., Schutz, A., Seguy, V., Sutherland, D. J., Tavenard, R., Tong, A., and Vayer, T. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 2021.
Freitag, M. A. Numerical linear algebra in data assimilation. GAMM MAMM--Miteillngen, 43200020000001, 2020.
Gingold, R. A. and Monaghan, J. Smoothed particle hydrodynamics: theory and application to non-spherical stars. Monthly notices of the royal astronomical society, 181(3):375-38,, 1977.
Green, M. S. Boltzmann equation from the statistical mechanical point of view. The Journal of Chemical Physics, 2565)5836-855, 1956.
Hoffman, M. D., Gelman, A., et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15():1193-1623, 2014.
Hoover, W. G. and Hoover, C. Links between microscopic and macroscopic fluid mechanics. Molecular Physics, (0(111):159-1553 2003.
Jagtap, A. D., Kharazmi, E., and Karniadakis, G. E. Conservative physics-infon informed neural net- works on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods Applied Mechanics and Engineering, 365:1 13028, 2020.
Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv:1412.6980, 2014.
arXiv preprint
Knott, M. and Smith, S. On the optimal mapping of distributions. Journal of Optimization Theory and Applications, 43(1):39-49, 1984.
Lagaris, I. E., Likas, A., and Fotiadis, D. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, -55)987-10000 1998.
10

---

Under review as a conference paper at ICLR 2023
Lagaris, I. E.,, Likas, A. C., and Papageorgiou, D. G. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 1165):041-1049, 2000.
Lao, J., Suter, C., Langmore, I,, Chimisov, C., Saxena, A., Sountsov, P., Moore, D., Saurous, R. A., Hoffman, M. D., and Dillon, J. V. tip. meme: Modern markov chain monte carlo tools built for modern hardware. arXiv preprint arXiv:2002.01 184, 2020.
Lind, S. J., Rogers, B. D., and Stansby, K. Review of smoothed particle hydrodynamics: towards converged lagrangian flow modelling, Proceedings of the Royal Society A, 476(2241):2019080) 2020.
Lu, L., Meng, X., Mao, Z., and Karniadakis, G. E. Deepxde: deep learning library for solving differential equations. SIAM Review, 63(1):20--228 2021.
Marchetti, M. C., Joanny, J.-F.. Ramaswamy, S.. Liverpool, T. B., Prost, J. Rao, M., and Simha, R. A. Hydrodynamics of soft active matter. Reviews of modern physics, 85(3):1143, 2013.
Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, K., Singh, S., Rathnayake, T., Vig. S., Granger. B. E., Muller, R. P., Bonazzi, F., Gupta, H., Vats, S., Johansson, F., Pedregosa, F., Curry, M. J.. Terrel, A. R., Roučka, v., Saboo, A., Fernando, I., Kulal, S., Cimrman, R., and Scopatz, A. Sympy: symbolic computing in python. PeerJ Computer Science, 3:3 January 2017. ISSN 2376-5992. doi: 10.7717/peerj-cs. 103..
Nabian, M. A., Gladstone, R. J., and Meidani, H. Efficient training of physics-informed neural networks via importance sampling. Computer-Aided Civil and Infrustructure Engineering, 36(8): 962-977, 2021.
Nussbaumer, R., Benoit, L.. Mariethoz, G., Liechti, F., Bauer, S., and Schmid, B. A geostatistical approach to estimate high resolution nocturnal bird migration densities from a weather radar network. Remote Sensing, 11(19):2233, 2019.
Nussbaumer, R., Bauer, S., Benoit, L., Mariethoz, G., Liechti, F., and Schmid, B. Quantifying year-round nocturnal bird migration with a fluid dynamics model. Journal of the Royal Society Interface, 18 18(179)2202101
2021.
Oosthuizen, P. H. and Carscallen, W. E. Introduction compressible fluid flow. CRC press, 2013.
Parisi, D. R., Mariani, M. C.,, and Laborde, M. A. Solving differential equations with unsupervised neural networks. Chemical Engineering and Processing: Process Intensification, 4788-1)7115-771 2003.
Parno, M. D. and Marzouk, Y. M. Transport map accelerated markov chain monte carlo. SIAM/ASA Journal on Uncertainty Quantification, (122:645-822, 2018.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein,
N., Antiga, L., Desmaison, A., Kopf. A., Yang. E., Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, nigh-performance deep learning library. In Wallach, H., Larochelle, H.. Beygelzimer, A., d'Alché and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
F., Fox, E.,
Raissi, M., Perdikaris, P., and Karniadakis, G. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Recktenwald, G. W. Finite-difference approximations to the heat equation. Mechanical Engineering, 10(01), 2004.
Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Rudd, K. Solving partial differential equations using artificial neural networks. PhD thesis, Duke University, 2013.
11

---

Under review as a conference paper at ICLR 2023
Särkkä, S. and Solin, A. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Sirignano, J. and Spiliopoulos, K. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 339-1339-136,, 2018.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33: 7462-7473, 2020.
Steele, J. M. Non-uniform random variate generation (luc devroye), 1987.
Szabó, B., Szöllösi, G., Gönci, B., Jurányi, Z., Selmeczi, D., and Vicsek, T. Phase transition in the collective migration of tissue cells: Experiment and model. Physical Review E, 74(6):061908, 2006.
Tadiparthi, V. and Bhattacharya, R. Optimal transport based refinement of physics-informed neural networks. arXiv preprint arXiv:2 105. /2307, 2021.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil. S., Raghavan, N., Singhal, U., Ramamoor- thi, R., Barron, J., and Ng, R. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
Toner, J. and Tu, Y. Long-range order in a two-dimensional dynamical XY model: how birds fly together. Physical review letters, 75(23):4326, 1995.
Toner, J. and Tu, Y. Flocks, herds, and schools: A quantitative theory of flocking. Physical review E, 58(4):4828, 1998.
Tu, Y., Toner, J., and Ulm, M. Sound waves and the absence of Galilean invariance in flocks. Physical review letters, 80(21):4819, 1998.
van Milligen, B. P., Tribaldos, V., and Jiménez, J. Neural network differential equation and plasma equilibrium solver. Physical review letters, 75(20):3594, 1995.
Wang, S., Teng, Y., and Perdikaris, Understanding and mitigating gradient flow pathologies in physice-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055 -3388,, 2021a.
Wang, S., Wang, H., and Perdikaris, P. On the eigenvector bias fourier feature networks: From regression solving multi-scale pdes with physics-informet neural networks. Computer Methods in Applied Mechanics and Engineering, 384: 13938, 2021b.
Wessels, H., Weißenfels, C., and Wriggers, P. The neural particle method-an updated lagrangian physics informed neural network for computational fluid dynamics. Computer Methods in Applied Mechanics and Engineering, 368:1 13127, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 BACKGROUND SAMPLING FOR PDPINNS
At initialization, the network prediction pe is random and thus does not carry any useful information, ie. sampling from this density would be meaningless. Therefore, we start training the PdPINNs with warm-up phase in which samples are obtained from a pre-specified background distribution:
x Phg (1, x:) = p()))ag())))
(17)
with p(() T). To avoid introducing mesh, we could rely on the previously estimated Gaussian distribution introduced in Section 4 i.e. Phg (t) = Pgauss (2=). As a second alternative, approach we consider random linear combinations of the convex hull of (z((*) Ji=1 spanned by data points summarized as rows of matrix Z E Rexd This leads to æ m2 with weight m E IR which can be drawn from a Dirichlet distribution, i.e. m ~ Dir(a 1). Of course, uniform sampling mechanism on a defined region is also suitable and the definitive choice depends on the data and PDE hand. However, we found that all of these methods work well in practice.
We initially draw all samples from the background distribution, and then slowly increase the pro- portion of samples obtained from the particle density, as we found that leaving some background samples slightly helps in the training.
A.2 IMPLEMENTATION OF RAR AND OT-RAR
For our comparison, we considered the adaptive refinement methods RAR and OT-RAR, proposed by Lu et al. (202I) and Tadiparthi & Bhattacharyal (202T, preprint). Both methods rely on consecutive refinements of fixed grid in the initial proposal. The number of collocation points is steadily increased and collocation points once added will not be removed. To allow for fairer comparison, we adapt both methods to use a limited budget of points, and in addition we regularly resample them. This leads slightly modified version of the methods which is similar in spirit. For learning the linear mapping proposed by Tadiparthi & Bhattacharya (202I), we rely on the PyOT (Flamary jet al. 202T) implementation of Knott Smith (1984). The pseudo-code for sampling a set of collocation points is given in Algorithm and Algorithm 2 The required input fe refers to the PDE approximated by the network, as discussed in Section T refer the original papers.
For more specific details on the methods we
Algorithm 1 Adapted RAR
Input: fe. uniform distribution Us. number of col. points k. previous col. points Xprev
X Srup [21 D2...... xx| with x;
Us
Sample proposals D Concatenate old and new points Keep top proposed points based on fe
Xcomh < concat(X Xonnct((ppeev... prev. X X
Xnew
<
Il,f (X k)
Output:
Xnew
A.3 EXPERIMENTS: CONSERVATION OF MASS
In the supplementary material we provide code in Python for the data generation and for the PdPINN model. Below we provide the details for all the experiments conducted. Furthermore, we provide short videos showing the predicted density movements for each different approach. More details on this can be found in the README. html provided in the supplementary files.
All experiments were run on a computing cluster using Nvidia GeForce GTX Titan X GPUs with 12 GB VRAM. Settings that required more memory were on RTX8000 with 48GB VRAM. Up to 16 Titan X GPUs could be used in parallel, or 4 RTX8000. In most settings, training in each experiment took less than 10 minutes.
13

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Adapted OT-RAR
Input: fe. uniform distribution Us.
number of col. points k number of points for empirical distribution 2k, previous col. points Xpre-
X peup
-- 1 I2.... Dg
7 with æs ~ Us
D Sample proposals Concatenate old and new points
X comb
concat( X prev X Xprop)
X
mageet
(X j) 1 22...... æ,I" 1 with æ; Us
> j samples for target empirical distribution
source
b j samples for source empirical distribution
Mor
LinOT (Xooncer Xiaget)
> Obtain linear operator that maps target distribution
Xnew
- [1 ....... xs4]" with x, Us
Sample uniformly
X Xmap
< MoT(XXew)
Map samples target distribution
Output:
X Xmap
A.3.1 ADDITIONAL EXPERIMENTAL RESULTS
3D Setting. Figure A.6 showcases the projection of the density in the onto the axis for a random run of the OT-RAR method and the Metropolis-Hasting: based pdPINN when using 2048 collocation points. The OT-RAR PINN shows disconnected density predictions that clearly violate mass con- servation, whereas the Metropolis Hastings based pdPINN is capable of mostly preserving it. The boxplot in Figure A.8 highlights the difference required number of collocation points of
10000
5030
05
14
Figure A.6: Mass conservation experiment (3D): Predictions (obtained with 2048 collocation points) summed over zy grid cells to obtain z-axis projection over time.
Toxt R2 Saytibrrake)
80322
4033t
Figure A.7: Mass conservation experiment (3D): Boxplot of test R2 of vp comparing PdPINN and uniform sampling with factor 8 difference for the number of collocation points. For each method, used PDE weight was selected based on the highest 1st quartile R² in a validation set.
14

---

Under review as a conference paper at ICLR 2023
2D Setting. As mentioned in Section slightly altered setting, where the data is projected onto the xy-plane, reducing to a 2D+Tima problem. The general setup is similar to the 3D setting, although smaller network and different training parameters are used, which are listed in the following sections below.
repeated the Conservation of Mass experiment in
0.75
0.70
Melhoc
R'
Baseline
RAR
0.60
mmpottnnc-samaling
(T-dPPNN
N11.dPPNN
0.55
4096
8192
Number Collocation
Figure A.8: Explained variance evaluatteed on the test set, for different number of collocation points for the 2D mass conservation experiment.
A.3.2 DATA GENERATION
Here we provide more detailed description for the generated data, namely the used velocity field, and the method for obtaining simulated "radar measurements".
Velocity field. The velocity field in the ry-plane was generated from a scalar potential field 1222 + and the z-component of a vector potential a : IR22 + R. Through the Helmholtz decomposition we can construct the velocity field Uxy I² + 1
xr 21
ga 6α/ōy -8a / dx
Uxy
V +
(18)
For both experiments the following fields were used:
x
yy 2 (x
2)-(2 2)):
(19)
[x]]
2 3
a
3
(20)
The derivatives obtained using the symbolic differentiation library SymPy (Meurer et al.) To add nonsteady component, the resulting velocity field is modulated in amplitude as a function of time E [0:
2017).
2
Uayt
uay
sim
mL
+ 0.05
(21)
The z (altitude) component of the velocity only depends on time and given by:
U. (1) 1.6 sin
(22)
3
Simulation. For the initial distribution the fluid, the particle positions were drawn from Gaussian mixtures. For & 30, these particles were simulated using the above constructed velocity field. Overall, the paths of the roughly 240000 parcels were simulated using basic backward Buler scheme.
'This is the 2D formulation of the Helmholtz decomposition. where the vector potential has non-zero components only along the z-axis in asa [0. 0, a)". The full decomposition is commonly written as
2y 2 + V azg-
15

---

Under review as a conference paper at ICLR 2023
Measurements. The measurements at the sensors were obtained by counting the number of particles within a given radius over multiple timesteps. The density corresponds to the mass divided by the sensor area, and the velocity is average over all the particle velocities. For the training data additional zero-mean isotropic Gaussian noise is added to all measurements. In the 3D setting, data measurements of density and velocity are obtained by 132 sensors on the xyyplane, within region -]² at 11 equidistant timesteps. In the 2D setting, the same set of sensors is used.
A.3.3 ARCHITECTURE AND TRAINING
In both experiments, the networks for density Pe, and velocity tee prediction (parameterized by θ and Og. respectively) are fully-comnected layers with sinusoidal activation functions, as proposed by SSizzmann et al. (020). The number of layers and units for each setting is shown in Table [ The sine frequency hyperparameter required in the SIREN architecture was tuned by hand according to the validation loss of the baseline model (i.e. without a PDE loss), leading to sine-freque finefreeyyncy of 12 for the 2D setting, and 5 for the 3D setting. We note that the proposed default value of 30 in Sitzmann al. (2020) heavily overfits our relatively wwwffrequency data and we thus recommend an adjustment of this hyperparameter for usage in PINNs.
For training the network, the ADAM optimizer (Kingma & Ba 2014) with learning rate of 8 x 10- (2D Setting) or 10 (3D Setting) was used. The learning rate was multiplied by a factor of 0.99 each epoch. All models were trained for 300 (3D setting) 500 (2D setting) epochs. The 2D setting was trained using full-batch gradient descent, whereas for the 3D setting we used a mini-hatch size of 6931. In all experiments trained and evaluated on 10 different random seeds.
Table A.1: Architecture for Particle Simulation Experiments.
Experiment
Input
Output Variable
# Hidden Layers
# Hidden Units
2D
[0,T] x R
Density pe, E R,,
2
256
Velocity Ue E RR²
64
3D
[0. x R3
Density pe, E R+
6
256
Velocity U02 E R
256
A.4 EXPERIMENTS: HEAT EQUATION
The dataset for the heat equation experiment was generated by numerically solving the heat equation through the finite difference method, precisely the Forward Time, Centered Space (FTCS) approxi- mation (Recktenwald) 2004). We used Dirichlet boundary conditions in form of zero temperature around squared shape far away from the relevant domain. These boundary conditions are not provided to the PINNs for slightly more difficult setting. Overall, the dataset is composed of 1000 training points, 1971120 test points and 492780 validation points. We made sure training points contained enough information about the initial condition, i.e. we selected a sufficient amount of points around the initial source of non-zero temperature. In contrast, validation and test points are taken uniformly in time and space. During the warm-up phase of the PdPINN training, collocction points were sampled uniformly, and afterwards 90% of the samples were drawn from the particle density distribution, which is proportional to the modeled temperature. Collocation points were re-sampled every 500 epochs. Differently from previous experiments, the employed architecture is a fully-connected two-layer neural network with 32 hidden units and tanh activations. The implementa- tion is in Py Torch (Paszke et al. 2019), using the ADAM optimizer (Kingma & Ba 2014) combined with exponential learning rate scheduler which multiplies the learning rate by factor of 0.9999 at each epoch, starting with rate of 10-4 and decreasing it until reaching a minimum value of 10-5 Training was terminated through early-stopping. soon the validation R² didn't improve for more than 3000 epochs.
Additional results. Figure A.9 illustrates the test R² of the predicted T averaged over 20 different seeds. Error bars correspond to 95% confidence interval for the mean estimation, based 1000 bootstrap samples, while colors indicate the different PDE weights w2 explored. As in previous settings, show that with few samples (16) the regularization enforced by the PDE loss is not strong
16

---

Under review as a conference paper at ICLR 2023
enough, leading to comparable results in both approaches (as expected). Hence PINNs and pdPINNs show similar results this regime. However, as the number of samples increases (32-64-128- (2--22-.5,), the PDE loss enforced by the proposed pdPINNS quickly and steadily outperforms uniform sampling. Lastly, we also verified that the limit of high samples (512-1024) the two sampling strategies converge, in such low-dimensional domain the uniform samples fully and densely covers the considered area. This, again, is in line with the observed results of the other experiments.
PdPINN
Uniform
0.95
0.90
Test R2 of sqrt(density)
0.85
0.80
0.75
0.70
0.65
0.60
0.55
0.50
16
32
128
256
512
1024
16
128
256
512
1024
Number of Collocation Points
Number of Collocation Points
Figure A.9: Test R² of predicted T in the heat equation experiment as a function of different number of collocation points. Results are averaged over 20 different seeds and the resulting error bars correspond to 95% confidence interval for the mean estimation, based on 1000 bootstrap samples. Different colors indicate different PDE weights w2.
A.5 EXPERIMENTS: FOKKER-PLANCK EQUATIONS TENSORFLOW
Within the Fokker- Planck experiment showcase the different training behaviors of uniform sampling, RAR, and multiple MCMC samplers. Due to the low dimensionality of the problem, we additionally consider the density. The IT sampler relies on the empirical edf estimated via uniform samples drawn over the whole domain. This method does not require building up Markov Chain, and thus very fast, but only works well in low dimensions.
(IT) sampler (Steele 1987) for efficiently sampling from
More specifically, we compare the following methods for selecting collocation points, with highly efficient implementation of the MCMC methods provided by TensorFlow probability:
I.) Uniform sampling
II.) Residual Adaptive Refinement (Lu et al. 2021
III.) PdPINN with
(IT) sampling (teele 1987)
IV.) PdPINN with Eatropolis-Hastinne (MH) MC with parallel tempering (Earl & Deem 2005)
PdPINN with Hamiltonian MC (HMC) with parallel tempering (Earl & Deem 2005) and dual averaging step-size adaptation (Hoffman et al. 2014
section 3.2)
A.5.1 SETTING AND ANALYTICAL SOLUTION
We consider the following setting over the time interval [to. f.,] = [-1, 1] with drift function noise and initial particle positions p(rlt- tu) given by
t) = (ut)= = sin (10t)
(23)
o(X,,t) = 0.06
(24)
p = to) =VV(0, 0.02 Ia)
(25)
17

---

Under review as a conference paper at ICLR 2023
The PDE has analytical solution (cf. Särkkä & Solin (2019)
which is given by
p(x[t) = (t),
(26)
p(t) u((o: (n)
(27)
(t cos() cos(10)
(28)
10
(t) = 0.0036t + 0.004.
(29)
For evaluating the deviation of our prediction the solution, we evaluate the KL. divergence between the analytical solution and the network approximation KLL t) pe t)) by sampling 10000 points from the true p(æ, t).
A.5.2 SETUP
We SIREN network and additionally sample (5000) collocation points the initial time-step, which is the default behavior of DeepXDE. An overview of the architecture and training details is given in Table A.2 Experiments were performed with NVIDIA GeForce RTX 2080 Ti and an Intel(R) Xeon(R) R) CPU E5-1660 v3 @ 3.00GHz processor.
Table A.2:
Architecture for Fokker-Planck experiments.
Experiment
Input
Output Variable
# Layers
# Units
col. points
epochs
ID
[0 IR
po(x, R2 5
64
5.000
30.000
A.5.3
WALL TIME
The wall times for the different methods provided in Figure A.10 Although Hutropolis-Hasting
and Hamiltonian Monte Carlo require more time per step compared to uniform sampling, the used inverse transform sampling achieves similar speed.
Walltime FP Experiment
uniform
RAR
sampler
IT-pdPINN
MHHpdPPINN
HMC-pdPINN
200
400
600
800
time in
Figure A.10: Total run-times for the Bokker-Planck experiment. Seeds selected randomly.
18


Reviews:
[{"review_id": "pSdY8wLxdJ", "reviewer": "Reviewer_ZR9E", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a network based method, in the realm of physics informed methods, to solve a class of PDEs. The proposed methods mainly extend the general framework of having PDE as constraints in the loss function by introducing a partial density and locality of particle interactions. It also proposes to use MCMC method to sample the collocation points.", "summary_of_the_review": "This paper proposes a novel loss function with underlying ideas of local partical density/interactions and mesh-free sampling strategy. It is a good paper with clearly written and organized context. The limitation of this paper could be due to its limited novelty and its general applicability.", "strength_and_weaknesses": "The strengths of this paper are:\n\n$\\mathbf{1}.$ The idea of using particle density and emphasizing locality of interactions in loss function is novel. It provides a rigorous view of defining a specific loss function for general PDE constrained loss function (of neural networks). The proposed method also incorporate a MCMC sampling strategy, which supports the idea very well. \n\n$\\mathbf{2}.$ The paper is very well organized and clearly written. The literature review is comprehensive along with arguments and to introduce the motivation of the proposed method. Overall, the paper is easy to read for audience in a broad area of deep learning. \n\nThe weakness of this paper:\n\n$\\mathbf{1}.$ There lacks novel proposal of network architecture, which might limit the novelty of this paper. Here, it is worthwhile noting that the novel definition of loss function does qualify as a good contribution. However, the architecture of neural network is not novel.\n\n$\\mathbf{2}.$ Author discussed the applicability of the propose methods. However, the limitation of applicability seems to be obvious as the authors stated. But, it is rather an unfamiliar area for me. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q3hKCKDqYd3", "reviewer": "Reviewer_N5cW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an extension to the physics-informed neural network approach, which utilizes a particle-based approach to more efficiently sample in the data space, and capture boundary conditions more accurately with less required samples as compared to traditional collocation points. The core contributions of the paper is the injection of microscopic fluid dynamics views rooted in smoothed particle hydrodynamics, into the mesh-free PINN approach to achieve more optimal adaptive sampling.", "summary_of_the_review": "The authors propose an extension to the PINN training approach rooted in particle-based adaptive refinement, utilizing Monte-Carlo sampling to capture the different regions, and boundaries of the problem more accurately. While well-motivated, and well-derived the approach is only a slight extension beyond the previous residual adaptive refinement approach of Lu et al., which furthermore also already utilized a Monte-Carlo integration step. The experiments, while illustrative, could take more difficult PDE-systems into consideration to showcase the true capability and limitations of the proposed approach.", "strength_and_weaknesses": "Strengths:\n- Strong embedding of the proposed approach into the current state of the literature with illustrative parallels being drawn to previous PINN approaches.\n- Well-motivated approach with a clear physical intuition behind it.\n\nWeaknesses:\n- As the approach is particle-based I would have expected an ablation analysis/comparison to particle-based inference approaches such as Sequential Monte-Carlo. The authors only consider inverse transform sampling, Metropolis Hastings, and Hamiltonian Monte-Carlo.\n- Limited set of experimental evaluations with fairly simple physical systems being considered. Would the proposed approach also work on the more difficult Korteweg-de-Vries equations? Would it work with Burgers equations, and how would it compare to the residual adaptive refinement approach of Lu et al. there? Would it scale to even more complex settings, such as the ones explored in XPINNs?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L27w9ujcqh", "reviewer": "Reviewer_cZb3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors claim to propose to sample directly from the distribution over the particle positions, eliminating  boundaries while adaptively\nfocusing on the most relevant regions. It looks higher sample efficiency and improved performance of PINNs.", "summary_of_the_review": "The author need to compare their work to compare the classical work about numerical solution of PDE. ", "strength_and_weaknesses": "Weaknesses: It looks that I do not understand what the authors' motivation and what they do to solve. Does mesh-free fluid dynamics really work?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DB2ACN2Q0gr", "reviewer": "Reviewer_ukjY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes an interesting idea based on the observation that PINNs are normally trained using uniform sampling, which might not be the optimal case. Using the proposed method, the region of interest will be naturally sampled with a higher density. This manuscript applied their method to a set of common PDE problems, including Fokker-Planck equations.", "summary_of_the_review": "This manuscript builds its idea based on an interesting observation. However, there are ablation studies that need to justify their choices and provide more advice to practitioners.", "strength_and_weaknesses": "Strength:\n- This manuscript is well-written.\n- The proposed method is easy to follow: it is true that the sampling matters in training PINNs.\n- The attached code showed good reproducibility.\n- The experiments validating this method are extensive.\n\nWeaknesses:\n- There are multiple algorithmic choices not justified in the manuscripts. For example, why use the square as the non-negative translator instead of the absolute? More ablation studies are expected to examine them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4gIxjYjfex", "reviewer": "Reviewer_J4Gd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the use of the expected molecular distribution function as the loss function to train a Physics Informed Neural Network. Then, the authors use MCMC to sample the molecular distribution function to compute the expectation using the output of the neural network for the molecular distribution function. The authors show three applications: mass conservation for simulated particles, heat equation and fokker planck equation (stochastic differential equation). ", "summary_of_the_review": "Overall, the paper lacks a good justification for the loss function choice and improvements to the learning representation this choice brings to the table. The experiments do not justify the use of the new loss function and the advantages are not sufficiently highlighted. Overall the idea of the paper is not a significant improvement in the field.", "strength_and_weaknesses": "The strength of the paper is the realization that existing PINNs with uniform sampling of the solution space is not sufficient for several applications and clearly discussed with the advection equation. The use of the expected molecular distribution function is also interesting. \n\nThe weaknesses of the paper are as follows:\n1. Why is the expected value of molecular distribution used? In the third application, the KL divergence is used for performance evaluation. Why not use that for the training itself. Are there any mathematical arguments/proofs on why training for expected distribution is good. For the hyperparamenter tuning, the quartile values are used. The choice seems arbitrary and not well explained or justified. \n\n2. The advection equation issue is well explained. So the reviewer (and any reader) will expect that the advection problem will be shown in the applications, at least in the appendix. The chosen application in section 5.1 seems to be a different version (not a soliton that struggles with PINNs). \n\n3. MCMC (or any variant including HMC) itself is too expensive to sample from. The authors also allude to the memory requirements in their high-end GPUs. Is MCMC the right approach for this problem? Is it not making the problem worse? Now if not expectation and some other information metric is used, the cost will go up more!\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "2InlfAwEIB6", "reviewer": "Reviewer_jfSc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider a modification of the PINN construction for solving certain classes of partial differential equations. Namely, they focus on PDEs formulated as a macroscopic description of an underlying microscopic physical process, such as diffusion and advection. They claim that the soft constraint on the PINN loss which informs the network of the original PDE is usually taken as uniformly discretized, which is a poor approximation when the true underlying solution distribution of collocation points is not uniformly spread. They suggest replacing the uniform sampling with a Monte Carlo sampling method, in which the distribution itself is updated during training. They claim that this method resolves the issues of unbalanced density solutions and the need to define a bounding region for the PDE loss.\nThe authors show improved performance over several other collocation point sampling methods, such as uniform, Residual adaptive refinement and some variants, as well as importance sampling.\n\n", "summary_of_the_review": "A well written paper with clear exposition and empirical results, albeit in a limited class of problems.", "strength_and_weaknesses": "Strengths:\n\n1) The paper is well written, clearly explains the objective and modifications made to existing methods.\n2) The presented method is straightforward to implement, and seems to outperform other methods in similar setups.\n\n\nWeaknesses/Questions:\n\n1) The suggested method seems to apply only to micro to macro phenomena, however PDEs are ubiquitous in other scenarios, making this solution very restrictive (albeit interesting).\n2) \"We argue that due to the locality of particle interactions, the regions with higher density are more relevant for regularizing the network.\" This statement is made as motivation for the class of PDEs under study, but it is not proven.\n3) How dependent is the adaptation process on the initial choice of distribution?\n4) While the authors compare their results against other PINN based solvers, it is unclear how the performance stands against other non-PINN methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject