Implicit Neural Spatial Representations for Time-dependent PDEs
Authors: Honglin Chen
Rundi Wu
Eitan Grinspun
Changxi Zheng
Peter Yichen Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: PDE
implicit neural representation
neural field
numerical methods

Abstract: Numerically solving partial differential equations (PDEs) often entails spatial and temporal discretizations. Traditional methods (e.g., finite
difference, finite element, smoothed-particle hydrodynamics) frequently adopt explicit spatial discretizations, such as grids, meshes, and point clouds, where each degree-of-freedom corresponds to a location in space. While these explicit spatial correspondences are intuitive to model and understand, these representations are not necessarily optimal for accuracy, memory-usage, or adaptivity. In this work, we explore implicit neural representation as an alternative spatial discretization, where spatial information is implicitly stored in the neural network weights. With implicit neural spatial representation, PDE-constrained time-stepping translates into updating neural network weights, which naturally integrates with commonly adopted optimization time integrators. We validate our approach on a variety of classic PDEs with examples involving large elastic deformations, turbulent fluids, and multiscale phenomena. While slower to compute than traditional representations, our approach exhibits higher accuracy, lower memory consumption, and dynamically adaptive allocation of degrees of freedom without complex remeshing.

Under review as a conference paper at ICLR 2023
IMPLICIT NEURAL SPATIAL REPRESENTATIONS FOR TIME-DEPENDENT PDEs
Anonymous authors Paper under double-blind review
ABSTRACT
Numerically solving partial differential equations (PDEs) often entails spatial and temporal discretizations. Traditional methods (e.g., finite difference, finite el- ement, mnoothed-partieen hydrodynamics) frequently adopt explicit spatial dis- cretizations, such as grids, meshes, and point clouds, where each degree-of- freedom corresponds to location in space. While these explicit spatial correspon- dences are intuitive to model and understand, these representations are not neces- sarily optimal for accuracy, memory-usage. adaptivity, In this work, explore implicit neural representation as an alternative spatial discretization, where spatial information is implicitly stored in the neural network weights. With implicit neu- ral spatial representation. PDEsstantddd time-stepping translates into updating neural network weights, which naturally integrates with commonly adopted opti- mization time integrators. Our approach requires neither training data nor train- ing/testing separation. Our method is the solver itself, just like the classical PDE solver. We validate our approach volving large elastic deformations, turbulent fluids, and multi-scale phenomena. While slower to compute than traditional representations, approach exhibits higher accuracy, lower memory consumption, and dynamically adaptive alloca- tion of degrees of freedom without complex remeshing.
variety of classic PDEs with examples in-
INTRODUCTION
Many science and engineering problems can be formulated as spatiotemporal partial differential equations (PDEs),
F V f............).......
f(x Rd
(1)
where Ω E Th and T E TR are the spatial and temporal domains, respectively. Examples include the inviscid Navier-Stokes equations for fluid dynamics and the elastodynamics equation for solid mechanics.
To numerically solve these PDEs, we oftentimes introduce temporal discretizations, {lz (T where T is the number of temporal discretization samples and At fx+1 - t, is the time step size. The solution to Equation then becomes a list of spatially dependent vector fields: {f"(()) {x)]
Traditional approaches represent these spatially dependent vector fields using grids, meshes, point clouds. For example, the grid-based linear finite element method (Hughes) (012) defines a shape function N each grid node and represents the spatially dependent vector field as f((x) D.e S PTN N where the number of spatial samples.
While widely adopted in scientific computing applications, these traditional spatial representations are not without drawbacks:
Spatial discretization emrors abound in fluid simulations as artificial numerical diffusion (Lantz 197I), dissipation (Fedkiw et al. (2001). and viscosity (Roache) 1998). These errors also appear in solid simulations as inaccurate collision resolution (Müller al. 2015) and numerical fractures (Sadeghirad et al. 201T).
2. Memory usage spikes with the number of spatial samples P Museth 2013

---

Under review as a conference paper at ICLR 2023
0.15
Ours Grid PINN
Mean Absolute Error
Grid PINN
9.
GT
23
05
X
0.2
3.00
1.23
Time (6)
= Os
= 35
== 12s
coussian-shapeer wave initially centered at x 1.5 moves rightward with a constant velocity of 0.25. From left to right, we show mean absolute error plot over time and solutions at t Os, 3s and 12s, respectively. The solution from grid-based finite difference method (green) tends to diffuse over time. PINN (yellow), trained within temporal range ~ 3s, fails to generalize for = 12s. Our solution (blue) approximates the ground truth (grey) the best over time. All three representations have the same memory footprint: approach and PINN RRaissi et al., 2019) both use a = 2 hidden layers width 3 = 20, and the finite difference grid resolution is 901.
Figure 1: ID advection example:
3. Adaptive meshing (Narain et al. 2012) and data structures (Setaluri et al. 20114) can reduce memory footprints but are often computationally expensive and challenging to implement.
We alleviate these limitations by exploring implicit neural representation (Park et al. 2019 Chen & Zhang 2019: Mescheder et al. 2019) an alternative spatial representation for PDE solvers. Un- like traditional representations that explicitly discretize the spatial vector via spatial primitives (e.g., points), neural spatial representations implicitly encode the field through neural network weights. In other words, the field is parameterized by a neural network (typically multilayer perceptrons), i.., fa)()) fa with 8 being the network weights. As such, the memory usage for storing the spatial field is independent
the number of spatial samples, but rather it is determined by the number of neural network weights. We show that under the same memory constraint, implicit neural representations indeed achieve higher accuracies than traditional discrete representations. Further- more, implicit neural representations are adaptive by construction (Xie al., 2021), allocating the network weights to resolve field details at any spatial location without changing the network archi- tecture.
Viewed from the lens of optimization- time integrators, our PDE solver seeks neural network weights that optimize an incremental potential over time (Kane et al. 2000b). Our solver does not employ the so-called training/testing split commonly appearing in many neural-network--bett approaches Baachez-Gonzalee et al. 2020 Li et al. 2020b). Our approach is the solver itself and does not require training in the machine learning sense. As such, we avoid using the word "training" in the exposition but rather use "optimizing". We employ exactly the same "optimization" integrator formulation as the classical solvers (e.g., finite element method (Bouaziz et al. 2014)).
PDE
We compare the proposed solver to grid, mesh and point cloud representations on time-dependent PDEs from various disciplines, and find that approach trades wall-clock runtime in favor of three benefits: lower discretization error, lower memory usage, and built-in adaptivity.
2 RELATED WORKS
Many prior works have explored representing continuous vector fields with neural networks. Here we highlight two lines of work: implicit neural representation and physics informed neural network.
Implicit Neural Representation uses neural networks to parameterize spatially-dependen func- tions. It has successfully captured the radiance fields (Mildenhall et al.) 2020) and the signed distance fields (Park et al., 2019) in computer vision and graphics settings. has also captured the solutions of strictly spatially dependent PDEs from elastostatics (Zehnder et al. (022). elliptic PDEs (Chiaramonte et al., 2013), and geometry processing (Yang et 2021). Chen et al. [202I), Pan et al. (2022), and Chen et al. (002) also explore neural networks spatial representations for dimension reduction, Dupont et al. (2022) develops a machine learning technique operating on data presented as implicit neural representations. Memory consumptions of traditional representations,
2

---

Under review as a conference paper at ICLR 2023
such as grids and point clouds, scale poorly with spatial resolutions. Adaptive discretizations can reduce memory but their generations are expensive. By contrast, neural representations are adaptive by construction and can use their representation capacities at arbitrary locations of interest without memory increases or data structures alterations. We refer to the recent review by Xie et al. (2021) for additional contexts.
Physics Informed Neural Network (PINN) is a powerful tool for solving PDEs. Traditional nu-
merical methods, such as finite difference and finite element methods, represent tensor fields with
well-studied polynomial basis functions (Hughes, 2012) constructed on meshes. Instead, PINN
represents tensor fields with neural networks and converts the PDE solution process into finding
network weights via IDE-based loss functions, Since the pioneering works by Raissi et al. (2019);
Sirignano & Spiliopoulos (2018) Lagaris et al. (1998); Dissanayake & Phan- ( (1994]
PINN
has been shown to excellently model forward simulation (Shin et al. 2020; Hennigh et al.
2021
Lu et al. 2021a Krishnapriyan et al., 202T), inverse design (Raissi et al.) 2020) Mao et al.
2020
Mishra & Molinaro 2022), optimal control (Mowlavi & Nabi 2021), and uncertain quantification
(Lye et al. 2020), PINN has found success in a wide range of application domains, including tur-
bulence (Hennigh et al.) 2021), elasticity (Rao et al. 2020), acoustics (Sitzmann et al. 2020), and
topology optimization (Zehnder et al.] 2021). Due to its mesh- free nature, PINN can robustly handle
high-dimensional PDEs. The recent review by Karniadakis al. (202T) offers more details.
When it comes to using neural networks to represent vector fields, we face an important design choice which dimension of the vector field is represented through the network. In the case of spatiotemporal vector fields, we have the choice of representing both the spatial and temporal di- mensions via neural networks; we can also represent just the spatial dimension or just the temporal dimension with a network.
We observe that the spatial variable æ oftentimes bounded, e.g., a fixed geometry with well-defined boundaries. However, the temporal variable t can be unbounded, c.g., in virtual reality application where the user interacts with a physical environment indefinitely (Sun et al. 2018). Modeling the additional temporal dimension also puts extra burden on the network. Motivated by these observa- tions, we opt to treat the spatial and the temporal dimensions differently. In particular, we use the neural network strictly as spatial representation and do not consider the temporal dimension as an input to the network. We then evolve this spatial representation by updating the network weights 8" (See Figure 2) potentially for an indefinite amount of time. Such an approach is different from standard PINN that takes both the spatial dimension æ and the temporal dimension as an imput to the network (Raissi et al. 2019 Karniadakis et al. 2021) which cannot resolve PDE solution outside a pre-defined temporal range (Kim et al. 2021) (See Higure I]
Du & Zaki (2021): Bruna al. 2022 K Krishnapriyan et al. 2021) h also explore evolution of neural network weights over time, with the goal of resolving PINN's 's limited time range as well as solving high-dimensional problems that classical solvers often suffer. Our work differs from these works by focusing on low-dimensional settings (ID-3D) that heavily rely on classical solvers (e.g.. finite element method). Our primary goal is to understand if we only replace classical solver's 's spatial rep- resentation with a neural network, while keeping the rest unchanged (e.g., time integrator, boundary condition), what tradeoffs do we get?
Since our approach only replaces the spatial representations of traditional numerical solvers with neural networks while keeping the rest of the solver intact, is compatible with any classical time integration schemes (e.g.. implicit Euler). In particular, we for- mulate time integration as an energy minimization problem (Radovitzky & Ortiz, 1999: Kane et al.) 2000b, Marsden & Wesi 200T: Kharevych et al.) 2006). These integrators find wide applications in PDE solvers on traditional representations, such as grids (Batty et al., 2007). tetrahedral meshes (Bouaziz et al. 2014), and point clouds (Gast et al. 2015). In the case of neural spatial representa- tions, time integration translates into optimizing the neural network weights at every time step.
Optimization Time Integrators.
Machine Learning for PDEs is emerging field with exciting techniques, such graph neural network "Sanchez-Gonzale et all] 2020), neural operator (Li et al.] (020b[c), and DeepONet Lu al. 2019). These techniques usually train on dataset and are then validated on test dataset. However, due to the machine learning nature, these methods's 's time-stepping schemes neither enforce
PDE constraints test time (Pffff et al., 2020 nor generalize to scenarios (c.g.. initial conditions, boundary conditions) drastically different from the training cases (Wang & Perdikaris 2021). As a
3

---

Under review as a conference paper at ICLR 2023
major point of departure, our approach does nor employ any training data. There is not so-called training/infernnce separation our approach, Our method is the solver itself, just like the classical solvers (e.g., FEM). As such, enjoy classical solver's 's unparalleled generalizability and explicit PDE constraints. See Table for a comparision of these techniques. Relatedly, Wandel et al. (2020)
also proposes data-free approach but still employs training testing split.
METHOD: TIEE-SGEPPING ON NEURAL SPATIAL REPRESENTATIONS
Our goal is to solve time-dependent PDEs on neural-network-baae -baed spatial representations, In Sec- tion B.I, we first discuss representing spatial vector fields with neural networks. Afterward, we will describe our time-stepping technique that evolves from one neural spatial representation to another.
3.1 NEURAL NETWORKS AS SPATIAL REPRESENTATIONS
We parameterize each of the time-discretized spatial vector fields with a neural network: f = fo-. where 0 are the neural network weights at time f.. Specifically, the field quantity at arbitrary spatial location æ E Ω can be queried via network inference p=.(2)) (2:).
Traditional representations explicitly discretize the spatial vector field using primitives such as points, tetrahedra, or voxels. These primitives explicitly correspond to spatial locations due to their compactly supported basis functions (Hughes) 2012). By contrast, neural spatial representations im- plicitly encode the vector field via neural network weights. These weights do not directly correspond to specific spatial locations. Instead, each weight affects the vector field globally. Such global sup- port is also an attribute of spectral methods (Canuto et al. 2007a b). Compared to spectral methods, our approach does not need to know the required complexity ahead of time in order to determine the ideal basis functions (Xie et al. 2021). Our neural representation automatically optimizes its parameters to where field detail is present.
Whereas memory consumption of traditional explicit representations scales poorly with the number of spatial samples, memory consumption for implicit neural representations is independent of the number of spatial samples (Xie et al. 202I). Rather, memory use is determined by the number of neural network weights.
Network Architecture Following the implicit neural representation literature, adopt mul- tilayer perceptron (MLP) architecture with SIREN activation function for its accuracy and quick convergence speed advantages (Sitzmann al. 2020). Each MIP has total of a hidden layers, each layer of width B. The specific choice of these hyper-parameters will be described Section
Spatial Gradients Traditional spatial representations (e.g., the finite element method) compute spatial gradients via basis functions. Higher-order gradients require higher-order basis functions. By contrast, a neural spatial representation is C by construction. We evaluate their gradients via computation graph-basee auto iitodiffeernntittoon with respect to the input (not the weights).
3.2 TEMPORAL EVOLUTION
Given previous-time spatial vector fields (f" (x) optimization-baseeee time integrators compute the next time-step (ln+1) vector field by optimizing
putl
argmin
Σ I(At,
( (V ( un+ JE
Jkzo (
BEMCR
Traditional time integrators, whether explicit and implicit, can be expressed in optimization forms (Kharevych et al. 2006). Furthermore, this optimization formulation applies to any spatial represen- tation, and has been explored thoroughly for traditional discretizations (Batty et al. 2007 Bouaziz et al. 2014: Gast al. 2015). which is defined over a finite number of the spatial integration sam- ples_ M == x E 92 spatial representation, optimize for
< < [M]). e.g., grids or meshes. Applying this formulation to a neural
0=+1_ argmin Σ
I(AI, Oz Jk ( V (V" f
Jk=0:
ga+t
aEMCR
4

---

Under review as a conference paper at ICLR 2023
force
where (0 Jk=o are the (fixed, not variable) neu- ral network weights from previous time steps. Figure 2 illustrates
time integration pro- cess. The particular choice of the objectíve function I depends the PDE of interest. In all the examples presented in this work, we solve this time-integration optimization prob- lem via Adam (Kingma & Ba 2014), a first- order stochastic gradient descent method.
f
gues
opabill sample field f(x) 8 weights stepm Figure 2: Time integration. We represent the field of interest using neural network føn, whose weights 0t are updated at each timestep via an optimization problem (Equation (3). In this case, the spatial domain is the interior of the initial object and the represented field f is the deformation map. The governing PDE the elas- todynamics equation (see Section
Spatial Sampling Explicit spatial representa- tions (e.g., tetrahedra mesh) are often tied particular spatial sampling; remeshing is some- times possible, but can also have drawbacks, especially in higher dimensions (Alliez et al.] 2002 Narain et al. 2012). By contrast, im- plicil spatial representations allow for arbitrary spatial sampling by construction (Equation (3))
Following Sitzmann al. (2020), dynamically sample M during optimization. For every gradi- ent descent iteration in every time step, we use a stochastic sample set M from the spatial domain Q2 M corresponds to the "mini-batch" in stochastic gradient descent, with batch size [M]. By directly drawing samples from the entire spatial domain Ω our approach is reminiscent of mesh-free Monte Carlo methods (Sawhney & Crane 2020).
Boundary Condition PDEs are typically accompanied by spatial (e.g., Dirichlet or Neumann) boundary conditions, which we formulate additional penalty terms in the objective Equation
g++1 argmin @n-1 #EMCR
Σ
I(Ai (foe k= (V k== (V fgz k=mo....)
(4)
+λ
Σ
C(S go+,, V gntt, V Sonti,...))
where À is the weighting factor and 0n is the boundary of the spatial domain. The particular choice of the boundary constraint function C depends on the problem interest.
Initial Condition The neural network is initialized using the given initial condition, i.e., the field value at time = by optimizing
0 argmin
Σ (1f
(x)11
go
MEECR
where f0 is the given initial condition. Similar to Equation 33. solve this optimization problem using Adam (Kingma & Ba 2014) and stochastically sample M at each gradient descent iteration.
EXPERIMENTS
In this section, we evaluate our method on three classic time-dependent PDEs: the advection equa- tion, the inviscid Navier-Stokes equation and, the elastodynamics equation. For each problem, we first discuss the continuous PDE and the specific objective function for temporal evolution (recall Equation ). Then we demonstrate the advantages of our approach by comparing with baselines using discrete spatial representations (i.e. a grid, tetrahedral mesh or point cloud). We refer readers to Appendices [ and for other implementation details (e.g., initial and boundary conditions) and additional results. The temporal evolutions of the PDEs are best illustrated by the supplementary video.
5

---

Under review as a conference paper at ICLR 2023
1.0
Mean Squared Emro
0.34
0.0
Timestops
Ground truth
Ours
Grid-based
Figure 3: 2D Taylor-Green vortex simulation. Left: mean squared error of the velocity field for 100 timesteps. Right: velocity magnitude of solutions from the ground truth, ours, and the grid-based method at timestep = 100. Under the same memory usage (for storing the spatial representation), our solution has significantly smaller error than the grid-based method.
4.1 ADVECTION EQUATION
Consider the classic ID advection equation,
du
+ (a V)u 0
(6)
Øt
where a is the advection velocity, and the vector field of interest the advected quantity f u. well known that traditional spatial representations, such as grid-based finite differences, exhibit numerical dissipation for the advection equation (Courant et al. 1952 Selle et al. 2008).
Time Integration We adopt the same time integration scheme in both the traditional representa- tion and ours. Choosing the energy peeegy-preserving midpoint method (Mullen et al., 2009) yields the time integration operator
(x)
u² (x) + (a. V)(
(x +
u" (x)
I
113
(7)
AL
2
Results comuperess our results with those of grid-based finite differences and PINN (Raissi al. 2019), subject to equal memory usage of the three methods. A Gaussian- shaped wave moves with constant velocity 0.25. Our approach uses finite difference grid resolution 901. We set PINN to use the same network architecture (with SIREN activation) ours. For ours and grid-based methods, require At but needs a pre-specified temporal range for training. For this temporal range, we use the solution from the grid-based method diffuses
2 hidden layers of width = 20, and
set At 0.05. PINN does not
[0, 3). As shown in Figure [1 its spatial discretization. While PINN can accurately capture the result up to f = 3s, it fails to produce meaningful solutions beyond its trained temporal range (see 12s) (Kim et al. 2021). By contrast, our solution does not suffer from numerical dissipation and agrees well with the ground truth all frames (see the error plot in Figure I.
time due to
4.2 INVISCID
EQUATIONS
In the incompressible and inviscid Navier-Stokes Equations
du pp(
+ u Vu) = --pp Pjg.
01
(8)
v u 0,
the vector field of interest is the fluid velocity field f = is the pressure, is the external force, and Øy is the fluid density. our experiments, we consider P 1 and g The pressure field p represented with another MLP network.
Time Integration We apply the Chorin-style operator splitting scheme (Chorin 1968; Stam 1999) both the neural spatial and finite- difee-eiffeeece grid representations. The scheme involves three sequential steps: advection (adv), pressure projection (pro), and velocity correction (cor).
Ad Advection uses semi-Lagranginn method, encoded by the operator (taaniforth & Côté 1991
I adv
Uadv n+1 u2" - (3

---

Under review as a conference paper at ICLR 2023
whose optimization yields the advected velocity Uade" =+1 The backtracked location is given by Tbacktrack = a Atu" (x:). While traditional spatial representations compute the backtracked velocity using interpolation (e.g., linear basis function), our approach requires no interpolation, only direct evaluation via network inference at the location Tbacktrack-
Pressure projection is encapsulated by the operator
Ipro- [[²pp*>> V Uadv "++ (a:)
(10)
Plugging pro into the optimization solver, we obtain the pressure p=+ that enforces incompress- ibility. Note that the MLP that represents the velocity field Uadu is kept fixed in this step.
Velocity correction is formulated by the operator
cor = || (Uade "+* - V
(11)
which adds the pressure gradient to the advected velocity yielding the incompressible velocity un+1
Results We first test our method on the 2D Taylor-Green vortex with zero viscosity (Taylor & Green 1937, Brachet et al. 1983). The closed-form analytical solution given by: u(x.t) (sin r COS y, COS z sin y) for æ E [0, 2T] x [0, 2m]. To compare under the same memory usage (for storing the veloeity field), we use a
hidden layers of width / = 32 for our MI.P and set grid resolution to 48 for the grid- based projection method. We set At = 0.05 and execute both methods for 100 timesteps. In Figure 3 we show the mean squared error of the solved velocity field over time. This example demonstrates that our method excellently preserves a stationary solution. Compared to the grid-based method, our method has less diffusion and achieves higher accuracy.
For discrete grid representation, efficiently capturing multi-scale details usually requires difficult- to-implement adaptive data structures (Setaluri et al. 2014). Instead, implicit neural representations adaptive by construction (Xie et al. (021) and enable us to capture more details under the same memory storage. We setup an example where the initial velocity field is composed by two approach
Taylor-Green vortices of different scales (see Figure & for illustration). We compare with PINN and the grid-based projection method under the same memory constraint for storing the spatial representations. Specifically, our approach and PINN uses MLP with a hidden layers of width 3 = 32 and the grid-basee projection method uses resolution 48. We execute our approach and the grid- method for 50 timesteps with At = 0.05, and train PINN with the same temporal range of 2.5 seconds. Using the solved velocity field, we advect a density field to visualize the amount of fine details captured by different representations. As shown in Figure A we are able to capture the fine details of the smaller vortex and best approximate the reference solution. The grid-based method (resolution 48) suffers from severe dissipation and fails to capture the vorticity. PINN is unable to correctly capture this vorticrsses field and we found its training loss remains high (~ le 1) after convergence. This is in agreement with previous findings (Chuang & Barba, 2022) that suggest PINN approaches have difficulty solving inviscid Ravier-Stokes equations for non-trivial examples involving turbulence.
4.3 ELASTODYNAMIC EQUATION
In the third experiment, we study the Elastodynamics equations
(12) that describe the motions of deformable solids (Gonzalez& & Stuart 2008). The vector field of interest the deformation map f = Φ. Here Po the density in the reference space, is the first Piola- Kirchhoff stress, F VΦ the deformation gradient, and Φ are the velocity and acceleration, and b is the body force.
Pn = V P(F) + pab
We assume a hyper-elasticity constitutive law, i..., P aF where v is the energy density function. In particular, we assume variant of the stable Neo-Hookean energy (Smith et al. 2018
=
tr I) + #(det(F) -)
(13)
where À and μ are the first and second lame parameters, E are the singular values of the deformation gradient F, and (et(F) is the determinant of the deformation gradient F. When f = 0, the clastic energy recovers the As -i--iid-----Psssiiii energy (Sorkine & Alexa 2007).
7

---

Under review as a conference paper at ICLR 2023
Reference
Ours
Grid-based (resolution 48)
PINN
Figure 4: Two vortices of different scales. We show the advected density field after 2.5 onds from the reference (top-left), our method (top-right), the grid-based method of resolution 48 (bottom-left) and PINN (bottom-right). The reference is obtained by running the high-resolution grid-based method (we use resolution 1024). Our MIP (a = 3, 3 32) has the same memory foot- print grids of resolution 48. PINN the same MIP network constraint, our approach suffers from less dissipation, captures more vorticity, and best resembles the reference solution, whose grids takes 450x memory compared our network. See Figure for the initial condition of this example.
Under the
memory
Undeformed
Deformed
Ours
mesh-based (FEM)
point-cloud-basse (MPM)
Figure 5: Elastic tension test. We compare the quasi-static simulation using our implicit neural rep- resentation (left) with the mesh representation (middle), and the point cloud representation (right). Our method handles large deformations matching the mesh-based finite element method (FEM), while the point cloud based material point method (MPM) suffers from incorrect numerical fracture.
Time Integration We apply the implicit Euler time integration scheme Gast et al. 2015 Kane
jet al., (2000)) the (1) tetrahedral finite element, (2) material point method, and (3) our neural representation, using the operator
②²) + ((b+t)) pob 8 gut
(14)
energy
energy
potential
n+1 /At, Po is the density, b is the external force. We can also incorporate boundary conditions, e.g., positional and contact constraints, by introducing additional energy terms [Bouaziz et al. 2014 Li et al. 2020a) (see Appendix B.4).
where ob" 1241
Results We first compare our implicit neural representation to the traditional tetraheral mesh rep- resentation (Finite Element Method, FEM (Hughes) 2012: Reddy 2019)) and the point cloud rep-
8

---

Under review as a conference paper at ICLR 2023
force
mesh-based (FEM)
force
Ours
0 sec
0.2 sec
0.6 sec
0.8 sec
Figure 6: When an elastic square collides with a circle, the finite element mesh (top) conforms poorly at the interface compared to the neural representation (bottom), for equal memory usage.
force
sec
0.2 sec
0.4 sec
Figure 7: A bunny collides with the ground in 3D. Our neural representation (green, left) captures more intricate geometry details and complex dynamics compared the traditional tetrahedral mesh representation (blue, right) under the same memory usage.
resentation (Material Point Method, MPM (Sulsky et al. 1995 Jiang et al., 2016 We use a 3 hidden layers of width | 68 for our MLP, which takes the same memory as the FEM mesh (0.8K vertices, 1.5K faces) and MPM point cloud (1.7K points). As shown in Figure 5 and Figure [1 our large elastic deformations and matches the result the traditional
method is capable of handling mesh-based method (FEM), while the point-cloud-batte merical fracture due to its meshless nature. To avoid these fractures, meshless methods require sophisticated modifications of the underlying kernel and basis functions (Gray al. 2001 Su al. 2022).
method (MPM) suffers from incorrect nu-
By virtue of its implicit nature, representation is able to represent more intricate details com- pared to the traditional explicit representations under the same memory usage. In Figure we show that implicit neural representation allows the deformed square gracefully fit the boundary of the sphere during the non-trivial collision. In contrast, the traditional mesh-based representation struggles produce smooth result due to its insufficient mesh resolution. To alleviate such arti- facts, the traditional mesh-based representation either needs to increase resolutions thusly inducing higher memory cost conducts complex remeshing (Narain et al., 2012). Figure 7 our implicit neural representation allows for more complex dynamics and fine geometry details compared to the traditional tetrahedral mesh representation.
Note that we adopt the same collision detection and handling strategy for both the neural representa- tion and the mesh-based representation (FEM). Specifically, we use spring-like penalty force and the corresponding energy move the collided point out of its collision surface, similar to (McAdams) cct al.) 2011 Xian et al.) 2019). Since our approach and FEM share the same time integration scheme and the stems from the underlying spatial representations.
collision handling method, the difference reported in Figure and Figure strictly
These advantages extend other complex 3D simulations. Figure and Figure 13 depict and statue deforming as they collide with the ground, exhibiting complex geometry, and rich contact- induced deformations.

---

Under review as a conference paper at ICLR 2023
5 DISCUSSION AND CONCLUSION
In this work, we explore implicit neural representations as spatial representations for numeri- cally modeling time-dependent PDEs. This representation naturally integrates with widely adopted optimization-based time integrators. PDE solvers with neural spatial representation offers improved aceuracy, reduced memory, and automatic adaptivity compared to traditional explicit representations such as mesh, grid, or point cloud.
While offering important benefits, repressttttlllrppeeetttttttooo based PDE time-stepping requires longer wall-clock computation time than existing methods (see also Table 1 by Zehnder et al. (2021) and Section by Yang et al. (202T).. Optimizing neural networks weights takes longer than opti- mizing grid values even if there are fewer number of neural network weights than the number of grid nodes. For instance, for the bunny example (Figure 7 our neural network optimization takes around 30 minutes per timestep while the corresponding FEM simulation takes less than I minute. Future work therefore lies in exploring advanced training techniques that reduce training time (Liu et al. 2020 Martel et al. 2021 Takikawa et al. 2021 In particular, Müller et al. (2022) offer promising direction where they show that we can reduce implicit neural representation training time from hours to seconds via advanced data structures and optimized implementation.
Our work demonstrates the effectiveness of neural spatial representations in solving time-dependent PDEs and observes empirical convergence under refinement (see Figure [2]. Future work should consider theoretical analysis (Mishra & Molinaro (2022) on convergence and stability. More chal- lenging physical phenomena, such turbulence (Wileox al., 1998), intricate contacts (Johnson &
Johnson 1987). and thin shells (Praff et al. 2020), are also important future directions. Currently, our work enforces "soft" boundary conditions. Enforcing "hard" boundary conditions on neural
architecture is another exciting direction (Lu et al. 2021b
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pierre Alliez, Mark Meyer, and Mathieu Desbrun. Interactive geometry remeshing. ACM Transac- tions on Graphies (TOG), 2173)347-334, 2002.
Christopher Batty, Florence Bertails, and Robert Bridson. A fast variational framework for accurate solid-fluid coupling. ACM Transactions on Graphics (TOG), 36(3):100-cs 2007.
Sofien Bouaziz, Sebastian Martin, Tiantian Liu, Ladislav Kavan, and Mark Pauly. Projective dy- namics: Fusing constraint projections for fast simulation, ACM transactions on graphics (TOG), 33(4):1-11, 2014.
Mare E Brachet, Daniel I Meiron, Steven A Orszag, BG Nickel, Rudolf H Morf, and Uriel Frisch. Small-scale structure of the taylor- green vortex. Journal of Fluid Mechanics, 130:411 1-252, 1983.
Joan Bruna, Benjamin Peherstorfer, and Eric Vanden-Eijndes Neural galerkin scheme with active learning for high-dimensional evolution equations. arXiv preprint arXiv:2203.01 2022.
Claudio Canuto, M Yousuff Hussaini, Alfio Quarteroni, and Thomas A Zang. Spectral methods; fundamentals in single domains. Springer Science Business Media, 2007a.
Claudio Canuto, M Yousuff Hussaini, Alfio Quarteroni, and Thomas A Zang. Spectral methods: evolution to complex geometries and applications to fluid dynamics. Springer Science & Business Media, 2007b.
Peter Yichen Chen, Maurizio Chiaramonte, Eitan Grinspun, and Kevin Carlberg. Model reduction for the material point method via an implicit neural representation of the deformation map. arXiv preprint arXiv: 109.009.2222 /2390, 2021.
Peter Yichen Chen, Jinxu Xiang, Dong Heon Cho, GA Pershing, Henrique Teles Maia, Maurizio Chiaramonte, Kevin Carlberg, and Eitan Grinspun. CROM: Continuous reduced-order modeling of PDEs using implicit neural representations. arXiv preprint arXiv:2206. 02607, 2022.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5939-5948, 2019.
M Chiaramonte, M Kiener, al. Solving differential equations using neural networks. Machine Learning Project, 1, 2013.
Alexandre Joel Chorin. Numerical solution of the navier-stokes equations. Mathematics of compu- tation, 22(104):755-76, 1968.
Pi- Yu-u Chuang and Lorena A Barba. Experience report of physics-informed neural networks in fluid simulations: pitfalls and frustration. arXiv preprint 1xXi::2205.14249 2022.
Richard Courant, Eugene Isaacson, and Mina Rees. On the solution nonlinear hyperbolic differ- ential equations by finite differences. Communications on pure and applied mathematics, 5(3): 243-255, 1952.
MWMG Dissanayake and Nhan Phan-Thien. Neural-network-based approximations for solving partial differential equations. communications in Numerical Methods in Engineering, 10(3):195- 201. 1994.
Yifan Du and Tamer A Zaki. Evolutional deep neural network. Physical Review E, 104(4):045303, 2021.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you should treat like one. arXiv preprint arXiv:2201.12204, 2022.
Ronald Fedkiw, Jos Stam, and Henrik Wann Jensen. Visual simulation of smoke. In Proceedings of the 28th annual conference on Computer graphies and interactive techniques, pp. 15-22, 2001.
11

---

Under review as a conference paper at ICLR 2023
Theodore F Gast, Craig Schroeder, Alexey Stomakhin, Chenfanfu Jiang, and Joseph M Teran. Opti- mization integrator for large time steps. IEEE transactions on visualization and computer graph- ics, (1110):1103-11::: 2015.
Oscar Gonzalez and Andrew M Stuart. A first course in continuum mechanics, volume 42, Cam- bridge University Press, 2008.
James P Gray, Joseph Monaghan, and RP1021 Swift. Sph elastic dynamics. Computer methods in (0149990))66611-411 2001.
applied mechanics and engineering.
Oliver Hennigh, Susheela Narasimhan, Mohammad Amin Nabian, Akshay Subramaniam, Kaustubh Tangsali, Zhiwei Fang. Max Rietmann, Wonmin Byeon, and Sanjay Choudhry. Nvidia simnetTM; An ai-accelerated multi-physics simulation framework. In International Conference on Compu- tational Science, pp. 447-461. Springer, 2021.
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: language for bigf-peeformance computation on spatially sparse data structures. ACM Transac- tions on Graphics (TOG), 38(6):1-16, 2019.
Thomas JR Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The ma- terial point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses, SIGGRAPH 16. Association for Computing Machinery, 2016.
Kenneth Langstreth Johnson and Kenneth Langstreth Johnson. Contact mechanics. Cambridge university press, 1987.
C. Kane, J. E. Marsden, M. Ortiz, and M. West. Variational integrators and the newmark algo- rithm for conservative and dissipative mechanical systems. International Journal for Numerical Methods in Engineering, 49(10)
2000a.
Couro Kane, Jerrold E Marsden, Michael Ortiz, and Matthew West. Variational integrators and the newmark algorithm for conservative and dissipative mechanical systems. International Journal for numerical methods in engineering,
1295-1325, 2000b.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, ((6) 4402 2021.
Liliya Kharevych, W Wei, Yiying Tong, Eva Kanso, Jerrold E Marsden, Peter Schröder, and Matthieu Desbrun. Geometric, variational integrators for computer animation. Eurographies Association, 2006.
Jungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jhin, and Noseong Park. Dpm: a novel train- ing method for physics-informed neural networks in extrapolation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8146-8154, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X1v:1412.6990, 2014.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances Neural In- formation Processing Systems, 34:26548-26564 2021.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 95):9877-0000, 1998.
RB Lantz. Quantitative evaluation of numerical diffusion (truncation error). Society of Petroleum Engineers Journal, 1103:3315-320, 1971.
David L.W. Levin. Bartels: A lightweight collection of routines for physics simulation, 2020. http::/githuh.comttileei//BBBB
12

---

Under review as a conference paper at ICLR 2023
Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy R Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M Kaufman. Incremental potential contact: intersection- and inversion-free, langedecformation dynamics. ACM Trans. Graph.. 39(4):49, 2020a..
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. XrXiv preprint arXiv:2010.08895, 2020b.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differ- ential equations. arXiv preprini arXiv:2003. 03485, 2020c.
Lingje Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15665 2020.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for iden- tifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.0323 2019.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: deep learning library for solving differential equations. SIAM Review, 63(1):208-223 2021a.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang. Francesc Verdugo, and Steven G Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on 100p:B1105-BBB11 2021b.
Scientific Computing,
Kjetil O Lye, Siddhartha Mishra, and Deep Ray. Deep learning observables in computational fluid dynamics. Journal of Computational Physics, 410:109333 2020.
Zhiping Mao, Ameya D Jagtap, and George Em Karniadakis. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering, 360: 12789, 2020.
Jerrold E Marsden and Matthew West. Discrete mechanics and variational integrators. Acta Numer- ica, 10:357-514, 2001.
Julien NP Martel, David В Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acom: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02788, 2021.
Aleka McAdams, Yongning Zhu, Andrew Selle, Mark Empey, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. Efficient elasticity for character skinning with contact and collisions. In ACM SIGGRAPH 2011 papers. pp. 1-12. 2011.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4470-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for pdes. IMA Journal of Numerical Analysis, 42(())981-100) 2022.
Saviz Mowlavi and Salch Nabi. Optimal control of pdes using physics-informed neural networks. arXiv preprint 1rXiv:2111.0985 1009980, 2021.
Patrick Mullen, Keenan Crane, Dmitry Pavlov, Yiying Tong, and Mathieu Desbrun. Energy-
preserving integrators for fluid animation. ACM Transactions Graphics (TOG), 28(3):1-8, 2009.
Matthias Müller, Nuttapong Chentanez, Tae- Yong Kim, and Miles Macklin. Air meshes for robust collision handling, ACM Transactions Graphics (TOG), 34(4):1-9, 2015.
13

---

Under review as a conference paper at ICLR 2023
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. arXiv preprint arXiv:2201.0988, 2022.
Ken Museth. Vdb: High-resolution sparse volumes with dynamic topology. ACM transactions on graphics (TOG), 32(3):1-2,, 2013.
Rahul Narain, Armin Samii, and James F O'brien. Adaptive anisotropic remeshing for cloth simu- lation. ACM transactions on graphics (TOG), 31(6):1-10, 2012.
Shaowu Pan, Steven L Brunton, and J Nathan Kutz. Neural implicit flow: mesh-agnostic dimen- sionality reduction paradigm of spatio-temporal data. arXiv preprint arXiv:2204.00 032266 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Tobias Pfaff, Meire Fortunato, Alvaro based simulation with graph networks. arXiv preprint 1iXix:20100034099 2020.
and Peter W Battaglia. Learning mesh-
Raul Radovitzky and Michael Ortiz. Error estimation and adaptive meshing in strongly nonlinear dynamic problems. Computer Methods in Applied Mechanics and Engineering, 172(1-4):203- 240, 1999.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481) 2020.
Chengping Rao, Hao Sun, and Yang Liu. Physics informed deep learning for computational elasto- dynamics without labeled data. arXiv preprint w-Xiv:2006.08472, 2020.
J. N. Reddy. Introduction the Finite Element Method. McGraw-Hill Education, New York, 4th
edition edition, 2019.
Patrick J Roache. Fundamentals of computational fluid dynamics. Hermosa Publishers, 1998.
Alireza Sadeghirad, Rebecca M Brannon, and Jeff Burghardt. A convected particle domain in- terpolation technique extend applicability of the material point method for problems involv- ing massive deformations. International Journal for numerical methods in Engineering, 86(12):
1435-1456, 2011.
Alvaro
Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Leariing simulate complex physics with graph networks. In International Confer- ence on Machine Learning, pp. 8459-8468. PMLR, 2020.
Rohan Sawhney and Keenan Crane. Monte carlo geometry processing: A grid-free approach to pde-based methods on volumetric domains. ACM Transactions on Graphics, 39(4). 2020.
Andrew Selle, Ronald Fedkiw, Byungmoon Kim, Yingjie Liu, and Jarek Rossignac, An uncondi- tionally stable maccormack method. Journal of Scientific Computing. 35(2):350-371, 2008.
Rajsekhar Setaluri, Mridul Aanjaneya, Sean Bauer, and Eftychios Sifakis. Spgrid: A sparse paged grid structure applied to adaptive smoke simulation. ACM Transactions on Graphics (TOG), 33 (6):1-12, 2014.
Yeonjong Shin, Jerome Darbon, and George Em Karniadakis. the convergence of physics in- formed neural networks for linear second-order elliptic and parabolic type pdes. arXiv preprint arXiv:2004.01806, 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 11::1339-1364 2018.
14

---

Under review as a conference paper at ICLR 2023
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Breannan Smith, Fernando De Goes, and Theodore Kim. Stable neo-hookean flesh simulation. ACM Trans. Graph., mar 2018.
Olga Sorkine and Mare Alexa. As-rigid-as-possiee surface modeling. In Symposium on Geometry processing, volume pp. 109-116, 2007.
Jos Stam. Stable fluids. Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 121-128, 1999.
Andrew Staniforth and Jean Côté. Semi-lagrangian integration schemes for atmospheric models review. Monthly weather review, 119(9):2206-2223, 1991.
Haozhe Su, Tao Xue, Chengguizi Han, and Mridul Aanjaneya. A-ulmpm: An adaptively updated
lagrangian material point method for efficient physics simulation without numerical fracture. In Computer Graphics Forum, volume 41, pp. 325-341. Wiley Online Library, 2022.
Deborah Sulsky, Shi-Jian Zhou, and Howard Schreyer. Application of particle-in in-cell method to solid mechanics. Computer physics communications, (7(1-2):236-23 252, 1995.
Qi Sun, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan McGuire, David Luebke, and Arie Kaufman. Towards virtual reality infinite walking: dynamic saccadic redirection. ACM Transactions on Graphics (TOG), 37(4): 2018.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahraii Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358-11367, 2021.
Geoffrey Ingram Taylor and Albert Edward Green. Mechanism of the production of small eddies from large ones. Proceedings of the Royal Society of London. Series A-Mathematical and Physical Sciences, 158(895)4499-51, 1937.
Nils Wandel, Michael Weinmann, and Reinhard Klein. Learning incompressible fluid dynam- ics from scratch-towards fast, differentiable fluid models that generalize. arXiv preprint arXiv:2006.08766 2020.
Sifan Wang and Paris Perdikaris. Long-time integration of parametric evolution equations with physics-informed deeponets. arXiv preprint arXiv:2106.0388, 2021.
David C Wilcox et al. Turbulence modeling for CFD, volume 2. DCW industries La Canada, CA, 1998.
Zangyueyang Xian, Xin Tong, and Tiantian Liu. scalable galerkin multigrid method for real-time simulation of deformable objects. ACM Trans. Graph., 38(6), nov 2019.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. arXiv preprint arXiv:21 2021.
Guandao Yang, Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.
Jonas Zehnder, Yue Li, Stelian Coros, and Bernhard Thomaszewski. Ntopo: Mesh-free topology optimization using implicit neural representations. Advances Neural Information Processing Systems, 001100010099 2021.
15

---

Under review as a conference paper at ICLR 2023
A COMPARISON OF DIFFERENT PDE SOLVERS
In the table below, we compare other ML-PDE solvers including MeshGraphNet (Pfaff et al. (020)
GraphNetworkSim (Sanchez-Gonzalee et al. (020)), DeepOnet Operator Li et al. 2020b)).
(Lu al. 2019)). Fourier Neural
Table 1: Comparison with other ML-PDE solvers.
Generalize to out- of-distribution initial/buudaary conditions, material parameters, and time spans
Require training data (from classical solvers real-world data)
Enforce PDE constraints
Methods
MeshGraphNeet
Yes
Limited
None
GraphNetworkSSim
Yes
Limited
None
DeepOnet
Yes
Limited
Good
Fourier Neural Operator
Yes
Limited
Good
Ours Classical solvers (e.g., FEM)
No
Good
Good
No
Very good
Very good
B IMPLEMENTATION DETAILS
B.1 OPTIMIZATION
We solve time-integration optimization problem (Equation (3) with the Adam optimizer (Kingma & Ba 2014). For all examples in our experiments, we set an initial learning rate Iro and reduce by factor of 0.1 if the loss value does not decrease for iter, iterations. We stop the optimization process when the learning rate is smaller than Immin or until it reaches a maximum of itermax iterations. Specific values of these hyper-parameters are described for each example below. We implement our method using Py PyTorch library and performed our experiments on an NVIDIA GeForce RTX 3090 GPU.
Algorithm 1: Temporal Evolution
Input: network weights 8", time step size At, time integrator I, spatial domain f
1
n < 0;
2
while frue do
3
g 0m;;
while converged do // network training with Adam optimizer
randomly sample M C Ωi
ga+1 + 00-1 aV Σ I(AI {S Jk=n: (V
1n+1 1=+1
// eq. 33)
end while
8
n + n + 1;
9 end while
B.2 ADVECTION EQUATION
For our advection example in Figure I the ID spatial domain is Ω = [- We consider the Dirichlet boundary condition, i.e., the advected quantity at boundaries equals zero Hence set the boundary constraint term in Equation (4) as
|| nt [.)
(15)
with the weighting factor 1. The initial condition for this example is
( 12)
(x))
(16)
2
16

---

Under review as a conference paper at ICLR 2023
Initial velocity field
Initial density field
0.
0.0
Figure 8: Initial condition for the example in Figure Left: velocity field (Equation 19
Right:
density field (Equation (20)).
with μ = -1 1.5 and a 0.1. We set the optimization hyper-parameters Iro = 1e-4, Im 1e-8, 8, iter,- 500 and iter mtx 20000. For each gradient descent iteration, randomly sample |M 5000 points within the spatial domain [-2, 2]. For this example, our method takes ~ 80s to compute per timestep, while the grid-based method (using the same memory) takes 4e-3s.
B.3 INVISCID NAVIER-STOKES EQUATIONS
For our 2D fluid examples, the spatial domain is Ω = x [-1.1]. We consider solid bound- ary conditions, i.e., the fluid cannot go through the boundaries. Recall that we adopt the operator splitting scheme. Therefore, the boundary constraint terms for the three sequential steps are
Cadv 1 n+1 (x)
Cpro = V
1V n+ 1
(17)
Coor |u""" - (x)113
where indicates the perpendicular direction against the boundary. The weighting factor
2D Taylor-Green vortex Standard 2D Taylor-Green is originally defined in domain [0, 2m] x
[0, 2m). We translate and scale the domain to -1, 1] x [-1, such that the input range fits our MLP with the SIREN activation (Sitzmann et al. 2020 Therefore, the initial condition for the velocity field becomes
(x) = sin[=(r + 1)] cy + 1)], cos + 1)] sin[sly + IJ]).
(18)
π
π
After the simulation, we convert it back to domain [0, 2T] x [0, 2m] for evaluation and comparison. We the optimization syper-parameters Ir, 1e-5, Irmin le-8, iter, = 500 and iter 20000. The size of sample set |M| 2562. For this example, our method takes 10min to compute per timestep, while the grid-based method (using the same memory) takes ~ 0.03s.
Two vortices of different scale For the example shown in Figure the initial condition for the velocity field is
(sin (ar + 1)] cos (2 + 1)), cos (or + 1)) sin (2r(y- 1)]) (sin(8r(x (x 7) cos (8s(88( [}, cos cos (8 - 7) sin ((t(8( D))
x x [-1
x x 1.1 (19)
(x))= =
(0.
otherwise.
The density field that we advect is initialized as
di (x) -{
|| + 1|1 0.5 or |8ar 7|| S 0.5
(20)
otherwise.
Figure visually illustrates the above initial conditions. After the simulation, we convert it back to [0, 2x) for evaluation and comparison. We set the optimization hyper-parameters
domain 0. 2m)
Ir. le-5, Irmin = 1e-8, iter, = 500 and iter 20000. The size of sample set |M = 128?.
For this example, method takes 10min to compute per timestep.
17

---

Under review as a conference paper at ICLR 2023
Table Experiment setup for the elasticity examples. Po the density, and μ are the first and second lame parameters. and 8 are the number of hidden layers and the dimension of the hidden features. Irg and iter are the initial learning rate and the maximum number of iterations. lay is average training time per time step. Note that the density Po and timestep size dt are reported as N/A for the quasistatic example Stretch (2D) (Figure 5p.
Example
Dim
M
dt
Pu
a
Iro
iter lvyg
itermax
Collision (2D) (Figure
2
100
0.1
lel
2e1
le3
68
le-5
le4
1.38e2
Stretch (2D) (Figure
2
1002
N/A
N/A
1e0
1e3
68
le-4
5e4
2.30e3
Bunny (Figure
3
203
0.1
1e0
le2
le3
66
le-5
2e4
1.70e3
Spot (Figure)
3
203
0.1
1e0
1e2
1e3
66
le 3
5e3
1.74e3
Lucy (Figure
3
20
0.1
1e0
1c3
1c3
128
1e-4 4
2c4
1.16e3
Undeformed
Twisted
Undeformed
Stretched
Compressed
Figure 10: Twisting test. Quasistatic simulation in 3D. The right end is twisted 45 degrees.
Figure 9: Elasticity patch test. Quasistatic simulation in 2D (undeformed, stretched, compressed).
B.4 ELASTODYNAMICS EQUATION
Initial and Boundary Conditions
For our 2D elasticity examples in Figure 5 Figure and Fig-
ure the 2D spatial domain is 92 [-1,
[-1, 11.
1]. For our 3D elasticity examples in Figure 12
the 3D spatial domain is Ω = 1. x - 1. For our 2D and 3D examples involving
nonregular geometry (Figure 7 Figure
2 and Figure 13), the spatial domain is the interior of the
shape, including the boundary. The initial condition for all the elasticity examples is
6 ( = (0,0) 0) (2D),
(21)
(x) = (0, 0, 0) (3D)
The boundary constraint for elasticity examples involves positional constraints or collision con- straints. Positional constraints, or Dirichlet boundary conditions, can be realized by defining the position of the constraint set 82 as the desired goal positions daai
I pos [ daall2
(22)
Collision constraints can be handled by adding unilateral constraints dynamically and viewing the collision penalty force as external force. Specifically, for colliding point ge first find the closest surface point be with normal n., and define our spring-like collision penalty force as:
fcol = hiat ((b. 4e) n..
(23)
where keat the ratio for the collision penalty force.
The corresponding collision energy can be defined as the work exerted by the collision force:
I n0 n+1
(24)
Experiment Setup For all the 2D comparison under the same memory usage, we use a = hidden layers of width 3 68 with SIREN activation function which takes the same memory (57 KB) as the FEM mesh (0.8K vertices, I.3K faces) and MPM point cloud (1.7K points) in We initialize the 2D deformation field of the network be
( (Sitzmann et al. 2020) for MLP,
18

---

Under review as a conference paper at ICLR 2023
02
00
Ground Truth
Ours
mesh-based (FEM)
Figure 11: Error of the elastic tension test. We visualize the L2 position error e of our result and the low-resolution FEM result (0.8K vertices) in Fig. [ with respect high-resolution FEM ground truth (3.2K vertices). Under the same memory footprint (for storing the spatial representation), our method (/e/ occ 8.89c-2) is closer the high-resolution ground truth than the mesh- method (||ell>c = 1.99e-1). 1).
Error
0.5
0.4
0.3
0.2
0.1
5
10
20
30
40
50
0.0
52
20a
30
40 so
#samples
#samples
Figure 12: Sampling convergence test. Quasi-static elasticity simulation using different number of samples. We trained the implicit neural representation using different number of samples and visualized the result using samples L 50 (left). We further compute the error with respect to MM = 50 when using different number of samples (right). As we increase the number of the training samples in use, the deformation field converges to the result trained on the highest resolution.
using [M] 1 uniform and random samples. Then we train the network using [M| 100 uniform and random samples at each training iteration. We use Bartels (Levin) (2220) and Taichi (Hu et al. 2019) perform the FEM and MPM simulation, respectively. We run our FEM and MPM comparison on CPU using a MacBook Pro with Apple M2 processor and 24GB of RAM.
For the 3D comparison under the same memory usage, for the bunny example (Figure we use a 3 hidden layers of width 3 66 with SIREN activation function for our MLP, which takes the same memory (53 KB) as the FEM mesh (0.5K vertices, 1.5K tetrahedra) use. For the statue example (Figure 13 we use = 3 hidden layers of width 3 128 with SIREN activation function for our MLP. which takes the same memory (197 KB) as the FEM mesh (2.0K vertices, 7.0K tetrahedra) in use. We initialize the 3D deformation field of the network to be using |M| 100 uniform and random samples. Then we train the network using [M| 20 uniform and random samples at each training iteration. Here for simplicity report all the parameters and experiment setup in Table 2 In addition, we set the hyper-parameters iterp 800 and Immin 1e-8 for all elasticity examples.
the mesh vertices as the uniform samples. We further
For sampling of the shapes involving nonregular geometry, for simplicity we choose to use a triangle or tetrahedral mesh and perform sampling within it. An ideal alternative would be adopting the implicit representation of the surface and performing rejection sampling based on it.
For rendering, we simply sample sufficient number of points from the undeformed shape and eval- uate the trained model at time on the sample positions to predict their deformation. Here we only sample the surface of the shape in 3D cases. Then we render the shape as dense point cloud.
C ADDITIONAL RESULTS
C.I ELASTODYNAMIE EQUATION
We validate the physical plausibility of our method using small 2D patch test (Figure 9 and 3D twisting example (Figure IO). We show that our implicit neural representation can exhibit volume- preserving property under both stretching, compression and twisting.
19

---

Under review as a conference paper at ICLR 2023
force
Tetrehedral Mesh
sec
0.2 sec
sec 0.6 sec
0.8 sec
1.0 sec
Figure 13: The statue collides with the ground and deforms. Our implicit neural representa- tion (green, right) is capable of capturing more fine geometry details compared to the traditional tetrahedral mesh representation (blue, left) under the same memory footprint.
We demonstrate qualitative and quatitative convergence of our method when increasing the num- ber of training samples in use. In Figure [2. we compare the quasi-static stretching results vi- sualized using [M| 50 uniform samples when using different number of training samples ([M] 53, 10 20 200,30 30", 408 50%), and report the error with respect to the high-resolution trained result (trained and visualized both [M] = 50%).
Finally, we provide an additional example for elasticity involving complex contact-induced defor- mations in Figure [3] Our implicit neural representation is able maintain more intricate geometry details compared to the traditional tetrahedral mesh representation under the same memory usage.
D QUANTITATIVE RESULTS
We present the quantitative results (error, runtime and memory usage) for each of our tested exam- ples separate tables.
Table 3: Quantitative results for 1D advection example (Figure D. Error: mean absolute error over total 240 time steps, compared to the ground truth analytical solution. Time: runtime for total 240 time steps. Memory: memory usage for storing the spatial representations.
Methods
Error
Time
Memory
Ours
0.0030
5.33h
3.520KB
Grid
0.0146
1.13s
3.520KB
PINN 0.0564
9.73m
3.598KB
Table Quantitative results for 2D Taylor-Green fluid example (Figure3). Error: mean squared error of velocity field over total 100 time steps, compared to the ground truth analytical solution. Time: runtime for total 100 time steps. Memory: memory usage for storing the spatial representa- tions.
Methods
Error
Time
Memory
Ours
3.35e-4
14.02h
25,887KB
Grid-based
4.83e-3
2.91s
27.00KB
Table 5: Quantitative results for two-vortices fluid example (Figure Error: mean absolute error of kinetic energy over total 50 time steps, compared to the reference solution (obtained by high resolution grid based method). Kinetic energy computed using 10242 uniform samples. Time: runtime for total 50 time steps. Memory: memory usage for storing the spatial representations.
Methods
Error
Time
Memory
Ours
2.24e2
10.8
25.887KB
Grid-based
1.07e4
1.78s
27.00KB
PINN
1.92e4
2.2h
26.137KB
PINN-seq
3.21e4
20.83h
26.137KB
20

---

Under review as a conference paper at ICLR 2023
Table 6: Quantitative results for 2d tension example (Figure 5
and Figure _. Error: Infinity
norm of L2 distance w.r.t high-resolution ground truth. Time:
Total runtime until convergence.
Memory: memory usage for storing the spatial representations.
Methods
Error
Time
Memory
Ours
8.82e-2
38.33m 22.04s
56.32KB
Mesh-based
1.99e-1
54.00KB
Table 7: Elastic square collides with circle (Figure 6.
Error: Infinity norm of L2 intersection
distance w.r.t. the circle boundary. Time: Average runtime for time step. Memory: memory usage for storing the spatial representations.
Error
Time
Memory
Methods
0.2s
0.4s
0.6s
0.8s
Ours
1.62e-2
1.10c-2
1.04e-2
1.06e-2
2.30m
56.32KB
Mesh-based
3.45e-2
1.39e-2
5,47e-2
4.23e-2
9.82s
54.00KB
21


Reviews:
[{"review_id": "mXjScNlbQy", "reviewer": "Reviewer_oNuL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work the authors propose to implicit neural representations to model the spatial representation, while approaching the temporal representation with classical optimization-based approaches to model time-dependent partial differential equations.", "summary_of_the_review": "This work present a new approach for implicit neural spatial representations for time-dependent partial differential equations. With the temporal representation modeled classically, there exist a number of similar approaches in existing literature, hence rendering the use of implicit representations the main novelty of the paper. In addition the paper severely lacks adequately chosen comparisons, only comparing to the original physics-informed neural network (PINN) approach from 2019. ", "strength_and_weaknesses": "Strengths:\n- Clarity of the mathematical exposition of the approach\n- Clarity in the embedding of the presented approaches intersection and background in PDE- and fluid dynamics theory\n\nWeaknesses:\n- Poor integration into the current state-of-the-art by omitting many similar approaches, such as Graph Network Simulations [2] which effectively encode the spatial representation with graph neural networks, and similarly use classical time-integrators for the temporal representation. The same applies for the even more modern MeshGraph [3], and Fourier Neural Operator [4], both would need to be compared to the herein presented approach to properly assess the capabilities of the presented approach. The usage of implicit representations for functions has also been explored by Dupont et al. [1] before. Said work would need to be put in context to the presented literature.\n- Improperly chosen comparisons to current state-of-the-art models due to an incomplete representation of the literature, see preceding point\n- No ablation analyses\n- Performance of the approach induces a 30x overhead, hence making it intractable for any practical problem at the current moment\n\n[1] Dupont, Emilien, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. \"From data to functa: Your data point is a function and you can treat it like one.\" In International Conference on Machine Learning, pp. 5694-5725. PMLR, 2022.\n[2] Pfaff, Tobias, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. \"Learning Mesh-Based Simulation with Graph Networks.\" In International Conference on Learning Representations. 2020.\n[3] Sanchez-Gonzalez, Alvaro, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. \"Learning to simulate complex physics with graph networks.\" In International Conference on Machine Learning, pp. 8459-8468. PMLR, 2020.\n[4] Li, Zongyi, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. \"Fourier Neural Operator for Parametric Partial Differential Equations.\" In International Conference on Learning Representations. 2020.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X1tU-U6rVET", "reviewer": "Reviewer_DwBW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper proposes an NN solver for the time-dependent PDE. The proposed method uses neural networks to parametrize the time-discretized spatial vector field. The loss function for training combined problem-dependent time integrator, penalty terms by boundary condition, and initial condition, which is the same with the classical solvers, yet one is allowed to choose different spatial samplings per time step.  The method is shown to work on several test problems, ranging from 1D advection function to 3D  Elastodynamics equation.   Overall, the authors show that the proposed method is promising for improving the flexibility and numerical accuracy of the solutions. \n", "summary_of_the_review": "To summarize, the paper proposed a NN solver that utilize neural network to parametrize the time-discretized spatial vector field and the adaptive meshes per time step.  I would recommend authors to present a more comprehensive and informative comparison with the existing methods, and more insights on why the proposed method has advantages and its possible limitations.", "strength_and_weaknesses": "\nStrengths:\n\nThe studied problem is interesting and important. The paper is well organized and easy to follow.  The numerical experiments have enough details for reproducibility. The visualizations  of the solution and mesh are nicely presented. \n\n\nWeakness:\n\n1. Limited technical novelty.  As far as I understand, the contribution of this work is to use neural network to represent the solution, and allows the flexibility of spatial samplings per time step. I would say both ideas are not new and has appeared in many works, and  this paper did not provide further insights on  how to select the sampling set and the design of neural network that yield better results. \n\n As authors mentioned: one could representing both the spatial and temporal dimensions via neural networks;  or just the spatial dimension or just the temporal dimension. It seems to me that consider spatial dimension may ignore the temporal correlation among different snapshots of the solution, and need more weights that what is actually need to represent a smooth spatiotemproal field.  \n\n\n2. The method is not  thoroughly evaluated. The evaluation metric is vague: there is no clear definition on  memory usage, numerical evaluation error metric for solutions and time complexity for each method.  Did the author use the state of the art solver for each example? How about the same type of PDEs but with different parameters and initial conditions. These PDEs may not allow explicit solutions but the classical solvers have theoretical guarantees to obtain the solution up to certain accuracy, so I think it is fine to compare with them to see if the proposed method have advantages in other aspects. \n\n3. The paper should discuss more on the limitations. For example,  this solver is only for smooth solutions, how to deal with non-smooth case? Also, compared to classical solvers, it  may be more difficult to incorporate certain physical constraints and boundary constraints.\n\n4. The PDE examples are up to 3D where the classical solvers also generally worked well.  It is believed that NN solver has advantages on deal with higher dimension problem (see ref 1) , can the authors comment on the potential of the proposed method on higher dimension problem? \n\nReferences:\n[1] Solving high-dimensional partial differential equations using deep learning\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WWJS0HZZAzu", "reviewer": "Reviewer_Tkaf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for partial differential equations by using a coordinate-based network instead of a standard grid as the representation for the solution to the equation. This is done by modeling the spatial dimensions as a SIREN MLP, and then evolving the parameters of the network over the time dimension by solving an optimization problem at each time step. This explicit separation allows the solution to better generalize to new times, as the coordinate-based network tends to overfit on bounded signals. The paper demonstrates that this works for a variety of partial differential equations, and that despite the slower speed, the approach outperforms grid-based solvers in terms of accuracy and memory consumption.", "summary_of_the_review": "Overall, I find the paper to be good: it very clearly proposes and describes a novel tweak to the existing SIREN architecture for PDE solutions (treating the temporal and spatial dimensions differently), evaluates this over a wide variety of different PDEs, and seems to demonstrate that the method performs significantly better. Since the technical contribution is based very heavily on the previous work, I think that the main key for the paper should be to demonstrate that this tweak does significantly improve the results. The paper does do this, but it could definitely be made stronger with some more explanation of what the baselines are (is PINN a SIREN), and some quantitative results to demonstrate more improved performance alongside the qualitative results.\n\nPOST REBUTTAL UPDATE:\nAfter reading the author response to my and other reviews, I am inclined to maintain my score. I appreciate the additional comparisons added in the Appendix which highlight the increase in performance (and differences in runtime, which I didn't recognize at first look of the manuscript without these tables). This improves my opinion of the work. However, I still believe that the amount of technical contribution is limited since the method is very similar to PINN with SIREN, and only offers increased performance at the expense of additional compute time. Overall, I still think it is a good paper that could be accepted and is above the threshold.", "strength_and_weaknesses": "In my opinion, the main strengths of the paper are:\n- The paper is written clearly, and communicates the proposed method well. The claims made in the paper are well supported by the experiments, and are well explained. The relationship of the proposed method to the related work, grid based methods and PINNs, is clearly outlined so the contribution and differences proposed are extremely clear.\n- The method seems to result in noticeable qualitative improvements in the solutions to various PDEs. The evaluation is thorough, and is done across three different PDEs demonstrating that the result is not just overfit for a particular PDE.\n\nIn my opinion, the main weaknesses of the paper are:\n- The amount of novelty is slightly limited. It seems like the method builds on the SIREN architecture heavily, and the only main contribution is the separation of the temporal dimension and treating it differently than the spatial dimensions. Do the PINN network baselines also use the SIREN architecture, and then just treat the time dimension as an input to the network? If not, this baseline absolutely needs to be compared to in detail in order to prove that this contribution of separating time and spatial dimensions is critical for performance. Overall, I would think the paper should make more clear that this is the main contribution over the prior state-of-the-art in using coordinate-based networks for PDE solutions.\n- There are very limited quantitative comparisons. While the qualitative comparisons are very nice and do show that there is a difference in performance, it’s difficult for me to understand the level of the failure without some kind of quantitative metric for each of the experiments. Addition of some quantitative results would definitely increase the strength of the paper by demonstrating that the method proposed really improves on the performance of the prior work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BpLuWdaGDcM", "reviewer": "Reviewer_uDb6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work,  a novel design of a time-dependent PDEs solver is proposed. It is achieved via a neural network and based on recent works on implicit neural representations (MLP + SIREN). This method is an alternative to a a well-known family of spatial discretization methods.", "summary_of_the_review": "The overall work is satisfactory. However,  the experimental part could be drastically consolidated with more challenging tasks and comparison with existing works. ", "strength_and_weaknesses": "The proposed method is straightforward either from modeling or implementation standpoints. It allows to encode the spatial information through a neural network where its weights store  informations of downstream tasks  in an implicit manner.  In contrary to classical  numerical discretization scheme where their success is highly related to the quality of mesh including the spatial sampling and the complexity of some local regions especially near the boundary layer,  implicit neural representation is independent of the number of points and spatial location since the learned weights are global and as a result affect the vector field globally. Moreover, implicit neural representations are powerful to mitigates the spectral biais of neural networks by leveraging high frequencies information and capturing well fine-grained information without scarifying the generalization capabilities of neural networks. The proposed work is also interesting if we look at the tedious task of numerical solver to build adaptive representation by considering different sampling (remeshing) strategies where their choice highly depends on the task, implicit neural representation are adaptive by construction w.r.t weights of the neural network, hence it is agnostic to the neural network architecture.\nAnother interesting aspect of this work is spatial gradient computation achieved through auto-differentiation. In traditional methods such VFM, FEM, higher order gradients requires higher order basis function such as Chebyshev polynomial which is very expensive in practice (complexity grows with the targeted order to achieve the desired accuracy). In the proposed work higher order gradients are cheap to get as neural implicit representations can achieve sufficiently high order easily by construction thanks to periodic function.\n\nOne of the drawbacks of the proposed work is in the computational time. The authors are aware of it and give some directions to tackle that in future works.\n\nThe claims are corroborated with experimental study including different physical systems such as Advection equation, N-S equations, and Elastodynamics equations. The results are in general satisfactory.\n\nI have some considerations and questions:\n\n1/ In N-S experiments, what is the Mach and Reynolds number ?\n\n2/ Have you tested or thought about SIREN variants including Multi-Scale implicit neural representations ? The latter could be more representative since Physical systems are multi-scale by construction.\n\n3/ Implicit Neural Representation are based on periodic functions. I am wondering if the good results you obtain on physical systems described with periodic boundary condition is intrinsically related to this property of periodicity of the neural network and boundary conditions. Have you any results on the same physical systems but described with non-periodic boundary conditions ?\n\n4/  It would be interesting to have more quantitative results and comparisons with recent PINN variants.\n\n5/ One other experiment that would be important to conduct is to assess the generalization capabilities of the proposed model on complex geometries (especially near the boundary layer).  For instance, considering the datasets and tasks described in \"Learning mesh-based simulation with graph networks\".\n  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9Gi8dwt9yi", "reviewer": "Reviewer_7x1Y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using implicit neural representations as a domain discretization, on which a PDE acts. For temporal evolution, classical PDE integrators are used. Next to requiring less memory for the representations, the model has the capacity to represent small but relevant features accurately. However, in the current implementation, the inference is more than one order of magnitude slower compared to using a classical PDE solver.", "summary_of_the_review": "The paper is very insightful, but with some minor details to mend before publication. I would already give the paper a 6, but if my comments are addressed appropriately, it would become an 8.", "strength_and_weaknesses": "The idea of using low-dimensional implicit spatial representations as a PDE domain discretization is really interesting because it starts with the assumption that there exists a low-dimensional manifold, on which the physical system lives. I could imagine using such representations to study the structure of chaotic systems such as turbulent flows, see Dynamic Mode Decomposition.\n\nLooking at this paper as a first step in the right direction (I have many ideas about how one could improve this method, but this paper is already a legitimate first step), there are some things that can be improved:\n1. Reading the paper for the first time, I didn't get how the inference is carried out up until Section 5, which clarifies that the implicit model is retrained at every time step, which of course is a time intense operation. Could you add an algorithm or just a step-by-step description of the inference step in Section 3, or at least in the appendix?\n2. In a pure computer vision paper, I could accept demonstrating performance only through visuals or graphs, but in a PDE paper, I would expect a table with results that other papers can compare against. Could you please add such data (even if it is in the appendix)?\n3.  Section 4.2 shows the performance of the method on the \"Navier-Stokes Equations\" (NSE). Having a solid background in fluid mechanics, I have to say that the *inviscid* NSE are called \"Euler Equations\" and have quite different behavior than the general NSE. Thus, it is for the least misleading to call Section 4.2 \"NSE\".\n4. In the same line of thought as 3., because the Euler Equations have zero viscosity (=> there is no dissipation), the 2D Taylor-Green system has a stationary solution, i.e. nothing will change at all. This is a nice example as a demonstration that the proposed method doesn't add too many errors as would happen if we use any finite spatial discretization, but this should be stated somewhere. I mean something like \"This example demonstrates that our method quite well preserves a stationary solution, but it does not yet show any dynamic behavior. For a dynamical 2D simulation we have the example with the two vortices.\". Pointing out the preservation of the stationary solution is indeed a strong argument compared to classical methods, which very often entail the so-called numerical diffusion term.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0AwTq_YxjjN", "reviewer": "Reviewer_8oNn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use spatial-coordinate neural networks as an alternative to the spatial representations used in traditional solvers for time-dependent partial differential equations. This work reformulates the PINN-like PDE solvers in a time-dependent way. It calculates the spatial derivatives as in PINNs at each time step and optimizes an INR w.r.t a physical loss involving a discretized time derivative at each time step. The proposed approach extends effectively PINNs beyond the limited temporal domain.", "summary_of_the_review": "This paper introduces the time-stepping into the original PINNs framework, which appears to outperform PINNs and some traditional solvers in many use cases. However, due to the limited novelty and comparison with the baseline methods, I think the paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "Strength:\n- The approach uses coordinate neural networks that can be evaluated at any location in space, which is more flexible than commonly used solvers defined on grid or meshes. \n- The optimization is more reasonable than the original PINNs for time-dependent PDEs: the residual loss based on time-stepping introduces the causal temporal relation into the optimization.\n\nWeaknesses:\n- As the paper proposes a new meshless PDE solver, it is then necessary to mention the background of the solvers of the same type. Also, the comparison to the existing meshless PDE solvers other than PINNs is somehow limited. If other meshless PDE solvers turn out to be as effective as the proposed method, can the claim about the trade-off between wall-clock runtime and other advantages still holds?\n- Comparison w.r.t approches proposed for the same objective of going beyond the training temporal horizon of PINNs is needed, e.g. (Wang & Perdikaris, 2021). \n- Question about the comparison with the PINNs in Section 4.2 inside the same training horizon: are PINNs trained as in the original paper (Raissi et al., 2019)? How the PINNs perform with sequence-like training compared to the proposed method (e.g. Krishnapriyan et al., 2021), which is closer to the setting of the paper?\n\n\nReferences:\n- (Raissi et al., 2019) Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations\n- (Krishnapriyan et al., 2021) Characterizing possible failure modes in physics-informed neural networks\n- (Wang & Perdikaris, 2021) Long-time integration of parametric evolution equations with physics-informed DeepONets", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject