Learning Time-dependent PDE Solver using Message Passing Graph Neural Networks
Authors: Pourya Pilva
Ahmad Zareei
Venue: ICLR 2022 Submitted (2022)
Keywords: graph neural networks
partial differential equations
time-dependent PDE
message passing graph neural networks

Abstract: One of the main challenges in solving time-dependent partial differential equations is to develop computationally efficient solvers that are accurate and stable. Here, we introduce a general graph neural network approach to finding efficient PDE solvers through learning using message-passing models. We first introduce domain invariant features for PDE-data inspired by classical PDE solvers for an efficient physical representation. Next, we use graphs to represent PDE-data on an unstructured mesh and show that message passing graph neural networks (MPGNN) can parameterize governing equations, and as a result, efficiently learn accurate solver schemes for linear/nonlinear PDEs. We further show that the solvers are independent of the initial training geometry and can solve the same PDE on more complex domains. Lastly, we show that a recurrent graph neural network approach can find a temporal sequence of solutions to a PDE.

Under review as a conference paper at ICLR 2022
LEARNING TIME-DEPENDENT PDE SOLVER USING (RECURRENT) MESSAGE PASSING GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
One of the main challenges in solving time-dependent partial differential equations is to develop computationally efficient solvers that are accurate and stable. Here, we introduce graph neural network approach to finding efficient PDE solvers through learning using message- -aassagggg models. We first introduce domain invariant features for PDE-data inspired by classical PDE solvers for an efficient physical representation. Next, we use graphs to represent PDE-data on an unstructured mesh and show that message passing graph neural networks (MPGNN) can parameterize governing equations, and as a result, efficiently Iearn accurate solver schemes for linear/nonlinear PDEs. We further show that the solvers are independent of the initial trained geometry, i.e. the trained solver can find PDE solution on different complex domains. Lastly, show that recurrent graph neural network approach can find a temporal sequence of solutions to a PDE.
INTRODUCTION
Physical phenomena are generally modeled through partial differential equations (PDEs) that govern the dynamic evolution or static solution of physical system. Numerically solving partial differential equations is an important aspect of scientific and mathematical modeling in a broad range of fields including physics, biology, material science, and finance. There have been many efforts to develop efficient and accurate numerical solvers for PDEs using different techniques including finite difference (Le Veque 2007 Shashkov & Steinberg 2018), finite volume (Eymard et al.) 2000 Brenner al.) 2008), and finite element schemes (Reddy) 2014; [Wriggers] 2008). While these methods have been successful in producing accurate solutions. major challenges for accelerating and reducing computational cost when the governing PDE is known, and also determining the governing PDE when the physical system is unknown, remains be addressed, problems such those in climate modeling, turbulent flow, contact problems, or plastic material deformation. With the recent developments in deep learning, faster algorithms have been proposed to evaluate the response of a physical system, using only observational data. While deep learning approaches, such as multi-layer perceptron (MLP) or convolutional neural networks (CNNs), are powerful in learning PDE solutions, they are, however, restricted to a specific discretization of the physical domain in which they are trained. As result, the learned model is limited to a specific domain and can not be generalized to solve on different domains for different discretizations, although the underlying physics remains to be the same. New training is required for any change in the physical domain or discretization. Here we propose a discretization and domain invariant neural network time-dependent PDE solver based on message passing graph neural nets (MPGNN) which is trained on a sample domain with different discretizations. The trained MPGNN can then used solve for different discretization or even on other domains as long as the underlying physics remains the same. We further show that recurrent version of MPGNN can be used to find a temporal sequence of solutions to a PDE.
RELATED WORKS
One class of neural-net-basee PDE solvers focuses using neural networks proxies of PDEs and aims at finding the solution by minimizing a loss that corresponds to the solution satisfying the governing equations and the boundary conditions (Raissi et al. 2017a Lagaris et al. 1998: Weinan

---

Under review as a conference paper at ICLR 2022
& Yu 2018 Sirignano & Spiliopoulos 2018 Khoo et al.) 2021).
Although such an approach helps to find the one-time solution of PDE with an instance of parameters, a slight modification to the PDE parameters, boundary conditions, the domain requires re-training of the network.
Another approach to solving PDEs is to use convolutional neural nets and snapshots of observations over the discretized input domain and to learn the dynamic evolution of PDE (Long et al. 2018 ( [Shi
et al. 2020). Further modifications such as using residual connections (Ruthotto & Haber] 2020p.
or autoregressive dense encoder-decoder (Geneva & Zabaras 2020), or symbolic multi-layer neural network (Long et al. 2019) in addition to the CNN can be used to improve the results. While these
models do not require prior knowledge of the PDE, they are limited to domain discretization (as a
result cannot be generalized to arbitrary domains) and are limited to certain time discretization (as a result unable to handle temporally and spatially sparse or non-uniform observations).
Inspired by the discretization techniques in solving PDEs, class of methods uses observational data to learn the discretization approximation required for the updates in classical computational PDE solver methods (Bar-Sinai et al., 2019; Kochkov et al. 2021] Zhuang et al.) 2021] Han et al. 2018). In this approach, a neural network is used for better interpolation at coarse scale to be used in the framework of traditional numerical discretization. These methods are used in conjunction with classical numerical methods and can improve the accuracy and accelerate the solutions of the traditional numerical schemes (Kochkov et al. 202T). Although these methods have been shown to generalize to new parameter regimes of a PDE, they are still bounded to the initially trained discretization and can not be used for arbitrary domains without re-training.
Lastly, a class of neural PDE solvers focus on graph representation of the discretized mesh data- structure to approximate the PDE solution (Li al. 2020a by Iakovlev et al.) 2020 Belbute-Peren et al. 2020). The numerical solution of a PDE is an approximation of the solution on discrete locations comprising a discretized mesh of continuous space. Each node represents a region in the continuous space and the approximate solution of the PDE in that region is assigned to the representative node. The discretized mesh forms a graph where each node is used to model the state of the system and forms a connectivity graph connecting to the neighboring nodes. This method has successfully been used to solve time-independent PDEs with different mesh sizes on the same physical domain (Li et al., (020aa). The connectivity and the location of the nodes can further be optimized to learn the solution with different levels of precision (Alet et al. 2019). If the PDE includes long-range interactions, which happens mostly in time- imeeinddeenndent PDEs, multi-level graph neural network framework to encapsulate long-range interactions can be used to improve the indeeedddeedd PDBs, in the case of time-dependent PDEs,
results (Li et al. 2020b). In contrast has been shown that a continuous-time model similar to physics informed neural nets but with graph neural network can be used to recover system's 's dynamics with sparse observational data recorded at irregular times (Iakoylev et al. [2020] Poli et al., 2019). Recently, it have been shown that message passing graph neural networks can be used implement powerful physical simulation engines (Praff et al. 2020; (aanhez- Gonzalez et al. (0200). The state of physical system can be expressed using particle- based method as reduced order model. The particles are then expressed as nodes in graph and the message passing neural network learns to compute the dynamics of the Saaticle(Sanchee--Gone Gonzalez et al. 2020). In addition to particle-bases based methods, mesh-based methods have been shown to be successful in physical simulations (Pfaff et al. 2020). Such graph-based models, first encodes the input data into a latent space and then process it in the latent space (reduced model), and to obtain the physical results decode the data back to the physical space. Here, we first show why graph neural networks can generalize to learn fast PDE solvers inspired by finite difference schemes. We introduce domain invariant features and boundary conditions inspired by classical PDE solvers to improve the generalization of the learned PDE solver operator. With the introduced features, we show that message passing graph neural network architecture efficiently fits the classical PDE solvers and can learn time- sieesssttppppleeeeeeeeeeeeee operators for linear and nonlinear PDEs with different boundary conditions. We further demonstrate that our trained graph neural network solver can be generalized to solve PDEs on physical domains different from the domain that it is trained on. This is beneficial to train GNN on a sample of small domains for even with unknown dynamics, and further, explore the dynamic behavior on different larger physical domains. Lastly, we show that recurrent version of our MPGNN can be used to predict the temporal sequence of solutions to a PDE.
2

---

Under review as a conference paper at ICLR 2022
3 TIME-DEPENDENT PDES
We consider continuous dynamical system u(x,t) E evolving over time E Rt and spatial coordinate x E S2 C IR where S is bounded d-dimensional domain. We assume the system is governed by a partial differential equation of the form
Ut = M(u
where MoeereN[[:: A] denotes the lincar/nonlincar differential operator(s) parameterized by the vector A. In the above general form, the temporal evolution of the current state u, depends the differential operator A M >] which may include various spatial derivatives of the state including Vu, Vau, etc. Depending on the differential operator N, appropriate boundary conditions well-posed PDE with a unique solution. Such PDE model is the cornerstone of mathematical models and is widely used to model various systems, from fluid dynamics, thermal sciences, acoustics, and quantum mechanics, As an example, U, = Nuu: A| = A v + X2 Vu constitutes convection diffusion equation for u R as variable of interest, where X1, X2 are the diffusitivity and the velocity field vector with which the quantity is moving with.
öΩ is required for
The state of the system at each time can be obtained using its initial state and time integration as uxx,t) = u(x, 0) + f [u: Ajdt. Numerous numerical techniques such as Finite Elements, Spectral Methods, Finite Difference, Finite Volume techniques have been developed efficiently approximate the differential operator (1; and solve for a dynamical system over time. In all numerical schemes, the domain is first discretized with mesh, the differential operator is approximated locally using neighboring points, and the solution is calculated over small time steps using a time integrator such as Euler's scheme, i.e..
n++ (x;) = (x;)) + 8tFF(u (x;), Vu" (x:), (x.",,))," .. (xx;))
where the superscript n shows the solution over discretised time t.., and the diffferential operator N(u; A) = Fuu, Vu, v.u.... ; A) shows that it contains information about local spatial deriva- tives. As an example, consider solving heat equation, u4 DV where D is the diffusion constant and 2 = /P2/ + 2 on an structured grid shown in Fig. Th. Let w be the discretized solution at time t = nôt and spa- tial location x iδz and y = jöy where öt, dx, and ōy are the time, horizontal, and vertical spa- tial discretization respectively. The time and spatial derivatives in the heat equation can be expanded using Taylor series at each discretized point, where Ouf, / Ouu = (u 2nL + 1.5) /8x2.
(b)
Figure 1: (a) A square domain with structured mesh discritization (b) A curved domain with an unstructured
(u /f8)
n+1 = u + StF and etc. Re-writing the equation for an arbitrary discretized point, we find u where F = (Mr.j41 Ui.;) + a (us,i-1 - + 3 (u:+1.5 - uz.j) + 3 (u:-1. Us.j), where 28 and β D6 Solving for this equation for all the points along with the boundary conditions the updates for the discretized points can be achieved. Note that here the update rule can be seen as the summation of updates that only depend on neighboring points. Although in this example with a simple linear equation and a structured grid was easy to find the update rule F, given an arbitrary domain that requires triangular mesh for discretization (see Fig. [b)) and nonlinear governing equation the update rule is not straight forward to be worked out. Our objective here is to learn the approximation of the differential operator using graph representation of the domain with message passing neural networks. Since in a general PDE the differential operator F(u, Vu, Vzu.... ... A) contains local spatial derivatives and only local neighboring values at point are relevant to approximate the differential operator at that point. As a result, a graph neural network promising framework approximate the above right-hand side for the next value predictions.
3

---

Under review as a conference paper at ICLR 2022
4 CONTRIBUTIONS
In this paper, propose graph-based model to learn domain invarrian and free-form solvers for a PDE on arbitrary spatial discretization using message passing neural networks. Our method here inspired by the finite difference method and the possibility of approximating partial differential equation using discrete point stencils. Here, we use graph neural networks as a nonlinear function approximator and use the simulated data learn the required stable stencils. In order to show that that the graph neural network has leamned the correct PDE solvers, we test our learned graph network domain with a different geometry and mesh discretization. We find that the trained model on sample domain can be used to predict the PDE solution on other physical domains and mesh discretization. This only possible by creating relevant features inspired by classical PDE solver techniques to represent the differential operator. Our contributions are: (i) introducing locally invariant feature representation inspired by classical PDE solvers to efficiently learn differential operator solver; (ii) showing graph representation with message passing neural networks that can parametrize PDE solvers with a domain- invariant representation; (iii) obtaining robust high-perte sech-eeffrmance models for various linear/nonlinear PDEs with arbitrary spatial discretization and showing that it can generalize to arbitrary physical domains, (iv) proposing a recurrent message passing graph neural network approach to predict temporal sequence of PDE solution over time.
to find PDE solutions
GRAPH NEURAL NETWORKS FOR PDES
Let G = (V, E) be graph representation of mesh with nodes V = ) where x; denotes the positions, and edges & {ei; } where Ejy represents the connecting neighboring points at x; and Xj. Given physical domain, use uniform random distribution of points for the nodes, and use Delaunay triangulation to find the neighboring points. We denote neighbors
node i with Mii { E E.. We further assign the node and edge attributes with u, and ety respectively (see Fig. D In a message-peee saassagg neural network (Gilmer et al. 2017), we propagate the latent state of nodes u1 of for K layers, where at each layer k, we have
(k) (6)
u (k-1)
1
Σ jEM(S)
g(h)
(u, (k-1)
(k-1) Eij))
(3)
[M(iI
where (a4) and ((0)) are differentiable deep neural networks. Note that -= -MM(i) represents the number of neighbors for node i, and furthermore instead of the average used in equation 3 other
permutation invariant aggregation function such as sum or max can also be used. Since equation (3)
an approximation of equation 2 where ((;;; X) includes various spatial differentials, we take
our edge features to include x,
for j E M(),, and also A(x,)) which is the PDE parameters at
(x: + xx) x 2/2. We further can include, higher derivatives of the PDE parameters such V VA(x;)) ), V (xxa) for a better approximation representation. At each point in time, we set u; = [u(xi, (), u ux1, 1 ot), ōt),.... w(xi, / not)] as the last n snapshots for the initial create the desired u,
the midpoint of the edge X
(K) [u(x;, T), u(x;, T' - ōt'),- ... u(x], mot')
latent space and use as the desired last m frames of the solution. The main assumption here is that the derivatives can be approximated using the graph nodes, which is possible in most physical simulations where the solutions are smooth. Additionally. note that the predicted values here include all inside and boundary nodes.
We use multilayer perceptron for the (k) and ((k) with three hidden layers. Note that three hidden layers, allows each point to connect the third order neighbors (i.e., neighbors of neighbors of neighbors), which potentially only allows for maximum 6th order derivative estimation. In general, the node features consist of the solution in previous time steps and edge features, motivated by the logic of solving PDEs, are made up of the distance between connecting neighboring nodes, along with the PDE parameters calculated at the center of the edge. We also add an extra feature to the node attributes showing that if the node lies the boundary not. This extra feature is 1 if the node lies on the boundary and 0 otherwise. Our decision on the node and edge features might slightly differ for different equations, and point out if there is any change in the features set.
4

---

Under review as a conference paper at ICLR 2022
(a)
" - (u;
Discrelization
F
pass &
(A)
mii
(R+-1)
Maj
(CM(
u4
(k)
((k) (u: uj c13)
(k+1) u
u,
m (k)
JCM(
Figure (a) Example of physical domain governed by the time dependent PDE ur NN: X). and the spatial discretization that represents graph structure. The inset figure shows the node features u, and the edge features on the mesh eij- (b) Graphical representation for one layer pass of message passing neural network: given node values u,
(k) and edge attributes eij. message is generated eij where (b(A) is neural network; next, given average message value N EjEM(S) my and the node features u, the next feature set for the nodes are obtained using another neural network (Ak)
for each edge such that mij (b(A)
(k)
(k+1)
TEST CASES
In this section, we through different linear/nonlinear PDEs for physical systems and evaluate the performance of our modeling. First, we start with the time-dependent heat equation and show how our model can learn to predict the future state(s) (see section 6.11. and more importantly, the learned model can be used for predictions in new physical domains different from initial learned domain (see figure 4 To predict the sequence of temporal data of a PDF, we use recurrent message passing graph neural network approach and show that our model is able to predict the sequence of PDE temporal data (see figure 6). Next, to show that the model is able to learn the nonlincarities, we focus on the Navier-Stokes equation to predict future solutions with an arbitrary discretization (see section 6.2). Lastly, to show that the model is able to include PDE parameters, we focus the advection- diffussion equation and by including the PDE parameters as edge attributes, we show that the model is able learn to predict for various PDE parameters (see section 6.3
6.1 HEAT EQUATION
Time-dependent heat equation in two dimensions can be written as
2 E Ω
(4)
which can be solved with initial condition u(x, 0) and the Dirichlet boundary condition, i.c.,
u(x E 02, t) = ug. where iQ denotes domain's 's boundary. We chose a square grid [0, 1] x [0, 1].
and use Fi redrake (Rathgeber et al. 2016) with characteristic length of dæ 0.0625 for
5

---

Under review as a conference paper at ICLR 2022
Input
Ground Truth
Prediction
146.5 1
10
te
Ed
644
04
<0
00
82
04
00
04
06
10
04
10
02
38
02
06
14
04
05
200
20
02
24
04
72
08
60
74
10
144
188
02
Figure 3: Message passing graph neural network learning predict the heat equation: Input data frames (first four columns), prediction (fifth column), and ground truth (the last column) for three different test in each row. The simulation data corresponds the heat equation with different boundary conditions. The first four columns correspond to the results with At = 206 time difference, and the network predicts the results for the next frame with At 200t after the last frame. The average MSE loss for the test data is 5.1 x 10
triangular mesh and time-stepping of öt = Note that the mesh in the simulations are generated using built-in mesh generator in construct a Delaunay triangulation of uniformly distributed nodes (Poisson point process) in the domain. We chose different Dirichlet boundary conditions where we set top, left, right, and bottom boundary conditions different constant values of ugg [0, 200]. Note that u for different simulation also remains in the same range generated mesh and set of boundary conditions. We set record the data every 2081, and use four subsequent observations for the input data (i.c., 4), and predict the next frame (i.c., a result, we input frames, 0, 20ot, 408t, 60ôt and predict the frame 808t. Since per each simulation, can further the initial frame to be any of 200t frames, can generate 20 data inputs per simulation. We generate 1000 simulations with different meshes and boundary conditions, and with that, we create 20,000 input data. We define MSE loss between the network output and the true values as >; ||ù, will will²/N where û; are the network prediction and ground truth for the node i value and N' is the number of nodes. We further three layers for the message passing graph neural network with 64 nodes, three message passing layers K = 3, and choose two three-layer neural networks for y (R) and (k) where the hidden layers are of the size 6 128, 128 x 128 and 128 x 256 0.001 using ADAM optimizer and step learning scheduler of 0.2 after every 5 epochs. Note that the network architecture and hyperparameters remains the same for different simulations. unless mentioned otherwise. We find that the MSE error starts at 179.2 and reduces to 7 x 10-7 after 20 epochs. The relative Lg on the test data after training falls below 6 x 10 The results for the prediction and ground truth for different boundary conditions on test simulations that the network has never seen are shown in figure 3 We further use the same generated data, however with more number of nodes (changing from 64 to 128) the graph neural network and find that the error the MPGNN the test data does not change (mean of relative L2 error on the test data changes from 5.2 x 10 to 5. X 10-6) 6).
F rerrraee In order to sample the data for training the MPGNN,
E [0, 200). For each simulation, we use newly
1). As
respectively. We set the learning

---

Under review as a conference paper at ICLR 2022
Input
Ground Truth
Prediction
175
123
155
0.6
02
0.2
02
1.0
1.0
0.5
100
30
Ground Truth
3.0
Ground Truth
Ground Truth
Pred.
Pred.
Pred.
375
120
25
2.5
256
20
80
15
1.5
10
1,0-
10
05
5
00
3.0
02 1.0
Figure 4: MPGNN trained on a square domain, predicting the values on a new unseen physical domain. We train MPGNN to predict the heat equation on square domain (see figure 3) Next, we use the sample MPGNN to predict the PDE values on an unseen geometry and boundary conditions.
Next, to show that the learned MPGNN can generalize to dif- ferent physical domains, create a distorted physical domain with a curved boundary, two inclined edges, and vertical wall. The newly generated geometry is different from the initial square geometry and the network has never seen such domains. We introduce random mesh on this new physical domain with similar characteristic mesh size and use the MPGNN (that is only trained on the square domain) to predict the results. We find that the MPGNN indeed predicts the output with high accu- racy (average MSE loss of 2.8 x 10-4). Examples of different simulations with different boundary conditions
10a1
10-2
same geometry
-Me-
different geometry
MSE Error
10-3
10-4
shown in figure 4 This shows that the graph neural network here learns predict the future values independent of the initial physical geometry and can be extended to predict a PDE results on new unseen physical domains. Here we have used four previous snapshots as the input to our network. In order to show the effect of number of input frames on the prediction power of our network, run tests by changing the number of input frames and calculating the final average MSE error on the test data set on both similar and different geometry. In all the cases predict the frame at 808/ after the final frame. When we change the number of input frames, the input frames are respectively 408t, 25öt, 205t. 158t, and 106t apart for two, three, four, five, and eight input frames respectively. Interestingly, for prediction on similar geometries, however to have a reliable MSE on a different geometry we need least three four frames.
10:1
Input frames
Figure 5: MSE error tried on the same and different geometry versus number of input frames
find that two input frames enough
7

---

Under review as a conference paper at ICLR 2022
Lastly, show that the MPGNN is capable of predicting sequence of temporal data, use MPGNN in a recurrent format (see figure 61). We first train our MPGNN with a sequence of three past results (i.e, = 3) to predict the next frame (i.e., given input data of initial three frames of a simulation (41, 12, (3), it can predict the next time step solution 44. Next, time frame and so on. The results for sample test case is shown in fig It is to be noted that we did
1). As result. in the trained network,
concatenate the last three obtained results, i.e. (12,43, f4) to predict the next
recurrent loss function train our network, and as result, the average MSE loss keeps growing with the number of predicted frames. The average MSE error for 1000 different simulations along with the range of error are shown in figure where the error slowly increases with the number of predicted frames. It is expected that training the MPGNN with recurrent loss function would improve the overall result.
(a) R-MPGNN
(6)
Impur
MPGNN
pred.
MPGNN
truth
(c)
MPGNN
% Error
NivMttx MSE Eno
Average MSE Enor
15
16
17
110 b11
Figure 6: (a) Schematic of a recurrent message passing graph neural network (R-MPGNN) learning predict the heat equation: input data the initial three frames (f1, (t) and the network predicts the next time step (t4). In the next layer the output of the previous layer f4 is concatenated to the last two frames 12, ta create the input for the next layer (12, f3,4)) f4), and so (b) Test data input and outputs for the R-MPGNN predicting the next frames in time in comparison with the ground truth. (c) The average of the MSE loss for each frame prediction over time.
6.2 NAVIER-STOKES EQUATION
In order to find MPGNN's performance on learning nonlinear PDEs, we use two-dimensional incompressible Navier-Stoker equation as
Do, + Vo- - -V -
(5)
where v (u, v, 0) is the velocity field, p pressure, p the fluid's density, and is the diffusion constant. The second equation known as the incompressibility equation which assures a divergence- free velocity field (i.e.. V.v 0). We use spectral methods to solve the above equation on square domain with periodic boundary conditions (see supplementary material for the details). We chose square domain with [0, 2T] [0, 2nr] and input random initial condition conditions we sample random numbers uniformly in the range [0, 5] for the nodes in the graph and then we then remove the high frequency patterns by taking a Fourier transformation and discarding the top 1/3 of high frequency wave numbers. Furthermore, in this setting and run our numerical PDE solver for 50 and record frames with öt = 0.002. Note that u, for different simulation also remains in the same range u; [0, 4]. We chose št such that the results become visually different. We first input five different frames (i.e., n = 5) and predict the next frame (i.e., = 1). We generate 20000 data points from 25 simulations similar the heat equation data
generate the initial
and =3e-4.
8

---

Under review as a conference paper at ICLR 2022
generation. In this case, as the one used for the heat equation. We find the relative Lg error for the test data after 25 epochs drops from 10.1 to below 5.4 10-5. The results for three different test cases are shown in figure 7 Note that in here in the plots, we are presenting the vorticity field Ç = v 0n instead of velocity fields separately. The result here shows that the nonlinear PDE structure also be learned using the MPGNN architecture.
the same network geometry for the message passing neural network
Input
Ground Truth
Prediction
1.0
2
0.8
0.4
4
0.2
Figure 7: MPGNN trained Navier-Stokes equation with periodic boundary condition. Each row represents a different test simulation, where the first five columns are the input frames and the last two columns respectively the MPGNN prediction and the ground truth result. We find that the average L2 error for the test data 5.4 10 10-5
6.3 ADVECTION-DITFUSIA EQUATION
Input
Ground Truth
Prediction
0.6
52
0.2
Figure 8: MPGNN trained to predict diveectionddifusion equation with horizontal periodic boundary. Each row represents different test simulation, where the first five columns are the input frames and the last two columns are respectively the MPGNN prediction and the ground truth result. We find that the average MSE loss for the test data 1.3 10 10-6

---

Under review as a conference paper at ICLR 2022
In all the PDEs tried far, we kept the PDE parameters as constant. In this section, we focus on the ddivectionddifusiio equation where the PDE includes different free parameters that can be used as edge features in the MPGNN. The advection-diffusiio equation is
Ug + 21 = X2 V
(6)
where A and X2 are the advection velocity and diffusion constant respectively. We chose our domain to be [0, 1] x [0, 1] and use characteristic mesh size of åz = 0.0625 and time stepping of öt = 10-4. We record the data every 2008t and use four subsequent input frames (i.c., m = 4) and predict the next frame (i.e., = 1). We use Firedrake (Rathgeber al. (016)) to simulate our PDE with random initial meshes and initial conditions. We use periodic boundary conditions at = 0, 1 and Neumann boundary conditions y 0, 1. The initial condition is created using 240 = a. sin(x) + 02 (2n ) + 0g cos(x) + 04 (2s(2x)) where all ai, i = 1ii=1,... 1,, .4 are drawn from a uniform distribution [1,-1, 1]. Note that for different simulation also remains in the same range as i E 3.3.5 We select Y1 and X randomly from uniform distributions 4 (0,5, 1.5]. In total we generate 300 simulations with random initial conditions and random PDE parameters and generate 300k input data for our network. We find that in training our network the average L2 error for the test data (new simulations that network has never seen) drops from 3.4 before training to 1.3 x 10-6 after training. Three different test results are shown here in figure g
7 BENCHMARK TESTS
In this section we compare our approach with other existing relevant studies. We have identified two relevant neural network approaches closely related to our work: [akoole et al. (2020) and (2) Li et al. (2020b). Since we are interested in generalization of the network to other domains and mesh sizes, perform two different tests on all of the networks and compare their performance by measuring average MSE for prediction on the test dataset. We select the heat equation as our basis (see Sec. 6.1 test different approaches on square geometry and test it the unseen domain discussed in Sec. 6.T We perform two set of tests: training on a square domain with a small characteristic mesh length (average edge size is 0.1 which we call high res), and testing the performance on the same geometry, and on different geometry (shown in Fig. 4 with similar resolution as well lower resolution (average edge distance of 0.2 which we call low res). Next, we repeat the same experiment but with training on low resolution data. The results are summarized in Table In repeating the work by Li et al. (2020b), we used their provided code at the corresponding git repository and used their suggested hyper-parame parramterssseers and made use of u, as node features, and the position of the nodes as edge features. As seen in table [ the proposed network has a lower MSE best when it is applied to the same geometry, however, in generalization it performs poorly which is due to the positional feature of the nodes instead of the invariant edge distance. In following Lakovlev et al. (2020), similar the previous case, we followed the proposed structure by the authors (i.e., three hidden layers with 60 neurons along with tanh activation function and one message passing layer). The main difference here is that the edge features contain relative positions the edge features instead of absolute positions. This is indeed the main reason for better generalization of this approach compared to the other method. Our method with a different activation function (ReLU instead of tanh) as well as three layers of message passing, along with extra feature flagging the boundary nodes shows an improved result.
Trained on high res (1)
Trained on low res (1)
MSE on test data
Our work
(2)
Our work
(2)
same geometry (same res)
4.01e-6
3.33e-5
1.49e-6
4.53e-6
6.27e-5
2.4le-6
different geometry (low res)
7.98e-5
1.57e-4
9.87e-3
1.57e-4
1.63e-4
1.47el
different geometry (high res)
5.56e-5
9.86c-5
5.34c-3
8.06e-5
1.11e-4
5.26c0
Table 1: Comparison between our MPGNN, with (1) Iakovlev et al. (220) and (2) et al. (2020b).
We train all networks using different mesh resolution and test it on similar resolution networks, moreover to a different geometry with two different resolution.
10

---

Under review as a conference paper at ICLR 2022
8 CONCLUSION
In this paper, we showed how message passing neural networks can parametrize time-dependent partial differential equations and their discrete classical PDE solvers. Inspired by numerical PDE solvers, we introduce domain invariant features for an efficient representation of PDE data. Us- ing graphs represent an unstructured mesh, we trained message passing graph neural networks (MPGNN) to efficiently learn accurate solver schemes for linear/nonlinear PDEs independent of the initial trained geometry. We showed that MPGNN can predict single/multtile frame(s) of the future data and furthermore trained model can predict results on unseen geometries. We further proposed a recurrent version MPGNN to march in time and find temporal sequence of solutions to PDE. In summary, the main important features of message passing graph neural network that make them suitable platforms for learning the time-dependent PDEs are: (i) MPGNNs similar to time-dependent PDEs spatially locally dependent where each point is only affected by the neighboring points for a small time-step, (ii) the permutation invariant combination of the messages signifies the rotational and translational symmetries in given physics-based PDE and helps the network to learn more efficiently, (iii) the neural networks used in creating the messages and the updates are general learners that can potentially learn the nonlinear update rules, and lastly (iv) the possibility of running several passes in message- passing neural networks helps the update of a node to see features of neighbors of neighbors and as a result MPGNN can find the update rule for larger time-steps. The possible future extension of our current MPGNN solver are: 1) including random long range connections in the initial mesh to enable long-ting time predictions with less number of message-passingg layers: 2) training recurrent loss function and addressing significant memory required in backpropagation order to improve long time predictions, ;3) using more accurate temporal schemes rather than Euler discretization improve predictions, 4) including symbolic regression analysis to uncover the discretized kernel learned by the MPGNN and exploring the possibility of uncovering new PDE solver schemes.
11

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ferran Alet, Adarsh K Jeewajee, Maria Bauza, Alberto Rodriguez, Tomas Lozano-Peree and Leslie Pack Kaelbling. Graph element networks: adaptive, structured computation and memory. arXiv preprint 00Xiv:/904000099 2019.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15444-1544 2019.
Filipe de Avila Belbute-Peres, Thomas Economon, and Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In International Conference on Machine Learning, pp. 2402-24222022 2411. PMLR, 2020.
Susanne C Brenner, L Ridgway Scott, and L. Ridgway Scott. The mathematical theory of finite element methods, volume 3. Springer, 2008.
Robert Eymard, Thierry Gallouut, and Raphaèle Herbin. Finite volume methods. Handbook of numerical analysis, 7:713-1018, 2000.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks. Journal of Computational Physics, 403:109056, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMLR, 2017.
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 11 115((4))8505-88)) 8885-88000 2018.
Valerii Iakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-time pdes from sparse data with graph neural networks. arXiv preprint arXiv:2006.0890 2020.
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mathematics, 32(3): 32(3):421-433 2021.
Dmitrii Kochkov, Jamie A Smith, Ayya Alieva, Qing Wang. Michael P Brenner, and Stephan Hoyer. computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021.
Machine
Isaac E Lagaris, Aristidis Likas, and Dimitrios Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions neural networks. 10009877-0000 1998.
Randall J LeVeque. Finite difference methods for ordinary and partial differential equations: steady- state and time-dependent problems. SIAM, 2007.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint wwXiv:2003.03483 2020a.
Zongyi Li. Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances Neural Information Processing Systems, 33, 2020b.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: International Conference on Machine Learning, pp. 3206-3216. PMLR, 2018.
Learning pdes from data. In
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
Tobias Pfaff, Meire Fortunato, Alvaro GonzalzZ and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
12

---

Under review as a conference paper at ICLR 2022
Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07332, 2019.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i); Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.1056.
2017a.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physies informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations, 2017b.
Florian Rathgeber, David A. Ham, Lawrence Mitchell, Michael Lange, Fabio Luporini, Andrew
T. T. McRae, Gheorghe-Teodon Bercea, Graham R. Markall, and Paul H. J. Kelly. Firedrake: automating the finite element method by composing abstractions. ACM Trans. Math. Softw, 43(3):
24:1-24:27, 2016. ISSN 0098-3500. doi: 10. 10.1145/2998441. URL http: http:://axxiv.. arxiv. http://wxxii.....aaa/// 1501.0180
Junuthula Narasimha Reddy. An Introduction to Nonlinear Finite Element Analysis Second Edition: with applications to heat transfer. fluid mechanics, and solid mechanics. OUP Oxford, 2014.
I Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62(3):352-364, 2020.
Alvaro Gonzzlez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. learning to simulate complex physics with graph networks. International Conference on Machine Learning, pp. 8459 8469. PMLR, 2020.
Mikhail Shashkov and Stanly Steinberg. Conservative mriit-diiffeeece methods on general grids. CRC press, 2018
Zheng Shi, Nur Sila Gulgec, Albert S Berahas, Shamim N Pakzad, and Martin Takáč. Finite difference neural networks: Fast prediction of partial differential equations. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 130-135. IEBE, 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 575:1339-1364 2018.
Weinan and Bing Yu. The deep ritz method: deep bearning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statisties, 6(1):1-12, 2018.
Peter Wriggers. Nonlinear finite element methods. Springer Science & Business Media, 2008
Jiawei Zhuang, Dmitrii Kochkov, Yohai Bar-Sinai, Michael P Brenner, and Stephan Hoyer. Learned discretizations for passive scalar advection in two-dimensional turbulent flow. Physical Review Fluids, 6(6):064605, 2021.
A SPECTRAL METHODS SOLUTION TO NAVIER-STOKES EQUATION
Spectral methods solution the Navier-Stokes equation. In this section, review the spectral methods used to solve the 2d incompressible Navier-Stokes (NS) equation (5). In a 2D-fluid motion with (u, v, 0), the NS equations (equation equation 6.2 in the expanded form are
0,u + ud,.u u + vd,u = -gp +v22
0. + uāzv + vd, = 0 +v
(8)
d_u 8yy =0
(9)
We define stream function as -0,4, d= The incomprehensibility condition, equation 9 is immediately satisfied, In order to remove the pressure from the equations, we 0, (Eq.
0, (Eq.
to find
a +u. 2 vo 2 =
(10)
13

---

Under review as a conference paper at ICLR 2022
Note that this 3 02v 0₂u (V v) 2.
We can simplify the last vorticity equation,
equation 10 in terms of Ç 24 to find
d,c + (-0 () 22 +
0g V () 005 - -VV
(1I)
Next, discretize the equation,
0c5 = (0, (2,V~ - (0. 20 + - + ( V
(12)
gut - SH - - -00 -
(13)
öt
where used the implicit Crank-Nicolson for the linear part and explicit part for the linear part. Assuming periodic boundary conditions in both x and direction, and taking the Fourier transform of the above equation, find
1/ [ + 1k)) Eatl 5m =F(())
ôt
2 P(() ( (0.v~2))005- - (0.V 20)
(14)
vol 1k/2 gntl = =P((") (1- vōl
k|2 8m
(15)
2
gat1 P(s") + (1
- vôt k|
(16)
(1+ vāt 2 k2
where {= F() is the Fourier transform of the vorticity field, and P = F (g((")) where g((") (0, ² - (a, V a,, We use equation 16 propagate in time and to solve for the vorticity field. Given the vorticity field, can further calculate the horizontal and vertical velocity with u = -, --, v== V-
B PERIODIC BOUNDARY CONDITION
In order to impose the periodic boundary condition, we add connections to the boundary nodes that would result in periodic boundary condition. Particularly, consider similar copies of the nodes and by repeating the Delaunay triangulation for the nodes, we find how the boundary nodes are connected with the other the inside and boundary nodes (see Fig.
Figure 9: Schematic of nodes and edges for simulation with periodic boundary condition.
14


Reviews:
[{"review_id": "s8-bPypWgV", "reviewer": "Reviewer_neAM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well organized with intuitive explanations of the network design approach and detailed experiments. The strength of the paper is on adopting graph neural networks. However, based on my understanding, there are a few weaknesses (and questions that could help reviewers understand the paper better).\n\n1. As mentioned by the authors, the domain invariant feature representation is the major contribution and is inspired by PDE solvers. However, except for highlighting the approximation by using the neighboring points in the domain, it either lacks more in-depth analysis or lacks novelty as, for example, using finite difference etc, in a neural network for PDEs is a not unknown approach. Could you be more specific with more details on the novelty?\n\n2. Section 5 introduces the proposed network with nice illustrations. Is it just a typical architecture of message passing neural network with some fully connected layers. Could you please be more specific on the novelty on the structure? As authors mention in section 6.1 that different domains were tested to show the generalization capability, could you be more specific on which part of the network that would allow good generalization to be domain invariant? \n\n3. The experiment section is well written with detailed explanation and visualization. However, there are a few questions:\n\n3.1 What is the scale of the data, i.e. $u_i$? Without reference, I find it is hard to measure the claimed MSE that was achieved.\n\n3.2 What is the training algorithm, or in other words, the optimizer chosen for training the network? \n\n3.3 What is the baseline that the proposed method is comparing with? How about other networks that was mentioned in the related works? Lacking these two important reference makes it hard to measure the performance of the proposed method.\n\n3.4 While the result visualization is nicely presented, without detailed numerical presentation, I am not sure how well the performance is and how efficient the training is? \n\nSome missing references:\n\n[1] Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations\n\n[2] Solving high-dimensional partial differential equations using deep learning\n\n[3] Finite Difference Neural Networks: Fast Prediction of Partial Differential Equations", "summary_of_the_paper": "This paper proposes to use a message passing neural network to predict solutions of both linear and nonlinear PDEs. The proposed method uses domain invariant features with graph based method for representing the temporal data of PDE. ", "summary_of_the_review": "To summarize, this paper basically applies message passing neural networks to solving PDEs. I strongly recommend authors to present motivations and design of networks with more details and analysis, and the experiment requires improvement. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "oIoFuljQpS4", "reviewer": "Reviewer_kLFD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The method is described fairly clearly and is easy to understand: applying a GNN to learn solutions to PDEs.\n\n- The authors evaluate their approach on a number of test problems and show that it can generalize to unseen domains.\n\n- The approach also extends to time-resolved simulation.\n\n**Weaknesses**\n\n- To me it seems that the main contributions of this work have already been demonstrated. Pfaff et al. (ICLR 2021) have already demonstrated a GNN for solving a wide array of simulation and PDE problems. The architecture of Pfaff et al. is generally similar to the proposed approach, they demonstrate generalization to unseen domains, and the experimental results go beyond that what is shown in the proposed method (including 3D, adaptive remeshing, and dynamic simulation domains). The authors should cite this work and explain if or how their contributions differ.\n\nPfaff, Tobias, et al. \"Learning mesh-based simulation with graph networks.\" In Proc. ICLR 2021.\n\n- The proposed recurrent approach to solving time-dependent PDEs is not trained with a recurrent loss. Thus the authors note that the MSE increases with increased unrolling steps. This is an open challenge for learned PDE solvers. Usually it is difficult to incorporate an unrolled loss because of the significant memory required to backpropagate through the GNN for multiple time steps. Is it possible to train the proposed method with a recurrent loss, or is that computationally prohibitive?\n\n**Misc.**\n\n- The authors note that there is a periodic boundary condition for the Navier-Stokes equations they solve. Do they incorporate this into the mesh they use in any way (e.g., using a periodic mesh to help the network learn these boundary conditions)?\n", "summary_of_the_paper": "The paper presents a method for learning mesh-based simulation of partial differential equations using graph neural networks (GNNs). The authors train the GNN to match a reference solution and demonstrate that the trained GNN can generalize to solving PDEs on different domains. Additionally, the authors show a recurrent variant of the network that can solve time-dependent PDEs. The method is shown to work on a number of test problems, including advection-diffusion and Navier-Stokes equations. Overall, the authors show that GNN-based simulation is a promising approach for improving the flexibility and generalization of learned PDE solvers.\n", "summary_of_the_review": "While the paper seems generally well written and the method appears sound and thoroughly evaluated, I'm not sure what the novelty of the paper is given that the contributions overlap significantly with Pfaff et al. (2021). Moreover, the authors do not cite this paper or explain how their work differs from it. Thus, my rating is to reject the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "OcoRXVnui0s", "reviewer": "Reviewer_LoDm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThis paper introduces a locally invariant feature representation inspired by classical PDE solvers to efficiently learn a differential operator solver. The method can obtain robust high-performance models for various linear/nonlinear PDEs with arbitrary spatial discretization and arbitrary physical domains. \n\nWeaknesses\n\nSome detailed instructions and experiments need to be added, such as how to construct meshes, how to construct boundary conditions, and sensitivity analysis of prediction results to data sets. \n\nComparisons with other methods need to be added.\n\nSome examples in irregular computational domains can be added.\n\nI would love to hear some limitations of the proposed method.\n", "summary_of_the_paper": "This paper presents a graph neural network approach by using a message-passing model to find efficient PDE solvers. It further shows that the trained solver can find a temporal sequence of PDE solutions in different domains. The proposed method is a nice idea and has been well verified. However, given the very limited scope of application in this paper, only toy environments with high-resolution numerical data sets can be used, more detailed ablation and comparison experiments may be required to ablate the model in a more refined manner.", "summary_of_the_review": "This paper proposes a new method, which is innovative, but only shows some simple experiments and lacks detailed discussions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "KhAAHk6Dia8", "reviewer": "Reviewer_Eu7u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages\n\n- The submission until the experimental section is well-written, easy to follow, and very clear. The written-style could include more PDE fundamentals for the uninitiated reader, but I think on the whole the authors have chosen for a clearer and simpler-to-follow read.\n\n- The proposed GNN architecture makes a lot of sense. Using a GNN on the mesh is intuitive and the particular features used are also sensible. The particular from of edge embedding used also makes the GNN locally shift equivariant. Furthermore, encoding the PDE parameters in the edge features allows the model to generalise across PDEs within a given class of PDEs.\n\n- The experiments demonstrate that the MPGNN can learn to make 1-step PDE prediction on multiple geometries. This is evidences in Figures 2 and 3.\n\nQueries\n\n- Just under equation 2: The argument that F(u, \\nabla u, …) contains local spatial derivatives an only local neighbouring values and so you only need a graph neural network operating on local neighbourhoods makes sense. I was wondering, what are the limits on this assumption? Does the solution space have to be analytic/smooth? I’m guessing the maximum derivative order and accuracy of approximation will also influence the size of neighbourhoods you should take.\n\n- Section 5: do you only use 1-step neighbourhoods in your model? I could imagine longer range message passing is important in PDEs with higher-order derivatives.\n\n- Experiments: It is not obvious to me what is going on in your experimental setup. I would not be able to reproduce these experiments. Specifically:\n\n— Does the MPGNN output 1 time frame at a time, but it takes in 4? \n\n— Why are the 4 input frames separated by 20 \\delta t each? \n\n— How are the coordinates for the mesh chosen? \n\n— From which distribution do you sample initial conditions and boundary conditions?\n\n— Which optimiser do you use?\n\n— Is the loss a 1-step loss?\n\n— Do you predict values on the boundary or not?\n\n- If you run the learned solver in recurrent mode, the error appears to increase over time as in Figure 4. c). Have you checked to see what happens over long rollouts? Does the error increase gradually or blow-up exponentially? I think it would be important to consider the long term error behaviour, since this is very important for time-dependent PDE solvers.\n\n- Baselines: I do not see any baselines. Without these it is difficult to make sense of the number in the submission. Sensible baselines could have been finite difference method, finite volume method, spectral method, off-the-shelf software, or other neural methods, such as the Fourier Neural Operator, or the method of Bar-Sinai et al. (2019).\n\n- What is the connection between this work and “Learning to Simulate Complex Physics with Graph Networks” (Sanchez-Gonzalez, 2020)? Here, the authors also use MPGNNs to solve complexity physical systems, driven by PDEs.\n\nMinor notes\n\n- Just under equation 2: n is a superscript, not subscript\n\n- Heat equation: Please qualify what Firedrake is. Is is a software package? This is not indicated in the text.\n\n", "summary_of_the_paper": "A message passing graph neural network is used to predict the time-dependent solution to a family of PDEs. The PDE parameters, initial conditions, and boundary conditions are input to the network, an it outputs the solution 1 time-step into the future. The authors show how to train such a system on the Heat equation, Navier-Stokes, and advection-diffusion equations. The flexibility of the message passing graph neural network permits its deployment on a diverse set of geometries and boundary conditions.\n", "summary_of_the_review": "The method is well laid out and written with enough clarity that the setup is understandable and intuitive. The method makes sense given the problem and is a logical step forward in this space. The experiments, however, are insufficient to back up the claims laid forth in the first section of the paper. I do not think I could reimplement the experiments based on the writeup and I do not see a separate supplementary material, where I may have missed these details (please correct me if I have overlooked them). Furthermore there is no comparison with any baselines (I’ve mentioned a few sensible ones in the main review), neural or classical, so it is hard to say how significant the numbers in the submission are. There are also no ablations for the reader to understand the impact of each design choice the authors make. It is mainly because of the experimental section that I recommend a reject for this submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to ICLR.  While all reviewers felt that there were some interesting aspects to the proposed work, the consensus was also that the work didn't properly situate itself within the existing literature on related methods.  In particular, I agree with Reviewer kLFD that a numerical comparison to Pfaff et al., is notably missing here; while the authors did provide qualitative comparisons in their discussion, it's not clear to me that these differences are ultimately that significant, and the methods need to be compared directly if a case is to be made for the advantages of the proposed approach.
Decision: Reject