Learning continuous-time PDEs from sparse data with graph neural networks
Authors: Valerii Iakovlev
Markus Heinonen
Harri Lähdesmäki
Venue: ICLR 2021 Poster (2021)
Keywords: dynamical systems
partial differential equations
PDEs
graph neural networks
continuous time

Abstract: The behavior of many dynamical systems follow complex, yet still unknown partial differential equations (PDEs). While several machine learning methods have been proposed to learn PDEs directly from data, previous methods are limited to discrete-time approximations or make the limiting assumption of the observations arriving at regular grids. We propose a general continuous-time differential model for dynamical systems whose governing equations are parameterized by message passing graph neural networks. The model admits arbitrary space and time discretizations, which removes constraints on the locations of observation points and time intervals between the observations. The model is trained with continuous-time adjoint method enabling efficient neural PDE inference. We demonstrate the model's ability to work with unstructured grids, arbitrary time steps, and noisy observations. We compare our method with existing approaches on several well-known physical systems that involve first and higher-order PDEs with state-of-the-art predictive performance.

Published as a conference paper at ICLR 2021
LEARNING CONTINUOUS-TING PDES FROM SPARSE DATA WITH GRAPH NEURAL NETWORKS
Valerii lakovlev, Markus Heinonen & Harri Lähdesmäki
Department of Computer Science Aalto University
Helsinki, Finland
[valerii. I iakovlev, markus. faokkus.o.hiinneee harri
waw...aaleeeeeeeeeee.... lahdesmahd @aalto. fi
ABSTRACT
The behavior of many dynamical systems follow complex, yet still unknown partial differential equations (PDEs). While several machine learning methods have been proposed to learn PDEs directly from data, previous methods are limited to discrete- time approximations
make the limiting assumption of the observations arriving at regular grids. We propose a general continuous time differential model for dy- namical systems whose governing equations parameterized by message passing graph neural networks. The model admits arbitrary space and time discretizations, which removes constraints the locations of observation points and time inter- vals between the observations. The model is trained with continuous-timg adjoint method enabling efficient neural PDE inference. We demonstrate the model's s ability to work with unstructured grids, arbitrary time steps, and noisy observations. We compare our method with existing approaches on several well-known physical systems that involve first and higher-order PDEs with state-of-the- art predictive performance.
INTRODUCTION
We consider continuous dynamical systems with state u(x, f) E R that evolves over time f e R. and spatial locations x E Ω Rº of a bounded domain Q. We assume the system is governed by an unknown partial diffcrential equation (PDE)
ú(x, t) = du(x, 1) dt
F(x, u, V
(1)
where the temporal evolution à of the system depends on the current state and its spatial first and higher-order partial derivatives w.r.t. the coordinates x. Such PDE models are the cornerstone of natural sciences, and are widely applicable to modelling of propagative systems, such as behavior of sound waves, fluid dynamics, heat dissipation, weather patterns, disease progression or cellular kinetics (Courant & Hilbert 2008). Our objective is to learn the differential F from data.
There long history of manually deriving mechanistic PDE equations for specific systems (Cajori 1928), such as the Navier-Stokes fluid dynamics or the Schrödinger's quantum equations, and approx- imating their solution forward time numerically (Ames) 2014). These efforts are complemented by
data-driven approaches to infer any unknown or latent coefficients in the otherwise known equations
2017|Santo [Santo et al.) 2019), or partially known equations (Freund ct al. 2019 Seo & Liu 2019b Seo et al. 2020). A series of methods have studied neural proxies of
(Isakov 2006 Berg & Nyström 2017|
known PDEs for solution acceleration (Lagaris et al. 1998 Raissi et al. 2017: Weinan & Yu 2018
Sirignano & Spiliopoulos 2018) for uncertainty quantification (Khoo al. 2017).
Related work. Recently the pioneering work Long et al. (2017) proposed fully non-mechanistic
method PDE-Net, where the governing equation is learned from system snapshot observations as a
convolutional neural network (CNN) over the input domain discretised into apatio-temport grid.
Further works have extended the approach with residual CNNs (Ruthotto & Haber (2019), symbolic neural networks (Long et al. 2019), high-order autoregressive networks (Geneva & Zabaras 2020),

---

Published as a conference paper at ICLR 2021
and feed-forwart networks (Xu al. 2019). These models fundamentally limited discretizing grid, while they also do not support continuous evolution
the input domain with a sample-ineffiiiee over time, rendering them unable to handle temporally spatially sparse non-uniform observations commonly encountered in realistic applications.
Models such (Battaglia al.) 2016 Chang et al. 2016 Sanchez-Gonzaler al. 2018) are related to the interaction networks where object's 's state evolves as function its neighboring objects, which forms dynamic relational graphs instead of grids. contrast the dense solution fields of PDEs, these models apply message-passing between small number of moving and interacting objects, which deviates from PDEs that are strictly differential functions.
In Poli ct al. 2019) graph neural ordinary differential equations (GNODE) were proposed as framework for modeling continuous-timg signals on graphs. The main limitations of this framework in application to learning PDEs are the lack of spatial information about physical node locations and lack of motivation for why this type of model could be suitable. Our work can be viewed as connecting graph-based continuous-time models with data-driven learning of PDEs in spatial domain through a classical PDE solution technique.
Contributions. In this paper propose to learn free-form, continuous- a priori fully unknown PDE model F from sparse data measured on arbitrary timepoints and locations of the coordinate domain Ω with graph neural networks (GNN). Our contributions are:
We introduce continuous-time representation and learning of the dynamics of PDE-driven systems
We propose efficient graph representation of the domain structure using the method of lines with message passing neural networks
We achieve state-of-the-ar learning performance on realistic PDE systems with irregular data, and our model highly robust data sparsity
Scripts and data for reproducing the experiments can be found in this github repository
Table 1: Comparison of machine- learning based PDE learning methods.
Unknown PDE learning
Continuous time
Free-form
Free-form
Model
spatial domain
initial/houndary conditions
Reference
PINN
x
Raissi al.
2017
AR
x
Geneva
PDE-net
Long
DPM
X
/
X
V
Freund al.
2019
DPGN
X
Seo & Liu
2019b
PA-DGN
X
/
Seo cf al.
2020
Ours
/
/
V
2 METHODS
In this Section we consider the problem of learning the unknown function F' from observations (y(to)...., E (MM+1) of the system's state u(t) ()) u u)),,t))² at N arbitrary spatial locations . and M time points .......ts! We introduce efficient graph convolution neural networks surrogates operating over continuous-time to learn PDEs from sparse data. Note that while we consider arbitrarily sampled spatial locations and time points, do not consider the case of partially observed vectors yt+)) ie. when data at missing at some time point. Partially observed vectors, however, could be accounted by masking the nodes with missing observations when calculating the loss. The function F is assumed to not depend on global values of the spatial coordinates i.e. we assume the system does not contain position-dependent fields (Section 2.1
location is
We apply the method of lines (MOL) Schiesserr 2012) to numerically solve Equation The MOL consists of selecting N nodes in Ω and discretizing spatial derivatives in F at these nodes. We
2

---

Published as a conference paper at ICLR 2021
place the nodes to the observation locations (x.....,N)) xN). The discretization leads to F being approximated by F and produces the following system of ordinary differential equations (ODEs)
whose solution asymptotically approximates the solution of Equation !
i1 (t)
du(x).)) de
F(x1, XN(I)) Un, (A(())
útt)
ERR.
(t)
Au(xy.))
F(xx, uN. uN(N))
di
As the discretized I' inherits its unknown nature from the true PDE function F, we approximate F by learnable neural surrogate function.
The system's 's state at x; is defined as ui, while (i) is a set of indices of neighboring nodes other than that are required to evaluate at and Xx(i) with uV(() are positions and states of nodes (t).. This shows that the temporal derivative ú; of depends not only on the location and state the node but also on locations and states of neighboring nodes, resulting in locally coupled system of ODEs.
Each ODE in the system follows the solution at fixed location x Numerous ODE solvers have been proposed (such as Euler and Runge- solvers) to solve the full system
u(t) u(0) + L. ú(T)dT,
(3)
where 0 < T S L is cumulative intermediate time variable. Solving equation 3 forward in time scales linearly both with respect to the number of nodes N and the number of evaluated time points M. while saturating the input space ft requires large number nodes. In practice, PDEs are often applied for two- and dinessoamenio spatial systems where the method is efficient.
2.1
INVARIAN GRAPH NEURAL NETWORK DIFFERENTIAL
After introducing Equation 2 we transition from learning F to learning P. The value of i at node i must depend only on the nodes and (i). Furthermore, the number of arguments and their order in F is not known in advance and might be different for each node. This means that model F must be able to work with an arbitrary number of arguments and must be invariant to permutations of their order. Graph neural networks (GNNs) (Wu et al. 2020) satisfy these requirements. In more restricted setting, where the number of neighbors and their order is known, (c.g. if the grid is uniform) other types models such as multilayer perceptrons and convolutional neural networks can be used as well.
We consider type of GNNs called message passing neural networks (MPNNs) Gilmer al. 2017 represent F as
Fo(xx()) X,, u4 "N()),
(4)
where XV(i) - x, - (x, X; E and 0 denote parameters of the MPNN.
This formulation assumes the absence of position-dependent quan- tities in f. but models based on this formulation are invariant to translations and rotations S2 which makes generalization to sys- tems with different node positions feasible, and prevents overfitting by memorizing caeition-specfiic dynamics.
We use MPNNs, which is type of spatial-based GNNs, due to their flexibility and computational efficiency. The main alternative spectral-based GNNs - have relatively poor scaling with the number of nodes and learn global, or domain-dependent, filters due to the need to perform eigenvalue decomposition of the Laplacian matrix.
Ω
Figure 1: Delaumay triangula- tion for and orange points are consid- ered to be neighbors as they share the same edge.
points. Green
2.2 MESSAGE PASSING NEURAL NETWORKS
Let graph G = (V, E) contain nodes = N defined by the measurement positions, and undirected edges P. = {eg)) and

---

Published as a conference paper at ICLR 2021
assume each node and edge associated with node feature v; and edge feature etj. respectively. We use the node neighborhood (i) to define edges. Neighbors for each node were selected by applying Delaunay triangulation to the measurement positions. Two nodes were considered to be neighbors if they lie on the same edge of at least one triangle (Figure D Delaunay triangulation has such useful properties as maximizing the minimum angle within each triangle in the triangulation and containing the nearest neighbor of each node which helps to obtain good quality discretization of S2.
In message passing graph neural networks we propagate latent state for K Z 1 graph layers, where cach layer kt consists of first aggregating messages m (k) for each node i, and then updating the corresponding node states h (8)
(k+1)
(((b)
(k) (h. h eis) ,es),
(5)
JEN(1)
h (k+1) 7(0 h m ((+tt))
(6)
where @ denotes permutation invariant aggregation function (e.g. sum, mean, max), and (i(), (a) are differentiable functions parameterized by deep neural networks. At any time т, we initialise the
latent states h (0)
(T) and node features to the current state uz(T) of the system. We define edge features e;j == X, x, as location differences. Finally, we use the node states at the last graph
layer of the MPNN to evaluate the PDE surrogate
du du(x;;t) t) = Fo(xN()) - X,, us uN(() = h::" -((K)
dt
which is used to solve Equation for the estimated states û(t) (ū(x), t)....,................ ((xx,,))...,,(xxx:
2.3 ADIOINT METHOD FOR LEARNING CONTINUOUS TIME MPNN SURROGATES
Parameters of Fg are defined by 0 which is the union of parameters of functions (i(*), 7(0), k 1...., in the MPNN. We fit 9 by minimizing the mean squared error between the observed states .... and the estimated states (ū(tu), ..... ú(tas))
aM [
ta
((0)
E(t, ú)dt
((((ts) y(4))lB5(t tt)dt
M
Σ -((t))
While discrete-time neural PDE models evaluate the system state only at measurement time points, more accurate continuous-time solution for the estimated state generally requires many more evalua- tions of the system state. If an adaptive solver is used to obtain the estimated states, the number of time steps performed by the solver might be significantly larger than M. The amount of memory required to evaluate the gradient of (()) by backpropagation scales linearly with the number of solver time steps. This typically makes backpropagation infeasible due to large memory requirements. We use alternative approach, which allows computing the gradient for memory cost, which is independent from the number of the solver time steps. The approach was presented in Chen et al. (2018) for neural ODEs and is based the adjoint method (Pontryagin, 2018). The adjoint method consists of single forward ODE pass 3 until state ú(!M) ) at the final time (., and subsequent backward ODE pass solving the gradients. The backward pass performed by first solving the adjoint equation
Oe 2(1)T oF Dū(t)
A(1)T
(10)
Oū(t)*
for the adjoint variables À from = ts until = 0 with A(tas) = 0, and then computing
dC
T
A(t) x(t)rāF
oF dt
(11)
d0
00
to obtain the final gradient.
4

---

Published as a conference paper at ICLR 2021
0.2
0.4 sec
0.8 sec
0.125
nodes
Data
0.100
Relalive Error
750 nodes
0.075
0.080
3000 nodes
0.025
0.000
750
0.0
0.2
0.4
0.6
nodes
(99c)
(a)
(b)
Figure a) Relative emrors for different grid sizes. b) Visualization of the true and learned system dynamics (grids are shown in the first column).
3 EXPERIMENTS
We evaluate our model's performance in learning the dynamics of known physical systems. We compare state-of-the-att competing methods, and begin by performing ablation studies to measure how our model's performance depends on measurement grid sizes, interval between observations, irregular sampling, amount of data and amount of noise.
3.1 CONVECTION-DIFFEE DIFFUSION ABLATION STUDIES
The convection-diffusion equation a partial differential equation that can be used to model variety of physical phenomena related to the transfer of particles, energy, and other physical quantities inside a physical system. The transfer occurs due to two processes: convection and diffusion. The convection-diffusion equation is defined
aulæ, y, DVZu(z,
Vu(a, y, t),
(12)
aL
where u is the concentration of some quantity of interest (full problem specification and setup are in Appendix(A) A Quality of the model's predictions evaluated using the relative error between the observed states y(4) and the estimated states ú(t;):
(t,(t) a(+))l [ty(+)]
Err
(13)
In all following experiments, unless otherwise stated, the training data contains 24 simulations the time interval [0, 0.2) sec and the test data contains 50 simulations on the time interval [0, 0.6) sec. The data is randomly downsampled from high fidelity simulations, thus all train and test simulations have different node positions while the number of nodes remains constant. Examples from the train and test are shown in Figure
Different grid sizes. This experiment tests our model's capability to learn from data with different density of observation points. The time step was set to 0.02 see resulting in training time points simulation. The number of observation points x; (and consequently nodes in the GNN) was set to 3000, 1500 and 750. The resulting grids are shown in the first column Figure 2b) Figure shows relative test enrors and models' predictions.
The performance of the model decreases with the number nodes in the grid. Nonetheless, even with the smallest grid, the model was able to learn reasonably accurate approximation of the system's dynamics and generalize beyond the training time interval.
Different measurement time interval. As will be shown in the following experiments, models with constant time step are sensitive to the length the time interval between observations. While showing good performance when the time step is small, such models fail generalize if the time step is increased. This experiment shows our model's ability to learn from data with relatively large time intervals between observations.
5

---

Published as a conference paper at ICLR 2021
0.2 sec
sec
0.8 sec
11 points points points
Relalive Error
0.10
6ime points
0.01
2 6ime points
0.0
0.2
0.4
0.6
(69c)
(a)
(b)
Figure 3: a) Relative errors for different time grids. Visualization of the and learned system dynamics (grids are shown in the first column).
We used 11, 4 and 2 evenly spaced time points for training. The number nodes was set to 3000. Figure shows relative test errors and models' predictions. The model is able to recover the continuous-time dynamics of the system even when trained with four time point per simulation. Increasing the frequency of observation does not significantly improve the performance. An example of a training simulation with four time points is shown Figure
Irregular time step. Observations used for training might not be recorded with a constant time step. This might cause trouble for models that are built with this assumption. This experiment tests our model's s ability to learn from data observed at random points time.
The model is trained on two time grids. The first time grid has a constant time step 0.02 sec. The second grid is the same the first one but with each time point perturbed by noise e ~ NO, (0.02 )²). This gives time grid with an irregular time step. The time step for test data
0.02
Relative Error
0.01
set to 0.01 sec. The number nodes was set to 3000. Relative test errors shown in Figure 4 In both cases the model achieves similar performance. This demonstrates the continuous-time nature of model as training and predictions are not restricted to evenly spaced time grids as with most other methods. None of the previous methods that learn free form (i.e., neural network parameterised) PDEs can be trained with data that is sampled irregularly over time.
Regular Irregular
0.00
0.2
0.4
(sac)
Figure Relative test errors for regular and irregular time grids.
Different amount of data. In this experiment, the model is trained test data contains 50 simulations. The time step was set 0.01 sec. The number of nodes was set to 3000. Rela- tive test errors are shown in Figure Performance of the model improves the amount training data increases. It should be noted that despite using more data, the relative error does not converge to zero.
5, 10 and 24 simulations. The
24 sim.
Relative Eroor
0.04
10 sim.
sim.
0.02
0.0
0.2
0.4
Varying amount of additive noise. We apply additive noise e ~ (0, o²) to training data with or set to 0.01, 0.02, and 0.04 while the largest magnitude of the observed states is The time step was set to 0.01 sec. The number of nodes
1 (sec)
Figure 5: Relative test errors for differ- ent amounts of training data.
to 3000. Noise was added only the training data. The relative test errors are shown in Figure 6 grows but even at 0.04 remains quite high.
The model's performance decreases as

---

Published as a conference paper at ICLR 2021
3.2 BENCHMARK METHOD COMPARISON
The proposed model compared sented in the literature: PDE-Net Long et al. 2017) and DPGN (Seo& & Liu 2019ap. PDE-Net is based volutional neural network and employs a constant time- stepping scheme resembling the Euler method. DPGN is based on a graph neural network and implements time- stepping as an evolution map in the latent space.
models pre-
0.04
Relative Error
0.02
No noise
0=0.02
0=0.01
a=0.04
We used the PDE-Net implementation provided in Long al. (017) except that we pass filter values through an MLP consisting of 2 hidden layers 60 neurons each and tanh nonlinearities which helps to improve stability and performance of the model. We use 5 X 5 and 3 x 3 filters without moment constraints and maximum PDE order set to 4 and 2 respectively. The number of öt-blocks was set to the number of time steps in the training data. Our implementation of DPGN followed that from Seo & Liu (2019a) with latent diffusivity a 0.001. The number of parameters in all models was close to 20k.
0.00
0.0
0.2
0.4
0.6
(sec)
Figure 6: Relative test errors for differ- ent amounts of noise in the training data.
The training data contains 24 simulations the time interval [0, 0.2] sec with the following time steps: 0.01, 0.02 and 0.04 sec. The test data contains 50 simulations on the time interval [0, 0.6] sec with the same time steps. The data was generated on 50 50 regular grid as PDE- net cannot be applied to arbitrary spatial grids. Separate models were trained for each time step. The performance of the models was evaluated using the mean of relative test error averaged over time.
10
PDE-Net (3x3)
Moan Rolative Error
10"
PDE-Net (5x5)
DPGN
10
Ours
10'
10
Mean relative test errors of the models are shown in Figure 7 The figure shows that performance of the discrete- time models is strongly dependent on the time step while performance of the continuous time model remains at the same level. At the smallest timestep, PDE-Net with 5 5 filters outperforms other models due to having access to a larger neighborhood of nodes which allows the model to make more accurate predictions. However, larger filter size does not improve stability.
10
0.01
0.02 Timestep (sec)
0.04
Figure 7: Mean relative errors of models trained with different time steps.
We note that some discrete-time models, e.g. DPGN, could be modified incorporate the time step as their input. Comparison with this type of models would be redundant since Figure already demonstrates the best case performance for such models (when trained and tested with constant time step).
Importance of relative positional information. We test our model with and without relative node positions that are encoded as the edge features in our MPNN on grids with different number of nodes. Smaller number of nodes results in higher distance variability between neighboring nodes (Figure T2p which should increase the dependence of the model accuracy on the relative spatial information. By removing spatial information from our model, we recover GNODE. The models were tested and convection-diffsiin equations.
Mean Relative Error
0.15
Heat eq. (ours)
0.10
Heat (GNODE)
Conv. diff, eq, (ours) Conv. diff. eq. (GNODE)
0.05
the heat (Appendix A full description of the experiment is in Appendix D The results are shown in Figure g
0.00
25%
75% Percentage of Nodes
50%
100%
Figure 8: Mean relative test errors of models with and without relative node positions.
Surprisingly, GNODE shows good results the purely diffusive heat equation. Nonetheless, the performance of GNODE noticeably differs from that of model that
7

---

Published as a conference paper at ICLR 2021
0 sec
0.1 sec
0.2 sec
0.3 sec
Relalive Error
0.010
Data
0.005
Pred.
0.0
0.1
0.2
0.3
((sec)
(a)
(b)
Figure 9: a) Relative test errors for heat equation. b) True and learned system dynamics.
0 sec
0.8 sec
1.6 sec
2.4 sec
0.15
Relative Eiror
0.10
Data
0.05
0.00
Pred.
0.0
0.5
1.0
1.5
2.0
2.5
(sec)
(a)
(b)
Figure 10: a) Relative test errors for Burgers' equations. b) True and learned system dynamics.
includes the spatial information. Furthermore, the performance difference almost doubles as the number of nodes is decreased from 100% to 50%.
When applied the -onfection-diffusion equation, GNODE fail to learn the dynamics irrespective of the number of nodes. This can be explained by the presence of the convective term which transports the field specific direction thus making positional information particularly important for accurately predicting changes in the field.
3.3 OTHER DYNAMICAL SYSTEMS
The model was tested on two more dynamical systems order evaluate its ability to work with wider range of problems. We selected the heat equation and the Burgers' equations for that purpose. The heat equation one of the simplest PDEs while the Burgers' equations are more complex than the convection -nfectioonniffusion equation due to the presence of nonlinear convective terms. The increase in the problems' difficulty allows trace the change in the model's performance as we move from simpler to more complex dynamics while keeping the number of model parameters fixed.
Heat equation. The heat equation describes the behavior of diffusive systems. The equation is = DV"u, where u is the temperature field (see Appendix B for details). Figure shows relative errors and model predictions for random test The heat equation describes simpler dynamics than the convection diffusion equation which allowed the model to achieve slightly smaller test emors.
defined as 8u 3
Burgers' equations. The Burgers' equations is system of two coupled nonlinear PDEs. It describes the behavior of dissipative systems with nonlinear propagation effects. The equations
defined in vector form du(2.y4) DVZ v, u(z, y, Vu(I, y, y,t), where is the velocity vector field (see Appendix CC for details). For visualization and error measurement purposes, the velocity vector field converted to scalar field defined by the velocity magnitude each node. Figure 10 shows relative errors and model predictions for a random test case.
The Burgers' equations describe more complex dynamics than the previous
which is reflected in higher relative test errors. Visual comparison of the true and predicted states shows that the model able achieve sufficient accuracy at approximating the unknown dynamics.
8

---

Published as a conference paper at ICLR 2021
4 CONCLUSION
We present continuous-timg model of dynamical systems whose behavior is governed by PDEs. The model accurately recovers the system's 's dynamics even when observation points are sparse and the data is recorded at irregular time intervals. Comparison with discrete-timg models reveals the advantage of continuous-tiin
models for datasets with larger time intervals between observations, which is typical for real-world applications where measurements can be either tedious costly, or both. Discretization of the coordinate domain with the method of lines provides general modeling
framework which arbitrary surrogate functions can be used for approximating The continuous- time nature of the model enables the use of various time integrators ranging from the Euler method to highly accurate adaptive methods. This allows to optimize the choice of the surrogate function and time integration scheme depending on the structure of the data.
REFERENCES
William Ames. Numerical methods for partial differential equations. Academic press, 2014.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp.
2016.
Jens Berg and Kaj Nyström. Neural network augmented inverse problems for pdes. arXiv preprint 17X2::1712.09685. 2017.
Florian Cajori. The early history of partial differential equations and integration. The American Mathematical Monthly, 35(9):459-455 1928.
Michael B Chang. Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.0034/ 2016.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary
differential equations. Advances in Neural Information Processing Systems, 2018.
Richard Courant and David Hilbert. Methods of Mathematical Physics: Partial Differential Equations.
John Wiley & Sons, 2008
Jonathan B Freund, Jonathan MacArt, and Justin Sirignano. Dpm: A deep learning pde aug- mentation method (with application to large-eddy simulation). arXiv preprint 1...4.591..09145. 2019.
Nicholas Geneva and Nicholas Zabaras. Modeling the dynamics of pde systems with physics- constrained deep auto-regressive networks. Journal of Computational Physics, 403:109056, 2020.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR, abs/1 704.01222, 2017. URL http://arxiv
arg/abs/ / 1704. 01212
Victor Isakov. Inverse problems for partial differential equations, volume 127. Springer, 2006.
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural networks. arXiv preprint arXiv: / 7707.0355, 2017.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks. 95)09987-1000 1998.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. arXiv preprint arXiv: 11.0vv:/11000966 2017.
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. Journal of Computational Physics, 399: 109:108925, 2019.

---

Published as a conference paper at ICLR 2021
Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532. 2019.
L.S. Pontryagin. Mathematical Theory of Optimal Processes. CRC Press, 2018. 9781351433068.
ISBN
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.1056. 2017.
Martin Riedmiller and Heinrich Braun. Rprop a fast adaptive learning algorithm. Technical report, Proc. of ISCIS VII, Universitat, 1992.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pp. 1-13, 2019.
Alvaro Sanchez-Gonzalle Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arXiv preprint arXiv: arXiv:/806.00 /806.0/24, 2018.
Niccolò Dal Santo, Simone Deparis, and Luca Pegolotti. Data driven approximation of parametrized pdes by reduced basis and neural networks, arXiv preprint arXivv1904.001 2019.
W.E. Schiesser. The Numerical Method of Lines: Integration of Partial Differential Equations. Elsevier Science, 2012. ISBN 9780128015513. BLLLtttpss//oooks.ooollee id=2YDNCgAAQBAJ
Sungyong Seo and Yan Liu. Differentiable physic----cicc-c--mneeee graph networks. CoRR, abs/1902.02950. 2019a. htthttp: / arwaa.ivvv... www.vvorg//bs//.o000.. 02950
Sungyong Seo and Yan Liu. Differentiable physics-informed graph networks. arXiv preprint 1xXi::1902.02900. 2019b.
Sungyong Seo, Chuizheng Meng, and Yan Liu. Physics-aware difference graph networks for sparsely- observed dynamics. In International Conference on Learning Representations, 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339 115,1139-136,, 1364, 2018.
E Weinan and Bing Yu. The deep ritz method: deep learning-base based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6 6(1):1112, 2018.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey graph neural networks. IEEE Transactions Neural Networks and Learning Systems, 2020.
Hao Xu, Haibin Chang, and Dongxiao Zhang. DI-pde: Deep-learning based data-driven discovery of partial differential equations from discrete and noisy data. arXiv preprint arXiv:1908.0m 044663 2019.
A CONVECTION-DIFFUSII ABLATION STUDIES
The cnurrcctio-ddifusiion equation is partial differential equation that be used to model variety of physical phenomena related to the transfer of particles, energy, and other physical quantities inside a physical system. The transfer occurs due two processes: convection and diffusion.
Training and testing data was obtained by solving the following initial-boundary value problem on 0, 2Tl
[0, 2T] with periodic boundary conditions:
au(x, y.t) Ø1
DVZ uV² f) - v Vu(z, f),
(x, E Ω, t 0, * E [0, 2m), 12 0, y E [0,27 2T., 1 12 0, (x, E 9
u(z, 0, = u(x, 2Tr 1); u(00, y, I) uu2T,, y, 1),
(14)
u(z, y, 0) = u0 y),
10

---

where N = 10 and Aw, 7he ~ (0, The generated data contains N, simulations. Each simulation contains values of u(x, y, at time points (f1,..., (lr) and locations (xI,..., .NN). where x,, (K,,, Vn). Numerical solutions that represent the true dynamics were obtained using the backward Published as a conference paper at ICLR 2021
where the diffusion coefficient D was set to 0.25 and the velocity field set to (5.0, 2.0)*. The initial conditions uo(2; y) were generated as follows:
uo (2, y) =
ARt cos (A: + 1y) + Tku sin (ka + ty)
(15)
k,1=-N
uz y) min (2,
(2, y)
(16)
max (2 v) min (o(
where = 4 and ARt, 7/e ~ (O, 1). The generated data contains N, simulations. Each simulation contains values of u(r y, time points (41,-.. -....--,a))) ((r) (In: Vn). Numerical solutions that represent the true dynamics were obtained using the backward Euler solver with the time step of 0.0002 seconds and testing data used in the following experiments is downsampled from these solutions. Quality of
and locations (X1,.... xNN), where X,,
computational grid with 4100 nodes. Training
the model's predictions was evaluated using the relative error between the observed states (t,) and the estimated states ú(t;):
[y(4)) - 6(4))l
Err
(17)
The model used for all following experiments contains single graph layer. The mean selected as the aggregation function. Functions (u,,)) and 7(2) (u;, U u4 Xj x;) were represented by multilayer perceptrons with 3 hidden layers and hyperbolic tangent activation functions. Input/output sizes for gf(1) and (2) were set to 4/40 and 41/1 respectively. The number of hidden neurons was set to 60. This gives approximately 20k trainable parameters.
We followed the implementation of the adjoint method and ODE solvers from torchdi Python package (Chen et al. 2018). In all following experiments. adaptive-order implicit Adams solver was used with rtol and atol set to 1.0-10-7 Rprop (Riedmiller & Braun 1992) optimizer was used with learning rate set to 1.0 10 10-6 and batch size set to 24.
B HEAT EQUATION EXPERIMENT
Training and testing data was obtained by solving the following initial-boundary value problem on = (0, 1) x (0, with Dirichlet boundary conditions:
Ou(a, y, t) Öt
DVVuuu,vV
(z,v) Q t
(18)
u(z y, = un (x, y),
(x, 02, t 0.
u(x, y. 0) = un (I,y),
(z, E Zt
where 02 denotes the boundaries of Ω and diffusion coefficient D was set to 0.2. The initial conditions uo(x, y) were generated as follows:
in ūo(2,)) y) = Σ
Aut cos (Aur + Iy) + Yku sin (kr + ly)
(19)
k.l=-N N
uo(2 y) min uo
u0 v) max uo (x, 3) min (o(x,))
(20)
Euler solver with the time step 0.0001 seconds on computational grid with 4100 nodes. Training and testing data used in the experiments with the heat equation is downsampled from these solutions.
The model used for all experiments with the heat equation contains single graph layer. The mean was selected the aggregation function. Functions (1 (u,,-) and ((1) uj - u4 X, were represented by multilayer perceptrons with 3 hidden layers and hyperbolic tangent activation functions. Input/output sizes for ((1) and (1) were set to 4/40 and 41/1 respectively. The number of 60. This gives approximately 20k trainable parameters.
hidden neurons
11

---

Published as a conference paper at ICLR 2021
We followed the implementation of the adjoint method and ODE solvers from torchdiffes Python package (Chen et al.] 2018). In all following experiments, adaptive-order implicit Adams solver used with rtol and at set to 1.0 10 Rprop (Riedmiller & Braun 1992 optimizer was used with learning rate set to 1.0 10-6 and batch size set to 24.
In the experiment, the training data contains 24 simulations the time interval [o, 0.1] with time step 0.005 sec resulting in 21 time point. The test data contains 50 simulations on the time interval [0, 0.3] sec with the same time step. The number of observation points x; was set to 4100.
C BURGERS' EQUATIONS EXPERIMENT
Training and testing data was obtained by solving the following initial-boundary value problem on = [0, 2T] x [0, 2T] with periodie boundary conditions:
@u(x,y y. !) Øt
y. Vu(z,v,t), (2, E Q t 0,
u(r, 0, u(x, 2T,, t),
t 0, (21)
(0, y, 1) = u(2 yı t),
120
u( y, 0) = un( v),
(x, E Ω2 0,
where the diffusion coefficient D was set to 0.15. The unknown function now rector-valued Therefore, the initial conditions (2, y) for each component were generated as follows:
uo 1) =
Σ
Auz cos (kr + (y) + The sin (kæ + Iy)
(22)
h.l=-N
(z v) - min uo 3) max üo (z. y) min uo (z,
uo 2) 6
0.5
(23)
where = 2 and Aus, Tkk ~ (0, 1). The generated data contains N, simulations. Each simulation contains values of u1x; y, at time points (f1, tas) and locations (X1,... ;.y) XN), where x,, (In, Mn). Numerical solutions that represent the true dynamics were obtained using the backward Euler solver with the time step of 0.0016 seconds and testing data used in the experiments with the heat equation is downsampled from these solutions.
computational grid with 5446 nodes. Training
The model used for all experiments with the Burgers' equations contains single graph layer. The mean was selected the aggregation function. Functions (1 (1,,)) ) and (0,,,, uj 14 Xy -x--xx x;) were represented by multilayer perceptrons with 3 hidden layers and hyperbolic tangent activation functions. Input/output sizes for ((1) and (1) were set to 6/40 and 41/2 respectively. The number of hidden neurons set to 60. This gives approximately 20k trainable parameters.
We followed the implementation of the adjoint method and ODE solvers from forchdiffed Python package (Chen et al. 2018). In all following experiments. adaptive-order implicit Adams solver was used with rtol and atol 1oooloo set to 1.0 1.0110- 10- Rprop (Riedmiller & Braun 1992) optimizer
was used with learning rate set to 1.0 10 10-6 and batch size set 24.
In the experiment, the training data contains 24 simulations on the time interval [o, 0.8) sec with time step 0.04 see resulting in 21 time point. The test data contains 50 simulations the time interval [0, 2.4] sec with the same time step. The number of observation points x, was set to 5000.
D RELATIVE POSITIONAL INFORMATION EXPERIMENT
Data generation, time intervals, models and hyper parameters for this experiment are described in Appendix B for the heat equation, and Appendix A and Section 3.1 for the convection diffusion
equation.
For the heat equation, 100% of nodes corresponds to 1000 nodes while for the convectionddiffsiii equation it corresponds to 3000 nodes. The number of training time points was set to 21 in both cases.
12

---

Published as a conference paper at ICLR 2021
E
EXTRA FIGURES
sec
0.0667 sec
0.1333 sec
0.2 8ec
Figure 11: Differences between observations in train case with 4 time points.
100000
80000
42000
20000
800
Node distence
(a)
50000
30000
20000
6.0
0.10
0.15
0.22
022
Noce distance
(b)
25000
30000
12000
800
6.66
6:10
0.15
0.23
0.8
(c)
Figure 12: Relative node distances for graphs with different number of nodes. a) 1000 nodes, b) 750 nodes, c) 500 nodes.
13

---

Published as a conference paper at ICLR 2021
sec
0.035 sec
0.07 sec
0.1 sec
sec
0.1 sec
0.2 sec
0.3 sec
(b)
Figure 13: Snapshots of train (a) and test (b) simulations for the heat equation.
sec
0.07 sec
0:13 sec
0.2 sec
sec
0.2 98c
0.4 sec
0.6 sec
(a)
(b)
Figure 14: Snapshots of train (a) and test (b) simulations for the convection-diflusion equation.
14

---

Published as a conference paper at ICLR 2021
5ec
0.27
0.53 sec
0.8 sec
5e0
0.8 sec
1.6 sec
2.4 sec
Figure 15: Snapshots of train (a) and test (b) simulations for the Burgers' equations.
F APPLYING TRAINED MODELS TO GRIDS OF DIFFERENT SIZES
Figure 2b shows grids with different numbers of nodes. The grid with 3000 nodes nodes contains neighborhoods of similar shapes and sizes while neighborhoods in the grid with 750 nodes differ in shapes and sizes over a much larger range. This suggests that models trained on the grid with 750 nodes would work reasonably well on grids with 1500 and 3000 nodes, but not vice versa. We demonstrate this in the table below. The data and models used for this experiments are the same as in Section 3.1
Table 2: Mean relative errors of models trained on some grid and applied to other grids.
Model
3000
1500
750
Grid size
3000
0.013 + 0.001
0.017 + 0.001
0.043 + 0.004
1500
0.050 + 0.005
0.032 = 0.001
0.036
0.001
750
0.142 + 0.034
0.086 0.004
0.073 + 0.004
The model trained on 3000 nodes generalizes poorly to coarser grids while the model trained on 750 grids performs fairly well on all grids. The model trained on 750 nodes performs better on test data with 3000 and 1500 nodes than with 750 nodes. This is because the finer grid allows to make more accurate predictions, therefore the error does not grow as large as for the coarse grid with 750 nodes.
15


Reviews:
[{"review_id": "i0rq250_F3S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review: \n\nThis papers proposes an algorithm to learn a model for spatio temporal data assumed to be described by a stationary spatio temporal PDE.\n\nData considered in the paper consists of vectors y(t) of observations at time t and a fixed set of spatial indices x.\nA model for discrete vector y(t) is proposed in the form of coupled ODEs (one for each x_i) with a sparse coupling arising from a neighbouring graph on spatial inputs x, and sharing the same transition function. \nThis transition acts on relative local spatial information and on the absolute function values in a neighbourhood.\n\nThe continuous time ODE can be solved using classing ODE solver. The model is fit to data using the adjoint method to backprop through the solution given a squared loss.\nThe ability of the model to learn is evaluated on data generated from PDEs.\n\n+ves: \n\n+ The proposed model is sensible and the paper motivates and describes the method well.\nThe obtained sparsity of the ODE coupling makes the method scalable. Preserving a continuous time dimension is useful \nif data is irregularly sampled\n\n+ Once learned, the model is quite flexible -> adapts to new grids, adapts to various time intervals\n \nConcerns: \n\n- The paper does not describe well the setting in which it is applicable. Spatio temporal data could be observations at random space and time locations. One has to read all the details to understand this while it should be highlighed.\nFor example, the data arrives in vectors of observations at input (x, t) for different t.\nThe method would not be able to deal with missing data (a partial vector y(t)) etc.. , the method assumes stationarity.\n\n- The methods cite other approaches that take a different route to the problem (e.g. for example learning the parameters of PDE directly) such approaches have their scaling issues but inherit from half a century of research on approximate solutions to pdes, and come with guarantees. Does this method come with any guarantees? A discussion on this aspect would be useful.\n\n\n- For infinitesimal time steps, it makes sense to use only local information to build the differential F.\nBut for longer time steps (think diffusion) you would need information that span further away to get accurate results.\nAlong these lines, how would a model trained on fine time grid deteriorate as you test it on data with bigger time steps\n\n- I m surprised of the experiment about removing the positional information. Removing it makes no sense when one has stationary PDEs in mind. Same with the noise: if you add noise, performance decreases.\n\n\n\nOverall, I like the method and think it can be very useful to members of the community,  but I find the paper lack a broader perspective when describing it.\nThe PDE connection could be used to discuss intuition on where it works or fails much more and to guide\nthe model experiments and validation.\nInstead, the current writing makes it sound a black box engineering solution has been proposed and is\n tested with no real guiding principle.\n\nFor this reason, I am not a big proponent of the paper but do not oppose acceptance.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TF_ZVnO4vwR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes extensions of PDE-net that relax some constraints that could help extend the range of applications of this approach. First, rather than fixing a spatial discretization in the form of a grid, the authors use a Delaunay triangulation to represent the domain. The updates to the nodes of this triangulation are performed using a message-passing GNN framework which couples neighboring nodes. Secondly, the authors use a classical adjoint method to allow for arbitrary time-discretizations (though this may be much more expensive in practice).\n\nI am not an expert on the experimental side of this area, but my sense was that the performance of the approach is relatively good, especially compared to other methods when the time step is large. For small time steps, however, PDE-Net is multiple orders of magnitude better than the present approach, a fact that should have been thoroughly discussed in the text rather than just mentioned, because the discrepancy is so large. \n\nAll the experimental results are presented as, essentially, measures of the training error. My sense is that the point of this methodology would be to meaningfully extend the time-horizon over which a given PDE could be intergrated, so I would have liked to see how the integrator performed outside the test set. \n\nSparsity is mentioned in the title and the \"contributions\" but appears basically nowhere in the main text. I did not actually see in what regard the data was sparse or see a clear theoretical justification as to why the MPNN approach would be superior on sparse data. I suppose the argument is that with arbitrary spatial and time discretization, the method still able to be formulated whereas PDE-Net requires dense spatial discretizations and time discretizations to train. However, the fact that the method could in principle be used on sparse data is not a demonstration that it works on sparse data. How does PDE-Net perform in the 2 and 4 time point cases?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LcfajJJFJGv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary\n\nThis paper presents a graph neural network model for learning to model PDE dynamical systems. Given its graph structure and its continuous time formulation (employing the adjoint method for differentiation/backpropagation), this method allows the usage of samples placed arbitrarily in space and time.\n \n\n- Pros\n\nThe continuous-time nature of the model allows for the usage of irregular time sample points.\n\nPreviously proposed methods either would not work on continuous time, or unstructured grids, or would not be applicable to settings with unknown governing PDEs. This work combines all these features. \n\nThe graph-based representation used makes the proposed method invariant to translations and rotations in the spatial domain.\n\n\n\n- Cons\n\nSimilar graph-based methods that used continuous-time to model differential equation dynamics had been previously presented, e.g. GNODE. The novelty of the proposed method might be limited.\n\nThe test cases are simple and the experimental details are somewhat lacking for a full evaluation of the results (more details below in the additional comments). \n\n\n\n- Reasons for score\n\n[Edit: Score updated, see discussion below]\n\nOverall, given the \"cons\" described above, notably the potential lack of strong novelty in the proposed method, and the lacking experimental description and results, I am for now classifying this paper as marginally below the acceptance threshold. \nOn the positive side, the method seems to perform favorably when compared to other baselines, in comparisons that are actually favorable to the other methods (e.g., using regular grids). Moreover, the method performs well on the tasks it is tested on.\nHowever, I'm concerned with some uncertainties I have regarding the experimental section and the presented results. These are discussed below in the comments.\nMoreover, the proposed method centers around using message-passing neural networks to model the differential equation dynamisc. As mentioned above, previous methods had already proposed the usage of graph neural networks with continuous time for the learning of differential equations, and I am not sure that the addition of spatial mesh information to such a graph neural network constitutes a significant enough modification at this point. \nDespite the concerns above, I am open to reading the authors' responses and the reviews/comments and changing my opinion depending on how those affect my current uncertainty.\n\n \n\n- Additional comments\n\nI believe an important element that is missing from the description of the experiments is a clearer of how much do train and test set actually differ? This would be important to understand how hard the tasks being performed are. Clearly, if training and test set are too similar, the results lose a lot of power.\nMorever, since we also dont see any traning vs test plots, it is also hard to see how much performance is different between these two are. (I am not claiming such a plot would be necessary, but merely that given the otherwise lack of information in this direction, it would be helpful information.) I am aware that the appendix includes a description of how the initial conditions for the data are generated, but lacking more information these are hard to grasp intuitively to be able to judge the tasks.\n\nMoreover, the error in model rollouts over time seems to spike at the beginning and then quickly flatten out. It seems strange that errors would spike up initially and then not compound significantly over time. Do the authors have any intuition as to why this is the case? Is it maybe a consequence of the data samples reaching a sort of steady state after some time? If so, wouldn't this weaken the case being made for a continuous-time model?\n\n\nHow would the Delauney triangulation being employed deal with possible obstacles present in the spatial domain? For example, an airfoil might have its opposing boundaries connected by edges (since they are close in space), even though that would supposedly be a solid. Would these types of solids have to be manually specified when extending this method to such scenarios? (This is not a \"drawback\", of course, it would be expected of most methods that such object boundaries would have to be defined.)\n\n\nWhat integrator is used for the experiments? \n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BlGjecSyHcO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use graph-based networks for evaluations of PDEs with continuous time formulations. In contrast to existing works on continuous time ODE formulations with graph structures, the proposed networks incorporate relative spatial information in order for the network to evaluate spatial derivatives in addition to the temporal dynamics. A key argument for this setup is the flexibility in spcae (via the graph nets) in addition to a variable time step. \n\nThe proposed setup is evaluated for the relatively simple PDE cases, a convection diffusion case, a pure diffusion case, and another advection diffusion case for transport. Despite the simplicity, the authors make an effort to illustrate the behavior of their method with an ablation study, and to compare to previous work. Here, they compare to PDE-net, which was originally proposed for system identification, and is used for predictions over time instead here. In addition, they compare to the GNODE approach by Poli et al., which omits the spatial information, but already incorporates ODEs into graph nets. For the latter, the authors demonstrate that for simple cases (pure diffusion) the GNODE approach does a good job to identify dynamics purely over time, while including advection terms significantly increases the error without spatial information. This is good to see and makes sense.\n\nFor the PDE-net comparison, I was wondering how the 3 time step sizes were incorporated into the PDE-net. Isn't this \"by-default\" a fixed time step architecture? Was it changed to receive the time step as an input, or does Figure 7 show 3 networks, i.e. one per timestep? The appendix unfortunately does not provide any additional details on how the comparison was executed. \n\nI also have to admit I couldnt follow the argumentation why graph-nets, e.g., from Sanchez-Gonalez et al. '18 aren't suitable in the context of the paper. Sure, they are evaluated on moving positions, but isn't that even more difficult compared to the static locations used here? So when keeping these positions fixed, wouldn't the networks potentially do an even better job than for the moving locations? As a plus, they wouldn't require a Delaunay triangulation or similar meshing step.\n\nWhile the ablation study is generally nice, I was missing one central component here: as the paper targets continuous time (it's highlighted in the title), I expected an evaluation regarding how much is gained from the introduction of continuous time. The larger time steps of Fig. 3 seem quite trivial, but what I instead hoped to see here was a comparison to a model that simply receives the chosen tilmestep dt as an additional input, and is trained in a supervised fashion with data from a few different time step sizes (i.e. non-continuous, but varying dt). This is maybe what was done for the PDE-net, but the paper is not clear here. I think it would be important to demonstrate the advantages of a continuous formulation, which introduces a significant amount for complexity, over a much simpler training with discrete but varying time steps. \n\nI hope the authors can shed light on this aspect during the rebuttal, as apart from this relatively central open question I like the paper. Thus, assuming that the authors can clarify this part and show that the proposed method yields benefits, I think this could be an interesting paper for ICLR. It presents an interesting evaluation of PDE-learning with graph-nets, which I would consider to be interesting for many ICLR attendees.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for learning a model for spatio-temporal data described by an (unknown) spatio-temporal PDE. The model learns a continuous time PDE using the adjunct method and uses graph networks to perform message passing between different discrete time steps on a grid obtained with Delaunay triangulation.

The method initially 3 favorable and 1 unfavorable ratings, but convincing responses to some of the raised issues led to unanimous recommendations for acceptance (not all reviewer feedback after the rebuttal has been made public, but feedback has been made to the privately AC on these issues by different reviewers). 

The reviewers appreciated novelty of the method and numerous ablations.

Initially perceived weaknesses were some key experiments on generalization over different grid discretizations; the simplicity of some experiments, and links to different prior art - many of these points have been dealt with by authors in their response.

The AC concurs and proposes acceptance.
Decision: Accept (Poster)