Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics
Authors: Sungyong Seo*
Chuizheng Meng*
Yan Liu
Venue:  (2020)
Keywords: physics-aware learning
spatial difference operators
sparsely-observed dynamics

Abstract: Sparsely available data points cause numerical error on finite differences which hinders us from modeling the dynamics of physical systems. The discretization error becomes even larger when the sparse data are irregularly distributed or defined on an unstructured grid, making it hard to build deep learning models to handle physics-governing observations on the unstructured grid. In this paper, we propose a novel architecture, Physics-aware Difference Graph Networks (PA-DGN), which exploits neighboring information to learn finite differences inspired by physics equations. PA-DGN leverages data-driven end-to-end learning to discover underlying dynamical relations between the spatial and temporal differences in given sequential observations. We demonstrate the superiority of PA-DGN in the approximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from weather stations.

Published as a conference paper at ICLR 2020
PHYSICS-AWARE DIFFERENCE GRAPH NETWORKS FOR SPARSELY-OBSERVED DYNAMICS
Sungyong Seo; Chuizheng Mengi Yan Liu
Department of Computer Science
University of Southern California
(sungyons, chui zhem, yan iu. as) @usc. edu
ABSTRACT
Sparsely available data points numerical emror finite differences which hinders us from modeling the dynamics of physical systems. The discretization error becomes defined on an unstructured grid, making it hard to build deep learning models to handle physics-governing observations the unstructured grid. In this paper, we propose novel architecture, Physics-awars Difference Graph Networks (PA-DGN), which exploits neighboring information to learn finite differences inspired by physics equations. PA-DGN leverages data-driven end-to-end learning to discover underlying dynamical relations between the spatial and temporal differences given sequential observations. We demonstrate the superiority of PA-DGN in the approximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from weather stations.
larger when the sparse data irregularly distributed
INTRODUCTION
Modeling real world phenomena, such as climate observations, traffic flow, physics and chemistry simulation (Li et al. 2018 Geng al. 2019: Long et al. 2018: de Bezenac et al. 2018: [Sanchez, Gonzalez et al. 2018; Gilmer et al. 2017). is important but extremely challenging. While deep learning has achieved remarkable successes in prediction tasks by learning latent representations from data-rich applications such as image recognition (Krizhevsky et al. 2012), text understanding (Wu al. 2016), and speech recognition (Hinton et al. 2012). we confront many challenging scenarios in modeling natural phenomena with deep neural networks when only a limited number observations available. Particularly, the sparsely available data points cause substantial numerical error when we utilize existing finite difference operators and the limitation requires a more principled way to redesign deep learning models.
While many methods have been proposed to model physics-simulated observations using deep learning, many of them designed under the assumption that input is on continuous domain. For example, Raissi et al. (2017a] b) proposed phfsics-ifformed neural networks (PINNs) to learn nonlinear relations between input (spatial- and temporal coordinaats (x, t)) and output simulated with given partial differential equation (PDE). Since Raissi et al. 22177a b) use the coordinates as input and compute derivatives based on the coordinates to represent the equation, the setting is only valid when the data are densely observed over spatial and temporal space.
Prior knowledge related to physics equations has been combined with data-driven models for var- ious purposes. Chen et al. (2015) proposed nonlinear diffusion process for image restoration and de Bezenac et al. (2018) incorporated the transport physics (advection-diffusion equation) with deep neural networks for forecasting
surface temperature by extracting the motion field. Lutter et al. (2019) introduced deep Lagrangian networks specialized to learn Lagrangian mechanics with learnable parameters. (Seo & Liu (019) proposed a physics-informed regularizer impose data- specific physics equations. In common, the methods in Chen et al. (2015); ); de Bezenac et al. (2018); Lutter et al. (2019) not efficiently applicable sparsely discretized input as only small number of data points are available and continuous properties on given space are not easily recovered. It is
"Equally contributed.

---

Published as a conference paper at ICLR 2020
unsuitable to directly use continuous differential operators to provide local behaviors because it is hard to approximate the continuous derivatives precisely with the sparse points (Shewchuk) [002] Amenta & Kill 2004: Luo et al., (009). Furthermore, they are only applicable when the specific physies equations are explicitly given and still hard to be generalized to incorporate other types of
equations.
As another direction to modeling physics-simulated data, Long et al. (0188) proposed PDE-Net which uncovers the underlying hidden PDEs and predicts the dynamics of complex systems. Ruthotto
& Haber (2018) derived new CNNs: parabolic and hyperbolic CNNs based on ResNet (He et al.
2016) architecture motivated by PDE theory. While [Long et al. (2018); Ruthotto & Haber (2018) are
flexible to uncover hidden physics from the constrained kernels, it is still restrictive to a regular grid
where the proposed constraints on the learnable filters are easily defined.
The topic of reasoning physical dynamics of discrete objects has been actively studied (Sanchez, Gonzalez et al., 2018 Battaglia et al. 2016, 20T6/Chang et al. (0166) as the appearance of graph-based Gilmer et al. (017). Although these models can handle sparsely located data points without explicitly given physics equations, they are purely data-driven so that the physics-inspired inductive bias for exploiting finite differences is not considered at all. In contrast, our method consists of physics- aware modules allowing efficiently leveraging the inductive bias to learn spatiotemporal data from the physics system.
neural networks Kipf & Welling| [011 2017/Santoro [Santoro et al. 2017/
In this paper, we propose physics- awwre difference graph networks (PA-DGN) whose architecture is motivated to leverage differences of sparsely available data from the physical systems. The differences are particularly important since most of the physics-relatee equations) handle differences of physical quantities in spatial and temporal space instead of using the quantities directly. Inspired by the property, we first propose spatial difference layer (SDL) to efficiently learn the local representations by aggregating neighboring information in the sparse data points. The layer is based on graph networks (GN) as it easily leverages structural features to learn the localized representations and share the parameters for computing the localized features. Then, the layer is followed by recurrent graph networks (RGN) to predict the temporal difference which another core component of physics-related dynamic equations. PA-DGN applicable to various tasks and we provide two representative tasks; the approximation of directional derivatives and the prediction of graph signals.
dynamic equations (e.g.. Navier--tooli Stokes
Our contributions are:
We tackle limitation of the sparsely discretized data which cause numerical error when we model the physical system by proposing spatial difference layer (SDL.) for efficiently exploiting neighboring information under the limitation of sparsely observable points.
We combine SDL with recurrent graph networks to build PA-DGN which automatically learns the underlying spatiotemporal dynamics in graph signals.
We verify that PA-DGN is effective in approximating directional derivatives and predicting graph signals in synthetic data. Then, we conduct exhaustive experiments predict climate observations from land-based weather stations and demonstrate that PA-DGN outperforms other baselines.
2 PHYSICS-AWARY DIFFERENCE GRAPH NETWORK
In this section, we introduce the building module used to learn spatial differences of graph signals and describe how the module is used to predict signals in the physics system.
2.1 DIFFERENCE OPERATORS ON GRAPH
As approximation of derivatives in continuous domain, difference operators have been used as a core role to compute numerical solutions of (continuous) differential equations. Since it is hard to derive closed-form expressions of derivatives in real-world data, the difference operators have been considered as alternative tools deseribe and solve PDEs in practice. The operators are especially important for physics-relate data (e.g., meteorological observations) because the governing rules behind the observations are mostly differential equations.
2

---

Published as a conference paper at ICLR 2020
(a) Original graph signals
(b) Detected edge
(c) Sharpened signals (d) Modulated gradients
Figure 1: Examples difference operators applied to graph signal. Filters used for the processing
(b) E,(, (5) (c) E, (1.15 fz), (d) [a 0.5fi-
Graph signals Given a graph 9 = (V, E) where is a set of vertices V = (1....,,, No) and is a of edges E S [(i.j) i.j E V) (E Ne), graph signals on all nodes at time are f(t) (f((t) i E V) where fi V + R. Graph signals on edges can also be defined similarly, F(t) (f) (i.5) E} where F.j:
E. Both signals can multidimensional.
Gradient on graph difference
The gradient (V) of function on nodes of graph represented by finite
V (VV (B))
(v) (f3- ()) if (i,j) E E and 0 otherwise,
where (V) and (E) denote Hilbert spaces for node/edge functions, respectively. The gradients on graph provide finite differences of graph signals and they become edge (i.j) features.
Daplace-Beltrame operator defined as
Caplace-Beltramt operator (or Laplacian, A) in graph domain is
A T. - (V),,,
(AA)
(fi- (1- Vi,jeW
J(LEEE
This operator is usually regarded as a matrix form in other literature, L D - A where A is an adjacency matrix and D diag( Σ Ejj Aij) is a degree matrix.
2.2 DIFFERENCE OPERATORS ON TRIANGULATED MESII
According to Crane 2018)) the gradient and Laplacian operators on the triangulated mesh can be discretized by incorporating the coordinates of nodes. To obtain gradient operator, the per-face gradient of each triangular face is calculated first. Then, the gradient on each node is the area-wert weighted average of all its neighboring faces, and the gradient on edge (i.j) is defined as the dot product between the per-node gradient and the direction vector eij- The Laplacian operator can be discretized with Finite Element Method (FEM):
(Aʃ)
(cot.a 00) (J² - -(), j((J)E
Σ
where node j belongs to node i's immediate neighbors (j E N2) and (az, B,) are two opposing angles of the edge (i,j)-
2.3 SPATIAL DIFFERENCE LAYER
While the difference operators are generalized in Riemannian manifolds (Lai et al. 2013 Lim 2015), there exist numerical error compared to those in continuous space and the error can be larger when the nodes are spatially far from neighboring nodes because the connected nodes (j E N,) of i-th node fail to represent local features around the node. Furthermore, the error is even larger if available data points are sparsely distributed (e.g., sensor-based observations). other words, the difference operators are unlikely discover meaningful spatial variations behind the sparse observations since they are highly limited to immediate neighboring information only. To mitigate the limitation, we propose spatial difference layer (SDL.) which consists of set of parameters to define learnalle

---

Published as a conference paper at ICLR 2020
aignake
Diftrrnnce grapn
Predizted gresh
signate time
Compatenaion
RGN
(..0 is v)
("VAO,) ("A)(e8)
10-1) 1) (f (043) ne
Modulaned spatia ciforenoes
Hittlen
Updineed Hittlen
SDL
(())) A1)()
(N,() W V)
Figure 2: Physics-aware Difference Graph Networks for graph signal prediction. Blue boxes have learnable parameters and all parameters are trained through end-to-end learning. The nodes/edges can be multidimensional.
difference operators as a form of gradient and Laplacian to fully utilize neighboring information:
(") = w/f (U; "w(/g>)0) S(),
("AN)=
Σ
Wij (1) (12)
10 Sn)
(1)
where Waj are the parameters tuning the difference operators along with the corresponding edge direction eij. The two forms (Eq [ are associated with edge and node features, respectively. The superscript in " and " denotes that the difference operators are functions the learnable parameters w. w (g) and wij are obtained by integrating local information as follow:
(t)
Way ((fk, Fmn (m,n) E h--hop neighborhood of edge (i,j)))
While the standard difference operators consider two connected nodes only (i and j) for each edge (i,j),, Eq uses larger view (h-hop) to represent the differences between and nodes. Since graph networks (GN) (Battaglia et al. 2018) are efficient networks to aggregate neighboring information, we use GN for g(+) function and Way are edge features of output of GN. Eq can viewed higher-order difference equation because nodes/edges which are multi-hop apart are considered.
Why has similar role of parameters in convolution kernels of CNNs. For example, while the standard gradient operator can be regarded as an example of simple edge-detecting filters, the operator can be
sharpening filter if (!()))
and (gx) [8++1 for node and the operators over each edge are [N.
summed. In other words, by modulating Wij. is readily extended to conventional kernels including edge detection sharpening filters and even further complicated kernels. On top of Wij. the difference forms in Eq] I make an optimizing process for learnable parameters based on the differences instead of the values intentionall.. Eq thus naturally provides the chysics-inspiret inductive bias which is particularly effective for modeling physics-related observations. Furthermore, it easily possible to increase the number of channels for w.y)) and wiy to be more expressive. Figure iillsstrates how the exemplary filters convolve the given graph signals.
(!)
2.4 RECURRENT GRAPH NETWORKS
Difference graph Once the modulated spatial differences (t), " (t) are obtained, they concatenated with the current signals [(t to construct node-wise (z,) and edge-wise (zi,) features and the graph is called a difference graph. The difference graph includes all information describe spatial variations.
Recurrent graph networks Given a snapshot (f(1), F(t) of a sequence of graph signals, one difference graph is obtained and is used to predict next graph signals. While non-linear layer can be used to combine the learned spatial differences to predict the next signals, it is limited to discover spatial relations only among the features in the difference graph. Since many equations describing physics-relatee phenomena are non-stati (e.g., Havier-Stoker equations), we adopt recurrent graph networks (RGN) (Sanchez- al. 2018) with graph state Gn, input to combine the spatial
4

---

Published as a conference paper at ICLR 2020
differences with temporal variations. RGN returns graph state (9) (h*(1) kx())) and next graph signal x and ziz. The update rule described as follow:
(zly h h(()) 2 (2ij. Zis 2z. h (c)) for all (i,j) pairs,
2. (z) (() + o x / ) for E V,
is an aggregated edge attribute related to the node i,
where 6º, x² are edge and node update functions, respectively, and they can be any recurrent unit. Finally, the prediction made through a decoder by feeding the graph signal, 2 and zig.
Learning objective Let f and F denote predictions of the target node/edge signals. PA-DGN is trained by minimizing the following objective:
L=
Σ 11 fall
Σ
1 F
(iJ)ER
For multistep predictions, & is summed over all predicting steps. If only type (node edge) of signal is given, the corresponding term in Eq used to optimize the parameters in SDL and RGN simultaneously.
3 EFFECTIVENESS OF SPATIAL DIFFERENCE LAYER
To investigate if the proposed spatial difference forms (Eq I can be beneficial to learning physics- related patterns, we use SDL on two different tasks: (1) approximate directional derivatives and (2) predict synthetic graph signals.
3.1 APPROXIMATION OF DIRECTIONAL DERIVATIVES
As we claimed in Section 2.33 the standard difference forms (gradient and Laplacian) on a graph can cause significant numerical error easily because they susceptible to a distance of points and variations of a given function. To evaluate the applicability of the proposed SDL, we train SDL approximate directional derivatives on graph. First, we define synthetic function and its gradients on 2D space and sample 200 points (r%, y:). Then, we construct graph on the sampled points by using
nx
Oga
Rxol
vixe)
k-NN algorithm (k: 4). With the known gradient
01 (V = (zz og at
each point (a node in the graph), we can compute directional derivatives by projecting f connected edge esj (See Figure B). We compare
against four baselines: (1) the finite gradient (FinGrad) (2) multilayer perceptron (MLP) (3) graph networks (GN) (4) different form of Eq (One-w). For the finite gradient ((f, fi)/ xz/)), there is learnable parameter and it only uses two points. For MLP, we feed (fr, æ;, x;,) as input to see whether learnable parameters can benefit the approximation or not. For GN, two connected points as edge features and function values on the points as node features. The edge feature output of GN is used as a prediction for the directional derivative modify the proposed form as (" ("V)is wiz ʃj - St. GN and the modified form are used to verify the effectiveness of Eq Note that define two synthetic functions (Figure (4) which have different property; (I) monotonically increasing from a center and (2) periodically varying.
Figure 3: Directional
derivative graph
distances of
the edge. Finally,
Table 1: Mean squared error (10-2) for approximation of directional derivatives.
Functions
FinGrad
MLP
GN
One-w
SDL
fi v) = 0.1x² + 0.5y² 6.42+10.7
2.1220.32
1.05+10.42
1.41
0.97=0.39
f2(2, v) sin(x) cos(y)
5.90+1.00
2.29
2.1710.34
6.73
1.26-0.05
5

---

Published as a conference paper at ICLR 2020
Figure 4: Gradients and graph structure of sampled points. Left: the synthetic function fi (x,
0.1x² + 0.5y². Right: the synthetic function is f2(2, v) sin(x:) + cos(y).
Approximation accuracy As shown in Table] | the proposed spatial difference layer outperforms others by a large margin. As expected, FinGrad provides the largest error since it only considers two points without learnable parameters. is found that the learnable parameters can significantly benefit to approximate the directional derivatives even if input is the same (FinGrad vs. MLP). Note that utilizing neighboring information (GN, One-w, SDL) is generally helpful to learn spatial variations properly. However, simply training parameters in GN is not sufficient and explicitly defining difference, which is important to understand spatial variations, provides more robust inductive bias. One important thing we found is that One-w is not effective as much as GN and it can be even worse than FinGrad. It is because of its limited degree of freedom. As implied in the form (V f) = Way f, fi, only one wiy adjusts the relative difference between fü and fy. and this is not enough to learn whole possible linear combinations of fi and fy. The unstable performance supports that the form of SDL is not ad-hoc but more rigorously designed.
3.2 GRAPH SIGNAL PREDICTION
We evaluate PA-DGN on the synthetic data sampled from the simulation of specific convection- diffusion equations, to provide if the proposed model can predict next signals of the simulated dynamics from observations discrete nodes only. For the simulated dynamics, we equation slightly modified based on the one in Long et al. (0018).
dfi(() a(i)())) + (VF) + c()AAS dt
f:(0) = fo(
(4)
where the index i points the i-th node whose coordinate is (x; y4) the 2D space (0,, 2n x (0, 2n and å and indicate x- and y-direction in the space. a(i) 0.5(cos (y:) + x, (2 x4 sin()sinn()) (z.)) 0.6.
(2, +)²
b(i) = 2(cos(y)) + sin(x:)) + 0.8, and c(i) = 0.5
Then, we uniformly
V2n
sample 250 points in the above 2D space. The task to predict signal values of all points in the future MM steps given observed values of the first N steps. For our experiments, we choose N = 5 and M 15. Since there is no priori graph structure on sampled points, we construct a graph with k-NN algorithm (k 4) using the Euclidean distance. Figure 5 the dynamics and the graph structure of sampled points.
To evaluate the effect of the proposed SDL on the above prediction task, we cascade SDL and linear regression model as our prediction model since the dynamics follows a linear partial differential equation. We compare its performance with four baselines: (1) vector auto-regressor (VAR); (2) multilayer perceptron (MLP); (3) StandardOP: the standard approximation of differential operators in Section 2.1 followed by linear regressor; (4) MeshOP: similar to StandardOP but use the discretization on triangulated mesh in Section 2.2 for differential operators.
Table 2: Mean absolute error (10 (10-2) for graph signal prediction.
VAR
MLP
StandardOP
MeshOP
SDL
16.8410.41
15.75400.33
11.99:1.29
12.82-10.06
10.87:10.98
Prediction Performance Table 2 shows the prediction performance of different models measured with mean absolute error. The prediction model with our proposed spatial differential layer outper- forms other baselines. All models incorporating any form of spatial difference operators (StandardOP,

---

Published as a conference paper at ICLR 2020
1 1= 1
1= 5
1. == 10
t == 15
= 20
Figure 5: Synthetic dynamics and graph structure of sampled points.
MeshOP and SDL) outperform those without spatial difference operators (VAR and MLP), showing that introducing spatial differences information inspired by the intrinsic dynamies helps prediction. However, in cases where points with observable signal are sparse in the space, spatial difference operators derived with fixed rules be inaccurate and sub-optimal for prediction since the locally linear assumption which they are based on no longer holds. Our proposed SDL, to the contrary, capable of bridging the gap between approximated difference operators and accurate ones by introducing learnable coefficients utilizing neighboring information, and thus improves the prediction performance.
PREDICTION: GRAPH SIGNALS ON LAND-BASED WEATHER SENSORS
We evaluate the proposed model on the task of predicting climate observations (Temperature) from the land-based weather stations located in the United States.
4.1 EXPERIMENTAL SET-UP
Data and task We sample the weather stations located in the United States from the Online Climate Data Directory of the National Oceanic and Atmospheric Administration (NOAA) and choose the stations which have actively measured meteorological observations during 2015. We choose two geographically close but meteorologically diverse groups of stations: the Western and Southeastern states. We use k-Nearest Neighbor (NN) algorithm (k = 4) to generate graph structures and the final adjacency matrix is A = (A + 4 make it symmetric where A₁ is the output adjacency matrix from N-NN algorithm.
Figure O shows the distributions of the land-based weather stations and their connectivity. Since the stations are not synchronized and have different timestamps for the obser- vations, we aggregate the time series hourly. The 1-year sequential data are split into the train set (8 months), the validation set (2 months), and the test set (2 months), re- spectively.
Our main task is to predict the next graph signals based on the current and past graph signals. All methods we evaluate are trained through the objective (Eq with the Adam optimizer and we use scheduled sampling (Bengio et al. 2015) for the models with recurrent modules. We evaluate PA-DGN and other baselines on two prediction tasks, (1) 1-step (2) multistep-ahead predictions. Furthermore, we demonstrate the ablation study that provides how much the spatial derivatives from our proposed SDL. are important signals to predict the graph dynamics.
Figure Weather stations in (left) west- ern (right) southeastern states in the United States and k-NN graph.
4.2 GRAPH SIGNAL PREDICTIONS
We compare against the widely used baselines (VAR, MLP, and GRU) for 1-step and multistep prediction. Then, structure beneficial. Finally, we evaluate PA-DGN to verify if the proposed architecture (Eq [ is able to reduce prediction loss. Experiment results for the prediction task are summarized in Table
RGN
et al. 2018) to examine how much the graph
Overall, RGN and PA-DGN are better than other baselines and implies that the graph structure provides useful inductive bias for the task. It is intuitive the meteorological observations
7

---

Published as a conference paper at ICLR 2020
continuously changing the space and time and thus, the observations the i-th station strongly related to those of its neighboring stations.
PA-DGN outperforms RGN and the discrepancy comes from the fact that the spatial derivatives (Eq we feed in PA-DGN are beneficial and this finding is expected because the meteorological signals at certain point function of not only its previous signal but also the relative differences between neighbor signals and itself. Knowing the relative differences among local observations is particularly essential understand physics-related dynamics. For example, Diffusion equation, which describes how physical quantities (e.g., heat) are transported through space over time, is also a function of relative differences of the quantities th = D&f) ) rather than values of the neighbor signals. In other words, spatial differences are physics-aware features and it desired leverage the features as input to learn dynamics related to physical phenomena.
Table 3:
Graph signal prediction results (MAF) on multistep predictions. In each row, we report the average with standard deviations from all baselines and PA-DGN. One step is 1-hour time interval.
Region
Method
1-step
6-step
12-step
VAR
0.1241 0.0234
0,4295 = 0.1004
0.4820 = 0.1298
MLP
0.1040 + 0.0003
0.3742 = 0.0238
0.4998 = 0.0637
GRU
0.0913 0.0047
0.1871 = 0.0102
0.2707 = 0.0006
West
RGN
0.0871 + 0.0033
0.1708 + 0.0024
0.2666 = 0.0252
RGN(StandardOP)
0.0860 0.0018
0.1674 + 0.0019
0.2504 = 0.0107
RGN(MeshOP)
0.0840 + 0.0015
0,2119 + 0.0018
0.4305 = 0.0177
PA-DGN
0.0840 + 0.0004
0.1614 0.0042
0.2439 = 0.0163
VAR
0.0889 0.0025
0.2250 + 0.0013
0.3062 0.0032
MLP
0.0722 + 0.0012
0.1797 + 0.0086
0.2514 + 0.0154
GRU
0.0751 0.0037
0.1724 0.0130
0.2446 = 0.0241
SouthEast
RGN
0,0790 + 0.0113
0,1815 = 0.0239
0.2548 = 0.0210
RGN(StandardOP)
0.0942 0.0121
0.2135 + 0.0187
0.2902 - 0.0348
RGN(MeshOP)
0.0905 + 0.0012
0.2052 + 0.0012
0.2602 - 0.0062
PA-DGN
0.0721 + 0.0002
0.1664 + 0.0011
0.2408 = 0.0056
4.3 CONTRIBUTION OF SPATIAL DERIVATIVES
We further investigate if the modulated spatial derivatives (Eq 1 are effectively advantageous compared to the spatial derivatives defined Riemannian manifolds. First, RGN without any spatial derivatives is assessed for the prediction tasks on Western and Southeastern states graph signals. Note that this model does not use any extra features but the graph signal, ft). Secondly, we add (1) StandardOP, the discrete spatial differences (Gradient and Laplacian) in Section 2.1 and (2) MeshOP, the triangular mesh approximation of differential operators Section 2.2 separately as additional signals to RGN. Finally, incorporate with RGN proposed Spatial Difference Layer.
Table the contribution each component. As expected, PA-DGN provides much higher drops in MAE (3.56%, 5.50%, 8.51% and 8.73%, 8.32%, 5.49% two datasets, respectively) compared to RGN without derivatives and the results demonstrate that the derivatives, namely, relative differences from neighbor signals effectively useful. However, neither RGN with StandardOP with MeshOP can consistently outperform RGN. We also found that PA-DGN consistently shows positive effects on the prediction еrгor compared to the fixed derívatives. This finding is piece of evidence support that the parameters modulating spatial derivatives in our proposed Spacial Difference Layer are properly inferred to optimize the networks and improve the prediction performance.
CONCLUSION
In this paper, we introduce a novel architecture (PA-DGN) that approximates spatial derivatives to them represent PDEs which have a prominent role for physics-aware modeling. PA-DGN
8

---

Published as a conference paper at ICLR 2020
effectively learns the modulated derivatives for predictions and the derivatives can be used to discover hidden physics describing interactions between temporal and spatial derivatives.
ACKNOWLEDGEMENTS
This work is supported in part by NSF Research Grant IIS- 1254206 and MINERVA grant N00014- 17- 1-2281, granted to co-author Yan Liu in her academic role at the University of Southern California. The views and conclusions are those of the authors and should not interpreted as representing the official policies of the funding agency, the U.S. Govemment. Last but not least, appreciate anonymous reviewers for your thorough comments and suggestions.
REFERENCES
Nina Amenta and Yong Joo Kil. Defining point-set surfaces. In ACM Transactions on Graphics (TOG). volume 23, pp. 264-270 ACM, 2004.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502 4510, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806.010000111 2018.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 1171-1179, 2015.
Michael B Chang. Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv: 2016.
Yunjin Chen, Wei Yu, and Thomas Pock. On learning optimized reaction diffusion processes for effective image restoration. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5261-5269, 2015.
Keenan Crane. Discrete differential geometry: An applied introduction. Notices of the AMS, Communication, 2018
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes:
Incorporating prior scientific knowledge. In International Conference Learning Representations, 2018. URL https: http://ppppeeeeeeeeeeeeeo
www.::eee/foooree iiieiiddd-yplae By4Hs FWAZ
Xu Geng, Yaguang Li, Leye Wang, Lingyu Zhang, Qiang Yang, Jieping Ye, and Yan Liu. Spa- tiotemporal multi -ulti-graph convolution network for ride-hailing demand forecasting, In 2019 AAAI Conference Artificial Intelligence (AAAI' 2019.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pp. 1263-1272. JMLR. org, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6)82-97, 2012.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky, lly Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Rongjie Lai, Jiang Liang, and Hongkai Zhao. local mesh method for solving pdes on point clouds. Inverse Problems & Imaging, 7(3), 2013.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations, 2018. URL
net htte:/orrmmmiid11.GG.
Lek-Heng Lim. Hodge laplacians graphs. arXiv preprint arXiv: 2015.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. Interna- tional Conference on Machine Learning, 2018
Chuanjiang Luo, Issam Safa, and Yusu Wang. Approximating gradients for meshes and point clouds via diffusion metric. Computer Graphics Forum, volume 28, pp. 1497-1508. Wiley Online Library, 2009.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics model prior for deep learning. In International Conference on Learning Representations, 2019. URL https: / /ppprrevieeeeeeeeeeeeee net/ forummid-Bkiiiii CC
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i):
Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561
2017a.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations. arXiv preprint arXiv:1711.10530
2017b.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. arXiv preprint Neo::040404777 2018.
Alvaro Sanchez-Gonzaler Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. International Conference on Machine Learning. 2018.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Sungyong Seo and Yan Liu. Differentiable physics-informed graph networks. arXiv preprint arXiv:1902.02990 2019.
Jonathan Richard Shewchuk. What is good linear finite element? interpolation, conditioning, anisotropy, and quality measures (preprint). University of California at Berkeley, 73:137, 2002.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.08144, 2016.
10

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1
SIMULATED DATA
For the simulated dynamics, discretize the following partial differential equation similar to the one in Long et al. (2018) to simulate the corresponding linear variable-coefficiee convection-diffusion equation graphs.
In a continuous space, we define the linear
cww.:/efccccccnooooocctnncoooini equation as: alx, y) b(zz ((r
at
(5)
fo(x.y)
with Q [0, [0, 2m.. (,, E [0.0.2] x Ω2 a(x,y) 0.5(cos(v) + x(2T x) sin(x)) +
(ar m))+ (vr x)}
0.6, b(z, v) = 2(cos(y) + sin( + 0.8, dx, y) = 0.5 1
22
We follow the setting of initialization in Long et 2018
fo(o, v) = Σ XR. cos(kr + ly) + Th, sin(ke + ly)
(6)
|&l,
where N = 9, AR, Thit ~ N (0, ) and l and are chosen randomly.
We use spatial difference operators approximate spatial derivatives:
1
2s (f v:) - f(:: -8,9:)) (f(xi, f(a; 8, v,))))
2s
1
fo(
(f v6) - (1riVii- - -8)) (f v/) -( ys +8)
28
2s
Sax (1i,
(f(rs, 31) - ((x1-,,9:)) +
(f Us) -f(x; + 8, vs))
1
(f( yi) - f(r, - 8)) + (f Uo) - yc 8))
82
22
where is the spatial grid size for discretization. Then we rewrite (5) with difference operators defined on graphs:
{
01 कर
(()) + + (Af)s))
(8)
f.(0)= fo(
where
a(zs, 34)
if x. xj + 8, 8: Vj
a()((1))3))
2s
(9)
a(Ts, y1
if z4 zj 8, y/ yj
28
b( Vi
if Tj. V1 yj +
(())(2)))))
28
(10)
b(r Mi)
if Ty: Vi = Uj
2s
c(i(
(11)
Then we replace the gradient w.r.t time in ( with temporal discretization:
{ f(++1)= At(a())(V)) = + (()(()))) + cc()((A))a + (A))ss)) + f()
(12)
1100=
where At is the time step in temporal discretization.
Equation is used for simulating the dynamics described by the equation Then, we uniformly sample 250 points in the above 2D space and choose their corresponding time series of as the dataset used in our synthetic experiments. We generate 1000 sessions on a 50 x 50 regular mesh with time step size At 0.01. 700 sessions used for training, 1.50 for validation and 150 for test.
11

---

We performed times for every experiment in this paper to report Published as a conference paper at ICLR 2020
A.2 EXPERIMENT SETTINGS
Here we provide additional details for models we used in this work, including model architecture settings and hyper-parameter settings.
A.2.1
MODEL. SETTINGS
Unless mentioned otherwise, all models use hidden dimension of size 64.
VAR: A vector autoregression model with 2 lags. Input is the concatenated features of previous 2 frames. The weights are shared among all nodes in the graph.
MLP: multilayer perceptron model with 2 hidden layers. Input is the concatenated
features of previous 2 frames. The weights are shared among all nodes in the graph.
GRU: A Gated Recurrent Unit network with 2 hidden layers. Input is the concatenated
features of previous 2 frames. The weights are shared among all nodes in the graph.
RGN: A recurrent graph neural network model with 2 GN blocks. Each GN block has an edge update block and node update block, both of which use 2-layer GRU cell as the update function. We set its hidden dimension to 73 so that it has the same number of learnable parameters as our proposed model PA-DGN.
RGN(StandardOP), Similar RGN, but the output of difference operators in Sec- tion t.. as extra input features. We set its hidden dimension to 73.
RGN(MeshOP): Similar RGN(StandardOP) but the extra input features calculated using opeartors in Section 2.2 We set its hidden dimension to 73.
PA-DGN: Our proposed model. The spatial derivative layer uses a message passing neural network (MPNN) with 2 GN blocks using 2-layer MLPs as update functions. The forward network part uses a recurrent graph neural network with 2 recurrent GN blocks using 2-layer GRU cells update functions.
The numbers learnable parameters all models are listed as follows:
Table 4: Numbers of learnable parameters.
Model
VAR
MLP
GRU
RGN
RGN(StandardOP)
RGN(MeshOP)
PA-DGN
# Params
4,417
37,889
345,876
341,057
342,152
340,001
A.2.2 TRAINING SETTINGS
The number of evaluation runs the mean and standard deviations.
Length of prediction For experiments on synthetic data, all models take first 5 frames input and predict the following 15 frames. For experiments on NOAA datasets, all models take first frames as input and predict the following 12 frames.
Training hyper-parametery We use Adam optimizer with learning rate 1e-3, batch size 8, and weight decay of 5e-4. All experiments are trained for maximum of 2000 epochs with early stopping. All experiments are trained using inverse sigmoid scheduled sampling with the coefficient 107.
Environments All experiments implemented with Python3.6 and PyTorch 1.1.0, and are with NVIDIA GTX 1080 Ti GPUs.
A.3 EFFECT OF DIFFERENT GRAPH STRUCTURES
In this section, evaluate the effect of 2 different graph structures baselines and our models: (1) k-NN: a graph constructed with N--N algorithm (k = 4); (2) TriMesh: a graph generated with Delaunay Triangulation. All graphs use the Euclidean distance.
12

---

Published as a conference paper at ICLR 2020
Table Mean absolute error (10-2) for graph signal prediction on the synthetic dataset.
StandardOP
MeshOP
SDL
VAR
MLP
k--N
TriMesh
k--N
TriMesh
k-NN
TriMesh
17.30
16.27
12.00
12.29
12.87
12.82
11.04
12.40
Table 6: Graph signal prediction results (MAE) on multistep predictions. In each row, we report the average with standard deviations from all baselines and PA-DGN. One step is hour time interval.
Region
Method
Graph
1-step
6-step
12-step
VAR
0.1241 0.0234
0.4295
0.1004
0.4820 + 0.1298
MLP
0.1040 + 0.0003
0.3742 = 0.0238
0.4998 = 0.0637
GRU
0.0913 + 0.0047
0.1871 = 0.0102
0.2707 = 0.0006
k--N
0.0871 + 0.0033
0.1708 0.0024
0.2666 + 0.0252
RGN
TriMesh
0.0897 = 0.0030
0.1723 = 0.0116
0.2800 = 0.0414
West
RGN
k-NN
0.0860 + 0.0018
0.1674 0.0019
0.2504 + 0.0107
(StandardOP)
TriMesh
0.0842 + 0.0011
0.1715 = 0.0027
0.2517 + 0.0369
RGN
k-NN
0.0840 + 0.0015
0.2119 = 0.0018
0.4305 = 0.0177
(MeshOP)
TriMesh
0.0846 + 0.0017
0.2090 0.0077
0.4051 + 0.0457
k-NN
0.0840 0.0004
0.1614 0.0042
0.2439 + 0.0163
PA-DGN
TriMesh
0.0849 = 0.0012
0.1610 = 0.0029
0.2473 = 0.0162
VAR
0.0889 + 0.0025
0.2250 + 0.0013
0.3062 + 0.0032
MLP
0.0722 + 0.0012
0.1797 = 0.0086
0,2514 = 0.0154
GRU
0.0751 0.0037
0.1724 = 0.0130
0.2446 = 0.0241
k-NN
0.0790 0.0113
0.1815 0.0239
0.2548 + 0.0210
RGN
TriMesh
0.0932 + 0.0105
0.2076 = 0.0200
0.2854 + 0.0211
SouthEast
RGN
k-NN
0.0942 = 0.0121
0.2135 = 0.0187
0.2902 + 0.0348
(StandardOP)
TriMesh
0.0868 + 0.0132
0.1885 = 0.0305
0.2568 = 0.0328
RGN
k--N TriMesh
0.0913 + 0.0016
0.2069 0.0031
0.2649 = 0.0092
(MeshOP)
0.0877 0.0020
0.2043 + 0.0026
0.2579 + 0.0057
k-NN
0.0721 0.0002
0.1664 0.0011
0.2408 = 0.0056
PA-DGN
TriMesh
0.0876 + 0.0096
0.2002 = 0.0163
0.2623 = 0.0180
Table and Table Table/ojshor show the effect of different graph structures on the synthetic dataset used in Section 3.2 and the real-world dataset in Section 4.2 separately. We find that for different models the effect of graph structures not homogeneous. For RGN and PA-DGN, k--N graph more beneficial to the prediction performance than TriMesh graph, because these two models rely more on neighboring information and k-NN graph incorporates it better than a Delaunay Triangulation graph. However, switching from TriMesh graph k-NN graph is harmful to the prediction accuracy of RGN(MeshOP) since Delaunay Triangulation is a well-defined method for generating triangulated mesh in contrast to k-NN graphs. Given the various effect of graph structures different models, proposed PA-DGN under k-NN graphs always outperforms other baselines using any graph structure.
13

---

Published as a conference paper at ICLR 2020
MAC este-tee Soteo avektes
MAC strectinn Solip predktin
XAX cbi 1r 32 prd
$110
1.50
1.0
1.00
135
111
Figure 7: MAE across the nodes.
A.4 THE DISTRIBUTION OF PREDICTION ERROR ACROSS NODES
Figure 7 provides the distribution of MAEs across the nodes of PA-DGN applied to the graph signal prediction task of the coast region of the real-world dataset in Section (.2) As shown in the figure, nodes with the highest prediction error for short-term prediction are gathered in the inner part where the observable nodes are sparse, while for long-term prediction nodes the area with a limited number of observable points no longer have the largest MAE. This implies that PA-DGN can utilize neighboring information efficiently even under the limitation of sparsely observable points.
A.5 EVALUATION ON NEMO SEA SURFACE TEMPERATURE (SST) DATASET
We tested proposed method and baselines the NEMO surface temperature (SST) datase We first download the data in the between 50N" -65N" and 75W -10W" starting from 2016-01- 01 to 2017-12-31, then we crop the [0, 550) x [100, 650] square from the area and sample 250 points from the square and truncate the tail. All models use the first 5-day SST as input and predict the SST the following 15 and 25 days. We use the data 2016 for training all models and the left for testing.
chosen dataset. We divide the data into 24 sequences, each lasting 30 days,
For StandardOP, MeshOP and SDL, we test both options using linear regression and using RGN for the prediction part and report the best result. The results in Table 7 show that all methods incorporating spatial differences gain improvement on prediction and that our proposed learnable SDL outperforms all other baselines.
Table 7: Mean absolute error (10 2) for SST graph signal prediction.
VAR
MLP
GRU
RGU
StandardOP
MeshOP
SDL
15 15-step
15.123
15.058 19.473
15.101
15.172 19.705
14.756 18.983
14.607
14.382
25 25-step
19.533
19.522
18.977
18.434
Available http:///wwcccc.tii.....ooooooooo
14

---

Published as a conference paper at ICLR 2020
A.6 EVALUATION ON DATASETS WITH DIFFERENT SPARSITY
We changed the number of nodes to control the sparsity of data. As shown in Table g our proposed model outperforms others under various settings of sparsity the synthetic experiment in Section
Table 8: Mean absolute error (10-2) for graph signal prediction with different sparsity.
#Nodes
VAR
MLP
StandardOP
MeshOP
SDL
250
0.1730
0.1627
0.1200
0.1287
0.1104
150
0.1868
0.1729
0.1495
0.1576
0.1482
100
0.1723
0.1589
0.1629
0.1696
0.1465
Furthermore, we sampled 400 points and trained SDL as described in Section 3.1 and resampled evaluate if SDL generalizes less sparse setting. As Table shows, fewer points (350,300,250.200)
MSE increases when fewer sample points are used. However, SDL is able to provide much more accurate gradients even it is trained under new graph with different properties. Thus, the results support that SDL is able to generalize the setting.
Table 9: Mean squared error ()0-3) for approximations of directional derivatives of function f2(2, y) = sin (x) + cos (y) with different sparsity.
Method
350 Nodes
300 Nodes
250 Nodes
200 Nodes
FinGrad
2.88 + 0.11
3.42 + 0.14
3.96 0.17
4.99 0.31
SDL
1.03 + 0.09
1.14 + 0.12
1,40 + 0.10
1.76 + 0.10
15


Reviews:
[{"review_id": "S1gtmEPptS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to reduce numerical error when predicting sequences governed by physical dynamics. The idea is that most physics simulators use finite difference differential operators, and the paper adds trainable parameters to them (the derivative and Laplacian operators). The added parameters are computed by a GNN. The paper concats the original graph feature and the output of the differential operators, and inputs them to a recurrent GNN to obtain the final prediction. \n\nI think the idea is interesting. Incorporating modulated derivative and Laplacian operators into physical simulators is novel and well justified. It could strengthen the argument is there is more justification of why this particular parameterization is selected. \n\nI think the experimental evaluation is somewhat adequate. There are a good selection of baselines including both manually designed iterators and GNNs. In particular, the weather prediction experiment show improved performance over several baselines. I am not familiar with this task or its state-of-the-art performance, but I am convinced that the proposed approach is superior compared to the claimed baselines (RGN, GRU). \n\nI have several confusions or concerns about the synthetic experiments\n\n1. In the synthetic experiments, is the evaluation task different from the training task? It is unclear from the description how well the learned parameters generalize. Does the method generalize to a. New functions/dynamics b. New graphs with similar properties (e.g. another graph draw from the same distribution) c. New graph with different properties (e.g. more or less sparse)? \n\n2. One short-coming of the synthetic experiment is the lack of error bars, or analysis of statistical significance. I think some of the improvements are not large enough to be statistically convincing without additional analysis. It seems necessary to experiment on multiple random problems (e.g. with random meshing, dynamics parameters). \n\nMinor comments:\n\nA related idea is “Learning Neural PDE Solvers with Guarantees” which modulates the finite difference iterative solver with deep networks, but the objective is solving PDEs with known dynamics instead of prediction with unknown dynamics. Conversely, the method the authors proposed seem also useful for speeding up PDE solvers. \n\nI think there is an error in the type definition of f and F in section 2.1. The two claimed types contradict each other.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByetFVF0tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper considers the problem of predicting node and/or edge attributes in a physics-governed system, where we have access to measurement data on discrete sensor locations. In contrast to other methods, they explore the usefulness of a Spatial Difference Layer (SDL) that learns a representation of the gradients on the edges and the Laplacian on the nodes, where the parameters to create those operators are learnable weights. The SDL layer is concatenated with the original graph and fed into a recurrent graph network (RGN) to predict node and/or edge properties. \n\nStrengths:\n- This research is very relevant for physics inspired machine learning, since many physical systems are governed by underlying differential equations.\n- The authors show on synthetic data experiments that SDL is capable of representing the derivatives of a physical system.\n- Real world use case for temperature prediction is presented with encouraging results.\n\nWeaknesses:\n- While comparison to RGN represents a rather strong benchmark, it would be interesting to see a comparison to a graph learning model that is specifically designed for weather forecast.\n- Just adding the Spatial Difference Layer using numerical methods (method RGN(StardardOP) and RGN(MeshOP)) can diminish prediction power for a long time horizon. This result suggests that those gradients might not help the prediction.\n- The inclusion of an h-hop neighborhood is not quite clear. What value for h was used in the experiments? Is this really necessary, when RGN by itself propagates the signal to neighbors that are further away?\n\nAdditional comments:\n- 2.1 and 2.2: On first reading, it's a bit confusing why there are two different equations for (∆f)i. The motivation of the second equation should be made more explicit.\n- 3.1 lacks explanation of what is train / test set, which is given only in the appendix. This is critical information to understand the use cases of the model and should definitely be in the main body.\n- In 3.2 formatting of a(i), b(i) and c(i) is confusing. Why is the setup only similar to Long et al? It would be nice to point out the differences and explain why it wasn't exactly the same.\n- 4.1: Was the train/validation/test split done in contiguous segments? I.e. are the 8 months of training data January to August? How is the problem of learning different seasons handled?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgw7nNRKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose new architectures that simulate difference operators. According to my experience, this research is important since PDEs are the most commonly used form to represent known physical relationships in dynamical systems.  The proposed method has novelty in that it uses advanced machine learning architectures to estimate physically meaningful variables. The authors further investigate how to use the proposed spatial difference layer in two tasks. \nI would suggest improving this research on these aspects: \n1.\tThe proposed method should be evaluated on more datasets. The difference information is used in almost any real-world dynamical systems and thus it would be more convincing to show the effectiveness on diverse applications, e.g., object tracking, the variation of energy and mass across space and time.\n2.\tIt would be interesting to design a test scenario where governing PDEs are known. Is it possible for your method to uncover the relationship of gradients that govern the system?\n3.\tHow sparse the data is? It would be better to have an experiment where data is intentionally hidden to control the sparsity and then evaluate the performance.\n4.\tA side question: In real-world systems, the observations are not only governed by PDE, but also unknown noisy factors, missing physics, etc. Can your method handle the noisy data/labels?\n5.\tThe proposed method has some complex components. I would encourage releasing the code and the simulated dataset upon acceptance. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this research is novel and well carried out, so this is a clear accept. Please ensure that the final version reflect the reviewer comments and the new information provided during the rebuttal
Decision: Accept (Poster)