Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes a new graph neural net (GNN) method to learn the dynamics of a spatio-temporal PDE-driven dynamical system directly from data.
main_review: This manuscript proposes a new graph neural net (GNN) method to learn the dynamics of a spatio-temporal PDE-driven dynamical system directly from data. The authors propose to do that using the finite element method (FEM). The proposed method builds on using: basis function approximation for the (unknown) field u, Galerkin method with the assumption that discrepancy between the dynamics F and basis function approximation is orthogonal to (finite) basis functions, method of lines, and message passing GNN as a proxy for the dynamics. The use of linear interpolation allows to express the time derivative of $Y$ as a solution of a system of linear equations, which is further approximated to gain computational additional efficiency. Authors also propose a method to incorporate inductive bias into model learning for models that are assumed to contain a convection component. Overall the proposed method is well-motivated, and for the most part the description is clear. To my knowledge, the proposed method is novel and contains some methodologically new ideas, and the performance seems to be on par with previous methods that learn free-form dynamics, and shows an improvement for models that contain a convection component when such prior knowledge is utilised in the model training. Authors could address and/or clarify the following aspects:
1. I understand that a piece-wise linear basis simplifies computational complexity by making some of the computational steps straightforward, but on the other hand the selected basis is the simplest and obviously not optimal from approximation accuracy point of view. Can the method be extended to other basis? For example, if we knew that the dynamics $F$ contains a diffusion term $\nabla^2u$ we would not be able to introduce it since the second derivative of PWL functions is zero everywhere. I think the discussion of possible limitations of the PWL basis and of possible extensions to higher-order bases is missing.
2. It seems that the measurement from the initial time point is used as the initial state? Why not introduce a separate free parameter for the initial state? 
3. As described towards the end of the manuscript, the system is initialed with an initial state and then the PDE dynamics define how the system evolves over time. However, below eq. (12) it is noted that "where $u$ encodes the data $Y$ at time $t$ in its coefficients." Is $u$ defined based on the dynamically evolving system state or by data? 
4. Paragraph "Network architecture" on page: scimitar-learn is used to compute inner product between basis functions. Provide a brief description of how the computation is done. 
5. End of page 6: gradients are back propagated through an ODE solver. Why not use adjoint method (possibly with checkpoints) as proposed in previous work and implemented e.g. in comparison methods? Discrete back-propagation may not scale to longer sequences. Is this the reason why data trajectories are shortened? 
6. Table 1: split the methods into two groups. Group 1 should includes PA-DGN, GWN, CT-MPNN and FEN, which do not assume prior knowledge about system dynamics. FEN performs similarly with CT-MPNN on ScalarFlow and perhaps slightly better on Black Sea data (MAE values within one std). Group 2 should include only T-FEN that is specifically designed to learn convection systems, and thus provides a small performance improvement. 
7) In experiments, the authors make their models time- and position-dependent while the strongest baseline models (CT-MPNN) does not utilize neither time nor positions. That makes it hard to tell whether the improvements in performance of FEN and T-FEN are due to the models's structure and inductive biases or due to time and position dependencies. Authors should provide an ablation study to address this.
8. Provide additional comparisons on systems with larger variety of dynamics using simulated data (for which the ground truth is known) to better understand when FEN performs better/worse than comparison methods. 
9. Datasets are subsampled to 1000 spatial points. Provide results for smaller and also larger spatial grids to demonstrate the argument that "approximation becomes arbitrarily good as the mesh resolution increases".  
summary_of_the_review: To my knowledge, the proposed method is novel and contains some methodologically new ideas, and the performance seems to be on par with previous methods that learn free-form dynamics, and shows an improvement for models that contain a convection component when such prior knowledge is utilised in the model training.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a graph / simplicial neural network based on the finite element method for learning dynamics from data when only a finite number of samples exist and the true dynamics are not known or only partially known.  
main_review: # Strenghts

- The paper looks at a very realistic setting for learning PDEs from data: a finite number of samples and (partially) unknown true dynamics. Tackling these problems concomitantly is of great practical importance and this makes the proposed method relevant for practical applications. 
- The introduction does a good job motivating the work and pinpointing the main challenges of learning dynamics from data. 
- As someone with little experience with the finite element method, Section 2.1 does a great job explaining the required background in the right amount of detail for understanding the paper.
- The connection that the paper makes between the finite element method and message passing neural networks is interesting and, to the best of my knowledge, original. 
- The authors show how inductive biases can be added to the model by using a certain prior over the structure of the function $F$. 
- I like that the paper focuses on real-world datasets. Also, the datasets themselves are extremely interesting to visualize and make the paper more interesting. 
- Figure 3 provides a very insightful qualitative understanding of the proposed model compared to the baseline. 
- I am glad that the paper includes a super-resolution experiment. Often, models that work with a discretised space can be very sensitive to changes in the resolution of the mesh. Figure 4 shows that the proposed model is relatively robust to changes in the number of triangles. 
- The authors show (in a relatively specific setting) that factorizing the dynamics achieves a disentanglement effect which allows some degree of interpretability of the model. 

# Weaknesses 

- While I appreciate the focus on real-world datasets, a synthetic experiment where the model could have been evaluated in a more systematic way would have been useful. 
- The trick to stabilize training described at the end of Section 3 is slightly peculiar. How important is this trick? Do the authors have any results for when this trick is not used? Could this trick improve the performance of the baselines as well? 
- The importance of the approximation from Equation (13) is not studied. Perhaps that is something that could have been tried in the more controllable synthetic setting I was suggesting above. In general, I would be interested to know what are the costs of this approximation and if more advanced approximations might be worth being considered to boost performance.
- Minor suggestion: The paper frames the model as a hypergraph neural network. However, the authors might want to be aware that there is a recent line of work developing simplicial and cell complex neural networks: https://arxiv.org/abs/2103.03212 (ICML 2021), https://arxiv.org/abs/2106.12575 (NeurIPS 2021), https://arxiv.org/abs/2010.03633. Since the model learns a function over the 2-simplices in the simplicial complex, the model is probably more accurately described as a type of simplicial neural network.   
summary_of_the_review: The weaknesses reported above are relatively minor and far outweighed by the strengths of the paper. Therefore, I recommend the paper for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new model for learning partial differential equations from data. The PDE is first discretized then solved as an ODE. The dynamics function is learned with Message-Passing Neural Networks, where the function is split into a sum of physically informed terms. This splitting both improves model performance and makes the model more interpretable by disentangling the dynamics. The model is tested rigorously against multiple baseline models, and the results show the new model performs well.
main_review: Strengths:

- The paper is well written and well presented. On the whole it was relatively easy to understand and the diagrams definitely contribute to that.
- The proposed model is well motivated and nicely extends ideas such as Graph Neural ODEs to the PDE domain.
- The paper considers the issues with a naive implementation in depth, (which is that the model would be slow). It then provides the solution to this.
- Rigorous tests on the proposed model are carried out against robust baselines.
- There is an extensive review of related work.
- A lot of effort has been put in to make the results reproducible, including all experimental details and code/datasets to come.

Weaknesses/Questions:

- My main concern is that training has been carried out over 10 time steps, was this a hyperparameter that was tuned? I agree that the correlation over say 30 steps will be minimal, however my understanding is that all models learn an update based on the current state and potentially a few steps in the past. Would it not still be possible to train over more time steps? Could T-FEN extrapolate better than GWN if it is trained on a larger time range?
- One of the proposed reasons GWN outperforms T-FEN at extrapolation is that it can use the past time-steps as input. This could be extended to T-FEN in the form of delay differential equations (https://arxiv.org/abs/2102.10801), where the dynamics takes the state at $(t)$ and the state at $(t-\tau)$ as input. This would make the model use the past as well. Would it be possible to carry out an ablation study using this?
- The paper could benefit from a discussion section. Saying what the model is good at and bad at. For example, we see that it performs very well in the times used to train, but is not as good at extrapolation.
- In table 1, for the ODE based method, the number of function evaluations are provided to show T-FEN is faster than FEN and CT-MPNN. Is it possible to time the evaluation to support this claim further? That way one could also compare to the other two baselines GWN and PA-DGN.
- Is it possible to move the related work section, either to the introduction or to just before the conclusion? It breaks the flow.
- What is the reason for using the $L_1$ loss over the $L_2$ loss? Could we expect better/worse results with $L_2$?
- Could the model be improved even further by using more physically informed terms? For example, like those in electro-magnetism $\dot{v}=qv \times B$, where $B$ is learnt (https://arxiv.org/abs/2109.07359). Or, because the experiments use data of fluids, could it help to include Laplacian or curl terms in the dynamics, which appear in Navier-Stokes? Is it possible to carry out an ablation? 
- The mass matrix $A$ is approximately inverted by lumping the matrix, are there situations where this approximation could lead to errors?
- Is it possible to extend this method where we expect higher order PDEs, that include terms such as $\frac{\partial^{2}u}{\partial t\partial x}$?
- The model disentangles the dynamics into a convection term and the remainder. How does this relate to disentangling the dynamics of an Augmented Neural ODE (https://arxiv.org/abs/1904.01681) into a velocity and an acceleration (https://arxiv.org/abs/2006.07220), which can also aid the interpretability of the ODE?
- I don’t entirely understand the Black Sea dataset. Is the mean temperature the mean temperature of the entire sea or of a region, say $1m^2$? If it is of the whole sea, does this not remove the spatial element of the task? Additionally are there any interesting effects appearing over long time periods due to global warming/concept drift? Given that the training regime is taken over 2012-2017 and the testing regime is in 2019?

Minor Points:

- The paragraph just above equation 15 has a small mistake. The manuscript says “This would prohibit training even with the adjoint equation (Chen et al., 2018)”, while the paragraph is about the speed of training. The benefit of the adjoint method is that it is memory efficient but slow compared to directly backpropagating through an ODE solver, which is fast but uses a lot of memory.
- There is a typo at the bottom of page 8, “disappears Extrapolationat the”
- There is a typo at the bottom of page 16, “ODE-base models” should be “ODE-based models”

summary_of_the_review: The paper is well written, the model (to my best knowledge) is novel, with the method building on existing work. The experiments rigorously test the model against the necessary baselines and information is given in the appendices on reproducing the results. Therefore I recommend acceptance, with a few clarifications to be made.

**EDIT** I have increased my confidence score from 3 to 4 after my initial questions have been answered.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The author proposes a method for forecasting in Partial Differential Equations by coupling Finite Element Method on an arbitrary grid with the learning of the dynamics from data. For this purpose a variant of message passing based graph networks is used. It is show in the paper that it's possible to incorporate priors on the structure of the PDE that results in an interpretable solution. The model also show more stability to changes of the mesh structure in test time (like superresolution) and to extrapolation than competitors.

main_review: The paper uses Message Passing Neural Networks to implement a Finite Element Method with learnable dynamics. It combines different models and techniques from the literature, but it clearly does that in a nontrivial way including modifications, therefore it is more than just derivative work.

I find it valuable that the author also considers performance of implementation, and discusses consideration on GPU architecture related performance issues. Also the effort put into making the work reproducible adds value to the paper.

The method can clearly have practical value, and the discussion of the method is clear and quite detailed. I am not as familiar with the PDE/NN literature as for example the Neural ODE literature, therefore I cannot rule out entirely that something similar already exists. 

Questions / actionable comments:

I)
"The results show that FEN and T-FEN provide the smallest prediction error on both datasets with a further boost due to the separate transport term in T-FEN."
I feel this is a bit too strong statement. While I tend to accept the paper with the results in Table 1 given the other clear benefits of the method (like robustness and interpretability) I am not comfortable with this statement. The table is based on 3 repeats, and have results for example:

Black Sea dataset  CT-MPNN 0.944 ± 0.003  vs.  FEN 0.938 ± 0.005, 

these confidence intervals are clearly overlapping and using 3 samples I am a bit skeptical. This is one of the strongest gaps in the table.
In the ScalarFlow FEN is really on par with CT-MPNN. Did you used a statistical test to bold the results?

Again, I have no problem supporting the paper even if the method is on par with the best competitor, but the statement in the paper should be supported by the statistics. 

Similarly, in the case of NFE CT-MPNN have approx +-60 deviation on the ScalarFlow dataset, I am not too comfortable to compare these numbers.

II)
What is the motivation of using L1 loss? instead of like MSE?

III) In Experiments/Multi-step Forecasting section:
Time horizon choice is well motivated, but still feels arbitrary in some sense. Does the author see some way to formalize what we accept as "meaningful dynamics"?
How one should choose a comparison horizon for example. Can it be done at least approximately without domain knowledge on the system? 

IV) In Model/Network architecture section: 
Order invariance of cell vertices is assured by ordering the nodes canonically. Would the author expect improvement if a permutation invariant network, like a Set Transformer be used here? Why or why not?
In some sense the message aggregation step, summation being permutation invariant, is a set network, but not a very expressive one. This  however makes the cell order invariant for a given node and not the other way around. 

summary_of_the_review: The paper give valuable contribution, the method expected to be practical, robust, and in some cases interpretable. I find the statement on raw prediction error overly strong.
review_title: null

Meta-review: This paper introduces a graph neural network (GNN) based on the finite element method (FEM) for learning partial differential equations from data. The proposed finite element network is based on a piecewise linear function approximation and a message passing GNN for dynamics' prediction. The authors also propose a method to incorporate inductive bias when learning the dynamical model, e.g. including a convection component.

The paper received three clear accept and one weak accept recommendations. The reviewers discussed the possible extensions of the method, and also raise several concerns regarding experiments, e.g. the added value of a synthetic dataset, implementation tricks or hyper-parameter settings. The rebuttal did a good job in answering reviewers' concerns: after rebuttal, there was a consensus among reviewers to accept the paper.

The AC's own readings confirmed the reviewers' recommendations. The paper is well written and introduces solid contribution at the frontier of GNNs and finite elements methods, especially a pioneer graph-based model for spatio-temporal forecasting derived from FEM. Therefore, the AC recommends acceptance.
Decision: Accept (Spotlight)