Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use a message passing neural network to predict solutions of both linear and nonlinear PDEs. The proposed method uses domain invariant features with graph based method for representing the temporal data of PDE. 
main_review: The paper is well organized with intuitive explanations of the network design approach and detailed experiments. The strength of the paper is on adopting graph neural networks. However, based on my understanding, there are a few weaknesses (and questions that could help reviewers understand the paper better).

1. As mentioned by the authors, the domain invariant feature representation is the major contribution and is inspired by PDE solvers. However, except for highlighting the approximation by using the neighboring points in the domain, it either lacks more in-depth analysis or lacks novelty as, for example, using finite difference etc, in a neural network for PDEs is a not unknown approach. Could you be more specific with more details on the novelty?

2. Section 5 introduces the proposed network with nice illustrations. Is it just a typical architecture of message passing neural network with some fully connected layers. Could you please be more specific on the novelty on the structure? As authors mention in section 6.1 that different domains were tested to show the generalization capability, could you be more specific on which part of the network that would allow good generalization to be domain invariant? 

3. The experiment section is well written with detailed explanation and visualization. However, there are a few questions:

3.1 What is the scale of the data, i.e. $u_i$? Without reference, I find it is hard to measure the claimed MSE that was achieved.

3.2 What is the training algorithm, or in other words, the optimizer chosen for training the network? 

3.3 What is the baseline that the proposed method is comparing with? How about other networks that was mentioned in the related works? Lacking these two important reference makes it hard to measure the performance of the proposed method.

3.4 While the result visualization is nicely presented, without detailed numerical presentation, I am not sure how well the performance is and how efficient the training is? 

Some missing references:

[1] Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations

[2] Solving high-dimensional partial differential equations using deep learning

[3] Finite Difference Neural Networks: Fast Prediction of Partial Differential Equations
summary_of_the_review: To summarize, this paper basically applies message passing neural networks to solving PDEs. I strongly recommend authors to present motivations and design of networks with more details and analysis, and the experiment requires improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method for learning mesh-based simulation of partial differential equations using graph neural networks (GNNs). The authors train the GNN to match a reference solution and demonstrate that the trained GNN can generalize to solving PDEs on different domains. Additionally, the authors show a recurrent variant of the network that can solve time-dependent PDEs. The method is shown to work on a number of test problems, including advection-diffusion and Navier-Stokes equations. Overall, the authors show that GNN-based simulation is a promising approach for improving the flexibility and generalization of learned PDE solvers.

main_review: **Strengths**

- The method is described fairly clearly and is easy to understand: applying a GNN to learn solutions to PDEs.

- The authors evaluate their approach on a number of test problems and show that it can generalize to unseen domains.

- The approach also extends to time-resolved simulation.

**Weaknesses**

- To me it seems that the main contributions of this work have already been demonstrated. Pfaff et al. (ICLR 2021) have already demonstrated a GNN for solving a wide array of simulation and PDE problems. The architecture of Pfaff et al. is generally similar to the proposed approach, they demonstrate generalization to unseen domains, and the experimental results go beyond that what is shown in the proposed method (including 3D, adaptive remeshing, and dynamic simulation domains). The authors should cite this work and explain if or how their contributions differ.

Pfaff, Tobias, et al. "Learning mesh-based simulation with graph networks." In Proc. ICLR 2021.

- The proposed recurrent approach to solving time-dependent PDEs is not trained with a recurrent loss. Thus the authors note that the MSE increases with increased unrolling steps. This is an open challenge for learned PDE solvers. Usually it is difficult to incorporate an unrolled loss because of the significant memory required to backpropagate through the GNN for multiple time steps. Is it possible to train the proposed method with a recurrent loss, or is that computationally prohibitive?

**Misc.**

- The authors note that there is a periodic boundary condition for the Navier-Stokes equations they solve. Do they incorporate this into the mesh they use in any way (e.g., using a periodic mesh to help the network learn these boundary conditions)?

summary_of_the_review: While the paper seems generally well written and the method appears sound and thoroughly evaluated, I'm not sure what the novelty of the paper is given that the contributions overlap significantly with Pfaff et al. (2021). Moreover, the authors do not cite this paper or explain how their work differs from it. Thus, my rating is to reject the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a graph neural network approach by using a message-passing model to find efficient PDE solvers. It further shows that the trained solver can find a temporal sequence of PDE solutions in different domains. The proposed method is a nice idea and has been well verified. However, given the very limited scope of application in this paper, only toy environments with high-resolution numerical data sets can be used, more detailed ablation and comparison experiments may be required to ablate the model in a more refined manner.
main_review: Strengths

This paper introduces a locally invariant feature representation inspired by classical PDE solvers to efficiently learn a differential operator solver. The method can obtain robust high-performance models for various linear/nonlinear PDEs with arbitrary spatial discretization and arbitrary physical domains. 

Weaknesses

Some detailed instructions and experiments need to be added, such as how to construct meshes, how to construct boundary conditions, and sensitivity analysis of prediction results to data sets. 

Comparisons with other methods need to be added.

Some examples in irregular computational domains can be added.

I would love to hear some limitations of the proposed method.

summary_of_the_review: This paper proposes a new method, which is innovative, but only shows some simple experiments and lacks detailed discussions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A message passing graph neural network is used to predict the time-dependent solution to a family of PDEs. The PDE parameters, initial conditions, and boundary conditions are input to the network, an it outputs the solution 1 time-step into the future. The authors show how to train such a system on the Heat equation, Navier-Stokes, and advection-diffusion equations. The flexibility of the message passing graph neural network permits its deployment on a diverse set of geometries and boundary conditions.

main_review: Advantages

- The submission until the experimental section is well-written, easy to follow, and very clear. The written-style could include more PDE fundamentals for the uninitiated reader, but I think on the whole the authors have chosen for a clearer and simpler-to-follow read.

- The proposed GNN architecture makes a lot of sense. Using a GNN on the mesh is intuitive and the particular features used are also sensible. The particular from of edge embedding used also makes the GNN locally shift equivariant. Furthermore, encoding the PDE parameters in the edge features allows the model to generalise across PDEs within a given class of PDEs.

- The experiments demonstrate that the MPGNN can learn to make 1-step PDE prediction on multiple geometries. This is evidences in Figures 2 and 3.

Queries

- Just under equation 2: The argument that F(u, \nabla u, …) contains local spatial derivatives an only local neighbouring values and so you only need a graph neural network operating on local neighbourhoods makes sense. I was wondering, what are the limits on this assumption? Does the solution space have to be analytic/smooth? I’m guessing the maximum derivative order and accuracy of approximation will also influence the size of neighbourhoods you should take.

- Section 5: do you only use 1-step neighbourhoods in your model? I could imagine longer range message passing is important in PDEs with higher-order derivatives.

- Experiments: It is not obvious to me what is going on in your experimental setup. I would not be able to reproduce these experiments. Specifically:

— Does the MPGNN output 1 time frame at a time, but it takes in 4? 

— Why are the 4 input frames separated by 20 \delta t each? 

— How are the coordinates for the mesh chosen? 

— From which distribution do you sample initial conditions and boundary conditions?

— Which optimiser do you use?

— Is the loss a 1-step loss?

— Do you predict values on the boundary or not?

- If you run the learned solver in recurrent mode, the error appears to increase over time as in Figure 4. c). Have you checked to see what happens over long rollouts? Does the error increase gradually or blow-up exponentially? I think it would be important to consider the long term error behaviour, since this is very important for time-dependent PDE solvers.

- Baselines: I do not see any baselines. Without these it is difficult to make sense of the number in the submission. Sensible baselines could have been finite difference method, finite volume method, spectral method, off-the-shelf software, or other neural methods, such as the Fourier Neural Operator, or the method of Bar-Sinai et al. (2019).

- What is the connection between this work and “Learning to Simulate Complex Physics with Graph Networks” (Sanchez-Gonzalez, 2020)? Here, the authors also use MPGNNs to solve complexity physical systems, driven by PDEs.

Minor notes

- Just under equation 2: n is a superscript, not subscript

- Heat equation: Please qualify what Firedrake is. Is is a software package? This is not indicated in the text.


summary_of_the_review: The method is well laid out and written with enough clarity that the setup is understandable and intuitive. The method makes sense given the problem and is a logical step forward in this space. The experiments, however, are insufficient to back up the claims laid forth in the first section of the paper. I do not think I could reimplement the experiments based on the writeup and I do not see a separate supplementary material, where I may have missed these details (please correct me if I have overlooked them). Furthermore there is no comparison with any baselines (I’ve mentioned a few sensible ones in the main review), neural or classical, so it is hard to say how significant the numbers in the submission are. There are also no ablations for the reader to understand the impact of each design choice the authors make. It is mainly because of the experimental section that I recommend a reject for this submission.
review_title: null

Meta-review: Thank you for your submission to ICLR.  While all reviewers felt that there were some interesting aspects to the proposed work, the consensus was also that the work didn't properly situate itself within the existing literature on related methods.  In particular, I agree with Reviewer kLFD that a numerical comparison to Pfaff et al., is notably missing here; while the authors did provide qualitative comparisons in their discussion, it's not clear to me that these differences are ultimately that significant, and the methods need to be compared directly if a case is to be made for the advantages of the proposed approach.
Decision: Reject