Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a message passing neural network to solve PDEs. As a graph neural network for solving PDEs, it proposes a few notable features, e.g. embedding in the encoder for equation-wise representation, a difference of inputs in the message function, and so on. The proposed method also adopts notable training methods to reduce error propagation. 
main_review: The paper is very well written with: (1) thorough and detailed literature review; (2) intuitive explanation of motivation; (3) clear methodology and approach used in designing the architecture; (4) extensive details in empirical study and design of experiment. Overall, it is a good paper  with strong motivation, clear objective and solid empirical results. It is worthwhile to mention that authors design the training approach, i.e. pushforward trick, to reduce the instability due to error propagation and allow the network to solve versatile tasks. 

summary_of_the_review: Overall, it is a good paper with a complete analysis of the design of architecture, training/testing approach and extensive experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper utilizes neural message passing to propose another neural solver for PDEs. 
 
main_review: I think the paper has a clarity issues and still in its early stages from that perspective. 

In particular, the METHOD section is very vague and not clearly explained. Notation are introduced without definition and no explanation for the background needed to understand the notation. I suggest you lead with an example and motivate from there instead of going the general route.

Also, it is not clear what is the difference between using GNN and other regular networks in solving PDEs--the authors did not mention that in the paper.
 
summary_of_the_review: The paper has a clarity issues and many notations are not introduced or defined.

--Edit-- The authors either explained or addressed the issues in the paper with other reviewers. I am happy to recommend the paper for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the stability of training neural PDE solvers. It uses the "pushforward trick": the adversarial-style loss is used by perturbing the input at step $k$ by a certain noise (note, that this idea looks similar to denoising autoencoders). Another trick is to predict K steps simulaneously in time. The architecture itself is graph-based: the nodes are distributed through the domain, and it is an encoder-processsor-decoder architecture. This helps to be closer to classical numerical solvers. The experiments show that the pushforward trick helps to increase "survival time" (which is defined as a time when a certain error exceed the threshold) clearly helps.


main_review: The ideas of the paper are interesting: the pushforward trick is appealing. The architecture, however, looks like a separate idea, which is not connected to the loss. I.e., the paper proposes two separate innovations, puts one into the title, but experiments confirm the effectiveness of only one. But I like the comparison with the standard numerical solver. However, the study of the architecture lacks the analysis of the systematic convergence of the solution, given by the new architecture, with the number of parameters. Classical solvers do have this property: increase the grid size, get better accuracy. Does the proposed model help with this respect? 

Comments:
- There is no ablation study (the loss or the architecture?)
- The dependence of the properties of the method on the number of parameters in the model is not studied (at least, comments are welcome).

summary_of_the_review: Reasonable idea, which proposes a loss that improves stability; however, it has two disconnected contributions and is not so easy to read. Some of the important points are not discussed/analyzed experimentally.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a framework to solve generic PDEs written in conservation form, in an autoregressive fashion. The grid domain is considered as a graph, whose nodes contain information about solutions to the PDE until a given times steps, node positions, time steps, and hyperparameters information about the PDE. The node’s labels are first embedded in a latent space. Then message passing is applied and the output is decoded to predict the next n time steps solutions together.
The network is trained with an additional loss which promotes zero stability of the solver, by constraining the network to be invariant to small perturbations in the input solutions of previous time steps.
The approach is validated on a parameterized family of 1d and 2d PDEs, showing accuracy and generalization results that are above the state of the art of neural solvers and classical solver schemes. In addition, the paper show promising results dealing with irregular domains with different types of boundary conditions.

main_review: ##########################################################################

Summary:

The paper proposes a framework to solve generic PDEs written in conservation form, in an autoregressive fashion. The grid domain is considered as a graph, whose nodes contain information about solutions to the PDE until a given times steps, node positions, time steps, and hyperparameters information about the PDE. The node’s labels are first embedded in a latent space. Then message passing is applied and the output is decoded to predict the next n time steps solutions together.
The network is trained with an additional loss which promotes zero stability of the solver, by constraining the network to be invariant to small perturbations in the input solutions of previous time steps.
The approach is validated on a parameterized family of 1d and 2d PDEs, showing accuracy and generalization results that are above the state of the art of neural solvers and classical solver schemes. In addition, the paper show promising results dealing with irregular domains with different types of boundary conditions.


##########################################################################

Reasons for score:

 

In my opinion, the paper brings significant advances to the field of neural PDE solvers. The proposed push forward trick, together with the temporal bundling trick, are valuable contributions as they impose favorably condition on autoregressive solvers (zero stability), improve the accuracy of other existing approaches as [Li et al 2020 a] and the inference time of the solver.
The approach is well evaluated experimentally,  showing accuracy and generalization results that are above the state of the art of neural solvers and beat classical solver schemes in inference time.
For these reasons I consider the manuscript to be accepted at ICLR 2022.


 

##########################################################################Pr

 Pros:

1. The paper tackles the important task of solving many families of PDEs, obtaining accuracy and generalization results upon the state-of-the-art of neural solvers.

2. The approach is well motivated as classical solver schemes (FDM, FVM, WENO) are instances of message passing, as the authors themselves stated.

3. The pushforward trick is by itself a valuable contribution as it enhances the performance of other existing neural solvers ([Li et al 2020 a]) as the authors show in Table 2.

4. The paper is clear and well written, providing sufficient background for people not experts in the field.


##########################################################################

Cons and questions:

1. It is not entirely clear to me how the network can generalize to multiple spatial resolutions: more specifically how the encoder and the message passing mechanism handle a varying number of nodes (i.e. cells) in the graph? The number of cells stays fixed but the input positions vary? If this is the case, please clarify this in the experimental section.


2. In the experimental section it would be good to provide more details regarding the time needed for training the model and the number of parameters of the network (also in comparison with [Li et al 2020 a])


3. The paper should fit within the 9-page limit. I suggest that the background section could be shortened a bit, moving some of its parts to the Appendix.


##########################################################################

Questions during rebuttal period:

 

Please address and clarify the cons above

 

#########################################################################

I spotted some typos:

(1) Figure 1(right): reduces -> reduce

(2) Page 5 Decoder section: an temporally -> a temporally

(3) Page 9 Experiments and result section: MP-PDE solvers obtains -> MP-PDE solvers obtain

(4) Page 9 section 4.4:  acccurately -> accurately

(5) Page 10 section 7: many diciplines-> many disciplines

(6) Page 10 section 7: paving a way -> pave the way


summary_of_the_review: In my opinion, the paper brings significant advances to the field of neural PDE solvers. The proposed push forward trick, together with the temporal bundling trick, are valuable contributions as they impose favorably condition on autoregressive solvers (zero stability), improve the accuracy of other existing approaches as [Li et al 2020 a] and the inference time of the solver.
The approach is well evaluated experimentally,  showing accuracy and generalization results that are above the state of the art of neural solvers and beat classical solver schemes in inference time.
For these reasons I consider the manuscript to be accepted at ICLR 2022.
review_title: null

Meta-review: This paper proposes a message passing neural network to solve PDEs. The paper has sound motivation, clear methodology, and extensive empirical study. However, on the other hand, some reviewers also raised their concerns, especially regarding the lack of clear notations and sufficient discussions on the difference between the proposed method and previous works. Furthermore, there is no ablation study and the generalization to multiple spatial resolution is not clearly explained. The authors did a very good job  during the rebuttal period: many concerns/doubts/questions from the reviewers were successfully addressed and additional experiments have been performed to support the authors' answers. As a result, several reviewers decided to raise their scores, and the overall assessment on the paper turned to be quite positive.
Decision: Accept (Spotlight)