Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper develops a set of benchmarks consisting of datasets and metrics for physics-informed graph neural networks (GNNs) for dynamical systems. In particular, they examine and compare the efficacy of Hamiltonian and Lagrangian GNNs and graph neural ODEs in terms of rollout error, and energy and momentum violations for scenarios including n-pendulum and n-spring systems. They provide the code to generate the datasets, train the models, and evaluate the metrics.
strengths: 1. Careful comparison and evaluation of multiple different physics-informed GNN models plus the impact of external constraints
2. Development of a standardized set of dynamical systems (n-pendulum and n-spring), with extrapolation beyond the training domain
3. Examination/explanation of relative strengths of different GNN models and impact of external constraints
weaknesses: 1. Other dynamical systems (like n-bodies in a gravitational field, etc.) would have been interesting to include
2. No discussion of how to incorporate measurement noise or bias or if it’s at all possible in any of these frameworks

correctness: Yes, the claims appear to be correct and the dataset is constructed in a sound way. The evaluation methods and experiment design are also appropriate and performed correctly.
clarity: Yes, the paper well written, but can benefit from another pass of grammar and spell check and minor corrections to formulas.
relation to prior work: Yes, the relation to prior work is discussed. This work differs in that it compares different architectures on a fair footing. However, there can be some additional citations to, e.g. https://arxiv.org/abs/1612.00222 
additional feedback: Additional feedback is as follows. In general, try to re-run grammar and spell checker.

Content

* L94 Could you clarify why constraints must be of the form $A(x)\dot{x}$ = 0 ?
Eq. (4) Missing closing right parentheses somewhere. Please check and correct.
* L99 from Eq. 2
* L103 In this paper, is an explicit time dependence for T and V or an explicit x dependence for T ever considered, i.e. $T(x, \dot{x}, t)$, $V(x, t)$? If not, can this be simplified to $T(\dot{x})$, $V(x)$ already? Or is the point that the model may not be constrained in this way in some situations?
* Eq. (7) The notation $\nabla_{\dot{x}\dot{x}}$ is confusing. Does this mean a double gradient or d'Alembertian, i.e. is it a vector again or a scalar?
* L113 This is a bit repetitive as it was already explained that $\mathbf{M} = \nabla_{\dot{x}\dot{x}} L$
* L119 Is there a sign error here? I think I get the correct equations if I assume $J = [0, I; -I, 0]$. Please check.
* Eq. (12) Choose one of $D_z$ or $D_Z$ for consistency
* Eq. (13) Correct error as stated in Supplementary material
* L312 Could this also be related to Noether’s theorem?
* Eq. (18) (Supplementary) Is this correct? If I naively differentiate Eq. (17), I get $(x_i - x_{i-1})(\dot{x} _i - \dot{x} _{i-1}) = 0$, which does not seem to reduce to Eq. (18)

Language/style

* L39 (HNNs)
* L40 Langrangian or Hamiltonian
* L52 they can potentially learn
* L61 based on the LNN, HNN, and NODE frameworks
* L159 improvises does not seem like the correct word here
* L183 Two sentences are repeated twice unnecessarily
* L437 (Supplementary) “Pfaffian” should not be in math mode
review_title: Valuable new benchmark for physics-informed GNNs, with some improvements possible

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors compare several existing variations of physics-informed GNNs on the problem of learning dynamical systems in a data-driven way. 
strengths: - A comparison of various physics-inspired GNNs is interesting. 
- I like the evaluation metrics used in the paper. 
- Efficient implementations in JAX are available. 
weaknesses: - The scope of the paper is very narrow. It looks at pendulum and spring systems. What about other types of systems that GNNs have been used for? Also, the models investigated represent a small subset of all existing surrogate models / learning from dynamical systems models. 
- The paper is neither a benchmark paper (it doesn't show how to use the benchmark, or how new methods could be evaluated using the benchmark) nor is it a dataset paper. It doesn't provide enough information on meta-data, data format, API access, maintenance, versioning, etc. of the data. 
- Access to the data is through an anonymous repo. Since this is the benchmark and dataset track, I would have expected a repo with DOI, maintenance plan, versioning, etc. 
- There seems to be crucial information missing from the main paper. For instance, what specific message-passing GNN was used as the base model?
- One weakness of ML-based models is that the values of hyperparameters matter a lot. For instance, in recent work on GNNs for dynamical systems, it was shown that adding noise to the input at each iteration of really important. The paper doesn't discuss and describe enough the choices made here and the search performed for the hyperparameters used. The appendix lists some of these, but how are they chosen? Randomly?
- The discussion/summary of the findings is cut short and disappoints. What have we now learned from these results? what matters? To have more space one could move a lot of the formulas to the appendix. 
correctness: There are some typos in the manuscript (I later saw in the appendix it was corrected). other than that, the paper seems correct. I have not checked the implementations though. 
clarity: The clarity of the paper could be improved. Details are missing (see above) and for non-expert readers, it might be really difficult to understand how GNNs and the various equations presented at the beginning of the paper are combined. The intuition of how physics-informed GNNs work is missing. Instead, a lot of space is "wasted" on the equations which could be moved to the appendix. 
relation to prior work: Related work is sufficiently discussed. What isn't clear to this reviewer: why not use existing dynamical systems datasets such as those used in prior work on physics-informed GNNs? Why "only" a comparison on spring and pendulum systems?
additional feedback: I am an advocate of taking a step back and benchmarking a large number of existing methods to truly understand "what's going on". However, the submission misses the mark here in execution due to its narrow scope, the suboptimal presentation, and the missing rigor in presenting the setup and results of the experiments.
review_title: Comparison of Physics-informed GNNs for spring and pendulum systems

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper evaluates different graph neural networks based systems for modeling dynamical systems. 

strengths: 1- The presentation of the problem, in general, is clear and well-motivated. 

2- The authors include three main models along with considering constraints. 

2- Including the preliminaries make the paper easier to follow.

3- The limitations and outlook paragraph in the conclusion is appreciated. 
weaknesses: 
1- The main observations of the experimental evaluation section are confusing and not presented properly. The section starts with mentioning that FGNN is inferior to all other models. Based on what results this conclusion is made? After that, it is mentioned that, like other works, LGN and CLGN were not trained for pendulum systems. Then, FGNN, LGN, and CLGAN are omitted in section 4 and then included in the Appendix. A justification to why these are not trainable is needed in the paper. When the results are included in the Appendix, there is no mention on how these models have become trainable. 


2- Since this is an evaluation paper for the goal of benchmarking already existing models that utilize neural networks in the form of MLPs, the impact of hyper-parameters such as number of layers, size of each layer, learning rates, and the type of activation functions (if used) need to be taken into account or at least discussed.


correctness: The claims of the paper sound correct.
clarity: The paper is, in general, clear. However, the observations and conclusions drawn from section 4 are not clear. 
relation to prior work: Yes.
additional feedback: Minor comments and suggested improvements:

1- The paragraph between lines 61 and 66 is a repetition of the contributions list. 

2- D is not defined in section 2.1.

3- In line 109, it should be "from L", not "from the L".

4- In Section 4.1, the sentence " All the simulations and training were carried out in
184 the JAX environment [22]." is repeated. 

5- A missing bracket in the denominator of the EE(t) equation in line 225.
review_title: The main observations in the experimental evaluation are not presented clearly. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this work, the authors rigorously demonstrate the utility of employing physics-informed graph neural networks for modeling complex dynamical systems. They do so by introducing readers to three main methodological approaches for representing system dynamics and investigating the behavior of these approaches in different physical settings and under physical constraints. Through their benchmarks, the authors present graph neural networks as a foundational architecture for modeling system dynamics, primarily for their inductive biases and subsequent generalizability. Notably, in an accessible manner, this work introduces readers to the field of physical modeling driven by deep learning.
strengths: This reviewer posits that the authors' contributions can be summarized with the following points.
1. The authors systematically analyze the performance of graph neural networks trained and evaluated on different physical systems.
2. The authors demonstrate that enforcing constraints improves the performance of all graph neural network types studied in this work.
3. The authors show that decoupling kinetic and potential energies also improves the performance of all graph neural networks studied.
4. The authors demonstrate that all graph neural networks investigated in this work can generalize to larger systems than the ones on which they were trained.
5. The authors provide readers with an accessible means by which to introduce themselves to the field of deep learning-driven physical modeling.
weaknesses: This reviewer's perceived limitations of this work are as follows.
1. The authors only conduct experiments on pendulum and spring systems. Rigid body systems, in all their complexities, are not considered in this work.
2. All constraints enforced in this work are specified, not learned.
3. The authors consider only standard variations of a graph neural network architecture. For example, transformer models and their graph locality-focused variants are not considered in this work, even though they have been shown to be equivalent in operations to graph neural networks operating on fully-connected graphs (Reference: https://graphdeeplearning.github.io/files/transformers-are-gnns-slides.pdf).
4. Contacts and deformations between entities in multi-body systems are not considered in this work.
correctness: The authors' claims throughout this work are correct. For example, their benchmarks make use of sound evaluation metrics and report averaged results across several experiments with different initial conditions.
clarity: The paper is fairly well written. However, this work contains a handful of redundant descriptions that should be removed. For example, Section 4.1 (Simulation Environment) repeats its first two sentences immediately thereafter.
relation to prior work: The authors discuss previous works on physics-informed neural networks and inductive graph neural networks. Further, the authors distinguish their work from that of earlier works by summarizing their contributions in terms of their joint investigation of (1) topology-aware modeling; (2) explicit constraints; (3) decoupling of potential and kinetic energies; and (4) the zero-shot generalizability of graph neural networks for physical systems of varying sizes.
additional feedback: This reviewer found the authors' discussion of the utility of graph neural networks for approximating physical quantities such as node-wise acceleration quite insightful. For this reviewer, this raises the question, "In the authors' opinion, are there physical quantities for which graph neural networks would not be optimal to use for approximation?".
review_title: An examination of physics-informed graph neural networks and the strength of their inductive biases

Meta-review: metareview: This paper evaluates the use of physics-informed GNNs for modeling dynamical systems (by simulating them) and explores the properties that are beneficial for achieving enhanced performances with them. The main concerns of the reviewers from the initial version have been addressed during the rebuttal and discussion periods, and at this point most reviewers are in favor of acceptance (with the remaining one only marking a marginal score). Therefore, I also recommend accepting the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept