Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to model the second-order ODE of multi-body dynamical systems by using Graph Neural Networks (GNNs). Different from previous LGN or HGN that directly predicts the Lagrangian/Hamiltonian of the system, this paper explicitly involves the inductive biases of the constraints, internal-external forces, and conservation of Newton’s third law.  Experiments are performed to support the benefit of considering the above biases, and the generalizability to unseen systems of large sizes.
strength_and_weaknesses: Strengths:

1. Indeed, GNNs have been explored widely for physical modeling. This paper applies GNN to characterize the terms of the second-order ODE in Eq. 5, which has contained sufficient terms that reflect conservative/non-conservative forces, internal/external forces, and constraints. This is novel.

2. The authors develop a series of models that take into account different physics priors. The discussion on the relation with Newton’s third law is interesting.

3. Experiments are conducted compactly but convincingly. It is able to justify the validity of the proposed models, particularly CGNODE, CDGNODE, MCGNODE. 


Questions:

Below, I raise some questions (not necessarily weakness) and hope the authors can provide answers to them.

1. Is there any gap like assumption or relaxation between Eq. (2) and Lagrangian dynamics in Eq. (1). If so, please specify.

2. How to compute the Coriolis-like forces in your models, and how to make M inversible in Eq. (4). 

3. How does the derivation of lambda by (4) differ from Eq. (7) in the paper [Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints].

4. N_i = Gamma_i =0 in all your experiments?

5. For your models, CGNODE, CDGNODE, MCGNODE, the inductive biases are added independently, or  enforced upon each other? Why not propose a eventual model that considers all biases?

6. For the baselines LGN and HGN, does it share the same GNN backbone as GNODE?

7. What is the implementation of NODE?

8. From Figure 2, it is hard to verify that GNODE is clearly better than NODE. If so, please modify the presentation in the paragraph of “Impact of topology”.

9. Eq. (3), there is a typo. There is no constraint error to check if the proposed model can really permit the constraint of the systems.

Suggestions:

1. Section 3.1 is not the central contribution. The authors are suggested to introduce more on Section 3.2-3.4 including moving their details and illustrated figures from appendix to the main body. 

2. The neural network in Figure 1 is just an MLP and unable to show the case of GNN or message passing.

3. It will be interesting to visualize the constraint errors, the difference between the internal force and external force distribution of the learned model. 

4. There are recent papers that focus on enhancing symmetry on physical modeling, leading to the application of equivariant GNNs; examples include EGNN, GMN that also involves constraints, SGNN that considers external force field. I am not asking the authors to conduct experimental comparisons wit them, but rather encourage the authors to add some related discussion with the methods in that domain, since symmetry is actually a central topic in physics. For example, if we enforce the Lagrangian to be invariant w.r.t. space translation/rotation, we will also derive the Newton’s third law just based on Lagrangian dynamics. This point is easily justified, by for example checking the Eq. (5) in GMN, where the force message f_ij on each edge is of the same orientation of the relative position x_ij of each two nodes, indicating f_ij=-f_ji.


[EGNN] E(n) Equivariant Graph Neural Networks, ICML 2021.
[GMN] Equivariant graph mechanics networks with constraints, ICLR 2022.
[SGNN] Learning Physical Dynamics with Subequivariant Graph Neural Networks, NIPS 2022.

clarity,_quality,_novelty_and_reproducibility: Please see above.
summary_of_the_review: I generally enjoy reading this paper and accept the ideas. I have some concerns on the writing and the experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to combine graph neural networks and neural ordinary differential equations (ODEs) to make the learned model generalize to particle systems with arbitrary sizes. It further studies how different inductive biases that could be embedded into the framework affect the performance. On two synthetic datasets, some benefits of the proposed model over two strong baselines are demonstrated.
strength_and_weaknesses: ## Strength
- The paper is clear and easy to follow.

## Weakness
- The motivation in the abstract is weak as generalizing to arbitrary-sized particle systems using graph neural networks with ODEs are already done by LGN/HGN as mentioned.
- The differences between the proposed method and LGN/HGN are not clear. It looks to me that the proposed method slightly generalize LGN/HGN by implementing different inductive biases. If this is the case, the novelty is limited.

## Questions
- Can you clarity the difference between MCGnode and LGN/HGN (in terms of inductive bias)? 
- In the result section it point that MCGnode outperforms both LGN and HGN (figure 5) without any explanation on potential reasons/conjectures. Why this happens? 
    - In general, the results section should not be a simple number comparison. You should draw some conclusion from the comparison.
- How are the different inductive biases studied here similar to [1]? Some comparisons could be useful as [1] is also a recent work with similar focus.

[1] Xu, K., Srivastava, A., Gutfreund, D., Sosa, F., Ullman, T., Tenenbaum, J. and Sutton, C., 2021. A Bayesian-symbolic approach to reasoning and learning in intuitive physics. Advances in Neural Information Processing Systems, 34, pp.2478-2490.
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is well-written and easy to read.
- Quality: The method is straightforward and the setup of ablation study of inductive biases is justified.
- Novelty: The proposed method seems to be very similar to existing ones in the literature.
- Reproducibility: The details in the appendix seems to be enough for reproducibility.
summary_of_the_review: It looks to me that the proposed method itself is not novel enough and the main contribution is the specific implementation to achieve several inductive biases and an ablation study on them. If this is not correct, the author could clarify this aspect by answering my questions in earlier section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses on inductive biases for effective learning of physical dynamics based on a data-driven approach. The proposed method, GNODE, is an extension of neural ODE and can describe physical dynamics using graph structures and can embed some physics-informed inductive biases. In the experiments, the effectiveness of the proposed method is verified by comparing it with NODE and graph-based NN models (LGN and HGN). It is also applied to transductive settings to achieve good performance.
strength_and_weaknesses: S1. A new graph-based NODE is proposed, which can embed several physics-informed inductive biases.
S2. In experiments, it is shown that the proposed method can predict trajectories more accurately than NODE while successfully capturing the energy conservation law.
S3. It is interesting that predictions can be made for unseen systems.

W1. Comparison methods seem to be lacking. For example, in Figure 2, it would be good to have a comparison not only with NODE, but also with models that can use physical knowledge (e.g. Symplectic ODE-Net [R1]).
W2. The task of predicting unseen systems is very interesting, but I think other approaches (e.g. meta-learning ) are working on similar tasks. I think it would be good to add a discussion of this in the related research section.
W3. There are several unclear points as follows:
  - This study is formulated based on the Lagrangian form. What is the reason for choosing the Lagrangian form? Also, are the ideas in this study applicable to Hamiltonian form?
  - Can you also briefly tell me about the advantages of studying physical systems based on GNN?
  - The more inductive biases you consider, the more robust you will be against sparsity and noise in your data. If you have any views or experimental results, please let me know.

[R1] Zhong et. al., Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control, ICLR, 2020.
[R2] Lee et. al., IDENTIFYING PHYSICAL LAW OF HAMILTONIAN SYSTEMS VIA META-LEARNING, ICLR, 2021.
clarity,_quality,_novelty_and_reproducibility: - This manuscript is readable.
- I think the proposed method contains novelty.
- There are some concerns about the evaluation. It is necessary to examine whether the comparison method is sufficient.
summary_of_the_review: Overall, it addresses an interesting problem and the method is novel. However, there are concerns about the choice of comparison methods and some ambiguities in the manuscript.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Neural ODEs are powerful models for learning dynamical systems, however, they cannot be generalized the systems with larger sizes when trained on the smaller systems. In order to solve this problem this paper proposes a graph-based neural ODE (GNODE) which can be inductive and generalize to larger systems with the similar dynamics such as pendulum and springs. Base on success from adding physics-based inductive biases in previous works such as Lagrangian neural networks (LNN) and Hamiltonian neural networks (HNN), this paper also explores the effectiveness of adding various types of physics-based inductive biases to the GNODEs. These inductive biases are (1) Decoupling the dynamics and constraints (CGNODE), (1) Decoupling the internal and body forces (CDGNODE), and (3) Newton’s third law (MCGNODE). They evaluate the models based on three metrics: rollout error, energy violation error, and momentum conservation error. The results show the performance of GNODE model, its different versions with distinct inductive biases, and two competing methods (LGN and HGN). Overall, the performance of MCGNODE is the best among GNODE models in terms of those three evaluation metrics, and it also outperforms the competing methods in the pendulum and spring systems. 
strength_and_weaknesses: Strength:

- The authors have done a good job in bridging the physics and ML by bringing such inductive biases which really help learn the dynamics in these systems effectively.

Weakness:

- These inductive biases work well in physics-based systems such as pendulum and spring. The authors have not investigated its efficacy in more general dynamical systems such as time-series. 

- Related to the previous point, I think the examples in this paper are not diverse and only show the physical systems, pendulum and spring.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. The novelty is sufficient as their way of introducing the physics-based inductive biases to the ODE models are different than previous work.
summary_of_the_review: I liked the idea of introducing three different types of physics-based inductive biases to the graph-based ODE models. It is always helpful to add known governing rules of the systems to the models in order to guide them to learn the true dynamics. However, I have the following concerns/questions:

- This is related to the usefulness of this method in other dynamical systems. I am convinced that this method should work great in the physical systems where we exactly know some of the underlying rules. However, can it still work so well in other systems such as time-series data? In other words, can bringing such inductive biases still help in the systems that we do not know for sure such assumptions do not hold?

- I think it would really help the reach of paper if it includes time-series data, as right now, the diversity of examples are limited to physical systems. 

Some minor comments:

- Typos: 
           an node  --> a node
           the learn --> to learn

- Should eq (11) have another summation over t?
- Why the authors use Verlet algorithm to compute the accelerations in eq (12)? Shouldn't the acceleration values be observed in such systems? How to choose \delta t and how accurate are the estimation of the accelerations using (12)? It would be great if the authors clarify this.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a graph-based neural ODE model which can be used to learn the time evolution of a physical system. The dynamics function is modeled with a graph neural network which allows applying a trained model to a different number of objects at test time. The authors propose several versions of the model:
- GNode: Each node of the GNN takes an object's position and velocity as inputs and the GNN predicts the acceleration of each object.
- CGNode: GNode + taking into account known constraints (holonomic or Pfaffian) based on the topology and nature of the system.
- CDGNode: CGNode + explicitly modeling external forces (for instance, gravitational or electromagnetic forces, which depend on the positions of objects rather than the topology of the system).
- MCGNode: CDGNode + Newton's third law (the internal forces between the particles are equal and opposite).

The experimental results suggest that introducing the inductive biases of the constraints, internal-external forces, and conservation of Newton’s third law improves the accuracy of the trained model.

During the discussion, the following concerns were raised by the reviewers:
- It was not always clear in the presentation what was assumed known and what was learned from data. The authors provided a good clarification of that.
- Trying more complex system. The authors provided experimental results with a 3D solid system subjected to isotropic compression.
- The differences between the proposed method and LGN/HGN are not clear. The authors responded that MCGnode incorporates three different inductive biases: Newton's third law (Sec 3.4), decoupling of internal and external forces (Sec 3.3) and Decoupling dynamics and constraints (Sec 3.2).
- Include more comparison models that can use physical knowledge (e.g. Symplectic ODE-Net) and study the robustness against noise. The authors added Symplectic ODE-Net as a baseline and included experiments studying robustness to noise.

The reviewers were generally convinced that the paper contains novel contributions and they were satisfied with the quality of the paper. The authors are encouraged to include the extra clarifications and experimental results produced during the discussion in the paper.
justification_for_why_not_higher_score: relatively small contribution
justification_for_why_not_lower_score: The models seem reasonable and perform well.
Decision: Accept: poster