Enhancing the Inductive Biases of Graph Neural ODE for Modeling Physical Systems
Authors: Suresh Bishnoi
Ravinder Bhattoo
Jayadeva Jayadeva
Sayan Ranu
N M Anoop Krishnan
Venue: ICLR 2023 poster (2023)
Keywords: Neural ODE
Graph neural network
physical systems
Graph Neural ODE

Abstract: Neural networks with physics-based inductive biases such as Lagrangian neural networks (LNNs), and Hamiltonian neural networks (HNNs) learn the dynamics of physical systems by encoding strong inductive biases. Alternatively, Neural ODEs with appropriate inductive biases have also been shown to give similar performances. However, these models, when applied to particle-based systems, are transductive in nature and hence, do not generalize to large system sizes. In this paper, we present a graph-based neural ODE, GNODE, to learn the time evolution of dynamical systems. Further, we carefully analyze the role of different inductive biases on the performance of GNODE. We show that similar to LNN and HNN, encoding the constraints explicitly can significantly improve the training efficiency and performance of GNODE significantly. Our experiments also assess the value of additional inductive biases, such as Newton’s third law, on the final performance of the model. We demonstrate that inducing these biases can enhance the performance of the model by orders of magnitude in terms of both energy violation and rollout error. Interestingly, we observe that the GNODE trained with the most effective inductive biases, namely MCGNODE, outperforms the graph versions of LNN and HNN, namely, Lagrangian graph networks (LGN) and Hamiltonian graph networks (HGN) in terms of energy violation error by ∼4 orders of magnitude for a pendulum system, and ∼2 orders of magnitude for spring systems. These results suggest that NODE-based systems can give competitive performances with energy-conserving neural networks by employing appropriate inductive biases.

Published as a conference paper at ICLR 2023
ENHANCING THE INDUCTIVE BIASES OF GRAPH NEU- RAL ODE FOR MODELING PHYSICAL SYSTEMS
Suresh Bishnoi, Ravinder Bhattoo, Jayadeva, Sayan Ranu, N. ML Anoop Krishnan Indian Institute of Technology Delhi, Hauz Khas, New Delhi, India 1 100116 (a (arz208500, c 175118, jayadeva, s sayanranu, kri krishnan) e itd. ac in
ABSTRACT
Neural networks with physics-based inductive biases such as Lagrangian neural networks (LNN), and Hamiltonian neural networks (HNN) learn the dynamics of physical systems by encoding strong inductive biases. Alternatively, Neural ODES with appropriate inductive biases have also been shown to give similar per- formances. However, these models, when applied to particle-based systems, are transductive in nature and hence, do not generalize to large system sizes. In this paper, we present a graph based neural ODE, GNODE, to learn the time evolution of dynamical systems. Further, we carefully analyze the role of different inductive biases on the performance of GNODE. We show that similar LNN and HNN, en- coding the constraints explicitly can significantly improve the training efficiency and performance of GNODE significantly. Our experiments also assess the value of additional inductive biases, such as Newton's 's third law, on the final performance of the model. We demonstrate that inducing these biases can enhance the per- formance of the model by orders of magnitude in terms of both energy violation and rollout error. Interestingly, we observe that the GNODE trained with the most effective inductive biases, namely MCGNODE, outperforms the graph versions of LNNN and HNN, namely, Lagrangian graph networks (LGGN) and Hamiltonian graph networks (HGN) in terms of energy violation error by 4 orders of magni- tude for a pendulum system, and orders of magnitude for spring systems. These results suggest that NODE- systems can give competitive performances with energy-conserve neural networks by employing appropriate inductive biases.
INTRODUCTION AND RELATED WORKS
Learning the dynamics of physical systems is a challenging problem that has relevance in several areas of science and engineering such as astronomy (motion of planetary systems), biology (move- ment of cells), physics (molecular dynamics), and engineering (mechanies, robotics)(LaVv) Valle 2006 Goldstein (011). The dynamics of a system are typically expressed as a differential equation, solutions of which may require the knowledge of abstract quantities such as force, energy, and drag(La Valle 2006) Zhong et al. 2020 Sanchez-Gonzalez et al. (2020). From an experimental per- spective, the real observable for a physical system is its trajectory represented by the position and velocities of the constituent particles. Thus, learning the abstract quantities required to solve the equation, directly from the trajectory, can extremely simplify the problem of learning the dynam- ics (Finzi et al. 2020).
Infusing physical laws as prior has been shown to improve learning terms of additional properties such as energy conservation, and symplectic nature (Karniadakis et al.) 202 Lutter et al.) 2019 Liu et al. 2021). To this extent, three broad approaches have been proposed, namely, Lagrangian
neural networks (LNN) (Cranmer et al. 2020aa Finzi et al. 2020 Lutter et al. 2019). Hamiltonian
neural networks (HNN) (Sanclez-Goonzlez et al. 2019 Greydanus et al. 2019 Zhong et al.) 2020 2021). and neural ODE (NODE) (Chen et al. |201 Gruver et al.] (021). 1). The learning efficieccy of
LNNS and HNNS is shown to enhance significantly by employing explicit constraints (Finzi et al.) 2020) and their inherent structure Zhong al. (2019). In addition, it has been shown that the superior performance of HNNs and LNNs is mainly due to their second-order bias, and not due to their symplectic or energy conserving bias (Gruver al.) (022). More specifically, an HNN with separable potential (V(g)) and kinetic (T(g. 4)) energies is equivalent to a second order NODE of the form í (g i). Thus, a NODE with second-order bias can give similar performances to
'S: School of Interdisciplinary Research, SR: Department of Computer Science, NMAK and RB: Depart- ment of Civil Engineering, J: Department of Electrical Engineering, SR and NMAK: Yardi School of Artificial Intelligence (joint appointment).

---

Published as a conference paper at ICLR 2023
that of an HNN (Gruver et al., 2021). Recently, a Bayesian-symbole approach, which assumes the knowledge of Newtonian mechanics along with symbolic features, has been used to learn the dynamics of physical systems Xu al. (202I).
An important limitation pervading these models is their transductive nature; they need to be trained on each system separately before simulating their dynamics. For instance, NODE trained on 5- pendulum can work only for 5-pendulum. This approach is referred to as transductive. In contrast, an inductive model, such as GNODE, when trained on 5-pendulum "learns" the underlying function governing the dynamics at node and edge level. Hence, GNODE, once trained, can perform infer- ence on any system size that is significantly smaller or larger than the ground truth as we demonstrate later in S This inductive ability is enabled through Graph neural networks (Scarselli et al.) 2008) due to their inherent topology-awan setting, which allows the learned system to naturally general- ize. Although the idea of graph-based -baee modeling has been suggested for physical systems (Cranmer) et al., 2020b; [Greydanus et al., 2019), the inductive biases induced due to different graph struc- tures and their consequences
the dynamics remain poorly explored. In this context, there are few recent works that aim to develop better representations of physical systems through equivariant graph neural networks Satorras et al. (2021); Han et al. (2022); Huang et al.) (022). However, the present work focuses on the inductive biases for physical systems rather than focusing on better representation learning. Some of the proposed inductive biases may also be natural consequence of forcing equivariance. For example, enforcing invariance to the Lagrangian w.r.t. space trans- lation and rotation also enforces Newton's 's third law, which we achieve through an inductive bias ($ 3. In the context of generalizability to unseen systems, there exists work on few-shot based on meta-learning (Lee et al. 202T). Meta-learning assumes the availability of limited amount of training data to adapt to an unseen system (task). This is different from our objective where we perform zero-shot generalizability, i.e., inference on an unseen system without any further training. To summarize, our key contributions are as follows:
Topelogy-aware modeling, where the physical system is modeled using a graph-based NODE (GNODE), enabling zero-shot generalizability to unseen system sizes.
2. Decoupling the dynamics and constraints, where the equation of motion of the GNODE is decoupled into individual terms such as forces, Coriolis-like terms, and explicit constraints.
3. Decoupling the internal and body forces, where the internal forces due to the interaction among particles are decoupled from the forces due to external fields.
4. Newton's third law, where the forces of interacting particles are enforced to be equal and oppo- site. We theoretically prove that this inductive bias enforces the conservation of linear momen- tum, in the absence of an external field, for the predicted trajectory exactly.
We analyze the role of these biases on n--enndulmm and spring systems. Interestingly, depending on the nature of the system, we show that some of these biases can either significantly improve the performance or have marginal effects. We also show that the final model with the appropriate inductive biases significantly outperforms the simple version of GNODE with no additional inductive biases and even the graph versions of LNN and HNN.
BACKGROUND ON DYNAMICAL SYSTEMS
A dynamical system can be defined by ordinary differential equations (ODEs) of the form, F(q, 4. t), where F represents the dynamics of the system. Thus, the time evolution of the system can be obtained by integrating this equation of motion. In Lagrangian mechanics, the dynamics of system are derived based on the Lagrangian of the system, which is defined as (Goldstein 201T):
L(g, a, () T(g. d) - (4)
Once the Lagrangian of the system is computed, the dynamics can be obtained using the Euler- VgI. Note that the EL equation is equivalent to D' Aleemerr'ss principle written in terms of the force as (N, m.ū)) dy; = O, where òg:
d Lagrange (EL) equation as di LVzL)
represents a virtual displacement consistent with the constraints of the system. Thus, an alternative to the EL equation can be written in terms of the forces and other components as (La 2006
Murray et al. 2017)
M(9) + Clg. ②) + N(2) + T 4
where (q) represents the mass matrix, C(q. q) the Coriolis-like forces, N (q))) the conservative forces, T()) represents the non-conservative forces such drag or friction, and IT represents any
2

---

Published as a conference paper at ICLR 2023
external forces. We note that Lagrangian dynamics is traditionally applied in systems with conser- vative forces, ie., where a Lagrangian is well-defined. However, Eq.2) is generic and can be applied to model any dynamical system irrespective of whether the system has a well-defined Lagrangian or not. Examples of systems where a well-defined Lagrangian may not exist include that of a colloidal system undergoing Brownian motion.
The constraints the system are given by A(g) = O, where A(g) E R represents & constraints.
Thus, A represents the non-normalized basis of constraint forces given by A Vo(Ø), where & is the matrix corresponding to the constraints of the system and À is the Lagrange multiplier corresponding to the relative magnitude of the constraint forces. Note that M, C, and N are directly derivable from the Lagrangian of the system L as M Vgg C = VggL, and N = VgL. Thus, decoupling the dynamic equation simplifies the representation of the dynamics F with enhanced interpretability. To learn the dynamics, Eq. can be represented as
i = M [T C -
(3)
To solve for A, the constraint equation can be differentiated with respect to time to obtain AQ+Aq Substituting for q from Eq. and solving for X, we get
(AM AT) (AM (II Cÿ N +)+ Ag)
(4)
Accordingly, i can be obtained as
q q= M (11- N Cq AT -A") (AM 'I1 -cg------ -N N -)) - AG))
(5)
3 INDUCTIVE BIASES FOR NEURAL ODE
To learn dynamical systems, NODES parameterize the dynamics F(g.. (t) using a neural network and learn the approximate function F(g., (t) by minimizing the loss between the predicted and actual trajectories, that is, L l1g:11 --+1ll- Thus, NODE essentially learns cumulative F, that consists of the contribution of all the terms contributing to the dynamics. In contrast, in an LNN, the learned L is used in the EL equation to obtain the dynamics. Similarly, in LNN with explicit constraints, the learned Lagrangian is used to obtain the individual terms the Eq. 5 by differentiating the LNN, which then substituted obtain the acceleration. However, this approach is transductive in nature. To overcome this challenge, we present a graph-based version of the NODE namely, GNODE, which models the physical system graph. We show, that GNODE can generalize to arbitrary system size after learning the dynamics F.
3.1 GRAPH NEURAL ODE (GNODE) FOR PHYSICAL SYSTEMS
In this section, we describe the architecture of GNODE. The graph topology is used learn the approximate dynamics F by minimizing the loss on the predicted positions. Graph structure. An rt-particle system is represented as an undirected graph g (V.E) where nodes represent particles and edges represent the connections or interactions between them. For instance, in pendulum springs system, the nodes correspond to bobs or balls, respectively, and the edges correspond to the bars or springs, respectively.
Input features. Each node is characterized by the features of the partieles themselves, namely, the particle type (I), position (q:), and velocity (q:). The type distinguishes particles of differing characteristics, for instance, balls or bobs with different masses. Further, each edge represented by the edge features Wij (4, 4z), which represents the relative displacement of the nodes connected by the given edge.
Neural architecture: Here, we provide a summary of the various neural layers within GNODE (see a detailed description in App. And Fig. Kin in the appendix provides a pictorial description. In the pre-processing layer, we construct a dense vector representation for each node and edge eij using MLPs. many cases, internal forces in a system that govern the dynamics are closely dependent on the topology of the structure. We model this through a deep, pessage-passiig GNN. Let z, hf denote the final representation constructed by the GNN over each node. These representations are passed through another MLP to predict the acceleration of each node.
Trajectory prediction and training. Based on the predicted ÿ. the positions and velocities are predicted using the velocity Verlet integration. The loss function of GNODE is computed by using
3

---

Published as a conference paper at ICLR 2023
the predieted and actual accelerations at timesteps ......., T in trajectory T, which is then back- propagated to train the MLPs. Specifically, the loss function is as follows.
1 n
T
n2
Here, ( is the predicted acceleration for the ht particle in trajectory T at time t and T. is the true acceleration. denotes a trajectory from 5, the set of training trajectories. Note that the accelerations are computed directly from the ground truth trajectory using the Verlet algorithm:
1 i(t) (At) la(t + At) + q(t - At) - [t)
Since the integration of the equations of motion for the predicted trajectory is also performed using the same algorithm as gt At) = 2t(t) g( åt) + ö(( ((At)? equivalent to training from trajectory/posiiions. should be noted that the training approach presented here may lead to learning the dynamics dictated by the Verlet integrator. Thus, the learned dynamics may not represent the "true" dynamics of the system, but one that is optimal for the Verlet integrator [
GNODE learns Newtonian dynamics directly without decoupling the constraints and other compo- nents governing the dynamics from the equation q F(q. q). Next, we analyze the effect of providing the constraints explicitly in the form of inductive biases and enhance the performance of GNODE. In the subsequent sections, we iteratively stack inductive biases on top of GNODE. Thus, the final proposed architecture, MCGNODE, is the eventual model that considers all biases. The detailed differences between all the graph architectures proposed in the work are included in App. K and Fig. 23
3.2 DECOUPLING THE DYNAMICS AND CONSTRAINTS (CGNODE).
To this extent, while keeping the graph architecture exactly the same, we use Eq. 5 to compute the acceleration of the system. In this case, the output GNODE is N; = squareplus (MLPv(Z.)) instead of 9s, where N; is the conservative force on particle i (refer Eq. D. Then, the acceleration is predicted using Eq. [ with explicit constraints. This approach, termed as constrained GNODE (CGNODE), enables us to isolate the effect of imposing constraints explicitly on the learning, and the performance of GNODE. It is worth noting that for particle systems in Cartesian coordinates, the mass matrix is diagonal one, with the diagonal entries, m,,8, representing the mass of the jth par- ticle. Thus, the mas are learned as trainable parameters. As a consequence of this design, the graph directly predicts the N₂ of each particle. Note that in the case of systems with drag, the graph would contain cumulative of N; and Y, Examples of such systems are demonstrated later (see App.
3.3 DECOUPLING THE INTERNAL
AND BODY FORCES (CDGNODE) In a physical system, internal forces arise due to interactions between stituent particles. On the contrary, the body forces are due the interac- tion of each particle with an external field such as gravity or electromag- netic field. While internal forces are closely related the system's 's topol- ogy, the effect of external fields on each particle is independent of the topology of the structure, In addition, while the internal forces are a function of the relative displacement between the particles, the external forces due to fields depend on the actual position of the particle.
Gaugh
redd
F.
SquasFiia
GNN
SquarFLa
Figure Architecture of CDGNODE (see App. K
To decouple the internal and body forces, we modify the graph architecture of the GNODE. Specifi- cally, the node inputs are divided into local and global features. Only the local features are involved passed through MLP to obtain an embedding,
in the message passing. The global features which is concatenated with the node embedding obtained after message passing. This updated em- bedding is passed through an MLP to obtain the particle level force, N; Here, the local node
See App. A for details why we use Verlet integrator
4

---

Published as a conference paper at ICLR 2023
features are the particle types and the global features are position and velocity. Note that in the case of systems with drag, the Y, can be learned using a separate node MLP using particle type and velocity as global input features. This architecture is termed as constrained and decoupled GNODE (CDGNODDE. The architecture of CDGNODE is provided in Fig. 1
3.4 NEWTON'S S THIRD LAW (MCGNODE)
According to Newton's 's third law, "every action has an equal and opposite reaction". In particular, the internal forces on each particle exerted by the neighboring partieles should be equal and opposite so that the total system is in equilibrium. That is, fy = -fji, where fij represents the force on particle due to its neighbor j. Note that this is stronger requirement than the EL equation itself, i... d (V L) V q. L. EL equation enforces that the rate of change of momentum of each particle is equal to the total force on the particle, N₂ ΣØ=1 fiy. where n; is the number of neighbors of i i- it does not enforce any restrictions on the individual component forces, fú, from the neighbors that add to this force. Thus, although the total force on each particle may be learned by the GNODE, the individual components may not be learned correctly. We show that GNODE with an additional inductive bias enforcing Newton's 's third law, namely momentum conserving GNODE (MCGNODE), conserves the linear momentum of the learned dynamics exactly.
de
Theorem In the absence of an external field, MCGNODE exactly conserves the momentum.
Proof of the theorem is provided in Appendix B To enforce this bias, we modify the graph archi- tecture as follows. In particular, the output of the graph after the message passing is taken from the edge embedding instead of taking from the node embedding. The final edge embedding is obtained after the message passing is passed through an MLP to obtain the force fij. Then the nodal force is computed as the sum of all the forces from the edges incident on the node. Note that while doing the sum, the symmetry condition fay ーデz is enforced. Further, the global forces coming from any external field are added separately to obtain the total nodal forces. Finally, the parameterized masses learned as the masses in particle system in Cartesian coordinates are diagonal in nature.
4 EMPIRICAL EVALUATION
In this section, we benchmark GNODE and the impact of the inductive biases, Our implementation is available at https://anonyymuss. wowsss.opnnn..... environment are provided in App.
ODE- Details of the simulation
4.1 EXPERIMENTAL SETUP
Baselines: To compare the performance of GNODE and its associated versions MCGNODE, CDGNODE, and CGNODE, compare with graph-based physics-informed neural networks, namely, (1) Lagrangian graph network (LGN) (Cranmer et al. 2020a), and (2) Hamiltonian graph network Sanchez-Gonzalee
et al.] 2019). Since the exact architecture and details of the graph ar- chitecture are not provided in the original works, a full graph architecture (Cranmer et al. 2020b Gonzalz-Gonzalez al. 2020) is used for modeling LGN and HGND] Since Cartesian coordinates are used, the parameterized masses are learned as diagonal matrix as performed in (Finzi et al.) 2020). We note that while in GNODE, the predicted acceleration is the direct output of the graph neural network, in LGN (or HGN) the output is the Lagrangian (or Hamiltonian) and further differ- entiation of the Lagrangian (or Hamiltonian) provides acceleration. These additional computational steps make LGN or HGN computationally expensive. We empirically establish this in App. [F] This result is consistent with earlier results that note neural ODEs (NODE) as being more efficient and generalizable than Hamiltonian neural networks if appropriate inductive biases injected Gruver let al. (202T). Thus, we also compare with NODE (Gruver et al., 2021). Finally, we also compare with SymODE-Net (Zhong al., 2019) (App. H and graph network simulator (GNS) (App. [
Datasets and systems: To compare the performance GNODES with different inductive biases and with the baseline models, we selected systems on which NODES are shown to give good perfor- mance (Zhong et al., 2021 Finzi et al. 2020), viz, n--endulums and springs, where n = (3, 4, 5). Further, to evaluate the zero-shot generalizability of GNODE to large-scale unseen systems, we sim- ulate 5, 20, and 50-link spring systems, and 5-, 10-, and 20-link pendulum systems. The details of the experimental systems are given Appendix. The training data is generated for systems without drag for each particle. The timestep used for the forward simulation of the pendulum system is 10 10-0g with the data collected every 1000 timesteps and for the spring system is 10 10-3, 8 with the data collected every 100 timesteps. The detailed data-generation procedure is given in App. [
We also evaluated the performance of LGN and HGN on the GNN of GNODE and observe that the perfor- mance is inferior. Details are provided in App-
5

---

Published as a conference paper at ICLR 2023
Linka
Links
Linka
5 Links
Linka
Linka
Spmm
Spmigg
Eneegy viletinn
Roloost
Pendulum
138
122
Time
Time
MODE
CONODE
MCGNODE
MCDE
CGNODE
MCONODE
GNODE
COONODE
ONODE
COONODE
Figure 2: Comparison of the energy violation and rollout error between NODE, GNODE, CGNODE, CDGNODE, MCGNODE for n- n-pendulum and spring systems with n 3, 4, 5 links. For the pen- dulum system, the model is trained on the 5-pendulum system and for the spring system, the model trained on the 5-spring system and evaluated on all other systems. The shaded region shows the 95% confidence interval from 100 initial conditions.
Following the work of (Finzi et al. 2020), we evaluate performance by computing the following three error metrics, namely, (1) the relative error in the trajectory, defined as the rollout error, (RE(t)), (2) Energy violation error (EE(t)), (3) the relative error in momentum
Evaluation Metric:
conservation, (ME(t)), given by: RE(t) = [l((t) -(ti)la EE(() (|] ME(() M =
|| Hlll
+
[[M-M]]= Note that all the variables with a hat, for example, à, represent the predicted values [IM la+ [|M]2 based on the trained model, and the variables without the hat, that is x, represent the ground truth.
Model architecture and training setup: For all versions of GNODE, LGN, and HGN, the MLPs two-layers deep. We 10000 data points generated from 100 trajectories to train all the models. This dataset is divided randomly in a 75:25 ratio as a training and validation set. Detailed training procedures and hyper-parameterr are provided in App. J] All models were trained till the decrease in loss saturates to less than 0.001 over 100 epochs. The model performance is evaluated on a forward trajectory, task it was not explicitly trained for, of 10s in the case of the pendulum and 20s in the case of spring. Note that this trajectory is 2-3 orders of magnitude larger than the training trajectories from which the training data has been sampled. The dynamics of the n-body system are known to be chaotic for n. 2 2. Hence, all the results are averaged over trajectories generated from 100 different initial conditions.
4.2 EVALUATION OF INDUCTIVE BIASES IN GNODE
Performance of (GNODE). Fig. 2 shows the performance of GNODE on the pendulum and spring systems of sizes 3 to 5. The GNODE for the pendulum is trained on 5-pendulum system and that of the spring is trained on a 5-spring system. Hence, the 3- and 4- 4-pendulum systems and 3- and 4-spring systems are unseen, and consequently, allow us to evaluate zero-shot generalizability. We observe that energy violation and rollout errors of GNODE for the forward simulations are comparable, both in terms of the value and trend with respect time, across all of the systems. Furthermore, when we compare with NODE, which has been trained on each system separately, while the errors are comparable in 5-pendulum, GNODE is better in 5-spring. It worth highlighting that NODE has an unfair ad advatagee on 5 5-pendulum since it is trained and tested on the same system, while 5-pendulum unseen for GNODE (trained on 5-pendulum). This confirms the generalizability of GNODE to simulate the dynamics of unseen systems of arbitrary sizes. More experiments comparing NODE with GNODE to further highlight the benefit of ropology-awars modeling are provided in App.D
GNoDE with explicit constraints (CGNODE). Next, we analyze the effect of incorporating the constraints explicitly in GNODE. To this extent, we train the 5-pendulum system using CGNODE model with acceleration computed using Equation 3 Further, we evaluate the performance of the model on 4- and 5-pendulum systems. Note that the spring system is an unconstrained system and hence CGNODE is equivalent GNODE for this system. We observe that (see Fig. 2) CGNODE exhibits 44 orders magnitude superior performance both in terms of energy violation and rollout error. This confirms that providing explicit constraints significantly improves both the learning efficiency and the predictive dynamics of GNODE models where explicit constraints are present, for example, pendulum systems, chains, rigid bodies.
Decoupling internal and body forces (CDGNODE). Now, we analyze the effect of decoupling local and global features in the GNODE by employing the CDGNODE architecture. It is worth

---

Published as a conference paper at ICLR 2023
Links
Links
5 Links
Linka
Linka
Linka
Sprinn
Energr violaton
Rollout arror
Pendbutu
Pavdulum
MCGNODE
LGW HGN
MCGNODE
LGN
MGN
Figure 3: Comparison of the energy violation and rollout error between GNODE, LGN, and HGN for n-pendulum and spring systems with n 3, 4, links. For both the pendulum and spring systems, the model is trained on a 5-link structure and evaluated on all other systems. The shaded region shows the 95% confidence interval from 100 initial conditions.
noting that the decoupling of the local and global features makes the GNODE model parsimonious with only the relative displacement and node type given input features for message passing to the graph architecture. In other words, the number of features given to the graph architecture is lower in CDGNODE in GNODE. Interestingly, observe that the CDGNODE performs better than CGNODE and GNODE by ~ order of magnitude for pendulum and spring systems, respectively.
Newton's 's third law (MCGNODE). Finally, empirically evaluate the effect of enforcing New- ton's 's third law, that is, fuj = -fj on the graph using the graph architecture namely, MCGNODE. Figure Figure[]show, shows the performance of MCGNODE for both pendulum and spring systems. Note that was theoretically demonstrated that this inductive bias enforces the exact conservation of momen- tum, in the absence of an external field. Figure 4 shows the ME of MCGNODE in comparison to NODE, GNODE, CGNODE, CDGNODE, and GCGNE for pendulum and spring systems. We observe that the spring system, the ME of MCGNODE is zero In the case of pendulum, al- though lower than NODE, GNODE, and CGNODE, the E of MCGNODE is comparable to that of CDGNODE. Correspondingly, MCGNODE exhibits improved performance for the spring system in comparison other models. However, in the pendulum system, we observe that the performance of MCGNODE is similar to that of CDGNODE. This is because, in spring systems, the internal forces between the balls govern the dynamics of the system, whereas in pendulum the dynamics erned mainly by the force exerted on the bobs by the external gravitational field. Interestingly, we ob- serve that the energy violation error of MCGNODE remains fairly constant, while this error diverges for other models. This confirms that strong enforcement of momentum conservation in turn ensures that the energy violation eтrоr stays constant over time, ensuring stable long-ten term equilibrium dy- namics of the system. In addition, MCGNODE exhibits significantly improved training efficiency than NODE (see App/F). Finally, we also evaluate the ability of MCGNODE to learn the dynamics of systems with drag and external forces (see App. ED. We observe that MCGNODE clearly outperforms NODE in this case as well. MCGNODE will, thus, be particularly useful for the statistical analysis of chaotic systems where multiple divergent trajectories correspond to the same energy state.
gov-
4.3 COMPARISON WITH BASELINES
Now, we compare the GNODE that exhibited the best performance, namely, MCGNODE with the baselines namely, LGN and HGN. Note that both LGN and HGN are energy-conserving graph neural networks, which are expected to give superior performance in predicting the trajectories of dynam- ical systems (Sanclez-Gonzalez et al. 2019). Especially, due to their energy mergy-conserving nature, these models are expected to have lower energy violation error than other models, where energy conservation bias not explicitly enforeed. F shows the energy violation and rollout error of GNODE, LGN, and HGN trained on a 5-link system and tested on 3. 4. 5-link pendulum and spring systems. Interestingly, we observe that MCGNODE outperforms both LGNN and HGN in terms of rollout error and energy error for both spring and pendulum systems. Specifically, in the case of pendulum systems, we observe that the energy violation and rollout errors are ~ orders mag- nitude lower in the case of MCGNODE in comparison to LGN and HGN. Further, we compare the ME of MCGNODE with LGN and HGN (see Fig. righht palette). We observe in both pendulum and spring systems that the ME of MCGNODE is significantly lower than that of LGN and HGN. Thus, in the case of spring systems, although the performance of HGN and MCGNODE are comparable for both R.F. and F.F. we observe that MCGNODE exhibits zero M., E, while in HGN and LGN M F.
7

---

Published as a conference paper at ICLR 2023
5 Links
Links
Links
Momertuu INTOR
238 111
85x
NODE
MCGNODE
Time
GNODE
COGNODE
HIN
Figure 4: Momentum error EM of NODE, GNODE, CGNODE, CDGNODE, and MCGNODE (top palette), and LGN, HGN, MCGNODE (bottom palette) for spring (top of both the palettes) and pendulum (bottom row of both the palettes) systems with 3, 4, and links. The shaded region represents the 95% confidence interval based on 100 forward simulations.
Enargi violatton
spring
Spiing
MCGNODE
LGN
HGN
MCONODE
LGN
HGN
Links
Linkks
Links
Enstop violatio
Rollout error
Pandstum
Penditum
124
MCGNODE
LGN
HGN
MCGNODE
LGN
HGN
Figure 5: Energy violation and rollout error between GNODE, LGN, and HGN for spring systems with 5, 20, 50 links and pendulum system with 5, 10, 20 links. Note that all the systems, GNODE, LGN, and HGN trained on 5-pendulum and 5-spring systems and tested others. The shaded region represents the 95% confidence interval based on 100 forward simulations.
diverges. The superior performance of MCGNODE could be attributed the carefully constructed inductive biases, namely, (i) explicit constraints imprroveess- data efficiency and reduces the error in pendulum systems, (ii) decoupling internal and body forces-leads superior performance in pendulum systems which act under gravity, (iii) momentum conservation-lcads to superior perfor- mance in spring systems by enforcing additional conservation law strongly. We note that these inductive biases may also be adapted suitably for LGN and HGN and thereby improve their perfor- mance. Thus, the contribution of our work is not the neural architecture isolation, but the core design principles that are generic enough to be abstracted beyond neural ODEs.
4.4 GENERALIZABILITY
Now, we focus on the zero-shot generalizability of GNODE, LGN, and HGN to large systems that are unseen by the model. To this extent, we model 5-, 10-, and 20-link pendulum systems and 5-, 20-, and 50-link spring systems. Note that all the models are trained on the 5-link systems. In congruence with the comparison earlier, we observe that MCGNODE performs better than both LGN and HGN when tested on systems with a large number of links. However, we observe that the variation in the error represented by the 95% confidence interval increases with an increase in the number of particles. This suggests that the increase in the number of particles essentially results in increased noise in the prediction of the forces. However, the error the MCGNODE significantly lower than the error in LGN and HGN for both pendulum and spring systems, as confirmed by the 95% confidence interval, which could again be attributed to the three inductive biases infused in MCGNODE, that are absent in LGN and HGN. This suggests that MCGNODE can generalize larger system sizes with increased accuracy.
4.5 LEARNING THE DYNAMICS 3D SOLID
8

---

Published as a conference paper at ICLR 2023
To demonstrate the applicability of MCGNODE to lear the dynamics of complex systems, consider challenging problem, that is, infer- ring the dynamics of elastically deformable body. Specifically, a 5 5 5 solid cube, discretized into 125 particles, is used for simulating the ground truth. 3D solid sys- tem is simulated using the peridy- namics framework. The system is compressed isotropically and then released to simulate the dynamics of the solid, that is, the con- traction and expansion in 3D. We observe that MCGNODE performs inference much superior to GNODE and GNS (see Fig. [and App. D. It should also be noted that although the momentum ror MCGNODE seems high, this is due to the fact that the actual momentum in these systems is significantly low. This is demonstrated by plotting the absolute momentum error in these systems which in MCGNODE is close to zero, while in GNODE it explodes (see Fig. 21). These results sug- gest that the MCGNODE could be applied to much more complex systems such as solids other particle-based systems where momentum conservation is equally important as energy conservation.
So5e5 total particles
Figure 6: (RE) and (ME) of GNODE, MCGNODE and GNS for particle-based system. Simulation performed us- ing peridynamics on 3D box of 125 particles (5 x 5 x 5).
4.6 ROBUSTNESS TO NOISE
Finally, show the robustness to noise of the MCGNODE in comparison GNODE, and additional baselines, namely, HGN and LGN. Fig. shows the performance of these models on 5-spring and 5-pendulum systems for varying noise values. Note that the noise is given in terms of % standard deviation of the distribution of the original output values. We observe that MCGNODE is robust against noise in comparison other models.
5-Pendulum
5-Spring
CONCLUSIONS
of Roltout GM Energy violation
In this work, we carefully analyze various inductive bi- ases that affect the performance of a GNODE. Specif- ically, we analyze the effect of a graph architecture, ex- plicit constraints. decoupling global and local features that makes the model parsimonious, and Newton's 's third law that enforces stronger constraint tion of momentum. We observe that inducing appro- priate inductive biases results in an enhancement the performance of GNODE by ~23 orders magnitude for spring systems and ~556 orders magnitude for pendu- lum systems in terms of energy violation error. Further, MCGNODE outperforms the graph versions of both La- grangian and Hamiltonian neural networks, namely, LGN and HGN. Altogether, the present work suggests that the performance of GNODE-based models can be enhanced and even made superior to energy-conse monggycconsrvving neural net- works by inducing the appropriate inductive biases.
the conserva-
Noise percentage
GNODE
MCONODE
LON
HON
Impact of noise on perfor- mance of GNODE, MCGNODE, LGN, and HGN. Note that the noise given in terms of % standard deviation of the dis- tribution the original output values.
Figure 7:
Limitations and Future Works: The present work fo- cuses particle-based systems such pendulum and spring systems. While these simple systems can be used to evaluate model performances, real life more complex in nature with rigid body motions, contacts, and collisions. Extending GNODE to address these problems can enable the modeling of realistic systems. A more complex set of systems involve plastically deformable systems such as steel rod, and viscoelastic materials where the deformation depends on the strain rate of materials. Further, instead of explicitly providing the constraints, learning the constraints directly from the trajectory can enhance the application of the models to unseen systems. Note that the present work is limited to physical systems. A sim- ilar approach could be employed in other dynamical systems as well. From the neural modeling perspective, the GNODE is essentially graph convolutional network. Using attention each message passing head may allow us to better differentiate neighbors based on the strength of their interactions. We hope pursue these questions in the future.

---

Published as a conference paper at ICLR 2023
REFERENCES
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne. Jax: composable transformations of python+ numpy programs, 2018. URL http://ggttub. com/google/jax, 4:16, 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6572-6583, 2018.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020a. URL httpps/://opeeeeiii.eeieieeeeeee wwtp:/nommdddiiieeeeo...
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. Advances in Neural Information Processing Systems, 33, 2020b.
Mare Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13880-13889, Curran Associates, Inc., 2020. URL hutpps:/prroceedingss.o https://pooeeddigssnuurips.... wwwpnisscccpppprr////......
Jonathan Godwin Thomas Keck*, Peter Battaglia, Victor Bapst, Thomas Kipf, Yujia Li, Kimberly Stachenfeld, Petar Veličković, and Alvaro Sanchez-Gonzalez. Jraph: A library for graph neural networks in jax., 2020. URL http::////gt http://hhwbb.oomm/eee/m/////////
Herbert Goldstein. Classical mechanics. Pearson Education India, 2011.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 04215379-15388 2019.
Nate Gruver, Marc Anton Finzi, Samuel Don Stanton, and Andrew Gordon Wilson. Deconstructing the inductive biases of hamiltonian neural networks, In International Conference on Learning Representations, 2021.
Jiaqi Han, Wenbing Huang, Hengbo Ma, Jiachen Li, Joshua Tenenbaum, and Chuang Gan. Learn- ing physical dynamics with subequivariant graph neural networks. 2022.
Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant graph mechanics networks with constraints. In International Conference on Learning Represen- tations, 2022.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informe informed machine learning. Nature Reviews Physics, 3(6):422 440, 2021.
Steven M Valale.. Planning algorithmss Cambridge university press, 2006.
Seungjun Lee, Haesang Yang. and Woojae Scong. Identifying physical law of hamiltonian systems via meta-learning. In International Conference Learning Representations, 2021. URL https: //open/eview..et//// Aistooruummmi455///it
Ziming Liu, Yuanqi Du, Yunyue Chen, and Max Tegmark. Physics-augmented learning: A new paradigm beyond chysics-informed learning, In NeurlPS 2021 Al for Science Workshop, 2021.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics model prior for deep learning. In International Conference on Learning Representations, 2019. URL https://ppenreiiewneeeee wwt.ne/foooummii-BBBHiCCC.
Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction robotic manipulation. CRC press, 2017.
Alvaro Sanchez- Fanchez-Gonzalez Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv e-prints, pp. arXiv-1909, 2019.
10

---

Published as a conference paper at ICLR 2023
Alvaro Sanchez-Gonziine Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- on Machine Learning, pp. 8459 8468. PMLR, 2020.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80) 2008
Samuel Schoenholz and Ekin Dogus Cubuk. Jax md: a framework for differentiable physics. Ad- vances in Neural Information Processing Systems, 33, 2020.
Kai Xu, Akash Srivastava, Dan Gutfreund, Felix Sosa, Tomer Ullman, Josh Tenenbaum, and Charles Sutton. A -ayesian--ymboolic approach to reasoning and learning in intuitive physics. Advances in Neural Information Processing Systems, 34:2478-2490, 2021.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference on Learning Representations, 2019.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Dissipative symoden: En- coding hamiltonian dynamics with dissipation and control into deep learning. arXiv preprint arXiv:2002.08880, 2020.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Benchmarking energy-conserving neural networks for learning dynamics from data. Learning for Dynamics and Control, pp. 1229-1229. PMLR, 2021.

---

Published as a conference paper at ICLR 2023
Initial node embedding
Flaw node featuree
Grach representation physical system
MLP
Inttaal edge embadding
Raw edge teatures
SquarePlus,
MLP
Figure 8: Architecture of GNODE.
A DETAILS OF GNODE
The architecture of GNODE is depicted Fig. g GNODE constitutes of the following neural layers.
Pre-Processing In the pre- pe-prooessing layer, we construct a dense vector representation for each node V and edge Eij using MLP em as:
h =
(8)
0 h i
squareplus(MIn (Wsj))
(9)
squareplus is an activation function. Note that the MLP corresponding to the node and edge embedding functions are parameterized with different weights. Here, for the sake of brevity, simply mention them as MLP
Acceleration prediction. In many cases, internal forces in a system that govern the dynamics are closely dependent on the topology of the structure. To capture this information, we employ multiple layers of message-passing between the nodes and edges. In the 7t layer of message passing, the node embedding is updated as:
h1th squareplus h
h/ w (h/fl1(s))
(10)
JEM.
Eij E} are the neighbors of W is a layer-specific learnable weight matrix. h represents the embedding of incoming edge eg us in the 1 layer, which is computed as follows.
where, .M. {U, EV
+++ squareplus (h/, + W (h/
(11)
Similar to W W. is a layer-specific learnable weight matrix specific to the edge set. The mes- sage passing is performed over layers, where L is a hyper-parameter. The final node and edge representations in the th layer are denoted as z, = h and Zij = h respectively. Finally, the acceleration of the particle q is predicted as:
i squareplus(MMRP(22)
(12)
Why Verlet integrator: The canonical coordinates employed in Newtonian mechanics are the po- sition (g) and velocity (ÿ) (or momentum) these quantities are the direct observables. Verlet algorithm is obtained by substracting the Taylor series expansion of r(t + At) with r(t At). The expression thus obtained is time-reversible and has error to the order of (O(At)). Due to the time- reversible nature, Verlet algorithm is energy conserving (symplectic), and hence performs better than other traditional integrators such as RK4 in predicting the long term trajectory of dynamical systems without drift.
B PROOF OF THEOREM 1
PROOF. Consider a system without any external field where the dynamics is governed by internal forces, for instance, n-sprrnn system. The conservation of linear momentum for this system implies
12

---

Published as a conference paper at ICLR 2023
Links
Links
Linka
Linkk
Linka
Linkk
Sgritng
Eneegs Villtton
Rolouut arror
Pendufum
LOMPONN)
HGMPPON)
LGN
MGN
LONIFORO HGNIPONO
LGN
HON
Figure 9: comparison of LGN and HGN with the full graph architecture.
that E de (miq)) = O, where q. represents the predicted velocity, Mu represents the learned mass of the particle i, and di represents the ground truth velocity. Assume the error in the rate of change of momentum of the predicted trajectory to be é, To prove that MCGNODE conserves the momentum exactly, is sufficient to prove that & = O. Consider,
12
d
12
n
n
(aaq: m.g.) IN. N
N
(13)
Note that in equation generality, Ei=1 N. 0, in the absence of an external field. Hence,
3 all terms except N₂ are either known or are zero. Thus, without loss of
Th
(Jis + fzz)
E
(14)
The inductive bias fúj = fez implies that (fs++fat)) 0. Thus,
Σ d dt
(mas))) =
C PERFORMANCE OF LGN AND HGN WHEN IMPLEMENT ON THE GNN OF GNODE.
In Fig. 9 we compare the performance of L.N and HGN implemented the full graph net- work (Cranmer al. 2020b, Sanchez-Gonzales al. 2020) against the versions implemented on the GNN used in GNODE. As visible, the full graph network version provides superior performance.
D ZERO-SHOT GENERALIZABILITT NODE VS GNODE VS MCGNODE
To showcase zero-shot generalizability, we compare the performance of NODE, GNODE, MCGN- ODE in Figure IO the case of NODE, the system is trained and evaluated each of the pendulum and spring systems separately. In the case of GNODE and MCGNODE, the training is performed on 5-spring and 5-pendulum system and evaluated on all other systems. First, we observe that the performance of NODE and GNODE are comparable for both rollout and energy eтror. This suggests that the performance of GNODE, with no additional inductive bias, is comparable that of NODE with the additional advantage that GNODE can generalize to any system size. Now, we compare the performance of NODE with MCGNODE, which has all the additional inductive biases discussed earlier. We observe that MCGNODE performs significantly better than NODE, expected. It is worth noting that while the inductive bias explicit constraints can be infused in NODE in a similar fashion in GNODE, other inductive biases such decoupling the global and local features and Newton's 's third law are unique to the graph structure. These inductive biases cannot be infused in NODE structure. This, in turn, demonstrates the superiority of GNODE which allows additional inductive biases due to the explicit representation of the topology through the graph structure.
13

---

Published as conference paper at ICLR 2023
3 Linka
4 Linka
5 Linka
3 Linka
4 Linka
Linka
Spring
vrclationn
Roleei error
Panduluum
MODE
ONOEE
MCGNODE
MODE
GNCDE
MCONODE
Figure 10: Energy and trajectory error for NODE, GNODE, and MCGNODE for spring and pendulum systems with 3-44, and 5-links. The shaded region represents the 95% confidence interval based 100 forward simulations.
Drag
Drag & external force
Drag
Drag external force
Energy violation
19
10"
Rollout
10
10
16
Time
Time
NODE
MCGNODE
MCGNODE
Figure 11: MCGNODE on 5-link spring with non-zero drag and external force with NODE base-
line.
E DRAG AND EXTERNAL FORCE
Here, we evaluate the ability of MCGNODE to learn the dynamics of systems subjected drag linear in velocity (drag force given by -0.1 and an external force. Fig. shows the performance of MCGNODE and NODE on a 5-spring system with linear drag and external force. We observe that the MCGNODE exhibits superior performance in both the cases.
TRAINING EFFICIENCY
Data Efficiency: Here, compare the training efficiency of MCGNODE with the NODE and HGN. Despite our best efforts, LGN was unable to learn the dynamics from the small amount of data provided and was exploding up on forward simulation leading to NaN values. Similar behavior was observed on HGN with low values data points. We focus on the best graph-based architecture and compare the performance with the baseline NODE 5-spring and 5-pendulum systems. Figure shows the performance of MCGNODE in comparison to NODE for the dataset sizes of 100, 500, 1000, 5000. and 10000 in terms of the geometric mean of the energy error and rollout error. We observe that the learning of MCGNODE, is significantly simplified in comparison to that of NODE- MCGNODE exhibits orders of magnitude better performance in terms of both energy and rollout error for both pendulum and spring systems. Further, we observe that MCGNODE performs orders of magnitude better than HGN.
Computational Efficiency:
We further compare the computation times of the training and infer- ence phases on GNODE, MCGNODE, HGN and LGN. Tables present the results. As visible, MCGNODE, is the fastest average. The difference with LGN and HGN is the highest in spring systems. MCGNODE benefits from the inductive biases as well as the fact that the neural architec- ture directly predicts the force (acceleration is computed from this) each particle. In contrast,
14

---

Published as a conference paper at ICLR 2023
Spring
Pendulum
GM of Energy violation
10
10"
10
10
10
10
10
GM of Rollout error
10
10
10
10"
10
10"
10"
10"
10
Data size
NODE
MCGNODE
HGN
Figure 12: Training efficiency of MCGNODE in comparison with the GNODE. The shaded region represents the 95% confidence interval based on 100 forward simulations.
Models
Training time (in sec)
Forward Simulation time (in sec)
MCGNODE
2433
0.87
GNODE
6341
1,02
HGN
2512
0.76
LoN
55042
14,82
Table 1: Training and inference times in pendulum systems.
Models
Training time
Forward Simulation sec)
MCGNODE
1576
GNODE
6977
0,10
HGN
14053
0.43
LON
141962
5.81
Table 2: Training and inference times in spring systems.
HGN and LGN predict the Hamiltonian and Langrangian, which needs be further differentiated to obtain acceleration. LGN is the slowest since it requires double differentiation.
G ERRORS ON INTERNAL AND EXTERNAL FORCES
this section, we report the errors in the predicted internal and external forces on each particle of
system. Figs. 13 16 presents the results GNODE, MCGNODE, LGN and HGN. As visible, the predicted force (represented using dashed lines) overlaps closely with the actual forces (solid line) in MCGNODE (Fig. 14). While the emror is also low in GNODE (Fig. 13) it is inferior to MCGNODE showcasing the usefulness the inductive biases. We also note that the error is the highest in LGN (Fig. 15
15

---

Published as a conference paper at ICLR 2023
S-Spring Exp
10.0
Averaged particles
7.0
Predicted
2.5
0.0
50
stap
(0)
S-Spring Exp
FRx particio
F
Predicted
Net
-2.5
-6.0
-7.5
50
100
125
150
175
Time step
(c)
S.Spring Exp
For particle
Not force
Predictad
50
100
125
150
175
200
Time
(6)
5.Spring Exp
5.0
For particle
F2
0.0
Predicted
Net
-6.0
100
Time step
(e)
S:Spring Exp
For particie
Not force
Predicted
60
100
125
160
175
200
Time step
5-Spring Exp
5.0
For
F,
2.5
Predicted
-5.0
50
step
Figure 13: Error in force calculation using GNODE. The solid lines represent the true force and the dashed lines represent the predicted ones. A close overlap among the solid and dashed lines indicate low error. F, and Fy denote the total forces in the x and direction respectively. Fz is always O.
16

---

Published as a conference paper at ICLR 2023
S Spring Exp
Averaged olher particles
F
Net forca
0.000
Predicted
008
100
step
(b)
S-Spring Exp
For particlo
F
Notforce
Predicled
0
25
50
75
100
125
150
175
200
Time stop
S-Spring Exp
particle
Not force
Predicted
step
(4)
5-Spring Exp
For particle
F
Net force
25
50
100
125
150
175
200
Time step
5. S.Bpring Exp
aattecl
F,
Not toros
o 2
Predicted
stap
S-Spring Exp
For particle
F,
Not force
o 20
Predicted
50
100 Time step
160
175
Figure 14: Error in force calculation using MCGNODE. The solid lines represent the true force and the dashed lines represent the predicted ones. close overlap among the solid and dashed lines indicate low enror. Fr and F, denote the total forces in the æ and direction respectively. F, is always O.
17

---

Published as a conference paper at ICLR 2023
S-Spring Exp
Averaged particles
force
Predicted
100
step
(0)
S S-Spring Exp
For particlo
F
Predicted
0.0
-2.-
-6.0
50
100
125
150
175
Time step
(c)
S.Spring Exp
8.0
For particle
2.
Predictad
0.0
2.5
-5.0
50
100
125
150
175
200
Time
(6)
5. Sprinng Exp
For particle
2.
Predicted
0.0
-255
-5.0
100 Time step
(e)
S:Spring
5.0
For particie
2.5
0.0
Predicted
60
100
120
160
175
200
Time step
5-Spring Exp
potticle
F,
2.5
Predicted
0.0
-2.
5.0
step
Figure 15: Error in force calculation using LGN. The solid lines represent the true force and the dashed lines represent the predicted ones. A close overlap among the solid and dashed lines indicate low error. F, and Fy denote the total forces in the x and direction respectively. Fz is always O.
18

---

Published as a conference paper at ICLR 2023
S-Spring Exp
0.015
Aovarrl particies
Net force
0.0010
Predicted
0.000
0.0000
stop
(0)
S S-Srring Exp
For particio
F
Not force
Predicled
50
100
125
150
175
Time step
(c)
5.Spring Exp
For particie22
force
Predictad
50
100
150
175
200
Time
(4)
5. Sprinng Exp
For particle
F2
Net force
Predicted
100
Time step
(e)
S:Spring
For Aorticto
F,
Not force
o mg
Predicted
60
100
125
160
175
200
Time step
5-Spring Exp
For particle
F,
Not force
Predicted
50
step
Figure 16: Error force calculation using HGN. The solid lines represent the true force and the dashed lines represent the predicted ones. A close overlap among the solid and dashed lines indicate low error. F, and Fy denote the total forces in the x and direction respectively. Fz is always O.
19

---

Published as a conference paper at ICLR 2023
Links
Links
Linka
Linka
Linka
Linka
Sgring
Eneegy Villtton
Eneegy
Pendstum
Pundufum
MCGRODE
LGN
HGN
SymODEN
MCGNODE
LGN
HON
SymODEN
Figure 17: Comparison of MCGNODE, LGN, HGN, and SymODEN. Note that SymODEN is trained and tested for 3-,4-55-links system, whereas in case of MCGNODE, LGN and HGN training is done only on 5-links system and tested on 3-14-5-llinks
H COMPARISON OF MCGNODE, LGN, HGN, SYMODEN
Fig. 17 shows the performance of an additional baseline, namely, SymODE-Net (SymODEN) et al. (2019) on 3,4,5-spring and pendulum systems. SymODE-Net being a feed-forward MLP is transductive in nature and thus cannot do zero-shot knowledge transfer. Hence, trained and tested on each system separately. We observe that the performance of SymODE-Net is comparable LGN but poorer than HGN and MCGNODE.
I EXPERIMENTAL SYSTEMS
I.1 SIMULATION ENVIRONMENT
All the simulations and training were carried out in the JAX environment (Schoenholz & Cubuk, 2020 Bradbury al. 2020). The graph architecture was developed using the jraph package (God- win et al. [2020). The experiments were conducted on a machine with Apple MI chip having 8GB RAM and running MacOS Monterey.
I.2 n- 2-PPENULUM
In n--pendulum system, repoint masses, representing the bobs, are connected by rigid bars which are not deformable. These bars, thus, impose a distance constraint between two point masses as
l|g: 9-11l =
(15)
where, 4 represents the length of the bar connecting the (i - 1)th and ith mass. This constraint can be differentiated to write in the form Pfaffian constraint as
9:4h + 9:-146-1
(16)
Note that such constraint be obtained for each of the masses considered obtain the A(g)-
The mass of each of the particles m, and the force on each bob due to the acceleration due to gravity in m,g. This force will be compensated by the constraint forces, which in turn governs the motion of the pendulum. For LGN, the Lagrangian of this system has contributions from potential energy due gravity and kinetic energy. Thus, the Lagrangian can be written
(1/2maqe -m,gy:)
(17)
And, for HGN, the Hamiltonian is given by
H. = Σ (1/2m²qi* m.gu)
(18)
where y represents the acceleration due gravity in the y direction.
20

---

Published as a conference paper at ICLR 2023
1.3 S-SPRING SYSTEM
In this system, 7tpoint masses are connected by elastic springs that deform linearly with extension or compression. Note that similar a pendulum setup, each mass my is connected only two masses m4-1 and m441 through springs so that all the masses form a closed connection. Thus, the force experienced by each bob due spring connecting to its neighbor is faj 1/2k(119: q3 -rr). For LGN, the Lagrangian of this system is given by
C
1/ /2m44 d1
Σ i=1
1 1/2k(119-11 - 9:ll -ro)²
(19)
for, HGN, the Hamiltonian by:
7 H.
T
1/ 4 2q +
Σ
1/2k(1lq--1 -ro1-
(20)
where ro and i represent the undeformed length and the stiffness, respectively, of the spring.
J IMPLEMENTATION DETAILS
J.1 DATASET GENERATION
ja..iia Software packages: 1umpy-1.22.1. jax-0.3.0, 1ax--.00 jax-m.-.. 1.2.. jaxlib-0.3.0. jraph-O.0.2.der
Hardware: Chip: Apple M1, Total Number of Cores: (4 performance and 4 efficiency), Memory:
8 GB, System Firmware Version: 101.3, OS Loader Version: 7459.101.3
All the datasets are generated using the known Lagrangian of the pendulum and spring systems, along with the constraints, as described in Section [] For each system, we create the training data by performing forward simulations with 100 random initial conditions. For the pendulum system, timestep of 10 10-Ss is used to integrate the equations of motion, while for the spring system, timestep 10-3 is used. The velocity- Verlet algorithm used to integrate the equations of motion due to its ability to conserve the energy in long trajectory integration.
For NODE, all versions GNODE, LGN:
While for HGN, datasets generated using Hamiltonian mechanics. Runge-Kutte Kutta integrator is used to integrate the equations of motion due to the first order nature of the Hamiltonian equations in contrast to the second order nature of LGN, NODE, and all versions of GNODE.
From the 100 simulations pendulum and spring system, obtained from the rollout starting from 100 random initial conditions, 100 data points extracted per simulation, resulting in total of 10000 data points. Data points were collected every 1000 and 100 timesteps for the pendulum and spring systems, respectively. Thus, each training trajectory of the spring and pendulum systems are 10s and 1s long, respectively. Here, we do not train from the trajectory. Rather, we randomly sample different states from the training predict the acceleration.
J.2 ARCHITECTURE
For NODE, use fully connected feed-forward neural network with hidden layers, each having 16 hidden units with square-plu activation function. For all versions of GNODE, graph architecture used with each MLP consisting of two layers with hidden units. One round of message passing is performed for both pendulum and spring. For LGN and HGN, full graph architecture is used with each MIPs consist of two layers with 16 hidden units and one round of message passing were performed. The hyper-parameters were optimized based on the good practices of the model development.
J.3
EDGE FEATURES: POSITION VS VELOCITY
Figure 18 shows the loss curve with relative position and relative velocity edge features. We observe that when relative velucity is provided as the edge feature, model struggles to learn and saturates quickly.
21

---

Published as a conference paper at ICLR 2023
a)
b)
10
Training
Training
Test
Test
10
Loss
Loss
10"
10
10
10"
2000
4000
6000
8000
10000
1000
2000
3000
4000
5000
6000
7000
Epoch
Epoch
Figure 18: Loss curve when (a) relative position is used edge feature and (b) relative velocity is used as edge feature.
J.4 TRAINING DETAILS
The training dataset is divided in 75:25 ratio randomly, where the 75% is used for training and 25% used as the validation set. Further, the trained models are tested on ability to predict the correct trajectory, a task it was not trained on. Specifically, the pendulum systems are tested for 1s, that is 10s timesteps, and spring systems for 20s, that is 2 x 10 timesteps on 100 different trajectories created from random initial conditions. All models are trained for 10000 epochs. A learning rate of 10-3 was used with the Adam optimizer for the training.
.ODE
Parameter
Value 2
Hidden layer neurons (MLP) Number of hidden layers (MLP)
Activation function Optimizer
squareplus ADAM
Learning rate
1.0e 3
Batch size
100
GNODE, CGNODE, CDGNODE, MCGNODE
Parameter
Value
Node embedding dimension Edge embedding dimension
Hidden layer neurons (MLP)
Number hidden layers (MLP)
Activation function Number of layers of message passing Optimizer
squareplus
ADAM
Learning rate
1.0e 3
Batch size
100
LLGN, HGN
22

---

Published as a conference paper at ICLR 2023
b)
10
[HAIL FIHAA
151, 1IRE
10
IM0-
10
10
10
10
50
100
150
200
50
200
Time
Time
c)
d)
10
[[H(2) HH2)ll2
[|2 1222
10
10
10
10
10
0
50
100
150
200
50
100
150
200
Time
Time
Figure 19: Plot comparing (a) relative energy error, (b) relative trajectory error, (c) absolute energy eтror, (d) absolute trajectory error for the 5-Spring-MCGNoDE system
23

---

Published as a conference paper at ICLR 2023
a)
b)
10
Training
Training
Test
10"
Test
10"
10"
10
10
Loss
Loss
10
10"
10
10
10
10
1000
2000
3000
4000
5000
2000
4000
6000
8000
10000
Epoch
Epoch
Figure 20: Loss curve for peridynamics system using (a) MCGNODE architecture and (b) GNODE architecture.
10 33
10"
10
10
10
||2122 - ||2122
|| 1|P11
10
10
10
10
10
10 -44
10
Time
Time
GNODE
MCGNODE
Figure 21: Absolute trajectory error error and absolute momentum error for peridynamics simulation using of GNODE, and MCGNODE.
24

---

Published as a conference paper at ICLR 2023
10 L1
10
10
Energy violation
Rollout error
10
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Time
Time
GNODE
GNODE woV
MCGNODE
MCGNODE_WoV
Figure 22: Comparison of Energy violation and rollout error for 5-link pendulum using GNODE, MCGNODE GNODE-wOV (GNODE without velocity input) and MCGNODE-woV (MCGNODE without velocity input).
Parameter
Value
Node embedding dimension Edge embedding dimension
Hidden layer neurons (MLP)
Number of hidden layers (MLP)
Activation function
squareplus
Number of layers of message passing Optimizer Learning rate Batch size
ADAM
1.0e-*
100
K GRAPH ARCHITECTURES: GNODE, CGNODE, CDGNODE, MCGNODE
The detailed differences between each of the architectures presented in the work are as follows.
GNODE
Summary: Graph version NODE that allows inductivity to unseen system sizes.
Node inputs: position (g), velocity (g), and type (1)
Edge inputs: wy = (g: 93)
Node output: acceleration of each particle (ÿ.). Note that do not use Eq. 5 here to predict q.
F(g. t), where F is the approximate dynamics learned by the GNODE. Here, predicting force and acceleration are equivalent problems since the acceleration is simple scaling of the force when the masses of the particles are equal as in the present case.
Instead, directly predict the acceleration using
Edge output: Nil. Architecture: Fig. 8 Here, GNODE directly predicts the acceleration and not the force although
they are equivalent problems.
Loss function: L - A EL
Assumed knowns: None, since we do not use Eq. here.
Inductive bias: Modeling the physical system graph with the particles nodes and connections
25

---

Published as a conference paper at ICLR 2023
between them edges.
CGNODE
Summary: Same as GNODE except for the property that the neural network predicts the force from which acceleration is computed using Eq.
Node inputs: Same as GNODE
Node output: The force N; on each particle i
Edge output: Nil, same as in GNODE
Architecture: Fig- [
Loss function: Same as in GNODE. The acceleration is obtained using Eq. 5 where N is obtained from the CGNODE, A(q) assumed to be known, M the learnable diagonal mass matrix, IT the known external force, C' is O, Y is the learnable external drag. Note that unless specified, TI and Y is considered be 0, that is, the system is not subjected to external force or drag. Details below. Assumed knowns: (i) Governing equation for acceleration
given by Eq. 5
(īi) k constraints
(holonomic or Pfaffian) in the physical system represented by A(g) R&D based on the topology
and nature (rigid/deformable)) of the system. For instance, the bar in the pendulum is rigid whereas the springs are deformable.
Inductive bias: Same GNODE explicit constraints as given by A(g) and acceleration as given by Eq. 5
CDGNODE
Summary: Same as CGNODE except that the architecture is modified to decouple the computations of internal and body forces. The raw node and edge inputs are the same. However, since we learn two representations per node, global and local, these inputs are used in the following manner: Input for global node representations: position (g) and velocity (ọ); these features are not involved message passing.
Input for local node representations: type (1).
Input for local edge representations: W'y = (q: - 9;). i.c., same as in GNODE. Note that node local input and edge inputs are involved in message passing.
Node output: Same as CGNODE, but the local node output and the global node output are concatenated and passed through an MLP to obtain the total node force (N,) shown in Fig. 1. The intuition is that the local node output represents the internal forces (for instance, the spring forces) and the global node output represents the body forces (for instance, the gravitational force), which when combined gives the total force on each particle. This design thus enables more expressive power for the neural network.
Edge outputs: Nil, same as in CGNODE. Architecture: Fig. 1.
Loss function: Same as in CGNODE.
Assumed knowns: Same as in CGNODE.
Inductive bias: Same as in CGNODE decoupling the computation of internal forces (which depend on the topology and is function of the Way) and body forces (which typically depends on the position of the particle rather than the topology of the system, for instance, gravitational or electromagnetic force).
MCGNODE
Summary: Same CDGNODE except that the architecture is modifed to enforce Newton's 's third law by incorporation of an additional MLP to predict the individual interaction forces at the edge level.
Node global input, node local input, edge input: Same as in CDGNODE.
Edge output: Force fy on particle due to the interaction from the neighboring particle j.
Node output: Total force N, for each particle given by N; = Ngi + Σx, fij. where N; represents of the set of all the neighbors of i and N,, represents the force due to external fields obtained by passing the global node output through an MLP.
Architecture: Fig. g with a difference that there are two MLPs, one to compute Ngs which takes as input the global node representation and second that takes the edge representation as input to predict the f-y- We will include a separate figure to explcitly differentiate between CDGNODE and MCGNODE.
26

---

Published as a conference paper at ICLR 2023
node embedding
Rew rode tentures
Graph representation physical
Initial edge
Raw edge festures
GNN
MLP
ScuarePlus
MLP
a) GNODE
Node embedding
Modele body forces
Feerr moddo Iestures
MLP
SquarePlus
Inttial node embedding
Graph representation of physical systom
node fseturon
MLP
MLP
SquarePlus
Initial edge embedaing
Fiew edgo features
GNN
F
Yay
MLP
SquarePlus
Models topology aware internal foroas
b) CGNODE and CDGNODE
embbediing
Models body
Rew nadia leaturee
MLP
SquarePlus
Inibal node embedding
Graph
Row node
representation aystem
Vis
MLP
F
Initial edge embedding
edge feeturos
GNN
F
MLP
MLP
SquarePlus
Modiels topology
c) MCGNODE
Figure 23: Architecture of (a) GNODE, (b) CGNODE, CDGNODE and (c) MCGNODE. Note that the architecture of CGNODE and CDGNODE are equivalent.
Loss function: Same as CGNODE. Assumed knowns: same as in CGNODE.
Inductive bias: Same as in CDGNODE Newton's third law, that is, the internal forces between the particles are equal and opposite.
L MCGNODE VS GNS
The tables demonstrate that GNS performs inferior to MCGNode.
27

---

Published as a conference paper at ICLR 2023
Time(s)
GNS
GNODE
MCGNODE
0.00
0.74
0.92
0.26
1.00
0.88
1.00
0.28
2.00
0.90
1.00
0.30
3.00
0.89
1.00
0.36
4.00
0.81
1.00
0.52
5.00
0.84
1.00
0.80
6.00
0.88
1.00
0.81
8.00
0.88
1.00
0.76
9.00
0.85
1.00
0.75
9.90
0.84
1.00
0.70
Table 3: Momentum error
Time(s)
GNS
GNODE
MCGNODE
0.00
0.00
0.00
0.00
1.00
1.00
0.96
0.01
2.00
1.00
1.00
0.02
3.00
1.00
1.00
0.04
4.00
1.00
1.00
0.06
5.00
1.00
1.00
0,06
6.00
1.00
1.00
0.09
7.00
1.00
1.00
0.12
8.00
1.00
1.00
0.16
9.90
1.00
1.00
0.21
Table 4: Rollout error
28


Reviews:
[{"review_id": "vWb5vNirTA", "reviewer": "Reviewer_zGuL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Neural ODEs are powerful models for learning dynamical systems, however, they cannot be generalized the systems with larger sizes when trained on the smaller systems. In order to solve this problem this paper proposes a graph-based neural ODE (GNODE) which can be inductive and generalize to larger systems with the similar dynamics such as pendulum and springs. Base on success from adding physics-based inductive biases in previous works such as Lagrangian neural networks (LNN) and Hamiltonian neural networks (HNN), this paper also explores the effectiveness of adding various types of physics-based inductive biases to the GNODEs. These inductive biases are (1) Decoupling the dynamics and constraints (CGNODE), (1) Decoupling the internal and body forces (CDGNODE), and (3) Newton’s third law (MCGNODE). They evaluate the models based on three metrics: rollout error, energy violation error, and momentum conservation error. The results show the performance of GNODE model, its different versions with distinct inductive biases, and two competing methods (LGN and HGN). Overall, the performance of MCGNODE is the best among GNODE models in terms of those three evaluation metrics, and it also outperforms the competing methods in the pendulum and spring systems. ", "summary_of_the_review": "I liked the idea of introducing three different types of physics-based inductive biases to the graph-based ODE models. It is always helpful to add known governing rules of the systems to the models in order to guide them to learn the true dynamics. However, I have the following concerns/questions:\n\n- This is related to the usefulness of this method in other dynamical systems. I am convinced that this method should work great in the physical systems where we exactly know some of the underlying rules. However, can it still work so well in other systems such as time-series data? In other words, can bringing such inductive biases still help in the systems that we do not know for sure such assumptions do not hold?\n\n- I think it would really help the reach of paper if it includes time-series data, as right now, the diversity of examples are limited to physical systems. \n\nSome minor comments:\n\n- Typos: \n           an node  --> a node\n           the learn --> to learn\n\n- Should eq (11) have another summation over t?\n- Why the authors use Verlet algorithm to compute the accelerations in eq (12)? Shouldn't the acceleration values be observed in such systems? How to choose \\delta t and how accurate are the estimation of the accelerations using (12)? It would be great if the authors clarify this.\n\n\n", "strength_and_weaknesses": "Strength:\n\n- The authors have done a good job in bridging the physics and ML by bringing such inductive biases which really help learn the dynamics in these systems effectively.\n\nWeakness:\n\n- These inductive biases work well in physics-based systems such as pendulum and spring. The authors have not investigated its efficacy in more general dynamical systems such as time-series. \n\n- Related to the previous point, I think the examples in this paper are not diverse and only show the physical systems, pendulum and spring.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OiC7lVIv_XU", "reviewer": "Reviewer_UJsG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to model the second-order ODE of multi-body dynamical systems by using Graph Neural Networks (GNNs). Different from previous LGN or HGN that directly predicts the Lagrangian/Hamiltonian of the system, this paper explicitly involves the inductive biases of the constraints, internal-external forces, and conservation of Newton’s third law.  Experiments are performed to support the benefit of considering the above biases, and the generalizability to unseen systems of large sizes.", "summary_of_the_review": "I generally enjoy reading this paper and accept the ideas. I have some concerns on the writing and the experiments. ", "strength_and_weaknesses": "Strengths:\n\n1. Indeed, GNNs have been explored widely for physical modeling. This paper applies GNN to characterize the terms of the second-order ODE in Eq. 5, which has contained sufficient terms that reflect conservative/non-conservative forces, internal/external forces, and constraints. This is novel.\n\n2. The authors develop a series of models that take into account different physics priors. The discussion on the relation with Newton’s third law is interesting.\n\n3. Experiments are conducted compactly but convincingly. It is able to justify the validity of the proposed models, particularly CGNODE, CDGNODE, MCGNODE. \n\n\nQuestions:\n\nBelow, I raise some questions (not necessarily weakness) and hope the authors can provide answers to them.\n\n1. Is there any gap like assumption or relaxation between Eq. (2) and Lagrangian dynamics in Eq. (1). If so, please specify.\n\n2. How to compute the Coriolis-like forces in your models, and how to make M inversible in Eq. (4). \n\n3. How does the derivation of lambda by (4) differ from Eq. (7) in the paper [Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints].\n\n4. N_i = Gamma_i =0 in all your experiments?\n\n5. For your models, CGNODE, CDGNODE, MCGNODE, the inductive biases are added independently, or  enforced upon each other? Why not propose a eventual model that considers all biases?\n\n6. For the baselines LGN and HGN, does it share the same GNN backbone as GNODE?\n\n7. What is the implementation of NODE?\n\n8. From Figure 2, it is hard to verify that GNODE is clearly better than NODE. If so, please modify the presentation in the paragraph of “Impact of topology”.\n\n9. Eq. (3), there is a typo. There is no constraint error to check if the proposed model can really permit the constraint of the systems.\n\nSuggestions:\n\n1. Section 3.1 is not the central contribution. The authors are suggested to introduce more on Section 3.2-3.4 including moving their details and illustrated figures from appendix to the main body. \n\n2. The neural network in Figure 1 is just an MLP and unable to show the case of GNN or message passing.\n\n3. It will be interesting to visualize the constraint errors, the difference between the internal force and external force distribution of the learned model. \n\n4. There are recent papers that focus on enhancing symmetry on physical modeling, leading to the application of equivariant GNNs; examples include EGNN, GMN that also involves constraints, SGNN that considers external force field. I am not asking the authors to conduct experimental comparisons wit them, but rather encourage the authors to add some related discussion with the methods in that domain, since symmetry is actually a central topic in physics. For example, if we enforce the Lagrangian to be invariant w.r.t. space translation/rotation, we will also derive the Newton’s third law just based on Lagrangian dynamics. This point is easily justified, by for example checking the Eq. (5) in GMN, where the force message f_ij on each edge is of the same orientation of the relative position x_ij of each two nodes, indicating f_ij=-f_ji.\n\n\n[EGNN] E(n) Equivariant Graph Neural Networks, ICML 2021.\n[GMN] Equivariant graph mechanics networks with constraints, ICLR 2022.\n[SGNN] Learning Physical Dynamics with Subequivariant Graph Neural Networks, NIPS 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E3wUdCfAjC", "reviewer": "Reviewer_gQW5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to combine graph neural networks and neural ordinary differential equations (ODEs) to make the learned model generalize to particle systems with arbitrary sizes. It further studies how different inductive biases that could be embedded into the framework affect the performance. On two synthetic datasets, some benefits of the proposed model over two strong baselines are demonstrated.", "summary_of_the_review": "It looks to me that the proposed method itself is not novel enough and the main contribution is the specific implementation to achieve several inductive biases and an ablation study on them. If this is not correct, the author could clarify this aspect by answering my questions in earlier section.", "strength_and_weaknesses": "## Strength\n- The paper is clear and easy to follow.\n\n## Weakness\n- The motivation in the abstract is weak as generalizing to arbitrary-sized particle systems using graph neural networks with ODEs are already done by LGN/HGN as mentioned.\n- The differences between the proposed method and LGN/HGN are not clear. It looks to me that the proposed method slightly generalize LGN/HGN by implementing different inductive biases. If this is the case, the novelty is limited.\n\n## Questions\n- Can you clarity the difference between MCGnode and LGN/HGN (in terms of inductive bias)? \n- In the result section it point that MCGnode outperforms both LGN and HGN (figure 5) without any explanation on potential reasons/conjectures. Why this happens? \n    - In general, the results section should not be a simple number comparison. You should draw some conclusion from the comparison.\n- How are the different inductive biases studied here similar to [1]? Some comparisons could be useful as [1] is also a recent work with similar focus.\n\n[1] Xu, K., Srivastava, A., Gutfreund, D., Sosa, F., Ullman, T., Tenenbaum, J. and Sutton, C., 2021. A Bayesian-symbolic approach to reasoning and learning in intuitive physics. Advances in Neural Information Processing Systems, 34, pp.2478-2490.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3TATb8WMsD", "reviewer": "Reviewer_okft", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses on inductive biases for effective learning of physical dynamics based on a data-driven approach. The proposed method, GNODE, is an extension of neural ODE and can describe physical dynamics using graph structures and can embed some physics-informed inductive biases. In the experiments, the effectiveness of the proposed method is verified by comparing it with NODE and graph-based NN models (LGN and HGN). It is also applied to transductive settings to achieve good performance.", "summary_of_the_review": "Overall, it addresses an interesting problem and the method is novel. However, there are concerns about the choice of comparison methods and some ambiguities in the manuscript.", "strength_and_weaknesses": "S1. A new graph-based NODE is proposed, which can embed several physics-informed inductive biases.\nS2. In experiments, it is shown that the proposed method can predict trajectories more accurately than NODE while successfully capturing the energy conservation law.\nS3. It is interesting that predictions can be made for unseen systems.\n\nW1. Comparison methods seem to be lacking. For example, in Figure 2, it would be good to have a comparison not only with NODE, but also with models that can use physical knowledge (e.g. Symplectic ODE-Net [R1]).\nW2. The task of predicting unseen systems is very interesting, but I think other approaches (e.g. meta-learning ) are working on similar tasks. I think it would be good to add a discussion of this in the related research section.\nW3. There are several unclear points as follows:\n  - This study is formulated based on the Lagrangian form. What is the reason for choosing the Lagrangian form? Also, are the ideas in this study applicable to Hamiltonian form?\n  - Can you also briefly tell me about the advantages of studying physical systems based on GNN?\n  - The more inductive biases you consider, the more robust you will be against sparsity and noise in your data. If you have any views or experimental results, please let me know.\n\n[R1] Zhong et. al., Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control, ICLR, 2020.\n[R2] Lee et. al., IDENTIFYING PHYSICAL LAW OF HAMILTONIAN SYSTEMS VIA META-LEARNING, ICLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster