Graph network simulators can learn discontinuous, rigid contact dynamics
Authors: Kelsey R Allen
Tatiana Lopez Guevara
Yulia Rubanova
Kim Stachenfeld
Alvaro Sanchez-Gonzalez
Peter Battaglia
Tobias Pfaff
Venue: CoRL 2022 Poster (2022)
Keywords: graph networks
contacts
rigid body dynamics
simulation

Abstract: Recent years have seen a rise in techniques for modeling discontinuous dynamics, such as rigid contact or switching motion modes, using deep learning. A common claim is that deep networks are incapable of accurately modeling rigid-body dynamics without explicit modules for handling contacts, due to the continuous nature of how deep networks are parameterized. Here we investigate this claim with experiments on established real and simulated datasets and show that general-purpose graph network simulators, with no contact-specific assumptions, can learn and predict contact discontinuities. Furthermore, contact dynamics learned by graph network simulators capture real-world cube tossing trajectories more accurately than highly engineered robotics simulators, even when provided with only 8 -- 16 trajectories. Overall, this suggests that rigid-body dynamics do not pose a fundamental challenge for deep networks with the appropriate general architecture and parameterization.  Instead, our work opens new directions for considering when deep learning-based models might be preferable to traditional simulation environments for accurately modeling real-world contact dynamics.

Graph network simulators can learn discontinuous, rigid contact dynamics
Kelsey R. Allen", Tatiana Lopez-Guevara"' Yulia Rubanova, Kimberly Stachenfeld, Alvaro Sanchez-Gonzaler, Peter Battaglia, Tobias Pfaff DeepMind, UK
Abstract: Recent years have seen a rise in techniques for modeling discontinuous dynamics, such as rigid contact switching motion modes, using deep learn- ing. A common claim is that deep networks are incapable of accurately modeling rigid-body dynamics without explicit modules for handling contacts, due the continuous nature of how deep networks are parameterized. Here we investigate this claim with experiments that general-purpose graph network simulators, with no contact-specific assump- tions, can learn and predict contact discontinuities. Furthermore, contact dynam- ics learned by graph network simulators capture real-world cube tossing trajecto- ries more accurately than highly engineered roboties simulators, even when pro- vided with only 8- 16 trajectories. Overall, this suggests that rigid-body dynamics do not pose fundamental challenge for deep networks with the appropriate gen- eral architecture and parameterization. Instead, our work opens new directions for considering when deep learning-base based models might be preferable to traditional simulation environments for accurately modeling real-world contact dynamics.
established real and simulated datasets and show
Keywords: graph networks, contacts, rigid body dynamics
1 Introduction
Model-hased control relies on the ability to predict how objects in the world are going to be affected by an agent's actions. Considerable effort has therefore been invested into building precise analyt- ical models of physical dynamics, such that these can be used effectively and efficiently for safe planning and control. However, some effects such as frictional contacts temperaturee-deeeddeeee in actuators are hard to model or to estimate via system identification, resulting in significant drift when such analytical models counteract these problems, the last few years have seen increased interest in learned dynamics mod- els which be directly fit real data [1, 2], on hardware accelerators significantly speed up policy, model training and inference [3, 4], and may side-step issues of inferring state representations by operating directly at the pixel level [5, 6, 7].
used to predict real-world dynamics (the "sim-to-ren gap"). To
With the increased attention on learned models, there is an incentive to understand which kinds of models can accurately capture the different types of dynamics a robot may encounter. In particular, many tasks robotics, from locomotion to manipulation, require making and breaking contact with rigid objects. This phenomenon is known to be discontinuous [8, 9]. the spatial and temporal scales on which rigid bodies deform during contact are imperceptibly small. Recent work has sug- gested that modelling contacts presents a fundamental challenge for deep learning models because of their tendency toward smooth interpolation [10, 11], but also even for analytic robotics simulators when compared to real- world data [12, 1].
To remedy these issues with body dynamics learning, several papers have proposed combining deep learning with analytic rigid-body dynamics models to varying degrees [1, 2. 11, 13, 14, 15]. While such techniques can improve stability and prediction accuracy in some cases, they also come
'Thee authors contributed equally. Correspondence to /crallen,zppllitatt/ddemmmi..
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

with some of the same drawbacks as analytical solvers: the analytical components may pose re- strictions on the type of objects and dynamics that can be simulated, may be hard parameterize correctly, or may induce strict time-stepping limits. This precludes the more general use of these models wider variety of physical phenomena, such as interactions between rigid bodies and flu- ids or deformable objects, and might hamper generalization to more complex, real-world situations in which e.g. contact forces are not perfectly preserved.
Here we show that there are simple, general ways to use deep learning to accurately model discon- tinuous contact dynamics without specific rigid con- tact parameterizations, and which lead to better perfor- mance for real-world tra- jectory modeling. We find that graph network simula- tors [16, 3]. are sufficient to capture rigid body dynam- ics on both simulated and real world data with excel- lent sample efficiency, out- performing learned mod- els with explicit handling of rigid contacts [2]. For modeling real world cube tosses, graph network sim- ulators can even outperform carefully tuned analytic simulators common in robotics, such as Drake [17], Bullet [18], and MuloCo [19]. We perform carefully controlled experiments on variety of rigid-body dynamics environments, and demonstrate the specific architectural choices for graph network simulators that support discontinuous dynamics learning. These results demonstrate that explicit modeling of rigid contacts is not necessary for deep learning methods to handle discontin- uous dynamics; instead, we find that general- penrrol-purpose, relational, translation-equivariae models are equally or more capable of handling these events precisely.
Ground truih (a)
(b)
Z velocity
-0.05-
Predicted
Ground truth Predicted 1s
-0.10-
Time
Number Timesteps
Figure 1: (a) A 2D projection of sample 3D trajectory from Pfrom- mer et al. [2] of a cube being tossed onto table (orange) and the pre- dictions obtained by Graph Network Simulator (GNS) (blue). The cube contacts the table along an edge, bouncing into the air. The red line shows the time of contact. (b) The GNS is capable of represent- ing and correctly resolving the velocity discontinuity introduced by the rigid contact event (dashed red line), closely matching the ground truth. The shaded area shows 95% confidence interval across 10 seeds.
2 Related work
Dynamics with discontinuities, such as rigid-body collisions frictional transitions, challeng- ing to learn due to the non-smooth and discontinuous nature of the dynamics involved [2. 10, I]. Classical approaches model such behaviour via rigid [8] or approximately rigid contact mod- els [20, 21, 22]. However, there is clear evidence that these methods struggle to capture the be- haviour of real world rigid impacts [1, 12, 23].
Another approach is to learn a function that approximates the rigid-body dynamics directly from data. For example, Gaussian processes (GPs) are popular technique that has been used to control robotic arms [24], learn slip-stick transitions [25] and model planar contact [1]. GPs are data- efficient but include smoothness assumptions that do not generalize well to discontinuous dynamics.
To better capture discontinuities while still leveraging access to data, a variety of recent methods use machine learning to augment analytic physics simulators. Some techniques parameterize contact dynamics as a linear complementarity program, and provide differentiable solvers that can be cou- pled with deep networks [13, 26]. Others incorporate a non-differentiable physics engine to make predictions, but also learn a residual model to augment those such that they more closely reflect ob- served dynamics [1, 14, 15]. Another class of approaches uses implicit models that embed analytic equations and solvers within end-to-end techniques [2. 11]. In all cases, these techniques explicitly encode rigid-body interactions, which limits their application richer physical phenomena.
Over the past several years, Graph Neural Networks (GNNs) have been increasingly applied to physical simulation. These models explicitly learn interactions between entities parameterized with

---

Graph Network Simulator: s(M r-hst
M
M
Updater
M
M4
Generate Mesh Seq
Encode Mesh
Per Node Prediction
T
Encoder
Processor
Decoder
LXK steps
MP
M
Figure 2: Graph Network Simulator (GNS) model for rigid shapes. We construct the input meshes 4 by applying the object center mass's translation and rotation T to the undeformed object shape M The encoder build graph from M and encodes relative positional features into graph nodes and edges. Next, the processor computes L x message passing steps on this graph. The decoder outputs accelerations a; for each node, which integrated into updated node positions.
neural networks. Early applications of GNNs modeled interactions between simple rigid entities like bouncing billiards [27]. and later work extended this more complex domains like pushed and falling shapes [28, 29J, moving robot bodies [30, 31, 32], and even fluids and deformable meshes [33, 16, 3]. Despite the successes of these models many areas, there have been careful analyses of how they perform in settings with hard discontinuities. Furthermore, most results are shown only in simulation relative to some ground truth simulator, which are themselves often inaccurate [1, 12]. Exacerbating this concern, recent papers have implied that there are simple cases where deep networks fail learn accurate collision dynamics [34, 10. 11]. These results cast doubt on whether end-to-end deep learning, models, including GNNs, are generally good fit for learning rigid- interactions, particularly if the rigid bodies are complex, volumetric objects like cubes or cylinders.
In this work, we show for the first time that GNN-based simulators, constructed with simple design choices, are capable of capturing real-world rigid contacts for some object shapes as effectively or better than analytic models [19, 18, 17] and learned models with explicit contact handling [2]. We conduct series of ablations and experiments shed light exactly which properties of GNNs enable this high performance.
3 Approach
rigid-body simulation, real-world experiment, can be described by the object shapes (e.g. rep- resented by an undeformed mesh M with nodes corresponding to object vertices, and edges corre- sponding object faces), and sequence of rigid transformations
(T.o...., T**) consisting of the translation and rotation at each time step l. Here we consider a GNN-based approach to simu- lation, trained using sequences meshes M (M ,M.',..., M**), where each mesh consists of the absolute position } for each node, as well as edges connecting these nodes. We can obtain such a sequence by applying to the initial object shape M as (T "...., Th (M1)).
3.1 Learning Forward Dynamics
We define the Graph Network simulator (GNS) model which given M and short history of states {M'-1 ..... Mt-h} (we use h = 2), outputs a next state mesh prediction 1/+11 This model can be iteratively applied for k steps to predict trajectory (M M Process-Decode GNN [3, 16, 35], adapted ..... N,*,)) (Figure 2). is rigid bodies. Below we describe the model components. We specifically consider the case of an object interacting with the floor, but multi-object collisions can be treated analogously using world edges as [3].
represented by an
The encoder constructs graph G (V. E) from the input mesh M using the mesh vertices nodes V. and mesh edges as graph edges Each node has features Vú2 [v.") b,), where vFD (x/ x x 2---11 x is the node's finiteddifferenc (FD) velocity over the input history, and b; is the distance to the domain boundaries (here, the floor). The boundary distance b, is
Encoder

---

clipped to the range 0.5) as in [16], to prevent overfitting for states far away from contact. An edge from node to j has features Eig = [d.z. ||d, a أz. ..Ild" y|lB, where dij x; x, is a vector of the relative displacement between the nodes and j. Similarly, dý denotes the displacement between these nodes on the undeformed initial mesh M which can help the model to retain the rigid shape. We note that this is local and translation-equivaria encoding of the simulation state, which is particularly important in the data-sparse setting. We also consider the commonly used absolute position encoding [33, 29, 28, 30], in which V; [v x,, x ). Paj = { which does not have these properties. All node and edges features, respectively, are concatenated and passed to encoder multi-layer perceptrons (MLPs) that produce latent vectors of size 128 for each node and edge in the graph.
Processor After encoding features, the processor computes several steps of pesssag--passing amongst the nodes in the graph as in [16]. In each message-pant passseg step, each latent edge Eig and node feature V; is updated to Egg and V respectively according to:
Figh + P (Fig, Vi), V + P (V, Σ, Fly)
(1)
with o and o implemented as MLPs. A message passing block consists of L message passing updates with unshared weights for all pe- Each step, this block repeated K times, resulting in L x K updates with (L) weights.
Decoder The decoder MLP 8V decodes the node latent features after the last message passing step to predict the por-node finite-difference acceleration estimate a, for each node Compared to predicting absolution position, this is soft bias for inertial dynamics.
Model training and data augmentation We compute the loss the per-node wean-squared-error on acceleration prediction, and train the model using the Adam optimizer. The ground-truth accel- eration estimated from the position data using finite differences. While the model is translation- cquivariant by design, we can exploit an additional symmetry of the problem: during data loading of training pair, we rotate both input and target around the (up) direction with an angle uniformly sampled from [- x.. This decreases overfitting without requiring additional data.
Updater To generate simulation trajectories at test time, the model is applied iteratively. After each step, the updater integrates the per-node acceleration predictions with an Euler integrator to compute the next-step position as x +1 a, + 2x x For long rollouts, positional error can accumulate and deform the object shape. As the only architectural distinction from Pfaff et al. [3], we prevent this by using shape matching [36] during rollout. That is, after each step we fit a transformation to the predicted node positions, and re-project the nodes to the transformed mesh to enforce shape consistency (see Appendix B for details). We do not use shape matching during training and compute the loss directly on network predictions.
Results
Rigid-body dynamics introduce two unique challenges for learning dynamics models: (I) discon- tinuous dynamics during rigid collisions, and (2) extreme sensitivity to initial conditions. Our experiments in this section provide a qualitative evaluation of the capabilities to handle the aforementioned problems, well as quantitative comparisons with models encod- ing rigid contact-specific priors. For videos of rollouts and access to code, please see https: com//vies/gmnnrriiidaa www.:/gnnniiiihoooooooeo homee
GNS models
// //sites google. www.coie//ee/eeece
4.1 Diagnostic qualitative results
We first perform a qualitative evaluation of whether a GNS can capture discontinuous dynamics and its sensitivity to initial conditions. The GNS in Figure I was trained on 256 real cube tossing trajec- tories from [2), while the GNS for Figure was trained on 8096 simulated MuJoCo trajectories from [10]. All figures show projections of 3D trajectories. See details on the training data in Appendix A.

---

(b)
Positional errer
(c)
Rotational error
(d)
Penetration
40
60
10
Mijico
Gege ContacsNela
Draks
40
degrees (+)
% wieth
% width
20
20
10
4
8
16
32
64
128256
4 8 16 32 64 128256 Training Trajectories
8 16
32 64 128256
Figure 4: (a) Real world cube throwing dataset (Figure 3 from Pfrommer et al. [2]). (b) (d): performance metrics from [2] indicating positional eπrоr, rotation error, and predicted penetration between the cube and the table. Lower is better. GNS (blue) more accurate than the ContactNets- Polytope method [2] (red) for > 64 training trajectories, and outperforms simulation with analytical solvers with between 8 and 128 trajectories (dotted lines).
Discontinuous dynamics Consider the scenario shown in Figure ((a). In this simple setup, cube thrown onto the ground such that it contacts the floor along one its edges. This contact produces sharp discontinuity in the velocity as the cube bounces upwards. In Figure 1(b), the graph network simulator is able to capture the discontinuities on each bounce accurately, matching the real data closely (even across 10 separate training seeds). This runs counter to prevailing belief (c.g. [10]) which suggests that deep networks struggle to learn such discontinuities.
(a)
(b)
Sensitivity to initial conditions Next, consider the sce- nario shown in Figure 3. In this setup. a cube is falling di- rectly downwards with a rotation 0 close to 45% For any angle < 45%, the cube will fall to the left, while for any angle 0 >> 45%, the cube will fall to the right. Here the model must be acutely sensitive to rotations differing by fractions of degree, these will significantly affect the final position of the cube. We see that the graph network simulator closely captures the discontinuous behavior in this diagnostic scenario: even for barely perceptible devi- ations of 0.25%, the simulator correctly predicts the final position of the block (Figure 3a,b). This further under- scores the ability of the learned model to handle sharp discontinuities and multi-modal dynamics without pro- ducing smoothed interpolations of the training data.
44.75
0= 45.25
Final Position f9
40
44
45
48
49
50
Angle (Degreas)
Figure 3: Graph Network Simulators learn to be appropriately sensitive to ini- tial conditions, including capturing dis- continuities. A cube is correctly pre- dicted fall to the left when angled 45 respective to the floor, and right for angles 45.
4.2 Quantitative accuracy comparisons
In this section, we measure the performance of Graph Network Simulators on two established cube tossing datasets from the robotics community designed to capture challenges in rigid-body dynamics: real-world cube tossing dataset from Pfrommer al. [2], and a MuJoCo-simulatte cube tossing dataset from Parmar et al. [10]. We also investigate performance on a subset of the more complex Kubric MOVi-A dataset [37].
Experiment setup The datasets from [2, 10] provide the center position and the rotation of the cube each time point, T (-). We convert these data into the positions for each node M as our model operates over nodes instead of center/rotation. We follow the experimental setup from [2, 10], and compute the following metrics: Positional error: mean-squared error (MSE) over the position of the cube center as percentage of the cube width; Rotational error: the angle between the ground- truth rotation and the prediction degrees; Penetration: the distance of cube penetration through the floor as percentage of the cube width; all averaged over the simulation trajectory. Following [2, 10], we train the model on a subset trajectories from the dataset and show how the error on the test set changes function of the training set size.
Real cubes Pfrommer al. [2] introduce a real cube tossing dataset, providing cube positions and orientations at each time step. We compare Graph Network Simulators to the implicit ContactNets
5

---

Positional error
Rotational error
(c)
Penetration
10
100
80
chirecthirss
width
60
degrees (+)
% width
GNS [-5m]
(abs)
GNS (3me)
GNS (vuus)
20
10
10
128256
128256
128256
Training Trajectories
Figure 5: Impact of different architectural choices for GNS. Lack of relative encoding (yellow) causes unstable predictions; the method is however relatively robust to other changes, with ablations decreasing performance but still allowing for better predictions than ContactNets at 256 trajectories.
model (which uses rigid-contact specific priors [2]) trained on this real data. We find that the GNS outperforms ContactNets across all the presented metrics: rotational and position enror, well as penetration depth. Perhaps more surprisingly, we find that the GNS outperforms carefully hard- coded physics engines (Drake [17]. PyBullet [18] and Mujoco [19]) in predicting the outcomes of held-out real cube tosses. To compare the real data to these engines, Acosta et al. [12] used system identification determine the best fitting parameters for each engine match the real cube tossing experiment. Their results are shown as the dashed horizontal lines in Figure 4 (penetration was not reported). The GNS trained on real cube data is able to outperform all analytic simulators in predicting position and rotation (requiring 256 trajectories for superior position prediction, and as few 64 for rotation prediction). This suggests that the GNS, in certain circumstances, may be better choice for real-world transfer than physics simulation engines. For example, a GNS may be preferable to circumvent error-prone system identification, or to adapt eccentricities of the real world setup which may not be explicitly modeled by analytical simulators.
Ablations In order understand the contribution of different components of the Graph Network Simulator, we conduct series of ablations (Figure 5) on this dataset. These include encoding and predicting absolute positions on node positions instead of encoding relative displacement and predicting accelerations (GNS [abs]), removing shape matching (GNS [-sm]), limiting message- passing steps to (GNS [1-mp]), and removing data augmentation (GNS [-aug]). GNS [abs] shows the largest difference: without the bias for inertial motion, the model has make predictions much larger dynamic range. Breaking translational equivariance also increases overfitting, leading to unstable rollouts. Intriguingly, almost all ablations still perform better than or comparably to ContactNets on the Penetration depth metric, reinforcing the observation that graph networks can learn to resolve rigid collisions without explicit contact priors using very few training trajectories. Further ablations can be found in Appendix C.
Robustness to noise Finally, performed two experiments investigate the robustness of the GNS model to different types of noise. First, to simulate imperfect shape recovery from sensors, add Gaussian noise with standard deviation individually the corners of the cube during training. There is a graceful degradation of the positional and rotational error as noise is increased (Figure 7a). Second, simulate imperfect object tracking, add noise the cube's tracked
(a)
(b)
Positional error
(c)
Rotational error
40
30
56 width
20
20
hard
mod um
10
sift
hard
medium
soft
10a Training Trajectories
10a
10a Training Trajectories
10a
Figure 6: (a) Renders from MuJoCo with different levels of ground penetration: hard, medium and
soft (Figure 3 from Parmar et al. [10]). (b), (c): performance metrics from [10] indicating positional and rotation error between the cube and the table, GNS (blue) shows significantly lower error rates
than GRU [10], and better sample efficiency, for all stiffness levels.

---

(a) Shape/Mesh Noise
(b) Position Noise
Positional error
Rotational error
Positional error
Rotational error
width
2.1
widch
15
(Mi the (,00)
10
0.005
Noise
Figure 7: Testing the model's robustness to noise. (a) Shape/Mesh noise is modeled by jittering the positions of the vertices in the cube with Gaussian noise centered at O with standard deviation o (x-axis; a relative cube side length of 2.0). (b) Position noise is modeled by jittering the position of the cube at each time-point in the trajectory relative to its current velocity. Noise is reported as a fraction of the cube's change in position between time-steps. Details in Appendix E.
position independently for each time-step of the trajectory during training. There is similarly graceful degradation the rollout error for both cube position and rotation noise is increased (Figure 7b). In both cases, performance is better or comparable to ContactNets even for moderate amounts of noise.
Simulated cubes with different stiffness In contrast to other rigid-body solvers, MuJoCo uses a "soft" penalty-based contact model, with the contact stiffness determining how rigid the contacts will be. Parmar et al. [10] explicitly vary this stiffness parameter in Mujoco to investigate the ability of deep learning systems to model rigid vs. soft contacts. They show that a gated recurrent neural network (GRU) performs worse when the data is generated using stiff contacts than when it is generated using soft contacts, and note that this isolates one the fundamental challenges for deep learning models of rigid-body dynamics.
Here we show that GNSs are not very sensitive to the rigidity with which contacts are simulated. As in Parmar et al. [10], we train the GNS on up to 10,000 trajectories (a total of 1e6 time steps) for each of the three stiffness levels provided, and compare the GNS to the GRU reported in Parmar et al. [10] in Figure 6b,c. The GNS not only outperforms the GRU, but more importantly shows little difference its ability to model soft, medium or hard contacts, particularly as the number of training trajectories grows.
Tossing other objects Finally, we train a model on the Kubric MOVi-A dataset [37] which pro- vides trajectories of complex shapes being tossed on the floor. This dataset is more diverse, consist- ing of three object shapes (a cube with beveled edges, cylinder, sphere) with 51-64 vertices each, two object sizes, and rollouts are shown Figure 8(b). For this dataset, we additionally append the friction and restitu- tion coefficients a node features, allow the model distinguish materials. We train a single model for all object shapes and variants, and observe that we can achieve similarly low error rates of 14.6% position error and 17.2 rotational
of material properties (friction and restitution coefficients). Example
even in this more complicated domain (Figure 8(c)). The dataset was generated using Bullet, which uses hard collision constraints; the model can learn replicate this behavior, showing an extremely low penetration error of 0.016% in this domain.
(a)
(b)
(c)
Positional error
Rotational error
Penetration
30
30
0.08
de width
20
degrees
20
de widtt
0.04
15
15
0.02
0.01
10
10
GNS
[L=5,6-11
Figure 8: Tossing geometric objects with 51 64 vertices (a) from the Kubric MOVi-A dataset [37]. Even in this complicated setup with multiple shapes and friction coefficients, GNS is able to produce accurate rollouts (b, blue: prediction, green: Bullet [18] ground truth), and low error rates (c, blue).
7

---

In general, we see similar trends in ablation experiments as for the cube datasets e.g. GNS [abs] also leads unstable rollouts. We do however note a few differences: as seen in Figure 8(c), the model benefits from more message passing steps (7. = 10, K = 3. blue) vs (7. 5, K 1, gray), to compute collision response on these more detailed shapes. Due to the higher shape complexity, we observe some (small) benefits from including rest position features (d ). And finally, while one might suspect that it would be harder to retain object rigidity for diverse shapes with more vertices, we observe very little shape deformation in this dataset, and model variants with and without shape matching perform equally well. This suggests that even the limited biases we built in to enforce rigidity may be lifted with more data diversity, paving the way for more general, unified simulation models involving rigid and deformable objects.
5
Discussion
We showed that Graph Network Simulators can precisely and accurately capture rigid-body dynam- ics, including discontinuities introduced by contact. A trained GNS outperforms not only other end- to-end approaches, but also other learning approaches with explicit contact assumptions [2]. Our model, trained with sufficient real-world data, is on average more accurate than real-to-sim using analytical physics engines (Drake [17], PyBullet [18] and MuJoCo [19]), matched to the same real- world experiment setup. In simulation, GNSs perform equally well on data simulated with variable levels of stiffness, and can also model other shapes, like cylinders, spheres, and beveled cubes.
We found that all aspects of the graph network simulator were necessary for best performance, especially when few trajectories were available. Remarkably, with as few as 8 real world trajectories, graph network simulators performed on par with a tuned MuJoCo model for predicting held-out real world samples (64-256 trajectories were required to outperform Bullet [18] and Drake [17]). This suggests that, at least in certain circumstances, graph network simulators may actually be a more appropriate model of real-world physical dynamics than specialized hard-coded simulators.
The effectiveness of graph network simulators for rigid contact stems from two factors: (1) operating on the explicit mesh geometry, which makes it easier to reason about contact and learn effective collision detection, and (2) a set of very general physical inductive biases, such inertial motion and translational equivariance. Notably, these biases are not specific to rigid bodies, allowing methods with similar architectures to be applied e.g. deformable objects. Contrary prior claims that deep models struggle with discontinuous dynamics, Graph Network Simulators seem able to model discontinuities in rigid body motion after training on very few sample trajectories. We emphasize that there are still settings for which one might want to Iearn simulation models that exactly respect analytical contact equations, as in [2, 11, 13]. However, for real-world data with dynamics which are hard to model analytically, or estimate parameters for, Graph Network Simulators are an attractive option for accurately learning dynamics end-to-end.
Limitations In this work, we focused on demonstrating that generic Graph Network Simulators can accurately represent discontinuities arising from rigid-body collisions, but there are other aspects to rigid-body dynamics as well. It would be interesting train GNS models on larger, more com- plex datasets beyond tossing single cubes, cylinders and spheres, including more complex shapes, multiple interacting rigid objects, and complex robotic manipulation locomotion data.
Robots rarely have access to precise state information (here, the locations and poses of all objects). Instead, they have sensors that can provide haptic or visual information. 'he graph network sim- ulators presented here rely on being able to extract the states of objects in the scene. While there has been some progress coupling visual estimation with graph network simulation [38, 39, 40], it remains unclear whether such combinations could support the level of precision needed to make predictions about rigid body dynamics.
Finally, despite being good forward models of dynamics, this does not guarantee that Graph Network Simulators will be useful for planning and control. Even when analytic differentiable rigid- body physics simulators are available [41, 4, 42, 43, 13], planning through contact poses a variety of challenges, such as combinatorics of local optima [44, 45, 8, 46) which can hamper finding globally optimal solution. While GNSs may be useful as differentiable models of more general physical phenomena, they will not solve these particular challenges from planning and control directly.
8

---

Acknowledgments
We would like to thank Luis Piloto for reviewing initial version of the paper and Yusuf Aytar for helpful discussions.
References
[1] N. Fazeli, E. Donlon, E. Drumwright, and A. Rodriguez. Empirical evaluation of common contact models for planar impact. In 2017 IEEE international conference on robolics and automation (ICRA), pages 3418-3425. IEEE, 2017.
[2] S. Pfrommer, M. Halm, and M. Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. Conference on Robot Learning (CoRL), 2020.
[3] T. Pfaff, M. Fortunato, A. Sanchez-Gonzales and P. Battaglia. Learning mesh-based simula- tion with graph networks. In International Conference on Learning Reprexentations, 2021.
[4] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax-a differen- tiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.132 2021.
[5] Ebert, Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.00566 2018.
[6] C. Finn, I. Goodfellow, and S. Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
[7] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
[8] P.B.. Wieber, R. Tedrake, and S. Kuindersma. Modeling and control of legged robots. In Springer handbook of robotics, pages 1203-120 1234. Springer, 2016.
[9] M. Posa, M. Tobenkin, and R. Tedrake. Stability analysis and control of rigid-body systems with impacts and friction. IEEE Transactions on Automatic Control (TAC), 61(6))1423-14437 jun 2016.
[10] M. Parmar, M. Halm, and M. Posa. Fundamental challenges in deep learning for stiff contact dynamics. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5181-5188. IEEE, 2021.
[11] Y. Jiang, J. Sun, and C. K. Liu. Data-augmented contact model for rigid body simulation. In Learning for Dynamics and Control Conference, pages 378-390. PMLR, 2022.
[12] B. Acosta, W. Yang, and M. Posa. Validating robotics simulators on real-world impacts. IEEE Robotics and Automation Letters, 2022.
[13] F. de Avila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z. Kolter. End-to-end differentiable physics for learning and control. Advances in Neural Information Processing Systems 31, pages 7
Curran Associates, Inc., 2018.
[14] A. Ajay, J. Wu, N. Fazeli, M. Bauza, P. Kaelbling, J. B. Tenenbaum, and Rodriguez. Augmenting physical simulators with stochastic neural networks: Case study of planar pushing and bouncing. In International Conference on Intelligent Robots and Systems (IROS), 2018.
[15] A. Kloss, S. Schaal, and J. Bohg. Combining learned and analytical models for predicting action effects. CoRR, abs/1710.04100 2017.
[16] A. Sanzaez- J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia. Learning to simulate complex physics with graph networks. International Conference on Machine Learning, pages 8459-8466 PMLR, 2020.

---

[17] R. Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL https: /trkee.rreeeee w.m.miiedeee edu.
[18] E. Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses, page 7.
[19] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[20] M. Anitescu and A. Potra. Formulating dynamic multi-rigid- contact problems with friction as solvable linear complementarity problems. Nonlinear Dynamics, 14(3):231-247, 1997.
[21] D. Stewart and J. J. Trinkle. An implicit time-stepping scheme for rigid body dynamics with Coulomb friction. International Journal for Numerical Methods Engineering, 39(1))22673- 2691, 1996.
[22] I. Mordatch, Z. Popović, and E. Todorov. Contact-Invarian Optimization for Hand Manipu- lation. SCA '12 Proceedings of the ACM SICGRAPH//Euographiss Symposium on Computer Animation, pages 137-144, 2012.
[23] A. Chatterjee. On the realism of complementarity conditions in rigid body collisions. Nonlin- ear Dynamics, 20(2):159-168, 1999.
[24] M. Deisenroth, C. Rasmussen, and D. Fox. Learning control low-cost manipulator using data-efficient reinforcement learning. In Robotics: Science and Systems (RSS), 2011.
[25] M. Bauza and A. Rodriguez. GP-SUM, gaussian processes filtering of non-gaussian beliefs. 2017.
[26] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization layer in neural networks. In International Conference on Machine Learning (ICML), 2017.
[27] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, and k. kavukcuoglu. Interaction net- works for learning about objects, relations and physics. In Advances in Neural Information Processing Systems 29, pages 4510- Curran Associates, Inc., 2016.
(28) D. Mrowca, C. Zhuang, F. Wang, N. Haber, L. F. Fei-Fei, J. Tenenbaum, and D. L.. Yamins. Flexible neural representation for physics prediction. Advances in neural information process- ing systems, 31, 2018.
[29] Li, Wu. J. Zhu, B. Tenenbaum, A. Torralba, and R. Tedrake. Propagation networks for bodel-based control under partial observation. In 2019 International Conference Roboties and Automation (ICRA), pages 1205-1211. IEEF, 2019.
[30] A. Sanzaez- N. Heess, J. T. Springenberg. J. Merel, M. Riedmiller, R. Hadsell, and Battaglia. Graph networks as learnable physics engines for inference and control. In Inter- national Conference on Machine Learning, pages 4470-4479, PMLR, 2018.
[31] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. Efros. Learning to control self assembiseniligg mor- phologies: A study of generalization via modularity. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Bus F. Fox, and R. Garnett, editors, Advances Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019.
[32] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet:
Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018.
[33] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019.
[34] B. Pfrommer and Daniilidis. Tagslam: Robust slam with fiducial markers. arXiv preprint arXiv: 1910.00679 2019.
10

---

[35] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1806.01261,, 2018.
[36] M. Mülllr, B. Heidelberger, M. Teschner, and M. Gross. Meshless deformations based on shape matching. ACM transactions on graphics (TOG), 24(3):47--775 2005.
[37] K. Greff, Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. J. Fleet, D. Gnanapra- gasam, F. Golemo, C. Herrmann, T. Kipf, A. Kundu, D. Lagun, I. Laradji, H.-T. D. Liu, H. Meyer, Y. Miao, D. Nowrouzezahrai, C. Oztireli, E. Pot, N. Radwan, D. Rebain, S. Sabour, M. S. M. Sajjadi, M. Sela, V. Sitzmann, A. Stone, D. Sun, S. Vora, Z. Wang, T. Wu, K. M. Yi, F. Zhong, and A. Tagliasacchi. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), pages 3749- 3761, June 2022.
[38] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint arXiv:2202. //855, 2022.
[39] Veličković, M. Bošnjak, T. Kipf, A. Lerchner, R. Hadsell, R. Pascanu, and C. Blundell. Reasoning-modulatee representations. arXiv preprint arXiv:2107.0888 2021.
[40] T. Kipf, E. van der Pol, and M. Welling. Contrastive learning of structured world models. arXiv preprint arXiv: 122447, 2019.
[41] Y. Hu, I.. Anderson, T.-M. Li, Q- Sun, N. Carr, J. Ragan-Kelley, and Durand. Difftaichi: Differentiable programming for physical simulation. In International Conference on Learning Representations, 2020.
[42] J. K. Murthy, M. Macklin, F. Golemo, V. Voleti, L. Petrini, M. Weiss, B. Considine, J. Parent- Lévesque, K. Xie, K. Erleben, et al. gradsim: Differentiable simulation for system identi- fication and visuomotor control. In International Conference on Learning Representations, 2020.
[43] T. Howell, S. Le Cleac'l h, Z. Kolter, M. Schwager, and Z. Manchester. Dojo: A differentiable simulator for robotics. arXiv preprint arXiv: 200::202300000 2022.
[44] M. Toussaint, K. R. Allen, K. A. Smith, and B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning extended abtract. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, ISCAI-19, pages 1235-6235. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
[45] M. Posa, S. Kuindersma, and R. Tedrake. Optimization and stabilization of trajectories for constrained dynamical systems. In 2016 IEEE International Conference on Robotics and Automation, volume 2016-June, pages 1366-1373, Stockholm, Sweden, may 2016. ISBN 9781 178146338023.
[46] H. Suh, M. Simchowitz, K. Zhang, T. Pang, and R. Tedrake. Pathologies and challenges of using differentiable simulators in policy optimization for contact-rich manipulation. In ICRA 2022 Workshop: Reinforcement Learning for Contact-Ria Manipulation, 2022.


Reviews:
[{"review_id": "tuSCQfNVdVA", "reviewer": "Reviewer_ETfg", "summary": "The paper investigates the applicability of general-purpose graph neural network simulators for modelling discontinuous dynamics. \n\nSimilar to existing work, the proposed method learns to predict the dynamics of a given mesh using a graph neural network that is trained with a one-step prediction loss. The resulting model is evaluated on different simulated and real-world datasets, performing on par or better than commonly used simulators, even when trained on relatively few samples. This result contradicts the previously held notion that this kind of learned simulator is unable to model discontinuous dynamics, and thus opens various avenues for further research.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper shows that general-purpose graph neural network simulators for can be used to model simple discontinuous dynamics. More precisely, it shows that a slight variation of the previously proposed MeshGraphNet [1] architecture can accurately model simple rigid contact dynamics, up to a precision that matches and sometimes surpasses that of commonly used solvers. This is an interesting finding, as it contradicts the previous believe that learned general-purpose simulators are ill-suited to model discontinuous dynamics.\n- The MeshGraphNet architecture is adapted to rigid body simulation by employing a shape-matching projection during the rollout phase. While the performance improvement from this additional step is minor in the measured metrics, this shape-matching is interesting as it ensures that the simulated rigid bodies can not deform over the course of a rollout.\n- The experiments are thorough, and both model architecture and chosen training hyperparameters are clearly explained in the text. Experiments are repeated for $10$ seeds, and confidence intervals are reported. Different parts of the proposed method are explored through a series of ablations.\n- The paper has a clear story and is well written. The figures are clean and complement the text nicely.\n\n[1] Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., & Battaglia, P. (2020, September). Learning Mesh-Based Simulation with Graph Networks. In *International Conference on Learning Representations*\n\n### Weaknesses\n\n- While the results are interesting and promising, the proposed method heavily builds on previous work thus lacks novelty. Section 3.1. essentially re-introduces [1] with some minor changes here and there.\n- The experiments are rather limited in scope. While they shows a clear proof of concept, it would be interesting to see how well the method works for more complex shapes.\n- Very few baselines are considered in the experimental section, making it hard to directly compare the approach to existing work. This is somewhat alleviated through various ablations and comparisons to classical solvers.\n- While most design decisions are ablated over, some interesting deviations from the MeshGraphNet paper remain unexplained. For example the message passing block is iterated over $K$ times, where $K=1$ in previous work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents and evaluates the use of general-purposes graph neural network simulators for modelling discontinuous dynamics. It does not present much in terms of novel architectures and only considers relatively simple tasks with few baselines.\n\nDespite these shortcomings, the paper highlights an interesting new application for graph network simulators and supports its claims with an overall thorough experimental section and various ablations.\n\nPut together, I currently vote to accept the paper since it gives a clear new insights in a field that is not yet well understood.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- As mentioned above, Section 3.1 is very similar to the MeshGraphNet paper. Clearly separating the basic MeshGraphNet architecture and the changes/adaptations made in this paper would make it easier to distinguish previous work from the contributions of this one.\n- An additional paragraph explaining these adaptations would make them easier to understand. For example, it is not quite clear why the message passing block is iterated over $K$ times per update (and what $K$ is set to, as $K=1$ in previous work), or why a history of the $h=2$ previous meshes is used instead of $h=1$ in previous work.\n- It should be clarified why some of the real-world tosses were removed from the dataset (as mentioned in Appendix A).\n- In the appendix, it is shown that data augmentation hurts performance when training on a lot of demonstrations. An additional sentence or two on why this may be the case would be useful for understanding and validating the choice of data augmentation.\n- The last paragraph of Section 5 before the `Limitations' hypothesizes why the used graph network simulators are able to model discontinuous dynamics. This hypothesis should not be stated as fact, as it is not directly proven in the paper.\n- In Figure 4, it would be interesting what the penetration value is for the other solvers.\n- While the experiments are generally convincing, they lack strong baselines and comparison to previous work. It would be interesting to see how the used GRU and ContactNets baselines perform on their respective other tasks. A baseline that considers velocity instead of acceleration prediction could also be useful. These baselines should also be considered for the MOVi-A dataset that showcases the behavior for the more complex shapes.\n\n### Typos and Style\n\n- l. 26 has two consecutive `and's\n- In ll. 129-131, `position coding' is introduced. The term `positional encoding' is probably more common.\n- l. 209 the `)' should come before the citation"}}, {"review_id": "NuqbT5MZoaJ", "reviewer": "Reviewer_8UbN", "summary": "The paper presents a learning-based approach to predicting object dynamics using graph neural networks. Object meshes are encoded using the finite difference of the mesh vertex velocity (over a short history) and distance to a boundary (e.g. floor), and edge features are relative displacements between nodes. Message passing is used to update node and edge features and the network predicts finite difference accelerations for every node. Extensive experiments are provided on simulated and real datasets of simple objects being tossed, and comparisons are made to both learned baselines and common physics engines used in robotics. The results suggest the graph network approach outperforms these baselines in terms of accuracy of predictions with relatively little training data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper was clearly written and enjoyable to read. The representation of meshes in the graph network is intuitive, and Figure 2 was quite helpful in quickly understanding what was being proposed. The experiments are very thorough in that they compare to a variety of relevant baselines on a variety of datasets. The expected ablations on the proposed method are performed and provide some insight on the necessity for the different components incorporated in training the GNN. The authors identify many of the relevant shortcomings of the method including making predictions for more complex objects and multi-object scenarios.\n\nWeaknesses: While the paper is relevant to robotics, there is minimal involvement of robots in the paper. Even the real-world dataset used from the ContactNet paper appears to just have a human tossing the cube. I find this limiting for robotics because then there is no action modality associated with any of the data, meaning it will be hard to learn predictive models for planning and control. There is also limited discussion on uncertainty and stochasticity in the dynamics, which is particularly important for tossing objects. For example, tossing the object 10 times in the exact same way will have a different outcome all 10 times, i.e. they'll vary at least a little bit due to slight variations in the execution and minute differences in the interactions with the environment. These sorts of dynamic manipulations are very sensitive to these subtle differences and can sometimes produce drastic differences in outcomes, particularly for more dynamic manipulations like dropping from higher heights, e.g. an object dropped on a table might bounce and come to rest on the table or fall off onto the floor, even when dropped from the same height in the same way. These dynamics are not fully predictable without infinite modeling precision, which is of course not possible. So it seems important to address matters of uncertainty in the predictions, and is worth discussing in more depth in the paper. This is also related to the first point about involving robots, one can control the actions to be more repeatable using a robot, which should make quantifying the uncertainty more feasible on a real-world setup, instead of having a human tossing objects for data collection. I am also curious how the approach can be applied to partial-view models, which is a very common situation in robot manipulation. The robot may only have a partial-view point cloud of the object and not have access to a ground truth mesh.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Making accurate predictions for object dynamics is undoubtedly relevant to robotics, and the paper does a very good job providing convincing evidence for their approach over learned baselines and common physics engines in robotics. My only reservation is that the explicit connection to robotics is tenuous. For this method to be useful for a robot, it needs to connect to the robot's action space, and none of the datasets have information about an action modality so far as I can tell. That being said, I think this paper motivates the use of a straightforward GNN architecture for simulating object dynamics, and I believe it could be a very useful tool for roboticists to build off of, which is why I'm strongly advocating for acceptance.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I would like for the authors to elaborate on the role of uncertainty in the model predictions, as I mentioned in the Strengths/Weaknesses section above. In particular that there is some limit to how accurate the predictions can be due to stochasticity from the environment, and this will be exacerbated in the real world since there is also the added uncertainty from noisy and incomplete observations. \n\nHere are some very minor issues:\nLines 108-113: It would be good to clearly state here how to interpret the \"node\"/\"edge\" terminology with respect to a mesh, presumably that a node is a vertex of the mesh and an edge between them means they are part of the same face?\nLine 265: \"models involving rigid and deformable objects.\" (make rigid non-plural)\nLine 295: Is \"rigids\" a term of art? It sounds awkward to me, I would go with \"rigid objects\"."}}, {"review_id": "I_fAAoaHZ8h", "reviewer": "Reviewer_ehVb", "summary": "This paper empirically exams the question of whether graph neural newtworks can learn discoutinous, rigid contact dynamics. Specifically, the GNS [1] architecture is studied on the following discountinous contact dynamcis: tossing a cube to the ground in simulation (with different stiffness parameters), tossing a cube to the table in the real world, and tossing objects with different geometris to the ground in simulation. The experimental results show clear constrast to previous conclusions that it is difficult for GNNs to learn such rigid contact dynamics, instead, learned GNN models can perform well on such dynamcis, even better than tuned analytic physics simulators.   \n\n[1] Sanchez-Gonzalez, et al, Learning to Simulate Complex Physics with Graph Networks, ICML 2020", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The question itself is interesting and the results help shed more light into the anwser -- GNN can indeed learn discountinous contact dynamcis well on the tested tossing tasks. \n- The ablation studies are complete and provide good understanding of which are the important factors for GNNs to perform well.\n\nWeakness:\n- My major concern is: is the result really surprising and makes enough contribution to a CoRL paper? In the original DPI [1] paper, the learned GNN model already demonstrates correct behaviour (i.e., non penetration and correct predicted cube pose) on modeling the contact interaction between the rigid cube and the water particles (the BoxBath experiment); and in the original GNS [2] paper, the GNN model also demonstrates correct behaviour (i.e., non penetration and correct predicted particle positions) between particles colliding with environment obstacles (the WaterRamps experiment). Although these are not strictly rigid objects colliding with another rigid object, but I would not be surpised that the GNN model should be good to handle the non-peneration part of rigid-to-rigid collision based on the results from these previous papers. With this being said, these previous papers did not study if GNN can learn the sensitivity to initial conditions for rigid-to-rigid collision, which is clearly demonstrated in this paper. But with this conclusion alone, I am not entirely sure it is enough contribution for a CoRL paper.\n- Also, most of the experiments considers tossing a rigid object into another. There are many more rigid contact dynamics other than this, e.g., a quadruped robot doing locomotion on the ground, a robotic hand manipulating a rigid object. Conducting some experiments in these more complicated senarios would make the result more convincing and stronger. \n- Another minor question: regarding figure 4, how is the sysID done for the physics simulators? More specifically, how many real-world trajectories are used to tune the simulation parameters? I would imagine with more real-world data used, the parameters can be tuned more accurate, i.e., the performance of the simulator in figure 4 should also be a curve instead of a horizontal line. More details to clarify this would be appreciated.\n\n[1] Li, et. al., Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids, ICLR 2019\n[2] Sanchez-Gonzalez, et al, Learning to Simulate Complex Physics with Graph Networks, ICML 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Please see the strength/weakness section above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the strength/weakness section above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "-w_pgxn7pTx", "reviewer": "Reviewer_MGH6", "summary": "This paper proposes to use a Graph Neural Network (GNN), similar to the mesh-based graph network proposed in [3], to learn rigid-body dynamics with contact in 3D. Following [3], each mesh vertex of the rigid body is modeled as one node of GNN, and each edge of the mesh as one edge of GNN. The authors argue that using spatial-equivariant relative displacement between nodes, as in [3], and predicting acceleration rather than velocity (same as [3] in the cloth case) are important to achieve high accuracy. The method is validated both with simulated datasets [10] and a real dataset [2], and outperforms recent baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths and weaknesses of the paper are both quite clear.\n\nStrength: \n\n- Simple algorithm and network structure, without heavy engineering and tuning.\n\n- Good results on both simulated and real datasets that outperforms baseline learning methods.\n\nWeaknesses:\n\n- Lack of novelty: not really a weakness in my opinion, if GNN works well already out of the box, that is worth sharing with the community. But I have to say the method and implementation details are both quite similar to [3].\n\n- Lack of insight: It’s hard for me to intuitively understand why GNN works much better than ContactNet and naive learning methods using e.g. GRUs. There is a diagnostic example (Sec. 4.1) but it does not provide hypothesis or theories why GNN is better at modeling discontinuities than *other neural nets* either.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "“GNNs are better at modeling discontinuities than other NNs” is certainly an unconventional, and to me counterintuitive, claim. As such, since the paper do not provide much insight on why, and the testing datasets are both relatively small, I’m a bit worried about whether findings from this paper can be applied to more complex rigid-body learning problems, or larger datasets.\n\nThat being said, the results seem solid, and I would like to see these results published if the following issues can be addressed during rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- L477 we “expect higher errors for later timesteps due to error accumulation”: could the authors share the final-frame errors in addition to average errors for ContactNet dataset? It’s okay if the improvement becomes less obvious compared with “those heavily tuned analytical simulators“.\n\n- L244: the ablation study is odd to me. GNS[abs] shows that using relative displacement between nodes, *and* predicting acceleration rather than velocity are important, since it is much worse than other ablations.\n\n   Could the authors split this important ablation into two? That is, what if we use relative displacement as input features but for output still predicts velocity/displacement, instead of acceleration? In [3], predicting acceleration and use it in second-order integration, instead of first-order seem to be an optional feature that can be easily turned on or off, which should be the same for this paper.\n\n- Could the authors also plot the accuracy metrics on training trajectories of ContactNet (i.e. training losses), in the format of Figure 3?\n\n- Is the Gaussian random noise (L463) identical to [16], or actually more similar to [3] which modifies upon [16]?\n\n- Since the paper does not have much theory or insight but only demonstrates good performance, for reviewers to trust the results, could the authors promise to publish their code if the paper is accepted?\n\n- If I understand correctly, Figure 3 (c) essentially says that if we know the initial state of a “dice”, this GNN can almost always predict correctly which face it lands on the table (otherwise rotational error would be quite large for that trajectory?), given the chaotic nature of a dice, could the authors share some insights on why you think this is not too good to be true?\n\n- Typo in abstract: Line 5 continuous -> discontinuous?"}}]

Author rebuttal:

Decision: Accept (poster)