Learning rigid dynamics with face interaction graph networks
Authors: Kelsey R Allen
Yulia Rubanova
Tatiana Lopez-Guevara
William F Whitney
Alvaro Sanchez-Gonzalez
Peter Battaglia
Tobias Pfaff
Venue: ICLR 2023 notable top 25% (2023)
Keywords: graph networks
rigid body dynamics
physics

Abstract: Simulating rigid collisions among arbitrary shapes is notoriously difficult due to complex geometry and the strong non-linearity of the interactions. While graph neural network (GNN)-based models are effective at learning to simulate complex physical dynamics, such as fluids, cloth and articulated bodies, they have been less effective and efficient on rigid-body physics, except with very simple shapes. Existing methods that model collisions through the meshes' nodes are often inaccurate because they struggle when collisions occur on faces far from nodes. Alternative approaches that represent the geometry densely with many particles are prohibitively expensive for complex shapes. Here we introduce the ``Face Interaction Graph Network'' (FIGNet) which extends beyond GNN-based methods, and computes interactions between mesh faces, rather than nodes. Compared to learned node- and particle-based methods, FIGNet is around 4x more accurate in simulating complex shape interactions, while also 8x more computationally efficient on sparse, rigid meshes. Moreover, FIGNet can learn frictional dynamics directly from real-world data, and can be more accurate than analytical solvers given modest amounts of training data. FIGNet represents a key step forward in one of the few remaining physical domains which have seen little competition from learned simulators, and offers allied fields such as robotics, graphics and mechanical design a new tool for simulation and model-based planning.

LEARNING RIGID DYNAMICS WITH FACE INTERACTION GRAPH NETWORKS
Kelsey R. Allen", Yulia Rubanova", Tatiana LopezaGu William Whitney, Alvaro Sanchez-Gonzale Peter Battaglia, Tobias Pfaff DeepMind, London, UK
ABSTRACT
Simulating rigid collisions among arbitrary shapes is notoriously difficult due to complex geometry and the strong non-lincarity of the interactions. While graph neural network (GNN)-based models are effective at learning to simulate complex physical dynamics, such fluids, cloth and articulated bodies, they have been less effective and efficient on rigid-body physics, except with very simple shapes. Existing methods that model collisions through the meshes' nodes often in- accurate because they struggle when collisions occur on faces far from nodes. Alternative approaches that represent the geometry densely with many particles are prohibitively expensive for complex shapes. Here we introduce the "Face In- teraction Graph Network" (FIGNet) which extends beyond GNN-based methods. and computes interactions between mesh faces, rather than nodes. Compared to learned node- and particle-based methods, FIGNet around 4x more accurate in simulating complex shape interactions, while also 8x more computationally effi- cient on sparse, rigid meshes. Moreover, FIGNet can learn frictional dynamics directly from real-world data, and can be more accurate than analytical solvers given modest amounts training data. FIGNet represents a key step forward of the few remaining physical domains which have seen little competition from learned simulators, and offers allied fields such as roboties, graphics and mechanical design new tool for simulation and model-based planning.
INTRODUCTION
Simulating rigid bodies accurately is vital in wide variety of disciplines from robotics to graphics
to mechanical design. While popular general-purpose tools like Bullet (Coumans) 2015), MuJoCo
(Todooov et al. 2012) and Drake (Tedrake 2019) can generate plausible predictions, predictions that
match real-worte world observations accurately are notoriously difficult (Wieber et al., 2016 Anitescu &
Potra 1997 Stewart & Trinkle 1996. Fazeli al., 2017 Lan 2022). Numerical approxima- tions necessary for efficiency are often inaccurate and unstable. Collision, contact and friction are challenging to model accurately, and hard to estimate parameters for. The dynamics are non-smooth and nearly discontinuous (Pfrommer et al.) 2020 Parmar et al., (021). and influenced heavily by the fine-grained structure of colliding objects' surfaces (Bauza & Rodriguez (017). Slight errors in the physical model or state estimates can thus lead to large errors in objects' predicted trajectories. This underpins the well- known sim-to-real gap between results from analytical solvers and real-world experiments.
Learned simulators can potentially fill the sim-to-real gap. They can be trained to correct imperfect state estimation, and can learn physical dynamics directly from observations, potentially producing more accurate predictions than analytical solvers (Allen et al., 2022 Kloss et al.) 2022). Graph neural network (GNN)-based) models, in particular,
effective at simulating liquids, sand, soft materials and simple rigids (Sanchez-Gonzalez et al.) [020 Mrowca et al., 2018 Li et al. 2019b] Pfaff al. 2021 Li al. 2019a). Many GNN-based models are node-based: they detect and resolve potential collisions based on whether two mesh nodes or particles are within a local neighborhood. However, collisions between objects do not only happen nodes. For example, two cubes may
"These authors contributed equally. Correspondence to [kral 1en, rubanova } @deepmind com

---

collide by corner hitting the other's face, or edge hitting the other's edge (in fact, corner- to-comerr collisions are vanishingly rare). For larger meshes, fewer collisions occur within the local neighborhood around nodes, and thus collisions may be missed. Some prior work thus restricts collisions to simple scenarios (e.g. a single object colliding with a floor (Allen et al. 2022 Pfrommer et al., )020)) Alternative approaches represent the object densely with particle nodes (Li et al.) 2019a Sanchez-Gonzalle et al. 2020).. but that leads to a quadratic increase in node-node collision lests, which computationally prohibitive for nontrivial scenes.
Here we introduce novel mesh-based approach to collision handling-Face Interaction Graph Net- works (eeNett)whiclh extends message passing from graphs with directed edges between nodes, to graphs with directed hyper-edges between faces. This allows FIGNet to compute interactions be- tween mesh faces (whose representations are informed by their associated nodes) instead of nodes directly, allowing accurate and efficient collision handling on sparse meshes without missing col- lisions. Relative to prior node-based models, we show that for simulated multi-rigid interaction datasets like Kubric (Greff et al., 2022), FIGNet is 8x more efficient at modeling interactions be- tween sparse, simple rigid meshes, and around 4x more accurate in translation and rotation error for predicting the dynamics of more complex shapes with hundreds or thousands of nodes. We ad- ditionally show that FIGNet even outperforms analytical solvers for challenging real-world robotic pushing experiments (Yu al. 2016). To our knowledge, FIGNet is the first fully learned simulator that can accurately model collision interactions of multiple rigid objects with complex shapes.
RELATED WORK
To address the weaknesses traditional analytic simulation methods, a variety of hybrid approaches that combine machine learning with analytic physics simulators have been proposed. Learned simu- lation models can be used to correct analytic models for a variety of real-world domains (Fazeli et al,,
2017
Ajay et al.] 2018) Kloss al., 20177] Golemo et al.] 20188: Zeng et al., (2020: Hwangbo et al. 2019 Heiden et al. (002)). Analytic equations can also be embedded into neural solvers that exactly preserve physical laws (Pfrommer al.) 2020 Jiang al., (202)). While hybrid approaches can improve over analytic models in matching simulated and real object trajectories, they still struggle in cases where the analytic simulator is not a good model for the dynamics, and cannot be trivially extended to non-rigid systems.
More pure learning-centric approaches simulation have been proposed in recent years support general physical dynamics, and GNN-based methods are among the most effective. GNNs represent entities and their relations with graphs, and compute their interactions using flexible neural network function approximators. Such methods can capture the dynamics of fluids and deformable meshes (Li al. [209a]] Gonzalzz et al.) 22200] Pfaff et al.] (02T). robotic systems (Pathak et al., | Gonzazzz et al. 20188] Wang et al. 2018). 8), and simple rigids (Rubanova et al. 1021 Mrowca et al.. 2018 Li et al. 2019b Battaglia et al. 2016 Allen et al. 2022 Bear et al. (021).
Despite these models' successes in many areas, the rigid dynamics settings they have been applied to generally use very simple shapes like spheres and cubes. Furthermore, most models are tested only in simulation, which may be poor proxy for real world rigid dynamics (Bauza & Rodriguez/ 2017
Fazeli et al.] 2017] AAcsst et al. 2022). It is therefore unclear whether end-to-end deep learning models, including GNNs, are generally good fit for learning rigid-body interactions, particularly if the rigid bodies are complex, volumetric objects like those we experience in the real world.
In this work, demonstrate how using insights from graphics for representing and resolving colli- sions as face-to-face interactions improves rigid body dynamics learning enough to not only very ac- curately capture simulated dynamics, but also outperform analytic simulators (Todorov al. 2012 Coumans] 2015; Lynch] 1992) for real-world rigid- body dynamics data. We conduct series of ablations and experiments which showcase how face-to-face collision representations dramatically improve rigid body dynamics prediction.
METHOD
For the purposes of rigid-body simulation, objects are represented as meshes M, consisting of the set of node positions (x; and set of faces (F) that describe how nodes are connected to
2

---

[b)
(c)
Face-Face
Meelh-Mesh
Object-Mesh
Face face edges
Mesh nodes
Par-object nodes 1.0
adges
Otject node
Closeel points
P. P.
Figure 1: FIGNet model with face-to-face edges. (a) Example meshes from the Kubric MOVi-B
dataset (Greff et al.) 2022). (b) FIGNet exploits three types of interactions: face-face collisions
between objects, mesh-mesh interactions within an object, and interactions with the virtual per- obiect node. (c) Rollout from FIGNet trained on Kubric MOVi-B (rrff et al. 2022
(8) 97
mpot
Update
gi
Updatee Mesh Mesh-Mesh
Update Object-Mesh
Cv.
Aggregatte
d/
Aggregste
Mesh update
92
Figure 2: (a) Calculating face-face collision edge features (b) Message-passingg with face-face edges. Face-face edges produce the updates for each node of the face (grey rectangles). After aggregation, the face-face update for the corresponding face node is combined with Mesh-Mesh and Object-Mesh edge representations, similarly to standard GNN message- pataugg....sing.... Notation is defined in text.
form objects mesh. Each face consists of three nodes F {sy}==1223, where 8j are the indices of the nodes in the face. We consider triangular faces but all aspects of the method generalize to faces with other numbers of nodes (quads, tetrahedra, points, segments, etc.) and their interactions.
The simulation trajectory is represented as sequence of meshes M = (M's M M where each mesh represents one of more objects. It is computed by applying sequence of object rotations and translations to reference static mesh M of each object. The goal of the simulator is to predict the next state of the system M from history of previous states (M',... Mt-A+1). We use a history length of h = 2, following (Allen et al., 2022). The rollout trajectory can be obtained by iteratively applying the simulator to the previous prediction; (M M , M4
3.1 FACE INTERACTION GRAPH NETWORK
We introduce the Face Interaction Graph Network (FIGNet) (Figs. and 2 as the simulator 8, which is designed extend existing general purpose simulators (Sanchez-Gonzalez et al., 2020 Pfaff et al., 2021) to model interactions between multiple rigid objects more efficiently. FIGNet is an evolution of the Sicode-Process-Deccee nodes and edges of the object meshes, adding two key extensions over MeshGraphNets<PPann et al.] 2021 First, model collisions between two objects, FIGNets generalizes message passing over directed edges to message passing over directed hyper-edges. This enables message passing on face- face edges, from face with three sender nodes to face with three receiver nodes. FIGNet replaces MeshGraphNets (Pfaff et al. 2021) node-based "world edges" with face-face edges when faces from distinct objects are within certain distance de- Second, FIGNet supports multiple node and edge types. We use this to add "object nodes" that are connected to all of the mesh nodes of given object. These also participate in message passing (Fig. 2p) to enable long-range communication!
architecture which performs message passing the
3.1.1 ENCODER
The encoder constructs the input graph = (V, E, QF)) from the mesh M The vertices V/ {vx} represent the mesh nodes, E -u, are bidirectional directed edges added for all pairs of nodes that are connected in the mesh and QF {qF,- F, } are edges between the faces of different
For simplicity on the notation will omit "object nodes" in the explanation of the model in the main text.
3

---

objects that are close spatially. Each edge in Q is defined between sets of nodes of the sender face F. (Us1. Maz, Ves) and the receiver face F, (Ur1, 0y2; Uy3)-
Node features and mesh edges We follow (Pafff et al. mesh node o; and mesh edge ev.-v,, The features of U; are the finite-difference velocities estimated from the position history v = [X} x x -h+1
2021 for constructing the features for each
=f where [] indicates concatenation. The features of Cv,-v, are computed as - [drs, d where dys = x, x, is the relative displacement between the nodes in the current mesh M and d is the relative displacement in as
the reference mesh M Note that this representation of the graph translation-invaria does not contain the absolute positions x,, leading to a translation equivariant model with better generalization properties (Pfaff et al. 2021). Finally, we use separate MI.P encoders to embed the features of each mesh node, and mesh edge into latent representations V; and -
Face-face edge features To model face-face collisions between two rigid objects, we intro- duce new type of face-face edges, QF. A face-face edge 9F,,FF- connects sender face F, = (Us1, 222; Vsa) to a receiver face Fr = (Ur1, Uy2, 0ra). To determine which faces to con- nect, we use the Bounding Volume Hierarchy (BVH) algorithm (Clark) 1976) find all pairs of faces from different objects that are within a certain pre-defined collision radius de-
To construct face-face edge features (Fig. Zh) we compute the closest points between the two faces, p. and pr for faces F. and F, respectively. Geometrically, the closest point for the sender/receiver face might be either inside of the face, on one of the face edges, at one of the nodes. Using the closest points as local coordinate systems for each of the faces involved in the collision, we build the following translation equivariant vectors: (1) the relative displacement between the closest points d Pr Ps at the two faces, (2) the spanning vectors of three nodes of the sender face F. relative to the closest point at that face d = Xs; Ps. (3) the spanning vectors of three nodes of the receiver face F, relative to the closest point at that face d = Xr. Pr, and (4) the face normal unit-vecton of the sender and receiver faces n, and n,. These vectors are the equivalent to the relative displacement for dys = X, X, in regular edges, and provide a complete, fully-local coordinate system while keeping the model translation equivariant. In summary:
features = jd/.. [d"" 1, 97,-F,
1,233 (d (d,,)==1,2,3 D.. n,.
(1)
We use an MLP followed by reshape operation to encode each face-face edge as three latent face-face edge vectors, one for each receiver node:
QF.+F. = 11, reshape(M1)) (ML
feattures (F.FF..).
For each face-face edge, and before computing features (2) and (3), we always sort the nodes of the sender face and the receiver face as function of the distance to their respective closest points. This achieves permutation equivariance of the entire model w.r.t. the order in which the sender and receiver nodes of each face are specified.
3.1.2 PROCESSOR
The procesor is a GNN generalized to directed hyper-edges that iteratively applies message passing across the mesh edges and face-face edges. We use 10 iterations of message passing with unshared learnable weights. Below we describe a single step of message passing to update the edge latents C,, the face-face edge latents QF.+F.., and the node latents v,, and produce c,, Q'r, FF. and v/-
The mesh edge latents e,,+u, are updated following the standard edge update approach from (Battaglia et al. 2018:
bby ML
proosssor
((cr, Vs. v,.)
(3)
The approach to update each face-face edge similar, except that each face-face edge has latent vectors, 3 sender mesh nodes and 3 receivers mesh nodes; and produces output vectors:
MittaapeMMMPPeeeeo ([|g,,,- -, Vay V,,li=1,2,3) This does not update the three latent vectors independently, but rather all nine input latent vectors [[9,,, -F Vaya Vr, 11,3)=1,2,3) contribute to all updated face-face edge latent vectors Qr
O7 (q).F.-A,lii 1,2,3 1,2,3
4

---

Finally, we perform the node update based on the incoming messages from face-face edges, well as other edges between the individual nodes:
MLPY Processo ([vi,
Σ V6,,-t9// /1
C
Σ
(5)
V4r, -
The second term corresponds to standard edge aggregatiinnBattaglia et al. 2018 and the third term sums over the set of face-face edges for which vj is receiver. Crucially, 9 F.F.F selects the specific part of the face-face edge latent corresponding to v2 receiver, i.e. + selects the first, second, or third vector, depending on whether Us is the first, second or third vector.
3.1.3 DECODER AND POSITION UPDATER
Following the approach of MeshGraphNets (Pfaff et al.) 2021D, the model predicts the acceleration a, by applying an MLP to the final mesh node latents which adds the bias for inertial dynamics to the model predictions. Finally, we update the node positions through a second order Euler integrator: x
+ 2x!
3.1.4 IMPLEMENTATION
Object-level nodes In perfect rigid, information propagates across the rigid at infinite speed. However, with n message passing steps, information can only propagate hops, and for very fine mesh when collision happens it may take many message passing steps to reach all other nodes in the mesh. To facilitate long range communication, we add virtual object-level node v at the center of each rigid object, and connect it to all of the mesh nodes of the object via bidirectional mesh-object edges. Ultimately, these extra edges and nodes will also participate in message passing equations 3 and described in the Appendix. This design allows completely decoupling the dynamics from mesh complexity as collision events can be computed on local neighborhoods using local message passing edges& E, and the resulting collision impulses can be aggregated and broadcasted using the virtual object node v Note that unlike (Mrowca et al. 2018 Li et al. 2019a), we use only single object node which itself only communicates with nodes on the surface of the object.
Loss and model training Following MeshGraphNets (Praff et al. 2021). we define the loss as the per-node nean-squured-error on the predicted accelerations a, whose ground truth values are estimated from the 2 previous positions and the future target position. We train the model using the Adam optimizer (Kingma Ba 2014). To stabilize long rollouts, we add random- walk noise to the positions during training (Sanchez-Gonzalez Gonzalez et al., 2020 Pfaff al.) 2021 Allen et al. 2022). We also augment the dataset by rotating the scenes by random angles B around the axis.
Generating rollouts and evaluation At time we generate trajectories by iteratively applying the model to previous model predictions. Following Allen et al. (202)); Müller et al. (2005), we use shape matching to preserve the shape of the object during rollouts. We use the predicted positions of the nodes in the object mesh to compute the position of the object center, as well as the rotation quaternion with respect to the reference mesh MU Then we compute the new node positions by applying this rotation to the reference mesh M and shifting the center of the object. This is performed only during inference, not during training.
3.2 BASELINES
Our baselines study the importance of how represent and resolve collisions in learned simulators. In particular, we consider the baseline outlined in Figure 3 which detect collisions in different ways by representing objects with particles, or meshes, and representing collisions between nodes or faces.
MeshGraphNets (MGNs) Plaff et al., 2021) are similar to FIGNet in that they represent the object as mesh with messages propagated along the mesh surface. However, when detecting collisions between objects, they connect nodes that are within some fixed distance d,- For objects that have sparse meshes (and may collide along edge face, see Figure 3). collisions may
5

---

go undetected if the edge is sufficiently long relative to de- MeshhrraphNets-LaageRaaaa attempt to fix this issue of MGNs by increasing the radius de to the longest edge length in the mesh to ensure collisions are detected. However, choosing such large radius can lead to explosion in the number of detected collisions, which significantly slows down computation (see Results). MGNs also do not
is an
object-level node for efficient message-passingg across the object.
DPI Reimplemented If we relax the requirement that objects are represented as meshes, another popular way to rep- resent objects as collections of parti- cles eLi al. 2019b). Particles are typi- cally densely sampled throughout an ob- ject's volume which allows for collision detection and efficient message propaga- tion, but this then requires using many more nodes. We reimplement DPI (Li al. 20196) as a particle-based base- line by converting meshes to particle point clouds using radius of 0.025. We use the loss formulation from DPI (see Appendix), and predict velocities rather than accelerations.
MeshhGraphhee Necollakuss detcctol
Particce Modell
Larger Badioss oagc "Jepezah skee
FiGNet Ouigital nuch "andependoo
Kixe olgos
Figure 3: Schematic of how baselines treat interacting objects. If one object's vertex intersects another object's face, small collision radius de will not detect collision (Naive MeshGraphNet). A dense particle representation fixes the problem but adds more edges and more nodes. Using a larger collision radius introduces quadratically more edges. By operating on faces, FIGNet can use small collision radius without missing collisions.
Representation of the floor In all of our datasets, the floor is the largest and simplest object. While FIGNet has the advantage of being able represent the floor using only two large triangles, the baselines require denser floor to ensure that collisions with the floor are detected. In practice, this means that we refine the floor mesh that triangle edges are not longer than 1.5 units (compared to maximum object size of 1.5). These variants are marked with + throughout the results.
For more complex datasets, refining the floor mesh can lead MGN and DPI models to run out of memory during training due to excessive numbers of nodes and collision edges. When this happens, we replace the floor mesh by an implicit floor representation eet al. 2022) where each node includes feature representing the distance to the floor, clamped to be within de. These variants are marked with throughout the results. Note that this technique fundamentally limited: it assumes that there is fixed number of immovable large objects (the floor, or optionally also the walls), and therefore this model cannot be applied to scenes with varying numbers of walls floor planes either during training inference.
Sanchez-Gonzalle
et al. 2020 Allen
RESULTS
Here we investigate FIGNer's ability to model both simulated rigid body scenes as well as real world rigid interactions. We focus on investigating three key aspects of FIGNet: its efficiency, accuracy, and generalization. Across simple and complex simulated scenes, FIGNet outperforms node-based baselines in computational efficiency (by up to 8x) and rollout accuracy (up to 4-8x). FIGNet also generalizes remarkably well across object and floor geometries far outside its training distribution. Finally, we demonstrate that FIGNet approximates real world planar pushing data better than carefully tuned analytic simulators like PyBullet (Coumans) 2015). MuJoCo (Todorov) et al. 2012), and hand-crafted analytic pushing models (Lynch, 1992). Throughout the results, we compare the representative model variations outlined in Figure For rollout videos, please refer to the website: sites. metsssoooll.ccom www.www//mwiggnctttiiiccc
Efficient modeling for simple meshes As alluded to previously, even relatively simple meshes can be challenging to model for node-based techniques. To accurately model the interaction between faces or edges of meshes, node-based models must increase their computational budget by either remeshing object the distance between nodes is smaller, or by increasing their collision radius d, to ensure that possible collisions are detected. This is particularly problematic when objects have simple meshes which mostly consist of large faces, like cubes and cylinders.

---

Ground Truih
FIGNet
MGN
Figure 4: Rollout examples on Movi-A dataset. The node- based MeshGraphNet model struggles to resolve the collision between the sides of the sparse cube meshes.
The Kubric MOVi-A dataset (Greff et al. 2022 perfectly encompasses this scenario. In this dataset, 3-10 rigid objects (cubes, spheres or cylinders) are tossed onto floor. The distance between nodes within an object's geometry can range from 0.25 to 1.4. To compare models, we measure the root mean squared error (RMSE) between the predicted and ground truth trajectories for translation and rotation after 50 rollout steps.
As expected, because the object geometries are sparse, using the node-based MGN with the same collision radius FIGNet (d.=0.1) exhibits poor prediction performance, with objects often inter- penetrating one another (Figure 4) leading to poor translation and rotation errors relative to ground truth (Figure 5ap. Increasing the collision radius to 1.5 solves these problems, but at great com- putational cost with an explosion in the number of collision edges that must now be considered (Figure 5ap. This is especially true for DPI-Nets (d.=0.5), the particle-based method, which can longer represent the floor as sets of particles without running out of memory during training. By comparison, FIGNet predictions are plausible with low translation and rotation errors (Figure 4). while not increasing the number of collision edges as dramatically, maintaining efficiency.
Modeling unprecedentedly com- plex scenes FIGNet's representa- tion of object geometries faces
Translation
10F
0.80
30.0
Collttion edges
10
(00g)
ables more accurate predictions for more complex meshes without sacri- ficing efficiency. The Kubric MOVi- dataset (Greff et al., (2022) contains much more complex shapes, includ- ing teapots, gears, and torus knots, with few hundred up to just over one thousand vertices per object.
0.40
20.0
10"
0.30
10.0
FIGNet
MEN LargoRadlus+
DPI Reimplementest
MGN+
(a) Kubric MOVi-A
1.00
Trensiation enoer
40.0
Rotation erfor
10
Collision edges
Despite the incredible complexity of this dataset, FIGNet produces ex- ceptionally good rollouts that closely track the ground truth dynamics (Fig- ure G.3) FIGNet has 4x lower translation RMSE, and 2x lower ro- tation rotation RMSE compared to other models (Figure 5b). Note that the meshes MOVi-B are complex enough that all baselines run out of memory if applied to MOVi-B di- rectly with a densely sampled floor. Therefore, all baselines use the im- plicit floor distance feature, which greatly reduces the number of over- all edges (since there will be de- tected collisions with the floor). Even with this adjustment, the node-based baselines (DPI- Reimplemented* (d.=0.25) and MGN-LargeRRdius (d.=0.3)) still generally require more collision
0.50
30.0
Collsiio edges
0.40
20.0
0.20
10.0
FIGNet
Largooogu
DPI Rolmpleemmmnn
MGN
(b) Kubric MOVi-B
Figure 5: Performance for FIGNet and several baseline methods: root mean squared-error on translation and rota- tion after 50 timesteps, and number of collision edges for (a) MOVi-A and (b) MOVi-B datasets. FIGNet has signif- icantly better translation and rotation error, while also cre- ating fewer collision edges overall. Models with use an implicit floor representation, while models with + use subdivided floor.

---

Ground
Truth
FIGNet
Figure 6: Rollout example on Kubric MoviC dataset (Greff et al. 2022) from FIGNet model trained on the MOVi-B dataset. FIGNet is able to generalize to the complex geometries of the MoviC objects which it has never seen during training.
edges than FIGNet Figure 5b). Moreover, DPI- "TT-Reimppeeeetedd" requires dense particle represen- tations of the objects (on average 4000 nodes per simulation compared to the 1500 nodes in the original mesh), increasing the model memory requirements.
The performance of FIGNet here partly driven by the object-level nodes that FIGNet uses to more efficiently propagate information through object meshes, but not entirely (see ablations in Figure E. and Figure E. 1). This partly explains why OPI-Reimplemmnnedd performs better than MGNs in this dataset, as it also uses object-level virtual nodes.
Generalization to different object and floor geometries Given FIGNet's strong performance on MOVi-A and MOVi-B, we next sought to understand how well FIGNet generalizes across different object shapes. We take the FIGNet models pre-trained the MOVi-A and MOVi-B datasets and apply them to the MOVi-B and MoviC respectively. In both the MOVi-A-to-MOVi-- and MOVi- B-to-MoviC generalization settings, FIGNet has 2-4x lower translation and rotation error than the baselines (Figure la Figure G.1b) and produces realistic-looking rollouts (Figure 6 Figure G.4). This is particularly remarkable in the MOVi-A-to-MOVi-B setting, because the model has only ob- served 3 different object shapes during training (cubes, cylinders and spheres). Furthermore, the MOVi-B translation error of FIGNet trained MOVi-A FFiuure G.lab is better than any baseline models trained directly on the MOVi- dataset (Figure 5b), once again demonstrating the general- ization ability of face-face collisions. In contrast, Dempleeelleettte and both versions MGNs struggle to generalize to scenes with new objects (Figure G.1a and F G.Ib).
Modeling real world dynamics better than analytic simulators In the preceeding sections, we saw that FIGNet significantly outperforms alternative learned models for complex, simulated datasets. However, the real world is complicated than simulation environments ple dynamics like planar pushing under friction are hard to model because of sub-scale variations object surfaces. Here we investigate how FIGNet compares analytic simulators in modeling real-world pushing dynamics.
sim-
For these experiments, we use the MIT Pushing Dataset ( Yu 2016); Figure 7 of 6000 real-world pushes for 11 different objects along 4 different surfaces (see (suseectio B.2
1), which consists
Median translation enror (relative zero movement)
Median rotation error (ralative to zero movement)
(a)
(b)
(c)
100
125
Anabyte hunning
Error (5% ml
Enar 0% deg)
75
40
20
25
128
512
2048
128
512 2048
train trajectories
train trajectories
Figure 7: MIT Pushing dataset results. (a) demonstration of the setup from Yu et al. (2016). (b) Example rollouts from FIGNet (blue) vs. ground truth (gray). The dashed outline indicates the ini- tial object position. (c) Performance on MIT Pushing dataset for FIGNet compared to analytic simulators MuJoCo (Todorov et al., 2012), PyBullet (Coumans) 2015), and the planar pushing an- alytic model developed by (Lynch 1992) and implemented for this dataset by Kloss et (2022).
8

---

for details). The objects are tracked for the full push. We sample different training dataset sizes to examine FIGNet's sample efficiency and test on a set of 1000 trajectories across objects. To mitigate the effect of outlier trajectories, the ground truth and predicted trajectories after 0.5 seconds, which corresponds to 125 timesteps. To provide interpretable unit for the metrics, scale the rotation and translation error by the error that would be predicted by a model that always outputs zeroes. 100% error therefore corresponds to performing no better than a model which always predicts O.
report the median translation and rotation error between
We compare FIGNet to two widely used analytic rigid body simulators, MuJoCo (Todorov et al. 2012) and PyBullet (Coumans) 2015). as well as the planar pushing specific model from Lynch (1992). For tuning system parameters of the analytic simulators, we use Bayesian Optimization on the training dataset with all available friction parameters, as well as the "timestep" for MuJoCo and PyBullet. Futhermore, while MuloCo and PyBullet trajectories are evaluated based on the tracked object positions directly (and are therefore prone to pereeption errors), the planar pushing analytic model uses the metadata for each pushing trajectory (giving information about the pushing direction, velocity, etc. without perception noise). This allows us to separately examine whether FIGNet improves on analytic simulators because it compensate for perception errors (where it would perform better than PyBullet and MuJoCo only) or because it can model pushing under friction more effectively (where it would perform better than all analytic models). We evaluate the analytic models on the same test dataset as FIGNet.
Remarkably, FIGNet outperforms all analytic simulators trained only 128 trajectories of real world data (Figure 7p. Unlike MuJoCo and PyBullet, FIGNet is robust to errors in perception which occasionally reveal the pusher moving an object even when not in contact, or when the pusher pene- trates the object. However, because FIGNet also improves on the analytic simulator (Lynch) 1992), this means that it is also simply modeling pushing under friction more accurately than can be analyt- ically modeled. Its particular improvement in rotation error relative to the analytic models suggests that can model torsional friction more accurately. This is exciting as opens new avenues for considering when learned models may be preferable to more commonly used analytic simulators.
5 DISCUSSION
We introduced FIGNet, which models collisions using showed it can capture the complex dynamics of multiple rigid bodies colliding with one another. Compared to prior node-based GNN simulators, FIGNet can simulate much more complex rigid body systems accurately and efficiently. It shows strong generalization properties both across object shapes, and even to different floor geometries. On real-world data, FIGNet outperforms analytic simulators in predicting the motion of pushed objects, even with relatively little data, presenting an exciting avenue for future development of learned simulators that model real world behavior better than current analytical options, such as MuJoCo (Todorov al., 2012 or PyBullet (Coumans) 2015).
face-to-face interactions, and
Our face-to-face approach is not limited to triangular faces, nor rigid body interactions. Extensions to quadric, tetrahedral, etc. faces can the same message-passing innovations we introduced here. Similarly, while this work's 's focus was on rigid body dynamics, in principle FIGNet naturally extends multi-material interactions. Soft bodies or fluids can be represented by collections of particles, and their interactions with faces can be modeled similarly to FIGNet. Exciting directions for future work could investigate whether different kinds of face interactions could even be learned from different datasets, and then composed to handle few-shot dynamics generalization.
In terms of limitations, FIGNet, like other function. In practice, this works well for the complex rigid systems studied here, however for highly stochastic or chaotic systems this may not be as effective. The datasets studied here include complex geometry, however the extent to which FIGNet will scale to extremely complex meshes (millions of faces) is unclear, and other approaches, e.g., implicit models, may be more suitable. While FIGNet works better than analytic simulators for real world data, it still requires access to state information, and future work should investigate how it can be coupled with perception to enable learning from raw pixel observations.
simulators, is trained with deterministic loss
FIGNet represents an important advance in the ability to model complex, rigid dynamics. By model- ing mesh geometries directly, FIGNet avoids the pitfalls of previous node-based learned simulators.

---

FIGNet opens avenues for learned simulators to be used as potential alternatives to existing analytic methods even for very complex scenes, with valuable implications for robotics, mechanical design, graphics, and beyond.
REFERENCES
Brian Acosta, William Yang. and Michael Posa. Validating robotics simulators on real-world im- pacts. IEEE Robotics and Automation Letters, 2022.
A. Ajay, J. Wu, N. Fazeli, M. Bauza, L.. P. Kaelbling, J. B. Tenenbaum, and A. Rodriguez. Aug- menting physical simulators with stochastic neural networks: Case study of planar pushing and bouncing. In International Conference on Intelligent Robots and Systems (IROS), 2018.
Kelsey R. Allen, Kevin A. Smith, and Joshua B. Tenenbaum. Rapid rial-and-erron learning with simulation supports flexible tool use and physical reasoning. Proceedings of the Na- 2020. doi: 10. 10.1073/pnas.111234111 1912341112 URL https: htttp::/wwww.paa.t http:/ppna..orr///////oo//// wwt.ooog//oooooooooooo /ttp://///.//////0.0////////////
tional Academy of Sciences, 117(47):
10130.1073333/3.0 1912341117
Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontin- uous, rigid contact dynamics. In 6th Annual Conference on Robot Learning, 2022. URL
hittps://epente wwt.iiewweeeee/eee.eee.. Foorummdd=rbbieeee
M Anitescu and F A Potra. Formulating dynamic multi-rigid-body contact problems with friction as solvable linear complementarity problems. Nonlinear Dynamics, 14(3):231-277, 1997.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
Jakob Andreas Bærentzen and Henrik Aanaes. Signed distance computation using the angle weighted pseudonormal. IEEE Transactions on Visualization and Computer Graphics, 11(3): 243-253, 2005.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physies. In Advances in Neural Information Processing Systems 29, pp. 4502-4510. Curran Associates, Inc., 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez- Gonzalzz, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: arXivv/806.001 1806.0/261, 2018.
Maria Bauza and Alberto Rodriguez. A probabilistic data-driven model for planar pushing, In IEEE International Conference on Robotics and Automation (ICRA), pp. 3008-3015, 2017.
Daniel M Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiau-V Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, al. Physion: Evaluating physical prediction from vision humans and machines. arXiv preprint arXiv:2 1rXiv:2106.08266 2021.
James H. Clark. Hierarchical geometric models for visible surface algorithms.
Commun. ACM, URL https:
19(10):477-55, oct 1976. ISSN 0001-0782. doi: 10.1145/360349.36055
[/doi.org/// 10. /1144//334449 360354
Erwin Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses, pp. 2015.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. arXiv preprint Dr.Xiv22044.1999 2022.
Nima Fazeli, Elliott Donlon, Evan Drumwright, and Alberto Rodriguez. Empirical evaluation of common contact models for planar impact. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3418-3425. IEEE, 2017.
10

---

Florian Golemo, Adrien Ali Taiga, Aaron Courville, and Pierre Yves Oudeyer. Sim-to-real transfer with aeural-d robot simulation. In Conference on Robot Learning, pp. 817-828. PMLR, 2018.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and David Scul- ley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1487-1495, 2017.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora. Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3749-3761, June 2022.
Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 202/ IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 9474 9481. IEEE, 2021.
Jemin Hwangbo. Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Roboties, x(26))eaau5872 2019.
Yifeng Jiang, Jiazheng Sun, and C Karen Liu. Data-augmented contact model for rigid body simu- lation. In Learning for Dynamics and Control Conference, pp. 378-390. PMLR, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv: 1412,690,, 2014.
Alina Kloss, Stefan Schaal, and Jeannette Bohg. Combining learned and analytical models for predicting action effects. CoRR, abs/1710.04102, 2017.
Alina Kloss, Stefan Schaal, and Jeannette Bohg. Combining learned and analytical models for predicting action effects from sensory data. The International Journal of Roboties Research, (8):778-797, 2022.
Lei Lan, Danny M Kaufman, Minchen Li, Chenfanfu Jiang, and Yin Yang. Affine body dynamics: Fast, stable & intersection-free simulation of stiff materials. ACM Trans. Graph, 2022.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019a
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference Robotics and Automation (ICRA), pp. 1205-1211. IEEE, 2019b.
Kevin M Lynch. The mechanics of fine manipulation by pushing. In ICRA, pp. 2269-222222222 2276. Citeseer, 1992.
Damian Mrowca, Chengxu Zhuang, Elias Wang. Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel Yamins. Flexible neural representation for physics prediction. Advances in neural information processing systems, 31, 2018.
Matthias Müller, Bruno Heidelberger, Matthias Teschner, and Markus Gross. Meshless deformations based on shape matching. ACM transactions graphics (TOG), 24(3):471-473 2005.
Mihir Parmar, Mathew Halm, and Michael Posa. Fundamental challenges in deep learning for stiff contact dynamics. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5181-5188. IEEE, 2021.
11

---

Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self-assembling morphologies: A study of generalization via modularity. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
Samuel Pfrommer, Mathew Halm, and Michael Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. Conference on Robot Learning (CoRL), 2020.
Yulia Rubanova, Alvaro Sanzhez-Gonzalez, Tobias Pfaff, and Peter W. Battaglia. Constraint-based graph network simulator. CoRR, abs/21 12.09161, 2021.
Alvaro Sanchez-Gonzalee Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470-4479, PMLR, 2018.
Alvaro Sanchez-Gonzalez,, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- ence on Machine Learning, pp. 1459-8468. PMLR, 2020.
D Stewart and JC J.C. Trinkle. An implicit time-stepping scheme for rigid body dynamics with Coulomb friction. International Journal for Numerical Methods in Engineering, (9(15)22 2691, 1996.
Russ Tedrake, Drake: Model-based design and verification for robotics, 2019.
URL https:
7/drake. miiraee.ii... edu
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033.
IEEE, 2012.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018.
Pierre- Brice Wieber, Russ Tedrake, and Scott Kuindersma. Modeling and control of legged robots. In Springer handbook of robotics, pp. 1200- 1234. Springer, 2016.
Kuan-Ting Yu, Maria Bauza, Nima Fazeli, and Alberto Rodriguez. More than a million ways to be pushed. fighffidliity experimental dataset of planar pushing. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 30-37, IEEE, 2016.
Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser. Tossingbot: Learning throw arbitrary objects with residual physics. IEEE Transactions Robotics, 36(4): 1307-1319, 2020.
12

---

A APPENDIX
B DATASET DETAILS
B.1 KUBRIC
Kubric dataset (Greff et al. 2022) consists of the rigid-body simulations of diverse 3D objects tossed simultaneously onto large plane. The simulations are created using the Bullet simulator (Coumans) 2015). Kubric provides several datasets with increasing complexity of the object meshes: MOVi-A, MOVi-B, MoviC, etc. In this work, we train the models on MOVi-A and MOVi-B, as those already present challenge to baseline models. We also provide the generalization experiments on MoviC.
Each trajectory in Kubric MOVi-A contains 3-10 objects different sizes tossed toward the given position the floor. In MOVi-A, only simple shapes used: cubes (51 nodes), spheres (64 nodes), and cylinders (64 nodes). MOVi-B and MoviC contain simulations with 11 and 1030 objects respectively (the latter taken from the Google Scanned Objects (Downs al. 2022)), with between 51 and several thousand nodes.
The objects have two types of material properties: metal (friction 0.4, restitution 0.3, density 2.7) or rubber (friction 0.8, restitution 0.7, density 1.1) for MOVi-A or MOVi-B, For MoviC, the material parameters are the same for all objects (friction 0.5, restitution 0.5 and density 1.0). We append mass, friction and restitution to the features for each node of the object.
The Kubric dataset was originally created for perception tasks and includes images of the generated trajectories, In this work, we use only the information about the physical state, consisting of the rest position of the mesh the object position and rotation quaterion. To obtain the state information, we re-generate the dataset using the github code github....... com/ gowgll-rreeeeeeeee aeteearc//// kubric For each of MOVi-A, A, MOVi-B and MoviC datasets, we use 1500 trajectories for training, 100 for validation and 100 for testing. Each trajectory consists of 96 time steps.
B.2 MIT PUSHING
Figure B.1: Objects used in the MIT pushing dataset. Figure credit to Yu al. 2016
The MIT Pushing Dataset introduced by Yu et al. (2016) consists of 6000 weal-world pushes for 11 different objects (Figure B. along 4 different surfaces, resulting in a total of 264000 trajectories. The trajectories vary that the pusher's velocity and acceleration of contact between the pusher and the object, and the angle between the pusher and the surface of the object where makes contact, Object poses and pusher poses were recorded at rate of 250 Hz, with each push unfolding over 5cm. We preprocess the data using the script provided by Kloss et al. 2022 at github..b... cwm. http://cuuulllbppppppap
change, well the point
Trajectories from this dataset consist of the positions and rotations of the object and robotic pushing tip. To compare results to Kloss al. (2022). we take the subset of trajectories which have pusher acceleration of O.. Each trajectory is cut off after 0.5 seconds have passed, corresponding to 125 timesteps. From the full set of trajectories, we randomly choose 10000 trajectories across all objects from the "abs" surface material for training, and 1000 for testing. Of these 10000 trajectories, sample different dataset sizes for training to examine FIGNet's sample efficiency.
For this dataset, we do not use static properties of objects in the node features, other than to indicate that nodes with mass 0 belong to the controlled pusher. However, we do subsample trajectories for training and rollouts to once every timesteps, which stabilizes training. To compensate for errors in state estimation the dataset, we found a larger node noise value was also critical (0.001). Because this dataset has controlled pusher, also do not predict the pusher's position. We predict only the
13

---

movement of the object mesh. Otherwise, all training details and architecture choices identical to those used for the Kubric dataset.
C METHOD DETAILS
C.I FULL MODEL DESCRIPTION
FIGNet uses an SncoddePrrceese Deccede approach similar to (Sanchez- et al., 2020 Pfaff ct al., 2021). Crucially the encoder and processor are adapted to able to support multiple node and edge types simultaneously. Also we introduce passing through face-face edges, which each have three sender and three receiver nodes, rather than just Note in case face-face edges always thangle-triange edges, hence the three senders and receivers, but this approach would generalize to any other type of edges, such as triangle-poini edges, satraheeroo-triangle edges, segment-triangl edges, etc. Also note that our approach is strict generalization of message passing between pair of nodes, and when applied to point-point edges, it recovers all aspects of regular edge updates.
type of edge update function for message
C.1.1 ENCODER
The encoder constructs the input graph 9 = (VM, 100..M. g M Q") that represents the scene from the mesh M with 2 different node types, 3 different regular edge types, and 1 face-face edge type. Each node type has some features associated with it, which encoded into fixed-size latent spaces.
Mesh nodes M.features
yM represents the set containing each of the mesh nodes v.M. with input features x xx,,
V
x Ps. k; f.]. where x is the position of the node time t, Pi static object properties, k; is binary "kinematic" feature that indicates whether the node subject to dynamics (e.g. the moving objects), or its position set externally (e.g. the floor), and ky (x, - x{) is feature that indicates how much kinematic nodes are going to move the next time step. We use an MLP to encode all of these features into an initial latent representation for mesh nodes:
[xx x x ....,
pencoder MI. VM
(v M.fearures
Object nodes D represents the set containing each of the object nodes representing each rigid body ep with input features V; O.input analogous to those of the mesh nodes, using the center of mass of the object as the position for the object node. We use an MLP encode all of these features into an initial latent representation for mesh nodes:
MLP.Roddr (v; Oiipptt
Mesh edges M are bidirectional edges added between mesh nodes that are connected in the mesh. For each mesh edge cM C connecting sender mesh node eM to receiver mesh node r.M we build W
edge features M,features = [dg, ax where dj r.M - is vector of position differences between
the nodes in the currently rotated mesh M (e.g. the positions of the nodes at the current of timestep, which are also affected by training noise); and d is the position difference in the reference mesh R (which is static and independent of the dynamics). We use an MLP to encode all of these features into an initial latent representation for mesh nodes:
M M
MLPemeoder (c (c MMM
M.feaurres)
Object-Mesh edges eUM are unidirectional edges that go from each of the object nodes, to all of the mesh nodes that belong to that object. There is object-mesh edge e.MM - for each mesh
OM. are analogous to the mesh computed from the positions node in M Input features c - of the object and mesh nodes. We use an MLP to encode all of these features into an initial latent representation for mesh nodes:
OM
ML.enncdee
OM.features 10---06
14

---

Mesh-Object edges eo are unidirectional edges that go from each of the mesh nodes, to the object node representing the object it belongs to. There is also one mesh-object edge eMO -o for Mo.features are analogous to the mesh edges, computed from each mesh node in yM_ Input features e, MM-
the positions of the mesh and object nodes. We use MLP to encode all of these features into an initial latent representation for mesh nodes:
MO 7 = MLPPE
Pencoder
M (c (c,
MO.fentures) -M
Face-face edges @ are edges that connect two mesh faces belonging to different objects. A face- face edge u is added if any point of the sender face FMM is within the collision radius de from any point of the receiver face F.M. This is a new type of edge, because from the point of view of the nodes of the graph, this "directed hyper edge", that connects the three mesh nodes of the sender mesh face FM (val 2, M), to the three mesh nodes of the receiver mesh face J.M
(u 12 )
For each face-face edge, we build features, 4 [d.,, (d., (11,,,, (d,,,y=1.2,3 D. n,). The features are defined with respect to the "closest collision points" between the two faces Ps (on the sender face F.), and p. (on the receiver face.F,). Geometrically, the closest point in the face might be either inside of the face, on one of the triangle edges or one of the nodes. Then the features are defined as follows: (1) the relative vector between the closest points d,s Pr Ps for the two faces, (2) the spanning vectors of three nodes of the sender face Fs relative to the closest point at that face d,, Xs, Ps. (3) the spanning vectors of three nodes of the receiver face Fr relative to the closest point at that face d, = X,, Pr. and (4) the face normal unit-vector the sender and receiver faces n, and n,, pointing towards the outside of the object.
Finpor
We use an MLP, followed by a reshape operation, to encode all of these features into an initial latent representation for each face-face edge three vectors, for each receiver node: I into three face-face edge latent vectors, one for each receiver node JM (u y. U of each face-face interaction:
Q.y-rx [9.. (4.... reshape(ML
features (g (UMM+JFM) )))
Having one edge vector per edge receiver node will be a crucial aspect the face-face edge update and mesh node update. Crucially, for each edge, and before computing features (2) and (3), we sort the nodes of the sender face F. = M..M .. M) and receiver face FM = (v 122 of the distance to the closest collision point in the corresponding face. This achieves permutation
..M. My as function
22 (33
equivariance of the entire model, w.r.t. to the order in which the sender, and receiver nodes of each face are specified.
C.1.2 MESSAGE PASSING AND ITERATIVE PROCESSOR
The goal of one step of message passing in the processor is to update all latent representa- tions in the encoded graph based on neighborhood information. This includes mesh edge latents M " object-mesh edge latents eoM - mesh-object edge latents Mo e-M++00 face-face edge la- tents QyN-IM,, object node latents vo and mesh node latents v.M. to produce updated (indicated as "prime") versions those: e eyM-soM - -.. QyM-IN v and v.M From a high level perspective the update occurs in two stages: (1) run an edge update (also referred to as "message function"") on all of the different edge types (mesh edges, object-mesh edges, mesh-object edges and face-face edges), gathering information about the nodes adjacent to the edge, and (2) run node update for each node type (mesh nodes and object nodes), aggregating information from edges of different types adjacent to the nodes.
' OM
MO
The previous paragraph describes a single layer of message passing, but following a similar approach to Sanzaez-Gonzalez et al.) (2020); Pfaff al. (2021) this layer can then be applied iteratively, as many times as necessary. with either shared or unshared neural network weights. Also similar to those we also add residual connections (sum the input to each layer to the output of the layer) at each layer. to improve gradient flow during training. Our processor consists 10 steps of message passing, with unshared weights.
15

---

Regular edge updates
used to update the mesh edge latents e - object-mesh edge la-
tents o0 -ot mesh-object edge latents
MO
Each edge is updated following the edge update
approach from (Battaglia et al. 2018):
peroesssoo gxy
(leux -ot V v.
where X, Y may take the role mesh (M) or object (O) nodes, depending on the type of edge.
Face-face edge updates are used to update the face-face edge latents Qyy- The approach to update each face-face edge is similar to regular edges, except that now each edge has three latent vectors, three senders and three receivers, and we also need to produce three output vectors:
-
re (M ( ey? V (,,==12)3)))
OF
noting that, this does not update the three latent vectors independently, but all nine input latent vectors [[g, vay vM [=122 contribute to all three updated face-face edge latent vectors QF.-F,'.
Object node updates used update the object node latents vP. Each node is updated fol- lowing the node update approach from (Battaglia et al.) 2018), aggregating all of the updated edge latents (or messages) for mesh-object edges that are adjacent that object node:
Processie 0
MLPPoces Do
(Iv)
Σ
MO UM -o
'))
/10==
Mesh node updates are used to update the mesh node latents v Mesh nodes may receive in- formation from object-mesh edges, mesh edges, and face-face edges, so update each mesh node, information of adjacent edges to that node is aggregated for each of the edge types:
processor M
MLP PM
(LVM
Σ
Σ
G
BoM--oM
Σ
VMOM x mo
VoM
Va -F, /TN -M
where the second and third terms correspond regular edge aggregation (just like in the object node update), and the last term sums over the set of face-face edges for which wM is one of the receivers and g/FF,FF, selects the specific face-face edge vector corresponding to u.M4 as receiver, i.e. from O'F,FFF. selects the first, second, or third vector, depending on whether v is the first, second or third vector in that receiver face (recall it is always sorted by distance to the closest point in the face, so the model remains permutation equivariant).
C.1.3 DECODER
Similar to output features. In our model we only require output features for the mesh nodes vi We produce this by applying MLP decoder from the latent space of the mesh nodes after the processor, which outputs finite-difference approximation to the acceleration:
et al. 2020 Plaff et al. 2021), the goal of the decoder is to produce
a, MM.plecoder M)
C.2 OTHER MODEL DETAILS
Norm features For all relative spatial feature vectors d, we also also concatenated their norm [d] part of the inputs.
16

---

Training noise To make models stable for long rollouts while training one-step data, the same strategy from (Sanchez-Gonzze et al. 2020; Pfaff et al. [202 to train with random walk noise in the imputs, asking the model to correct for noise in the input velocities.
Predicted targets Similar to (Sanchez-Gonzale et al., 2020 Pfaff et al. dicts a finite- difee-difference acceleration that is used to update the position x; a, 2xf x!
2021 our models pre-
Normalization Similar to (Sanchez-Gonzale, et al. 2020; Pfaff et al. 2021) we normalized all inputs and targets zero-mean unit-variance. The loss is computed in the normalized space of the targets. This means our entire model and loss is agnostic to the scale of the data.
MLPs We MLPs with 2 hidden layers, and 128 hidden and output sizes (except the decoder MLP, with an output size of 3). All MLPs, except for those in the decoder, are followed by a LayerNorm Ba et al. 2016) layer.
Optimization All models are trained to 1M steps with a batch size of 128 across TPU devices. We use Adam optimizer, and an exponential learning rate decay from le-3 to le-4.
D BASELINE DETAILS
D.1 MuJoCo AND BULLET SIMULATORS FOR REAL- WORLD MIT PUSHING DATASET
For the MIT Pushing Dataset we compare with the MuJoCo (Todorov et al.] 2012) and Bullet (Coumans) 2015) simulators. While the objects have a provided mesh geometry, the physical prop- erties of this real-world system, as well as the optimal internal simulator parameters, are unknown and thus must be estimated by system identification.
We construct this system identification problem as a black-box optimization procedure as follows. The objective we minimize is the sum of the relative translation error and relative rotation error of object being pushed. measure this objective we use a 50-trajectory subset of the full training dataset; making predictions for these 50 trajectories takes 5-10 minutes for each simulator. Using more trajectories for fitting did not yield improved results. As our optimizer, we choose Bayesian optimization with Gaussian processes as implemented by Vizier (Golovin et al., 2017). We perform 500 trials of optimization, then take the best-performing hyperparameters and evaluate them on a 100-trajectory validation set.
For MuJoCo we model the externally-controlled pusher as a non-physical motion capture body attached via an equality constraint the physically-modeled pusher geometry. This allows the simulator to impute velocities for the geometry which are unavailable when using motion capture positions directly, making the physics stabler and more realistic. We find that modeling contacts with three degrees of freedom and neglecting the differential effects of rotational and rolling friction gives strictly superior results and so perform our final optimization using this setting. We use the expensive but more accurate RK4 integrator rather than the default Euler integrator for greater precision in modeling hard contact. We search over the space of scalar friction coefficients objects, as well as over the number of physics substeps {1, 10, 100, 1000]. The best-performin solution has Kobject = 1.91, kpwsher = 0.41, and Anoor = 0.O, and uses 1000 physics substeps per data timestep. For Bullet create fixed constraint for the tip and directly update the parent location of the fixed constraint with the positions of the tip given by the input trajectories. We search over the space of lateral friction coefficients & E [0,5] for all three objects, as well over the number of engine substeps {0, 1, 2, 3, 4, 5). The best cessfrperooning solution has hiobjoce 0.28, kipuster = 0.0, and knoor 1.106, with 0 substeps.
[0, 5] for all three
D.2 MESHGRAPHNETWORKSS (MGN)
To make the comparison more informative and fair, we reimplement MeshNets refer to model with the same "bells and whisles" to ours, except for two joint ablations of important differences with (Pfaff al. 2021
17

---

Translation error
Rotation error
Collision edges
FIGNet
Callsaa argeRaddin FiGNct Redbjecthode
30.00
10a
0.80
F1C24et-Wi Nof accCollaiine
Collision edges
Error (m)
Error (deg)
0.40
20.00
10a
15.00
0.20
10.00
Figure F.1: Model ablations on MOVi-A dataset. Replacing the face-face collisions with the node- node collisions makes the performance deteriorate the most. Models with use implicit floor representation, while models with use subdivided floor.
Remove object nodes (FIGNettNoOOjjecNWude) The ablation consists of removing the object nodes, as well all of the object- edges from the graph.
The ablation consists of computing collisions according to the distance between the mesh nodes, rather than the faces, and replacing the face-face edges, by regular mesh-mesh edges, referred to as "world edges" in (Pfaff et al., 202T). Unlike MeshGraphNets, this model has the per-object node, similarly to FIGNet. In order give this ablation chance, we (1) use larger collision radius de (LargeRadius) and (2) subdivide the two gigantic triangles that make up the floor into smaller triangles of similar in size the collision radius (1.5), or (3) when (2) becomes too we use an "implicit floor" by adding an extra feature to the nodes indicating if they are within the collision radius distance d, from the floor and by how much. We also provide some results from applying these ablations independently.
Node based collisions (FIGNet-NoFaceColliiin
D.3 DPI-REIMPLEMENTED
To make the comparison more informative and fair, we use DPI-Reimplemented to refer to model with the same "bells and whisles" to ours, except for three joint ablations important differences with (Li et al. (019b)).
Object-level predictions (FIGNet-DPILLoss) The ablation consists of: (1) use decoder for the object nodes, to produce a velocitylacceleration, for the position and rotation quaternion of the center of mass of the object, (2) used that prediction update the mesh nodes positions, (3) compute the effective mesh node acceleration, and (4) use this to build the loss.
We replaced the mesh data by a dense particle representation of the object, and use those particles as nodes. As there aren't any faces we also use node-based collisions. Note in this case, we still keep the central object node, following (Li et al. 2019b) which would correspond to k 1 level hierarchy their model.
Particles (FIGNet-Particlkl
We also provide some results from applying these ablations independently.
ABLATION RESULTS
To motivate the importance of the per-object nodes and face-face collisions, we provide the ablations of our model where remove either of these properties from the model.
Figure E. and Figure E.2 demonstrate the ablation on MOVi-A and MOVi-B datasets respectively. Removing the face-face collisions, but keeping the same collision radius (NoFaceCollision model) makes the performance deteriorate on both MOVi-A A and MOVi-B, presumably because the current collision radius does allow capture all the nodes that necessary to resolve the collision.
18

---

Translation error
Rotation error
Collision edges
FIGNet
Calliia argeRadiun FiGNct Redbjesthoee
30.00
0.80
F1C24el-Ae Nof scetollaigne
Collision edges
10*
Error (m)
Error (deg)
0.40
20.00
10a
15.00
0.20
Figure F.2: Model ablations on MOVi-B dataset. Removing the object node and replacing face-face collisions with node-node drastically affects the performance, making it worse. Models with use an implicit floor representation, while models with + use a subdivided floor.
Translation error
Rotation error
Collision edges
1.60
40.00
10a
FIGNet
FiGNet- Particles* OPL-Reimplementer
0.80
30.00
Collision edges
Error (m)
Losss
Error (deg)
10
0.40
20.00
10a
0.20
0.10
10.00
10a
Figure E.3: Model ablations towards DPI LLi et al. 2019b) on MOVi-A dataset. While the loss formulation has little effect, replacing the mesh with densely sampled particles creates huge issues with memory requirements, and makes overall translation and rotation error worse. Models with" use an implicit floor representation.
Increasing the collision radius NofFaceColision-LLageReaaa restores the performance on MOVi- A, but it is not sufficient on MOVi-B to match the performance of FIGNet.
Next, consider the ablation of removing the per-object node from FIGNet. On MOVi-A this modification does not affect the performance as much, presumably because the small number of nodes per object in MOVi-A relatively small and it does not require additional message-passing through the object node. However, on MOVi-B we observe much higher translation and rotation errors compared FIGNet and other baselines. This result highlights the importance of object-level node in the complex datasets like MOVi-B, as provides "shortcut" the message-pasee paaeeeppssnngggg between nodes of the object.
Note that the FiGNet-NoFaceCollen the face-face message-passe passing with node-node interactions. They suffer from the same issues related the param-
and CCCCNee-oaaaeClllssioon---LLLLL LaigeRRRauss baselines replace
to floor parameterizations as MeshNets, desribed in the main text. Therefore eterization with subdivided floor for MOVi-A, and implicit floor for MOVi-B for these models, similarly to MeshNets (see explanation for Figure 5a Figure 5b in the main text).
We also consider various ablations that gradually move the FIGNet model towards the DPI- Reimplemented marticle-bssed model, outlined in the ablations sections. Note that changing to particle based representation makes it impossible to represent the floor using a dense representation, so all models with particles implicit floor (marked with in the main text).
19

---

Translation error
Rotation error
Collision edges
1.60
10a
FIGNet
PiGNe--Particles
0.80
DPl-Beiinllmmented PiGNet-DPLLos
30.00
Collision edges
Error (deg)
10
Emor (m)
0.40
20.00
10a
0.20
0.10
10.00
10a
Figure E.4: Model ablations towards DPI on MOVi-B dataset. While the loss formulation has little effect, replacing the mesh with densely sampled particles creates huge issues with memory requirements, and makes overall translation and rotation error worse. Models with use an implicit floor representation.
F GENERALIZATION ACROSS SURFACE GEOMETRIES
To test generalization across different surface geometries, we created custom scenes where sphere dropped in location and rolls along the different surfaces (Figure F.Ip. Despite never seeing inclined, curved or concave planes during training, FIGNet produces plausible trajectories of how the ball should roll. This supports our findings on the remarkable ability of FIGNet to generalize to new scenes, shapes and floor landscapes.
Figure F.I: Demonstration of generalization rollouts from Kubric MOVi-B to 3D adaptation of the Bridge level from the Virtual Tools Game (Allen et al. 2020). Grey and blue are static and dynamic objects respectively.
G GENERALIZATION ACROSS OBJECT SHAPES
To test generalization across different shapes, train models the MOVi-A or MOVi-B datasets, and test their generalization to MOVi-B MoviC respectively, with further fine-tuning. In Figure G.la and Figure G.Ib, FIGNet clearly outperforms alternative methods in generalization. Even more remarkably, FIGNet's performance on MOVi-B, when trained only on MOVi-A, still surpasses all baseline performances when the baselines are trained on MOVi-B directly. This sug- gests that FIGNet is not only accurate when trained, but provides compelling reasons to believe it will generalize to further complex dynamics in future.
20

---

Demonstration of generalization rollouts from Kubric MOVi-B to manually de- signed u-slide and imported asset (hippo obj asset imported from github. ctt.coom//aaam. mmmackinn/ tinsel. git)). Grey and blue static and dynamic objects respectively.
Figure F.2:
Figure F3: Demonstration of generalization rollouts from Kubric MOVi-B to an unseen ramp.
Translation
Rotation
Hobes Micificts Larpefistes MashWetts
40.00 00
0.80
Error (m)
Error (deg)
30.00
20.00
0.20
15.00
10.00
(a) MOVi-A to MOVi-B
Translation
error
80.00
FIGMA
Lergetakaat
1.60
Mushivess
Eror (m0)
0.80
Eroor (dee)
40.00
0.40
20.00
0.20
0.10
30.00
(b) MOVi-B to MoviC
Figure G.1: Generalization performance of the FIGNet model (a) trained on Kubric MOVi-A and tested on MOVi-B (b) trained on Kubric MOVi-B and tested on MoviC.
21

---

Ground
Truth
FIGNet
MGN
MGN
LargeRa-
dius
DPI
Reimple- mented
Figure G.2: Rollout examples on MOVi-A dataset
Ground
Truth
FIGNet
MGN
MGN
LargeRa-
dius
DPI Reimple- mented
Figure G.3: Rollout examples on MOVi-B dataset
22

---

Ground
Truth
FIGNet
MeshNets
MeshNets
LargeRa-
dius
DPI
Reimple- mented
Figure G.4: Generalization from MOVi-A to MOVi-B dataset
Ground
Truth
FIGNet
MeshNets
MeshNets
LargeRa-
dius
Figure G.5: Generalization from MOVi-B to Movi-C dataset
23

---

H PENETRATION MEASURES
To measure penetration, we implement the algorithm from (Bærentzen & Aanaes 2005). Since
this algorithm is imperfect when faces are small, we report penetration statisties with respect the ground truth trajectories given the same initial scene. FIGNet penetration distances are only 3% greater than the ground truth simulator, with only 4% more faces in penetration with each other relative to ground truth. By comparison, MeshGraphNets performs considerably worse when using a small collision radius, with penetration distances greater than the ground truth model. These results are reported in Table H.T
Table H.I: Penetration distances and counts reported as a ratio to the distances and counts for the ground truth PyBullet simulator.
(a) MOVi-A dataset
Model
Penetration distance ratio
Penetration count ratio
FIGNet
1.037 + 0.033
1.041 + 0.037
MGN-I LargeRadius+
1.071 + 0.020
1.073 + 0.018
MGN+
4.613 + 0.143
5.246 + 0.187
(b) MOVi-B dataset
Model
Penetration distance ratio
Penetration count ratio
FIGNet
1.121 + 0.034
1.147 0.039
3.813 + 0.985
4.473 1 1.197
MGN
5.614 1 0.473
6.703 1 0.573
I RUNTIME
We additionally report runtime performance (on CPU) to more directly compare the amount of time taken to run single forward step of each model. These results are reported in Table I.1 for the MOVi-A dataset and Table 1.2 for the MOVi-B dataset.
FIGNet has the shortest inference runtime in comparison to baselines for the MOVi-A dataset. MGN-LargeRadius+ which is closest to FIGNet in terms of accuracy (Figure 3(a)), requires 2.7x more time to perform inference step. DPI-Reimplemented* and MGN+ are inferior to FIGNet in terms of both runtime and accuracy.
For the MOVi-B dataset, FIGNet is slower than the alternative models, despite using fewer collision edges on average. This can be explained since the majority of the complexity for MOVi-B is due to the complexity of the mesh objects. Unlike the baselines, FIGNet contains edges connecting nodes of the mesh (mesh edges) and edges from the object node to each of the mesh nodes. Thus, with added object complexity, FIGNet contains more edges. Furthermore, FIGNet the only model of the four to represent the floor explicitly, and many of the collisions that occur are between the floor and the objects. All alternative models need represent the floor implicitly (marked with "), as otherwise they run out of memory. Future work could investigate improvements to object representations which require fewer mesh nodes to be connected, which we expect would improve FIGNet's runtime performance relative to baselines.
Table I.1: Computational performance on MOVi-A dataset (runtime and number of collision edges)
Model
Runtime (seconds)
Collision Edges
FIGNet
0.091 +: 0.005
233 : 2.1
MGN-L MGGN-LargRRaduss
0.258 + 0.010
10637 317
DPI- Reimplemented*
0.126 0.006
2515 34
MGN+
0.160 = 0.007
3.8 0.39
24

---

Table L.2: Computational performance on MOVi-B dataset (runtime and number of collision edges)
Model
Runtime (seconds)
# Collision Edges
FIGNet
0.342 0.012
1385.610 + 23.818
PPI--Rempleeented
0.145 + 0.009
2250.688 + 64.507
MGN-Larg LargeaaggRadius
0.218 1 0.033
1797.985 + 59.699
MGN
0.175 1 0.018
34.367 1 4.075
J EFFECTS OF DIFFERENT COLLISION RADII
Figure J.1 demonstrates the ablations over different collision radii for the MeshGraphNets (MGN+) (Plaff et al.] (02T) model compared FIGNet with collision radius 0.1 on the Kubric MOVi-A dataset. Notably, there is no collision radius where MeshGraphNets would simultaneously have comparable runtime FIGNet while also having comparable accuracy. Large values for the colli- sion radius Iead to better accuracy, but at the cost of runtime performance. The increase in collision radius also leads to an exponential increase in collision edges MeshGraphNets,, as more nodes fall into the collision radius, leading to higher memory consumption. In experiments, the increase in collision edges manifested as 50% increase runtime from collision radius 0.05 to 1.5.
Translation error
collisinn edges
0.3
(m)
0.0
1000.0
Runtime
100.0
0.2
15.0
1.0
0.1
10.0
0.1
0.05
redius
Figure J.1: Ablation over collision radii on MOVi-A dataset for MGN, compared to FIGNet with collision radius 0. If the nodes (in MGN) faces (in FIGNet) between different objects lie within the collision radius, we connect them with an edge in the graph.
By comparison FIGNet is not particularly sensitive to the collision radius. We trained models with collision radius of 0.05 and 0.5 relative to the main model's radius of 0.1. The errors for these different collision radii are given Table
Table J.1: FIGNet collision radius sensitivity analysis for MOVi-A dataset.
Metric
Collision radius
0.05
0.1
0.5
Translation error (m)
0.151 1 0.014
0.115 0.008
0.105 - 0.006
Rotation error (deg)
16.8 + 0.3
14.4 0.2
13.4 1 0.4
25

---

K RESULTS FROM MAIN TEXT AS TABLES
For ease of comparison between models, we provide the results on Kubric from the main text as tables.
Table K.1: Results on Kubric datasets from main text as table
Kubric MOVi-A results
Model
Trauslation error
Rotation error
Collision edges
DPP-Reimplemmnted
0.180 0.017
20.817 0.926 15.069 + 0.649
2515.354 34.341
MGN-- arr--aaageRddiu++
0.119 + 0.009
10637.350 + 317.336 3.792 + 0.396 232.644 + 2.135
MGN+
0.705 + 0.069
31.210 + 0.989
FIGNet
0.115 0.008
14.387 1 0.175
Kubric MOVi-B results
Model
Translation error
Rotation error
Collision edges
Reippeiipleeeetec
0.368 + 0.057
26.928 2.740
2250.688 64.507
Lar-LaageRadius***
0.460 0.045
26.342 1.397
1797.98 - 59.699
MGN
0.538 1 0.035
26.914 0.783
34.367 1 4.075
FIGNet
0.127 + 0.006
13.990 1 0.464
1385.610 + 23.818
26


Reviews:
[{"review_id": "uIiXwB_anQ", "reviewer": "Reviewer_edAB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Face Interaction Graph Network (FIGNet) to extend GNN-based rigid object collisions simulators for interaction computation between mesh faces, rather than nodes.  The proposed FIGNet is 4x more accurate in simulating complex shape interactions and 8x more computationally efficient on sparse and rigid meshes than learned node- and particle-based methods. ", "summary_of_the_review": "Based on the strength mentioned above, my current decision is accept.", "strength_and_weaknesses": "Strength\n\n1.\tThe paper is well written so that it is easy to follow and understand the motivation and the proposed method.\n\n2.\tThe proposed FIGNet is novel and experimentally demonstrated to be effective.\n\nWeakness\n\n1.\tThe experimental scenarios are a little simple, it would be better to conduct experiments on more realistic data.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "n6vmJscV-k", "reviewer": "Reviewer_sivF", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an extension of MeshGraphNet by adding explicit modeling of rigid body faces. The authors show that this modification significantly improves simulation performances over a large number of tasks while reducing the computational burden.", "summary_of_the_review": "The method presented in this paper is very interesting. I particularly appreciate the clarity of the writing as well as the extensive empirical study, and the care taken to offer solid and comparable baselines. My score is however reduced by a doubt as to the practical interest of FIGNet against MeshGraphNet in the case where $d_c$ is increased, and also particularly by the liberties taken by the authors regarding the editorial rules, which is not very fair with other submissions. If the authors provide satisfactory answers to these two points, I will gladly increase my score!\n\n## EDIT\nThe authors answer my remark in a very complete and convincing way. I believe that the new experiments add great value to the paper and thus highly recommend this paper for acceptance.", "strength_and_weaknesses": "I very much appreciated this paper which introduces with great clarity a method which seems reasonable. The addition of face-level interactions in the structure of GNNs is a significant improvement over standard models, and seems to me somehow related to previous works on dual graphs, e.g. [1]. A discussion of the key differences with existing works on the subject could perhaps be added.\n\nFIGNet seems relevant and efficient, and the design choices presented are reasonable. I only have two questions about it:\n\n- The collision distance $d_c$ seems to be an important parameter. Did the authors measure the effect of this parameter on learning and inference? Is it a sensitive hyper-parameter? Can we imagine learning $d_c$ directly with the weights of the model?\n\n- Sorting the nodes of a face according to the distance from the nearest point on the other face is said to be \"crucial\". Can the authors discuss more clearly the significance of this? Have ablation experiments been performed?\n\nRegarding the experimental results, these are indeed \"plausible\". But this also seems to be the case for MeshGraphNet when the collision distance is increased. The comparison with the baselines would be easier if the results were aggregated in a table, which could be added in the appendix. The authors justify the advantage of FIGNet by evoking the exponential complexity of the baseline when $d_c$ increases. Nevertheless, FIGNet trades off the increase in the number of edges against an increase in the complexity of the model. Finally, it seems to me possible that the gain of FIGNet is quite limited in practice. Thus, a more in-depth analysis of the difference between these two models seems necessary. It would be particularly interesting to measure their inference time (and why not the number of FLOPs).\n\nFinally, one of the weaknesses of the approach is the need to have access to a rich and detailed representation of the scene, in the form of a mesh. Have the authors considered solutions to train their models from weaker signals, or even directly from images?\n\nIn addition to the quality of the work offered, I note the use of certain tricks to keep the number of pages to the limit. In particular, the referencing of figures in the appendix in the main text is questionable, but I recognize that this is common practice. What concerns me more is the modification of the style of bibliography compared to the standards of ICLR. It seems to me that this style is mandatory (while it requires a lot of space). If this is indeed the case, I would appreciate if the authors regularize their paper, for the sake of fairness with other submissions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "g0tzbZuZLE0", "reviewer": "Reviewer_zfn2", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a “Face Interaction Graph Network” (FIGNet)\" architecture for accurately and efficiently modeling collisions between faces of meshes. Previous SOTA methods based on Graph Neural Networks (GNNs) model only collisions between nodes of the meshes of different objects and hence cannot deal effectively and efficiently with objects with large faces without requiring significant sub-division of the mesh's surfaces. There are two key innovations that the authors introduce into GNNs to solve this problem. The first is to introduce a mechanism for message passing between the face and the nodes that comprise it and the second is to introduce and object-level virtual node for efficient passing of information between nodes without requiring too many message passing iterations. The authors compare their approach to SOTA learning-based graph and particle methods for this task on a simulated dataset and against an analytical solver on a real-world object pushing task. They show SOTA performance in terms of accuracy, computational efficiency and generalizability of the proposed method versus all existing approaches.", "summary_of_the_review": "Overall this paper introduces a significant and broadly impactful novel contribution to the computer vision research community, which can potentially open the doors to new research ideas and applications. Hence, I recommend accepting this work.\n", "strength_and_weaknesses": "Strengths:\nNovelty: The work solves the previously unsolved problem of how to enable effective and efficient message passing between colliding mesh faces in a GNN. To do so they introduce a task-specific module of encoding \"Face-face edge features\" along with mechanism for effective message passing between these new features and the existing nodes and edges of the graph. The proposed design is conceptually correct and well-motivated. The authors also (potentially) introduce a virtual object node at the center of each object to pass messages between distant nodes more efficiently. This too makes sense.\n\nClarity: The paper is well-written and the material is presented in a well-organized fashion.\n\nSignificance: The method shows impressive performance improvements over the SOTA methods in terms of accuracy and computational efficiency and generalization, outperforming many previous learning-based and analytical baselines. The authors' results on effectively learning real-world dynamics besides the ones from simulation are exciting and can open the door to many new applications.\n\nWeaknesses:\n1. It is a bit unclear from the text and experiments if the virtual object node is the author's novel contribution or it comes from prior works on particle based approaches. It would be good if the authors could state this more clearly in the text.\n\n2. The authors have clearly acknowledged several limitations of their presented technique in terms of not being able to deal with stochastic outputs, requiring state information as opposed to predicting from images alone, and in terms of learning with even more complex objects with many orders of magnitude more points. I am satisfied with with this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "YfeJNGxYJ5D", "reviewer": "Reviewer_dbQg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel face interaction mechanism for GNN-based rigid body simulators to solve the penetration issue existing when the contact point is far from nodes. The new framework significantly increases the accuracy of contact resolution and enables the simulation of complex shapes. The framework is even more expressive than analytical simulators. It can capture subtle dynamic behaviors shown in real-world data that cannot be interpret by some existing rigid-body simulators.", "summary_of_the_review": "The paper proposes an extension to MeshGraphNets, which significantly increases the contact resolution accuracy. However, quantitative statistics about the efficiency and accuracy of the proposed framework as a rigid body simulator are needed.", "strength_and_weaknesses": "**Strength:**\n\n- The illustrations are helpful in understanding the idea, and the technical details are complete.\n\n- This extension to MeshGraphNets is elegant.\n\n\n**Weaknesses:**\n\n- The following reference may be closely related to the topic:\n\n*Lei Lan, et al. 2022. Affine Body Dynamics: Fast, Stable and Intersection-free Simulation*\n\nThe above simulator can achieve competitive or faster rigid-body dynamics than Bullet, Mujoco, and PhysX, and can guarantee non-penetration for arbitrarily complex shapes. I would like to see a wall-clock time cost comparison to Bullet, Mujoco, and PhysX as well.\n\n- There are only two contact scenarios for discrete mesh: point-triangle contacts and edge-edge contacts, as shown in the following reference:\n\n*Minchen Li, et al. 2020. Incremental Potential Contact: Intersection- and Inversion-free, Large-Deformation Dynamics.*\n\nThe experiments would be more thorough if the authors could specifically examine how these two contacts are resolved.\n\n- Mathematically, contact is just a non-penetration constraint for simulation. I would like to see **quantitative** statistics on penetration counts or penetration depths when making comparisons.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%