Learning rigid-body simulators over implicit shapes for large-scale scenes and vision
Authors: Yulia Rubanova
Tatiana Lopez-Guevara
Kelsey R Allen
William F Whitney
Kim Stachenfeld
Tobias Pfaff
Venue: NeurIPS 2024 oral (2024)
Keywords: graph networks
learned simulation
physics
rigid body simulation
scaling
Primary area: graph_neural_networks

Abstract: Simulating large scenes with many rigid objects is crucial for a variety of applications, such as robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simulation parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to hand-designed simulators like MuJoCo and Bullet. They are able to accurately capture dynamics of real objects directly from real-world observations. However, current state-of-the-art learned simulators operate on meshes and scale poorly to scenes with many objects or detailed shapes. Here we present SDF-Sim, the first learned rigid-body simulator designed for scale. We use learned signed-distance functions (SDFs) to represent the object shapes and to speed up distance computation. We design the simulator to leverage SDFs and avoid the fundamental bottleneck of the previous simulators associated with collision detection.
For the first time in literature, we demonstrate that we can scale the GNN-based simulators to scenes with hundreds of objects and up to 1.1 million nodes, where mesh-based approaches run out of memory. Finally, we show that SDF-Sim can be applied to real world scenes by extracting SDFs from multi-view images.

Learning rigid-body simulators over implicit shapes for large-scale scenes and vision
Yulia Rubanova" Tatiana LopezaGuevara Kimberly Stachenfeld
Kelsey Allen Tobias Pfaff
William F, Whitney
Google Deepmind
Abstract
Simulating large scenes with many rigid objects is crucial for a variety of appli- cations, such robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simula- tion parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to hand-designed simulators like MuJoCo [36] and PyBullet [13]. They are able accurately capture dynamics of real objects directly from real-world observations. However, current state-of-the-an learned simulators operate on meshes and scale poorly to scenes with many objects or detailed shapes. Here we present SDF-Sim, the first learned rigid-body simulator designed for scale. We use learned signed- distance functions (SDFs) to represent the object shapes and to speed up distance computation. We design the simulator to leverage SDFs and avoid the fundamental bottleneck of the previous simulators associated with collision detection. For the first time literature, we demonstrate that we can scale the to scenes with hundreds of objects and up to 1.1 million nodes, where mesh-based approaches run out of memory. Finally, we show that SDF-Sim can be applied real world scenes by extracting SDFs from multi-view images.
simulators
Introduction
Simulating real-world environments, such as bookshelf with books and decorations dinner table with plates and glasses, presents significant challenge for traditional physics simulators. These simulators require precise 3D shape, location, and corresponding physical parameters of each object, making the simulation arbitrary scenes a difficult task.
Recently, learned simulators based on graph networks (GNNs) [4, 19, 29] have been introduced as a powerful alternative to traditional hand-designed simulators like MuJoCo [36] or PyBullet [13]. Graph networks can capture range of complex physical dynamics, learn physical properties directly from real data and generalize to new scenes. However, current -Ns-based methods do not scale well to large scenes. Similarly to the traditional simulators, they rely on 3D meshes to describe object shapes. In scenes with large number of objects or objects with detailed meshes, these scenes might contain thousands of nodes and mesh triangles, making collision detection between objects extremely costly. In the context of graph networks, large meshes also lead input graphs that might contain hundreds of thousands of nodes and edges, crippling runtime and memory performance.
To remedy the issue with expensive collision detection, a common approach in the classic simulation literature is to use sisned-distance functions or fields (SDFs). SDFs implicitly represent the object
"Corresponding email: rubanovalgoogle.com
'Co-senior authorship
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Objects represented as SDFs
SDF-Sim
(Signed functionn)
MLP |II-
MLP
timestep
Graph NN
timestep [+1
Figure 1: Overview of SDF-Sim pipeline. SDFs parameterized by MLPs are learned for each object to implicitly represent the object shape and the distance field. A GNN-based simulator uses learned SDFs to predict object dynamics for the next simulation step.
shapes and allow to find the distance and the closest point the object surface from arbitrary location in 3D space in constant-time Since distance queries are a main driver of the compute cost in traditional rigid body simulation, they can be significantly sped up using SDFs. However, in practice, SDFs are frequently pre-computed from a mesh and stored as a 3D grid, which trades off the runtime efficiency for increased memory cost and limits their usefulness for large real scenes.
An orthogonal line of work started to explore SDFs learned from set of images to reconstruct the 3D shape from vision. Those SDFs store the shape implicitly in the MLP weights and are fast to train and query. They are less memory-hunrry compared to 3D grids, making them a perfect candidate for simulation. However, despite these advantages, learned SDFs were applied for dynamics scenes only in a limited context [14,23]
We present SDF-Sim, a learned hroph-network-basee simulator for rigid interactions that uses learned SDFs represent object shapes, Figure Using special design of the input graph, SDF-Sim allows us to substantially reduce the runtime and memory requirements compared to simulators. This is the first demonstration of a learned simulator generalizable to extremely large scenes up to 1.1 million nodes in Figure 2, orders of magnitude larger than what have been shown in any previous work on learned rigid simulation. These simulations include concave shapes (shoes, a hanger) and thin structures (screwdriver, baking form). Finally, we show that SDF-Sim can directly work with SDFs obtained from nulti-view RGB images, Figure 3, supporting rich 3D simulation of objects extracted from real scenes.
learned
2 Background
Mesh-based simulation A simulation can be represented a time series of system states S!...., S" The goal is to learn a neural simulator that predicts the next state St+1 given a history of previous states (S*-h+1 ..... St.. Simulators based graph networks (GNNs) [8, 29] encode the system state into a graph G = (V, E) with nodes V and edges E. For rigid body simulation, this graph can be constructed from the triangle mesh of the individual objects: mesh vertices become the graph nodes, and mesh edges act as graph edges. The object motion is computed by message passing across the nodes and edges in the graph. Within individual objects, the position, rotation and velocity of the object can be computed by message passing through the nodes and edges of that object.
Why learned simulators? Analytical simulators like MuJoCo, PyBullet or Drake, are most com- monly used for modelling rigid bodies. However, the traditional simulators rely on hard-coded approximations of physical interactions that might not match the properties of real objects, even with careful hyperparameter tuning[1]. The predictions from these simulators inevitably diverge from observations of real objects- a so-called sim-to-real gap [17].
Learned simulators have unique advantages that analytical simulators don't 't provide. First, learned simulators can be trained directly on real-world observations. They can track the real object trajectory better than analytical simulators, solving well-known sim-to-real gap[17]. Another common issue precisely estimating the initial states, which analytical simulators rely on learned simulators can compensate for these inaccuracies [3]. Finally, learned simulators are differentiable and can be used for optimization and [2s At the same time, learned simulators are not optimized for runtime and memory, are slower and more centor-con than analytical simulators. In this work, we address that limitation of memory constraint and unlock the ability to run learned simulators on larger scenes in comparison to previous learned simulators.

---

Figure 2: Example of rollouts from SDF-Sim scaled to large simulations, all simulated for 200 steps. (Top) 300 shoes (object from Movi-C), with 851k nodes, falling onto the floor (Middle) 270 knots from Movi-B, 384k nodes (Bottom) 380 objects from Movi-C, 1.1M nodes. See simulation videos com/view/sddf sii.
a VaISDF
b. SDF Sim Rollout
J00
150
Figure 3: Simulating assets extracted from vision. (a) We extract the SDF from the images of real-world scene with a garden table [7]. (b) We simulate a virtual shoe object falling onto a vase and a table using SDF-Sim. SDF-Sim is able predict realistic dynamies, even for the collision of the shoe with the intricate shape of the vase (frames 50-70). See section 4.4 for details and the video on the website.
Modelling collisions in GNN simulators The most challenging component of the simulation is computation of collision impulses between objects. To so, GNN simulators introduce collision edges gooll between nodes [31] or triangles [4] on the mesh surface that belong to different objects and are within predefined distance threshold. However, the amount of these edges is the main bottleneck of GNN-based simulators. Asymptotically, the number of potential collision edges grows quadratically with the number of simulated nodes, leading to prohibitive compute and memory costs. Another challenge is identifying which pairs of triangles/nodes to connect with collision edges in the first place, by computing the distance the closest point on mesh. Typically, this procedure implemented with tree search methods over all mesh triangles in the scene, such BVH [[]], that are difficult to integrate into deep learning pipelines. In this work, we address both of these challenges by using SDFs.
Signed distance functions Signed-distanee functions (SDFs) are widely used in computer graphics, game engines, and robotics for fast collision detection and computation of distances to an object [33, 45]. SDF defines field f(y) R - that represents the signed distance from an arbitrary point y to the closest point on the surface of the object. The sign of the SDF determines whether point outside (positive) or inside (negative) of the object. The zero level of the SDF (y E R { = 0} implicitly represents the object surface. SDFs permit constant-time queries of the distance to an object surface, irrespective of the number of nodes/faces in the object mesh, which is an essential component of collision detection.
3

---

Intra-object edges glatts /
Collision edges between objects Ej
Edge connectivity
Edge features
Surface nodes of the object O₂
mikk
Ti
0 Center node for object O2
Edge in the input graph
* C The closest point on object O to
Object O
computed via SDF
Object O
Object O
Coy +t
[AI I1
(
Figure 4: Construction of graph edges in SDF-Sim.
Constructing an SDF In the computer graphics and simulation literature, SDF is often pre- computed as a high-resolution 3D grid containing signed distances from the points on the grid the object surface [26, 37J. The grid SDF allows to speed up distance queries by trading off memory: for example, 512x5 12x512 grid would take s=14M voxels (0.5Gb of memory) for a single object.
Learned SDFs started to gain popularity for reconstruction of water-tight 3D shapes from images. They approximate the continuous distance field f (y; 8) with an MLP parameterized by 0 [23, 28, 43). Unlike 3D-grid-SDF, learned SDFs are not tied to a fixed grid resolution and can represent detailed shapes using a small of parameters 0. Despite these advantages, learned SDFs have not been sufficiently explored to speed up physical simulations. Limited available works combine learned SDFs with classic physics solvers [5, 33J. They demonstrate that learned SDFs massively reduce the distance query time thanks to parallelization on a GPU compared to mesh-based computation, while taking ~32 less memory than traditional 3D-eoxel voxel-grid SDFs.
Computing closest points For any point in 3D space y an SDF fo allows us easily compute the closest point y" on the object surface that it represents, as:
y = foly)V
where, by definition, fe(y) is the distance between and y", and the gradient fa(y) points in the opposite direction of the shortest path from y to the surface of the object and is unit-norm. If fe(y) parameterized as an MLP, this calculation requires only one forward and one backward pass of the network. This provides the simulation. In this work. for the first time the literature, we use learned SDFs to accelerate distance computation in SOTA graph-network simulators.
efficient way to calculate the closest points for collision resolution in
3 SDF-Sim
We introduce SDF-Sim, netph-networkk-o based simulator for rigid objects that uses learned SDFs to represent object shapes and to perform fast and memory-efficiet distance computation between the objects. By leveraging SDF properties, we propose a new way to construct the input graph for the graph network, allowing us to use a smaller graph size and to get an order-of order-of-magnitude reduction in runtime and memory on large simulations.
3.1 Training SDF functions per object
We represent SDF fo(y) as MLP that takes in 3D point y E R3 and outputs scalar SDF value. Learned SDFs are pre-trained separately for each object and remain fixed throughout the simulation.
Meshes To compare to the existing mesh-based baselines, we apply SDF-Sim on benchmarks where meshes are available and train the learned SDF fe(y) from a mesh. To train an SDF, we sample points in 3D space and compute the ground-truth signed distances from these points to the mesh surfaces using a classic B BVH [11] algorithm. Finally, we train an MLP fe(y) to fit these distances with supervision. See more details in section C. We use the same architecture and the model size for each object shape of 8 MLP layers with 128 units each, unless otherwise stated.
Vision We use VoISDF [43] to distill an SDF from a set of images representing 360-degree view of the outdoor garden scene first described [7] with camera poses estimated via COLMAP [32]. See visualisation in Figure 3 and section C.2 for details.

---

3.2 Learned simulator
Object representation We represent the shape of the object O₂ in the reference pose (centered at zero) as learned SDF function fe.. At simulation step t, rigid transformation T (pf R transforms the object from reference pose the current pose in the simulation. Here, P is an object translation, corresponding to the object's center of mass at timestep I; R. is rotation. The task of the learned simulator is predicting the next-step transformation T for each object. In the following text, we omit the time index t for brevity.
To represent / objects in the input graph, we introduce object nodes =o= -..). located at position p., and a set of surface nodes Vs- {na E O,ii 1..I, = 1..K.) on the surface of the objects, where the number of nodes K; may differ for each object. These surface nodes {na) move with their corresponding object according to the transformation T,'. With a slight abuse of notation, we will refer to {n both as the node entities and their 3D position in the simulation space.
Nodes and edges within the object To construct the graph connectivity within object, we follow an established line of work on learned simulators [4, 19, 29]. We connect surface nodes (n,z] } to their corresponding object node o; using edges & gintra {eo, + na E O:}. Thus, all the information about object motion, e.g. impulses from collision events, is propagated between the nodes via the object node o,. As shown by [19] we can omit the surface edges between the nodes {nx} without loss of accuracy.
In graph networks, nodes and edges are associated with feature vectors that can hold the information about the motion and the relation between the nodes. We follow the approach of [4, 19, 29] to construct a set of nodes and edge features that are necessary for simulation: To construct the node features for surface nodes nıj, we compute the Lnite-difference velocity estimates in the simulation space using a history of the latest three timesteps Va = (n/s n n (-11
n 2²) We set node features to be [Va, ||v II. dh] where di are the constant object parameters: mass, friction and restitution. We use the same procedure for the object nodes {or} using their positions p,. For intra-object edges Eintra, object center as the edge feature co,+n. = [o, nak: lo, n,
displacement vector between the surface node position and the
based inter-object edges. Here we introduce a new way to construct collision edges between the objects by leveraging their SDF representations (Figure 4). We design these edges such that they contain sufficient information to detect collisions, while their number remains linear in the number of nodes. This unlike quadratic number of collision edges in mesh-based simulators.
Consider two objects O, and Oy. For node n. on O,, we directly query the SDF fe, of object O, to get the distance d' from Das the closest point Oj. We note that unlike mesh-based approaches, this is a single test, and we do not need to calculate the distance from n2 to each node/triangle on O₃- Then, if this distance de is within predefined distance threshold D, we connect the surface node na directly to the opposing object node oj and refer to this edge as Co,+ta.- Thus, we define the set of inter-object edges as E = {eo, + VVn E O : fg (n - D). This approach is different from mesh-based simulators [4, 19, 29] that connect pairs of nodes or triangles on the two surfaces.
We construct the features for collision edges eo,+n such that they contain information about potential points of collision. First, we compute the closest point d/2 from n2 E O₂ to the surface of object Oy. To do so, we transform the position of n, into the reference space of O2 using T (n:x). We call an SDF function to get the distance from the node n, to the closest point on Oş as d' fo,(TT (D.))). The closest point on the surface of O, is then computed similarly to Eq. 1:
DiK dig (V (Tx))
(2)
Note that c/s lies on the surface defined by for- but does not have coincide with any surface node and is not part of node set V. Thus, SDFs allow us to test contact between two objects at higher fidelity, without relying on the density of the surface nodes.
Finally, we construct the features for the collision edges as follows: co,e-m |ch D. c/2 Oj: ||o/ n& Ill ||c/ - 1,]]] These features provide information about the relative position of the closest point (a potential collision point) within the object O, as well as its relative location
5

---

to the node Nu on the opposing object. This information is sufficient for the neural network to resolve collisions. Through message-passing over such imput graph, the object node o can collect the information from all the nodes that are within a collision radius relative to the object Oj. The model also has access to the velocity and rotation history for both objects through node features, so is able to infer how fast the node n. is changing its position w.r.t. Oj.
With such construction, SDF-Sim requires significantly fewer edges than mesh-based methods, because a single collision edge can test a node against an entire object surface. Asymptotically, SDF-Sim has O K) edges in the worst case, with the number of objects I being magnitudes smaller than the total number of surface nodes K = E; K;, instead of ( for mesh- based simulators. As we demonstrate below, this choice of imput graph and simulator unlocks the ability to scale to very large scenes that has not been previously shown in the literature.
Graph network simulator We encode the input graph using MLPs for each node and edge type. We process the graph using 10 message passing steps as in [4, 29]. We decode the processed surface node features into acceleration estimate a& for each surface node nak- and compute per-node position update i using Euler integration. Finally, next-step rigid transformations (f are computed from fi using shape matching [25], following [4]. The simulator trained on single-step prediction task using per-node 1.2 loss on the acceleration estimate a,ı-
4 Results
Ground-truth rollout (frrst and last frames)
Predictions (last frame) FIGNer
We start by evaluating the accuracy and efficiency of SDF-Sim on small-scale simulation benchmarks, where evalua- tion of the baselines is feasible as well. Subsequently, we demonstrate the scal- ing properties of the models on scenes with an increasing number of objects. We show that SDF-Sim produces realistic rollouts of extremely large scenes with up to 1.1M nodes, which was not possi- ble with the previous generation of learned simulators. We emphasize that the benefits of SDF-Sim are not specific to simulations with similar object shapes, where separate SDFs are used for each object. Finally, we present ablations for how the quality of the learned SDFs affects the simulation. All simulation videos are available on the website https:/ h/ppss////ttaa..t www.te.ooogllee.....o.oooooooeee cam/ wwt.wcww//v./eeeeeeeeeeee wwwee/adf-ssii
SDF-Sim
Figure 5: Comparison of the last frames of rollouts pre- dicted on Movi-C. See more frames in Figure S9 and on the website.
Kubric datasets We evaluate SDF-Sim on the benchmark Kubric datasets [16] (shared with Apache 2.0 license), consisting of simulated trajectories of rigid objects thrown towards the center of the scene. We evaluate two difficulty tiers: Movi-B and Movi-C. Movi-B simulations contain eleven synthetic shapes (e.g., cube, torus, cow) with up to 142 nodes each. Movi-C comprises 930 scanned real-world household objects, including hollow, flat, or otherwise non-trivial surfaces with thousands of triangles. Both Movi B/C contain only 3-10 objects per simulation. We provide more details about mesh sizes in Movi-B/C in supplement A.
We perform many of our quantitative comparisons on Movi-B/C datasets with small-scale scenes because these domains are small enough for us to run also baseline methods to quantify efficiency and accuracy. On Movi-C we compare only to FIGNet* [19], because other baselines run out of memory during training due to the large number of collision edges, as reported by Allen et al. [4].
Baseline models We compare SDF-Sim to the existing state-of-the-ar learned simulators for rigid- body interactions: FIGNet [4] and FIGNet [19]. Both methods are based on graph networks. Unlike SDF-Sim, they operate directly over object meshes and use special type of graph edges between mesh triangles. Their runtime and memory costs grow with the number and size of the object meshes used for the simulation. FIGNet* is memory-efficien version FIGNet that omits edges between the surface nodes of each object. In their original publication, FIGNet was demonstrated to scale up to 10 objects with a few thousands nodes each [4], while FIGNet was tested up to a larger table scene with 40k triangles, but with only moving object [19].

---

Graph Edges
Poak Memory (MiB)
Runtime per step (sec)
Transiation Errar (meters) Rotation Error (deg)
25000
100
0.3
0.5
15000
60
0.2
0.3
15
0.1
5000
20
0.1
5
Movi-B
Movi-C
Movi-8
Movi-C
Movi-8
Movi-C
Movi-B
Mevi-C
Movi-B
Movi-C
MGN-Large-Radius
MGN
FIGNet
FIGNet
SDF-Sim
Figure 6: Accuracy, memory and runtime comparisons between the SDF-Sim model and the mesh- based baselines on the Movi-B/C benchmarks. On Movi-C, most baselines except FIGNer run of memory and are not shown. As "Peak Memory" we report the peak memory used by the model per single step of the simulation. DPI, BEGN-Large-Raiiee and MGN results were reported by [4]. See Tables 3 and 4 for the exact numbers.
collision edges [10%
edges [10%6]
dynamics per step (sec) time per step
10-
SoiSin
1.0
0.6
0.4
0:5
3
200
420
200
400
200
402
200
402
objects
objects
objects
objects
Figure 7: Large-scale simulation of Spheres-in- step of SDF-Sim rollout on the scene with 512 spheres. Right: number of edges and runtime w.r.t the number of spheres in the simulation (max 512). In complex simulations with lots of contacts, FIGNet and FIGNet generate an excessive number of collision edges, quickly exceeding GPU memory (end of the orange and blue lines). SDF-Sim generates an order-of-magnitude fewer collision edges, and can easily simulate scenes with 100s of objects without running out of memory.
simulated for 200 timesteps. Left: the final
We additionally include previously reported results from Allen et al. [4] on Movi-B for the following models: DPI [18] that represents the objects as a set of disjoint particles; as well as MeshGraphNets (MGN) and MGN-Large-Rade Radius [29] that use inter-object edges between mesh surface nodes. As reported by Allen et al. [4], DPI, MGN and MGN-Large-Ree Radius baselines suffer from a prohibitively large size of the inputs graph and can only
small Movi-B simulations.
4.1 Baseline comparison datasets of small scenes.
First, we evaluate SDF-Sim on Movi-B/C datasets with up to 10 objects. As shown in Figure 6, our SDF-Sim uses substantially sparser graphs represent the scenes, with 28% fewer graph edges compared to FIGNet* on Movi-B and 33% fewer on Movi-C. This translates into reduction of peak memory required execute one step of the simulation: 39% reduction on Movi-B and 42.8% on Movi-C. The reduction in the number of edges in SDF-Sim is enabled by object-to-node collision edges that scale linearly with the number of nodes, as opposed to quadratic number of face-face collision edges in FIGNet and FIGNet" (in the worst case, see section 2). Sparser graphs in SDF-Sim also lead lower average runtime per step of a rollout, by 36% in Movi-B and 43% in Movi-C. because the graph network performs edge updates for/over fewer edges even though the number of nodes in the graph remains the same. Figure S2(a) shows that the runtime of SDF-Sim consistently lower than that of FIGNet for varying sizes of the input graph. In the next section we show how these efficiency improvements translate into order-or order-or-magninne or-manituud gains on large-scale simulations.
In terms of simulation accuracy, measured as object translation and rotation RMSE errors, SDF-Sim has substantially lower errors than previous baselines DPI, MGN and MGN-Large-Radd Radius on Movi-B. However, in comparison to SOTA models FIGNet and FIGNet SDF-Sim has slightly higher error. Note that the errors of SDF-Sim are already very low: the translation error of SDF-Sim is 0.24 meters, which is only 4.9% of the average object travel distance of 4.92 meters in Movi-C dataset (see tables Tables 3 and for exact numbers). In rigid-body systems, even small discrepancies in predicted positions/rotations can lead to a drastically different object trajectory after collision, and we consider 4.9% deviation to good result. Finally, in the next section we will demonstrate that on large scenes SDF-Sim is actually more accurate in comparison to FIGNet*.
7

---

4.2 SDF-Sim scales to scenes with up to 512 objects
To study the scaling properties of the model, we created Spheres-in-Bon -nww:::: set of simulations with a variable number of spheres, ranging from 1 to 512, being dropped into bowl (Fig- ure 7). We create the ground-truth for these scenes using PyBullet, the same simulator used for Kubric Movi- B/C. We evaluate a set of learned simulators: SDF-Sim and FIGNet* models trained on Movi-C, as well as FIGNet trained Movi-B. Note that the largest version, with 512 spheres, has over 50 times more objects than in Movi-B/C datasets used for training.
Average penetration (10-f m)
(8)
Rollout RMSE (m)
0.06
FIGNet
FIGNNet
SDF-Sim tours)
04
0.02
0 00
50
100
150
200
100
150
time step
time step
Figure Accuracy metries w.r.t simulation time step for Spheres-in-Bowl simulation shown in Figure 7. (a) Penetra- tion metrics. (b) Rollout RMSE. Both metrics are averaged over simulation runs with up to 140 spheres, the maximum for which all baselines could be run.
As shown in Figure 7, SDF-Sim has an order-of- maggnniude fewer collision edges, compared to FIGNer and FIGNet. This difference is substantial, as the number of edges dominates the memory cost. FIGNet and FIGNet run out of memory for simulations with >140 and >160 objects, respec- tively, whereas SDF-Sim can simulate the entire set of 512 interacting objects using the same Nvidia V100 GPU. In terms of total runtime, SDF-SinI is up to 5 times faster than FIGNet even including the time for querying learned SDFs. To the best of our knowledge, this is the first demonstration of a learned simulator successfully scaling scenes with hundreds of objects and thousands of collisions, despite heing trained only on ten objects per scene.
Next, we evaluate the accuracy of these simulations. SDF-Sim has the lowest penetration distance throughout the simulation, as shown in Figure 8(a). Figure 8(b) shows the rollout errors across different steps of the simulation. Notably, SDF-Sim generalizes to these larger scenes better than FIGNer and has lower error, despite slightly lower accuracy on datasets with smaller scenes (section 4.1). FIGNet has the lowest error out of learned models, likely because it was trained on Movi-B, which contains a sphere object, while Movi-C dataset, used for training Fignet* and SDF-Sim -Dmm does not. Despite not having seen sphere object in training, SDF-Sim performs very well on both penetration and rollout metrics, indicating that the SDF representation is not only efficient, but also shows good generalization over geometry.
4.3 Scaling to extra-large scenes with up to 1.1 million nodes
Next, we provide a further qualitative demonstration that SDF-Sim can scale to extra-large scenes and produce realistic rollouts. We design three scenes with falling stacks of shoes, metal knots, and mixed objects taken from the Movi-C C dataset, shown in Figure 2. Such contact-rich simulations are challenging even for analytical simulators and classic test examples in the computer graphics literature. These simulations consist of: 300 shoes with 851k nodes, 270 knots with 384k nodes, and 380 various Movi-C objects with 1.IM nodes, respectively. All of these scenes are orders of magnitude larger than those in Movi-C dataset used for SDF-Sim training.
Simulations in Figure 2 show that SDF-Sim can scale to these massive scenes without running out of memory and produce qualitatively realistic rollouts of 200 steps; see videos. Note that we cannot run any other learned baseline on these scenes, as due to the large number of potential contacts, these models produce vast number of collision edges and exceed the GPU memory.
4.4 Simulating real-world scenes from vision
Simulating large real-world scenes is the primary area where the ability to generalize to new, larger scenes crucial, e-g., in virtual reality applications. Here we present a proof-of-concept that SDF-Sim can successfully handle real scenes, despite being trained only on small simulated assets. We take a Garden scene [7], represented by a sequence of RGB images and use VolSDF [43] to distill a 3D SDF representation of the table and the vase from this scene. SDF-Sim allows to cleanly integrate this 3D representation into a simulation, where it would interact with another object. Finally, we add shoe object from Movi-C to this scene. Figure shows rollout from the SDF-Sim model of a shoe
8

---

(4)
SDF-Sim
(b)
5DF-Sim
(c)
(d)
(e)
0.20
Transiation Error
Rotation Error
SDF Error
Example Learned SDFs
Heatmap (128)
Error [m)
Error (m)
128
15.0
02
10.0
30
0.05
-2
32
-4
Layer
5DF
Size
Figure 9: Ablation learned SDF model sizes: 8 layer MLPs with layer sizes of 32, 64, 128 hidden units. (a. b) Translation and rotation error for SDF-Sim trained on Movi-B with different SDF sizes. (c) Mean squared error of the predicted SDF estimates near the surface. (d) Visualisations of the cow shape from Movi-B with different SDF sizes. (e) A cross-section of the learned SDF field for the Movi-B cow shape.
falling on top of the vase and the table SDF extracted by VoISDE With its over 80k nodes, this mesh exceeds the limits of FIGNet baseline that runs out of memory. In contrast, SDF-Sim capture the nuanced interactions between the objects, which is particularly evident in frames 50 to 70; see the video on the website. This experiment demonstrates that SDF-Sim can generalize new scenes, despite being trained on synthetic data of small scenes.
4.5 Ablation of learned SDF quality
We investigate the quality of the estimated SDF values and their impact on simulator performance. We train SDF using MLPs with 32, 64 and 128 units per layer, and 8 layers total. Figure 9C)) shows that larger SDF models help to improve the quality of estimated SDFs. Overall, the learned SDF models of all three layer sizes can reproduce the shape, although the reconstruction from smallest 32-umit model is more coarse (Figure (d), see Figure S8 for more examples). However, Figure (a, shows that SDF-Sim accuracy is similar across different SDF MLP sizes, suggesting that the learned simulator can learn to compensate for noise in the estimated SDFs. We provide additional metries on SDF accuracy, SDF gradients and SDF projected closest points in section D.2. We also provide the comparison of memory footprint of SDF versus storing meshes in section D.5.
Related Work
Dynamics over learned SDFs Overall, using implicit shapes, e.g., learned SDFs, for simulation has been explored only in limited settings. DiftSDFSim [35] and DANO [12] use learned object shapes (SDFs and NeRFs respectively) in combination with analytic contact model on scene with single object. Argila et al. [5] and Qiao et al. [30] model the interaction of learned SDF with cloth via an analytic simulator. In the space of learned simulators, FIGNer [19]
CNNNbased model on 3D scene reconstructed via NeRF and converted into a mesh. Driess et al. [15] learn latent dynamics over NeRFs. Whitney al. [41] convert a NeRF 3D scene into set of particles and use them in GNN simulator. Unlike these pipelines, SDF-Sim avoids the costly simulation of large meshes or sets of particles and operates directly over the SDFs. Mani et al. [23] use learned SDFs to represent large rigid scenes for the sake of efficiency and use the SDF values directly in the GNN simulator. However, this model is designed to use only single SDF per scene. In contrast, we focus on more challenging interactions of many rigid objects, that is much more prevalent in real applications.
Learning SDFs from vision Learned SDFs started to gain popularity for reconstructing an 3D surfaces from a sequence of images. Unlike NeRFs [7], SDFs are more likely to represent closed, watertight, smooth surfaces. Wang et al. [39], Yariv et al. [43], Yu et al. [44] train an MLP to represent an SDF of a single object. Other works train a single generative model for many SDF shapes, allowing to amortise the training cost, generate new shapes and perform shape completion from partial inputs [9, 10, 28, 34]. Recent works also offer ways to use SDFs for deformable objects [42], articulated objects [24], to edit SDFs [38].
One can also reconstruct SDFs from other modalities, e.g., [24, 40] demonstrate how to reconstruct
unseen object SDF using RGB-D video from single camera view. SDFs can be

---

recovered for large room-scale scenes [6] or even in real time as the camera moves through the space [27], or from noisy point clouds, e.g. from Lidar [21, 22]. We note that although we can leverage the SDFs extracted by these methods, these works pursue a different goal and focus only on accurate reconstruction of the 3D surface and do not aim to use these SDFs for simulation.
6 Discussion and Limitations
We introduce a rigid-body simulator, SDF-Sim, that uses learned SDFs to represent the object shapes and employs graph network to learn dynamics over these objects. We re-design the construction of the imput graph for the simulator to make use of SDFs and reduce the number of inter-object edges from quadratic linear, unlocking the ability to scale to large scenes. We provide detailed study on both small-scale benchmarks and large scenes with up to 50x more objects than seen in the training. Additionally, we investigate the impact of SDF quality on the simulation results. Our work demonstrates, for the first time in the learning simulation literature, that we can use these simulators on scenes with hundreds of objects and up to 1.1 million nodes and produce realistic rollouts, potentially unlocking applications to virtual reality, film, robotics, and more.
Limitations One limitation of SDF-Sim is that it requires training SDF MLP for every new shape we want to include in a simulation. A promising future direction is to train one model on a dataset of many shapes, amortising the cost of training SDFs, similarly to [28, 34J. We also note that SDF-Sim has slightly higher error on small-scale datasets (section 4.1), although SDF-Sim is closer to the ground-truth on large scenes than FIGNet (section 4.2). We believe it is justifiable trade slight increase in model error in exchange for the ability to run on larger scenes than previously possible. We believe that SDF-Sim is therefore best suited for the applications that favour scaling over physical accuracy, such as animation and film. Finally, in this paper we only focus on rigid-rigid interactions, but SDFs be extended to incorporate deformable articulated objects [24, 42], fluids [23], and even a mix of SDFs, meshes, and particles.
work as a step in making applications that require simulation, such as augmented reality or animation, more accessible to everyday consumers, As such, our work allows directly use 3D assets filmed mobile phone, and simulate large scenes using commodity hardware with a single GPU, without requiring a specialized skill of creating meshes appropriate for simulation. Overall, we SDF-Sim as important step towards learning physically realistic dynamics models from real-world data, and for enabling physical reasoning over complex, large-scale physical scenes.
Broader impact We
Acknowledgements
The authors would like to thank Ladislav Rampášek for immense help on proof-reading the paper and revising the notation for the final paper submission.
References
[1] Brian Acosta, William Yang, and Michael Posa. Validating robotics simulators real-world impacts. IEEE Robotics and Automation Letters, 7:6471-6478 2021. URL https: //api.
semanticscholar arggCorpusIDD:. :t:p:/pppuI:::22211223333 2
[2] Kelsey Allen, Tatiana Lopez-Guevara, Kimberly L. Stachenfeld, Alvaro Sanchez Gonzalez, Peter Battaglia, Jessica B Hamrick, and Tobias Pfaff. Inverse design for fluid-structure interactions using graph network simulators. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 13759- 13774. Curran Associates, Inc., 2022. 2
[3] Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can lear discontinuous, rigid contact dynamics. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1157-1167. PMIR, 14-18 Dec 2023. URL https::/ htaps::/pproceeingg mlr. press/v205 allen23a. html. 2
10

---

[4] Kelsey R. Allen, Yulia Rubanova, Tatiana Lopez- Guevara, William Whitney, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. In International Conference on Learning Representations, 2023. 1, 3, 5, 6, 7, 14, 15,
[5] Hugo Bertiche Argila, Meysam Madadi, and Sergio Escalera. Neural implicit surfaces for efficient and accurate collisions in physically based simulations. ArXiv, abs/21 10.01614, 2021. URL. https: //ppi. semanticschol. org/ CorpusID 238354360. 4, 9
[6] Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 6290-6301, June 2022. 10
[7] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 3, 4. 8, 9, 16
[8] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Matcusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1r0iv: 01261, 2018.
[9] Yen-Chi Cheng, Hsin-Ying Ying Lee, Sergey Tulyakov, Alexander Schwing, and Liang- Yan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pages 4456-4465. 2023.
[10] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. 2023. 9
[11] James H. Clark. ACM, 19(10):547-554, oct 1976. ISSN 0001-0782. doi: 10.1145/36339:3360354 URL
Hierarchical geometric models for visible surface algorithms. Commun.
360354. attps://doi.orge: 1.....1111 1145/360349 4444346049.336334 3, 4, 16
[12] Simon Le Cleac'h, Hong Yu, Michelle Guo, Taylor A. Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamics- augmented neural objects. JEEE Robotics and Automation Letters, 8:2780-2787, 2022. URL
https://api. semanticschol orgooog///ooo Corpps 2522996791..
[13] Erwin Coumans. Bullet physics simulation. ACM SIGGRAPH 2015 Courses, page 7, 2015, 1
[14] Danny Driess, Jung-Su Ha, Marc Toussaint, and Russ Tedrake. Learning models as functionals of signed-distance fields for manipulation planning. In 5th Annual Conference on Robot Learning, 2021. URL
[15] Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi- object dynamics with compositional neural radiance fields. In 6th Annual Conference on Robot Learning, 2022. URL https: //tp:/eveppeeeeeeeeeeeeeeee net fest/fooummicom httpiiiddduuuTTyGppniiimm
[16] Klaus Greff al. Kubric: scalable dataset generator. 2022. 6, 14
[17] Peide Huang, Xilun Zhang, Ziang Cao, Shiqi Liu, Mengdi Xu. Wenhao Ding, Jonathan Francis, Bingging Chen, and Ding Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Jie Tan, Marc Toussaint, and Kourosh Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 734-760. PMLR, 06 09 Nov 2023. URL https: httprc/eecccc......//me mlr. wwt.srresssv//// huang23c. html. 2
[18) Yunzhu Li, Jiajun Wu, Jun- Yan 7hu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pages 2205-1211, IEEF, 2019. 7
[19] Tatiana LupezaGu Yulia Rubanova, William F. Whitney, Tobias Pfaff, Kimberly Stachen- feld, and Kelsey R. Allen. Scaling face interaction graph networks to real world scenes, 2024.
1, 5. 6, 9, 14
11

---

[20] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field. pages 347-353. 1998. 17
[21] Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pulll Learning signed distance function from point clouds by learning pull space onto surface. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 7246-7257. PMLR, 18-24 Jul 2021. URL https: http:////pooenneeeeeeeeee mlr press/ hates//v19/mmaall htt22....mm.......m.m.m 10 www.m.ee../mee.........
[22] Baorui Ma, Yu-Shen Liu, and Zhizhong Han, Learning signed distance functions from noisy 3d point clouds via noise to noise mapping. In International Conference on Machine Learning (ICML), 2023. 10
[23] Arjun Mani, Ishaan Preetam Chandratreya, Elliot Creager, Carl Vondrick, and Richard Zemel. Surfsup: Learning fluid simulation for novel surfaces. arXiv preprint arXiv:2303.08/28 2023.
2, 4, 9. 10
[24] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representa- tion. pages 12981-12991 2021. 9, 10
[25] Matthias Müller, Bruno Heidelberger, Matthias Teschner, and Markus Gross. Meshless defor- mations based on shape matching, ACM transactions graphics (TOG), 24()):771-488 2005.
[26] Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Adam Fishman, and Dieter Fox. CabiNet: Scaling neural collision detection for object rearrangement with procedural scene generation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), May 2023. URL https: arxiv. org/abs/ absaab///233// 22304.09302. 4
[27] Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for robot perception. In Robotics: Science and Systems, 2022. 10
(28) Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019. 9, 10
[29] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh- based simulation with graph networks. International Conference on Learning Representations, 2021. 1, 2, 5, 7
[30] Yi-Ling Qiao, Alexander Gao, Yiran Xu, Yue Feng, Jia-Bin Huang, and Ming C. Lin. Dynamic mesh-aware radiance fields. ICCV, 2023.
[31] Alvaro Sanchez-Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning simulate complex physics with graph networks. In International Conference on Machine Learning pages 8459 8468. PMLR, 2020. 3
[32] Johannes L Schonberger and Jan-Michael Frahm. Structure-from motoomn revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[33] Nicholas Sharp and Alec Jacobson. Spelunking the deep: Guaranteed queries on general neural implicit surfaces via range analysis. ACM Trans. Graph., 41(4), jul 2022. ISSN 0730-0301. doi: 10.1145/35222235533555 URL https::///oii orw/o0..//11... 1145/3528223.... 35301555. 3,
[34] Jaehyeok Shim, Changwoo Kang, and Kyungdon Joo. Diffusion-based signed distance fields for 3d shape generation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 20887-20897, June 2023, 9, 10
12

---

[35] Michael Strecke and Joerg Stueckler. Diffsdfsim: Differentiable rigid-body dynamics with implicit shapes. In 3DV, pages 96-105, 2021. URLattps://bb..orgg// 1ttp//000110///33322222
2021. 2021.00020. 9
[36] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[37] Dylan Turpin, Tao Zhong, Shutong Zhang, Guanglei Zhu, Eric Heiden, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Fast-grasp'd d: Dexterous multi-finger grasp generation through differentiable simulation. In ICRA, 2023.
[38] Petros Tzathas, Petros Maragos, and Anastasios Roussos. 3d neural sculpting (3dns); Editing neural signed distance functions, 2023. 9
[39] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021. 9
[40] Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. CVPR, 2023. 9
[41] William F. Whitney, Tatiana Lopez-Guevara, Tobias Pfaff, Yulia Rubanova, Thomas Kipf, Kimberly Stachenfeld, and Kelsey R. Allen. Learning 3d particle-based simulators from rgb-d videos, 2023.
[42] Youngsun Wi, Peter R. Florence, Andy Zeng, and Nima Fazeli. Virdo: resentations of deformable objects. 2022 International Conference on Robotics and Automation (ICRA), pages 3583-3590, 2022. URL https:/ /ttp//appp /api. semanticscholar. wwtpsachooaaaoroo/eeeeeoeoee ID: 246473027. 9, 10
Visio-tactile implicit rep-
[43] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. 4, 8, 9, 16
[44] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2022. 9
[45] Meng Zhang, Duygu Ceylan, Tuanfeng Y. Wang, and Niloy Jyoti Mitra. Dynamic neural garments. ACM Transactions on Graphics (TOG), 40:1 15, 2021. URL //api. semanticscholar org/ wwrg/CorpusID::300 232013515.
[46] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: modern library for 3D data processing, arXiv:1801.0988 2018. 22
13

---

A Datasets and custom simulations
A.1 Kubric dataset and evaluations [16]
We use Movi-B and Movi-C datasets from Kubric [16] for small-scale evaluation and fair comparison to baselines [4] and [19]. Simulations in Movi-B and Movi-C datasets contain between and 10 objects per scene. In all simulations objects are thrown on the floor towards the center of the scene. The simulations consist of 100 time steps, or 2 seconds of the simulation time, See the visualisations of Movi-B shapes in Figure S8 and selected shapes from Movi-C in Figure S7.
In both Movi-B and Movi-C we use 1500 trajectories for training, 100 for validation and 100 for test, as provided in the original Kubric dataset. Kubric dataset is distributed according to the Apache 2.0 license".
# object types
nodes per object max
# triangles per object max
mean
mean
Movi-B Movi-C
11 930
3,483
13,952 84,950
6,938 15,955
28,224
8,982
101,142
Table Sizes of the detailed meshes used to train SDFs.
object types
nodes per object max
# triangles per object
mean
mean
max
Movi-B Movi-C
11
453
1,142
841 1,002
2,160
930
543
1,908
3,324
Table 2: Sizes of the collision meshes used for simulation.
150
150
Count
100
Count
100
50
# nodes
triangles
(a) Detailed meshes used for SDF training
300
300
Count
100
100
0
500
1000
1500
2000
1000
2000
3000
nodes
triangles
(b) Collision meshes used simulation
Figure 10: Distribution of mesh sizes used in Movi-C (930 objects).
A.2 Large scenes and vision
For evaluations large-scale scenes, we create several custom scenes in Blender with large number of objects. For each simulation, we arrange the objects in a stacked formation above the floor, to
14

---

create a large number of collisions. Note that modelling a falling stack of objects is an extremely hard task even for classic simulators due large number of collisions.
Spheres-in-Bowl For evaluations in section 4.2, we create a scene with variable numher of spheres (from to 512) falling into bowl. Spheres objects from Movi-B, with 64 nodes each, resized to 0.52 meters in diameter. A bowl is an object from Movi-C.
Falling shoes (Figure 2 top). We use 300 identical shoe objects from Movi-C. This simulation contains 85lk nodes and 200 frames
Falling knots (Figure 2 middle). We use 270 torus-knots from Movi-B, The resulting simulation has 384k nodes and 200 frames.
Heaps of Stuff (Figure 2 bottom). We use 380 various objects from Movi-C. The scene consists of 1.1M nodes and is simulated for 400 time steps.
B SDF-Sim simulator architecture and training
Architecture Following [4], we use graph network with node and edge updates without global pooling. For node and edge updates, we use MLPs with 2 hidden layers and 128 units each with LayerNorm. GNN consists 10 pesssage-passing steps.
We add a small amount gaussian noise to the inputs in (0 0,003) to help the network to correct inaccuracies from predictions in the previous steps and guarantee stable rollouts.
We normalise both inputs and outputs to zero-mean and unit-variance. The input graph is constructed using history of positions for each node for the three latest time steps [n . nnJ] that convert to a history of two previous velocities [n] in(-1_ n n n The model predicts the acceleration for each node a
Loss We train the model on 1-step prediction task. To construct loss, we estimate ground-truth accelerations via finite difference: a = (n - n (nf n = n 2n + n f-1 We mean-squared error loss the accelerations per node (normalised as mentioned above) train the model. Note that the loss is computed per-node rather than per object.
Generating rollout To generate rollout of multiple steps, we convert the predicted the per-node accelerations a1+11 into the next simulation state (positions and translations of the objects). To do so, we estimate the positions of each node at the next time point niς 1+1 a + 2n n t-1 Finally, we use shape matching, similarly to [4] to estimate the positions and rotation the objects from the node positions.
Metries Following [4], report translation and rotation error between the predicted and ground- truth rollouts of 50 steps. For translation, we report root mean- meean-sqareed error over the object translation of. For rotation, we report the mean angle between the predicted and ground-truth object rotations. Translation vector and rotation quaternions per object are computed from predicted node positions via shape matching, as described above. As in prior work [4], translation and rotation accuracy is reported on rollouts of 50 time steps, while memory and runtime metrics are reported on the full rollouts of 100 steps.
Training time and hardware We train the network for 1M steps on 8 A100 GPUs devices with a batch size of 8. We use Adam optimizer, and exponential learning rate decay from le-3 le-4. The approximate training time is 5 days.
'Here use "hat" : notation to denote the quantities that are estimated or predicted as opposed to the ground-trun truth values.
15

---

C Learning SDFs
C.1 Training SDFs from meshes
For our experiments, pre-train the SDFs each object that will be used in the simulation. For SDF training, we use the meshes in the canonical pose centered around the [0,0,0] point, rescaled so that the maximum vertex coordinate is 0.5. Thus, the entire object mesh is located within [-0.5, 0.5] interval for each axis.
Sampling query points We sample query points y on and near the surface of the mesh to use for SDF training. First, take the mesh nodes V/ and sample more points on the surface of the mesh uniformly using smmplepoinnae wtt.-pooooooooootttttttoo per sq. meter of the object surface area. Then, we add gaussian noise with a 0.1 to the surface points. We use both surface points and noisy points as query points for training. We re-sample the noisy points every 100 steps of training.
function in open3D library. We sample 10,000 points
Computing ground-truth SDF We compute the ground-truth SDF d to the object mesh for all query points. First, for each query point y, we use classic Boundary Vector Hierarchy (BVH) [11] method to find the closest triangle on the object mesh and then calculate the closest point y" on the triangle. The ground-truth distance from the point y to the mesh is ||y - y"Il. Finally, we use the normal N f of the closest triangle (pointing outside of the object) to estimate the SDF sign. We compute the dot product N, (y y"). If the dot product positive, the point is outside the object and the SDF should have positive sign, and if it is negative, it is inside the object and its SDF should have a negative sign. The ground-truth SDF is therefore |ly - y"ll multiplied by the sign.
Training SDF We train the SDF function fo(y) by supervising it using the ground-truc truth SDF d. We use the L2 loss on the signed distances (d od--fol [y)? as well as the L2 loss on the absolute distances (ld) - Iff (y) The reason that we found that sign estimation can be unstable if vector (y -y) is close to being orthogonal to the triangle normal vector n,, Therefore, we use the loss unsigned distances on all query points, but the loss on the signed distances is only used on the points where the sign can be reliably estimated. Specifically, we use signed-distanee loss on the points where the angle between (y - y") and n is less than 75 degrees or more than 105 degrees.
SDF architecture Unless otherwise stated, the Iearned SDFs consisted of MLPs with 8 layers and 128 hidden units. The models were trained for 400K steps and a learning rate of 10 5 for Kubric Meshes. For the Garden Vase scene, MLPs with 8 layers and 256 units were used, but training details were otherwise the same. For quantitative results in Figure 6, we run the all the model with 3 seeds and report the mean and the 95% confidence interval.
SDF training time and hardware For all meshes in Movi-C, trained the MLPs for 400k iterations, which requires 18 hours of training time on average on a single V100 GPU. It is likely that this training time could be reduced, SDFs in this work. Instead, we chose to train the SDF to maximize the quality for complex Movi-C meshes that have 8,982 nodes and 15,955 triangles on average.
did not aim minimize the computational cost of the
Other approaches tested In NeRF literature common regularisation train SDFs is the Eikonal regularisation that enforces unit-norm of the SDF gradient. However, we found that Eikonal regulari- sation leads to over-smoothed sphere-like shapes and did not use it in our training. Furthermore, we find that our approach of directly fitting the distances already results in mait-norm--radicn and does not require an additional regularisation term. We also tested another common technique of periodic encodings as the MLP inputs, however, we found that it leads to checkerboard -ikke artifacts for points outside of the object and did not use this method.
C.2 Training SDFs from vision
For an experiment in section 4.4, we take a Garden Scene with a 360-degree view of the scene. We train a VoISDF model [43] to distill an SDF from this scene. VoISDF leverages a volumetric representation that explicitly encodes the volume density as
from [7]. represented as set of 2D images
16

---

transformed SDF. This explicit formulation provides an inductive bias over the geometry of an object and built-in preference for capturing smooth and clean shapes.
VoISDF already represents a learned SDF and, in principle, can be directly used in SDF-Sim. However, simplify the pipeline and have consistent SDF architecture for all objects, we train another learned SDE, as described in Section C. To do so, we extract a high quality mesh (with over 80k vertices) of the table and the vase from VoISDF using the Marching Cubes algorithm [20]. Then train learned SDF with the architecture described in section C and use it in the SDF-Sim.
17

---

D Additional results
D.1
Quantitative Results on small benchmark datasets.
Movi-B
Model
Translation RMSE (m)
Rotation Err (deg)
DPI
0.368 1 0.057
26.928 1: 2.740
MGr-LargeRadius
0.460 1 0.045
26.342 1.397
MGN
0.538 0.035
26,914 0.783
FIGNet
0.14 = 0.0087
15.0 = 0.675
FIGNet
0.13 + 0.008
15.94 = 0.8849
SDF-Sim (ours)
0.16 = 0.0124
18.03 = 0.9433
Model
Collision Edges
# Graph Edges
Peak Memory (MiB)
Runtime per step (ms)
DPI
2250.688 65
0.145 0.009
Marg-aargRasius
1797.985
0,218 0.033
MGN
34.367 - 4
0.175 + 0.018
FIGNet
1360.99 + 196
24569.59 + 938
94,45 + 2.9698
0.23 + 0.0044
FIGNer
1409.87 + 267
8613.05 + 512
63.83 + 3.7489
0.22 + 0.0077
SDF-Sim (ours)
201.14 +
6195.58 + 193
38.8 0.5711
0.14 0.004
Table 3: Quantitative comparison of SDF-Sim and the baselines a Movi-B dataset. This table
presents the numbers for the Figure in the main text. The error bars show the 95% confidence interval from running all models with 3 random seeds. DPI, MGN-LargeRading and MGN results are reported by [4]. Results marked "-" were not reported by the previous work [4]
Movi-C
Model
Translation RMSE (m)
Rotation Err (deg)
DPI- APt-Reimplemennte* MGN-Larged Larg-LargRRddus*
OOM
0OM
MGN
0OM
FIGNet
0OM
FIGNet"
0.18 + 0.0097
19.68 0.6483
SDF-Sim (ours)
0.24 1: 0.0151
23.15 + 0.668
Model
# Collision Edges
Graph Edges
Peak Memory (MiB)
Runtime per step (ms)
DPI-Reimplemented
0OM
MGN LargeRadius*
0OM
M
0OM
FIGNet
OOM
FIGNet*
2220.96 469
11502. + 944
75.87 + 9.587
0.30 + 0.0163
SDF-Sim (ours)
300.46 34
7661.17 266
43.35 0.7466
0.17 0.0038
Table 4: Quantitative comparison of SDF-Sim and the baselines Movi-C dataset. This table presents the numbers the Figure the main text. The error bars show the 95% confidence interval from running all models with 3 random seeds. DPI, MGN-LargeRadius and MGN and FIGNet baselines run OOM on the Movi-C simulations and are not reported.
18

---

D.2 Additional metrics on SDF quality
Projection MSE
SDF MSE
(0)
SDF Grad MSE
10
109
MSE
MSE
10"
MSE
10-4
10"7
128
64
129
32
Layer
Projection Err v. Dist
(e)
SDF Err v. Dist
(
Gradient Err Dist
0.20
0.005
0.10
0.18
0.09
(9)
0.16
0.14
Grddert Erer
0.07
0.12
Sortror
0.003
0.10
0.05
0.08
0.002
0.06
0.03
0.002
0.02
0.01
-02
0.2
0.0
0.4
Signed Distance
Signed
Signed (m)
Figure S1: Additional metrics on SDF quality
(a-c) Metrics computed across learned SDF models of different sizes for different objects in Movi-B. Each black marker denotes different Movi-B B shape. For each metric, loss decreases increase the size of the model.
(a) MSE between the projected surface points predicted from the learned SDFs and the true closest
point on the surface.
(b) MSE between the signed distance outputted by the learned SDF and the true signed distance.
(c) MSE between the gradient of the learned SDF and the gradient of the ground truth SDF function.
(d-f) SDF evaluation metrics as a function of distance from the object surface. SDF error, projection error and the gradient errors remain constant up distance 0.2 from the object surface and start to increase afterwards. Generally all errors remain low in comparison to the object size of 0.5 meter.
(d) Error as distance between the learned SDF projected surface point and the true closest surface point, plotted as function of signed distance from the surface.
(e) Error as the absolute difference in magnitude between the learned SDF prediction and ground truth SDF, plotted function of signed distance from the surface.
(f) Error as the cosine distance between the gradient of learned SDF and the ground truth SDF gradient.
19

---

D.3 Runtime versus number of nodes in simulation graph.
We measure the time to execute one step of simulation as a function of the input graph on different scenes in Movi-C dataset. For both SDF-Sim and FIGNet*, the runtime grows linearly with the number of the nodes in the graph. The runtime of SDF-Sim is consistently lower than FIGNet On large meshes with ~8000 nodes and ~ 18000 peak edges, SDF-Sim has 52% smaller runtime compared to the FIGNet* baseline.
Runtime per step (sec)
0.8
FIGNet
Runtime (sec)
SDF-Sim (ours)
0.6
0.4
0.2
2000
4000
6000
# Nodes per graph
Figure S2: Runtime per simulation step w.r.t. the total number of nodes in the scene. The runtime evaluations were performed the Movi-C test Nvidia A100 GPU. FIGNet baseline 00M on Movi-C and is not shown here.
D.4 Runtime comparison to Bullet simulator Spheres-in-Bowl.
We compare the runtime of SDF-Sim, FIGNet and FIGNet to the analytical simulator Bullet. SDF- Sim is substantially faster in comparison FIGNet and FIGNet and is similar Bullet up scenes with around 30 objects. Note that Bullet simulator is highly optimized for efficiency, and we do not expect learned simulators match it in terms of runtime.
Total runtime per step [s]
FiGNet
FIGNet*
1.5
Bullet
Runtime
1.0
0.0
objests
Runtime comparisons, measured per-step Spheres-in-Bowl. X axis shows the number of spheres in Spheres-in -n-wo... SDF-Sim has similar runtime to Bullet on scenes up to 30 objects.
Figure S3:
D.5 Memory of the SDF versus mesh
key advantage learned SDFs that they provide a compact representation of complex object shapes compared to meshes. In Figure $4, we compare the sizes of meshes from Movi-C dataset (red) to the sizes of the SDF model parameters (purple) in MiB, assuming both are stored as pickle files on disk. While the size of the meshes grows with the number of mesh nodes, memory requirements of the SDFs do not depend on the complexity of the object, as both simple and intricate shapes are modeled with the same size MLP. We see that our MLP with 8 layers of 128 units has similar memory demands to meshes with 15,000 nodes, which is 4x less memory than the largest meshes in Movi-C. Thus, SDF-Sim can potentially use less memory to store the object shapes: while FIGNet and FIGNet require to store the entire object mesh SDF-Sim needs only a compact SDF and a set surface nodes, which, will show in section D.7, can be relatively small.
20

---

1e6
Mesh file size bytes
2.0
Mesh Size
SDF Size
1.5
15k nodes
0.0
20000
40000
60000
80000
mesh nodes
Figure S4: Memory footprint of storing the mesh versus storing the SDF weights w.r.t the number mesh nodes, evaluated on 930 objects from Movi-C.
D.6
Penetration and error comparison Bullet
Average penetration (10
Rollout RMSE (m)
FIGNet
RIGNet"
0.05
PIGNet"
5DF-5im
Bullet perturbed
0.04
Bullet-perturedd
0.03
0.02
0.01
0.00
100 150
25
50
100
125
150
175
200
step
Lime slep
Figure S5: Penetration and Rollout RMSE on Spheres-In-Bow simulation. Bullet simulation is used as ground-truth. This plot extends Figure with an additional Bullet-pertureed baseline, where we add tiny amount of gaussian noise to the initial states in Bullet simulation. (Left) SDF-Sim has smaller penetrations than Bullet, presumably because Bullet is optimized for speed instead. (Right) Both SDF-Sim and Bullet-perturbed eventually diverge from the original simulation, and the error of SDF-Sim higher than Bullet-pente perturbbe (as expected), but lower than FIGNet
21

---

D.7 Re-sampling object surface nodes using an SDF
Simulation accuracy heavily depends on the quality of the object shapes, typically mesh. However, not all meshes are suitable for simulation. Object meshes, either created by artists or obtained by scanning a real-world object, often consist of thousands of nodes, and the nodes are concentrated around on the fine details of the shape (an example shown in Figure S6(a)). To be able to simulate the objects, the shapes are often approximated by hand-crafted convex collision meshes (Figure S6(b)). But such meshes might not always exist, and might not be optimal for GNN-based simulators like FIGNet. However, mesh-based simulators like FIGNet, typically use these collision meshes, as changing a mesh while retaining a connected surface is not easy.
In contrast, SDF-Sim does not require a connected mesh; it relies only on surface nodes and the learned SDF to perform simulation step. This gives us full flexibility to choose the number and the location of graph nodes. Varying the number of surface nodes gives us a tighter handle controlling the tradeoff between accuracy, simulation speed and memory.
To create a new set of surface nodes, we can sample the surface nodes from a learned SDF. We initialize K = No equally-spaced points in the volume of cube which encompasses the object. We project these points to the object surface using Eq. I, with minor filtering (more details on this sampling procedure described below). With this strategy, we get points that evenly cover the entire object surface (Figure S6(c)). Finally, we take the SDF-Sim simulator trained on Movi-C and replace each object's 's mesh nodes with the ones sampled from an SDF at test time. Figure 6 shows that with this sampling strategy, we can maintain the simulation accuracy using 73.5% fewer nodes (hence smaller memory) compared SDF-Sim with Movi-C collision meshes, used elsewhere in the paper. Specifically, in Figure S6(d) the models with K- = 7 and K = 8 (dark brown bars) have similar translation error (0.23) compared to the model with original node distribution (green bar, 0.24). However, in Figure S6(e) the model K = 7 requires only 934.19 nodes per scene on average (fourth orange column), while the model with the original Movi-C nodes needs 3530.13 nodes (green column) to perform the same simulation.
Here we perform this experiment test time only, but it would be straightforward to also train the SDF-Sim with the nodes sampled from an SDF, and experiment with sampling strategies for optimal prediction performance. This creates the opportunity to use a learned SDF as the sole representation of shape without ever requiring a mesh.
Details on sampling the surface nodes from an SDF We start with a uniform 3D grid of NxNxN points, located within an interval [-0.7, 0.7]. to ensure that the grid is larger than the object (recall that the object meshes were rescaled to [-0.5, 0.5] before training an SDF). Placing the query points on grid ensures that the corresponding projected points would cover the entire object surface, including concave parts like inner side of a shoe or a knot. Then, we project these points on the surface: y" = y fe (y) V fo (y). We generally find that the trained SDF functions are fairly accurate and the projected points are close to the surface. Nevertheless, we additionally filter the points fo(y") 0.01 ensure that yd are exactly the surface as it otherwise would impact the accuracy of the simulation. Finally, some areas of the shape might have clusters of points. We additionally downsample projected points using voxel_ down_ noon_ssmplee function from open3D library [46], such that at most one point is present per voxel of size 0.025 to further reduce the number of points.
(d)
Translation RMSE
# Nodes per Scene
(a) Detailed Mesh
(b) Simulation Mesh K=759
(c) Sampled from SDF
3000
2000
K=21285
K=676
0.2
1000
0.1
Kod
Kega
MoviC
Kmug
Km5
Kegs
Ka
Kega
C
Sampled from SDF
Sampled from 5DF
Figure S6: Shoe object from Movi-C, with vertices shown green: (a) the original high- mesh for rendering, (b) collision mesh for simulation, (c) nodes sampled on a grid and projected to the learned SDF). We can achieve favorable tradeoffs between error (d) and node count (e) by varying the dimensions K of the sample grid (orange bars). The green bar represents SDF-Sim that uses the nodes from the original Movi-C collision mesh as used elsewhere in the paper.
22

---

D.8 Quantifying the source of SDF-Sim error
natural question to ask is, where does the increased error (in comparison FIGNet*) of SDF-Sim on Movi B/C comes from. The distances predicted by learned SDFs may not be perfectly accurate, because they come from learned model. To quantify this, we compute the distances using brute- force distance computation between query points and each triangle in the mesh. Then we train the simulator using these distances instead of learned SDFs. is tricky to estimate the SDF sign using this approach (whether the point is inside/outside of the mesh), so we use the unsigned distances.
We ran an experiment where we train SDF-Sim on Movi-C with estimates of the unsigned distance computed from the mesh via BVH. We found that using learned SDFs versus pre-computed distances makes little difference in practice. We achieve similar Translation RMSE on Movi-C (0.23 with mesh-based UDFs versus 0.24 with learned SDF). While it is not one-to-one comparison, this result does indicate that our method relatively robust to SDF accuracy, and the learned simulator can learn to correct for minor inaccuracies.
23

---

E Learned SDF visualisations
(a) Learned SDF reconstructions
(b) Ground truth meshes
Figure S7: (a) Learned SDF reconstructions for randomly chosen Movi-C shapes. For visualisation purposes, convert the object surface defined by leamee SDF: {y : fe(y) 0} into mesh using Marching Cubes and show this mesh here. (b) Corresponding ground -ruth meshes.
24

---

E.1 SDF shapes through training
As the SDF network has to encode the distances only for single object, we find that a simple MLP is sufficient to encode complex shapes to the necessary extent for the simulation. Figure S8(a) demonstrates that the most accurate shapes are at 400k iterations of training, however already after 4000 training steps the network is able to capture the outlines of most shapes. For simple shapes like cones or cubes, a model learns an accurate shape already by 8000 training iterations. Shapes with holes (like torus) can also be learned within the first 8000 iterations. Although in this paper we chose to train the SDFs for 400k iterations with a small learning rate for best results, these visualisations suggest that faster training for fewer iterations might be sufficient to use for simulation.
Leamed SDFS scmos training
Leamed for stees
4,000
40,000
Layer
Figure S8: (a) Learned SDFs for Movi-B visualized different points during training. (b) Learned SDFs for Movi-B visualized for different MLP sizes (different number of units per layer, all MLPs have 8 layers). For visualisation purposes in both figures, we convert the object surface defined by leamed SDF: {y: : fy = 0} into a mesh using Marching Cubes.
25

---

Rollout Examples
Ground truth
FIGNet*
SDF-Sim
Ground truth
FIGNet*
SDF-Sim
Figure S9: Rollout comparisons between baselines and SDF-Sim on Kubric Movi-C.
26

---

Ground truth
FIGNet
SDF-Sim
Ground
truth
FIGNet*
SDF-Sim
Figure S10: Rollout comparisons between baselines and SDF-Sim on Kubric Movi-C.
27

---

Ground truth
FIGNet
FIGNet
SDF-Sim
Ground truth
FIGNet
FIGNet*
SDF-Sim
Figure S11: Rollout comparisons between baselines and SDF-Sim on Kubric Movi-B.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction reflect the scope of the paper and are supported by the experimental results.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes explicit Limitations sections and states the assumptions (c.g. access to 3D meshes). The paper provides detailed investigations on scaling and efficiency.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not contain theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper introduces new model architecture. We provide the detailed description of the architecture, sufficient to reproduce the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We provide detailed description of the method sufficient to reproduce it. Open-source release of the code is not possible for this work.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the training details, data splits, type of optimizer, etc.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: Yes]
Justification:
We report the confidence intervals over multiple random seeds for our experi- ments.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Question:
29

---

Answer:
[Yes]
Justification: We provide the information about the memory, runtime and hardware for our experiments.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://neurppsppp http:pp///.orrrs...pcaaiae//eeee httppp//pllle/eiiiccccdedde..e..
Answer: [Yes]
Justification: The authors reviewed the Code of Ethics and confirm that this work follows the code.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yss]
Justification: We provide an overview of the potential societal impact, although do not foresee our work on physical simulation to have adversarial consequences.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: To best judgement, the presented approach (scalable 3D simulation) presents no risks for misuse.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the datasets and the models used as baselines are properly credited and the terms of use respected.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Current paper does not use human participants and does not require an approval.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Current paper does not use human participants and does not require an approval.
30


Reviews:
[{"review_id": "Hw9jwOeeiP", "reviewer": "Reviewer_mwTT", "summary": "This paper presents a neural network-based simulator of rigid body dynamics with contacts that is memory-efficient and thus scalable to scenes with hundreds of objects and up to 1.1 million nodes on a single GPU. The key idea is to use SDF as the geometry representation to simplify collision detection compared with mesh representations.", "questions": "I have a question about the surface nodes. In the paper, the authors didn't explain how the surface nodes are generated. From Appendix E.5, it seems like the surface nodes used in the experiments in the paper are the mesh vertices used for collision simulation. Is it correct? In E.5, using surface nodes as samples on the SDF can achieve similar accuracy with much fewer nodes. Why isn't it the default design in the main paper? Is there a trade-off?", "limitations": "The authors adequately addressed the limitations. The authors and the reviewer did not foresee the potential negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper addressed an important and interesting problem, which is an efficient end-to-end simulator of rigid body dynamics. It has significant potential value in various fields, including robotics, AR, etc. Replacing mesh representations with SDFs allows the framework to largely reduce the computational complexity when modeling the collisions between objects. This key idea has been clearly conveyed and experiments validated this claim. \n\nSection 2 provided a nice context that educates readers about the basic knowledge required to understand the framework of this work. The methodology presentation in Section 3 is clear, compact, and easy to follow. The experiments laid out the performance of the proposed method from small-scale scenes to significant larger-scale scenes, which is beneficial for readers to appreciate the merits of this work. \n\nThe authors provided extensive details in the appendix about the implementation of the geometric representation, network modeling, training, and evaluation, which made the whole paper much more informative and improved reader's understanding.", "weaknesses": "I found weaknesses mainly in the experiment section, mainly in the limited explanation and interpretation of the experimental results. \n\nFirst, I found the wording \"small-scale datasets\" and \"small datasets\" misleading. The conventional interpretation of a \"small-scale dataset\" refers to the size of the training data, while in this paper, I believe it actually means the simulation scene is of a relatively small scale, i.e., involving a limited number of objects and nodes. It is a core concept in the experiment section, and the potential gap between the interpretations could lead readers to totally different takeaway messages and create inconsistency between the claim and the evidence. According to Appendix B, 1500 trajectories are used for training in both Movi-B and Movi-C datasets, which reveals the actual scale of the training data. \n\nOn top of this point, there isn't enough discussion about why the proposed method performs slightly worse than the baseline methods on datasets with small scenes and outperforms in larger scenes. What is the difference between smaller scenes and larger scenes for these models? It is a key finding in this paper, but the discussion is largely omitted. \n\nAppendix E.6 seems to try to answer this question, but I didn't quite understand this paragraph. What do you mean by \" train an SDF-Sim architecture using the accurate distances directly computed from a mesh\"? I thought it was exactly what was done in the paper.  Why do an experiment with unsigned signed distance? I think this section created more confusion. \n\nA minor issue is in Appendix E.4. and Figure S3. The figure shows the baseline in purple (blue to me) and SDF-sim in red (orange to me), while the text says the opposite. Please make them consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highly appreciating the value of the paper and for valuable comments.\n\n**The wording \"small-scale datasets\"**\n\nIndeed, here we mean the datasets have small-scale scenes with up to 10 objects, while later in the paper we scale the simulator to scenes with hundreds of objects. We will update the wording in the paper.\n\n**What is the difference between smaller scenes and larger scenes?**\n\nThe key difference is that having an order of magnitude more objects in large scenes leads to many more collisions. We specifically design large scenes to have stacked objects, as this setup is considered to be a particularly hard problem in rigid simulation. It requires resolving many collisions within the same time step and propagating the collision response across a chain of multiple objects. In contrast, in small scenes, the collisions are sparse and involve only pairs of objects. \n\nWe suspect that FIGNet* and FIGNet may have some slight accuracy advantage in pairwise collisions due to the explicit computation over the mesh. SDF-Sim is better at propagating those collisions across many objects and therefore has better performance on more challenging scenes with stacked objects.\n\n**Appendix E.6: What do you mean by \"train an SDF-Sim architecture using the accurate distances directly computed from a mesh\"? Why do an experiment with unsigned signed distance?**\n\nThank you for pointing out the confusion – we will update the section to make it clearer.\n\nThe distances predicted by learned SDFs may not be perfectly accurate, because they come from a learned model. In Appendix E.6 we aim to test whether this is an issue for the simulator. To do so, we compute the distances using a brute-force distance computation between query points and each triangle in the mesh. Then we train the simulator using these distances instead of learned SDFs. It is tricky to estimate the SDF sign using this approach (whether the point is inside/outside of the mesh), so we use the unsigned distances. We found that using learned SDFs versus pre-computed distances makes little difference in practice.\n\n**Surface nodes … are the mesh vertices used for collision simulation? In Appendix E.5 surface nodes are samples from the SDF. Why isn't it the default design? Is there a trade-off?**\n\nThis is correct – for most experiments in the paper we used mesh vertices from the original collision meshes as surface nodes for SDF-Sim. The only exception is the experiment in section E.5, where the nodes were sampled from an SDF. \n\nIndeed, we have shown that we can potentially scale the simulator further by re-sampling the nodes from an SDF. However, the tradeoff of node re-sampling is that it introduces an additional step of the pipeline with its own set of hyperparameters (grid size, downsampling, etc.) that might need to be tuned for more complex shapes. In the main paper we chose to focus on a clear message that we can scale the simulator solely by using SDFs and adapting the GNN, as this architecture works universally well for different object shapes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oku6Au2nvn", "reviewer": "Reviewer_SAmo", "summary": "This work tackles the problem of learning how to simulate rigid-body objects, scaling up to very large scenes that may consist in hundreds of objects and around a million mesh vertices. Graph neural network (GNN) baselines were usually used but become intractable for large scene, with high memory and computational costs. To solve this, this paper presents a new method based on the signed distance function (SDF): first, a learnable SDF is fitted to each object, which can be queried quickly and is memory efficient. Then, the SDFs are leveraged to build a graph between the objects without connecting all vertices, reducing the amount of edges needed. The method is evaluated against baselines on different simulation setups. It is noted that the proposed approach does not perform better than SOTA method based on GNN when the scene is small enough for them to be applied to. However, when scaling the scenes up, these baselines fail due to memory constraints, while the proposed method still runs. Some accompanying videos also show realistic qualitative results.", "questions": "1. See Weakness 4., why is that important to build learned simulators in the context of rigid-object simulations? Are the simulators also memory-constrained and that makes them unusable in the large scenes discussed in the work?\n   1. A learnable surrogate model that approximates a simulator can also be useful as a *differentiable* simulation, e.g., in order to optimize the scene or some objects. Could the presented approach be use in such a pipeline?\n2. A more open-ended question: assuming no intersection, all the SDFs are used in their positive domain. Would the proposed method also work for *open surface* objects, using their UDFs (unsigned distance functions)? In practice, as there might be tiny amount of penetration, the SDF can detect it (negative output) but not the UDF. I wonder if the method is resilient to this or may fail.\n\nA minor suggestion: for Fig. 8, adding the standard-deviation or similar metric to the plots can be helpful, as they represent an aggregate of multiple simulations.", "limitations": "Limitations are discussed in the paper: the need to train an SDF MLP per object in the scene, that the current accuracy is slightly below SOTA (when those GNN model can be applied), and is currently limited to rigid-objects only.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Generally, I found the paper easy to follow and the ideas clearly presented.\n2. The new **SDF-based inter-object edges** that project a node of object 1 to object 2 using the object 2's SDF is an interesting idea to reduce edges number and get nearly continuous information about where node 1 is approaching object 2.\n   1. It is also noted that the worst case complexity is reduced from quadratic to linear w.r.t. the number of nodes per object.\n3. Results show a reduction in memory and computational cost regarding baselines. The experiment with growing number of objects (section 4.2) displays clearly that the proposed method scales better than the baselines and allow to simulate more complex cases where the GNNs cannot be run.\n4. The predicted simulation rollouts are qualitatively realistic without intersection, as seen on the provided videos.\n5. The proof-of-concept vision experiment suggests that once trained on clean objects from a dataset, the method could still generalize to \"real\" meshes coming from 3D reconstruction.", "weaknesses": "1. As is noted in the work, the proposed method is still a bit behind the SOTA GNNs in the accuracy of the prediction w.r.t. the ground-truth simulation. Thus, the proposed method is more aimed at larger scenes only where the baselines cannot be run.\n2. Learned SDFs for the objects, while fast to query, still need to be fitted first for each object and its accuracy is limited by the network's capacity.\n   1. In addition, every object's SDF-model need to be pushed on the GPU--or swapped in-and-out. This work seems to use mostly a smaller set of repeatable object, and thus can reuse the same models. This could potentially be solved by weight sharing between models or, as noted in the limitations.\n3. The *vision* experiment, simulating in an environment extracted from a multi-view NeRF reconstruction, is a bit underwhelming. While it is indeed just a proof-of-concept, it uses only a single object in a single scene, which is directly extracted as a rigid environment. I am unsure what is the main point of this result as the pipeline seems to be mostly the same as the previous ones.\n4. There is not much motivation behind needing *learned* simulators. Maybe expending on the limitations of the simulators themselves could make the work stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for positive evaluation of our paper and for insightful comments.\n\n**Learned SDFs ….  accuracy is limited by the network's capacity.**\n\nWe agree that the accuracy of Learned SDFs depends on network capacity. However, we found that even a basic MLP with 8 layers and 32 units per layer is sufficient to achieve <10e-5 error on SDF estimates and perform accurate simulation (Figures 9(a-c)). We did not find the SDF network capacity to be a limitation.\n\n**Every object's SDF-model need to be pushed on the GPU--or swapped in-and-out**\n\nEvaluating each SDF sequentially and swap them in-and-out would be very inefficient and slow.\n\nSDF weights are small, and we keep all object SDF on the GPU throughout the simulation. Even with many different object shapes in a given scene, keeping SDFs in GPU memory is not an issue. To compute the distances, we also evaluate all SDFs in parallel, leveraging the GPU accelerators. As mentioned in Limitations, it is possible to further reduce the memory consumption, if needed, by using amortized models, weight sharing, or faster SDF queries.\n\n**This work seems to use mostly a smaller set of repeatable object**\n\nWe emphasize that SDF-Sim is not specific to using repeatable objects. We provide an example of a large-scale simulation with many different objects in Figure 10 in the original paper (also replicated in the Rebuttal PDF for reviewer’s reference). \n\n\n**Why is it important to build learned simulators in the context of rigid-object simulations? Are the simulators also memory-constrained?**\n\nThis is a great question. The limitation of traditional simulators like MuJoCo or Bullet is that the simulations inevitably diverge from observations of real objects – a so-called sim-to-real gap [1]. The traditional simulators rely on hard-coded approximations of physical interactions that might not match the properties of real objects. It was shown that even with careful parameter tuning, the analytical simulators cannot precisely model a real cube tossed on a table [2]. This is a major problem for robotics, which heavily relies on simulated environments, and a long line of research in robotics is dedicated to mitigate the sim-to-real gap.\n\nIn contrast, learned simulators do not suffer from sim-to-real gap, as they can be directly trained or fine-tuned on real-world data. In fact, learned simulators can be better at tracking real objects than traditional simulators like MuJoCo, Drake or Bullet, even in the low data regime [3]. \n\nAnother advantage of learned simulators is that they are differentiable by nature and can be used for solving inverse problems, such as design or control (as noted in reviewer’s question below).\n\nHowever, learned simulators used to be memory-constrained and work only on small scenes with up to 10 objects – this is exactly the problem we are targeting in the paper.\n\n**A learnable surrogate model …  can also be useful as a differentiable simulation, e.g., in order to optimize the scene or some objects.**\n\nThis is a great point. SDF-Sim is fully differentiable: GNN and learned SDFs are differentiable by  nature of neural networks. We can also pass gradients through the construction of input graph from GNN to the learned SDFs. One can use a differentiable version of SVD for the shape-matching step.\n\nGenerally, GNN-based models can be successfully used as differentiable simulators to optimize an object shape, such as an airplane wing, and can provide more stable gradients than traditional differentiable simulators [4]. We believe that SDF-Sim inherits these properties and can be used  for differentiable simulation in the areas of mechanical design, robotics and more.\n\n**The vision experiment, simulating in an environment extracted from a multi-view NeRF reconstruction, is a bit underwhelming.  I am unsure what is the main point of this result.**\n\nThis experiment is a proof-of-concept that demonstrates 1) SDF-Sim can generalize to real-world object shapes derived from 3D reconstruction, although it was trained on “clean” shapes 2) we can directly connect the VolSDF outputs to SDF-Sim, making a fully-differentiable vision-to-simulation pipeline.\n\nThe alternative approach to simulating this scene would be to convert VolSDF output into a mesh and run a mesh-based simulator. However, generating clean meshes can be tricky and the meshes may need to be cleaned-up before they can be used in the simulator. Additionally, most rigid solvers like Bullet or MuJoCo rely on convex meshes, requiring additional convex mesh decomposition which is often done manually. In contrast, SDF-Sim directly takes SDF as an input, and we can bypass the step of converting SDFs into meshes.  \n\n**Would the proposed method also work for open surface objects, using their UDFs (unsigned distance functions)? Can it detect penetration?**\n\nWe hypothesize that SDF-Sim will work with UDF representations, assuming that UDFs are used for all objects and SDF-Sim is trained on UDF representations. \n\nThe learned simulators generally are able to detect penetrations using unsigned distances. For instance, mesh-based FIGNet baseline uses only the unsigned distances between individual triangles and can successfully reason about penetrations by pooling the information from pairs of faces on the surfaces of the two objects. In the current SDF-SIm paper, we performed an experiment with unsigned distances computed from a mesh (Appendix E.6) and found that the accuracy is similar to our current model with SDF representations.\n\n\n[1] What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery. Huang et al.CoRL 2023\n\n[2] B. Acosta, W. Yang, and M. Posa. Validating robotics simulators on real-world impacts. IEEE\nRobotics and Automation Letters, 2022.\n\n[3] Graph network simulators can learn discontinuous, rigid contact dynamics. Allen et al. CoRL 2023\n\n[4] Inverse Design for Fluid-Structure Interactions using Graph Network Simulators. Allen et al. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqQwJH6zyV", "reviewer": "Reviewer_nWDb", "summary": "In this paper, a learning-based simulator based on graph neural networks (GNNs) is proposed that leverages signed distance function (SDF) shape representations of objects for efficient parallel processing. The GNN operates on a set of nodes and edges where the nodes are defines by the object center of masses and contact points on the object surfaces. Edges are formed between object centers and collision points on other objects within a distance threshold. The SDF is used to query distances to the object shapes and determine approximate collision points using the gradients of the signed distance fields in a single step. The approach facilitates scenes with 100s of objects with millions of sample points for collision checking on a V100 GPU. Results indicate that the method scales to larger scenes than mesh-based learning-based simulator baselines. Comparison to classical analytical simulators like NVidia PhysX, MuJoCo or bullet is not provided.", "questions": "Please address questions raised in paper weaknesses.", "limitations": "The paper discusses several limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach of using SDFs for learning-based simulation is novel and interesting.\n- The results section provides an interesting set of evaluations of scaling, run-time and accuracy performance. Results indicate improvements in scaling performance over the state-of-the-art learning-based methods.\n- The paper is well written and easy to follow.", "weaknesses": "- l. 182ff, why not differentiate through the euler integration and shape matching steps and take these processing steps into account for the learning approach? Are these steps differentiable? \n- The paper should also compare with classical analytical state-of-the-art simulators such as NVidia PhysX, MuJoCo, bullet. SDFs can be converted into mesh representations for these simulators. How does SDF-Sim relate to these approaches in performance?\n- The run-time evaluation is unclear. The plots in Fig. 7 suggest that the simulator requires several seconds per time step. What is the simulated time per time step ? Is the simulator real-time capable? At which number of objects / collision sample nodes does the method loose real-time property?\n- l. 207, please provide a reference for FIGNet* on first occurence.\n- It is unclear if the edge pruning in FIGNet* is fairly adjusted to these datasets for the comparison with the proposed method. How does FIGNet* perform with variations in the pruning parameters? Can a similar distance threshold be used as in SDF-Sim ? How does the performance tradeoff change with such parameter variation in Fig. 7 ? Can FIGNet* outperform SDF-Sim overall ?\n- Sec. 4.4. how do classical analytical simulators such as PhysX, MuJoCo, bullet perform on this \"large real-world scene\" when transforming the extracted scene SDF to a mesh? \n- l. 314, DANO [10] uses NeRF representation of object shape, not SDF.\n- l. 327, the requirement of watertight objects can be a disadvantage too, e.g., if the objects are thin. Please discuss.\n- The broader impact argues that extracting a mesh for simulation from VolSDF reconstruction would be a specialized skill. This is questionable, as a simple MarchingCubes reconstruction might be sufficient to extract such a mesh.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We will add the discussions below into the paper. We will also correct the reference to FIGNet* and DANO description in the related work. \n\n**Are euler integration and shape matching differentiable?**\n\nBoth Euler integration and shape-matching (using differentiable SVD) are differentiable, and we could potentially include those steps into the learning process. However, we suspect that the current loss on the nodes before shape-matching provides a fine-grained learning signal and forces the model to correct errors for each node individually.\n\n**Edge pruning in FIGNet$\\*$ is fairly adjusted? Influence on Fig 7? Can FIGNet$\\*$ outperform SDF-Sim overall?**\n\nThe edge pruning has one main parameter, which is the collision radius. We use the same collision radius of 0.1 for FIGNet, FIGNet* and SDF-Sim, matching the collision radius used in FIGNet and FIGNet* papers.\n\nIf we change the collision radius, we expect the relation between the models in Figure 7 to stay the same, because FIGNet* constructs O(K^2) edges within the collision region, while SDF-Sim has only a linear number of edges O(K), where K is the number of nodes. Given such asymptotic complexity, we do not think FIGNet* can outperform SDF-Sim.\n\n**Is the simulator real-time capable?**\n\nWe perform the simulation for 200 time steps, 5 seconds of total simulation time. \n\nWe have added a runtime comparison to PyBullet in the attached PDF (Figure 2). SDF-Sim has a similar runtime to PyBullet up to ~30 objects per scene. Generally, learned simulators are not real-time, except on very small scenes. \n \nWe note that optimizing our code for speed was not our goal. MuJoCo or Bullet use low-dimensional collision meshes to achieve real-time, while all of our large simulation scenes use relatively detailed node sets. Additionally, a large part of SDF-Sim runtime is spent on constructing the edges of the input graph, which can be sped up by faster edge pruning, batching SDF queries or using specialized cuda kernels.\n\n\n**Comparison to the analytical simulators such as NVidia PhysX, MuJoCo, bullet**\n\nThis is an interesting question. \nNote that we use Bullet simulations as the ‘ground-truth’ in experiments with Movi B/C (Figure 6) and Spheres-in-Bowl (Figure 8). As shown in Figure 6, SDF-Sim remains consistent with the Bullet simulation, having low translation and rotation error after 50 rollout steps.\n\nIn the attached PDF, we also added *new* comparisons to Bullet in terms of runtime, penetrations and simulation stability (Figures 2 and 3).\n\nWe note that it was not our goal to outcompete the state-of-the-art analytical simulators in runtime. We develop learned simulators because they have their own unique advantages that analytical simulators don’t provide, specifically for Robotics. The main difference is that learned simulators can be trained directly on real-world observations. They can track the real object trajectory better than analytical simulators, solving a well-known sim-to-real gap [1]. Another common issue is precisely estimating the initial states, which analytical simulators rely on – learned simulators can compensate for these inaccuracies [1]. Finally, learned simulators are differentiable and can be used for optimization and design [2]. At the same time, we agree that learned simulators were not optimized for runtime and are slower than analytical simulators. Therefore, in this paper we chose to only compare to other *learned* simulators.\n\n[1] Graph network simulators can learn discontinuous, rigid contact dynamics. Allen et al. CoRL 2023\n\n[2] Inverse Design for Fluid-Structure Interactions using Graph Network Simulators. Allen et al. NeurIPS 2022.\n\n\n**Sec. 4.4. comparison to PhysX, MuJoCo, bullet on SDFs from vision, where extracted SDF is transformed into a mesh**\n\nWe expect that PhysX, MuJoCo, Bullet to be able to successfully handle the scene, since 80k nodes is feasible for these simulators.\n\nWe are not claiming that SDF-Sim is the only simulator that could do that. Instead, this experiment shows that 1) SDF-Sim can generalize to new scenes, despite being trained on synthetic data 2) We can plug in output from VolSDF directly into SDF-Sim, bypassing the mesh conversion 3) FIGNet already runs OOM on this scene\n\nWe will remove the wording “large” for this section and clarify the goals of this experiment in the paper.\n\n\n**The requirement of watertight objects can be a disadvantage too, if the objects are thin**\n\nModeling thin surfaces, e.g. cloths, is non-trivial, and it is outside of scope for this paper. Here we focus on modeling rigid objects, for which volumetric shapes are often a good approximation.\n\nAlthough SDF-Sim is not designed to represent cloths, other graph-network simulators, e.g. MeshGraphNet [1] are able to do so. One can imagine combining the ideas from SDF-Sim and MeshGraphNet, where rigid objects are represented as an SDF for efficiency, while thin/deformable objects are represented as a meshes.\n\n[1] Learning mesh-based simulation with graph networks. Pfaff et al. ICLR, 2021\n\n\n**A simple MarchingCubes reconstruction might be sufficient to extract a mesh from VolSDF**\n\nExtracting meshes intended for simulation require special handling for two reasons:\n\n1. Naive extraction with Marching Cubes leads to noisy meshes. VolSDF is a learned model and its zero values might not be *exactly* on the surface. Therefore, the meshes produced by MarchingCubes can contain holes or floating faces and might require substantial clean-up before they can be used in the simulator, because analytical simulators rely on the meshes to be watertight to perform inside-outside tests. \n\n2. Analytical simulations need a “collision” mesh consisting of convex subcomponents, requiring an additional convex mesh decomposition which is often done manually.\n\nThese steps are feasible to do, but they require knowledge and tools that a deep learning researcher might not have encountered before.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qo0GVCPNhR", "reviewer": "Reviewer_HA7E", "summary": "This work presents a GNN-based rigid-body simulator augmented with learned signed distance fields (SDFs). The core idea is to connect surface nodes between objects by testing whether their signed distances are within a certain threshold. This way, the resultant graph networks have sparser edges than prior works, enabling larger-scale simulations of rigid bodies with contacts and collisions. The paper evaluated the proposed GNN simulator on synthetic and real-world scenes.", "questions": "I am not sure about one critical technical detail in the method. From what I understand, the collision edges need to be dynamically created and updated during simulation: at each time step, based on the current relative positions of these objects, one checks the vertices from one object in the SDFs of another object. If the distance is smaller than a threshold, collision edges are then created. This process has to be computed on the fly. Could you confirm whether my understanding is correct?\n\nLines 69-71 suggest acceleration structures like BVH often rely on CPU implementations that are difficult to accelerate. BVH trees on GPUs are actually common, well-known, and efficient, e.g., “Fast Parallel Construction of High-Quality Bounding Volume Hierarchies” and “ \"Maximizing Parallelism in the Construction of BVHs, Octrees, and k-d Trees.\" It might be good to tone down the claim here.\n\nConstructing SDFs: Did the paper construct a narrow-band SDF or the full SDF on the whole grid as the training data for the network to fit? Also, did the network encourage unit SDF gradient norm during training?\n\nComputing closest points: Equation 1 is correct in theory. In practice, due to the numerical representation of SDFs, one needs to apply Eqn 1 multiple times before obtaining a fairly accurate closest point on the zero-level set.\n\nSDF-based inter-object edges: I am not sure I get the O(K^2) complexity for mesh-based simulators. Such simulators rarely go over all O(K^2) pairs in a brute-force manner. Instead, they apply broad-phase and narrow-phase collision detections to prune many pairs.", "limitations": "The two large-scale scenes (Spheres-in-Bowl and Shoes) are nice, but both of them have some limitations. For the sphere scene, using a neural SDF is an overkill: collision detections between spheres can be computed in its closed form. For the Shoes, only one SDF is needed. These scenes are visually pleasing, but I don’t think they have touched the “true” large-scale rigid-body simulation with contact for the two reasons above. A “true” large-scale scene would be many rigid bodies with various shapes.\n\nOverall, I think I lean positive. Moderately rewriting certain text should address most of my concerns with the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written. The main idea is very straightforward to follow and easy to implement. Obviously, using SDFs for rigid-body collision detection is not new in physics animation, but marrying it to a GNN rigid-body simulator looks new to me. For this particular problem setting in this paper (many rigid bodies with frequent contact), using SDF to prune the collision edges is very reasonable.\n\nThe large-scale scenes are also nice. While classic numerical simulators can handle these scenes without trouble, I don’t know any existing GNN-based simulators that can solve such big scenes and generate visually plausible results. However, I am not an expert in GNN, so I may have missed some state-of-the-art GNN simulation papers.", "weaknesses": "I have one particular concern with the overall storytelling in the paper. The paper keeps drawing comparisons with mesh-based GNN simulators in its introduction, method description, and experiments, e.g., choosing MeshGraphNet and DPI as baselines. I think the overall storytelling is a bit biased and occasionally misleading, without stating the pros and cons of mesh-based GNNs fairly. Methods like MeshGraphNet and DPI can represent deformable objects and fluids, which are more complicated physical systems than rigid bodies. The proposed method in this paper enjoys the benefits of SDFs in collision handling because it focuses on rigid bodies only. For collisions between deformable bodies like those in MeshGraphNet and DPI, a precomputed SDF is not possible because shapes deform over time. Therefore, comparing this paper with MeshGraphNet and DPI is neither fair nor necessary: one wouldn’t use particle systems like DPI to simulate rigid bodies in the first place. I think the paper needs to provide a much more balanced statement about these methods. At a minimum, it should inform readers that by introducing SDFs in this new GNN simulation framework, it is no longer obvious to resolve deformable solids and fluids with contact that previous mesh-based GNNs can typically handle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback and insightful comments.\n\n**Representation of methods like MeshGraphNet and DPI is misleading. SDF-Sim cannot resolve deformable solids and fluids**\n\nWe fully agree that MeshGraphNet and DPI can handle broader types of physical systems. We are happy to add a more balanced statement to reflect that.\n\nWe would like to clarify that when we draw comparisons mesh-based GNN simulators, we generally refer to FIGNet and FIGNet* as these two models have shown the most complex rigid dynamics so far among the learned models. MeshGraphNet and DPI have shown amazing results in the realm of deformable simulation, however rigid bodies have their own unique challenges. Rigid collisions and contacts are non-smooth and are notoriously hard to derive numerical approximations for. Rigid dynamics is chaotic in nature: tiny errors in the model or in the initial states can quickly accumulate and lead to large deviations in later steps of the object trajectory. Hence, in this work we focus on rigid bodies only.\n\n**A clarification about the construction of the collision edges.**\n\nThe reviewer’s understanding is entirely correct: collision edges are created dynamically at every step of the simulation.\n\n\n**Constructing SDFs: Did the paper construct a narrow-band SDF or the full SDF on the whole grid as the training data for the network to fit?**\n\nWe don’t use SDF grids for training, as the reviewer might be suggesting. Instead, we randomly sample the ‘query’ points around the object by taking the points on the object surface and adding gaussian noise with zero-mean and sigma=0.1. We compute the SDFs for these query points and use them for training learned SDFs. Therefore, learned SDFs are not constrained to be narrow-band, although most of the training points fall within $2 \\sigma = 0.2$ distance from the object surface. Figure S1(d-e) shows that learned SDFs are most accurate within [-0.2, 0.2] distance interval from the surface. In practice, we only care about the accurate SDF estimates within the collision radius=0.1, therefore the current SDF accuracy is sufficient.\n\nWe provide more details on SDF training in Appendix D.1. \n\n**Did the network encourage unit SDF gradient norm during training?**\n\nWe did not find that it is necessary to specifically encourage the unit SDF gradient norm. We see this property naturally emerging during the training.  We think it is due to our training strategy where we randomly sample the query points near the object surface instead of training on grid points.\n\n\n**Computing closest points: Equation 1 is correct in theory. In practice, … one needs to apply Eqn 1 multiple times before obtaining a fairly accurate closest point.**\n\nEmpirically, we find that it is sufficient to apply Eq. 1 *once* to find the closest point on the object surface, thanks to the accurate learned SDFs. We use only one projection step in all our experiments. We invite the reviewer to inspect the Figure S1(a), where we compare the true closest point to the *one-step* projection* from a learned SDF and show that the mean-squared error stays within [1e-4, 1e-3] for different network sizes.\n\n\n**O(K^2) complexity of SDF-based inter-object edges: … simulators rarely go over all O(K^2) pairs in a brute-force manner.**\n\nHere we refer to the number of the constructed edges between the objects. Within the collision region, FIGNet an FIGNet* connect all mesh triangles of one object to all mesh triangles on another object, so the number of edges between the objects is O(K^2). In SDF-Sim we connect nodes of one object to the *center* of another object (within the collision region), so the number of edges grows only linearly in the number of nodes O(K). \n\nFor mesh-based simulators, K^2 edges would be a worst-case scenario in case the collision region is huge and spans the entire scene, which is indeed unlikely to happen.\n\nHowever, we can see that these asymptotic trends clearly manifest in Figure 7. The quadratic number of edges in FIGNet pose a significant problem, leading to 1M collision edges for 160 objects and causing the model to run OOM. SDF-Sim has <100k edges for the same simulation.\n\n\n**Spheres-in-Bowl and Shoes use repeated objects. These scenes … did not represent the “true” large-scale rigid-body simulation with …many rigid bodies with various shapes.**\n\nWe have an example of a large-scale scene with many objects with various shapes. It is shown in Figure 10 and replicated in the attached PDF for reviewer’s reference. The corresponding simulation video is shown on the project webpage (“Heaps of stuff” section). This simulation includes concave shapes (shoes, hanger) and thin structures (screwdriver, baking form). We will highlight it more in the final manuscript. \n\nSpheres-in-Bowl: Although sphere SDFs are trivial, Spheres-in-Bowl is a nice example to study the scaling properties of the models due to a large number of collisions within any given time step. We show other large-scale examples with more complex shapes, such as a pile of shoes (Figure 2), a pile of knots (Figure 10 top) and a mix of different objects (Figure 10 bottom). Finally, the bowl itself does not have an analytical SDF expression and still requires an SDF for collision detection.\n\n\n**Lines 69-71 suggest acceleration structures like BVH often rely on CPU implementations that are difficult to accelerate. BVH trees on GPUs are actually common.**\n\nThank you for pointing this out – we will remove this claim from the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the thoughtful and constructive comments.

We are happy that the reviewers appreciated the contributions of our paper, specifically that the paper *addressed an important and interesting problem … and has significant potential value in various fields, including robotics, AR* (mwTT); *marrying SDFs to a GNN rigid-body simulator is new* (HA7E, nWDb); *SDF-Sim allows to largely reduce the cost of modeling collisions* (mwTT, SAmo); *predicted simulations are qualitatively realistic*  (SAmo); *other existing GNN-based simulators cannot model such large scenes* (HA7E). Reviewer SAmo also pointed out that our vision experiment in section 4.4 as a strength, noting that SDF-Sim could generalize to "real" meshes coming from 3D reconstruction.

**A large-scale simulation with many different object shapes**

We would like to draw reviewers’ attention that we have an example of large-scale simulation with a mix of different object shapes from Movi: “Heaps of stuff” video on the webpage, also shown in Figure 10. This simulation includes concave shapes (shoes, hanger) and thin structures (screwdriver, baking form). We emphasize that the benefits of SDF-Sim are not specific to simulations with similar object shapes. We will bring up this example to the first page in the final manuscript.

**Why learned simulators?**

We additionally describe why it is important to build learned simulators in response to reviewers nWDb and SAmo.

**Additional results**

We have added a PDF to address reviewer's questions:
- Re-iteration of “Heaps of stuff” simulation with a mix of objects (reviewers SAmo, HA7E)
- New results with runtime comparison between SDF-Sim and Bullet on large scenes (reviewer nWDb)
- Comparison of Penetration and Rollout RMSE metrics, with a new comparison to Bullet on large simulations  (reviewer nWDb)



We will update the final manuscript to include reviewers’ suggestions.

Meta-review: This paper describes an approach to significantly improving the scale of simulations that can be successfully addressed by GNN based methods by employing learned Signed Distance Functions which effectively capture the geometry of the objects and lead to accelerated computational performance. The paper was reviewed by a panel of experts who felt that the method was well explained and that the advantages in terms of the scale of the simulations that could be addressed were quite compelling. It was also noted that the approach could provide an avenue to more accurate simulations of physical systems since it could be used to learn appropriate representations from sensor data thus addressing the sim-to-real gap. The provided rebuttal effectively addressed  all of the reviewers comments.
Decision: Accept (oral)