Title: Learning rigid-body simulators over implicit shapes for large-scale scenes and vision
Venue: NeurIPS 2024
Accepted: yes
Keywords: graph networks

learned simulation

physics

rigid body simulation

scaling
Abstract: Simulating large scenes with many rigid objects is crucial for a variety of applications, such as robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simulation parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to hand-designed simulators like MuJoCo and Bullet. They are able to accurately capture dynamics of real objects directly from real-world observations. However, current state-of-the-art learned simulators operate on meshes and scale poorly to scenes with many objects or detailed shapes. Here we present SDF-Sim, the first learned rigid-body simulator designed for scale. We use learned signed-distance functions (SDFs) to represent the object shapes and to speed up distance computation. We design the simulator to leverage SDFs and avoid the fundamental bottleneck of the previous simulators associated with collision detection.
For the first time in literature, we demonstrate that we can scale the GNN-based simulators to scenes with hundreds of objects and up to 1.1 million nodes, where mesh-based approaches run out of memory. Finally, we show that SDF-Sim can be applied to real world scenes by extracting SDFs from multi-view images.

Learning rigid-body simulators over implicit shapes for large-scale scenes and vision

Abstract

Simulating large scenes with many rigid objects is crucial for a variety of applications, such as robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simulation parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to hand-designed simulators like MuJoCo [36] and PyBullet [13]. They are able to accurately capture dynamics of real objects directly from real-world observations. However, current state-of-the-art learned simulators operate on meshes and scale poorly to scenes with many objects or detailed shapes. Here we present SDF-Sim, the first learned rigid-body simulator designed for scale. We use learned signeddistance functions (SDFs) to represent the object shapes and to speed up distance computation. We design the simulator to leverage SDFs and avoid the fundamental bottleneck of the previous simulators associated with collision detection. For the first time in literature, we demonstrate that we can scale the GNN-based simulators to scenes with hundreds of objects and up to 1.1 million nodes, where mesh-based approaches run out of memory. Finally, we show that SDF-Sim can be applied to real world scenes by extracting SDFs from multi-view images.

1 Introduction

Simulating real-world environments, such as a bookshelf with books and decorations or a dinner table with plates and glasses, presents a significant challenge for traditional physics simulators. These simulators require precise 3D shape, location, and corresponding physical parameters of each object, making the simulation of arbitrary scenes a difficult task.

Recently, learned simulators based on graph networks (GNNs) [4, 19, 29] have been introduced as a powerful alternative to traditional hand-designed simulators like MuJoCo [36] or PyBullet [13]. Graph networks can capture a range of complex physical dynamics, learn physical properties directly from real data and generalize to new scenes. However, current GNN-based methods do not scale well to large scenes. Similarly to the traditional simulators, they rely on 3D meshes to describe object shapes. In scenes with a large number of objects or objects with detailed meshes, these scenes might contain thousands of nodes and mesh triangles, making collision detection between objects extremely costly. In the context of graph networks, large meshes also lead to input graphs that might contain hundreds of thousands of nodes and edges, crippling runtime and memory performance.

To remedy the issue with expensive collision detection, a common approach in the classic simulation literature is to use signed-distance functions or fields (SDFs). SDFs implicitly represent the object

cow

SDF-Sim

timestep t+1

timestep t

Graph NN

knot

Objects represented as SDFs

(signed distance functions)

MLP

MLP

shapes and allow to find the distance and the closest point on the object surface from an arbitrary location in 3D space in constant-time. Since distance queries are a main driver of the compute cost in traditional rigid body simulation, they can be significantly sped up using SDFs. However, in practice, SDFs are frequently pre-computed from a mesh and stored as a 3D grid, which trades off the runtime efficiency for increased memory cost and limits their usefulness for large real scenes.

An orthogonal line of work started to explore SDFs learned from a set of images to reconstruct the 3D shape from vision. Those SDFs store the shape implicitly in the MLP weights and are fast to train and query. They are less memory-hungry compared to 3D grids, making them a perfect candidate for simulation. However, despite these advantages, learned SDFs were applied for dynamics scenes only in a limited context [14, 23].

We present SDF-Sim, a learned graph-network-based simulator for rigid interactions that uses learned SDFs to represent object shapes, Figure 1. Using a special design of the input graph, SDF-Sim allows us to substantially reduce the runtime and memory requirements compared to mesh-based learned simulators. This is the first demonstration of a learned simulator generalizable to extremely large scenes up to 1.1 million nodes in Figure 2, orders of magnitude larger than what have been shown in any previous work on learned rigid simulation. These simulations include concave shapes (shoes, a hanger) and thin structures (screwdriver, baking form). Finally, we show that SDF-Sim can directly work with SDFs obtained from multi-view RGB images, Figure 3, supporting rich 3D simulation of objects extracted from real scenes.

2 Background

Mesh-based simulation A simulation can be represented as a time series of system states S 1 , . . . , S T . The goal is to learn a neural simulator that predicts the next state S t +1 given a history of previous states {S t -h +1 , . . . , S t } . Simulators based on graph networks (GNNs) [8, 29] encode the system state into a graph G = ( V , E ) with nodes V and edges E . For rigid body simulation, this graph can be constructed from the triangle mesh of the individual objects: mesh vertices become the graph nodes, and mesh edges act as graph edges. The object motion is computed by message passing across the nodes and edges in the graph. Within individual objects, the position, rotation and velocity of the object can be computed by message passing through the nodes and edges of that object.

Why learned simulators? Analytical simulators like MuJoCo, PyBullet or Drake, are most commonly used for modelling rigid bodies. However, the traditional simulators rely on hard-coded approximations of physical interactions that might not match the properties of real objects, even with careful hyperparameter tuning[1]. The predictions from these simulators inevitably diverge from observations of real objects - a so-called sim-to-real gap [17].

Learned simulators have unique advantages that analytical simulators don't provide. First, learned simulators can be trained directly on real-world observations. They can track the real object trajectory better than analytical simulators, solving a well-known sim-to-real gap[17]. Another common issue is precisely estimating the initial states, which analytical simulators rely on - learned simulators can compensate for these inaccuracies [3]. Finally, learned simulators are differentiable and can be used for optimization and design[2]. At the same time, learned simulators are not optimized for runtime and memory, are slower and more memory-constrained than analytical simulators. In this work, we address that limitation of memory constraint and unlock the ability to run learned simulators on larger scenes in comparison to previous learned simulators.

a. VolSDF

0

100

150

t

50

60

70

Modelling collisions in GNN simulators The most challenging component of the simulation is computation of collision impulses between objects. To do so, GNN simulators introduce collision edges E coll between nodes [31] or triangles [4] on the mesh surface that belong to different objects and are within a predefined distance threshold. However, the amount of these edges is the main bottleneck of GNN-based simulators. Asymptotically, the number of potential collision edges grows quadratically with the number of simulated nodes, leading to prohibitive compute and memory costs. Another challenge is identifying which pairs of triangles/nodes to connect with collision edges in the first place, by computing the distance to the closest point on a mesh. Typically, this procedure is implemented with tree search methods over all mesh triangles in the scene, such as BVH [11], that are difficult to integrate into deep learning pipelines. In this work, we address both of these challenges by using SDFs.

Signed distance functions Signed-distance functions (SDFs) are widely used in computer graphics, game engines, and robotics for fast collision detection and computation of distances to an object [33, 45]. SDF defines a field f ( y ) : R 3 → R that represents the signed distance from an arbitrary point y to the closest point on the surface of the object. The sign of the SDF determines whether a point is outside (positive) or inside (negative) of the object. The zero level of the SDF { y ∈ R 3 | f ( y ) = 0 } implicitly represents the object surface. SDFs permit constant-time queries of the distance to an object surface, irrespective of the number of nodes/faces in the object mesh, which is an essential component of collision detection.

Intra-object

edges

Object

Collision edges between objects

Object

Object

Edge connectivity

Edge features

Surface nodes of the object

Center node for object

The closest point on object       to

computed via SDF

Edge in the input graph

= [   ,||   ||,   , ||   ||]

Constructing an SDF In the computer graphics and simulation literature, SDF is often precomputed as a high-resolution 3D grid containing signed distances from the points on the grid to the object surface [26, 37]. The grid SDF allows to speed up distance queries by trading off memory: for example, 512x512x512 grid would take ≈ 134M voxels (0.5Gb of memory) for a single object.

Learned SDFs started to gain popularity for reconstruction of water-tight 3D shapes from images. They approximate the continuous distance field f ( y ; θ ) with an MLP parameterized by θ [23, 28, 43]. Unlike 3D-grid-SDFs, learned SDFs are not tied to a fixed grid resolution and can represent detailed shapes using a small set of parameters θ . Despite these advantages, learned SDFs have not been sufficiently explored to speed up physical simulations. Limited available works combine learned SDFs with classic physics solvers [5, 33]. They demonstrate that learned SDFs can massively reduce the distance query time thanks to parallelization on a GPU compared to mesh-based computation, while taking ∼ 32x less memory than traditional 3D-voxel-grid SDFs.

Computing closest points For any point in 3D space y an SDF f θ allows us to easily compute the closest point y ∗ on the object surface that it represents, as:

where, by definition, f θ ( y ) is the distance between y and y ∗ , and the gradient ∇ f θ ( y ) points in the opposite direction of the shortest path from y to the surface of the object and is unit-norm. If f θ ( y ) is parameterized as an MLP, this calculation requires only one forward and one backward pass of the network. This provides an efficient way to calculate the closest points for collision resolution in the simulation. In this work, for the first time in the literature, we use learned SDFs to accelerate distance computation in SOTA graph-network simulators.

3 SDF-Sim

We introduce SDF-Sim, a graph-network-based simulator for rigid objects that uses learned SDFs to represent object shapes and to perform fast and memory-efficient distance computation between the objects. By leveraging SDF properties, we propose a new way to construct the input graph for the graph network, allowing us to use a smaller graph size and to get an order-of-magnitude reduction in runtime and memory on large simulations.

3.1 Training SDF functions per object

We represent SDF f θ ( y ) as an MLP that takes in a 3D point y ∈ R 3 and outputs a scalar SDF value. Learned SDFs are pre-trained separately for each object and remain fixed throughout the simulation.

Meshes To compare to the existing mesh-based baselines, we apply SDF-Sim on benchmarks where meshes are available and train the learned SDF f θ ( y ) from a mesh. To train an SDF, we sample points in 3D space and compute the ground-truth signed distances from these points to the mesh surfaces using a classic BVH [11] algorithm. Finally, we train an MLP f θ ( y ) to fit these distances with supervision. See more details in section C. We use the same architecture and the model size for each object shape of 8 MLP layers with 128 units each, unless otherwise stated.

Vision We use VolSDF [43] to distill an SDF from a set of images representing a 360-degree view of the outdoor garden scene first described in [7] with camera poses estimated via COLMAP [32]. See visualisation in Figure 3 and section C.2 for details.

3.2 Learned simulator

Object representation We represent the shape of the object O i in the reference pose (centered at zero) as a learned SDF function f θ i . At a simulation step t , a rigid transformation T t i = ( p t i , R t i ) transforms the object from reference pose to the current pose in the simulation. Here, p t i is an object translation, corresponding to the object's center of mass at timestep t ; R t i is a rotation. The task of the learned simulator is predicting the next-step transformation T t +1 i for each object. In the following text, we omit the time index t for brevity.

To represent I objects in the input graph, we introduce object nodes V O = { o i | i = 1 ..I } , located at position p i , and a set of surface nodes V S = { n ik ∈ O i | i = 1 ..I, k = 1 ..K i } on the surface of the objects, where the number of nodes K i may differ for each object. These surface nodes { n ik } move with their corresponding object according to the transformation T t i . With a slight abuse of notation, we will refer to { n ik } both as the node entities and their 3D position in the simulation space.

Nodes and edges within the object To construct the graph connectivity within an object, we follow an established line of work on learned simulators [4, 19, 29]. We connect surface nodes { n ik } to their corresponding object node o i using edges E intra i = { e o i ↔ n ik | n ik ∈ O i } . Thus, all the information about object motion, e.g., impulses from collision events, is propagated between the nodes via the object node o i . As shown by [19] we can omit the surface edges between the nodes { n ik } without loss of accuracy.

In graph networks, nodes and edges are associated with feature vectors that can hold the information about the motion and the relation between the nodes. We follow the approach of [4, 19, 29] to construct a set of nodes and edge features that are necessary for simulation: To construct the node features for surface nodes n ik , we compute the finite-difference velocity estimates in the simulation space using a history of the latest three timesteps v ik = ( n t ik -n t -1 ik , n t -1 ik -n t -2 ik ) . We set node features to be [ v ik , || v ik || , ϕ i ] where ϕ i are the constant object parameters: mass, friction and restitution. We use the same procedure for the object nodes { o i } using their positions p i . For intra-object edges E intra i , we use displacement vector between the surface node position and the object center as the edge feature e o i ↔ n ik = [ o i -n ik , || o i -n ik || ] .

SDF-based inter-object edges. Here we introduce a new way to construct collision edges between the objects by leveraging their SDF representations (Figure 4). We design these edges such that they contain sufficient information to detect collisions, while their number remains linear in the number of nodes. This is unlike quadratic number of collision edges in mesh-based simulators.

Consider two objects O i and O j . For a node n ik on O i , we directly query the SDF f θ j of object O j to get the distance d j ik from n ik to the closest point on O j . We note that unlike mesh-based approaches, this is a single test, and we do not need to calculate the distance from n ik to each node/triangle on O j . Then, if this distance d j ik is within a predefined distance threshold D , we connect the surface node n ik directly to the opposing object node o j and refer to this edge as e o j ↔ n ik . Thus, we define the set of inter-object edges as E coll ji = { e o j ↔ n ik |∀ n ik ∈ O i : f j ( n ik ) ≤ D} . This approach is different from mesh-based simulators [4, 19, 29] that connect pairs of nodes or triangles on the two surfaces.

We construct the features for collision edges e o j ↔ n ik such that they contain information about potential points of collision. First, we compute the closest point c j ik from n ik ∈ O i to the surface of object O j . To do so, we transform the position of n ik into the reference space of O j using T -1 j ( n ik ) . We call an SDF function to get the distance from the node n ik to the closest point on O j as d j ik = f θ j ( T -1 j ( n ik )) . The closest point on the surface of O j is then computed similarly to Eq. 1:

Note that c j ik lies on the surface defined by f θ j , but does not have to coincide with any surface node and is not part of node set V . Thus, SDFs allow us to test contact between two objects at higher fidelity, without relying on the density of the surface nodes.

Finally, we construct the features for the collision edges as follows: e o j ↔ n k = [ c j ik -n ik , c j ik -o j , || c j ik -n ik || , || c j ik -o j || ] . These features provide information about the relative position of the closest point (a potential collision point) within the object O j as well as its relative location

to the node n ik on the opposing object. This information is sufficient for the neural network to resolve collisions. Through message-passing over such input graph, the object node o j can collect the information from all the nodes that are within a collision radius relative to the object O j . The model also has access to the velocity and rotation history for both objects through node features, so it is able to infer how fast the node n ik is changing its position w.r.t. O j .

With such construction, SDF-Sim requires significantly fewer edges than mesh-based methods, because a single collision edge can test a node against an entire object surface. Asymptotically, SDF-Sim has O ( I · K ) edges in the worst case, with the number of objects I being magnitudes smaller than the total number of surface nodes K = ∑ i K i , instead of O ( K 2 ) for mesh-based simulators. As we demonstrate below, this choice of input graph and simulator unlocks the ability to scale to very large scenes that has not been previously shown in the literature.

Graph network simulator We encode the input graph using MLPs for each node and edge type. We process the graph using 10 message passing steps as in [4, 29]. We decode the processed surface node features into an acceleration estimate a ik for each surface node n ik , and compute a per-node position update ˆ n t +1 ik using Euler integration. Finally, next-step rigid transformations { ˆ T t +1 i } are computed from ˆ n t +1 ik using shape matching [25], following [4]. The simulator is trained on a single-step prediction task using a per-node L2 loss on the acceleration estimate a ik .

4 Results

We start by evaluating the accuracy and efficiency of SDF-Sim on small-scale simulation benchmarks, where evaluation of the baselines is feasible as well. Subsequently, we demonstrate the scaling properties of the models on scenes with an increasing number of objects. We show that SDF-Sim produces realistic rollouts of extremely large scenes with up to 1.1M nodes, which was not possi-

FIGNet*

SDF-Sim

Ground-truth rollout

Predictions (last frame)

(first and last frames)

ble with the previous generation of learned simulators. We emphasize that the benefits of SDF-Sim are not specific to simulations with similar object shapes, where separate SDFs are used for each object. Finally, we present ablations for how the quality of the learned SDFs affects the simulation. All simulation videos are available on the website https://sites.google.com/view/sdf-sim .

Kubric datasets We evaluate SDF-Sim on the benchmark Kubric datasets [16] (shared with Apache 2.0 license), consisting of simulated trajectories of rigid objects thrown towards the center of the scene. We evaluate on two difficulty tiers: Movi-B and Movi-C. Movi-B simulations contain eleven synthetic shapes (e.g., cube, torus, cow) with up to 1142 nodes each. Movi-C comprises 930 scanned real-world household objects, including hollow, flat, or otherwise non-trivial surfaces with thousands of triangles. Both Movi B/C contain only 3-10 objects per simulation. We provide more details about mesh sizes in Movi-B/C in supplement A.1.

We perform many of our quantitative comparisons on Movi-B/C datasets with small-scale scenes because these domains are small enough for us to run also baseline methods to quantify efficiency and accuracy. On Movi-C we compare only to FIGNet* [19], because other baselines run out of memory during training due to the large number of collision edges, as reported by Allen et al. [4].

Baseline models We compare SDF-Sim to the existing state-of-the-art learned simulators for rigidbody interactions: FIGNet [4] and FIGNet* [19]. Both methods are based on graph networks. Unlike SDF-Sim, they operate directly over object meshes and use a special type of graph edges between mesh triangles. Their runtime and memory costs grow with the number and size of the object meshes used for the simulation. FIGNet* is a memory-efficient version of FIGNet that omits edges between the surface nodes of each object. In their original publication, FIGNet was demonstrated to scale up to 10 objects with a few thousands nodes each [4], while FIGNet* was tested up to a larger table scene with 40k triangles, but with only one moving object [19].

Movi-B

Movi-C

5000

15000

25000

Movi-B

Movi-C

20

60

100

Movi-B

Movi-C

0.1

0.2

0.3

Runtime per step (sec)

Movi-B

Movi-C

0.1

0.3

0.5

Translation Error (meters)

Movi-B

Movi-C

5

15

25

35

Rotation Error (deg)

DPI

MGN-Large-Radius

MGN

FIGNet

FIGNet*

SDF-Sim

#

total edges [10^6]

dynamics time per step (sec)

total time per step (sec)

#

objects

#

collision edges [10^6]

#

objects

#

objects

#

objects

We additionally include previously reported results from Allen et al. [4] on Movi-B for the following models: DPI [18] that represents the objects as a set of disjoint particles; as well as MeshGraphNets (MGN) and MGN-Large-Radius [29] that use inter-object edges between mesh surface nodes. As reported by Allen et al. [4], DPI, MGN and MGN-Large-Radius baselines suffer from a prohibitively large size of the inputs graph and can only run on small Movi-B simulations.

4.1 Baseline comparison on datasets of small scenes.

First, we evaluate SDF-Sim on Movi-B/C datasets with up to 10 objects. As shown in Figure 6, our SDF-Sim uses substantially sparser graphs to represent the scenes, with 28% fewer graph edges compared to FIGNet* on Movi-B and 33% fewer on Movi-C. This translates into reduction of peak memory required to execute one step of the simulation; 39% reduction on Movi-B and 42.8% on Movi-C. The reduction in the number of edges in SDF-Sim is enabled by object-to-node collision edges that scale linearly with the number of nodes, as opposed to quadratic number of face-face collision edges in FIGNet and FIGNet* (in the worst case, see section 2). Sparser graphs in SDF-Sim also lead to lower average runtime per step of a rollout, by 36% in Movi-B and 43% in Movi-C, because the graph network performs edge updates for/over fewer edges even though the number of nodes in the graph remains the same. Figure S2(a) shows that the runtime of SDF-Sim is consistently lower than that of FIGNet* for varying sizes of the input graph. In the next section we show how these efficiency improvements translate into order-or-magnitude gains on large-scale simulations.

In terms of simulation accuracy, measured as object translation and rotation RMSE errors, SDF-Sim has substantially lower errors than previous baselines DPI, MGN and MGN-Large-Radius on Movi-B. However, in comparison to SOTA models FIGNet and FIGNet*, SDF-Sim has a slightly higher error. Note that the errors of SDF-Sim are already very low: the translation error of SDF-Sim is 0.24 meters, which is only 4.9% of the average object travel distance of 4.92 meters in Movi-C dataset (see tables Tables 3 and 4 for exact numbers). In rigid-body systems, even small discrepancies in predicted positions/rotations can lead to a drastically different object trajectory after a collision, and we consider 4.9% deviation to be a good result. Finally, in the next section we will demonstrate that on large scenes SDF-Sim is actually more accurate in comparison to FIGNet*.

4.2 SDF-Sim scales to scenes with up to 512 objects

To study the scaling properties of the model, we created Spheres-in-Bowl : a set of simulations with a variable number of spheres, ranging from 1 to 512, being dropped into a bowl (Figure 7). We create the ground-truth for these scenes using PyBullet, the same simulator as used for Kubric MoviB/C. We evaluate a set of learned simulators: SDF-Sim and FIGNet* models trained on Movi-C, as well as FIGNet trained on Movi-B. Note that the largest version, with 512 spheres, has over 50 times more objects than in Movi-B/C datasets used for training.

(b)

(a)

As shown in Figure 7, SDF-Sim has an order-of-magnitude fewer collision edges, compared to FIGNet* and FIGNet. This difference is substantial, as the number of edges dominates the memory cost. FIGNet and FIGNet* run out of memory for simulations with >140 and >160 objects, respectively, whereas SDF-Sim can simulate the entire set of 512 interacting objects using the same Nvidia V100 GPU. In terms of total runtime, SDF-Sim is up to 5 times faster than FIGNet*, even including the time for querying learned SDFs. To the best of our knowledge, this is the first demonstration of a learned simulator successfully scaling to scenes with hundreds of objects and thousands of collisions, despite being trained only on ten objects per scene.

Next, we evaluate the accuracy of these simulations. SDF-Sim has the lowest penetration distance throughout the simulation, as shown in Figure 8(a). Figure 8(b) shows the rollout errors across different steps of the simulation. Notably, SDF-Sim generalizes to these larger scenes better than FIGNet* and has lower error, despite slightly lower accuracy on datasets with smaller scenes (section 4.1). FIGNet has the lowest error out of learned models, likely because it was trained on Movi-B, which contains a sphere object, while Movi-C dataset, used for training Fignet* and SDF-Sim, does not. Despite not having seen a sphere object in training, SDF-Sim performs very well on both penetration and rollout metrics, indicating that the SDF representation is not only efficient, but also shows good generalization over geometry.

4.3 Scaling to extra-large scenes with up to 1.1 million nodes

Next, we provide a further qualitative demonstration that SDF-Sim can scale to extra-large scenes and produce realistic rollouts. We design three scenes with falling stacks of shoes, metal knots, and mixed objects taken from the Movi-C dataset, shown in Figure 2. Such contact-rich simulations are challenging even for analytical simulators and are classic test examples in the computer graphics literature. These simulations consist of: 300 shoes with 851k nodes, 270 knots with 384k nodes, and 380 various Movi-C objects with 1.1M nodes, respectively. All of these scenes are orders of magnitude larger than those in Movi-C dataset used for SDF-Sim training.

Simulations in Figure 2 show that SDF-Sim can scale to these massive scenes without running out of memory and produce qualitatively realistic rollouts of 200 steps; see videos. Note that we cannot run any other learned baseline on these scenes, as due to the large number of potential contacts, these models produce a vast number of collision edges and exceed the GPU memory.

4.4 Simulating real-world scenes from vision

Simulating large real-world scenes is the primary area where the ability to generalize to new, larger scenes is crucial, e.g., in virtual reality applications. Here we present a proof-of-concept that SDF-Sim can successfully handle real scenes, despite being trained only on small simulated assets. We take a Garden scene [7], represented by a sequence of RGB images and use VolSDF [43] to distill a 3D SDF representation of the table and the vase from this scene. SDF-Sim allows to cleanly integrate this 3D representation into a simulation, where it would interact with another object. Finally, we add a shoe object from Movi-C to this scene. Figure 3 shows a rollout from the SDF-Sim model of a shoe

(a)

(b)

(c)

(d)

(e)

falling on top of the vase and the table SDF extracted by VolSDF. With its over 80k nodes, this mesh exceeds the limits of FIGNet baseline that runs out of memory. In contrast, SDF-Sim can capture the nuanced interactions between the objects, which is particularly evident in frames 50 to 70; see the video on the website. This experiment demonstrates that SDF-Sim can generalize to new scenes, despite being trained on synthetic data of small scenes.

4.5 Ablation of learned SDF quality

We investigate the quality of the estimated SDF values and their impact on simulator performance. We train SDF using MLPs with 32, 64 and 128 units per layer, and 8 layers total. Figure 9(c) shows that larger SDF models help to improve the quality of estimated SDFs. Overall, the learned SDF models of all three layer sizes can reproduce the shape, although the reconstruction from a smallest 32-unit model is more coarse (Figure 9(d), see Figure S8 for more examples). However, Figure 9(a,b) shows that SDF-Sim accuracy is similar across different SDF MLP sizes, suggesting that the learned simulator can learn to compensate for noise in the estimated SDFs. We provide additional metrics on SDF accuracy, SDF gradients and SDF projected closest points in section D.2. We also provide the comparison of memory footprint of SDF versus storing meshes in section D.5.

5 Related Work

Dynamics over learned SDFs Overall, using implicit shapes, e.g., learned SDFs, for simulation has been explored only in limited settings. DiffSDFSim [35] and DANO [12] use learned object shapes (SDFs and NeRFs respectively) in combination with an analytic contact model on a scene with a single object. Argila et al. [5] and Qiao et al. [30] model the interaction of a learned SDF with a cloth via an analytic simulator. In the space of learned simulators, FIGNet* [19] uses a GNN-based model on a 3D scene reconstructed via NeRF and converted into a mesh. Driess et al. [15] learn latent dynamics over NeRFs. Whitney et al. [41] convert a NeRF 3D scene into a set of particles and use them in a GNN simulator. Unlike these pipelines, SDF-Sim avoids the costly simulation of large meshes or sets of particles and operates directly over the SDFs. Mani et al. [23] use learned SDFs to represent large rigid scenes for the sake of efficiency and use the SDF values directly in the GNN simulator. However, this model is designed to use only a single SDF per scene. In contrast, we focus on more challenging interactions of many rigid objects, that is much more prevalent in real applications.

Learning SDFs from vision Learned SDFs started to gain popularity for reconstructing an 3D surfaces from a sequence of images. Unlike NeRFs [7], SDFs are more likely to represent closed, watertight, smooth surfaces. Wang et al. [39], Yariv et al. [43], Yu et al. [44] train an MLP to represent an SDF of a single object. Other works train a single generative model for many SDF shapes, allowing to amortise the training cost, generate new shapes and perform shape completion from partial inputs [9, 10, 28, 34]. Recent works also offer ways to use SDFs for deformable objects [42], articulated objects [24], or to edit SDFs [38].

One can also reconstruct SDFs from other modalities, e.g., [24, 40] demonstrate how to reconstruct a new unseen object as an SDF using an RGB-D video from a single camera view. SDFs can be

recovered for large room-scale scenes [6] or even in real time as the camera moves through the space [27], or from noisy point clouds, e.g., from Lidar [21, 22]. We note that although we can leverage the SDFs extracted by these methods, these works pursue a different goal and focus only on accurate reconstruction of the 3D surface and do not aim to use these SDFs for simulation.

6 Discussion and Limitations

We introduce a rigid-body simulator, SDF-Sim, that uses learned SDFs to represent the object shapes and employs a graph network to learn dynamics over these objects. We re-design the construction of the input graph for the simulator to make use of SDFs and reduce the number of inter-object edges from quadratic to linear, unlocking the ability to scale to large scenes. We provide detailed study on both small-scale benchmarks and large scenes with up to 50x more objects than seen in the training. Additionally, we investigate the impact of SDF quality on the simulation results. Our work demonstrates, for the first time in the learning simulation literature, that we can use these simulators on scenes with hundreds of objects and up to 1.1 million nodes and produce realistic rollouts, potentially unlocking applications to virtual reality, film, robotics, and more.

Limitations One limitation of SDF-Sim is that it requires training an SDF MLP for every new shape we want to include in a simulation. A promising future direction is to train one model on a dataset of many shapes, amortising the cost of training SDFs, similarly to [28, 34]. We also note that SDF-Sim has slightly higher error on small-scale datasets (section 4.1), although SDF-Sim is closer to the ground-truth on large scenes than FIGNet* (section 4.2). We believe it is justifiable to trade a slight increase in model error in exchange for the ability to run on larger scenes than previously possible. We believe that SDF-Sim is therefore best suited for the applications that favour scaling over physical accuracy, such as animation and film. Finally, in this paper we only focus on rigid-rigid interactions, but SDFs can be extended to incorporate deformable or articulated objects [24, 42], fluids [23], and even a mix of SDFs, meshes, and particles.

Broader impact We see our work as a step in making applications that require simulation, such as augmented reality or animation, more accessible to everyday consumers. As such, our work allows us to directly use 3D assets filmed on a mobile phone, and simulate large scenes using a commodity hardware with a single GPU, without requiring a specialized skill of creating meshes appropriate for simulation. Overall, we see SDF-Sim as an important step towards learning physically realistic dynamics models from real-world data, and for enabling physical reasoning over complex, large-scale physical scenes.

References

Brian Acosta, William Yang, and Michael Posa. Validating robotics simulators on real-world impacts. IEEE Robotics and Automation Letters , 7:6471-6478, 2021. URL https://api. semanticscholar.org/CorpusID:238253129 . 2

Kelsey Allen, Tatiana Lopez-Guevara, Kimberly L Stachenfeld, Alvaro Sanchez Gonzalez, Peter Battaglia, Jessica B Hamrick, and Tobias Pfaff. Inverse design for fluid-structure interactions using graph network simulators. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 13759-13774. Curran Associates, Inc., 2022. 2

Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro SanchezGonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning , volume 205 of Proceedings of Machine Learning Research , pages 1157-1167. PMLR, 14-18 Dec 2023. URL https://proceedings.mlr. press/v205/allen23a.html . 2

Kelsey R. Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William Whitney, Alvaro SanchezGonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. In International Conference on Learning Representations , 2023. 1, 3, 5, 6, 7, 14, 15, 18

Hugo Bertiche Argila, Meysam Madadi, and Sergio Escalera. Neural implicit surfaces for efficient and accurate collisions in physically based simulations. ArXiv , abs/2110.01614, 2021. URL https://api.semanticscholar.org/CorpusID:238354360 . 4, 9

Dejan Azinovi´ c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6290-6301, June 2022. 10

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. CVPR , 2022. 3, 4, 8, 9, 16

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018. 2

Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4456-4465, 2023. 9

Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance functions. 2023. 9

James H. Clark. Hierarchical geometric models for visible surface algorithms. Commun. ACM , 19(10):547-554, oct 1976. ISSN 0001-0782. doi: 10.1145/360349.360354. URL https://doi.org/10.1145/360349.360354 . 3, 4, 16

Simon Le Cleac'h, Hong Yu, Michelle Guo, Taylor A. Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamicsaugmented neural objects. IEEE Robotics and Automation Letters , 8:2780-2787, 2022. URL https://api.semanticscholar.org/CorpusID:252967901 . 9

Erwin Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses , page 7, 2015. 1

Danny Driess, Jung-Su Ha, Marc Toussaint, and Russ Tedrake. Learning models as functionals of signed-distance fields for manipulation planning. In 5th Annual Conference on Robot Learning , 2021. URL https://openreview.net/forum?id=FS30JeiGG3h . 2

Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multiobject dynamics with compositional neural radiance fields. In 6th Annual Conference on Robot Learning , 2022. URL https://openreview.net/forum?id=qUvTmyGpnm7 . 9

Klaus Greff et al. Kubric: a scalable dataset generator. 2022. 6, 14

Peide Huang, Xilun Zhang, Ziang Cao, Shiqi Liu, Mengdi Xu, Wenhao Ding, Jonathan Francis, Bingqing Chen, and Ding Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Jie Tan, Marc Toussaint, and Kourosh Darvish, editors, Proceedings of The 7th Conference on Robot Learning , volume 229 of Proceedings of Machine Learning Research , pages 734-760. PMLR, 06-09 Nov 2023. URL https://proceedings.mlr.press/v229/ huang23c.html . 2

Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA) , pages 1205-1211. IEEE, 2019. 7

Tatiana Lopez-Guevara, Yulia Rubanova, William F. Whitney, Tobias Pfaff, Kimberly Stachenfeld, and Kelsey R. Allen. Scaling face interaction graph networks to real world scenes, 2024. 1, 5, 6, 9, 14

William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field , pages 347-353. 1998. 17

Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-pull: Learning signed distance function from point clouds by learning to pull space onto surface. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7246-7257. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ma21b.html . 10

Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Learning signed distance functions from noisy 3d point clouds via noise to noise mapping. In International Conference on Machine Learning (ICML) , 2023. 10

Arjun Mani, Ishaan Preetam Chandratreya, Elliot Creager, Carl Vondrick, and Richard Zemel. Surfsup: Learning fluid simulation for novel surfaces. arXiv preprint arXiv:2303.08128 , 2023. 2, 4, 9, 10

Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. pages 12981-12991, 2021. 9, 10

Matthias Müller, Bruno Heidelberger, Matthias Teschner, and Markus Gross. Meshless deformations based on shape matching. ACM transactions on graphics (TOG) , 24(3):471-478, 2005. 6

Adithyavairavan Murali, Arsalan Mousavian, Clemens Eppner, Adam Fishman, and Dieter Fox. CabiNet: Scaling neural collision detection for object rearrangement with procedural scene generation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , May 2023. URL https://arxiv.org/abs/2304.09302 . 4

Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. isdf: Real-time neural signed distance fields for robot perception. In Robotics: Science and Systems , 2022. 10

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 4, 9, 10

Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning meshbased simulation with graph networks. In International Conference on Learning Representations , 2021. 1, 2, 5, 6, 7

Yi-Ling Qiao, Alexander Gao, Yiran Xu, Yue Feng, Jia-Bin Huang, and Ming C. Lin. Dynamic mesh-aware radiance fields. ICCV , 2023. 9

Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning , pages 8459-8468. PMLR, 2020. 3

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4104-4113, 2016. 4

Nicholas Sharp and Alec Jacobson. Spelunking the deep: Guaranteed queries on general neural implicit surfaces via range analysis. ACM Trans. Graph. , 41(4), jul 2022. ISSN 0730-0301. doi: 10.1145/3528223.3530155. URL https://doi.org/10.1145/3528223.3530155 . 3, 4

Jaehyeok Shim, Changwoo Kang, and Kyungdon Joo. Diffusion-based signed distance fields for 3d shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20887-20897, June 2023. 9, 10

Michael Strecke and Joerg Stueckler. Diffsdfsim: Differentiable rigid-body dynamics with implicit shapes. In 3DV , pages 96-105, 2021. URL https://doi.org/10.1109/3DV53792. 2021.00020 . 9

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012. 1

Dylan Turpin, Tao Zhong, Shutong Zhang, Guanglei Zhu, Eric Heiden, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Fast-grasp'd: Dexterous multi-finger grasp generation through differentiable simulation. In ICRA , 2023. 4

Petros Tzathas, Petros Maragos, and Anastasios Roussos. 3d neural sculpting (3dns): Editing neural signed distance functions, 2023. 9

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS , 2021. 9

Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Muller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. BundleSDF: Neural 6-DoF tracking and 3D reconstruction of unknown objects. CVPR , 2023. 9

William F. Whitney, Tatiana Lopez-Guevara, Tobias Pfaff, Yulia Rubanova, Thomas Kipf, Kimberly Stachenfeld, and Kelsey R. Allen. Learning 3d particle-based simulators from rgb-d videos, 2023. 9

Youngsun Wi, Peter R. Florence, Andy Zeng, and Nima Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. 2022 International Conference on Robotics and Automation (ICRA) , pages 3583-3590, 2022. URL https://api.semanticscholar.org/CorpusID: 246473027 . 9, 10

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021. 4, 8, 9, 16

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in Neural Information Processing Systems (NeurIPS) , 2022. 9

Meng Zhang, Duygu Ceylan, Tuanfeng Y. Wang, and Niloy Jyoti Mitra. Dynamic neural garments. ACM Transactions on Graphics (TOG) , 40:1 - 15, 2021. URL https://api. semanticscholar.org/CorpusID:232013515 . 3

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847 , 2018. 22

A Datasets and custom simulations

A.1 Kubric dataset and evaluations [16]

We use Movi-B and Movi-C datasets from Kubric [16] for small-scale evaluation and fair comparison to baselines [4] and [19]. Simulations in Movi-B and Movi-C datasets contain between 3 and 10 objects per scene. In all simulations objects are thrown on the floor towards the center of the scene. The simulations consist of 100 time steps, or 2 seconds of the simulation time. See the visualisations of Movi-B shapes in Figure S8 and selected shapes from Movi-C in Figure S7.

In both Movi-B and Movi-C we use 1500 trajectories for training, 100 for validation and 100 for test, as provided in the original Kubric dataset. Kubric dataset is distributed according to the Apache 2.0 license * .

0

20000 40000 60000 80000

# nodes

0

50

100

150

Count

0

25000

50000

75000 100000

0

50

100

150

Count

(a) Detailed meshes used for SDF training

0

500

1000

1500

2000

# nodes

0

100

200

300

Count

0

1000

2000

3000

0

100

200

300

Count

(b) Collision meshes used for simulation

A.2 Large scenes and vision

For evaluations on large-scale scenes, we create several custom scenes in Blender with large number of objects. For each simulation, we arrange the objects in a stacked formation above the floor, to

create a large number of collisions. Note that modelling a falling stack of objects is an extremely hard task even for classic simulators due to a large number of collisions.

Spheres-in-Bowl . For evaluations in section 4.2, we create a scene with variable number of spheres (from 1 to 512) falling into a bowl. Spheres are objects from Movi-B, with 64 nodes each, resized to 0.52 meters in diameter. A bowl is an object from Movi-C.

Falling shoes (Figure 2 top). We use 300 identical shoe objects from Movi-C. This simulation contains 851k nodes and 200 frames

Falling knots (Figure 2 middle). We use 270 torus-knots from Movi-B. The resulting simulation has 384k nodes and 200 frames.

Heaps of Stuff (Figure 2 bottom). We use 380 various objects from Movi-C . The scene consists of 1.1M nodes and is simulated for 400 time steps.

B SDF-Sim simulator architecture and training

Architecture Following [4], we use a graph network with node and edge updates without global pooling. For node and edge updates, we use MLPs with 2 hidden layers and 128 units each with LayerNorm. GNN consists of 10 message-passing steps.

We add a small amount gaussian noise to the inputs in N (0 , 0 . 003) to help the network to correct inaccuracies from predictions in the previous steps and guarantee stable rollouts.

We normalise both inputs and outputs to zero-mean and unit-variance. The input graph is constructed using a history of positions for each node for the three latest time steps [ n t -2 k , n t -1 k , n t k ] that we convert to a history of two previous velocities [ n t -1 k -n t -2 k , n t k -n t -1 k ] . The model predicts the acceleration for each node ˆ a t +1 k .

Loss We train the model on 1-step prediction task. To construct a loss, we estimate ground-truth accelerations via finite difference: a t +1 k = ( n t +1 k -n t k ) -( n t k -n t -1 k ) = n t +1 k -2 n t k + n t -1 k . We use mean-squared error loss on the accelerations per node (normalised as mentioned above) to train the model. Note that the loss is computed per-node rather than per object.

Generating a rollout To generate a rollout of multiple steps, we convert the predicted the per-node accelerations ˆ a t +1 k † into the next simulation state (positions and translations of the objects). To do so, we estimate the positions of each node at the next time point as ˆ n k t +1 = ˆ a t +1 k +2 n t k -n t -1 k . Finally, we use shape matching, similarly to [4] to estimate the positions and rotation of the objects from the node positions.

Metrics Following [4], we report translation and rotation error between the predicted and groundtruth rollouts of 50 steps. For translation, we report root mean-squared error over the object translation o t i . For rotation, we report the mean angle between the predicted and ground-truth object rotations. Translation vector and rotation quaternions per object are computed from predicted node positions via shape matching, as described above. As in prior work [4], translation and rotation accuracy is reported on rollouts of 50 time steps, while memory and runtime metrics are reported on the full rollouts of 100 steps.

Training time and hardware We train the network for 1M steps on 8 A100 GPUs devices with a batch size of 8. We use Adam optimizer, and an an exponential learning rate decay from 1e-3 to 1e-4. The approximate training time is 5 days.

C Learning SDFs

C.1 Training SDFs from meshes

For our experiments, we pre-train the SDFs on each object that will be used in the simulation. For SDF training, we use the meshes in the canonical pose centered around the [0,0,0] point, rescaled so that the maximum vertex coordinate is 0.5. Thus, the entire object mesh is located within [-0.5, 0.5] interval for each axis.

Sampling query points We sample query points y on and near the surface of the mesh to use for SDF training. First, we take the mesh nodes V S and sample more points on the surface of the mesh uniformly using sample_points_uniformly function in open3D library. We sample 10,000 points per sq. meter of the object surface area. Then, we add gaussian noise with σ = 0 . 1 to the surface points. We use both surface points and noisy points as query points for training. We re-sample the noisy points every 100 steps of training.

Computing ground-truth SDF We compute the ground-truth SDF d to the object mesh for all query points. First, for each query point y , we use a classic Boundary Vector Hierarchy (BVH) [11] method to find the closest triangle on the object mesh and then calculate the closest point y ∗ on the triangle. The ground-truth distance from the point y to the mesh is || y -y ∗ || . Finally, we use the normal N f of the closest triangle (pointing outside of the object) to estimate the SDF sign. We compute the dot product N f · ( y -y ∗ ) . If the dot product is positive, the point is outside the object and the SDF should have a positive sign, and if it is negative, it is inside the object and its SDF should have a negative sign. The ground-truth SDF d is therefore || y -y ∗ || multiplied by the sign.

Training SDF We train the SDF function f θ ( y ) by supervising it using the ground-truth SDF d . We use the L2 loss on the signed distances ( d -f θ ( y )) 2 , as well as the L2 loss on the absolute distances ( | d | - | f θ ( y ) | ) 2 . The reason is that we found that sign estimation can be unstable if vector ( y -y ∗ ) is close to being orthogonal to the triangle normal vector n f . Therefore, we use the loss on unsigned distances on all query points, but the loss on the signed distances is only used on the points where the sign can be reliably estimated. Specifically, we use signed -distance loss on the points where the angle between ( y -y ∗ ) and n f is less than 75 degrees or more than 105 degrees.

SDF architecture Unless otherwise stated, the learned SDFs consisted of MLPs with 8 layers and 128 hidden units. The models were trained for 400K steps and a learning rate of 10 -5 for Kubric Meshes. For the Garden Vase scene, MLPs with 8 layers and 256 units were used, but training details were otherwise the same. For quantitative results in Figure 6, we run the all the model with 3 seeds and report the mean and the 95% confidence interval.

SDF training time and hardware For all meshes in Movi-C, we trained the MLPs for 400k iterations, which requires 18 hours of training time on average on a single V100 GPU. It is likely that this training time could be reduced, as we did not aim to minimize the computational cost of the SDFs in this work. Instead, we chose to train the SDF to maximize the quality for complex Movi-C meshes that have 8,982 nodes and 15,955 triangles on average.

Other approaches tested In NeRF literature a common regularisation to train SDFs is the Eikonal regularisation that enforces unit-norm of the SDF gradient. However, we found that Eikonal regularisation leads to over-smoothed sphere-like shapes and did not use it in our training. Furthermore, we find that our approach of directly fitting the distances already results in unit-norm-gradient and does not require an additional regularisation term. We also tested another common technique of periodic encodings as the MLP inputs, however, we found that it leads to checkerboard-like artifacts for points outside of the object and did not use this method.

C.2 Training SDFs from vision

For an experiment in section 4.4, we take a Garden Scene from [7], represented as a set of 2D images with a 360-degree view of the scene. We train a VolSDF model [43] to distill an SDF from this scene. VolSDF leverages a volumetric representation that explicitly encodes the volume density as a

transformed SDF. This explicit formulation provides an inductive bias over the geometry of an object and a built-in preference for capturing smooth and clean shapes.

VolSDF already represents a learned SDF and, in principle, can be directly used in SDF-Sim. However, to simplify the pipeline and have a consistent SDF architecture for all objects, we train another learned SDF, as described in Section C. To do so, we extract a high quality mesh (with over 80k vertices) of the table and the vase from VolSDF using the Marching Cubes algorithm [20]. Then we train a learned SDF with the architecture described in section C and use it in the SDF-Sim.

D Additional results

D.1 Quantitative Results on small benchmark datasets.

Movi-B

.

Movi-C

.

D.2 Additional metrics on SDF quality

(a)

Projection MSE

(b)

SDF MSE

(c)

10-3

100

10-5

MSE

MSE

10-6

MSE

8

10-4

10-1

10-7

32

64

128

32

64

128

32

64

128

(d)

(e)

(f

0.20

0.005

0.10

0.18

0.09

Projection Error(m)

0.16

0.004

0.08

0.14

SDFError(m)

0.10

0.12

0.003

GradientE

0.05

0.06

0.08

0.002

0.04

0.06-

50.03-

0.04

0.001-

0.02

0.02

0.01

0.00

0.000

0.00

-0.2

0.0

0.20.4

0.2

0.0

0.2

0.4

-0.2

0.00.20.4

GT Signed Distance (m)

GT Signed Distance (m)

GT Signed Distance (m)

(a-c) Metrics computed across learned SDF models of different sizes for different objects in Movi-B. Each black marker denotes a different Movi-B shape. For each metric, loss decreases as we increase the size of the model.

(a) MSE between the projected surface points predicted from the learned SDFs and the true closest point on the surface.

(b) MSE between the signed distance outputted by the learned SDF and the true signed distance.

(c) MSE between the gradient of the learned SDF and the gradient of the ground truth SDF function.

(d-f) SDF evaluation metrics as a function of distance from the object surface . SDF error, projection error and the gradient errors remain constant up to distance 0.2 from the object surface and start to increase afterwards. Generally all errors remain low in comparison to the object size of 0.5 meter.

(d) Error as distance between the learned SDF projected surface point and the true closest surface point, plotted as a function of signed distance from the surface.

(e) Error as the absolute difference in magnitude between the learned SDF prediction and ground truth SDF, plotted as a function of signed distance from the surface.

(f) Error as the cosine distance between the gradient of learned SDF and the ground truth SDF gradient.

D.3 Runtime versus number of nodes in simulation graph.

We measure the time to execute one step of simulation as a function of the input graph on different scenes in Movi-C dataset. For both SDF-Sim and FIGNet*, the runtime grows linearly with the number of the nodes in the graph. The runtime of SDF-Sim is consistently lower than FIGNet*. On large meshes with ∼ 8000 nodes and ∼ 18000 peak edges, SDF-Sim has a 52% smaller runtime compared to the FIGNet* baseline.

2000

4000

6000

# Nodes per graph

0.2

0.4

0.6

0.8

Runtime (sec)

Runtime per step (sec)

FIGNet*

SDF-Sim (ours)

D.4 Runtime comparison to Bullet simulator on Spheres-in-Bowl.

We compare the runtime of SDF-Sim, FIGNet and FIGNet* to the analytical simulator Bullet. SDFSim is substantially faster in comparison to FIGNet and FIGNet* and is similar to Bullet up to scenes with around 30 objects. Note that Bullet simulator is highly optimized for efficiency, and we do not expect learned simulators to match it in terms of runtime.

Total runtime per step [s]

Runtime [s]

Bullet

FIGNet

FIGNet*

SDF-Sim

D.5 Memory of the SDF versus a mesh

A key advantage of learned SDFs is that they provide a compact representation of complex object shapes compared to meshes. In Figure S4, we compare the sizes of meshes from Movi-C dataset (red) to the sizes of the SDF model parameters (purple) in MiB, assuming both are stored as pickle files on disk. While the size of the meshes grows with the number of mesh nodes, memory requirements of the SDFs do not depend on the complexity of the object, as both simple and intricate shapes are modeled with the same size MLP. We see that our MLP with 8 layers of 128 units has similar memory demands to meshes with 15,000 nodes, which is 4x less memory than the largest meshes in Movi-C. Thus, SDF-Sim can potentially use less memory to store the object shapes: while FIGNet and FIGNet* require to store the entire object mesh, SDF-Sim needs only a compact SDF and a set surface nodes, which, as we will show in section D.7, can be relatively small.

~15k nodes

D.6 Penetration and error comparison to Bullet

Bullet-perturbed

FIGNet

FIGNet*

SDF-Sim

Bullet-perturbed

FIGNet

FIGNet*

SDF-Sim

D.7 Re-sampling object surface nodes using an SDF

Simulation accuracy heavily depends on the quality of the object shapes, typically a mesh. However, not all meshes are suitable for simulation. Object meshes, either created by artists or obtained by scanning a real-world object, often consist of thousands of nodes, and the nodes are concentrated around on the fine details of the shape (an example shown in Figure S6(a)). To be able to simulate the objects, the shapes are often approximated by hand-crafted convex collision meshes (Figure S6(b)). But such meshes might not always exist, and might not be optimal for GNN-based simulators like FIGNet. However, mesh-based simulators like FIGNet, typically use these collision meshes, as changing a mesh while retaining a connected surface is not easy.

In contrast, SDF-Sim does not require a connected mesh; it relies only on surface nodes and the learned SDF to perform a simulation step. This gives us full flexibility to choose the number and the location of graph nodes. Varying the number of surface nodes gives us a tighter handle controlling the tradeoff between accuracy, simulation speed and memory.

To create a new set of surface nodes, we can sample the surface nodes from a learned SDF. We initialize K = N 3 equally-spaced points in the volume of a cube which encompasses the object. We project these points to the object surface using Eq. 1, with minor filtering (more details on this sampling procedure described below). With this strategy, we get points that evenly cover the entire object surface (Figure S6(c)). Finally, we take the SDF-Sim simulator trained on Movi-C and replace each object's mesh nodes with the ones sampled from an SDF at test time . Figure 6 shows that with this sampling strategy, we can maintain the simulation accuracy using 73.5% fewer nodes (hence smaller memory) compared to SDF-Sim with Movi-C collision meshes, used elsewhere in the paper. Specifically, in Figure S6(d) the models with K = 7 3 and K = 8 3 (dark brown bars) have similar translation error (0.23) compared to the model with original node distribution (green bar, 0.24). However, in Figure S6(e) the model K = 7 3 requires only 934.19 nodes per scene on average (fourth orange column), while the model with the original Movi-C nodes needs 3530.13 nodes (green column) to perform the same simulation.

Here we perform this experiment at test time only, but it would be straightforward to also train the SDF-Sim with the nodes sampled from an SDF, and experiment with sampling strategies for optimal prediction performance. This creates the opportunity to use a learned SDF as the sole representation of shape without ever requiring a mesh.

Details on sampling the surface nodes from an SDF We start with a uniform 3D grid of NxNxN points, located within an interval [ -0 . 7 , 0 . 7] , to ensure that the grid is larger than the object (recall that the object meshes were rescaled to [ -0 . 5 , 0 . 5] before training an SDF). Placing the query points on a grid ensures that the corresponding projected points would cover the entire object surface, including concave parts like inner side of a shoe or a knot. Then, we project these points on the surface: y ∗ = y -f θ ( y ) ∇ f θ ( y ) . We generally find that the trained SDF functions are fairly accurate and the projected points are close to the surface. Nevertheless, we additionally filter the points f θ ( y ∗ ) < 0 . 01 to ensure that y ∗ are exactly on the surface as it otherwise would impact the accuracy of the simulation. Finally, some areas of the shape might have clusters of points. We additionally downsample projected points using voxel_down_sample function from open3D library [46], such that at most one point is present per voxel of size 0.025 to further reduce the number of points.

(a) Detailed

Mesh

K=21285

(b) Simulation

Mesh

(c) Sampled

from SDF

K=676

(d)

(e)

Sampled from SDF

Sampled from SDF

D.8 Quantifying the source of SDF-Sim error

A natural question to ask is, where does the increased error (in comparison to FIGNet*) of SDF-Sim on Movi B/C comes from. The distances predicted by learned SDFs may not be perfectly accurate, because they come from a learned model. To quantify this, we compute the distances using a bruteforce distance computation between query points and each triangle in the mesh. Then we train the simulator using these distances instead of learned SDFs. It is tricky to estimate the SDF sign using this approach (whether the point is inside/outside of the mesh), so we use the unsigned distances.

We ran an experiment where we train SDF-Sim on Movi-C with estimates of the unsigned distance computed from the mesh via BVH. We found that using learned SDFs versus pre-computed distances makes little difference in practice. We achieve similar Translation RMSE on Movi-C (0.23 with mesh-based UDFs versus 0.24 with learned SDF). While it is not a one-to-one comparison, this result does indicate that our method is relatively robust to SDF accuracy, and the learned simulator can learn to correct for minor inaccuracies.

E Learned SDF visualisations

(a) Learned SDF reconstructions

(b) Ground truth meshes

多

E.1 SDF shapes through training

As the SDF network has to encode the distances only for a single object, we find that a simple MLP is sufficient to encode complex shapes to the necessary extent for the simulation. Figure S8(a) demonstrates that the most accurate shapes are at 400k iterations of training, however already after 4000 training steps the network is able to capture the outlines of most shapes. For simple shapes like cones or cubes, a model learns an accurate shape already by 8000 training iterations. Shapes with holes (like torus) can also be learned within the first 8000 iterations. Although in this paper we chose to train the SDFs for 400k iterations with a small learning rate for best results, these visualisations suggest that faster training for fewer iterations might be sufficient to use for simulation.

Layer size = 32

Layer size = 64

Layer size = 128

t = 4,000

t = 40,000

t = 400,000

Learned SDFs for different model sizes

Learned SDFs across training

(b)

(a)

●

司

F Rollout Examples

Ground

truth

FIGNet*

SDF-Sim

Ground

truth

FIGNet*

SDF-Sim

Ground

truth

FIGNet*

SDF-Sim

Ground

truth

FIGNet*

SDF-Sim

Ground

truth

FIGNet

FIGNet*

SDF-Sim

Ground

truth

FIGNet

FIGNet*

SDF-Sim

O

O

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction reflect the scope of the paper and are supported by the experimental results.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes an explicit Limitations sections and states the assumptions (e.g. access to 3D meshes). The paper provides detailed investigations on scaling and efficiency.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not contain theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper introduces a new model architecture. We provide the detailed description of the architecture, sufficient to reproduce the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We provide a detailed description of the method sufficient to reproduce it. Open-source release of the code is not possible for this work.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the training details, data splits, type of optimizer, etc.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the confidence intervals over multiple random seeds for our experiments.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information about the memory, runtime and hardware for our experiments.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors reviewed the Code of Ethics and confirm that this work follows the code.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide an overview of the potential societal impact, although we do not foresee our work on physical simulation to have adversarial consequences.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: To our best judgement, the presented approach (scalable 3D simulation) presents no risks for misuse.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, the datasets and the models used as baselines are properly credited and the terms of use respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Current paper does not use human participants and does not require an approval.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Current paper does not use human participants and does not require an approval.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a GNN-based rigid-body simulator augmented with learned signed distance fields (SDFs). The core idea is to connect surface nodes between objects by testing whether their signed distances are within a certain threshold. This way, the resultant graph networks have sparser edges than prior works, enabling larger-scale simulations of rigid bodies with contacts and collisions. The paper evaluated the proposed GNN simulator on synthetic and real-world scenes.\n\nSTRENGTHS:\nThe paper is well written. The main idea is very straightforward to follow and easy to implement. Obviously, using SDFs for rigid-body collision detection is not new in physics animation, but marrying it to a GNN rigid-body simulator looks new to me. For this particular problem setting in this paper (many rigid bodies with frequent contact), using SDF to prune the collision edges is very reasonable.\n\nThe large-scale scenes are also nice. While classic numerical simulators can handle these scenes without trouble, I don’t know any existing GNN-based simulators that can solve such big scenes and generate visually plausible results. However, I am not an expert in GNN, so I may have missed some state-of-the-art GNN simulation papers.\n\nWEAKNESSES:\nI have one particular concern with the overall storytelling in the paper. The paper keeps drawing comparisons with mesh-based GNN simulators in its introduction, method description, and experiments, e.g., choosing MeshGraphNet and DPI as baselines. I think the overall storytelling is a bit biased and occasionally misleading, without stating the pros and cons of mesh-based GNNs fairly. Methods like MeshGraphNet and DPI can represent deformable objects and fluids, which are more complicated physical systems than rigid bodies. The proposed method in this paper enjoys the benefits of SDFs in collision handling because it focuses on rigid bodies only. For collisions between deformable bodies like those in MeshGraphNet and DPI, a precomputed SDF is not possible because shapes deform over time. Therefore, comparing this paper with MeshGraphNet and DPI is neither fair nor necessary: one wouldn’t use particle systems like DPI to simulate rigid bodies in the first place. I think the paper needs to provide a much more balanced statement about these methods. At a minimum, it should inform readers that by introducing SDFs in this new GNN simulation framework, it is no longer obvious to resolve deformable solids and fluids with contact that previous mesh-based GNNs can typically handle.\n\nQUESTIONS:\nI am not sure about one critical technical detail in the method. From what I understand, the collision edges need to be dynamically created and updated during simulation: at each time step, based on the current relative positions of these objects, one checks the vertices from one object in the SDFs of another object. If the distance is smaller than a threshold, collision edges are then created. This process has to be computed on the fly. Could you confirm whether my understanding is correct?\n\nLines 69-71 suggest acceleration structures like BVH often rely on CPU implementations that are difficult to accelerate. BVH trees on GPUs are actually common, well-known, and efficient, e.g., “Fast Parallel Construction of High-Quality Bounding Volume Hierarchies” and “ \"Maximizing Parallelism in the Construction of BVHs, Octrees, and k-d Trees.\" It might be good to tone down the claim here.\n\nConstructing SDFs: Did the paper construct a narrow-band SDF or the full SDF on the whole grid as the training data for the network to fit? Also, did the network encourage unit SDF gradient norm during training?\n\nComputing closest points: Equation 1 is correct in theory. In practice, due to the numerical representation of SDFs, one needs to apply Eqn 1 multiple times before obtaining a fairly accurate closest point on the zero-level set.\n\nSDF-based inter-object edges: I am not sure I get the O(K^2) complexity for mesh-based simulators. Such simulators rarely go over all O(K^2) pairs in a brute-force manner. Instead, they apply broad-phase and narrow-phase collision detections to prune many pairs.\n\nLIMITATIONS:\nThe two large-scale scenes (Spheres-in-Bowl and Shoes) are nice, but both of them have some limitations. For the sphere scene, using a neural SDF is an overkill: collision detections between spheres can be computed in its closed form. For the Shoes, only one SDF is needed. These scenes are visually pleasing, but I don’t think they have touched the “true” large-scale rigid-body simulation with contact for the two reasons above. A “true” large-scale scene would be many rigid bodies with various shapes.\n\nOverall, I think I lean positive. Moderately rewriting certain text should address most of my concerns with the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a learning-based simulator based on graph neural networks (GNNs) is proposed that leverages signed distance function (SDF) shape representations of objects for efficient parallel processing. The GNN operates on a set of nodes and edges where the nodes are defines by the object center of masses and contact points on the object surfaces. Edges are formed between object centers and collision points on other objects within a distance threshold. The SDF is used to query distances to the object shapes and determine approximate collision points using the gradients of the signed distance fields in a single step. The approach facilitates scenes with 100s of objects with millions of sample points for collision checking on a V100 GPU. Results indicate that the method scales to larger scenes than mesh-based learning-based simulator baselines. Comparison to classical analytical simulators like NVidia PhysX, MuJoCo or bullet is not provided.\n\nSTRENGTHS:\n- The proposed approach of using SDFs for learning-based simulation is novel and interesting.\n- The results section provides an interesting set of evaluations of scaling, run-time and accuracy performance. Results indicate improvements in scaling performance over the state-of-the-art learning-based methods.\n- The paper is well written and easy to follow.\n\nWEAKNESSES:\n- l. 182ff, why not differentiate through the euler integration and shape matching steps and take these processing steps into account for the learning approach? Are these steps differentiable? \n- The paper should also compare with classical analytical state-of-the-art simulators such as NVidia PhysX, MuJoCo, bullet. SDFs can be converted into mesh representations for these simulators. How does SDF-Sim relate to these approaches in performance?\n- The run-time evaluation is unclear. The plots in Fig. 7 suggest that the simulator requires several seconds per time step. What is the simulated time per time step ? Is the simulator real-time capable? At which number of objects / collision sample nodes does the method loose real-time property?\n- l. 207, please provide a reference for FIGNet* on first occurence.\n- It is unclear if the edge pruning in FIGNet* is fairly adjusted to these datasets for the comparison with the proposed method. How does FIGNet* perform with variations in the pruning parameters? Can a similar distance threshold be used as in SDF-Sim ? How does the performance tradeoff change with such parameter variation in Fig. 7 ? Can FIGNet* outperform SDF-Sim overall ?\n- Sec. 4.4. how do classical analytical simulators such as PhysX, MuJoCo, bullet perform on this \"large real-world scene\" when transforming the extracted scene SDF to a mesh? \n- l. 314, DANO [10] uses NeRF representation of object shape, not SDF.\n- l. 327, the requirement of watertight objects can be a disadvantage too, e.g., if the objects are thin. Please discuss.\n- The broader impact argues that extracting a mesh for simulation from VolSDF reconstruction would be a specialized skill. This is questionable, as a simple MarchingCubes reconstruction might be sufficient to extract such a mesh.\n\nQUESTIONS:\nPlease address questions raised in paper weaknesses.\n\nLIMITATIONS:\nThe paper discusses several limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the problem of learning how to simulate rigid-body objects, scaling up to very large scenes that may consist in hundreds of objects and around a million mesh vertices. Graph neural network (GNN) baselines were usually used but become intractable for large scene, with high memory and computational costs. To solve this, this paper presents a new method based on the signed distance function (SDF): first, a learnable SDF is fitted to each object, which can be queried quickly and is memory efficient. Then, the SDFs are leveraged to build a graph between the objects without connecting all vertices, reducing the amount of edges needed. The method is evaluated against baselines on different simulation setups. It is noted that the proposed approach does not perform better than SOTA method based on GNN when the scene is small enough for them to be applied to. However, when scaling the scenes up, these baselines fail due to memory constraints, while the proposed method still runs. Some accompanying videos also show realistic qualitative results.\n\nSTRENGTHS:\n1. Generally, I found the paper easy to follow and the ideas clearly presented.\n2. The new **SDF-based inter-object edges** that project a node of object 1 to object 2 using the object 2's SDF is an interesting idea to reduce edges number and get nearly continuous information about where node 1 is approaching object 2.\n   1. It is also noted that the worst case complexity is reduced from quadratic to linear w.r.t. the number of nodes per object.\n3. Results show a reduction in memory and computational cost regarding baselines. The experiment with growing number of objects (section 4.2) displays clearly that the proposed method scales better than the baselines and allow to simulate more complex cases where the GNNs cannot be run.\n4. The predicted simulation rollouts are qualitatively realistic without intersection, as seen on the provided videos.\n5. The proof-of-concept vision experiment suggests that once trained on clean objects from a dataset, the method could still generalize to \"real\" meshes coming from 3D reconstruction.\n\nWEAKNESSES:\n1. As is noted in the work, the proposed method is still a bit behind the SOTA GNNs in the accuracy of the prediction w.r.t. the ground-truth simulation. Thus, the proposed method is more aimed at larger scenes only where the baselines cannot be run.\n2. Learned SDFs for the objects, while fast to query, still need to be fitted first for each object and its accuracy is limited by the network's capacity.\n   1. In addition, every object's SDF-model need to be pushed on the GPU--or swapped in-and-out. This work seems to use mostly a smaller set of repeatable object, and thus can reuse the same models. This could potentially be solved by weight sharing between models or, as noted in the limitations.\n3. The *vision* experiment, simulating in an environment extracted from a multi-view NeRF reconstruction, is a bit underwhelming. While it is indeed just a proof-of-concept, it uses only a single object in a single scene, which is directly extracted as a rigid environment. I am unsure what is the main point of this result as the pipeline seems to be mostly the same as the previous ones.\n4. There is not much motivation behind needing *learned* simulators. Maybe expending on the limitations of the simulators themselves could make the work stronger.\n\nQUESTIONS:\n1. See Weakness 4., why is that important to build learned simulators in the context of rigid-object simulations? Are the simulators also memory-constrained and that makes them unusable in the large scenes discussed in the work?\n   1. A learnable surrogate model that approximates a simulator can also be useful as a *differentiable* simulation, e.g., in order to optimize the scene or some objects. Could the presented approach be use in such a pipeline?\n2. A more open-ended question: assuming no intersection, all the SDFs are used in their positive domain. Would the proposed method also work for *open surface* objects, using their UDFs (unsigned distance functions)? In practice, as there might be tiny amount of penetration, the SDF can detect it (negative output) but not the UDF. I wonder if the method is resilient to this or may fail.\n\nA minor suggestion: for Fig. 8, adding the standard-deviation or similar metric to the plots can be helpful, as they represent an aggregate of multiple simulations.\n\nLIMITATIONS:\nLimitations are discussed in the paper: the need to train an SDF MLP per object in the scene, that the current accuracy is slightly below SOTA (when those GNN model can be applied), and is currently limited to rigid-objects only.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a neural network-based simulator of rigid body dynamics with contacts that is memory-efficient and thus scalable to scenes with hundreds of objects and up to 1.1 million nodes on a single GPU. The key idea is to use SDF as the geometry representation to simplify collision detection compared with mesh representations.\n\nSTRENGTHS:\nThe paper addressed an important and interesting problem, which is an efficient end-to-end simulator of rigid body dynamics. It has significant potential value in various fields, including robotics, AR, etc. Replacing mesh representations with SDFs allows the framework to largely reduce the computational complexity when modeling the collisions between objects. This key idea has been clearly conveyed and experiments validated this claim. \n\nSection 2 provided a nice context that educates readers about the basic knowledge required to understand the framework of this work. The methodology presentation in Section 3 is clear, compact, and easy to follow. The experiments laid out the performance of the proposed method from small-scale scenes to significant larger-scale scenes, which is beneficial for readers to appreciate the merits of this work. \n\nThe authors provided extensive details in the appendix about the implementation of the geometric representation, network modeling, training, and evaluation, which made the whole paper much more informative and improved reader's understanding.\n\nWEAKNESSES:\nI found weaknesses mainly in the experiment section, mainly in the limited explanation and interpretation of the experimental results. \n\nFirst, I found the wording \"small-scale datasets\" and \"small datasets\" misleading. The conventional interpretation of a \"small-scale dataset\" refers to the size of the training data, while in this paper, I believe it actually means the simulation scene is of a relatively small scale, i.e., involving a limited number of objects and nodes. It is a core concept in the experiment section, and the potential gap between the interpretations could lead readers to totally different takeaway messages and create inconsistency between the claim and the evidence. According to Appendix B, 1500 trajectories are used for training in both Movi-B and Movi-C datasets, which reveals the actual scale of the training data. \n\nOn top of this point, there isn't enough discussion about why the proposed method performs slightly worse than the baseline methods on datasets with small scenes and outperforms in larger scenes. What is the difference between smaller scenes and larger scenes for these models? It is a key finding in this paper, but the discussion is largely omitted. \n\nAppendix E.6 seems to try to answer this question, but I didn't quite understand this paragraph. What do you mean by \" train an SDF-Sim architecture using the accurate distances directly computed from a mesh\"? I thought it was exactly what was done in the paper.  Why do an experiment with unsigned signed distance? I think this section created more confusion. \n\nA minor issue is in Appendix E.4. and Figure S3. The figure shows the baseline in purple (blue to me) and SDF-sim in red (orange to me), while the text says the opposite. Please make them consistent.\n\nQUESTIONS:\nI have a question about the surface nodes. In the paper, the authors didn't explain how the surface nodes are generated. From Appendix E.5, it seems like the surface nodes used in the experiments in the paper are the mesh vertices used for collision simulation. Is it correct? In E.5, using surface nodes as samples on the SDF can achieve similar accuracy with much fewer nodes. Why isn't it the default design in the main paper? Is there a trade-off?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations. The authors and the reviewer did not foresee the potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}