Grounding Graph Network Simulators using Physical Sensor Observations
Authors: Jonas Linkerhägner
Niklas Freymuth
Paul Maria Scheikl
Franziska Mathis-Ullrich
Gerhard Neumann
Venue: ICLR 2023 poster (2023)
Keywords: graph network simulators
deformable object simulation
point clouds

Abstract: Physical simulations that accurately model reality are crucial for many engineering disciplines such as mechanical engineering and robotic motion planning. In recent years, learned Graph Network Simulators produced accurate mesh-based simulations while requiring only a fraction of the computational cost of traditional simulators. Yet, the resulting predictors are confined to learning from data generated by existing mesh-based simulators and thus cannot include real world sensory information such as point cloud data. As these predictors have to simulate complex physical systems from only an initial state, they exhibit a high error accumulation for long-term predictions. In this work, we integrate sensory information to ground Graph Network Simulators on real world observations. In particular, we predict the mesh state of deformable objects by utilizing point cloud data. The resulting model allows for accurate predictions over longer time horizons, even under uncertainties in the simulation, such as unknown material properties. Since point clouds are usually not available for every time step, especially in online settings, we employ an imputation-based model. The model can make use of such additional information only when provided, and resorts to a standard Graph Network Simulator, otherwise. We experimentally validate our approach on a suite of prediction tasks for mesh-based interactions between soft and rigid bodies. Our method results in utilization of additional point cloud information to accurately predict stable simulations where existing Graph Network Simulators fail.

Published as a conference paper at ICLR 2023
GROUNDING GRAPH NETWORK SIMULATORS USING PHYSICAL SENSOR OBSERVATIONS
Jonas Linkerhägner
Niklas Freymuth'
Paul Maria Scheikl
Franziska Mathis-Ullrich
Gerhard Neumann'
Instiiutee for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Karlsruhe, Germany
'Department Artificial Intelligence in Biomedical Engineering, Aleeht-AAeexade---niie Brlangen-Nürnberg Erlangen, Germany
ABSTRACT
Physical simulations that accurately model reality are crucial for many engineer- ing disciplines such as mechanical engineering and robotic motion planning. In recent years, Iearned Graph Network Simulators produced accurate mesh-based simulations while requiring only a fraction of the computational cost of traditional simulators. Yet, the resulting predictors are confined to learning from data gener- ated by existing mesh-based simulators and thus cannot include real world sensory information such as point cloud data. As these predictors have to simulate complex physical systems from only initial state, they exhibit high error accumulation for long-term predictions, In this work, we integrate sensory information to ground Graph Network Simulators on real world observations. In particular, we predict the mesh state of deformable objects by utilizing point cloud data. The resulting model allows for accurate predictions over longer time horizons, even under uncertainties in the simulation, such unknown material properties. Since point clouds are usually not available for every time step, especially in online settings, we employ imputation-bart
model. The model can make use of such additional infor- mation only when provided, and resorts to a standard Graph Network Simulator, otherwise. We experimentally validate approach for mesh-based interactions between soft and rigid bodies. Our method results in utilization of additional point cloud information to accurately predict stable simulations where existing Graph Network Simulators fail.
suite of prediction tasks
INTRODUCTION
Mesh-based simulation of complex physical systems lies at the heart of many fields in numerical sci-
ence and engineering (Liu et al. 2022 Reddy 2019 Rao 2017
Sabat & Kundu 2021). Applications
include structural mechanics (Zienkiewicz & Taylor 2005
Stanova al. 2015), electromagnet-
ics (Jin) 2015 Xiao et al. 2022 Coggon 1971). fluid dynamics (Chung
1978//awaw et al. 2018)
Long al. 2021 and biomedical engineering (Van Staden al. 2006 Soro al. 2018), most of
which traditionally depend on highly specialized task-dependent simulators. Recent advancements in deep learning brought rise to more general learned dynamic models such as Graph Network Simulators (GNSs) (Sanchez-Gonzalee et al.) (2018) 2020: Pfaff et al.) (02T).. GNSs Iearn to predict the dynamics of system from data by encoding the system state as a graph and then iteratively computing the dynamics for every node in the graph with a Graph Neural Network (GNN) (Scarselli cet al. 22099] [Battaglia et al. [2018] Wu et al. 2020b). Recent extensions include long-term fluid flow predictions (Han et al. 2022) and dynamics on different scales (Fortunato et al. (022). Yet, these approaches assume full knowledge of the initial system state, making them ill-suited for applications
"correspondence jonas. 1 linkerhaegneree ummni .it.

---

Published as a conference paper at ICLR 2023
Figure I: A robot's end-effector (grey, red) grasps 3-dimensional deformable cavity. The robot maintains internal simulated prediction of the cavity (orange) for two consecutive simulation steps (left, right). This prediction can deviate from the true state of the cavity over time due to an accumulation of error. However, the true cavity state can infrequently be observed from point cloud data (blue), which the model can use to correct its prediction. Here, the point cloud is used to contract the simulated cavity at the bottom and extend it at the top, causing the points better align with the mesh surface. We repeat the point cloud from the earlier simulation step in both images for clarity.
like model-predicite control (Camacho & Alba, 2013 Schwenzer et al. 2021) and model-based Reinforcement Learning (Polydoros & Nalpantidis [2077] Mocrland et al. 2020) where accurate predictions must be made based on partial initial states and observations.
In this work, we present Grounding Graph Network Simulators (GGNSs), a new class of GNS that can process sensory information input to ground predictions in the scene observations. More precisely, we extend the graph of the current system state with point cloud data before predicting the system dynamics from it. Since point clouds do not provide correspondences over time, it is difficult to learn dynamics from point clouds alone. Thus, we use mesh-based data to learn the general system dynamics and utilize point clouds correct the predictions. As the sensory data is not always available, particularly not for future predictions, our architecture is trained with imputed point clouds, i.e. for each time step the model receives point clouds only with a certain probability. This training scheme allows the model to efficiently integrate the additional information whenever provided. During inference, the model iteratively predicts the next system state, using point clouds whenever available to greatly improve the simulation quality, especially for simulations with incomplete initial state information. Furthermore, our architecture addresses a critical research topic for GNSs by alleviating common challenges such drift and error accumulation during long-term predictions.
As practical example, consider a robot grasping a deformable object. For optimal planning of the grasp, the robot needs to model the state of the deformable object over time and predict the influence of interactions between object and gripper. This prediction not only depends on the initial shape of the object, but also on the forces the robot applies, the kind of material to grasp and external factors such as the temperature, making it difficult to accurately predict how the material will deform over time. However, once the robot starts deforming the object, it may easily observe the deformations in the form of e.g., point clouds. These observations can then be integrated into the state prediction, i.c.. they can ground the simulation whenever new information becomes available. An example given prediction is similar in nature to c.g., Kalman Filters (Kalman 1960, Jazwinski 1970, Becker et al. 2019) as the belief of the system state is updated based on partial observations about the system. However, while Kalman Filters explicitly integrate novel information into the belief in a mathematical fashion, we instead simply provide this information to a learned model as additional unstructured sensor input.
in Figure I Such observation-aide
We evaluate GGNS on suite of 2d and 3d deformation prediction tasks created in the Simulation Open Framework Architecture (SOFA) (Faure et 2012). Comparing approach to an existing GNS (Pfaff et al.) 2021), we find that adding sensory information in the form of point clouds to our model improves the simulation quality for all tasks. We investigate this behavior through extensive ablation studies, showing the importance of different parameter choices and design decisions. Code and data can be found under https: / //t/tttttb//////// com/ http//linkkk wwiiiinkniNNNSS
Our list of contributions is as follows: (I) We extend the GNS framework to include sensory infor- mation to ground predicted simulations in observations of the system state, allowing for accurate
2

---

Published as a conference paper at ICLR 2023
predictions of the full simulation. (II) We propose a simple but effective imputation training scheme that naturally integrates sensory information to GNSs whenever available without substantially increasing training cost or model complexity. (III) We construct and experiment different defor- mation prediction tasks and find that the inclusion of sensory information improves performance in all settings, and that it is particularly crucial when the initial system state is not fully known.
RELATED WORK
Learned Physics Simulation. In recent years there has been a steady increase in research concerning deep learning for physical simulations. Early work in physical reasoning aims at teaching systems
to understand physical relations on N-body systems (Battaglia et al.] 2016) and deformable ob- jects (Mrowea al., 2018). A more direct approach is to instead train learnable simulator from data
provided by some existing ground truth simulator. Here, Convoluational Neural Networks (CNNs)
have been extensively studied for fluid flow simulation (Tompson al. 2017: Chu & Thuerey 2017
Ummenhofer et al. 2020 Kim et al. 2019) Xie et al. 2018) and aerodynamic flow fields (Guo et al.
2016|Zhang et al. 2018 Bhatnagar et al. 2019). Further approaches use standard neural networks for
liquid splash simulations (Um et al. 2018) and latent space physics simulation (Wiewel et al. 2019).
Such learned physics simulators are considerably faster than their ground counterparts, and that they are usually fully differentiable. Thus, they have been applied to model-based Reinforcement
Learning (Mora et al. 2021) and for Inverse Design problems (Baqué et al., 2018 Durasov et al. 202
Allen et al. 2022b).
Graph Network Simulators. Graph Network Simulators (GNS) Sanchez-Gonzalezz et al. 2020) are
special case of learned physics simulators that utilize GNNs (Sarselli et al. (2099) to efficiently encode the graph-like structure of many physical problems. They have found wide-spread application in calculating atomic forces (Hu et al. 202I1, particle-based simulations (Li et al. 2019 Sanchez
Gonzalez et al. 2020) and based simulations (Pfaff et al. 2021 Weng et al. [021]] Han et al.
2022; Fortunato et al. 2022; Allen et al., 2022a). Other works in this field directly solve partial differential equations (Alet et al.) 2019). and integrates explicit domain knowledge into the learned
simulator to improve the predictions (de Avila Selbute-Peres et al., 2020 Li & Farimani, T
2022). Similarly, CNNs have been used to predict particle masses from images to subsequently
simulate physical systems with a GNN eLi al. (020)) via visual grounding. This approach assumes access to series of images to predict particles and their behavior, whereas GGNS integrates sensor observations into an existing mesh- besh- simulation. The work most closely related to our research is MeshGraphNet (MGN) (Pfaff et al.) (02T), which combines a graph- based encoding of the system state with the next-step prediction of dynamic quantities to produce realistic predictions of mesh-based simulations.
Simulation from Observation. Another variant of learned physics simulation is simulation from observation. Learning directly from observations instead of a ground truth simulator requires less expert knowledge for the design of the simulator and is more applicable to real-world scenarios. Different approaches exist for this type of simulation, including Physical reasoning (Li al. 2020) and particle- based simulation (Martinkus et al. 202T). Point clouds have been used in CNN- based simulation (Watters al., 2017 Wang et al.) 2019). and combined with PointNet (Charles al.
2017 Qi et al. (017) to predict object deformations purely from observational data PPak et al. 2021))
Further approaches make use of GNNs to predict object relations (Fetaya et al. 2018) and future
frames in a point cloud sequence (Gomes et al. 2021).
Simulation of Deformable Objects. Simulating deformable objects is crucial for many applications such as robotic manipulation tasks (Sanchez et al.] 2018). Yet, recent approaches do not take explicit deformation into account (Matas et al. [2018), 1), or only consider highly simplified geometries such as ropes (Sundaresan et al.] 2020) or a square piece of cloth (Wu et al. 2220aa) Lin et al. 2020 2022). One reason for this is the high computational cost of existing simulators, which may be alleviated by fast and accurate learned simulators (Pfaff et al. |021 Weng et al. 2021). Another recent work trains the parameters of differentiable simulator to align its simulations with real-world observations of deformable objects based point cloud information (Sundaresan et al.) 2022). In this work, instead utilize point cloud information to improve upon existing mesh-based GNSs in settings where additional point cloud data is available.
3

---

Published as a conference paper at ICLR 2023
3 FOUNDATIONS
3.1 MESSAGE PASSING NETWORK
Let 9 EV, Xv, Xg) be directed graph with nodes V, edges E S V x V, node features R " of dimension dg. A Message Passing Network (MPN) (Snnalez-Gonzzlez et al. 2020 Praff al. 202T is a GNN consisting of I. Message Passing Blocks that receives the graph g as input and outputs a learned representation for each node V and edge E. Each block computes updated features for all nodes
Xv : V Rdv of dimension dy and edge features X E
V and edges
e E E as
x+1 -/((x Xe), with (u, v) and x -x
(e=(O,)EEB)
where xL and x are embeddings of the initial node and edge features of G and @ is permutation- invariant aggregation such as a sum, max, or mean operator. Furthermore, each f. is a learned function that is generally parameterized as simple Multilayer Perceptron (MLP).
3.2 GRAPH NETWORK SIMULATOR
GNSs simulate a system's 's dynamics by repeatedly applying the following three steps. First, they encode the system state S in graph G. If the system state given e.g., triangular or tetrahedral mesh M of the underlying entities, this graph is naturally constructed by using the nodes of . as nodes of the graph, and the connection between these nodes as edges. The node and edge features Xv,Xp XE can be constructed based the concrete simulation. In general, encoding purely relative properties such as relative distances and velocities per edge rather than absolute positions per node have been shown greatly improve training speed and generalization (Sanchez- Gonzalez al.) 2020). Next, the encoded graph 9 is used as input for a learned MPN, which computes final latent interpreted (potentially
representations _ for each node higher-order) derivatives of dynamic quantities, which are used by a simple forward-Fun integrator to derive an updated system state S'. Note that for some tasks, only fraction of mesh nodes need to be predicted, as the others are either fixed or belong to known entity such as a gripper or collider. In this case, only the latent representations of the nodes with otherwise unknown dynamics are used.
V. These latent representations
GNS$ are trained on a node-wise next-step Mean Squared Error (MSE) objective, i.e., they minimize the 1-step prediction error of the next system state to that of a given ground truth trajectory. During inference, simulations over potentially hundreds of steps can be generated by iteratively repeating the above-mentioned steps, using the updated dynamics of one step as the input for the next. We note that the model does not predict the movement of fixed entities such e.g., a collider, which is instead assumed to be known and combined with the model's prediction about the unknown parts of the system. Due to this iterative dependence accumulation. common strategy to tackle this limitation is to apply additional noise to the dynamic variables of the system for each training step (Sanchez-Gonzalez et al. 2020 Pfaff al., (021). Intuitively, adding training noise acts as form data augmentation that allows the learned model to compensate for small prediction errors prediction leads to plausible and visually realistic predictions. However, the resulting predictions can be arbitrarily inaccurate with respect to the true dynamics of the system, since the model has no reference for its simulation other than some potentially incomplete initial state Sg.
previous outputs, the model is prone to error
time. This kind of eroor-compnnaaingg next-step
4 GROUNDING GRAPH NETWORK SIMULATOR
Our approach combines recent advances in graph-based learned physics simulation with additional partial observations of the system state to generate highly accurate simulations from incomplete initial states. To this end, we extend the existing GNS framework to naturally and efficiently integrate auxiliary point cloud data whenever available. This auxiliary information grounds the predictions of the model in observation of the true system state, guiding it towards predictions that not only look realistic but also closely match the actual dynamics of the system. Figure @ illlustrates an overview of approach. A more detailed description of the GNN-part of the method is found in Appendix A
4

---

Published as a conference paper at ICLR 2023
Point-cloud at Step
Neighborhood Graph
Update
Predicted Mesh at Step t+1
GNN
Optional Point-cloud Information
Training MSE
Inference
Inference [=t+|
Initial System State
True State Step 1+1
State Step
Figure 2: Schematic of GGNS. Given system state S, (red) and optional point cloud observations (dashed box), a GNN (orange) predicts how the system S,41 will look like at the next step (blue). For object deformation tasks, the state include boundary conditions (gray) such colliders or walls. When provided, the point cloud (green) is transformed into a neighborhood graph (yellow) in the same coordinate system the mesh, connecting each point in the point cloud to the nearest mesh nodes. The model is trained to predict the next system state based on a true state (purple) provided by ground truth simulator. During inference, the model iteratively predicts updates from potentially incomplete initial system state (purplc), using additional point cloud observations when available.
4.1 POINT CLOUDS AND NEIGHBORHOOD GRAPHS
In order to utilize point-based data addition to meshes we first have to transfer both into common graph. Following previous work (Sanalez-Gonzalez et al. 2018), we do this by creating neighborhood graph based on spatial proximity. Given graph g = V E, Xv, , Xg) that encodes a
predicted system state and point cloud observation P {pI..... p. Pj E Rd of the true system state, we set V' VUP and
E' Eu
E paId(p,,p) Pj) S rp] u ((c,p))
ss).
Here, d is some distance measure, usually the euclidean distance, and rp and r's task task-specific neighborhood radii. The corresponding features Xv, X of the added nodes and edges in V and E' depend the concrete task. The different node and edge types are one-hot encoded into their respective features to allow the model to differentiate between them. Similar to the original features, information can be encoded in a relative fashion the form of edge features aid generalization. More concretely, we encode relative distances in world space along all edges, additionally adding mesh-space distances for edges between two mesh nodes. This connectivity slightly different from MGN (Plaff et al. [202T), which make use of additional world edges between mesh-nodes by creating a similar radius-based neighborhood graph for the mesh nodes in world space.
4.2 IMPUTATION-BAN TRAINING AND INFERENCE
For most realistic applications, point clouds are typically not available at each time step during inference. For example, we may have access to observed point clouds from the previous i time steps and want to use them infer the state of the system in the future. We adapt model to this constraint by employing an imputation-bast randomly replace the graph G of S with the corresponding extended graph G' = (V', E', Xv., with equal probability during training. In both cases, the model is only trained to predict the system dynamics for the original nodes V. Intuitively, this allows each system node v e to utilize the additional information of close-by points of point cloud when available, while at the same time forcing to also make sensible predictions when there is no additional information. During inference. we construct g' from the (predicted) system state S and corresponding observed point cloud 'P of the true object whenever available and use G otherwise. This enables the model to reason about the true system state that is observed via P. adapting its prediction to the otherwise unknown behavior of the system. This grounding of the prediction also alleviates common errors of GNS such as drift and more generally error accumulation. An example can be seen in Figure [ Here, the system state consists of a predicted mesh and a gripper, and the point cloud consists of points sampled from the true object. The mismatch between point cloud and predicted mesh indicates the prediction error, and
training scheme. Our model still uses a single GNN, but we now
5

---

Published as a conference paper at ICLR 2023
GGNS(K=5)
Ground Truth
MGN
Figure 3: Final simulated meshes (t = 50) for GGNS (A = 5) (left), the ground truth simulation (middle) and MGN (right) for 2 test rollouts with different material properties for the Deformable Plate task. Our model closely matches the ground truth simulations for both materials, while MGN predicts the same material every time.
the model uses this additional information to correct the current state estimate, Similar figures for the other two tasks can be found in Appendix C
We compare this simple imputation-basee method to another training scheme our experiments, which we call GGNS+LSTM. Here we use an LSTM (Hochreiter & Schmidhuber 1997) layer on the node output features of the GNN to explicitly include recurrency into the model. This modification allows information such the material properties to be inferred and propagated over time, which can be utilized to improve the predictions in time steps without point clouds. The resulting model trained on the same 1-step prediction loss and also uses training noise to generate stable rollouts during inference. However, it is significantly more costly to train, as makes use of backpropagation in time compute the gradients for the recurrency. We find experimentally that this recurrent model performs worse than the imputation-base based method. An explanation for this is that the potential benefit of propagating information over time is offset by the additional training and model complexity, especially with respect to the next-step prediction objective. For this reason, GGNS relies on this simple but effective imputation-ban approach.
EXPERIMENTS
We evaluate GGNS on complex 2d and 3d mesh-based object deformation prediction tasks modelled in the Simulation Open Framework Architecture (SOFA) (Faure et al., 2012). For each task, the true system state given by tetrahedral FEM mesh of a deformable object with rigid boundary conditions combined with triangular surface mesh of rigid collider. The point clouds generated by raycasting using one virtual camera for 2d and up to five cameras for 3d tasks arranged around the scene. More details the generation of the point clouds enviromment-spest
presented in Appendix B Additional
details, including node and edge features and dataset properties can also be found Appendix [ We assume that, while the initial mesh of the object known, its material properties are not. We model these unknown properties via the Poisson's 's ratio (Lim) 2015) -1 < // < 0.5, which scalar value describing the ratio of contraction (u < 0) or expansion (v 0) under compression (Mazaev et al.) (020). For all datasets, we randomly assign Poisson's 's ratios from E { (-0.9, 0.O, 0.49) equally to all rollouts.
We train all models all tasks using the Adam optimizer (Kingma & Ba 2015) with a learning rate of 10-4 and a batch size of 32, using early stopping on a held-out validation set to save the best model iteration for each setting. The models passing blocks with 1-layer MLPs and a latent dimension of 128 for node and edge updates. We use mean aggregation for the edge features and training noise of 0.01. All tasks space of [] An overview of the network hyperparameters can be found in Appendix
LeakyReLU activation function, five message
normalized task
Evaluation Metrics. We evaluate the performance of all trained models 10 different seeds per experiment. We report the means and standard deviations of the different runs, where, for each run, average the results over all available steps of trajectory and over all trajectories in the test set of

---

Published as a conference paper at ICLR 2023
GGNS (k=5)
Ground Truth
MGN
Figure 4: Visualization for a test trajectory at time step 1 = 70 for GGNS (left), the ground truth simulation (middle) and MGN (right). While MGN accumulates large prediction error time, GGNS is able to utilize the additional point cloud information to stay close to the ground truth for the full length of the simulation. Different angles of this visualization can be found Appendix C
the respective data set. For all experiments, we report the full rollout loss, where the model starts with the initial state Sy and predicts the states up to a final state Sy. Here, we provide point cloud to the model every & 2 1 steps and resort to mesh-only prediction otherwise. This corresponds to a setting in which the deformation of an object is tracked with both cameras which provide the position of the rigid collider and point-cloud information respectively.
sensors and freeureyyncyy
We also consider application where robot observes an object's deformation up some point in time and then reasons about future deformations without additional point-cloud information. For this setting, the initial system state So is provided to the model, followed by m point clouds for its next m predictions. Then, 10 more steps are predicted without point clouds to predict a state Sm++ and and compute the corresponding 10-step prediction loss. The reported losses are the average MSE over every step along the trajectory averaged over all possible rollouts. This metric reduces to the average loss for m + 10-step prediction for methods that cannot make use of point cloud data, as the state 1m+10 needs to be predicted from the initial So.
Baselines. We compare to MGN, slate-of-the-an GNS, which utilizes additional world edges between close-by mesh nodes, but does not incorporate point cloud observations. Comparing these world edges to Section 4.1 MGN assumes an edge partition E = E, ÜEE and separate edge update functions J and L The edge-aggregation for the node update is then computed by aggregating both types of edges separately and concatenating the result. We adopt this
the latent features explicit representation of edge types for the MGN baseline and experiment with it for GGNS in Appendix[] D As it does not provide any significant advantages for our model, GGNS instead resorts to simple one-hot encoding of the type of input edge for the remaining experiments.
Additionally, we evaluate a variant of MGN that has additional access to the underlying Poisson's ratio as node feature, called MGN (M). This additional information leads to a deterministic ground truth simulation w.r.t. the initial system state, and upper bounds the performance of MGN. We also compare to GGNS+LSTM, which integrates recurrency into our imputation technique. Here, investigate whether this recurrency helps the model predicting c.g., material properties over time.
As point cloud based baseline, we use non-learned method to directly generate a mesh from the point cloud of each time step. We voxel-subsample the point cloud so that we observe approximately the same number of points as nodes in the ground truth mesh and then use Alpha Shapes (Akkiraju et al. 1995) to create (potentially non-convex) mesh for this time step. This baseline shows how much information can be directly inferred from just the point cloud information.
Deformable Plate. We consider a family of 2-dimensional trapezoids that are deformed by circular collider with constant velocity. Besides the trapezoidal shapes, diversity in the dataset is introduced by varying the size and starting positions of the collider. For this task, additionally consider the Intersection over Union (IoU) between the predicted and the ground truth mesh as an evaluation metric. We find that this metric is less sensitive to individual mesh nodes and that it instead measures how well the predicted object shape matches that of the real system state. We use a total of 675/ 15/5//1/1/ 1135 trajectories for training, validation and test sets. Each trajectory consist of T 50 timesteps.
Tissue Manipulation. An important application for the prediction of deformable objects is medical robotics. We simulate robot-assisted surgery scenario where piece of tissue is deformed by
7

---

Published as a conference paper at ICLR 2023
Rolbet Men Sqaaed Ebnee
10-3
10-11
10-1
10
10
10
10"4
10a4
(km1)
(on2) (6x5)
(ee)) (be2) (ked) MÚN MCN
(k-1) (A-2) (a-5) (a-10) MGN MON
(o-1) (k=2) (ae5) (k-10) MGG MGX
GGNS
LGNS-LLIM
(KI
GEINS
(KI
COINS
(M)
(a) Deformable Plate
(h) Tissue Manipulation
(c) Cavity Grasping
Figure 5: Rollout Mean Squared Error of GGNS and MGN baselines evaluated on the test set of the three datasets. We report the results for GGNS using point clouds in every k-th time step. MGN(M) indicates the baseline method of MGN that uses the ground truth material as input feature. GGNS outperforms the MGN baseline all settings and in most cases even if it has access to the complete initial state. For the Deformable Plate task additionally report the errors for GGNS+LSTM, which perform worse than GGNS for all k but still outperforming the MGN baseline.
solid gripper. Varying the direction of the gripper's motion and its gripping position on the tissue results in additional diversity. Here, 600/120/122 trajectories are used, each of which is rolled out for T 100 timesteps. This task is visualized in Figure
Cavity Grasping. Robotic manipulation of deformable objects is an important application of deformable physics simulation. Here, a simulated Panda robot gripper grasps and deforms a cavity. For this purpose, we randomly generate cone-shaped cavities with different radii, which are deformed by a gripper from different positions. An example simulation step by GGNS for this task is illustrated We use the same amount of samples and data split as in the Tissue Manipulation task.
in Figure I
6 RESULTS
Main Results. We test our method on the three deformation prediction tasks described in Section 5 and compare it to MGN with and without material information. We find that GGNS can use the point cloud information produce high quality rollouts that closely match the true system states. An example is shown in Figure which visualizes the final simulated meshes for our method and the ground truth simulation. Additionally, GGNS outperforms the baselines even when they have access to the complete initial state, which our model has not. Figure 4 shows the qualitative differences between GGNS and MGN the Tissue Manipulation task. Additional visualizations for all tasks and both methods can be found in Appendix C The evaluations for full rollouts are given in Figure 5 Table shows results for the m + 10-step evaluation. Appendix shows the performance of GGNS for different model hyperparameters. Similar to Plaff et al. (202I). we find that GGNS is robust to most parameter choices, and that a modest amount of training noise is crucial for long-term rollouts. To show the applicability of our method for more realistic point cloud data, we provide additional ablations on noisy and partial observable point clouds in Appendix D We find that our model is quite robust to the quality of the point clouds and can still reliably use their information to ground the simulation. On the Deformable Plate dataset, we additionally evaluate the mean Intersection over Union (IoU) during the rollouts to emphasize the compliance with the overall shape of the object rather than that of individual mesh nodes. The results are illustrated in Figure 6a
Recurrent Imputation Model. For the 2d data of the Deformable Plate task, additionally compare our imputation model to the GGNS+LSTM approach, which can use the recurrence of LSTMs to pass information over time. Figure 5a shows that GGNS outperforms this alternative approach for each k. We find that our simple architecture outperforms the recurrent one while requiring significantly less time to train, likely due the additional complexity of training the recurrent model. The qualitative results in Appendix confirm these findings.
Initial Mesh Generation. Using the IoU metric, we can compare objects across different mesh representations. The results in Figure 6b show that GGNS produces accurate rollouts even if the initial mesh is generated directly from the initial point cloud. For this, we compute a mesh with similar resolution to the training meshes from the convex hull of the initial point cloud, avoiding
FRANKA EMIKA GmbH, Munich, Germany
8

---

Published as a conference paper at ICLR 2023
+ 10-step MSE x10-
Approach
Plate
Tissue
Cavity
GGNS
2.907 = 0.172
0.514 + 0.052
0.923 + 0.040
MGN (M)
10.663 1.063
5.027 1.489
6.294 + 0.668
MGN
282.684 = 18.112
5.885 + 0.723
11.528 + 0.747
Table I: Evaluation the 10-step prediction setting on the test for all three tasks. GGNS clearly outperforms the baselines on all tasks even if they have access to the full initial simulation state.
0.35
50-aag ft-bustat
Nomulied Bundi
50-tpp
(b-11
(k-2) MGN
GENNS
GENS
(a) Deformable Plate IoU
(b) ToU without initial mesh
(c) Different grounding frequencies
Figure 6: (a) Rollout loU for the Deformable Plate dataset for GGNS and the MGN baseline. The main findings here are similiar to the MSE results but enable to compare against the values in (b). where the initial mesh created from initial the point cloud. The achieved IoUs are lower compared to using the ground truth mesh, but GGNS (k S 2) still outperforms all baselines including the Alpha Shapes in this setting. (c) Normalized benefit of using a point cloud in every k-th timestep, where k 1 means point cloud is available in every time step.
the dependence on any simulation data. This procedure marks an important step towards using these models on real world data. The results indicate that generating the initial mesh from point cloud information results in degradation the performance compared to an evaluation that uses a provided mesh. Yet, it still allows for high-quality prediction of the deformation. The comparison to Alpha Shapes shows that combining infrequent point cloud information (k = 5) with a simulator leads better and more consistent results than directly creating the mesh from the point cloud in each time step. Additionally, our model naturally tracks the correspondences of mesh nodes over time, whereas Alpha Shapes cannot observe the evolution of individual particles in the system. As such, GGNS allows for a more thorough understanding of the modeled process.
Grounding Frequency. Figure 66c shows the normalized performance of GGNS for grounding frequencies l E (1..10} across tasks. Here, value of 1.0 corresponds to the performance for k = and 0.0 to the performance MGN. For all tasks there clear advantage utilizing the point cloud information, and the performance increases with the frequency of available point clouds.
7 CONCLUSION
We propose Grounding Graph Network Simulator (GGNS), an extension of the popular Graph Network Simulator framework that can utilize auxiliary observations to accurately simulate complex dynamics from incomplete initial system states. Utilizing neighborhood graph computed from point cloud information and imputation- based training scheme, our model able to ground its prediction in an observation of the true system state. We show experimentally that this leads to high-quality simulations in challenging 2d and 3d object deformation tasks, outperforming existing approaches even when these provided with full information about the system.
In future work, we will extend GGNSs to explicitly model uncertainty and maintain belief over the latent variables of the system, e.g., by employing Kalman filter in a learned latent space (Becker et al. 2019). Another promising direction is to adapt the current next-step prediction loss to instead predict trajectory over small period of time increase the long-term consistency of the model. Finally, we will employ our model for predicttre control and model- -based Reinforcement Learning in both simulation and real robot.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Vincent Kreuziger for the helpful discussions on the visualizations and for the high-quality blender renderings. The authors acknowledge support by the state of Baden- WellenWWirttebbergg through bwHPC. GN was supported by the DFG research unit DFG-FOR 5339 (AI-based Methodology for the Fast Maturation of Immature Manufacturing Processes) and GN and NF were supported by the BMBF project Davis (Datengetriehene Vernetzung für die ingenieurtechniscee Simulation).
REFERENCES
N. Akkiraju, Edelsbrunner H., Facello M., P. Fu, E. P. Mticke, and C. Varela. Alpha shapes: Definition and software. In Proceedings of the 1st International Computational Geometry Software Workshop. pp. 63-66, 1995. URL http: shapes95def/
/www: www. geom. uiuc. wdw. wttp://ttwwwateeeeoooeseeeeeeeee wet.t//aaae//llss/eeeeelto
Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas Lozano- Perez, and Leslie Kaelbling. Graph element networks: adaptive, structured computation and memory. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learn- ing Research, pp. 212 222. PMLR, 09-15 Jun 2019. URL https: https::/proceediing mlr
2xess/v97/aletlge html
Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kimberly Stachenfeld, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. Conference on Robot Learning (CoRL)., 2022a.
Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Pe- ter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv:2202.0077 2022b.
Pierre Baqué, Edoardo Remelli, François Fleuret, and Pascal Fua. Geodesic convolutional shape optimization. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmässuun Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 481-494199 PMLR, 2018. URL http: http:://prcceeiing.... m.r press hat.///aaueeaa..tt....
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL /t///tttps///rrcceeeinggg.. neurips /c/
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Goonzlee,, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CORR, abs/ 100.01121, 2018. URL http: http//axivv.. org. wttp::///..b.......02202.. 01261
Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, James Taylor, and Gerhard Neumann. Recurrent kalman networks: Factorized inference in high-dimensional deep feature spaces. In International Conference on Machine Learning, pp. 344-552.1 2019.
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational jun 2019. doi: 10. 10.10077300466-119--07444
Mechanics, 64(2):525-545,
URL. https
7/doi . orw.liorg11100 1t0p::/0000000000000000000000000 01740-0
Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media, 2013.
10

---

Published as a conference paper at ICLR 2023
R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In 2017 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77-85, 2017. doi: 10.1109/CVPR.2017...6
Mengyu Chu and Nils Thuerey. Data-driven synthesis of smoke flows with emn- enn-based feature descriptors. ACM Truns. Gruph., 36(4), jul 2017. ISSN 0730-0301, doi: 10. 10.114553072955:3073444 btRthttps:/ /tte:s::/doi.oom org 0011111///725555 3073643
TJ Chung. Finite element analysis in fluid dynamics. NASA STI/Recon Technical Report A, 78:44102, 1978.
JH Coggon. Electromagnetic and electrical modeling by the finite element method. Geophysics, 36 (1):132-155, 1971.
Filipe de Avila Belbute-Peres, Thomas D. Economon, and J. Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
Nikita Durasov, Artem Lukoyanov, Jonathan Donier, and Pascal Fua. Debosh: Deep bayesian shape optimization. arXiv preprint arXiv:2 2021.
François Faure, Christian Duriez, Hervé Delingette, Jérémie Allard, Benjamin Gilles, Stéphanie Marchesseau, Hugo Talbot, Hadrien Courtecuisse, Guillaume Bousquet, Igor Peterlik, and Stéphane Cotin. SOFA: A Multi-Model Framework for Interactive Physical Simulation. In Yohan Payan (ed.), Soft Tissue Biomechanical Modeling for Computer Assisted Surgery, volume 11 of Studies in Mechanobiology, Tissue Engineering and Biomaterials,, pp. 283-321. Springer, June 2012. doi: 10.1007/8415, 2012) 125. htLhhtts::///me inriiniiii..... fr hal.:rraaallaaa-
T. Fetaya, Elias Wang. K.-C. Welling, Michelle Zemel, Thomas Kipf, Ethan Fetaya, Chan-Chieh Wang, Max Welling, and Richard S. Zemel. Neural relational inference for interacting systems. arXiv: Machine Learning, 2018.
Meire Fortunato, Tobias Pfaff, Peter Wirnsberger, Alexander Pritzel, and Peter Battaglia. Mul- tiscale meshgraphnets. In ICML 2022 2nd Al for Science Workshop, 2022. URL https
//openreview m
Pedro Gomes, Silvia Rossi, and Laura Toni. Spatio-temporal graph-rnn for point cloud prediction. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 3428-3432, 2021. doi: 10.1 10.1199//1112222220000 2506990000.2
Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 481-490, New York, NY, USA, 2016. Associa- tion for Computing Machinery. ISBN 9781450342322. doi: 10.1145/29396722299977 URL
Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. CoRR, abs/2201.091 2022. URL http:/ts///axiv..oo///
abs/2201. 09113
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735-80, 12 1997. doi: 10. 162/neco. 1at.t....9999999 997.9.8.1733.
Weihua Hu, Muhammed Shuaibi, Abhishek Das, Siddharth Goyal, Anuroop Sriram, Jure Leskovec, Devi Parikh, and C Lawrence Zitnick. Forcenet: A graph neural network for large-scale quantum calculations. In ICLR 2021 SimDL Workshop. volume 20, 2021.
AH Jazwinski. Stochastic processes and filtering theory. ACADEMIC PRESS, INC.,, 1970.
Jian-Ming Jin. The finite element method electromagnetics. John Wiley & Sons, 2015.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of basic Fngineering, 82(1):35-45, 1960.
11

---

Published as a conference paper at ICLR 2023
Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep Fluids: Generative Network for Parameterized Fluid Simulations. Computer Graphics Forum (Proc. Eurographics), 38(2), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Leurning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. h
1412. 6980
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2019. URI TRIhhtsps: potttps://ppnreeee http//pppeeevveeeeeeeeeee ew. forrmm?
id cd-rJgbSnO99
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel Yamins, Jiajun Wu, Joshua Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5927-5936. PMLR, 13-18 Jul 2020. URL attips:/pproeeednggsst wtt.://sws.......sss/s/l///0 wwtpppseweeeeeeeeeee
Zijie Li and Amir Barati Farimani. Accelerating lagrangian fluid simulation with graph neural networks. In ICLR 202/ SimDL. Workshop, volume 20, 2021.
Zijie Li and Amir Barati Farimani. Graph neural network-accellertte lagrangian fluid simulation. Computers & Graphics, 103:201-211, 2022.
CeinnCheng Lim. Auxetic Materials and Structures. Springer Singapore, 01 2015. ISBN 978- doi: 00.1007/978-9981-88--7777
981-287-274-6.
URL
978 911-988--888880 275-3
Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning. 2020. URL
/tmps://aariinn org. abs 2011.11111 1120111070 07215
Xingyu Lin, Yufei Wang, Zixuan Huang, and David Held. Learning visible connectivity dynamics for cloth smoothing. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 256-266. PMLR, 08-11 Nov 2022. URL hattts::/prrceeecingg ml httpsss........ssseeeeemm wrwss/vv66ll v 1im:2a. html
Wing Kam Liu, Shaofan Li, and Harold S Park. Eighty years of the finite element method: Birth, evolution, and future. Archives of Computational Methods in Engineering, pp. 1-23, 2022.
Ting Long, Can Huang, Dean Hu, and Moubin Liu. Coupling edge-based smoothed finite element method with smoothed partiele hydrodynamics for fluid structure interaction problems. Ocean 2021.
Engineering,
Karolis Martinkus, Aurelien Lucchi, and Nathanaël Perraudin. Scalable graph networks for par- ticle simulations. Proceedings of the AAAI Conference on Artificial 3510 8920, May 2021. URL URL htttt::t//ooss aaai wwwpoorgggnngi.... http////ppp///.t/aaaaaaeeee///te 17078
Jan Matas, Stephen James, and Andrew J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzer- land, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 734-743. PMLR, 2018. URL hww.hhtpp//// htttp://proceedingg: .aaiggsmmrr.... ml html
A V Mazaev, O Ajeneza, and M V Shitikova. Auxetics materials: properties and applications. IOP Conference Series: Materials Science and Engineering, 747 (1):012008, jan 2020. doi: 1001///777719997777711010010 1088/1757- /
classification, mechanical
URL BtLl/attpp:///i..o////
12

---

Published as a conference paper at ICLR 2023
Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Model-based reinforcement learning: A survey. arXiv preprint 20Xi::2006.1671,, 2020.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7805 78105-887. PMLR, 18 24 Jul 2021. URL http://proceedlngg htt.://.........eee////////22222
Damian Mrowca, Chengxu Zhuang, Elias Wang. Nick Haber, Li F Fei-Fei, Josh Tenen- baum, and Daniel L Yamins. Flexible neural representation for physics prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL htttps:://proceeddnngg wtt.erppss.ccpppeee//0////////// 100.ddd666af11111334444000555553
Jinhyung Park, Dohae Lee, and In-K Lee. Flexible networks for learning physical dynamics of deformable objects. CoRR, abs/2112.03728,, 2021. URL httph:htpp:/satt /tt::///xxxiy/ http://awxx...../baaa./a org. 2112
03728
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalezz and Peter W. Battaglia. Learning mesh- based simulation with graph networks. In International Conference on Learning Representations, 2021. BtLpbtts::// //tp:v/aaiivvvvvaav wwwp////gaaaaeeeeeeeeeeeeeeeeees
03409
Athanasios S Polydoros and Lazaros Nalpantidis. Survey of model-based reinforcement learning: Applications on robotics. Journal of Intelligens Robotic Systems, 86(2):153- 173, 2017.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances Neurul Information Processing Systems, volume 30. Curran Associates, Inc., 2017. hwtphhttss//ppoceeeeeeeeeeeeeeee 111pa//e0000//////////cc22201200
ips
Singiresu S Rao. The finite element method engineering. Butterworth hetpwoorthheeinemann. 2017.
Junuthula Narasimha Reddy. Introduction the finite element method. McGraw-Hill Education, 2019.
Lovely Sabat and Chinmay Kumar Kundu. History of finite element method: a review. Recent Developments in Sustainable Infrastructure, pp. 395-404, 2021.
Jose Sanchez, Juan Antonio Corrales Ramon, B. Chedli BOUZGARROU, and Y. Mezouar. Robotic manipulation and sensing of deformable objects in domestic and industrial applica- tions: A survey. The International Journal of Robotics Research, 37:688 716, 06 2018. doi:
10.1177/0278364918779999
Alvaro Sanzhez-Gonzzlez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4470-4479. PMLR, 10-15 Jul 2018. URL saacchez-gonalee21B.. html
ml www.prresssv...
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Proceedings of the 37th International Conference Machine Learning, pp. 8459-8468. PMLR, 2020.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1 20(1):61-80, 2009. doi: 10.1109/TNN 2WW.N.2008.0005560
Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology. 11765):33771134, 2021.
13

---

Published as a conference paper at ICLR 2023
Nicolas Soro, Laurence Brassart, Yunhui Chen, Martin Veidt, Hooyar Attar, and Matthew S Dargusch. Finite element analysis porous commercially pure titanium for biomedical implant application. Materials Science and Engineering: A, 725:43-50, 2018.
Eva Stanova, Gabriel Fedorko, Stanislav Kmet, Vieroslav Molnar, and Michal Fabian. Finite element analysis of spiral strands with different shapes subjected axial loads. Advances in engineering software, 83:45-58, 2015.
Priya Sundaresan, Jennifer Grannen, Brijen Thananjeyan, Ashwin Balakrishna, Michael Laskey, Kevin Stone, Joseph E. Gonzalez, and Ken Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9411-9415 2020. doi: 10. (0.1100/CCAA44445220000009999999
Priya Sundaresan, Rika Antonova, and Jeannette Bohg. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects. CoRR, abs/2204.03139, 2022. doi: 10. 10-48550/r2Xi.2224.31111 URL
200.04400000/21..22222.... 22w.ll...22...... 03139
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating eulerian fluid simulation with convolutional networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' pp. 3424 3433. JMLR.org, 2017.
Kiwon Um, Xiangyu Hu, and Nils Thuerey. Liquid splash modeling with neural networks. Computer Graphics Forum, 3718):71 2018. doi: https://doi.or//0..111111 wtt.//....//1////fff...oo 13522. URL https
//otlinelibrary wi Ley com/
10 http:/11111111111111 11w1/ 13522
13522
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference Learning Representations, 2020.
URL httt:s::s::// htttps::/oppnreeiieee net. forum? d=B1 DoJSYDH
RC Van Staden, Hong Guan, and Yew-Chaye Loo. Application of the finite element method in dental implant research. Computer methods in biomechanics and biomedical engineering, (4):2577-770 2006.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. ACM Trans. Graph. 38(5), oct 2019. ISSN 0730-0301. doi: 10.1145/332636.. URL http::ttt://oo.. wttp:....rg/..... 1145/3326662
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and An- drea Tacchetti. Visual interaction networks: Learning a physics simulator from video. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: htttps://proceeeeeee hww.eeeeedddinne..nee..ee www.ips..i.p.. cc/ pttpppper/201//E1e 8:ba005a556ccco4 ce F30 000aa000-00ccaccaaa--eee. pd
Zehang Weng, Fabian Paus, Anastasiia Varava, Hang Yin, Tamim Asfour, and Danica Kragic. Graph- based lask-specific prediction models for interactions between deformable and rigid objects. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5741- 5748, 2021. doi: 10.1109/IROS1 168.208222222 16.219966660021 URL
2103. 02932
Steffen Wiewel, Moritz Becher, and Nils Thuerey, Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow. Computer Graphics Forum, 2019, ISSN 1467-8659, doi: 10.1 (e111//fff.1330000
Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning Manipulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020a. doi: 10.15607/RSS.2025 200...20000.......3.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1 ():4))4444 24, 2020b.
14

---

Published as a conference paper at ICLR 2023
Longying Xiao, Gianluca Fiandaca, Bo Zhang, Esben Auken, and Anders Vest Christiansen. Fast 2.5 and 3d inversion transient electromagnetic surveys using the octree-based finite-llement method. Geophysics, 87(4):E267-E277, 2022.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempoGAN: A Temporally Coherent, Volu- metric GAN for Super-resolution Fluid Flow. ACM Transactions on Graphics (TOG), 37(4):95, 2018.
Mohd Hafiz Zawawi, Saleha, A Salwa, NH Hassan, Nazirul Mubin Zahari, Mohd Zakwan Ramli, and Zakaria Che Muda. A review: Fundamentals of computational fluid dynamics (cfd). In A/P conference proceedings, pp. 020252. AIP Publishing LLC, 2018.
Yao Zhang, Woong Je Sung, and Dimitri N. Mavris. Application convolutional neural network to predict airfoil lift coefficient. In 2018 ALAA/ASCE/AHS/SSC Structures, Structural Dynamics, and Materials Conference, 2018. doi: 10.251446.2018-1903. URL
/oi/abs/10.220 2514 6. 2018-1903
Qian- Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. CoRR, abs/ 80100987,, 2018. bhttt::/arxii. wrg./br///0//00/oo 09847
Olek C Zienkiewicz and Robert Leroy Taylor. The finite element method for solid and structural mechanics. Elsevier, 2005.
15

---

Published as a conference paper at ICLR 2023
g'
Encoder
Message Paosing Network
Deesder
X
Linear Node Embeddinn
Message Paesing Stack (Exx
Prediction
Message Passing Dluck
Presing Block
Presing Block
X
Lincar Edge
Buler Integration
Eimbalding
Figure 7: A detailed view of the GNN part of GGNS. Given a graph g', the node and edge features Xv. and X are linearly embedded into latent space and then updated with L Message Passing Blocks. The resulting predictions are interpreted as dynamic quantities that are used to update the system.
A MODEL DETAILS
The Message Passing Network employed by GGNS is displayed in As node-wise predictions we use velocities, which are Euler-integrated once to update the positions of the mesh of the deformable object.
B ENVIRONMENT DETAILS
Here, describe all key aspects, which are valid for all three environments. All datasets simulated using SOFA and include different material properties. Therefore, we choose discrete Poisson's ratios from E { (-0.9, 0.0, 0.49) for one-third of all simulated trajectories each. Other material parameters are kept constant, c.g., for the mass we choose large values for the solid object and smaller values for the deformable to ensure sufficient deformation. The chosen parameters do not represent the full reality, there are other material parameters that could be varied. However, as we want to showcase the capabilities of our method, we selected these parameters they displayed the biggest impact the deformation behavior.
B.1 POINT CLOUD GENERATION
The required point clouds are not directly available in SOFA, but instead rendered from the scene of the meshes using Raycasting from Open3D (Zhou et al. 2018). We therefore place virtual cameras around and on top of the scene to generate partial point clouds from different directions. For Deformable Plate dataset one camera is sufficient, while the other two tasks rely on four cameras around and one camera on top the scene. This results in a good, but not complete coverage of the entire surface with points of the point cloud. Even though there are five cameras around the scene, there are areas that are not covered: For the tissue, the parts that are occluded by the red liver, and for the cavity, parts of the inner surface depending one another. Also, as there can be no camera from below, there are naturally no points on the lower surface for both datasets. In Appendix D we additionally provide results for less cameras the cavity dataset, leading to only partially observable point clouds. If more than one point cloud camera used, the resulting point clouds are fused and subsampled accordingly to achieve processable number of points. We voxel subsample in world space, so the points do not belong to any specific part of the mesh, but can rather seen as some "interpolation" between mesh vertexes. The main challenge is that there are no point correspondences and that the model needs to figure out which point of the point cloud belongs to which vertex the mesh to do the correction of the mesh nodes for grounding the simulation. Still, voxel subsampling leads to the most structured results compared to other subsampling techniques, which helps the model to account for correspondences between points over time.
how the upper and lower radii deviates from
16

---

Published as a conference paper at ICLR 2023
B.2 INPUT FEATURES
In addition to encoding the node or edge type as one-hot features, we add an encoding to static nodes and encode the velocity of the collider in its node features. We encode the positions in space as relative features in the edges instead of absolute encodings in the node features following previous work (Sanche7-
al., 2020). All edges thus receive their relative world coordinates, while mesh edges additionally contain relative coordinates in mesh space.
B.3 COLLISION HANDLING
SOFA as the ground truth simulator handles collision between objects using triangular surface meshes of all objects involved to detect collisions. The detection is implemented using the LocalMinDistance method and detected collisions are included in the constraints of the system. Using Lagrangian multipliers, the constraints are then processed together with the other forces from the deformation to solve the complete FEM system (Faure et al. 2012). In contrast to that, GGNS uses one-hot encoded edges between the rigid and the soft body that are used by the model to compute the dynamics. There is no explicit handling of collisions, the network learns to avoid them and adapts the mesh accordingly.
B.4 DEFORMABLE PLATE
For this enviromment, we simulate a family of 2-dimensional trapezoids deformed by circular collider with constant velocity. We vary the size of the collider by sampling from triangular distribution between 15 and 60 of the edge length of the deformable object. For the collider start position we sample from a uniform distribution between the left and right corner of deformable object. We record 50 time steps per trajectory and 945 trajectories in total, which are split in 675 1755/35/11 135/135 135 trajectories per train, evaluation and test set. A single data sample contains approx. 700 nodes: 57 nodes for the collider, 81 nodes for the mesh oft the deformable object and around 600 points in the subsampled point cloud. The mesh itself consists of 416 edges, the total number of edges is about 3 K depending on the deformation in the according time step. In contrast to the Poisson's 's ratio, the other adjustable material parameter in SOFA, the Young's 's modulus kept constant for all samples at E = 5 000Pa. It describes the compressive stiffness when force is applied lengthwise. The different material properties together with the different trapezoidal shapes introduce uncertainty in the form of multi-modality into the data. The reason for this is that different deformations result in states that cannot be clearly assigned to a single trapez-materia combination. We construct this dataset because it comes with lower computational cost due to the restriction to 2d, but already allows for more general statements due to the non-trivial deformations and the malti-moddlity. Therefore, is especially suitable as proof-of-concepl and for ablations.
B.5 TISSUE MANIPULATION
Here, a piece of tissue deformed by rigid gripper which could be part of a robot-assisted surgery scenario. To generate diversity, we generate random motions in 2d plane and sample a random gripping point from the 19 top mesh points. We record 100 time steps per trajectory and 840 trajectories in total, which we split in 600/120) 20/120 trajectories per train, evaluation and test set. A single data sample consists of approx. 1 200 nodes: 361 for the mesh one for the gripper and about 850 for the point cloud. The mesh consists of 2 154 edges, which leads
total number of about 3 800 edges depending on the time step. To ensure physically plausible deformation, each Poisson's 's ratio is assigned its specific Young's modulus from E 010,00, 80 000, 30 000) Pa. If instead it were kept the same for each Poisson's 's ratio, the gripper could penetrate the deformable object or pull without touching it. The uncertainty in this dataset is mainly in the initial state, which can result in different deformations depending on the material from the same initial state.
B.6 CAVITY GRASPING
We randomly generate cone-shaped cavities with radii between 87.5% and 50% of the maximum possible gripping width. The cone shape helps to increase uncertainty in the form of multi-modality in the data, because the states resulting from deformation cannot be clearly assigned to single
17

---

Published as a conference paper at ICLR 2023
cone-material combination. The deformable cavities are deformed by gripper located random positions in space. The positions are sampled form a hexahedron around the geometrical center of the cavity ensuring collision free starting positions. For the grasping, the gripper moves quickly as is allowed to the gripping position and then closes its fingers with constant velocity. We record 100 time steps per trajectory and 840 trajectories in total, which train, evaluation and test set. A single data sample consists of approx. 2.4 K nodes: 750 for the mesh, 636 for the gripper and about number of edges in the graph is about 8.5 K depending on the exact time step. The motivation for the creation of this environment is that successful use of our method in this setting an important step on the way weal-world application.
split in 600 11/0112///// 120 trajectories per
for the point cloud. The mesh consists of 4 500 edges, the overall
18

---

Published as a conference paper at ICLR 2023
GGNS (k=5)
Ground Truth
MGN
Figure 8: Visualization of a test trajectory in the Tissue Manipulation dataset from three different viewing angles (rows) time step 70 for GGNS (left), the ground truth simulation (middle) and MGN (right).
C QUALITATIVE RESULTS
In addition the qualitative illustrations in the main paper, we also provide further views and examples here: Figure 8 shows the same trajectory as Figure Hu from three additional viewing angles. Figure 9 and Figure[O] show an overlay of point cloud on the deformable object during the time step where the simulation is grounded by the point cloud. This representation is comparable to Figure foo the Cavity Grasping dataset. Furthermore, we provide example visualizations for test rollout over time for the Deformable Plate task in Figure I1 for the Tissue Manipulation task in Figure
12 and for the Cavity Grasping in Figure [3 Throughout all tasks, GGNS closely matches the ground truth simulation for the complete rollout, achieving close optimal results when provided with frequent point cloud information (k: 2). Opposed to this, MGN sometimes fails to predict the correct material, leading poor predictions time and large mismatches the final system states.
19

---

Published as a conference paper at ICLR 2023
Figure Overlay of the point cloud and the predicted mesh for two consecutive time steps [10, 11] in the Deformable Plate dataset. We repeat the point cloud from the earlier simulation step in both images for clarity. The illustration shows the correction behavior of GGNS by including the point cloud to ground the mesh based simulation in this time step. This can be observed particularly well the upper left and right corners of the plate.
Figure 10: Overlay of the point cloud and the predicted mesh for two consecutive time steps t= [70, 71] the Tissue Manipulation dataset. We repeat the point cloud from the earlier simulation step in both images for clarity. The illustration shows the correction behavior of GGNS by including the point cloud to ground the mesh based simulation in the time step.
20

---

Published as a conference paper at ICLR 2023
1- 10
t-
t=
1 40
t- 50
Ground Trush
GONS-LSTM GGNS (l=22
[K-2)
GGNS (k-5)
(k--5) GONS LSTM
MGN
Figure 1I: Test rollout visualization for the Deformable Plate task. The last column depicts a close-up of the final time step, which is shown in full in the previous column. Here, additionally show qualitative results for the GGNS+LSTM model. We can see that for & = 2 it matches the ground truth quite well, while for k 5 large occurs due prediction of the wrong material.
21

---

Published as a conference paper at ICLR 2023
Ground Trush
GGNS(E2)
CONS (k=10)
MON
Figure 12: Test rollout visualization for the Tissue Manipulation task. The last column depicts
close-up of the final time step, which is shown in full in the previous column.
Ground Troth
GGNS GGNS (K-2)
GUNS (K-100)
000000
MON
Figure 13: Test rollout visualization for the Cavity Grasping task. The last column depicts a close-up of the final time step, which is shown in full in the previous column.
22

---

Published as a conference paper at ICLR 2023
Latent Size
MLP Layers
MP Blocks
Noise Std
Graph Connectivity
1-step MSE
10
10
g
50-step Rollout MSI
10-3
10
10->
10-6
64
128
256
0.0
0.0
0.0
0.1
Full Graph
Equal Radii
Reducod Graph
Figure 14: Performance for different changes in hyperparameter choices (grey) on the Deformable Plate dataset in comparison our default model (blue) with k: 1. Error bars indicate standard deviation. The top row shows the eгror for the next-step prediction, the bottom row that of full rollouts. We find that suitable noise scale is crucial for stable rollouts, and that more information in the form of additional edges between the different types of graph nodes generally improves performance. Given enough Message Passing (MP) blocks, further increases model capacity only lead modest improvements.
D ABLATIONS
D.1 HYPERPARAMETER CHOICES
Figure 14 compares the performance of GGNS for different hyperparameter choices. We find that the most importance parameters are the number of Message Passing (MP) blocks and the scale of the noise used in training. Both are crucial to achieve a good performance over multi-step rollouts. In terms of training noise, there is I-step/mulli-step loss trade-off. Other than that, approach is robust to variations of the different hyperparameters. that all settings achieve similar performance. Additional information in the form of more local edges helps slightly, while larger connectivity radii do not do much. A detailed listing of the used edge radii display in Table 2 In particular, the of significantly more edges in the Equal Radii setting does not provide significant advantage, which is why we use weaker connectivity Full Graph that saves computation time. The results for the Reduced Graph settings show that edges within the point cloud are not mandatory. For this reason, we omit these edges in the more complex 3d tasks in favor of shorter computation time.
terms of graph connectivity, can be seen
D.2 NOISY POINT CLOUDS
Besides the ablations on our hyperparameter choices, we present further ablations more realistic point cloud data. For this purpose, we use point clouds with additional noise and only partial observability to get closer to real world point clouds. Figure 15 shows the results for additional ablations on different scales noise on the point cloud data of the Deformable Plate dataset. We add noise to the point cloud positions during training, evaluation and testing. This makes it more difficult to infer the correct behavior from the point cloud, but provides a more realistic scenario for, because real world point clouds often exhibit large noise. The results show the robustness of
23

---

Published as a conference paper at ICLR 2023
10"
Max Sqarea Erte
10"4
0.05 MON MGN (M)
0,055
MGN MGN (M)
0.055
MON MGN (M)
Nisi Std
Nosie
Nilie
(a) 1/
(b) 2
(c) l:
Figure 15: Additional ablations for realistic point cloud data two datasets. Here, four different noise levels on the point cloud are evaluated on the Deformable Plate datset. Different grounding frequencies of k = 1 in (ap. k 2 in (b) and in (c) GGNS performs better than the baseline even when noise in the scale of the training noise of a = 0.01 is applied to the point cloud.
method: Even when a noise level of = 0.01 applied to the point cloud during testing, it clearly outperforms the baseline. This noise level corresponds to the amount of noise used on the mesh during training.
D.3 PARTIAL OBSERVABLE POINT CLOUDS
For the ablations on the partial observability, use the Cavity Grasping dataset. We generate the partial point clouds by using only one, two or five virtual point cloud cameras when using raycasting. The resulting point clouds are visualized for better clarity in Fig an example test trajectory at time step f = 0. One camera results in coverage from only one half of the outer surface of the cavity and two cameras cover almost the complete outer hull but not the inner surface. With five cameras, the point cloud covers almost the entire mesh completely, except for the inside and bottom. The resulting point clouds have a very different number of points: About 400 for one camera, about 600 for two cameras, and about 1000 for five cameras compared to 750 mesh nodes for the cavity, The results in Figure 16] show that even with these much less complete point clouds, GGNS still outperforms the baseline. For l this is the case if the baseline has access to the full initial state, which GGNS has not.
24

---

Published as a conference paper at ICLR 2023
10-1
10"
10
10
10
MGN
(a) & =
(b) k =
(c) k 10
Figure 16: Additional ablations for more realistic point cloud data on the Cavity Grasping dataset. For this purpose, different numbers of cameras used when generating the point cloud using raycasting. Comparison for three different grounding frequencies:k = in (a), k = 5 in (b) and k = 10 (c). GGNS outperforms the baseline for all camera setrings and grounding frequencies k.
One Camera
Two Camerus
Five Cameras
Mesh
Figure 17: Visualization the point clouds using one, two or five cameras for the raycasting and the corresponding mesh for reference. It clearly visible how better coverage of the object is achieved as the number of cameras increases.
Table 2: Edge radii for the connectivities between point clouds P and meshes M on the 2D Deformable Plate Dataset.
Setting
p P-P P
M - P
World
Full Graph
0.1
0.08
Equal Radii
0.2
Reduced Graph
0.0
0.08
MGN
0.0
0.0
0.35
25

---

Published as a conference paper at ICLR 2023
Intersetioon over Union
0.95
Intersettion over Union
0.95
0.9
0.9
0.85
0.85
0.8
0.8
GGNS
GGNS
(mat)
(mar)
(a) m + 10-step IoU Evaluation
(b) 772
10-step IoU Evaluation without initial mesh
Figure 18: (a) Comparison of model the baseline results on the Plate cataset using the 10- step Evaluation routine. (b) Results when using an initial mesh generated from the point cloud.
GGNS outperforms the MGN baseline even if has access the initial ground truth mesh.
10-2
GGNS GGNS GGNS MGNNM MIN
Lolop Miaz Sqparel Elnor
500-ate Mox Squard Exto
20
103
200
200
300
300
400
(a) Train loss
(b) 1-step Evaluation Loss
(c) 100-step Evaluation Loss
Figure 19: Exemplary learning curves for the Cavity Grasping task. The light shaded area indicates one standard deviation. Both GGNS and the basclines learn the task pretty similarly in terms of 1-step predictions. Our model only evaluated for the and variant during full rollout evaluation. Here, clearly the advantage of using the point cloud information.
10
Rollout Mean Squared Ervee
10
10-
10
10
MGN(IH) MGN(IH)
Plate
Tissue
Cavity
Figure 20: Comparison of the MGN baseline with version using the one-hot encoded edge types instead of an explicit edge type partitioning indicated by MGN (1H). Both are compared for all three tasks and no significant advantage of the explicit edges partitioning could found. For this reason, GGNS uses the one-hot encoding, because it is both conceptually simpler and requires less computational power. The MGN baseline still uses explicit edge type partitioning throughout this work, following Pfaff et al. (202T.
26

---

Published as a conference paper at ICLR 2023
Table Configuration of the hyperparameters and key information the training of our model for all experiments.
Parameter
Value
Batch Size
32
Optimizer
Adam
Learning Rate
x 10
Activation Function
LeakyReLU
Aggregation Function
Mean
Encoder
Linear Layer
MP-Blocks
MLP Layers
Latent Dimension
128
Decoder
1-layer MLP
Residuals Connections
Around each MP block
Training Noise Std
0.01
Table 4: Task specific configuration and hyperparameters for our experiments. We vary the graph connectivity and the number of training epochs for different tasks to control the total training time of our method.
Parameter
Plate
Tissue
Cavity
Connectivity Setting Number of Epochs
Full Graph
Reduced
Reduced
1000
800
400
Approx. Training Time
21 00 h
40 00 h
38 : 00 h
E HYPERPARAMETERS
Table 3 gives overview of hyperparameters shared across tasks. Since GNS generally robust to the choice of hyperparameters (c.f. [D). we use the same hyperparameters for all task and for both, GGNS and MGN for simplicity. The only hyperparameters that vary tasks the graph connectivity and the number training epochs, as shown in Table We adapt these parameters to control for the total training time on single GPU.
27


Reviews:
[{"review_id": "QWd91LxKZ2", "reviewer": "Reviewer_vQH9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the Grounding Graph Network Simulator (GGNS) that aims to improve the simulation quality by integrating point cloud sensor information for making predictions of the mesh state of deformable objects. An effective imputation training schema is proposed for leveraging the sensor information.  Experimental results demonstrate that the resulting model improves the simulation quality in several deformation prediction tasks, partially when the initial system state is incomplete.", "summary_of_the_review": "The reviewer tends to give acceptance to the paper given that it addressed the key issue of simulation by introducing a simple solution with additional sensory information.", "strength_and_weaknesses": "The paper is well presented and motivated. It highlights the imperfection of simulation exhibiting high error accumulation particularly when the initial system state might be incomplete. In response to this, the authors seek to add additional point cloud information to guide the simulation via simple graph combination or fusion and imputation-based training schema. Both quantitive and qualitative results verify the effectiveness of these simple designs.\n\n“Our method, however, omits this explicit representation in favor of a simple one-hot encoding of the type of input edge.”. The authors should have provided an explanation of why they prefer the one-hot encoding.\n\nMore insights should have been brought into the paper when the authors make a comparison against the recurrent imputation model. It is suggested to provide some visualization of the simulation results between the recurrent imputation model and GGNS.\n\nIt remains unclear to the reviewer the influence of the simulated result on downstream applications in model predictive control and model-based reinforcement learning, which seems to be an obvious limitation to demonstrate the model’s practical value.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LfjKpgSbniZ", "reviewer": "Reviewer_ZGJG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines graph neural networks with point cloud information to improve the performance of the popular graph network simulators. Standard graph network simulators accumulate prediction errors, causing predicted dynamic states to drift over time. The key idea in this paper is using point cloud information whenever available to counteract such drift and error accumulation. The paper provides three experiments involving 2D and 3D deformable objects to evaluate the proposed approach.", "summary_of_the_review": "The paper suggested an interesting direction, but I feel the technical method is not ready yet. I also have concerns with the experiments.", "strength_and_weaknesses": "**Strength**\nThe paper tackles a critical research problem: error accumulation from long-term prediction of graph neural simulators. Fusing sensor data into dynamic prediction is a reasonable strategy. The paper is also well-written and articulates its core idea clearly.\n\n**Weakness**\nRegarding the technical method:\nI think using point cloud to enhance dynamic prediction is a good direction to explore, but I feel the proposed method is not quite ready and the problem setup in this paper is a bit contrived: the point cloud data is generated from the ground truth mesh in a simulator and seems much denser than the mesh nodes (the DoFs in simulation). I would be much more excited if the paper could tackle noisy, sparse, partially observable point clouds (e.g., from meshes with self-occlusion) that are common in real-world 3D point clouds.\n\nOn a related note, I am trying to understand this paper from another perspective: Consider using the node positions from the ground-truth simulator as the point cloud input to this approach, i.e., once in a few frames, the method sees the ground-truth mesh node locations (are these exactly what we want the network to predict?) but without edge information. In this case, I am unsure whether predicting new states is still a challenging problem. Since the point cloud used in this paper seems denser, I guess the problem setup in the paper is even easier. Of course, the authors should feel free to correct me if I misunderstand something.\n\nRegarding the experiment:\n- The physics parameters used in the dataset are not very realistic (5000Pa Youngs modulus and {-0.9, 0, 0.49} Poisson’s ratio). A few more physics parameters, e.g., the density of the material and the size of the object, seem missing.\n- Fig. 4: Could you show the side view of this scene, e.g., seeing the deformable object from its left?\n- Overlaying the point cloud data on the deformable object in Fig. 1 is nice and informative. Is there a similar figure for the other two examples?\n- It looks like the point cloud is typically denser than the mesh nodes (600 vs. 81 in B.1 and 850 vs. 361 in B.2), so I guess “voxel-subsample” in the text describing the alpha-shape baseline means downsampling the point cloud to have similar node numbers as the mesh. Why not creating alpha shapes directly from the original point cloud?\n- Fig. 6 (b): Does k = 1 mean a point cloud is available in every time step?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ec42Mw-dcxj", "reviewer": "Reviewer_GZAL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the grounding graph network simulator (GGNS), which extends the previous graph network simulator by using auxiliary point cloud data to simulate dynamics of deformable objects with incomplete initial system states. ", "summary_of_the_review": "Although novelty of this may need to be further justified, the paper introduces several interesting ideas for grounding graph network simulators that can simulate soft objects.", "strength_and_weaknesses": "+ The studied problem on simulating deformable objects is important for robot manipulation.\n+ The idea of applying the imputation-based training scheme seems to be a promising solution to address the problem of incomplete initial system states.\n\nWeakness:\n\n- Technical novelty of the paper is marginal. For example:\n-- The complicated dynamics of the deformable mesh seems to be addressed by the previous GNS method, which is inherited by the proposed GGNS. \n-- Although it seems to be new for graph network simulators, the implementation of the imputation-based training scheme is straightforward: the proposed approach basically learns two graph networks with and without auxiliary point cloud data, and optimizes them to have the probability during training. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster