Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the grounding graph network simulator (GGNS), which extends the previous graph network simulator by using auxiliary point cloud data to simulate dynamics of deformable objects with incomplete initial system states. 
strength_and_weaknesses: + The studied problem on simulating deformable objects is important for robot manipulation.
+ The idea of applying the imputation-based training scheme seems to be a promising solution to address the problem of incomplete initial system states.

Weakness:

- Technical novelty of the paper is marginal. For example:
-- The complicated dynamics of the deformable mesh seems to be addressed by the previous GNS method, which is inherited by the proposed GGNS. 
-- Although it seems to be new for graph network simulators, the implementation of the imputation-based training scheme is straightforward: the proposed approach basically learns two graph networks with and without auxiliary point cloud data, and optimizes them to have the probability during training. 

clarity,_quality,_novelty_and_reproducibility: This paper seems to provide an original work for grounding graph network simulators, but the work may also be incremental. The paper is well written, and the approach is relatively well explained. 
summary_of_the_review: Although novelty of this may need to be further justified, the paper introduces several interesting ideas for grounding graph network simulators that can simulate soft objects.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes the Grounding Graph Network Simulator (GGNS) that aims to improve the simulation quality by integrating point cloud sensor information for making predictions of the mesh state of deformable objects. An effective imputation training schema is proposed for leveraging the sensor information.  Experimental results demonstrate that the resulting model improves the simulation quality in several deformation prediction tasks, partially when the initial system state is incomplete.
strength_and_weaknesses: The paper is well presented and motivated. It highlights the imperfection of simulation exhibiting high error accumulation particularly when the initial system state might be incomplete. In response to this, the authors seek to add additional point cloud information to guide the simulation via simple graph combination or fusion and imputation-based training schema. Both quantitive and qualitative results verify the effectiveness of these simple designs.

“Our method, however, omits this explicit representation in favor of a simple one-hot encoding of the type of input edge.”. The authors should have provided an explanation of why they prefer the one-hot encoding.

More insights should have been brought into the paper when the authors make a comparison against the recurrent imputation model. It is suggested to provide some visualization of the simulation results between the recurrent imputation model and GGNS.

It remains unclear to the reviewer the influence of the simulated result on downstream applications in model predictive control and model-based reinforcement learning, which seems to be an obvious limitation to demonstrate the model’s practical value.

clarity,_quality,_novelty_and_reproducibility: The authors conducted extensive evaluations across multiple tasks covering Deformable Plate,  Tissue Manipulation, and Cavity Grasping. 

The results support the effectiveness of the proposed model.
summary_of_the_review: The reviewer tends to give acceptance to the paper given that it addressed the key issue of simulation by introducing a simple solution with additional sensory information.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines graph neural networks with point cloud information to improve the performance of the popular graph network simulators. Standard graph network simulators accumulate prediction errors, causing predicted dynamic states to drift over time. The key idea in this paper is using point cloud information whenever available to counteract such drift and error accumulation. The paper provides three experiments involving 2D and 3D deformable objects to evaluate the proposed approach.
strength_and_weaknesses: **Strength**
The paper tackles a critical research problem: error accumulation from long-term prediction of graph neural simulators. Fusing sensor data into dynamic prediction is a reasonable strategy. The paper is also well-written and articulates its core idea clearly.

**Weakness**
Regarding the technical method:
I think using point cloud to enhance dynamic prediction is a good direction to explore, but I feel the proposed method is not quite ready and the problem setup in this paper is a bit contrived: the point cloud data is generated from the ground truth mesh in a simulator and seems much denser than the mesh nodes (the DoFs in simulation). I would be much more excited if the paper could tackle noisy, sparse, partially observable point clouds (e.g., from meshes with self-occlusion) that are common in real-world 3D point clouds.

On a related note, I am trying to understand this paper from another perspective: Consider using the node positions from the ground-truth simulator as the point cloud input to this approach, i.e., once in a few frames, the method sees the ground-truth mesh node locations (are these exactly what we want the network to predict?) but without edge information. In this case, I am unsure whether predicting new states is still a challenging problem. Since the point cloud used in this paper seems denser, I guess the problem setup in the paper is even easier. Of course, the authors should feel free to correct me if I misunderstand something.

Regarding the experiment:
- The physics parameters used in the dataset are not very realistic (5000Pa Youngs modulus and {-0.9, 0, 0.49} Poisson’s ratio). A few more physics parameters, e.g., the density of the material and the size of the object, seem missing.
- Fig. 4: Could you show the side view of this scene, e.g., seeing the deformable object from its left?
- Overlaying the point cloud data on the deformable object in Fig. 1 is nice and informative. Is there a similar figure for the other two examples?
- It looks like the point cloud is typically denser than the mesh nodes (600 vs. 81 in B.1 and 850 vs. 361 in B.2), so I guess “voxel-subsample” in the text describing the alpha-shape baseline means downsampling the point cloud to have similar node numbers as the mesh. Why not creating alpha shapes directly from the original point cloud?
- Fig. 6 (b): Does k = 1 mean a point cloud is available in every time step?

clarity,_quality,_novelty_and_reproducibility: I think the paper is easy to follow with most of the low-level details properly left out in the supplemental materials.

Fig. 1 was a bit confusing when I read the paper for the first time, but most of it became understandable after reading the corresponding sections:
- It is still unclear to me what solid and dashed lines in Fig. 1 attempt to imply, although it is not a very big deal.
- Why is there an inference arrow pointing from GNN outputs back to “State at Step t”?

The technical method does not seem complicated to implement. Therefore, I think the paper is probably reproducible. It could be better if the authors could release the code.

One thing that is unclear to me is how the collisions between the rigid and soft bodies are handled in their experiments, and it would be nice if the supplemental materials could include more detailed descriptions about it.

summary_of_the_review: The paper suggested an interesting direction, but I feel the technical method is not ready yet. I also have concerns with the experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers unanimously recommend acceptance (6-8-6). The author-reviewer discussion was productive and the authors implemented many improvements to address the reviewers' concerns. No major concerns remain. The paper is ready for publication.
justification_for_why_not_higher_score: This work is solid and well-executed. However, I wouldn't recommend a higher score as it is not groundbreaking either -- it is an extension of the Graph Network Simulator framework.
justification_for_why_not_lower_score: Technically solid. All reviewers' concerns have been addressed.
Decision: Accept: poster