Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper empirically exams the question of whether graph neural newtworks can learn discoutinous, rigid contact dynamics. Specifically, the GNS [1] architecture is studied on the following discountinous contact dynamcis: tossing a cube to the ground in simulation (with different stiffness parameters), tossing a cube to the table in the real world, and tossing objects with different geometris to the ground in simulation. The experimental results show clear constrast to previous conclusions that it is difficult for GNNs to learn such rigid contact dynamics, instead, learned GNN models can perform well on such dynamcis, even better than tuned analytic physics simulators.   

[1] Sanchez-Gonzalez, et al, Learning to Simulate Complex Physics with Graph Networks, ICML 2020
strengths_and_weaknesses: Strength:
- The question itself is interesting and the results help shed more light into the anwser -- GNN can indeed learn discountinous contact dynamcis well on the tested tossing tasks. 
- The ablation studies are complete and provide good understanding of which are the important factors for GNNs to perform well.

Weakness:
- My major concern is: is the result really surprising and makes enough contribution to a CoRL paper? In the original DPI [1] paper, the learned GNN model already demonstrates correct behaviour (i.e., non penetration and correct predicted cube pose) on modeling the contact interaction between the rigid cube and the water particles (the BoxBath experiment); and in the original GNS [2] paper, the GNN model also demonstrates correct behaviour (i.e., non penetration and correct predicted particle positions) between particles colliding with environment obstacles (the WaterRamps experiment). Although these are not strictly rigid objects colliding with another rigid object, but I would not be surpised that the GNN model should be good to handle the non-peneration part of rigid-to-rigid collision based on the results from these previous papers. With this being said, these previous papers did not study if GNN can learn the sensitivity to initial conditions for rigid-to-rigid collision, which is clearly demonstrated in this paper. But with this conclusion alone, I am not entirely sure it is enough contribution for a CoRL paper.
- Also, most of the experiments considers tossing a rigid object into another. There are many more rigid contact dynamics other than this, e.g., a quadruped robot doing locomotion on the ground, a robotic hand manipulating a rigid object. Conducting some experiments in these more complicated senarios would make the result more convincing and stronger. 
- Another minor question: regarding figure 4, how is the sysID done for the physics simulators? More specifically, how many real-world trajectories are used to tune the simulation parameters? I would imagine with more real-world data used, the parameters can be tuned more accurate, i.e., the performance of the simulator in figure 4 should also be a curve instead of a horizontal line. More details to clarify this would be appreciated.

[1] Li, et. al., Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids, ICLR 2019
[2] Sanchez-Gonzalez, et al, Learning to Simulate Complex Physics with Graph Networks, ICML 2020
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Please see the strength/weakness section above.
issues: Please see the strength/weakness section above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to use a Graph Neural Network (GNN), similar to the mesh-based graph network proposed in [3], to learn rigid-body dynamics with contact in 3D. Following [3], each mesh vertex of the rigid body is modeled as one node of GNN, and each edge of the mesh as one edge of GNN. The authors argue that using spatial-equivariant relative displacement between nodes, as in [3], and predicting acceleration rather than velocity (same as [3] in the cloth case) are important to achieve high accuracy. The method is validated both with simulated datasets [10] and a real dataset [2], and outperforms recent baselines.
strengths_and_weaknesses: The strengths and weaknesses of the paper are both quite clear.

Strength: 

- Simple algorithm and network structure, without heavy engineering and tuning.

- Good results on both simulated and real datasets that outperforms baseline learning methods.

Weaknesses:

- Lack of novelty: not really a weakness in my opinion, if GNN works well already out of the box, that is worth sharing with the community. But I have to say the method and implementation details are both quite similar to [3].

- Lack of insight: It’s hard for me to intuitively understand why GNN works much better than ContactNet and naive learning methods using e.g. GRUs. There is a diagnostic example (Sec. 4.1) but it does not provide hypothesis or theories why GNN is better at modeling discontinuities than *other neural nets* either.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: “GNNs are better at modeling discontinuities than other NNs” is certainly an unconventional, and to me counterintuitive, claim. As such, since the paper do not provide much insight on why, and the testing datasets are both relatively small, I’m a bit worried about whether findings from this paper can be applied to more complex rigid-body learning problems, or larger datasets.

That being said, the results seem solid, and I would like to see these results published if the following issues can be addressed during rebuttal.
issues: - L477 we “expect higher errors for later timesteps due to error accumulation”: could the authors share the final-frame errors in addition to average errors for ContactNet dataset? It’s okay if the improvement becomes less obvious compared with “those heavily tuned analytical simulators“.

- L244: the ablation study is odd to me. GNS[abs] shows that using relative displacement between nodes, *and* predicting acceleration rather than velocity are important, since it is much worse than other ablations.

   Could the authors split this important ablation into two? That is, what if we use relative displacement as input features but for output still predicts velocity/displacement, instead of acceleration? In [3], predicting acceleration and use it in second-order integration, instead of first-order seem to be an optional feature that can be easily turned on or off, which should be the same for this paper.

- Could the authors also plot the accuracy metrics on training trajectories of ContactNet (i.e. training losses), in the format of Figure 3?

- Is the Gaussian random noise (L463) identical to [16], or actually more similar to [3] which modifies upon [16]?

- Since the paper does not have much theory or insight but only demonstrates good performance, for reviewers to trust the results, could the authors promise to publish their code if the paper is accepted?

- If I understand correctly, Figure 3 (c) essentially says that if we know the initial state of a “dice”, this GNN can almost always predict correctly which face it lands on the table (otherwise rotational error would be quite large for that trajectory?), given the chaotic nature of a dice, could the authors share some insights on why you think this is not too good to be true?

- Typo in abstract: Line 5 continuous -> discontinuous?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a learning-based approach to predicting object dynamics using graph neural networks. Object meshes are encoded using the finite difference of the mesh vertex velocity (over a short history) and distance to a boundary (e.g. floor), and edge features are relative displacements between nodes. Message passing is used to update node and edge features and the network predicts finite difference accelerations for every node. Extensive experiments are provided on simulated and real datasets of simple objects being tossed, and comparisons are made to both learned baselines and common physics engines used in robotics. The results suggest the graph network approach outperforms these baselines in terms of accuracy of predictions with relatively little training data.
strengths_and_weaknesses: Strengths: The paper was clearly written and enjoyable to read. The representation of meshes in the graph network is intuitive, and Figure 2 was quite helpful in quickly understanding what was being proposed. The experiments are very thorough in that they compare to a variety of relevant baselines on a variety of datasets. The expected ablations on the proposed method are performed and provide some insight on the necessity for the different components incorporated in training the GNN. The authors identify many of the relevant shortcomings of the method including making predictions for more complex objects and multi-object scenarios.

Weaknesses: While the paper is relevant to robotics, there is minimal involvement of robots in the paper. Even the real-world dataset used from the ContactNet paper appears to just have a human tossing the cube. I find this limiting for robotics because then there is no action modality associated with any of the data, meaning it will be hard to learn predictive models for planning and control. There is also limited discussion on uncertainty and stochasticity in the dynamics, which is particularly important for tossing objects. For example, tossing the object 10 times in the exact same way will have a different outcome all 10 times, i.e. they'll vary at least a little bit due to slight variations in the execution and minute differences in the interactions with the environment. These sorts of dynamic manipulations are very sensitive to these subtle differences and can sometimes produce drastic differences in outcomes, particularly for more dynamic manipulations like dropping from higher heights, e.g. an object dropped on a table might bounce and come to rest on the table or fall off onto the floor, even when dropped from the same height in the same way. These dynamics are not fully predictable without infinite modeling precision, which is of course not possible. So it seems important to address matters of uncertainty in the predictions, and is worth discussing in more depth in the paper. This is also related to the first point about involving robots, one can control the actions to be more repeatable using a robot, which should make quantifying the uncertainty more feasible on a real-world setup, instead of having a human tossing objects for data collection. I am also curious how the approach can be applied to partial-view models, which is a very common situation in robot manipulation. The robot may only have a partial-view point cloud of the object and not have access to a ground truth mesh.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Making accurate predictions for object dynamics is undoubtedly relevant to robotics, and the paper does a very good job providing convincing evidence for their approach over learned baselines and common physics engines in robotics. My only reservation is that the explicit connection to robotics is tenuous. For this method to be useful for a robot, it needs to connect to the robot's action space, and none of the datasets have information about an action modality so far as I can tell. That being said, I think this paper motivates the use of a straightforward GNN architecture for simulating object dynamics, and I believe it could be a very useful tool for roboticists to build off of, which is why I'm strongly advocating for acceptance.
issues: I would like for the authors to elaborate on the role of uncertainty in the model predictions, as I mentioned in the Strengths/Weaknesses section above. In particular that there is some limit to how accurate the predictions can be due to stochasticity from the environment, and this will be exacerbated in the real world since there is also the added uncertainty from noisy and incomplete observations. 

Here are some very minor issues:
Lines 108-113: It would be good to clearly state here how to interpret the "node"/"edge" terminology with respect to a mesh, presumably that a node is a vertex of the mesh and an edge between them means they are part of the same face?
Line 265: "models involving rigid and deformable objects." (make rigid non-plural)
Line 295: Is "rigids" a term of art? It sounds awkward to me, I would go with "rigid objects".
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper investigates the applicability of general-purpose graph neural network simulators for modelling discontinuous dynamics. 

Similar to existing work, the proposed method learns to predict the dynamics of a given mesh using a graph neural network that is trained with a one-step prediction loss. The resulting model is evaluated on different simulated and real-world datasets, performing on par or better than commonly used simulators, even when trained on relatively few samples. This result contradicts the previously held notion that this kind of learned simulator is unable to model discontinuous dynamics, and thus opens various avenues for further research.
strengths_and_weaknesses: ### Strengths

- The paper shows that general-purpose graph neural network simulators for can be used to model simple discontinuous dynamics. More precisely, it shows that a slight variation of the previously proposed MeshGraphNet [1] architecture can accurately model simple rigid contact dynamics, up to a precision that matches and sometimes surpasses that of commonly used solvers. This is an interesting finding, as it contradicts the previous believe that learned general-purpose simulators are ill-suited to model discontinuous dynamics.
- The MeshGraphNet architecture is adapted to rigid body simulation by employing a shape-matching projection during the rollout phase. While the performance improvement from this additional step is minor in the measured metrics, this shape-matching is interesting as it ensures that the simulated rigid bodies can not deform over the course of a rollout.
- The experiments are thorough, and both model architecture and chosen training hyperparameters are clearly explained in the text. Experiments are repeated for $10$ seeds, and confidence intervals are reported. Different parts of the proposed method are explored through a series of ablations.
- The paper has a clear story and is well written. The figures are clean and complement the text nicely.

[1] Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A., & Battaglia, P. (2020, September). Learning Mesh-Based Simulation with Graph Networks. In *International Conference on Learning Representations*

### Weaknesses

- While the results are interesting and promising, the proposed method heavily builds on previous work thus lacks novelty. Section 3.1. essentially re-introduces [1] with some minor changes here and there.
- The experiments are rather limited in scope. While they shows a clear proof of concept, it would be interesting to see how well the method works for more complex shapes.
- Very few baselines are considered in the experimental section, making it hard to directly compare the approach to existing work. This is somewhat alleviated through various ablations and comparisons to classical solvers.
- While most design decisions are ablated over, some interesting deviations from the MeshGraphNet paper remain unexplained. For example the message passing block is iterated over $K$ times, where $K=1$ in previous work.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents and evaluates the use of general-purposes graph neural network simulators for modelling discontinuous dynamics. It does not present much in terms of novel architectures and only considers relatively simple tasks with few baselines.

Despite these shortcomings, the paper highlights an interesting new application for graph network simulators and supports its claims with an overall thorough experimental section and various ablations.

Put together, I currently vote to accept the paper since it gives a clear new insights in a field that is not yet well understood.
issues: - As mentioned above, Section 3.1 is very similar to the MeshGraphNet paper. Clearly separating the basic MeshGraphNet architecture and the changes/adaptations made in this paper would make it easier to distinguish previous work from the contributions of this one.
- An additional paragraph explaining these adaptations would make them easier to understand. For example, it is not quite clear why the message passing block is iterated over $K$ times per update (and what $K$ is set to, as $K=1$ in previous work), or why a history of the $h=2$ previous meshes is used instead of $h=1$ in previous work.
- It should be clarified why some of the real-world tosses were removed from the dataset (as mentioned in Appendix A).
- In the appendix, it is shown that data augmentation hurts performance when training on a lot of demonstrations. An additional sentence or two on why this may be the case would be useful for understanding and validating the choice of data augmentation.
- The last paragraph of Section 5 before the `Limitations' hypothesizes why the used graph network simulators are able to model discontinuous dynamics. This hypothesis should not be stated as fact, as it is not directly proven in the paper.
- In Figure 4, it would be interesting what the penetration value is for the other solvers.
- While the experiments are generally convincing, they lack strong baselines and comparison to previous work. It would be interesting to see how the used GRU and ContactNets baselines perform on their respective other tasks. A baseline that considers velocity instead of acceleration prediction could also be useful. These baselines should also be considered for the MOVi-A dataset that showcases the behavior for the more complex shapes.

### Typos and Style

- l. 26 has two consecutive `and's
- In ll. 129-131, `position coding' is introduced. The term `positional encoding' is probably more common.
- l. 209 the `)' should come before the citation
review_title: null

Meta-review: metareview: This paper provides empirical studies to show that a GNN architecture can provide better contact behavior modeling for rigid bodies than other deep learning architectures and, sometimes, can be better than physics-based methods.

All the reviewers agree that the paper poses a powerful and interesting hypothesis, and showed convincing evidence that it is true, and agree that the work is highly relevant to robotics. Many of the reviewers praise the clarity of the presentation and thoroughness of the empirical studies.  Several reviewers appreciate the simplicity of the architecture.  The reviewers also noted some concerns, but the authors addressed these concerns well in their rebuttal.  However, the authors are highly encouraged to open source their code for the final paper for the sake of reproducibility, and to encourage other researchers to build on their work. 

In summary, this work is a welcome addition to CoRL, and will give valuable insights into the potential for GNN-based contact physics modeling in robotics in the future. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)