Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a differentiable contact model and demonstrates its compatibility with deep learning frameworks and its usefulness in downstream applications. The proposed contact model is similar to that in MuJoCo but with several enhancements. The final contact model formulates the task of solving contact forces as a convex optimization problem and is differentiable. The authors demonstrate the usefulness of their proposed contact model in several standard 2D and 3D examples.
main_review: Differentiable contact models are not new to the ML/robotics/graphics communities, and previous works have proposed quite a few candidates. One of my biggest concerns for this paper is that it focuses on showing the performance of the proposed contact model but without comparing it to other differentiable contact models from previous papers. The authors are aware of the differentiable contact model presented in [35] but did not seem to comment [35] in their related work section. The original “hit” example in [35], which this paper got inspiration from (line 326), is a 3D deformable ball bounces between walls and grounds more than once, which is much more sophisticated than the two examples this paper presented in Fig. 7. Another more recent, state-of-the-art differentiable contact model “Incremental Potential Contact: Intersection- and Inversion-free Large Deformation Dynamics” (IPC), is missing in this paper’s related work but I believe IPC deserves a careful discussion. I do agree this submission has some unique contributions in its machine learning experiments which are not covered by [35] or IPC, which is also why I choose to give a positive score. However, I think it should also pay more attention to other existing differentiable contact models and position its contributions more properly.
limitations_and_societal_impact: From what I understand, developing contact models typically focuses on one of the two goals: it either aims at physical accuracy and targets at real-world applications, or focuses on speed/robustness/simplicity and considers animation/learning problems as its main application. The position of the contact model presented in this work is not very clear to me: it is definitely more physically plausible than some of the simplest contact models in robotics/graphics/ML, but not accurate enough for a real-world application. I look forward to seeing a clarification and, even better, a real-world example in their future work.
 
Another limitation in this work is that a comprehensive discussion about the role of collision detection is missing. However, I notice that the authors are aware of this and consider it as future work, which I think is fair, so I won’t hold it against them.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces Lagrangian/Hamiltonian neural networks for simulating systems with frictional contacts. Previous Constrained Hamiltonian Neural Networks (CHNN) are able to learn constrained motion while this paper extends to the case of constrained motions with frictional contacts. A first phase leverages the NeuralODE framework to simulate in a fully differentiable way, the constrained motion associated with the learnt Lagrangian/Hamiltonian. Contacts are handled by incorporating an additional contact-solving phase solved using differentiable optimization layers (cvxpylayers).  The approach is benchmarked against an MLP baseline and an Interaction-Network on the task of learning a differentiable simulator from data. The learnt differentiable simulator is then applied to control tasks and compared to the DIffTaichi simulator. 
main_review: A consequence of the use of CHNN is the need for using Cartesian coordinates instead of generalized ones. This results in soft joint constraints when considering multi-body systems in robotics and also requires solving artificially higher dimension problems. Moreover, the proposed approach does not account for external or control forces  and such efforts could make it more challenging to enforce constraints. In order to consider control applications, would it still be possible for CHNN to learn the dynamics of a system under these forces? 

The Signorini condition which enforces the complementarity between normal contact forces and velocities accounts for the non-inter penetrability between objects [The non-smooth contact dynamics method, Michel Jean 1999]. This condition is necessary to simulate rigid bodies and is ignored in this paper. The same approximation is done in the simulator Mujoco [Mujoco: a physics engine for model-based control, Todorov et al. 2012] and leads to simulation artefacts such as objects interacting even when they are not in contact. Did you observe similar effects? Could the Authors report on it?

Does the SCS solver have convergence guarantees for positive semi-definite problems (case when the A matrix has a null eigenvalue)? This is a common case in robotics that corresponds to over-constrained problems and one should avoid using a regularization matrix R as it leads to less precise solutions.

Experiments on dynamics and parameters learning demonstrate the superiority of the method when compared to a simple MLP or a more sophisticated Interaction-Networks. However, a comparison with a differentiable simulator such as gradSim [gradSim: differentiable simulation for system identification and visuomotor control, Murthy et al. 2021] would be interesting because the approach should be more competitive than other baselines (as shown by experiments from 5.). Durations considered during the evaluation of energy conservation are short when compared to the ones involved in robotic applications. Is this property still valid for longer time horizons? 

Concerning the experiment on robustness, why were the 3D examples generated from an LCP simulator and not with one accounting for the cone constraint? From my understanding, the learnt friction parameter does not have the same meaning as the original one because it corresponds to different friction models which makes it difficult to compare them [Differentiable simulation for physical system identification, Le Lidec et al. 2021]. 

A billiard task is considered to evaluate the approach and compare it against the DiffTaichi differentiable simulator. This simulator includes gradients from the collision detection routine while it is not clear if this information is available in the collision detection module used in the approach proposed in the paper. I would expect this information to be important to solve a problem where one needs to reason about contact points as is the case in the billiard task. Could it be the reason behind the better performance of Difftaichi on this task? Did you consider different initial guesses or targets to assert the robustness of the approach on this task?

limitations_and_societal_impact: Overall, the limitations as presented by the Authors seems a bit limited.
Indeed, the Authors did not discuss for instance the fact they did consider complementarity between motion and forces at the contact level.
Additionally, they clearly did not discuss the fact they did a mix between acceleration-based simulation and time-stepping methods which will compute a corrective impulse at velocity level. What is the physical and mathematical meaning of such a choice?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a differentiable contact model, whose unknown parameters (restitution and friction coefficients) can be learned together with other system parameters using Hamiltonian and/or Lagrangian neural networks. The proposed contact model consists of two constrained convex optimization problems, which are made differentiable by the recently proposed differentiable convex optimization layers. Variants of the proposed methods outperform baselines on multiple 2D and 3D physical systems. The learned dynamics model can also be used for downstream planning and control tasks.
main_review: Originality: The proposed method is a novel combination of the two-phase contact model, the differentiable convex optimization layers and the Hamiltonian & Lagrangian neural networks. Related works are sufficiently discussed.

Quality:
The paper is technically sound, and the empirical evaluation is thorough. Some weaknesses are listed below:
- The choice of the target velocity $v_C^*$ to fix penetration seems to require a lot of engineering and not very principled. It is also unclear how sensitive the performance is to the choice of the target velocities, as it is not discussed in the experiments.
- One of the improvements made from the previous submission is that the authors fixed an error in the code base. However, the code is still not included in this submission. 

Clarity: The paper is generally well-written. Some improvements can be made on the presentation of the proposed method. For example, an algorithm box outlining the training procedure would help. Also, the current introduction repeats part of the related work section, and can be made cleaner.

Significance: The contributions of this paper are incremental and moderately significant. The proposed method outperforms the baselines on prediction error and energy conservation, but also requires a known collision detection module and the nontrivial choice of the target velocity. In general, it doesn’t seem to have a clear advantage over existing methods, but can still be a meaningful contribution.

==== Post rebuttal ====

My concerns are well addressed by the author response. I have increased the score to 7.

limitations_and_societal_impact: I think the authors have adequately addressed the limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper adds explicit contact modelling to lagrangian/hamiltonian neural networks.
Specifically, they add a contact resolution step to CHNN using an impulse-based micro collision maximum dissipation contact model inspired by the contact model used in MuJoCo.
They also handle elastic collisions with a 2 phase collision based on poisson restitution.
Each phase of contact resolution are convex programs and CvxpyLayers is leveraged to compute analytic gradients through them.
Experimental evaluations are performed to test the proposed models ability to infer physical parameters from trajectories and predict future trajectories.
Two small experiments are also included to show the capability of the model to be used for control.

main_review: Strengths:
1. The choices of the contact model are inspired by MuJoCo and novel for differentiable rigid body physics models and most other methods typically don’t handle elastic collisions.
2. Method is thoroughly detailed in the appendix.
3. Video shows qualitative results on the various toy tasks.

Issues:
1. The only test data for comparing trajectory error is generated using the proposed model itself. In this case there is zero model mismatch. An additional prediction task using real world data (ideally) or trajectories generated from a separate physics simulator should be used to compare the method against baseline.
2. It would be interesting to extend Figure 5 to see when/if MLP and IN baseline could catch up to the proposed method (if the CM method is too expensive at least the baselines could be extended).
3. This is cleared up in the appendix but it would be clearer if the notation used in Algorithm1 ($\Delta v$) were explicitly related in the section 3 description.
4. In the appendix S.10 shows $f_E$ can be expressed in terms of $f_C$ but doesn’t $f_E$ depend also on impulse from unconstrained dynamics f.e. acceleration due to gravity (the Mujoco paper includes this as “s” term)?
5. Some details on the IN and MLP baselines are missing. What features are being passed to them?
6. The regularized version CMr is briefly introduced and not discussed too much. In which scenarios would the regularization be important?
7. I understand that the LCP baseline was infeasible due to nan issues. Another possible baseline which should be comparatively stable is soft contact model.
8. I believe [34] do test energy conservation for their contact model somewhat.

Overall:
I think the paper provides a useful contribution to differentiable rigid body simulation which has seen a lot of interest. A new differentiable contact model is proposed and evaluated experimentally. My main issue is that comparisons and prediction errors are only evaluated on data generated by the proposed model itself.
limitations_and_societal_impact: Perhaps the authors could elaborate on what cases the contact model could be inaccurate/unstable for. For example, would you expect any issues around resting contacts, large numbers of simultaneous contacts etc.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper propose a novel method for learning simultaneously the properties of dynamics and contacts.
This method is implemented by a combination between C[HL]NN, Differentiable Convex Optimization Layers
and neural ODE. The main contribution is the ability to model frictional contacts both with or without elasticity.

The experiments demonstrate the superiority of the proposed method over the relevant baselines.



main_review: originality:

the proposed combination between the existing frameworks (C[HL]NN, Differentiable Convex Optimization Layers
and neural ODE) for handling contacts with friction is novel 

quality:
the main claims of the paper are demonstrated to be true, and the method/model is reasonable.


clarity:
I strongly recommend to add a description of the entire method by a block diagram/picture/pseudo code.
Also, frequent jumping between the paper and the appendices affects the readability. 

significance:
even though the method is original, it is limited to particular classes of contact dynamics, and/or dimensionality of dynamics.
Specifically, there is no improvement in "Rope" environment, which indicates that the proposed solution is not general and/or scalable.  Consequently,  it is unclear when this method improves the baselines. Does it work only with a small number of contacts ( <= 8)? What is the main limitation? Does it perform better with more training data and/or larger networks? E.g., Table 3 might shed light on this issue, if the corresponding errors (as in Fig2+Fig3) would be added. Addressing this concern and providing a conclusion will improve the score. 


post rebuttal: following the clarifications and improvements by the authors I change my score to 7.


limitations_and_societal_impact: the authors adequately addressed the limitations.
review_title: null

Meta-review: This paper proposes a differentiable contact model that can be used together with previously proposed Lagrangian and Hamiltonian neural networks, enabling gradient-based system identification and model-based control for hybrid dynamical systems. All reviewers recommend acceptance and by the end of the rebuttal process became satisfied with the additional experiments and clarifications by the authors. Some limitations on the scalability of the method in contact-rich (and high-dimensional state) scenarios were also illustrated through additional experiments, and reviewers were satisfied. I am recommending this paper for acceptance.    
Decision: Accept (Poster)