Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ============ Update ===========
The authors have done a good job at addressing my concerns, and the revised version of the submission is substantially improved. I have adjusted my score and recommend accepting the paper.
==============================

Recent work has explored encoding analytic mechanics formulae into neural networks as inductive biases to learn physics models that generalize better. Neural networks are implemented to learn quantities like kinetic and potential energy rather coordinate derivatives. In this paper, the work of [1] is extended to incorporate a more generalizable approach to modeling functions on angles, integral approach where errors are backpropagated through an ODE solver rather than fitting errors in the derivatives, and modeling response to controls. 

For Lagrangian and Hamiltonian systems it’s often easier to work with non-euclidean generalized coordinates rather than a constrained Euclidean system, however it can be difficult to design well parametrized neural network functions on a manifold like a circle. The authors address this by still expressing the having the Hamiltonian expressed in terms of circular generalized coordinates, but parametrizing the functions on the Euclidean embeddings. The paper shows that this approach does not have a problem with generalizing to large angles that the naïve approach does. 

The integral approach to computing errors seems sensible, and appears to work well but no comparison is made to the previous method working with derivatives. This would be useful in demonstrating that the predictive performance is at least no worse than the approach taken in [1] and doesn’t require knowing or estimating derivatives. Also it would be good to have an ablation study investigating predictive performance as a function of tau, the number of integration timesteps for computing the error.

The paper shows evaluation of the learned dynamics system for control on two examples, an inverted pendulum and CartPole. In the inverted pendulum example, the learned potential energy and the control response is used to design a control that shapes the potential energy and with additional damping. Since the control output is closely related to a PD controller, it would be good to compare against a standard P(I)D controller that doesn’t depend on the model. For the CartPole system, the control is exactly a PD controller and it’s not clear how the Hamiltonian/dynamics model are used at all in this example. Generally the paper does a good job at demonstrating benefits in modeling ability and generalization, but the experiments applying this model to control are not very convincing. Having an example where the model is applied for a standard approach like MPC for one of these problems would useful for gauging efficacy in possible control applications.

There are some promising leads explored in this paper for learning physical system dynamics effectively with neural nets. I think there is a lot of promise in the approach, however some of the improvements over past work have not been adequately tested (integral approach and sensitivity to # of integration steps) and the control experiments are not very convincing although it seems like they could be. I lean towards a weak reject for the paper as is, but if the authors flesh out the control experiments and do some ablation studies I will improve my score.

Minor Comments and Questions:
Is f tilde in equation 11 parametrized to have 0 output on u dot or is it expected that this relationship would simply be approximated by the model?

[1] Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. arXiv:1906.01563, 2019.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Update: I have read the author's response. Thank you!
***

This is an excellent paper that integrates inductive biases from physics into learning dynamical models that can then be integrated into deep RL-based control tasks.

The model approximates the dynamical function f(q, p, u), where q are the generalised coordinates of the system (mixture of positions in R^1 and angles in S^1), p are the generalised momenta and u is the external control input. Function f can then be integrated by a numerical solver, and used as the dynamical function in a Neural ODE (ordinary differential equation) for modelling the continuous time evolution of the nonlinear dynamical system. The dynamics function is explicitly written as the equations of the Hamiltonian dynamical system, involving the 1) inverse of the mass function, 2) potential energy and 3) control function, in a complex graph (Figure 1) that transforms positional and angular coordinates and momenta x and the external control into f(x, u).

The derivation of the method is long but very well written and didactic. The experiments on the control suite of OpenAI Gym  are simple (pendulum and cart-pole only) but thorough, and compare the proposed method with a non-inductive bias method (still relying on Neural ODEs), a simpler naive and geometric baselines. Overall, the paper is very easy to follow (even for someone who does not work on control experiments in deep RL) while addressing complex physics.

Minor remarks:
Can you define g in 2.1?
Can you add the derivation of equations (8) through (10) in the appendix?
The second to last sentence on page 4 seems unfinished.
Can the authors comment on how this model would scale to larger (e.g. multiple joints) dynamical systems?
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: In this paper, the authors propose a framework for learning the dynamics of a system with underlying Hamiltonian structure subjected to external control. Based on the extended equations of motion, the authors suggest how to apply NeuralODE in a way that makes use of the prior information that the unconstrained system is Hamiltonian and subjected to a control term. For a range of tasks, the authors then demonstrate that the proposed SymODEN framework can learn the dynamics and recover the known analytical solution, and that they can derive a controller that allows them to drive the system to a target configuration. 

I find this paper very interesting and the formulation elegant. In particular, I appreciate that the paper is pretty much self-contained and that the authors derive the theory from first principles. However, there are a couple of points (listed below) that should be addressed prior to publication to improve the clarity of the paper, and to help the reader to fully appreciate the depth of the experimental section. If these points were addressed in sufficient detail, I would be willing to increase the score:

1) Reading the abstract and the introduction, I got the impression that SymODEN can be applied to any physical system while the method is in fact only applicable to systems governed by Eq. (11). I think it’s worth mentioning in the text that not every physical system follows (constrained) Hamiltonian Dynamics. 

2) You mention that a controller is designed based on the learned dynamics. For example, in the first sentence in Sec. 2.2: ‘Once the dynamics of a system have been learned, it can be used to synthesize a controller to maneuver the system to a reference configuration q*’. I think it is important to specify here which dynamics you are referring to, i.e. the constrained or unconstrained dynamics. Later on it becomes clear that you use constant-u training data and that u is part of the input to the model, so it has to be constrained dynamics, but at this stage it is still unclear to the reader.

3) In Eq. (11), you set du/dt = 0 without motivating this restriction. Can you provide at least one sentence on why this is an interesting choice and back it up with a reference? Furthermore, the sentence above Eq. (11) is broken and needs fixing. 

4) Tasks (general): You consider a range of tasks and I appreciate that you start with a simple and intuitive system. However, I think a bit more guidance throughout the tasks section would be very helpful. In particular, I would suggest that you provide a short (one or two sentences) summary at the beginning of each task to say what exactly it is that you are trying to test or demonstrate. Furthermore, I would suggest showing the summary of  results, i.e. Sec. 4.2, after you introduce the individual tasks rather than before.

5) Task 2: This task addresses multiple things in one go: Initially, you demonstrate that you can recover the results of Task 1 without access to the generalised momenta, and explain why this can only be done up to a constant scaling factor. This is very interesting and clear. However, then you jump straight into the controller and things become a little unclear because, at this stage, it still seems that the dynamics are the same as in Task 1, i.e. unconstrained. Please add a sentence or two for clarification. Another important aspect that is not commented on at all is the behaviour of u(t) in Eq. (27). In particular, isn’t u(t) expected to satisfy du/dt = 0 based on Eq. (11)? The results in Fig. 6 suggest that this is not the case (see time interval [2, 6]). This seems like an interesting and surprising behaviour, especially because SymODEN was only trained with constant-u training data. I would appreciate if the authors could comment on this. I would also suggest to add horizontal lines to Fig. 6 to indicate the expected results.

6) Task 4: Why did you not explain this task in a dedicated section like you did for all other tasks?

7) Symplectic: Since both the method and the title of the paper contain the word ‘symplectic’, it would be good if you explained what the term actually means.

8) ‘Our results show that incorporation of such physics-based inductive bias can provide knowledge about relevant physical properties (mass, potential energy) and laws (conservation of energy)...’. To me, this statement is slightly misleading. You did not demonstrate that SymODEN ‘provides knowledge’ of laws of the system; energy conservation (for u = 0) as a law is hard-coded into your network. The specific value of the energy can be inferred but that I would consider a physical property. I would suggest to change the wording to reflect this clearly.

9) Introduce the acronym ODE much earlier than in Sec. 3.1.

10) Model training: What happens if you use unseen initial conditions rather than the ones in the training data? Perhaps you could add a comment to clarify.

11) There are many typos and grammar mistakes in the paper. Please revise it carefully. To give you a few examples:
‘are both reformulation’ -> ‘are both reformulations’
Sec 2.1: Decide on whether you use plural or singular for ‘dynamics’ and be consistent.
‘on a equal footing’ -> ‘on an equal footing’ 
‘beyond classical mechanics, the Hamiltonian’ -> ‘Beyond classical mechanics, Hamiltonian …’
‘Hamiltonian is same as’ -> ‘Hamiltonian is the same as’
‘represents potential energy’ -> ‘represents the potential energy’
‘trajectory actually converge to’ -> ‘trajectory actually converges to’
‘a ODE solver -> ‘an ODE solver’.
‘Lagrangian and Hamiltonian formulation’ -> ‘Lagrangian and Hamiltonian formulations’
‘assume that q and p evolves’ -> ‘assume that q and p evolve’
‘translational coordinate’ -> ‘translational coordinates’
‘naive baseline model approximate’ -> ‘naive baseline model approximates’
etc.


*************************************
The authors addressed my comments and answered  my questions clearly.  I therefore increased the score. *************************************
review_title: Official Blind Review #2

Meta-review: This paper proposes a novel method for learning Hamiltonian dynamics from data. The data is obtained from systems subjected to an external control signal. The authors show the utility of their method for subsequent improved control in a reinforcement learning setting. The paper is well written, the method is derived from first principles, and the experimental validation is solid. The authors were also able to take into account the reviewers’ feedback and further improve their paper during the discussion period. Overall all of the reviewers agree that this is a great contribution to the field and hence I am happy to recommend acceptance.
Decision: Accept (Poster)