Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control
Authors: Yaofeng Desmond Zhong
Biswadip Dey
Amit Chakraborty
Venue:  (2020)
Keywords: Deep Model Learning
Physics-based Priors
Control of Mechanical Systems

Abstract: In this paper, we introduce Symplectic ODE-Net (SymODEN), a deep learning framework which can infer the dynamics of a physical system, given by an ordinary differential equation (ODE), from observed state trajectories. To achieve better generalization with fewer training samples, SymODEN incorporates appropriate inductive bias by designing the associated computation graph in a physics-informed manner. In particular, we enforce Hamiltonian dynamics with control to learn the underlying dynamics in a transparent way, which can then be leveraged to draw insight about relevant physical aspects of the system, such as mass and potential energy. In addition, we propose a parametrization which can enforce this Hamiltonian formalism even when the generalized coordinate data is embedded in a high-dimensional space or we can only access velocity data instead of generalized momentum. This framework, by offering interpretable, physically-consistent models for physical systems, opens up new possibilities for synthesizing model-based control strategies.

Published as a conference paper at ICLR 2020
SYMPLECTIC ODE-NET: LEARNING HAMILTONIAN DYNAMICS WITH CONTROL
Yaofeng Desmond Zhong*
Biswadip Dey
Princeton University
Siemens Corporate Technology b swadidd deye@s eemens.
y. r.zoongerinnettonn.ee edu
Amit Chakraborty Siemens Corporate Technology
amit. chakraborty weeeaeeertttssssess.eeeemmmm com
ABSTRACT
In this paper, we introduce Symplectid[ODE.r ODE-Net (SymODEN), deep learning framework which infer the dynamics of a physical system, given by or- dinary differential equation (ODE), from observed state trajectories. To achieve better generalization with fewer training samples, SymODEN incorporates appro- priate inductive bias by designing the associated computation graph in physics- informed manner. In particular, we enforce Hamiltonian dynamics with control learn the underlying dynamics in transparent way, which can then be leveraged draw insight about relevant physical aspects of the system, such as mass and po- tential energy. In addition, we propose parametrization which can enforce this Hamiltonian formalism even when the generalized coordinate data embedded high-dimensionnal space or we can only access velocity data instead of general- ized momentum. This framework, by offering interpretable, physically.consisee models for physical systems, opens up new possibilities for synthesizing model- based control strategies.
INTRODUCTION
In recent years, deep neural networks (Goodfellow al. 2016) have become very accurate and widely used many application domains, such as image recognition (He et al., 2016). language comprehension (Devlin et al., 2019)), and sequential decision making (Silver al. 2017 To learn underlying patterns from data and enable generalization beyond the training set, the learning ap- proach incorporates appropriate inductive bias (Haussler, [9888 Baxter 2000) by promoting repre- sentations which are simple in some sense. It typically manifests itself via set of assumptions, which in turn can guide learning algorithm predicting an outcome for previously unseen data then depends on how well the inductive bias cap- tures the ground reality. Inductive bias can be introduced as the prior Bayesian model, or via the choice of computation graphs in neural network.
pick one hypothesis over another. The success in
In a variety of settings, especially in physical systems, wherein laws of physics are primarily re- sponsible for shaping the outcome, generalization in neural networks can be improved by leveraging underlying physics for designing the computation graphs. Here, by leveraging a generalization of the Hamiltonian dynamics, we develop a learning framework which exploits the underlying physics in the associated computation graph. Our results show that incorporation of such physics-base based in- ductive bias offers insight about relevant physical properties of the system, such as inertia, potential energy, total conserved energy. These insights, in turn, enable a more accurate prediction future behavior and improvement out-oles behavior. Furthermore, learning a physically-consistee model of the underlying dynamics can subsequently enable usage of model- controllers which can provide performance guarantees for complex, nonlinear systems. In particular, insight about
* "This research was carried out during Y. D. Zhong's 's internship at Siemens Corporate Technology.
We use the word Symplectic to emphasize that the learned dynamics endows symplectic structure Ammol 200T) the underlying space.

---

Published as a conference paper at ICLR 2020
kinetic and potential energy of a physical system can be leveraged synthesize appropriate control strategies, such as the method controlled Lagrangian (Bloch et al.) [000] and interconnection & damping assignment (Ortega et al., 2002), which reshape the closed-loop energy landscape to achieve a broad range of control objectives (regulation, tracking, etc.).
RELATED WORK
Physics-based Priors for Learning in Dynamical Systems: The last few years have witnessed a significant interest in incorporating physics-based priors into deep learning frameworks. Such approaches, in contrast to more rigid parametric system identification techniques (Söderström & Stoica 1988), use neural networks to approximate the state- transition dynamics and therefore are more expressive, Saanaee--onzalez et al. (2018), by representing the causal relationships in a phys- ical system as a directed graph, use a recurrent graph network to infer latent space dynamics of robotic systems. Lutter et al. (2019) and Gupta et al. (2019) leverage Lagrangian mechanics to learn the dynamics of kinematic structures from time- series data of position, velocity, and acceleration. A more recent (concurrent) work by Grryydanu et al. (2019) uses Hamiltonian mechanics to learn the dynamics of autonomous, energy-conserved mechanical systems from time-series data of position, momentum, and their derivatives. A key difference between these approaches and the proposed one is that our framework does require any information about higher-order derivatives (e.g.. acceleration) and can incorporate external control into the Hamiltonian formalism.
Neural Networks for Dynamics and Control: Inferring underlying dynamics from time-series data plays a critical role in controlling closed-loop response of dynamical systems, such as robotic manipulators (Lillierap al., 2015) and building HVAC systems (Wei al. 2017). Although the use of neural networks towards identification and control of dynamical systems dates back to more than three decades ago (Narendra & Parthasarathy, 1990). recent advances in deep neural networks have led to renewed interest in this domain. Watter et al. (2015) learn dynamics with control from high-dimensional observations (raw image sequences) using variational approach and synthesize an iterative LQR controller to control physical systems by imposing a locally linear constraint. Karl ct al. (2016) and Krishnan et al. (2017) 7) adopt a variational approach and use recurrent architectures to learn state-space models from noisy observation. SE3-Nets (Byravan & Fox 2017) learn SE()) transformation of rigid bodies from point cloud data. Ayed et al. (2019) use partial information about the system state to learn nonlinear state-space model. However, this body of work, while attempting to learn state-space models, does not take physics-based priors into consideration.
CONTRIBUTION
The main contribution of this work is two-fold. First, we introduce a learning framework called Symplectic ODE-Net (SymODEN) which encodes generalization of the Hamiltonian dynamics. This generalization, by adding an external control term to the standard Hamiltonian dynamics, al- lows to learn the system dynamics which conforms Hamiltonian dynamics with control. With the learned structured dynamics, we are able to synthesize controllers to control the system to track reference configuration. Moreover, by encoding the structure, we can achieve better predictions with smaller network sizes. Second, we take one step forward in combining the physics- prior and the data-driven approach. Previous approaches (Lutter al. 2019 Greydanus et al., 2019) require data in the form of generalized coordinates and their derivatives up to the second order. However, large number of physical systems accommodate generalized coordinates which are non-Euclided (cos g, sin 4) instead of the coordinate (q) itself. The underlying reason is that an angular coordi- nate lies on 31 instead of RI. In contrast to previous approaches which do not address this as- pect, SymODEN has been designed to work with angle data in the embedded form. Additionally, leverage differentiable ODE solvers to avoid the need for estimating second-order derivatives of generalized coordinates. Code for the SymODEN framework and experiments is available at http:///whhbb.comcccccss//////mp
(e.g., angles), and such angle data is often obtained in the embedded form, i.e.,
2

---

Published as a conference paper at ICLR 2020
2 PRELIMINARY CONCEPTS
2.1 HAMILTONIAN DYNAMICS
Lagrangian dynamics and Hamiltonian dynamics are both reformulations of Newtonian dynamics. They provide novel insights into the laws of mechanics. In these formulations, the configuration of a system is described by its generalized coordinates. Over time, the configuration point of the system moves in the configuration space, tracing out a trajectory. Lagrangian dynamics describes the evolution of this trajectory, i.e., the equations of motion, in the configuration space. Hamiltonian dynamics, however, tracks the change of system states in the phase space, i.e. the product space of generalized coordinates q = (41. 92.... qn) and generalized momenta p = (p1, P2. .p.......... In other words, Hamiltonian dynamics treats and p on an equal footing. This not only provides symmetric equations of motion but also leads to a whole new approach to classical mechanics (Goldstein et al, 2002). Hamiltonian dynamics is also widely used in statistical and quantum mechanics.
In Hamiltonian dynamics, the time-evolution of system is described by the Hamiltonian H (q. p), scalar function of generalized coordinates and momenta. Moreover, in almost all physical systems, the Hamiltonian is the same as the total energy and hence can be expressed as
H(q. = =(
" (q) + V(q).
where the mass matrix M(q) is symmetric positive definite and V(q) represents the potential energy of the system. Correspondingly, the time-evolution of the system is governed by
0H
0H
P
3p
Ōq
where we have dropped explicit dependence on and for brevity of notation. Moreover, since
HH T 0q
4 Öpp ) p= =0 0,
T
H
the total energy is conserved along a trajectory of the system. The RHS of Equation 2 is called the symplectic gradient (Rowe et al. 1980) of H, and Equation (3) shows that moving along the symplectic gradient keeps the Hamiltonian constant.
In this work, we consider a generalization of the Hamiltonian dynamics which provides means to incorporate external control (u), such as force and torque. As external control is usually affine and only influences changes in the generalized momenta, we can express this generalization as
q P
aH
8p
0H
[(())
u,
pq
where the input matrix g(q) is typically assumed to have full column rank. For 0, the general- ized dynamics reduces to the classical Hamiltonian dynamics (2) and the total energy is conserved; however, when A O, the system has dissipation-fie energy exchange with the environment.
2.2 CONTROL VIA ENERGY SHAPING
Once have learned the dynamics of system, the learned model can be used to synthesize controller for driving the system to a reference configuration q" As the proposed approach offers insight about the energy associated with a system, is natural choice to exploit this information for synthesizing controllers via energy shaping (Ortega et al. 2001). As energy is a fundamental aspect of physical systems, reshaping the associated energy landscape enables us specify broad range of control objectives and synthesize nonlinear controllers with provable performance guarantees.
If ra))(g(()) rank(q), the system is fully-actuated and we have control over any dimension of "acceleration" in P. For such fully-actuated systems, controller u(q. p) = B(q) + (p)) can be synthesized via potential energy shaping B(q) and damping injection v(p). For completeness, we restate this procedure (Ortega et al., 2001) using our notation. As the name suggests, the goal of potential energy shaping is to synthesize B(()) such that the closed-loop system behaves as its time-evolution is governed by desired Hamiltonian Ha. With this, we have
a8
0H
P
=
öp
0H
0 g(q)
B(q) =
8p
0Hg
(5)
0q
0q
3

---

Published as a conference paper at ICLR 2020
where the difference between the desired Hamiltonian and the original one lies in their potential energy term, i.c
Ha(q. = =2 2p'
1 (qp + Va(g).
(6)
In other words, B(g) shape the potential energy such that the desired Hamiltonian Ha(g.p) p) has minimum at (q",0) 0). Then, by substituting Equation (1 and Equation (6) into Equation 5 we get
( = g" (gg") ) @V
aV
g
0q
Ōq
Thus, with potential energy shaping, we ensure that the system has the lowest energy at the desired reference configuration. Furthermore, to ensure that trajectories actually converge to this configura- tion, we add an additional damping term given by
v(p) -g-
However, for underactuated systems, potential energy shaping alone canno drive the system to desired configuration. We also need kinetic energy shaping for this purpose CChang et al. 2002).
Remark If the desired potential energy is chosen to be quadratic of the form
1 (q q")² Kp -'),,
(g(g)=
(9)
2
the external forcing term can be expressed as
g" (gg")
av
( -----"")p- Kap).
-Kap)
(10)
Ōq
This can be interpreted PD controller with additional energy compensation term.
SYMPLECTIC ODE-NET
In this section, we introduce the network architecture of Symplectic ODE-Net. In Subsection 3.1 we show how learn ordinary differential equation with constant control term. In Subsection 3.2 we assume we have access to generalized coordinate and momentum data and derive the network architecture. In Subsection 3.3 we take one step further to propose data-driven approach to deal with data of embedded angle coordinates. In Subsection 3.4 we put together the line of reasoning introduced in the previous two subsections to propose SymODEN for learning dynamics on the hybrid space R" x T
3.1 TRAINING NEURAL ODE WITH CONSTANT FORCING
Now we focus on the problem of learning the ordinary differential equation (ODE) from time series data. Consider an ODE: = f(x). Assume we don't know the analytical expression of the right hand side (RHS) and we approximate it with neural network. If we have time series data X (Xt, Xt,, ..., ). how could we learn fxx from the data?
Chen al. (2018) introduced Neural ODE, differentiable ODE solvers with O(1)-memory back- propagation. With Neural ODE, we make predictions by approximating the RHS function using a neural network fo and feed it into ODE solver
&1 &ty...... .... &t. ODESolve(Xto fo, L1 (2...).,.))
We can then construct the loss function L = X X 12 and update the weights 0 by backpropagating through the ODE solver.
In theory, we can learn fa in this way. In practice, however, the neural net is hard to train if n is large. we have a bad initial estimate of the fe, the prediction error would general large. Although [e, & might be small, &x would be far from X error accumulates, which makes the neural network hard to train. In fact, the prediction error of & is not as important as & In other words, should weight data points in short time horizon more than the rest of the data points. In order
á instead in Equation 3 As gg" is not invertible, we cannot solve the matching condition given by Equation *Please refer Appendix B for more details.
IIf we have access instead of p.
4

---

Published as a conference paper at ICLR 2020
to address this and better utilize the data, introduce the time horizon 7 as a hyperparameter and predict X1 X2,42 ..,, X/+++ from initial condition Xt,, where i = O.... 72 - т.
One challenge toward leveraging Neural ODE to learn spate- models is the incorporation of the f(x, u) with x = (q. p). A function
control term into the dynamics. Equation 4 has the form = of this form cannot be directly fed into Neural ODE directly since the domain and range of f have different dimensions. In general, if our data consist trajectories of (x, u) to.....t where u remains the same in trajectory, we can leverage the augmented dynamics
x
(x. u) fx( u).
(11)
match the input and output dimension of fo. which enables us feed into Neural ODE. The idea here is to use different constant external forcing to get the system responses and use those responses to train the model. With trained model, we can apply a time- varying u to the dynamics x = fx u) and generate estimated trajectories. When we synthesize the controller, remains constant each integration step. As long as our model interpolates well among different values of constant we could get good estimated trajectories with time-varying u. The problem is then how to design the network architecture of fg, equivalently fo such that we can learn the dynamics in an efficient way.
With Equation (1
3.2 LEARNING FROM GENERALIZED COORDINATE AND MOMENTUM
Suppose we have trajectory data consisting of (q, p, to...... where remains constant in tra- jectory. If we have the prior knowledge that the unforced dynamics of and is governed by Hamiltonian dynamics, we can use three neural nets (4)'(()), Vo, (q) and ge, (q) as function approximators to represent the inverse of mass matrix, potential energy and the input matrix. Thus,
aHl,, Up
fp(q.p.u) =
alle,
+
go (q)
u
(12)
0q
where
Ho, 0, (q. = 2 M,, (q)p + (g)
(13)
The partial derivative in the expression can be taken care of by automatic differentiation. by putting the designed fg (q. P. u) into Neural ODE, we obtain systematic way of adding the prior knowledge of Hamiltonian dynamics into end andtto-end learning.
3.3 LEARNING FROM EMBEDDED ANGLE DATA
In the previous subsection, we assume (q, p, u).a........ In lot of physical system models, the state variables involve angles which reside in the interval [-T, X). In other words, each angle resides on the manifold S². From data-driven perspective, the data that respects the geometry is a dimensional embedding (cos q. sin g). Furthermore, the generalized momentum data is usually not available. Instead, the velocity often available. For example, in OpenAl Gym (Brockman et al.) 2016) Pendul um task, the observation is (cos q, sin g, Q).
From a theoretical perspective, however, the angle itself is often used, instead of the 2D embedding. The reason being both the Lagrangian and the Hamiltonian formulations are derived using general- ized coordinates. Using an independent generalized coordinate system makes it easier to solve for the equations of motion.
In this subsection, we take the data-driven standpoint and develop an angle-aware method to accom- modate the underlying manifold structure. We assume all the generalized coordinates are angles and the data comes in the form (x (q), (2))). xs(4), u)to...... = (cos q, sin q, q, u)tg....... We aim to incorporate our theoretical prior Hamiltonian dynamics into the data-driven approach. The goal is to learn the dynamics of x1, X2 and X3. Noticing = M(1,, x2 ), we can write down the derivative of x1, X2 and X3,
X = sin q q q= X2 qq
X2 cos aoqqo=sse xI à
(14)
d (1,
x3 di
(M (M- (x1, M (x
di
5

---

Published as a conference paper at ICLR 2020
where "o" represents the elementwise product (i.e., Hadamard product). We assume q and p evolve with the generalized Hamiltonian dynamics Equation 44. Here the Hamiltonian Hx1,, X2, p) is function of x1, X2 and p instead of and p.
ÖH
q 0p
(15)
0H
0x, 0H
0x2 0H
p
+ g(X1, X2)u
gxxx
dq
dq 0xi
dq 0x2
0H
0H
0H - -x1 0x22
0H
= sin o σx1 cos
+ g(x. x2)
+ g(xx,x2))
(16)
dxg
σx1
Then the right hand side of Equation 14 can be expressed as function of state variables and control (2,, X3, u). Thus, it can be fed into the Neural ODE. We use three neural nets X2), Ve, X2) and ge, (x1, x2) as function approximators. Substitute Equation 15
and Equation (16) into Equation (14). then the RHS serves as (x1, X2, X2, X3, u
HHH,. .2
HHe
(x1, X2, X3, u), u)=
X1
dp HH,
d (Mo, (xIxx)) M (x1x2)
HHo 8xa
tgg, (x1x2)u
de
X2
8x1
X1
(17)
where
Ho, (x1, p) = pp'MM )
Xx1, (x,, + (x1,
(18)
p Mo. (2),x2)x0
(19)
3.4 LEARNING ON HYBRID SPACES R X T
In Subsection 3.2 we treated the generalized coordinates translational coordinates. In Subsection 3.3 we developed an angle-aware method to better deal with embedded angle data. In most of physical systems, these two types of coordinates coexist. For example, robotics systems are usually modelled as interconnected rigid bodies. The positions of joints or center of mass are translational coordinates and the orientations of each rigid body angular coordinates. In other words, the generalized coordinates lie on R" x T where T denotes the m-torus, with T = S and T subsections. We
31 $1. In this subsection, we put together the architecture of the previous assume the generalized coordinates are q (r,Φ) E IR" x T and the data comes in the form of (x1, X2, (2, X3, X4, X5, u) t)........ (r,cos, Ø, s, F, Φ u).o....f.. With similar line of reasoning, use three neural nets Mj x2,, Vo, (x1, xx) approximators. We have
X3) and ge, (X1, X2, X3) as function
X4
= Ma, (x1, Xs)
(20)
X5
Ho, (x1, (x,, X3, p) 5D" (x,, + Vo, X2,
(21)
with Hamiltonian dynamics, we have
Ho,
q
(22)
Öp
0
p
0Ho, 02
8h
OHo R2 8kg
+ go, (x1, X2, Xz)u
(23)
Xg D
8x2
X2
Then
x
r
X2
x3
Xs
(x1, X2, Xg. X4, u)
(24)
x
X4
d
(Mp. (x1, X2, Xx)
)ppp Mj (x), X2, X3
X5_
di
where the and o come from Equation (22). Now we obtain a fo which can be fed into Neural ODE. Figure shows the flow the computation graph based on Equation (20 (4))
[
'In Equation
17 the derivative of M (x1, can be expanded using chain rule and expressed as
function of the

---

Published as a conference paper at ICLR 2020
M
H
X1
X2
Xy
Pg
X
fx
ODESolvee
9os
X4
Xn
B
Xt.
(1(-) (4)
11
Figure 1: The computation graph of SymODEN, Blue arrows indicate neural network parametrization. Red arrows indicate automatic differentiation. For a given (x, u), the computation graph outputs a fo (x, u) which follows Hamiltonian dynamies with control. The function itself is an input to the Neural ODE to generate estimation of states each time step. Since all the operations differentiable, weights of the neural networks can updated by backpropggation.
3.5 POSITIVE DEFINITENESS OF THE MASS MATRIX
In real physical systems, the mass matrix M is positive definite, which ensures a positive kinetic energy with non-zero velocity. The positive definiteness of M implies the positive definiteness of p, Thus, we impose this constraint in the network architecture by M = Lg, L where Lg, lower-triangular matrix. The positive definiteness ensured the diagonal elements of M are positive. In practice this can be done by adding small constant to the diagonal elements of M It not only makes Mg, invertible, but also stabilize the training.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We use the following four tasks to evaluate the performance of Symplectic ODE-Net model (i) Task I: pendulum with generalized coordinate and momentum data (learinn on R-); (ii) Task 2: a pendulum with embedded angle data (learning on S²); (iii) Task 3: CartPolc system (learning on R ) and (iv) Task 4: an Acrobot (learning on T
Model Variants. Besides the Symplectic ODE-Net model derived above, we consider a variant by approximating the Hamiltonian using fully connected neural net He,,0, We call Unstructured Symplectic ODE-Net (Unstructured SymODEN) since this model does not exploit the structure of the Hamiltonian ①
In order to show that we can learn the dynamics better with less parameters by leveraging prior knowledge, we set up baseline models for all four experiments. For the pendulum with generalized coordinate and momentum data, the naive baseline model approximates Equa- tion 112 Eg (x, u) - by a fully connected neural net. For all the other experiments, which involves embedded angle data, set up two different baseline models: naive baseline approximates fx u) by a fully connected neural net. It doesn't 't respect the fact that the coordinate pair, cos & and sin d. lie on T Thus, set up the geometric baseline model which approximates and with fully connected neural net. This ensures that the angle data evolves on T'm
Baseline Models.
Data Generation. For all tasks, we randomly generated initial conditions of states and subsequently combined them with 5 different constant control inputs, i.e., u = 22.0, -1.0, 0.0, 1.0, 2.0 to produce the initial conditions and input required for simulation. The simulators integrate the corresponding dynamics for 20 time steps to generate trajectory data which is then used to construct the training set. The simulators for different tasks are different. For Task we integrate the true generalized Hamiltonian dynamics with time interval of 0.05 seconds to generate trajectories. All the other tasks deal with embedded angle data and velocity directly, so we use OpenAl Gym (Brockman al.) 2016) simulators to generate trajectory data. One drawback of using OpenAl Gym is that not all environments use the Runge- Kutta method (RK4) to carry out the integration. OpenAl Gym favors other numerical schemes over RK4 because of speed, but is harder to learn the dynamics with
For information on model details, please refer Appendix

---

Published as a conference paper at ICLR 2020
inaccurate data. For example, if we plot the total energy as a function of time from data generated by sendulum-vo environment with zero action, we see that the total energy oscillates around a constant by significant amount, even though the total energy should be conserved. Thus, for Task 2 and Task 3, we use Pendul um um-v0 and Cart e-vee, respectively, and replace the numerical integrator of the environments to RK4. For Task 4, we use the Acrobot-y environment which is already using RK4. We also change the action space of Pendul 1 un,v0, Acrobot- v1 to continuous space with a large enough bound.
vO, CartPole-vz vv and
Model training. In all the tasks, we train our model using Adam optimizer (Kingma & Ba 2014) with 1000 epochs. We set a time horizon т 3, and choose "RK4" as the numerical integration scheme in Neural ODE. We vary the size of the training set by doubling from 16 initial state con- ditions to 1024 initial state conditions. Each initial state condition combined with five constant control = -2.0, 1.0,, 0.0, 1.0, 2.0 to produce initial condition for simulation. Each trajectory is generated by integrating the dynamics 20 time steps forward. We set the size of mini-batches to be the number of initial state conditions. We logged the train error per trajectory and the prediction error per trajectory in each case for all the tasks. The train error per trajectory is the mean squared error (MSE) between the estimated trajectory and the ground truth over 20 time steps. To evaluate the performance of each model in terms of long time prediction, we construct the metric of predic- tion error per trajectory by using the same initial state condition in the training set with a constant control of u = 0.0, integrating 40 time steps forward, and calculating the MSE over 40 time steps. The reason for using only the unforced trajectories is that a constant nonzero control might cause the velocity to keep increasing or decreasing over time, and large absolute values of velocity are of little interest for synthesizing controllers.
4.2 TASK 1: PENDULUM WITH GENERALIZED COORDINATE AND MOMENTUM DATA
In this task, use the model de- scribed in Section and present the predicted trajectories of the learned models as well as the learned func- tions of SymODEN. We also point out the drawback of treating the angle data as a Cartesian coordinate. The dynamics of this task has the follow- ing form
Tajectry Resicion
Trajestrr Fresicsin
Trajectoy Fredktion
Seaun trem Save ta
Smord trem UNIITALURRE SymdDEs
triedd Fr Sym
gl4)
M -G6g)
V(4)
Symd0os Pall
SYMODEN F1
q 33,
-5 q + (25)
with Hamiltonian (q) = 1.5p² + 5(1 - cos v). In other words M(4) =
SyMOOsH Mg100
3. V()) 5(1 - g) and g(g) 1.
Figure 2: Sample trajectories and learned functions of Task 1.
In Figure 2 The ground truth is an unforced trajectory which is energy-conserved. The predic- tion trajectory of the baseline model does not conserve energy, while both the SymODEN and its unstructured variant predict
trajectories. For SymODEN, the learned go, (q) and M (q) matches the ground truth well. Vo, (g) differs from the ground truth with a constant. This is acceptable since the potential energy is a relative notion. Only the derivative of Ve, plays a role in the dynamics.
Here we treat as a variable in R and our training set contains initial conditions of E [-T,, 3m). The learned functions do not extrapolate well outside this range, as we can see from the left part in the figures of M7 (g) and (z). We address this issue by working directly with embedded angle data, which leads us to the next subsection.
4.3 TASK 2: PENDULUM WITH EMBEDDED DATA
In this task, the dynamics is the same Equation 25 but the training data are generated by the OpenAl Gym simulator, i.e. we use embedded angle data and assume we only have access to instead of p. We use the model described in Section 3.3 and synthesize an energy-based controller (Section 2.2). Without true data, the learned function matches the ground truth with a scaling 3, as shown in Figure 3 To explain the scaling, let us look at the following dynamics
q p/a,
p - 155a sin q + 3cx
(26)
8

---

Published as a conference paper at ICLR 2020
with Hamiltonian H (2)( + 15a(1- - cos q). If we only look at the dynamics of q. - 15 + 3u, which is indepen- dent of If we don't have access to the generalized momentum our trained neural network may converge to Hamiltonian with a Oe which is different from the true value, a = 1/3, in this task. By a scaling 5 = az/a, = 0.357. the learned functions match the ground truth. Even we are not learning the true Ct,, we can still perform predic- tion and control since we are learning the dynamics of g correctly. We let Va2 = - Vo, (q). then the desired Hamiltonian has minimum energy when the pendulum rests at the upward position. For the ( and
p(c)
M ((g)
V10)
Groand Trus Nyedtoo phuydd
Gread P0A0E
have
Thir Symdton Ng00e
-2
Figure 3: Without true generalized momentum data, the learned
functions match the ground truth with scaling. Here 3 = 0.357
damping injection, we let Ka = 3. Then from Equation
(8) the controller we synthesize is
VVo2 sin +
aVo₂
u(cos q, sin q, q) = 90, (cos q, sin 4) (
cos q) - 31)
(27)
a cos q
i Ssin g
Only SymODEN out of all models consider provides the learned po- tential energy which is required to synthesize the controller. Figure 4 shows how the states evolve when the controller is fed into the OpenAI Gym simulator. We can successfully control the pendulum into the inverted position using the controller based on the learned model even though the absolute maximum control 7.5, is more than three times larger than the absolute maximum u in the training set, which is 2.0. This shows SymODEN extrapolates well.
Figure 4: Time-evolution of the state variables (cos g, sin v, q) when the closed-loop control input u{cos q, sin g. q) is governed by Equation (27). The thin black lines show the expected results.
4.4 TASK 3: CARTPOLE SYSTEM
The CartPole system is an underactuated system and to synthesize a controller to balance the pole from arbitrary initial condition requires trajectory optimization kinetic energy shaping. We show that we can learn its dynamics and perform prediction in Section 4.6 We also train SymODEN in fully-actuated version of the CartPole system (see Appendix [ The corresponding energy-based
controller can bring the pole to the inverted position while driving the cart to the origin.
4.5 TASK 4: ACROBOT
The Acrobot is an underactuated double pendulum. As this system exhibits chaotic motion, it is not possible to predict its long-term behavior. However, Figure [6] shows that SymODEN can provide reasonably good short-term prediction. We also train SymODEN in a fully-actuated version of the Acrobot and show that we can control this system to reach the inverted position (see Appendix
4.6 RESULTS
In this subsection, we show the train error, prediction error, as well as the MSE and total energy of a sample test trajectory for all the tasks. F the variation in train error and prediction error with changes in the number of initial state conditions in the training set. We can see that SymODEN yields better generalization in every task. In Task 3, although the Geometric Baseline Model yields lower train error in comparison to the other models, SymODEN generates more accurate predictions, indicating overfitting in the Geometric Baseline Model. By incorporating the physics-based prior of Hamiltonian dynamics, SymODEN learns dynamics that obeys physical laws and thus provides better predictions. In most cases, SymODEN trained with a smaller training dataset performs better than other models in terms of the train and prediction error, indicating that better generalization can be achieved even with fewer training samples.
Figure 6 shows the evolution of MSE and total energy along a trajectory with previously unseen initial condition. For all the tasks, MSE of the baseline models diverges faster than SymODEN. Un- structured SymODEN performs well in all tasks except Task 3. As for the total energy, in Task 1 and Task 2, SymODEN and Unstructured SymODEN conserve total energy by oscillating around a con- stant value. In these models, the Hamiltonian itself is learned and the prediction of the future states

---

Published as a conference paper at ICLR 2020
Pondulum
Task Pondulum/embeet)
Task CartPole
Task Acrobot
101
16"
Syndars
10a
109
10
10-3
10a
10
102
10a
102
3.2
10P
10
1.0
10.
Lake Fortin
100
Predictoon
102
10¹
101
10f
10'
10'
10
number Initaal sate condit D1
number intiaal state conditions
number of inttlal state conditinns
number of Intha shate conditions
Figure 5: Train enror per trajectory and prediction error per trajectory for all tasks with different number of training trajectorics. Horizontal axis shows number of initial state conditions (16, 32, 64, 128, 256, 512, 1024) the training set. Both the horizontal axis and vertical axis are in log scale.
Task1: Pendulum
Task 2: Pendulum(embedd
Task 3: CartPolc
Task 4: Acrobot
50
40-
7.5
5.0
20-
2.5
2-
0.0
0
0-
Total treege
0.7
0.6
0-
10
Figure 6: Mean square error and total energy of test trajectories. SymODEN works the hest in terms of both MSF and total energy. Since SymODEN has learned the Hamiltonian and discovered the conservation from data the predicted trajectories match the ground truth. The ground truth of energy all four tasks stay constant.
stay around a level of the Hamiltonian. Baseline models, however, fail to find the conservation and the estimation of future states drift away from the initial Hamiltonian level set.
5 CONCLUSION
Here have introduced Symplectic ODE-Net which provides systematic way to incorporate prior knowledge of Hamiltonian dynamics with control into a deep learning framework. We show that SymODEN achieves better prediction with fewer training samples by learning an interpretable, physically-consistent state-space model. Future works will incorporate broader class of physics- based prior, such as the Hort-Hamitonnian system formulation, of physical systems. SymODEN can work with embedded angle data or when we only have access velocity instead of generalized momentum. Future works would explore other types of embed- ding, such as embedded 3D orientations. Another interesting direction could be to combine energy shaping control (potential as well as kinetic energy shaping) with interpretable end-to-end learning frameworks.
learn dynamics of larger class
REFERENCES
Vladimir I. Arnold, Alexander B. Givental, and Sergei P. Novikov. Symplectic geometry. In Dy- namical systems IV. pp. 1-138. Springer, 2001.
Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, Julien Brajard, and Patrick Gallinari. Learning 2019.
dynamical systems from partial observations.
10

---

Published as a conference paper at ICLR 2020
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12: 149-198, 2000.
Anthony M. Bloch, Naomi E. Leonard, and Jerrold E. Marsden. Controlled lagrangians and the sta- bilization of euler-poincaré mechanical systems. International Journal of Robust and Nonlinear Control, 11(3)191-221, 2001.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv:1606.01550 2016.
Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 173- 180. IEEE, 2017.
Dong E. Chang, Anthony M. Bloch, Naomi E. Leonard, Jerrold E. Marsden, and Craig A. Woolsey. The equivalence of controlled lagrangian and controlled hamiltonian systems. ESAIM: Control, Optimisation and Calculus of Variations, 8:393 422, 2002.
Tian Q. Chen, Yulia Rubanova, Jesse Bettencourt. and David K. Duvenaud. Neural ordinary dif- ferential equations. Advances in Neural Information Processing Systems 31, pp. 6571-6583. 2018.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4188 2019.
Herbert Goldstein, Charles Poole, and John Safko. Classical mechanics, 2002.
Ian Goodfellow, Aaron Courville, and Yoshua Bengio. Deep learning, volume MIT Press, 2016.
Sam Greydanus, Misko Dzamba, and Jason Yosinski.
Hamiltonian Neural Networks.
1rXiv:1906.01500 2019.
Jayesh K. Gupta, Kunal Menda, Zachary Manchester, and Mykel J. Kochenderfer. A general frame- work for structured learning of mechanical systems. arXiv:1902.08705 2019.
David Haussler. Quantifying inductive bias: Al learning algorithms and Valiant's learning frame- work. Artificial Intelligence, 3772):177-221, 1988.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 770-778, 2016.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv:1605.06432, 2016.
Diederik P. Kingma and Jimmy Ba. 141.v.1412.6988) 2014.
Adam:
A Method for Stochastic Optimization.
Rahul G. Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. (eXiv:1509.02971 2015.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In 7th International Conference on Learning Representations (ICLR), 2019.
Kumpati S. Narendra and Kamnan Parthasarathy. Identification and control dynamical systems using neural networks. IEEE Transactions Newral Networks, 1(1):4277, 1990.
11

---

Published as a conference paper at ICLR 2020
Romeo Ortega, Arjan J. Van Der Schaft, Iven Mareels, and Bernhard Maschke. Putting energy back in control. IEEE Control Systems Magazine, 21(2):18 100:18833, -3, 2001.
Romeo Ortega, Arjan J. Van Der Schaft, Bernhard Maschke, and Gerardo Escobar. Interconnection and damping assignment passivity-basee control of port-controlled hamiltonian systems. Auto- matica, 38(4):585-596 2002.
David J. Rowe, Arthur Ryman, and George Rosensteel. Many-body quantum mechanics as sym- plectic dynamical system. Physical Review A, 22(6):2362, 1980.
Alvaro Sanzhez-Gonzalez, Nicolas Heess, Jost T. Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning (ICML). pp. 4467-4476, 2018.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Torsten Söderström and Petre Stoica. System identification. Prentice-Hall, Inc., 1988.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing 29, pp. 2746-2754, 2015.
Tianshu Wei, Yanzhi Wang, and Qi Zhu. Deep Reinforcement Learning for Building HVAC Control. In Proceedings of the 54th Annual Design Automation Conference (DAC), pp. 22:1-22:6, 2017.
Appendices
A EXPERIMENT IMPLEMENTATION DETAILS
The architectures used for experiments are shown below. For all the tasks, SymODEN has the lowest number of total parameters. To ensure that the learned function is smooth, we use Tanh activation function instead of ReLu. As we have differentiation in the computation graph, smooth activation functions would lead to discontinuities in the derivatives. This, in turn, would result in an ODE with discontinuous RHS which is not desirable. All the architectures shown below are fully-connected neural networks. The first number indicates the dimension of the input layer. The last number indicates the dimension of output layer. The dimension of hidden layers shown in the middle along with the activation functions.
Task 1: Pendulum
Input: state dimensions, 1 action dimension
Baseline Model (0.36M parameters): 600Tanh 600Tanh
2Linear
Unstructured SymODEN (0.20M parameters):
Ha, : 2 400Tanh 400Tanh 1Linear
:og: 200Tanh 200Tanh 1Linear
SymODEN (0.13M parameters):
-M 1 300Tanh 300Tanh 1Linear
Ve,: 50Tanh 50Tanh 1Linear
- ga,: 200Tanh 200Tanh LLinear
Task 2: Pendulum with embedded data
Input: state dimensions, I action dimension
12

---

Published as a conference paper at ICLR 2020
Naive Baseline Model (0.65M parameters): 4 800Tanh 800Tanh 3Linear
Geometric Baseline Model (0.46M parameters):
~M Lo, 1. where Lo, 2 300Tanh 300Tanh 300Tanh LLinear
approximate (g. p): 4 600Tann 600Tanh 2Linear
Unstructured SymODEN (0.39M parameters):
Mø. Le, L% where L 2 300Tanh 300Tanh 300Tanh 1Linear
No.: 500Tanh 500Tanh LLinear
geg: 2 200Tanh 200Tanh 1Linear
SymODEN (0.14M parameters):
M Le, - where Lø, 2 3000Thn 300Tanh 300Tanh 1Linear
Veg: 2 50Tanh 50Tanh 1Linear
gog 2 200Tanh 200Tanh LL.inar
Task 3: CartPole
Input: state dimensions, 1 action dimension
Naive Baseline Model (1.01M parameters): 6 1000Tanh 1000Tanh -Linear
Geometric Baseline Model (0.82M parameters):
M Lo, T.g, where Lo, 3 400Tanh SonTannh4001aan 3Linear
approximate (q, p): 6 700Tanh 700Tanh 4Linear
Unstructured SymODEN (0.67M parameters):
-MØ Le, L7 where Lø, 3 400Tanh 400Tanh 400Tanh 3Linear
Ho.: 500Tanh 500Tanh LLinear
ges: 3 300Tanh 300Tanh 2Linear
SymODEN (0.5IM parameters):
-M Le, D7 where Ln,: 3 1-000ann 400Tanh 400Tanh 3Linear
Vo,: 300Tanh 300Tanh 1Linear
go,: 3 300Tanh 300Tanh - 2neer
Task 4:Acrobot
Input: 6 state dimensions, 1 action dimension
Naive Baseline Model (1.46M parameters): 7 1200Tanh- 1200Tanh -6Linear
Geometric Baseline Model (0.97M parameters):
M Lo, I where Lo, 4 400Tanh 400Tanh 400Tanh 3Linear
approximate (q, p): 7 800Tanh 800Tanh 4Linear
Unstructured SymODEN 70.78M parameters):
-MØ Le, - where Le, 4 400Tanh 400Tanh 400Tanh 3Linear
Ha.: 6- 600Tanh 600Tanh 1Linear
ga.: 4 300Tanh 300Tanh 2Linear
SymODEN (0.5IM parameters):
-M Le, L where Lø, 4 - 400Tanh 3Linear
-Vo; 4- 300Tanh 300Tanh Linner
go, 4- 300Tanh 300Tanh 21 Lnnea
13

---

Published as a conference paper at ICLR 2020
B SPECIAL CASE OF ENERGY-BASED CONTROLLER PD CONTROLLER WITH ENERGY COMPENSATION
The energy-based controller has the form u(q. = B(q) + v(p), where the potential energy shaping term (()) and the damping injection term v(p) are given by Equation 7 and Equation (8) respectively.
If the desired potential energy (q) is given by quadratic, in Equation 9
then
0V
0Va
g = g gg T)
0q 0q
0V
g" gg
Kp(q (q q'))
(28)
0q
and the controller can be expressed as
0V
u(q. p) = ((q) + v(p) = (gg") )
Kp(q -q") - K.P)
(29)
0q
The corresponding external forcing term is then given by
2v
g(q)u
3q (K K.P.
(30)
which is same Equation (TO) in the main body of the paper. The first term in this external forcing provides an energy compensation, whereas the second term and the last term are proportional and derivative control terms, respectively. Thus, this control can be perceived PD controller with an additional energy compensation.
C ABLATION STUDY OF DIFFERENTIABLE ODE SOLVER
In Hamiltonian Neural Networks (HNN), Greydanus (2019) incorporate the Hamiltonian struc- ture into learning by minimizing the difference between the symplectic gradients and the true gra- dients. When the true gradient not available, which often the case, the authors suggested using finite difference approximations. In SymODEN, true gradients or gradient approximations are not necessary since we integrate the estimated gradient using differentiable ODE solvers and set up the loss function with the integrated values. Here we perform an ablation study of the differentiable ODE Solver.
Both HNN and the Unstructured SymODEN approximate the Hamiltonian by a neural network and the main difference the differentiable ODE solver, so we compare the performance HNN and the Unstructured SymODEN. We set the time horizon T 1 since it naturally corresponds to the finite difference estimate of the gradient. larger T would correspond to higher-order estimates of gradients. Since there is angle-aware design in HNN, use Task compare the performance of these two models.
We generate 25 training trajectories, each of which contains 45 time steps. This is consistent with the HNN paper. In the HNN paper GGryydanus et al. (2019), the initial conditions of the trajectories are generated randomly in an annulus, whereas in this paper, we generate the initial state conditions uniformly in reasonable range in each state dimension. We guess the reason that the authors of HNN choose the annulus data generation is that they do have an angle-aware design. Take the pendulum for example; all the training and test trajectories they generate do not pass the inverted position. If they make prediction on a trajectory with a large enough initial speed, the angle would go over +22,, 4:4x, etc. in the long run. Since these are away from the region where the model gets trained, we can expect the prediction would be poor. In fact, this motivates us to design the angle-aware SymODEN in Section 3.3 In this ablation study, we generate the training data in both ways.
Table I shows the train error and the prediction error per trajectory of the two models. We can Unstructured SymODEN performs better than HNN. This is an expected result. To see why this the case, let assume the training loss per time step of HNN is similar to that of Unstructured SymODEN. Since the training loss is on the symplectic gradient, the emror would accumulate while integrating the symplectic gradient get the estimated state values, and MSE of the state values
14

---

Published as a conference paper at ICLR 2020
MSE wy annulus training data
Total Energy ml arnulus triining dala
MSE wi rie Langle training ditia
Total Energy wi weccang traring deta
Figure 7: MSE and Total energy of sample test trajectory. Left two figures: the training data for the models are randomly generated in an annulus, the same in HNN. Right two figures: the training data for the models are randomly generated in a rectangle - the same way that we use in SymODEN.
would likely be one order of magnitude greater than that of Unstructured SymODEN. Figure shows the MSE and total energy of a particular trajectory. It clear that the MSE of the Unstructured SymODEN is lower than that of HNN. The MSE of HNN periodically touches zero does not mean has good prediction that time step. Since the trajectories in the phase space closed circles, those zeros mean the predicted trajectory of HNN lags behind (or runs ahead of) the true trajectory by one
circles. Also, the energy of the HNN trajectory drifts instead of staying constant, probably because the finite difference approximation is not accurate enough.
Table 1: Train error and prediction error per trajectory of Unstructured SymODEN and HNN. The train error per trajectory is the sum of MSE of all the 45 timesteps averaged over the 25 training trajectories. The prediction error per trajectory is the sum of MSE of 90 timesteps in trajectory.
annulus training data
rectangle training data
Models
train error prediction error
train error prediction error
Unstructured SymODEN
56.59
440.78
502.60
4363.87
HNN
290.67
564.16
5457.80
26209.17
D EFFECTS OF THE TIME HORIZON T
Incorporating the differential ODE solver also introduces two hyperparameters: solver types and time horizon For the solver types, the Euler solver is not accurate enough for our tasks. The adaptive solver "dopri5" lead to similar train error, test error and prediction error as the RK4 solver, but requires more time during training. Thus, in our experiments, we choose RK4.
Time horizon 7 is the number of points we use to construct our loss function. Table shows the train error, test error and prediction eигоr per trajectory in Task when is varied from to 5. We can see that longer time horizons lead to better models. This is expected since long time horizons penalize worse long term predictions. We also observe in experiments that longer time horizons require more time to train the models.
Table 2: Train emror test and prediction per trajectory of Task 2
Time Horizon
T
T
T
Train Error
0.744
0.136
0.068
0.033
0.017
Test Error
0.579
0.098
0.052
0.024
0.012
Prediction Error 3.138
0.502
0.199
0.095
0.048
E FULLY-ACTUATED CARTPOLE AND ACROBOT
CartPole and Acrobot are underactuated systems. Incorporating the control of underactuated sys- tems into the end-to-end learning framework is our future work. Here trained SymODEN on fully-actuated versions of Cartpole and Acrobot and synthesized controllers based on the learned model.
For the aully-actuated CartPole, Figure shows the snapshots of the system of a controlled trajectory with initial condition where the pole below the horizon. Figure 9 the time series of state
15

---

Published as a conference paper at ICLR 2020
variables and control inputs. We can successfully learn the dynamics and control the pole to the inverted position and the cart to the origin.
Figure Snapshots controlled trajectory fully-actuated CartPole system with 0.3s time interval.
U1
1.0
2.5
-2.5
10 U2
sinlg)
-5.0
Figure 9: Time series of state variables and control inputs of controlled trajectory shown in Figure 8 reference lines indicate expected value in end.
Black
For the fully-actuated Acrobot, Figure I0 shows the snapshots of controlled trajectory. Figure 1] shows the time series of state variables and control inputs. can successfully control the Acrobot from the downward position to the upward position, though the final value of q2 is little away from zero. Taking into account that the dynamics has been learned with only 64 different initial state conditions, is most likely that the upward position did not show up in the training data.
Figure 10: Snapshots a controlled trajectory of the fully-actuated Acrobot system with Is time interval.
91
15
20
10
15
10
15
20
Q2
d2
U2
1.0
56/))
0.5
0.0
-0.
-25
10
Figure 11: Time series state variables control inputs of a controlled trajectory shown in Figure 10 Black reference lines indicate expected value the end.
F TEST ERRORS OF THE TASKS
Here show statistics of train, test, and prediction per trajectory all four tasks. The train errors are based on 64 initial state conditions and 5 constant inputs. The test errors are based on 64 pre- viously unseen initial state conditions and the same constant inputs. Each trajectory in the train and test set contains 20 steps. The prediction error is based on the same 64 initial state conditions (during training) and zero inputs.
16

---

Published as a conference paper at ICLR 2020
Table 3: Train, Test and Prediction of the Four Tasks
Naive Baseline
Geometric Baseline
Unstructured Symplectic-ODS
Symplectic
Task 1: Pendulum
Model Parameter
0.36M
N/A
0.20M
0.13M
Train error
30.82 + 43.45
N/A
0.89 2.76
1.50 4.17
Test error
40.99 + 56.28
N/A
2.74 9.94
2.34 5.79
Prediction error
37.87 + 117.02
N/A
17.17 + 71.48
23.95 + 66.61
Task 2: Pendulum (embed)
Model Parameter
0.65M
0.46M
0.39M
0.14M
Train error
2.31 + 3.72
0.59 + 1.634
1.76 + 3.69
0.067 + 0.276
Test етror
2.18 + 3.59
0.49 + 1.762
1.41 2.82
0.052 + 0.241
Prediction error
317.21 + 521.46
14.31 + 29.54
3.69 - 7.72
0.20 + 0.49
Task3: CartPole
Model Parameter
1.01M
0.82M
0.67M
0.51M
Train error
15.53 = 22.52
0.45 + 0.37 1.20 + 2.67
4.84 + 4.42
1.78 + 1.81
Test error
25.42 + 38.49
6.90 8.66
1.89 1.81
Prediction error
332.44 245.24
52.26 + 73.25
225. - 194.24
11.41 + 16.06
Task 4: Acrobot
Model Parameter
1.46M
0.97M
0.78M
0.51M
Train
2.04 + 2.90
2.07 + 3.72
1.32 2.08
0.25 + 0.39
Test eтror
5.62 + 9.29
5.12 + 7.25
3.33 + 6.00
0.28 + 0.48
Prediction error
64.61 + 145.20
26.68 + 34.90
9.72 16.58
2.07 5.26
17


Reviews:
[{"review_id": "r1gsylCe9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "============ Update ===========\nThe authors have done a good job at addressing my concerns, and the revised version of the submission is substantially improved. I have adjusted my score and recommend accepting the paper.\n==============================\n\nRecent work has explored encoding analytic mechanics formulae into neural networks as inductive biases to learn physics models that generalize better. Neural networks are implemented to learn quantities like kinetic and potential energy rather coordinate derivatives. In this paper, the work of [1] is extended to incorporate a more generalizable approach to modeling functions on angles, integral approach where errors are backpropagated through an ODE solver rather than fitting errors in the derivatives, and modeling response to controls. \n\nFor Lagrangian and Hamiltonian systems it’s often easier to work with non-euclidean generalized coordinates rather than a constrained Euclidean system, however it can be difficult to design well parametrized neural network functions on a manifold like a circle. The authors address this by still expressing the having the Hamiltonian expressed in terms of circular generalized coordinates, but parametrizing the functions on the Euclidean embeddings. The paper shows that this approach does not have a problem with generalizing to large angles that the naïve approach does. \n\nThe integral approach to computing errors seems sensible, and appears to work well but no comparison is made to the previous method working with derivatives. This would be useful in demonstrating that the predictive performance is at least no worse than the approach taken in [1] and doesn’t require knowing or estimating derivatives. Also it would be good to have an ablation study investigating predictive performance as a function of tau, the number of integration timesteps for computing the error.\n\nThe paper shows evaluation of the learned dynamics system for control on two examples, an inverted pendulum and CartPole. In the inverted pendulum example, the learned potential energy and the control response is used to design a control that shapes the potential energy and with additional damping. Since the control output is closely related to a PD controller, it would be good to compare against a standard P(I)D controller that doesn’t depend on the model. For the CartPole system, the control is exactly a PD controller and it’s not clear how the Hamiltonian/dynamics model are used at all in this example. Generally the paper does a good job at demonstrating benefits in modeling ability and generalization, but the experiments applying this model to control are not very convincing. Having an example where the model is applied for a standard approach like MPC for one of these problems would useful for gauging efficacy in possible control applications.\n\nThere are some promising leads explored in this paper for learning physical system dynamics effectively with neural nets. I think there is a lot of promise in the approach, however some of the improvements over past work have not been adequately tested (integral approach and sensitivity to # of integration steps) and the control experiments are not very convincing although it seems like they could be. I lean towards a weak reject for the paper as is, but if the authors flesh out the control experiments and do some ablation studies I will improve my score.\n\nMinor Comments and Questions:\nIs f tilde in equation 11 parametrized to have 0 output on u dot or is it expected that this relationship would simply be approximated by the model?\n\n[1] Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. arXiv:1906.01563, 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxvhJ_0YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I have read the author's response. Thank you!\n***\n\nThis is an excellent paper that integrates inductive biases from physics into learning dynamical models that can then be integrated into deep RL-based control tasks.\n\nThe model approximates the dynamical function f(q, p, u), where q are the generalised coordinates of the system (mixture of positions in R^1 and angles in S^1), p are the generalised momenta and u is the external control input. Function f can then be integrated by a numerical solver, and used as the dynamical function in a Neural ODE (ordinary differential equation) for modelling the continuous time evolution of the nonlinear dynamical system. The dynamics function is explicitly written as the equations of the Hamiltonian dynamical system, involving the 1) inverse of the mass function, 2) potential energy and 3) control function, in a complex graph (Figure 1) that transforms positional and angular coordinates and momenta x and the external control into f(x, u).\n\nThe derivation of the method is long but very well written and didactic. The experiments on the control suite of OpenAI Gym  are simple (pendulum and cart-pole only) but thorough, and compare the proposed method with a non-inductive bias method (still relying on Neural ODEs), a simpler naive and geometric baselines. Overall, the paper is very easy to follow (even for someone who does not work on control experiments in deep RL) while addressing complex physics.\n\nMinor remarks:\nCan you define g in 2.1?\nCan you add the derivation of equations (8) through (10) in the appendix?\nThe second to last sentence on page 4 seems unfinished.\nCan the authors comment on how this model would scale to larger (e.g. multiple joints) dynamical systems?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1l1Ek92KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a framework for learning the dynamics of a system with underlying Hamiltonian structure subjected to external control. Based on the extended equations of motion, the authors suggest how to apply NeuralODE in a way that makes use of the prior information that the unconstrained system is Hamiltonian and subjected to a control term. For a range of tasks, the authors then demonstrate that the proposed SymODEN framework can learn the dynamics and recover the known analytical solution, and that they can derive a controller that allows them to drive the system to a target configuration. \n\nI find this paper very interesting and the formulation elegant. In particular, I appreciate that the paper is pretty much self-contained and that the authors derive the theory from first principles. However, there are a couple of points (listed below) that should be addressed prior to publication to improve the clarity of the paper, and to help the reader to fully appreciate the depth of the experimental section. If these points were addressed in sufficient detail, I would be willing to increase the score:\n\n1) Reading the abstract and the introduction, I got the impression that SymODEN can be applied to any physical system while the method is in fact only applicable to systems governed by Eq. (11). I think it’s worth mentioning in the text that not every physical system follows (constrained) Hamiltonian Dynamics. \n\n2) You mention that a controller is designed based on the learned dynamics. For example, in the first sentence in Sec. 2.2: ‘Once the dynamics of a system have been learned, it can be used to synthesize a controller to maneuver the system to a reference configuration q*’. I think it is important to specify here which dynamics you are referring to, i.e. the constrained or unconstrained dynamics. Later on it becomes clear that you use constant-u training data and that u is part of the input to the model, so it has to be constrained dynamics, but at this stage it is still unclear to the reader.\n\n3) In Eq. (11), you set du/dt = 0 without motivating this restriction. Can you provide at least one sentence on why this is an interesting choice and back it up with a reference? Furthermore, the sentence above Eq. (11) is broken and needs fixing. \n\n4) Tasks (general): You consider a range of tasks and I appreciate that you start with a simple and intuitive system. However, I think a bit more guidance throughout the tasks section would be very helpful. In particular, I would suggest that you provide a short (one or two sentences) summary at the beginning of each task to say what exactly it is that you are trying to test or demonstrate. Furthermore, I would suggest showing the summary of  results, i.e. Sec. 4.2, after you introduce the individual tasks rather than before.\n\n5) Task 2: This task addresses multiple things in one go: Initially, you demonstrate that you can recover the results of Task 1 without access to the generalised momenta, and explain why this can only be done up to a constant scaling factor. This is very interesting and clear. However, then you jump straight into the controller and things become a little unclear because, at this stage, it still seems that the dynamics are the same as in Task 1, i.e. unconstrained. Please add a sentence or two for clarification. Another important aspect that is not commented on at all is the behaviour of u(t) in Eq. (27). In particular, isn’t u(t) expected to satisfy du/dt = 0 based on Eq. (11)? The results in Fig. 6 suggest that this is not the case (see time interval [2, 6]). This seems like an interesting and surprising behaviour, especially because SymODEN was only trained with constant-u training data. I would appreciate if the authors could comment on this. I would also suggest to add horizontal lines to Fig. 6 to indicate the expected results.\n\n6) Task 4: Why did you not explain this task in a dedicated section like you did for all other tasks?\n\n7) Symplectic: Since both the method and the title of the paper contain the word ‘symplectic’, it would be good if you explained what the term actually means.\n\n8) ‘Our results show that incorporation of such physics-based inductive bias can provide knowledge about relevant physical properties (mass, potential energy) and laws (conservation of energy)...’. To me, this statement is slightly misleading. You did not demonstrate that SymODEN ‘provides knowledge’ of laws of the system; energy conservation (for u = 0) as a law is hard-coded into your network. The specific value of the energy can be inferred but that I would consider a physical property. I would suggest to change the wording to reflect this clearly.\n\n9) Introduce the acronym ODE much earlier than in Sec. 3.1.\n\n10) Model training: What happens if you use unseen initial conditions rather than the ones in the training data? Perhaps you could add a comment to clarify.\n\n11) There are many typos and grammar mistakes in the paper. Please revise it carefully. To give you a few examples:\n‘are both reformulation’ -> ‘are both reformulations’\nSec 2.1: Decide on whether you use plural or singular for ‘dynamics’ and be consistent.\n‘on a equal footing’ -> ‘on an equal footing’ \n‘beyond classical mechanics, the Hamiltonian’ -> ‘Beyond classical mechanics, Hamiltonian …’\n‘Hamiltonian is same as’ -> ‘Hamiltonian is the same as’\n‘represents potential energy’ -> ‘represents the potential energy’\n‘trajectory actually converge to’ -> ‘trajectory actually converges to’\n‘a ODE solver -> ‘an ODE solver’.\n‘Lagrangian and Hamiltonian formulation’ -> ‘Lagrangian and Hamiltonian formulations’\n‘assume that q and p evolves’ -> ‘assume that q and p evolve’\n‘translational coordinate’ -> ‘translational coordinates’\n‘naive baseline model approximate’ -> ‘naive baseline model approximates’\netc.\n\n\n*************************************\nThe authors addressed my comments and answered  my questions clearly.  I therefore increased the score. *************************************", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for learning Hamiltonian dynamics from data. The data is obtained from systems subjected to an external control signal. The authors show the utility of their method for subsequent improved control in a reinforcement learning setting. The paper is well written, the method is derived from first principles, and the experimental validation is solid. The authors were also able to take into account the reviewers’ feedback and further improve their paper during the discussion period. Overall all of the reviewers agree that this is a great contribution to the field and hence I am happy to recommend acceptance.
Decision: Accept (Poster)