Hamiltonian Generative Networks
Authors: Peter Toth
Danilo J. Rezende
Andrew Jaegle
Sébastien Racanière
Aleksandar Botev
Irina Higgins
Venue:  (2020)
Keywords: Hamiltonian dynamics
normalising flows
generative model
physics

Abstract: The Hamiltonian formalism plays a central role in classical and quantum physics. Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserved quantities, and they come equipped with many useful properties, like time reversibility and smooth interpolation in time. These properties are important for many machine learning problems - from sequence prediction to reinforcement learning and density modelling - but are not typically provided out of the box by standard tools such as recurrent neural networks. In this paper, we introduce the Hamiltonian Generative Network (HGN), the first approach capable of consistently learning Hamiltonian dynamics from high-dimensional observations (such as images) without restrictive domain assumptions. Once trained, we can use HGN to sample new trajectories, perform rollouts both forward and backward in time, and even speed up or slow down the learned dynamics. We demonstrate how a simple modification of the network architecture turns HGN into a powerful normalising flow model, called Neural Hamiltonian Flow (NHF), that uses Hamiltonian dynamics to model expressive densities. Hence, we hope that our work serves as a first practical demonstration of the value that the Hamiltonian formalism can bring to machine learning. More results and video evaluations are available at: http://tiny.cc/hgn

Published as a conference paper at ICLR 2020
HAMILTONIAN GENERATIVE NETWORKS
Peter Toth
Danilo J. Rezende" DeepMind danilorê wowgle.oooooo.ooooooooooooooooo com
Andrew Jaegle DeepMind d jrewjaegge@@ooogllo
DeepMind
petertothhgoogle. com
Sébastien Racanière
Aleksandar Botev DeepMindd botevêgooglle.co
Irina Higgins DeepMind ir szinan@google.com
DeepMind
sracaniere@google.com
ABSTRACT
The Hamiltonian formalism plays central role in classical and quantum physies. Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserved quantities, and they come equipped with many useful prop- erties, like time reversibility and smooth interpolation in time. These properties are important for many machine learning problems from sequence prediction rein- forcement learning and density modelling but are not typically provided out of the box by standard tools such as recurrent neural networks. In this paper, we introduce the Hamiltonian Generative Network (HGN), the first approach capable of consis- tently learning Hamiltonian dynamics from high-dimensional observations (such images) without restrictive domain assumptions. Once trained, we can use HGN to sample new trajectories, perform rollouts both forward and backward in time and even speed up slow down the learned dynamics, We demonstrate how simple modification of the network architecture turns HGN into a powerful normalising flow model, called Neural Hamiltonian Flow (NHF), that uses Hamiltonian dynamics model expressive densities. We hope that our work serves as a first practical demon- stration of the value that the Hamiltonian formalism can bring deep learning.
INTRODUCTION
Any system capable of wide range of intelli- gent behaviours within dynamic environment requires good predictive model of the environ- ment's dynamics. This is true for intelligence in both biological (Friston 2009 2010 Clark] 2013) and artificial (Hafner et al. 2019 Battaglin ha et al. 2019 systems. Predicting environmental dynam- ics is also of fundamental importance in physics, where Hamiltonian dynamies and the structure- preserving transformations it provides have been used unify, categorise and discover new phys- ical entities (Noether 1915) Livio 2012).
Image manifold
2013 Watter et al.) 2015 Watters et al.)
Path on manifold induced by Hamiltonian
Hamilton's fundamental result was system of two first-order differential equations that, in stroke, unified the predictions made by prior Newtonian and Lagrangian mechanics (Hamil- ton 1834). After well over a century devel- opment, it has proven to be essential for parsi- monious descriptions of nearly of physics.
Figure I:
The Hamiltonian manifold hypothesis: natural images lie on a low-dimensional manifold pixel space, and natural image sequences (such as one produced by watching a two-body system, as shown red) correspond movement on the manifold according to Hamiltonian dynamics.
"Equal contribution.
More results and video evaluations are available http::/tiny.cc///g/

---

Published as a conference paper at ICLR 2020
Hamiltonian
%
Inference
Decoder
2- Ki
Forward rollout
+dt
-dt
Reverse rollout
Figure Hamiltonian Generative Network schematic. The encoder takes a stacked sequence images and infers the posterior over the initial state. The state is rolled out using the learnt Hamiltonian. Note that we depict Euler updates of the state for schematic simplicity, while in practice this is done using leapfrog integrator. For each unroll step we reconstruct the image from the position q state variables only and calculate the reconstruction error.
Hamilton's 's equations provide a way to predict a system's 's future behavior from its current state in phase space (that is, its position and momentum for classical Newtonian systems, and its generalized position and momentum more broadly). Hamiltonian mechanics induce dynamics with several nice properties: they are smooth, they include paths along which certain physical quantities are conserved (symmetries) and their time evolution is fully reversible. These properties are also useful for machine learning systems. For example, capturing the time-reversible dynamics of the world state might be useful for agents attempting to account for how their actions led to effects in the world; recovering an abstract -i-dimennional manifold with paths that conserve various properties tightly connected to outstanding problems in representation learning (see e.g. Higgins et al. (2018) for more discussion); and the ability to conserve energy is related to expressive density modelling in generative approaches (Rezende & Mohamed 2015). Hence, we propose reformulation of the well-known image manifold hypothesis by extending it with Hamiltonian assumption (illustrated in Fig. [ natural images lie on a low-dimensional manifold embedded within a high-dimensional pixel space and natural sequences of images frace out paths on this manifold that follow the equations of an abstract Hamiltonian.
Given the rich set of established tools provided by Hamiltonian descriptions of system dynamics, can we adapt these to solve outstanding machine learning problems? When it comes to adapting the Hamiltonian formalism to contemporary machine learning, two questions need to be addressed: 1) how should system's 's Hamiltonian be learned from data; and 2) how should system' 's abstract phase space be inferred from the digensiimnnion observations typically available machine learning systems? Note that the inferred state may need to include information about properties that play no physical role in classical mechanics but which can still affect their behavior function, like the colour or shape of an object. The first question was recently addressed by the Hamiltonian Neural Network (HNN) (Greydanus al. 2019) approach, which was able to learn the Hamiltonian of three simple physical systems from noisy phase space observations. However, address the second question, HNN makes assumptions that restrict to Newtonian systems and appear to limit its ability to scale to more challenging video datasets.
In this paper we introduce the first model that answers both of these questions without relying on re- strictive domain assumptions. Our model, the Hamiltonian Generative Network (HGN), is a generative model that infers the abstract state from pixels and then unrolls the learned Hamiltonian following the Hamiltonian equations (Goldstein 1980). We demonstrate that HGN is able reliably learn the Hamil-
2

---

Published as a conference paper at ICLR 2020
tonian dynamics from noisy pixel observations on four simulated physical systems: a pendulum, mass-spring and two- and three- body systems. Our approacch outperforms HNN by a significant margin. After training, we demonstrate that HGN produces meaningful samples with reversible dynamics and that the speed of rollouts can be controlled by changing the time derivative of the integrator at test time. Finally, show that small modification of our architecture yields flexible, normalising flow-based generative model that respects Hamiltonian dynamics. We show that this model, which we call Neural Hamiltonian Flow (NHF), inherits the beneficial properties of the Hamiltonian formalism (including volume preservation) and is capable of expressive density modelling, while offering computational benefits over standard flow-based models.
RELATED WORK
Most machine learning approaches to modeling dynamics use discrete time steps, which often results in an accumulation of the approximation errors when producing rollouts and, therefore, to a fast drop in accuracy. Our approach, on the other hand, does not discretise continuous dynamics and models them directly using the Hamiltonian differential equations, which leads to slower divergence for longer rollouts. The density model version of HGN (NHF) uses the Hamiltonian dynamics as normalising flows along with numerical integrator, making our approach somewhat related to the recently published neural ODE work (Chen et al. 2018; Grathwohl et al. 2018). What makes our approach different is that Hamiltonian dynamics are both invertible and puuueeprrsrriing (as discussed in Sec. (.3), which makes our approach computationally cheaper than the alternatives and more suitable as a model of physical systems and other processes that have these properties. Also related is recent work attempting to learn a model of physical system dynamics end-to-end from image sequences using an autoencoder (de Avila Belbute- Peres et al. 2018). Unlike our work, this model does not exploit Hamiltonian dynamics and is trained in supervised or semi-supervised regime.
2.1 HAMILTONIAN NEURAL NETWORK
One of the most comparable approaches to ours is the Hamiltonian Neural Network (HNN) (Greydanus 2019). This work, done concurrentyy to ours, proposes way to learn Hamiltonian dynamics from data by training the gradients of neural network (obtained by backpropagation) to match the time derivative of target system in supervised fashion. In particular, HNN learns a differentiable function H((q.p) that maps a system's 's state (its position, q. and momentum, p) to a scalar quantity interpreted as the system's 's Hamiltonian. This model is trained that H (p.q) satisfies the Hamiltonian equation by minimizing
1. H dg
d
OH dp 0g dt 2
CHNN = 1 0p dt
+
(1)
where the derivatives BR and 8H By Op are computed by backpropagation. Hence, this learning procedure is most directly applicable when the true state space (in canonical coordinates) and its time derivatives known. Accordingly, in the majority of the experiments presented by the authors, the Hamiltonian was learned from the ground truth state space directly, rather than from pixel observations.
The single experiment with pixel observations required modification of the model. First, the input to the model became concatenated, flattened pair of images o [x1,, which was then mapped to dimeeeenniiaal embedding space 2, = [qc.Pr] using an encoder neural network. Note that the dimensionality of this embedding (= E R in the case of the pendulum system presented in the paper) was chosen to perfectly match the ground truth dimensionality of the phase space, which was assumed to be known priori. This, however, is not always possible. The latent embedding was then treated as an estimate of the position and the momentum of the system depicted in the images, where the momentum was assumed to be equal to the velocity of the system an assumption enforced by the additional constraint found necessary to encourage learning, which encouraged the time derivative of the position latent to equal the momentum latent using finite differences the split latents:
Lec (pc 0(141 -
(2)
3

---

Published as a conference paper at ICLR 2020
This assumption is appropriate in the case of the simple pendulum system presented in the paper, however it does not hold more generally. Note that our approach does not make any assumptions on the dimensionality of the learned phase space, the form of the momenta coordinates, which makes our approach more general and allows to perform well on a wider range of image domains as presented in $
3
METHODS
3.1 THE HAMILTONIAN FORMALISM
The Hamiltonian formalism describes the continuous time evolution of system in an abstract phase space 8 = p) E - where q E R" is a vector of position coordinates, and p e R" is the corresponding vector of momenta. The time evolution of the system in phase space is given by the Hamiltonian equations:
0q
8H.
0p
0H
(3)
Ot
0p"
Ot
0q
where the Hamiltonian H: 2 R maps the state s= (q.p) to scalar representing the energy of the system. The Hamiltonian specifies vector field over the phase space that describes all possible dynamics of the system. For example, the Hamiltonian for an undamped mass-spring system is H(q.p)== = kkg² + e² where m is the mass, E is its position, P IR is its momentum and k the spring stiffness coefficient. The Hamiltonian can often be expressed as the sum of the kinetic T and po- tential V energies 7 = (p) + (q). ()). as is the case for the mass- sprinn example. Identifying system's 's Hamiltonian in general a very difficult problem, requiring carefully instrumented experiments and researcher insight produced by years of training. In what follows, we describe method for modeling system's 's Hamiltonian from raw observations (such as pixels) by inferring a system's 's state with generative model and rolling out with the Hamiltonian equations.
3.2 LEARNING HAMILTONIANS WITH THE HAMILTONIAN GENERATIVE NETWORK
Our goal is to build a model that can learn a Hamiltonian from observations. We assume that the data X= [=x]
(T) x=((=])....t)) "........)) (x. ...,,.f } comes in the form of high-dimensional noisy observations, where each x; = G(,,) = -((;) a determttrii function of the generalised position in the phase space, and the full state is non-deterministic function of sequence of images s, (q.P,;) = =D(x))....x)) since the momentum (and hence the full state) cannot in general be recovered from a single observation. Our goal is to infer the abstract state and learn the Hamiltonian dynamics in phase space by observing K motion sequences, discretised into T'+ 1 time steps each. In the process, we also want to learn an approximation the generative process G(s) in order be able move in both directions between the high dimensional observations and the now-dimensional abstract phase space.
Although the Hamiltonian formalism general and does not depend on the form of the observations, we present our model in terms of visual observations, since many known physical Hamiltonian systems, like mass-spring system, can be easily observed visually. In this section we introduce the Hamiltonian Generative Network (HGN), a generative model that is trained to behave according to the Hamiltonian dynamics an abstract phase space learned from raw observations of image sequences. HGN consists of three parts (see Fig. 2 inference network, a Hamiltonian network and decoder network, which are discussed next.
The inference network takes in sequence of images (x) concatenated along the channel dimension, and outputs a posterior over the initial state ~ (- corresponding to the system's is coordinates in phase space at the first frame of the sequence. We parametrise go(2) as a diagonal Gaussian with a unit Gaussian prior p(z) = AO, II) and optimise it using the usual reparametrisation trick (Kingma & Welling 2014). To increase the expressivity of the abstract phase space Sg, we map samples from the posterior with another function su = f (2) to obtain the system's initial state. As mentioned in Sec. the Hamiltonian function expects the state to be of the form s= = (q.p). hence we initialise sn E 2m and arbitrarily assign the first half of the units to represent abstract position g and the other half represent abstract momentum p.
4

---

Published as a conference paper at ICLR 2020
A n((u)
B
p(x)
% x(5))
8,- p(s)
f
x
90
Figure 3: A: standard normalising flow, where the invertible function fi implemented by neural the initial density is transformed using the learned Hamiltonian
network. B: Hamiltonian flows, where dynamics. Note that we depict Fuler updates of the state for schematic simplicity, while in practice this done using leapfrog integrator.
an2 dt
H4
Pg TR..
Inference
-d
+dt
Sampling
Figure 4: A schematic representation of NHF which can perform expressive density modelling by using the learned Hamiltonians as normalising flows. Note that we depict Euler updates of the state for schematic simplicity, while in practice this is done using leapfrog integrator.
The Hamiltonian network is parametrised inferred abstract state and maps it to a scalar H, (s;)) E IR. We can use this function to do rollouts in the abstract state space using the Hamiltonian equations (Eq. 3), for example by Euler integration: SH dt). In this work we assume a separable Hamiltonian, so in practice we use more sophisticated leapfrog integrator to roll out the system, since it has better theoretical properties and results in better performance in practice (see Sec. A.6 in Supplementary
neural network with parameters 7 that takes in the
8t4 g++ 1 1P++1) (q: 5
op, dt, zq:
Materials for more details).
The decoder network a standard deconvolutional network (we use the architecture from Karras al. (018)) that takes in low-dimension representation vector and produces high-dimensional pixel reconstruction. Given that each instantaneous image does not depend on the momentum in- formation, restrict the decoder to take only the position coordinates of the abstract state as input: =
The objective function. objective:
Given sequence of T images, HGN is trained optimise the following
C(o.b.7 FT)
go (xle)....)))) [log Por ( qq) |- KL (z) (L( )
T+1
which can be seen as temporally extended variational autoencoder (VAE) Kingmma & Welling 2014
Rezende al. (214)) objective, consisting reconstruction term for each frame, and an additional term that encourages the inferred posterior to match a prior. The key difference with a standard VAE lies in how we generate rollouts these are produced using the Hamiltonian equations of motion in learned Hamiltonian phase space.
3.3 LEARNING HAMILTONIAN FLOWS
In this section, we describe how the architecture described above can be modified to produce a model for flexible density estimation. Learning computationally feasible and accurate estimates of complex
5

---

Published as a conference paper at ICLR 2020
Pendulum
Mass-spring
2- & 3-Body Problem
((pup) myl(1 cvela)) 2nd!!
Hiyaa
Misdi
2m1
Figure 5: Ground truth Hamiltonians and samples from generated datasets for the ideal pendulum, mass-spring. and two- and three-body systems used to train HGN.
densities is an open problem in generative modelling. A common idea address this problem is to start with a simple prior distribution π (u) and then transform it into more expressive form p(æ) through series of composable invertible transformations fi (() called normalising flows (Rezende & Mohamed 2015) (see Fig. A). Sampling can then be done according to x fr....o o fi fu(,, where [~- Density evaluation, however, requires more expensive computations of both inverting the flows and calculating the determinants of their Jacobians. For a single flow step, this equates to the
following: p(=)= - (f (z:) det 05
While lot of work has been done recently into proposing
better alternatives for flow-based generative models in machine learning (Rezende & MMohamed] [0155]
Kingma et al., 2016 Papamakarios et al. 2017] Dinh et al. 2017 Huang et al. [2018] Kinggaa &
Dhariwal 2018 Hoogeboom et al. 2019 Chen al. 2019: Grathwohl et al. (018), none of the
approaches manage to produce both sampling and density evaluation steps that are computationally scalable.
The two requirements for normalising flows are that they are invertible and volume preserving, which exactly the two properties that Hamiltonian dynamics possess. This can be seen by computing the determinant of the Jacobian of the infinitesimal transformation induced by the Hamiltonian H. By Jacobi's ' formula, the derivative of the determinant at the identity is the trace and:
8PH Squ 8P Bp.app
8 Bq 8 3p.
1
]
8PH Baipp 8'H Sp.Ops
8P
det
I+ dt
-1
1+++t dt Tr
0q.aas 8 Sp.Das
+0(df)=1+0(() (5)
where j j are the off- diagonal entries of the determinant of the Jacobian. Hence, in this section we describe simple modification of HGN that allows it to act normalising flow. We will refer to this modification as the Neural Hamiltonian Flow (NHF) model. First, we assume that the initial state 8p is a sample from simple prior So (-))). We then chain several Hamiltonians Hi to transform the sample to new state $T = HT c Hi ( (sp) which corresponds to sample from the more expressive final density 8T p(x)) ) (see Fig. B for illustration of single Hamiltonian flow). Note that unlike HGN, where the Hamiltonian dynamics are shared across time steps (a single Hamiltonian is learned and its parameters are shared across time steps of rollout), NHF each step of the flow (corresponding to single time step of a rollout) can be parametrised by different Hamiltonian. The inverse of such a Hamiltonian flow can be easily obtained by replacing dt by -dt in the Hamiltonian equations and reversing the order of the transformations, Sy = Hi o... H. (sT) (we will use the appropriate dt or ddt superscript from now on to make the direction of integration of the Hamiltonian dynamics more explicit). The resulting density p(sT) is given by the following equation:
In p(s)) In . (8o) =In (HH, o.. (
Our proposed NHF is more computationally efficient that many other flow- approaches, because it does not require the expensive step of calculating the trace of the Jacobian. Hence, the NHF model constitutes a more structured form of Neural ODE flow (Chen et al.) (018)), but with a few notable differences: (i) The Hamiltonian ODE is volume- prenneepeeer,ing, which makes the computation log- likelihood cheaper than for general ODE flow. (ii) General ODE flows are only invertible in the limit dt - 0, whereas for some Hamiltonians we can use more complex integrators (like the symplectic leapfrog integrator described in Sec. A.6) that are both invertible and volume preserving for any dt > 0. The structure s = (q.p) on the state-space imposed by the Hamiltonian dynamics can be constraining from the point of view of density estimation. We choose to use the trick proposed in the Hamiltonian

---

Published as a conference paper at ICLR 2020
Pendulum
Mass-spring
Train MSE
0.20
0.20
0.22
00:00
Tost
0.53
HNN
HGN (lespfrog)
HGN (Euler)
HGN (deter))
HGN (no f
Two-Bon Two-Body
Three-Body Body
MS6
1.50
0.44
105 MIE
Figure 6: Average pixel MSE for each step of single train and test unroll on four physical systems. All versions of HGN outperform HNN, which often learned reconstruct an average image. Note the different scales of the plots comparing HGN to HNN and different versions of HGN.
Monte Carlo (HMC) literature (Neal et al. 2011 Salimans et al. 2015 Levy et al. 2017), which treats the momentum p latent variable (see Fig. 4h. This is an elegant solution which avoids having to artificially split the density into two disjoint sets. As a result, the data density that our Hamiltonian flows modelling becomes exclusively parametrised by p(qr), ) which takes the following form: This integral is intractable, but the
p(gr) = f [p(rr-Pu (ellggrPr)ddd Pdpr ff [ o.. (((---r))dd)) model can still be trained via variational methods where we introduce a variational density f (Pr (Tr)
with parameters and instead optimise the following ELBO:
ELBO(qr)= =Es (pr ar) In (HH," o...o H..
-n (prlar) $ In p(gr),
(6)
Note that, in contrast to VAEs (Kingma & Welling 2014 Rezende al. 2014), the ELBO in Eq 6 is
not explicitly in the form of a reconstruction error term plus a KL. term.
4 RESULTS
In order to directly compare the performance of HGN to that of its closest baseline, HNN, we generated four datasets analogous the data used in Greydanus al. (2019). The datasets contained observations of the time evolution of four physical systems: mass-spring. pendulum, two- and three-body (see 5). In order generate each trajectory. we first randomly sampled an initial state, then produced 30 step rollout following the ground truth Hamiltonian dynamics, before adding Gaussian noise with standard deviation 2² =0.1 1 each phase-space coordinate, and rendering a corresponding 64x64 pixel observation. We generated 50 000 train and 10 000 test trajectories for each dataset. When sampling initial states, we start by first sampling the total energy of the system denoted as radius r in the phase space, before sampling the initial state (q.p) uniformly on the circle of radius Note that our pendulum dataset is more challenging than the one described in Greydanus et al. (2019), where the pendulum had fixed radius and was initialized maximum angle of 30" from the central axis.
Mass-spring. The dynamics of frictionless mass-spring system are modeled by the Hamiltonian
p² H. = 2kq + 2m where is the spring constant and m is the mass. We fix & = and m = 0.5, then sample radius from uniform distribution -~ UO(...11..)
7

---

Published as a conference paper at ICLR 2020
Pendulum.
The dynamics of frictionless pendulum modeled by the Hamiltonian 7/ 1122mgl(1- cos (q)) + 2m/22 where g is the gravitational constant and is the length of the pendulum. We fix g=3. 0.5.5, 1= then sample a radius from uniform distribution r~ (1.3.333)
y²
Two- and three- body problems. In an n-body problem, particles interact with each other through an attractive force, like gravity. The dynamics are represented by the following Hamiltonian H =
2m; Eisicien bg T We set m= = =1 and g g= 1 for both systems. For the two-body problem, we set r ~ 1,5,,, and we also change the observation noise to o² 0.05. For the three-body problem, we set U~~-~~~000000000 (0.9.11.)) and set the observation noise to o² 0.2.
Σx l1p:1:
Learning the Hamiltonian We tested whether HGN and the HNN baseline could learn the dynamics of the four systems described above. To ensure that our e--imllemettatinn of HNN was correct, we replicated all the results presented in the original paper GGryydanu et al. 2019) by verifying that it could learn the dynamics of the mass-spring, pendulum and two-body systems well from the ground truth state, and the dynamics of a restricted pendulum from pixels. We also compared different modifications of HGN: version trained and tested with an Euler rather than a leapfrog integrator (HGN Fuler), a version trained with no additional function between the posterior and the prior (HGN no fw) and a deterministic version (HGN determ), which did not include the sampling step from the posterior qo (z ET)
MnDRI
M MASS-SPRING
PENDULUM
TWO-RODY
THREI-RODY
ToxT
TRAIN
TRAIN
TUST
TRAIN
HNN
50.3811175
504.37=52.61
09
61.6220.87
80.52= 1.79
78.921.93
61.6313.31
57.5412.30
HNN (CONV)
16.97
119.09::9.11
13.14
106.07 41.994
0.3.
120.714
15.22 1.99
02.77:37.57
HGN (BULL3))
3.67+1.09
6.2+2.69
2.53
10.93+4.32
6.6243.93
15.0647.01
7.61=3.40
9.443.02
HGN (USTERM)
0.23-4023
1.07
0.79+1.24
10.68
2.34= 2.3
14.47+6.24
4.142.05
5.1741.96
HGN (NO f()
4.96+1..7
7.04
6.83
13.9844.94
6.36
16.40::6.6
8.37=3.13
10.41
HGN (LEAPFROG)
3.84:11.07
6.23412.0
1.9.1.66
11.72144.14
6.36
16.47:7.115
3.88
9.813.72
Table 1: Average pixel MSE over a 30 step unroll on the train and test data on four physical systems. All values are multiplied by 1.+44. We evaluate two versions of the Hamiltonian Neural Network (HNN) (Greydanus et al. 2019): the original architecture and a convolutional version closely matched the architecture of HGN. We also compare four versions of our proposed Hamiltonian Generative Network (HGN): the full version, version trained and tested with an Fuler rather than a leapfrog integrator, a deterministic rather than generative version, and version of HGN with extra network between the posterior and the initial state.
Tal Fig. demmonttrtee that HGN and its modifications learned well on all four datasets. However, when we attempted to train HNN on the four datasets described above, its Hamiltonian often collapsed to 0 and the model failed to reproduce any dynamics, defaulting to a static single image. We were unable to improve on this performance despite our best efforts, including a modification of the architecture to closely match ours (referred to as HNN Conv) (see Sec. A.3 of the appendix for details). Tbl. [shows that the average mean squared error (MSE) of the pixel reconstructions on both the train and test data is an order of magnitude better for HGN compared to both versions of HNN. The same holds when visualising the average per-frame MSE of single train and test rollout for each dataset shown in Fig. 6
Note that the different versions of HGN have different trade-offs. The deterministic version produces more accurate reconstructions but it does not allow sampling. This effect is equivalent to a similar distinction between autoencoders (Hinton & Salakhutdinov 2006) and VAEs. Using the simpler Euler integrator rather than the more involved leapfrog one might be conceptually more appealing, however it does not provide the same energy conservation and reversibility properties as the leapfrog integrator, as evidenced by the increase by an order of magnitude of the variance of the learned Hamiltonian throughout sequence rollout as shown in Tbl. [] The full version of HGN, on the other hand, is capable of reproducing the dynamics well, is capable of producing diverse yet plausible rollout samples (Fig. 8) and its rollouts can be reversed in time, sped up or slowed down by either changing the value or the sign of dt used in the integrator (Fig.
Expressive density modelling using learned Hamiltonian flows We evaluate whether NHF is capable of expressive density modelling by stacking learned Hamiltonians into series of normalising
8

---

Published as a conference paper at ICLR 2020
MODEL
TWO-BODY
TRAIN
TIST
TRAIN
TEST
INNN HNN (OONV)
N/A
N/A
NIA
N/A
N7A
2.17
28.34
HGN (BULas))
157.28
120.16
135.53
19.53
465.83
682.02
1.18
8.36
HGN
HGN (ND fe)
0.02
0.02
0.02
0.04
0.04
0.02
4.03
1.82
TGGN
Table 2: Variance of the Hamiltonian on four physical systems over single train and test rollouts shown in Fig. 6 The numbers reported scaled by factor of 1e++4. High variance indicates that the energy not conserved by the learned Hamiltonian throughout the rollout. Many HNN Hamiltonians have collapsed to 0, indicated by N/A. HGN Hamiltonians meaningful, and different versions of HGN conserve energy to varying degrees.
Original Cpaind
Origial
sped uncolf
forwaduurel
alooed cown foroare uneal (-Rce) (-kat)
Figure 7: Example of train and test sequence from the dataset of three-body system, its inferred forward, backward, double speed and half speed rollouts in time from HGN, and forward rollout from
HNN. HNN did not learn the dynamics the system and instead learned to reconstruct an average image.
Pendulum
Iwo-body
Sample Saniaa
Mass-spring
Sampe: Sencil
Figure 8: Examples of sample rollouts for all four datasets from a trained HGN.
flows. Fig. 9 demonstratts that NHF can transform a simple soft- uniform prior distribution тT T(S0:a,,3 9,3, into significantly more complex densities with arbitrary numbers modes. The soft-uniform density, n (Su;o, (3) f(ā(s a f)){(( (-B(s a :1 ))), where is the sigmoid function and ,3 is constant, was chosen to make easier to visualise the learned attractors. The model also performed well with other priors, including Normal distribution. It is interesting to note that the trained model is very interpretable. When decomposed into the equivalents the kinetic and potential energies, can be seen that the learned potential energy V(q) learned to have several local minima, one for each mode of the data. As a consequence, the trajectory of the initial samples through the flow has attractors at the modes of the data. We have also compared the performance of NHF to that of the RNVP baseline

---

Published as a conference paper at ICLR 2020
Terget density
Learned density
Kinetic energy
Potential energy
Single step
Intag ated fiow
D
8
0
Figure 9: Multimodal density learning using Hamiltonian flows. From left to right: KDE estimators of the target and learned densities; learned kinetic energy K (p) and potential energy V V(g): single leapfrog step and an integrated flow. The potential energy learned multiple attractors, also clearly visible in the integrated flow plot. The basins attraction are centred at the modes of the data.
Terpet dlensitty
Hamiltonian Flow(nt 1)
4.5
Hamiltonian Flow(nt 2)
Hamiltonian Flow(nt
Neg-ELBO (nats)
4.4
NVP(2 blocks)
4.3
NVP(3 blocks)
(nt=2)
(nte3)
4.2
4.1
NVP (2 blocks)
NVP (3 blocks)
4.0
3.9 0
2000
4000
6000
8000
10000
Iterations (x50)
Figure 10: Comparison between RNVP (Dinh al. The RNVPs use alternating masks with two layers (red) or layers (purple). NHF usFuses 1, or leapfrog steps (blue, yellow and green respectively).
2017) and NHF on Gaussian Mixture Dataset.
(Dinh et al. 2017 Fig. shows that the two approaches comparable in their performance, but NHF is more computationally efficient as discussed at the end of Sec. 3.3
5 CONCLUSIONS
We have presented HGN, the first deep learning approach capable of reliably learning Hamiltonian dynamics from pixel observations. We have evaluated our approach four classical physical systems and demonstrated that outperformed the only relevant baseline by a large margin. Hamiltonian dynamics have number of useful properties that can be exploited more widely by the machine learning community. For example, the Hamiltonian induces smooth manifold and a vector field in the abstract phase space along which certain physical quantities are conserved. The time evolution along these paths is also completely reversible. These properties can have wide implications in such areas of machine learning as reinforcement learning, representation learning and generative modelling. We have demonstrated the first step towards applying the learnt Hamiltonian dynamics normalising flows for expressive yet computationally efficient density modelling. We hope that this work serves as the first step towards wider adoption of the rich body of physics literature around the Hamiltonian principles in the machine learning community.
ACKNOWLEDGEMENTS
We thank Alvaro Sanchez for useful feedback.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenhaum. Simulation as an engine of physical scene understanding. PNAS, 110, 2013.
Jacobsen. Residual flows for Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018.
Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and invertible generative modeling. arXiv, 2019.
Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 2013.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiable physics for learning and control. NeurlPS, 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. ICLR, 2017.
K. Friston. The free-energy principle: rough guide the brain? Trends in cognitive sciences, 13, 2009.
Karl Friston. The free-energy principle: unified brain theory? Nature Reviews Neuroscience, 2010.
Herbert Goldstein. Classical Mechanics. Aadison-Wesley, Pub. Co, Reading, Mass., 1980.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. ICLR, 2018.
Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. NeurIPS, 2019. 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. PMIR, 2019.
William R. Hamilton. On general method in dynamics. Philosophical Transactions of the Royal Society, 11, pp. 247-308, 1834.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv, 2018.
G. F. Hinton and R.R. Salakhutdinov. "reducing the dimensionality of data with neural networks.
Science, 313:504-507, 2006.
Emiel Hoogeboom, Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. ICML, 2019.
Chin Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. ICML, 2018.
Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientific tools for Python, 2001. TRRltttp://wwwscipppt
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. ICLR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, Improving variational inference with inverse autoregressive flow. NewrIPS, 2016.
11

---

Published as a conference paper at ICLR 2020
Daniel Levy, Matthew D Hoffman, and Jascha Sohl- Generalizing hamiltonian monte carlo with neural networks. arXiv preprint arXiv:1711.09266. 2017.
Mario Livio. Why symmetry matters. Nature, 490, 2012.
Radford M Neal et al. MCMC using hamiltonian dynamics. Handbook of markov chain monte carlo, 2 (11):2, 2011.
Emmy Noether. The finiteness theorem for invariants finite groups. Mathematische Annalen, 77: 89-92, 1915.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. NeurIPS, 2017.
geco. Workshop Danilo Rezende and Fabio Viola. Generalized elbo with constrained optimization, gecio, on Bayesian Deep Learning. NeurlPS, 2018.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint
2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. ICML., 2014.
Tim Salimans, Diederik Kingma, and Max Welling, Markov chain Monte Carlo and variational inference: Bridging the gap. In /CML, 2015.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. NeurIPS, 2015.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration. arxiy, 2019.
12

---

Published as a conference paper at ICLR 2020
A SUPPLEMENTARY MATERIALS
A.1 HAMILTONIAN GENERATIVE NETWORK
The Hamiltonian Generative Network (HGN) consists of three major parts, an encoder, the Hamiltonian transition network and decoder. During training the encoder starts with a sequence
(21.22..... R x3xxx
of raw training images and encodes into a probabilistic prior representation transformed with an additional network on top
q= q (=p. (0.1))
(8)
into a start state
Su (x4x
consisting of a downsized spatial representation in latent space (4 4), where each abstract pixel is the the concatenation of abstract position (g) and momentum (p) (each of dimension 16). The encoder network is a convolutional neural network with 8 layers, with 32 filters on the first layer, then 64 filters on each subsequent layer, while in the last layer we have 48 filters. The final encoder transformer network is convolutional neural network with 3 layers and 64 filters each layer. Starting from this initial embedded state, the Hamiltonian transition network generates subsequent states using a symplectic integrator approximating the Hamiltonian equations. The Hamiltonian transition network represents the Hamiltonian function
H: E R (x4xx(2 + R
(10)
as function from the abstract position and momentum space to the real numbers at any time step t. The Hamiltonian transition network is a convolutional neural network of 6 layers, each consisting of 64 filters. The discrete timestep we use for the symplectic integrator update step dt= 0.1255
At each time step t the decoder network dg takes only the abstract position part of the state and decodes back to an output image 3; ep 13323323 of the same shape as the input images.
dg: -
(11)
The decoder network is progressive network consisting of 3 residual blocks, where each residual block resizes the current input image by a factor of using the nearest neighbor method (at the end we have to upscale our latent spatial dimension of 4 to the desired output image dimension of 32 in these steps), followed by 2 blocks of one layer convolutional neural network with 64 filters and a leaky ReLU activation function, closing by a sigmoid activation in each block. After the 3 blocks a final one layer convolutional neural network outputs the output image with the right number of channels.
We use Adam optimisier (Kingma & Ba 2014) with learning rate 1.5e-4. When optimising the loss, in practice we do not learn the variance of the decoder (= and fix it to 1, which makes the reconstruction objective equivalent to a scaled L.2 loss. Furthermore, we introduce Lagrange multiplier in front of the KL. term and optimise using the same method in infeezedde & Viola 2018).
A.2 NEURAL HAMILTONIAN FLOW
For all NHF experiments the Hamiltonian was of the form H(g.p) = K())+ V((g. The kinetic energy term K and the potential energy term V are soft-plus MLPs with layer-sizes (d, (d,128. 128, 1] where dl is the dimension of the data. Soft-plus non-linearities were chosen because the optimisation of Hamiltonian flows involves second-ords -oder derivatives of the MLPs used for parametrising the Hamiltonians. This makes ReLU ooneeinasss unsuitable. The encoder network (<lp))=N(pp)) (q)), where μ and a are ReLU MLPs with size 128 128.d]. The Hamiltonian flow dt was approximated using a leapfrog integrator (Neal et al. 2011 since it preserves volume and is invertible for any dt (see also section We found that only two leapfrog steps where sufficient for all our examples. Parameters were optimised using Adam (Kingma & Ba 2014) (learning rate 3e-4) and Lagrange multipliers were optimised using the same method as in Rezende & Viola (2018). All
parametrized as
13

---

Published as a conference paper at ICLR 2020
shown kernel density estimate (KDE) plots used 1000 samples and isotropic Gaussian kernel bandwidth of 0.3. For the RNVP baseline (Dinh et al. (017) we used alternating masks with two or three layers. Each RNVP layer used an affine coupling parametrized by twolayyer relu that matched those used in the leapfrog.
A.3 HAMILTONIAN NEURAL NETWORK
The Hamiltonian Neural Network (HNN) (Greydanus et al., 2019) learns differentiable function H((.p) that maps system's 's state in phase space (its position q and momentum p) to scalar quantity interpreted as the system's 's Hamiltonian. This model is trained so that H(q.P) satisfies the Hamiltonian equation by minimizing
1 11000 ) ) 0H dp + )
0H dq
CNNN 2 0p dt Ōq
(12)
HR and HR where the derivatives 2g op are computed by backpropagation. In the original paper, these targets are either assumed to be known or are estimated by finite differences using the state at times t and ++ 1. Accordingly, in the majority of the experiments presented by the authors, the Hamiltonian was learned from the ground truth state space directly, rather than from pixel observations.
The original HNN model is trained in supervised fashion on the ground truth state of a physical system and its time derivatives. As such, it is not directly comparable to our method, which learns a Hamiltonian directly from pixels. Instead, we compare the PixelHNN variant of the HNN, which is introduced in the same paper, and which is able to learn a Hamiltonian from images and in the absence of true state time derivative in some settings.
This required a modification of the model. First, the input to the model became concatenated, flattened pair of images X, (æ,, (t+1), which was then mapped to a low-dimensional embedding space z, = (qtpPt) using an encoder neural network. Note that the dimensionality of this embedding (2 E R² in the case of the pendulum system presented in the paper) is chosen to perfectly match the ground truth dimensionality of the phase space, which was assumed to be known a priori. This, however, is not always possible, as when a system has not yet been identified. The latent embedding was then treated as an estimate of the position and the momentum of the system depicted in the images, where the momentum was assumed to be equal to the velocity of the system an assumption enforced by the additional constraint found necessary to encourage learning, which encouraged the time derivative of the position latent to equal the momentum latent using finite differences on the split latents:
(cc=(p.-(qt1-q)))
(13)
This loss is motivated by the observation that in simple Newtonian systems with unit mass, the system's state is fully described by the position and its time derivative (the system's velocity). An image embedding that corresponds to the position and velocity of the system will minimize this loss. This assumption is appropriate in the case of the simple pendulum system presented in the paper, however it does not hold more generally.
As mentioned earlier, PixelHNN takes input concatenated, flattened pair of images and maps them to an embedding space 2, = (t). which is treated as an estimate of the position and momentum of the system depicted in the images. Note that X, always consists of two images in order to make the momentum observable. This embedding space is used as the input to an HNN, which is trained to learn the Hamiltonian of the system before, but using the embedding instead of the true system state.
To enable stable learning in this configuration, the PixelHNN uses a standard mean-squared error autoencoding loss:
CAE
X
x1)
(14)
N
1=1
14

---

Published as a conference paper at ICLR 2020
where X, is the autoencoder output and X; is the value of pixel i of N total pixels in X,- This loss encourages the network embedding to reflect the content the input images and to avoid the trivial solution to
The full PixcIHNN loss is:
CPiscHINN + NUNN +ADD
(15)
where ChNN is computed using the finite difference estimate of the time derivative of the embedding. AHNN is Lagrange multiplier, which set to O. 1, as the original paper. LwD a standard L2 weight decay and its Lagrange multiplier ÄwD is set to 1e-5, as in the original paper.
In the experiments presented here, we reimplemented the PixcIHNN architecture as described in Greydanus et al. (2019) and trained it using the full loss (5). As in the original paper. used a PixelHNN with HNN, encoder, and decoder subnetworks each parameterized by a multi-layer perceptron (MLP). The encoder and decoder MLPs use ReLU nonlinearities. Each consists of layers, with 200 units in each hidden layer and an embedding of the same size as the true position and momentum of the system depicted (2 for mass-spring and pendulum, 8 for two-body, and 12 for 3-body). The HNN MLP uses tanh nonlinearities and consists of two hidden layers with 200 units and one-dimensional output.
To ensure the difference in performance between the PixeIHNN and HGN are not due primarily archiectural choices, also compare variant of the PixcIHNN architecture using the same convolutional encoder and decoder as used in HGN. We used identical hyperparameters to those described in section A.I We map between the convolutional latent space used by the encoder and decoder and the vector-valued latent required by the HNN using one additional linear layer for the encoder and decoder.
In the original paper, the PixelHNN model is trained using full-batch gradient descent. To make it more comparable to our approach, we train it here using stochastic gradient descent using minibatches of size 64 and around 15000 training steps. As in the original paper, train the model using the Adam optimizer (Kingma & Ba 2014) and a learning rate of Ie-3. As in the original paper, we produce rollouts of the model using Runge-Kor Kunge-Kutta integrator (RK4). See (ection/A.6foom description of RK4. Note that, as in the original paper, we use the more sophisticated algorithm implemented in scipy (scipy integrate....... solve solve_ivp (Jones al. 2001).
A.4 DATASETS
The datasets for the experiments described in were generated in a similar manner to Greydanus et al. (2019) for comparative purposes. All of the datasets simulate the exact Hamiltonian dynamics the underlying differential equation using the default scipy initial value problem solver solverJones et al. (2001). After creating dataset trajectories for each system, we render those into a sequence of images. The system depicted in each dataset can be visualized by rendering circular objects:
For the mass-spring the mass object is rendered as a circle and the spring and pivot are invisible.
For the pendulum only the weight (the bob) is rendered as a circle, while the massless rod and pivot are invisible.
For the two and three body problem we render each point mass as a circle in a different color.
Additionally, we smooth out the circles such that they do not have hard edges, can be seen in Fig.
A.5 CONVERGENCE OF HGN AND HNN
In order to obtain the results presented in Tbl. [e trained both HGN and HNN for I 15000 iterations, with batch size of 16 for HGN, and 64 for the HNN. Given the dataset sizes, this means that HGN was trained for around 5 epochs and HNN was trained for around 19 epochs, which took around 16 hours. Figs.
plot the convergence rates for HGN (leapfrog) and HNN (conv) on the four datasets.
II experiments, which use RGB images, this expectation is taken over color channels well pixels.
15

---

Published as a conference paper at ICLR 2020
Pendulum
Mass-
0.008
0.007
0.006
0.005
0.005
0.004
0.004
0.003
0.002
0.001
0.001
0.000
20
80
100
120
140
160
120
Two-Body
Three-Body
0.007
0.007
0.006
0.006
0.005
0.005
0.004
0.004
hade Mhb wm mwm
0.003
Mmw mm
0.003
0.002
0.002
0.001
0.001
0.000
0.000
20
80
100
120
140
160
100
120
140
160
Figure 11: Average MSE for pixel reconstructions during training of HGN (leapfrog). The axis indicates the training iteration number 1e+2.
A.6 INTEGRATORS
Throughout this paper, we estimate the future state of systems from inferred values of the system position and momentum by numerically integrated the Hamiltonian. We explore three methods of numerical integration: (i) Fuler integration, (ii) Runge- Kutta integration and (iii) leapfrog integration.
Euler integration estimates the value of a function at time d+ by incrementing the function's 's value with the value accumulated by the function's 's derivative, assuming it stays constant in the interval df). In the Hamiltonian framework, Euler integration takes the form:
OH
Pi+di +
(16)
0p
p=p:
0H
Pitdi = Pt -
dt
(17)
0q
q=
Because Euler integration estimates function's future value by extrapolating along its first derivative, the method ignores the contribution of higher-order derivatives to the function's 's change in time. Accordingly, while Euler integration can reasonably estimate a function's 's value over short periods, its enrors accumulate rapidly as it is integrated over longer periods or when it is applied multiple times. This limitation motivates the use of methods that are stable over more steps and longer integration times.
One such method is four-step Runge-Kutte integration (RK4), the most widely used member of the Lunge-Kuuta family of integrators. Whereas Euler integration estimates the value of function at time t+ dt using only the function's derivative at time t, RK4 accumulates multiple estimates of the function's 's value in the interval [t,t dt). This integral more correctly reflects the behavior of the function in the interval, resulting in more stable estimate of the function's value. RK4 estimates the state at time 1 dl
16

---

Published as a conference paper at ICLR 2020
Pendulum
Mass-spring
0.018
0.020
0.018
0.016
0.044
0.014
0.012
0.012
0.010
0.008
0.008
0.006
0.006
40
60
80
100
120
140
160
20
60
80
100
120
140
160
Two-Body
Three-Body
0.018
0.018
0.016
0.016
0.014
0.014
0.012
0.012
0.010
0.010
W/
0.008
W Mm
wm
Mama Aw
0.006
0.004
Figure 12: Average MSE for pixel reconstructions during training HNN (conv). The x axis indicates the training iteration number 1e+22.
1 +. dlll -k + +2 ks k4):
(18)
where, for compactness, we write the full system state as x, = [q and the Hamiltonian as H(x)). In the Hamiltonian framework, k ka. are obtained by evaluating the derivative at four points in the interval dt]:
dH((r) k
(19)
dt
==
dH(())
12 dt +== 4 k:
(20)
dH((x)
k: dt memmet
(21)
d'H(x))
k1 dt
(22)
2= dt-kks
While RK4 may produce reasonably stable estimates over short periods of time, it is not guaranteed to behave stably indefinitely. Neither RK4 nor Euler integration is guaranteed to preserve the energy of the system being integrated, and in practice both will produce estimates that drift away from the true system dynamics over timescales that are relevant for simulating real systems.
nell- methods for numerical integration that preserve energy and can be applied to Hamiltonian systems, like the one we propose here. One such method leapfrog integration, which is special method for integrating differential equations of the form:
Fortunately, there
17

---

Published as a conference paper at ICLR 2020
Non-symplectic
Symplectic
Padi
B
dt
Figure 13: A: example of using symplectic (leapfrog) and non-symplectic (Euler) integrators on the Hamiltonian of harmonic oscillator. The blue quadrilaterals depict volume in phase space over the course of integration. While the symplectic integrator conserves the volume of this region, but the non-symplectic integrator causes to increase volume with each integration step. The symplectic integrator clearly introduces less divergence in the phase space than the non-sympeectic alternative over the same integration window. B: an illustration of the leapfrog updates in the phase space, where is position and is momentum.
dy = F(),, dt
dx
(23)
dt
we assume that the Hamiltonian equations take this form, we can integrate them using the leapfrog integrator, which in essence updates the position and momentum variables at interleaved time points in a way that resembles the updates "leapfrogging" over each other (see Fig. 138 for an illustration).
In particular, the following updates can be applied to Hamiltonian of the form H. +T()). where V is the potential energy and T is the kinetic energy the system:
dt QV
Pitdi = Pt
(24)
0q
q=q:
qt +
(25)
As discussed above, leapfrog integration is more stable and accurate over long rollouts than integrators like Euler or RK4. This because the leapfrog integrator an example symplectic integrator, which means is guaranteed to preserve the special form of the Hamiltonian even after repeated application. An example visual comparison between a symplectic (leapfrog) and non-symplectic (Euler) integrator applied over the Hamiltonian for harmonic oscilator is shown in Fig. [3A. For more thorough discussion of the properties of leapfrog integration, see (Neal et 2011
A.7 PROOF OF THE HAMILTONIAN FLOW ELBO
The Hamiltonian Flow consists of two components. Firstly, it defines a density model over the joint
space $ = (Qr-P) using the Hamiltonian Flow as described section 3.3 However, we assume that the observable variable represents only qr and treat pr as a latent variable which we have to marginalize over. Since the integral is intractable using the introduced variational distribution we can derive the lower bound the marginal likelihood:
18

---

Published as a conference paper at ICLR 2020
In p(p(d))= Imn p(ar-Pr) dpr
=In /
p(
Lo (p (r)) dpr MArr-PT)d
=In (prlaa
(pr (q))
2 2E wr) In pQr )dpr
(prll9r)
=Booerlar)) (
In h (H1 dt o... o, di (r)) -m In (erlgg))
= =ELBO(qT)
19


Reviews:
[{"review_id": "Sye2f6xAur", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes two ideas: 1) Hamiltonian Generative Networks (HGN) and 2) Neural Hamiltonian Flow (NHF). \n\nHamiltonian Generative Networks are generative models of high-dimensional timeseries which use hamiltonian differential equations to evolve latent variables (~position and ~momentum vectors) through time (using any differentiable integration scheme). Given the ~position vector a decoder network generate predictions. The initial latent variables are inferred using a VAE style inference network that takes a sequence of images as the input. The decoder, inference network and crucially the hamiltonian energy function is learned by minimizing a VAE style ELBO on observed sequences. The model induces a strong hamiltonian physics prior and is quite elegant all in all. The model is evaluated on 4 simulated physics tasks, and beats the only baseline the Hamiltonian Neural Network (HNN).\n\nNeural Hamiltonian Flow notes that hamiltonian dynamics are invertible and volume preserving, which is the properties you need for neural flow models. As such it propose to use a series of hamiltonian update steps with multiple learned energy functions as a flexible density estimator. The resulting density estimator is subjectively evaluated on three 2d toy density estimation tasks.\n\nI propose a weak accept as I think the paper is interesting and well written, but could be much better. The paper explains how both HGN and NHF work, but not much more. The HGN is only compared to a single other method (the closely related HNN), on four toy benchmarks. The NHF is barely evaluated, and not compared to anything.\n\nDoes the authors actually care about modelling physics and think their method is superior at this? If so, they should compare and contrast to some of the many, many papers on modelling physics, e.g. [1,2,3,4] and references herein. If not, what do they care about? Where do they think this model can be useful? Why should anyone use this model over some of the many, many other models one could use?\n\nSimilarly for the NHF, if I only read this paper I have no idea whether it's better than any of the other flow based models. Is it faster (to sample? to eval likelihood?) is it a better estimator? Why should I use it?\n\nI think the paper would benefit from being split into two papers, each thoroughly examining one idea.\n\nA few questions and minor comments\n\n - While the hamiltonian dynamics expect position and momentum vectors, the neural network is free to use those however it sees fit. Actually, if I understand correctly, the position vector must also encode the color of the objects for the 2 and 3 body problem. Is that correct? It would be interesting if you could examine how predictive the q and p vectors were of the true position and momentum vectors. \n - Successful experiments with n-body problems with n randomly sampled during training and unseen n used in testing would be very powerful in showing generalization. I'm afraid that the current setup doesn't generalize well.\n - I'm surprised that the generated images start showing artifacts after some time, e.g. pendulum sample 4 and 6 in https://docs.google.com/presentation/d/e/2PACX-1vRD2FnKgymgR2lU8lE6-XM8Cz-UWLTI6n_Uht3v6Gu4hIyMHmOcNL5D-0eG6Z4WHDAWS4qFosU-lxXP/pub?start=false&loop=false&delayms=3000&slide=id.g61bbdf339d_0_426. How can those appear if the hamiltonian dynamics preserve energy?\n - Equation 3 is given as self evident. It's not clear to me why 1) det(I+dt*A) = 1+dt*Tr(A)+O(dt^2). Can the authors give a reference? Also, doesn't the O(dt^2) term accumulate over multiple timesteps or longer rollouts? if so, how can the multiple steps proposed be said to be volume preserving? \n\n[1] - Battaglia, Peter, et al. \"Interaction networks for learning about objects, relations and physics.\" Advances in neural information processing systems. 2016.\n[2] - de Avila Belbute-Peres, Filipe, et al. \"End-to-end differentiable physics for learning and control.\" Advances in Neural Information Processing Systems. 2018.\n[3] - Santoro, Adam, et al. \"A simple neural network module for relational reasoning.\" Advances in neural information processing systems. 2017.\n[4] - Fraccaro, Marco, et al. \"A disentangled recognition and nonlinear dynamics model for unsupervised learning.\" Advances in Neural Information Processing Systems. 2017.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HJgFZBKTKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors present a method for learning Hamiltonian functions that govern a dynamical directly from observational data.  The basic approach uses three networks: 1) an inference network (I'm not clear why this is not just called an encoder), that maps past observations to a latent p,q space in a VAE-like fashion; 2) a Hamiltonian network that governs the time-evolution of the system over this latent state; and 3) a decoder network that outputs the observation from the latent state.  In addition to introducing this basic formalism, the \n\nComments: I have mixed opinions on this paper, though am leaning slightly toward acceptance.  The overall notion of learning a Hamiltonian network directly is a great one, though really this is due to the Hamiltonian Neural Networks paper of Greydanus et al., 2019.  Although the focus in that work is on applying learned Hamiltonian networks directly to physics-based data, they also have an encoder-decoder network just using a classical autoencoder instead of a VAE.  So my first impression is that the benefits of the proposed HGN over HNNs in Figure 6 is really just an artific of this replacement.\n\nPerhaps because the authors also felt this was a marginal contribution, the paper's ultimate value may prove to be in the consideration of such networks for the purposes of normalizing flow models.  This portion seemed a little bit underdeveloped in the paper, to be honest, but overall the idea of parameterizing a normalizing flow with a Hamiltonian dynamical system seems like a good one (e.g., allowing for easier large-timestep inference).  But on the flipside, it does seem like the presentation here is rather brief, i.e., just defining the ELBO without much context or detail, etc.\n\nThus, while I'm very much on the fence on this paper, I think the marginal improvement over HNNs via a better encoder/decoder model, plus the realization that these methods are a good fit for normalizing flow models, altogether put the paper slightly above bar for me.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgU-6KRKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a novel way of learning Hamiltonian dynamics with a generative network. The Hamiltonian generative network (HGN) learns the dynamics directly from data by embedding observations in a latent space, which is then transformed into a phase space describing the system's initial (abstract) position and momentum. Using a second network, the Hamiltonian network, the position and momentum are reduced to a scalar, interpreted as the Hamiltonian of the system, which can then be used to do rollouts in the phase space using techniques known from, e.g., Hamiltonian Monte Carlo sampling. Finally, a decoder network can use the system's phase space location at any rollout step to generate images of the system. The HGN can further be modified, leading to a flow-based model, the Neural Hamiltonian Flow (NHF).\nThe authors evaluate the HGN on four simulated physical systems, showing substantial improvements over the competing Hamiltonian Neural Network (HNN). Lastly, the NHF is shown to be able to model complex densities, which can further be interpreted in terms of the kinetic and potential energy.\n\nThis paper is a rare treasure. It tackles a well-motivated problem and introduces a, to my knowledge, completely new framework for embedding Hamiltonian dynamics in a generative model. This is hugely inspiring! The paper is a joy to read and includes very informative figures providing a high-level understanding of the proposed models. Accept is a no-brainer.\n\nThat being said, I have a few questions and suggestions for improvements. My biggest complaint is the evaluation of the NHF model. I would have liked to see a comparison to a state-of-the-art flow-based model in terms of density modelling. The authors state that the NHF offers more expressiveness and computational benefits over standard flow-based models, but this is never shown. While I am willing to believe the claim, it is not intuitive to me, and I would have liked to see experimental verification of it.\n\nFigure 6 needs a bit of love. It is quite challenging to read. Larger font sizes, conversion to vector format, and more distinguishable colours will help a lot.\nAdditionally, I think it would be helpful to have the derivation of the ELBO in Eq. (4) written out, e.g. in the supplementary material.\n\nAdditional questions:\n- In the experimental section, I am not sure what is meant by the deterministic version of HGN. Which part if the model is deterministic?\n- On p 6, it is mentioned that the Euler integrator results in an increased variance of the learnt Hamiltonian and that this can be seen in Fig. 6. How exactly is this seen in the figure?\n- How many epochs were HNN and HGN trained for to produce table 1? How do the convergence rates look, and how long time did they take to train?\n\nMinor comments:\n- p 5: Reference to \"Salimans et al.\" is missing the year.\n- p 6: There is a hanging ')' after \"as shown in Fig. 6).\"\n- p 6: \"reversed it time\" -> \"reversed in time\"\n- In the reference for Glow, \"Durk P Kingma\" should be \"Diederik P. Kingma\".\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel way of learning Hamiltonian dynamics with a generative network. The Hamiltonian generative network (HGN) learns the dynamics directly from data by embedding observations in a latent space, which is then transformed into a phase space describing the system's initial (abstract) position and momentum. Using a second network, the Hamiltonian network, the position and momentum are reduced to a scalar, interpreted as the Hamiltonian of the system, which can then be used to do rollouts in the phase space using techniques known from, e.g., Hamiltonian Monte Carlo sampling. An important ingredient of the paper is the fact that no access to the derivatives of the Hamiltonian is needed. 

The reviewers agree that this paper is a good contribution, and I recommend acceptance.
Decision: Accept (Spotlight)