Hamiltonian Latent Operators for content and motion disentanglement in image sequences
Authors: Asif Khan
Amos Storkey
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep generative models
Variational Autoencoder
Symplectic Geometry
Hamiltonian Dynamics
Latent Space Disentanglement

Abstract: We introduce \textit{HALO} -- a deep generative model utilising HAmiltonian Latent Operators to reliably disentangle content and motion information in image sequences. The \textit{content} represents summary statistics of a sequence, and \textit{motion} is a dynamic process that determines how information is expressed in any part of the sequence. By modelling the dynamics as a Hamiltonian motion, important desiderata are ensured: (1) the motion is reversible, (2) the symplectic, volume-preserving structure in phase space means paths are continuous and are not divergent in the latent space. Consequently, the nearness of sequence frames is realised by the nearness of their coordinates in the phase space, which proves valuable for disentanglement and long-term sequence generation. The sequence space is generally comprised of different types of dynamical motions. To ensure long-term separability and allow controlled generation, we associate every motion with a unique Hamiltonian that acts in its respective subspace. We demonstrate the utility of \textit{HALO} by swapping the motion of a pair of sequences, controlled generation, and image rotations.

Hamiltonian latent operators for content and motion disentanglement in image sequences
Asif Khan School of Informatics University of Edinburgh asif. khan@ed ac. uk
Amos Storkey School of Informatics University of Edinburgh storkey@ed ac uk
Abstract
We introduce HAL.O a deep generative model utilising HAmiltonian Latent Oper- ators reliably disentangle content and motion information in image sequences. The content represents summary statistics of sequence, and motion is a dynamic process that determines how information expressed in any part of the sequence. By modelling the dynamics as Hamiltonian motion, important desiderata are ensured: (1) the motion is reversible, (2) the symplectic, volume-pree preseerie struc- ture in phase space means paths are continuous and are not divergent in the latent space. Consequently, the nearness of sequence frames is realised by the nearness of their coordinates in the phase space, which proves valuable for disentanglement and long-term sequence generation. The sequence space generally comprised different types of dynamical motions. To ensure long-term separability and allow controlled generation, we associate every motion with a unique Hamiltonian that acts in its respective subspace. We demonstrate the utility of HALO by swapping the motion of a pair sequences, controlled generation, and image rotations.
1 Introduction
The ability to learn to generate artificial image sequences has diverse uses, from animation, keyframe generation, and summarisation restoration that has been explored in previous work over many decades (Hogg) 1983 Hurri and Hyvärinen 2003] Cremers and Yuille 2003), Storkey and Williams 2003 Kannan al. 2005). However, learning generate arbitrary sequences not enough; to hold a practical application, the user must be able to control aspects of the sequence generation, such as the motion being enacted or the characteristies of the agent doing an action. To enable this, must learn to decompose image sequences into content and motion characteristics so that we can apply learnt motions to new objects vary the motions being applied.
Deep generative models (DGMs) such as variational autoencoders (VAEs) Kingma and Welling 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) use neural networks
(NNs) to transform the samples from a prior distribution over lower-dimensional latent factors to samples from the data distribution itself. Recent developments (Chung al. 2015) Srivastava et al.
2015 Hsu et al. 2017]] Yingzhen and Mandt (0018) extend VAEs to sequences using Recurrent Neural Networks (RNNs) on the representation of temporal frames. Similar approaches have been taken for
GAN models (Tulyakov et al.) 2018: Yoon et al. 2019 Dandi et al., 2020).
The dynamical processes creating the evolution of image sequences are highly constrained. Consider the simplistic case of a person walking in a scene with camera moving around that individual. The walking pose will return to similar positions periodically, and likewise, the revolving camera will revisit previous positions. Even without strict periodicity, many dynamical processes are reversible. Any time that a dynamic could conceivably return to an earlier state suggests implicit conservation law the conservation of information in the underlying scene generator-ae as it must be capable of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

14
Figure I: Left: we empirically demonstrate the benefits of learning Hamiltonian operator in the latent space. Consider (32, 32) image sequence of ball rotating in fixed orbit; the (i. j) centre of a ball moves under constraint 2 + j = c, where c is a constant. We generated a data set of sequences with different initial condition and same number of time steps. Next, trained VAE with Hamiltonian operator in the latent space. We use an encoder to transform sequences to a 2-dimensional phase space, next unroll trajectory using learnable Hamiltonian operator and finally use decoder to obtain the sequence. Top row is the original sequence, second row is the reconstructed sequence and below are three sequences generated from random initial states. Middle: the coordinates in the phase space are coloured by the energy value, alongwith a representation of a sequence generated from a random initial coordinate. Right: is the probabilistic graph of our generative and inference model.
returning to and regenerating the same scene with non-negligibbe probability. The critical observation we wish to capture in this paper is that understanding the conservation occurring in the context of a set of sequences is a vital ingredient to decomposing content from motion.
Given any conserved quantity, any motion must be modelled maintain the conserved quantity. In physics, such a motion is called a Hamiltonian motion; it keeps the corresponding Hamiltonian function constant. Hence we argue that flexible latent Hamiltonian model provides good inductive bias to learn a representation space that enables conservation of the right quantities (which are themselves learnt) and models the dynamic evolution. This mathematically equivalent to saying that we can learn to represent the underlying motions as combinations of differentiable symmetry groups; all differentiable symmetry transformations follow conservation law (Noether 1918).
We next illustrate the benefit of using Hamiltonian dynamics in the latent space of DGM in Figure Here, we observe using the Hamiltonian dynamics model can discover constant energy in latent space from a set of image sequences proving critical for generating novel energy preserving sequences. This example demonstrates identifying symmetries is suitable inductive bias for developing expressive DGMs that understand the motion constraints and generalise beyond the training data. Higgins et al. (0118) : Toth al. (2019) Botev al. (202T) have discussed the benefits of such inductive biases for learning disentangles representation.
The existing sequential DGMs do not impose any structural prior for constraining the dynamics in motion space and, therefore, accumulate errors as the sequence length grows, quickly deviating from the relevant path (Karl al. 2016, FFaccaro et al., 2017. Yildiz et al. 2019: Bird and Williams 2019). The attractive property of Hamiltonian dynamics is that they are symplectic that is, the divergence of a vector field is zero, and the evolving dynamics preserve the infinitesimal volume element. Consequently, the motion paths are restricted to oowddimeniional manifold in the latent space, and we can predict the dynamics forward and backwards in time.
In this paper, we intimate the more general applicability of latent Hamiltonian models; previous applications have been limited to somewhat constrained physical systems. We propose HALO - VAE framework to model the dynamics of image sequences using a collection of learnable linear Hamiltonian operators in the latent space. Specifically, for any motion sequence. the transition from a time step f to a step 1. + 1 using a group action of a Hamiltonian operator. The evolution of the dynamics of sequence leaves certain information unchanged, identified as content, and specific properties that evolve in conjunction (i.e. motion). Since the space of image sequences can comprise various types of dynamical actions, we split the motion space into subspaces where each subspace models a unique action and is unaffected by other actions. This formulation explicitly ensures the separability of dynamics. It further reduces the computational cost since the Hamiltonian of the space is now in a block diagonal form where each block is Hamiltonian of a symmetry subgroup. Here, we focus on a discrete, identified set of actions that we can then compose at generation time. We want to remark our method can also work without action labels,
model
We discuss the specifics of dynamical operators in Section

---

as empirically demonstrated in the results. The benefit of identifying actions apriori is that we can use it for a controlled generation. We empirically demonstrate the advantages of our approach through i) generation of diverse dynamics from a starting frame and ii) demonstrating successful disentanglement of the content and motion representation.
2 Related Work
Hamiltonian Neural Networks Several deep learning (DL) methods have recently been proposed to learn the dynamics of physical systems using Hamiltonian mechanics. Greydanus et al. (2019) use NNs to predict Hamiltonian from shase-space coordinates s = (p. q) and their derivatives, In similar work (Bondesan and Lamacraft (219)) used NNs to discover symmetries of Hamiltonian mechanical systems. More recently, Hamiltonian NNs have been used for simulating complex physical systems Sanchez-Gonzalee the states of particles as a graph and use a graph neural network (GNN) to predict the change from the current state to the next state. In a follow-up Crammer et al. (2200). introduce sparsity on the messages in a graph and use the symbolic regression method to search for physical laws that describe the messages in the graph. Recently Toth et al. (2019) 9) developed the Hamiltonian generative network (HGN), where they proposed to learn a Hamiltonin a latent representation and then projects it to the phase space to unroll the dynamics using a symplectic ODE integrator with Hamilton's 's equation. In another work Yildiz et al. (2019) use second-order ODE parameterised as a BNN for modelling dynamics of high dimensional sequence data in the latent space of VAE. Most of the developments are built the neural ODE (Chen et al. 2018), idea to view layers of NNs as internal states of an ODE. These methods rely on the numerical integration scheme and the stability of the ODE solver. A Hamiltonian formalism dictates additional requirement that the dynamics of an ODE should be volume preuse-in and reversible. We want to clarify that, unlike HGN, which mainly focuses on sequence generation and relies on symplectic ODE integrators in the latent space, we use linear Hamiltonian operators with matrix exponentials and demonstrate its relevance for disentanglemennt
et al. 2019 2020). The key idea of this work is to represent
from image sequences. HGN maps sequence to
Latent Space Models There is a long history of latent state space models for modelling se- quences (Kalman 1960; Sttarner and Pentland 1997| Roweis and Ghahramani 1999; Elliott and Krishnamurthy) 1999; Pavlovic et all 2000). More recently, these methods have been combined with deep generative models for generating high dimensional sequences well as learning a disentangled
representation (Karl et al. 2016, Villegas et al. 2017] Tulyakov et al. 2018; Hsieh et al., 2018
Yingzhen and Mandi 2018 MMladdnovii et al. [ et al. [019 Franceschi al. 2020
Zhu et al. 2020). MoCoGAN (Tulyakov et al. 2018) developed an adversarial framework, combining random content noise with sequence of random motion noise to generate videos. More recently, DSVAE (Yingzhen and Mandi 2018) proposed to split a latent space into time-variant and invariant representations and use LSTM (Hochreiter and Schmidhuber 1997) to learn the prior on time- variant representation. S3VAE (Zhu et (022) improves the disentanglement of DSVAE by minimising a mutual information loss between content and motion variables. Some Hamiltonian methods (Toth) ct al. 2019; Yildiz et al. 2019) also model the dynamics of high dimensional sequential data in a latent space. However, the focus in those cases is only on sequence generation; to our knowledge, this has not been investigated for disentanglement.
Group Transformations in Latent Space Models Rao and Ruderman 1999) proposed the algorithm
to model the infinitesimal movement on data manifold using learnable Lie group operators Culpepper and Olshausen (2099) use the matrix exponents learn the transport operators for modelling the manifold trajectory. Many other similar methods have investigated the use of geometric operators for learning the manifold representation from data (Rao and Ruderman 1999 Culpepper and Olshausen 2009; Mcmisevic 2012; Sohl- Dicksteii et al. 2010: Cohen and Welling 2014). The use of symme- tries for learning disentangled factors of variations has recently been considered. A disentangiement is generally identified as learning representations with independent latent factors. The main goal is that each latent factor should control distinct data factor, and single latent variable should control no two data factors, Various authors (Bengio et al. 2013 Lake et al. 2017]] Eastwood and Williams definition of disentanglement. The goal in these settings was to decompose a latent space into subspaces and on each subspace, to learn a unique group transformation such that the subspace is unchanged by the action of other groups Casclles-Dupte et al. (2019), build such model using interaction with the environment. Some other
2018). Higgins et al. (2018) have proposed a symmetry-
3

---

MOTION SPACE
H
H
Hx
Pins
Plel
P1-1
CONTENT
POSITION
MOMENTUM
Rewact
ENCODER
Figure 2: The framework for our model. We first encode each time step of a sequence to a respective feature vector h1Tr Next, to unroll the dynamies of an action k we map the feature representation to the respective phase space. Specifically, we sample starting index t and map h, to position coordinate q For momentum p we use temporal convolution with kernel size w on h:-w:t- We then use the operator H₂ to trace out the forward and backward trajectory. At last, we combine the position coordinates of all timesteps 91.7 with the content representation z and pass through decoder network to generate the sequence.
similar approaches were recently proposed to learn group transformations a latent space (Connor and Rozell, [2200 Quessard et al.) 2020; Dupont et al. 2020). However, the applications were restricted to relatively toy problems and, to our knowledge, have not been investigated on higher dimensional videos.
3 Method
Here, we introduce HALO for sets of sequential image data. Each set of sequences depicts the temporal evolution associated with one several actions. this context, an action simply label associated with particular sequence set, but where it is understood, the sequences within a set may have very different content but the same dynamic form, e-g. in the sprites data (discussed later) the actions are 'walking', 'spell cast', and 'slash' and the sequences within set are different individuals performing the relevant action. In the following, we assume the separation into action sets known, but that assumption is relaxed later.
Let xi.r denote the ith image sequence, with x the (th frame in the sequence. Let u² be indicator vector denoting the action associated with the ith sequence; i.c. ui iff sequence i follows action A and w 0 for all other k' * k. These sequences and corresponding actions are collected into dataset [(x u²) ) of size N, where, for the sake of simplicity in description, we assume they all are of same length T. this paper, we use a latent space to aid the modelling of each sequence and decompose that latent space into two parts, which we call content space (denoted by Z) and a motion space (denoted by S). As the data comprises sequences of various actions that take different dynamical forms, we further decompose the latent motion space S S S + ..... @ S with one subspace for each action. In modelling sequence corresponding action k; only the subspace S will be allowed to change across the length of that sequence. Each motion subspace is further decomposed into generalised position and momentum parts: S* = (p* a²t.. Only the position component this phase space is used together with content to create individual images a sequence generatively. The momentum part only affects the dynamics.
The above formulation provides many advantages; prevents the neural network from leaking constant content information via the motion representation, and it ensures the possibility of preserving key conservation quantities that we argue are implicit the constraints of most motion dynamics. This is discussed further in Paragraph The full framework of our model is illustrated in Figure

---

We next introduce the generative model, followed by the variational formalism for inference and learning.
Generative Model For completeness we first present the full probabilistic model in before describing each component. Each dynamic is categorised by a particular action enumerated by k encoded in indicator vector (i.e. u4 for action k). The generative model is conditioned on this action vector. First, in ( we sample the content variable z from a prior p(z). The content variable will describe the constant appearance characteristies expressed throughout the sequence. Next, we sample a starting position from a prior p(q"). ). and momentum from a prior p(p)) ) (we initialise the actions not represented in the sequence zero). The full state-space representation for the dynamic of action is then given by s = (p q² The dynamical model (355) then traces out the forward trajectory in the phase space. Finally, combine the position trajectory with the content representation and use decoder neural network to get the emission distribution of the data space sequence, In summary,
GIVEN: & denoting action label for sequence.
z p(z), qf *N la), p N((, la), $ ip dil s = 0, vk' + k,
(2)
5 1 1), st === = M>1,K #
[4) M. ~ N(x (z).q,)) 2m), 11
where d is the dimensionality of hthh subspace, m is the dimensionality of data space, I is a dynamical model (5) and We are the parameters of f to be used for the t subspace. We use an emission distribution that spherical Gaussian, with a parameterised mean ((.,)) and a covariance a amIm-
Dynamical Model In image sequences, we can view each frame of a sequence as a point in an abstract representation space; the temporal dynamics trace a path connecting the frames forming 1 I-submanifold of the image manifold. Most dynamical models either try to capture this geometry
deterministically (Srivastava al. 2013) probabilistically (Chung et al., 2015 Hsu al. 2017] Yingzhen and Mandt (018)) via linear or non-linear state-space models. In either case, small errors in dynamical steps can accumulate and result in significant deviation from the manifold when unrolling
long-term trajectories at inference time (Karl et al., 2016; Fraccaro et al., (017). Interestingly, Hamiltonian systems alleviate these issues by constraining the dynamics to be symplectic and reversible. The symplectic geometry ensures the dynamics are volume- preseervigge.. preventing any deviation from the manifold, and reversibility is useful in understanding how the state of an object changes under dynamical evolution. By reversing the arrow of time, the object could return to its previous state, this awareness provides sense of accountability to an object for its actions. In our work, without significant loss of generality, we propose linear Hamiltonian system in the latent layer, relying on deep neural network mapping data space handle all nonlinear aspects. The linearity of dynamics also enhances the interpretability of the dynamics.
Definition matrix E 2d22d is an Hamiltonian matrix if H2 JH J. where J is skew-
symmetric matrix J = 0 la and la is an identity matrix.
0
Consider a coordinate vector s E R2dd in the phase space S at time / that evolves under constant Hamiltonian energy E = 5/2" M(t)s, where M(t) is a symmetric matrix. In Hamiltonian mechanics, the coordinates are specified in terms of position and momentum p variables
(q. p). Using ds(t) (t)ss Let
the fact energy E constant over time we can express equation of motion as, di
H(t) JM(t), we rewrite the equation of motion as, dt
ds(t) H(t)s. The closed-fome solution is
given by matrix exponential s(t) = s0)) The matrix exponent has a connection to Lie algebras, and for small i. can interpret eH
infinitesimal transformation of s(0) under the action of Lie group of H. We discuss this further in Appendix 2 For a detailed introduction to the topic, we refer to Chevalley (2016). We use fast Taylor approximation (Bader et al. 2019) to compute matrix exponential that provides stable solution under matrix norms.
In this work, we consider K Hamiltonians H,...., HR. each acting on unique subspace of the phase space S S² s?....,,** . To unroll the trajectory of motion k, we use the group action defined by
5

---

the matrix exponent of the operator H3 on starting phase space representation s E S* given by,
(5) The backward dynamics can simply be obtained by negating time, i.e. replacing t with -t in the above. We assume all time steps are equally spaced. The above formulation provides an explicit disentanglement of the motion space. It further allows us to parallelise the computation of matrix exponential by leveraging the block diagonal form of H. Specific to our work, we parameterise a symmetric matrix M2 and obtain its Hamiltonian matrix as H₂ = JM, where is a fixed skew-symmetric matrix as stated in definition (1). The group of such real Hamiltonian matrices form a symplectic Lie group under multiplication Sp(2d) with 2d + d independent elements. The symplectic geometry proves useful for long-term sequence generation. We also consider the symplectic orthogonal group SpO(2d) that further restricts the Hamiltonian matrix to a skew- symmetric form with (d² /2/2 independent elements. The benefit of this restriction is that the resulting transforms reduce to rotations which is more interpretable. We briefly introduce the details in Appendix 2 For more comprehensive overview, we refer readers to Easton (1993).
$ = [ 1kkk 1) = tHe MMM -1 VI> 1; s =0, Vl,K t..
Inference In order to learn the model parameters, we need to infer the distribution over latent variables. We use variational inference to learn the model parameters that leads to maximising the where q(-1.) is the
evidence lower bound (ELBO) objective, max, E (q((,.((xx.rr.)) log p(Kur.z.s
9(2.5 (
approximate posterior distribution and s; = [q., Pe]. It remains to define the approximate posterior we use. Since the Hamiltonian dynamics are reversible, at inference time, we randomly sample
choice of frame and use forward and backward action of Hamiltonian to trace the trajectory of states after and before that frame for the respective action stated in Equation [
For a sequence XI,T, we use the process (6) to draw samples from a variational distribution 4(z, s u). Simply, we sample the content variable z conditioned on the observed data and independently sample the motion states s = [q pf] for the reference frame conditioned on the observed data and the relevant action k. Motion states corresponding to other actions are set zero. In equations, this is.
(6) where t is a starting index, q(q x,, u). is the posterior distributions of Atth position subspace con- ditioned on the frame x and action variable u, q(p" x u) is the posterior distributions of k momentum subspace conditioned on previous frames and action variable u and g(z)x1:)) is the posterior distribution of the content space conditioned on the entire sequence. We parameterise the factorised posterior as spherical Gaussian distribution learned using encoder neural network. Specifically, q(z (1:T) as content network, q(q² | xx, u) as a position network, and g(p x u) as momentum network. We use reparametrisation trick (Kingma and Welling 2013) to sample from latent distribution Z = f + o @ e where e ~ (0, I).
q(z)xiT), U(II, TT),, at q(q xx, u), P. <((p (x- u). = =Iq p
Learning Objective The learning problem reduces to the optimisation of the following objective,
max
max-xIlg(a |x. u)]l)((?() - KL/gpp/ Ix - [l(((p]]] (1 + be
Σ logg(xx.quu (q...
We have provided the derivation of ELBO in Appendix I Figure right side, is the probabilistic graph of the generative and inference model.
4 Experiments
The implementation details of neural network architecture and training procedure are discussed in Appendix 3] Our code available GitHub. P We first demonstrate the application of HALO on disentangling content and motion in sequences of rotating balls evolving under constant energy. Next, we investigate the applicability of our approach on complex datasets: Sprites and MUG Aifanti
et al. (210)).
Rotating Balls We construct set of sequences of images of ball that moves in orbit under constraint + j2 = e, where (i, is a centre of a ball and c is the distance from the centre of an

---

orbit. Each sequence is drawn from a different initial condition decided uniformly at random, and all sequences are of length 16. To introduce the content element, we colour half of the sequence "red" and the remaining as "blue". In Figure [] we show the result of swapping the content variable of two held-out sequences that demonstrates the effectiveness of HALO in disentangling motion from the content while keeping the dynamics intact.
Next, we introduce details of the two datasets, followed by a discussion of the results in Sec- tion 4.1 Sprites is a sequence of animated char- acters performing different actions ("walking', 'spell cast' and 'slashing' from three viewing angles 'left', 'right' and 'straight') as per sprites SheetsP/The sequences are of length RGB im- ages of size 64 x 64 3. Each character's is appear- ance has four attributes: skin colour, hairstyle, tops and pants. Each attribute can take six values resulting in 1296 unique characters. We used 1000 characters for training and the rest for eval- uation. MUG (Aifanti et al. 2010) a dataset of six facial expressions (anger, disgust, fear, hap- piness, sadness and surprise) of 52 individuals. Sequences are of variable lengths ranging from 50 to 160 frames. We downsample the sequences by a factor of two and then take a random subse- quence of length 8, crop the face region and resize it to 64 x 64. The training and evaluation splits are based on (Tllykovv et al.) 2018). We also demonstrate the application of our model in predicting rotations on MNIST digits where the symplectic structure proves useful for sampling long trajectories in the latent space. We refer to Appendix 3,5 for results.
Figure 3: First two rows are original sequences, the next two rows are respective reconstructions, and the last two rows are generated by swapping the content variables. Swapping the content changes the colour but the dynamics are intact.
4.1 Results and Discussion
We first compare two choices of Hamiltonian structure, symplectic group using H and the symplectic orthogonal group by restricting H to a akew-syymettric form that we refer to as skew-H. Here, we map a starting frame to the latent space, unroll the trajectory and then map the timesteps to data space using a decoder: We generate sequences of length 16 (twice the length used for training purposes). The sprites consist periodic sequences of length 8 where the start and end frames are identical; in
Model Dataset
SSIM
PSNR1
MSE
this case, we duplicate the sequence to get a ground
Sprises 0.962 1. 0.005
36 76.76.. 1.096
0 0005 1 0 0002
truth of length 16. For MUG, we draw a sequence of length 16 from the evaluation set. We compare the generated sequences with target sequences using per-frame structural similarity index measure (SSIM), peak signal-to to-ial--tnoisee ratio (PSNR) and mean squared error (MSE). The SSIM scores are between -1 and 1, with more significant score indicating more sim- ilarity between the ground truth and generated sequence. Likewise, higher PSNR and lower MSE describes the performance under different scores, demonstrating our model can generate high-quality sequences from an input image. We observe that H performs better than H. We hypothesise that the superior performance of H can be attributed to the fact that skew-H introduces additional restrictions on the parameters of H that might be reducing the expressiveness of the model. This result is an interesting finding-the question we wish to investigate in future work. For the rest of the paper, we consider the dynamical operator H.
MUG 0.707 0.003
24.49
0.099
0.0040
0.0001
Sprites 0.950 0.021
33.88 - 2.03
0,0125 + 0.002
Skew H
MUG 0.791 0.003
24.25 = 0.094 0.0044 =: 0.0001
Table 1: We can see both choices of an opera- tor can generate sequences close to the ground truth.
imply better generation. Table [
To evaluate disentanglemenn we compare with the state-of-ther tfrt baselines DSVAE YYngzhhnn and Mandi 2018), MoCoGAN(Tulyakoo et al. 2018) and S3VAE (Zhu et al., 2020).
Quantitative Evaluation We use pretrained action prediction classifier for evaluating disentan- glement. The architecture provided Table of Appendix. To begin with, we draw a starting position and momentum from a prior distribution and use dynamical model to unroll the trajectory in the phase space. Next, we sample the content variable z from real sequences and combine
Paups//githuub.oo///oowwwaaa Saiivessa-LPC-spritese
7

---

Method
Data
Accuracyt
Hly/xi|
Hly)
1S
HALO (cundiional)
0.929
0.108
1 1.778
5.312
Sprites (Attr.)
Accuracy!
RALO (mncondiiona)
0.750
0.187
1.762
4.890
DSVAE
Yingzzeen and Mundi
2018
MUG
0.543
0.374
1.057
3.007
Skin Color
0.925
McCoGAN
Tulyakuv al
0018
0:631
0.183
1.721
4.655
Shirt
0.948
SEVAE
Zha a K0AD
0.706
0.135
1,780
5.178
Pant
0.968
Hair
0.992
MALO (canditiona))
1.000
7.574
DSVAE
Yengzhen and Mandi 2018
Sprites
0.907
0.072
2.192
8.331
Identity (MUG)
0.998
McCodRNN Tolvakav di 2018
0.928
0.090
2,192
8.182
SIVAE Zhuer
0.994
0.041
2.197
636
Table 2: Left shows the disentanglement of content and motion components of latent space. The high
score of accuracy and Inter-Entropy Hyy) while low scores of Intra-Entropy H H(ylx) are expected from better model. Our model performs best across all three scores on MUG. On sprites, we comparable to S3VAE. This is due to the simplicity of classes in sprites. We want to remark our unconditional model, as demonstrated on MUG, significantly outperforms other baselines showing the benefit of Hamiltonian even when labels are not available. On the right, we investigate the extent which content is preserved when we switch motion variables with an arbitrary sequence. We report the accuracy of individual attributes in sprites and the identity of actors in the MUG dataset. The results show the content space can capture attributes that don't 't change under dynamics.
with position variables to generate image sequences. We report the performance of the classifier in predicting the action from these generated sequences. The score is a useful measure of the model's tendency to keep the motion intact with the modified content. We use the same classifier to report the intra-Entropp H (y) x) and inter-Entropy H (y) that estimates the diversity of generated sequences. H H(y)x |x) measures the closeness of generated sequences to the real sequences, and (y) measures the diversity of generated sequences (He et al., 2018). The two scores can be combined together to obtain inception score (IS) Salimans et al. (2016) a commonly used evaluation criterion for the diversity of generated samples(IS = exp(H(y)
Hyy x)) Barratt and Sharma (2018). The results are reported Table 2 We observe HALO outperforms the baselines on the MUG and is comparable with S3VAE on sprites. This improvement results from explicitly associating an action with unique subspace that allows separability of the dynamics and avoids any mixing or ambiguity of action in the motion space. The results on sprites are comparable; we attribute this to the simplicity of sprites' classes that result in high performance across all models. We want to remark that our formulation is not constrained by action variables u. Table 2 also describes the results for unconditional version on the MUG (Aifanti et al. 2010) that significantly outperforms the baseline. The details of the unconditional model are provided in Appendix 3.2 The benefit of incorporating action variables is that it allows controlled generation of sequences, as demonstrated in Figure 6
We, next evaluate the tendency to preserve the identity of sequences. For sprites, identity refers to four different attributes, and for MUG (Aifanti et al.] 2010) it is the sequence label of the individual. We pre-train classifier on the task of identity prediction and use it for evaluating the generated sequences. This way, we measure the model's ability to keep the identity intact when the motion is changed. For sprites, we report the accuracy of individual attributes. Table outtlnes the results. We can see on MUG that our model can preserve the identity with high accuracy. We can make a similar observation for different attributes of sprites sequences. Thus, good performance indicates that the content is preserved when traversing the motion subspace, and the motion space is invariant when changing the content variables. Also validated by the qualitative results.
Qualitative Evaluation We first evaluate the quality of sequence prediction by comparing the original sequence, its reconstruction and the generation from initial frame. To predict the future timesteps of a sequence, we apply the motion operator on the latent encoding of the first time step. Figure 4 on the left are the results for sprites, and on the right of MUG video sequences. Next, we report sequences reconstructed by swapping motion variables to evaluate disentanglement, We start by encoding two sequences x and x to their latent representations (z', (qir)) and (z², qi.r), next we swap the motion variables (z², q..)) and (z², . between the two representation spaces, and then pass the resulting representations through the decoder generate the sequences x x and x Figure 5 on the left, are the pair of consecutive rows of original sequences and on the right of the sequences generated by swapping the motion representations. We can see that swapping the motion part does not affect the identity of the sequences.
We now evaluate the image-to-sequence task to investigate the suitability of our model for controlled generation. We first encode the image to its content and phase space coordinate in different motion spaces. Next, use the respective operators to unroll the trajectories in phase space, which are combined
8

---

Figure 4: In each patch, the first row is the original sequence, the second row is reconstruction, and the third row is a sequence generated by an action of the operator on the starting time step. The reconstruction demonstrates demonstrates the dynamical operator can capture realistic motions from an arbitrary starting frame. More examples are in Appendix 3.2
model can learn good representations, and the generation
S
S
%
O
Figure 5: Qualitative demonstration of content and motion disentanglement. On the left side, we show rows of original sequence pairs and on the right are the reconstructions after swapping the motion variables in the latent space. More examples are in Appendix 3.2
with content and mapped to the image space using a decoder network. 6 examples of decoding different motions from the same input image. For sprites, the actions are in order "walk', 'spell card', 'slash' and for MUG they are ordered 'anger', 'disgust', 'fear', 'happiness', 'sadness' and 'surprise". We observe that the visual dynamics associated with all the operators are well separated. We also evaluate our model for long-term sequence generation; results are presented in Appendix (.22 where it is apparent that longer-term sequences maintain the consistency associated with the content-motion pair.
Ablation We earry out ablation to identify the benefits of using HALO over other dynamical methods. Figure 6 we compare the motion transfer by replacing Hamiltonian with a linear dynamical model and RNN. We observe that, unlike HALO, the variations in dynamics get constant for Linear and RNN models. This result shows the benefit of Hamiltonian dynamics as, by definition, they ensure the phase space coordinates change over time, preventing the encoder neural network from channelling any static information in the motion space, which is vital for explicit disentanglement of content and motion vari- ables. More results are provided in Appendix 3.4 We outline the key findings in Table 3 Figure 4 of the appendix, we show the caveat with other dynamical models the controlled generation task.

---

Figure 6: Left: examples of sequences generated by an action of Hamiltonian operator on the phase space representation of the starting frame. We observe that all dynamics well separated, demonstrating the disentanglement different actions. This result shows the benefit of the symplectic structure in motion space. Right: we demonstrate that our method outperforms other dynamical model baselines used for disentanglement. In each patch, rows one and two are original and rows three and four are obtained by swapping the motion components of latent space. The top is Linear Model, the centre is RNN, and the last is HALO.
Our formulation proves useful for learning disentangled repre- sentations outperforming various baselines. We demonstrated its use for the controlled generation of image sequences. The effectiveness of our approach is a direct consequence of the symplectic geometry the phase space, which prevents trajec- tories from deviating from the motion manifold. The choice of the quadratic form of energy provides a relationship among latent components, which we speculate is critical for the inter-
Dynumies
Metion Swap
Sinezom
Hale
Synglooi
Linzar
RNN
Phetiomal Eecoday
Table 3: The benefits of our formu- lation over other dynamical models. The details on positional encoding are provided in Appendix 3.3
pretability of actions.
Conclusions and Future Work
We introduced HALO a DGM to disentangle motion from the content in image sequences. Our formulation utilises Hamiltonian latent operators to associate conserved quantities with the dynamics. Moreover, by the definition of the Hamiltonian dynamics, the motion space has to vary over time; this prevents the encoder neural network from channelling static information in the motion variables and therefore provides helpful notion of disentanglement of content and motion. Our quantitative results with both conditional and unconditional models outperform the existing baselines. We furthermore demonstrate disentanglement qualitatively using motion swapping. In the conditional model, we associate every action with a unique Hamiltonian that proves critical for the controlled generation task of image-to- oosseueence generation. HALO can generate long-term trajectories and traverse the motion manifolds of different actions in the latent space. We look forward to future applications to other sequential data types, such as molecular trajectories. A potential limitation of our model is that is less able to deal with irregularly sampled sequences, changes in tempo or reversals. In future work, we wish address this issue by allowing more flexible prior the spacing between time steps.
Societal Impact The applications of generative models for the realistic generation of images or videos an immediate concern for society. The proposed application, like motion swap and controlled generation of videos, can be potentially misused for generating fake data. scope to research and use data within the condition in the license agreement.
We restrict our paper's s
Acknowledgements
The authors would like to thank Joseph Mellor and William Toner for the helpful discussion during the project and Elliot J. Crowley for the valuable feedback on the paper. This research was funded in part by an unconditional gift from Huawei Noah's 's Ark Lab, London.
10

---

Checklist
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] After conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [Yes] Provided derivation of ELBO in appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] Data is publicly available and we provide github link to the code.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Ys]] All training details provided in appendix.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report error bars Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In the appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Provided link to obtain license of
MUG dataset.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We provide details of our network, implementation additional results in supplementary.
We provide such references in the paper.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] There is no offensive content. The data of face images used in experiments is available from license by an external organisation. We provide reference to the concerned paper.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
11

---

References
Aifanti, N., Papachristou, C., and Delopoulos, A. (2010). The MUG facial expression database. In 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pages 1-4. IEEE.
Bader, P., Blanes, S., and Casas, F. (2019), Computing the matrix exponential with an optimized Taylor polynomial approximation. Mathematics, 7(12):1174,
Barratt, S. and Sharma, R. (2018). A note the inception score. arXiv preprint arXiv: 180100199
Bengio, Y., Courville, A., and Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-188
Bird, A. and Williams, C. K. (2019). Customizing sequence generation with multi-task dynamical systems. arXiv preprint arXiv:1910.0502.
Bondesan, R. and Lamacraft, A. (2019). Learning symmetries of classical integrable systems. arXiv
preprint urXiv:1906. 04645.
Botev, A., Jaegle, A., Wirnsberger, P., Hennes, D., and Higgins, I. (2021). Which priors matter? benchmarking models for learning latent dynamics.
Caselles-Dupré, H., Garcia-Ortiz, M., and Filliat, D. (2019). Symmetry-based disentangled represen- tation learning requires interaction with environments. arXiv preprint arXiv: 104v003904.00233. 00243.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural ordinary differential equations. arXiv preprint arXiv:1806.07300
Chevalley, C. (2016). Theory of Lie Groups (PMS-8), Volume 8. Princeton University Press.
Chung, J., Kastner, K., Dinh, L.., Goel, K., Courville, A., and Bengio, Y. (2015). A recurrent latent variable model for sequential data. arXiv preprint arXiv: 500.0026226
Cohen, T. and Welling, M. (2014). Learning the irreducible representations of commutative Lie groups. In International Conference Machine Learning, pages 1755-1763. PMLR.
Connor, M. and Rozell, C. (2020). Representing closed transformation paths in encoded network latent space. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3666-3675.
Cranmer, M., Sanchez-Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., and Ho, S. (2020). Discovering symbolic models from deep learning with inductive biases. arXiv preprint arXiv:2006.112 11287.
Cremers, D. and Yuille, A. (2003). A generative model based approach to motion segmentation. In Joint Patlern Recognition Symposium, pages 313-320. Springer.
Culpepper, B. J. and Olshausen, B. A. (2009). Learning transport operators for image manifolds. In NIPS, pages 423-431.
Dandi, Y., Das, A., Singhal, S., Namboodiri, V., and Rai, P. (2020). Jointly trained image and video generation using residual vectors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3028-3042.
Dupont, E., Martin, M. B., Colburn, A., Sankar, A., Susskind, J., and Shan, Q. (2020). Equivariant neural rendering. In International Conference on Machine Learning, pages 2761-2770. PMLR.
Easton, R. W. (1993). Introduction to Hamiltonian dynamical systems and the N-body problem (KR Meyer and GR Hall). SIAM Review, 35(4):659-65 -659.
Eastwood, C. and Williams, C. (2018). A framework for the quantitative evaluation of disentangled representations. International Conference on Learning Representations.
Elliott, R. and Krishnamurthy, (1999). New dimes-aamanll filters for parameter estimation of discrete-time time linear Gaussian models. IEEE Transactions on Automatic Control, 44(5):938-955
12

---

Fraccaro, M.. Kamronn, S., Paquet, U., and Winther, O. (2017). A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint arXiv: 171.055710.
Franceschi, J.-Y. Delasalles, E., Chen, M., Lamprier, S., and Gallinari, (2020). Stochastic latent residual video prediction. In International Conference on Machine Learning, pages 3233-3246. PMLR.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, (2014). Generative adversarial networks. arXiv preprint arXiv: wrXiv:1406.2661.
Greydanus. S., Dzamba, M., and Yosinski, (2019). Hamiltonian neural networks. arXiv preprint arXiv:1906.0/563
He, J., Lehrmann, A., Marino, J., Mori, G., and Sigal, L. (2018). Probabilistic video generation using holistic attribute control. In Proceedings of the European Conference on Computer Vision (ECCV), pages 452-467.
Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L.,, Rezende, D., and Lerchner, A. (2018). Towards a definition of disentangled representations. arXiv preprint arXiv: 18/ 2.02230.
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8):1335 1780.
Hogg, D. (1983). Model-based vision: program see a walking person. Image and Vision Computing, 1(1):5-20.
Hsich, J.-T., Liu, B., Huang, D.-A., Fei-Fei, L., and Niebles, J. C. (2018). Learning decompose and disentangle representations for video prediction. arXiv preprint arXiv: 1806.04666.
Hsu, W.-N., Zhang, Y., and Glass, J. (2017). Unsupervised learning of disentangled and interpretable representations from sequential data, arXiv preprint 170.0770...
Hurri, J. and Hyvärinen, (2003). Temporal and spatiotemporal coherence in simple-cell responses:
generative model of natural image sequences. Network: Compulation in Neural Systems, 1473)5527-551. PMID: 12938770.
Kalman, R. E. (1960). A new approach to linear filtering and prediction problems.
Kannan, A., Jojic, N., and Frey, B. J. (2005). Generative model for layers of appearance and deformation. In AISTATS, volume 2005.
Karl, M., Soelch, M., Bayer, J., and Van der Smagt, P. (2016). Deep variational Bayes filters:
Unsupervised learning of state space models from raw data. arXiv preprint 1rXiv:1605.06432.
Kingma, D. P. and Welling, M. (2013). Auto-encoding variational Bayes. 131.61131..6114.
arXiv preprint
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. (2017). Building machines that learn and think like people. Behavioral and brain sciences, 40.
Memisevic, R. (2012). On multi-view feature learning. arXiv preprint arXiv:1206.4000
Miladinović, D., Gondal, W., Schölkopf, B., Buhmann, M., and Bauer, S. (2019). Disentangled state space models: Unsupervised learning of dynamics across heterogeneous environments.
Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K., and Lee, H. (2019). Unsupervised learning of object structure and dynamics from videos. arXiv preprint
Noether, E. (1918). Invariant variation problems, gott.
Pavlovic, V., Rehg. J. M., and MacCormick, J. (2000). Learning switching linear models of human motion. In NIPS, volume 2, page
Quessard, R., Barrett, T., and Clements, (2020). Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33.
13

---

Rao, R. P. and Ruderman, D. L. (1999). Learning Lie groups for invariant visual perception. Advances in neural information processing systems, pages 810-816.
Roweis, and Ghahramani, Z. (1999). A unifying review of linear Gaussian models. Neural computation, 11(2):305-345.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training gans. Advances in neural information processing systems, 29.
Sanchez-Gonnalez, A., Bapst, V., Cranmer, K., and Battaglia, P. (2019). Hamiltonian graph networks with ODE integrators. arXiv preprint 1909. 12790.
Sanchez-Gonzalee A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., and Battaglia, P. (2020). Learning to simulate complex physics with graph networks. In International Conference on Machine Learning, pages 8469. PMLR.
Sohl-Dickstein, J., Wang. C. M., and Olshausen, B. A. (2010). An unsupervised algorithm for learning Lie group transformations. arXiv preprint arXiv:1001.1027.
Srivastava, N., Mansimov, E., and Salakhudinov, R. (2015). Unsupervised learning of video rep- resentations using LSTMs. In International conference on machine learning, pages 843-852.
PMLR.
Starner, T. and Pentland, A. (1997). Real-time American sign language recognition from video using hidden Markov models. In Motion-based recognition, pages 227-243. Springer.
Storkey, A. and Williams, C. (2003). Image modeling with position-encoding dynamic trees. IEEE Transactions on Puttern Analysis and Machine Intelligence, 25(7):859-871.
Toth, P., Rezende, J., Jaegle, A., Racanière, S., Botev, A., and Higgins, I. (2019). Hamiltonian generative networks. arXiv preprint arXiv: 1909.13999
Tulyakov, S., Liu, M.-Y., Yang, X., and Kautz, J. (2018). MoCoGAN: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 1526-1535.
Villegas, R., Yang, J., Hong, S., Lin, X., and Lee, H. (2017). Decomposing motion and content for natural video sequence prediction. arXiv preprint
Yildiz, C., Heinonen, M., and Lähdesmäki, H. (2019). ODE2VAE: Deep generative second order ODEs with Bayesian neural networks.
Yingzhen, L. and Mandt, S. (2018). Disentangled sequential autoencoder. International Conference Machine Learning. pages 5670-5679 PMLR.
Yoon, J., Jarrett, D., and der Schaar, M. (2019). Time-series generative adversarial networks.
Zhu, Y., Min, M. R., Kadav, A., and Graf, H. (2020), S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6538-68 -5577.
14


Reviews:
[{"review_id": "Xy5b3_kPEjz", "reviewer": "Reviewer_FnRf", "summary": "This paper deals with the task of generating image sequences. Specifically, the authors propose a method called Halo that allows to disentangle the content from the motion in image sequences, in the VAE framework. They do so by separating the latent space in two spaces: 1) the content space, a global content vector that summarizes the image sequence;  2) the motion space, a sequence of time-dependent vectors that capture the dynamics of the sequence. The main contribution of the authors is to model the motion space with Hamiltonian dynamics. The authors claim that Hamiltonian dynamics have good inductive biases for sequence generation, such as reversibility of the motion. Experiments on simple image sequences are performed to prove the quality of their model. ", "questions": "1) Are $H_k$ matrices learned, or are they kept as drawn from the initialization with symmetric matrix $M_k$ and $H_k = J M_k$. If learned, how  is $H_k$ constrained during optimization?\n2) Does the inductive bias of the Hamiltonian operator help in a setting with few training examples? ", "limitations": "The authors addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* 1) The latent Hamiltonian operator is quite generic. It could be extended and used with other families of deep generative models for sequences, and thus be of great interest for practitioners. \n* 2) Halo achieves SOTA scores on motion/content disentanglement metrics. Ablations with similar architecture and other sequence models are convincing (especially Table 1 of Supplementary Material). \n\n\nWeaknesses\n* 1) Stochasticity is only allowed by the gaussian sampling, but there is no stochasticity in the Hamiltonian operator. Thus, Halo can only generate one motion vector given input frames. However, trajectory prediction is a highly stochastic process. This could be a limitation that makes scaling to more complex environments difficult.\n* 2) How does the method guarantee that no motion information leak in the global content vector? Since the encoder + LSTM that generate the global vector see the whole input sequence, it could also capture some information about the motion.\n* 3) Limited evaluation: the model is tested on three simple datasets. Could be interesting to see how it performs on more complex datasets, with different/moving backgrounds.\nIt also misses comparisons with recent works. E.g.:\na) Franceschi et al., \"Stochastic latent residual video prediction\". ICML 2020.\nb) Wang et al., \"G3AN: Disentangling appearance and motion for video generation\". CVPR 2020.\n* 4) Clarity. While the paper is well written, there lacks some implementation details on the main component of the paper: the hamiltonian operator (see questions). This affects the understandability.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SB_BBFqS0Jn", "reviewer": "Reviewer_pzKP", "summary": "The paper proposes Halo, a novel type of variational autoencoder with structured latent space and demonstrates its applications to different types of (controlled) video generation tasks. The main contribution is a principled decomposition of the latent space into a content space and a motion space, where the motion space is modeled using Hamiltonian dynamics. The structural constraints (e.g, symmetries) imposed by these dynamics induce desirable properties like reversibility and volume-preservation, enabling the conservation of (learned) quantities.", "questions": "- l.201f: *“without significant loss of generality, we propose a linear Hamiltonian system in the latent layer, relying on the deep neural network mapping to data space to handle all nonlinear aspects.”* This comment raises a larger question: how does the optimization process ensure that not *all* relevant information is pushed to the decoder, while the carefully designed latent space models only trivial dependencies? In the same vein, how does the optimization process guarantee that no motion data is being pushed into ${\\bf z}$, thus completely circumventing the phase space?\n\n- It is not entirely clear to me how the Hamiltonian nature of ${\\bf H}_k$ is preserved during the optimization process. My understanding is that the symmetry of ${\\bf M}_k$ ensures this, but I would appreciate a confirmation.\n\n- l.224: *“The symplectic geometry proves useful for long term term *(sic!)* sequence generation.”* The sequences in the main paper are all relatively short and even the experiments on longer sequences shown in the Appendix are only 32 frames, i.e., a little over a second at 25fps. What is the limit before the generated sequences visibly deteriorate or become static?\n\n- What architecture does the pre-trained action prediction classifier use and how was it trained?", "limitations": "- The paper flags potential misuse in the area of fake video data generation. \n- The paper does not contain a limitations section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ Video generation is an important and challenging task with a rich history. The proposed approach takes a fresh perspective on this topic and explores the benefits of inductive bias based on principles rooted in the physics community.\n\n+ Sections 1-3 (introduction, related work, method) are well-organized and easy to follow: the main contributions are clearly formulated, the figures are helpful in understanding architectural details, and the mathematical notation is (mostly) consistent. The related work section is commendable and provides a comprehensive overview of the field. The paper does have a fairly strong physics flavour and I would recommend to provide stronger guidance for an audience which may not be familiar with topics that are not part of core ML, such as group action, phase space, conservation law, and symplectic geometry.\n\n+ The Hamiltonian design of the latent space is interesting and novel, and the advantages of reversible and symplectic latent dynamics make intuitive sense. I also appreciate the principled derivation of the dynamical model $f$ from a constant-energy perspective (l.206-l.214). The variational inference section is less clear and I would encourage the authors to move at least some intuition about the ELBO from the Appendix into the main paper.\n\n**Weaknesses**\n\n- The weakest part of the paper are its experiments, both in terms of their presentation and design.\n\t- Presentation: \n\t\t- The structure of the experiments is confusing throughout section 4. For example, the description of the Sprites and MUG datasets starts in the middle of the “Rotating Balls” paragraph (l.262). Likewise, the description of the baseline comparison starts in the middle of the “Quantitative Evaluation” section (l.302). Grammar and text flow in the experiment section also feel unpolished. Finally, none of the figures in this section have proper axes/labels and the reader needs to count rows and infer the content from the caption or even the main text (l.348-350 for Figure 5 (left)).\n\n\t- Design: \n\t\t- Since Table 1 does not include a comparison to other baselines it is not possible to assess whether the presented SSIM/PSNR/MSE scores are competitive or not. Why not use the same metrics as in Table 2?\n\t\t- Table 3 is not mentioned in the text and seems to be based on the single example of Figure 5 (right), which is not enough to make any general statements. The positional encoding mentioned in this table is not explained and not supported by any qualitative evidence.\n\t\t- In Figure 4 (left) the reconstructed and generated sequences look fairly similar, which can be an indication of low diversity.\n\t\t- It is unclear how the sequences of the rotating balls dataset were generated as the mentioned constraint does not specify any temporal pattern. What is the dynamic model used here?\n\t\t- The sequences are very short (8/16 frames) and small (64 x 64). What is the main bottleneck that prevents application to high-fidelity image sequences?\n\n**Minor comments**\n\n- Typos: Figure 1 (“alongwith”), l.224 (“long term term”), l.234 (“(6))”), l.258 (as”blue”), l.283 (“EvaluationWe”)\n\n- The Appendix provides valuable information about the ELBO objective, terminology, and network structures, but the main paper does not refer to it often enough (e.g., content/position/momentum network).\n\n- The paper follows a top-down approach, first introducing high-level structures and then filling in the details. While that is a reasonable approach, it does mean that readers will have to read the paper twice (or go back to previous paragraphs), because the motivation for some design choices remains initially unclear. One example is the structure of the phase space.\n\n**Summary**. I appreciate the technical formulation of this paper but am on the fence due to the weak and unconvincing experiments. I encourage the authors to address the concerns above as well as the questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AZ6bD7BOLBa", "reviewer": "Reviewer_pj74", "summary": "This paper proposes a deep state space model for videos. The dynamics are defined by linear Hamiltonian Dynamics, and the motion matrix is further assumed to be block diagonal in order to separate different categories of actions. Like previous works, a latent variable z is introduced for explaining content and kept fixed for all frames. Experiments are carried out on Sprites and MUG to demonstrate the efficacy.", "questions": "Please see comments above.", "limitations": "Yes. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and easy to follow. The idea of introducing Hamiltonian dynamics as an inductive bias for explaining repetitive or cycled motions in videos are reasonable and natural. The theoretical derivation is technically sound.\n\nWeaknesses:\n\nMy main concern is that the proposed method is not well supported by the experimental results:\n- I don't understand how SSIM and PSNR can be used for evaluating \"generation quality\", as generated samples are supposed to be different from the training datasets. I can only imagine that the numbers in Table 1 are reported for reconstruction, in which case it is not for generation quality as described by the paper. Also, no baseline methods are compared in terms of reconstruction.\n- There's no commonly used metrics reported that are designed for really evaluating sample quality, such as FVD or Inception scores.\n- For disentanglement evaluation, it is not fair to use the conditional Halo model to compared with the baselines which are trained unconditionally. The only fair way is the compare unconditional Halo models with the baselines where the performance of the proposed model does not stand out. \n- In the ablation study section, it makes me confused that the paper mentioned RNN or linear dynamic model cannot make image move but it also showed the results of swapping motions of the two baselines where the video sequences are changing over time. Also it sounds wired to me that the linear/RNN dynamics cannot do image-to-seq as those have been applied by many classical state space models. \n\nIn experiments, two operators H and skew-H are compared but there's no official definition of skew-H in the previous sections. \n\nThe model assumes that the action space can be divided into subspaces where each subspace represents a unique action. This representation can be highly ineffective if the number of actions goes huge. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept