Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a deep state space model for videos. The dynamics are defined by linear Hamiltonian Dynamics, and the motion matrix is further assumed to be block diagonal in order to separate different categories of actions. Like previous works, a latent variable z is introduced for explaining content and kept fixed for all frames. Experiments are carried out on Sprites and MUG to demonstrate the efficacy.
strengths_and_weaknesses: Strengths:

The paper is well written and easy to follow. The idea of introducing Hamiltonian dynamics as an inductive bias for explaining repetitive or cycled motions in videos are reasonable and natural. The theoretical derivation is technically sound.

Weaknesses:

My main concern is that the proposed method is not well supported by the experimental results:
- I don't understand how SSIM and PSNR can be used for evaluating "generation quality", as generated samples are supposed to be different from the training datasets. I can only imagine that the numbers in Table 1 are reported for reconstruction, in which case it is not for generation quality as described by the paper. Also, no baseline methods are compared in terms of reconstruction.
- There's no commonly used metrics reported that are designed for really evaluating sample quality, such as FVD or Inception scores.
- For disentanglement evaluation, it is not fair to use the conditional Halo model to compared with the baselines which are trained unconditionally. The only fair way is the compare unconditional Halo models with the baselines where the performance of the proposed model does not stand out. 
- In the ablation study section, it makes me confused that the paper mentioned RNN or linear dynamic model cannot make image move but it also showed the results of swapping motions of the two baselines where the video sequences are changing over time. Also it sounds wired to me that the linear/RNN dynamics cannot do image-to-seq as those have been applied by many classical state space models. 

In experiments, two operators H and skew-H are compared but there's no official definition of skew-H in the previous sections. 

The model assumes that the action space can be divided into subspaces where each subspace represents a unique action. This representation can be highly ineffective if the number of actions goes huge. 

questions: Please see comments above.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes Halo, a novel type of variational autoencoder with structured latent space and demonstrates its applications to different types of (controlled) video generation tasks. The main contribution is a principled decomposition of the latent space into a content space and a motion space, where the motion space is modeled using Hamiltonian dynamics. The structural constraints (e.g, symmetries) imposed by these dynamics induce desirable properties like reversibility and volume-preservation, enabling the conservation of (learned) quantities.
strengths_and_weaknesses: **Strengths**
+ Video generation is an important and challenging task with a rich history. The proposed approach takes a fresh perspective on this topic and explores the benefits of inductive bias based on principles rooted in the physics community.

+ Sections 1-3 (introduction, related work, method) are well-organized and easy to follow: the main contributions are clearly formulated, the figures are helpful in understanding architectural details, and the mathematical notation is (mostly) consistent. The related work section is commendable and provides a comprehensive overview of the field. The paper does have a fairly strong physics flavour and I would recommend to provide stronger guidance for an audience which may not be familiar with topics that are not part of core ML, such as group action, phase space, conservation law, and symplectic geometry.

+ The Hamiltonian design of the latent space is interesting and novel, and the advantages of reversible and symplectic latent dynamics make intuitive sense. I also appreciate the principled derivation of the dynamical model $f$ from a constant-energy perspective (l.206-l.214). The variational inference section is less clear and I would encourage the authors to move at least some intuition about the ELBO from the Appendix into the main paper.

**Weaknesses**

- The weakest part of the paper are its experiments, both in terms of their presentation and design.
	- Presentation: 
		- The structure of the experiments is confusing throughout section 4. For example, the description of the Sprites and MUG datasets starts in the middle of the “Rotating Balls” paragraph (l.262). Likewise, the description of the baseline comparison starts in the middle of the “Quantitative Evaluation” section (l.302). Grammar and text flow in the experiment section also feel unpolished. Finally, none of the figures in this section have proper axes/labels and the reader needs to count rows and infer the content from the caption or even the main text (l.348-350 for Figure 5 (left)).

	- Design: 
		- Since Table 1 does not include a comparison to other baselines it is not possible to assess whether the presented SSIM/PSNR/MSE scores are competitive or not. Why not use the same metrics as in Table 2?
		- Table 3 is not mentioned in the text and seems to be based on the single example of Figure 5 (right), which is not enough to make any general statements. The positional encoding mentioned in this table is not explained and not supported by any qualitative evidence.
		- In Figure 4 (left) the reconstructed and generated sequences look fairly similar, which can be an indication of low diversity.
		- It is unclear how the sequences of the rotating balls dataset were generated as the mentioned constraint does not specify any temporal pattern. What is the dynamic model used here?
		- The sequences are very short (8/16 frames) and small (64 x 64). What is the main bottleneck that prevents application to high-fidelity image sequences?

**Minor comments**

- Typos: Figure 1 (“alongwith”), l.224 (“long term term”), l.234 (“(6))”), l.258 (as”blue”), l.283 (“EvaluationWe”)

- The Appendix provides valuable information about the ELBO objective, terminology, and network structures, but the main paper does not refer to it often enough (e.g., content/position/momentum network).

- The paper follows a top-down approach, first introducing high-level structures and then filling in the details. While that is a reasonable approach, it does mean that readers will have to read the paper twice (or go back to previous paragraphs), because the motivation for some design choices remains initially unclear. One example is the structure of the phase space.

**Summary**. I appreciate the technical formulation of this paper but am on the fence due to the weak and unconvincing experiments. I encourage the authors to address the concerns above as well as the questions below.
questions: - l.201f: *“without significant loss of generality, we propose a linear Hamiltonian system in the latent layer, relying on the deep neural network mapping to data space to handle all nonlinear aspects.”* This comment raises a larger question: how does the optimization process ensure that not *all* relevant information is pushed to the decoder, while the carefully designed latent space models only trivial dependencies? In the same vein, how does the optimization process guarantee that no motion data is being pushed into ${\bf z}$, thus completely circumventing the phase space?

- It is not entirely clear to me how the Hamiltonian nature of ${\bf H}_k$ is preserved during the optimization process. My understanding is that the symmetry of ${\bf M}_k$ ensures this, but I would appreciate a confirmation.

- l.224: *“The symplectic geometry proves useful for long term term *(sic!)* sequence generation.”* The sequences in the main paper are all relatively short and even the experiments on longer sequences shown in the Appendix are only 32 frames, i.e., a little over a second at 25fps. What is the limit before the generated sequences visibly deteriorate or become static?

- What architecture does the pre-trained action prediction classifier use and how was it trained?
limitations: - The paper flags potential misuse in the area of fake video data generation. 
- The paper does not contain a limitations section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper deals with the task of generating image sequences. Specifically, the authors propose a method called Halo that allows to disentangle the content from the motion in image sequences, in the VAE framework. They do so by separating the latent space in two spaces: 1) the content space, a global content vector that summarizes the image sequence;  2) the motion space, a sequence of time-dependent vectors that capture the dynamics of the sequence. The main contribution of the authors is to model the motion space with Hamiltonian dynamics. The authors claim that Hamiltonian dynamics have good inductive biases for sequence generation, such as reversibility of the motion. Experiments on simple image sequences are performed to prove the quality of their model. 
strengths_and_weaknesses: Strengths
* 1) The latent Hamiltonian operator is quite generic. It could be extended and used with other families of deep generative models for sequences, and thus be of great interest for practitioners. 
* 2) Halo achieves SOTA scores on motion/content disentanglement metrics. Ablations with similar architecture and other sequence models are convincing (especially Table 1 of Supplementary Material). 


Weaknesses
* 1) Stochasticity is only allowed by the gaussian sampling, but there is no stochasticity in the Hamiltonian operator. Thus, Halo can only generate one motion vector given input frames. However, trajectory prediction is a highly stochastic process. This could be a limitation that makes scaling to more complex environments difficult.
* 2) How does the method guarantee that no motion information leak in the global content vector? Since the encoder + LSTM that generate the global vector see the whole input sequence, it could also capture some information about the motion.
* 3) Limited evaluation: the model is tested on three simple datasets. Could be interesting to see how it performs on more complex datasets, with different/moving backgrounds.
It also misses comparisons with recent works. E.g.:
a) Franceschi et al., "Stochastic latent residual video prediction". ICML 2020.
b) Wang et al., "G3AN: Disentangling appearance and motion for video generation". CVPR 2020.
* 4) Clarity. While the paper is well written, there lacks some implementation details on the main component of the paper: the hamiltonian operator (see questions). This affects the understandability.  
questions: 1) Are $H_k$ matrices learned, or are they kept as drawn from the initialization with symmetric matrix $M_k$ and $H_k = J M_k$. If learned, how  is $H_k$ constrained during optimization?
2) Does the inductive bias of the Hamiltonian operator help in a setting with few training examples? 
limitations: The authors addressed the limitations.
review_title: null

Meta-review: metareview: This paper proposes a novel type of variational auto encoder, referred to as HELO. The latent space is decomposed into a content space and a motion space, and the main contribution is the proposal to model the motion space using Hamiltonian dynamics. All reviewers agree that the idea of using Hamiltonian dynamics is interesting and novel. One main critique, that the authors agreed on, was that the operator does not contain any stochasticity and that this might be a limitation when applying the idea to model more complex data. Another remark was that the experiments are limited and experiments on less constraint data are missing. A quick look at the baseline methods revealed that they also use the same kind of data sets to evaluate their methods, so this latter concern might be of minor importance. 
All in all, the potential positive outcomes of this paper outweight its current limitations, so we recommend acceptance at this point, while urging authors to address the remaining concerns in the final version. 

confidence: Certain
Decision: Accept