Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new framework for spatiotemporal disentanglement. In particular, it contributes to the disentanglement of content and dynamics using neural ODEs. 

Strength: 
1. I think it is a well-written paper with interesting experimental design, especially the “swap” and “multi-view” experiments that measure the degree of disentanglement. 
2. The proposed model consistently outperforms well-established approaches for video prediction, including DrNet, DDPAE, and PhyDNet, which is also a PDE-based model for spatiotemporal disentanglement.

Weakness:
1. Although the effectiveness of the model has been validated on several datasets, most of them are synthetic. I strongly encourage the authors to include real human motion datasets. For example, DrNet was evaluated on KTH, and PhyDNet was evaluated on Human3.6M.
2. The model was only compared with PKnl and PhyDNet on the Wave and SST datasets, and I am curious about how models beyond disentanglement, such as SVG and MIM, perform in this case. Also, the results in Figure 2 appear to be quite blurry and do not show significant improvement. These experiments mainly show the general ability of the proposed model for video prediction rather than spatiotemporal disentanglement. 

Correctness: 
In Eq. (12), the authors use a Gaussian prior to convey dynamic information and exclude the spatial information. Is it a good thing or a bad thing from the view of modeling temporal dynamics? A question is: can such a simple dynamic model be applied to datasets with complex motion information?


review_title: Well-written paper with interesting but insufficient experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a generative model for videos where the latent trajectories have two components - a term without a slowness loss that represents "content" and a term with a slowness loss that represents "style". They present results on a dataset simulating the wave equation and on videos of moving MNIST digits and 3D chairs. The results are generally good, especially for long roll-outs, and they demonstrate something like disentangling by showing that the identities of the digits can be swapped in the moving MNIST data.

My main objection with the paper is that it has nothing to do with PDEs or separation of variables. The actual latent trajectories are simulated as *ODEs*, not PDEs, which are then used to generate images. The justification in terms of separation of variables is also a bait-and-switch...a slowness penalty is added to the loss for one of the latent trajectories, that is all. Factored latent trajectories are a well-established modeling technique for time series already. The use of ODEs parameterized by neural networks rather than, say, LSTMs or other RNN architectures is also more a difference in degree than in kind from other sequence models. Much like how ResNets become neural ODEs in the limit of very deep networks, simply using Runge-Kutta updates parameterized by a neural network is still technically a kind of RNN (if you define RNN very loosely as any nonlinear iterated function learnable by gradient descent). So I'm not sure that this paper actually does most of what it claims to be doing in the motivation.
review_title: Another generative model for videos with a slow feature loss

final_score: rating: 7: Good paper, accept
review_content: The paper presents a spatiotemporal disentanglement method for handling sequence data. Solving high-dimensional PDEs, for deriving the exact dynamics, is difficult; hence, this work proposes learning time-invariant and time-dependent representations separately to solve this problem. To achieve this goal, the authors devised a model that incorporates a temporal ODE process. The provided experiments indicate that the method achieves good performance, although the gain is not consistent.

The overall derivation and methodology of this paper are technically sound, and I guess discarding $S^\prime$ was a practical choice for learning time-invariant representation. Although the proposed model is generally applicable, the experiments only cover prediction results on classical image datasets. Nevertheless, the underlying timescales of datasets are sufficiently varied to demonstrate the effectiveness in general temporal sequences.

[Quality]

The paper is clearly written overall. However, Section 4 was a bit hard to follow, which is the core part of this paper. For example, "the system coordinates" and reasons for the relaxation (Sec. 4.2) were not clearly explained for people who are not familiar with the area. It would be harder to understand the architecture if Fig. 1 was not given.

[Originality]

The originality of the paper is not stellar, but sufficient for acceptance. 

[Significance]

The significance of this work is mainly for model architecture. I believe these kinds of approaches, which can internally model continuous dynamics, are heavily preferred when solving real-world problems. Therefore, the significance of the paper is sufficient for acceptance.

review_title: Review

Meta-review: This paper proposes a model for disentangling content and dynamics, but unlike the majority of previous work, the dynamics are modeled using ODEs rather than their discrete approximations - RNNs. The reviewers agree that the paper is well written, and the results look good, especially for longer trajectories. Hence, I am happy to recommend this paper for acceptance.
Decision: Accept (Poster)