PDE-Driven Spatiotemporal Disentanglement
Authors: Jérémie Donà
Jean-Yves Franceschi
sylvain lamprier
patrick gallinari
Venue: ICLR 2021 Poster (2021)
Keywords: disentanglement
spatiotemporal prediction
representation learning
dynamical systems
separation of variables

Abstract: A recent line of work in the machine learning community addresses the problem of predicting high-dimensional spatiotemporal phenomena by leveraging specific tools from the differential equations theory. Following this direction, we propose in this article a novel and general paradigm for this task based on a resolution method for partial differential equations: the separation of variables. This inspiration allows us to introduce a dynamical interpretation of spatiotemporal disentanglement. It induces a principled model based on learning disentangled spatial and temporal representations of a phenomenon to accurately predict future observations. We experimentally demonstrate the performance and broad applicability of our method against prior state-of-the-art models on physical and synthetic video datasets.

Published as a conference paper at ICLR 2021
PDE-DRIVEN SPATIOTEMPORAL DISENTANGLEMENT
Jérémie Donå, 1 Jean-Yver Franceschi, i* Sylvain Lamprier's & Patrick Gallinari
*oobonnne Université, CNRS, LIP6, F-75005 Paris, France
*Criteo AI Lab, Paris, France firstname. last name fr
ABSTRACT
A recent line of work in the machine learning community addresses the problem of predicting high-dimensional spatiotemporal phenomena by leveraging specific tools from the differential equations theory. Following this direction, we propose in this article novel and general paradigm for this task based resolution method for partial differential equations: the separation of variables. This inspiration allows to introduce dynamical interpretation of spatiotemporal disentanglement. It induces a principled model based on learning disentangled spatial and temporal representations of phenomenon to accurately predict future observations. We experimentally demonstrate the performance and broad applicability of our method against prior oottt--f--hearr models on physical and synthetic video datasets.
INTRODUCTION
The interest of the machine learning community in physical phenomena has substantially grown for the last few years (Shi et al., 2015; Long et al., 2018; Greydanus et al., 2019). In particular, an increasing amount of works studies the challenging problem of modeling the evolution of dynamical systems, with applications in sensible domains like climate or health science, making the understanding of physical phenomena key challenge in machine learning. To this end, the community has successfully leveraged the formalism of dynamical systems and their associated differential formulation as powerful tools to specifically design efficient prediction models. In this work, we aim at studying this prediction problem with a principled and general approach, through the prism of Partial Differential Equations (PDEs), with focus learning spatiotemporal disentangled representations.
Prediction via spatiotemporal disentanglement was first studied in video prediction works, in order to separate static and dynamic information (Denton & Birodkar, 2017) for prediction and interpretability purposes, Existing models are particularly complex, involving either adversarial losses or variational inference. Furthermore, their reliance on Recurrent Neural Networks (RNNs) hinders their ability to model spatiotemporal phenomena (Yildiz et al., 2019; Ayed et al., 2020; Franceschi et al.. 2020). Our proposition addresses these shortcomings with a simplified and improved model by grounding spatiotemporal disentanglement in the PDE formalism.
Spatiotemporal phenomena obey physical laws such as the conservation of energy, that lead to describe the evolution of the system through PDEs. Practical examples include the conservation of energy for physical systems (Hamilton, 1835), or the equation describing constant illumination in scene (Horn & Schunck, 1981) for videos that has had longstanding impact in computer vision with optical flow methods (Dosovitskiy et al., 2015; Finn et al., 2016). We propose to model the evolution of partially observed spatiotemporal phenomena with unknown dynamics by leveraging formal method for the analytical resolution of PDEs: the functional separation of variables (Miller, 1988). Our framework formulates spatiotemporal disentanglement for prediction learning separable solution, where spatial and dynamic information are represented in separate variables. Besides offering a novel interpretation of spatiotemporal disentanglement, it confers simplicity and performance compared to existing methods: disentanglement is achieved through the sole combination of a prediction objective and regularization penalties, and the temporal dynamics is defined by a learned Ordinary Differential Equation (ODE). We experimentally demonstrate the applicability, disentanglement capacity and
"Equal contribution.

---

Published as a conference paper at ICLR 2021
forecasting performance of the proposed model various spatiotemporal phenomena involving standard physical processes and synthetic video datasets against prior state-of-the-arr models.
RELATED WORK
Our contribution deals with two main directions of research: spatiotemporal disentanglement and the coupling of neural networks and PDEs.
Spatiotemporal disentanglement. Disentangling factors of variations is an essential representation learning problem (Bengio et al., 2013). Its cardinal formulation for static data has been extensively studied, with state-of-the-art solutions (Locatello et al., 2019) being essentially based on Variational Autoencoders (VAEs; Kingma & Welling, 2014; Rezende et al., 2014), As for sequential data, several disentanglement notions have been formulated, ranging from distinguishing objects in a video (Hsich et al., 2018; van Steenkiste et al., 2018) to separating and modeling multi-scale dynamics (Hsu et al., 2017; Yingzhen& & Mandt, 2018).
We focus in this work on the dissociation of the dynamics and visual aspects for spatiotemporal data. Even in this case, dissociation can take multiple forms. Examples in the video generation community include decoupling the foreground and background (Vondrick al., 2016), constructing structured frame representations (Villegas et al., 2017b; Minderer et al., 2019; Liu et al., 2019), extracting physical dynamics (Le Guen & Thome, 2020), or latent modeling of dynamics in a state-space manner (Fraccaro et al., 2017; Franceschi et al., 2020). Closer to our work, Denton Birodkar (2017), Villegas al. (2017a) and Hsich et al. (2018) introduced in their video prediction models explicit latent disentanglement of static and dynamic information obtained using adversarial losses (Goodfellow et al., 2014) or VAES. Disentanglement has also been introduced in more restrictive models relying on data-specific assumptions (Kosiorek et al., 2018; Jaques et al., 2020), and in video generation (Tulyakov et al., 2018). We aim in this work at grounding and improving spatiotemporal disentanglement with more adapted inductive biases by introducing a paradigm leveraging the functional separation of variables resolution method for PDEs.
Spatiotemporal prediction and PDE-based neural network models. An increasing number of works combining neural networks and differential equations for spatiotemporal forecasting have been produced for the last few years. Some of them show substantial improvements for the prediction of dynamical systems or videos compared to standard RNNs by defining the dynamics using learned ODEs (Rubanova et al., 2019; Yildız et al., 2019; Ayed et al., 2020; Le Guen & Thome, 2020), following Chen et al. (2018), or adapting them to stochastic data (Ryder et al., 2018; Li et al., 2020: Franceschi al., 2020). Most PDE-based spatiotemporal models exploit some prior physical knowledge. It can induce the structure of the prediction function (Brunton et al., 2016; de Avila Belbute-Peres et al., 2018 8) specific cost funetions, thereby improving model performances. For instance, de Bézenac et al. (2018) shape their prediction function with an advection-diffusiio mechanism, and Long et al. (2018; 2019) estimate PDEs and their solutions by learning convolutional filters proven to approximate differential operators. Greydanus et al. (2019), Chen et al. (2020) and Toth al. (2020) introduce non-regression losses by taking advantage of Hamiltonian mechanics (Hamilton, 1835), while Tompson et al. (2017) and Raissi et al. (2020) combine physically inspired constraints and structural priors for fluid dynamic prediction. Our work deepens this literature by establishing a novel link between resolution method for PDEs and spatiotemporal disentanglement thereby introducing a data-agnostic model leveraging any static information in observed phenomena.
BACKGROUND: SEPARATION OF VARIABLES
Solving high dimensiona PDEs is difficult analytical and numerical problem (Bungartz & Griebel, 2004). Variable separation aims at simplifying it by decomposing the solution, e.g., as a simple combination of lower-dimensional functions, thus reducing the PDE to simpler differential equations.
3.1 SIMPLE CASE STUDY
Let us introduce this technique through a standard application, with proofs in Appendix A. on the one-dimensional heat diffusion problem (Fourier, 1822), consisting in bar of length L,, whose
2

---

Published as a conference paper at ICLR 2021
temperature at time t and position is denoted by u(x; t) and satisfies:
au
Ouu
= dr.
u t) = =((,t) = 0,
u(2,,) f(x).
(I)
0t
Suppose that solution is product-separable, i..., it can be decomposed as: u(z,t) ij (z) (t))).
Combined with Equation (1), it leads to (x) (z) (t))) (t)/u2(t). The left- and right-hand sides of this equation are respectively independent from and æ. Therefore, both sides are constant, and solving both resulting ODEs gives solutions of the form, with μ E IR and E N:
u( t) = μ sin (nTz/ L)) x (xpp-(enT/L)))
The superposition principle and the uniqueness of solutions under smoothness constraints allow then build the set of solutions of Equation (1) with linear combinations of separable solutions (Le Dret & Lucquin, 2016). Besides this simple example, separation of variables can be more elaborate.
3.2 FUNCTIONAL SEPARATION OF VARIABLES
The functional separation of variables (Miller, 1988) generalizes this method. Let be a function obeying a given arbitrary PDE. The functional variable separation method amounts to finding parameterization 2, a functional U, an entangling function ç, and representations & and & such that:
sepatitar
(((x),(())
u(x, 1) = U()).
(3)
Trivial choices & = and identity function U. Φ and d ensure the validity of this reformulation. Finding suitable é, 1b, U, and & with regards to the initial PDE can facilitate its resolution by inducing separate simpler PDEs on o, t0, and U. For instance, sepdaai-separaaility is retrieved with U = exp. General results on the existence of separable solutions have been proven (Miller, 1983), though their uniqueness depends on the initial conditions and the choice of functional separation (Polyanin, 2020).
Functional separation of variables finds broad applications. helps to solve refinements of the heat equation, such generalizations with advection term (see Appendix A.2) with complex diffusion and source terms forming a general transport equation (Jia et al., 2008). Besides the heat equation, functional separation of PDEs is also applicable in various physics fields like reaction- diffusion with non-linear sources or convection-diffusie -iffiuuin phenomena (Polyanin, 2019; Polyanin & Zhurov, 2020), Hamiltonian physics (Benenti, 1997), or even general relativity (Kalnins et al., 1992).
Reparameterizations such as Equation (3) implement a separation of spatial and temporal factors of variations, i.e., spatiotemporal disentanglement. We introduce in the following a learning framework based on this general method.
4 PROPOSED METHOD
We propose to model spatiotemporal phenomena using the functional variable separation formalism. We first describe our notations and then derive principled model and constraints from this method.
4.1 PROBLEM FORMULATION THROUGH SEPARATION OF VARIABLES
We consider a distribution P of observed spatiotemporal trajectories and corresponding observation samples (Utor Ato++At. ... vr,) ), with v, E V C ITTh and ( to + vÂt. Each sequence P corresponds to an observation of a dynamical phenomenon, assumed to be described by a hidden functional u. (also denoted by for the sake of simplicity) of space coordinates x E X C R and time 1. E R that characterizes the trajectories. More precisely, u, describes an unobserved continuous dynamics and e corresponds to instantaneous discrete spatial measurements associated to this dynamics. Therefore, we consider that V; results from time-independent function Ç of the u. (.)t). For example, might consist in temperatures measured at some points of the sea surface, while u. would be the complete ocean circulation model. In other words, v provides partial information about u, and is a projection of the full dynamics. We seek to learn a model which, when conditioned on prior observations, can predict future observations.
a mapping
To this end, we posit that the state u of each observed trajectory v is driven by hidden PDE, shared among all trajectories; we discuss this assumption in details in Appendix C.I. Learning such PDE
3

---

Published as a conference paper at ICLR 2021
Egg
C rog
D
Ve
LAB
Viy
S
T2
Uto
Tp
S
ET
D
Sree
2t
Tto CT
f
Figure 1: Computational graph of the proposed model. Es and Er take contiguous observations as input; time invariance is enforced on S; the evolution of T, is modeled with an ODE and is constrained coincide with Er; Tio regularized; forecasting amounts decoding from S and Tr.
and its solutions would then allow us to model observed trajectoriess v. However, directly learning solutions to high-dimensional unknown PDEs is complex task (Bungartz & Griebel, 2004; Sirignano & Spiliopoulos, 2018). We aim in this work at simplifying this resolution. We propose to do so by relying on the functional separation of variables of Equation (3), in order leverage a potential separability of the hidden PDE. Therefore, analogously to Equation (3), we propose to formulate the problem as learning wwwervattion-oostraaiiin Ø, 3b and U, as well as & and Ç, such that:
= {()(z), (tt),
u(x t) U(z),
122 ((u(,,t))
(4)
with & and 1 allowing disentangle the prediction problem. the formalism of the functional separation of variables, this amounts to decomposing the full solution thereby learning spatial PDE on @, temporal ODE on v. and a PDE on U, as well as their respective solutions.
4.2 FUNDAMENTAL LIMITS AND RELAXATION
Directly learning u is, however, restrictive choice. Indeed, when formulating PDEs such in Equation (I), spatial coordinates (I, y, etc.) and time appear as variables of the solution. Yet, unlike in fully observable phenomena studied by Sirignano & Spiliopoulos (2018) and Raissi (2018), directly accessing theses variables in practice can be costly or infeasible in our partially observed setting. In other words, the nature and number of these variables unknown. For example, the dynamic of the observed sea surface temperature is highly dependent on numerous unobserved variables such temperature deeper levels wind intensity. Explicitly taking into account these unobserved variables can only be done with prior domain knowledge. To maintain the generality of the proposed approach, we choose not make any data-specific assumption on these unknown variables.
We overcome these issues by eliminating the explicit modeling of spatial coordinates by learning dynamic and time-invariant representations accounting respectively for the time-dependent and space- dependent parts of the solution. Indeed, Equation (4) induces that these spatial coordinates, hence the explicit resolution of PDEs on or U, can be ignored, as it amounts to learning ø, vp and such that:
(C o U o (Ø(-), (t)) = D (t))).
(5)
In order to manipulate functionals é and sb practice, we respectively introduce learnable time- invariant and time-dependent representations of and v/, denoted by S and T, such that:
= S S C Rd
= T. E C IR",
(6)
where the dependence of = on time will be modeled using a temporal ODE following the separation of variables, and the function ø, and consequently its spatial PDE, are encoded into vectorial representation S. Besides their separation of variables basis, the purpose of and T is to capture spatial and motion information of the data. For instance, S could encode static information such as objects appearance, while typically contains motion variables.
S and T because of their dependence on in Equations (5) and (6), are inferred from observation history, or conditioning frames, V, (tu), where Vt( = (v,, U2+ Ar.... (e+rrt), using respectively encoder networks Es and Ey. We parameterize D of Equation (5) as neural network that acts both S and T,, and outputs the estimated observation &, D(S, T.). Unless specified otherwise, S and T, fed concatenated into D, which then learns the parameterization of their combination.
4

---

Published as a conference paper at ICLR 2021
4.3 TEMPORAL ODE
The separation of variables allows us to partly reduce the complex task of learning and integrating PDEs to learning and integrating ODE dt, which has been extensively studied in the literature, as explained in Section 2. We therefore model the evolution of Tú, thereby the dynamics of our system, with first-order ODE:
OT. 00
T. T [ s(Te)
S(T.)
Note that the first-order ODE assumption can be taken without loss of generality since any ODE is equivalent to higher-dimensional first-order ODE. Following Chen et al. (2018), f is implemented by neural network and Equation (7) is solved with an ODE resolution scheme. Suppose initial ODE conditions 8 and Tio have been computed with Es and ËT. This leads to the following simple forecasting scheme, enforced by the corresponding regression loss:
-D(S.TT. [( f(Te) dt')
M ==
0. D S.
Ppeed
1 Utotieell
(8)
where v + is the number of observations, and is the dimension of the observed variables v.
Equation (8) ensures that the evolution of T coherent with the observations; we should enforce its consistency with Er. Indeed, the dynamics of T; is modeled by Equation (7), while only its initial condition Th, is computed with Er. However, there is no guaranty that T,, computed via integration, matches Er Vr (t)) at any other time t, while they should principle coincide. We introduce the following autoencoding constraint mitigating their divergence, thereby stabilizing the evolution of T::
LAE
m Es (Vo +itt)) 1eotise
1
with i v ~(10, rI).
(9)
4.4 SPATIOTEMPORAL DISENTANGLEMENT
As indicated hereinabove, the spatial PDE on ó is assumed to be encoded into S. Nonetheless, since inferred from an observation history, we need explicitly enforce its time independence. In the PDE formalism, this is equivalent to:
i Es (t) = O
dE (1) dt = 0.
(10)
at
Ot
However, enforcing Equation (10) raises crucial issues. Firstly, in our partially observed setting, there can be variations of observable content, for instance when an object conceals another one. Therefore, strictly enforcing null time derivative not desirable it prevents Es to extract accessible information that could be obfuscated in the sequence. Secondly, estimating this derivative in practice in our setting unfeasible and costly because of the coarse temporal discretization of the data and the computational cost of Es; see Appendix B for more details. We instead introduce discretized penalty in our minimization objective, discouraging variations of content between two distant time steps, with d being the dimension of S:
1 Es (v(((()o)) Es -A))))))))))))) -VV)(((-=A)))
(11)
allows us to overcome the previously stated issues by not enforcing a strict invariance S and removing the need to estimate any time derivative. Note that this formulation actually originates from Equation (10) using the Cauchy-Setie -Shhwaz inequality (see Appendix B for a more general derivation).
Abstracting the spatial ODE on & from Equation (4) into generic representation S leads, without additional constraints, to an underconstrained problem where spatiotemporal disentanglement cannot be guaranteed. Indeed, Es can be set to zero satisfy Equation (11) without breaking any prior constraint, because static information is not prevented to be encoded into T. Accordingly, information in S and T needs to be segmented.
Thanks to the design of our model, it suffices to ensure that S and T are disentangled at initial time to for them be to be disentangled at all f. Indeed, the mutual information between two variables
5

---

Published as a conference paper at ICLR 2021
is preserved by invertible transformations. Equation (7) is ODE and as neural network, is Lipschitz-contimuuus so the ODE flow T, ++ is invertible. Therefore, disentanglement between S and T, characterized by low mutual information between both variables, is preserved through time; see Appendix C for a detailed discussion. We thus only constrain the information quantity in Tt by using a Gaussian prior encourage it to exclusively contain necessary dynamic information:
- T 112 = / (v-((t)))
(12)
p
4.5 LOSS FUNCTION
The minimized loss linear combination of Equations (8), (9), (11) and (12):
((v) Enpp Spredd AAE LAR X L Xrog Fing
(13)
as illustrated in Figure 1. In the following, we conventionally set At 1. Note that the presented approach could generalized to irregularly sampled observation times thanks to the dedicated literature (Rubanova et al., 2019), but this is out of the scope of this paper.
5 EXPERIMENTS
We study in this section the experimental results of our model on various spatiotemporal phenomena with phys- ical, synthetic video and real-world datasets, which are briefly presented in this section and more details in Appendix D. We demonstrate the relevance of our model with ablation studies and its performance by comparing with more complex state-of-the-art models. Perfor- mances are assessed thanks to standard metrics (Den- & Fergus, 2018; Le Guen & Thome, 2020) Mean Squared Error (MSE, lower is better) or its alternative Peak Signal-to-Noise Ratio (PSNR, higher better), and Structured Similarity (SSIM, higher is better). We refer to Appendix for more experiments and prediction ex-
t=0
t=4
t=6
t=8
Input frames
Truth
PKnl
PhyDNet
MIM
amples, to Appendix E for training information and to
supplementary material for the corresponding code! and datasets.
Ours
Content input
5.1 PHYSICAL DATASETS:
Swap
WAVE EQUATION AND SEA SURFACE TEMPERATURE
We first investigate two synthetic dynamical systems and real-world dataset in order to show the advantage of PDE-driven spatiotemporal disentanglement for forecast- ing physical phenomena. To analyze our model, we first lean on the wave equation, occurring for example acoustic or electromagnetism. with source term like Saha et al. (2020), to produce the WaveEq dataset consisting in 64 x 64 normalized images of the phenomenon. We additionally build the WaveEq-100 100 dataset by extracting 100 pixels, chosen uniformly at random and shared among sequences, from WavelEq frames; this experimental setting be thought of as measurements from sensors partially observing the phenomenon. We also test and compare our model on the real-world dataset SST, derived from the data assimilation engine NEMO (Madec & Team) and introduced by de Bézenac et al. (2018), consisting in 64 x 64 frames showing the evolution of the surface temperature. Modeling its evolution is particularly challenging as its dynamic is highly non-linear, chaotic, and involves several unobserved quantities (e.g., forcing terms).
Figure 2: Example of predictions of compared models on SST. Content swap preserves the location of extreme temper- ature regions which determine the move- ment while modifying the magnitude of all regions, especially in temperate areas.
Our source code is also publicly released at the following URI:: https: /tte:///gtthtt com/ Jerembona/ spat i emporal a 1_variar aabiaablee sepp

---

Published as a conference paper at ICLR 2021
Table 1:
Forecasting performance WaveFq-1000 Waveliq and SST of compared models with respect to indicated prediction horizons. Bold scores indicate the best performing method.
WaveFiq 100
WaveFq
SST
Models
MSE
MSE
SSIM
++ 40
+
40
1 6
+ 10
t
+ 10
PKnl
1.28
2.03
0.6686
0.5844
PhyDNet
1.27
1.91
0.5782
0.4645
SVG
1.51
2.06
0.6259
0.5595
MIM
0.91
1.45
0.7406
0.6525
Ours
4.33
10-5
1.44
10
0.86
1.43
0.7466
0.6577
Ours (without S)
1.33 x 10
5.09
10
0.95
1.50
0.7204
0.6446
Table 2:
Prediction and content swap scores of all compared models on Moving MNIST. Bold scores indicate the best performing method.
Pred. (L + 10)
Pred. (t + 95)
Swap 10)
Swap (1 + 95)
Models
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SVG
18.18
0.8329
12.85
0.6185
MIM
24.16
0.9113
16.50
0.6529
DrNet
14.94
0,6596
12.91
0.5379
14.12
0,6206
12.80
0.5306
DDPAE
21.17
0.8814
13.56
0.6446
18.44
0.8256
13.25
0.6378
PhyDNet
23.12
0.9128
16,46
0.3878
12.04
0.5572
13.49
0.2839
Ours
21.70
0.9088
17.50
0.7990
18.42
0.8368
16.50
0.7713
We compare our model on these three datasets to alternative version with S removed and integrated into T, thus also removing Ligg and LT We also include the state-of-the-art PhyDNet (Le Guen & Thome, 2020), MIM (Wang et al., 2019b), SVG (Denton Fergus, 2018) and SST-specific PKnl (de Bézenac et al., 2018) in the comparison SST; only PhyDNet and PKnl were originally tested on this dataset by their authors. Results are compiled Table and an example of prediction is depicted in Figure 2.
reg-
On these three datasets, our model produces more accurate long-term predictions with S than without it. This indicates that learning invariant component facilitates training and improves generalization. The influence of S can be observed by replacing the of sequence by another one extracted from another sequence, changing the aspect of the prediction, as shown in Figure (swap row). We provide in Appendix F further samples showing the influence of S in the prediction. Even though there is no evidence of intrinsic separability component. Indeed, our model outperforms PKnl despite the data-specific structure of the latter, the stochastic SVG and the high-cape saghccapacities PhyDNet and MIM model, whereas removing its static component suppresses our advantage.
SST, our trained algorithm takes advantage of its time-invarian
We highlight that MIM way 64 times larger latent states than ours, hence its better reconstruction ability at the first time step. However, its sharpness and movement gradually vanish, explaining its lower performance than ours. We refer Appendix F.3 for additional discussion on the application of our method and its performance on SST.
computationalll--eavy model that manipulates in an autoregressive
5.2 A SYNTHETIC VIDEO DATASET: MOVING MNIST
We also assess the prediction and disentanglement performance of model on the Moving MNIST dataset (Srivastava et al., 2015) involving MNIST digits (LeCun et al., 1998) bouncing over frame borders. This dataset particularly challenging in the literature for long-term prediction tasks. We
7

---

Published as a conference paper at ICLR 2021
T-4
t=6
t-10
t-14
t-24
tw39
Dynamic input
Inpur fraves
8
5
5
5
5
5
MIM
8
5
8
$
4
5
5
5
5
5
5
5
Content input
DOPAE
5
5
5
5
8
5
5
PhyoNes
8
5
5
$
Ours
8
5
8
5
5
5
85
5
Content
Swep
2
& 7
V
%
7
&
Figure 4: Fusion of content (first column) and dynamic (first row) variables in our model on 3D Warehouse Chairs
Figure 3: Predictions of compared models on Moving
MNIST, and content swap experiment for our model.
compare our model to competitive baselines: the non-disentangled SVG (Denton & Fergus, 2018) and MIM (Wang et al., 2019b), as well as forecasting models with spatiotemporal disentanglement ablities DrNet (Denton & Birodkar, 2017), DDPAE (Hsieh et al., 2018) and PhyDNet. We highlight that all these models leverage powerful machine learning tools such as adversarial losses, VAEs and high-capacity temporal architectures, whereas ours is solely trained using regression penalties and small-size latent representations. We perform as well a full ablation study of our model to confirm the relevance of the introduced method.
Results reported in Table 2 and illustrated in Figure 3 correspond to two tasks: prediction and disentanglement, at both short and long-term horizons. Disentanglement evaluated via content swapping, which consists in replacing the content representation of sequence by the one of another sequence. which should result for a perfectly disentangled model in swapping digits of both sequences. This is done by taking advantage of the synthetic nature of this dataser that allows us to implement the ground truth content swap and compare it to the generated swaps of the model.
Reported results show the advantage of our model against all baselines. Long-term prediction challenges them as their performance and predictions collapse in the long run. This shows that the baselines, including high-capacity models MIM and PhyDNet that leverage powerful LTnVLTSMs (Shi et al., 2015), have difficulties separating content and motion. Indeed, a model separating correctly content and motion should maintain digits appearance like DDPAE which alters only marginally the digits in Figure 3. In contrast, ours manages to produce consistent samples even at 95, making it reach state-of-the-ar performance. Moreover, significantly outperform all baselines in the content swap experiment, showing the clear ad of the proposed PDE-insppred simple model for spatiotemporally disentangled prediction.
when it miscalculates their trajectories,
Ablation studies developed in Table confirm that this advantage is due to the constraints motivated by the separation of variables, Indeed, the model without S fails at long-term forecasting, and removing any non -onedrecittion penalty of the training loss substantially harms performances. In particular, the invariance loss on the static component and the regularization of initial condition Te, are essential, as their absence hinders both prediction and disentanglement The auto-encoding constraint makes predictions more stable, allowing accurate long-term forecasting and disentanglement. This ablation study also confirms the necessity to constrain the & norm of the dynamic variable (see Equation (12)) for the model to disentangle. Comparisons of Table actually show that enforcing this loss on the first time step only is sufficient to ensure ofte--of-the-ar disentanglement, advocated in Section 4.4.
Finally, we assess whether the temporal ODE of Equation (7) induced by the separation of variables is advantageous by replacing the dynamic model with standard GRU RNN (Cho et al., 2014). Results reported Table show substantially better prediction and disentanglement performance for the original model grounded the separation of variables, indicating the relevance of our approach.
8

---

Published as a conference paper at ICLR 2021
Table 3: Prediction MSE (x100 32 32 2) of compared models TaxiBJ, with best MSE highlighted in bold.
Ours 39.5
Ours (without S)
PhyDNet
MIM
E3D
C. LSTM
PredRNN
ConvISTM
43.7
41.9
42.9
43.2
44.8
46.4
48.5
5.3 A MULTI-VIEL DATASET: 3D WAREHOUSE CHAIRS
We perform an additional disentanglement experiment on the 3D Warehouse Chairs dataset introduced by Aubry et al. (2014). This dataset contains 1393 three-dimensional models of chairs seen under various angles. Since all chairs are observed from the same set of angles, this constitutes a multi-view dataset enabling quantitative disentanglement experiments. We create sequences from this dataset for our model by assembling adjacent views of each chair to simulate its rotation from right to left. We then evaluate the disentanglement properties of our model with the same content swap experiments as for Moving MNIST. It is similar to one of Denton & Birodkar (2017)'s experiments who qualitatively tested their model on similar, but smaller, multi-view chairs dataset. We achieve 18. 70 PSNR and 0.7746 SSIM on this task, outperforming DrNet which only reaches 16.35 PSNR and 0.6992 SSIM. This is corroborated by qualitative experiments in Figures and 11. We highlight that the encoder and decoder architectures of both competitors are identical, validating our PDE-grounded framework for spatiotemporal disentanglement of complex three-dimensional shapes.
5.4 A CROWD FLOW DATASET: TAXIBJ
We finally study the performance our spatiotemporal model on the real-world TaxiBJ dataset (Zhang et al., 2017), consisting in taxi traffic flow in Beijing moni- tored on 32 x 32 grid with an observation every thirty minutes. It is highly structured as the flows are de- pendent on the infrastructures of the city, and complex since methods have to account for non-local dependen- cies and model subtle changes in the evolution of the flows. It is standard benchmark in the spatiotemporal prediction community (Wang et al., 2019b; Le Guen & Thome, 2020).
t-e
t-4
t-5
t-6
t-7
Input
franes
Truth
Diff.
Pred.
Figure Example of ground truth and pre- diction of our model on TaxiBJ. The middle row shows the scaled difference between our predictions and the ground truth.
We compare our model in Table against PhyDNet and MIM, as well as powerful baselines E3D-LSTM (E3D, Wang et al., 2019a), Causal LSTM (C. LSTM, Wang et al., 201 8), PredRNN (Wang et al., 2017) and ConVLTSM (Shi et al., 2015), using results reported by Wang et al. (2019b) and Le Guen & Thome (2020), An example of prediction given in Figure 5. We observe that significantly overcome the state of the art on this complex spatiotemporal dataset. This improvement is notably driven by the disentanglement abilities of our model, model without S achieves results comparable to E3D and worse than PhyDNet and MIM.
observe in Table that the alternative version of our
CONCLUSION
We introduce novel method for spatiotemporal prediction inspired by the separation of variables PDE resolution technique that induces time invariance and regression penalties only. These constraints ensure the separation of spatial and temporal information. We experimentally demonstrate the benefits of the proposed model, which outperforms prior state-of-the-art methods on physical and synthetic video datasets. We believe that this work, by providing dynamical interpretation of spatiotemporal disentanglementt could serve as the basis of more complex models further leveraging the PDE formalism. Another direction for future work could be extending the model with more involved tools such as VAEs to improve its performance, or adapt it to the prediction of natural stochastic videos (Denton & Fergus, 2018).

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like thank all members of the MLIA team from the LIP6 laboratory of Sorbonne Université for helpful discussions and comments, as well as Vincent Le Guen for his help to reproduce PhyDNet results and process the TaxiBJ dataset.
We acknowledge financial support from the LOCUST ANR project (ANR-15 15 I-C-EE-022)07 and the European Union's s Horizon 2020 research and innovation programme under grant agreement 825619 (AI4EU). This study has been conducted using E.U. Copernicus Marine Service Information. This work was granted access to the HPC resources of IDRIS under allocations 2020-AD01 101 1360 and 2021-AD01 101 1360RI made by GENCI (Grand Equipement National de Calcul Intensif), Patrick Gallinari is additionally funded by the 2019 ANR Al Chairs program via the DLACLIM project.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50)1-34, 2018.
Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell, and Josef Sivic. Sceing 3D chairs: Exemplar part-based 2D-3D alignment using a large dataset CAD models. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3762-3769, June 2014.
Ibrahim Ayed, Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Learning the spatio- temporal dynamics of physical processes from partial observations. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3232-3236, 2020.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and förn-Henrik Jacobsen. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 573-582, Long Beach, California, USA, June 2019. PMLR.
Sergio Benenti. Intrinsic characterization of the variable separation in the Hamilton-Jacoti equation. Journal of Mathematical Physics, 38(12):6577 66008, 1997.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(8):1798-188 1828,
August 2013.
Steven L. Brunton, Joshua L. Proctor, and Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932-399) 2016.
Hans-Joachim Bungartz and Michael Griebel. Sparse grids. Acta Numerica, 13:147-269, 2004.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 31, pp. 6571 65533. Curran Associates, Inc., 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN:
Interpretable representation learning by information maximizing generative adversarial nets. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2172-2180. Curran Associates, Inc., 2016.
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Léon Bottou. Symplectic recurrent neural networks. In International Conference on Learning Representations, 2020.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in
10

---

Published as a conference paper at ICLR 2021
Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, October 2014. Association for Computational Linguistics.
Filipe de Avila Belbute- Peres, Kevin A. Smith, Kelsey R. Allen, Joshua B. Tenenbaum, and Zico Kolter. End-to-end differentiable physics for learning and control. In Samy Bengio, Hanna Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Aasa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7178-77 7189. Curran Associates, Inc., 2018.
Emmanuel de Bézenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. In International Conference on Learning Representations, 2018.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017.
pp. 4414-4423,
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1174-1183, Stockholmsmässan, Stockholm, Sweden, July 2018. PMLR.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. FlowNet: Learning optical flow with convolutional networks. In The IEEE International Conference Computer Vision (ICCV), pp.
7758-2766, December 2015.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 64-72. Curran Associates, Inc., 2016.
Jean Baptiste Joseph Fourier. Théorie analytique de la chaleur. Didot, Firmin, 1822.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3601-3610 Curran Associates, Inc., 2017.
Jean-Y Yves Franceschi, Edouard Delasalles. Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1233-3245 PMLR, July 2020.
Ian Goodfellow, Jean Pouget-Abiis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances Neural Information Processing Systems 27, pp. (672-2680. Curran Associates, Inc., 2014.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 15379-15389. Curran Associates, Inc., 2019.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34 (1):014004, December 2017.
Ernst Hairer, Syvert P. Nørsett, and Gerhard Wanner. Solving Ordinary Differential Equations I: Nonstiff Problems, chapter Kunge-Kutta and Extrapolation Methods, pp. 129-353. Springer Berlin Heidelberg, Berlin, Heidelberg, 1993.
William Rowan Hamilton. Second essay on a general method in dynamics. Philosophical Transactions of the Royal Society, 125:95-144, 1835.
11

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016.
Berthold K. P. Horn and Brian G. Schunck. Determining optical flow. Artificial Intelligence, 17(1-3): 185-203, August 1981.
Jun-Ting Hsich, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In Samy Bengio, Hanna Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 517-526. Curran Associates, Inc., 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1878-1889. Curran Associates, Inc., 2017.
Miguel Jaques, Michael Burke, and Timothy Hospedales. Physics-as-inverss--raaaa Unsupervised physical parameter estimation from video. In International Conference on Learning Representa- tions, 2020.
Huabing Jia, Wei Xu, Xiaoshan Zhao, and Zhanguo Li. Separation variables and exact solu- tions to nonlinear diffusion equations with z z-dependent convection and absorption. Journal of Mathematical Analysis and Applications, 339(2):982 995, March 2008.
E. G. Kalnins, Willard Miller, Jr., and G. C. Williams. Recent advances in the use of separation of variables methods general relativity. Philosophical Transactions: Physical Sciences and Engineering, 340(16588:337-35,, 1992.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.
Adam R. Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Samy Bengio, Hanna Wallach, Hugo Larochelle, Kristen Grauman, Nicolò desa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 3606-8616 Curran Associates, Inc., 2018.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E. 69:066138, June 2004.
Martin Wilhelm Kutta. Beitrag zur näherungweisen Integration totaler Differentialgleichungen Zeitschrift für Mathematik und Physik,
1901.
Hervé Le Dret and Brigitte Lucquin. Purtial Differential Equations: Modeling, Analysis and Numerical Approximation, chapter The Heat Equation, pp. 219-251. Springer International Publishing, Cham, 2016.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11474-11484, June 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11 (0(1)):227823244 November 1998.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbecl, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 1 2018.
Xuechen Li. Ting-Kan Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. arXiv preprint arXiv:2001.01 01828, 2020.
12

---

Published as a conference paper at ICLR 2021
Zhijian Liu, Jiajun Wu, Zhenjia Xu, Chen Sun, Kevin Murphy, William T. Freeman, and Joshua B. Tenenbaum. Modeling parts, structure, and system dynamics via predictive learning. In Interna- tional Conference on Learning Representations, 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 41 4124,4124, Long Beach, California, USA, June 2019. PMLR.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-Net: Learning PDEs from data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2216-3216. Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
Zichao Long, Yiping Lu, and Bi Dong. PDE-Net 2.0: Learning PDEs from data with numeric- symbolic hybrid deep network. Journal of Computational Physics, 399: 108925, 2019.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3276-3266 -285,, Stockholmsmässaan, Stockholm Sweden, July 2018. 8. PMLR.
Gurvan Madec and NEMO System Team. NEMO ocean engine. Technical Report 27, Scientific Notes of Climate Modelling Center, Institut Pierre-Simon Laplace (IPSL). Zenodo.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In International Conference on Learning Representations, 2016.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference Learning Representations, 2018.
Willard Miller,
The technique of variable separation for partial differential equations. In Kurt Bernardo Wolf (ed.), Nonlinear Phenomena, pp. 184-208, Berlin, Heidelberg, 1983. Springer Berlin Heidelberg.
Willard Miller, Jr. Mechanisms for variable separation in partial differential equations and their relationship group theory. In Decio Levi and Pavel Winternitz (eds.), Symmetries and Nonlinear Phenomena: Proceedings of the International School on Applied Mathematics, pp. 188-221, Singapore, 1988. World Scientific.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 92-102. Curran Associates, Inc., 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, nigh-performance deep learning library. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 0026-8037 Curran Associates, Inc., 2019.
Andrei D. Polyanin. Functional separable solutions of nonlinear convection-diffusin equations with variable coefficients. Communications in Nonlinear Science and Numerical Simulation, 73: 379-390, July 2019.
Andrei D. Polyanin. Functional separation of variables in nonlinear PDEs: General approach, new solutions of diffusion-type equations. Mathematics, 8(1):90, 2020.
13

---

Published as a conference paper at ICLR 2021
Andrei D. Polyanin and Alexei I. Zhurov. Separation of variables in PDEs using nonlinear transfor- mations: Applications to reaction-diffusion diffusion type equations. Applied Mathematics Letters, 100: 106055, February 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Learning Represen- tations, 2016.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. Journal of Machine Learning Research, 19(25): 2018
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481):1026-10000 1030, 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara (eds.), Pro- ceedings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Machine Learning Research, pp. 1286, Bejing, China, June 2014. PMLR.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Cosputer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing.
Yulia Rubanova, Ricky T. Q Chen, and David Duvenaud. Latent ordinary differential equations for irreuulaly-sampled time series. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ahh Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 5320-5330. Curran Associates, Inc., 2019.
Tom Ryder, Andrew Golightly, A. Stephen McGough, and Dennis Prangle. Black- variational inference for stochastic differential equations. In Jennifer Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4423 4432, Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
Priyabrata Saha, Saurabh Dash, and Saibal Mukhopadhyay, PhICnet: Plysics-incorporrteed convolu- tional recurrent neural networks for modeling dynamical systems. arXiv preprint 2rX1v:2004.06243, 2020.
Christian Schüldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: A local SVM approach. In Proceedings of the 17th International Conference on Puttern Recognition, 2004. ICPR 2004., volume 3. pp. 32-36, August 2004.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 802-810. Curran Associates, Inc., 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339 11541339-136,, 1364, 2018.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using LSTMs. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 843-853 852, Lille, France, July 2015. PMLR.
Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating Eulerian fluid simulation with convolutional networks. In Doina Precup and Yee Whye Teh (eds.), Pro- ceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3424-3433. International Convention Centre, Sydney, Australia, August 2017. PMLR.
14

---

Published as a conference paper at ICLR 2021
Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Hig- gins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 1526-1535, June 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:/8/2.017/ 2018.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. In International Conference on Learning Representations, 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference Learning Representations, 2017a.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lec. Learning to generate long-term future via hierarchical prediction. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3560-3569, International Convention Centre, Sydney, Australia, August 2017b. PMLR.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lec. High fidelity video prediction with large stochastic recurrent neural networks. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 81-91. Curran Associates, Inc., 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 613-621. Curran Associates, Inc., 2016.
Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and Philip S Yu. PredRNN: Recurrent neural networks for predictive learning using spatiotemporal LSTMs. Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 879-888. Curran Associates, Inc., 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. PredRNN++ Towards a resolution of the deep-in- in- time dilemma in spatiotemporal predictive learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5123-5132, Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3D LSTM: A model for video prediction and beyond. In International Conference on Learning Representations, 2019a.
Yunbo Wang, Jianjin Zhang, Hongyu Zhu, Mingsheng Long, Jianmin Wang, and Philip S. Yu. Memory memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models, In International Conference on Learning Representations, 2020.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5670-5679, Stockholmsmässsan, Stockholm Sweden, July 2018. PMLR
15

---

Published as a conference paper at ICLR 2021
Cagatay Yıldiz, Markus Heinonen, and Harri Lahdesmaki. ODE Deep generative second order odes with Bayesian neural networks. In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 14412-13421 Curran Associates, Inc., 2019.
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In Proceedings of the Thirty-Fine AAAI Conference Artificial Intelligence, 1A pp. 1655-1661. AAAI Press, 2017.
A PROOFS
A.1
RESOLUTION OF THE HEAT EQUATION
In this section, we succinetly detail proof for the existence and uniqueness for the solution to the two-dimensional heat equation. shows that product solutions allow build the entire solution space for this problem, highlighting our interest in the research of separable solutions.
Existence through separation of variables. Consider the heat equation problem:
uu
82m u
2
u(0, t) = t( = 0,
u(zz 0) = f(x).
(14)
ŌL
dr
Assuming product separability of u with u(æ, t) = 4n (z) in Equation (14) gives:
(x)
(t)
(15)
ui (z) (1)
Both sides being independent each other variables, they are equal to a constant denoted by -a. If a negative, solving the right side of Equation (15) results to non-phn son-physical solutions with exponentially increasing temperatures, and imposing border condition of Equation (14) makes this solution collapse to the null trivial solution. Therefore, we consider that a O.
Both sides of Equation (15) being equal to a constant leads to a second-order ODE on on and a first-order ODE on u2, giving the solution shapes, with constants A, B and D:
(x) A cos( (aax) + B sin (Vax)
(16)
Ug
-e=a
Link with initial and boundary conditions. We now link the above equation to the boundary conditions of the problem. Because our separation is multiplicative, we can omit D for non-trin non-trivial solutions and set it without loss of generality to as it only scales the values of A and B.
Boundary condition w(0,t) t) = u(L,t) t) = O, along with the fact that for all t > 0, u2(() * give:
=
Ba sin (VoL)
(17)
which means that, for a non-trivial solution (i.e., B * 0), we have for some n E N: va= L.. We finally express product-separable solution the heat equation without initial conditions
TUT L
u(z, t) = B sin
exp
(18)
Considering the superposition principle, because the initial problem is homogeneous, all linear combinations Equation (18) are solutions of the heat equation without initial conditions. Therefore, any following function is solution of the heat equation without initial conditions.
u(x,t) Σ ==
B sin
7:T
exp
CN. L
(19)
Finally, considering the initial condition u(x 0) fxx, Fourier decomposition of allows to choose appropriate values for all coefficients B,, showing that, for any initial condition f. there exists solution to Equation (14) of the form of Equation (19).
16

---

Published as a conference paper at ICLR 2021
We present here elements of proof for establishing the uniqueness of the solutions of Equation (14) that belong to C² (10, 1] x R+). Detailed and rigorous proofs are given by Le Dret & Luequin (2016).
Uniqueness.
The key element consists in establishing the so-called Maximum Principle which states that, consid- ering sufficiently smooth solution, the minimum value of the solution is reached on the boundary of the space and time domains.
For null border condition in our case, this means that the norm of the solution is given by the norm of the initial condition f. Finally, let us consider two smooth solutions Un and U2 of Equation (14). Then, their difference
U1 U2 follows the heat equation with null border and initial conditions (i.e, v(z, 0) 0). Because v is as regular as U1 and U2 it satisfies the previous fact about the norm of the solutions, i.e, the norm of equals the norm of its initial condition: ||o/|| Therefore, is null and so is U1 U2 O, showing the uniqueness of the solutions.
Therefore, this shows that solutions of the form of Equation (19) shape the whole set of smooth solutions of Equation (14).
A.2 HEAT EQUATION WITH ADVECTION TERM
Consider the heat equation with a complementary advection term, for x E (-,, t E (0, and a constant R.+.
Ou
Ou
8au Ox
(20)
Øt
dx
We give here details for the existence of preduct-separable solutions of Equation (20). To this end, let us choose real constants a and 3, and consider the following change of variables for w:
u(z, t) = v(x =
(21)
The partial derivatives from Equation (20) can be rewritten as functions of the new variable v:
du dv
e
az+
+ a0
(22)
Ot ot
Du =
dv
ou+ft + a+
(23)
dr Or
Su. 8Lu
cztBt + 2a Ov dr
ax4St a 00
(24)
dx
Ox²
Using these expressions in Equation (20) and dividing it by t lead to:
dv +
du 8v -²xx + (c (2-2xx) dr dr²
(25)
Øt
a and 3 can then be set such that:
8 + ca x=
c 2ax = 0,
(26)
to retrieve the standard two-dimensional heat equation of Equation (14) given by:
Ov 8. at Ox2*
(27)
which is known to have product-separaae solutions as explained in the previous section. This more generally shows that all solutions Equation (20) can be retrieved from solutions to Equation (14).
B ACCESSING TIME DERIVATIVES OF S AND DERIVING A FEASIBLE WEAKER CONSTRAINT
Explicitly constraining the time derivative of Es Es (V)()) explained in Section 4.4 is difficult matter in practice. Indeed, Es does not take as input neither the time coordinate nor spatial coordinates and y done by Raissi (2018) and Sirignano & Spiliopoulos (2018), which allows
17

---

Published as a conference paper at ICLR 2021
them to directly estimate the networks derivative thanks to automatic differentiation. In our case, Es rather takes as input a finite number of observations, making this derivative impractical to compute.
To discretize Equation (10) and find weaker constraint, we chose to leverage the SeuuhyySchwarz incquality. We presented and used a version where we applied this incquality on the whole integration domain, i.e., from to to 1 -tt. We highlight that this inequality can also be applied on subintervals of the integration domain, generalizing our proposition. Indeed, let p E No and consider a sequence
of {(k) for & E [0. such that to = x()) S {(1) ... I
{(r) = t1 - -At. Then, using the
Cauchy-Schwand inequality, obtain:
h1 h1-ra
(V.
2
k=p
(())
oEs (V)
2
dt
dt
to
0t
k=0
((--1)
ōt
2
kmp
1
((k)
FE (V)
dt
{() ((k-1)
e(k-1)
Öt
Es V4 (i(((tt)
Es V (f((-11)
{()) {(k 1)
k=
(28)
Our constraint is special case of this development, with p 1. Nevertheless, we experimentally found that our simple penalty is sufficient to achieve state-of-the-ar performance at substantially reduced computational cost. We notice that other invariance constraints such as the of Denton & Birodkar (2017) can also be derived thanks to framework, showing the generality of our approach.
C
OF SPATIOTEMPORAL DISENTANGLEMENT
C. MODELING SPATIOTEMPORM PHENOMENA WITH DIFFERENTIAL EQUATIONS
Besides their increasing popularity to model spatiotemporal phenomena (see Section 2), the ability of residual networks to facilitate learning (Haber & Ruthotto, 2017) as well as the success of their continuous counterpart (Chen et al., 2018) motivate our choice. Indeed, learning ODEs or discrete approximations as residual networks has become standard for a variety of tasks such as classification, inpainting, and generative models. Consequently, their application to forecasting physical processes and videos is only a natural extension of its already broad applicability discussed in Section Furthermore, they present interesting properties, detailed below.
C.2 SEPARATION OF VARIABLES PRESERVES THE MUTUAL INFORMATION OF S AND T THROUGH TIME
C.2.1 INVERTIBLE FLOW OF AN ODE
We first highlight that the general ODE Equation (7) admits, according to the Cauchy-Lipschizz theorem, exactly one solution for given initial condition, since f is implemented with standard neural network (see Appendix E), making Lipschitz-continuunl Consequently, the flow of this ODE, denoted by Φ, and defined as:
Φ: x RP R
R
(4o, (T) =
a bijection for all 1. Indeed, let Tr, fixed and lg. l be two timesteps; thanks to the existence and unicity of the solution to the ODE with this initial condition: 2to-6 L 2r, 2r, 2 Therefore, 4, is bijection and Φ; $,.. Moreover, the flow is differentiable if f is continuously differentiable as well, which is not restrictive assumption if it implemented by a neural network with differentiable activation functions.
C.2.2 PRESERVATION OF MUTCAL INFORMATION BY INVERTIBLE MAPPINGS
A proof of the following result is given by Kraskov et al. (2004). We indicate below the major steps of the proof. Let X and Y be two random variables with marginal densities μxx py. Let F be
18

---

Published as a conference paper at ICLR 2021
diffeomorphism acting on Y. Y FY)). If JF is the determinant of the Jacobian of F, we have:
(z( 3¹) u(a,y)Jr(()
Then, expressing the mutual information in integral form, with the change of variables y' = F(y) (F being dificomorphism), results in:
((X,Y') = JJ 1 (x,11/) log
pl (a,v') dr
1xx (2) (yr)(1)) p(z, y)
11 p(a,y) log (x x (y
do dy
((x = I(X,Y).
C.3 ENSURING DISENTANGLEMENT AT ANY TIME
As noted by Chen et al. (2016) and Achille & Soatto (2018), mutual information is a key metric to evaluate disentanglement. We show that our model logically preserves the mutual information between S and T through time thanks the flow of the learned ODE T. Indeed, with the result of mutual information preservation by diffeomorphisms, and o, being diffeomorphism as demonstrated above, we have, for all t and t':
T(S, T2) [X,, (T.) I(S, Tx))
(29)
Hence, if S and T, are disentangled, then so are S and Te.
The flow Φ being dicretized in practice, its invertibility can no longer be guaranteed in general. Some numerical schemes (Chen et al., 2020) or residual networks with Lipschitz-constrainee residual blocks (Behrmann et al., 2019) provide sufficient conditions to concretely reach this invertibility. In our case, we did not observe the need to enforce invertibility. We can also leverage the data processing inequality show that, for any t to:
I(,, I(S,T,),
(30)
since Tr is deterministic function of Try Since we constrain the very first T value Tr, (i.e., do not need to go back in time), there is no imperative need to enforce the invertibility of 4, in practice: the inequality also implies that, if S and Tí, are disentangled. then so are S and T, for t 2 to. Nevertheless, should the need to disentangle for t < to appear, the aforementioned mutual information conservation properties could allow, with further practical work to ensure the effective invertibility of tb, to still regularize Tú, only. This is, however, out of the scope of this paper.
D DATASETS
D.1 WAVEEQ AND WAVEEQ-100
These datasets are based on the two-dimensional wave equation on a functional w(z y, f)t)
8w
01 = 2222 + f(x 1)
(31)
where 2 is the Laplacian operator, denotes the wave celerity, and is an arbitrary time-dependent source term. It has several application in physics, modeling a wide range of phenomena ranging from mechanical oscillations to electromagnetism. Note that the homogeneous equation, where f = admits product-separable solutions.
We build the WaveEq dataset by solving Equation (31) for t [0, 0.299998 and x, y E [0, 63). Sequences generated using drawn uniformly at random in [300, 400] for each sequence to imitate the propagation of acoustic waves, with initial and Neumann boundary conditions:
w(I, y. 0) = w0, 0, t) = (22,322, = 0,
(32)
and, following Saha et al. (2020), we make use of the following source term:
f(x, v, = foc :o if (α,z) E B (33,,
(33)
otherwise
19

---

Published as a conference paper at ICLR 2021
with To 0.05 and fo ~ ((11,,0) The source term is taken non-null in circular central zone only in order to avoid numerical differentiation problems in the case of a punctual source.
We generate 300 sequences of 64 64 frames of length 150 from this setting by assimilating pixel (i.j) E [00 63] x [o. 63] to a point (z, y) E [0, 63) x [0, 63) and selecting a frame per time interval of size 0.002. This discretization is used solve Equation (31) its spatial derivatives thanks to finite differences; once computed, they are used in an ODE numerical solver to solve Equation (31) l. Spatial derivatives are estimated with finite differences of order and the ODE solver is the fourth-order Runge-Kute method with the 3/8 rule (Kutta, 1901; Hairer et al.. 1993) and step size 0.001. The data are finally normalized following a min-max [0. 1] scaling per sequence.
estimated
The dataset is then split into training (240 sequences) and testing (60 sequences) sets. Sequences sampled during training are random chuncks of length + 1 = 25, including 1 = 5 conditioning frames, of full-size training sequences. Sequences used during testing are all possible chunks of length + 1 + 40 = 45 from full-size testing sequences.
Finally, WaveEq- 100 is created from WaveEq by selecting 100 pixels uniformly at random. The extracted pixels are selected before training and are fixed for both training and testing. Therefore, train and test sequences for Waveliq- 100 consist of vector of size 100 extracted from WaveF2q frames. Training and testing sequences are chosen to be the same as those of WaveFq.
D.2 SEA SURFACE TEMPERATURE
SST is composed of sea surface temperatures of the Atlantic ocean generated using E.U. Copernicus Marine Service Information thanks to the state-of-the-art simulation engine NEMO. The use of a so-called reanalysis procedure implies that these data accurately represent the actual temperature measures. For more information, we refer to the complete description of the data by de Bézenac et al. (2018). The data history of this engine is available online.? ? Unfortunately, due recent maintenance, data history is limited to the last three years; prior histories should be manually requested.
The dataset uses daily temperature acquisitions from Thursday 28th December, 2006 Wednesday 5th April, 2017 of a 481 x 781 zone, from which 29 zones of size 64 x 64 zones are extracted. We follow the same setting as de Bézenac et al. (2018) by training all models with r 1 = 4 conditioning steps and v т = 6 steps to predict, and evaluating them only on zones 17 to 20. These zones are particularly interesting since they are the places where cold waters meet warm waters, inducing more pronounced motion.
We normalize the data in the same manner as de Bézenac et al. (2018). Each daily acquisition of zone is first normalized using the mean and standard deviation of measured temperatures in this zone computed for all days with the same date of the year from the available data (daily history climatological normalization). Each zone is then normalized so that the mean and variance over all acquisitions correspond to those of a standard Gaussian distribution. These normalized data are finally fed the model; MSE scores reported in Table are computed once the performed normalization of the data and model prediction is reverted to the original temperature measurement space, in order to compute physically meaningful scores.
Training sequences correspond to randomly selected chunks of length = 10 in the first 2987 acquisitions (corresponding to 80% of total acquisitions). and testing sequences all possible chunks of length / = 10 in the remaining 747 acquisitions.
D.3 MOVING MNIST
This dataset involves two MNIST digits (LeCun et al., 1998) of size 28 x 28 that linearly move within 64 x 64 frames and deterministically bounce against frame borders following reflection laws. We use the modified version of the dataset proposed by Franceschi et al. (2020) instead of the original one (Srivastava al., 2015). We train all models in the same setting as Denton & Birodkar (2017), with + == 5 conditioning frames and v - - 10 frames to predict, and test them to predict either 10 or 95 frames ahead. Training data consist in trajectories of digits from the MNIST training set, randomly generated on the fly during training. Test data are produced by computing a trajectory for
httpp: pottps://resaurcent mar coperni cus eo/topt on camn-com_cswvvii ew-detal product_ -ie.iiii-GGOHAAAA ANAILY
20

---

Published as a conference paper at ICLR 2021
each digit of the MNIST testing set, and randomly pairwise combining them, thus producing 5000 sequences.
To evaluate disentanglement with content swapping. report PSNR and SSIM metries between the swapped sequence produced by our model and ground truth. However, having two digits in the image, there is ambiguity truth. To account for this ambiguity and thanks to the synthetic nature of the dataset, we instead build two ground truth sequences for both possible digit swap permutations, and report the lowest metric between the generated sequence and both ground truths (i.e., we choose the closest ground truth to compare with respect the considered metric).
in which order target digits should be swapped in the ground
D.4 3D WAREHOUSE CHAIRS
This multi-view dataset introduced by Aubry et al. (2014) contains 1393 three-dimensional models of chairs seen under the same periodic angles. We resize the original 600 x 600 images by center- cropping them to 400 x 400 images, and downsample them to 64 x 64 frames using the Lanczos filter of the Pillow library."
We create sequences from this dataset for our model by assembling the views of each chair to simulate its rotation from right to left until it reaches its initial position. This process repeated for each existing angle to serve as initial position for all chairs. We chose this dataset instead of Denton & Birodkar (2017)'s multi-view chairs dataset because the latter contains too few objects to allow both tested methods to generalize on the testing set, preventing us to draw any conclusion from the experiment. We train models on this dataset with T + = conditioning frames and v - - 10 frames to predict, and test them predict 15 frames within the content swap experiment. Training and testing data are constituted by randomly selecting 85% of the chairs for training and 15% of the remaining ones for testing. Disentanglement metrics are computed similarly to the MNIST, but with only one reference ground truth corresponding to the chair given as content input at the position of the chair given as dynamic input.
Moving
D.5 TAXIBJ
This crowd flow dataset provided by Zhang et al. (2017) consists in two-channel 32 x 32 frames representing the inflow and outflow of taxis in Beijing, each pixel corresponding square region of the city. Observations are registered every thirty minutes. It is highly structured as the flows are dependent on the infrastructure of the city, and complex since methods have to account for non-local dependencies and model subtle changes in the evolution of the flows.
We follow the preprocessing steps of Wang al. (2018) and Le Guen Thome (2020) by performing a min-max normalization of the data to the [0, 1] range. We train models on this dataset with T = 4 conditioning frames and competitors the last four weeks of data which are excluded from the training set. MSE on this dataset is reported in the [o, no-moamal space and multiplied by a hundred times the dimensionality of frame, i.c. by 100 x 32 x 32 x 2.
r = 4 frames to predict, and test them to predict frames like our
E TRAINING DETAILS
Along with the code in the supplementary material, we provide in this section sufficient details in order to replicate our results.
E REPRODUCTION OF BASELINES
PKnl. We retrained PKnl (de Bézenac et al., 2018) on SST using the official implementation and the indicated hyperparameters.
SVG, MIM and DDPAE. We trained SVG (Denton & Fergus, 2018), MIM (Wang et al., 2019b) and DDPAE (Hsieh et al., 2018) on version of Moving MNIST using the official implementation and the same hyperparameters that the authors used for the original version of Moving MNIST.
hhttps://pi ow. readthedocs.1er
21

---

Published as a conference paper at ICLR 2021
We trained MIM SST using the recommended hyperparameters of the authors, and SVG by retaining the same hyperparameters as those used on KTH.
DrNet. We trained DrNet (Denton & Birodkar, 2017)
version of Moving MNIST using the same hyperparameters originally used for the alternative version of the dataset on which it was originally trained (with digits of different colors). To this end, we reimplemented the official Lua implementation into a Python code in order to train it with a more recent infrastucture. We also trained DrNet on 3D Warehouse Chairs using the same hyperparameters used by its authors on the smaller multi-view chairs dataset on which they trained their method.
PhyDNet. We trained PhyDNet (Le Guen & Thome, 2020) SST and our version of Moving MNIST using the official implementation and the same hyperparameters that the authors used for SST and the original version of Moving MNIST. We removed the skip connections used by the authors on the Moving MNIST dataset in order to perform a fairer comparison with other models, such as ours, in our experimental study that do not incorporate skip connections on this dataset.
E.2 MODEL SPECIFICATIONS
E.2.1 IMPLEMENTATION
We used Python 3.8.1 and PyTorch 1.4.0 (Paszke et al., 2019) implement model. Each model was trained on an Nvidia GPU with CUDA 10.1. Training is done with mixed-precision training (Micikevicius et al., 2018) thanks the Apex library.
E.2.2 ARCHITECTURE
Combination of S and T. As explained in Section 4, the default choice of combination of S and T as decoder inputs is the concatenation of both vectorial variables: it is generic, and allows the decoder to learn an appropriate combination function as in Equation (4).
Nonetheless, further knowledge of the studied dataset can help to narrow the choices of combination functions. Indeed, we choose to multiply S and I' before giving them as input to the decoder for both datasets WaveEq and WaveEq- solutions to the homogeneous version of equation (i.e., without source). This shows that it is possible to change the combination function of and T, and that existing combination functions in the PDE literature could be leveraged for other datasets.
given the knowledge of the existence of product-separable
Encoders F.s and Fr,, and decoder D. For WaveEq, the encoder and decoder outputs are consid- ered to be vectors; images are thus flattened before encoding and reshaped after decoding to 64 x 64 frames. The encoder is a MultiLayer Perceptron (MLP) with two hidden layers of size 1200 and internal ReLU activation functions. The decoder is MLP with three hidden layers of size 1200, internal ReLU activation functions, and a final sigmoid activation function for the decoder. The encoder and decoder used for WaveEq- 100 are similar to those used for WaveEq, but with two hidden layers each, of respective sizes 2400 and 150.
We used for SST VGG16 architecture (Simonyan & Zisserman, 2015), mirrored between the encoder and the decoder, complemented with skip connections integrated into S (Ronneberger et al., 2015) from all internal layers of the encoder corresponding decoder layers, also leveraged by de Bézenac et al. (2018) in their PKnl model. We adapted this VGG16 architecture without skip connections for the 32 32 frames of TaxiBJ by removing the shallowest upsampling and downsampling operations in the VGG encoder and decoder. For Moving MNIST, the encoder and its mirrored decoder are shaped with the DCGAN discriminator and generator architecture (Radford et al., 2016), with an additional sigmoid activation after the very last layer of the decoder; this encoder and decoder DCGAN architecture is also used by DrNet and DDPAF. We highlight that we leveraged in both SST and Moving MNIST architectural choices that are also used in compared baselines, enabling fair comparisons.
For the two-dimensional latent space experiments SST (see Appendix F.3), we use modified version of the VGG encoder / decoder network by removing the two deepest maximum pooling layers,
hhttps:// // thub. http:/c///wvvaaaagegoemm
22

---

Published as a conference paper at ICLR 2021
thus preserving the two-dimensional latent structures. The decoder mirrors the encoder complemented with skip connections.
Regarding 3D Warchouse Chairs, also followed the same architectural choices as DrNet with a ResNet18- architecture for the encoders and DCGAN architecture, followed by a sigmoid activation after the last layer for the decoder.
Encoders Es and Ey taking as input multiple observations, we combine them by either concatenating them for the vectorial observations of WaveEq- 100, or grouping them the color channel dimensions for the other datasets where observations are frames. Each encoder and decoder layer was initialized from normal distribution with standard deviation 0.02 (except for biases initialized to 0, and batch normalizations weights drawn from Gaussian distribution with unit mean and standard deviation of 0.02).
ODE solver. Following the recent line of work assimilating residual networks (He et al., 2016) 6) with ODE solvers (Lu et al., 2018; Chen et al., 2018), we use residual network as an integrator for Equation (7). This residual network is composed of given number K of residual blocks, each block i E [1, K] implementing the application id + Gr. where is an MLP with a two hidden layers of size H and internal ReLU activation functions. The parameter values for each dataset are:
WaveEq and WaveEq-100: K = 3 and H = 512;
SST (with linear latent states): K = 3 and H = 1024;
Moving MNIST, 3D Warchouse Chairs and TaxiBJ: K and H = 512.
Each MLP is orthogonally initialized with the following gain for each dataset:
WaveEq, WaveEq-1000 SST (with linear latent states), 3D Warchouse Chairs and TaxiBJ: 0.71;
Moving MNIST: 1.41.
For SST with wwo-dimensional states, the MLPs are replaced by convolutional layers with kernel size 3. padding and number of hidden channels equal to H = 128. We set K and an orthogonal initialization gain of 0.2. ReLU activations are replaced by Leaky ReLU activations and preceded by batch normalization layers.
Latent variable sizes. S and T have the following vectorial dimensions for each dataset:
WaveEq and WaveEq- 100: 32;
SST, respectively 196 16 x 16 and 64 16 x 16; for the linear version, both are set to 256.
Moving MNIST and TaxiBJ: respectively, 128 and 20;
3D Warehouse Chairs: respectively, 128 and 10.
Note that, in order to perform fair comparisons, the size of T for basclines without static component chosen to the sum of the vectorial sizes of S and the full model. The skip connections of S for SST cannot, however, be integrated into T, as its evolution is only modeled in the latent space, and it out of the scope of this paper to leverage low-level dynamics.
E.3 OPTIMIZATION
Optimization is performed using the Adam optimizer (Kingma & Ba, 2015) with initial learning rate 4 x 10-4 for WaveFq. WaveFq- 100, Moving MNIST, 3D Warehouse Chairs and SST and 4 x 10-5 for TaxiBJ, and with decay rates 31 0.9 (except for the experiments on Moving MNIST where we choose 31 = 0.5) and B2 0.99.
Loss function.
Chosen coefficients values of Xpred AAE. Aeg. and XT are the following:
Xpred 45;
23

---

Published as a conference paper at ICLR 2021
Table 4:
Prediction and content swap PSNR and SSIM scores of variants of our model.
Pred. (L + 10)
Pred. (l + 95)
Swap
10)
Swap (4 + 95)
Models
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Ours
21.70
0.9088
17.50
0.7990
18.42
0.8368
16.50
0.7713
Ours (without S)
20.46
0.8867
14.95
0.6707
Ours (ΛĄE 0)
21.61
0,9058
16.58
0.7611
18.21
0.8309
15.79
0.7399
Ours (A
reg = 0)
15.99
0.6900
12.31
0.5702
13.73
0.5476
12.07
0,5556
Ours (1/2g = 0)
15.63
0.7369
14.02
0.7253
14.91
0.7154
13.95
0.7234
Ours (GRU)
21.66
0.9088
15,45
0.4888
17.70
0.8178
14.77
0.4718
AAE = for TaxiBJ; 10 for SST (linear)and Moving MNIST; for WaveEq, WaveEq-100
and 3D Warehouse Chairs; 0.1 for SST;
reg 100 for SST; Aeg = 45 for WaveFq, WaveFqq-100, SST (linear) and Moving MNIST;
for 3D Warehouse Chairs; 0.0001 for TaxiBJ;
XTg = 1 10-3 for WaveEq, WaveEq-100, Moving MNIST, 3D Warehouse Chairs and TaxiBJ (where p is the dimension of 7); 2p 10-2 for SST (linear); 10-6 for SST.
The batch size is chosen to be 128 for WaveEq, WaveEq- 100, Moving MNIST and 3D Warehouse Chairs, and 100 for SST and TaxiBJ.
Training length. The number of training epochs for each dataset is:
WaveEq and WaveEq-100: 250 epochs;
SST: 30 epochs; SST (linear): 80 epochs;
Moving MNIST: 800 epochs, with an epoch corresponding to 200 000 trajectories (the dataset being infinite), and with the learning rate successively divided by at epochs 300, 400, 500, 600, and 700;
3D Warehouse Chairs: 120 epochs;
TaxiBJ: 550 epochs, with the learning rate divided by 5 epochs 250, 300, 350, 400 and 450.
E.4 PREDICTION OFFSET FOR SST
Using the formalism of our work, our algorithm trains to reconstruct = (Uto...., Ut,)) from conditioning frames V, (to). Therefore, first learns to reconstruct V=(to).
However, the evolution of SST data is chaotic and predicting above an horizon of 6 with coherent and sharp estimations is challenging. Therefore, for the SST dataset only, we chose supervise the prediction from / la (T + 1) i.e, our algorithm trains to forecast (r++()+)) (+()). It simply consists in making the temporal representation Ey (V-(to) match the observation (ro+ instead Vt,- This index offset does not change our interpretation of spatiotemporal disentanglement through separation of variables.
Ut, from
F ADDITIONAL RESULTS AND SAMPLES
F.1 ABLATION STUDY ON MOVING MNIST
We report in Table the results of an ablation study our model on Moving MNIST, that we comment in Section 5.2.
24

---

Published as a conference paper at ICLR 2021
Table 5: method.
FVD score of compared models KTH. The bold score indicates the best performing
Ours
PhyDNet
SVG
DrNet
330
384
375
383
F.2 PRELIMINARY RESULTS ON KTH
The application of our method to natural videos an interesting perspective, but would motivate further adaptation of the model (see perspectives in the conclusion), in particular regarding the integration of stochastic dynamics. Indeed, there a consensus in the literature (e.g.: Denton Fergus (201 8); Villegas et al. (2019); Weissenborn et al. (2020)) indicating that human motion datasets require stochastic modeling because of the inherently highly random events occurring these videos. Tackling this issue would require incorporate stochasticity in our model, for example leveraging variational autoencoders like Denton & Fergus (2018), or supplement it with adversarial losses on the image space, for instance like Mathieu et al. (2016) and Lee et al. (2018). These changes are feasible, but are out of the scope of this paper.
Nonetheless, investigate the realistic video dataset KTH (Schüldt al., 2004), which is an action recognition video database featuring various subjects performing actions in front of different backgrounds. We trained our model, SVG, DrNet and PhyDNet on this dataset. DrNet and PhyDNet are powerful deterministic approaches, while SVG is a standard stochastic video prediction model. We compare all models in terms of FVD (Unterthiner et al., 2018, lower better), which is metric based on deep features that evaluates the realism of the generated videos.
Results are reported in Table 5. We observe that our model substantially outperforms the considered baselines. These significant results against powerful deterministic baselines, and even the standard stochastic method SVG, confirm our advantage at modeling complex dynamics and support our claim that our model lays the foundations for domain-specific methods, such as stochastic version for natural videos.
Reproductibility.
We use the following training parameters for KTH:
we follow the same dataset processing and evaluation procedure as Denton & Fergus (2018);
we train our model on 125 epochs with batch size 100, with an epoch being defined as 100 000 training sequences;
we set the learming rate to 2 x 10 and the same optimizer parameters as for SST;
Apred 45, λAE 10- و 10, XT. p 10-4;
the size of 5 and are respectively 128 and 50;
the ODE is solved with flat latent architecture and parameters K and H 512;
the encoder and decoder architecture is VGG16 with skip connections integrated into S from Es to D, and with the decoder output being given final sigmoid activation.
We reproduced SVG, DrNet and PhyDNet using the recommended hyperparameters of their authors. We trained PhyDNet for 125 epochs, like model, to obtain fair evaluation despite its low efficiency (six times slower than ours).
F.3 MODELING SST WITH SEPARATION VARIABLES
We present in Table 6 results of Table for SST, complemented with an alternative version of our model obtained using vectorial representation for S and T and MLPs to compute the derivative of strictly enforced separation of spatial and dynamical variables,
T. The latter setting corresponds with results significantly outperforming powerful methods PhyDNet, PKnl and SVG thanks to this separation, attested by the corresponding ablation without static component.
25

---

Published as a conference paper at ICLR 2021
Table 6: Forecasting performance SST of PKnl, PhyDNet and model with respect to indicated prediction horizons. Bold scores indicate the best performing method.
MSE
SSIM
Models
+
t + 10
+ 6
+ 10
PKnl
1 1.28
2.03
0.6686
0.5844
PhyDNet
1.27
1.91
0.5782
0,4645
SVG
1.51
2.06
0.6259
0.5595
MIM
0.91
1.45
0.7406
0.6525
Ours
0.86
1.43
0.7466
0.6577
Ours (without S)
0.95
1.50
0.7204
0.6446
Ours (linear)
1.15
1.80
0.6837
0,5984
Ours (linear, without S)
1.46
2.19
0.6200
0,5456
t=0
t=20
Input fraies
Truth
ours
Figure 6: Example of predictions of our model on WaveEq.
However, sea surface temperature exhibits highly local structure that can be assimilated to a flow in coarse approximation. For example, there is transport of large bodies of hot and cold water. Accordingly, performances may be enhanced by considering local dependencies in the dynamics, as also implemented by MIM and PhyDNet. We propose to do by considering like the latter methods two-dimensional latent states for the static S and the dynamical T, and convolutional networks to model the derivative of T.
Accounting for such locality in the dynamics amounts to implementing another separation than the usual separation between and spatial variables. Indeed, it rather excludes unknown content variables from the dynamics. The resulting dynamics is then a PDE over time and the observation coordinates and that implement using convolutional neural networks, following Long et al. (2018) and Ayed et al. (2020). This different kind of separation of variables simplifies learning by estimating PDE that is simpler than the original one, since it acts on fewer variables. It highlights the generality of our intuition of using the separation of variables, which may be used in other settings that strict spatiotemporal disentanglement. This approach, while still maintaining disentangling properties, significantly improves prediction performances.
Note that our proposition remains computationally much lighter than the alternatives MIM, PhyDNet and SVG.
F.4 ADDITIONAL SAMPLES
F.4.1 WAVREQ
We provide in Figure a sample for the WaveEq dataset, highlighting the long-term consistency in the forecasts of our algorithm.
We also show in Figure 7 the effect in forecasting of changing the spatial code S from the one of another sequence.
26

---

Published as a conference paper at ICLR 2021
Swap difference
t=5
t=10
t=15
t=20
T=25 t=30
t=35
t=48
t=45
t=50
Figure 7: Evolution of the scaled difference between the forecast of sequence and the same forecast with spatial code coming from another sequence for the WaveEq dataset.
t=0
t=4
t=6
t=8
Input frames
Truth
PKnl
PhyDNet
MIM
Ours
Content input
Swap
Figure 8: Example of predictions of compared models on SST.
27

---

Published as a conference paper at ICLR 2021
t-
t=2
t=4
t-6
t=8
t-10
t-12
t-14
t-24
t-54
t-99
Input frames
7
7
Truth
7
7
/
7
7
17
7
h
71
MIM
7
7
7
7
7
7
/
/
DrNiet
4
1
1
/
7
7
7
DDPAE
7
7
7
7
7
/
71
7
PhyONet
77
7
7
7
77
7
/
Ours
7
7
7
7
/7
7
h
71
/
Content input
Swap
6
7
9
5
9
4
/
/
55
Figure 9: Example of predictions of compared models Moving MNIST.
t=0
t=2
t=4
t=6
t=8
t=10
t-12
t-14
t-24
t-54
t-99
Input franes
70
70
7
Truth
o
O1
MIM
b
DrNet
O
10
10
DDPAE
1
PhyONet
Ours
7
6
0
ō
O
Content input
0
0
Swap
0
8
g
5
5
5
5
50
Figure 10: Example of predictions of compared models Moving MNIST.
F.4.2
SST
We provide an additional sample for SST in Figure 8.
F.4.3 MOVING MNIST
We provide additional samples for Moving MNIST in Figures 9 and 10.
28

---

Published as a conference paper at ICLR 2021
Dynamic input
Dynamic input
Content input
Content input
(a) DrNet.
(b) Ours.
Figure 11: Fusion of content (first column) and dynamic (first row) variables in DrNet and our model on 3D Warehouse Chairs.
F.4.4 3D WAREHOUSE CIAIIR
We provide qualitative comparison for the content swap experiment between model and DrNet for 3D Warehouse Chairs in Figure 11. We notice that DrNet produces substantially more blurry samples than our model and has difficulties capture the exact dynamic of the chairs.
29


Reviews:
[{"review_id": "unq7FMrkb8n", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework for spatiotemporal disentanglement. In particular, it contributes to the disentanglement of content and dynamics using neural ODEs. \n\nStrength: \n1. I think it is a well-written paper with interesting experimental design, especially the “swap” and “multi-view” experiments that measure the degree of disentanglement. \n2. The proposed model consistently outperforms well-established approaches for video prediction, including DrNet, DDPAE, and PhyDNet, which is also a PDE-based model for spatiotemporal disentanglement.\n\nWeakness:\n1. Although the effectiveness of the model has been validated on several datasets, most of them are synthetic. I strongly encourage the authors to include real human motion datasets. For example, DrNet was evaluated on KTH, and PhyDNet was evaluated on Human3.6M.\n2. The model was only compared with PKnl and PhyDNet on the Wave and SST datasets, and I am curious about how models beyond disentanglement, such as SVG and MIM, perform in this case. Also, the results in Figure 2 appear to be quite blurry and do not show significant improvement. These experiments mainly show the general ability of the proposed model for video prediction rather than spatiotemporal disentanglement. \n\nCorrectness: \nIn Eq. (12), the authors use a Gaussian prior to convey dynamic information and exclude the spatial information. Is it a good thing or a bad thing from the view of modeling temporal dynamics? A question is: can such a simple dynamic model be applied to datasets with complex motion information?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VaqLW4BhWcR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a generative model for videos where the latent trajectories have two components - a term without a slowness loss that represents \"content\" and a term with a slowness loss that represents \"style\". They present results on a dataset simulating the wave equation and on videos of moving MNIST digits and 3D chairs. The results are generally good, especially for long roll-outs, and they demonstrate something like disentangling by showing that the identities of the digits can be swapped in the moving MNIST data.\n\nMy main objection with the paper is that it has nothing to do with PDEs or separation of variables. The actual latent trajectories are simulated as *ODEs*, not PDEs, which are then used to generate images. The justification in terms of separation of variables is also a bait-and-switch...a slowness penalty is added to the loss for one of the latent trajectories, that is all. Factored latent trajectories are a well-established modeling technique for time series already. The use of ODEs parameterized by neural networks rather than, say, LSTMs or other RNN architectures is also more a difference in degree than in kind from other sequence models. Much like how ResNets become neural ODEs in the limit of very deep networks, simply using Runge-Kutta updates parameterized by a neural network is still technically a kind of RNN (if you define RNN very loosely as any nonlinear iterated function learnable by gradient descent). So I'm not sure that this paper actually does most of what it claims to be doing in the motivation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MrpnIesKol5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a spatiotemporal disentanglement method for handling sequence data. Solving high-dimensional PDEs, for deriving the exact dynamics, is difficult; hence, this work proposes learning time-invariant and time-dependent representations separately to solve this problem. To achieve this goal, the authors devised a model that incorporates a temporal ODE process. The provided experiments indicate that the method achieves good performance, although the gain is not consistent.\n\nThe overall derivation and methodology of this paper are technically sound, and I guess discarding $S^\\prime$ was a practical choice for learning time-invariant representation. Although the proposed model is generally applicable, the experiments only cover prediction results on classical image datasets. Nevertheless, the underlying timescales of datasets are sufficiently varied to demonstrate the effectiveness in general temporal sequences.\n\n[Quality]\n\nThe paper is clearly written overall. However, Section 4 was a bit hard to follow, which is the core part of this paper. For example, \"the system coordinates\" and reasons for the relaxation (Sec. 4.2) were not clearly explained for people who are not familiar with the area. It would be harder to understand the architecture if Fig. 1 was not given.\n\n[Originality]\n\nThe originality of the paper is not stellar, but sufficient for acceptance. \n\n[Significance]\n\nThe significance of this work is mainly for model architecture. I believe these kinds of approaches, which can internally model continuous dynamics, are heavily preferred when solving real-world problems. Therefore, the significance of the paper is sufficient for acceptance.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model for disentangling content and dynamics, but unlike the majority of previous work, the dynamics are modeled using ODEs rather than their discrete approximations - RNNs. The reviewers agree that the paper is well written, and the results look good, especially for longer trajectories. Hence, I am happy to recommend this paper for acceptance.
Decision: Accept (Poster)