Long-horizon video prediction using a dynamic latent hierarchy
Authors: Alexey Zakharov
Qinghai Guo
Zafeirios Fountas
Venue: Submitted to ICLR 2023 (2023)
Keywords: long-term video prediction
hierarchical generative model
spatiotemporal disentaglement
event-based model

Abstract: The task of video prediction and generation is known to be notoriously difficult, with the research in this area largely limited to short-term predictions. Though plagued with noise and stochasticity, videos consist of features that are organised in a spatiotemporal hierarchy, different features possessing different temporal dynamics. In this paper, we introduce Dynamic Latent Hierarchy (DLH) -- a deep hierarchical latent model that represents videos as a hierarchy of latent states that evolve over separate and fluid timescales. Each latent state is a mixture distribution with two components, representing the immediate past and the predicted future, causing the model to learn transitions only between sufficiently dissimilar states, while clustering temporally persistent states closer together. Using this unique property, DLH naturally discovers the spatiotemporal structure of a dataset and learns disentangled representations across its hierarchy. We hypothesise that this simplifies the task of modeling temporal dynamics of a video, improves the learning of long-term dependencies, and reduces error accumulation. As evidence, we demonstrate that DLH outperforms state-of-the-art benchmarks in video prediction, is able to better represent stochasticity, as well as to dynamically adjust its hierarchical and temporal structure. Our paper shows, among other things, how progress in representation learning can translate into progress in prediction tasks.

Under review as a conference paper at ICLR 2023
LONG-HORIZON VIDEO PREDICTION USING A DYNAMIC LATENT HIERARCHY
Anonymous authors Paper under double-blind review
ABSTRACT
The task of video prediction and generation is known to be notoriously difficult, with the research in this area largely limited to short-term predictions. Though plagued with noise and stochasticity, videos consist of features that are organised in a spatiotemporal hierarchy, different features possessing different temporal dy- namics. In this paper, we introduce Dynamic Latent Hierarcky (DLH) a deep hierarchical latent model that represents videos as a hierarchy of latent states that evolve over separate and fluid timescales. Each latent state is a mixture distri- bution with two components, representing the immediate past and the predicted future, causing the model to learn transitions only between sufficiently dissimi- lar states, while clustering temporally persistent states closer together. Using this unique property, DLH naturally discovers the spatiotemporal structure of dataset and learns disentangled representations across its hierarchy. We hypothesise that this simplifies the task of modeling temporal dynamics of a video, improves the learning of long-term dependencies, and reduces error accumulation. As evidence, we demonstrate that DLH outperforms state-of-the art benchmarks in video pre- diction, able better represent stochasticity, as well to dynamically adjust its hierarchical and temporal structure. Our paper shows, among other things, how progress in representation learning can translate into progress in prediction tasks.
INTRODUCTION
Video data is considered one of the most difficult modalities for generative modelling and prediction, characterised by high levels of noise, complex temporal dynamics, and inherent stochas- ticity. Even more so, modelling long-term vidcos poses significant challenge due to the problem of sequential error accumulation, largely restricting the research in this topic to short-term predictions.
Deep learning has given rise to generative Aatent-variable models with the capability to learn rich latent representations, allowing to model ligensinensi data by means of more efficient, lower- dimensional states (Kingma & Welling 2014) Higgins et al. 2022 Vahdat & Kautz 2020) Rasmus al. 2015). Here, of particular interest hierarchical latent models, which possess higher de- gree of representational power and expressivity. Employing hierarchies has so far proved to be an effective method for generating high-fidelity visual data, as well concurrently producing more meaningful and disentangled latent representations in both static (Vahdat & Kautz 2020) and tem- poral (Zakharov et al. 2022) datasets.
Unlike images, videos possess spatiotemporal structure, in which collection of spatial fea- tures adhere the intrinsic temporal dynamics of dataset often evolving at different and fluid timescales. For instance, consider a simplistic example shown in Figure [ in which the features of a video sequence evolve within a strict temporal hierarchy: from the panda continuously changing its position to the background elements being static over the entire duration of the video.
Discovering such a temporal structure in videos complements nicely the research into hierarchical generative models, which have been shown capable of extracting and disentangling features across hierarchy of latent states. Relying on this notion of inherent spatiotemporal organisation of features, several hierarchical architectures have been proposed to either enforce a generative temporal hier- archy explicitly (Saxena et al.) 202I), or discover it in an unsupervised fashion (Kim et al.) 2019 Zakharov et al. 2022). In general, these architectures consist of collection of latent states that

---

Under review as a conference paper at ICLR 2023
Background
Airplane
Panda
Figure I: Videos can be viewed as a collection of features organised in a spatiotemporal hierarchy. This graphic illustrates a sequence of frames, in which the components of the video possess different temporal dynamics (white circles indicate feature changes). Notice the irregularities in their dynam- ies the panda continuously changes its position, the airplane is seen only for few timesteps, while the background remains static throughout. Similar to this, our model learns hierarchically disentan- gled representations of video features with the ability to model their unique dynamics.
transition over different timescales, which has been shown to benefit long-term predictions (Saxena et al. 2021 Zakharov et al. 2022).
Building upon these notions, we propose an architecture of hierarchical generative model for long-horizon video prediction Dynamic Latent Hierarchy (DLH). The principle ideas underly- ing this work are two-fold. First, we posit that learning disentangled hierarchical representations and their separate temporal dynamics increases the model's expressivity and breaks down the prob- lem video modelling into simpler sub-problems, thus benefiting prediction quality. As such, our model is capable of discovering the appropriate hierarchical spatiotemporal structure of the dataset, seamlessly adapting its generative structure to a dataset's dynamics. Second, the existence of a spatiotemporal hierarchy, in which some features can remain static for an arbitrary period of time (e.g. background in Fig. []. implies that predicting the next state at every timestep may introduce unnecessary accumulation of error and computational complexity. Instead, our model learns to transition between states only if a change in the represented features has been observed (e.g. air- plane in Fig. D. Conversely, if no change in the features has been detected, the model clusters such temporally tttrsssellypessssteett states closer together, thus building more organised latent space. Our contributions are summarised as follows:
A novel architecture of a hierarchical latent-variable generative model employing temporal Gaussian mixtures (GM) for representing latent states and their dynamics;
Incorporation of a non-parametric inference method for estimating the discrete posterior distribution over the temporal GM components;
The resulting superior long-horizon video prediction performance, emergent hierarchical disentanglement properties, and improved stochasticity representation.
2 DYNAMIC LATENT HIERARCHY
We propose an architecture of a hierarchical latent model for video prediction Dynamic Latent Hierarchy. DLH consists of a hierarchy of latent states that evolve over different and flexible timescales. Each latent state is mixture of two Gaussian components that represent the immediate past and the predicted future in single belief state. Using this formalisation, the model dynamically assigns every newly inferred posterior state to one of these clusters, and thus implicitly learns the temporal hierarchy of the data in an unsupervised fashion.
2.1 GENERATIVE MODEL
We consider sequences of observations, (01, or), modelled by hierarchical generative model with a joint distribution in the form (Fig. 2
N
po o., 5, E.)
I pa (o. (.)
II
p
S spa(e"|s2
temporal hierarchical
2

---

Under review as a conference paper at ICLR 2023
a
b
4
#2
d
Figure 2: (a) Generative model of DLH. (b) Inference models of DLH. (c) Architectural components of DLH, showing the deterministic variables that correspond to bottom-up. top-down, and temporal information. (d) Example of two-level DLH model rolled-out over three timesteps.
where s, ~ (.(,-)) is a diagonal Gaussian latent state, e. ~ Ber(-) is the corresponding Bernoulli variable at hierarchical level n and timestep 6, 5 {s} ...,; .*v and {e ,..., e^ denote collections of all variables in a hierarchy, and 0 are the parameters of the model. Notice that each state sy conditioned on all of the hierarchical states above, past states in the same level, and an indicator variable e."
One of the key features DLH is the representation of latent state as a temporal mixture of Gaus- sians (MoG). In particular, variables sa and eă together define a MoG, p(s," e²") p(s" "e"""e""))))) peee) with just two components such that,
(p(st_1) (po(s)|s2
ee 0, (previous state: static prior) if ee = 1, (predicted state: change prior).
p(s
As such, every timestep, DLH holds prior beliefs over the state of the world: (1) it can remain static, or (2) it can progress through time and thus change. In this view, variable of can be informally described as the probability of whether state St should be updated or remain fixed at timestep & (Fig. 3). This property allows DLH to model data as a collection of hierarchical latent states that evolve over different and flexible timescales, determined by the indicator variable e".
2.2 INFERENCE
In order to train the model using variational lower bound, must estimate the posterior distri- bution q(s," lo,),, for which we assume a mean-field factorisation ((s")<(e"); therefore, the two distributions approximated separately (Fig. (
Estimating (s) In DLH, posterior (s,") assumed to be a diagonal Gaussian, amortised using a neural network go (s) ss?" 0,) with parameters vb, conditioned on hierarchical states above and the latest data point o,- In line with the established procedure, the approximate posterior is trained using the reparametrisation trick (Kingma & Welling 2014).
Estimating q(c) Using reparametrisation tricks for discrete latent variables poses a significant challenge for stable training procedure of deep learning models, which can be further exacerbated in hierarchical models (Falck et al. 2021). To avoid this, we estimate q(e) using a non non-parametric method.
Inferring distribution g(e") ) can be conceptualised as a clustering problem of q("") with respect to the static and change priors of the model, with the central question being: under which temporal mixture component in Eq. is the inferred state most likely? Has the state of the world changed or
has remained the same? As such, formulate the approximation p(e" (s as model selection using expected likelihood ratio, where the two components of the MoG (Eq. are the competing
Stripping away the hierarchical and temporal conditioning for elarity.
3

---

Under review as a conference paper at ICLR 2023
models. Under the inferred state, q' (s{))) (s s .z),, the expected log- likelihoodd ratio is,
Ellog A Ass))] = (gp) log ('(s = 0) p' =
(3)
where p'(s;) (er) po (S, c <t: S²") from the definition of the generative model. Assuming
the selection of the most likely component under the inferred posterior, we come to the following selection criterion (see full derivation and further clarifications in Appendix B).
D (sp) le = 1)] D (sP)
0)].
(4)
where the most likely component is approximated have probability q(e; i) This approximation relates to the VaDE trick, which is similarly a non-parametric method of estimating the posterior component variable of MoG (Jiang et al. 2017 Falek al., 202T]. In particular, our method can be viewed as taking a sample from the most likely component of the VaDE-estimated q(e") under the assumption of equal prior probabilities (see Appendix B.3). Though this method introduces bias, in practice, we found that it performs better than the VaDE trick. We hypothesise that this relates to relatively fast convergence of the parametrised prior po(e.)-) model, which becomes overly confident in its predictions (even before any video features have been learned), thus irreversibly skewing the approximation of g(e")). Nevertheless, we believe this direction future work may merit further investigation.
2.3 NESTED TIMESCALES
We add a constraint on the hierarchical temporal structure of the generative model similar to Saxena
ett al. (2021): Zakharov al. (2022). In particular, q(e"+1 1|e" 0) 0. Enforeing this constraint has been shown to be an effective method to promote spatiotemporal disentanglement of features in hierarchical models, encouraging the representation of progressively slower features in the higher levels of the model. Furthermore, to reduce the computational complexity of the model, we block any further inference above the hierarchical level where e" is inferred, such that:
(5) Lastly, model continuously changing videos, we assume q(e² 1) 1, which allows for the bottom level of DLH to always be in use. It is worth noting that the proposed model constraints may be relaxed in different implementations of DLH. which could be explored in future work.
e = 0, then e," and ((s") + q(s
2.4 LOWER BOUND ESTIMATION
To train the model, we derive variational lower bound (ELBO), for which we introduce an approx- imate posterior distribution q(5,, c) so that
logp(o.) Σ log [Ealsh
((s 0.) po (Or, Sc, ec)
(6)
E Eg R. log lo + Ey(E. log Po(S e )oo (ē:)
g(5, E,)
Assuming posterior factorisation of q(s)" e²¹) = " we write the complete formulation of the ELBO,
T
LELOO
E (3) (og
(8a)
T
N
E Eg 1n 182 DKL ",."
le, s2. s]"]]]
(8b)
N
E Byier- (e0 DKL -e
(8c)
4

---

Under review as a conference paper at ICLR 2023
ef
0
cj
Level 2
Pruned
2
s
Level 1
=1
=
+
Figure 3: Sampling MoG components can be seen as changing the generative structure of the model. The diagram shows two-level DLH unrolled over three timesteps each box indicating temporal MoG. Bold arrows indicate the sampled component of a MoG. At n, t = 2, the model samples the static component 0 ~ plez), thus state sf remains fixed before being updated s (indicated by the 'Pruned' label on the right).
To better understand the optimisation objective and the role of temporal Gaussian mixture from Eq. it is useful to break the down the three components of the ELBO. First, component [a is the likelihood of the data under the inferred posteriors (5x), which improves the quality of frame recon- structions. Second, component &c] is the KL divergence between the posterior and prior Bernoulli distributions over allowing the parametrised prior model to learn the evolution of static and change priors over time. Lastly, component [b) regularises the latent belief space by bringing the posterior either closer the static or the change component of prior MoG. This can be if we expand the expectation,
clearly
=q(e" 0) DKL (dd (pp))' le, 0)] +q(e," = 1) DKL (au[9'' (g'(s
1)]
(9)
posterior static prior
posterior 4+ change prior
Depending on the inferred posterior distribution q(e,)). the model will employ the appropriate part of Eq. | in the optimisation. For example, if inferred that state s has not changed (eī 0), the model will bring the new posterior and the static prior closer together, and vice versa. Ultimately. this allows the model to naturally cluster similar temporal states together, while learning transition between states that are sufficiently separated in the belief space.
2.5 MODEL COMPONENTS
DLH's architectural implementation similar to that of NVAE (Vahdat & Kautz 2020 and VPR Zakharov et al., 2022). which employ deterministic variables for propagating information through model. More specifically, DLH consists of the following model components,
Encoder,
Lt
n+1
f (x²))
(10) Posterior state,
(13)
Decoder,
f (5) ct
(11) Prior state,
Po di ())
(14)
Temporal,
di frem (s) d.t
(12) Prior factor,
pe (e(e" (d")
(15)
where deterministic variables x²; c,, di correspond to the bottom-up, top-down, and temporal variable transformations, as shown in Figure 2 and a! and ef correspond to the output image o,- Variables of and d, are non-linear transformations of samples from si" and s respectively. We GRU model (Cho et al. 2014) for the transition and prior factor models, convolutional layers for the encoder and decoder, and fully-connected ML. layers for all other models.
RELATED WORK
Video prediction Early works in video prediction largely focused on different variants of deter- ministic models (Oh et al., 2015 Finn et al., 2016 Byravan & Fox] 2017] Vondrick & Torralaa 2017]: however, it has been widely suggested that these models are poorly suited for capturing stochasticity that is often present in video datasets.
5

---

Under review as a conference paper at ICLR 2023
GT
12
DLH
DLH GT
DLH GT
T
1
T 1
Y
T
t
Y Y
1
1
7
1=1 5
10
15
20
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
Figure 4: Open-loop video prediction given 30 context frames (not shown). GT; ground-truth se- quence, DLH: our model's prediction. DLH maintains the important contextual information about the video, without significant degeneration in the reconstruction quality.
The problem of stochastic video prediction has been addressed using variety of generative ar- chitectures. Models autoregressive in image space (Babaeizadeh al. 2021
Reed al., 2017
Weissenborn et al. [0200] Kalchbrenner et al. 2016 Denton & Fergus 2018 demonstrate good results but suffer from high computational complexity, particularly for long-term predictions. GAN- based (Goodfellow et al. 2014) approaches have been popular due to their ability to produce sharp predictions (Clark et al. [2019] Arr Mathieu et al. 2016 Lee et al. 2018). 8). More recently, transform- (Vaswani et al. 2017) have been used model video datasets, both in latent (Rakhimov al., 2020 Yan et al. 2021 and pixel space (Weissenborn et al. 2020). A fairly large category of video using Variational Autoencoders (Kingma & Welling (014), which have been shown to produce meaningful latent representations on image (Vahdat & Kautz 2020; Higgins
architectures is based
ct al. 2022) and video data (Zakharov et al.) [0222). Variational autoencoders (VAE)-based models
that attempt to learn temporal dependencies in the latent space ( (Wu et al. 2021 Villegas et al. 2019
al,, 2022) generate good performance but generally suffer from blurry predictions, referred to as the 'un- derfitting problem' (Babaeizadeh al., [021 Wu et al. 20 Villegas al.] 2019). Nevertheless, these models benefit from computational efficiency since the learning of temporal video dynamics commonly happens
Castrejon et al., 2019 Franceschi al., 2020; Saxena al., 2021: Yan al. [021 2021
sower-dimensional latent space. Most recently, diffusion models have been
shown to produce great performance on both short (Yang et al. 2022
Höppe et al., 2022) and long
(Harvey et al. 2022) videos.
Hierarchical generative models Hierarchical generative models have been shown to be ef- fective way of modelling high- digh-dimensional data, including images (Rasmus et al., 2015 Sønderby al. 2016; Maaløe al. 2019; Vahdat & Kautz 2020 Child 2021) and videos (Saxena al.)
2021 Kim et al. 2019; Pertsch et al. 2020 Hsu et al. 2019 Zakharov et al. 2022), producing rich
latent representations and demonstrating strong representational power.
Temporal abstraction The topic of learning temporal abstractions from sequential data has been harmoniously rising in popularity alongside the progress in deep and hierarchical latent models. Temporal abstraction models often operate number of hierarchical latent variables updating over different timescales, with the goal of capturing the temporal features of dataset (Chung al.] 2017] Mujika et al.) 2017 Kim et al.] 2019; Saxena et al. 2021 Fountas et al., [022] Zakharov et al. 2022], though other models learn the relevant prediction timescales without resorting to hierarchical methods (Chung et al. 2017; Neitz et al. 2018 Jayaraman et al.) 2018 Shang et al. 2019 Kipf
ct al. 2019 Pertsch et al. 2020 Zakharov et al. 2021).
Gaussian Mixtures in VAES Our work similarly touches on the topic of VAEs with Gaussian Mixture latent states. Generally, these models are aimed at producing meaningful structure of the latent space, in which data points are clustered in unsupervised fashion (Dilokthanakul et al. 2016 Jiang et al. 2017, Falck et al. 2021). Though highly relevant conceptually, these works deal with non-temporal data and therefore have fundamentally different formulations.

---

Under review as a conference paper at ICLR 2023
Level (L1)
Level 2 (L2)
Level 3 (L3)
f
KTH Action
1
1
T
1
1
1
Y
1
M-MNIST
3
8 7
J
2
7
3
3
3
6
5
7
3
8
3 9
3
3
3
2
8
2
2
5
5
6
5
8
8 3
2
5
3
3
9
DML Mazes: Level rollouts
L3
L2
L1
La
Figure 5: Hierarchical disentanglement in DLH. (A): Random samples independently drawn from the different hierarchical levels while keeping all other levels fixed. They qualitatively illustrate what information contained within a particular level. In KTH, LI and L2 tend to encode motion; L3 encodes the general context of the frame. In M-MNIST, L1 encodes slight variations in the position and digits; 1.2 represents position; I.3 contains both style and digit types. (B): Rolling out hierarchical levels in the DML Mazes under all other levels being fixed. L.1 includes minor variations in the view angle: L2 changes the position of the observer and wall shape; L3 predicts transitions the different parts of the maze. (C): Inferred components of the temporal MoGs at every level of the model. Black circles indicate e" = 1 (change component) inferred by DLH. L2 infers the static component when the person's arms are not moving (green ellipse). Similarly, L3 is static throughout the duration of the video.
EXPERIMENTS
In this section, we showcase the representational properties DLH, and their resulting impact on the performance of the model for long-term video prediction. In particular, we demonstrate that DLH: (a) outperforms benchmarks long-term video prediction, (b) produces an organised hierarchical latent space with spatiotemporal disentanglement and temporal abstraction, (c) generates coherent videos even in datasets characterised by temporal stochasticity, and (d) dynamically regulates its structural complexity. In the analysis probing DL.''s expressivity and representations, emphasise how the presented formulation of the generative model, particular the use of temporal MoG, naturally results in the emergent properties of the model.
4.1 DATASETS AND BENCHMARKS
Datasets To test the model's ability in long-term video prediction, vastava et al. 2016) with 300 timesteps, KTH Action (Schuldt et al. (2004) with 300 timesteps, and DeepMind Lab (DML) Mazes (Eslami et al. 2018) with 200 timesteps. For more detailed analysis of the model's properties, we use toy Moving Ball dataset (Zakharov et al. 2022).
Moving MNIST (Sri-
Benchmarks Clockwork Variational Autoencoder (CW-VAB) Saxenaa al. 202 is hierarchical
VAE for video prediction, which latent variables operate over iixed-tempora schedules, similarly subject nested timescales. CW-VAR demonstrated state-of-the-arr performance in long-term video

---

Under review as a conference paper at ICLR 2023
prediction, indicating the merit of the slower-evolving context states. VTA (Kim al. 2019 is
two-level hierarchical model for vidco prediction that employs a parametrised boundary detector to learn sub-sequences of video and generate lemporally-abstractee (Lee et al. 2021) learns and stores long-term motion context for better long-horizi video predic- tion, which has been shown to outperform other RNN-based approaches.
representations. LMC-Memory
Metries To evaluate stochastic video prediction, we em- ploy the standard procedure of sampling 100 condition- ally generated sequences and picking the best one to re- port (Denton & Fergus) 2018). For metrics, we use Struc- tural Similarity (SSIM) and Peak Signal-to-Noise Ratio (PSNR) test the performance of model with respect the ground-truth videos.
Table 1: Open-loop video prediction.
Stars denote <5% standard deviation.
M-MNIST
SSIM
PSNR1
DLH (Ours)
0.78*
15.7*
CW-VAE
0.68*
13.11* *
VTA
0.58
12.18
LMC-Memory
0.75*
13.73*
4.2 VIDEO PREDICTION AND GENERATION
KTH Action
SSIM
PSNR1
Table shows the evaluation of DLH and its benchmarks in the task of long-horizon video prediction. As evi- dent, DLH outperforms other models across the presented shows some examples of long-horizon
DLH (Ours)
0.84*
24.7
CW-VAE
0.80
22.0
VTA
0.77
22.41
datasets. Figure 4 open-toop rollouts. For Moving MNIST, DL.H maintains the information about the digits throughout the sequence, while also accurately predicting their positions. For DML Mazes, DLH correctly predicts the colours and wall po- sitions, without switching to configuration of another maze. Similarly, for KTH, our model preserves the im- portant contextual knowledge (e.g. background) and ac- curately predicts the long sequence of arm swings. Ap- pendices C.I and C.2 include qualitative comparisons of the models and the per-frame metric plots.
LMC-Memory
0.83*
23.44
DML Mazes
SSIM
PSNR1
DLH (Ours)
0.59
14.3
CW-VAE VTA
0.44
13.71
0.55
13,51 *
Parameter count: DLH (7M), CW-VAE (12M), VTA (3M),
LMC-Memory (34M).
4.3
HIERARCHICAL ABSTRACTION AND DYNAMIC STRUCTURE
DLH exhibits characteristics of model that learns temporally-abstracted and hierarchically dis- entangled representations. Figure 5 demonstrates reconstructed frames retrieved by sampling the different hierarchical levels of the model. Here, we observe the variations in the samples that corre- spond to meaningful and interpretable spatiotemporal features of the videos. In Figure we show rollouts the model's levels (other levels being fixed) using DMI. Mazes, which indicate that DLH learns to transition between progressively slower features in the higher levels of its hierarchy.
Figure 5 demonstrates another telling qualitative evalua- tion of DLH and its representations the inferred compo- nents of e" (static or change) for a given video. In par- ticular, it shows that the model continuously detects fea- ture changes in the second level of its hierarchy (L.2) when the person is moving their arms, and conversely when the person's are motionless. Furthermore, it can be seen that the the top level (L.3) remains static throughout. Notably, these results are in agreement with the random samples shown above, and more clearly illustrate the property of hierarchical disentanglement present in the model.
Table 2: Average number of employed
levels (L) and the total KL loss in the instances of DLH with different num- ber of hierarchical levels (Moving Ball)
Levels
L
KI. loss
2
1.22 + 0.05
41.9 0.3
3
1.24 + 0.05
42.8 = 1.1
1.32
4 0.12
41.1 1.0
5
1.38 + 0.07
43.1 2.1
The capacity of DLH to learn the spatiotemporal representation of features along its hierarchy is largely driven by the dynamic manipulation of its hicrarchical and temporal structure (Figure B3) In- terestingly, we observe that DLH consistently converges to similar structures even when possessing different number of levels. Table 2 shows the average hierarchical depth employed by the model (L = Σ Σx e over video length given the total number of hierarchical levels it has (trained using the Moving Ball dataset). As evident, the models converge to similar values despite their size differences, indicating that DLH naturally simplifies its structure and does not employ
8

---

Under review as a conference paper at ICLR 2023
DLH
CW-VAE
Two randomly-selected open-loop rollouts in stochastic Moving Ball dataset using DLH (top) and CW- VAE (bottom) models. DLH successfully produces realistic rollouts, while CW-VAE struggles to produce rollouts with sharp and random colour changes (frames highlighted in red).
Figure 6:
more resources than necessary. This is similarly substantiated by the comparable magnitudes of the total KI. loss, which is commonly used to indicate the amount of information stored in the latent states (see Appendix C.3 for more in-depth analysis).
4.4 TEMPORAL STOCHASTICITY
Videos often contain temporal stochasticity, where features may change at seemingly random times. How would gener- ative model represent such uncertainty? In the context of em- ploying Gaussian latent state, the uncertainty about the next state would have to be reflected the higher variance, or- der to cover both possible outcomes; however, this necessarily increases the chance of sampling areas of the latent space that do not correspond to any meaningful states, harming the pre- DLH, by virtue of the temporal MoG,
Table 3: Average predicted proba- bility of po(e" = 1) under the dif- ferent levels of temporal stochastic- ity (A) in the Moving Ball dataset
Po = 1)
change
static
diction performance. such stochasticity be effectively captured by variable e". which decides whether the latent state s should be updated 0.3
0.0
.97 0.01
.007 + .003
0.1
.82 = 0.02
.074 + .007
.73 = 0.02
.167 + .011
or remain fixed, alleviating the need to sample from degener- ate regions of the latent space. To demonstrate this, we modify the Moving Ball dataset to include random colour changes that can occur at every timestep with probability and train a two-level shows a comparison of open-loop rollouts generated by DLH and CW- WAE, trained on the stochastic Moving Ball with = 0.1. While CW-VAE struggles to generate rollouts with consistent and sharp colour changes, DLH faces no such problems, producing sequences with both deterministic and random colour switches. This experiment shows the important role MoGs in representing temporal stochasticity, and highlights the superior representational capacity of DLH.
DLH using it. Fioors
Similarly, the behaviour of prior (e()))-) under temporal stochasticity can be more clearly under- stood using the results in Table 3] which shows the average predicted probability of the change component, at level 2, under the inferred posterior q(e²) being either change (e² = 1) or static (e2 = 0). More stochasticity necessarily implies the reduced ability to predict when the observed video features would change (as would be signalled by the inferred posterior component), which should be reflected in the average probabilities predicted by the prior component model. Indeed, in Table 3 we observe that as the stochasticity of the dataset, X, rises, the model becomes more cautious its predictions.
DISCUSSION
Our work demonstrates that building generative models with better representational properties, such as spatiotemporal and hierarchical disentanglement, translates to better predictive capabilities in long and complex time series. Furthermore, we believe that improving the quality of latent rep- resentations is of high importance for model-based reinforcement learning agents, where accurate predictions of the future lead to better planning and offline credit assignment, while hierarchical and nested treatment of time could allow for temporally-abstrae reasoning. Nevertheless, one of the limitations facing VAE-base based models, and by extension our own, the lack of sharpness in the predictions. Though significant progress has been made in the recent years (Babaeizadeh al.) 2021 Wu et al., 2021). addressing this problem DLH can a significant next step for further improving the performance of the model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
ArrowGAN: Learning to generate videos by learning Arrow of Time Elsevier Enhanced Reader.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. FitVid: Overfitting in Pixel-Level Video Prediction, June 2021.
Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. 2017 IEEE International Conference on Robolics and Automation (ICRA), pp. 173- 180, 2017.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved Conditional VRNNs for Video Prediction, April 2019.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. ArXiv, abs/2011. 1. 10650 2021.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder for statistical machine translation. In EMNLP, 2014.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hicrarchical multiscale recurrent neural net- works. In International Conference on Learning Representations, ICLR, 2017.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Datasets, September 2019.
Adversarial
Video Generation Complex
Emily Denton and Rob Fergus. Stochastic Video Generation with Learned Prior, March 2018.
Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, J. Lee, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. ArXiv,
2016.
M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360(6394):1204-DB 1210, June 2018. doi: 10.1126/ ac0
Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Christo- pher C. Holmes. Multi-facet clustering variational autoencoders. ArXiv, abs/2 106.05246. 2021.
Chelsea Finn, Ian J. Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. ArXiv, abs/1605.00757, 2016.
Zafeirios Fountas, Anastasia Sylaidi, Kyriacos Nikiforou, Anil K. Seth, Murray Shanahan, and War- rick Roseboom. A Predictive Processing Model of Episodic Memory and Time Perception. Neural Computation, 34(7):5001-1544, 06 2022. ISSN 0899-7667. doi: 162/nco// a a_01514. URL _0154
httt::::/aol.orr/100.1111
Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic Latent Residual Video Prediction, August 2020.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, June 2014.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible Diffusion Modeling Long Videos, May 2022.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE. Learning Basic Visual Concepts with Constrained Variational Framework. In International Conference on Learning Representations, July 2022.
10

---

Under review as a conference paper at ICLR 2023
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion Models for Video Prediction and Infilling, August 2022.
Wei-Ning Hsu, Y. Zhang, Ron J. Weiss, H. Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Z. Chen, Jonathan Shen, P. Nguyen, and Ruoming Pang. Hierarchical generative modeling for controllable speech synthesis. ArXiv, abs/1810.07211, 2019.
Dinesh Jayaraman, Frederik Ebert, Alexei Efros, and Sergey Levine. Time-agnostic prediction: Predicting predictable video frames. arXiv preprint arXiv: 1808.077844 2018.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In IJCAI, 2017.
Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video Pixel Networks, October 2016.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 32: 12:11570-11579,220 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ (bb//141,,6980, 2015.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes, May 2014.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: execution. In International Conference on Machine Learning, pp. 3418-3428. PMLR, 2019.
Compositional imitation learning and
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic Adversarial Video Prediction, April 2018.
Sangmin Lee, Hak Gu Kim, Dae Hwi Choi, Hyung-ll Kim, and Yong Man Ro. Video predic- tion recalling long-term motion context via memory alignment learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In NewrIPS, 2019.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, February 2016.
Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. arXiv preprint arXiv: (et::000500833 08639, 2017.
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adap- tive skip intervals: Temporal abstraction for recurrent dynamical models. arXiv preprint arXiv:1808.044 04768, 2018.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-Conditional Video Prediction using Deep Networks in Atari Games, December 2015.
Karl Pertsch, Olch Rybkin, Jingyun Yang, Shenghao Zhou, Konstantinos Derpanis, Kostas Dani- ilidis, Joseph Lim, and Andrew Jaegle. Keyframing the future: Keyframe discovery for visual prediction and planning. Learning for Dynamics and Control, pp. 969-979. PMLR, 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent Video Transformer, June 2020.
Antti Rasmus, Mathias Berglund, M. Honkala, H. Valpola, and T. Raiko. Semi-supervised learning with ladder networks. In NIPS, 2015.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner. Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, and Nando de Freitas. Parallel Multiscale Autoregressive Density Estimation, March 2017.

---

Under review as a conference paper at ICLR 2023
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork Variational Autoencoders, February 2021.
C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: A local SVM approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. volume 3, pp. 32-36 Vol.3, August 2004. doi: 10.1
Wenling Shang, Alex Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Learning world graphs to accelerate hierarchical reinforcement learning. arXiv preprint arXiv: 2019.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NIPS, 2016.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised Learning of Video Representations using LSTMs, January 2016.
Arash Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. ArXiv, abs/2007.0380 008998, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need, December 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lee. High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks, November 2019.
Carl Vondriek and Antonio Torralba. Generating the future with adversarial transformers. 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2992-3000, 2017.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling Autoregressive Video Models, February 2020.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction, June 2021.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. using VQ-VAE and Transformers, September 2021.
VideoGPT: Video Generation
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion Probabilistic Modeling for Video Generation, May 2022.
Alexey Zakharov, Matthew Crosby, and Zafeirios Fountas. Episodic memory for subjective- timescale models. In ICMI. 202/ Workshop Unsupervised Reinforcement Learning, 2021.
Alexey Zakharov, Qinghai Guo, and Zafeirios Fountas. Variational predictive routing with nested subjective timescales. In International Conference on Learning Representations, 2022. URL
httt:s/s.geeeereee..eeee/eeeeatt net/ fott://oorunmllddd-drrbbbbb--
12

---

Under review as a conference paper at ICLR 2023
A TRAINING AND ARCHITECTURAL DETAILS
A.1 IMPLEMENTATION AND TRAINING
For all datasets, DLH was trained with video sequences of length 100 and batch 100. We used Adam optimiser (Kingma & Ba 2013) with learning rate of 0.0005 and 0.0001. We also find that it is beneficial to multiply the KL loss with parameter 3. which is slowly increased to the value of 1.0 the course of the first 10k training iterations. This promotes the model to learn good reconstructions before being severely restricted in increasing its latent complexity. The image reconstruction model predicts the means of a Gaussian with fixed standard deviation, which is optimised separately for each dataset.
For this paper, we used three-level DLH for all datasets (except for Moving Ball, where we used just two levels). Each level possesses deterministic variables, for which we set |." |d") Ic"l 200, and random variables, where |s" 20. For encoding and decoding image data, we use 4 convolutional and 4 transposed-convolttine details of all other model components, including the sizes of neural networks. The total number of parameters of three-level DLH is 7M, and its training takes ~3 days single Tesla V100.
layers, respectively. Table shows the architectural
Table 4: Architecture of models from Eq 10 15
Model
Hidden layers
Encoder, f (xx?)
MLP [200]
Decoder, fice (s," Ct)
MLP [200]
Temporal, f (s)) d
GRU [200]
Posterior, 9 (s," [. s 2n))
MIP [40, 40, 40]
Prior state, po (s) S7
MI.P [40, 40, 40)
Prior factor, (ee S
GRU [200]
A.2 PSEUDOCODE
Algorithm 1: Pseudocode of DLH
for (oirh) }; do
for t 1 to 7' do
for to N do
> Compute the log- likelihoodd ratio under approximate posterior
=0))"")""""))0"""" |s,""" 0r) using
DKLla (s)) (s, ee, 1)] and (s" le" 0)].
& Approximate the posterior q(c"))] [e] using Eq, and the nested timescales constraint, q(e," = 1 = = 0.
ir q(e")) then
b Approximate q(e?" = 11.) = and q((s?") + d((s?"))
> K < break
for 72 K to / do
D Compute the state posterior qu (s] oz)
Compute the ELBO (Eq. 8) using the inferred posteriors qu (5.))))()))
& Apply gradient step on 0, t.
13

---

Under review as a conference paper at ICLR 2023
B APPROXIMATION OF THE POSTERIOR COMPONENT OF MoG
B.1 EXPECTED LOG-LIKELIHOOD RATIO
In Section 2.2 we propose an approximation of the posterior q(e;) using the expected log likelihhoddd ratio in Eq. We can arrive at this formulation via either (a) considering the log-likelihood ratio of the two components of the MoG, or (b) the VaDE trick from the Gaussian Mixture VAE literature (Appendix B.3 In this section, we briefly explain the former perspective.
We start by considering the two components in the prior MoG as competing models. Their log- likelihood ratio is defined as,
p' ('( = 0) = Σ log p( e" = =)) i= p/ (s e" -
D
p'(s, e = =0)
log A(s") = log II
(16)
(s"), D is the number of posterior samples, and A(s)) = I d'6""""=0)
where ST
p (b)== By
applying the Law of Large numbers, taking D 20, we can write this as an expectation with respect to the posterior q' (s").
Ellog A(s")] (sg log l' =0) e' = 1)'
(17)
where the static and prior components, p'(s" le" 0) and p' (s" e" 1), are viewed as the compet- ing models under the inferred posterior q' ). This expected likelihood ratio can be computed in terms of KL divergences, if add and subtract the entropy of g' (s
Ellog As = (sr) logp'(6 (s" e" = 0) (sy) log (s")
(18)
(sg(ss)) logp' (s" = 1) + (or) log (8")
(19)
DKLld'(s" IIp') (s" =1]] DKLld (5'" (ll''s"" 0)]
(20)
Assuming minimum probability of error test, the selection of the most likely component realised
via
Dxt.ld(s")(l''(""(" 1)] (s 0]]
$ 0.
(21)
e"=0
B.2 EXPECTED LOG-LIKELIHOOD RATIO WITH HIERARCHICAL DEPENDENCIES
DLH, posteriors are factorised hierarchically, such that
N
(5.)) = II
go (s,1s"" o.).
(22)
==
In turn, this implies that the expected log- likelihhkhoo ratio must be computed with respect to the hierarchical posteriors,
p'(s" e" 0)
E Elog A(s")] By(,,) log
(23)
p/ (s e
(s Egg log p'(s" e 0) p'(s" e" 1)
(24)
bicrarchical context
However, this estimation implies that all hierarchical posteriors, g ()(s?"), must first be inferred in top-down process, which may be computationally expensive, especially for large N. To resolve this, approximate the hierarchical posteriors, above the level which the posterior component q(e") is being estimated, using the static priors,
N
N
II j=
p ('s 0) = II d' (((s
(25)
j=++1
14

---

Under review as a conference paper at ICLR 2023
which are already known. This simple assumption alleviates the need infer all hierarchical poste- riors before the estimation g(e") and has been shown to work well in practice.
More specifically, the computational savings come from the combination of two factors: (1) approx- imating hicrarchical context using Eq. 25 during the inference of q(ee)). and (2) the nested timescales assumption in Eq. that blocks any further inference beyond the level which e, 0. As such, the computations pertaining to the inference procedure are required only up to some level k, where ef 0 is first inferred.
B.3 RELATIONSHIP TO VADE TRICK
VaDE trick is non paraamamttric technique for estimating posterior p(els) in Gaussian Mixture VAE models (Jiang et al. 2017 Falck et al., 2021). In particular, Falck et al. (2021) proved that an approximate posterior distribution ((e) that minimises the KL divergence to the true posterior ((els) will take the form,
exp(Bg(s) log p(els))
arg arg min DKL (p(ee(e)) Eeer exp(E,)s) log p(els))
(26)
((e)
where K E is the number of components in Gaussian mixture p(s, e), and (()) is an approxi- mate posterior over s.
In DLH, the number of components in MoG is limited K = Using Eq. 26 can compute
the posterior odds and solve for one of the two components (e.g. q(( = 0)),
g(e = 0) exp(Eg(s) logple = 0|s))
(27)
g(e 1)
(s))) log p(e 1[s))
exp
Fys)) log p(e 0/s) (yy(s) logp(e 1|s)
(28)
Ey(s) log plse 0)p(e 0)
(usepeyep Eyss) log p(s/e = 1)p(e 1) [uerp((e))p(t)
exp
(29)
exp
Ey(s) log p(s)e 0)p(c = 0) Egg(s) log = 1)p(e = 1)
(30)
= exp Eg(s) log p(s|e = 0) Eyis) logp(s|e 1) log p(e 0)
(31)
p'e = 1)
Adding and subtracting gg()) log (s) inside the exponent,
= =exp [RLlls())ll(sse= 1)] Dwlla(s))ll(ssttt 0)] + log ple = 0) p(e 1)
(32)
Finally, we use the fact that EoEF q(e) to get q(== 0),
g(e- 0)
q exp (kill(s))lp(sle= 1)] Dxslg())ll(sssee 0)] + log ple = 0) p(e 1)
(33)
and solving for q(e 0) yields,
exp (KLl 1)] D 0j] + log p)e=0)
p(e=1)
q( = 0) + exp ( 1)] Drala(s)llosss = 0)) + log p(e=0)
(34)
p(e=
Equation 34 shows the formulation of distribution q(e) in terms of the VaDE trick. We note that
under the assumption that p(e = 0) = p(c = 1) = 0.5, the formulation is softmax function of the difference between the KL divergences indicated in Eq. 4 As such, the proposed estimation of g(e) may be seen as taking a sample from the most likely component in the VaDE-based distribution in Eq. 34 under the assumption of equal prior probabilities.
15

---

Under review as a conference paper at ICLR 2023
C
EXTRA RESULTS
C.1
QUALITATIVE COMPARISONS
GT
DLH
+
8
8
8
CWW.E
8
e
LMC-M
8
VTA
O
9
6
60
GT
7
7
?
?
7
7
17
7
7
7
7
7,
7
7
77
CWVAE
7
7
17
17
P
3
7
3
17
h
5
LMC-M
7
17
VTA
7
7
2
30
256
285
280
Figure 7: Qualitative comparisons of open-loop predictions using Moving MNIST. All models given 30 context frames (36 for CW-VAE). The figures show the first 50 and last 45 timesteps in the produced rollouts. The context frames shown here. GT denotes the ground-truth sequence.
GT
T
T
DLH
CWWEE
LMC-M
7
DLH
1
CWVAE
LMC-M
Y
T
VTA
1
5
10
15
20
25
30
35
40
45
50
155
160
165
170
175
180
185
190
195
200
Figure 8: Qualitative comparisons of open-loop predictions using KTH. All models are given 30 context frames (36 for CW-VAE, The figures show the first 50 and last 45 timesteps in the produced rollouts. The context frames are not shown here. GT denotes the ground-truth sequence.
16

---

Under review as a conference paper at ICLR 2023
C.2
PER-FRAME PREDICTION EVALUATION
Moving MNIST SSIM
Moving MNIST PSNR
Moving MNIST LPIPS
1.0
20
0.5
55TH
0
PSNR
26
LPDS
0.2
0.5
0.1
50
200
250
300
Timesteps
Timestegs
Timesteps
KTH SSIM
KTH PSNR
KTH LPIPS
0.5
0
28
55T
PSWR
0.8
20
0.2
07
50
16
200
250
300
0.1
250
Timestops
DLH
LMC-Memory
CWVAE
VTA
Figure 9: Per-frame video prediction performance against the benchmarks on the Moving MNIST and KTTH datasets. They y-axes correspond to the SSIM1, PSNR1, and LPIPS metrics. The x-axes correspond to the number of open-loop prediction steps.
C.3 SELECTION OF THE NUMBER OF LEVELS
We demonstrate in detail the ability of DLH converge its structure and employ only the necessary amount of resources in processing data. In Section 2.3, using the Moving Ball dataset, we showed that irrespective of the number of levels the model possesses, only partially uses its latent hierarchy. In particular, we reported two metrics: the average number of levels used by the model and the average total KL divergence. For both metrics, we reported similar values across the instances of DLH with different total number of levels.
Table 5: Per-level KL divergences of DLH (Moving Ball dataset)
Levels
KL (LI)
KL (L2)
KL (L3)
KL (L4)
KL (L5)
35.1
6.8
3
34.7
7.8
0.3
34.3
6.5
0.2
0.1
36.1
6.6
0.1
0.1
0.2
Notably, the KI. divergence often used measure the amount of information stored in the latent variables of the model. By breaking down the contributions of each latent level the total value the KL loss, we can get glimpse into how DLH employs the different hicrarchical levels. Table (5 shows the average per-level KL loss of DLH. As be observed, levels > are signifi- cantly lower, suggesting that no information is being stored in those levels, as the model naturally "collapses' them. More visually,
can sample from these low-KL levels (> 2), while keeping KL levels fixed (S 2), and vice versa, in order to see if these levels contribute to the variations in the reconstructed images. Figures 10 and IT show that, indeed, samples from the low-KL. levels exhibit minimum variations, while high-KL levels seem to encode most of the important informa- tion. These results once again illustrate how DLH naturally tends to use the minimum amount of resources necessary for modelling the data. Furthermore, this property hints at a potential method for selecting the appropriate number of levels by monitoring the values of the KL. for models with different number of hierarchical levels.
17

---

Under review as a conference paper at ICLR 2023
(a) 3-Ievel DLH Samples from levels: Fixed levels: 1,2
(b) 4-level DLH Samples from levels: 3, Fixed levels: 1,2
(c) 5-level DLH Samples from levels: 5
Fixed levels: 1,
Figure 10: Samples from levels As be seen, there reconstructions, indicating that no information stored in these levels. This is corroborated by the low values of the KL divergence associated with the hierarchical levels.
variations in the observed
(a) 3-1evel DLH Samples from levels: 1, 2
(b) 4-level DLH Samples from levels: 1, 2 Fixed levels: 3,4
(c) 5-level DLH Samples from levels: 1, Fixed levels: 3, 4,
Fixed levels: 3
observe high variation in the reconstructed images, indicating that the data is primarily modelled by the bottom two levels. This is in line with the high values of the KL divergences associated with these levels.
Figure 11: Samples from levels 2. In contrast samples in Figure 10
18


Reviews:
[{"review_id": "xijN3NFPyC", "reviewer": "Reviewer_2ayA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for hierarchical representation learning of spatiotemporal features in long-term video prediction. The proposed method is called: Dynamic Latent Hierarchy (DLH). The method distinguishes between features that are changing and those that are not changing in the video sequence. DLH is able to handle multiple objects moving at different speeds and differentiate moving objects from a static environment. The advantages of DLH include: long-term video prediction, improved modeling of stochasticity, and dynamic, efficient latent structure. DLH outperforms baseline approaches on Moving MNIST, KTH Action, and DML Mazes datasets. ", "summary_of_the_review": "Overall, this is a good, clearly written paper that proposes a reasonable approach for video prediction that outperforms the considered baselines across several datasets. I am currently inclined to accept it. I encourage the authors address the first two weaknesses listed above, in particular, regarding the complexity of the datasets used for evaluation. ", "strength_and_weaknesses": "Strengths:\n* I really enjoyed reading this paper. It is well written and the ideas are easy to follow. \n* The problem is well-motivated and the literature review does a good job at contextualizing the paper in prior work.\n* The methods section was presented with intuition for the design choices and notation. This intuition was then supported empirically.\n* An attempt was made to interpret the hierarchical levels semantically in the experiments.\n* Given the considered datasets, the experiments section is constructed thoughtfully and the results are convincing.\n* The figures are informative and effectively illustrate the benefits of the proposed approach. Particularly, the intuition provided by the results in Fig. 5 was helpful in understanding how the method works.\n\nWeaknesses:\n* Overall, the datasets considered are fairly uncluttered and simplistic. The video examples do not highlight the capabilities of DLH to handle many objects at many different speeds (e.g., in crowded urban scenes). The data also does not showcase what happens when the background is not stationary and there are moving objects. I would recommend considering a more complicated, dynamic dataset, for example, from the autonomous driving setting (i.e., Waymo Open Dataset [1], NuScenes [2], or KITTI [3]). The simplicity of the toy Moving Ball dataset is also underscored by the results of Table 2, where the full capacity of the hierarchical model is not necessary to model the data. Although it is great to see that the model can dynamically adapt to use less of the latent space when the underlying data distribution is simpler, it would be compelling to see how the full latent space would be used in a more complex setting.\n* Although there is a discussion on the importance of stochasticity, this capability is only explored in a toy-dataset setting with random color changes. No discussion of multimodality in the distribution is included. Could DHL handle multimodal outputs (e.g., multiple equally valid possibilities for the future)? This would again be relevant to more complex video datasets (e.g., in the urban setting), where given observations of a person walking straight, they could choose to continue walking straight or turn in the future.\n* It would be helpful in Fig. 4 or in an appendix to show the outputs of the baseline approaches for comparison.\n* In Fig. 5, it is not entirely clear that levels 1 and 2 for the KTH Action dataset are disentangled.\n* It would be interesting to benchmark against a deterministic video prediction method in Table 1 to see if the considered datasets are sufficiently stochastic to warrant modeling of stochasticity.\n* Is there a way to report a measure of statistic significance of the proposed method's metric performance over the baselines in Table 1?\n* In Fig. 6, it is not very clear to me what is wrong with some of the highlighted frames output by the CW-VAE. Is the issue that the ball reduces in size for those frames? Why were the other baseline results not shown?\n* Further explanation for the values in Table 3 would be helpful.\n\n[1] Sun, Pei, et al. \"Scalability in perception for autonomous driving: Waymo open dataset.\" CVPR, 2020.\n\n[2] Caesar, Holger, et al. \"nuScenes: A multimodal dataset for autonomous driving.\" CVPR, 2020.\n\n[3] Geiger, Andreas, et al. \"Vision meets robotics: The KITTI dataset.\" IJRR, 2013.\n\nSome typos and minor points of confusion are listed below:\n\n1. I am not sure I fully followed the diagrams in Fig. 2. Are there temporal indices missing from the states?\n2. Are there hierarchical levels and $\\psi$ parameters missing in the 'Estimating' paragraphs in Sec. 2.2 and similarly dropped indexes in $p(e \\mid s)$ in the paragraph before Eq. 3?\n3. In Eq. 3, $\\Lambda$ is not defined.\n4. Missing period at the end of Eq. 3.\n5. In Sec. 2.3, I am having some trouble understanding the notation. Should $q(\\bf{e}^{n+1} \\mid \\bf{e}^n = 0) = 0$ be $q(\\bf{e}^{n+1} = 1 \\mid \\bf{e}^n = 0) = 0$?\n6. It would be helpful to derive Eq. 8 from Eq. 7 in the appendix for completeness.\n7. Fig. 3 was referenced much earlier than it appears.\n8. Am I correct in understanding that at the first hierarchical level $e^1 = 1$ always?\n9. In Fig. 4, it should be made clear whether the 30 past context frames are included in the visualization or only the 100 predicted frames are shown.\n10. In Sec. 3, temporal abstraction paragraph, the sentence \"Temporal abstraction models ...\" has a grammatical typo, is a bit long, and is missing a period at the end.\n11. Missing periods at the end of table captions. Unclear what the * symbol signifies.\n12. In Sec. 4.3, I did not fully understand what it means that \"DLH learns transition between progressively slower features in the higher levels of its hierarchy\". Does this mean that minor variations in the scene are faster features than location changes of the view?\n13. The references should be proofread (e.g., to ensure the year is not entered twice in a citation, the conference venue is listed instead of ArXiv when available, the confererence name formatting is consistent, etc.).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kx69eZU1POP", "reviewer": "Reviewer_1qre", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new video prediction model in the form of neural networks, which combines the advantages of hierarchical clockwork models (from CW-VAE) and jumpy recurrent modeling controlled by binary indicators (from VTA). \n- Major: It provides a novel graphical model for spatiotemporal prediction. To avoid the unnecessary accumulation of prediction errors and improve computational efficiency, the authors propose to disentangle the temporal dynamics into hierarchical latent states and predict whether the states would change or remain static at each future timestep. \n- Minor: The proposed model is shown effective on four datasets, improving previous benchmarks including CW-VAE and VTA.", "summary_of_the_review": "Overall, it is a good paper that improves the existing benchmarks on multiple video prediction datasets. However, it needs significant improvement, specifically in the following aspects: \n1. The technical contribution of the paper. Although the proposed approach is new in terms of the graphical model shown in Fig 1, the key insight of combining hierarchical latent representations and the jumpy state transitions for modeling video dynamics adaptively at different timescales has been explored by existing work with similar techniques (using binary indicators for the state changes). \n2. More explanation of the connection between the motivations shown in Fig 1 and the proposed approach.\n3. Additional experiments on model efficiency and more qualitative comparisons for long-term prediction should be included to understand the effectiveness of the approach.", "strength_and_weaknesses": "Strength: \n1. This paper is clearly written and easy to follow. \n2. By considering modular representations, this paper has proposed a simple yet effective hierarchical framework to model the mixed spatiotemporal dynamics in video sequence at different timescales, and also achieves great quantitative results.\n3. Detailed validations are performed on multiple datasets. In addition to the significant improvement in SSIM/PSNR, in Fig 5, the authors also tried to demonstrate the advantages of the proposed method in decoupling different spatiotemporal dynamics from the qualitative visualization.\n\nWeaknesses:\n1. My main concern lies in the novelty of the proposed approach, which seems to be a simple extension of the hierarchical model from Zakharov et al. (2022). If I understand correctly, the biggest difference between them is the binary indicator e_t^n, which is used to detect event (change or static). However, similar techniques has been well explored by Kim et al. (2019). The technical contributions of the model are not convincing to me and require further clarifications. \n2. I also have some doubts about the connections between the example explained in Fig 1 and the method of \"nested timescales\" proposed in Sec 2.3. I understand that the authors attempted to describe the hierarchical state organization, but in Fig 1, it seems that the dynamics of the panda has nothing to do with that of the airplane. Therefore, if Fig 1 correctly states the motivation of the method, it should learn the dynamics at different timescales in parallel state transition branches, instead of the hierarchical one. In other words, in Eq (5), the indicator of state changes at layer n (e_t^n=0) should not be completely determined by e_t^{n-1}=0. \n3. Although in Sec 2.3, the method is claimed to reduce the computational complexity, no corresponding empirical evidence was given in Sec 4 that could support the efficiency of the proposed model. \n4. In my view, the numbers of state levels that are used to model complex scenes (such as KTH or even datasets with more complex visual dynamics) and simple scenes (such as Moving MNIST) may be significantly different. How should the hierarchies be determined for different scenarios? In other words, how many levels of latent states should be selected for different datasets? As shown in Table 1 and Fig 5 for selecting 3 levels throughout the first three datasets, can the conclusions be extended to general scenarios? If not, can the authors provide some reference schemes for selecting the number of levels. Besides, it would improve the fairness of the comparison to implement a VTA model that also employs a 3-level architecture with similar number of parameters. \n5. What does * indicate in Table 1?\n6. In Figs 4-5, I strongly encourage the authors to include more qualitative comparisons for long-term video prediction with existing benchmarks. The proposed method is expected to generate future frames with higher visualization quality. In addition, it would be nice if the authors could give clearer textual explanations in Figs 4-5. I am confused on what each image sequence represents...Are those in the first row the ground-truth images, or does each row represent a different prediction sequence by the model based on the same input sequence? \n\nOther suggestions (NOT weaknesses):\n1. If the experimental conditions allow, it is essential to show the model performance on more real-world datasets with large spatiotemporal uncertainty, such as RoboNet or BAIR. The authors may consider the action-conditioned video prediction setup because it would be interesting to analyze the correspondence between the learned indicators and the given action signals. \n2. In addition to using the best prediction among 100 prediction samples, the authors may evaluate the models using the average performance of the 100 samples as well as the worst cases.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fmaJz23DwJN", "reviewer": "Reviewer_uT2U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a hierarchical VAE for video prediction. The latent structure of the model is formulated as a mixture of gaussians. The authors show that the proposed model is competitive to some previous hierarchical video VAE models on relatively simple datasets.", "summary_of_the_review": "Overall the method presented in this paper is not that different from other previous hierarchical VAE models, and while it shows some improvements over these methods, in general it is unclear if this model will be of interest to the community compared to newer methods (FitVid, diffusion video models) that can generate long video sequences on complex datasets and have simpler latent spaces. ", "strength_and_weaknesses": "**Strengths:**\n\n[+] Clear method formulation and good empirical results on relatively simple datasets.\n\n\n**Weaknesses:**\n\n[-] The model is fairly similar to previous approaches that use hierarchical latent variables in time and/or space.\n[-] Results are on relatively simple datasets, with newer video prediction methods being capable of generating sequences for long horizons on more complex datasets (e.g. Kinetics)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xv0rxHYRcY", "reviewer": "Reviewer_UWJy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Dynamic Latent Hierarchy, a latent-variable video prediction model. This model can learn a hierarchy of latent variables where the latent variable at each level of the hierarchy, operates at a different timescale. Unlike related work, ClockWork VAE’s, (Saxena et al. 2021), whether or not a latent variable at a certain level is activated is not hard coded. Instead, the paper proposes to cast this as a Bernoulli latent variable and learn it using variational inference.\n\nImprovements are shown on KTH, DML Mazes and Moving Mnist on SSIM and PSNR over 3 competing methods. Qualitiative experiments on KTH show that the higher levels get activated lesser as compared to the lower levels. Similarly, on a synthetic moving ball dataset, when the color is changed stochastically, on higher levels of stochasticity, the emission probability is higher.\n", "summary_of_the_review": "I've provided detailed feedback above with respect to my concerns regarding the paper. I'm open to adjusting my rating if authors can clarify my concerns.", "strength_and_weaknesses": "**Strengths**\n\nThe problem is well motivated as stated in the introduction. In a video, it is expected that latent variables operate at different timescales. Providing this as an inductive bias in video modeling, could help in video prediction and could lead to interesting insights.\n\n**Weaknesses**\n\nI have some concerns with the novelty, clarity and experiments in the paper. See below for detailed feedback and for actionable suggestions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject