Video Prediction with Variational Temporal Hierarchies
Authors: Vaibhav Saxena
Jimmy Ba
Danijar Hafner
Venue:  (2021)
Keywords: latent dynamics
temporal abstraction
video prediction
probabilistic modeling
variational inference
deep learning

Abstract: Deep learning has shown promise for accurately predicting high-dimensional video sequences. Existing video prediction models succeeded in generating sharp but often short video sequences. Toward improving long-term video prediction, we study hierarchical latent variable models with levels that process at different time scales. To gain insights into the representations of such models, we study the information stored at each level of the hierarchy via the KL divergence, predictive entropy, datasets of varying speed, and generative distributions. Our analysis confirms that faster changing details are generally captured by lower levels, while slower changing facts are remembered by higher levels. On synthetic datasets where common methods fail after 25 frames, we show that temporally abstract latent variable models can make accurate predictions for up to 200 frames.

Under review as a conference paper at ICLR 2021
VIDEO PREDICTION WITH VARIATIONAL TEMPORAL HIERARCHIES
Anonymous authors
Paper under double-blind review
Abstract
Deep learning has shown promise for accurately predicting high-dimensional video sequences. Existing video prediction models succeeded in generating sharp but often short video sequences. Toward improving long-term video prediction, we study hierarchical latent variable models with levels that process at different time scales. To gain insights into the representations of such models, we study the information stored at each level of the hierarchy via the KI. divergence, predictive entropy, datasets of varying speed, and generative distributions. Our analysis confirms that faster changing details are generally captured by lower levels, while slower changing facts are remembered by higher levels. On synthetic datasets where common methods fail after 25 frames, we show that temporally abstract latent variable models can make accurate predictions for up 200 frames.
INTRODUCTION
Deep learning has enabled predicting video sequences from large datasets (Chiappa et al., 2017; Oh et al., 2015; Vondrick et al., 2016). For high-dimensional inputs such as video, there likely exists a more compact representation of the scene that facilitates long term prediction. Instead of learning dynamics in pixel space, latent dynamics models predict ahead in a more compact feature space (Doerr Buesing al., 2018; Karl et al., 2016; Hafner et al., 2019). This has the added benefit of increased computational efficiency and lower memory footprint, allowing to predict thousands of sequences in parallel using a large batch size.
Moving MNIST Open-loop Prediction
level fector
lavel fodtor
Shuctura Simteaty b3/00
level foitor
RSSM
SVG4P
0.75
rondom
0.70
al., 2018;
0.85
0.60
Figure 1: Mean SSIM over test set of 100 sequences of open-loop prediction with Moving MNIST. All 3-level latent dynamics models, with temporal abstrac- tion factors 4, 6, and have the same number of model parameters.
lot of work in deep learning has focused on spatial ab- straction, following the advent of convolutional networks (LeCun al., 1989), such as the Variational Ladder Au- toencoder (Zhao al., 2017) that learns a hierarchy of features in images using networks of different capacities, along with playing important role in the realm of video prediction models (Castrejón et al., 2019). Recent sequential models have incorporated temporal abstraction for learning dependencies in temporally distant observations (Koutník al., 2014; Chung et al., 20.6). Kim et al. (2019) proposed Variational Temporal Abstraction (VTA), in which they explored one level of temporal abstraction above the latent states, the transition of which was modeled using a Bernoulli random variable. In this paper, we intend to work in more controlled setup than VTA for a qualitative and quantitative analysis of temporally abstract latent variable models.
In this paper, we study the benefits of temporal abstraction using hierarchical latent dynamics model, trained using a variational objective. Each level in the hierarchy of this model temporally abstracts the level below by an adjustable factor. This model can perform long-horizon video prediction of 200 frames, while predicting accurate low-level information for a times longer duration than the baseline model. We study the information stored at different levels of the hierarchy via KL divergence, predictive entropy. datasets of varying speeds, and generative distributions. In our experiments show that this amounts to object location and identities for the Moving MNIST dataset, and the wall or floor patterns for the GQN mazes dataset (Eslami al., 201 8), stored at different levels.

---

Under review as a conference paper at ICLR 2021
90
120
150
180
210
240
270
300
330
360
390
420
Figure 2: Long-horizon open-loop video prediction for the MineRL. Navigate dataset (Guss et al., 2019), using our 3-level TALD model with temporal abstraction factor 6, compared with VTA, RSSM, and SVG-LP. We observe that TALD accurately predicts the movement of the scene from the ocean to the forest, and maintains that context until 420 frames. However, VTA predicts implausible scene after 240 steps with blue and black skies in the same frame. RSSM predicts plausible future as well where the player stays in the ocean the distant forest moves out of the scene, whereas SVG-LP learns to copy the initial frame indefinitely and does not predict any new events in the future.
Our key contributions summarized as follows:
Temporal A bstract Latent Dynamics (TALD) We introduce a simple model with different
clock speeds at every level to study the properties of variational hierarchical dynamics.
Accurate long-term predictions Our form of temporal abstraction substantially improves for how long the model can accurately predict video frames into the future.
Adaptation to sequence speed We demonstrate that our model automatically adapts the amount of information processed at each level to the speed of the video sequence.
Separation of information We visualize the content represented at each level of the hierarchy find location information in lower levels and object identity in higher levels.
2 RELATED WORK
Generative video models A variety of methods have successfully approached video prediction using large datasets (Chiappa et al., 2017; Oh et al., 2015; Vondrick et al., 2016; Babaeizadeh et al., 2017: Gemici et al., 2017; Ha & Schmidhuber, 2018). Denton & Fergus (2018) proposed stochastic video generation model with learned prior that transitions in time, and is conditioned past observations. Lee et al. (2018) proposed to use an adversarial loss with a variational latent variable model produce naturalistic images, while Kumar et al. (2019) used flow-based generative modeling to directly optimize the likelihood of a video generation model. Recently, Weissenborn et al. (2020) scaled autoregressive models for video prediction using three-dimensional self-attention mechanism and showed competitive results on real-world video datasets. On similar lines, Xu et al. (2018) proposed use an entirely CNN-based architecture for modeling dependencies between sequential inputs.
Latent dynamics models Latent dynamics models have evolved from latent space models that had access to low-dimensional features (Deisenroth & Rasmussen, 2011; Higuera et al., 2018). 8). to models that can build a compact representation of visual scenes and facilitate video prediction purely in the latent space (Doerr et al., 2018; Buesing et al., 2018; Karl et al., 2016; Franceschi et al., 2020). The Variational RNN (Chung al., 2015) uses auto-regressive state transition that takes inputs from observations. making it computationally expensive to be used as an imagination module. Hafner al. (2019) proposed latent dynamics model, which is combination of deterministic and stochastic states, that enables the model to deterministically remember all previous states and filter that information obtain a distribution over the current state.
Hierarchical latent variables Learning per-frame hierarchical structures has proven to be helpful in generating videos on long-term horizon (Wichers et al., 2018). Zhao et al. (2017) proposed the Variational Ladder Autoencoder (VLAE) that uses networks of different capacities at different levels of the hierarchy, encouraging the model to store high-level image features the top level, and simple features at the bottom. Other recently proposed hierarchical models use a purely bottom- up inference approach with interaction between the inference and generative models (Kingma & Welling,
2

---

Under review as a conference paper at ICLR 2021
Second
Lovel
top-down modulation
Find Level
temporal
temperal
Vileo
Detta
bottom-up observations
672
Connection used for both prtor and posterior
Connection only used for posterior (given data)
Figure 3: Temporally Abstract Latent Dynamics (TALD). Left is the structure of our recurrent model, in which each latent state sf in the second level abstracts two latent states in the first level. The solid arrows represent the generative model, while both solid and broken arrows comprise the inference model. On the right, we illustrate the internal components of the state variable, which comprises deterministic state hú and stochastic state 2t The deterministic state processes all contextual information and passes it to the stochastic state to used for either generation or inference.
2014; Rezende Mohamed, 2015; Rezende al., 2014). contrast, Sønderby et al. (2016, LVAE) and Vahdat & Kautz (2020, NVAE) proposed to use combination of bottom-up and top-down inference procedures, sharing parameters between the inference and generative distributions during the top-down pass. We incorporate this conditional structure in our model design well
Temporal abstraction Identifying complex dependencies between temporally distant observations challenging task and has inspired a variety of fundamental work recurrent models (Koutník et al., 2014; Chung et al., 2016). However, relatively few works have demonstrated modeling long- term dependencies using temporally abstract latent dynamics models (Wichers al., 2018; Jaderberg et al., 2018). Recently, Kim et al. (2019) introduced Variational Temporal Abstraction (VTA) to learn temporally abstract latent spaces. They explored one level of temporal abstraction above the latent states, the transition of which was modeled using Bernoulli random variable, that chose between "copy' or 'update" steps. Inspired by this work, we aim to gain a deeper understanding latent dynamics models. We perform our analysis simplified to using fixed time scales for every level. Moreover, the lower level is a continuing chain in our model, whereas VTA resets transitions lower level when transitioning higher level.
of such temporally-abstree
model that is
TEMPORALLY ABSTRACT LATENT DYNAMICS
Long video sequences contain both information that is local to few frames as well as global information that is shared among many frames. Traditional video prediction models that prediet ahead at the frame rate of the video can struggle to retain information long enough to learn such long- term dependencies. We introduce Temporally Abstract Latent Dynamics (TALD) learn long-term correlations of videos. Our model predicts ahead on multiple time scales to learn dependencies at different temporal levels, as visualized in Figure We build our work upon the recurrent state-space model (RSSM; Hafner et al., 2019), the details of which can be found in Appendix A.
TALD consists of hierarchy of recurrent latent variables, where each level transitions at different clock speed. We slow down the transitions exponentially as we go up in the hierarchy, i.e. every level being slower than the level below by a factor of k. We denote a set of active timesteps for every level I E [1, L] as those steps in time where the state transition generates a new latent state,
Active timesteps:
Ti (t E [1,T] mod k =
(1)
At each level, we condition every window of & latent states on a single latent variable in the level above. This can also be thought of hierarchy of latent variables where each level has the same clock speed, but performs a state transition every kd-1 timesteps and copies the same state variable otherwise, so that VL & Tr:
Inactive states:
Joint distribution We can factorize the joint distribution of a sequence of observations and (active) latents every level into terms: (1) decoder term conditioned on the latent states in the lowest
3

---

Under review as a conference paper at ICLR 2021
level, and (2) state transitions at all levels conditioned on the latent state of the last active timestep at the current level and the level above.
p s $ = [T- plze [ (IE Ien p(s) 1 s
(3)
Inference For inference, TALD embeds observed frames using a CNN. A hierarchical recurrent network then summarizes the input embeddings, for which each (active) latent state at a level / receives embeddings from k observation frames (dashed lines in Figure 3). The latent state the previous timestep at the current level, and the state belief at the level above also condition the posterior belief (solid lines in Figure 3). The input embeddings combined with this top-down and temporal context together condition the posterior belief of over the latent state.
Generation The prior transition p is computed by conditioning over the latent state at the previous timestep at the current level, and the state belief at the level above (solid lines in Figure 3).
Decoding Finally, the state beliefs at the bottom- level are decoded using a transposed CNN to provide training signal. To summarize, utilize the following components in model, E [1, L), E TT
Encoder:
= e(("aa+--1-11
Posterior transition df=
q(s; st ,e))
(4)
Prior transition p\: p(s) s/-11
Decoder:
p(az s.))
Training objective Since cannot compute the likelihood of the training data under the model
in closed form, we use the ELBO as our training objective. This training objective optimizes a reconstruction loss at the lowest level, and KL. regularizer every level in the hierarchy summed across active timesteps,
max ET [l: ||)] -L= EeET, KL pt]. e.i.g.p
(5)
The KI. regularizer at each level limits the amount of information that filters through the encoder and stays in the posterior at that level. This encourages the model to utilize the state transitions and context from the level above as much as possible. Since the number of active timesteps decreases as we go higher in the hierarchy, the number of KL terms per level decreases as well. Hence it is easier for the model to push global information high up in the hierarchy and pay lesser KL penalty, instead of transitioning those bits with an identity transformation at lower level.
Stochastic and Deterministic Path As illustrated in Figure 3 (right), we split the state sf into stochastic (=)) and deterministic (ht) parts (Hafner et al., 2019). The deterministic state is computed using the top-down and temporal context, which then conditions the stochastic state at that level.
The stochastic states follow a diagonal Gaussian, with mean and variance predicted by neural network. We use GRU (Cho et al., 2014) per level to update the deterministic state at every active timestep. All components in Equation are trained jointly by optimizing Equation using stochastic backpropagation with reparameterized sampling. Please refer to Appendix B for architectures details.
4 EXPERIMENTS
We aim to evaluate temporally-abstracc latent dynamics models modeling long-term depen- dencies in video. Moreover, we aim to under- stand how they separate information into dif- ferent levels of the hierarchy. To investigate these questions, we train TALD described in VTA model
Table 1: KL divergence (in bits) between the pos-
terior and prior at different levels of the hierarchy (summed in time). We observe that the amount of information at the (lowest) level decreases as the model gets deeper.
Section 3, the temporally-abstie (Kim et al., 2019), the RSSM model without temporal abstraction (Hafner et al., 2019), and the image-space video prediction model SVG- LP (Denton & Fergus, 2018) four datasets of varying complexity. We consider the well-
TALD
LEVEL
LEVEL
LEVEL
LEVELS
372.89
59.44
3.84
1.76e-4
LEVELS
529.04
65.33
9.07
LEVELS
561.79
56.52
635.51
established Moving MINST dataset (Srivastava et al., 2015), the KTH Action dataset (Schuldt et al.,
4

---

Under review as a conference paper at ICLR 2021
GF
D
/
@
6
/
SVG RSSM 31-H1 3L
01
O
6
O
O
/
D
6
1
O
O
9
P
D
01
6
OF
9
1
O
6
8
0
6
6
2
6
6
6
Z
6
3
8
0
8
0
3
9
9
9
7
t= 0 20 40
60
80
100
150
200
250
300
400
500
600
700
800
900
Figure 4: Long-horizon horizon open-loop prediction for Moving MNIST. (L for levels, for abstraction factor.) We illustrate samples from our TALD model with levels and temporal abstraction factors: 6 (3L-F6), and 1 (3L-F1) (i.e. no temporal abstraction). We compare those with samples from the RSSM and SVG-LP baselines. Red boxes show instances in time from where models lose accurate object identity. We observe that TALD, with abstraction factor 6, is able to maintain accurate long-term dependencies in the form of object identities for 900 frames into the future.
2004), the GQN mazes dataset (Eslami et al., 2018), and the MineRL Navigate dataset (Guss et al., 2019). We evaluate open-loop video predictions on these datasets using four quantitative metrics: Structural Similarity index (SSIM; higher is better), Peak Signal-to- to-oose Ratio (PSNR; higher is better). Learned Perceptual Image Patch Similarity (LPIPS; lower is better) (Zhang et al., 2018), and Frechet Video Distance (FVD; lower is better) (Unterthiner et al., 2018). Section 4.5, we investigate how the amount of information stored at different levels of temporal hierarchy adapts to changes in sequence speed. In Section 4.6, we visualize the information stored at different levels by resetting individual levels of the hierarchy.
We trained all our models using sequences length 100. We used convolutional frame encoders and decoders, with architectures very similar the DCGAN (Radford al., 2016) discriminator and generator, respectively. Our implementations made use of TensorFlow Probability (Dillon et al., 2017) and CuDNN, and used the Adam optimizer (Kingma & Ba, 2014) for training. The training time for a 3-level TALD model with temporal abstraction 6 amounted to around 24 hours for 100 epochs on single NVIDIA TITAN Xp GPU. Refer to Appendix C for hyperparameters and experimental setup.
4.1 MOVING MNIST DATASET
The Moving MNIST dataset consists of two digits moving in square with velocities sampled in range of to pixels per frame. We trained different versions of TALD with levels in the hierarchy and temporal abstraction factors 1, 2, 4. 6 and 8, all of which use the same number of model parameters. We compare samples of long- horizon open-loop video predictions of 900 frames with RSSM and SVG-LP in Figure 4. All samples were conditioned using posterior beliefs inferred after observing 36 context frames. Please refer to Appendix for more details and discussion.
TALD GI
SVG VIA
1
1
3
1=0
3
4
17
20
30
32
33
48
50
Figure 5: Open-loop video prediction for the KTH Action dataset. While TALD predicts accurately for 50 time frames, we observe jumpy transitions in VTA. where in this example the person disappears after the 17th frame. SVG predicts accurately for 18 frames, but starts to forget the task thereafter, as the person in the video starts move in the opposite direction.
5

---

Under review as a conference paper at ICLR 2021
SSIM on KTH Action dataset
PSNR on KTH Action dataset
LPIPS on KTH Action dataset
32.5
0.5
0.90
30.0
0.85
5SM
27.5
LPIPS
0.80
25.0
0.3
0.75
22.5
0.2
0.70
20.0
0
50
100
150
200
250
300
0
50
100
150
200
250
300
50
100
150
200
250
300
Distance Frames
Distance Frames
Distance Frames
TALD w/ levels, factor 6 (ours)
Baseline (no temp. abstraction) (ours)
RSSM
VIA
SVGLLP
Figure 6: Quantitative comparison between our temporally abstract latent dynamies model (TALD) and baselines over open-loop video prediction of 300 frames for the KTH Action dataset.
We observe that SVG typically forgets object identity within 50 timesteps, while TALD with abstrac- tion factor 6 maintains digit identity over 900 timesteps. RSSM clearly outperforms SVG, however starts to forget object identities after 250 time frames. TALD with abstraction factor (i.e. no temporal abstraction) also starts forget object identity after around 600 frames. With regards to the object positions, TALD with abstraction factor 6 predicts accurate digit positions until around 90 steps, and predicts a plausible sequence thereafter. RSSM and TALD without temporal abstraction predict the correct location of digits for at least as long as TALD with temporal abstraction. However, SVG starts lose track of positions much sooner. We also note that our predictions compared to those generated by SVG. Please refer to Appendix D for more experimental results.
bit blurry
We report the KL divergence value per level (summed over active time steps) for our TALD models in Table 1. Each value was obtained after training over sequences of length 100, for 200 epochs. The 2 and 3-level models were trained with temporal abstraction factor of 6, and the 4-level model with a factor of (to fit into memory). Figure 1 compares the Structural Similarity index (SSIM) for different versions of TALD with RSSM and SVG-LP. We note that SSIM decreases models with higher temporal abstractions. As a baseline, we compute SSIM between ground truths and random sequences from the training set. It is interesting to note that quality of video predictions from TALD stay better than random for a 6 times longer duration than SVG.
lower rate for
Adapting to Sequence Speed
4.2 KTH ACTION DATASET
102
We trained 3-level TALD model with temporal abstraction factor for the KTH Action dataset. In Figure 6, we report the SSIM, PSNR, and LPIPS, of TALD compared to SVG- LP. RSSM, and VTA. We also illustrate open-loop video predictions in Figure 5, conditioned using 36 context frames. While TALD predicts plausible frames for 50 timesteps, we observe jumpy transitions with VTA. probably because of breaks in the transition chain at the lower level. We also observe that SVG predicts accurately for 18 frames, while switching to a different task thereafter. We also note that SVG uses the DCGAN architecture for MNIST and the much larger VGG for KTH, whereas TALD works well even with the smaller DCGAN encoder/decoder. Refer to Appendix D for more results.
KL divergence
20'
30a
20-1
Level
10--
Level
9%.
10%
3.0
100%
500%
Speed of Noving Digits
Figure 8: KL divergence at each level of the TALD model (3 levels, tem- poral abstraction factor 6), trained slower/faster versions of Moving MNIST. Observe that the KL. term at higher levels decreases with in- crease in the speed of the digits, sug- gesting less global information being pushed up in the hierarchy.
4.3 GQN 3D MAZES DATASET
We trained 2-level TALD model with temporal abstraction
factor 6, and compared it with RSSM and VTA, the GQN mazes dataset. Figure 7 shows open-loop video prediction samples, conditioned using 36 context frames. We observe that while our model maintain global information of wall and floor colors for 200 frames, RSSM starts to forget the same after ~0 frames. Even though the open-loop predictions from TALD differ from ground truth in terms of camera viewpoints, the model does not forget the wall and floor patterns. We

---

Under review as a conference paper at ICLR 2021
GF
TALD
VIA
RSSM
0
3 4
20
40
60
80
100
120
140
160
180
200
Figure 7: Long horizon open-loop video prediction for the GQN mazes dataset (Eslami al., 2018) using our 2-1evel TALD model with temporal abstraction factor of 6. RSSM does not use temporal abstraction and thus starts to forget wall and floor patterns after frames, while TALD maintains this global information over 200 frames.
compare quantitative metries on this dataset Figure 12d. Please refer to Appendix for detailed experimental results.
Entropies (in different
Level 1 Lovel 2 Lovel
4.4 MINERL NAVIGATE DATASET
We trained 3-level TALD model with temporal abstraction factor 6, and compared it with RSSM, SVG-LP and VTA, on a Minecraft dataset (MineRI. Navigate) (Guss et al., 2019). This dataset fea- tures videos in a variety of world environments with complex moving backgrounds. We show long-horizon open-loop video predictions upto 420 frames (conditioned on 36 context frames) on this dataset in Figure We also compare quantitative metrics on this dataset in Figure 12c in Appendix D. Please refer to Figure 15 in Appendix for more open-loop video prediction examples.
Distance frames
(a) GQN mazes
Entropies (in nats) at di different levels
Level
aba
4.5 ADAPTING TO SEQUENCE SPEED
Level2
In order understand how model adapts changing temporal correlations in the dataset, we trained our model with slower and faster versions of moving MNIST, with speeds varied by factors of 3. For this experiment, our model consisted of 3 levels in the hierarchy, with each level temporally abstracting the level below by factor of 6.
Loqul)
Distance
(b) Moving MNIST
Figure 10: Entropy of the prior during open-loop video genera- tion at different levels of TALD (3 levels, temporal abstraction 2). (a) With GQN mazes, con- stant entropy at level (stored wall and floor patterns). (b) With Moving MNIST, constant entropy at level 2 (stored digit identities), and level 3 (suffered posterior collapse).
Figure 8 shows the KL divergence summed across the active timesteps at every level in the hierarchy. We observe that there is correlation between the KL divergence at every level and the speed at which the digits move. There is more information at level I when the digits move faster, and consequently lesser information at the levels above Also, even though the KI. divergence at level 3 is small, it still follows the same trend the other two levels. It is also important to note that the KL divergence between the prior and the posterior only an upper bound the information stored by the encoder in posterior belief state.
4.6 RESETTING INDIVIDUAL LEVELS
We visualize the information stored at a certain level by replacing the posterior belief at one level with the prior belief, i.e. all but one level receive observations (Zhao et al., 2017). Conditioned on those posterior beliefs, we sample open-loop video predictions using the trained prior model, which should show variations in the attributes learned that level. We expect model to store global information high up in the hierarchy, allowing the model to perform fewer transitions over that information, making it easier to pay less cost in the form of KI. divergence during training.
7

---

Under review as a conference paper at ICLR 2021
Ground sequence
Conditioned sample using all levels
Conditioned samples setting level-3 to prior
1=
2
3
10 15
20
25
30
35
40
45
50
Figure 9: Visualizing the information stored at the higher level of our 3-level model with temporal abstraction factor 2, using the GQN mazes dataset. We computed a posterior belief at each level using 8 observation frames, and which were then used to condition open-loop predictions. Changing wall textures show that they are stored the highest level.
of the levels the prior (by not feeding it with observations),
Ground truth sequence
0
4
0
4
Q
0
0
a
4
8
Conditioned samples using levels
0
9
9
9
4
4
9
0
0
9
9
0
0
40
4
0
0
0
4
4
%
0
0
6
0
4
8
9
9
0
Conditioned samples setting level- to prior
9
0
9
0
0
8
0
0
0
8
8
0
9
6
8
9
0
0 9
2
9
0 9
0
9
90
90
7
7
0
8
8
8
9
Conditioned samples setting level-2 prior
0
2
2
0
0
2
9
22
2
0
2
0
0.
0
2
9
8
2
2
6
6
6
9
8
3
6
5
6
3
6
3
3
3
3
6
3
6
3
4
20
30
40
60
70
80
100
Figure 11: Visualizing the information stored at different levels TALD with 3-levels and temporal abstraction factor 6, for Moving MNIST. We computed posterior belief at each level using 36 observation frames, while setting one of the levels to the prior (by not feeding with observations), which were then used condition open-loop predictions.
Figure shows video predictions with different levels reset to the prior for the GQN mazes dataset. With a 3-level model and temporal abstraction factor observe that when level 3 was not fed with observations, the conditioned open-loop predictions started with the correct viewpoint, but differed w.r.t. wall and floor colors. This suggests that those characteristics were stored in the higher level of the hierarchy.
Figure 11 shows video predictions with different levels reset to the prior for the Moving MNIST dataset. The 3-level model with temporal abstraction factor obtained a separation of information in the bottom two levels, while the posterior at the third level nearly collapsed to the prior. When level I was not fed with observations, we observe that the conditioned open-loop predictions maintained the same digit identity, but showed variations w.r.t. digit positions. On the other hand, when level
8

---

Under review as a conference paper at ICLR 2021
was not fed with observations, the samples maintained the same digit positions but produced digits with different identities in every sample. This suggests that lower level stored digit positions, high frequency details which changed frequently in time, while the level above it stored the digit identities, i.e. long-term information. We also observed that, when resetting level 2 to the prior, the digits start to differ in position sooner (~60 frames) than when all levels receive observations (~800 frames). This suggests that this level does have some information about digit positions, and that there is still mixing of information between different levels of the hierarchy.
Predictive entropy To corroborate our understanding of the latent representations, we observe the entropy of the prior distribution as it varies over time during open-loop video generation in Figure 10, With GQN mazes, the entropy at the top level remained relatively constant as the model remained certain about the high-level details. With Moving MNIST, the top two levels showed relatively constant entropy, with level 2 storing the digit identities and level 3 suffering from posterior collapse. Please refer to Appendix D.6 for a more detailed analysis.
DISCUSSION
In this work, we presented a hierarchical latent dynamics model with temporal abstraction (TALD), where each level in the hierarchy temporally abstracted the level below by an adjustable factor.
We evaluated long-horizon open-loop predictions using our model, and observed that TALD was able predict far into the future while accurately maintaining global information.
We also observed that the amount of information at the higher levels decreased as the speed of the sequence was increased.
We analyzed the separation of information at different levels of the hierarchy, by generating open-loop video predictions with different levels reset to the prior. With Moving MNIST, the bottom level in the hierarchy stored high frequency details (digit positions) and the level above stored more global information (digit identities). With the GQN mazes dataset, TALD stored wall and floor patterns at the top level in the hierarchy.
Temporally abstract models complex datasets and environments. We hope that our work can refuel interest in temporally abstract latent dynamics models and motivate the development of effective deep learning systems for high- dimensional data more generally.
intuitive approach obtaining high-level representations of

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. CoRR, abs/1710.11252, 2017. URL http:://axxii.
hrg/abs/17100 11252.
Lars Buesing, Theophane Weber, Sébastien Racanière, M. Ali Eslami, Danilo Jimenez Rezende, David P. Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, and Daan Wierstra. Learning and querying fast generative models for reinforcement learning. CoRR, abs/1 0bs//882.0335 802.03006, 2018. URL http: arxiv. org/ aabs///////// 1802. C3006.
Lluís Castrejón, Nicolas Ballas, and Aaron C. Courville. Improved conditional vrnns for video prediction. CoRR,
2019. URL http:://xxxv..oog/aaaa/.o .111911116685.
Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed, Recurrent enviromment simulators. CoRR, abs/1 704.02254, 2017. httttt/://aaae http://wwwww....r//////////////. 02254.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406. 1078, 2014. URL atttp://arae
org/ abs/ /106 1078.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. CoRR, abs/1506.02216, 2015. URL http:// //tpiivaaaiii///aa//aaeeeariiiia org/ abs 1506.0.0 02216.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks CoRR, (aa//609.00 71704, 2016. URL http: /tttp:p//aaiiii /wtxiv...vv//v////////eee/aawavl
httpa//wwwwwwaaaaa/aaa//aaaaaaaa otg/abs/////eoooooo.e....ssse 9 9. 01704.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-II, pp. 465-472, 2011.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. CoRR. abs/1 0000010027777,. 2018. URL http:://axxv..rr/// hbrgg/bb//80////// 07687.
Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous. Tensorflow distributions. arXiv preprint arXiv:1711.10600 2017.
Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, Stefan Schaal, Marc Toussaint, and Sebastian Trimpe. Probabilistic recurrent state-space models, arXiv preprint 1rXiv:1801.1 10395, 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (201)633))1204-1111
2018.
Yven Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. arXiv preprint arXiv:2002.09219, 2020.
Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo Jimenez Rezende, David Amos, and Timothy P. Lillicrap. Generative temporal models with memory. CoRR, abs/1702.04649, 2017. URL http: htt/xaxivvv..o abs 1702. 04649.
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang. Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. Tighty-Eighth International Joint Conference on Artificial Intelligence, 2019. URL http: //minerl.io
David Ha and Jürgen Schmidhuber. World models. CoRR, 0tbs/1803.10110 2018. URL http:
// //axiiv.. wrgg abs/ /ete/1103..0 10122.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2566 2019.
10

---

Under review as a conference paper at ICLR 2021
Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Synthesizing neural network con- trollers with probabilistic model based reinforcement learning. arXiv preprint arXivv1803.0000 02291, 2018.
Max Jaderberg, Wojeiech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio García Castañeda, Charles Beattie, Neil Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukeuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. CoRR, abs/ 807.01281, 2018. URL http:// //ap:v//axvv..// org/ abs /1077 01281.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep varia- tional bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint 1605::6605.06433 2016.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. In Advances in Neural Information Processing Systems, pp. 11566-11575, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, 6980.
abs//1412.6980, 2014. URL htttt:://axiiv.rr///a/t/
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In Proceedings of the International Conference Learning Representations (ICLR), 2014.
Jan Koutník, Klaus Greff, Faustino J. Gomez, and Jürgen Schmidhuber. A clockwork RNN. CoRR, abs/1402.351 2014. URL http: http:://aaaiii/ wttg///////////////
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. CoRR, abs/1903.01434 2019. URL http::////// /ttp///axvvv/e.//////eee/...ev.a owg.eiiiioooooooooooo.eeoeeooeer wooooogg//s...o..ooooo..
/abs///////// / 1903. C1434.
Yann LeCun, Bernhard Boser, John $ Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. CoRR, abs/1804.01523, 2018. URL http://arxix org/abs/ /1804 01523.
Junhyuk Oh, Xiaoxiao Guo. Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances Neural Information Processing Systems, pp. 2863-2871 2015.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 244, 2016, Conference Track Proceedings, 2016. URL http: /ttw//arvvvv.////////// crg/abs/ /1511. 06434.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint urXiv: 14X.::4401.4022.. 2014.
C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: local approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pp. 32-36 Vol.3, 2004.
11

---

Under review as a conference paper at ICLR 2021
Casper Kaae Sønderby, Tapani Raiko, Lars Maalde, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Proceedings of the 30th International Conference on Neural Informa- fion Processing Systems, NIPS' 16, pp. 3745-3753, USA. 2016. Curran Associates Inc. ISBN 978- 1-5108-3881-9 URL http: di cm. org/ ci tat ion cfm? 31 57511 6.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms. CoRR, abs/1502.04661,, 2015. URL 1502.04680 04881.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly, Towards accurate generative models of video: A new metric challenges. CoRR, abs/1111.00117,, 2018. URL http: arxiv. org/ abs 1812.. 01717.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder, 2020.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics, In Advances In Neural Information Processing Systems, 2016.
Dirk Weissenborn, Jakob Uszkoreit, and Oscar Täckström. Scaling autoregressive video models. In ICLR, 2020.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision. CoRR, fbs/1806.04768, 2018. URL http::/arxiv..ee abs/1806 04 768.
Ziru Xu, Yunbo Wang, Mingsheng Long, and Jianmin Wang. Predcnn: Predictive learning with cascade convolutions. Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI' pp. 2940-2944 AAAI Press, 2018. ISBN 9780999241127.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11.5586999 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. CoRR, abs/ Abs.1700, 2017. URL http:://aaaal..ooo//////10000.00
12

---

Under review as a conference paper at ICLR 2021
A BACKGROUND
We build our work upon the recurrent spate-space model (RSSM; Hafner al., 2019) that has been shown to successfully learn environment dynamics from raw pixels in reinforcement learning. This model acts as an important baseline for our evaluation. RSSM explains the video sequence T1rT using a latent sequence of compact Markovian states S1.T. Importandly the model is autoregressive in latent space but not in image space, allowing us predict into the future without generating images along the way,
(6) Given a training sequence, RSSM first individually embeds the frames using a CNN. A recurrent network with deterministic and stochastic components then summarizes the image embeddings. Hafner et al. (2019) argued that the stochastic component helps with modeling multiple futures, while the deterministic state helps remember information over many timesteps so that it is not erased by noise, Finally, the recurrent states are decoded using a transposed CNN provide a training signal.
p(ZETT S1:)) = II'= p(æ, st)p(se I 8t-1).
Encoder:
e, enc(2;)
Posterior transition ge:
g(st st-1, E2)
Prior transition pe: p(s: s(-1) p st).
Decoder:
The posterior and prior transition models share the same recurrent model. The difference is that the posterior incorporates images, while the prior tries to predict ahead without knowing the correspond- ing images. This lets to predict ahead purely in latent space inference time.
As typical with deep latent variable models, we cannot compute the likelihood of the training data under the model in closed form. Instead, we use the evidence lower bound (ELBO) as training objective. The ELBO encourages to accurately reconstruct each image from its corresponding latent state, while regularizing the latent state distributions to stay close the prior dynamics.
max ET Egg (In(r, +t)] Σ KL ll Pe]- 9:P
(8)
The KL regularizer limits the amount of information that the posterior transition incorporates into the latent state at each time step, thus encouraging the model to mostly rely on information from past time steps and only extract information from each image that cannot be predicted from the preceding images already. All components jointly optimize Equation 8 using stochastic backpropagation with reparameterized sampling (Kingma & Welling, 2013; Rezende et al., 2014).
B MODEL ARCHITECTURES
We use convolutional frame encoders and decoders, with architectures very similar to the DCGAN (Radford et al., 2016) discriminator and generator, respectively. To obtain the input embeddings ej particular level, k' input embeddings are rre-processed using a feed-forward network and then added up to obtain a single embedding. We also want to emphasize that we do not use any skip connections between the encoder and decoder bypassing the latent states as we believe this would motivate the model to make better use of the temporal hierarchy.
C HYPER PARAMETERS AND EXPERIMENTAL SETUP
We kept the output dimensionality of the encoder at each level of TALD as |e|| = 1024, that of the
stochastic states as 1po1 = ln'l 20, and that of the deterministic states as [h] 200. All hidden layers inside the cell, both for prior and posterior transition, were set to 200. We trained our models using the Adam optimizer (Kingma& & Ba, 2014), with learning rate of 5 x 10-4 and 10-4 for the Moving MNIST dataset. With the KTH Action and GQN mazes datasets, we lowered the learning rate to 3 x 10-4, For all three datasets, batch size was set to be 100 sequences, each sequence being of length 100.
The decision of using 36 context frames for conditioning all open-loop video predictions was taken considering the minimum number of observation frames required to transition at least once in the highest level of the hierarchy. With 3 levels in the hierarchy and temporal abstraction factor of 6, each latent state at the highest level corresponds to 36 images in the sequence, and thus its encoder network expects 36 images input.
13

---

Under review as a conference paper at ICLR 2021
D ADDITIONAL EXPERIMENTS
D.1
QUANTITATIVE METRICS FOR LONG-HORIZON VIDEO PREDICTION
Moving
1c3
Moving
0.9
0.8
PSNN
2.4
0.7
2.3
0.6
0.25
0.2
100
150
200
250
300
50
100
150
200
250
300
50
100
150
200
250
330
50
100
150
200
250
300
Dastace
Datance
Tramiss
Tammes
Dicannc Framess
TALD
SVG
(a) Moving MNIST Datasel.
SSIM
Adton
Action
1+2
Action
32.5
05
2.5
0.85
27.5
30.0
0.4
2.0
2. 0.80
25.0
0.3
1.0
22.5
0.5
1000
200
302
107
200
50
352
250
300
TALD ii lerds, fuctor lours)
Beseinn temp. abstractin) busi
ASSN
VTA
SVG.U
(b) KTH Action Dataset.
SSIM on MineRL NavigateDense--vi
PSNR on MineRL NavigateDense-vd
LPIPS on MineRL NevigsteDensee vo
IVD MineRL Navigate
9.5
0.8
9.4
PSWN
0.4
0.2
10
Frames
TALD w/ 3 leven, factor & (ouns)
Deseline the samp. acstracten) toun)
RSSK
VTA
SVGLP
(c) MineRL NavigateDense Dataset.
SSIM on GON Maces
PSNR on GON Mazes
LPIPS GON Mazes
le3
FVD on GON Mazes
1.0
2.0
0
0.6
1.0
0.3
0.4
10
150
30
Distarnc in Fraims
Datanne Framnos
Distanoe Famms
Disancc Frames
TALD w/ levels, factor (ours) Baseline the temp abstraction Jours)
RSSM
VTA
SAGUP
(d) GQN Mazes Datasct.
Figure 12: Quantitative comparison between temporally abstract latent dynamics model (TALD) and other video prediction models, for long-horizon open-loop video prediction. Higher values are preferred for SSIM and PSNR, whereas lower values are preferred for LPIPS and FVD.
14

---

Under review as a conference paper at ICLR 2021
D.2
LONG-HORIZON PREDICTION OF MOVING MNIST
33
33
3
3
3
33
3
3
3
-
33
3
3
3
3
33
3
SVG RSSM 3L-FI 3L-16
33
33
33
33
3
33
3
3
3
3 3
33
3
3
33
3
3
33
33
33
3
3
3
3
3
33
3
3 3
3
7
3
3
33
3
33
3
3
3
3
3
3
3
33
3 3
3
3 3
3 3
2 3
2
2
3
3
7
3
7
3
3
9
3
3
8
3
3
3
3
7
3
43
3
0
20
40
60
80
100
150
200
250
300
400
500
600
700
800
900
Figure 13: Long-horizon open-loop prediction on Moving MNIST. for levels, for abstraction factor.) The top row shows the ground truth sequence. The next two rows illustrate samples generated from our TALD model: levels with temporal abstraction factor of 6 (3L-F6), and 3 levels with temporal abstraction factor of (3L-FI) (essentially a deeper model with no temporal abstractions). Following that we have samples from the baselines of RSSM and SVG-LP. It is interesting to note here that TALD with abstraction factor 6 is able maintain accurate long-term correlation in the form of object identities for 900 frames into the future.
D.3 LONG-HORIZON PREDICTION OF 3D MAZES
GT
TTALD
RSSM
3
4
20
40
60
80
100
120
140
160
180
200
GT
RSSM TALD
t
3
4
5
20
40
60
80
100
120
140
160
180
200
Figure 14: Long horizon open-loop video prediction for the GQN mazes dataset (Eslami et al., 2018) using our 2-level TALD model with temporal abstraction factor of In both the examples, the mazes switch between two types of wall patterns as the camera moves around the maze. We observe that while TALDD could remember the alternate wall patterns for 200 timesteps, RSSM could not predict any frame with the previously generated wall patterns. This suggests that TALD was able to maintain long-term information for at least 200 frames.
D.4 LONG-HORIZON PREDICTION OF MINERL NAVIGATE
15

---

Under review as a conference paper at ICLR 2021
VTA
90
120
150
180
210
240
270
300
330
360
390
420
t
30
60
90
120
150
180
210
240
270
300
330
360
390
420
30
60
90
120
150
180
210
240
270
300
330
360
390
420
30
60
90
120
150
180
210
240
270
300
330
360
390
420
30
60
90
120
150
180
210
240
270
300
330
360
390
420
Figure 15: Long- horizon open-loop video prediction for the MineRL Navigate dataset (Guss et al., 2019). using our 3-level TALD model with a temporal abstraction factor of 6, along with some established baselines.
16

---

Under review as a conference paper at ICLR 2021
90
120
150
180
210
240
270
300
330
360
390
420
t
30
60
90
120
150
180
210
240
270
300
330
360
390
420
t
0
30
60
90
120
150
180
210
240
270
300
330
360
390
420
30
60
90
120
150
180
210
240
270
300
330
360
390
420
GT
60
90
120
150
180
210
240
270
300
330
360
390
420
Figure 15: (contd.) Long- open-loop video prediction for the MineRL Navigate dataset (Guss et al., 2019), using our 3-level TALD model with a temporal abstraction factor of 6, along with some established baselines.
17

---

Under review as a conference paper at ICLR 2021
D.5 UNDERSTANDING THE LATENT REPRESENTATIONS 3D MAZES
Ground truth sequence
Conditioned sample using all levels
Conditioned samples setting level-3 prior
2
3
10
15
20
30
35
40
45
50
Ground truth sequence
haus
Conditioned sample using all levels
Conditioned samples setting level-3 to prior
2
3
4
5
10 15
20
25
30
35
40
45
50
Figure 16: Visualizing the information stored at the higher level of our 3-level model with temporal abstraction factor 2, using the GQN mazes dataset. We computed posterior belief at each level using 8 observation frames, and set one the levels to the prior (by not feeding it with observations), which were then used to condition open-loop predictions. Changing wall textures show that they stored at the highest level.
D.6 ANALYSING THE ENTROPY CURVES
Figure 10b shows the entropy of the prior distribution at all levels of the 3-level TALD model with temporal abstraction factor 2, as varies over time, with the Moving MNIST dataset. We observe that the entropy at the lowest level varies between 9 13 nats, whereas for the higher levels in the hierarchy, the entropy remained relatively constant. In Figure 11 we presented some visualizations that supported the claim that the higher level in the hierarchy stores global information (digit identities) which, in this case, does not change over time, and hence the model should remain rather certain about as time progresses. Figure 10b corroborates that claim showing that the model's uncertainty at higher levels does not dither significantly over time compared to the lowest level in the hierarchy.
Figure 10a shows the entropy of the prior distribution at all levels of the 3-level TALD model with temporal abstraction factor 2, as varies over time, with the GQN mazes dataset. While the bottom two levels showed a similar variance in entropy with time, the entropy of the prior distribution at the top level did not change significantly. We showed in Figure 9 how the top level in the hierarchy stores wall and floor colors, which the model should remain certain about over time. In support of this claim, here also that the entropy of the prior distribution at the topmost level does not change significantly over time.
18


Reviews:
[{"review_id": "zAJifCBnlnQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of long term video prediction. It proposes a novel sequential hierarchical VAE-based model called Temporal Abstract Latent Dynamics (TALD), that can keep long term consistency by having different levels of the hierarchy update at different frequencies. \nEmpirically, the authors validate that the higher levels of the hierarchy do indeed encode for slower changing features in the videos. They also show that using temporal abstraction allows TALD to produce better predictions of long sequences (hundreds of frames) and outperform prior models in that setting.\n\n################################################\n\n\nStrong points:\n\n- The paper is clear and easy to follow. \n\n- It deals with an important topic, as long-term consistency is a challenge in many settings for sequential deep learning models.\n\n- The idea behind the model is simple and appealing.\n\n- Thorough experimental results validate that the model behaves as expected.\n\n\nConcerns:\n\n- Mainly, evaluation of performance seems inadequate:\n\nAll of the considered models (including the proposed one) are of stochastic nature. They are trained to provide plausible sequence, and not retrieve the exact ground-truth sequence. Yet, the proposed evaluation metric ignores the stochastic aspect, assuming the models should be able to correctly retrieve the ground-truth sequence given the initial frames. \nThose metrics might be ill-suited, as small positional shift due to the stochasticity can propagate through time and lead to huge drops in performances even if the digits are correctly conserved the entire sequence and the trajectories mostly correct. This actually shows up when comparing Figure 1 (where all models have scores comparable to random frames at the end)  and Figure 4 that shows that the digits mostly remain unchanged.\nWhy weren't stochastic variants of PSNR, LPIPS and SSIM (as used in the original publication for SVG-LP), or Frechet Video Distance (maybe on subsampled frames of the sequences) considered instead? \n\nAlso, the long-term consistency is mostly evaluated by human observation on samples. Maybe, a digit detector (and a wall/ground texture detector) could be used to provide a quantitative evaluation?\n\n################################################\n\nScore motivation:\n\nThe paper tackle an important problem with a well-motivated novel model. However, I have concerns about the quantitative evaluation methods.\n\n################################################\n\nMinor remarks:\n\n- How are the initial states h_0 computed? This information might be needed for reproducibility.\n\n\nThe authors might also want to acknowledge (or compare to) some recent works in video generation:\n\n- Stochastic Adversarial Video Prediction\nAlex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, Sergey Levine\n\n- VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation\nManoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, Durk Kingma\n\n- Stochastic Latent Residual Video Prediction\nJean-Yves Franceschi, Edouard Delasalles, Mickael Chen, Sylvain Lamprier, Patrick Gallinari\n\n- Scaling Autoregressive Video Models\nDirk Weissenborn, Oscar Täckström, Jakob Uszkoreit", "extra_scores": {}, "extra_text": {}}, {"review_id": "ourdPEn4U-G", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method called Temporal Abstract Latent Dynamics (TALD). TALD is built up on RSSM (Hafner et al. 2019) but with hierarchical dynamics. The experiments are conducted on moving MNIST, GQN 3D Mazes, and KTH. Results are qualitatively better than other methods in term of maintaining long-term consistent prediction. Quantitative comparison is reported only on KTH dataset (Figure 5). Written presentation is clear and easy to understand.\n\nPros:\n- The idea of modeling hierarchical latent variables for long-term video prediction is novel and interesting.\n- TALD qualitatively and quantitatively (on KTH) outperforms other methods, such as RSSM (direct baselines), VTA, SVG.\n- The experiments of removing one level of hierarchical latent (by replacing the posterior) is particularly interesting which proved that TALD can abstract different semantic information at different level its hierarchical representation.\n\nCons: \n- The comparisons on moving MNIST and GQN Mazes are only qualitatively (which may be biased to the selected sequences). The supplementary provides two more sequences (1 from moving MINIST and 1 from GQN Mazes) with qualitative results, but not quantitative results on these benchmarks. It should be better to report some sort of quantitative comparisons and evaluated over the full dataset instead (as done with KTH).", "extra_scores": {}, "extra_text": {}}, {"review_id": "nawgHD3xIG1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present build on the recurrent state space model by Haffner et al. (2019) to include a hierarchy of latent spaces. The idea is that the lower levels model short-term changes while the higher levels of the hierarchy model longer temporal dependencies.\n\nPros:\n1. The paper is largely well-written and easy to follow.\n2. The experiments do show that the proposed method is able to capture the long-term trends better than the chosen baselines for the chosen datasets. \n3. Experiments showing different amounts of information present at different levels of the latent space are interesting and show that desired hierarchical structure in the latent space is achieved.\n\nCons:\n1. In light of the paper \"Improved conditional VRNNs for video prediction\" by Castrejon et al., I do not believe there is enough novelty in this paper in terms of the latent space architecture, as this paper also applies a hierarchical latent space model for video prediction. The main contributions are in the analysis of the hierarchical architectures using information-theoretic measures for three datasets, which I believe are not sufficient for accepting this paper. The authors should clarify the main differences in the proposed method and the paper by Castrejon et al.\n\n2. Related to the above point, the datasets for the experiments should be stronger as this paper's novelty is mainly empirical. The KTH action dataset is the closest to a real-world dataset, but is still simplistic with just one moving object and a fixed background. The authors should consider experimenting with more challenging datasets. This is necessary to understand the limitations of the proposed method. \n\nUPDATE AFTER AUTHOR RESPONSE:\nI appreciate the authors' response and for clarifying the contributions. However I still feel that the experiments and datasets are too simple and not realistic. I am increasing my rating to reflect this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "i_m1OQN2lhV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a recurrent network for long-term video prediction, which learns hierarchical latent states in space-time. Although it shows strong experimental results, I have the following concerns about the novelty of the proposed model and the completeness of the experiments: \n1. To me, the proposed TALD architecture can be seen as a combination of two previous models, RSSM and VTA, with small extensions. While I understand that they have many differences in technical details, such as the number of layers and the dependencies between the latent states, the key insight behind these approaches is similar. \n2. Hierarchical latent variables have long been used for video prediction, as in Wichers et al. (2018) and Xu et al. (2018). I think the authors should make more comparisons with these similar works both in terms of methods and experimental results.\n[Xu et al., 2018] PredCNN: Predictive Learning with Cascade Convolutions\n3. For both Moving MNIST and KTH datasets, the proposed model tasks as input 36 context frames, which is not a common practice of most previous models, including those from Wichers et al. (2018) and Denton et al. (2018). Given that the temporal dynamics is relatively simple in these datasets, it may not be necessary to use so much context information. What would happen if we reduce the number of input frames?\n4. The authors put the descriptions of Figure 1 in Section 4.1 and the descriptions of Figure 2 in Section 4.3, which caused me some reading difficulties at the very beginning of the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the properties and advantages of temporal abstraction in hierarchical latent variable-based video prediction approaches, producing interesting results on various simulated environments and the KTH action dataset. 

The two key drawbacks of this paper are: limited visual complexity of datasets used for evaluation (a real video dataset like BAIR pushing would have really helped), and lack of comparison (conceptually or empirically) to relevant prior work including those raised by various reviewers, plus see more below.

Aside from this, does the paper claim to propose a new model, or perform an empirical study to evaluate an existing model class, or both? The answer to this question is not always clear from the paper and this confusion is reflected both in the reviews and reviewer discussions and also in the authors' own responses to these.

Some potentially relevant works that don't find mention in this paper (aside from those pointed out by reviewers):
- NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
https://arxiv.org/pdf/1901.10946.pdf
- Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
https://arxiv.org/pdf/2006.13205.pdf 

I do think these are all relatively easily fixed shortcomings, and I urge the authors to fix them in future versions.
Decision: Reject