Clockwork Variational Autoencoders
Authors: Vaibhav Saxena
Jimmy Ba
Danijar Hafner
Venue: NeurIPS 2021 Poster (2021)
Keywords: temporal abstraction
variational inference
recurrent neural networks
deep learning

Abstract: Deep learning has enabled algorithms to generate realistic images. However, accurately predicting long video sequences requires understanding long-term dependencies and remains an open challenge. While existing video prediction models succeed at generating sharp images, they tend to fail at accurately predicting far into the future. We introduce the Clockwork VAE (CW-VAE), a video prediction model that leverages a hierarchy of latent sequences, where higher levels tick at slower intervals. We demonstrate the benefits of both hierarchical latents and temporal abstraction on 4 diverse video prediction datasets with sequences of up to 1000 frames, where CW-VAE outperforms top video prediction models. Additionally, we propose a Minecraft benchmark for long-term video prediction. We conduct several experiments to gain insights into CW-VAE and confirm that slower levels learn to represent objects that change more slowly in the video, and faster levels learn to represent faster objects.

Clockwork Variational Autoencoders
Vaibhav Saxena University of Toronto vaibhavθcs toronto. edu
Jimmy Ba University of Toronto j jba@cs. toronto. edu
Danijar Hafner University of Toronto Google Research, Brain Team mai mail@dar mail@danijarr com
Abstract
Deep learning has enabled algorithms to generate realistic images. However, accurately predicting long video sequences requires understanding long-term dependencies and remains open challenge. While existing video prediction models succeed generating sharp images, they tend fail accurately predicting far into the future. We introduce the Clockwork VAE (CW- VAE), a video prediction model that leverages hierarchy of latent sequences, where higher levels tick slower intervals. We demonstrate the benefits of both hierarchical latents and temporal abstraction on diverse video prediction datasets with sequences up to 1000 frames, where CW-VAE outperforms top video prediction models. Additionally, we propose Minecraft benchmark for long-term vidco prediction. We conduct several experiments gain insights into CW-VAE and confirm that slower levels learn to represent objects that change more slowly the video, and faster levels learn to represent faster objects.'
Introduction
Moving MNIST Open-loop Prediction
0.90
Structural Simllanty (SSIM)
CW-VAE (3 levels, factor
Video prediction involves generating high- dimensional sequences for many steps into the future. A simple approach is to predict one im-
0.84
CW-VAE (3 levels, factor
CW-VAE (3 levels, factor 4)
(3 levels, facter
0.78
RSSM
age after another, based on the previously ob- served generated images (Oh al., 2015; Kalchbrenner et al., 2016; Babaeizadeh et al., 2017; Denton and Fergus, 2018; Weissenborn et al., 2020). However, such temporally au- toregressive models expensive because they operate the high- dimensional image space and at the frame rate of the dataset. In contrast, humans have the abil- ity reason using abstract concepts and predict their changes far into the future. without having visualizing the details in the input space or at the input frequency.
SVG-LP random
0.72
0.66
0.60
be computationally
80
120
160
200
Distance
Figure 1: Video prediction quality function of the distance predicted. We show versions of Clockwork VAE with temporal abstraction fac- tors 2, 4, 6, and 8. Larger temporal abstraction directly results in predictions that remain accurate for longer horizons. Clockwork VAE further out- performs the top video models RSSM and SVG.
Latent dynamics models predict sequence of learned latent states forward that is then decoded into the video, without feeding generated frames back into the model (Kalman, 1960; Krishnan et al., 2015: Karl et al., 2016). These models are typically trained using variational inference. similar VRNN (Chung al., 2015), except that the generated images are not fed back into the model. Latent dynamics models have recently achieved success in reinforcement learning for learning world models from pixels (Ha and Schmidhuber, 2018; Zhang et al., 2019; Buesing et al., 2018; Mirchev et al., 2018; Hafner et al., 2019b;a; 2020).
All code is publicly available https:: /tt::/gitttutt com/ btweaaxnnaa////aaa 35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Inputs
Video Prediction without Further Inputs
SVG RSSM VIA VAE
1=1
36
50 80
140 170 200 230 260 290 320 350
380 410
440 470 500
Figure 2: Long-horizon video predictions on the MineRL Navigate dataset. In the dataset, the camera moves straight ahead most of the time. CW-VAE accurately predicts the camera movement from the ocean to the forest until the end of the sequence. In contrast, VTA outputs artifacts in the sky after 240 steps and shows less diversity in the frames. RSSM fails to predict the movement toward the island after 290 frames. SVG simply copies the initial frame and does not predict any new events in the future.
To better represent complex data, hierarchical latent variable models learn multiple levels of features. Ladder VAE (Sønderby et al., 2016), VLAE (Zhao et al., 2017), NVAE (Vahdat and Kautz, 2020), and very deep VAEs (Child, 2020) have demonstrated the success of this approach for generating static images. Hierarchical latents have also been incorporated into deep video prediction models (Serban al., 2016; Kumar et al., 2019; Castrejón et al., 2019). These models can learn to separate high-level details, such as textures, from low-level details, such as object positions. However, they operate at the frequency of the input sequence, making challenging to predict far into the future.
Temporally abstract latent dynamics models predict learned features at slower frequency than the input sequence. This encourages learning long-term dependencies that can result in more accurate predictions and enable computationally efficient long-horizon planning. Temporal abstraction has been studied for dineensiiio sequences (Koutník al., 2014; Chung et al., 2016; Mujika et al., 2017). VTA (Kim et al., 2019) models videos using two levels of latent variables, where the fast states decide when the slow states should tick. TD-VAE (Gregor and Besse, 2018) models high-dimensional input sequences using jumpy predictions, without hierarchy. Refer to Appendix B for further related work. Despite this progress, scaling temporally abstract latent dynamics to complex datasets and understanding how these models organize the information about their inputs remain open challenges.
In this paper, we introduce the Clockwork Variational Autoencoder (CW-VAE), a simple hierarchical latent dynamics model where all levels tick at different fixed clock speeds. We conduct an extensive empirical evaluation and find that CW-VAE outperforms existing video prediction models. Moreover, we conduct several experiments to gain insights into the inner workings of the learned hierarchy.
Our key contributions summarized as follows:
Clockwork Variational Autoencoder (CW-VAE) We introduce a simple hierarchical video prediction model that leverages different clock speeds per level to learn long-term dependencies in video. A comprehensive empirical evaluation shows that on average, CW- CW-VAE outperforms strong baselines, such SVG-LP, VTA, and RSSM across several metrics.
Long-term video benchmark In the past, the video prediction literature has mainly focused on short-term video prediction of under 100 frames. To evaluate the ability to capture long-term dependencies, we propose using the Minecraft Navigate dataset as a challenging benchmark for video prediction of 500 frames.
Accurate long-term predictions Despite the simplicity of fixed clock speeds, CW-VAE im- proves over the distance of accurate video prediction of prior work. On the Minecraft Navigate dataset, CW-VAE accurately predicts for over 400 frames, whereas prior work fails before or around 150 frames.
Adaptation sequence speed We demonstrate that CW-VAE automatically adapts to frame rate of the dataset. Varying the frame rate of a synthetic dataset confirms that the slower latents are used more when objects the video are slower, and faster latents are used more when the objects are faster.

---

Second Level
top-down context
40
S
First Level
$
$
temporal context
h
temporal context
Video Data
bottom-up
top dop-down context
1=2
observations
Connection used for both prior and posterior
Connection only used for posterior (given data)
Figure Clockwork Variational Autoencoder VAE). We show 2 levels with temporal abstraction factor 2 (but we use up to 4 levels and abstraction factor of 8 in our experiments). Left is the structure of the model, where each latent state sy in the second level conditions two latent states in the first level. The solid arrows represent the generative model, while both solid and dashed arrows comprise the inference model. On the right, we illustrate the internal components of the state, containing a deterministic variable ht and a stochastic variable 2t. The deterministic variable aggregates contextual information and passes to the stochastic variable be used for inference and stochastic prediction.
Separation of information We visualize the content represented at higher levels of the temporal hierarchy. This experiment shows that the slower levels represent content that changes more slowly in the input, such as the wall colors of maze, while faster levels represent faster changing concepts, such as the camera position.
2 Clockwork Variational Autoencoder
Long video sequences contain both, information that is local to a few frames, as well as global information that is shared among many frames. Traditional video prediction models that prediet ahead at the frame rate of the video can struggle to retain information long enough to learn such long-term dependencies. We introduce the Clockwork Variational Autoencoder (CW-VAE) to learn long-term correlations of videos. Our model predicts ahead on multiple time scales, as visualized in Figure 3. We build our work upon the Recurrent State-Space Model (RSSM; Hafner et al., 2019b), the details of which can be found in Appendix A. CW-VAE consists of a hierarchy of recurrent latent variables, where each level transitions at different clock speed. We slow down the transitions exponentially as we go up the hierarchy, i.c. each level is slower than the level below by factor k. We denote the latent state at timestep and level by sf and the video frames by 2y. We define the set of active timesteps Ti for each level 1 E [1, T.] as those instances in time where the state transition generates new latent state,
Active steps: T = (1 E [1, mod k = =1}.
(1)
At each level, we condition consecutive latent states on a single latent variable in the level above. For example, in the model shown in Figure with = 2, Ti (1,2,3.... ... ..... To = {1,3, ...... and both 8 and si are conditioned the same of from the second level.
The latent chains separate state variable per timestep, but only updates the previous state every kl-1 timesteps and otherwise copies the previous state, that Vt ş Tí:
also be thought of as hierarchy of latent variables where each level has
Copied states:
5 s max
(2)
Joint distribution We can factorize the joint distribution of sequence of images and active latents at every level into two terms: (1) the reconstruction terms of the images given their lowest level latents, and (2) state transitions all levels conditioned on the previous latent and the latent above.
p(urr, 1:f) = (Ie pla. +))) (4))(II'
(II I p(st o1 at1))
3
3

---

To implement this distribution and its inference model, CW-VAE utilizes the following components, 1A E [1, Ti,
Encoder:
et e(21,048-1111 q( s/-1, st,,,ttt,e)) .b)
Posterior transition o/=
(4)
Prior transition pt: p(s 11 stt)
Decoder:
p(az s!).
Inference CW-VAE embeds the observed frames using CNN. Each active latent state at level I receives the image embeddings of its corresponding k' observation frames (dashed lines in Figure 3). The diagonal Gaussian belief of is then computed as a function of the input features, the posterior sample at the previous step, and the posterior sample above (solid lines in Figure 3), We reuse all weights of the generative model for inference except for the output layer that predicts the mean and variance.
Generation The diagonal Gaussian prior p is computed by applying the transition function from the latent state at the previous timestep in the current level, as well as the state belief at the level above (solid lines in Figure 3). Finally, the posterior samples at the lowest level are decoded into images using a transposed CNN.
Training objective Because cannot compute the likelihood of the training data under the model in closed form, we use the ELBO as our training objective. This training objective optimizes a reconstruction loss at the lowest level, and KL regularizer at every level in the hierarchy summed across active timesteps,
Σt Egg [I p(1. [ Et- Eien, of KLLd Il P8]] c-g-P
max
(5)
The KI. regularizers limit the amount of information about the images that enters via the encoder. This encourages the model utilize the "free" information from the previous and above latent and only attend to the input image to the extent necessary. Since the number of active timesteps decreases as we go higher the hierarchy, the number of KL terms per level decreases as well. Hence is easier for the model to store slowly changing information high up in the hierarchy than to pay KL penalty to repeatedly extracting the information from the images at the lower level or trying to remember it by passing it along for many steps at the lower level without accidental forgetting.
Stochastic and Deterministic Path As shown in Figure 3 (right), split the state s into stochastic (z() and deterministic (ht) parts (Hafner et al., 2019b). The deterministic state is computed using the top-down and temporal context, which then conditions the stochastic state that level. The stochastic variables follow diagonal Gaussians with predicted means and variances. We use one GRU (Cho et al., 2014) per level to update the deterministic variable at every active step. All components of Equation& 4 jointly optimize Equation 5 by stochastic backprop with reparameterized sampling (Kingma and Welling, 2013; Rezende et al., 2014). Refer to Appendix C for architecture details.
3 Experiments
We compare the Clockwork Variational Autoencoder (CW-VAE) on 4 diverse video prediction datasets to state-of-the-art video prediction models in Sections 3.2 to 3.5. We then conduct an extensive analysis to gain insights into the behavior of CW-VAE. We study the content stored each level of the hierarchy in Section 3.6, the effect of different clock speeds in Section 3.7, the amount of information at each level in Section 3.8, and the change in information as function of the dataset frame rate Section 3.9. The source code and video predictions are available on the project website.?
Datasets We choose 4 diverse video datasets for the benchmark. The MineRL Navigate dataset 4.0 license) was crowd sourced by Guss et al. (2019) for reinforcement learning applications. We process this data to create long- horizon video prediction dataset that contains 750k frames. The sequences show players traveling to goal locations in procedurally generated 3D worlds of the video game Minecraft (traversing forests, mountains, villages, oceans). The KTH Action video prediction dataset (Schuldt et al., 2004) (available under the CC Atrribution-NnnCommeree show humans walking, jogging, running, boxing hand-waving, and clapping. GQN Mazes (Eslami
(available under the CC
www.Nooooooooiaa..
license) contains 290k frames. The 600 videos
Phttps: h/tp://daaiiieeee jar. wwt.ao/ww...aeeeeeeeeeeae

---

Inputs
Video Prediction without Further Inputs
SVG RSSM VIACW- VAE GT
t=1 18
36
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
Figure 4: Open-loop video predictions for KTH Action. CW-VAE predicts accurately for 50 time steps. VTA fails to predict coherent transitions from one frame to the next around time step 55, which we attribute the fact that its lower level is reset when the higher level steps. SVG and RSSM accurately predict for 20 frames and their subsequent predictions appear plausible but fail to capture the long-term dependencies in the walking movement that is present in the dataset.
et al., 201 8) (available under the Apache License 2.0) contains 9M frames of videos of scripted policy that traverses procedurally generated mazes with randomized wall and floor textures. For Moving MINST (Srivastava et al., 2015) (available under the CC AntriuttionShaaeellin 3.0 license) we generate 2M frames where two digits move with velocities sampled uniformly the range to 6 pixels per frame and bounce within the edges.
Baselines We compare CW- VAB to well-established video prediction models, and an ablation of our method where all levels tick at the fastest scale, which call NoTmpAbs. VTA (Kim et al., 2019) is the sate-of-theearr for video prediction using temporal abstraction. It consists of two levels, with the lower level predicting when the higher level should step. The lower level is reset when the higher level steps. RSSM (Hafner et al., 2019b) commonly used as a world model in reinforcement learning. It predicts forward using sequence of compact latents without temporal abstraction. SVG-LP (Denton and Fergus, 2018), or SVG for short, has been shown to generate sharp predictions on visually complex datasets. It autogressively feeds generated images back into the model while also using a latent variable at every time step. The parameter counts are shown in Table D.2. NoTmpA simply the temporal abstraction factor and thus the same number of parameters as its temporally-abstract counterparts.
Training details We train all models on all datasets for 300 epochs on training sequences of 100 frames of size 64 x 64 pixels. For the baselines, we tune the learning rate in the range [10-4 10 and the decoder stddev the range [0.1.1]. We use a temporal abstraction factor of 6 per level for CW-VAE, unless stated otherwise. Refer to Appendix D for hyperparameters and training durations. 3-level WW-VAE with abstraction factor 6 takes 2.5 days train on one Nvidia Titan Xp GPU, Higher temporal abstraction factors train faster because fewer state transitions need to be computed.
Evaluation We evaluate the open-loop video predictions under 4 metrics: Structural Similarity index (SSIM, higher is better), Peak Signal-to-Noiss Ratio (PSNR, higher is better), Learned Percep- tual Image Patch Similarity (LPIPS, lower is better; Zhang al., 2018), and Frechet Video Distance (FVD, lower is better; Unterthiner et al., 2018), All video predictions are open-loop, meaning that the models only receive the first 36 frames as context input and then predict forward without access to intermediate frames. The number of input frames equals one step of the slowest level of CW-VAE for simplicity; see Appendix D.
3.1 Benchmark Scores
We evaluate the 5 models on 4 datasets and 4 metrics. To aggregate the scores, we compute how each model ranks compared to the other models, and average its ranks across datasets and metrics, shown in Figure 5c. Individual scores are shown in Table E.
Averaged over datasets and metrics, CW- substantially outperforms the existing methods, which we attribute to its hierarchical latents and temporal abstraction. The second best model is NoTmpAbs, which uses hierarchical latents but no temporal abstraction. It is followed by RSSM, which has only one level. The performance improvement due to hierarchical latents is smaller than the additional
5

---

Sequences
Moving MNIST Digit Identities
Rank (lower is better)
4.0
Accuracy
Accuracy
0.60
3.5
3,0
544
6.50
63-
CW-VAE
NoTmpAbs
RSSM
VTA
SVG
Prediction Horizon
CW-VAE
NoTmpths
RSSM
SVG
VTA
CW-VAE (ours) RSSM SVG
NoTmpAbs (ours)
VTA
Figure 5: Evaluating video prediction models. (a) Accuracy of sequence of rooms in GQN Mazes. VAE predicts video with a significantly higher accuracy high-level details than the second best video prediction model for this evaluation (VTA). (b) Accuracy of digit identities in two-digit Moving MNIST. CW-VAE generates video with a significantly higher accuracy of digit identities until 1000 timesteps. (c) Ag Aggregated performance ranks of all methods across datasets and 4 metrics. The best possible rank is I and the worst is 5. Our CW-VAE uses temporally abstract hierarchy and substantially outperforms hierarchy without temporal abstraction (NoTmpAbs), well the top single-level model RSSM, the image-space model SVG, and the temporally abstract model VTA.
improvement due temporal abstraction. Specifically, NoTmpAbs is sometimes outperformed by RSSM but CW-VAE matches or outperforms RSSM on all datasets and metrics.
While SVG achieves high scores on GQN Mazes due to its sharp predictions, it performs poorly compared to CW-VAE and RSSM on the other 3 datasets. This result indicates the benefits of predicting forward purely in latent space instead of feeding generated images back into the model. While VTA makes reasonable predictions, lags behind the other methods our experiments. One reason could be that its low level latents do not carry over the recurrent state when the high level ticks, which we found to cause sudden jumps in the predicted video sequence.
3.2 MineRL Navigate
Video predictions for sequences of 500 frames are shown in Figure 2. Given only an island on the horizon as context, the temporally abstract models CW-VAE and VTA correctly predict that the player will enter the island (as the player typically navigates straight ahead in the dataset). However, the predictions of VTA lack diversity and contain artifacts. In contrast, CW-VAE predicts diverse variations in the terrain, such as grass, rocks, trees, and pond. RSSM generates plausible images but fails to capture long-tern term-termmm dependencies, such as the consistent movement toward the island. SVG only predicts small to changes after the first few frames. Additional samples in Figure F.1 further confirm the findings. The predictions of all models are bit blurry, which we attribute to the model capacity, which was restricted due our limited computational resources.
3.3 KTH Action
Figure shows video prediction samples for the walking task of KTH Action. CW-VAI accurately predicts the motion of the person they walk across the frame until walking out of the frame. VTA predicts that the person suddenly disappears. which we attribute to it resetting the low level every time the high level steps. Both RSSM and SVG tend to forget the task demonstrated in the context frames, with RSSM predicting a person standing still, and SVG predicting the person starting to move in the opposite direction, after open-loop generation of about 20 frames. We also point out that SVG required the slower VGG architecture for KTH, whereas CW-VAE works well even with the smaller DCGAN architecture.
3.4 GQN Mazes
Figure G.1 shows open-loop video prediction samples of GQN Mazes, where the camera traverses two rooms connected by a hallway with distinct randomized textures in a maze. CW-VAE maintains and predicts wall and floor patterns of both rooms for 200 frames, whereas RSSM and VTA fail to predict the transition to the textures of the second room. SVG the model with the sharpest predictions this dataset which is reflected by the metries. However, it confuses textures over

---

Inputs
Video Prediction without Further Inputs
VTA CW-VAV GT
/
O
/
/
/6
6
0
6
6
i
d
/
8
o
O
q
7
0
/
d
o
01
O
/
/
0z
6
10
10
/
8
6
X
5
8
2
X
>
5 X
0
x
x
3
3
5
SVG RSSM
1
8
O
f
/
1/
0 /
/ O
8
0
10
9
8 X
a
X
4
8 x
4
2
1
8
0
f
//
8
&
5
&
8
18
36
40
60
80
100
120
140
160
200
300
400
500
600
700 800 900 1000
Figure 6: Long-horizon open-loop video predictions on Moving MNIST. We compare samples of CW-VAE, VTA, RSSM, and SVG. Red crosses indicate predicted frames that show incorrect digit identities. We observe that CW- VAE remembers the digit identities across all 1000 frames of the prediction horizon, whereas all other models forget the digit identities before or around 300 steps. The image-space model SVG is the first to forget the digit identities, supporting the hypothesis that prediction errors accumulate more quickly when predicted images are fed back into the model compared to predicting forward purely in latent space.
long horizon, generating mixed features the walls and floor of the maze as visible in the video predictions in Figure G.I. While the open-loop predictions of CW-VAE differ from the ground truth in their camera viewpoints, the model remembers the wall and floor patterns for all 200 timesteps, highlighting its ability to maintain global information for a longer duration.
To better evaluate CW-VAF, we compute the prediction accuracy for the following three high-level categories of sequences of rooms in the ground-truth video: 1) agent moves across the same room throughout the video, 2) agent traverses into the hallway but does not transition into another room, 3) agent goes into the hallway and traverses back into a room. Because the rooms and hallways use different textures, we can easily identify these classes via the color histogram. Figure 5a shows that CW-VAE predicts video with a significantly higher accuracy than the second best model, VTA, which is then followed by other baselines with similar accuracy. This shows that CW-VAE immensely benefits from its temporal hierarchy for accurately predicting high-level details such as the sequence of rooms in video.
3.5 Moving MNIST
Figure 6 shows samples of open-loop video prediction of 1000 frames. We observe that CW-VAE remembers the digit identity for all 1000 timesteps. RSSM clearly outperforms SVG, which typically forgets digit identity within 50 timesteps, but starts to forget object identities much sooner than CW-VAE. Figure 5b shows the two-digit classification accuracy of video prediction as it varies over time, averaged over the test set. We observe that while RSSM initially has the highest accuracy, falls below CW-VAE after 200 frames. CW-VAE maintains stable accuracy over 1000 timesteps, ending with significantly more accurate predictions at the horizon than the second best model, VTA. With regards to digit positions, as shown in Figure 6, CW- VAE predicts accurate positions until 100 steps, and predicts a plausible sequence thereafter. RSSM also predicts the correct location of digits for at least as long as CW-VAE, whereas SVG starts to lose track of positions much sooner. We note
Model
Level 1
Level
Level 3
Level 4
level CW-VAE
397.10
41.20
2.66
0.0001
3-level CW-VAE
366.70
45.28
6.29
2-vel CW-VAE
389.40
39.18
1-level CW-VAE
440.50
Table 1: KL loss at each level of the hierarchy for CW-VAES with different numbers of levels, summed over time steps of a training sequence. Deeper hierarchies incorporate less information about the inputs into the lowest level and instead distribute the information content across levels. Higher levels tend store less information than lower levels, suggesting that the dataset contains more short-term dependencies than long-term dependencies.
7

---

Inputs
Ground truth
Inputs
Video prediction
Inputs
Video predictions when top level does not receive inputs
4 8
10 11
12
13
14
15
16
17 18
19
20
21
22
23
24
25
Figure 7: Visualization of the content stored at the top-level of Clockwork VAH trained on GQN Mazes. The first row shows ground truth and the second row shows a normal video prediction. The remaining rows show video predictions where the top-level stochastic variables are drawn from the prior rather than the posterior. In other words, only the lower and middle levels have access the context images but the top level is blind. We find that the positions of camera and nearby walls remain unchanged, so this information must have been represented lower and middle levels. In contrast, the model predicts textures that differ from the context, meaning that this global information must have been stored at the top level.
that the generations by models that predict purely in latent space are slightly blurry compared to those generated by SVG.
Adapting Sequence Speed
3.6 Content Stored at Different Levels
KL divergence in nats
We visualize the information stored at differ- ent levels of the hierarchy. We generate video predictions where only the lower and middle layer have access to the context images, but
100
10
the high level follows its prior. This way, the
0.1
Bottom
prediction will only be consistent with the con- text inputs for information that extracted by the lower and middle level. Information that held by the top level is not informed by the context frames and thus will follow the train- ing distribution. We use instead of 36 input images for this experiment because we do not need to compute the top level posterior here. Figure 7 shows these samples for one input sequence and more examples are included Figure G.I. We find that the video predictions correctly continue the positions of the camera and nearby walls visible in the input frames, but that the textures appear randomized. This confirms that the top level stores the global information about the textures whereas middle levels. We also experimented with resetting the lower or middle level but found that they store similar information, suggesting that 2-level model may be sufficient for this dataset.
Middle level
0.01
Top level
3%
10%
30%
100%
300%
Speed of Moving Digits
Figure 8: KL loss each level of CW-VAE when trained on slower or faster variants of Moving MNIST. The KL value at level indicates the amount of information stored at that level. Faster moving digits result in higher KLs at the lower level and lower KLs at higher levels. Slower moving dig- its result in
information at higher levels.
ephemeral information is stored in the faster ticking lower and
3.7 Temporal Abstraction Factor
Figure compares the quality of open-loop video predictions on Moving MNIST for CW- VAE with temporal abstraction factors temporal abstraction factor directly increases the duration for which the predicted frames are accurate for. Comparing to RSSM and SVG, CW-VAEs model long-term dependencies for 6x as many frames as the baselines before losing temporal context. The point at which the models lose temporal context when they approach the "random" line, which shows the quality using randomly sampled training images native baseline for video prediction that has temporal dependencies.
and all with equal number of model parameters. Increasing the

---

3.8 Information Amount per Level
Table 1 shows the KL regularizers for each level for CA-V of varying number of levels. The KL. regularizers are summed across evaluation sequences on the Moving MNIST dataset. The KL regularizers provide an upper bound the amount of information incorporated into each level of the hierarchy, which we use as an indicator. The 2-level and 3-level CW- WAES were trained with a temporal abstraction factor of 6, and the 4-level model with factor of 4 to fit into GPU memory. We observe that the amount of information stored at the lowest level decreases as we use a deeper hierarchy, and further decreases for larger temporal abstraction factors. Using larger number of levels means that the lowest level does not need to capture as much information. Moreover, increasing the amount of temporal abstraction makes the higher levels more useful the model, again reducing the amount of information that needs to be incorporated at the lowest level.
3.9 Adapting to Sequence Speed
To understand how our model adapts to changing temporal correlations in the dataset, we train CW- VAE on slower and faster versions of moving MNIST. Figure 8 shows the KL divergence summed across the active timesteps of each level the hicrarchy. The KL regularizer each level correlates with the frame rate of the dataset. The faster the digits move, the more the fast ticking lowest level of the hierarchy is used. The slower the digits move, the more the middle and top level are used. Even though the KI. divergence at the top level is small, it follows consistent trend. We conjecture that the top level stores relatively little information because the only global information is the two digit identities, which take about 5 nats to store. The experiment shows that the amount of information stored at any temporally abstract level adapts to the speed of the sequence, with high-frequency details pushed into fast latents.
Discussion
This paper introduces the Clockwork Variational Autoencoder (CW-VAE) that leverages temporally abstract hierarchy of latent variables for long-term video prediction. We demonstrate its empirical performance on diverse video prediction datasets with up to 1000 frames and show that CW-VAE outperforms top video prediction models from the literature. Moreover, we confirm experimentally that the slower ticking higher levels of the hicrarchy learn to represent content that changes more slowly in the video, whereas lower levels hold faster changing information.
We point out the following limitations of our work promising directions for future work:
The typical video prediction metrics are not ideal at capturing the quality of video predictions, especially for long horizons. To this end, we have experimented with evaluating the best out of 100 samples for each evaluation video but have not found significant differences in the results, while evaluating even more samples is computationally infeasible to us. Using datasets where underlying attributes of the scene available would allow evaluating the multi-step predictions by how well underlying attributes can be extracted from the representations using a separately trained readout network.
our experiments, we train the Clockwork VAE end-to-end on training sequences of length 100. With 3 levels and a temporal abstraction factor of 8, the top level can only step once within each training sequence. Our experiments show clear benefits of the temporally abstract model, but we conjecture that its performance could be further improved by training the top level on more consecutive transitions.
We used relatively small convolutional neural networks for encoding and decoding the images. This results in a relatively light- weight model that can easily be trained on single GPU in a few days. However, we conjecture that using larger architecture could increase the quality of generated images, resulting in a model that excels at predicting both high frequency details and long-term dependencies in the data.
Temporally abstract latent hierarchies are an intuitive approach for processing high-dimensional input sequences. Besides video prediction, we hope that our findings can help advance other domains that deal with high-dimensional sequences, such as representation learning, video understanding. and reinforcement learning.
Acknowledgments We thank Ruben Villegas and our anonymous reviewers for their valuable input.

---

References
M. Babacizadch, C. Finn, D. Erhan, R. H. Campbell, and S. Levine. Stochastic variational video prediction. CoRR, abs/1710. 11252, 2017. URL http://p://axx....o httpe/i..vwg/as////eeeee htt///abs/111... 11252.
S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1171-1179, 2015.
Y. Bengio. Markovian models for sequential data, 1996.
H. Bourlard and N. Morgan. Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic Publishers, 01 1994. ISBN 978-1-4613-64099- doi: 00.1007/978-14411--327
L. Buesing, T. Weber, S. Racaniere, S. Eslami, D. Rezende, D. P. Reichert, F. Viola, F. Besse, K. Gregor, D. Hassabis, et al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:
2018.
I.. Castrejón, N. Ballas, and A. C. Courville. Improved conditional vrans for video prediction. CoRR, abs/1904. 12165, 2019. URL http:://aaixi.....////////// wttp:/awwwww...r/arr//aa/aaaaa/e /904. 12165.
R. Child. Very deep vaes generalize autoregressive models and can outperform them on images, 2020.
K. Cho, B. Merrienboer, Ç. Güilgehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078, 2014. URL httt:://axxii: org/ wtrg/abss/4ee /10// 1078.
J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pages 2980-2988, 2015.
Chung, S. Ahn, and Y. Bengio. Hicrarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01744 2016.
A. Clark, J. Donahue, and K. Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: 06571, 2019.
E. Denton and R. Fergus. Stochastic video generation with a learned prior. arXiv preprint 1e02:000277885 2018.
S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Gamelo, A. Ruderman, A. A. Rusu, Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science, 360(6394); 1204-1210, 2018
J.-Y. Franceschi, E. Delasalles. M. Chen, S. Lamprier, and P. Gallinari. Stochastic latent residual video prediction. arXiv preprint 20XX:22000200229,, 2020.
M. Gemici, C.-C. Hung, A. Santoro, G. Wayne, S. Mohamed, D. Rezende, D. Amos, and T. Lillicrap. Generative temporal models with memory. arXiv preprint arXiv: 2017.
A. Graves, G. Wayne, and Danihelka. Neural turing machines. CoRR, abs/ 2014. URL http://arxiv. www.aaiwww....aaaaaaaaaaaeeaaaae httgg//b///// 1411111111........
A. Graves, G. Wayne, M. Reynolds, Т. Harley, I. Danihelka, A. Grabska-Barwinska. S. Gómez, E. Grefenstette, T. Ramalho, J. Agapiou, A. Badia, K. Hermann, Y. Zwols, G. Ostrovski, Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 538, 10 2016. doi: 10. 1038//aauure20001..
K. Gregor and Besse. /806.0800..3111/ 2018.
Temporal difference variational auto-encoder.
arXiv preprint
W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. MineRL: large-scale dataset of Minecraft demonstrations, Eivett-Eigght International Joint Conference on Artificial Intelligence, 2019. URL http:/ http://mmieriiii
10

---

D. Ha and J. Schmidhuber. World models. arXiv preprint 1803. 2018.
D. Hafner, T. Lillierap, J. Ba, and M. Norouzi. Dream control: Learning behaviors by latent imagination. arXiv preprint
2019a.
D. Hafner, T. Lillierap. I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. 2019b.
D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint ar arXiv:2010.02193 2020.
G. E. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference Computational learning theory, pages 5-13, 1993.
S. Hochreiter and Schmidhuber. Long short-term memory. Neural Comput., 98)) Nov. 1997. ISSN 0899-7667. doi: 10.1162/neco. 1997.9.8.1777 1735. URL https://doii htt.///ii.or/11..11..1.1.... neco. 1997 9. 9.8.8......... 1735.
D. Jayaraman, Ebert, Efros, and S. Levine, Time-agnostic prediction: Predicting predictable video frames. arXiv preprint 1rXiv:1808.07784, 2018.
N. Kalchbrenner, V. d. Oord, K. Simonyan, I. Danihelka, O. Vinyals, Graves, and
K. Kavukcuoglu. Video pixel networks. arXiv preprint arXiv: 2016.
R. E. Kalman. A new approach linear filtering and prediction problems. Journal of basic Engineering, 82(1):35-45 1960.
M. Karl, M. Soelch, Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint
2016.
T. Kenter, V. Wan, C.-A. Chan, R. Clark, and J. Vit. Chive: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. International Conference Machine Learning, pages
PMLR, 2019.
Kim, S. Ahn, and Y. Bengio. Variational temporal abstraction. In Advances in Neural Information Processing Systems, pages 11566-11575, 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/ (bb///42..690, 2014. URL http::/arxiv.. wwt//o//////a//////////////////e abs//412../1 6980.
D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Koutník, K. Greff, J. Gomez, and Schmidhuber. A clockwork RNN. CoRR, abs/ 140235511 2014. URL http://arxiv. org/ abs/ 14022.3511
R. G. Krishnan, U. Shalit, and D. Sontag. Deep kalman filters. arXiv preprint arXiv:1511.0512 2015.
M. Kumar, M. Babaeizadeh, D. Erhan, C. Finn, S. Levine, L. Dinh, and D. Kingma. Videoflow: A conditional flow-based model for stochastic video generation. arXiv preprint arxiv:1903.01494 2019.
A. X. Lee, R. Zhang, Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. CoRR, abs/1804.01523, 2018. http://///// wwwpooggg/bbb//ooeeeeeeeeeaa /t11//00440111....
W. Lotter, G. Kreiman, and D. Cox. Deep predictive coding networks for video prediction and 2016.
unsupervised learning, arXiv preprint
A. Mirchev, Kayalibay, M. Soelch, van der Smagt, and Bayer. Approximate bayesian inference in spatial environments. arXiv preprint ArXiv:1805.07220 2018.

---

A. Mujika, F. Meier, and A. Steger. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, pages 5915-5924, 2017.
J. Oh, X. Guo, H. Lee, R. L.. Lewis, and S. Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems, pages 2863-2871, 2015.
K. Pertsch, O. Rybkin, F. Ebert, C. Finn, D. Jayaraman, and S. Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. arXiv preprint arXiv:2006. 13205, 2020.
A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: /ttp://aaxivvvvv. wwapiii..rg//aieeo wwwg/b////////ooooo/go/////e 1111444411.........1
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 1401.4082, 2014.
C. Schuldt, I. Laptev, and B. Caputo. Recognizing human actions: local svm approach. In Proceedings of the 17th International Conference on Puttern Recognition, 2004. ICPR 2004., volume 3, pages 32-36 Vol.3, 2004.
I. V. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pineau, A. Courville, and Y. Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv preprint arXiv: 1xXiv:605.06000 06066, 2016.
E. Shelhamer, K. Rakelly, Hoffman, and T. Darrell. Clockwork convnets for video semantic segmentation. CoRR, abs/1608.. 00000, 2016. URL http: //ap:i/////ivvav/aa org/abs/16088
K. Sonderby, T. Raiko, L. Maalće, S. K. Sønderby, and O. Winther. Ladder variational autoencoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pages $745-3753, USA, 2016. Curran Associates Inc. ISBN 978-1-5108--388119 URL
http://dl. acm. wwwg///t//iiiiooooooo/eeoooo.e on cfm?id-3157382. 3157516..
N. Srivastava, Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using Istms. CoRR, abs/1502.046811 2015. URL http://arxixz org/ abaa 015520006633
T. Unterthiner, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. CoRR, abs/1812.01171, 2018. URL http::/ttseee //axtp://rxxv.... orwooogg//aooo
A. Vahdat and Kautz. Nvae: A deep hierarchical variational autoencoder, 2020.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. CoRR, abs/1706.03766 2017. URL httt::::/axxvv.rr/aaat 03762.
C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, 2016.
D. Weissenborn, J. Uszkoreit, and O. Täckström. Scaling autoregressive video models. In ICLR, 2020.
N. Wichers, R. Villegas, D. Erhan, and H. Lee. Hierarchical long-term video prediction without supervision. CoRR, abs/1806.04768, 2018. URL http://arxiv. org/abs/18066 04768.
M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, 2019.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
S. Zhao, Song, and Ermon. Learning hierarchical features from generative models. CoRR, abs/1702. 2017, URL http:://axis/
12


Reviews:
[{"review_id": "OltLMQjmtYv", "reviewer": "Reviewer_qetJ", "summary": "The paper proposed a method for video prediction called Clockwork Variational Autoencoders. The method adapts Clockwork-RNNs to sequential VAEs and defines a hierarchy of latents operating at different temporal scales. This allows the model to generate long term videos.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nThe idea of having different temporal scales in sequential models has been proposed before in Chung et al. or Koutnik et al. However, the adaptation of this idea to sequential VAEs for video prediction is not straightforward and therefore novel.\n\n**Clarity**\n\nThe paper is well written. The problem statement, the model description and the experimental section are clear.\n\n**Quality**\n\nThe technical contribution of the paper is sound. The experimental section of the paper is extensive and adequately highlights the properties of the proposed model, not only comparing to different baselines but also investigating different aspects of the model such as the utilization of the different levels of latents. \n\n**Significance**\n\nOverall the paper is clearly written, presents a novel approach for video prediction specifically designed to model long-term videos, and the experimental section properly supports the validity of the model. I believe the community will be interested in this work.\n\n**Review Updates**\n\nAfter reading the rest of the reviews and the authors' response, I stand by my original rating and argue for the acceptance of the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some of the limitations of their model but do not acknowledge any potential negative societal impact of their work. Video prediction models in general might be used with malicious intents (for example \"deepfakes\"), and might reflect biases found in the training data. Please include a discussion on potential negative impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "There are no major ethical concerns with the paper. The authors use a dataset showing videos of humans but do not mention the license for this dataset (it might be unclear), as opposed to the rest of the dataset for which they mention the license."}}, {"review_id": "Hpc_hnoTiGW", "reviewer": "Reviewer_UP8B", "summary": "This paper proposes Clockwork-VAEs, a hierarchical latent dynamics video prediction model. CW-VAE uses different clock speeds at each level to model dependencies occurring at different frequencies. This allows it to perform well on long-term video prediction. The paper presents results on four diverse datasets, including using a new dataset (MineRL) for long term video prediction.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed architecture is intuitive, simple, and achieve strong results. Experiments are also fairly comprehensive, and I appreciate the visualizations provided. I think this model and the proposed use of the MineRL dataset are valuable contributions for the community, and I lean towards acceptance.\n\n## Strengths\n\n* The proposed CW-VAE automatically adapts to the frame rate of the dataset, which is a neat feature to sidestep annoying hyperparameter tuning across different datasets.\n* The quantitative results are strong, and the ablation experiments are comprehensive. I particularly appreciate the efforts in sections 3.6, 3.8 and 3.9 which examine different levels of CW-VAE, to show that the proposed hierarchy does indeed encode information of differing content and temporal speeds.\n* The model itself is intuitive and simple, yet appears to achieve strong results. Training time and resources are also modest and comparable to prior work.\n\n\n## Weaknesses\n\n* The experiments seem to be conducted on fairly simple datasets, which have minimal moving objects. The only dataset with more than one moving object is Moving MNIST, which is somewhat of a toy dataset. Experiments on more complex datasets such as KITTI would be valuable. It would be interesting to see how the different hierarchies perform with faster / slower moving cars, in particular, or how it operates when there are multiple moving objects in the video.\n* While the quantitative metrics seem strong, it is also valuable to conduct a human study over the generated results, especially for long term generation results on MineRL, which is a new dataset.\n* There is missing comparison to prior work on stronger hierarchical models, such as [1].\n* The results in the paper are presented on 64x64 images, which seem rather small. How does CW-VAE perform if scaled up to larger video resolutions, such as 128x128 or 256x256?\n\n## Concerns / questions\n\n* Are results presented in Figures F.1 and G.1 randomly sampled, or cherry picked? If they are cherry picked, please also share some randomly sampled results.\n* How is the video diversity of CW-VAE generated videos? There is a claim in the paper (L198) that VTA lacks diversity on MineRL. Please substantiate CW-VAE’s improved diversity with qualitative results for the different models, or with some measure of video diversity (e.g. LPIPS of predicted video frames at long horizons) when sampling different values from the Gaussian prior.\n* MineRL seems like a natural setting for action-conditioned video prediction. It would be interesting to hear the authors’ thoughts on whether they attempted to include the action condition in CW-VAE, or if there is a natural way to do so.\n\n## Presentation\n\nThere are some minor typos in the paper:\n* L169: “factors training faster” -> “factors *train* faster”\n\n\n[1] Lee, W., Jung, W., Zhang, H., Chen, T., Koh, J. Y., Huang, T., ... & Hong, S. (2020, September). Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction. In International Conference on Learning Representations.\n\n\n*Post rebuttal*\n\n* The authors have addressed most of my concerns. I think this is a meaningful paper and I would be happy to accept it. I have raised my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of CW-VAE and potential directions for future work are provided in Section 4 of the main paper. There does not seem to be any potential negative societal impact that would directly arise from this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DomWFjBAO-F", "reviewer": "Reviewer_ugeK", "summary": "In this paper, the authors tackle the problem of long-term video generation. To this end, they propose a hierarchical latent variational autoencoder: The Clockwork Variation Autoencoder. This model incorporates a hierarchy of latents separated at different frame rates. This frameworks allows the VAE to model dependencies in the video at different timescales. The authors evaluate their approach on a variety of datasets: a new benchmark from Minecraft, KTH, Moving MNIST, and GQN Mazes. The paper presents promising qualitative and quantitative results with the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is well written and easy to understand. It flows well and properly structured.\n\n2. The problem is relevant and important to the community: There has been a cornucopia of video modeling papers, but relatively few have addressed the problem of long-term coherence in video generation. This is a fairly unexplored problem relevant to other fields such as Reinforcement Learning.\n\n3. The quantitative and qualitative results on the chosen datasets is promising.\n\nWeaknesses:\n\n1. Choice of metrics. This is an issue common with video modeling papers in general, but metrics such as SSIM, PSNR, and\nLPIPS assume a single ground truth future video. They assume the problem of future video generation is a deterministic problem\nwhen more stochastic enviroments (i.e. real-world videos from UCF101, Kinetics, etc) clearly have multiple valid outcomes. Video\nprediction should be treated as a generative model that estimates the probability distribution of future video. Metrics such as Log-likelihood (or approximations to this), and even crowdsourced human ratings would set a precedent for future work on more complex datasets.\n\n2. The datasets are simplistic. From a computer vision standpoint, the datasets featured have very low visual complexity.\nFrom an event standpoint, KTH, Moving MNIST, and GQN Mazes are domains with very low levels of uncertainty.\n\n\nOverall:\n\nThe paper has a few flaws. The authors choose simple datasets for evaluation and use flawed metrics that may not be the best foundation for future work. However, they explore an important, underexplored problem in video generation. Modeling long-term dependencies in many domains-text, images, and video-is non-trivial. In spite of these issues, I believe this paper is a step in the right direction for research in video generation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed any limitations or social impact.", "review_text": "", "extra_scores": {"ethical_concerns": "I see no ethical issues with this paper. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 Hour"}, "extra_text": {}}, {"review_id": "5WeY6unJlIA", "reviewer": "Reviewer_oxfd", "summary": "This paper presents a video prediction model named Clockwork VAE, which leverages hierarchical latent variables and temporal abstraction to learn long-term dependency. It also proposes the new Minecraft benchmark for long-term video prediction. The model is compared with existing video prediction methods on four datasets, and achieves remarkable long-term prediction results.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. The overall idea of the proposed model is reasonable. It uses a hierarchy of latent variables with a temporally abstract structure to capture long-term video dynamics. It also uses the clockwork recurrent transitions to accommodate different sequence speeds. All the above network designs are integrated in a VAE framework and can be trained in an end-to-end manner. \n2. Impressive qualitative results for long-term prediction.\n\nCons: \n1. The idea of combining temporal abstraction, clockwork transition (state copy/update), and variational models has been introduced by VTA. Though a clear difference is that CW-VAE can be equipped with more than two levels of temporal abstraction (that can better capture hierarchical dynamics), it would be good if the authors could further clarify other differences between these two approaches.\n2. Another concern of mine is about the completeness of the experiments. It seems that VTA is the most important baseline model, so it would be a good choice if the datasets used by VTA can also be included in this paper. Besides, RSSM is mostly used for model-based RL, where long-term prediction is less important due to the use of re-planning and model predictive control. Since none of the compared approaches were originally designed for long-term prediction, it would be good to include other existing models in the experiments, such as the work from Wichers et al. (2018), which also uses hierarchical latent variables to facilitate long-term prediction. \n\nA minor concern: The authors are encouraged to present the quantitative results of the four metrics and to give more description and analysis in the main text, rather than in the supplementary material. Also, from TableE.1, the performance of the proposed model is not much improved compared with RSSM, especially on the MineRL Navigate dataset.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel architecture for long term video generation. Reviewers agree that the approach is novel, it focuses on an important and under-explore problem of video generation and that the experimental section supports the value of the contribution.
Decision: Accept (Poster)