Video Prediction with Variational Temporal Hierarchies

1 INTRODUCTION Deep learning has enabled predicting video sequences from large datasets ( Chiappa et al. , 2017 ; Oh et al. , 2015 ; Vondrick et al. , 2016 ) . For high-dimensional inputs such as video , there likely exists a more compact representation of the scene that facilitates long term prediction . Instead of learning dynamics in pixel space , latent dynamics models predict ahead in a more compact feature space ( Doerr et al. , 2018 ; Buesing et al. , 2018 ; Karl et al. , 2016 ; Hafner et al. , 2019 ) . This has the added benefit of increased computational efficiency and a lower memory footprint , allowing to predict thousands of sequences in parallel using a large batch size . A lot of work in deep learning has focused on spatial abstraction , following the advent of convolutional networks ( LeCun et al. , 1989 ) , such as the Variational Ladder Autoencoder ( Zhao et al. , 2017 ) that learns a hierarchy of features in images using networks of different capacities , along with playing an important role in the realm of video prediction models ( Castrejón et al. , 2019 ) . Recent sequential models have incorporated temporal abstraction for learning dependencies in temporally distant observations ( Koutník et al. , 2014 ; Chung et al. , 2016 ) . Kim et al . ( 2019 ) proposed Variational Temporal Abstraction ( VTA ) , in which they explored one level of temporal abstraction above the latent states , the transition of which was modeled using a Bernoulli random variable . In this paper , we intend to work in a more controlled setup than VTA for a qualitative and quantitative analysis of temporally abstract latent variable models . In this paper , we study the benefits of temporal abstraction using a hierarchical latent dynamics model , trained using a variational objective . Each level in the hierarchy of this model temporally abstracts the level below by an adjustable factor . This model can perform long-horizon video prediction of 200 frames , while predicting accurate low-level information for a 6 times longer duration than the baseline model . We study the information stored at different levels of the hierarchy via KL divergence , predictive entropy , datasets of varying speeds , and generative distributions . In our experiments we show that this amounts to object location and identities for the Moving MNIST dataset , and the wall or floor patterns for the GQN mazes dataset ( Eslami et al. , 2018 ) , stored at different levels . Our key contributions are summarized as follows : • Temporal Abstract Latent Dynamics ( TALD ) We introduce a simple model with different clock speeds at every level to study the properties of variational hierarchical dynamics . • Accurate long-term predictions Our form of temporal abstraction substantially improves for how long the model can accurately predict video frames into the future . • Adaptation to sequence speed We demonstrate that our model automatically adapts the amount of information processed at each level to the speed of the video sequence . • Separation of information We visualize the content represented at each level of the hierarchy to find location information in lower levels and object identity in higher levels . 2 RELATED WORK . Generative video models A variety of methods have successfully approached video prediction using large datasets ( Chiappa et al. , 2017 ; Oh et al. , 2015 ; Vondrick et al. , 2016 ; Babaeizadeh et al. , 2017 ; Gemici et al. , 2017 ; Ha & Schmidhuber , 2018 ) . Denton & Fergus ( 2018 ) proposed a stochastic video generation model with a learned prior that transitions in time , and is conditioned on past observations . Lee et al . ( 2018 ) proposed to use an adversarial loss with a variational latent variable model to produce naturalistic images , while Kumar et al . ( 2019 ) used flow-based generative modeling to directly optimize the likelihood of a video generation model . Recently , Weissenborn et al . ( 2020 ) scaled autoregressive models for video prediction using a three-dimensional self-attention mechanism and showed competitive results on real-world video datasets . On similar lines , Xu et al . ( 2018 ) proposed to use an entirely CNN-based architecture for modeling dependencies between sequential inputs . Latent dynamics models Latent dynamics models have evolved from latent space models that had access to low-dimensional features ( Deisenroth & Rasmussen , 2011 ; Higuera et al. , 2018 ) , to models that can build a compact representation of visual scenes and facilitate video prediction purely in the latent space ( Doerr et al. , 2018 ; Buesing et al. , 2018 ; Karl et al. , 2016 ; Franceschi et al. , 2020 ) . The Variational RNN ( Chung et al. , 2015 ) uses an auto-regressive state transition that takes inputs from observations , making it computationally expensive to be used as an imagination module . Hafner et al . ( 2019 ) proposed a latent dynamics model , which is a combination of deterministic and stochastic states , that enables the model to deterministically remember all previous states and filter that information to obtain a distribution over the current state . Hierarchical latent variables Learning per-frame hierarchical structures has proven to be helpful in generating videos on long-term horizon ( Wichers et al. , 2018 ) . Zhao et al . ( 2017 ) proposed the Variational Ladder Autoencoder ( VLAE ) that uses networks of different capacities at different levels of the hierarchy , encouraging the model to store high-level image features at the top level , and simple features at the bottom . Other recently proposed hierarchical models use a purely bottom-up inference approach with no interaction between the inference and generative models ( Kingma & Welling , 2014 ; Rezende & Mohamed , 2015 ; Rezende et al. , 2014 ) . In contrast , Sønderby et al . ( 2016 , LVAE ) and Vahdat & Kautz ( 2020 , NVAE ) proposed to use a combination of bottom-up and top-down inference procedures , sharing parameters between the inference and generative distributions during the top-down pass . We incorporate this conditional structure in our model design as well . Temporal abstraction Identifying complex dependencies between temporally distant observations is a challenging task and has inspired a variety of fundamental work in recurrent models ( Koutník et al. , 2014 ; Chung et al. , 2016 ) . However , relatively few works have demonstrated modeling longterm dependencies using temporally abstract latent dynamics models ( Wichers et al. , 2018 ; Jaderberg et al. , 2018 ) . Recently , Kim et al . ( 2019 ) introduced Variational Temporal Abstraction ( VTA ) to learn temporally abstract latent spaces . They explored one level of temporal abstraction above the latent states , the transition of which was modeled using a Bernoulli random variable , that chose between ‘ copy ’ or ‘ update ’ steps . Inspired by this work , we aim to gain a deeper understanding of such temporally-abstract latent dynamics models . We perform our analysis on a model that is simplified to using fixed time scales for every level . Moreover , the lower level is a continuing chain in our model , whereas VTA resets transitions at a lower level when transitioning at a higher level . 3 TEMPORALLY ABSTRACT LATENT DYNAMICS . Long video sequences contain both information that is local to a few frames as well as global information that is shared among many frames . Traditional video prediction models that predict ahead at the frame rate of the video can struggle to retain information long enough to learn such longterm dependencies . We introduce Temporally Abstract Latent Dynamics ( TALD ) to learn long-term correlations of videos . Our model predicts ahead on multiple time scales to learn dependencies at different temporal levels , as visualized in Figure 3 . We build our work upon the recurrent state-space model ( RSSM ; Hafner et al. , 2019 ) , the details of which can be found in Appendix A. TALD consists of a hierarchy of recurrent latent variables , where each level transitions at a different clock speed . We slow down the transitions exponentially as we go up in the hierarchy , i.e . every level being slower than the level below by a factor of k. We denote a set of active timesteps for every level l ∈ [ 1 , L ] as those steps in time where the state transition generates a new latent state , Active timesteps : Tl . = { t ∈ [ 1 , T ] | tmod kl−1 = 1 } . ( 1 ) At each level , we condition every window of k latent states on a single latent variable in the level above . This can also be thought of as a hierarchy of latent variables where each level has the same clock speed , but performs a state transition every kl−1 timesteps and copies the same state variable otherwise , so that ∀t /∈ Tl : Inactive states : slt . = slmax τ { τ∈Tl|τ≤t } . ( 2 ) Joint distribution We can factorize the joint distribution of a sequence of observations and ( active ) latents at every level into two terms : ( 1 ) a decoder term conditioned on the latent states in the lowest level , and ( 2 ) state transitions at all levels conditioned on the latent state of the last active timestep at the current level and the level above , p ( x1 : T , s 1 : L 1 : T ) . = ( ∏T t=1 p ( xt | s1t ) ) ( ∏L l=1 ∏ t∈Tl p ( s l t | slt−1 , sl+1t ) ) . ( 3 ) Inference For inference , TALD embeds observed frames using a CNN . A hierarchical recurrent network then summarizes the input embeddings , for which each ( active ) latent state at a level l receives embeddings from kl−1 observation frames ( dashed lines in Figure 3 ) . The latent state at the previous timestep at the current level , and the state belief at the level above also condition the posterior belief ( solid lines in Figure 3 ) . The input embeddings combined with this top-down and temporal context together condition the posterior belief qlt over the latent state . Generation The prior transition plt is computed by conditioning over the latent state at the previous timestep at the current level , and the state belief at the level above ( solid lines in Figure 3 ) . Decoding Finally , the state beliefs at the bottom-most level are decoded using a transposed CNN to provide a training signal . To summarize , we utilize the following components in our model , ∀ l ∈ [ 1 , L ] , t ∈ Tl , Encoder : elt = e ( xt : t+kl−1−1 ) Posterior transition qlt : q ( s l t | slt−1 , sl+1t , elt ) Prior transition plt : p ( s l t | slt−1 , sl+1t ) Decoder : p ( xt | s1t ) . ( 4 ) Training objective Since we can not compute the likelihood of the training data under the model in closed form , we use the ELBO as our training objective . This training objective optimizes a reconstruction loss at the lowest level , and a KL regularizer at every level in the hierarchy summed across active timesteps , max e , h , q , p ∑T t=1 Eq1t [ ln p ( xt | s 1 t ) ] − ∑L l=1 ∑ t∈Tl KL [ q l t ‖ plt ] . ( 5 ) The KL regularizer at each level limits the amount of information that filters through the encoder and stays in the posterior at that level . This encourages the model to utilize the state transitions and context from the level above as much as possible . Since the number of active timesteps decreases as we go higher in the hierarchy , the number of KL terms per level decreases as well . Hence it is easier for the model to push global information high up in the hierarchy and pay lesser KL penalty , instead of transitioning those bits with an identity transformation at a lower level . Stochastic and Deterministic Path As illustrated in Figure 3 ( right ) , we split the state slt into stochastic ( zlt ) and deterministic ( h l t ) parts ( Hafner et al. , 2019 ) . The deterministic state is computed using the top-down and temporal context , which then conditions the stochastic state at that level . The stochastic states follow a diagonal Gaussian , with mean and variance predicted by a neural network . We use a GRU ( Cho et al. , 2014 ) per level to update the deterministic state at every active timestep . All components in Equation 4 are trained jointly by optimizing Equation 5 using stochastic backpropagation with reparameterized sampling . Please refer to Appendix B for architectures details . 4 EXPERIMENTS We aim to evaluate temporally-abstract latent dynamics models at modeling long-term dependencies in video . Moreover , we aim to understand how they separate information into different levels of the hierarchy . To investigate these questions , we train TALD described in Section 3 , the temporally-abstract VTA model ( Kim et al. , 2019 ) , the RSSM model without temporal abstraction ( Hafner et al. , 2019 ) , and the image-space video prediction model SVGLP ( Denton & Fergus , 2018 ) on four datasets of varying complexity . We consider the well- established Moving MINST dataset ( Srivastava et al. , 2015 ) , the KTH Action dataset ( Schuldt et al. , 2004 ) , the GQN mazes dataset ( Eslami et al. , 2018 ) , and the MineRL Navigate dataset ( Guss et al. , 2019 ) . We evaluate open-loop video predictions on these datasets using four quantitative metrics : Structural Similarity index ( SSIM ; higher is better ) , Peak Signal-to-Noise Ratio ( PSNR ; higher is better ) , Learned Perceptual Image Patch Similarity ( LPIPS ; lower is better ) ( Zhang et al. , 2018 ) , and Frechet Video Distance ( FVD ; lower is better ) ( Unterthiner et al. , 2018 ) . In Section 4.5 , we investigate how the amount of information stored at different levels of a temporal hierarchy adapts to changes in sequence speed . In Section 4.6 , we visualize the information stored at different levels by resetting individual levels of the hierarchy . We trained all our models using sequences of length 100 . We used convolutional frame encoders and decoders , with architectures very similar to the DCGAN ( Radford et al. , 2016 ) discriminator and generator , respectively . Our implementations made use of TensorFlow Probability ( Dillon et al. , 2017 ) and CuDNN , and used the Adam optimizer ( Kingma & Ba , 2014 ) for training . The training time for a 3-level TALD model with temporal abstraction 6 amounted to around 24 hours for 100 epochs on a single NVIDIA TITAN Xp GPU . Refer to Appendix C for hyperparameters and experimental setup .

Summary: This paper proposes a method called Temporal Abstract Latent Dynamics (TALD). TALD is built up on RSSM (Hafner et al. 2019) but with hierarchical dynamics. The experiments are conducted on moving MNIST, GQN 3D Mazes, and KTH. Results are qualitatively better than other methods in term of maintaining long-term consistent prediction. Quantitative comparison is reported only on KTH dataset (Figure 5). Written presentation is clear and easy to understand.