VideoGen: Generative Modeling of Videos using VQ-VAE and Transformers

1 INTRODUCTION . Deep generative models of multiple types ( Goodfellow et al. , 2014 ; van den Oord et al. , 2016b ; Dinh et al. , 2016 ) have seen incredible progress in the last few years on multiple modalities including natural images ( van den Oord et al. , 2016c ; Zhang et al. , 2019 ; Brock et al. , 2018 ; Kingma & Dhariwal , 2018 ; Ho et al. , 2019a ; Karras et al. , 2017 ; 2019 ; Van Den Oord et al. , 2017 ; Razavi et al. , 2019 ; Vahdat & Kautz , 2020 ; Ho et al. , 2020 ; Chen et al. , 2020 ) , audio waveforms conditioned on language features ( van den Oord et al. , 2016a ; Oord et al. , 2017 ; Bińkowski et al. , 2019 ) , natural language in the form of text ( Radford et al. , 2019 ; Brown et al. , 2020 ) , and music generation ( Dhariwal et al. , 2020 ) . These results have been made possible thanks to fundamental advances in deep learning architectures ( He et al. , 2015 ; van den Oord et al. , 2016b ; c ; Vaswani et al. , 2017 ; Zhang et al. , 2019 ; Menick & Kalchbrenner , 2018 ) as well as the availability of compute resources ( Jouppi et al. , 2017 ; Amodei & Hernandez , 2018 ) that are more powerful than a few years ago . However , one notable modality that has not seen the same level of progress in generative modeling is high fidelity natural videos . The complexity of natural videos requires modeling correlations across both space and time with much higher input dimensions , thereby presenting a natural next challenge for current deep generative models . The complexity of the problem also demands more compute resources which can be considered as one important reason for the slow progress in generative modeling of videos . It is useful to build generative models of videos , both conditional and unconditional , as it implicitly solves the problem of video prediction and forecasting . Video prediction ( Kalchbrenner et al. , 2017 ; Sønderby et al. , 2020 ) can be seen as learning a generative model of future frames conditioned on the past frames . Architectures developed for video generation can be useful in forecasting applications for autonomous driving , such as predicting the future in more semantic and dense abstractions like segmentation masks ( Luc et al. , 2017 ) . Finally , building generative models of the world around us is considered as one way to measure understanding of physical common sense ( Lake et al. , 2015 ) . Multiple classes of generative models have been shown to produce strikingly good samples such as autoregressive models ( van den Oord et al. , 2016b ; c ; Menick & Kalchbrenner , 2018 ; Radford et al. , 2019 ; Chen et al. , 2020 ) , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ) , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Kingma et al. , 2016 ; Vahdat & Kautz , 2020 ) , Flows ( Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ) , vector quantized VAE ( VQ-VAE ) ( Van Den Oord et al. , 2017 ; Razavi et al. , 2019 ) , and lately diffusion and score matching models ( Sohl-Dickstein et al. , 2015 ; Song & Ermon , 2019 ; Ho et al. , 2020 ) . These different generative model families have their tradeoffs : sampling speed , sample diversity , sample quality , ease of training , compute requirements , and ease of evaluation . To build a generative model for videos , we first make a choice between likelihood-based and adversarial models . Likelihood-based models are convenient to train since the objective is well understood , easy to optimize across a range of batch sizes , and easy to evaluate . Given that videos already present a hard modeling challenge due to the nature of the data , we believe likelihood-based models present fewer difficulties in the optimization and evaluation , hence allowing us to focus on the architecture modeling . Among likelihood-based models , autoregressive models that work on discrete data in particular have shown great success and have well established training recipes and modeling architectures . Second , we consider the following question : Is it better to perform autoregressive modeling in a downsampled latent space without spatio-temporal redundancies compared to modeling at the atomic level of all pixels across space and time ? Below , we present our reasons for choosing the former : Natural images and videos contain a lot of spatial and temporal redundancies and hence the reason we use image compression tools such as JPEG ( Wallace , 1992 ) and video codecs such as MPEG ( Le Gall , 1991 ) everyday . These redundancies can be removed by learning a denoised downsampled encoding of the high resolution inputs . For example , 4x downsampling across spatial and temporal dimensions results in 64x downsampled resolution so that the computation of powerful deep generative models is spent on these more fewer and useful bits . As shown in VQ-VAE ( Van Den Oord et al. , 2017 ) , even a lossy decoder can transform the latents to generate sufficiently realistic samples . Furthermore , modeling in the latent space downsampled across space and time instead of the pixel space improves sampling speed and compute requirements due to reduced dimensionality . 1 The above line of reasoning leads us to our proposed model : VideoGen , a simple video generation architecture that is a minimal adaptation of VQ-VAE and GPT architectures for videos . VideoGen employs 3D convolutions and transposed convolutions ( Tran et al. , 2015 ) along with axial attention ( Clark et al. , 2019 ; Ho et al. , 2019b ) for the autoencoder in VQ-VAE in order to be able to learn a downsampled set of discrete latents . These latents are then autoregressively generated by a GPT-like ( Radford et al. , 2019 ; Child et al. , 2019 ; Chen et al. , 2020 ) architecture . The latents are then decoded to videos of the original resolution using the decoder of the VQ-VAE . Our results are highlighted below : 1 . On the widely benchmarked BAIR Robot Pushing dataset ( Ebert et al. , 2017 ) , VideoGen can generate realistic samples that are competitive with existing methods such as DVD-GAN ( Clark et al. , 2019 ) , achieving an FVD of 112 when benchmarked with real samples , and an FVD * ( Razavi et al. , 2019 ) of 94 when benchmarked with reconstructions . 2 . VideoGen can easily be adapted for action conditional video generation . We present qualitative results on the BAIR Robot Pushing dataset and Vizdoom simulator ( Kempka et al. , 2016 ) . 3 . We present ablations showing that employing axial attention blocks in the VQ-VAE and spatiotemporal position encodings in the Transformer are helpful design choices in VideoGen . 4 . Our results are achievable with a maximum of 8 Quadro RTX 6000 GPUs ( 24 GB memory ) , significantly lower than the resources used in prior methods such as DVD-GAN ( Clark et al. , 2019 ) ( 32 to 512 16GB TPU ( Jouppi et al. , 2017 ) cores ) . 1Modeling long sequences is a challenge for transformer based architectures due to quadratic memory complexity of the attention matrix ( Child et al. , 2019 ) . 2 BACKGROUND . 2.1 VQ-VAE . The Vector Quantized Variational Autoencoder ( VQ-VAE ) ( Van Den Oord et al. , 2017 ) is a model that learns to compress high dimensional data points into a discretized latent space and reconstruct them . The encoder E ( x ) → h first encodes x into a series of latent vectors h which is then discretized by performing a nearest neighbors lookup in a codebook of embeddings C = { ei } Ki=1 of size K. The decoder D ( e ) → x̂ then learns to reconstruct x from the quantized encodings . The VQ-VAE is trained using the following objective : L = ‖x−D ( e ) ‖22︸ ︷︷ ︸ Lrecon + ‖sg [ E ( x ) ] − e‖22︸ ︷︷ ︸ Lcodebook +β ‖sg [ e ] − E ( x ) ‖22︸ ︷︷ ︸ Lcommit where sg refers to a stop-gradient . The objective consists of a reconstruction loss Lrecon , a codebook loss Lcodebook , and a commitment loss Lcommit . The reconstruction loss encourages the VQ-VAE to learn good representations to accurately reconstruct data samples . The codebook loss brings codebook embeddings closer to their corresponding encoder outputs , and the commitment loss is weighted by a hyperparameter β and prevents the encoder outputs from fluctuating between different code vectors . An alternative replacement for the codebook loss described in Van Den Oord et al . ( 2017 ) is to use an EMA update which empirically shows faster training and convergence speed . In this paper , we use the EMA update when training the VQ-VAE . 2.2 GPT . GPT and Image-GPT ( Chen et al. , 2020 ) are a class of autoregressive transformers that have shown tremendous success in modelling discrete data such as natural language and high dimensional images . These models factorize the data distribution p ( x ) according to p ( x ) = ∏d i=1 p ( xi|x < i ) through masked self-attention mechanisms and are optimized through maximum likelihood . As in Vaswani et al . ( 2017 ) , the architectures follow the standard design of employing multi-head self-attention blocks followed by pointwise MLP feedforward blocks . 3 VIDEOGEN . Our primary contribution is VideoGen , a new method to model complex video data in a computationally efficient manner . An overview of our method is shown in Fig 2 . Learning Latent Codes In order to learn a set of discrete latent codes , we first train a VQ-VAE on the video data . The encoder architecture consists of a series of 3D convolutions that downsample over space-time , followed by attention residual blocks . Each attention residual block is designed as shown in Fig 3 , where we use LayerNorm ( Ba et al. , 2016 ) , and axial attention layers follow Ho et al . ( 2019b ) . The architecture for the decoder is the reverse of the encoder , with attention residual blocks followed by a series of 3D transposed convolution that upsample over space-time . The position encodings are learned spatio-temporal embeddings that are shared between all axial attention layers in the encoder and decoder . Learning a Prior The second stage of our method is to learn a prior over the latents . The prior is learned by training a transformer model over the VQ-VAE latents . We follow the iGPT architecture with added dropout after the feedforward and attention block layers for regularization . Although the VQ-VAE is trained unconditionally , we can generate conditional samples by training a conditional prior . We use two types of conditioning : • Concatenation : We concatenate a conditional vector before every feedforward block in the transformer . This conditioning method is primarily used for frame conditioning , where the conditioned frame is encoded into a conditioning vector by a ResNet ( He et al. , 2016 ) backbone and then concatenated . • Conditional Norms : Similar to conditioning methods used in GANs , we parameterize the gain and bias in the transformer Layer Normalization ( Ba et al. , 2016 ) layers as affine functions of the conditional vector . This conditioning method is used for action conditioning .

Summary: This paper proposes a generative model to synthesize videos using VQ-VAEs. The scheme works in latent space by using embeddings for video sequences learnt by the VQ-VAE. For inference, an autoregressive transformer prior for video sequences is learnt, which upon sampling from and sending to the VQ-VAE decoder, generates unconditional (or conditional) samples of video. To learn video embeddings, the paper uses a 3D convolutional network, with an extra dimension for time.