VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation

1 INTRODUCTION . Exponential progress in the capabilities of computational hardware , paired with a relentless effort towards greater insights and better methods , has pushed the field of machine learning from relative obscurity into the mainstream . Progress in the field has translated to improvements in various capabilities , such as classification of images ( Krizhevsky et al. , 2012 ) , machine translation ( Vaswani et al. , 2017 ) and super-human game-playing agents ( Mnih et al. , 2013 ; Silver et al. , 2017 ) , among others . However , the application of machine learning technology has been largely constrained to situations where large amounts of supervision is available , such as in image classification or machine translation , or where highly accurate simulations of the environment are available to the learning agent , such as in game-playing agents . An appealing alternative to supervised learning is to utilize large unlabeled datasets , combined with predictive generative models . In order for a complex generative model to be able to effectively predict future events , it must build up an internal representation of the world . For example , a predictive generative model that can predict future frames in a video would need to model complex real-world phenomena , such as physical interactions . This provides an appealing mechanism for building models that have a rich understanding of the physical world , without any labeled examples . Videos of real-world interactions are plentiful and readily available , and a large generative model can be trained on large unlabeled datasets containing many video sequences , thereby learning about a wide range of real-world phenoma . Such a model could be useful for learning representations for further downstream tasks ( Mathieu et al. , 2016 ) , or could even be used directly in applications where predicting the future enables effective decision making and control , such as robotics ( Finn et al. , 2016 ) . A central challenge in video prediction is that the future is highly uncertain : a short sequence of observations of the present can imply many possible futures . Although a number of recent works have studied probabilistic models that can represent uncertain futures , such models are either extremely expensive computationally ( as in the case of pixel-level autoregressive models ) , or do not directly optimize the likelihood of the data . In this paper , we study the problem of stochastic prediction , focusing specifically on the case of conditional video prediction : synthesizing raw RGB video frames conditioned on a short context ∗A majority of this work was done as part of the Google AI Residency Program . of past observations ( Ranzato et al. , 2014 ; Srivastava et al. , 2015 ; Vondrick et al. , 2015 ; Xingjian et al. , 2015 ; Boots et al. , 2014 ) . Specifically , we propose a new class of video prediction models that can provide exact likelihoods , generate diverse stochastic futures , and accurately synthesize realistic and high-quality video frames . The main idea behind our approach is to extend flow-based generative models ( Dinh et al. , 2014 ; 2016 ) into the setting of conditional video prediction . To our knowledge , flow-based models have been applied only to generation of non-temporal data , such as images ( Kingma & Dhariwal , 2018 ) , and to audio sequences ( Prenger et al. , 2018 ) . Conditional generation of videos presents its own unique challenges : the high dimensionality of video sequences makes them difficult to model as individual datapoints . Instead , we learn a latent dynamical system model that predicts future values of the flow model ’ s latent state . This induces Markovian dynamics on the latent state of the system , replacing the standard unconditional prior distribution . We further describe a practically applicable architecture for flow-based video prediction models , inspired by the Glow model for image generation ( Kingma & Dhariwal , 2018 ) , which we call VideoFlow . Our empirical results show that VideoFlow achieves results that are competitive with the state-ofthe-art in stochastic video prediction on the action-free BAIR dataset , with quantitative results that rival the best VAE-based models . VideoFlow also produces excellent qualitative results , and avoids many of the common artifacts of models that use pixel-level mean-squared-error for training ( e.g. , blurry predictions ) , without the challenges associated with training adversarial models . Compared to models based on pixel-level autoregressive prediction , VideoFlow achieves substantially faster test-time image synthesis 1 , making it much more practical for applications that require real-time prediction , such as robotic control ( Finn & Levine , 2017 ) . Finally , since VideoFlow directly optimizes the likelihood of training videos , without relying on a variational lower bound , we can evaluate its performance directly in terms of likelihood values . 2 RELATED WORK . Early work on prediction of future video frames focused on deterministic predictive models ( Ranzato et al. , 2014 ; Srivastava et al. , 2015 ; Vondrick et al. , 2015 ; Xingjian et al. , 2015 ; Boots et al. , 2014 ) . Much of this research on deterministic models focused on architectural changes , such as predicting high-level structure ( Villegas et al. , 2017b ) , energy-based models ( Xie et al. , 2017 ) , generative cooperative nets ( Xie et al. , 2020 ) , ABPTT ( Xie et al. , 2019 ) , incorporating pixel transformations ( Finn et al. , 2016 ; De Brabandere et al. , 2016 ; Liu et al. , 2017 ) and predictive coding architectures ( Lotter et al. , 2017 ) , as well as different generation objectives ( Mathieu et al. , 2016 ; Vondrick & Torralba , 2017 ; Walker et al. , 2015 ) and disentangling representations ( Villegas et al. , 2017a ; Denton & Birodkar , 2017 ) . With models that can successfully model many deterministic environments , the next key challenge is to address stochastic environments by building models that can effectively reason over uncertain futures . Real-world videos are always somewhat stochastic , either due to events that are inherently random , or events that are caused by unobserved or partially observable factors , such as off-screen events , humans and animals with unknown intentions , and objects with unknown physical properties . In such cases , since deterministic models can only generate one future , these models either disregard potential futures or produce blurry predictions that are the superposition or averages of possible futures . A variety of methods have sought to overcome this challenge by incorporating stochasticity , via three types of approaches : models based on variational auto-encoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , generative adversarial networks ( Goodfellow et al. , 2014 ) , and autoregressive models ( Hochreiter & Schmidhuber , 1997 ; Graves , 2013 ; van den Oord et al. , 2016b ; c ; Van Den Oord et al. , 2016 ) . Among these models , techniques based on variational autoencoders which optimize an evidence lower bound on the log-likelihood have been explored most widely ( Babaeizadeh et al. , 2017 ; Denton & Fergus , 2018 ; Lee et al. , 2018 ; Xue et al. , 2016 ; Li et al. , 2018 ) . To our knowledge , the only prior class of video prediction models that directly maximize the log-likelihood of the data are autoregressive models ( Hochreiter & Schmidhuber , 1997 ; Graves , 2013 ; van den Oord et al. , 2016b ; c ; Van Den Oord et al. , 2016 ) , that generate the video one pixel at a time ( Kalchbrenner et al. , 2017 ) . However , synthesis with such models is typically inherently sequential , making synthesis substantially 1We generate 64x64 videos of 20 frames in less than 3.5 seconds on a NVIDIA P100 GPU as compared to the fastest autoregressive model for video ( Reed et al. , 2017 ) that generates a frame every 3 seconds t ) . We model those levels through a sequential process∏ t ∏ l p ( z ( l ) t | z ( l ) < t , z ( > l ) t ) . inefficient on modern parallel hardware . Prior work has aimed to speed up training and synthesis with such auto-regressive models ( Reed et al. , 2017 ; Ramachandran et al. , 2017 ) . However , ( Babaeizadeh et al. , 2017 ) show that the predictions from these models are sharp but noisy and that the proposed VAE model produces substantially better predictions , especially for longer horizons . In contrast to autoregressive models , we find that our proposed method exhibits faster sampling , while still directly optimizing the log-likelihood and producing high-quality long-term predictions . 3 PRELIMINARIES : FLOW-BASED GENERATIVE MODELS . Flow-based generative models ( Dinh et al. , 2014 ; 2016 ) have a unique set of advantages : exact latentvariable inference , exact log-likelihood evaluation , and parallel sampling . In flow-based generative models ( Dinh et al. , 2014 ; 2016 ) , we infer the latent variable z corresponding to a datapoint x , by transforming x through a composition of invertible functions f = f1 ◦ f2 ◦ · · · ◦ fK . We assume a tractable prior pθ ( z ) over latent variable z , for eg . a Logistic or a Gaussian distribution . By constraining the transformations to be invertible , we can compute the log-likelihood of x exactly using the change of variables rule . Formally , log pθ ( x ) = log pθ ( z ) + K∑ i=1 log |det ( dhi/dhi−1 ) | ( 1 ) where h0 = x , hi = fi ( hi−1 ) , hK = z and |det ( dhi/dhi−1| is the Jacobian determinant when hi−1 is transformed to hi by fi . We learn the parameters of f1 . . . fK by maximizing the log-likelihood , i.e Equation ( 1 ) , over a training set . Given g = f−1 , we can now generate a sample x̂ from the data distribution , by sampling z ∼ pθ ( z ) and computing x̂ = g ( z ) . 4 PROPOSED ARCHITECTURE . We propose a generative flow for video , using the standard multi-scale flow architecture in ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) as a building block . In our model , we break up the latent space z into separate latent variables per timestep : z = { zt } Tt=1 . The latent variable zt at timestep t is an invertible transformation of a corresponding frame of video : xt = gθ ( zt ) . Furthermore , like in ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) , we use a multi-scale architecture for gθ ( zt ) ( Fig . 1 ) : the latent variable zt is composed of a stack of multiple levels : where each level l encodes information about frame xt at a particular scale : zt = { z ( l ) t } Ll=1 , one component z ( l ) t per level . 4.1 INVERTIBLE MULTI-SCALE ARCHITECTURE . We first briefly describe the invertible transformations used in the multi-scale architecture to infer { z ( l ) t } Ll=1 = fθ ( xt ) and refer to ( Dinh et al. , 2016 ; Kingma & Dhariwal , 2018 ) for more details . For convenience , we omit the subscript t in this subsection . We choose invertible transformations whose Jacobian determinant in Equation 1 is simple to compute , that is a triangular matrix , diagonal matrix or a permutation matrix as explored in prior work ( Rezende & Mohamed , 2015 ; Deco & Brauer , 1995 ) . For permutation matrices , the Jacobian determinant is one and for triangular and diagonal Jacobian matrices , the determinant is simply the product of diagonal terms . • Actnorm : We apply a learnable per-channel scale and shift with data-dependent initialization . • Coupling : We split the input y equally across channels to obtain y1 and y2 . We compute z2 = f ( y1 ) ∗ y2 + g ( y1 ) where f and g are deep networks . We concat y1 and z2 across channels . • SoftPermute : We apply a 1x1 convolution that preserves the number of channels . • Squeeze : We reshape the input from H ×W × C to H/2×W/2× 4C which allows the flow to operate on a larger receptive field . We infer the latent variable z ( l ) at level l using : Flow ( y ) = Coupling ( SoftPermute ( Actnorm ( y ) ) ) ) ×N ( 2 ) Flowl ( y ) = Split ( Flow ( Squeeze ( y ) ) ) ( 3 ) ( h ( > l ) , zl ) ← Flowl ( h ( > l−1 ) ) ( 4 ) where N is the number of steps of flow . In Equation ( 3 ) , via Split , we split the output of Flow equally across channels into h ( > l ) , the input to Flow ( l+1 ) ( . ) and z ( l ) , the latent variable at level l. We , thus enable the flows at higher levels to operate on a lower number of dimensions and larger scales . When l = 1 , h ( > l−1 ) is just the input frame x and for l = L we omit the Split operation . Finally , our multi-scale architecture fθ ( xt ) is a composition of the flows at multiple levels from l = 1 . . . L from which we obtain our latent variables i.e { z ( l ) t } Ll=1 .

Summary: This paper extended the flow-based generative model for stochastic video prediction. The proposed model takes an advantage of the flow-based models which provide exact latent-variable inference, exact log-likelihood evaluation, and efficiency. The paper used the autoregressive model and the multi-scale Glow architecture. The experiments on the stochastic movement dataset (synthetic) and the BAIR Robot push dataset show the performance improvement against other state-of-the-art stochastic video generation models (SV2P and SAVP-VAE).