Diverse Video Generation using a Gaussian Process Trigger

1 INTRODUCTION . Humans are often able to imagine multiple possible ways that the scene can change over time . Modeling and generating diverse futures is an incredibly challenging problem . The challenge stems from the inherent multi-modality of the task , i.e. , given a sequence of past frames , there can be multiple possible outcomes of the future frames . For example , given the image of a “ person holding a cup ” in Figure . 1 , most would predict that the next few frames correspond to either the action “ drinking from the cup ” or “ keeping the cup on the table. ” This challenge is exacerbated by the lack of real training data with diverse outputs – all real-world training videos come with a single real future and no “ other ” potential futures . Similar looking past frames can have completely different futures ( e.g. , Figure . 1 ) . In the absence of any priors or explicit supervision , the current methods struggle with modeling this diversity . Given similar looking past frames , with different futures in the training data , variational methods , which commonly utilize ( Kingma & Welling , 2013 ) , tend to average the results to better match to all different futures ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2017 ; Gao et al. , 2018 ; Lee et al. , 2018 ; Oliu et al. , 2017 ) . We hypothesize that explicit modeling of future diversity is essential for high-quality , diverse future frame generation . In this paper , we model the diversity of the future states , given past context , using Gaussian Processes ( GP ) ( Rasmussen , 2006 ) , which have several desirable properties . They learn a prior on potential future given past context , in a Bayesian formulation . This allows us to update the distribution of possible futures as more context frames are provided as evidence and maintain a list of potential futures ( underlying functions in GP ) . Finally , our formulation provides an interesting property that is crucial to generating future frames – the ability to estimate when to generate a diverse output vs. continue an on-going action , and a way to control the predicted futures . In particular , we utilize the variance of the GP at any specific time step as an indicator of whether an action sequence is on-going or finished . An illustration of this mechanism is presented in Figure . 2 . When we observe a frame ( say at time t ) that can have several possible futures , the variance of the GP model is high ( Figure . 2 ( left ) ) . Different functions represent potential action sequences that can be generated , starting from this particular frame . Once we select the next frame ( at t+2 ) , the GP variance of the future states is relatively low ( Figure . 2 ( center ) ) , indicating that an action sequence Potential Future 1 Potential Future 2 Person Holding Cup Person Keeping Cup on the Table Person Drinking from the Cup Past Figure 1 : Given “ person holding cup , ” humans can often predict multiple possible futures ( e.g. , “ drinking from the cup ” or “ keeping the cup on the table. ” ) . is on-going , and the model should continue it as opposed to trying to sample a diverse sample . After the completion of the on-going sequence , the GP variance over potential future states becomes high again . This implies that we can continue this action ( i.e. , pick the mean function represented by the black line in Figure . 2 ( center ) ) or try and sample a potentially diverse sample ( i.e. , one of the functions that contributes to high-variance ) . This illustrates how we can use GP to decide when to trigger diverse actions . An example of using GP trigger is shown in Figure . 2 ( right ) , where after every few frames , we trigger a different action . Now that we have a good way to model diversity , the next step is to generate future frames . Even after tremendous advances in the field of generative models for image synthesis ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ; Vondrick & Torralba , 2017 ; Lu et al. , 2017 ; Vondrick et al. , 2016 ; Saito et al. , 2017 ; Tulyakov et al. , 2018 ; Hu & Wang , 2019 ) , the task of generating future frames ( not necessarily diverse ) conditioned on past frames is still hard . As opposed to independent images , the future frames need to obey potential video dynamics that might be on-going in the past frames , follow world knowledge ( e.g. , how humans and objects interact ) , etc .. We utilize a fairly straightforward process to generate future frames , which utilizes two modules : a frame auto-encoder and a dynamics encoder . The frame auto-encoder learns to encode a frame in a latent representation and utilizes it to generate the original frame back . The dynamics encoder learns to model dynamics between past and future frames . We learn two independent dynamics encoders : an LSTM encoder , utilized to model on-going actions and the GP encoder ( similar to ( Srivastava et al. , 2015 ) ) , and a GP encoder , utilized to model transitions to new actions . The variance of this GP encoder can be used as a trigger to decide when to sample new actions . We train this framework end-to-end . We first provide an overview of GP formulation and scalable training techniques in §3 , and then describe our approach in §4 . Comprehensively evaluating diverse future frames generation is still an open research problem . Following recent state-of-the-art , we will evaluate different aspects of the approach independently . The quality of generated frames is quantified using image synthesis/reconstruction per-frame metrics : SSIM ( Wang et al. , 2019 ; Sampat et al. , 2009 ) , PSNR , and LPIPS ( Zhang et al. , 2018 ; Dosovitskiy & Brox , 2016 ; Johnson et al. , 2016 ) . The temporal coherence and quality of a short video clip ( 16 neighboring frames ) are jointly evaluated using the FVD ( Unterthiner et al. , 2018 ) metric . However , high-quality , temporarily coherent frame synthesis does not evaluate diversity in predicted frames . Therefore , to evaluate diversity , since there are no multiple ground-truth futures , we propose an alternative evaluation strategy , inspired by ( Villegas et al. , 2017b ) : utilizing action classifiers to evaluate whether an action switch has occurred or not . A change in action indicates that the method was able to sample a diverse future . Together , these metrics can evaluate if an approach can generate multiple high-quality frames that temporally coherent and diverse . Details of these metrics and baselines , and extensive quantitative and qualitative results are provided in §5 . To summarize , our contributions are : ( a ) modeling the diversity of future states using a GP , which maintains priors on future states given the past frames using a Bayesian formulation ( b ) leveraging the changing GP distribution over time ( given new observed evidence ) to estimate when an on-going action sequence completes and using GP variance to control the triggering of a diverse future state . This results in state-of-the-art results on future frame generation . We also quantify the diversity of the generated sequences using action classifiers as a proxy metric . 2 RELATED WORK . Understanding and predicting the future , given the observed past , is a fundamental problem in video understanding . The future states are inherently multi-modal and capturing their diversity finds direct applications in many safety-critical applications ( e.g. , autonomous vehicles ) , where it is critical to model different future modes . Earlier techniques for future prediction ( Yuen & Torralba , 2010 ; Walker et al. , 2014 ) relied on searching for matches of past frames in a given dataset and transferring the future states from these matches . However , the predictions were limited to symbolic trajectories or retrieved future frames . Given the modeling capabilities of deep representations , the field of future frame prediction tremendous progress in recent years . One of the first works on video generation ( Srivastava et al. , 2015 ) used a multi-layer LSTM network to learn representations of video sequences in a deterministic way . Since then , a wide range of papers ( Oliu et al. , 2017 ; Cricri et al. , 2016 ; Villegas et al. , 2017a ; Elsayed et al. , 2019 ; Villegas et al. , 2019 ; Wang et al. , 2019 ; Castrejón et al. , 2019 ) have built models that try to incorporate stochasticity of future states . Generally , this stochasticity lacks diverse high-level semantic actions . Recently , several video generation models have utilized generative models , like variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) and generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) , for this task . One of the first works by Xue et al . ( 2016 ) utilized a conditional VAE ( cVAE ) formulation to learn video dynamics . Similar to our approach , their goal was to model the frame prediction problem in a probabilistic way and synthesizing many possible future frames from a single image . Since then , several works have utilized the cVAE for future generation ( Babaeizadeh et al. , 2017 ; Denton & Fergus , 2018 ) . The major drawback of using the cVAE approach is that its objective function marginalizes over the multi-modal future , limiting the diversity in the generated samples ( Bhattacharyya et al. , 2018 ) . GAN-based models are another important class of synthesis models used for future frame prediction or video generation ( Vondrick & Torralba , 2017 ; Lu et al. , 2017 ; Vondrick et al. , 2016 ; Saito et al. , 2017 ; Tulyakov et al. , 2018 ; Hu & Wang , 2019 ) . However , these models are very susceptible to mode collapse ( Salimans et al. , 2016 ) , i.e. , the model outputs only one or a few modes instead of generating a wide range of diverse output . The problem of mode collapse is quite severe for conditional settings , as demonstrated by ( Isola et al. , 2016 ; Zhu et al. , 2017 ; Mathieu et al. , 2015 ) . This problem is worse in the case of diverse future frame generation due to the inherent multi-modality of the output space and lack of training data . Another class of popular video generation models is hierarchical prediction ( Walker et al. , 2017 ; Villegas et al. , 2017b ; Wichers et al. , 2018 ; Cai et al. , 2018 ) . These models decompose the problems into two steps . They first predict a high-level structure of a video , like a human pose , and then leverage that structure to make predictions at the pixel level . These models generally require additional annotation for the high-level structure for training . Unlike these approaches , Our approach explicitly focuses on learning the distribution of diverse futures using a GP prior on the future states using a Bayesian formulation . Moreover , such GP approaches have been used in the past for modeling the human dynamics as demonstrated by ( Wang et al. , 2008 ; Hardy et al. , 2014 ; Moon & Pavlovic , 2008 ) . However , due to the scalability issues in GP , these models were limited to handling low dimensional data , like human pose , lane switching , path planning , etc . To the best of our knowledge , ours is the first approach that can process video sequences to predict when an on-going action sequence completes and control the generation of a diverse state .

Summary: The authors propose to use a Gaussian Process (GP) to model the uncertainty of future frames in a video prediction setup. In particular, they employ a GP to model the uncertainty of the next step latent in a latent variable model. This allows them to use the GP variance to decide when to change an "action sequence", corresponding to a deterministic dynamics function implemented using an LSTM.