Consistent Jumpy Predictions for Videos and Scenes

1 INTRODUCTION . The ability to fill in the gaps in high-dimensional data is a fundamental cognitive skill . Suppose you glance out of the window and see a person in uniform approaching your gate carrying a letter . You can easily imagine what will ( probably ) happen a few seconds later . The person will walk up the path and push the letter through your door . Now suppose you glance out of the window the following day and see a person in the same uniform walking down the path , away from the house . You can easily imagine what ( probably ) happened a few seconds earlier . The person came through your gate , walked up the path , and delivered a letter . Moreover , in both instances , you can visualize the scene from different viewpoints . From your vantage point at the window , you can imagine how things might look from the gate , or from the front door , or even from your neighbour ’ s roof . Essentially , you have learned from your past experience of unfolding visual scenes how to flexibly extrapolate and interpolate in both time and space . Replicating this ability is a significant challenge for AI . To make this more precise , let ’ s first consider the traditional video prediction setup . Video prediction is typically framed as a sequential forward prediction task . Given a sequence of frames f1 , ... , ft , a model is tasked to generate future frames that follow , ft+1 , ... , fT . Existing models ( Babaeizadeh et al. , 2018 ; Kalchbrenner et al. , 2017 ; Finn et al. , 2016 ) carry out the prediction in an autoregressive manner , by sequentially sampling frame ft+n from p ( ft+n|f1 , ... , ft+n−1 ) . This traditional setup is often limiting – in many cases , we are interested in what happens a few seconds after the input sequence . For example , in model based planning tasks , we might want to predict what frame fT is , but might not care what intermediate frames ft+1 , ... , fT−1 are . Existing models must still produce the intermediate frames , which is inefficient . Instead , our model is “ jumpy ” – it can directly sample frames in the future , bypassing intermediate frames . For example , in a 40-frame video , our model can sample the final frame 12 times faster than an autoregressive model like SV2P ( Babaeizadeh et al. , 2018 ) . More generally , existing forward video prediction models are not flexible at filling in gaps in data . Instead , our model can sample frames at arbitrary time points of a video , given a set of frames at arbitrary time points as context . So it can , for example , be used to infer backwards or interpolate between frames , as easily as forwards . Our model is also not autoregressive on input or output frames – it takes in each input frame in parallel , and samples each output frame in parallel . In our setup of “ jumpy ” video prediction , a model is given frames f1 , ... , fn from a single video along with the arbitrary time-points t1 , ... , tn at which those frames occurred . To be clear , t1 , ... , tn need not be consecutive or even in increasing order . The model is then asked to sample plausible frames at arbitrary time points t′1 , ... , t ′ k. In many cases there are multiple possible predicted frames given the context , making the problem stochastic . For example , a car moving towards an intersection could turn left or turn right . Although our model is not autoregressive , each set of k sampled frames is consistent with a single coherent possibility , while maintaining diversity across sets of samples . That is , in each sampled set all k sampled frames correspond to the car moving left , or all correspond to the car moving right . At a high level , our model , JUMP , takes in the input frames and samples a stochastic latent that models the stochasticity in the video . Given an arbitrary query time-point , the model uses the sampled latent to render the frame at that time-point . The model is not autoregressive on input or output frames , but still captures correlations over multiple target frames . Our method is not restricted to video prediction . When conditioned on camera position , our model can sample consistent sets of images for an occluded region of a scene , even if there are multiple possibilities for what that region might contain . We test JUMP on multiple synthetic datasets . To summarize our results : on a synthetic video prediction task involving five shapes , we show that JUMP produces frames of a similar image quality to modern video prediction methods . Moreover , JUMP converges more reliably , and can do jumpy predictions . To showcase the flexibility of our model we also apply it to stochastic 3D reconstruction , where our model is conditioned on camera location instead of time . For this , we introduce a dataset that consists of images of 3D scenes containing a cube with random MNIST digits engraved on each of its faces . JUMP outperforms GQN ( Eslami et al. , 2018 ) on this dataset , as GQN is unable to capture several correlated frames of occluded regions of the scene . We focus on synthetic datasets so that we can quantify how each model deals with stochasticity . For example , in our 3D reconstruction dataset we can control when models are asked to sample image frames for occluded regions of the scene , to quantify the consistency of these samples . We strongly encourage the reader to check the project website https : //bit.ly/2O4Pc4R to view videos of our experiments . To summarize , our key contributions are as follows . 1 . We motivate and formulate the problem of jumpy stochastic video prediction , where a model has to predict consistent target frames at arbitrary time points , given context frames at arbitrary time points . We observe close connections with consistent stochastic 3D reconstruction tasks , and abstract both problems in a common framework . 2 . We present a model for consistent jumpy predictions in videos and scenes . Unlike existing video prediction models , our model consumes input frames and samples output frames entirely in parallel . It enforces consistency of the sampled frames by training on multiple correlated targets , sampling a global latent , and using a deterministic rendering network . 3 . We show strong experimental results for our model . Unlike existing sequential video prediction models , our model can also do jumpy video predictions . We show that our model also produces images of similar quality while converging more reliably . Our model is not limited to video prediction – we develop a dataset for stochastic 3D reconstruction . Here , we show that our model significantly outperforms GQN . 2 RELATED WORK . Video Prediction A common limitation of video prediction models is their need for determinism in the underlying environments ( Lotter et al. , 2017 ; Srivastava et al. , 2015 ; Boots et al. , 2014 ; Finn et al. , 2016 ; Liu et al. , 2017 ) . Creating models that can work with stochastic environments is the motivation behind numerous recent papers : On one end of the complexity spectrum there are models like Video Pixel Networks ( Kalchbrenner et al. , 2017 ) and its variants ( Reed et al. , 2017b ) that are powerful but computationally expensive models . These models generate a video frame-by-frame , and generate each frame pixel-by-pixel . SV2P ( Babaeizadeh et al. , 2018 ) and SAVP ( Lee et al. , 2018 ) do not model each individual pixel , but autoregressively generate a video frame-by-frame . On the other end of the spectrum , sSSM ( Buesing et al. , 2018 ) is a faster model that generates an abstract state at each time step which can be decoded into a frame when required . All these stochastic models still generate frames ( or states ) one time-step at a time and the input and output frames must be consecutive . By bypassing these two issues JUMP extends this spectrum of models as a flexible and faster stochastic model . Additionally , unlike prior methods , our models can be used for a wider range of tasks – we demonstrate our approach on stochastic 3D reconstruction . 3D Reconstruction Research in 3D reconstruction can be broadly divided into two categories : on the one hand there are traditional 3D reconstruction methods that require some predefined description of the underlying 3D structure . This type of algorithms include structure-from-motion , structure-from-depth and multi view geometry techniques and while these models have achieved impressive results , they require researchers to handcraft a feature space in advance ( Pollefeys et al. , 2004 ; Wu et al. , 2016 ; Zhang et al. , 2015 ) . On the other hand there are more recent neural approaches that learn an implicit representation from data directly . The aim of these models however is usually to learn the distribution over images as opposed to learning the 3D structure of an environment ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ; Goodfellow et al. , 2014 ; Gregor et al. , 2016b ) . Viewpoint transformation networks do focus on learning this structure , however the possible transformations are limited and the models have only been non-probabilistic so far ( Choy et al. , 2016 ; Tatarchenko et al. , 2016 ; Fouhey et al. , 2016 ) . Finally , GQN ( Eslami et al. , 2018 ) is a recent deep learning model used for spatial prediction . However , GQN was primarily used for deterministic 3D reconstruction . In stochastic 3D reconstruction , GQN is unable to capture several correlated frames of occluded regions of a scene . Meta-Learning Finally , our task can be framed as a few-shot density estimation problem and is thus related to some ongoing research in the area of meta learning . Meta learning is often associated with few-shot classification tasks ( Vinyals et al. , 2016 ; Koch et al. , 2015 ) but these algorithms can be extended to few-shot density estimation for image generation ( Bartunov & Vetrov , 2018 ) . Additional approaches include models with variational memory ( Bornschein et al. , 2017 ) , attention ( Reed et al. , 2017a ) and conditional latent variables ( Rezende et al. , 2016 ) . Crucially , while these models can sample the estimated density at random they can not query it at specific target points and their application is limited to visually less challenging datasets like omniglot . Concurrent work on neural processes ( Garnelo et al. , 2018 ) , can query the density at specific target points , but on low-dimensional data-sets like 2D function regression and 2D contextual bandit problems . 3 METHODS . 3.1 PROBLEM DESCRIPTION . We consider problems where we have a collection of “ scenes ” . Scenes could be videos , 3D spatial scenes , or in general any key-indexed collection . A scene S ( i ) consists of a collection of viewpointframe ( key-value ) pairs ( v ( i ) 1 , f ( i ) 1 ) , ... , ( v ( i ) n , f ( i ) n ) where v ( i ) j refers to the indexing ‘ viewpoint ’ information and f ( i ) j to the frame . For videos the ‘ viewpoints ’ are timestamps . For spatial scenes the ‘ viewpoints ’ are camera positions and headings . For notational simplicity , we assume that each scene has fixed length n ( but this is not a requirement of the model ) . The viewpoint-frame pairs in each scene are generated from a data generative process D , as formulated below . Note that the viewpoint-frame pairs are typically not independent . ( v ( i ) 1 , f ( i ) 1 ) , ... , ( v ( i ) n , f ( i ) n ) ∼ D = P ( ( v1 , f1 ) , ... , ( vn , fn ) ) ( 1 ) Each scene S is split into a context and a target . The context C contains m viewpoint-frame pairs C = { ( vi , fi ) } mi=1 . The target T contains the remaining n − m viewpoints V = { vi } ni=m+1 and corresponding target frames F = { fi } ni=m+1 . At evaluation time , the model receives the context C and target viewpoints V and should be able to sample possible values F̂ corresponding to the viewpoints V . In particular , the model parameterizes a ( possibly implicit ) conditional distribution Pθ ( F |C , V ) , from which the frames are sampled . Given a training set of N example scenes from data distribution D , the training objective is to find model parameters θ that maximize the log probability of the data E C , V , F∼D [ logPθ ( F |C , V ) ] . ( 2 ) 3.2 MODEL ( GENERATION ) We implement JUMP as a latent variable model . For each scene S , JUMP encodes them viewpointframe pairs of the context C by applying a representation function Mθ to each pair independently . The resulting representations r1 , ... , rm are aggregated in a permutation-invariant way to obtain a single representation r. The latent variable z is then sampled from a prior Pφ that is conditioned on this aggregated representation r. The idea behind z is that it can capture global dependencies across the n−m target viewpoints , which is crucial to ensure that the output frames are generated from a single consistent plausible scene . For each corresponding target viewpoint vi , the model applies a deterministic rendering network Mγ to z and vi to get an output frame fi . Our model , JUMP , can thus be summarized as follows . rj =Mθ ( vj , fj ) for all j ∈ 1 , ... , m r = r1 + ... + rm z ∼ Pφ ( z|r ) fi =Mγ ( z , vi ) for all i ∈ m+ 1 , ... , n In JUMP , the representation network Mθ is implemented as a convolutional network . The latent z is sampled using a convolutional DRAW ( Gregor et al. , 2015 ; 2016b ) prior , an LSTM-like model that recurrently samples latents over several iterations . The rendering network Mγ is implemented as an LSTM where the inputs z and v are fed in at each recurrent step . We give details of these implementations in the Appendix . Note that these building blocks are easily customizable . For example , DRAW can be replaced with a regular variational autoencoder , albeit at a cost to the visual fidelity of the generated samples .

Summary: The paper motivates and provides a model to generate video frames and reconstructions from non-sequential data by encoding time/camera position into the model training. The idea is to allow the model to interpolate, and more importantly, extrapolate from frames and learn the latent state for multiple frames together. The same techniques are also applicable to 3d-reconstruction.  JUMP is very closely related to GQN with the main difference being that the randomness in JUMP is learned better using a "global" prior. The evaluation is reasonable on multiple synthetic experiments including a 3d-scene reconstruction specially created to showcase the consistency capabilities in a stochastic generation. Paper is mostly clear but more priority should be given to the discussion around convergence and the latent state.