Goal-Conditioned Video Prediction

Many processes can be concisely represented as a sequence of events leading from a starting state to an end state . Given raw ingredients , and a finished cake , an experienced chef can surmise the recipe . Building upon this intuition , we propose a new class of visual generative models : goal-conditioned predictors ( GCP ) . Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video . GCP instead treats videos as start-goal transformations , making video generation easier by conditioning on the more informative context provided by the first and final frames . Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned , but GCP models can also utilize structures that are not linear in time , to accomplish hierarchical prediction . To this end , we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively , splitting the video iteratively into finer and finer segments delineated by subgoals . In experiments across simulated and real datasets , our GCP methods generate high-quality sequences over long horizons . Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs , making training and inference very efficient , and allowing the model to train on sequences that are thousands of frames in length . Finally , we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions . Videos are on the supplementary website : https : //sites.google.com/view/video-gcp 1 INTRODUCTION . Many phenomena , both natural and artificial , are naturally characterized as transformations — the most salient information about them is contained in the start and end states , given which it is possible to fill in intermediate states from prior experience . For example , ending up in San Francisco after starting in Oakland entails getting into a car and crossing the Bay Bridge . Similarly , to an expert engineer observing a bridge , the task of reverse-engineering how it was built is well-defined and tractable . In contrast , consider the task of predicting forward in time , having observed only the steel and concrete that went into making the bridge . Such forward prediction tasks are severely underconstrained , leading to high uncertainties that compound with time , making it impossible to make meaningful predictions after only a few stages of iterative forward prediction ( see Fig . 1 ) . This is aggravated in highdimensional settings such as forward video prediction , which despite being the most widely studied setting for video synthesis , struggles to produce coherent video longer than a few seconds . We propose to condition video synthesis instead on the substantially more informative context of the start and the goal frame . We term such models goal-conditioned predictors ( GCP ) . Much like the engineer observing the bridge , GCPs treat long videos as start-goal transformations and reverseengineer the full video , conditioned on the first and final frames . The simplest instantiation of GCPs modifies existing forward prediction approaches to also observe the final frame . More broadly , once we consider conditioning on the goal frame , we can devise new types of GCP models that more efficiently leverage the hierarchical structure present in real-world event sequences ( Fig . 1 , right ) . Just as coarse-to-fine image synthesis ( Karras et al. , 2017 ) generates a high-resolution image by iteratively adding details to a low-resolution image , we can synthesize a temporally downsampled video in the form of sequences of keyframes , and fill it in iteratively . We propose to implement this as a tree-structured GCP : the start-goal input frames serve as the initial downsampled sequence and new frames are recursively added until the full video is generated . Intuitively , this exploits the hierarchical temporal structure of natural video by breaking a long sequential procedure into its constituent steps , as illustrated in Fig 1 . However , procedural steps do not all occur on a regularly spaced schedule or last for equal lengths of time . To model this , we further propose to allow the model to select which frames to generate at each level in the tree . In our experiments , all GCP variants successfully generate longer and higher-quality video than has been demonstrated with standard auto-regressive video prediction models , which only utilize the starting frames for context . Furthermore , we show that tree-structured GCPs are more parallelizable than auto-regressive models , leading to very fast training and inference . We show that we can train tree-structured GCPs on videos consisting of thousands of frames . We also study the applications of GCPs , demonstrating that they can be utilized to enable prediction-based control in simulated imitation learning scenarios . In these settings , the GCP models can be trained without access to demonstrator actions , and can synthesize visual plans directly from start and goal images , which can then be tracked using an inverse model . 2 RELATED WORK . Video generation . Several existing neural video generation approaches either generate the entire video from scratch ( Vondrick et al. , 2016 ; Clark et al. , 2019 ) or conditioned on the beginning of the video ( Ranzato et al. , 2014 ; Srivastava et al. , 2015 ; Mathieu et al. , 2015 ; Finn et al. , 2016 ; Oh et al. , 2015 ) . Some approaches also model the uncertainties inherent in video prediction through variational inference ( Denton & Fergus , 2018 ; Villegas et al. , 2019 ; Xue et al. , 2016 ; Lee et al. , 2018 ; Larsen et al. , 2016 ; Babaeizadeh et al. , 2018 ) . As argued above , video prediction approaches struggle to synthesize videos longer than a few seconds due to the ill-defined nature of this task . To rectify this , we propose a different class of models that conditions the prediction on both the first and the last frame of the desired video . This is related to prior work on neural video interpolation . However , such work has focused on short-term interpolation , often using models based on optical flow . Liu et al . ( 2017 ) ; Jiang et al . ( 2018 ) propose techniques that transform the start and the goal images via flow fields predicted by a convolutional neural network ( CNN ) . Niklaus et al . ( 2017a ; b ) transform the start and the goal images with convolutional kernels predicted with a CNN . These methods rely on simple warping-based techniques that , while well-suited for short-term video in-filling , are not effective for generating longer video sequences . In contrast , we propose a powerful goal-conditioned video prediction technique that scales to videos as long as a minute in length . We compare against video interpolation in our experiments . Visual imitation and planning . We show an application of GCP for imitation learning in the case where we have expert data without actions . Prior methods for solving the visual imitation learning problem have tried to learn reward functions from demonstrations to train reinforcement learning agents to perform imitation ( Sermanet et al. , 2016 ; Peng et al. , 2018 ; Sermanet et al. , 2018 ) , suffering from high sample complexity , and exploration problems of reinforcement learning . Our proposed goal-conditioned imitation framework , does not need to perform reinforcement learning and also leverages expert demonstrations to overcome the exploration problem . Visual goal-directed behavior can also be achieved using visual planing methods , such as visual foresight ( Ebert et al. , 2018 ) or Causal InfoGan ( Kurutach et al. , 2018 ) . Unlike GCP-based imitation however , these methods do not attempt to imitate the expert , but instead compute plans internally by sampling imaginary states or actions , exhibiting difficulties for longer control horizons as targeted with our proposed method . Inverse models such as zero-shot visual imitation proposed by Pathak et al . ( 2018 ) require access to actions and are therefore insufficient to work in the action-free imitation setting . 3 GOAL-CONDITIONED PREDICTION . In this section , we formalize the goal-condition prediction problem , and propose several models for goal-conditioned prediction , including both auto-regressive models and tree structured models . To define the goal-conditioned prediction problem , consider a sequence of observations [ o1 , o2 , ... oT ] of length T . Standard forward prediction approaches observe the first k observations and synthesize the rest of the sequence , i.e. , they model p ( ok+1 , ok+2 , . . . oT−1|o1 , o2 , . . . ok ) . Instead , we would like our goal-conditioned predictors to produce intermediate observations given the first and last elements in the sequence . In other words , they must model p ( o2 , o3 , . . . oT−1|o1 , oT ) . We propose several designs of goal-conditioned predictors that operate in learned compact state spaces for scalability and accuracy . Fig 2 shows schematics of these GCP designs , as well as of a standard forward predictor . 3.1 GOAL-CONDITIONED SEQUENTIAL PREDICTION . We first present a simple auto-regressive model for goal-conditioned prediction . In this model , we predict latent state representations sequentially in chronological order , from the start to the end , with the prediction at each point in time conditioned on the first and final observations as well as the previous latent state . The resulting model ( GCP-sequential , shown in Fig 2 , b ) can be factorized as follows : p ( o2 , o3 , . . . oT−1|o1 , oT ) = ∫ p ( o2|s2 ) p ( s2|o1 , oT ) T−1∏ t=3 p ( ot|st ) p ( st|st−1 , o1 , oT ) ds2 : T−1 ( 1 ) We show in Sec 3.4 that this model is simple to implement , and can build directly on previously proposed auto-regressive video prediction models . However , its computational complexity scales with the sequence length , as every state must be produced in sequence . Furthermore , this approach does not account for the hierarchical structure of natural video , which contains events and sub-events , as depicted in Fig 1 . 3.2 GOAL-CONDITIONED PREDICTION BY RECURSIVE INFILLING . To account for this hierarchical structure and also devise a computationally more efficient method , we now design a tree-structured GCP model . Suppose that we have an intermediate state prediction operator p ( st|pa ( t ) ) that produces an intermediate state st halfway in time between its two parent states pa ( t ) . Then , consider the following alternative generative process for goal-conditioned prediction : at the beginning , the observed first and last frames are encoded into the latent state space as s1 and sT , and the prediction operator p ( st|pa ( t ) ) generates s0.5T . The same operator may now be applied to two new sets of parents ( s1 , s0.5T ) and ( s0.5T , sT ) . As this process continues recursively , the intermediate prediction operator fills in more and more temporal detail until the full video is synthesized . Fig 2 , c depicts this model , which we call GCP-tree , since it has a tree-like shape where each predicted state is dependent on its right and left parents , starting with the start and the goal . GCP-tree factorizes the goal-conditioned video generation problem as : p ( o2 , o3 , . . . oT−1|o1 , oT ) = ∫ p ( s1|o1 ) p ( sT |oT ) T−1∏ t=2 p ( ot|st ) p ( st|pa ( t ) ) ds2 : T−1 . ( 2 ) Computational efficiency . While the sequential forward predictor performs O ( T ) sequential operations to produce a sequence of length T , the hierarchical prediction can be more efficient due to parallelization . As the depth of the tree is dlog T e , it only requires O ( log T ) sequential operations to produce a sequence , assuming all operations that can be conducted in parallel are parallelized perfectly . We therefore batch the branches of the tree and process them in parallel at every level to utilize the benefit of efficient computation on modern GPUs . We note that the benefits of the GCP-tree runtime lie in parallelization , and thus diminish with large batch sizes , where the parallel processing capacity of the GPU is already fully utilized . We notice that , when predicting sequences of 500 frames , GCP-sequential can use up to 4 times bigger batches than GCP-Tree without significant increase in runtime cost . Adaptive binding . We have thus far described the intermediate prediction operator as always generating the state that occurs halfway in time between its two parents . While this is a simple and effective scheme , it may not correspond to the natural hierarchical structure in the video . For example , in the navigation example in Figure 1 , we might prefer the first split to correspond to traversing the bridge , which partitions the prediction problem into two largely independent halves . We can design a version of GCP-tree that allows the intermediate frame predictor to select which of the several states between the parents to predict , each time it is applied . In other words , the predicted state might bind to one of many observations in the sequence . In this more versatile model , we represent the time steps of the tree nodes with discrete latent variable w that selects which nodes bind to which frames : p ( ot|s1 : N , wt ) = p ( ot|swt ) . We can then express the prediction problem as : p ( o2 : T−1|o1 , oT ) = ∫ p ( s1|o1 ) p ( sN |oT ) ∏ n p ( sn|pa ( n ) ) T−1∏ t=2 p ( ot|s1 : N , wt ) p ( wt ) ds1 : Ndw2 : T−1 ( 3 ) Appendix C shows an efficient inference procedure for w based on dynamic programming .

Summary: The following work proposes a model for long-range video interpolation -- specifically targetting cases where the intermediate content trajectories may be highly non-linear. This is referred to as goal-conditioned in the paper. They present an autoregressive sequential model, as well as a hierarchical model -- each based on a probabilistic framework. Finally, they demonstrate an application in imitation learning by introducing an additional model that maps pairs of observations (frames) to a distribution over actions that predicts how likely each action will map the first observation to the second. Their imitation learning method is able to successfully solve mazes, given just the start and goal observations.