Keyframing the Future: Discovering Temporal Hierarchy with Keyframe-Inpainter Prediction

To flexibly and efficiently reason about temporal sequences , abstract representations that compactly represent the important information in the sequence are needed . One way of constructing such representations is by focusing on the important events in a sequence . In this paper , we propose a model that learns both to discover such key events ( or keyframes ) as well as to represent the sequence in terms of them . We do so using a hierarchical Keyframe-Inpainter ( KEYIN ) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes . We propose a fully differentiable formulation for efficiently learning the keyframe placement . We show that KEYIN finds informative keyframes in several datasets with diverse dynamics . When evaluated on a planning task , KEYIN outperforms other recent proposals for learning hierarchical representations . 1 INTRODUCTION . When thinking about the future , humans focus their thoughts on the important things that may happen ( When will the plane depart ? ) without fretting about the minor details that fill each intervening moment ( What is the last word I will say to the taxi driver ? ) . Because the vast majority of elements in a temporal sequence contains redundant information , a temporal abstraction can make reasoning and planning both easier and more efficient . How can we build such an abstraction ? Consider the example of a lead animator who wants to show what happens in the next scene of a cartoon . Before worrying about every low-level detail , the animator first sketches out the story by keyframing , drawing the moments in time when the important events occur . The scene can then be easily finished by other animators who fill in the rest of the sequence from the story laid out by the keyframes . In this paper , we argue that learning to discover such informative keyframes from raw sequences is an efficient and powerful way to learn to reason about the future . Our goal is to learn such an abstraction for future image prediction . In contrast , much of the work on future image prediction has focused on frame-by-frame synthesis ( Oh et al . ( 2015 ) ; Finn et al . ( 2016 ) ) . This strategy puts an equal emphasis on each frame , irrespective of the redundant content it may contain or its usefulness for reasoning relative to the other predicted frames . Other recent work has considered predictions that “ jump ” more than one step into the future , but these approaches either used fixed-offset jumps ( Buesing et al. , 2018 ) or used heuristics to select the predicted frames ( Neitz et al. , 2018 ; Jayaraman et al. , 2019 ; Gregor et al. , 2019 ) . In this work , we propose a method that selects the keyframes that are most informative about the full sequence , so as to allow us to reason about the sequence holistically while only using a small subset of the frames . We do so by ensuring that the full sequence can be recovered from the keyframes with an inpainting strategy , similar to how a supporting animator finishes the story keyframed by the lead . One possible application for a model that discovers informative keyframes is in long-horizon planning . Recently , predictive models have been employed for model-based planning and control ( Ebert et al . ( 2018 ) ) . However , they reason about every single future time step , limiting their applicability to short horizon tasks . In contrast , we show that a model that reasons about the future using a small set of informative keyframes enables visual predictive planning for horizons much greater than previously possible by using keyframes as subgoals in a hierarchical planning framework . To discover informative frames in raw sequence data , we formulate a hierarchical probabilistic model in which a sequence is represented by a subset of its frames ( see Fig . 1 ) . In this two-stage model , a keyframing module represents the keyframes as well as their temporal placement with stochastic latent variables . The images that occur at the timepoints between keyframes are then inferred by an inpainting module . We parametrize this model with a neural network and formulate a variational lower bound on the sequence log-likelihood . Optimizing the resulting objective leads to a model that discovers informative future keyframes that can be easily inpainted to predict the full future sequence . Our contributions are as follows . We formulate a hierarchical approach for the discovery of informative keyframes using joint keyframing and inpainting ( KEYIN ) . We propose a soft objective that allows us to train the model in a fully differentiable way . We first analyze our model on a simple dataset with stochastic dynamics in a controlled setting and show that it can reliably recover the underlying keyframe structure on visual data . We then show that our model discovers hierarchical temporal structure on more complex datasets of demonstrations : an egocentric gridworld environment and a simulated robotic pushing dataset , which is challenging for current approaches to visual planning . We demonstrate that the hierarchy discovered by KEYIN is useful for planning , and that the resulting approach outperforms other proposed hierarchical and non-hierarchical planning schemes on the pushing task . Specifically , we show that keyframes predicted by KEYIN can serve as useful subgoals that can be reached by a low-level planner , enabling long-horizon , hierarchical control . 2 RELATED WORK . Hierarchical temporal structure . Hierarchical neural models for efficiently modeling sequences were proposed in Liu et al . ( 2015 ) ; Buesing et al . ( 2018 ) . These approaches were further extended to predict with an adaptive step size so as to leverage the natural hierarchical structure in language data ( Chung et al. , 2016 ; Kádár et al. , 2018 ) . However , these models rely on autoregressive techniques for text generation and applying them to structured data , such as videos , might be impractical . The video processing community has used keyframe representations as early as 1991 in the MPEG codec ( Gall , 1991 ) . Wu et al . ( 2018 ) adapted this algorithm in the context of neural compression ; however , these approaches use constant offsets between keyframes and thus do not fully reflect the temporal structure of the data . Recently , several neural methods were proposed to leverage such temporal structure . Neitz et al . ( 2018 ) and Jayaraman et al . ( 2019 ) propose models that find and predict the least uncertain “ bottleneck ” frames . Gregor et al . ( 2019 ) construct a representation that can be used to predict any number of frames into the future . In contrast , we propose an approach for hierarchical video representation that discovers the keyframes that best describe a certain sequence . In parallel to our work , Kipf et al . ( 2019 ) propose a related method for video segmentation via generative modeling . Kipf et al . ( 2019 ) focus on using the discovered task boundaries for training hierarchical RL agents , while we show that our model can be used to perform efficient hierarchical planning by representing the sequence with only a small set of keyframes . Also concurrently , Kim et al . ( 2019 ) propose a similar method to KEYIN for learning temporal abstractions . While Kim et al . ( 2019 ) focuses on learning hierarchical state-space models , we propose a model that operates directly in the observation space and performs joint keyframing and inpainting . Video modeling . Early approaches to probabilistic video modeling include autoregressive models that factorize the distribution by considering pixels sequentially ( Kalchbrenner et al. , 2017 ; Reed et al. , 2017 ) . To reason about the images in the video holistically , latent variable approaches were developed based on variational inference ( Chung et al. , 2015 ; Rezende et al. , 2014 ; Kingma & Welling , 2014 ) , including ( Babaeizadeh et al. , 2018 ; Denton & Fergus , 2018 ; Lee et al. , 2018 ) and large-scale models such as ( Castrejon et al. , 2019 ; Villegas et al. , 2019 ) . Kumar et al . ( 2019 ) is a recently proposed approach that uses exact inference based on normalizing flows ( Dinh et al. , 2014 ; Rezende & Mohamed , 2015 ) . We build on existing video modeling approaches and show how they can be used to learn temporal abstractions with a novel keyframe-based generative model . Visual planning and model predictive control . We build on recent work that explored applications of learned visual predictive models to planning and control . Several groups ( Oh et al. , 2015 ; Finn et al. , 2016 ; Chiappa et al. , 2017 ) have proposed models that predict the consequences of actions taken by an agent given its control output . Recent work ( Byravan et al. , 2017 ; Hafner et al. , 2018 ; Ebert et al. , 2018 ) has shown that visual model predictive control based on such models can be applied to a variety of different settings . In this work , we show that the hierarchical representation of a sequence in terms of keyframes improves planning performance in the hierarchical planning setting . 3 KEYFRAMING THE FUTURE Our goal is to develop a model that generates sequences by first predicting key observations and the time steps when they occur and then filling in the remaining observations in between . To achieve this goal , in the following we ( i ) define a probabilistic model for joint keyframing and inpainting , and ( ii ) show how a maximum likelihood objective leads to the discovery of keyframe structure . 3.1 A PROBABILISTIC MODEL FOR JOINT KEYFRAMING AND INPAINTING . We first describe a probabilistic model for joint keyframing and inpainting of a sequence I1 : T . The model consists of two parts : the keyframe predictor and the sequence inpainter ( see Fig . 2 ) . The keyframe predictor takes in C conditioning frames Ico and produces N keyframes K1 : N as well as the corresponding time indices τ1 : N : p ( K1 : N , τ1 : N |Ico ) = ∏ n p ( Kn , τn|K1 : n−1 , τ1 : n−1 , Ico ) . ( 1 ) From each pair of keyframes , the sequence inpainter generates the sequence of frames in between : p ( Iτn : τn+1−1|Kn , Kn+1 , τn+1 − τn ) = ∏ n p ( It|Kn , Kn+1 , Iτn : t−1 , τn+1 − τn ) , ( 2 ) which completes the generation of the full sequence . The inpainter additionally observes the number of frames it needs to generate τn+1− τn . The temporal spacing of the most informative keyframes is data-dependent : shorter keyframe intervals might be required in cases of rapidly fluctuating motion , while longer intervals can be sufficient for steadier motion . Our model handles this by predicting the keyframe indices τ and inpainting τn+1−τn frames between each pair of keyframes . We parametrize the prediction of τn in relative terms by predicting offsets δn : τn = τn−1 + δn . 3.2 KEYFRAME DISCOVERY . To produce a complex multimodal distribution over K we use a per-keyframe latent variable z with prior distribution p ( z ) and approximate posterior q ( z|I , Ico ) .1 We construct a variational lower bound 1For simplicity , the variable representing the full sequence is written without indices ( I is the same as I1 : T ) . on the likelihood of both I and K as follows2 : ln p ( I , K|Ico ) ≥ Eq ( z|I , Ico ) [ N∑ n=1 lnEp ( τn , τn+1|z1 : n , Ico ) [ p ( Iτn : τn+1 |Kn , n+1 , τn+1 − τn ) ] ︸ ︷︷ ︸ inpainting + ln p ( K|z , Ico ) ︸ ︷︷ ︸ keyframing ] −DKL ( q ( z|I , Ico ) ||p ( z ) ) ︸ ︷︷ ︸ regularization . ( 3 ) In practice , we use a weight β on the KL-divergence term , as is common in amortized variational inference ( Higgins et al. , 2017 ; Alemi et al. , 2018 ; Denton & Fergus , 2018 ) . If a simple model is used for inpainting , most of the representational power of the model has to come from the keyframe predictor . We use a relatively powerful latent variable model for the keyframe predictor and a simpler Gaussian distribution produced with a neural network for inpainting . Because of this structure , the keyframe predictor has to predict keyframes that describe the underlying sequence well enough to allow a simpler inpainting process to maximize the likelihood . We will show that pairing a more flexible keyframe predictor with a simpler inpainter allows our model to discover semantically meaningful keyframes in video data .

Summary: The authors address the problem of discovering and predicting with hierarchical structure in data sequences of relevance to planning. Starting with the kinds of data that have been used recently in video prediction, the authors aim at learning a sequence of keyframes (i.e., subsets of frames forming the overall sequence) that in a suitable sense "summarize" the overall trace. As they rightly note, many alternate models struggle with making good long term predictions in part because they focus on all levels of prediction equally.