Keyframing the Future: Discovering Temporal Hierarchy with Keyframe-Inpainter Prediction

To flexibly and efficiently reason about temporal sequences , abstract representations that compactly represent the important information in the sequence are needed . One way of constructing such representations is by focusing on the important events in a sequence . In this paper , we propose a model that learns both to discover such key events ( or keyframes ) as well as to represent the sequence in terms of them . We do so using a hierarchical Keyframe-Inpainter ( KEYIN ) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes . We propose a fully differentiable formulation for efficiently learning the keyframe placement . We show that KEYIN finds informative keyframes in several datasets with diverse dynamics . When evaluated on a planning task , KEYIN outperforms other recent proposals for learning hierarchical representations . 1 INTRODUCTION . When thinking about the future , humans focus their thoughts on the important things that may happen ( When will the plane depart ? ) without fretting about the minor details that fill each intervening moment ( What is the last word I will say to the taxi driver ? ) . Because the vast majority of elements in a temporal sequence contains redundant information , a temporal abstraction can make reasoning and planning both easier and more efficient . How can we build such an abstraction ? Consider the example of a lead animator who wants to show what happens in the next scene of a cartoon . Before worrying about every low-level detail , the animator first sketches out the story by keyframing , drawing the moments in time when the important events occur . The scene can then be easily finished by other animators who fill in the rest of the sequence from the story laid out by the keyframes . In this paper , we argue that learning to discover such informative keyframes from raw sequences is an efficient and powerful way to learn to reason about the future . Our goal is to learn such an abstraction for future image prediction . In contrast , much of the work on future image prediction has focused on frame-by-frame synthesis ( Oh et al . ( 2015 ) ; Finn et al . ( 2016 ) ) . This strategy puts an equal emphasis on each frame , irrespective of the redundant content it may contain or its usefulness for reasoning relative to the other predicted frames . Other recent work has considered predictions that “ jump ” more than one step into the future , but these approaches either used fixed-offset jumps ( Buesing et al. , 2018 ) or used heuristics to select the predicted frames ( Neitz et al. , 2018 ; Jayaraman et al. , 2019 ; Gregor et al. , 2019 ) . In this work , we propose a method that selects the keyframes that are most informative about the full sequence , so as to allow us to reason about the sequence holistically while only using a small subset of the frames . We do so by ensuring that the full sequence can be recovered from the keyframes with an inpainting strategy , similar to how a supporting animator finishes the story keyframed by the lead . One possible application for a model that discovers informative keyframes is in long-horizon planning . Recently , predictive models have been employed for model-based planning and control ( Ebert et al . ( 2018 ) ) . However , they reason about every single future time step , limiting their applicability to short horizon tasks . In contrast , we show that a model that reasons about the future using a small set of informative keyframes enables visual predictive planning for horizons much greater than previously possible by using keyframes as subgoals in a hierarchical planning framework . To discover informative frames in raw sequence data , we formulate a hierarchical probabilistic model in which a sequence is represented by a subset of its frames ( see Fig . 1 ) . In this two-stage model , a keyframing module represents the keyframes as well as their temporal placement with stochastic latent variables . The images that occur at the timepoints between keyframes are then inferred by an inpainting module . We parametrize this model with a neural network and formulate a variational lower bound on the sequence log-likelihood . Optimizing the resulting objective leads to a model that discovers informative future keyframes that can be easily inpainted to predict the full future sequence . Our contributions are as follows . We formulate a hierarchical approach for the discovery of informative keyframes using joint keyframing and inpainting ( KEYIN ) . We propose a soft objective that allows us to train the model in a fully differentiable way . We first analyze our model on a simple dataset with stochastic dynamics in a controlled setting and show that it can reliably recover the underlying keyframe structure on visual data . We then show that our model discovers hierarchical temporal structure on more complex datasets of demonstrations : an egocentric gridworld environment and a simulated robotic pushing dataset , which is challenging for current approaches to visual planning . We demonstrate that the hierarchy discovered by KEYIN is useful for planning , and that the resulting approach outperforms other proposed hierarchical and non-hierarchical planning schemes on the pushing task . Specifically , we show that keyframes predicted by KEYIN can serve as useful subgoals that can be reached by a low-level planner , enabling long-horizon , hierarchical control . 2 RELATED WORK . Hierarchical temporal structure . Hierarchical neural models for efficiently modeling sequences were proposed in Liu et al . ( 2015 ) ; Buesing et al . ( 2018 ) . These approaches were further extended to predict with an adaptive step size so as to leverage the natural hierarchical structure in language data ( Chung et al. , 2016 ; Kádár et al. , 2018 ) . However , these models rely on autoregressive techniques for text generation and applying them to structured data , such as videos , might be impractical . The video processing community has used keyframe representations as early as 1991 in the MPEG codec ( Gall , 1991 ) . Wu et al . ( 2018 ) adapted this algorithm in the context of neural compression ; however , these approaches use constant offsets between keyframes and thus do not fully reflect the temporal structure of the data . Recently , several neural methods were proposed to leverage such temporal structure . Neitz et al . ( 2018 ) and Jayaraman et al . ( 2019 ) propose models that find and predict the least uncertain “ bottleneck ” frames . Gregor et al . ( 2019 ) construct a representation that can be used to predict any number of frames into the future . In contrast , we propose an approach for hierarchical video representation that discovers the keyframes that best describe a certain sequence . In parallel to our work , Kipf et al . ( 2019 ) propose a related method for video segmentation via generative modeling . Kipf et al . ( 2019 ) focus on using the discovered task boundaries for training hierarchical RL agents , while we show that our model can be used to perform efficient hierarchical planning by representing the sequence with only a small set of keyframes . Also concurrently , Kim et al . ( 2019 ) propose a similar method to KEYIN for learning temporal abstractions . While Kim et al . ( 2019 ) focuses on learning hierarchical state-space models , we propose a model that operates directly in the observation space and performs joint keyframing and inpainting . Video modeling . Early approaches to probabilistic video modeling include autoregressive models that factorize the distribution by considering pixels sequentially ( Kalchbrenner et al. , 2017 ; Reed et al. , 2017 ) . To reason about the images in the video holistically , latent variable approaches were developed based on variational inference ( Chung et al. , 2015 ; Rezende et al. , 2014 ; Kingma & Welling , 2014 ) , including ( Babaeizadeh et al. , 2018 ; Denton & Fergus , 2018 ; Lee et al. , 2018 ) and large-scale models such as ( Castrejon et al. , 2019 ; Villegas et al. , 2019 ) . Kumar et al . ( 2019 ) is a recently proposed approach that uses exact inference based on normalizing flows ( Dinh et al. , 2014 ; Rezende & Mohamed , 2015 ) . We build on existing video modeling approaches and show how they can be used to learn temporal abstractions with a novel keyframe-based generative model . Visual planning and model predictive control . We build on recent work that explored applications of learned visual predictive models to planning and control . Several groups ( Oh et al. , 2015 ; Finn et al. , 2016 ; Chiappa et al. , 2017 ) have proposed models that predict the consequences of actions taken by an agent given its control output . Recent work ( Byravan et al. , 2017 ; Hafner et al. , 2018 ; Ebert et al. , 2018 ) has shown that visual model predictive control based on such models can be applied to a variety of different settings . In this work , we show that the hierarchical representation of a sequence in terms of keyframes improves planning performance in the hierarchical planning setting . 3 KEYFRAMING THE FUTURE Our goal is to develop a model that generates sequences by first predicting key observations and the time steps when they occur and then filling in the remaining observations in between . To achieve this goal , in the following we ( i ) define a probabilistic model for joint keyframing and inpainting , and ( ii ) show how a maximum likelihood objective leads to the discovery of keyframe structure . 3.1 A PROBABILISTIC MODEL FOR JOINT KEYFRAMING AND INPAINTING . We first describe a probabilistic model for joint keyframing and inpainting of a sequence I1 : T . The model consists of two parts : the keyframe predictor and the sequence inpainter ( see Fig . 2 ) . The keyframe predictor takes in C conditioning frames Ico and produces N keyframes K1 : N as well as the corresponding time indices τ1 : N : p ( K1 : N , τ1 : N |Ico ) = ∏ n p ( Kn , τn|K1 : n−1 , τ1 : n−1 , Ico ) . ( 1 ) From each pair of keyframes , the sequence inpainter generates the sequence of frames in between : p ( Iτn : τn+1−1|Kn , Kn+1 , τn+1 − τn ) = ∏ n p ( It|Kn , Kn+1 , Iτn : t−1 , τn+1 − τn ) , ( 2 ) which completes the generation of the full sequence . The inpainter additionally observes the number of frames it needs to generate τn+1− τn . The temporal spacing of the most informative keyframes is data-dependent : shorter keyframe intervals might be required in cases of rapidly fluctuating motion , while longer intervals can be sufficient for steadier motion . Our model handles this by predicting the keyframe indices τ and inpainting τn+1−τn frames between each pair of keyframes . We parametrize the prediction of τn in relative terms by predicting offsets δn : τn = τn−1 + δn . 3.2 KEYFRAME DISCOVERY . To produce a complex multimodal distribution over K we use a per-keyframe latent variable z with prior distribution p ( z ) and approximate posterior q ( z|I , Ico ) .1 We construct a variational lower bound 1For simplicity , the variable representing the full sequence is written without indices ( I is the same as I1 : T ) . on the likelihood of both I and K as follows2 : ln p ( I , K|Ico ) ≥ Eq ( z|I , Ico ) [ N∑ n=1 lnEp ( τn , τn+1|z1 : n , Ico ) [ p ( Iτn : τn+1 |Kn , n+1 , τn+1 − τn ) ] ︸ ︷︷ ︸ inpainting + ln p ( K|z , Ico ) ︸ ︷︷ ︸ keyframing ] −DKL ( q ( z|I , Ico ) ||p ( z ) ) ︸ ︷︷ ︸ regularization . ( 3 ) In practice , we use a weight β on the KL-divergence term , as is common in amortized variational inference ( Higgins et al. , 2017 ; Alemi et al. , 2018 ; Denton & Fergus , 2018 ) . If a simple model is used for inpainting , most of the representational power of the model has to come from the keyframe predictor . We use a relatively powerful latent variable model for the keyframe predictor and a simpler Gaussian distribution produced with a neural network for inpainting . Because of this structure , the keyframe predictor has to predict keyframes that describe the underlying sequence well enough to allow a simpler inpainting process to maximize the likelihood . We will show that pairing a more flexible keyframe predictor with a simpler inpainter allows our model to discover semantically meaningful keyframes in video data .

Summary: The paper introduces a model trained for video prediction hierarchically: a series of significant frames called “keyframes” in the paper are first predicted and then intermediate frames between keyframes couples are generated. The training criterion is maximum likelihood with a variational approximation. Experiments are performed on 3 different video datasets and the evaluation is performed for 3 tasks: keyframe detection, frame prediction and planning in robot videos.