Keyframing the Future: Discovering Temporal Hierarchy with Keyframe-Inpainter Prediction
Authors: Karl Pertsch
Oleh Rybkin
Jingyun Yang
Konstantinos G. Derpanis
Kostas Daniilidis
Joseph J. Lim
Andrew Jaegle
Venue:  (2020)
Keywords: representation learning
variational inference
video generation
temporal hierarchy

Abstract: To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed. One way of constructing such representations is by focusing on the important events in a sequence. In this paper, we propose a model that learns both to discover such key events (or keyframes) as well as to represent the sequence in terms of them.  We do so using a hierarchical Keyframe-Inpainter (KeyIn) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes. We propose a fully differentiable formulation for efficiently learning the keyframe placement. We show that KeyIn finds informative keyframes in several datasets with diverse dynamics. When evaluated on a planning task, KeyIn outperforms other recent proposals for learning hierarchical representations.

Under review as a conference paper at ICLR 2020
KEYFRAMING THE FUTURE: DISCOVERING TEMPORAL HIERARCHY WITH KEYFRAME-INPAINTER PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
To flexibly and efficiently reason about temporal sequences, abstract representations that compactly represent the important information in the sequence are needed. One way of constructing such representations is by focusing on the important events in a sequence. In this paper, we propose a model that learns both to discover such key events (or keyframest as well as to represent the sequence in terms of them. We do so using hierarchical Keyframe-Inpainter (KEYIN) model that first generates keyframes and their temporal placement and then inpaints the sequences between keyframes. We propose fully differentiable formulation for efficiently learning the keyframe placement. We show that KEYIN finds informative keyframes several datasets with diverse dynamics. When evaluated planning task. KEYIN outperforms other recent proposals for learning hierarchical representations.
INTRODUCTION
When thinking about the future, humans focus their thoughts on the important things that may happen (When will the plane depart?) without fretting about the minor details that fill each intervening moment (What is the last word will say to the taxi driver?). Because the vast majority of elements in a temporal sequence contains redundant information, temporal abstraction can make reasoning and planning both easier and more efficient. How can we build such an abstraction? Consider the example of lead animator who wants to show what happens in the next scene cartoon. Before worrying about every low-level detail, the animator first sketches out the story by keyframing, drawing the moments time when the important events occur. The scene can then be easily finished by other animators who fill in the rest of the sequence from the story laid out by the keyframes. In this paper, we argue that learning to discover such informative keyframes from raw sequences is an efficient and powerful way to learn to reason about the future.
Our goal is to learn such an abstraction for future image prediction. In contrast, much of the work on future image prediction has focused frame-by-frame synthesis (Oh et al. (2015): Finn al. (2016). This strategy puts an equal emphasis on each frame, irrespective of the redundant content it may contain its usefulness for reasoning relative to the other predicted frames. Other recent work has considered predictions that "jump" more than one step into the future, but these approaches either used fixed-offset jumps (Buesing al. 2018) used heuristics select the predicted frames (Neitz et al. 2018 Jayaraman et al. [019 Gregor et al., 2019). In this work, we propose a method that selects the keyframes that are most informative about the full sequence, so as to allow us to reason about the sequence holistically while only using small subset of the frames. We do so by ensuring that the full sequence can be recovered from the keyframes with an inpainting strategy, similar to how supporting animator finishes the story keyframed by the lead.
One possible application for a model that discovers informative keyframes is in long-horizon planning. Recently, predictive models have been employed for model-based planning and control (Ebert et al. (2018) ). However, they reason about every single future time step, limiting their applicability to short horizon tasks. In contrast, show that model that reasons about the future using small set of informative keyframes enables visual predictive planning for horizons much greater than previously possible by using keyframes as subgoals in hierarchical planning framework.

---

Under review as a conference paper at ICLR 2020
Keyframe Predictor
Sequence Inpainter
Figure Keyframing the future. Instead of predicting one frame after the other, we propose to represent the sequence with the keyframes that depict the interesting moments of the sequence. The remaining frames can inpainted given the keyframes.
To discover informative frames in raw sequence data, we formulate a hierarchical probabilistic model in which sequence is represented by subset of its frames (see Fig. [p. In this two-stage model, a keyframing module represents the keyframes as well as their temporal placement with stochastic latent variables. The images that occur the timepoints between keyframes are then inferred by an inpainting module. We parametrize this model with a neural network and formulate a variational lower bound on the sequence oog-liilihhod. Optimizing the resulting objective leads model that discovers informative future keyframes that can be easily inpainted to predict the full future sequence.
follows. We formulate a hierarchical approach for the discovery of infor- mative keyframes using joint keyframing and inpainting (KEYIN), We propose a soft objective that allows us to train the model in fully differentiable way. We first analyze our model on a simple dataset with stochastic dynamics in a controlled setting and show that it can reliably recover the underlying keyframe structure on visual data. We then show that our model discovers hierarchical temporal structure on more complex datasets of demonstrations: ment and a simulated robotic pushing dataset, which is challenging for current approaches to visual planning. We demonstrate that the hierarchy discovered by KEYIN is useful for planning, and that the resulting approach outperforms other proposed hierarchical and non-hierarchical planning schemes on the pushing task. Specifically, show that keyframes predicted by KEYIN can serve useful subgoals that can be reached by a low-level planner, enabling long- horizon, hierarchical control.
Our contributions
egocentric gridworld environ-
2 RELATED WORK
Hierarchical temporal structure. Hierarchical neural models for efficiently modeling sequences were proposed in Liu et al. 2015): Buesing et al. (2018). These approaches were further extended to predict with an adaptive step size so to leverage the natural hierarchical structure in language data (Chung et al. 20166] Kádár et al. 2018). However, these models rely on autoregressive techniques for text generation and applying them structured data, such videos, might be impractical.
The video processing community has used keyframe representations as early as 1991 the MPEG codec (Gall 1991). Wu al. (2018) adapted this algorithm in the context of neural compression; however, these approaches use constant offsets between keyframes and thus do not fully reflect the temporal structure of the data. Recently, several neural methods were proposed to leverage such temporal structure. Neitz et al. (2018) and Jayaraman et al. (2019) propose models that find and predict the least uncertain "bottleneck" frames. Gregor et al. (2019) construct a representation that can be used to predict any number of frames into the future. In contrast, we propose an approach for hierarchical video representation that discovers the keyframes that best describe certain sequence.
In parallel to our work, Kipf et al. (2019) propose related method for video segmentation via generative modeling. Kipf et al. (2019) focus on using the discovered task boundaries for training hierarchical RL agents, while we show that our model can be used to perform efficient hierarchical planning by representing the sequence with only a small set of keyframes. Also concurrently, Kim al. (2019) propose similar method to KEYIN for learning temporal abstractions. While Kim et al. (2019) focuses on learning hierarchical state-space models, we propose a model that operates directly the observation space and performs joint keyframing and inpainting.
2

---

Under review as a conference paper at ICLR 2020
Video modeling. Early approaches to probabilistic video modeling include autoregressive models that factorize the distribution by considering pixels sequentially (Kalchbrenner et al. 2017 Reed
et al. 2017). To reason about the images in the video holistically, latent variable approaches were developed based on variational inference (Chung et al., 2015 Rezende et al., 2014: Kingma &
Welling. 2014), including (Babaeizadeh al. 2018 Denton & Fergus 2018: Lee al.
2018) and
large-scale models such as (Castrejon et al.J [019 Villegas et al. 2019). Kumar et al.
(019)) is
recently proposed approach that uses exact inference based on normalizing flows (Dinh al. 2014
Rezende & Mohamed, 2013). We build on existing video modeling approaches and show how they can be used learn temporal abstractions with novel keyframe- based generative model.
Visual planning and model predictive control. We build on recent work that explored applica- tions of learned visual predictive models to planning and control. Several groups (Oh et al. 2015]
Finn et al. 2016 Chiappa et al. 2017) have proposed models that predict the consequences of actions taken by an agent given its control output. Recent work (Byravan et al.) 2017 Hafner et al., 201
Ebert et al. 2018) has shown that visual model predictive control based such models can be
applied to a variety of different settings. this work, we show that the hierarchical representation of sequence in terms of keyframes improves planning performance in the hierarchical planning setting.
KEYFRAMING THE FUTURE
K" 7"t
K r
Our goal is to develop a model that generates se- quences by first predicting key observations and the time steps when they occur and then filling in the remaining observations in between. To achieve this goal, in the following we (i) define probabilistic model for joint keyframing and inpainting, and (ii)
I
N
show how maximum likelihood objective leads to the discovery of keyframe structure.
A probabilistic model for jointly keyframing and inpainting future sequence. First, a sequence of keyframes K 11:N is gen- erated, as well corresponding temporal in- dices -lrN, defining the structure of the un- derlying sequence. In the second stage, for each pair of keyframes K" and Ka+1, the frames 1--:p=1-11 are inpainted.
Figure 2:
3.1 A PROBABILISTIC MODEL FOR JOINT KEYFRAMING AND INPAINTING
We first describe a probabilistic model for joint keyframing and inpainting of sequence /TT- The model consists of two parts: the keyframe predictor and the sequence inpainter (see Fig. 2
The keyframe predictor takes in C' conditioning frames I.o and produces N keyframes K1:N as well as the corresponding time indices ,I:N;
p(K
I:N
(co)
I(K", [ lin-1 Ico).
(I)
From each pair of keyframes, the sequence inpainter generates the sequence of frames in between:
(14mm11111/4 kn+1 n1+1 -") I[((// Kn+1 Lyna-1, n+1 - 7"),
which completes the generation of the full sequence. The inpainter additionally observes the number of frames it needs to generate -*+1 - -_" The temporal spacing of the most informative keyframes is data-dependent: shorter keyframe intervals might be required in cases of rapidly fluctuating motion, while longer intervals can be sufficient for steadier motion. Our model handles this by predicting the keyframe indices and inpainting 7 -*+1 -7 frames between each pair of keyframes. We parametrize the prediction of 7" in relative terms by predicting offsets 8": 7 -- 8".
3.2 KEYFRAME DISCOVERY
To produce a complex multimodal distribution over K we use a per-keyframe latent variable a with prior distribution p(z) and approximate posterior q(> I, Lox) [ We construct variational lower bound
For simplicity, the variable representing the full sequence is written without indices (I is the same T1:T).
3

---

Under review as a conference paper at ICLR 2020
on the likelihood of both I and K as follows
N
In ppp k() 2 Te
In E p(r rn / )oo)
[p( | Kmmut putl - -7))
,
inpainting
+ In In 2| 1oo)
DKL. (a(=)T, llo)))))))
keytramiin
regularization
In practice, we use weight 3 the KL-divergence term, is common in amortized variational inference (Higgins et al. 2017 Alemi et al., 2018 Denton & Fergus 2018).
If simple model is used for inpainting, most of the representational power of the model has to come from the keyframe predictor. We use relatively powerful latent variable model for the keyframe predictor and a simpler Gaussian distribution produced with a neural network for inpainting. Because of this structure, the keyframe predictor has to predict keyframes that describe the underlying sequence well enough to allow a simpler inpainting process maximize the likelihood. We will show that pairing more flexible keyframe predictor with simpler inpainter allows our model to discover semantically meaningful keyframes in video data.
4 CONTINUOUS RELAXATION BY LINEAR INTERPOLATION IN TIME
Our model can dynamically predict the keyframe placement 7 However, learning a distribution over the dis- crete variable 7" is challenging due to the expensive evaluation of the ex- pectation over 1[ Ico) in the objective in Eq. 3 To be able to eval- uate this term efficiently and in a dif- ferentiable manner while still learn- ing the keyframe placement, we pro- pose continuous relaxation of the objective. The placement distribution T" defines probability for each pre- dicted frame to match to a certain frame in the ground truth sequence. Instead of sampling from this distri- bution to pick target frame we pro- duce soft target for each predicted frame by computing the expected tar- get frame, i.e. the weighted sum of all frames in the true sequence, each mul- tiplied with the probability of match- ing to the predicted frame. When the entropy of 7" converges to zero, the continuous relaxation objective is equivalent the original, discrete objective.
Ground Truth Sequenoe
Keyffarne Predictor
1
1
1
P eccemen Distribution
n"d
K
Li 7
Keyfreme
L
K"n
Target
Figure 3: Soft keyframe loss in the relaxed formulation. For each predicted keyframe K" we compute target image K" as the sum of the ground truth images weighted with the corresponding distribution over index r" Finally, we compute the reconstruction loss between the estimated image K" and the soft target KT
Keyframe targets. To produce keyframe target, K", we lincarly interpolate between the ground truth images according to the predicted distribution over the keyframe's 's temporal placement r": T" E, 7/" It, where 7"" is the probability that the n keyframe occurs at timestep This process depicted in Fig. B
We parametrize temporal placement prediction in terms of offsets 8 with maximum offset of J. Because of this, the maximum possible length of the predicted sequence is N... It desirable for to be large enough to be able to capture the distribution of keyframes in the data, but this may lead to
2 Keyframes K and time indices 7 are modeled as independent given the corresponding latent variable such that the latter captures the dependency between the former two.
We find this occurs most of the time in practice.
4

---

Under review as a conference paper at ICLR 2020
Keyframe
Keyfrume
Segment
Segment
Segment
Ground Truth
Predicted, Keyln (ours)
Predicted, Jumpy
Figure 4: Sequences generated by KEYIN and a method with constant temporal keyframe offset (Jumpy) Brownian Motion data. Generation is conditioned the first five frames. The first half of the sequence is shown. Movement direction changes are marked red in the ground truth sequence and predicted keyframes are marked blue. We see that KEYIN correctly reconstruct the motion as selects an informative set of keyframes. The sequence generated by the Jumpy method does not reproduce the direction changes since they cannot be inferred from the selected keyframes.
the generation of sequences longer than the target NJ T. To correctly compute the value of the relaxed objective in this case, we discard predicted frames at times > ' and normalize the placement probability output by the network so that it sums one over the first I' steps. Specifically, for each keyframe we compute this probability as c²: e" Eist Ti' The loss corresponding to the last two terms of Eq.
(3) then becomes:
2 En c² (1IK 11 DRL. ( ((p(==))))
Lkey
(4)
En c"
Inpainting targets. To complete our relaxed objective, for each ground truth frame, we produce We note that offsets å have maximum range of J, and in general have non-zero probability on each timestep. the inpainting network needs to produce J frames ir,, between each pair keyframes (K" Km++). As in the previous section, the targets for ground truth images are given as interpolation between generated images weighted by the probability of the predicted frame In being matched ground truth frame Ie: I. (Eng, m",e-"") // En. m" Here, m",, is the probability that the j-th predicted image in segment n has an offset of f from the beginning of the predicted sequence, which can be computed from r" To obtain probability distribution over produced frames, we normalize the result with En. m, The full loss for our model is:
target image composed from the inpainted frames F
Liotal Lkey 33
Σ 114 E114 1.11. t
(5)
DEEP VIDEO KEYFRAMING
We show how instantiate KEYIN with deep neural networks and train it on high-dimensional observations, such as images. We further describe an effective training procedure for KEYIN.
5.1 ARCHITECTURE
We use a common mncoder-recurrre--deeem architecture (Denton & Fergus (2018) Hafner et al. (2018)). Video frames first processed with convolutional encoder module produce image embeddings 4 = CNNNane(T)) Inferred frame embeddings i are decoded with convolutional de- coder in = (C,) The keyframe predictor p( K ,I:N | 2 Ico) is parametrized with condi- tion the keyframe predictor on past frames, we initialize its state with the final state of another LSTM that processes the conditioning frames. Similarly, we parametrize the sequence inpainter p(L K k -1+1 embeddings, -n-1 and Rn, , as well as the temporal offset between the two, 8", by passing these inputs through layeirlayer perceptron that produces the initial state of the inpainting LSTM.
a Long Short-Term Memory network (LSTM, [Hochreiter & Schmidhuberi (1997)).
r²) with LSTM. We condition the inpainting on both keyframe
*This makes that each ground truth frame contributes equally the final loss.

---

Under review as a conference paper at ICLR 2020
Keyframe
Kcyframe
Start Frame
Segment
Segment 2
Ground
Truth
Predicted
Reconstruction
Keyframe Predictions
Ground
Truth
Predicted
26
Figure 5: Example generations by KEYIN on (top) Pushing and (bottom) Gridworld data. The generation conditioned on a single ground truth frame. Twelve of the 30 predicted frames are shown. We observe that for each transition between pushes and each action of the Gridworld agent our network predicts keyframe either exactly at the timestep of the event or one timestep apart. Note, although agent position is randomized, objects not visible in the first image be predicted in Gridworld because the maze is fixed across episodes.
We use a Gaussian distribution with identity variance as the output distribution for both the keyframe predictor and the inpainting model and multinomial distribution for 8 We parametrize the inference q(= [Tc+1:7) with an LSTM with attention over the entire input sequence. The inference distribution diagonal covariance Gaussian, and the prior p(2 D(21:N) is unit Gaussian.
Further details of the inference procedure are given in Sec. and Fig. 8 of the Appendix.
5.2 TRAINING PROCEDURE
We train our model two stages. First, we train the sequence inpainter to inpaint between ground truth frames sampled with random offsets, thus learning interpolation strategies for variety of different inputs. In the second stage, we train the keyframe predictor using the loss from Eq. by feeding the predicted keyframe embeddings the inpainter. In this stage, the weights of the inpainter are frozen and are only used to backpropagate errors to the rest of the model. We found that this simple two-stage procedure improves optimization of the model.
We use LI reconstruction losses to train the keyframe predictor. We found that this and adding reconstruction loss on the predicted embeddings of the keyframes, weighted with a factor 3k., improved the ability of the model to produce informative keyframes. Target embeddings are computed using the same soft relaxation used for the target keyframes, More details the loss computation are given in Sec. Eland Algorithm of the Appendix.
EXPERIMENTS
We evaluate the quality KEYIN'S representation for future sequences by addressing the following questions: (i) Can it discover and predict informative keyframes? (ii) Can it model complex data distributions? (iii) the discovered hierarchy useful for long-horizon hierarchical planning?
Datasets. We evaluate our model on three datasets containing structured long-term behavior. The Structured Brownian motion (SBM) dataset consists of binary image sequences of size 32 x 32 pixels which a ball randomly changes directions after periods of straight movement of six to eight frames.
The Gridworld Dataset consists of 20k sequences of agent traversing maze with different objects. The agent sequentially navigates to objects and interacts with them following a task sketch. We use the same maze for all episodes and randomize the initial position of the agent and the task sketch. We use 64 x 64 pixel image observations and further increase visual complexity by constraining the field of view to 5 x 5-cells egocentric window.

---

Under review as a conference paper at ICLR 2020
The Pushing Dataset consists of 50k sequences of a robot pushing puck towards goal on the opposite side of a wall. Each sequence consists of six consecutive pushes. We vary start and target position of the puck, well as the placement of the wall. The demonstrations were generated with the MuloCo simulator (Todorov et al. 2012) at resolution of 64 x 64 pixels. For more details on the data generation process, see Sec D of the Appendix.
Further details about the experimental setup are given Sec. Cf the Appendix.
6.1 KEYFRAME DISCOVERY
To evaluate KEYIN'S ability to dis- cover keyframes, we train KEYIN on all three datasets with N = 6, which can be interpreted as selecting the N most informative frames from se- quence. We show qualitative exam- ples of keyframe discovery for the SBM dataset in Fig. and for the Grid- world and Pushing datasets in Fig. 5
Table 1: Fl accuracy score for keyframe discovery all three datasets. Higher is better.
METHOD
BROWNIAN
PUSH
GRIDWORLDD
RANDOM
0.15
0.18
0.12
STATIC SURPRISE
0.21
0.18
0.25
0.73
0.10
0.32
KEYIN (OURS)
0.94
0.43
0.42
On all datasets the model discovers meaningful keyframes which mark direction changes of the ball, transitions between pushes or interactions with objects, adapting its keyframe prediction patterns to the data. Consequently, the inpainter network is able to produce frames of high visual quality. Misplaced keyframes yield blurry interpolations, as can seen for the jumpy prediction Fig. This suggests that keyframes found by KEYIN describe the overall sequences better.
To show that KEYIN discovers informative keyframes, we compare keyframe predictions against alternative approach that measures the surprise associated with ob- serving frame given the previous frames. This ap- proach selects keyframes as the N frames with the largest peaks in "surprise" measured by the KL-ddieegeence DKL. [nn]((2,///1. (<==///==)lp(((// [[p(=))) between the prior and the posterior of stochastic predictor based on Denton & Fergus (2018) (see Sec. and Algorithm 2 of the Appendix for details), We provide comparisons to alternative formulations of surprise in Appendix Sec. Tab.
Figure 6: Distribution of trajectories sampled from KEYIN. Each black line denotes one of 100 trajectories of the nipulated object. The obstacle is shown in blue and the initial position in pink. We see that our model covers both modes of the distribution, producing both trajec- tories that go to the right and to the left of the obstacle.
For quantitative analysis, we define approximate ground truth keyframes to be the points of direction change for the SBM dataset, the moments when the robot lifts its arm to transitions between pushes, or when the agent interacts with objects in the gridworld. We report Fl scores that capture both the precision and recall of keyframe discovery. We additionally compare to random keyframe placement, and learned but static baseline that is the same for all sequences. The evaluation in Tab. shows that KEYIN discovers better keyframes than alternative methods. The difference is especially large on the more complex Pushing and Gridworld datasets. The burprise-based method does not reason about which frames are most helpful to reconstruct the entire trajectory and thus is unable to discover the correct structure on the more complex datasets. In addition to the Fl scores, we report temporal distance between predicted and annotated keyframes in Appendix, Tab. 4 also indicating that KEYIN IN is better able to discover the temporal structure in both datasets.
6.2 KEYFRAME-BANE BASED VIDEO MODELING
Even though the focus of this work is on discovering temporal structure via keyframing and not on improving video prediction quality, we verify that KEYIN can represent complex data distributions in terms of discovered keyframes and attains high diversity and visual quality. We show sample generations from our model on the Pushing and Gridworld datasets on the supplementary website
httpss:/siites googloooooooooooooooooooooooo wowogle.coo/uue www./////ww/////////

---

Under review as a conference paper at ICLR 2020
We that Keyln is able to faithfully model complex distributions of video sequences. We further visualize multiple sampled Pushing sequences from our model conditioned on the same start position in Fig. [6] showing that KEYIN is able to cover both modes of the demonstration distribution. We further show that KEYIN compares favorably to prior work on video prediction metrics on sequence modeling in Tab. of the Appendix, and outperforms prior approaches in terms of keyframe modeling in Appendix, Tab. 6
6.3 ROBUSTNESS OF KEYFRAME DETECTION
In the previous sections, we showed that when the sequence can indeed be summarized with N keyframes, KEYIN predicts the keyframes that correspond to our notion of salient frames. However, what happens if we train KEYIN to select a larger or smaller amount of keyframes?
Table 2: Keyframe discovery for varied number of predicted keyframes. The data has approximately 6 keyframes. Uninterpretable entries are omitted for clarity: see the text for details.
KEYFRAMES
6
8
PRECISION
0.92
0.92
To evaluate this, we measure KEYIN recall with extra and precision with fewer available keyframes. We note that high precision is un- achievable in the first case and high recall is un- achievable in the second case, since these prob- lems are misspecified. As these numbers are not informative, we not report them. In Tab. we see that KEYIN able to find informative keyframes when N does not exactly match the structure of the data. We further qualitatively show that KEYIN selects a superset or a subset the original keyframes respectively in Sec. G This underlines that method's ss ability to discover keyframe structure is robust to the choice of the number of predicted keyframes.
BROWNIAN
RECALL
0.96
0.90
PRECISION
0.30
0.38
PUSH
RECALL
0.48
0.46
PRECISION
0.37
0.43
GRIDWORLD
RECALL
0.41
0.40
As first step towards analyzing the robustness of KEYIN under more realistic conditions we report keyframe discovery when trained and tested on sequences with additive Gaussian noise, a noise characteristic commonly found in real-world camera sensors. We find that KEYIN is still able to discover the temporal structure on both the Pushing and the Gridworld dataset. For qualitative and quantitative results, see Appendix Fig. and Tab.
6.4 HIERARCHICAL KEYFRAME-BASED PLANNING
We have seen that KEYIN can find frames that correspond to an intuitive notion of keyframes. This demonstrates that the keyframes discovered by KEYIN do indeed capture abstraction that compactly describes the sequence. In light of this, we hypothesize that an informative set of keyframes contains sufficient information about a sequence effectively follow the trajectory shows. To test this, we use the inferred keyframes as subgoals for hierarchical planning in the pushing environment. During task execution, we first plan a sequence of keyframes that reaches the target using our learned keyframe predictor. Specifically, we generate keyframe trajectories from our model by sampling latent variables from the prior and using them to roll out the keyframe prediction model. We optimize for a sequence of latent variables 2 that results in keyframe trajectory which reaches the goal using the Cross- Eatsosy Method (CEM, [Rubinstein & Kroese (2004). (2004). We then execute the plan by using the keyframes as subgoals for a low-level planner. This planner reaches each subgoal via model predictive control using ground truth dynamics, again employing CEM for optimization the action trajectory. This planning procedure is illustrated in Fig. 7 (left). For more details, see Sec. and Algs. 3 and of the Appendix.
We find that KEYIN is able to plan coherent subgoal paths towards the final goal that often lead to successful task execution (executions are shown on the supplementary website To quantitatively evaluate the keyframes discovered, we compare alternative subgoal selection schemes: fixed time offset (Jumpy, similar to Buesing et al. (2018), a method that determines points of peak surprise (Surprise, see Sec. 6.11 and a nottleneck-basel subgoal predictor (time-agnostic prediction TAP, Jayaraman et al. (2019).. We additionally compare to an approach that plans directly towards the final goal using the low-level planner (Flat). We evaluate all methods with the shortest path between
https://sitee: https://sites..ooggee www.iitee..gogl....... wwtp:///////eee/////////e/
8

---

Under review as a conference paper at ICLR 2020
Keyln
METHOD
POSITION ERROR
SUCCESS RATE
Keyframing
INTITIAL
1.32 + 0.06
RANDOM
1.32 + 0.07
1.
K
KN
FLAT
0.90 + 0.14
15.0%
TAP
0.80 1: 0.16
23.3% %
Control
SURPRISE
0.64 +: 0.28
50.8 %
JUMPY
0.62 + 0.33
58.8
ag
KEYIN (OURS)
+
We use the model to produce keyframes that represent the sequence between the current observation image and the goal. A low-level planner based on model predictive control produces the actions, a,, executed to reach each keyframe, until the final goal is reached. Right: Planning performance Pushing task. The hierarchy discovered by KEYIN outperforms comparable planning approaches.
Figure 7: Hierarchical planning on the Pushing dataset. Left:
the target and the actual position of the object after the plan is executed. All compared methods use the same low-level planner as we only want to measure the quality of the predicted subgoals.
As shown in Fig. 7 (right), our method outperforms all prior approaches. TAP shows only moderate increase in performance over the Flat planner, which we attribute to the fact that it fails to predict good subgoals and often simply predicts the final image as the bottleneck. This likely due to the relatively large stochasticity of our dataset and the absence of the clear bottlenecks that TAP is designed to find. Our method outperforms the planners that use Jumpy and Surprise subgoals. This further confirms that KEYIN is able to produce keyframes that are informative about the underlying trajectory, such that planning toward these keyframes makes easier to follow the trajectory.
DISCUSSION
We presented KEYIN, method for representing sequence by its informative keyframes by jointly keyframing and inpainting. KEYIN first generates the keyframes of a sequence and their temporal placement and then produces the full sequence by inpainting between keyframes. We showed that KEYIN discovers informative keyframes on several datasets with stochastic dynamics. Furthermore, by using the keyframes for planning, we showed our method outperforms several other hierarchical planning schemes. Our method opens several avenues for future work. First, an improved training procedure that allows end end-to-end training is desirable. Second, more powerful hierarchical planning approaches can be designed using the keyframe representation to scale to long-term real-world tasks. Finally, the proposed keyframing method can be applied to a variety of applications, including video summarization, video understanding, and multi-stage hierarchical video prediction.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy. Fixing broken ELBO. In Proceedings of International Conference on Machine Learning (ICML), 2018.
Mohammad Babacizadch, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In Proceedings of International Conference on Learning Representations (ICLR), 2018. 2018.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning align and translate. Proceedings of International Conference on Learning Representations (ICLR), 2015.
Lars Buesing, Theophane Weber, Sébastien Racanière, S. M. Ali Eslami, Danilo Jimenez Rezende, David P. Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, and Daan Wierstra. Learning and querying fast generative models for reinforcement learning. arXiv:/802.0088 002000006, 2018.
Arunkumar Byravan, Felix Leeb, Franziska Meier, and Dieter Fox. Sc3-pose-nets: Structured deep dynamics models for visuomotor planning and control. Proceedings of IEEE International Conference on Robotics and Automation, 2017.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vmns for video prediction. arXiv preprint arXiv: 12165,, 2019.
Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed, Recurrent environment simulators. Proceedings of International Conference on Learning Representations (ICLR), 2017.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Proceedings of Neural Information Processing Systems (NeurIPS), 2015.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hicrarchical multiscale recurrent neural networks. 2016.
E. Denton and R. Fergus. Stochastic video generation with a learned prior. In Proceedings of International Conference on Machine Learning (ICML), 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14X.v:1410.855 16, 2014.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robotic Learning (CoRL), 2017.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Vi- sual foresight: Model-based deep reinforeement learning for vision-based robotic control. arXiv:1812.00568, 2018
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In Proceedings of IEFE International Conference on Robotics and Automation, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Proceedings of Neural Information Processing Systems (NeurIPS). 2016.
D. Le Gall. MPEG: video compression standard for multimedia applications. Commun, ACM, 34 (4):46-58, 1991.
Karol Gregor, George Papamakarios, Frederic Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 1ete1111100555 2018.
10

---

Under review as a conference paper at ICLR 2020
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of International Conference Learning Representations (ICI.R), 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1780, 1997.
D. Jayaraman, F. Ebert, A. A. Efros, and S. Levine. Time-agnostic prediction: Predicting predictable video frames. Proceedings of International Conference on Learning Representations (ICLR), 2019.
Akos Kádár, Mare-Alexandre Côté, Grzegorz Chrupata, and Afra Alishahi. Revisiting the hierarchical multiscale Istm. arXiv preprint arXiv: 1807 885,03395, 2018.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1771-1779. JMLR. org, 2017.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. In Proceedings of Neural Information Processing Systems (NeurIPS), 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR), 2015.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of Interna- tional Conference on Learning Representations (ICLR), 2014.
Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi. Edward Grefenstette, Pushmeet Kohli, and
Peter Battaglia. Compositional imitation learning: Explaining and executing one task at a time. Proceedings of International Conference Machine Learning (ICML), 2019.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint 1933011903.01444 2019.
A. X. Lee, R. Zhang. F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. arXiv: 1804.015455 Abs/1804.01523. 2018
Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. Multi-timescale long short-term memory neural network for modelling sentences and documents. In Proceedings of the 2015 conference on empirical methods natural language processing, pp. 2326 2335-2 2015.
Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to iitention-basedd neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1421, 2015.
Alexander Neitz, Giambattista Parascandolo, Stefan Bauer, and Bernhard Schölkopf. Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. In Proceedings of Neural Information Processing Systems (NeurIPS). 2018.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Proceedings of Neural Information Processing Systems (NeurIPS), 2015.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. In Proceedings of the 34th International Conference on Machine Sourning-Volume 70, pp. 2912-2921. JMLR. org, 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of International Conference on Machine Learning (ICML), 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of International Conference on Machine Learning (ICMI., 2014.
11

---

Under review as a conference paper at ICLR 2020
Reuven Y. Rubinstein and Dirk P. Kroese. The Cross Etoss-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning. Springer-Vene Verlag New York, 2004.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V. Le, and Honglak Lee. High fidelity video prediction with large neural nets. In Proceedings of Neural Information Processing Systems (NeurIPS), 2019.
Chao-Yure Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpo- lation. In Proceedings of the European Conference Computer Vision (ECCV), pp- 416-431, 2018.
12

---

Under review as a conference paper at ICLR 2020
Io
1
I
I3
IT
Image (Future)
CNN enc
LSTM
Key-Value Attention
Softmax
Approximate Posterior g(a) To:N)
ka-1
kz
kn+1
Keyframes
Figure 8: Structure of the keyframe inference network. This diagram depicts the procedure to infer the embedding of the n-th keyframe, x", given the previously inferred keyframe embedding ===1 and the future images. The initial state of LST is produced by LSTMcam (not shown), which takes the embedding of past images as input. This ensures that the See the text for more details.
A VIDEOS
We include video results on the supplementary website at hittps://sin google.ggo..oooooooooooooooo.... com/ view keyin The website includes inference samples, prior samples, and exectued trajectories for all methods.
B ARCHITECTURE DETAILS
We found simple attention over LSTM outputs to be an effective inference procedure. Our approximate inference network LSTM outputs (x."*, (t)SsT, where ginf is an embedding used compute an attention weight and the G, are values to be attended over. We compute the posterior distribution over zt using key-value attention mechanism (Bahdanau et al. 2015 Luong et al. 2015
an.t exp(d(R"-1 K(
(6)
11" a
Σ an, Σ an
(7)
The distance metric, d, is the standard inner product. The architecture used for keyframe inference, including the attention mechanism, depicted in Supplemental Fig,
C EXPERIMENTAL SETUP
We set the prediction horizon to T 30 frames and predict N segments with J = 10 frames each for the SBM dataset and 6 frames each for the Pushing dataset. We pre-train the interpolator on segments of two to eight frames for Structured Brownian motion data, and two to six frames for Pushing data. The weight the KL-diiergence term for the interpolator VAE is 1e-3. 3. For training
13

---

Under review as a conference paper at ICLR 2020
the keyframe predictor, set 3x = 0, 3.c = 1, 3. The hyperparameters We activate the generated images with a sigmoid and use BCE losses on each color channel to avoid saturation. The convolutional encoder and decoder both have three layers for the Structured Brownian motion dataset and four layers for the Pushing dataset. We use a simple two-layer LSTTM with a 256-dimensional state in each layer for all recurrent modules. Each LSTTM has linear projection layer before and after it that projects the observations to and from the correct dimensions. We use the Adam optimizer (Kingma & Ba 2015) with 31 0.9 and 32 0.999, batch size of 30, and learning rate of 2c 2c-4. 4. For more details please refer to the appendix. Each network was trained on single high-end NVIDIA GPU. We trained the interpolator for 100K iterations, and the keyframe predictor for 200K iterations. The toal training time took about a day.
hand- sund-tuned.
In the Pushing environment, we use a held-out test set of 120 of sequences. The Structured Brownian Motion dataset is generated automatically and is potentially infinite. We used 1000 testing samples on the Structured Brownian Motion generated using a different random seed.
D DATA COLLECTION IN THE MUJoCo ENVIRONMENT
The data collection for our pushing dataset was performed in an environment simulated in MuJoCo Todorov et al. (2012). In the environment, robot arm initialized at the center of the table pushes an object to a goal position located at the other side of a wall-shaped obstacle.
The demonstrations followed a rule-based algorithm that first samples subgoals between the initial position of the object and the goal and then runs a deterministic pushing procedure to the subgoals in order. The ground truth keyframes of the demonstrations were completed.
defined by frames which subgoals
We subsampled demonstration videos by factor of when saving them the dataset, dropping every other frame in the trajectory and averaging actions of every two consecutive frames. For all datasets we generated for this environment following a rule-based algorithm, we only kept successful demonstrations and dropped the ones that fail to push the object to the goal position within predefined horizon.
E DETAILS OF THE LOSS COMPUTATION ALGORITHM
We describe the details of the continuous relaxation loss computation Algorithm I
Note that efficiently implement computing of the cumulative distributions which allows us to vectorize much the computation. Computational complexity the proposed implementation scales linearly with the number of keyframes N, and number of allowed frames per segment J and number of ground truth frames T. The final complexity is (N.J), which we find in practice to be negligible compared to the time needed for the forward and backward pass.
convolution,
F SURPRISE BASELINE
Standard stochastic video prediction methods do not attempt to estimate keyframes, as they are designed to densely estimate future videos frame-by-frame, Accordingly, they cannot be used directly as baselines for keyframe prediction methods, such KEYIN. However, |Denton & Fergus (2018) observe that the variance of the learned prior of a stochastic video prediction model tends to spike before an uncertain event happens. We exploit this observation to find the points of high uncertainty for our strong Surprise baseline. We use the KI. divergence between the prior and the approximate posterior KL[[g(z) (zel Ti:e)) } [[(:)) to measure the surprise. This quantity can interpreted as the number of bits needed to encode the latent variable describing the next state, it will be larger if the next state is more stochastic.
We train stochastic video prediction network with fixed prior (SVG-FP, Denton & Fergus (0188) with the same architectures of encoder, decoder, and LSTM as our model. We found that selecting the peaks of suprise works the best for finding true keyframes. The procedure use to select the keyframes is described in Algorithm In order to find the keyframes in a sequence sampled from the prior, we the inference network on the generated sequence.
14

---

Under review as a conference paper at ICLR 2020
Algorithm I Continuous relaxation loss computation
Parameters: Number of ground truth frames T, Number of keyframes N Input: Ground truth frames I1.,, Generated frames I, generated offset distributions 8
Convert the distributions of interframe offsets 8 to keyframe timesteps T", For the first keyframe, -² 81.
for / = 2..... .M do
Compute further 7" with chain rule. This can be efficiently computed via convolution:
T" 8"h i.e. 7
Σ En
end for Compute probabilities of keyframes being within the predicted sequence: c² EIST Ti.
Compute soft keyframe targets: K" = E. T/' It.
Compute the keyframe loss: (,n c" | K>1B//
Get probabilities of segments ending after particular frames: 6j EEii Get distributions of individual frames timesteps: m x Compute soft individual frames: Σ., my,e Compute the sequence loss: E, 11 I
nn- Tt-jt
Algorithm 2 Selecting keyframes via Surprise
Parameters:
Number of ground truth frames N, Desired number of keyframes M
Input:
Input sequence IITT, Stochastic Video Prediction model SVG(.)
Run the inference network over the sequence: q(arT)/ SVG())T)
Get the surprise measure: 8, (L=g((((4==)llp(aa)) Find the set of peak surprise points S where: >> $141 A 8, St-1.
SS] < M then
add M - |S] maximum surprise points S. end
Return: The M keyframes from S with maximum surprise.
G ABLATION OF THE NUMBER OF PREDICTED KEYFRAMES
We show qualitative results of training KEYIN with N = 4, 6 (optimal number), and 8 on the SBM dataset in Fig. 9 We observe that if train KEYIN to select smaller larger number of keyframes than needed. it learns to predict subset or a superset of the true keyframes, respectively. This property follows from the structure of the model, which encourages the model predict the keyframes that allow the full sequence to be inpainted. When too few keyframes are available, the model will be unable to put keyframes at all important times. but those it picks must still be good for inpainting. When more keyframes are available than necessary, the model can place the additional keyframes at any time, as only a subset of the keyframes are needed to ensure good inpainting.
Keyframes
Keyframes
Keyframes
Figure 9: Qualitative keyframe discovery the Structured Brownian Motion dataset for varying
number of predicted keyframes. Top: Ground truth sequence, keyframes with bold white frame. Bottom: KEYIN keyframe predictions at their predicted temporal placement. Even if the number of predicted keyframes does not match the true number of keyframes KEYIN correctly discovers the keyframes and their temporal placement.
15

---

Under review as a conference paper at ICLR 2020
Table 3: We compare different formulations for surprise-based keyframe detection method: (1) detecting maxima of the KL divergence between prior and posterior in stochastic prediction model, (2) detecting maxima in the lower bound on data likelihood log (ELBO) stochastic prediction model, (3) the formulation proposed in Denton & Fergus (2018) that detects maxima of the variance of learned prior distribution.
DATASET
PUSH
GRIDWORLD
METHOD
F1 1
min dxF
min
PRED OKF
F1
min STUE
min PRED
KL-SUPRISE
0.25
1.44
2.05
0.32
1.42
1.86
logp-Sune P-SUPRISSE
0.24
1.42
2.08
0.31
1.45
1.83
DENTON & FERGUS 2018)
0.17
L.73
2.01
0.35
1.10
1.53
Table 4: In addition to the Fl scores report the minimal temporal distance to the next keyframe as an additional metric that more graceful with respect to "close misses". Specifically, we report the distance the next annotated keyframe averaged across predicted keyframes, min KKF and, inversely, the distance the next predicted keyframe for each annotated keyframe, min kF For both datasets the distance metrics support the Fl results: KEYIN discovers keyframes that are better aligned with the annotated keyframes than the baselines.
DATASET
PUSH
GRIDWORLD
METHOD
F1 f
min dKF
FREED min dKF
F1
min dgF
min dKF
STATIC
0.18
1.67
1.25
0.25
1.22
1.07
SURPRISE
0.25
1.44
0.32
1.42
1.86
KEYIN (OURS)
0.43
1.25
1.86
0.42
1.03
0.99
H VIDEO MODELING PERFORMANCE
We further report quantitative results standard video prediction metrics, Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), in Tab. KEYIN able to match performance of two comparable prior approaches, showing that the keyframe- based modeling is able to represent complex data distributions.
I PLANNING ALGORITHM
Algorithm 3 Planning in the subgoal space.
Input: Keyframe model KEYIN(., .), cost function
Input: Start and target images I, and Inrget
Set the sampling distribution to the prior:
f4 = 0, 0. I
for i 1 H do
Sample sequences of latent variables:
21:N (N(u, os)
Produce L subgoal plans: kINN KEYIN I::)
Compute cost between generated and true targets: c( N Junget)
Choose L' best plans, refit sampling distribution:
Wi+10044 fit(z/)
end for
Return: Best subgoal plan K1 I:N
16

---

Under review as a conference paper at ICLR 2020
Ground Truth
Predicted
Keyframes
Legend
Input
Predicted
Figure 10: A training sequence and two samples from our model on the Structured Brownian motion dataset. Each image shows an entire trajectory. Our model first samples the keyframes (shown in red), and then deterministically predicts the rest of the sequence. The image resolution was enhanced for viewability.
Table SSIM and PSNR scores on pushing and gridworld dataset. Higher better.
DATASET
PUSH
GRIDWORLD
METHOD
PSNR
SSIM
PSNR
SSIM
DENTON FERGUS (2018)
33.3 0.1
0.956 + 0.001
29.4 0.1
0.812 0.01
JUMPY
33.7 0.1
0.960 0.001
0.831 0.001
KEYIN (OURS)
33.4 + 0.7
0.959 + 0.001
29.3 0.1
0.820 0.001
To apply the KEYIN model for planning, we use the approach for visual planning outlined in Algorithm 1. At the initial timestep. we use the cross-entropy method (CEM) Rubinttenn &
Start
Planned Trajectory
EVE
(20e4)](2004) to select subgoals for the task. To do so, we sample M latent sequences from
Executed Trajectory
41
42
Figure 11: Example keyframe detections on noisy sequences. Red frames mark annotated keyframes. Top: Pushing dataset. Bottom: Gridworld dataset. Each triplet depicts top: ground truth sequence with additive Gaussian noise, middle: predicted keyframes at the predicted time steps, bottom: predicted full sequence. KEYIN reliably able detect keyframes and reconstruct the full sequence.
17

---

Under review as a conference paper at ICLR 2020
Table 6: Keyframe SSIM and PSNR scores on pushing and gridworld dataset. Higher is better.
DATASET
PUSH
GRIDWORLD
METHOD
PSNR
SSIM
PSNR
SSIM
JUMPY
28.25 + 0.11
0.904 = 0.001
16.3 + 0.17
0.632 + 0.001
KEYIN (OURS)
29.5 + 0.16
0.911 + 0.001
18.4 + 0.17
0.636 + 0.001
Table 7: Fl score and distance closest annotated / predicted keyframe when trained and tested on sequences with additive Gaussian noise. KEYIN is able to reliably find keyframes on both datasets when trained and tested on noisy sequences. Even though the Fl score is lower on the Pushing dataset, the distances indicate that the discovered keyframes are well aligned with the annotated keyframes under noise.
DATASET
PUSH
GRIDWORLD
METHOD
F1 f
min dRF +
PRED min dKF
F1 1
TRUS min dKF
PPRD min dKF
+
KEYIN, NO-NOISE
0.43
1.25
1.86
0.42
1.03
0.99
KEYIN, GAUSS-NOISE NOISE
0.25
1.21
1.34
0.43
1.00
0.96
the prior (0, 1) and use the keyframe model to retrieve M corresponding keyframe sequences, each with L frames. We define the cost of an image trajectory as the distance between the get image and the final image of each keyframe sequence defined under a -ppecic- dis- tance function (see below). In the update step of the CEM algorithm, we rank the trajectories based their cost and fit diagonal Gaussian distribution to the latents z' that generated the M' = rM best sequences, where r is the elite ratio. We repeat the procedure above for a total of N iterations.
We define the cost between two frames used during planning as the Fuclidean distance between the center pixels the object in both frames. We recover the center pixel via color-based segmentation of the object. While this cost function designed for the particular planning environment we are testing on, our algorithm can be easily extended to use alternative, more domain-agnostic cost formulations
that have been proposed the literature Finn & Levine 2017
Ebert al. (2017 2018).
After subgoals are selected, we use CEM based planner to produce rollout trajectories. Similar to the subgoal generation procedure, at each time step, we initially sample M action sequences un from the prior N (0, I) and use the ground truth dynamics of the simulator to retrieve M corresponding image sequences, each with 7 frame We define the cost of an image trajectory as the distance between the target image and the final image of each trajectory. In the update step, we rank the trajectories based on their cost and fit a diagonal Gaussian distribution to the actions u' that generated the . = M best sequences. After sampling new set of actions un+ from the fitted Gaussian distributions we repeat the procedure above for a total of iterations.
Finally, execute the first action in the action sequence corresponding to the best rollout of the final CEM iteration. The action at the next time step is chosen using the same procedure with the next observation input and reinitialized action distributions. The algorithm terminates when the specified maximal number of planning steps Trax has been executed or the distance to the goal is below set threshold.
In practice, we clip the sampled actions maximal action range [-amas + 0 before passing them to the simulator.
18

---

Under review as a conference paper at ICLR 2020
Table 8: Hyperparameters for the visual planning experiments.
planning Parameters
Max. planning timesteps max
60
Max. per subgoal timesteps (Ie max)
10
Keyframe prediction horizon (I.)
6
# keyframe sequences (M)
200
planning horizon (l)
8
# planning sequences (M)
200
Elite fraction (r M' /M)
0.05
# refit iterations (N)
3
max. action (0max)
1.0
dywitch
5
We switch between planned subgoals if (i) the subgoal is reached, i.e. the distance to the subgoal is below a threshold dewitch measured in pixels, or (ii) the current subgoal was not reached for T, mmax execution steps. We use the true goal image as additional, final subgoal.
Algorithm Full two-stage planning algorithm
Input: Keyframe model KLLL- = LSTMMey 21.1)
Input: Video prediction model fect1 LSTM (11--1, u2a+1)
Input: Subgoal index update heuristics ixe+1 = sixx,, It, KI.t)
Input: Start and goal images and gaal- Initialize latents from prior: Zg ~ (O,
for 1. Nu. do
Rollout keyframe model for L steps, obtain M future keyframe sequences KILL.
Compute distance between final and goal image: dist( K Igoat).
Choose M best sequences, refit Gaussian distribution: pit o fit( (((K)).
key Rey
Sample new latents from updated distribution: zi+1 (N ..1)) end for
key key
Feed best sequence of latents into keyframe model to obtain subgoals: Ki-L
LSTMkey (4,, N o).
Set current subgoal to iX 1. 1... do
Tpian
Perform subgoal update ix, f(ix, 1, I--1, (.L)) Initialize latents from prior: 140 ~ (0,
for i = O N₂. do
Rollout prediction model for steps, obtain M future sequences Lestt.
Compute distance between final and subgoal image: dist(Lett, Kix)).
Choose M' best sequences, refit Gaussian distribution: 1441,
J.41 N441,04411 fit(u().
Sample new latents from updated distribution: u,+1 ~ N (pi+1, end for
Exccute "N..0 and observe next image It- end for
The parameters used for visual planning experiments listed in Supplemental Table
19

---

Under review as a conference paper at ICLR 2020
Start
Executed Trajectory
Figure 13: Sample planning task executions from the test set. From start state depicted on the left, the robot arm successfully pushes the object into the goal position (semi-transparee object) guided by the KEYIN subgoals. The right side of the figure shows intermediate frames the execution trajectories.
20


Reviews:
[{"review_id": "rklBAZ8AYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a model trained for video prediction hierarchically: a series of significant frames called “keyframes” in the paper are first predicted and then intermediate frames between keyframes couples are generated. The training criterion is maximum likelihood with a variational approximation. Experiments are performed on 3 different video datasets and the evaluation is performed for 3 tasks: keyframe detection, frame prediction and planning in robot videos.\nThe idea of generating an abstraction or a summary of a video via a sequence of important frames is attractive and could probably be used in different contexts. The proposed model is new and the authors introduce some clever ideas in order to train it. The evaluation work is important and the authors propose different settings for this evaluation. \nThe paper also present weaknesses. First the motivation for keyframes generation should be better developed: the model does not perform better than baselines for video frames prediction so that keyframes generation should be motivated by other applications. Planning as proposed by the authors could be one, but in this case it should be more developed. The main weakness is however the technical presentation which is painful to follow. When it is possible to get a general picture of what is done, it is quite difficult to figure out exactly how the model works. A global rewriting and maybe a better focus are required for publication. The probabilistic model (section 3.1) is relatively clear, even if it could be improved. It seems that the generation of a keyframe and the prediction of the corresponding time (tau^n)  are independent (eq. 3). This could be commented. Also it seems that in eq. 3 the log(K|z..) term should be inside an expectation. Section 4 was difficult to decipher for me. My understanding is that instead of sampling from a multinomial during training, you bypass this non differentiable operation by using what you call “soft targets” thus obtaining a differentiable objective (eq. 4). Is that true? In any case, the procedure should be made a lot clearer. The “intermediate frame” passage also remained confuse for me.\nConsidering the experiments, the authors make an important effort in order to evaluate different aspects of their model. In a fisrt step, they evaluate the ability of the model to generate significant keyframes using a detection setting.  It is not clear how they define ground truth frames for this evaluation. Those ground truth frames are defined as the frames where the movement in the image changes, which is easy on the Brownian movement dataset but what about the others? Also the baselines used in this comparison are weak. In the paper of Denton, they suggest some way to detect surprise and apparently this is not what you used. This should be justified/ commented. For keyframe modeling the proposed model behaves similarly to the baselines and even performs worse than the simpler “jumpy” model. Concerni g the paragraph about the selection of the number of predicted keyframes, it is not clear what is the reference (ground truth) number of target keyframes.\n The planning experiments are interesting, but difficult to follow at least from the main text.\nOverall, I think that there are several interesting ideas and realizations. They should be better put in perspective and explained.\n\n----- post rebuttal -----------\n\nThanks for the detailed answer. The paper is largely improved both for the style and the comparisons. But still requires further improvements. I will keep my score.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklIUCrW5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a variational objective to train a model which can jointly select keyframes of a video and generate the intervening frames to produce a resultant video. The model is provided an initial set of frames as context. At training time the model always learns to produce N*J frames, where N is the number of keyframes and J is a fixed number of frames to generate for each keyframe. The authors compare their method for selecting informative keyframes on a number of baselines and show an improvement over these baselines. The problem is interesting and well-motivated, but I have some concerns with the proposed approach and experiments. As such, I am a weak reject.\n\ncomments / questions:\n- Equation 3 lacks context. Initially, when looking at the authors' objective it seems that the inner expectation should be taken with respect to the joint time indices for the current and next keyframe. Only later after equation 4 do they mention that they always predict a fixed number of frames J. \n- The need for normalizing over the first T timesteps in equation 4 seems quite messy. Is it guaranteed that all of the needed keyframes will actually be within the first T timesteps? How does this work in practice?\n- Many important details of the inference procedure are relegated to the appendix. For example, there are no details for extracting which of the 60 keyframes that were trained for a sequence (due to the fixed length sequences) should be selected at test time. Looking at the appendix, it is clear that the approach requires an extensive planning algorithm at inference time, which seems like an important component.\n- The authors prominently highlight that their method is fully differentiable, yet they train in two stages while freezing weights. Why isn't the model trained end-to-end? The stated reason for doing so is that this \"simple\" two-stage procedure improves optimization. What exactly happens if you don't do this two stage training process? Does it fail to learn? Some experimental numbers would be nice to see. \n- The authors do not compare their method to any strong keyframe prediction baselines. Considering there is existing work in keyframe prediction, it seems important to highlight the difference between other competing models, rather than relying on simple baselines. Why don't they use self-information/surprisal as a baseline i.e., by training an autoregressive model on the frames and then picking the N frames with the largest -log(p)? This is a metric that has been investigated frequently and has better interpretability than defining a new measure of surprise. Note that Kipf et al. (2019) uses this notion of surprisal as well.\n- Sauer et al. (BMVC 2019) should likely be cited as it does very similar keyframe analysis. Also, as the ICML 2019 conference had already concluded by the ICLR submission deadline, is it really fair to state the work with Kipf et al. (2019) was conducted in parallel?\n- Why does the model trained to learn a fixed number of timesteps for the intermediate frames? Did they investigate jointly predicting the indices for the current and next timesteps? It seems like it would greatly simplify their inference scheme if they did this. If they tried that approach and it failed, maybe that should be mentioned in the paper (with an explanation as to why it fails).\n- In the literature review, when discussing hierarchical temporal structure, the authors state: \"However, these models rely on autoregressive techniques for text generation and are not applicable to structured data, such as videos.\" Autoregressive techniques have been investigated in relation to videos; in fact, the authors later describe papers that have used autoregressive techniques for modeling videos.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJl6wpNVcS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address the problem of discovering and predicting with hierarchical structure in data sequences of relevance to planning. Starting with the kinds of data that have been used recently in video prediction, the authors aim at learning a sequence of keyframes (i.e., subsets of frames forming the overall sequence) that in a suitable sense \"summarize\" the overall trace. As they rightly note, many alternate models struggle with making good long term predictions in part because they focus on all levels of prediction equally.\n\nThe technical approach is to pose the problem as one of inferring the temporal location of each of these key frames and then to interpolate with a model to generate intermediate frames. One could try to make either step sophisticated - the authors choose to make the keyframe selection more sophisticated and interpolation simpler. The paper first described the KeyIn model in terms of a probabilistic model of jointly finding the Ks and then the inpainted Is. This can become delicate, so the authors propose a relaxation that is more forgiving when the keyframe locations are being searched for. Learning is driven by a reconstruction loss of finding the approximate location, locally interpolating and then seeing if this accords with the training data. This is all implemented with an LSTM based NN architecture which seems sound to me.\n\nI feel the paper is taking on the right kinds of questions, looking for ways to inject the right kind of structure. I do have some concerns about the overall formulation:\n\n1. Much of the paper is focussed on rather clean images where nothing extraneous is happening. In reality, the backgrounds of real images is not so benign and other extraneous dynamics might interfere. While I understand this is a step towards the long term goal, I wonder if the end result is a bit too incremental in the absence of some attempt to explore this source of (lack of) robustness.\n\n2. In §6.3, the authors try to demonstrate that the number of keyframes parameter can be wrong by a little bit but these are still small ranges. In realistic images it is likely that the total number of keyframes selected by such an algorithm is much larger due to extraneous events. This is why a proper robustness study is crucial on more realistic input. As it, in anything other than the trivial dot on black background, the precision-recall numbers are fairly modest. This will likely degenerate into noise in most camera-based images of the kind seen by a real robot. So, how much confidence should we expect to have in the approach's generality?\n\n3. For the baselines, the true good baseline might have been a human annotation that tells us how people really conceptualise the structure. With data such as pushing, this might not be so different from the simple visual inspection, but again with real data this will vary. The paper would really be much stronger if these were addressed.\n\n\n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is interesting in video prediction, introducing a hierarchical approach: keyframes are first predicted, then intermediate frames are generated. While it is acknowledge the authors do a step in the right direction, several issues remain: (i) the presentation of the paper could be improved (ii) experiments are not convincing enough (baselines, images not realistic enough, marginal improvements) to validate the viability of the proposed approach over existing ones.

Decision: Reject