Goal-Conditioned Video Prediction
Authors: Oleh Rybkin
Karl Pertsch
Frederik Ebert
Dinesh Jayaraman
Chelsea Finn
Sergey Levine
Venue:  (2020)
Keywords: predictive models
video prediction
latent variable models

Abstract: Many processes can be concisely represented as a sequence of events leading from a starting state to an end state. Given raw ingredients, and a finished cake, an experienced chef can surmise the recipe. Building upon this intuition,  we propose a new class of visual generative models: goal-conditioned predictors (GCP). Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video. GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames.  Not only do existing forward prediction approaches synthesize better and longer videos when modified to become goal-conditioned,  but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons.  Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training  and  inference  very  efficient, and allowing the model to train on sequences that are thousands of frames in length.Finally, we demonstrate the utility of GCP approaches for imitation learning in the setting without access to expert actions. Videos are on the supplementary website: https://sites.google.com/view/video-gcp

Under review as a conference paper at ICLR 2020
GOAL-CONDITIONED VIDEO PREDICTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Many processes can be concisely represented as a sequence of events leading from a starting state end state. Given ingredients, and finished cake, an experienced chef can surmise the recipe. Building upon this intuition, we propose new class of visual generative models: goal-conditioned predictors (GCP). Prior work on video generation largely focuses on prediction models that only observe frames from the beginning of the video. GCP instead treats videos as start-goal transformations, making video generation easier by conditioning on the more informative context provided by the first and final frames. Not only do existing forward prediction approaches synthesize better and longer videos when modified become goal-conditioned, but GCP models can also utilize structures that are not linear in time, to accomplish hierarchical prediction. To this end, we study both auto-regressive GCP models and novel tree-structured GCP models that generate frames recursively, splitting the video iteratively into finer and finer segments delineated by subgoals. In experiments across simulated and real datasets, our GCP methods generate high-quality sequences over long horizons. Tree-structured GCPs are also substantially easier to parallelize than auto-regressive GCPs, making training and inference very efficient, and allowing the model to train on sequences that are thousands of frames in length. Finally, we demonstrate the utility of GCP approaches for imitation learning the setting without access to expert actions. Videos are on the supplementary website: ht tps: / //sites. google. com wwwn///vw///iee wwwwwwwwwwwee.. video-gcp
INTRODUCTION
Many phenomena, both natural and artificial, are naturally characterized as transformations - the most salient information about them is contained in the start and end states, given which is possible to fill in intermediate states from prior experience. For example, ending up in San Francisco after starting in Oakland entails getting into a car and crossing the Bay Bridge. Similarly, to an expert engineer observing bridge, the task of reverse-engineering how was built is well-defined and tractable.
In contrast, consider the task of predicting forward in time, having observed only the steel and concrete that went into making the bridge. Such forward prediction tasks are severely undervonstrained, leading high uncertainties that compound with time, making it impossible make meaningful predictions after only a few stages of iterative forward prediction (see Fig. 1). This is aggravated in high- dimensional settings such as forward video prediction, which despite being the most widely studied setting for video synthesis, struggles to produce coherent video longer than few seconds.
We propose to condition video synthesis instead on the substantially more informative context of the start and the goal frame. We term such models goal-conditioned predictors (GCP). Much like the engineer observing the bridge, GCPs treat long videos as start-goal transformations and reverse- engineer the full video, conditioned the first and final frames. The simplest instantiation of GCPs modifies existing forward prediction approaches to also observe the final frame.
More broadly, once we consider conditioning the goal frame, we can devise new types of GCP models that more efficiently leverage the hierarchical structure present in real-world event sequences (Fig. 1, right). Just coarse-to-fine fo-neee image synthesis (Karras et al., 2017) generates nigh-resolution image by iteratively adding details to a row-resouttion image, we can synthesize a temporally downsampled video in the form of sequences of keyframes, and fill it in iteratively. We propose to

---

Under review as a conference paper at ICLR 2020
Non-Goal-Conditoone
Goal-Conditioned Sequential
Goal-Conditioned Hierarchical
Start
Goal
Predicted First
Predicted Last
Figure I: We study the task of prediction towards given goal. When conditioning predictions the goal, the distribution of possible trajectories reduces significantly (left middle), making the modeling task easier and therefore allowing to scale prediction to much longer time horizons. Additionally, such goal-conditioned prediction enables more efficient, hierarchical prediction schemes (right).
implement this as tree-structured GCP: the start-goal input frames serve as the initial downsampled sequence and new frames recursively added until the full video is generated. Intuitively, this exploits the hicrarchical temporal structure natural video by breaking a long sequential procedure into its constituent steps, illustrated in Fig 1. However, procedural steps do not all occur regularly spaced schedule or last for equal lengths time. To model this, we further propose to allow the model select which frames to generate at each level in the tree.
In our experiments, all GCP variants successfully generate longer and higher-quality video than has been demonstrated with standard auto-regressive video prediction models, which only utilize the starting frames for context. Furthermore, we show that trec-structured GCPs are more parallelizable than auto-regressive models, leading to very fast training and inference. We show that can train tree-structured GCPs on videos consisting of thousands of frames. We also study the applications of GCPs, demonstrating that they can be utilized to enable prediction based control in simulated imitation learning scenarios. In these settings, the GCP models can be trained without access to demonstrator actions, and can synthesize visual plans directly from start and goal images, which can then be tracked using an inverse model.
RELATED WORK
Video generation. Several existing neural video generation approaches either generate the entire video from scratch (Vondrick et al., 2016; Clark et al., 2019) conditioned on the beginning of the video (Ranzato et al., 2014; Srivastava et al., 2015; Mathicu et al., 2015; Finn et al., 2016; Oh et al., 2015). Some approaches also model the uncertainties inherent in video prediction through variational inference (Denton & Fergus, 2018: Villegas et al., 2019; Xue et al., 2016; Lee et al., 2018; Larsen al., 2016; Babaeizadeh et al., 2018). As argued above, video prediction approaches struggle to synthesize videos longer than a few seconds due to the ill-defined nature of this task. To rectify this, we propose a different class models that conditions the prediction on both the first and the last frame of the desired video. This is related to prior work on neural video interpolation. However, such work has focused on short-term interpolation, often using models based on optical flow. Liu et al. (2017); Jiang et al. (2018) 8) propose techniques that transform the start and the goal images via flow fields predicted by convolutional neural network (CNN). Niklaus et al. (2:17;;) transform the start and the goal images with convolutional kernels predicted with CNN. These methods rely techniques that, while well-suited for short-term video in-filling, are not
on simple warping-ber effective for generating longer video sequences. In contrast, we propose powerful goal-conditioned video prediction technique that scales to videos as long as a minute in length. We compare against video interpolation in our experiments.
Visual imitation and planning. We show application of GCP for imitation learning in the case where we have expert data without actions. Prior methods for solving the visual imitation learning problem have tried learn reward functions from demonstrations train reinforcement learning
2

---

Under review as a conference paper at ICLR 2020
0
S
22
S2
Sg
ST-
ST
O
O
O
b)
Figure 2: Graphical models for state-space space video generation. Left: video continuation. Right: the proposed goal-conditioned predictors (GCPs). b) sequential goal-conditioned prediction. <) a hierarchical goal-conditioned predictor. Shaded circles denote observations, and others are unobserved latent states.
agents to perform imitation (Sermanet et al., 2016; Peng et al., 2018; Sermanct et al., 2018), suffering from high sample complexity, and exploration problems of reinforcement learning. Our proposed goal-conditioned imitation framework, does not need to perform reinforcement learning and also leverages expert demonstrations overcome the exploration problem. Visual goal-directed behavior can also be achieved using visual planing methods, such as visual foresight (Ebert et al., 2018) or Causal InfoGan (Kurutach al., 2018). Unlike GCP-based imitation however, these methods do not attempt to imitate the expert, but instead compute plans internally by sampling imaginary states or actions, exhibiting difficulties for longer control horizons targeted with our proposed method. Inverse models such as zero-shot visual imitation proposed by Pathak et al. (2018) require access to actions and are therefore insufficient work in the action-free imitation setting.
GOAL-CONDITIONED PREDICTION
In this section, we formalize the goal-condition prediction problem, and propose several models for goal-conditioned prediction, including both auto-regressive models and tree structured models. To define the goal-conditioned prediction problem, consider a sequence of observations [o), 02, ...T] of length T. Standard forward prediction approaches observe the first & observations and synthesize the rest of the sequence, i.e., they model p(0k+1, 0b+2. 0r-1 01 02. ....)). 0k). Instead, we would like conl-condii predictors to produce intermediate observations given the first and last elements in the sequence. other words, they must model p(02, 03, 0r 111, or). We propose several designs of coal-conditioned predictors that operate in learned compact state spaces for scalability and accuracy. Fig 2 shows schematics of these GCP designs, as well as of standard forward predictor.
3.1 GOAL-CONDITIONS SEQUENTIAL PREDICTION
We first present simple regresseee model for goal-conditioned prediction. In this model, predict latent state representations sequentially in chronological order, from the start to the end, with the prediction at each point in time conditioned the first and final observations previous latent state. The resulting model (GCP-sequential, shown in Fig 2, b) can be factorized as follows:
well as the
T-1 p(o2, 02........p 0r oT-101, = plo2)12))(12/011.0 oT) II
or. (1)
We show in Sec 3.4 that this model is simple implement, and can build directly previously proposed auto-regressive video prediction models. However, its computational complexity scales with the sequence length, every state must be produced in sequence. Furthermore, this approach does not account for the hierarchical structure natural video, which contains events and sub-events, depicted in Fig 1.
3

---

Under review as a conference paper at ICLR 2020
3.2 COADIIONTT PREDICTION BY RECURSIVE INFILLING
To account for this hierarchical structure and also devise a computationally more efficient method, we now design a tree-structured GCP model.
Suppose that we have an intermediate state prediction operator p((*/ppa(() that produces an inter- mediate state St halfway in time between its two parent states pa(t). Then, consider the following alternative generative process for goal-conditioned prediction: at the beginning, the observed first and last frames are encoded into the latent state space as $1 and sy, and the prediction operator p(5e (pa(t))() generates $0.52. The same operator may now be applied to two new sets of parents ($1, $0.57) and (S0.5T, $T). As this process continues recursively, the intermediate prediction operator fills more and more temporal detail until the full video is synthesized.
Fig 2, e depicts this model, which we call GCP-tree, since it has tree-like shape where each predicted state is dependent on its right and left parents, starting with the start and the goal. GCP-tree factorizes the goal-conditioned video generation problem as:
T Il ploclse)o(ss.pp(()))da)
p(o2, 03, OT 11-1 p(s))o1)p(ST(or)
Computational efficiency. While the sequential forward predictor performs (TT) sequential op- erations to produce a sequence of length T. the hierarchical prediction can be more efficient due to parallelization. As the depth of the tree [log 7", it only requires (log T') sequential operations to produce a sequence, assuming all operations that can be conducted in parallel are parallelized perfectly. We therefore batch the branches of the tree and process them in parallel at every level to utilize the benefit of efficient computation
modern GPUs We note that the benefits of the GCP-tree runtime lie in parallclization, and thus diminish with large batch sizes, where the parallel processing capacity of the GPU is already fully utilized. We notice that, when predicting sequences of 500 frames, GCP- -equennulttial can use up to times bigger batches than GCP- Tree without significant increase in runtime cost.
Adaptive binding. We have thus far described the intermediate prediction operator as always generating the state that occurs halfway in time between its two parents. While this is a simple and effective scheme, it may not correspond to the natural hierarchical structure in the video. For example, in the navigation example in Figure 1, we might prefer the first split to correspond to traversing the bridge, which partitions the prediction problem into two largely independent halves. We can design version of GCP-tree that allows the intermediate frame predictor to select which of the several states between the parents to predict, each time it is applied. In other words, the predicted state might bind to one of many observations in the sequence. In this more versatile model, we represent the time steps of the tree nodes with discrete latent variable w that selects which nodes bind to which frames: p(o,) $1:N, wx) = p(o (w..). We can then express the prediction problem as:
T-1
p(02 / p(sslo))p(swlo> (o) II p(s (pa(n)) II
p(or
(3)
Appendix C shows an efficient inference procedure for w based on dynamic programming.
3.3 LATENT VARIABLE MODELS FOR GCP
To build powerful probabilistic models for goal-conditioned prediction, we propose to model the stochasticity with a per-frame latent variable = (see Fig 3), inspired by prior work on forward prediction Buesing et al. (2018); Denton & Fergus (2018). To train the models, we maximize lower bound on the likelihood the sequence that is computed using amortized variational inference. In practice, we use weight 3 on the KL-divergence term, is common in amortized variational inference (Higgins et al., 2017; Alemi et al., 2018; Denton & Fergus, 2018).
Inp(o2r-1lo..)) Eg Ex(=ar-1=)) [Inp(o2T-1101 22T-11) 3KL (g(2lorr) ((222-1101))
4

---

Under review as a conference paper at ICLR 2020
5
S
S
Z
S
1
Figure 3: Proposed latent variable models. Circles represent stochastic variables, shaded circles represent variables observed at training and squares represent deterministic variables. Left: Sequential prediction. Right: Hierarchical prediction.
3.4 ARCHITECTURES FOR GOND-IINDTT PREDICTION
Having introduced the probabilistic framework of goal-conditioned prediction, and several instantia- tions of that framework as graphical models, we now describe how these graphical models can be implemented deep neural networks, to make it possible to predict sequences of high-dimensional observations 01:7, such as images. Our predictive models depicted in Fig. 3 consists the follow- ing parts: the prior p( 2a sx)), the deterministic recurrent predictor p(sa 22 pa())), and the decoding distribution p(or st). Additionally, we use an amortized variational inference network g( (1:T).
We parameterize these distributions follows. The prior diagonal covariance Gaussian whose parameters are predicted with multi-layer perceptron (MLP). The recurrent predictor p( [t, pa(()) long short-term memory network (LSTM, (Hochreiter & Schmidhuber, 1997)). The decoding distribution p(or)s:) is unit variance Gaussian with the mean ōt predicted by convolutional decoder. We condition the recurrent predictor on the start and goal images via first encoding them into embeddings e enc(o:). We denote the corresponding embedding space of the decoder dec(e;).
The posterior distributions for each node, q(2 (1:r), is computed using an attention mechanism over the embeddings of the evidence sequence (Bahdanau et al., 2015; Luong et al., 2015): g(zx) = Att(ene(o1:T). pa()))) For simplicity, we reuse the same frame embeddings e, for the attention mechanism.
We found that using TreeLSTM (Tai et al., 2015) as the backbone of the hierarchical predictor significantly improved performance, perhaps due to better capturing long-term temporal dependencies. To increase the visual quality the generated results, we also use skip-connections from the encoder of first and the last frame (Villegas et al., 2017: Denton & Fergus, 2018) to all decoded frames and a foreground-backggruudd generation procedure similar to (Wang et al., 2018). We activate the generated images with a hyperbolic tangent.
GOAL-CONDITIONED IMITATION LEARNING WITH GCP
We demonstrate natural application of GCP for imitation learning, where the goal is learn controller that can select actions that reach a user-indicated goal observation. We can utilize a GCP trained on expert optimal behavior to generate sequence of predicted observations that must occur in optimal trajectories between the current observation and the goal, i.e., a plan. Crucially, such GCP can be trained on data that is not annotated with any actions for example, a robot could learn behaviors from raw videas downloaded from YouTube. After the predictions are generated, we still need to actually select the action to take, but this is now a much simpler problem, since the GCP predicts the very next observation, and can be accomplished using a one-step inverse model.
Formally, at each time step, our imitation learning agent receives the current observation o, and goal observation or. The GCP model outputs a sequences of observations ôz..ôō leading from the start to the goal. To infer the actions necessary to execute this plan, we train a separate inverse model that estimates p(alo, o') the probability that the action will lead from observation to o'. This model can be trained on any dataset that includes actions, including random behavior, since it does not need to perform any long-hoo nong-horizon reasoning. In our implementation, is trained a supervieed fashion on trajectories from a random controller, without any expert supervision, while the GCP model is trained on expert data but without access the expert's actions.
inverse model
5

---

Under review as a conference paper at ICLR 2020
Human 3.6
Plak&
3x3 Maze
10x10 Maze
Figure 4: Prior samples from GCP-tree on the four datasets: Human 3.6, pick&place, 3x3 Maze and 10x10 Maze. Each sequence is subsampled to 9 frames.
The complete imitation method is summarized in Algorithm Instead of targeting the predicted observations 6, with an inverse model, it is also possible to train the inverse model to infer actions based on the encodings of the observations o dec(é,). The advantage of using the encodings is that uncertainties and inaccuracies induced in the image generation process cannot negatively affect the inverse model's performance.
Algorithm 1 Goal Conditioned Imitation Learning
1: Inputs: Goal-conditioned predictor g, inverse model p(a|s, s'), goal observation or
2: while True do
3: Ōg. = g(,,, OT)
4: for i Orepalreplan
do 5: Compute = arg max p(a)ó; or)
6: Execute action dz-
EXPERIMENTAL EVALUATION
The aim of our experiments study the following questions: 1) Are goal-conditioned models able to generate long- horizon video sequences? 2) Does tree-structured prediction improve efficiency long-horizon video prediction? 3) Does goal-conditioned prediction enable long-term imitation without access to the expert's actions?
Datasets. We evaluate our model on the three datasets depicted in Fig. 5). First, the Human 3.6M dataset (Ionescu et al., 2013) is a visu- ally complex dataset of human actors that is commonly used for evaluation of visual pre- dictive models. We further design two simu- lated long-term video datasets. The pick&place dataset contains videos of simulated Sawyer robot arm placing objects into bin. Demonstra- tions are collected using rule-based policy that has access the underlying world state. Our third dataset, Maze, collected an environ- ment based the Myym-iniworlld (Chevalier-
Figure 5: Datasets used for our evaluation of goal- conditioned prediction. Left right: Human 3.6M (64x64px), pick&place dataset (64x64px) and Maze dataset (16x 16px)
Boisvert, 2018) simulator which consists of number of sparsely connected rooms located on grid. The maze layout is constructed such that one single path exists between every pair of rooms and the

---

Under review as a conference paper at ICLR 2020
Table I: Prediction performance. Plak& data 80 frames 64x64, H3.6M data 500 frames 64x64, 3x3 Maze data are 80 frames 32x32, 10x10 Maze data are 1000 frames 16x16.
DATASET
PICK&PLACE
HUMAN 3.6M
3x3 MAZE
10x10 MAZE
METHOD
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SEQUUENNL
34.45
0.965
27.57
0,924
20.83
0.664
17.08
0.48
GCP-TREE
34.34
0.965
28.34
0.928
20.80
0.652
17.29
0.495
DVF
26.15
0.858
26.74
0,922
15.798
0,485
14.50
0.428
CIGAN
21.16
0.613
16.89
0.453
15.57
0.415
13.81
0,331
SVG
31.41
0.960
19.2
0.600
17.01
0.496
WICHERS ET AL. (2018)
19.37
0.815
12.38
0.50
wall texture of each room is unique. Demonstrations between random start and goal positions are collected using the probabilistic roadmap (PRM) planner (Kavraki et al., 1996), leading to demon- strations with substantial noise. Due its flexible layout the maze dataset can scale to arbitrarily long-houe long-horizon tasks. We evaluate our method on a 3 room and 10x 10 room layout.
We use 64 x 64px spatial resolution for Human 3.6M and pick&place and train on sequences of 500 frames and 80 frames respectively. For Maze, we generate sequences up to 100 frames on the 3x3 layout and 1000 frames for the 10x 10 rooms. Due to the very long sequences we evaluate on resolution of 16x 16px.
5.1 GOAL-CONDITIONED PREDICTION
We compare GCP-sequential and GCP-trec to state- of-the-art deep video interpolation method, DVF (Liu al., 2017)' and a method for generation of visual plans, CIGAN Kurutach et al. (2018) (see Tab. 1). Following the standard procedure for evaluation of stochastic predic- tion models, we report top-of- Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Metric (SSIM). We observe that the proposed goal-conditioned prediction models outperform the video interpolation baseline by large margin. We find that the interpolation method fails to learn meaningful long-term dynamics and instead merely blends over between the start and the goal image (see Fig. 6). CIGAN, which uses latent space interpolation, similarly struggles to capture long-term transformations, predicting unphysical changes in the scene. In contrast, both GCP-sequential and GCP-tree learn to predict rich scene dynamics in between distant start and goal frames, synthesizing sequences that traverse tens of different rooms (see Fig. 4). We attribute these results to the more powerful stochastic latent variable model our methods employ.
Table Ablation of prediction perfor- mance on pick&place
METHOD
PSNR
SSIM
TREE
34.34
0.965
TREE w/O SKIPS
32.64
0.955
TREE W/O LSTM
31.44
0.947
We also include comparison Stochastic Video Generation (SVG, Denton & Fergus (2018)), a prior approach that employs a stochastic latent variable model, but without goal conditioning. As expected, without the goal information the prediction problem is too underconstrained and the resulting predictions rarely match the ground truth sequene. We show that incorporating goal information leads directed predictions of higher accuracy across all datasets. We provide evaluations of all models on the perceptual metrics Fréchet Video Distance (FVD, Unterthiner et al. (0) and Learned Perceptual Image Patch Similarity (LPIPS, Zhang al. (2018)) in the appendix, Tab.
To further inspect the properties of the proposed approach, we ablate different architectural choices in Tab. using the tree-structured predictor. We show that both skip connections from the parents and start and goal as well as recurrence in the predictive module are essential to achieve high prediction performance.
5.1.1 RUNTIME COMPARISON
We note that all methods including DVF were trained on the training split of the datasets they were tested allow for fair comparison.
7

---

CONTROL Under review as a conference paper at ICLR 2020
0
124
186
249
311
373
435
498
Data
GCP-tree
GCP-seq
DVF
CIGAN
Figure 6: Predictions on Human 3.6M. We see that the GCP models are able faithfully capture the human trajectory. The optical flow-based method (DVF) captures the background but fails to generate complex motion needed for long-tern conditiond prediction. Causal InfoGan also struggles to capture the structure of these long sequences and produce implausible interpolations.
20
GCP-Tree
One advantage the hierarchical prediction approach of GCP. Tree over the came-by--frmm prediction of GCP- GCP-sequential is that the former allows for heavy parallelization of computation, leading to substantially reduced sequential computational com- plexity (see Sec. 3.2). We validate that we can realize these runtime benefits by comparing the time required for each train- ing iteration of forward and tree-structured prediction across different sequence lengths (see Fig. 7). We find that, espe- cially for very long sequences, GCP-Tree has a much lower iteration time, enabling more efficient training long-horizon prediction tasks.
GCP sequential
15
BachBarch
10
10a
10a
Sequence Length
Figure 7: Runtime GCP for dif- ferent sequence lengths 10x10 Maze (16 x 16px) with a batch size of four. Experiments were performed on a standard NVIDIA P100 GPU.
5.2 BOTTLENECK DISCOVERY
We evaluate the ability of the model with adaptive binding to learn the structure of the dataset.
so, we set decoder distribution variance of the first two layers of the tree to 50, forcing this node to bind to the frame for which the prediction is the most confident. In Fig. 8, we see that this indeed causes the model to bind the top nodes to the frames that is the easiest predict- the semantic bottleneck where the robot arm is just about to drop the object in the bin. We found that the two other nodes find similar described in the Appendix Experimentally,
bottleneck points. Further details of this experiment we found that adaptive binding did not substantially improve quantitative accuracy of the predictions on our datasets, though the ability discover meaningful bottlenecks may itself be useful property in future work, as discussed in the literature on hierarchical RL (Barto & Mahadevan, 2003) or exploration (Goyal et al., 2019).
5.3
We evaluate whether goal-cote conditiod visual prediction is suitable for imitating long-term, goal- directed expert behavior without access to the expert's actions. We first collect dataset of 30k expert
8

---

Under review as a conference paper at ICLR 2020
Real trajectory
Time
Goal Image
fo
plan
foo plan
too plan
100 plan
Figure 9: Illustration of the agent successfully following plans generated by GCP-sequential model through 5 rooms in succession. The agent's position in the map is marked with the red diamond, the goal is marked in green; time progresses from left right; re-planning is performed every 5 steps. We show temporally subsampled plans and executed sequence; the full trajectory has a length of 200 steps.
trajectories with up to 100 frames in the 3x3 Maze environment. For planning and execution follow the procedure detailed in Sec. 4. We train the inverse model that is used to follow the visual plans using 20k trajectories of 15 frames each, collected by taking random actions from random initial positions in the maze.
We quantitatively evaluate the performance of goal-conditioned prediction for visual imitation learning by randomly sampling 100 start and goal positions in the map and measuring the dif- ference between the initial and final distance to the goal after the episode finished, computed as the shortest path distance through the maze. We compare GCP- seq-uequettial to two alternative visual planning methods: visual foresight (Ebert et al., 2018) and Causal InfoGan (CIGAN, Ku- rutach et al. (2018)). We additionally report performance of behavior cloning approach, but note that requires ground truth action an- notations for the demonstration dataset while all other approaches can learn from pure visual demonstrations only.
Figure Bottleneck discovery on pick&place. Four predicted sequences for adaptive GCP-tree are shown, with a node the second layer of the tree marked red. We see that this node reliably identifies and predicts a semantic bottleneck the trajectory, which is the position where the agent is about to drop the object in the bin.
We find that none of the alternative methods for visual planning is able scale long-horizon tasks that require the traversal of multiple rooms. In particular, we observe that CIGAN is unable to produce coherent, long-horizon plans and instead generates smooth interpolations between start and goal image. Consequently, both prior methods fail to complete most of the tasks while GCP- sequential can generate long-horizon, coherent plans (see Fig. 9) and complete substantially more of the test tasks (see Tab. 3). Finally, behavior cloning achieves higher performance but requires access to ground truth demonstration actions at training time which can be very costly or impossible obtain in realistic settings.
DISCUSSION AND FUTURE WORK
We presented conlicondi predictors (GCPs) predictive models that generate video sequences between a given start and goal frame. GCPs must learn to understand the mechanics of the environ- ment that they are trained in, in order to accurately predict the intermediate events that must take

---

Under review as a conference paper at ICLR 2020
place in order to bring about the goal images from the start images. GCP models not only allow for substantially more accurate video prediction than conventional models that are conditioned only on the beginning context, but also allow for novel model architectures. Specifically, we explore how, in addition to more conventional auto-regressive GCPs, we can devise tree-structured GCP models that predict video sequences hierarchically, starting with the coarsest level subgoals and recursively subdividing until a full sequence is produced.
Our experimental results show that GCPs can make more accurate predictions. We also demonstrate that they can be utilized in an imitation learning scenario, where they can learn behaviors from video demonstrations without example actions. Imitation from ob- servations, without actions, is applicable in a wide range of realistic scenarios. For example, robot could learn the mechanics of cook- ing from watching videos on YouTube (Damen al., 2018), and then use this model to learn how to cook on its own. We hope that the imitation framework presented in our work can be a step in towards effectively leveraging such data for robotic control.
Table 3: Control Performance the Maze task
METHOD
SUCCESS
GCP-SEQ
59
CIGAN
10%
VMPC
7%
BC
87%
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy. Fixing broken ELBO. In Proceedings of International Conference on Machine Learning (ICML) 2018.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine Stochastic variational video prediction. ICLR, 2018.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. Proceedings of International Conference on Learning Representations (ICLR), 2015.
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(1-2):41-77, Jamuary 2003, ISSN 0924-6703, doi: 10.1023/A 1022140919877 URL httpt:tts//// htttpp::/doii.rg/1....oo. 1023 A 211/////1112221919997777
Lars Buesing, Theophane Weber, Sébastien Racanière, S. M. Ali Eslami, Danilo Jimenez Rezende, David P. Reichert, Fabio Viola, Frederie Besse, Karol Gregor, Demis Hassabis, and Daan Wierstra. Learning and querying fast generative models for reinforcement learning. arXiv:/802.03006 2018.
Maxime Chevalier-Boisverr gym-miniworld environment for openai gym. https://githu:
com/ma wiweeb///////mcc gym-min worrlddordd 1, 2018.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets.
arXiv preprint
2019.
Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In Proceedings of the 34th International Conference on Machine Leurning- Volume 70, pp. 894-903.
JMLR. org, 2017.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic- epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 720-736, 2018
E. Denton and Fergus. Stochastic video generation with a learned prior. In Proceedings of International Conference on Machine Learning (ICML), 2018.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Vi- sual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv:1812.0056 2018.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Proceedings of Neural Information Processing Systems (NewrIPS), 2016.
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick. Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottle- neck. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of International Conference Learning Representations (ICLR), 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:/502.03 1502.03167. 2015.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7): 1325-1339, 2013.
11

---

Under review as a conference paper at ICLR 2020
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming- Hing--ssann Yang. Erik Learned-Miller and Jan Kautz. Super slomo: High quality estimation multiple intermediate frames for video interpolation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9000-9008, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10100196,,,16 2017.
Lydia E. Kavraki, Petr Svestka, J-C Latombe, and Mark H Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE transactions on Robotics and Automation, 588,, 1996.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference Learning Representations (ICLR), 2015.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. In Advances in Neural Information Processing Systems, pp- 8733-8744, 2018.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. ICML, 2016.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 2018.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 190.00690, 2019.
Ziwei Liu, Raymond A Ych, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4463 4471, 2017.
Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421 2015.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, arXiv preprint arXiv:1511.05444 2015.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In IEEE Conference on Computer Vision and Puttern Recognition, 2017a
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolu- tion. In IEEE International Conference on Computer Vision, 2017b.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. Advances Neural Information Processing Systems, 2015.
Deepak Pathak, Parsa Mahmoudich, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 2050-200 2050,2053, 2018
Xue Bin Peng. Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Reinforce- ment learning of physical skills from videos. In SIGGRAPH Asia 2018 Technical Papers, pp. 178. ACM, 2018.
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv: 141666444, 2014.
12

---

Under review as a conference paper at ICLR 2020
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. arXiv preprint arXiv:1612.06699 2016.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
N. Srivastava, Mansimov, and R. Salakhutdinov. Unsupervised learning of video representations using LSTMs. In Proceedings of International Conference on Machine Learning (ICML), pp. 843-852, 2015.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. Assosiation Computational Linguistics (ACL), 2015.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric challenges. arXiv preprint arXiv:18/2.001711 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lec. Decomposing motion and content for natural video sequence prediction. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V. Le, and Honglak Lee. High fidelity video prediction with large neural nets. In Proceedings of Neural Information Processing Systems (NeurIPS), 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, pp. 613-621, 2016
Ting-Chun Wang. Ming- Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. Proceedings of Neural Information Processing Systems (NeurIPS), 2018.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision. ICML, 2018.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in Neural Information Processing Systems, pp. 91-99, 2016.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
13

---

Under review as a conference paper at ICLR 2020
4
1T
E
E
TrooLSTM
MI P
Treel STM
M
Treel STM
Treel STM
LSTM Core
1
i
Figure 10: Schematic visualization of the model architecture for GCP-Tree (first two layers depicted). Dashed lines represent passing the LSTM state from one iteration to the next, dotted lines depict skip connections (skip connections to the decoder in the first iteration omitted for clarity).
A DATA PROCESSING
For the Human 3.6 dataset, we downsample the original videos to 64 by 64 resolution. We obtain videos of length roughly 800 1600 frames, which we randomly crop in time 500-frame sequences.
We split the Human 3.6 into training, validation and test set by correspondingly 95%, 5% and 5% of the data. On the TAP dataset, we use 48949 videos for training, 200 for validation and 200 for testing.
B ARCHITECTURE
We use a convolutional encoder and decoder similar to the standard DCGAN discriminator and generator architecture respectively. The latent variables 2, as well as e, are 32-dimensional. All hidden layers in the Multi-Layer Perceptron have 32 neurons. We add skip-connections from the encoder activations from the first image to the decoder for all images. For the inference network implemented with attention, we found it beneficial to use 2-layer ID temporal convolutional network that adds temporal context into the latent vectors e, before attention. For the recursive predictor that predicts en, we found it crucial for the stability of the training activate c, with hyperbolic tangent (tanh), and use group normalization Wu He (2018). We observed that without this, the magnitude of activations can explode in the lower levels of the and conjecture that this is due recursive application of the same network. We found that batch normalization loffe & Szegedy (2015) does not work as well as group normalization for the recursive predictor and conjecture that this is due the activation distributions being non-i.i.d. for different levels of the tree. We use batch normalization in the convolutional encoder and decoder, and use local per-image batch statistics at test time.
Hy Hyperparameters. For each method and dataset, we performed a manual sweep of the hyperpa- rameter 3 in the range from 1c-0 to 1e-4. The convolutional encoder and decoder both have five layers. We use the Rectified Adam optimizer (Liu et al., 2019; Kingma Ba. 2015) with 3, = 0.9 and B2 0.999, batch size of 16 for GCP-sequential and 4 for GCP-tree, and learning rate of
14

---

Under review as a conference paper at ICLR 2020
Figure 11: Prior samples from GCP-tree on the Human 3.6M dataset. Each row is different prior sample conditioned on the same information.
2. On each dataset, we trained each network for the same number of epochs on a single high-end NVIDIA GPU.
C ADAPTIVE BINDING WITH DYNAMIC PROGRAMMING
To optimize the model with adaptive binding, we perform variational inference on both and
logp(w) (x (p(a/,, =)] DKz (r((e()))ll)(()) (xrl
To infer quular z), we want to produce a distribution over possible alignments between the tree and the evidence sequence. Moreover, certain alignments, such as the ones that violate the ordering of the sequence are forbidden. We define such distribution over aligment matrices A via Dynamic Time Warping. We define the energy of an alignment matrix as the cost, and the following distribution over alignment matrices:
p(A[2, v) z Avc(z,2)
where = EAle Amc(2,=)], [A(e_Axc(,,)) and is the MSE error between the ground truth frame I, and the decoded frame associated with 2n- We are interested in computing marginal edge distributions w = [A]. Given these, we can compute the reconstruction error efficiently. We next show how to efficiently compute the marginal edge distributions.
Given two sequences Fo.T, Z0:N, denote the partition function of aligning two subsequences To:i, 20:j as Si,j = EAc.oo. e F0)( Cuturi & Blondel (2017) shows that these can be computed efficiently as:
fi, c(),, * (fi-1.j1+ ft-15).
denote the partition function of aligning IgT, 2j:N as bi EAEATT A3, Analogously, we can compute as:
Futhermore,
big cl 2z) (b:41541 + b.41.)).
The total unnormalized density all alignment matrices including the edge (i.j) can be computed as ei fi.j * br./ cCIz ) c(T,, (fr-1.j-1 + fi-1,5) )+ (b++1 + bs+1.). Moreover, the probability of the edge (i, be computed as Wi.j = Gig/
Proposition 1
15

---

Under review as a conference paper at ICLR 2020
s
S 11
S
12
S 111
S 112
S
S
121
122
Prior samples from GCP-tree the 2D maze dataset. Each pair of rows shows two samples given the same context.
Figure 12:
Table 4: Prediction performance on perceptual metrics. Pick&Place data are 80 frames 64x64, H3.6M data are 500 frames 64x64, 3x3 Maze data are 80 frames 32x32, 10x10 Maze data 1000 frames 16x16.
DATASET
PICK&PLACE
HUMAN 3.6M
3x.3 MAZE
10x 10 MAZE
METHOD
FVD
LPIPS
FVD
LPIPS
FVD
LPIPS
FVD
LPIPS
GCP-SEQUENTIAL
328.9
0.02
1541.8
0.06
359.2
0.17
472.7
0.26
GCP-TREE
430.3
0.02
1314.3
0.05
535.0
0.18
473.7
0.26
DVF
2879.9
0.06
1704.6
0.05
470.7
0.28
529.5
0.32
CIGAN
3252.6
0.12
2528.5
0.17
756.0
0.36
486.9
0.33
SVG
713.6
0.01
498.1
0.19
488.5
0.28
Proposition enables us to compute the expected reconstruction cost in quadratic time:
w c(x, y).
16


Reviews:
[{"review_id": "rJxnKiMaKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper reformulates video prediction problem by conditioning the prediction on the start and end (goal) frame. This essentially changes the problem from extrapolation to interpolation which results in higher quality predictions. \n\nThe motivation behind the paper is not clear. First of all, the previous work in video predicted is typically formulated as \"conditioned frame prediction\" where the prediction of the next frame is conditioned on \"a set of context frames\" and there is no reason why this set cannot contain the goal frame. Their implementation, however, is motivated by their application and therefore these models are usually only conditioned on the start frames. Unfortunately, besides the reverse planning in imitation learning, the authors did not provide a suite of applications where such a model can be useful. Hence, I think the authors should answer these two questions to clear up the motivation:\n1. Why conditioning on the goal frame is interesting? It specifically helps to provide more concrete details than getting from Oakland to San Fransico.\n2. Where the current conditional models suffer by conditioning on the goal image?\n\nMore experiments are required to support the claims of the paper as well. \nGiven my point regarding context frames, a more fair experiment would be to compare the proposed method with them when they are conditioned on the goal frame as well. This explicitly has been avoided in 5.1.\n The used metrics are not a good evaluation metric for frame prediction as they both do not give us an objective evaluation in the sense of the semantic quality of predicted frames. The authors should present additional quantitative evaluation to show that the predicted frames contain useful semantic information. FVD and Inception score come to my mind as good candidates. \n\nOn quality of writing, the paper is well written but it can use a figure that demonstrates proposed architecture. The authors provided the code which is always a plus. \n\nIn conclusion, I believe the impact of the paper, in the current form, is marginal at best and for sure does not meet the requirements for a prestigious conference such as ICLR. However, a more clear motivation, a  concrete set of goals and claims, as well as more comprehensive experiments,  can push the quality above the bar. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyedaO2pKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The following work proposes a model for long-range video interpolation -- specifically targetting cases where the intermediate content trajectories may be highly non-linear. This is referred to as goal-conditioned in the paper. They present an autoregressive sequential model, as well as a hierarchical model -- each based on a probabilistic framework. Finally, they demonstrate an application in imitation learning by introducing an additional model that maps pairs of observations (frames) to a distribution over actions that predicts how likely each action will map the first observation to the second. Their imitation learning method is able to successfully solve mazes, given just the start and goal observations.\n\nStrengths:\n-The extension to visual planning/imitation learning was very interesting\n-Explores differences between sequential and hierarchical prediction models\n\nWeaknesses/questions/suggestions:\n-In addition to SSIM and PSNR, one might also want to consider the FVD and LPIPS, both which should correlate better with human perception.\n-How does the inverse model in section $ p(a | o,o')$ account for the case in which multiple actions may eventually result in o -> o', given than o' is sufficiently far from o? Does the random controller need to be implemented in a specific way to handle this?\n-I think a fairly important unstated limitation is that latent-variable based methods tend not to generalize well outside of their trained domain. In table 1, I assume DVF was taken off-the-shelf, but all other methods were trained specifically on H3.6M?\n\n\nLPIPS: https://github.com/richzhang/PerceptualSimilarity\nFVD: https://github.com/google-research/google-research/tree/master/frechet_video_distance\n\n\nOverall, I think the results seem pretty promising -- most notably the imitation learning results. I hope that the authors can address some of my concerns stated above.\n\n\n** Post Rebuttal:\nThe authors have adequately addressed my concerns regarding clarity and metrics. The current draft also better motivates the task of long-range interpolation vs short range interpolation. I maintain my original rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklBPG5AFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nREFERENCES ARE LISTED AT THE END OF THE REVIEW\n\n\nSummary:\nThis paper proposes a method for video prediction that, given a starting and ending image, is able to generate the frame trajectory in between. They propose two variations of their method: A sequential and a tree based methods. The tree-based method enables efficient frame sampling in a hierarchical way. In experiments, they outperform the used baselines in the task of video prediction. Additionally, they used the learned pixel dynamics model and an inverse dynamics model to plan actions for an agent to navigate from a starting frame to an ending frame.\n\n\nPros:\n+ Novel latent method for goal conditioned prediction (sequential and hierarchical)\n+ Really cool experiments on navigation using the predicted frames\n+ Outperforms used baselines\n\nWeaknesses / comments:\n- Missing baseline:\nThe Human 3.6M experiments are missing the baseline from Wichers et al., 2018. I would be good to compare against them for better assessment of the predicted videos.\n\n- Bottleneck discovery experiments (Figure 8):\nThe visualizations shown in Figure 8 are very interesting, however, I would like to see if the model is able to generate multiple trajectories from the same frame. It looks like the starting frames (left) are not the same.\n\n\nConclusion:\nThis paper proposes a novel latent variable method for goal oriented video prediction which is then used to enable an agent to go from point A to point B. I feel this paper brings nice insights useful for the model based reinforcement learning literature where the end goal can be guided by an image rather than predefined rewards. It would be good if the authors can include the suggested video prediction baseline from Wichers et al., 2018 in their quantitative comparisons.\n\n\nReferences:\nNevan Wichers, Ruben Villegas, Dumitru Erhan, Honglak Lee. Hierarchical Long-term Video Prediction without Supervision. In ICML, 2018\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses a video generation setting where both initial and goal state are provided as a basis for long-term prediction. The authors propose two types of models, sequential and hierarchical, and obtain interesting insights into the performance of these two models. Reviewers raised concerns about evaluation metrics, empirical comparisons, and the relationship of the proposed model to prior work.

While many of the initial concerns have been addressed by the authors, reviewers remain concerned about two issues in particular. First, the proposed model is similar to previous approaches with sequential latent variable models, and it is unclear how such existing models would compare if applied in this setting. Second, there are remaining concerns on whether the model may learn degenerate solutions. I quote from the discussion here, as I am not sure this will be visible to authors [about Figure 12]: "now the two examples with two samples they show have the same door in the middle frame which makes me doubt the method learn[s] anything meaningful in terms of the agent walking through the door but just go to the middle of the screen every time."
Decision: Reject