Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
Authors: Wonkwang Lee
Whie Jung
Han Zhang
Ting Chen
Jing Yu Koh
Thomas Huang
Hyungsuk Yoon
Honglak Lee
Seunghoon Hong
Venue: ICLR 2021 Poster (2021)
Keywords: Video prediction
generative model
long-term prediction

Abstract: Learning to predict the long-term future of video frames is notoriously challenging due to the inherent ambiguities in a distant future and dramatic amplification of prediction error over time. Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to a longer future quickly leads to destruction in structure and content. In this work, we revisit the hierarchical models in video prediction. Our method generates future frames by first estimating a sequence of dense semantic structures and subsequently translating the estimated structures to pixels by video-to-video translation model. Despite the simplicity, we show that modeling structures and their dynamics in categorical structure space with stochastic sequential estimator leads to surprisingly successful long-term prediction. We evaluate our method on two challenging video prediction scenarios, \emph{car driving} and \emph{human dancing}, and demonstrate that it can generate complicated scene structures and motions over a very long time horizon (\ie~thousands frames), setting a new standard of video prediction with orders of magnitude longer prediction time than existing approaches. Video results are available at https://1konny.github.io/HVP/.

Published as a conference paper at ICLR 2021
REVISITING HIERARCHICAL APPROACH FOR PERSISTENT LONG-TERM VIDEO PREDICTION
Wonkwang Lee', Whie Jung', Han Zhang?, Ting Chen', Jing Yu Koh², Thomas Huang", Hyungsuk Yoon', Honglak , SSenghhon Hong' KAISTT "Google Research, UUnivesitty of Michigan, *MOLOCO, LG Al Research {wonkwang .ee, whieya, seunghoon. hong) @kaist aactt.ac... ac.kr (zhanghan, iamt ingche,,,, jykoh @google. com thomasehe wwwhhhumihhhhhccmcc edu byungsukemoloccadd..o com hong sanglakkeesss umich.ede
ABSTRACT
Learning to predict the long-term future of video frames is notoriously challeng- ing due to inherent ambiguities in the distant future and dramatic amplifications of prediction error through time. Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to longer future quickly leads to destruc- tion in structure and content. In this work, we revisit hierarchical models in video prediction. Our method predicts future frames by first estimating a sequence of semantic structures and subsequently translating the structures to pixels by video- to-video translation. Despite the simplicity, we show that modeling structures and their dynamics in the discrete semantic structure space with stochastic recurrent estimator leads to surprisingly successful long-term prediction. We evaluate our method on three challenging datasets involving car driving and human dancing, and demonstrate that it can generate complicated scene structures and motions over very long time horizon (i.e., thousands frames). setting new standard of video prediction with orders of magnitude longer prediction time than existing approaches. Full videos and codes are available at https://1 hitygggiuhh....P....
INTRODUCTION
Video prediction aims to generate future frames conditioned on a short video clip. has received much attention in recent years as forecasting the future of visual sequence is critical in improving the planning for model-based reinforcement learning (Finn et al., 2016; Hafner et al., 2019; Ha & Schmidhuber, 2018), forecasting future event (Hoai & Torre, 2013), action (Lan al., 2014), and activity (Lan et al., 2014; Ryoo, 2011). To make it truly beneficial for these applications, video prediction should be capable of forecasting long-term future. Many previous approaches have for- mulated video prediction as a conditional generation task by recursively synthesizing future frames conditioned on the previous frames (Vondrick al., 2016; Tulyakov al., 2018; Denton & Fergus, 2018; Babaeizadeh et al., 2018; Castrejon et al., 2019; Villegas et al., 2019). Despite their success in short-term forecasting, however, none of these approaches have been successful in synthesizing convincing long-term future, due to the challenges in modeling complex dynamics and extrapolating from short sequences to much longer future. As the prediction errors easily accumulate and amplify through time, the quality of the predicted frames quickly degrades over time.
One way to reduce the error propagation is to extrapolate in a low dimensional structure space in- stead of directly estimating livel-level dynamics in video. Therefore, many hierarchical modeling approaches are proposed (Villegas et al., 2017b; Wichers et al., 2018; Liang et al., 2017; Yan et al., 2018; Walker al., 2017; Kim et al., 2019). These approaches first generate sequence using a low-dimensional structure representation, and subsequently generate appearance conditioned on the predicted structures. Hicrarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future. However, previous approaches often employed too specific and incompre- hensive structures such as human body joints (Villegas et al., 2017b; Yan et al., 2018; Yang et al., 2018; Walker et al., 2017; Kim et al., 2019) or face landmarks (Yan et al., 2018; Yang et al., 2018).
"The author contributed to this work while Google Research.

---

Published as a conference paper at ICLR 2021
3. Patuze Label Maps
Future
Frames
Sc-111
Sc-12
Context Frames
AC
Scy
X1C
RC++
Gatruct
Gimage
Xx-1
Segmentation
Semantic Label Maps
8p
Gatruct
Gimage
Xy
Structare 2.1)
Image
The overall framework of the proposed hierarchical approach. Given the context frames and label maps extracted by the segmentation network, model predicts the future frames by timating the semantic label maps using a stochastic sequence estimator (Section 2.1) and converting the predicted labels RGB frames by using conditional image sequence generator (Section 2.2).
Figure 1:
Moreover, they made oversimplified assumptions of the future by using deterministic loss or as- suming homogeneous content. We therefore argue that the benefit of hierarchical models has been underestimated and their impact on long-term video prediction has not been properly demonstrated.
In this paper, we propose hierarchical model with a general structure representation (i.e., dense semantic label map) for long-term video prediction complex scenes. We abstract the scene as categorical semantic labels for each pixel, and predict the motion and content change in this label space using variational sequence model. Given the context frames and the predicted label maps, generate the textures by translating the sequence of label maps to the RGB frames. As dense label maps generic and universal, we can learn comprehensive scene dynamics from object motion to even dramatic scene change. We can also capture the multi-modal dynamics in the label space with the stochastic prediction of the variational sequence model. Our experiments demonstrate that we can generate a surprisingly long-term future of videos, from driving scenes to human dancing, including the complex motion of multiple objects and even evolution of the content in distant future. We also show that predicted frame quality is preserved through time, which enables persistent future prediction virtually mear-infinite time horizon. For scalable evaluation of long-term prediction at this scale, we also propose novel metric called shot-wise FVD, which enables the evaluation of spatio-temporal prediction quality without ground-truie truths and consistent with human perception.
METHOD
x our goal is to synthesize the future frames Xe+ir = {x0+1> XC42, XT} up to an arbitrary long-term future T'. Let S; E & xXW denote dense label map of the frame X; defined over N categories, which is inferred by the pre-trained semantic segmentation model'. Then given the context frames X and the label maps $ic, our hierarchical framework synthesizes the future frames ce ir by the following steps.
Given the context frames X,.c {x1
A structure generator takes the context label maps as inputs and produces a sequence of the future label maps by $c-1. Gs,uass,.c, 2, where z, denotes the latent variable encoding the stochasticity of the structure.
Given the context frames and the predicted structures, an image generator produces RGB frames by feEuT
Gimage (X.:;;
Figure I illustrates the overall pipeline. Note that there are various factors beyond motions that make spatio-temporal variations in the label maps, such as an emergence of new objects, partial observability, or even dramatic scene changes by the global camera motion (e.g., panning). By learning to model these dynamics in the semantic labels with the stochastic sequence estimator and conditioning the video generation with the estimated labels, the proposed hierarchical model can synthesize convincing frames into the very long-term future. Below, we describe each component.
We employ the pre-trained network (7hu et al., 2019) on still images obtain segmentations in videos.
2

---

Published as a conference paper at ICLR 2021
2.1 SEQUENTIAL DENSE STRUCTURE GENERATOR
Our structure generator models dynamies in label maps by p(ser (ggr) ) employ sequence model proposed by Denton & Fergus (2018) since it (1) provides probabilistic framework to handle stochasticity in structures and (2) can easily incorporate discrete sequences. Specifically, we optimize 3-VAE objective (Higgins et al., 2017; Denton & Fergus, 2018) below:
- p( zg.). We
Σ B Beolegriegr) poo Sc.) -
where qs (Zgr(ssr)) is the approximated posterior distribution and Po (Ze(s.c) is the prior distribu- tion. At each step, it learns to represent and reconstruct each label map s through CNN-based encoder fene and decoder flc,, respectively, while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows:
( = L.SM, (h,)
where h, = - (8t);
Zt ~ (N) (()
(2)
g: = ( = (g)))
where h f (sc-
where LSTM, and LSTMs respectively approximate the generative and the posterior distributions recurrently up the time step f. Unlike Denton & Fergus (2018); Villegas al. (2019) that exploit skip-connection from the last observed context frame during both training and testing time, we skip hidden representations of the encoder to the decoder at every time step during testing handle long-term dynamics in structure as in (Villegas et al., 201 7a; Finn et al., 2016).
During training, we apply teacher-forcing by feeding ground-trutt label maps and sampling the la- tent from the posterior distribution NV (t), (t)) = LSTM, (s:). During inference, we recursively generate label maps by (1) sampling z, from the prior distribution A (uo(t), ot) (f
(2) producing the frame 6; through the decoder, and (3) discretizing the pre- dicted label map by taking pixel-wis wise maximum. Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints (Ville- gas et al., 2017b; Yan et al., 2018; Yang et al., 2018) or optical flow (Walker et al., 2017).
Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available, we can optionally extend the structure generator to jointly predict the object boundary maps. Such structure can add a notion of object instance, and is useful to improve the denotes the object
image generator in sequences with many occluding objects. Let e; E (0. 1) boundary map at frame x,. Then we train the conditional generator é; = Gedye that produces boundary map given the label map for each frame'. We train Gedge using the conditional GAN objective to match the generated distribution to the joint distribution of the real label and real bound- ary maps pl (S ez). The boundary and label maps are then combined as the output of the structure generator S = [š,, @.]. and are used as inputs to the image generator described below.
[ HWW
2.2 TTRUCTURE-CONDTTIONN CONDITIONAL PIXEL SEQUENCE GENERATOR
Given a sequence of the structures and the context frames, the image generator learns model the conditional distribution of the RGB frames by p(xgr(sgr) = II P[== p(x: (g.,, We formulate this task video-to-video translation problem, and employ ofate-of-the-arr conditional video generation model (Wang et al., 2018). Specifically, the video synthesis network F in Wang et al. (2018) consists of three main components; the generator H, occlusion mask predictor M, and optical flow estimator W, which are combined to generate each frame X2 by the following operation:
x 11 = (1 m:) O W + m, O h;
where w,-1 (( s.)..) is the warped previous frame x;- using the estimated optical flow, h = ( soft occlusion mask blending w,-1 and h, Unlike models synthesizing future frames by transform- ing the context frames xc (Villegas et al., 2019; 2017b; Yan et al., 2018: Tulyakov et al., 201 8), this
- S -..)) is the hallucinated frame at t, and m, = (x S, .,..... is the
We do not feed the predicted boundary map input to the structure generator since it makes the structure generator prune error propagation thus prevents extrapolation long-term future.
3

---

Published as a conference paper at ICLR 2021
model is appropriate synthesize the long-term future since it can handle both transformation of the existing objects (via We-1) and synthesis of the emerging objects (via h2).
To ensure both frame-level and vidco-level generation quality, the video synthesis network F is
trained against a conditional image discriminator Dj and a conditional video discriminator Dy through adversarial learning by
(F D1) = (xr(xeraer)) Dog (r(x;, (xr(eer (lg (1 D1(&,, s:))l, (v(F, E [log DV(Xx-1n-1) Wi-
(4)
E [log (1 (x(x--n-r") W
(5)
where Lr(F, and (F. Dv) are frame-level and video-level adversarial losses, respectively.
For efficient training. we follow Wang et al. (2018) to adopt sampling operators di (x Sgr) (x) and Φv Wer,,Kgr,S0 8gr) = (W-zz-r'' - x.
" for frame and video-level ad-
versarial learning objectives, respectively, where i an integer sampled from U(1, T) and U(r's +
1, + 1) for frame sampling operator Φi and video sampling operator Φv, respectively. Then the final learning objective is formulated by
min max Li(F, + (v). Dr.Dv
(6)
During training, the model is trained estimate RGB frames from the GT structures, while the predictions from the structure generator are used during testing. See Appendix C.3 for more details.
3 RELATED WORK
The task of predicting future video frames has been extensively studied over the past few years (Vil- legas et al., 2019; Denton & Fergus, 2018 8; Lee et al., 2018; Babacizadeh et al., 2018; Vondrick et al., 2016; Finn et al., 2016; Clark et al., 2019; Castrejon et al., 2019). Early approaches focus on model- ing simple and deterministic dynamics using regression loss (Srivastava et al., 2015; Ranzato et al., 2014) predictive coding (Lotter et al., 2017). However, such deterministic models may not be appropriate for modeling stochastic variations in real-world videos. Recently, deep generative mod- els have been employed model dynamies in complex videos. Babaeizadeh al. (2018) proposed a variational approach for modeling stochasticity in a sequence. Denton & Fergus (2018) incorpo- rated more flexible frame-wise inference models. The prediction quality of the variational models has been further improved by employing rich structure in latent variables (Castrejon et al., 2019), maximally increasing the network parameters (Villegas et al., 2019), or incorporating adversarial loss (Vondrick et al., 2016; Clark et al., 2019; Tulyakov et al., 2018; Villegas et al., 20) Despite their success, these approaches still fail to generate long-term videos and the common artifacts were losing object structures, switching object categories, etc. Our approach addresses these issues by explicitly predicting dynamics in the low-dimensional label map using the hierarchical model.
Hicrarchical models studied in the past for video prediction were usually in specific video do- mains (Villegas et al., 2017b; Wichers et al., 2018; Liang et al., 2017: Yan et al., 2018; Walker et al., 2017; Minderer al., 2019). Villegas et al. (2017b) employed LSTMs to predict human body joints and visual analogy making to create textures, which is extended by Wichers et al. (2018) and Min- derer et al. (2019) unsupervised approaches. Other approaches employed sequential VAE (Yan et al., 2018) or GAN (Yang et al., 2018) to predict the human body posture. However, these ap- proaches are designed specifically for certain objects, and evaluated under simple videos containing only single moving object. Also, they mostly utilized deterministic models to learn structure-level dynamics. These simplified assumptions on video content and dynamics limit their application to real-world videos. We propose to resolve these limitations by using dense semantic label maps as universal representation and stochastic sequential estimator for modeling video dynamics.
4 EXPERIMENTS
4.1 EVALUATION METRICS
Below we describe the evaluation metrics used in the experiment. Unless otherwise specified, we use 64 x 64 images for all quantitative evaluation for fair comparison to the existing works.
Short-term prediction We employ three conventional metrics in the literature to evaluate the prediction performance short-term videos (i.e., less than 50 frames): VGG cosine similarity (CSIM) measuring the frame-wise perceptual similarity using VGG features, mean Intersection- over-Union (mloU) measuring the structure-level similarity using the label maps extracted by the
4

---

Published as a conference paper at ICLR 2021
pre-trained segmentation network (Zhu et al., 2019) and Fréchet Video Distance (FVD) (Unterthiner et al., 2018) measuring a Fréchet distance between the ground-truth videos and the generated ones in video representation space. Detailed evaluation protocols are described in Appendix A.
Long-term prediction Compared to short-term prediction, evaluating prediction quality of arbi- trary long-term video is challenging for number of reasons. First, the ground-truth videos are seldom available at this scale, making it impossible to adopt metrics based on trame-wise compari- son (e.g. CSIM and mloU). Second, the uncertainty of future prediction increases exponentially with time, making it intractable to employ density- bassed metrics (e.g... FVD) as it requires exponentially many samples. Since our goal is demonstrate video prediction the scale of hundreds thou- sands of frames, we introduce a novel metric based on FVD, which enables evaluation of temporal and frame-level synthesis quality without ground-trute sequences and allows tractable evaluation through time. Specifically, we introduce shot-level video-quality evaluation metric as
shot-wise FVD(L) FVD (X::)(-1+111
which computes a FVD between the ground-truth shots X = -X_,.... , _ dictions X X {X - X.M -- in sliding window manner through time, where I, denotes the total number of overlapping shots in the training video and M denotes the number of predicted shots. The shot FVI) evaluates the synthesis quality in short interval defined by w through time. We show in the experiment that it is indeed aligned well with human perception. We also compute Inception Score (Salimans et al., 2016) for frame-level quality evaluation, which does not require ground- truths thus is appropriate for long-term evaluation. Please find Appendix A for more details.
and the shots of pre-
User study We conduct an user study on Amazon Mechanical Turk (AMT). To evaluate the quality different time scales, we present 5-second (50 frames) videos extracted at 1, 250, 400th predicted frames for all methods and counted their chosen ratio as the best. More details are in Appendix A.
4.2 RESULTS ON HUMAN DANCING SEQUENCES
We first present our results on human dancing videos collected from the internet (Wang et al., 2018).
Baselines We compare our method with two state-of-the-art video prediction models. SVG- extend (Villegas et al., 2019) is directly operating on RGB frames via a stochastic estimator and serves as our baseline for minhheearchical model. Following the paper, we employ the largest model with maximum parameters for fair comparison. We employ Villegas et al. (2017b) as a hier- archical model designed specifically for long-term prediction of human motion using pose (Newell et al., 2016). All models are trained to predict 40 future frames given 5 context frames.
Short-term prediction results We evaluate the short-term prediction performance by comparing the ground- -tuths with synthesized 50 future frames given 5 context frames. Table 1 summarizes the quantitative evaluation results (see Figure A for qualitative results). Even in short prediction interval, our method generates substantially higher quality samples than the baselines terms of modeling appearance (CSIM), structure (mloU), and motion (FVD). This is because danc- ing sequences contain rapid and complex variations in dynamics, making prediction task particularly challenging. We discuss more detailed amalysis across methods in long-term prediction task below.
Table I: short-term prediction results.
Quantitative comparisons of
Model
CSIM
mloU(t:
PVD(
SVCi-xtend
0.6654
0.0519
2125.20
Villegas
al.
0,7637
0,1555
1987,55
Ours
0.8164
0,3454
1398.98
Long-term prediction results We evaluate 500 frame prediction results based on shot-wise FVD, frame-wise Inception score, and human evaluation. Figure 2 and 3 summarize the quantitative and qualitative comparisons, respectively. We notice that the SVG-extend fails to model complex dy- namics in dancing even early in prediction. This is because the dancing motions involve fast transi- tion and frequent self-occlusions, resulting in catastrophic error propagation through time. On the other hand, as shown in the Inception score, Villegas et al. (2017b) produces much reasonable future frames as it exploits human body joints for generation. However, the deterministic LSTM module is not strong enough to capture complex dancing motions, and ends up generating static postures in the long term that leads to very high FVD scores. In contrast, we observe that our method gen- erates both realistic frames and convincing motions, leading to stable Inception and FVD scores in the long-term future. The human evaluation (Figure 2(c)) also shows the consistent results that method outperforms all methods from the short- to long-term prediction. When method is
5

---

Published as a conference paper at ICLR 2021
RVG-nctend Vileggx
SWTSebbend
Villegss
Ourx
30:00
Inceptton score
shot-wise PVD
Model
[=1
1=250
1=400
SVG-Extend
3.9
2.3
3.1
2.
2600
Villegas et al.
6.6
9.9
9.1
1000
Oure
89.5
87.8
100
100
200
500
Lize
Lime
(8) Inception score
(6) Shot-wise FVD
(c) Human ovaluation (mosst-pefferee ratio)
Figure 2: Quantitative comparisons of the long-term prediction on human dancing sequences.
t-I-25
tw251-275
tw401-425
SVG- extend
A
A
8
Villegas al.
Ouzs
Figure 3: Qualitative comparisons of long-term video generation results across models on human dancing sequences. All models are conditioned on the same context frames. Click the image to play the video in browser. More results available at httpps//Ikonny htt.:://ooonyggggbbo//HHHHH github.io/HVP/..
applicd to predict up to 2040 frames 128 x 128 resolution (see Figure C in the appendix), we ob- serve that it generates future frames that are convincing and involve diverse dance motions without noticeable quality degradation through time. In addition to the synthesis quality, Figure B illustrates that method generate diverse and interesting motions with the stochastic structure estimator.
4.3 RESULTS ON KITTI BENCHMARK
Baselines In addition SVG-extend, we employ the future segmentation model (Bhattacharyya et al., 2019) (Bayes-WD-SL that predicts future dense label maps as a strong baseline for hierarchi- cal model. Since generates only the label maps, employ the same image generator with ours to produce RGB frames from the predicted label maps. All models are trained to predict 15 future frames given context frames. Please refer the appendix for the detailed architecture settings.
Short-term prediction results We evaluate the short-term prediction quality by synthesizing 50 frames given context frames. Table 2 summarizes the quantitative results (see Fig- for qualitative analysis). We observe that SVG-extend performs worse in structural accuracy (mloU) and motion (FVD), as it loses the object structure and generates arbi- trary pixel dynamics. The Bayes-WD-SSI performs better in mloU, as it conditions the frame prediction with the structure estimation. However, as shown high FVD score, it fails to model temporal variations in the structure, largely due to its feedforward chitecture. This unstructured motion leads to disastrous failure in a long-term prediction, as we will discuss later. In contrary, substantially outperforms the others in all metrics.
Table 2: Quantitative comparisons of
short-term prediction results.
Model
CSIMif)
mloo(())
PVD())
SVG-estend
0.6664
0.3529
1448.84
Beyes-WD-SL.
0.6333
0.4225
956.0
Ours
0.5137
762.73
method generates semantically accurate motions and structures, and
Long-term prediction results Figure 4 and 5 summarize the quantitative and qualitative com- parisons, respectively. In all metrics, our method outperforms the other baselines with substantial margins, showing that our method synthesizes both high-quality frames and motion. We notice that SVG-extend simulates arbitrary pixel motions, resulting in relatively constant Inception and shot- wise FVD scores through time. However, these unstructured motions lead rapid destruction in

---

Published as a conference paper at ICLR 2021
inception score
3.0
shot FVD
3000
Model
t-1
1-250
t-400
SVG-Extend
27.2
2.
2000
Bayes- WD-SL
23.0
6.6
6.8
Ours 63.9
70.5
66.0
1.8
500
100
2013
500
time
(0) Inception score
(b) Shot-wise FVD
(c) Human evaluation (most-prefered ratio)
Figure 4: Quantitative comparisons of the long-term generation quality on KITTI Benchmark.
-1
290
401
440
SVC- extend
Ours Bayes-WyD WD---
Figure 5: Qualitative comparisons of long-term video generation results across models. Although all models succeed in generating plausible frames in a short-term (t 40), only our approach can generate persistent and convincing futures even in the end (1 251 ~ 500). Click the image to play the video browser. More results are available at https://lkonny. wwwryygittuu.ooooo ioHVV/
Context
1-501-516
t-1001-1016
1-1001-1516
1-2001-2016
L-2501-2516 2516
Figure 6: Long- term-term prediction results high- nigh-resolution KITTI sequence. Both the frames and the label maps are predicted by our method. Our method can generate high-resolution frames (256 x 256 pixels) into the long-term future without particular quality degradation. Click the image to play the video in a browser. More results are available at https://1 konny githh..itth....//P...
recognizable concepts the synthesized frames (Figure 5). The hierarchical model (Bayes- WD- SL) generates more convincing frames in a short-term as shown in lower shot-wise FVD, but fails extrapolate in long-term. Importantly, such trends in shot-wiss FVD (Figure 4(b)) aligns well with human evaluation results (Figure 4(c)), showing that it is appropriate metric for evaluation of long-term prediction. Interestingly, we observe that the Inception Score of our method increases through time. It is because the long-term prediction by our method often leads to scenes with simple and typical structures, such as a highway, where the image generator can produce more high-quality frames than complicated scenes appearing early in the test videos. Note that such transition of the scene is still reasonable as it is frequently observed in the training data. Nonetheless, we observe that our method generates reasonable sequences through time while maintaining its quality in rea- sonable range even in a distant future. Finally, Figure 6 illustrates the prediction results over 2000 future frames. We observe that the generated sequences are reasonable in both structure and motion, and capture interesting translations of the scenes through time (e.g., from suburban to rural areas).

---

Published as a conference paper at ICLR 2021
Table Comparison future segmentation methods on Cityscapes dataset.
person
rider
track
train
motoreycle
bicycle
mīoU
S28 (Lac al., 2017)
0.37
0.18
0.70
0.43
0.55
0.26
0.27
0.38
0.39
F2F (Luc al., 2018)
0.35
0.20
0.72
0.53
0.58
0.38
0.30
0.25
0.41
Ours
0.41
0.28
0.73
0.49
Last Observed
Predicted Frames
Frame
11th
71th
128th
185th
242th
299th
GG
Figure 7: Long-term prediction results 256 x 512 Cityscapes dataset. Click the image to play the video browser. For more qualitative results and details, please refer to Figure H and Figure in the appendix and the project website: https://1 konny. www.gt.tthu.h.......
4.4 RESULTS ON CITYSCAPES DATASET
To further evaluate the quality of structure prediction, compare structure generator with existing future segmentation methods that directly predict the segmentation map of the future frames.
Baselines We compare our method with S2S (Luc et al., 2017) and F2F (Luc et al., 2018), which the state-of-the-ard in the future segmentation literature. S2S is deterministic model based on fully-convolutionnll network that predicts future semantic label maps in a multi-scale and auto- regressive manner. F2F is extension of S2S to future instance segmentation task by predicting the high-level feature maps of Mask R-CNN (He et al., 2017). Similar to KITTI, we use the label maps extracted by the pre-trained semantic segmentation network (Zhu al., 2019) for the training.
We follow the standard evaluation protocols in the literature (Luc al., 2017: 2018) for quantitative and qualitative comparisons. For each validation sequence, each model is provided with 4 contexts, and produces up 29th frame. Then, we compute mloUs between ground-truths and predictions at 20th time-step. For fair comparisons with the future instance segmentation model (i.e., F2F), we follow Luc al. (2018) and measure mloUs only for moving objects. For S2S and F2F, we use publicly available tre-trained models provided by the authors.
Evaluation
Results Table presents the quantitative evaluation results (See Figure H for qualitative results). The two baselines produce blurry predictions, resulting inaccurate structures and mislabelings. These problems have been widely observed in the literature and are attributed to the inability to handle stochasticity (Denton & Fergus, 2018). Our method outperforms the two baselines classes it can handle highly stochastic nature of complex driving scenes. Figure and Figure J illustrates the long-ten terg-termmp prediction results our full model including the boundary map and image generator on 256 x 512 resolution. It shows that method can generate convincing future even in extremely complex and high-resolution videos. See Appendix B.3 for more detailed discussion.
5
ABLATION STUDY
As demonstrated in the previous sections, the proposed hierarchical model succeeds at predicting frames persistently up to an arbitrary long-term future. To identify the source of this persistency, we conduct an ablation study. Specifically, we note that the model from Villegas et al. (2019), which suffers from dramatic error propagation through time, performs surprisingly well when applied to predicting semantic structures. Therefore, we focus on the different choices of structure generator and its hierarchical composition. We follow Villegas et al. (2019)and ablate effects of (1) stochastic estimation and (2) recurrent estimation. In addition, we investigate benefits of (3) discretization of predicted label maps and (4) our hierarchical model design that first predicts the structures and
8

---

Published as a conference paper at ICLR 2021
CSN C37M
CR
D9G
Gata
inception score
50%
shot-wise FVD
3.5
2750
2.5
1250
14
188
282
376
470
94
188
262
376
470
time
time
Figure 8: Quantitative comparisons of long-term prediction performances among ablated models.
motions via structure generator and then translates them into pixel-level RGB frames. The latter can naturally prevent the pixel-level errors from propagating through the structure predictions.
To this end, we construct four ablated baselines from our structure generator as follows: (1) LSTMM where stochastic estimation modules are removed, (2) CNN where stochastic and recurrent estimation modules are removed, (3) SVG where the discretization process is removed and the model is modified to observe and estimate continuous logits the semantic segmentation model, and (4) SVGzgiprop where an output of the structure generator at time & is translated to RGB frame by the image generator and then further processed by the semantic segmentation model before used as an imput time + 1. We train all the models in KITTI Benchmark on the 64 x 64 resolution and compare them using the inception score and shot-wise FVD. All baselines share the same image generator (Section 2.2) translate the predicted structures to RGB sequence. Figure summarizes the quantitative results. For qualitative comparisons, please refer to Figure G in the appendix.
Stochastic estimation. As shown in the quantitative comparison results, deterministic baselines (CNN and LSTM) suffer from drastic degradation of shot-wise FVD through time compared to the stochastic ones. This is because the deterministic models tend to seek the most likely future and thus are prone to bad local minima, which are very difficult to be recovered from (e.g. when trapped in a loop and recursively generates the same structures and motions shown in Figure G). It shows that the stochastic estimation is critical especially in a very long-term prediction.
Recurrent estimation. Thanks to the ability to capture temporal dependencies among time steps, we find that the recurrent model (LSTM) is much beneficial than the feedforward counterpart (CNN) in (1) predicting more plausible sequences in short to mid-term (shot-wise FVD at 200) and (2) generating more recognizable scenes (inception score).
Discretization. We observe that predicting the structure in the discrete space is very critical in a persistent prediction. To validate this, we compare our method with the baselines (1) unrolling in RGB space (SVGRça) and (2) unrolling in structure but using soft class logits without discretization (SVGaoft). As shown in Figure G, both baselines perform much worse than our method. Sur- prisingly, the behaviors of both baselines are very similar in both measures, although they unroll in different spaces (RGB and soft class logit). It shows that unrolling in discrete space is one of the keys for persistent prediction, as it prevents error propagation and eliminates complex temporal variations except the ones caused by structure and motion.
Decomposed hierarchical prediction. Comparison to SVGegbprop shows that errors induced from pixel-level predictions gradually add up through time and have a significant impact predicting plausible structures and motions. It shows that our architecture having the independent unrolling loop of the structure and image generator important for persistent prediction.
CONCLUSION
We proposed a hierarchical approach for persistent video prediction. We revisit the hierarchical model with stochastic estimation of dense label maps and integration of image generator robust against temporal mis-prediction in pixels and structures. Our experimental results show that care- fully designed hicrarchical model can learn synthesize a very long-term future with convincing structures and dynamics even in complex videos. By scaling up the video prediction to order-of- magnitudes longer, we believe that our work can help the future research focus on more challeng- ing problems of learning a long-term dependency and eliminating explicit structure estimation.
Acknowledgment This work was supported in part by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (2020-0- 00153 and 2016-0-00464), Samsung Flectronics, and NSF CARRER under Grant 1453651.
by

---

Published as a conference paper at ICLR 2021
REFERENCES
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, 2018.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018.
Apratim Bhattacharyya, Mario Fritz. and Bernt Schiele. Bayesian prediction future street scenes using synthetic likelihoods. In ICLR, 2019.
Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrnns for video predic- tion. In ICCV, 2019.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets. arXiv preprint arXiv: (001:00907006777 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. ICML, 2018.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NewrIPS. 2016.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurlPS. 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixcls. In ICML, 2019.
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In ICCV, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Minh Hoai and Fernando Torre. Max-margin early event detectors. IJCV, 2013.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Yunji Kim, Seonghyeon Nam, In Cho, and Seon Joo Kim. Unsupervised keypoint learning for guiding class-conditional video prediction. In NeurIPS. 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS. 2012.
Tian Lan, Tsung- Chung Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In ECCV, 2014.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: / 2018.
10

---

Published as a conference paper at ICLR 2021
Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P. Xing. Dual motion gan for future-flow embedded video prediction. In JCCV. 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning. In ICLR, 2017.
Pauline Luc, Natalia Neverova, Camille Couprie, Jacob Verbeek, and Yann LeCun. Predicting deeper into the future semantic segmentation. In ICCV, 2017.
Pauline Luc, Camille Couprie, Yann LeCun, and Jakob Verbeek. Predicting future instance segmen- tation by forecasting convolutional features. In ECCV. 2018.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. In NewrIPS, 2019.
Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Struc- ture guided image inpainting using edge prediction. ICCV, 2019.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estima- tion. In ECCV, 2016.
Mare' Aurelio Ranzato, Arthur Szlam, Joan Bruna, Michaël Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: baseline for generative models of natural videos. arXiv preprint 1eriiv:1412.666 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJC,, 2015.
M. S. Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In ICCV, 2011.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yeung, Wai-kin Wong, and Wang-chun Woo. Convolutional istm network: A machine learning approach for precipitation nowcasting. In NeurIPS, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In ICMI, 2015.
C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In CVPR, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric challenges. arXiv preprint arXiv:1812.00711 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In ICLR. 2017a.
Ruben Villegas, Jímei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to Generate Long-term Future via Hierarchical Prediction. In ICML. 2017b.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In NeurIPS. 2019.

---

Published as a conference paper at ICLR 2021
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurlPS. 2016.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video fore- casting by generating pose futures. In ICCV, 2017.
Ting- Bong-Chun Wang, Ming-Yir Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. In NewrlPS, 2018.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lec. Hierarchical long-term video prediction without supervision. In ICML. 2018
Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap. Ersin Yumer, and Honglak Lee. Mt-vae: Learning motion transformations to generate multimodal human dynamics. In ECCV, 2018.
Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation. In ECCV, 2018.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TPAMI, 2017.
Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih. Shawn D. Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A EVALUATION PROTOCOLS
Short-term prediction. Following the previous works, we evaluate the short-term prediction qual- ity by comparing the ground-truth and predicted frames in test videos. We employ three metries: VGG cosine similarity (CSIM), mean Intersection-over-Uniin (mloU), and Fréchet Video Distance (FVD) (Unterthiner et al., 2018). The first metric measures the frame-wise perceptual similarity using VGG features, while the second one measures the structure-level similarity using the label maps extracted by the pre-trained semantic segmentation network (Zhu et al., 2019). To handle stochasticity, we sample 100 predictions for each test clip and report the scores of the best scoring predictions. For the video-level evaluation, we adopt FVD that measures a Fréchet distance between the ground-truth videos and the generated ones in video representation space. We an Inflated 3D Convnet (I3D) (Carreira & Zisserman, 2017) network pre-trained on Kinetics 400 dataset (Car- reira & Zisserman, 2017) obtain representations on videos, and use all 100 predictions compute FVD against ground-trus sequences.
Long-term prediction. To evaluate the shot-wise FVD, employ the I3D Convnet as the short- term prediction. For the models trained on KITTI dataset, we use a ResNet-18 (He et al., 2016) pre-trained Places 365 (Zhou al., 2017) to compute the Inception Score. For the models trained on the Human Dancing dataset, we use the Inception V3 (Szegedy et al., 2015) pre-trained on ImageNet (Russakovsky et al., 2015) for measuring the Inception Score.
Human evaluation. We conducted a human evaluation using Amazon Mechanical Turk (AMT). For evaluating both the short- and long-term prediction performances, we generated 500 predicted frames for all compared methods (SVG-Extend, Villegas al. (2019), and ours). We extracted 5- second clips (50 frames encoded in 10 fps) at the 1st, 250th, and 400th frames, and asked users to rank the videos according to perceived quality and realism of the videos.
On the KITTI dataset, we conducted the evaluation over the 133 unique sequences present in the validation set. On the Human Dancing dataset, we conducted evaluation over 97 videos used for validation. Responses for each video was aggregated from 5 independent human evaluators for each clip.
B MORE DETAILS OF EXPERIMENT RESULTS
This section provides additional experiment details and results that could not be accommodated in the main paper due to space restriction. Please find our website to assess full videos for qualitative analysis: https:// httess//koony..ce htth://kony.gthuub//HWWW.. gtthb.
B. HUMAN DANCING DATASET
B.1.1 DATASET
To evaluate if our model can learn complex and highly structured motion, we used videos of human dancing". We construct this dataset by crawling set of videos from the web containing a single person covering various dance moves. Following the pre-processing used in Wang al. (2018), we crop the center of the video and resize it to square frames. We collect approximately 240 videos in total for training. For structure representation, we use the body parts segmentation obtained by DensePose (Alp Güler et al., 2018). We sub-sampp sub-samplc each sequence by factor of two.
B.1.2 IMPLEMENTATION DETAILS
SVG-extend (Villegas et al., 2019). We use the same architectural and hyperparameter settings described in Section C.1 except that SVG-extend in this case is trained to model pixel-level RGB
hhtps:/wwwwiinsgggr...mmmmiiiiia We obtained the vidcos under the permission of the creator/owner.
13

---

Published as a conference paper at ICLR 2021
sequences directly unsupervised non non-hierarchical baseline, unlike ours that works semantic segmentation sequences as sequential dense structure generator.
Villegas et al. (2017b). This method also adopts hierarchical approach that leverages human key- points as additional supervision for predicting future video frames. Specifically, this method consists of five modules, namely (1) the Hourglass network (Newell al., 2016) that estimates human key- points from pixel-level images, (2) a single layer deterministic encoder-decoder LSTM with 1024 hidden units and tanh activations that predicts sequence of future human keypoints given a few observed ones, (3) an image generator that synthesizes pixel-level image frames given the predicted and observed keypoints based on visual-analogy scheme, (4) an image discriminator for adversarial learning, and (5) the AlexNet (Krizhevsky et al., 2012) pre- rre-traieed on ImageNet (Russakovsky et al., 2015) that is used for perceptual loss (Johnson et al., 2016). For training this model on our human dancing dataset, we directly follow the setting used in the paper (Villegas et al., 201 7h), and use the publicly available code provided by the authors.
B.1.3 QUALITATIVE RESULTS
Short-ferm prediction. We show in Figure A the qualitative comparisons of short-term predic- tions across models, which corresponds to the Table 1 in the main paper. As shown in the figure, SVG-extend starts to fail in producing plausible samples be attributed to its implicit modeling of spatio-temporal variations of structures and appearances. Villegas et (2017b) improves the quality of both structures and appearances as well as extrapo- lation through time via the explicit modeling of structures followed by the translation of predicted structures into pixel-level frames in a hierarchical manner. However, still fails to produce realis- tic pixel-level frames due the adoption of (1) coarse- coarse-grained human landmarks and deterministic model which are insufficient to fully describe the fine details of stochastic and structured motions, and (2) image synthesis performed by transforming the last observed frames into the future, which may not be appropriate when there exist large appearance changes due to self-occlusion and move- ment. On the other hand, approach can produce high-fidelity predictions with plausible motions thanks to the stochastic and comprehensive estimation of scene dynamics under the semantic seg- mentation space.
very short-term (t 8 ~), which can
Diversity of samples. Figure B illustrates the multiple prediction results conditioned on the same context frames. We observe that our model produces diverse and plausible future frames, where such stochasticity mainly stems from the structure generator.
Long-term prediction results. Figure C illustrates the prediction results on a very long-term fu- ture (over 2000 frames). We observe that our method generates convincing frames without particular quality degradation and error propagation over very long-term future. We also observe that the predicted frames contain interesting dance motions, such as the ones with horizontal translation (t = 1000 ~ 1040), dynamics on legs (t 500 ~ 540) and arms (t = 2000 ~ 2040), etc.
14

---

Published as a conference paper at ICLR 2021
t=1
t=8
t=15
t=22
t=29
t=35
t=43
t=50
Ground-Truth Truth
SVG-Extend
Villegas et al.
Ours
Ground-Truth
SVG-Extend
Villegas et al.
Ours
Figure A: Qualitative comparisons of frame-wise video prediction results across models: (top pane) given the same context frames, we sample 100 predictions for each model and show their best predictions terms of the cosine similarity metric. (bottom pane) for each best prediction, we show the semantic label maps predicted by Densepose (Alp Güler et al., 2018).
15

---

Published as a conference paper at ICLR 2021
t=6
t=12
t=18
t=48
t=54
t=60
Ground-Truth
Sample
Sample
Sample 3
Ground-Truth
Sample
Sample
Sample
Figure B: Diverse video prediction results on 128 x 128 dance sequence. All samples are condi- tioned on the same context frames.
16

---

Published as a conference paper at ICLR 2021
Context
t=500-545
--100-1040
- 1510001134
t=2000-2044
Figure C: Long-term prediction results on a 128 x 128 human dancing sequence. website to assess the full video.
Please find our
17

---

Published as a conference paper at ICLR 2021
B.2 KITTI BENCHMARK
B.2.1 DATASET
An ability to cope with partial observability and high stochasticity is highly important in the video prediction task. To evaluate model under this criterion, we adopt KITTI dataset (Geiger et al., 2013) consisting of driving scenes captured by front-view camera in residential and rural areas. Since the segmentation labels are available for only small subset of frames in this dataset (200 frames in total), we extract the labels using the semantic segmentation model (Zhu et al., 2019) pre-trained on still images (Cordts et al., 2016; Geiger et al., 2013) for both training (the entire sequences) and testing (only context frames). We follow the pre- pre-procsssing and train/test splits used in Lotter et al. (2017), and sub-sample each sequence by a factor of two.
B.2.2 IMPLEMENTATION DETAILS
SVG-extend (Villegas et al., 2019). Same as Section B.1.2, we follow the implementation details specified in Section C.I for training this model on KITTI Benchmark dataset.
Bayes-Wil WD-SL (Bhattacharyya et al., 2019). This work is a future segmentation method that ex- ploits odometry information additional supervision and enables stochastic estimation of uncertain futures through the Bayesian approach. On top of ResNet-based deterministic video prediction model (Lue et al., 2017), this work introduces a novel Bayesian formulation to better handle inherent uncertainties when predicting future frames. Specifically, this method introduces (1) weight-dropout (WD) which assumes a finite set of weights for approximating model space, followed by Bernoulli variational distribution on those finite set of weights, and (2) synthetic likelihoods (SL) which miti- gates the constraint on each model to explain all the data points and thus allows sampled models to be diverse to better deal with the uncertainties. For training this model on KITTI Benchmark, we follow the same settings used in the original paper for the experiments on Cityscapes dataset except that we omit odometry supervision for fair comparisons. We use the publicly available code."
B.2.3 QUALITATIVE RESULTS
Short-term prediction. Figure D illustrates the qualitative comparisons for short-term prediction, which correspond Table the main paper. As shown in Figure D. SVG-extend performs very well at the very beginning of the prediction (t = 1 ~ 8), since it directly exploits the content from the last context frame via temporal skip-connection. However, its prediction quality degrades rapidly through time, as it loses the object structure and simulates arbitrary, unstructured pixel-wise motions in the long-term. The future segmentation baseline (Bayes-WD-SLL) preserves a much rea- sonable structure, since its generation is conditioned on the estimated label maps similar to ours. However, it fails to model structures in motion and thus loses such structures in the end. These results demonstrate that both the unsupervised and hierarchical basclines fail to extrapolate to the long-term horizon than the one used during the training. Compared to these baselines, our method generates semantically more accurate motions and structures, and substantially outperforms the oth- ers especially in the latter prediction steps.
Diversity of samples. We show diverse prediction results in Figure E, which are conditioned on the same context frames. As can be seen in the figure, predicted sequences not only are accurate in near future (t 18), but also show diverse motions and scene dynamics through time. For example, we can see diverse scene dynamics such emergence of novel vehicles (f = 54 ~ 144), as well as transition to the novel scene (1 144 180).
Long-term prediction results. Figure F illustrates the long-term prediction results over 2000 frames in 256 x 256 resolution, which corresponds to Figure in the main paper. We observe that the generated sequences are reasonable in both structure and motion. Interestingly, unrolling the prediction over a very long-term leads to very interesting transitions of the scene, such as trans- lation from suburban (t 1035) to rural areas 1500), which cannot be captured in short-term period. We believe that these results illustrate well why the long-term prediction is important.
hhtp:///ggtiaaccmmmaaaaimnnntnnn pred
18

---

Published as a conference paper at ICLR 2021
contexts
t=1
t=8
t=15
t=29
t=36
t=43
t=50
Ground-Truth
SVG-extend
Bayes-WD-SL
Ours
Ground-Truth
SVG-extend
Bayes-WD-ee -D
Figure D: Qualitative comparisons of frame-wise video prediction results across models: (top pane) given the same context frames, we sample 100 predictions for each model and show their best predictions terms of the cosine similarity metric. (bottom pane) for each best prediction, we show the semantic label maps predicted by pre-trained semantic segmentation network (7hu et al., 2019).
t=18
t=36
t=54
1=14
t=162
1=180
Ground-Truth
Sample 1
Sample
Sample 3
Ground-Truth
Sample 1
Sample
Sample 3
Figure E: Diverse video prediction results on a 256 256 KITTI sequence. All samples are condi- tioned on the same context frames.
19

---

Published as a conference paper at ICLR 2021
Context
t=500 5300~535
--1000-1035
1=1500 ~535
-220002033
Figure F:
Long-term prediction results on high-resolution KITTI sequence. Both the frames and the label maps are predicted by our method. Our method can generate high-resolution frames (256 x 256 pixels) into the long-term future without particular quality degradation.
20

---

Published as a conference paper at ICLR 2021
t=1-40
t= 251
290
t= 401 440
CNN
LSTM
SVG Soft
Drop prbo
SVG 1ab
Ours
Figure G: Qualitative comparisons of long-tern generation results across the ablated models. We ablate SVG-extens to produce four baselines: LSTM where stochastic estimation modules moved, CNN where stochastic and recurrent modules are removed, SVGaett where the discretization process is removed and the model is modified to observe and estimate continuous logits of semantic segmentation model, and SVGzgborop where an output of the structure generator at time is trans- lated RGB frame by the pixel generator and then further processed by the semantic segmentation model before used as an input at time t. + 1. Please refer to the Section 5 for the discussion.
21

---

Published as a conference paper at ICLR 2021
B.3 CITYSCAPES DATASET
B.3.1 DATASET
To evaluate the quality of forecasting dense labels by the structure generator, we employ the Cityscapes (Cordts et al., 2016), a widely-used benchmark in future segmentation tasks. The dataset consists of 2.975 training. 1,525 testing, and 500 validation sequences where each sequence is 30- frame-long and has ground-trum truth segmentation labels only in the 20th frame. Similar to KITTI dataset, we employ the pre-trained segmentation network (Zhu et al., 2019) to generate segmenta- tion labels of 256 x 512 image resolution for training our SVG-extend, Following (Luc et al., 2017; 2018), subsample frames by factor of three for each sequence and test all models on 128 x 256 image resolution.
B.3.2
IMPLEMENTATION DETAILS
S2S (Luc et al., 2017). We use the best performing S2S model proposed by the authors, namely S25- dil, AR. fine-tune, which improves the performance of the single-frame prediction baseline (S2S) by introducing the followings: (I) dilated convolution to enlarge receptive fields of the model. (2) autoregressive fine-tuning that uses the prediction time t as an input to predict the future frame at time t + 1 during training. which enables the single-frame prediction model to predict an arbi- trary number of frames into the future. We use a publicly available pre-trained model* provided by the authors, which was trained using the predicted semantic segmentation maps by Dilation10 network (Yu & Koltun, 2016) on 128 x 256 image resolution.
F2F (Luc et al., 2018). This method targets future instance segmentation task, which extends fu- ture segmentation method S2S (Luc et al., 2017) by formulating the task of the model as predicting intermediate feature maps of Mask R-CNN (He et al., 2017) on the target future instance segmen- tation masks. We use a publicly available pre-trained model and code' provided by the authors, which first trained on MS-COCO dataset (Lin et al., 2014) and then finetuned the Cityscapes images of 1024 x 2048. Following (Luc et al., 2017; 2018), we down-sample the predicted future instance segmentation masks to 128 x 256 resolution for the evaluation.
B.3.3 QUALITATIVE RESULTS
Qualitative comparison. We present the qualitative comparisons of our method to existing fu- ture segmentation models, which correspond to Table 3 in the main paper. As shown in the table, our method outperforms the two baselines in all classes, thanks its ability to handle the highly stochastic nature of complex driving scenes. We also compare qualitative samples of each model in Figure H. As shown in the figure, our model predicts convincing structures and their temporal dynamics are accurate. On the other hand, the two baselines produce blurry predictions, resulting in (1) inaceurate structures (S2S at all timestamps and F2F at 17-29th timestamps), and (2) multiple se- mantics in single structure (S2S at all timestamps and F2F at 23-29th timestamps). These blurred prediction problems have been widely observed in the literature and attributed to the inability of deterministic models to handle stochasticity in data (Denton & Fergus, 2018).
Short-term prediction. To evaluate the future segmentation results in Section 4.4, we extract 6 future semantic or instance segmentations ahead (14, 17, 20, 23, 26, and 29th) from each model given 4 contexts (2, 5, 8, and 1lth). Then, following (Luc et al., 2018), we compute per-class and mean IoUs for 8 moving object categories only: person, rider, car, truck bus, train, motorcycle, bicycle.
Long-term video prediction. Figure J illustrates the qualitative results of video prediction results on the Cityscapes dataset. As the Cityscapes contains many occluding objects, we employ the extended structure generator with instance boundary prediction to improve the frame prediction quality. We show in Figure J the predicted sequences up to 96 future frames conditioned on context frames. As shown in the figure, our method can casily incorporate the additional structures (i.e. instance boundary map), and can be generalized to complex and high-resolution sequences
5 httpp://gghhbhcooomcccccccccccnp
22

---

Published as a conference paper at ICLR 2021
14th
17th
20th
23th
26th 29th
(27)
PxF (30)
Ours
Oraele (38]
Figure H: Quantitative comparisons of future segmentation models. All models predict structures of moving objects (8 classes) up 29th frame given context frames (2, 8, and 1 1th). The oracle obtained by running the pre-trained segmentation model on the ground-truth frames. The results of compared methods (S2S and F2F) are based on the pre-trained models provided by the authors.
to predict instance structures into the long future without error propagations (e.g. blurred predictions in the structure space). On the other hand, we also observe some undesirable biases the predicted motion dynamics in the structure space, such as static or cyclic motions of structures (e.g.red dashed boxes in Figure J). We notice that it is because the training sequences in Cityscapes datasets are considerably short (i.e. 30 frames), which makes the structure generator favor short-term dependencies in the prediction (e.g. short-term skip connections). It leads biases in the model and artifacts in long-ten extrapolation, such as static motions in the distant objects (e.g. buildings and trees far away from the camera), fixed content (e.g. repeatedly generated objects), and etc., which are not observed in the other datasets with reasonably long training sequences.
B.3.4 THE EFFECT OF INSTANCE BOUNDARY MAP PREDICTION
To analyze the effect of the instance boundary map pro- duced by Godge. compare the prediction performance of our model with and without Gedge. Specifically, we
Table Quantitative results ablating
Gedge on the 256 x 512 Cityscapes dataset.
extract 6 future frames ahead (14, 17, 20, 23, 26, and
Model
PSNR(1)
SSIM(1)
CSIM(1)
29-th) given 4 context frames (1, 4, 7, and 11-th) from
With Gate
20.427
0.610
0.938
the models, and compare frame-wise evaluation results measured by Peak Signal-to-noise ratio (PSNR), Struc- tural Similarity (SSIM), and VGG cosine similarity (CSIM). We average the measured scores across the test sequences and timestamps. Table A and Figure show the quantitative and qualitative re- sults, respectively. As shown in the table, we observe marginal improvements those metrics due to the relatively short length of Cityscapes sequences (up to 30 frames). However, as can be seen in the figure, we find that the generated frames contain more clear instance boundaries with Gedge especially in crowded scenes.
Without Gatge
19.236
0.602
0.937
Semantic
Label Map
Boundary Map
RGB Prediction
Without Gedge
With Godge
Figure Qualitative results of ablating Gedge the 256 512 Cityscapes dataset.
23

---

Published as a conference paper at ICLR 2021
Last Observed
Predicted
Frame
Frames
11th
71th
128th
185th
242th
299th
Figure J: Long-term prediction results on 256 x 512 Cityscapes dataset. We predict 96 future frames (14, 17,..., ..., 299th) given 4 context frames. Our structure generator extended produce additional instance boundary map for improved instance boundary demarcation in the image generator. Red dashed boxes indicate some biases in the prediction, such as static and eyclic motions in the predicted structures, which are attributed to the short training sequences in the Cityscapes dataset. Please refer Section 4.4 for the description.
24

---

Published as a conference paper at ICLR 2021
C MODEL ARCHITECTURE
In this section, we detail the architectures and hyperparameters used in our experiments.
C.I SEQUENTIAL DENSE STRUCTURE GENERATOR
We use the modified version of SVG (Denton Fergus, 2018) by Villegas (2019) as our sequential dense structure generator, which we refer to SVG-extend throughout the paper. The two main differences between SVG and SVG-extend are two-fold: (I) SVG-extend adopts the shal- lower auto-encoder architecture than that of SVG. (2) SVG-extend replaces Fully-Connected with Convolutional-L.5 information contained in 3D feature maps, which is not maintained when inputs mapped to low-dimensional I-D image embedding vectors.
(Shi et al., 2015). These two modifications are to better exploit spatial
Since the detailed architectural configuration of SVG-extend is not described in (Villegas et al., 2019), we summarize our reproduced version of this framework in Table B, C, D, and E. On the 64 x 64 image resolution, and 256 x 256 image resolutions, we simply stack one and two more convolutional blocks (non- linear convolutional layers followed by upsampling or downsampling layers) in the Encoder and Decoder networks, respectively, so that the shape of h, Zz, is kept to R 8x8 agnostic to the image resolutions. For hyperparameters, we use C 5; we set B 0.0005; we use ADAM optimizer (Kingma & Ba, 2015) with the learning rate of 0.0001 and (81,82) (0.9, 0.999).
the architecture described in the tables. On 128 x 128
C.2 BOUNARY MAP PREDICTION
When the pre-trained instance-wise segmentation model available, we can optionally extend the structure generator to predict the object boundary maps well. Such structures add notion of object instance and is useful to improve the image generator in sequences with many occluding objects. Let e {0, 1 H>W denotes the object boundary map at frame X,. Then train the denoising autoencoder Godge to produce an edge map of each frame by
é Godge ($:)-
(8)
We employed the encoder-decoder network (Nazeri et al., 2019) as the conditional generator, and train it based on adversarial loss by
Lade
Ea,s [og D(e, s)] + [o[g[ - D(e, Gedge
where D is the discriminator. Note that the edge generator in Eq. applied to each frame inde- pendently; observe that adding the predicted boundary map as input the structure generator makes the model prune to error propagation. The boundary and label maps are then combined as the output of the structure generator $; [s,, c.).
C.3 STRECTURE--CNDDITONNA PIXEL SEQUENCE GENERATOR
We use Vid2Vid (Wang et al., 2018) for our structure-conditional pixel sequence generator. For Vid2Vid on KITTI dataset of x 64 resolution, we reduce the number downsampling layers in the generator and the scale parameter of the patch discriminator to 1. We use T = 5 for KITTI and Human Dancing and use + for Cityscapes. We use 7' = for experiments. If not specified otherwise, we do not modify any hyperparameters and simply follow the default settings for the training.
25

---

Published as a conference paper at ICLR 2021
Table B: Encoder Architecture.
Input: S, E R66x616
3 X Conv 64, ReLU
3 X Conv 64, ReLU + h
2 X MaxPool
3 X 3 Conv 128, ReLU
3 X Conv 128, ReLU - h?
2 X 2 MaxPool
3 X Conv 256, ReLU
3 x Conv 256, ReL.U
3 X 3 Conv 256, ReLU + h²
2 x MaxPool
3 x 3 Conv 128, ReLU + h,
Table C: Posterior/Prior LSTM Architecture.
Input: h, E R 128x8x8
3 CSnVLSTM 256, 3 3 Conv 256, ReLU :(()), (())), N (u((), (()) E R 12888x8
Table Predictor LSTM Architecture.
Input: [h, 2,] E (122++128)88x8
3 x ConvLSTM 256, ReLU
3 3 ConvLSTM 256, ReLU
3 x 3 Conv 128, ReLU + E I 128x8x8
Table Decoder Architecture.
Input: gr
R 128x8x8
2 NN Upsample g? Concatenate(g] h²)
3 x Conv 256, ReLU
3 Conv 256, ReLU
3 x Conv 128, ReLU
2 x NN Upsample - g?
Concatenate(g;; h((
3 x Conv 128, ReLU
3 x Conv 64, ReLU
2 x NN Upsample + g! Concatenate(g:: h
3 Conv 64, ReLU
3 x Conv 3, LogSoftmax + St
26

---

Published as a conference paper at ICLR 2021
Table F: Time required to train the model.
Dataset
Model
GPU (V100 16GB) Training Time
KITTI & Human Dancing 64 64
SVG-Extend
9 hours
Vid2Vid
12 hours
Human Dancing 128 128
SVG-Extend
48 hours
Vid2Vid
4
48 hours
SVG-Extend
8
48 hours
KITTI 256 x 256
Vid2Vid
8
48 hours
SVG-Extend
8
48 hours
CittyScapes 256 x 512
Edge Predictor
12 hours
Vid2Vid
48 hours
27


Reviews:
[{"review_id": "xSoqMhhlT28", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---- Summary ----\n\nThe paper extends video-to-video translation model of (Wang’18) to video prediction by first generating a sequence of segmentation masks and then translating them into videos. Variational video prediction is used to generate a sequence of segmentation masks. The model produces impressive high-resolution and long-horizon results, and is extensively evaluated on Kitti, Cityscapes, and dancing data, outperforming some previously proposed methods.\n\n---- Decision ----\n\nThe paper proposes a relevant method for hierarchical video prediction with impressive high-resolution and long-horizon results. As the paper notes, this sets a new standard for video prediction methods, and will likely spur more research into achieving similar results without the labeled data requirement. I am willing to accept the paper provided the author’s response clarifies my questions.\n\n---- Strengths ----\n\nThe paper presents a modern version of Villegas’17a, powered by the advances in probabilistic video prediction and generative adversarial networks. The proposed model significantly outperforms prior work, scaling to high-resolution images (256x256) and long horizons (2500 frames). \n\n---- Weaknesses ----\n\nA weakness in the experimental setup is that the compared baselines are not controlled for the number of parameters. In particular, SVG-extend, which also works well on Kitti data, seems to contain the same number of parameters as the segmentation prediction network of the proposed model. It is also unclear how the baselines were tuned and what was the range of considered hypeparameters for all methods. More generally, it would be good for the paper to present some simple setting in which SVG-extend works well to analyze where hierarchical prediction is most important. \n\n---- Additional comments ----\n\nThere is a number of minor inaccuracies in the paper: \n\nThe authors of Wichers’18 are cited as “Ruben Wichers, Nevan Villegas”. The real authors of this paper are Nevan Wichers and Ruben Villegas.\n\nPage 2, “We employ the sequence model based on VAE (Denton & Fergus, 2018)”. Denton and Fergus did not invent VAEs, neither they invented sequential VAEs. You likely want to cite the original VAE papers (Kingma’14, Rezende’14), or the sequential VAE papers (Chung’15, Fraccaro’15). Somewhat strangely, none of these papers are cited elsewhere in the paper either. If you do want to cite Denton & Fergus, the sentence would need to be “We employ the sequence model based on VAE proposed by Denton & Fergus (2018)””.\n\nPage 3, “we skip hidden representations of the encoder to the decoder at every time step during testing to handle longterm dynamics in structure”. Would be great to cite some papers that do also that, e.g. Villegas’17a or Finn’16.\n\n“Extension to object boundary prediction”. This paragraph seems to be hastily written and full of incorrect statements. The G is not an autoencoder (and not a denoising autoencoder either). It predicts e from s, maybe just call it a boundary prediction network? The conditional GAN objective does not maximize p(e,s). In fact, p(s) is a constant. Instead, it tries to match p(e|s), but not by maximizing likelihood, but by minimizing an approximation to Jensen-Shannon divergence (see Goodfellow’14).\n\nEq (1) is not a variational lower bound for beta <> 1.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bWvitgCfd5z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a hierarchical framework for long-term video prediction. The structure is firstly predicted in the form of semantic map. It lies in a categorical structure space which is easier to predict. Then the authors translate the predicted semantic map to a real video sequence in a frame-by-frame manner. The proposed model is \"surprisingly successful\" for long-term video prediction, as claimed by the authors (thousands frames). \n\nPros:\n\n+ clarity: This paper is overall easy to understand. The proposed method, which firstly predicts the categorical map and then translate  the map to video frame, is very straightforward and clear. The motivation behind such kind of design is convincing, i.e., to explicitly reduce the modelling difficulty of the prediction module.\n\n+ significance: I have carefully checked the prediction results provided by the authors. I think the predicted video sequences are convincing and appealing, which are of >1000 frames. The video content does not freeze even after thousands frames. The visual quality on the cityscape dataset looks very promising.\n\nCons:\n\n- originality: The proposed framework (first predicting semantic label then translating) is a simple extension compared to the previous work[1,2]. The model used in this work is a typical combination of LSTM and VAE, where the uncertainty is modeled with Gaussian Distribution and KL-divergence loss. For the second part, the auto-encoder based translation model, along with adversarial training scheme, is also commonly used in the image transfer task. Overall, the novelty of the proposed method is limited in my point of view. \n\n- quality: Regarding this aspect, my major concern lies in the lack of the ablation study. In the experiment, the authors present sufficient and extensive results compared with other methods. However, the contribution of each module in the proposed method is completely unknown. Which part is the most important? Which part is the key to long-term video prediction? A comprehensive analysis about this part is highly recommended.\n\n[`1] Predicting deeper into the future of semantic segmentation.\n[2] Predicting future instance segmentation by forecasting convolutional features.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KpGEUM4GBer", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a VAE based hierarchical model for video prediction. The model employs recurrent model to predict intermediate representations (in the form of label maps) and these representations are mapped to pixel level information, i.e., videos. The paper presents an interesting idea of using representations that do not use any domain knowledge. The authors demonstrate the value of modeling temporal evolution of these representations which enables long term video prediction.\n\n**Strengths**\n+ The paper is clearly written and the implementation details are clearly described \n+ The model outperforms relevant baselines quantitatively\n+ The model is able to generate realistic and temporally consistent samples\n+ The evaluation of the model is thorough and performs better consistently across challenging datasets.\n\n\n**Weaknesses**\n- How important is the warped optical flow in the video discriminator?\n- How is $\\tau'$ decided? Does it have any impact on the performance or computation requirements of the model? \n- The role of $G_{edge}$ is unclear. An ablation study would be useful. \n- Given the complexity of the task, it would be good to report the time required to train the models.\n\nOverall, the paper presents impressive results on long-term video prediction and the evaluation of the paper is thorough. However, some ablations studies are required to convince the importance of each component used in the overall model. Therefore, my initial rating for this paper is 6.\n\n\n=======================================**Post-rebuttal Comments**===============================\n\n\nI appreciate the revisions and additional results reported by the authors. The authors have addressed the concerns raised by me in the revision. While I agree with R1 that novelty of the method is limited, after considering the reviews collectively, I believe this paper presents very impressive results given the fact that the problem is challenging. Therefore, I would like to improve my final rating to 7.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8K9eLKC7GEo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###Summary###\n\n\nThe paper proposes the hierarchical video prediction model.\nSpecifically, the model first generates a sequence of semantic segmentation maps.\nAnd then it generate a sequence of future frames corresponding to the semantic segmentation maps.\nSince the sequence is generated stochastically, it can generate various futures given same condition.\nWith this hierarchical method, they could successfully generated thousands of future frames.\n\n\n###Pros###\n\n\n-\nBy using low-level information for modeling of dynamics, the proposed method generates realistic images not suffering from severe error amplification issue.\n\n\n\n###Questions###\n\n-\nIt is not clearly pointed why this method successfully generates long-term sequences.\nIs it the result of learning the approximate posterior distribution of the following semantic segmentation map, or the result of the hierarchical generation mechanism?\n\n-\nIs the modeling motion only using the LSTM or conditional CNN inferior to this method? (deterministic one, no modeling for the approximate posterior distribution)\n\n-\nRegarding the equation 4, have you tried training the model without applying the teacher-forcing?\nWhich means, training with the generated semantic segmentation maps as the condition.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new implementation of a previously proposed two-stage process for video prediction: first predict future segmentation maps, then map them to video frames. Combined with other advances in video prediction and image generation, this simple idea is shown empirically to work very well, producing video predictions up to many hundreds of frames into the future in real stochastic settings with unprecedented quality. Strong ablation studies over the course of the review process further serve to confirm the value of various design choices involved in the implementation. 
Decision: Accept (Poster)