Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction

1 INTRODUCTION . Video prediction aims to generate future frames conditioned on a short video clip . It has received much attention in recent years as forecasting the future of visual sequence is critical in improving the planning for model-based reinforcement learning ( Finn et al. , 2016 ; Hafner et al. , 2019 ; Ha & Schmidhuber , 2018 ) , forecasting future event ( Hoai & Torre , 2013 ) , action ( Lan et al. , 2014 ) , and activity ( Lan et al. , 2014 ; Ryoo , 2011 ) . To make it truly beneficial for these applications , video prediction should be capable of forecasting long-term future . Many previous approaches have formulated video prediction as a conditional generation task by recursively synthesizing future frames conditioned on the previous frames ( Vondrick et al. , 2016 ; Tulyakov et al. , 2018 ; Denton & Fergus , 2018 ; Babaeizadeh et al. , 2018 ; Castrejon et al. , 2019 ; Villegas et al. , 2019 ) . Despite their success in short-term forecasting , however , none of these approaches have been successful in synthesizing convincing long-term future , due to the challenges in modeling complex dynamics and extrapolating from short sequences to much longer future . As the prediction errors easily accumulate and amplify through time , the quality of the predicted frames quickly degrades over time . One way to reduce the error propagation is to extrapolate in a low dimensional structure space instead of directly estimating pixel-level dynamics in a video . Therefore , many hierarchical modeling approaches are proposed ( Villegas et al. , 2017b ; Wichers et al. , 2018 ; Liang et al. , 2017 ; Yan et al. , 2018 ; Walker et al. , 2017 ; Kim et al. , 2019 ) . These approaches first generate a sequence using a low-dimensional structure representation , and subsequently generate appearance conditioned on the predicted structures . Hierarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future . However , previous approaches often employed too specific and incomprehensive structures such as human body joints ( Villegas et al. , 2017b ; Yan et al. , 2018 ; Yang et al. , 2018 ; Walker et al. , 2017 ; Kim et al. , 2019 ) or face landmarks ( Yan et al. , 2018 ; Yang et al. , 2018 ) . ∗The author contributed to this work while at Google Research . Moreover , they made oversimplified assumptions of the future by using a deterministic loss or assuming homogeneous content . We therefore argue that the benefit of hierarchical models has been underestimated and their impact on long-term video prediction has not been properly demonstrated . In this paper , we propose a hierarchical model with a general structure representation ( i.e. , dense semantic label map ) for long-term video prediction on complex scenes . We abstract the scene as categorical semantic labels for each pixel , and predict the motion and content change in this label space using a variational sequence model . Given the context frames and the predicted label maps , we generate the textures by translating the sequence of label maps to the RGB frames . As dense label maps are generic and universal , we can learn comprehensive scene dynamics from object motion to even dramatic scene change . We can also capture the multi-modal dynamics in the label space with the stochastic prediction of the variational sequence model . Our experiments demonstrate that we can generate a surprisingly long-term future of videos , from driving scenes to human dancing , including the complex motion of multiple objects and even an evolution of the content in a distant future . We also show that predicted frame quality is preserved through time , which enables persistent future prediction virtually near-infinite time horizon . For scalable evaluation of long-term prediction at this scale , we also propose a novel metric called shot-wise FVD , which enables the evaluation of spatio-temporal prediction quality without ground-truths and is consistent with human perception . 2 METHOD . Given the context frames x1 : C = { x1 , x2 , ... , xC } , our goal is to synthesize the future frames xC+1 : T = { xC+1 , xC+2 , ... , xT } up to an arbitrary long-term future T . Let st ∈ RN×H×W denote a dense label map of the frame xt defined over N categories , which is inferred by the pre-trained semantic segmentation model1 . Then given the context frames x1 : C and the label maps s1 : C , our hierarchical framework synthesizes the future frames x̂C+1 : T by the following steps . • A structure generator takes the context label maps as inputs and produces a sequence of the future label maps by ŝC+1 : T ∼ Gstruct ( s1 : C , z1 : T ) , where zt denotes the latent variable encoding the stochasticity of the structure . • Given the context frames and the predicted structures , an image generator produces RGB frames by x̂C+1 : T ∼ Gimage ( x1 : C , { s1 : C , ŝC+1 : T } ) . Figure 1 illustrates the overall pipeline . Note that there are various factors beyond motions that make spatio-temporal variations in the label maps , such as an emergence of new objects , partial observability , or even dramatic scene changes by the global camera motion ( e.g. , panning ) . By learning to model these dynamics in the semantic labels with the stochastic sequence estimator and conditioning the video generation with the estimated labels , the proposed hierarchical model can synthesize convincing frames into the very long-term future . Below , we describe each component . 1We employ the pre-trained network ( Zhu et al. , 2019 ) on still images to obtain segmentations in videos . 2.1 SEQUENTIAL DENSE STRUCTURE GENERATOR . Our structure generator models dynamics in label maps by p ( s≤T |z≤T ) = ∏T t=1 p ( st|s < t , z≤t ) . We employ a sequence model proposed by Denton & Fergus ( 2018 ) since it ( 1 ) provides a probabilistic framework to handle stochasticity in structures and ( 2 ) can easily incorporate discrete sequences . Specifically , we optimize β-VAE objective ( Higgins et al. , 2017 ; Denton & Fergus , 2018 ) as below : T∑ t=1 Eqφ ( z≤T |s≤T ) [ log pθ ( st|z≤t , s < t ) ] − βDKL ( qφ ( zt|s≤t ) ||pψ ( zt|s < t ) ) . ( 1 ) where qφ ( z≤T |s≤T ) is the approximated posterior distribution and pψ ( zt|s < t ) is the prior distribution . At each step , it learns to represent and reconstruct each label map st through CNN-based encoder f enc and decoder f dec , respectively , while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows : µφ ( t ) , σφ ( t ) = LSTMφ ( ht ) zt ∼ N ( µφ ( t ) , σφ ( t ) ) , gt = LSTMθ ( ht−1 , zt ) st = f dec ( gt ) , where ht = f enc ( st ) , where ht−1 = f enc ( st−1 ) , ( 2 ) where LSTMθ and LSTMφ respectively approximate the generative and the posterior distributions recurrently up to the time step t. Unlike Denton & Fergus ( 2018 ) ; Villegas et al . ( 2019 ) that exploit skip-connection from the last observed context frame during both training and testing time , we skip hidden representations of the encoder to the decoder at every time step during testing to handle long-term dynamics in structure as in ( Villegas et al. , 2017a ; Finn et al. , 2016 ) . During training , we apply teacher-forcing by feeding ground-truth label maps and sampling the latent zt from the posterior distribution N ( µφ ( t ) , σφ ( t ) ) = LSTMφ ( f enc ( st ) ) . During inference , we recursively generate label maps by ( 1 ) sampling zt from the prior distribution N ( µψ ( t ) , σψ ( t ) ) = LSTMψ ( f enc ( ŝt−1 ) ) , ( 2 ) producing the frame ŝt through the decoder , and ( 3 ) discretizing the predicted label map by taking pixel-wise maximum . Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints ( Villegas et al. , 2017b ; Yan et al. , 2018 ; Yang et al. , 2018 ) or optical flow ( Walker et al. , 2017 ) . Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available , we can optionally extend the structure generator to jointly predict the object boundary maps . Such structure can add a notion of object instance , and is useful to improve the image generator in sequences with many occluding objects . Let et ∈ { 0 , 1 } H×W denotes the object boundary map at frame xt . Then we train the conditional generator êt = Gedge ( ̂st ) that produces a boundary map given the label map for each frame1 . We train Gedge using the conditional GAN objective to match the generated distribution to the joint distribution of the real label and real boundary maps p ( st , et ) . The boundary and label maps are then combined as the output of the structure generator s̃t = [ ŝt , êt ] , and are used as inputs to the image generator described below . 2.2 STRUCTURE-CONDITIONAL PIXEL SEQUENCE GENERATOR . Given a sequence of the structures and the context frames , the image generator learns to model the conditional distribution of the RGB frames by p ( x≤T |s≤T ) = ∏T t=1 p ( xt|x < t , s≤t ) . We formulate this task as a video-to-video translation problem , and employ a state-of-the-art conditional video generation model ( Wang et al. , 2018 ) . Specifically , the video synthesis network F in Wang et al . ( 2018 ) consists of three main components ; the generatorH , occlusion mask predictorM , and optical flow estimator W , which are combined to generate each frame x̂t by the following operation : x̂t = F ( x̂t−τ : t−1 , st−τ : t ) = ( 1−mt ) ŵt−1 +mt ht ( 3 ) where ŵt−1 = W ( x̂t−τ : t−1 , st−τ : t ) is the warped previous frame x̂t−1 using the estimated optical flow , ht = H ( x̂t−τ : t−1 , st−τ : t ) is the hallucinated frame at t , and mt = M ( x̂t−τ : t−1 , st−τ : t ) is the soft occlusion mask blending ŵt−1 and ht . Unlike models synthesizing future frames by transforming the context frames xC ( Villegas et al. , 2019 ; 2017b ; Yan et al. , 2018 ; Tulyakov et al. , 2018 ) , this 1We do not feed the predicted boundary map as input to the structure generator since it makes the structure generator prune to error propagation thus prevents extrapolation to long-term future . model is appropriate to synthesize the long-term future since it can handle both transformation of the existing objects ( via ŵt−1 ) and synthesis of the emerging objects ( via ht ) . To ensure both frame-level and video-level generation quality , the video synthesis network F is trained against a conditional image discriminator DI and a conditional video discriminator DV through adversarial learning by LI ( F , DI ) =EφI ( x≤T , s≤T ) [ logDI ( xi , si ) ] + EφI ( x̂≤T , s≤T ) [ log ( 1−DI ( x̂i , si ) ) ] , ( 4 ) LV ( F , DV ) =EφV ( w < T , x≤T , s≤T ) [ logDV ( xi−1 : i−τ′ , wi−2 : i−τ′ ) ] + EφV ( w < T , x̂≤T , s≤T ) [ log ( 1−DV ( x̂i−1 : i−τ′ , wi−2 : i−τ′ ) ) ] , ( 5 ) where LI ( F , DI ) and LV ( F , DV ) are frame-level and video-level adversarial losses , respectively . For efficient training , we follow Wang et al . ( 2018 ) to adopt sampling operators φI ( x≤T , s≤T ) = ( xi , si ) and φV ( w < T , x≤T , s≤T ) = ( wi−2 : i−τ′ , xi−1 : i−τ′ , si−1 : i−τ′ ) for frame and video-level adversarial learning objectives , respectively , where i is an integer sampled from U ( 1 , T ) and U ( τ ′ + 1 , T + 1 ) for frame sampling operator φI and video sampling operator φV , respectively . Then the final learning objective is formulated by min F max DI , DV LI ( F , DI ) + LV ( F , DV ) . ( 6 ) During training , the model is trained to estimate RGB frames from the GT structures , while the predictions from the structure generator are used during testing . See Appendix C.3 for more details .

Summary: The paper extends video-to-video translation model of (Wang’18) to video prediction by first generating a sequence of segmentation masks and then translating them into videos. Variational video prediction is used to generate a sequence of segmentation masks. The model produces impressive high-resolution and long-horizon results, and is extensively evaluated on Kitti, Cityscapes, and dancing data, outperforming some previously proposed methods.