Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction

1 INTRODUCTION . Video prediction aims to generate future frames conditioned on a short video clip . It has received much attention in recent years as forecasting the future of visual sequence is critical in improving the planning for model-based reinforcement learning ( Finn et al. , 2016 ; Hafner et al. , 2019 ; Ha & Schmidhuber , 2018 ) , forecasting future event ( Hoai & Torre , 2013 ) , action ( Lan et al. , 2014 ) , and activity ( Lan et al. , 2014 ; Ryoo , 2011 ) . To make it truly beneficial for these applications , video prediction should be capable of forecasting long-term future . Many previous approaches have formulated video prediction as a conditional generation task by recursively synthesizing future frames conditioned on the previous frames ( Vondrick et al. , 2016 ; Tulyakov et al. , 2018 ; Denton & Fergus , 2018 ; Babaeizadeh et al. , 2018 ; Castrejon et al. , 2019 ; Villegas et al. , 2019 ) . Despite their success in short-term forecasting , however , none of these approaches have been successful in synthesizing convincing long-term future , due to the challenges in modeling complex dynamics and extrapolating from short sequences to much longer future . As the prediction errors easily accumulate and amplify through time , the quality of the predicted frames quickly degrades over time . One way to reduce the error propagation is to extrapolate in a low dimensional structure space instead of directly estimating pixel-level dynamics in a video . Therefore , many hierarchical modeling approaches are proposed ( Villegas et al. , 2017b ; Wichers et al. , 2018 ; Liang et al. , 2017 ; Yan et al. , 2018 ; Walker et al. , 2017 ; Kim et al. , 2019 ) . These approaches first generate a sequence using a low-dimensional structure representation , and subsequently generate appearance conditioned on the predicted structures . Hierarchical approaches are potentially promising for long-term prediction since learning structure-aware dynamics allows the model to generate semantically accurate motion and content in the future . However , previous approaches often employed too specific and incomprehensive structures such as human body joints ( Villegas et al. , 2017b ; Yan et al. , 2018 ; Yang et al. , 2018 ; Walker et al. , 2017 ; Kim et al. , 2019 ) or face landmarks ( Yan et al. , 2018 ; Yang et al. , 2018 ) . ∗The author contributed to this work while at Google Research . Moreover , they made oversimplified assumptions of the future by using a deterministic loss or assuming homogeneous content . We therefore argue that the benefit of hierarchical models has been underestimated and their impact on long-term video prediction has not been properly demonstrated . In this paper , we propose a hierarchical model with a general structure representation ( i.e. , dense semantic label map ) for long-term video prediction on complex scenes . We abstract the scene as categorical semantic labels for each pixel , and predict the motion and content change in this label space using a variational sequence model . Given the context frames and the predicted label maps , we generate the textures by translating the sequence of label maps to the RGB frames . As dense label maps are generic and universal , we can learn comprehensive scene dynamics from object motion to even dramatic scene change . We can also capture the multi-modal dynamics in the label space with the stochastic prediction of the variational sequence model . Our experiments demonstrate that we can generate a surprisingly long-term future of videos , from driving scenes to human dancing , including the complex motion of multiple objects and even an evolution of the content in a distant future . We also show that predicted frame quality is preserved through time , which enables persistent future prediction virtually near-infinite time horizon . For scalable evaluation of long-term prediction at this scale , we also propose a novel metric called shot-wise FVD , which enables the evaluation of spatio-temporal prediction quality without ground-truths and is consistent with human perception . 2 METHOD . Given the context frames x1 : C = { x1 , x2 , ... , xC } , our goal is to synthesize the future frames xC+1 : T = { xC+1 , xC+2 , ... , xT } up to an arbitrary long-term future T . Let st ∈ RN×H×W denote a dense label map of the frame xt defined over N categories , which is inferred by the pre-trained semantic segmentation model1 . Then given the context frames x1 : C and the label maps s1 : C , our hierarchical framework synthesizes the future frames x̂C+1 : T by the following steps . • A structure generator takes the context label maps as inputs and produces a sequence of the future label maps by ŝC+1 : T ∼ Gstruct ( s1 : C , z1 : T ) , where zt denotes the latent variable encoding the stochasticity of the structure . • Given the context frames and the predicted structures , an image generator produces RGB frames by x̂C+1 : T ∼ Gimage ( x1 : C , { s1 : C , ŝC+1 : T } ) . Figure 1 illustrates the overall pipeline . Note that there are various factors beyond motions that make spatio-temporal variations in the label maps , such as an emergence of new objects , partial observability , or even dramatic scene changes by the global camera motion ( e.g. , panning ) . By learning to model these dynamics in the semantic labels with the stochastic sequence estimator and conditioning the video generation with the estimated labels , the proposed hierarchical model can synthesize convincing frames into the very long-term future . Below , we describe each component . 1We employ the pre-trained network ( Zhu et al. , 2019 ) on still images to obtain segmentations in videos . 2.1 SEQUENTIAL DENSE STRUCTURE GENERATOR . Our structure generator models dynamics in label maps by p ( s≤T |z≤T ) = ∏T t=1 p ( st|s < t , z≤t ) . We employ a sequence model proposed by Denton & Fergus ( 2018 ) since it ( 1 ) provides a probabilistic framework to handle stochasticity in structures and ( 2 ) can easily incorporate discrete sequences . Specifically , we optimize β-VAE objective ( Higgins et al. , 2017 ; Denton & Fergus , 2018 ) as below : T∑ t=1 Eqφ ( z≤T |s≤T ) [ log pθ ( st|z≤t , s < t ) ] − βDKL ( qφ ( zt|s≤t ) ||pψ ( zt|s < t ) ) . ( 1 ) where qφ ( z≤T |s≤T ) is the approximated posterior distribution and pψ ( zt|s < t ) is the prior distribution . At each step , it learns to represent and reconstruct each label map st through CNN-based encoder f enc and decoder f dec , respectively , while the temporal dependency of the label maps is modeled stochastically by the two LSTMs as follows : µφ ( t ) , σφ ( t ) = LSTMφ ( ht ) zt ∼ N ( µφ ( t ) , σφ ( t ) ) , gt = LSTMθ ( ht−1 , zt ) st = f dec ( gt ) , where ht = f enc ( st ) , where ht−1 = f enc ( st−1 ) , ( 2 ) where LSTMθ and LSTMφ respectively approximate the generative and the posterior distributions recurrently up to the time step t. Unlike Denton & Fergus ( 2018 ) ; Villegas et al . ( 2019 ) that exploit skip-connection from the last observed context frame during both training and testing time , we skip hidden representations of the encoder to the decoder at every time step during testing to handle long-term dynamics in structure as in ( Villegas et al. , 2017a ; Finn et al. , 2016 ) . During training , we apply teacher-forcing by feeding ground-truth label maps and sampling the latent zt from the posterior distribution N ( µφ ( t ) , σφ ( t ) ) = LSTMφ ( f enc ( st ) ) . During inference , we recursively generate label maps by ( 1 ) sampling zt from the prior distribution N ( µψ ( t ) , σψ ( t ) ) = LSTMψ ( f enc ( ŝt−1 ) ) , ( 2 ) producing the frame ŝt through the decoder , and ( 3 ) discretizing the predicted label map by taking pixel-wise maximum . Such discretization provides additional merits for being more robust against error propagation than continuous structures such as 2D keypoints ( Villegas et al. , 2017b ; Yan et al. , 2018 ; Yang et al. , 2018 ) or optical flow ( Walker et al. , 2017 ) . Extension to object boundary prediction When the pre-trained instance-wise segmentation model is available , we can optionally extend the structure generator to jointly predict the object boundary maps . Such structure can add a notion of object instance , and is useful to improve the image generator in sequences with many occluding objects . Let et ∈ { 0 , 1 } H×W denotes the object boundary map at frame xt . Then we train the conditional generator êt = Gedge ( ̂st ) that produces a boundary map given the label map for each frame1 . We train Gedge using the conditional GAN objective to match the generated distribution to the joint distribution of the real label and real boundary maps p ( st , et ) . The boundary and label maps are then combined as the output of the structure generator s̃t = [ ŝt , êt ] , and are used as inputs to the image generator described below . 2.2 STRUCTURE-CONDITIONAL PIXEL SEQUENCE GENERATOR . Given a sequence of the structures and the context frames , the image generator learns to model the conditional distribution of the RGB frames by p ( x≤T |s≤T ) = ∏T t=1 p ( xt|x < t , s≤t ) . We formulate this task as a video-to-video translation problem , and employ a state-of-the-art conditional video generation model ( Wang et al. , 2018 ) . Specifically , the video synthesis network F in Wang et al . ( 2018 ) consists of three main components ; the generatorH , occlusion mask predictorM , and optical flow estimator W , which are combined to generate each frame x̂t by the following operation : x̂t = F ( x̂t−τ : t−1 , st−τ : t ) = ( 1−mt ) ŵt−1 +mt ht ( 3 ) where ŵt−1 = W ( x̂t−τ : t−1 , st−τ : t ) is the warped previous frame x̂t−1 using the estimated optical flow , ht = H ( x̂t−τ : t−1 , st−τ : t ) is the hallucinated frame at t , and mt = M ( x̂t−τ : t−1 , st−τ : t ) is the soft occlusion mask blending ŵt−1 and ht . Unlike models synthesizing future frames by transforming the context frames xC ( Villegas et al. , 2019 ; 2017b ; Yan et al. , 2018 ; Tulyakov et al. , 2018 ) , this 1We do not feed the predicted boundary map as input to the structure generator since it makes the structure generator prune to error propagation thus prevents extrapolation to long-term future . model is appropriate to synthesize the long-term future since it can handle both transformation of the existing objects ( via ŵt−1 ) and synthesis of the emerging objects ( via ht ) . To ensure both frame-level and video-level generation quality , the video synthesis network F is trained against a conditional image discriminator DI and a conditional video discriminator DV through adversarial learning by LI ( F , DI ) =EφI ( x≤T , s≤T ) [ logDI ( xi , si ) ] + EφI ( x̂≤T , s≤T ) [ log ( 1−DI ( x̂i , si ) ) ] , ( 4 ) LV ( F , DV ) =EφV ( w < T , x≤T , s≤T ) [ logDV ( xi−1 : i−τ′ , wi−2 : i−τ′ ) ] + EφV ( w < T , x̂≤T , s≤T ) [ log ( 1−DV ( x̂i−1 : i−τ′ , wi−2 : i−τ′ ) ) ] , ( 5 ) where LI ( F , DI ) and LV ( F , DV ) are frame-level and video-level adversarial losses , respectively . For efficient training , we follow Wang et al . ( 2018 ) to adopt sampling operators φI ( x≤T , s≤T ) = ( xi , si ) and φV ( w < T , x≤T , s≤T ) = ( wi−2 : i−τ′ , xi−1 : i−τ′ , si−1 : i−τ′ ) for frame and video-level adversarial learning objectives , respectively , where i is an integer sampled from U ( 1 , T ) and U ( τ ′ + 1 , T + 1 ) for frame sampling operator φI and video sampling operator φV , respectively . Then the final learning objective is formulated by min F max DI , DV LI ( F , DI ) + LV ( F , DV ) . ( 6 ) During training , the model is trained to estimate RGB frames from the GT structures , while the predictions from the structure generator are used during testing . See Appendix C.3 for more details .

Summary: This paper proposes a VAE based hierarchical model for video prediction. The model employs recurrent model to predict intermediate representations (in the form of label maps) and these representations are mapped to pixel level information, i.e., videos. The paper presents an interesting idea of using representations that do not use any domain knowledge. The authors demonstrate the value of modeling temporal evolution of these representations which enables long term video prediction.