Long History Short-Term Memory for Long-Term Video Prediction

While video prediction approaches have advanced considerably in recent years , learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions . To address this challenge , existing algorithms rely on extra supervision ( e.g. , action or object pose ) , motion flow learning , or adversarial training . In this paper , we propose a new recurrent unit , Long History Short-Term Memory ( LH-STM ) . LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies . To capture spatiotemporal dynamics in videos , we combined LH-STM with the Context-aware Video Prediction model ( ContextVP ) . Our experiments on the KTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions , but also farther into the future compared to the state-of-the-art methods . 1 INTRODUCTION . Learning the dynamics of an environment and predicting consequences in the future has become an important research problem . A common task is to train a model that accurately predicts pixel-level future frames conditioned on past frames . It can be utilized for intelligent agents to guide them to interact with the world , or for other video analysis tasks such as activity recognition . An important component of designing such models is how to effectively learn good spatio-temporal representations from video frames . The Convolutional Long Short-Term Memory ( ConvLSTM ) network ( Xingjian et al. , 2015 ) has been a popular model architecture choice for video prediction . However , recent stateof-the-art approaches produce high-quality predictions only for one or less then ten frames ( Lotter et al. , 2016 ; Villegas et al. , 2017a ; Byeon et al. , 2018 ) . Learning to predict long-term future video frames remains challenging due to 1 ) the presence of complex dynamics in high-dimensional video data , 2 ) prediction error propagation over time , and 3 ) inherent uncertainty of the future . Many recent works ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ) focus on the third issue by introducing stochastic models ; this issue is a crucial challenge for long-term prediction . However , the architectures currently in use are not sufficiently powerful and efficient for long-term prediction , and this is also an important but unsolved problem . The model needs to extract important information from spatio-temporal data and retain this information longer into the future efficiently . Otherwise , its uncertainty about the future will increase even if the future is completely predictable given the past . Therefore , in this paper , we attempt to address the issue of learning complex dynamics of videos and minimizing long-term prediction error by fully observing the history . We propose a novel modification of the ConvLSTM structure , Long History Short-Term Memory ( LH-STM ) . LH-STM learns to interconnect history states and the current input by a History SoftSelection Unit ( HistSSel ) and double memory modules . The weighted history states computed by our HistSSel units are combined with the history memory and then used to update the current states by the update memory . The proposed method brings the power of higher-order RNNs ( Soltani & Jiang , 2016 ) to ConvLSTMs , which have been limited to simple recurrent mechanisms so far . The HistSSel unit acts as a short-cut to the history , so the gradient flow in the LSTM is improved . More powerful RNNs are likely to be necessary to solve the hard and unsolved problem of long-term video prediction , which is extremely challenging for current architectures . Moreover , by disentangling the history and update memories , our model can fully utilize the long history states . This structure can better model long-term dependencies in sequential data . In this paper , the proposed modification is integrated into the ConvLSTM-based architectures to solve the long-term video prediction problem : Context-aware Video Prediction ( ContextVP ) model . The proposed models can fully leverage long-range spatio-temporal contexts in real-world videos . Our experiments on the KTH human actions and the BAIR robot pushing datasets show that our model produces sharp and realistic predictions for more frames into the future compared to recent state-of-the-art long-term video prediction methods . 2 RELATED WORK . Learning Long-Term Dependencies with Recurrent Neural Networks ( RNN ) : While Long ShortTerm Memory ( LSTM ) has been successful for sequence prediction , many recent approaches aim to capture longer-term dependencies in sequential data . Several works have proposed to allow dynamic recurrent state updates or to learn more complex transition functions . Chung et al . ( 2016 ) introduced the hierarchical multiscale RNN that captures a hierarchical representation of a sequence by encoding multiple time scales of temporal dependencies . Koutnik et al . ( 2014 ) modified the standard RNN to a Clockwork RNN that partitions hidden units and processes them at different clock speeds . Neil et al . ( 2016 ) introduced a new time gate that controls update intervals based on periodic patterns . Campos et al . ( 2017 ) proposed an explicit skipping module for state updates . Zilly et al . ( 2017 ) increased the recurrent transition depth with highway layers . Fast-Slow Recurrent Neural Networks ( Mujika et al. , 2017 ) incorporate ideas from both multiscale ( Schmidhuber , 1992 ; El Hihi & Bengio , 1996 ; Chung et al. , 2016 ) and deep transition ( Pascanu et al. , 2013 ; Zilly et al. , 2017 ) RNNs . The advantages of the above approaches are efficient information propagation through time , better long memory traces , and generalization to unseen data . Alternative solutions include the use of history states , an attention model or skip connections . Soltani & Jiang ( 2016 ) investigated a higher order RNN to aggregate more history information and showed that it is beneficial for long range sequence modeling . Cheng et al . ( 2016 ) deployed an attention mechanism in LSTM to induce relations between input and history states . Gui et al . ( 2018 ) incorporated into an LSTM , dynamic skip connections and reinforcement learning to model long-term dependencies . These approaches use the history states in a single LSTM by directly adding more recurrent connections or adding an attention module in the memory cell . These models are used for one dimensional sequence modeling , whereas our proposed approach separates the history and update memories that learn to encode the long-range relevant history states . Furthermore , our approach is more suitable for high-dimensional ( e.g. , video ) prediction tasks . Video Prediction : The main issue in long-term pixel-level video prediction is how to capture longterm dynamics and handle uncertainty of the future while maintaining sharpness and realism . Oh et al . ( 2015 ) introduced action-conditioned video prediction using a Convolutional Neural Network ( CNN ) architecture . Villegas et al . ( 2017b ) and Wichers et al . ( 2018 ) focused on a hierarchical model to predict long-term videos . Their model estimates high-level structure before generating pixel-level predictions . However , the approach by Villegas et al . ( 2017b ) requires object pose information as ground truth during training . Finn et al . ( 2016 ) used ConvLSTM to explicitly model pixel motions . To generate high-quality predictions , many approaches train with an adversarial loss ( Mathieu et al. , 2015 ; Wichers et al. , 2018 ; Vondrick et al. , 2016 ; Vondrick & Torralba , 2017 ; Denton et al. , 2017 ; Lee et al. , 2018 ) . Weissenborn et al . ( 2019 ) introduced local self-attention on videos directly for a large scale video precessing . Another active line of investigation is to train stochastic prediction models using VAEs ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ) . These models predict plausible futures by sampling latent variables and produce long-range future predictions . The Spatio-temporal LSTM ( Wang et al. , 2017 ; 2018a ) was introduced to better represent the dynamics of videos . This model is able to learn spatial and temporal representations simultaneously . Byeon et al . ( 2018 ) introduced a Multi-Dimensional LSTM-based approach ( Stollenga et al. , 2015 ) for video prediction . It contains directional ConvLSTM-like units that efficiently aggregate the entire spatio-temporal contextual information . Wang et al . ( 2018b ) recently proposed a memory recall function with 3D ConvLSTM . This work is the most related to our approach . It uses a set of cell states with an attention mechanism to capture the long-term frame interaction similar to the work of Cheng et al . ( 2016 ) . With 3D-convolutional operations , the model is able to capture short-term and long-term information flow . In contrast to this work , the attention mechanism in our model is used for a set of hidden state . We also disentangle the history and update memory cells to better memorize and restore the relevant information . By integrating a double memory LH-STM into the context-aware video prediction ( ContextVP ) model ( Byeon et al. , 2018 ) , our networks can capture the entire spatio-temporal context for a long-range video sequence . 3 METHOD . In this section , we first describe the standard ConvLSTM architecture and then introduce the LH-STM . Finally , we explain the ConvLSTM-based network architectures for multi-frame video prediction using the Context-aware Video Prediction model ( ContextVP ) ( Byeon et al. , 2018 ) . 3.1 CONVOLUTIONAL LSTM . Let Xn1 = { X1 , ... , Xn } be an input sequence of length n. Xk ∈ Rh×w×c is the k-th frame , where k ∈ { 1 , ... , n } , h is the height , w the width , and c the number of channels . For the input frame Xk , a ConvLSTM unit computes the current cell and hidden states ( Ck , Hk ) given the cell and hidden states from the previous frame , ( Ck−1 , Hk−1 ) : Ck , Hk = ConvLSTM ( Xk , Hk-1 , Ck-1 ) , ( 1 ) by computing the input , forget , output gates ik , fk , ok , and the transformed cell state Ĉk : ik = σ ( Wi ∗Xk +Mi ∗Hk-1 + bi ) , fk = σ ( Wf ∗Xk +Mf ∗Hk-1 + bf ) , ok = σ ( Wo ∗Xk +Mo ∗Hk-1 + bo ) , Ĉk = tanh ( Wĉ ∗Xk +Mĉ ∗Hk-1 + bĉ ) , Ck = fk Ck-1 + ik Ĉk , Hk = ok tanh ( Ck ) , ( 2 ) where σ is the sigmoid function , W and M are 2D convolutional kernels for input-to-state and state-to-state transitions , ( ∗ ) is the convolution operation , and ( ) is element-wise multiplication . The size of the weight matrices depends on the size of convolutional kernel and the number of hidden units . 3.2 LONG HISTORY SHORT-TERM MEMORY ( LH-STM ) . LH-STM is an extension of the standard ConvLSTM model by integrating a set of history states into the LSTM unit . The history states include the spatio-temporal context of each frame in addition to the pixel level information in the frame itself . Figure 1 illustrates the differences between a standard RNN , Higher-Order RNN ( Soltani & Jiang , 2016 ) , and our proposed model ( Double and Single LH-STM ) . History Soft-Selection Unit ( HistSSel ) : The HistSSel unit computes the relationship between the recent history and the earlier ones using dot-product similar to ( Vaswani et al. , 2017 ) 1 . This mechanism can be formulated as SoftSel ( Q , K , V ) = softmax ( WQQ ·WQK ) ·WQV . It consists of queries ( Q ) , keys ( K ) and values ( V ) . It computes the dot products of the queries and the keys ; and then applies the a softmax function . Finally , the values ( V ) are weighted by the outputs of softmax function . The queries , keys , and values can be optionally transformed by the WQ , WK , and WV matrices . Using this mechanism , HistSSel computes the relationship between the last hidden state Hk-1 and the earlier hidden states Hk-m : k-2 at time step k ( See Figure 2 ) . Hk-m : k-2 is the set of previous hidden states , ( Hk-m , Hk-m-1 , · · ·Hk-3 , Hk-2 ) . We will show in Section 4.2 the benefit of using history states versus using the past input frames directly . The history soft-selection mechanism can be formulated as follows : HistSSel ( Hk-1 , Hk-m : k-2 ) = n∑ i=2 softmaxi ( H̃ Q k-1 · H̃ K k-i ) · H̃Vk-i , H̃ji =W j i Hi + b j i , j ∈ { Q , K , V } . 2 ( 3 ) Single LH-STM : A simple way to employ HistSSel in ConvLSTM ( Equation 1 ) is to add HistSSel ( Hk-1 , Hk-m : k-2 ) in addition to the input and the previous states ( Xk , Hk-1 , Ck-1 ) in the Equation 6 Figure 1c shows a diagram of the computation . This direct extension is named Single LH-STM : Hk = SingleConvLSTM ( Xk , Hk-1 , Ck-1 , HistSSel ( Hk-1 , Hk-m : k-2 ) ) . ik = σ ( Wi ∗Xk +Mi ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + bi ) , fk = σ ( Wf ∗Xk +Mf ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + bf ) , ok = σ ( Wo ∗Xk +Mo ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + bo ) Ĉk = tanh ( Wĉ ∗Xk +Mĉ ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + bĉ ) , Ck = ik Ĉk + fk Ck-1 , Hk = ok tanh ( Ck ) . ( 4 ) Double LH-STM : To effectively learn dynamics from the history states , we propose Double LH-STM . It contains two ConvLSTM blocks , History LSTM ( H-LSTM ) and Update LSTM ( U-LSTM ) . The goal of the Double LH-STM is to explicitly separate the long-term history memory and the update memory . By disentangling these , the model can better encode complex long-range history and keep track of the their dependencies . Figure 1d illustrates a diagram of Double LH-STM . 1The purpose of this unit is to compute the importance of each history state . While we used a selfattention-like unit in this paper , this can be achieved with other common layers as well , e.g. , fully connected or convolutional layers . 2bji can be omitted . The H-LSTM block explicitly learns the complex transition function from the ( possibly entire ) set of past hidden states , Hk-m : k-1 , 1 < m < k. If m = k , H-LSTM incorporates the entire history up to the time step k − 1 . The U-LSTM block updates the states Hk and Ck for the time step k , given the input Xk , previous cell state Ck−1 , and the output of the H-LSTM , H ′k−1 . The History-LSTM ( H-LSTM ) and Update-LSTM ( U-LSTM ) can be formulated as : H-LSTM i′k-1 = σ ( M ′ i ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + b′i ) , f ′k-1 = σ ( M ′ f ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + b′f ) , o′k-1 = σ ( M ′ o ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + b′o ) Ĉ′k-1 = tanh ( M ′ ĉ ∗Hk-1 + HistSSel ( Hk-1 , Hk-m : k-2 ) + b′ĉ ) , C′k-1 = f ′ k−1 C′k-2 + i′k−1 Ĉ′k−1 , H ′k-1 = o ′ k-1 tanh ( c′k-1 ) . ( 5 ) U-LSTM ik = σ ( Wi ∗Xk +Mi ∗H ′k-1 + bi ) , fk = σ ( Wf ∗Xk +Mf ∗H ′′k-1 + bf ) , ok = σ ( Wo ∗Xk +Mo ∗H ′k-1 + bo ) , Ĉk = tanh ( Wĉ ∗Xk +Mĉ ∗H ′′k-1 + bĉ ) , Ck = fk Ck-1 + ik Ĉk , Hk = ok tanh ( Ck ) . ( 6 )

Summary: The paper proposes a type of recurrent neural network module called Long History Short-Term Memory (LH-STM) for longer-term video generation. This module can be used to replace ConvLSTMs in previously published video prediction models. It expands ConvLSTMs by adding a "previous history" term to the ConvLSTM equations that compute the IFO gates and the candidate new state. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism. As such, it is not clear if there are significant differences between LH-STMs and previously proposed LSTMs with attention on previous hidden states. The authors propose recurrent units that include one or two History Selection (soft-attention) steps, called single LH-STM and double LH-STM respectively. The exact formulation of the double LH-STM is not clear from the paper.  The authors then propose to use models with LH-STM units for longer term video generation. They claim that LH-STM can better reduce error propagation and better model the complex dynamics of videos. To support the claims, they conduct empirical experiments where they show that the proposed model outperforms previous video prediction models on KTH (up to 80 frames) and the BAIR Push dataset (up to 25 frames).