Title: LONG HISTORY SHORT-TERM MEMORY FOR LONG-TERM VIDEO PREDICTION

Abstract: While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging -ambiguous future or error propagation over time yield blurry predictions. To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion flow learning, or adversarial training. In this paper, we propose a new recurrent unit, Long History Short-Term Memory (LH-STM). LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies. To capture spatiotemporal dynamics in videos, we combined LH-STM with the Context-aware Video Prediction model (ContextVP). Our experiments on the KTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the-art methods.

1. INTRODUCTION:

Learning the dynamics of an environment and predicting consequences in the future has become an important research problem. A common task is to train a model that accurately predicts pixel-level future frames conditioned on past frames. It can be utilized for intelligent agents to guide them to interact with the world, or for other video analysis tasks such as activity recognition. An important component of designing such models is how to effectively learn good spatio-temporal representations from video frames. The Convolutional Long Short-Term Memory (ConvLSTM) network (Xingjian et al., 2015) has been a popular model architecture choice for video prediction. However, recent stateof-the-art approaches produce high-quality predictions only for one or less then ten frames (Lotter et al., 2016; Villegas et al., 2017a; Byeon et al., 2018) . Learning to predict long-term future video frames remains challenging due to 1) the presence of complex dynamics in high-dimensional video data, 2) prediction error propagation over time, and 3) inherent uncertainty of the future.

Many recent works (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018) focus on the third issue by introducing stochastic models; this issue is a crucial challenge for long-term prediction. However, the architectures currently in use are not sufficiently powerful and efficient for long-term prediction, and this is also an important but unsolved problem. The model needs to extract important information from spatio-temporal data and retain this information longer into the future efficiently. Otherwise, its uncertainty about the future will increase even if the future is completely predictable given the past. Therefore, in this paper, we attempt to address the issue of learning complex dynamics of videos and minimizing long-term prediction error by fully observing the history.

We propose a novel modification of the ConvLSTM structure, Long History Short-Term Memory (LH-STM). LH-STM learns to interconnect history states and the current input by a History Soft-Selection Unit (HistSSel) and double memory modules. The weighted history states computed by our HistSSel units are combined with the history memory and then used to update the current states by the update memory. The proposed method brings the power of higher-order RNNs (Soltani & Jiang, 2016) to ConvLSTMs, which have been limited to simple recurrent mechanisms so far. The HistSSel unit acts as a short-cut to the history, so the gradient flow in the LSTM is improved. More powerful RNNs are likely to be necessary to solve the hard and unsolved problem of long-term video prediction, which is extremely challenging for current architectures. Moreover, by disentangling the history and update memories, our model can fully utilize the long history states.

This structure can better model long-term dependencies in sequential data. In this paper, the proposed modification is integrated into the ConvLSTM-based architectures to solve the long-term video prediction problem: Context-aware Video Prediction (ContextVP) model. The proposed models can fully leverage long-range spatio-temporal contexts in real-world videos. Our experiments on the KTH human actions and the BAIR robot pushing datasets show that our model produces sharp and realistic predictions for more frames into the future compared to recent state-of-the-art long-term video prediction methods.

2. RELATED WORK:

Learning Long-Term Dependencies with Recurrent Neural Networks (RNN): While Long Short-Term Memory (LSTM) has been successful for sequence prediction, many recent approaches aim to capture longer-term dependencies in sequential data. Several works have proposed to allow dynamic recurrent state updates or to learn more complex transition functions. Chung et al. (2016) introduced the hierarchical multiscale RNN that captures a hierarchical representation of a sequence by encoding multiple time scales of temporal dependencies. Koutnik et al. (2014) modified the standard RNN to a Clockwork RNN that partitions hidden units and processes them at different clock speeds. Neil et al. (2016) introduced a new time gate that controls update intervals based on periodic patterns. Campos et al. (2017) proposed an explicit skipping module for state updates. Zilly et al. (2017) increased the recurrent transition depth with highway layers. Fast-Slow Recurrent Neural Networks (Mujika et al., 2017) incorporate ideas from both multiscale (Schmidhuber, 1992; El Hihi & Bengio, 1996; Chung et al., 2016) and deep transition (Pascanu et al., 2013; Zilly et al., 2017) RNNs. The advantages of the above approaches are efficient information propagation through time, better long memory traces, and generalization to unseen data.

Alternative solutions include the use of history states, an attention model or skip connections. Soltani & Jiang (2016) investigated a higher order RNN to aggregate more history information and showed that it is beneficial for long range sequence modeling. Cheng et al. (2016) deployed an attention mechanism in LSTM to induce relations between input and history states. Gui et al. (2018) incorporated into an LSTM, dynamic skip connections and reinforcement learning to model long-term dependencies. These approaches use the history states in a single LSTM by directly adding more recurrent connections or adding an attention module in the memory cell. These models are used for one dimensional sequence modeling, whereas our proposed approach separates the history and update memories that learn to encode the long-range relevant history states. Furthermore, our approach is more suitable for high-dimensional (e.g., video) prediction tasks.

Video Prediction::

The main issue in long-term pixel-level video prediction is how to capture longterm dynamics and handle uncertainty of the future while maintaining sharpness and realism. Oh et al. (2015) introduced action-conditioned video prediction using a Convolutional Neural Network (CNN) architecture. Villegas et al. (2017b) and Wichers et al. (2018) focused on a hierarchical model to predict long-term videos. Their model estimates high-level structure before generating pixel-level predictions. However, the approach by Villegas et al. (2017b) requires object pose information as ground truth during training. Finn et al. (2016) used ConvLSTM to explicitly model pixel motions. To generate high-quality predictions, many approaches train with an adversarial loss (Mathieu et al., 2015; Wichers et al., 2018; Vondrick et al., 2016; Vondrick & Torralba, 2017; Denton et al., 2017; Lee et al., 2018) . Weissenborn et al. (2019) introduced local self-attention on videos directly for a large scale video precessing. Another active line of investigation is to train stochastic prediction models using VAEs (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018) . These models predict plausible futures by sampling latent variables and produce long-range future predictions.

The Spatio-temporal LSTM (Wang et al., 2017; 2018a) was introduced to better represent the dynamics of videos. This model is able to learn spatial and temporal representations simultaneously. Byeon et al. (2018) introduced a Multi-Dimensional LSTM-based approach (Stollenga et al., 2015) for video prediction. It contains directional ConvLSTM-like units that efficiently aggregate the entire spatio-temporal contextual information. Wang et al. (2018b) recently proposed a memory recall function with 3D ConvLSTM. This work is the most related to our approach. It uses a set of cell states with an attention mechanism to capture the long-term frame interaction similar to the work of Cheng et al. (2016) . With 3D-convolutional operations, the model is able to capture short-term and long-term information flow. In contrast to this work, the attention mechanism in our model is used for a set of hidden state. We also disentangle the history and update memory cells to better memorize and restore the relevant information. By integrating a double memory LH-STM into the context-aware video prediction (ContextVP) model (Byeon et al., 2018) , our networks can capture the entire spatio-temporal context for a long-range video sequence.

3. METHOD:

In this section, we first describe the standard ConvLSTM architecture and then introduce the LH-STM. Finally, we explain the ConvLSTM-based network architectures for multi-frame video prediction using the Context-aware Video Prediction model (ContextVP) (Byeon et al., 2018) .

3.1 CONVOLUTIONAL LSTM Let X n 1 = {X 1 , ..., X n } be an input sequence of length n. X k ∈ R h×w×c is the k-th frame,

where k ∈ {1, ..., n}, h is the height, w the width, and c the number of channels. For the input frame X k , a ConvLSTM unit computes the current cell and hidden states (C k , H k ) given the cell and hidden states from the previous frame, (C k-1 , H k-1 ):

EQUATION

by computing the input, forget, output gates i k , f k , o k , and the transformed cell state Ĉk :

EQUATION

where σ is the sigmoid function, W and M are 2D convolutional kernels for input-to-state and state-to-state transitions, ( * ) is the convolution operation, and ( ) is element-wise multiplication.

The size of the weight matrices depends on the size of convolutional kernel and the number of hidden units.

3.2. LONG HISTORY SHORT-TERM MEMORY (LH-STM):

LH-STM is an extension of the standard ConvLSTM model by integrating a set of history states into the LSTM unit. The history states include the spatio-temporal context of each frame in addition to the pixel level information in the frame itself. Figure 1 illustrates the differences between a standard RNN, Higher-Order RNN (Soltani & Jiang, 2016) , and our proposed model (Double and Single LH-STM).

…. History Soft-Selection Unit (HistSSel): The HistSSel unit computes the relationship between the recent history and the earlier ones using dot-product similar to (Vaswani et al., 2017) 1 . This mechanism can be formulated as SoftSel

H k-1 H k-2 H k-3 x k-2 xk x k-1 x k (a) Standard RNN …. x k-2 xk H k-1 H k-2 H k-3 x k-1 x k (b) Higher-Order RNN …. H k-1 H k-2 H k-3 x k-2 xk x k-1 x k (c) Single LH-STM …. H k-2 H k-3 H k -1 H ' k -1 x k-2 xk x k-1 x k (d) Double LH-STM

H k-m H k-2 H k-m V H k-1 H k-2 V H k-m K H k-2 K …. …. …. H k-1 Q H k-1 Q …. Softmax H k-m : k-2 W k-m V W k-m K W k-2 V W k-2 K W k-1 Q Queries (Q) Keys (K) Values (V) ( H k -1 , H k -m : k -2 ) HistSSel Dot-product Dot-product

(Q, K, V ) = softmax(W Q Q • W Q K) • W Q V . It consists of queries (Q), keys (K)

and values (V). It computes the dot products of the queries and the keys; and then applies the a softmax function. Finally, the values (V) are weighted by the outputs of softmax function. The queries, keys, and values can be optionally transformed by the W Q , W K , and W V matrices.

Using this mechanism, HistSSel computes the relationship between the last hidden state H k-1 and the earlier hidden states H k-m:k-2 at time step k (See Figure 2 ).

H k-m:k-2 is the set of previous hidden states, (H k-m , H k-m-1 , • • • H k-3 , H k-2

). We will show in Section 4.2 the benefit of using history states versus using the past input frames directly. The history soft-selection mechanism can be formulated as follows:

HistSSel(H k-1 , H k-m:k-2 ) = n i=2 softmax i ( HQ k-1 • HK k-i ) • HV k-i , Hj i = W j i H i + b j i , j ∈ {Q, K, V }. 2

(3)

Single LH-STM: A simple way to employ HistSSel in ConvLSTM (Equation 1) is to add HistSSel(H k-1 , H k-m:k-2 ) in addition to the input and the previous states (X k , H k-1 , C k-1 ) in the Equation 6Figure 1c shows a diagram of the computation. This direct extension is named Single LH-STM: The U-LSTM block updates the states H k and C k for the time step k, given the input X k , previous cell state C k-1 , and the output of the H-LSTM, H k-1 .

EQUATION

The History-LSTM (H-LSTM) and Update-LSTM (U-LSTM) can be formulated as: (Hochreiter et al., 2001) and allow long-term prediction: (a) between previous and the current recurrent layers (blue dotted line in Figure 9 ) and (b) across layers (green solid line in Figure 9 ). See Appendix B for the effectiveness of both skip connections.

EQUATION

ContextVP-4 (Byeon et al., 2018) consists of 4 layers of five directional Parallel Multi-Dimensional LSTM (PMD) units (Stollenga et al., 2015) along the h+, h-, w+, w-, and trecurrence directions. A PMD unit along the time direction (t-) is mathematically equivalent to a standard ConvLSTM (Equation 6). By using LSTM connectivities across the temporal and spatial dimensions, each processing layer covers the entire context in a video. Following Byeon et al. (2018) , we also included weighted blending, directional weight sharing (DWS), and two skip connections between the layers 1 -3 and 2 -4 (see Figure 9 ). See Appendix B for details of the network architectures.

Loss: The loss function to train the networks is the combination of L1-norm, L2-norm, and perceptual loss (Johnson et al., 2016; Zhang et al., 2018) :

EQUATION

where Y and X are the target and the predicted frames, respectively. λ is a weight for each function. Lp-norm is defined as L p (Y, X) = ||Y -X|| p . p = 1 for L1-norm, and p = 2 for L2-norm. The perceptual loss computes the cosine distance between the feature maps extracted from a VGG-16 network pre-trained on the ImageNet dataset (Simonyan & Zisserman, 2014) as

L pl (y, x) = 1 -1 l l 1 h l ×w l h l ,w l (φ(y) l • φ(x) l )

, where φ(y) l and φ(x) l are the feature maps of the target and the predicted frames Y and X, respectively at the layer l. The size of the feature map at layer l is h l × w l .

4. EXPERIMENTS:

We evaluated our LH-STM model on the KTH human actions (Laptev et al., 2004) and BAIR actionfree robot pushing (Ebert et al., 2017) datasets. Videos in the KTH human actions dataset consist of six human actions including walking, jogging, running, boxing, hand waving, and hand clapping. The BAIR action-free robot pushing dataset includes randomly moving robotic arms, which push objects on a table. The videos have a resolution of 64 × 64 pixels. We show quantitative comparisons by computing frame-wise Structural Similarity (SSIM) (Wang et al., 2004) , Peak Signal-to-Noise Ratio (PSNR), Visual Information Fidelity (VIF) (Sheikh & Bovik, 2005) , and LPIPS (Zhang et al., 2018) between the ground truth and generated video sequences.

4.1. KTH HUMAN ACTIONS DATASET:

We follow the experimental setup of Lee et al. (2018) . We evaluate our methods on 128 × 128 pixels for 40 frame prediction (Table 2 ) We train all models to predict the next 10 frames, given 10 prior ones. For testing, all models recursively predict 40 frames.

Our Double LH-STM outperforms all competing methods on all metrics including the base model (ContextVP). Our Double LH-STM model consistently outperforms other methods for the 40 predicted frames. The improvement is larger for longer-term frame prediction into the future (40 frames). Qualitative results can be found in Figure 7 and Appendix E (Figure 14 and Figure 15 ). These results show that motion and human shape predicted by Double LH-STM are the closest to the ground truth.

Our models are compared with FRNN (Oliu et al., 2018) , PredRNN (Wang et al., 2017) , Pre-dRNN++ (Wang et al., 2018a) , E3D-LSTM (Wang et al., 2018b) , SAVP-VAE and SAVP-Deterministic models (Lee et al., 2018) 3 . Our model outperforms all other methods.

…. 

H k-1 H k-2 H k-3 x k-2 xk x k-1 x k …. H k-2 H k-3 H k -1 H ' k -1 x k-2 xk x k-1 x k

4.2. STATE HISTORY VS INPUT HISTORY:

The proposed LH-STM models use past hidden states. Alternatively, input frames can be directly applied to LH-STM as illustrated in Figure 3 . We compared the performance of using input and state history with the same LH-STM models and the same model size in Table 1 on the KTH human actions dataset. Figure 7 shows a qualitative comparison of using state history (Double LH-STM) and input history (Double, input history). The results show the importance of using history states with LH-STM and that the state history is more suitable for videos.

4.3. LONGER PREDICTION:

To test the performance of longer-term prediction (more than 40 frames), we additionally trained the KTH human actions dataset with a resolution of 64 × 64 pixels4 . Figure 4 and Figure 5 compares the performance of 80 frame prediction with Double LH-STM and two state-of-the-art models ( 3 SAVP-VAE and SAVP-Deterministic (Lee et al., 2018) ) on all action classes and three action classes only (hand waving, hand clapping, and boxing), respectively. We compare only these three action classes with all action classes to show that the high performance at longer frames is not due to background copy. The human in other action class videos (running, jogging, and walking) may disappear after certain frames. Additionally, Figure 13 shows the performance comparisons of each action class. The experimental setup is the same as the experiments of a resolution 128 × 128 (Section 4.1).

From these comparisons, we can observe that Double LH-STM results are generally better than SAVP-Deterministic and -VAE on all metrics. These also show that our model is not just copying the background when predicting more than 40 frames. Our experimental setup follows that of Lee et al. (2018) . All models are conditioned on five frames to predict the next ten frames5 . The models are tested to predict 25 frames. We evaluated our LH-STM models (Single and Double) and compared them with the base model (ContextVP), the 3 SAVP-VAE, and SAVP-Deterministic models (Lee et al., 2018) in Table 3 . Our Double LH-STM achieved better PSNR and SSIM scores than the SAVP-Deterministic and SAVP-VAE models. It can be seen from the per-frame prediction results (Figure 6 ), that the performance LH-STM is higher over time, especially on the PSNR and SSIM metrics. On the LPIPS metric, the performance of Double LH-STM is comparable to SAVP-Deterministic and better than SAVP-VAE. Qualitative results can be found in Appendix E (Figure 17 and Figure 18 ). This dataset contains random robot motion which makes long-term prediction extremely hard with deterministic models. However, the overall motion prediction with Double LH-STM and SAVP-Deterministic is still reasonable compared to SAVP-VAE. 

Method PSNR SSIM VIF # parameters:

ConvLSTM (Xingjian et al., 2015) 22.85 0.639 --FRNN (Oliu et al., 2018) 23.77 0.678 --PredRNN (Wang et al., 2017) 24.16 0.703 --PredRNN++ (original) (Wang et al., 2018a) 25.21 0.741 -15.1M *PredRNN++ (retrained) (Wang et al., 2018a) 26.37 0.734 -15.1M 3 SAVP-VAE (retrained) (Lee et al., 2018) 26.23 0.777 0.289 9.8M 3 SAVP-Det (retrained) (Lee et al., 2018) 27.59 0.796 0.328 8.9M E3D-LSTM (Wang et al., 2018b) 27.24 0.810 --ContextVP-4 (base model) (Byeon et al., 2018) 

5. CONCLUSION:

In this paper, we have introduced a new RNN structure, LH-STM, which incorporates long history states into the recurrent unit to learn longer range dependencies. This module is combined with the ContextVP architecture allowing context aware long-term video prediction. The resulting model produces more realistic predictions for a longer period of time compared to the recent state-of-the-art methods. The proposed approach can easily be extended to standard LSTM or Gated Recurrent Units (GRU) and applied to tasks that require a long-term memory trace, e.g., language modeling or speech recognition. 8 ). The parameters of opposite directions are shared. We followed the same network architecture as Byeon et al. (2018) except for the number of neurons. 

EQUATION

)

where H is the output of the PMD units from h+, h-, w+, w-, and tdirections, and W is the weight matrix.

Directional Weight-Sharing: Weights and biases of the PMD units in opposite directions (i.e., the h+ and hdirections, w+ and wdirections) are shared.

X k-2:

Xk-1 Skip connection in a recurrent block (R-Skip): A skip connection is commonly used in LSTM layers (Melis et al., 2017; Merity et al., 2017; Wang et al., 2018a) for long-term gradient propagation.

LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM LH-STM Xk Xk+1 X k -1 X k LH-STM

Our proposed model concatenates outputs of the previous and current LSTM layers (see Figure 9 ).

Gated skip connection across layers (L-Skip): Unlike in (Byeon et al., 2018) , We added the gated skip connection across layers. In our model, the multiplicative gate is added to control the flow of information across layers. It stabilizes learning as depth increases. A layer is formulated as:

EQUATION

where

H l = X l • W H l + b H and T l = σ(X l • W T l + b T

). X l is the input at the l th layer, T is a gate, and H l is the transformation of the input at the layer l.

We adjusted the number of neurons, so all models have similar model size (≈ 17M). LH-STM Double contains 64-64-128-128 neurons for the input size of 64 × 64 pixels and 64-128-128-128 for 128 × 128. LH-STM Single contains 128-128-128-128 neurons for the input size of 64 × 64 pixels and 64-128-128-256 for 128 × 128.

The initial learning rate is 1e -4 found by grid search between 1e -3 and 1e -5. The learning rate is decayed every 10 epochs with the decay rate 0.95. Weights are initialized using the Xavier's normalized initializer (Glorot & Bengio, 2010) and the recurrent states are all initialized to zero.

B ABLATION STUDY:

We performed a series of ablation studies to evaluate the importance of each component of our proposed model 1) RNN type (standard ConvLSTM, Single LH-STM, or Double LH-STM); 2) different losses (L 1 , L 2 , and the perceptual loss); adding skip connections 3) in recurrent blocks; and 4) across layers (see Section 3.3 for details). We evaluated performance on the KTH human actions dataset with a resolution of 64 × 64 pixels. The weights for the loss, λ 1 , λ 2 , and λ pl , are tested between zero to one. The results of this experiment are summarized in Table 4 . We found that performance improves by adding skip connections in recurrent blocks and across layers. Double LH-STM outperforms the standard ConvLSTM and Single LH-STM. Combination of L 1 , L 2 , and L pl also improves the performance (with λ 1 = 1, λ 2 = 1, λ pl = 1).

C EVALUATION OF STOCHASTIC MODELS:

SAVP-VAE numbers in the paper by Lee et al. (2018) are obtained by first generating 100 output samples per input and then selecting the best generated sample for each input based on the highest score between the output and the ground truth. This evaluation measures the best prediction the model can do, given 100 tries. However, it does not provide an overall picture of the generations from the model. In real world scenarios, the ground truth is not available, so the 'best' samples cannot be picked as done in such evaluations. Therefore, we re-computed the scores for stochastic models based on their 'median' output sample (instead of best) among the 100 randomly generated ones, compared to the ground truth. This strategy measures how well the model can be expected to perform on average, and so it is a more representative score. 

D EVALUATION WITH LARGE AND SMALL MOTION:

We split the video samples into five bins by motion magnitude computed as averaged L2 norm between target frames similar to the evaluations by Villegas et al. (2017a) . Figure 12 shows the comparisons on the KTH human action dataset, and ?? shows on BAIR action free robot pushing dataset. Overall, across all metrics, all models perform worse when the motion is larger. For PSNR and SSIM, Double LH-STM achieves the best performance except for the largest motion. For LPIPS, Double LH-STM performs better for small motions (first two bins). SAVP-Deterministic performs the best on this metric for larger motions.

E QUALITATIVE RESULTS:

Figure 14 and Figure 15 show prediction results on the KTH human actions dataset, and Figure 17 and Figure 18 presents the results on the BAIR action free robot pushing dataset. Figure 16 and Figure 19 include the samples with one of the largest motions for both datasets. When motion is larger, all models fail to produce reasonable predictions, whether they are deterministic or stochastic.

F RETRAINING PREDRNN++:

As mentioned in the paper, we re-trained PredRNN++ on the KTH human actions dataset using the provided code by Wang et al. (2018a) (https://github.com/Yunbo426/predrnn-pp).

We conducted grid hyper-parameter search for the learning rate and the patch size. We found in the code that there is an option to divide the input frames into smaller patches. The learning rate search was performed in the range between 1e -3 and 1e -4. The range of patch size was 1, 2, 4 for both frame resolutions (64 × 64 and 128 × 128 pixels). The reported results in the paper are for a learning rate of 1e -4 and the patch size 1 (full resolution). (2017a) . Top row shows the ground truth, and the rest is the prediction results. The model is trained for 10 frames and predicted for 40 frames given 10 frames. When motion is larger, all models fail to produce reasonable predictions.

:

The purpose of this unit is to compute the importance of each history state. While we used a selfattention-like unit in this paper, this can be achieved with other common layers as well, e.g., fully connected or convolutional layers.

2 b ji can be omitted.

We note thatLee et al. (2018) reported the performance of VAE+GAN, VAE only, GAN only, and of their deterministic models. We select their best (VAE only) and the most comparable (deterministic) models to compare against. To evaluate the VAE model, we take 100 random input samples for each video followingLee et al. (2018). The reported numbers in this paper are for their method's "median" output among the 100 randomly generated ones, compared to the ground truth. Please see Appendix C for the discussion of this evaluation setup

Due to lack of GPU memory, we could not use the original resolution for 80 frame prediction.

Lee et al. (2018) used two input frames. We increase the number of input frames as motion in two frames is almost not visible. We increased the number of input frames to provide enough history to the model. We also re-trained the models ofLee et al. (2018) with five input frames.

Figures and Tables:

Figure 2: A graphical illustration of the History Soft-Selection Unit (HistSSel). It computes the relationship between the last H k-1 and the earlier hidden state H k-m:k-2 .

Double LH-STM:To effectively learn dynamics from the history states, we propose Double LH-STM. It contains two ConvLSTM blocks, History LSTM (H-LSTM) and Update LSTM (U-LSTM). The goal of the Double LH-STM is to explicitly separate the long-term history memory and the update memory. By disentangling these, the model can better encode complex long-range history and keep track of the their dependencies. Figure1dillustrates a diagram of Double LH-STM.The H-LSTM block explicitly learns the complex transition function from the (possibly entire) set of past hidden states, H k-m:k-1 , 1 < m < k. If m = k, H-LSTM incorporates the entire history up to the time step k -1.

Figure 6: Per-frame comparisons on BAIR robot pushing dataset (25 frame prediction) (•): averaged score

Figure 7: Qualitative comparison of Double LH-STM, Single LH-STM, Double LH-STM with input history (See Section 4.2 for the details), the base model (ContextVP (Byeon et al., 2018)), and the state-of-the-art model (SAVP-VAE and SAVP-Deterministic (Lee et al., 2018)). Double LH-STM in overall produces the sharper images and greater realism compared to other models. Compare to the Deterministic model (Lee et al., 2018), our model can capture more precise motion.

Figure 8: ContextVP-4 network archtecture

Figure 9: A graphical illustration of skip connections for the direction t-. Blue dotted lines indicate concatenation-based skip connection in the LH-STM block. A green solid line indicates a gated skip connection across the 1-3 and 2-4 layers.

Figure 11: Per-frame comparisons of BAIR robot pushing dataset with motion-based test set separation. We split the video samples into five bins by motion magnitude computed as averaged L2 norm between target frames following Villegas et al. (2017a). The first row is with smallest motion and the bottom row with largest motion. (•) indicates averaged score.

Comparison of using input vs state history with LH-STM Single and Double on the KTH human actions dataset (resolution 128 × 128).

Table content: <table><tr><td colspan="3">History LH-STM PSNR SSIM</td><td>VIF</td></tr><tr><td>Input</td><td>Single Double</td><td colspan="2">25.78 0.779 0.302 26.45 0.788 0.308</td></tr><tr><td>State</td><td>Single Double</td><td colspan="2">27.83 0.816 0.354 28.64 0.826 0.367</td></tr><tr><td>Figure 3: The use of input history with LH-STM</td><td/><td/></tr><tr><td>Single (left) and Double (right)</td><td/><td/></tr></table>

Evaluation of multi-frame prediction on the BAIR robot pushing dataset. All models are trained for 10 frame prediction given 5 input frames. They are then tested to predict 25 future frames. The reported numbers are averages across the predicted frames. Higher values indicate better results.

Table content: <table><tr><td>Method</td><td>PSNR SSIM</td><td>VIF</td><td># parameters</td></tr><tr><td>3 SAVP-VAE (Lee et al., 2018)</td><td colspan="2">17.32 0.779 0.524</td><td>6.7M</td></tr><tr><td colspan="3">3 SAVP-Deterministic (Lee et al., 2018) 18.08 0.801 0.544</td><td>6.4M</td></tr><tr><td>ContextVP-4 (Byeon et al., 2018)</td><td colspan="2">18.24 0.817 0.513</td><td>17.3M</td></tr><tr><td>Single LH-STM (ours)</td><td colspan="2">18.64 0.817 0.523</td><td>17.6M</td></tr><tr><td>Double LH-STM (ours)</td><td colspan="2">18.82 0.825 0.547</td><td>16.3M</td></tr></table>

An ablation study on KTH human actions dataset. The frame resolution is 64 × 64 pixels. All models are trained to predict 10 future frames given 10 input frames. They are then tested to recursively predict 40 future frames. The reported numbers are averages over all the predicted frames.

Table content: <table><tr><td>RNN type</td><td>Loss</td><td>R-Skip L-Skip PSNR SSIM</td></tr><tr><td/><td>L 1</td><td>27.74 0.812</td></tr><tr><td>ConvLSTM</td><td>L 1</td><td>27.96 0.812</td></tr><tr><td/><td>L 1</td><td>28.41 0.812</td></tr><tr><td>Single LH-STM</td><td>L 1 L 1 +L pl</td><td>28.59 0.818 28.46 0.817</td></tr><tr><td/><td>L 1</td><td>28.69 0.812</td></tr><tr><td>Double LH-STM</td><td>L 1 +L pl L 1 +L 2</td><td>28.20 0.811 29.01 0.818</td></tr><tr><td/><td>L 1 +L 2 +L pl</td><td>29.05 0.820</td></tr></table>

Peer review (iclr2020):
The paper proposes a type of recurrent neural network module called Long History Short-Term Memory (LH-STM) for longer-term video generation. This module can be used to replace ConvLSTMs in previously published video prediction models. It expands ConvLSTMs by adding a *previous history* term to the ConvLSTM equations that compute the IFO gates and the candidate new state. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism. As such, it is not clear if there are significant differences between LH-STMs and previously proposed LSTMs with attention on previous hidden states. The authors propose recurrent units that include one or two History Selection (soft-attention) steps, called single LH-STM and double LH-STM respectively. The exact formulation of the double LH-STM is not clear from the paper. The authors then propose to use models with LH-STM units for longer term video generation. They claim that LH-STM can better reduce error propagation and better model the complex dynamics of videos. To support the claims, they conduct empirical experiments where they show that the proposed model outperforms previous video prediction models on KTH (up to 80 frames) and the BAIR Push dataset (up to 25 frames). Overall I believe there are serious flaws with the paper that prevent acceptance in its current form. First, I believe the paper starts from the wrong assumption, namely that current video prediction models are limited by their capacity to limit the propagation of errors and to capture complex dynamics. Instead, it is well known that the main difficulty for longer term video prediction is to manage the increasing uncertainty in future outcomes. Stochastic models such as SVG-LP or SAVP are currently the state-of-the-art in video generation, with deterministic models not being able to generate more than a few non-blurry frames of video. While the authors mention that they do not focus on future uncertainty here, it is not clear how the proposed model helps to generate better longer-term videos when it does not deal with what actually makes long-term video generation difficult. In addition, it*s misleading to claim that current models produce high quality generations for *only one or less than ten frames*, especially without defining high quality. Models such as SVG [1] or SAVP[2] can produce non-blurry videos for 30-100 frames for the BAIR dataset, for example. The experiments are missing 1) SVG as a baseline, 2) metrics that correlate with human perception such as LPIPS or FVD [3] and 3) qualitative samples that compare to stochastic models. Deterministic models can achieve very high PSNR/MSE/SSIM scores but produce very bad samples, as these scores are maximized by blurry predictions that conflate all possible future outcomes. This is highly apparent when looking at samples, and metrics that correlate better with human perception are usually better to compare video prediction methods. Comparisons to SAVP are found in Table 1 and 3 but there are no figures comparing samples from this model to the proposed model. The samples from the proposed model on the BAIR Push dataset for example (found in the appendix) are of significant lower quality than those reported from SAVP or SVG, and at the same time they are not longer-term than the predictions from these models. Consequently, the experimental section does not correctly assess how this model can generate better longer-term prediction than current models and it also does not give an accurate assessment of the model with respect to the current state-of-the-art. To sum up, the paper does not adequately address how the proposed model allows for longer-term video generation. It is missing critical qualitative comparisons to state-of-the-art models such as SVG and it is unclear how the proposed model is different from a ConvLSTM with attention on previous hidden states. [1] Stochastic Video Generation with a Learned Prior. E.Denton and R. Fergus. ICML 2018 [2] Stochastic Adversarial Video Prediction. Lee et al. Arxiv 2018 [3] Towards Accurate Generative Models of Video: A New Metric & Challenges. Unterthiner et al. Arxiv 2018 --- Post-discussion update --- The authors have addressed a number of points raised by the reviewers and I*m raising my score to a weak reject from a reject. There are important remaining issues with the experimental section and the conclusions reached from their results, and therefore I still think the paper is below the acceptance bar.