Year: 2019

Abstract:
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions. To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion flow learning, or adversarial training. In this paper, we propose a new recurrent unit, Long History Short-Term Memory (LH-STM). LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies. To capture spatiotemporal dynamics in videos, we combined LH-STM with the Context-aware Video Prediction model (ContextVP). Our experiments on the KTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the-art methods.

text: While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions. To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion flow learning, or adversarial training. In this paper, we propose a new recurrent unit, Long History Short-Term Memory (LH-STM). LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies. To capture spatiotemporal dynamics in videos, we combined LH-STM with the Context-aware Video Prediction model (ContextVP). Our experiments on the KTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the-art methods.

heading: 1 INTRODUCTION
text: Learning the dynamics of an environment and predicting consequences in the future has become an important research problem. A common task is to train a model that accurately predicts pixel-level future frames conditioned on past frames. It can be utilized for intelligent agents to guide them to interact with the world, or for other video analysis tasks such as activity recognition. An important component of designing such models is how to effectively learn good spatio-temporal representations from video frames. The Convolutional Long Short-Term Memory (ConvLSTM) network (Xingjian et al., 2015) has been a popular model architecture choice for video prediction. However, recent stateof-the-art approaches produce high-quality predictions only for one or less then ten frames (Lotter et al., 2016; Villegas et al., 2017a; Byeon et al., 2018). Learning to predict long-term future video frames remains challenging due to 1) the presence of complex dynamics in high-dimensional video data, 2) prediction error propagation over time, and 3) inherent uncertainty of the future.
Many recent works (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018) focus on the third issue by introducing stochastic models; this issue is a crucial challenge for long-term prediction. However, the architectures currently in use are not sufficiently powerful and efficient for long-term prediction, and this is also an important but unsolved problem. The model needs to extract important information from spatio-temporal data and retain this information longer into the future efficiently. Otherwise, its uncertainty about the future will increase even if the future is completely predictable given the past. Therefore, in this paper, we attempt to address the issue of learning complex dynamics of videos and minimizing long-term prediction error by fully observing the history.
We propose a novel modification of the ConvLSTM structure, Long History Short-Term Memory (LH-STM). LH-STM learns to interconnect history states and the current input by a History SoftSelection Unit (HistSSel) and double memory modules. The weighted history states computed by our HistSSel units are combined with the history memory and then used to update the current states by the update memory. The proposed method brings the power of higher-order RNNs (Soltani & Jiang, 2016) to ConvLSTMs, which have been limited to simple recurrent mechanisms so far. The HistSSel unit acts as a short-cut to the history, so the gradient flow in the LSTM is improved. More powerful RNNs are likely to be necessary to solve the hard and unsolved problem of long-term video prediction, which is extremely challenging for current architectures. Moreover, by disentangling the history and update memories, our model can fully utilize the long history states.
This structure can better model long-term dependencies in sequential data. In this paper, the proposed modification is integrated into the ConvLSTM-based architectures to solve the long-term video prediction problem: Context-aware Video Prediction (ContextVP) model. The proposed models can fully leverage long-range spatio-temporal contexts in real-world videos. Our experiments on the KTH human actions and the BAIR robot pushing datasets show that our model produces sharp and realistic predictions for more frames into the future compared to recent state-of-the-art long-term video prediction methods.

heading: 2 RELATED WORK
text: Learning Long-Term Dependencies with Recurrent Neural Networks (RNN): While Long ShortTerm Memory (LSTM) has been successful for sequence prediction, many recent approaches aim to capture longer-term dependencies in sequential data. Several works have proposed to allow dynamic recurrent state updates or to learn more complex transition functions. Chung et al. (2016) introduced the hierarchical multiscale RNN that captures a hierarchical representation of a sequence by encoding multiple time scales of temporal dependencies. Koutnik et al. (2014) modified the standard RNN to a Clockwork RNN that partitions hidden units and processes them at different clock speeds. Neil et al. (2016) introduced a new time gate that controls update intervals based on periodic patterns. Campos et al. (2017) proposed an explicit skipping module for state updates. Zilly et al. (2017) increased the recurrent transition depth with highway layers. Fast-Slow Recurrent Neural Networks (Mujika et al., 2017) incorporate ideas from both multiscale (Schmidhuber, 1992; El Hihi & Bengio, 1996; Chung et al., 2016) and deep transition (Pascanu et al., 2013; Zilly et al., 2017) RNNs. The advantages of the above approaches are efficient information propagation through time, better long memory traces, and generalization to unseen data.
Alternative solutions include the use of history states, an attention model or skip connections. Soltani & Jiang (2016) investigated a higher order RNN to aggregate more history information and showed that it is beneficial for long range sequence modeling. Cheng et al. (2016) deployed an attention mechanism in LSTM to induce relations between input and history states. Gui et al. (2018) incorporated into an LSTM, dynamic skip connections and reinforcement learning to model long-term dependencies. These approaches use the history states in a single LSTM by directly adding more recurrent connections or adding an attention module in the memory cell. These models are used for one dimensional sequence modeling, whereas our proposed approach separates the history and update memories that learn to encode the long-range relevant history states. Furthermore, our approach is more suitable for high-dimensional (e.g., video) prediction tasks.
Video Prediction: The main issue in long-term pixel-level video prediction is how to capture longterm dynamics and handle uncertainty of the future while maintaining sharpness and realism. Oh et al. (2015) introduced action-conditioned video prediction using a Convolutional Neural Network (CNN) architecture. Villegas et al. (2017b) and Wichers et al. (2018) focused on a hierarchical model to predict long-term videos. Their model estimates high-level structure before generating pixel-level predictions. However, the approach by Villegas et al. (2017b) requires object pose information as ground truth during training. Finn et al. (2016) used ConvLSTM to explicitly model pixel motions. To generate high-quality predictions, many approaches train with an adversarial loss (Mathieu et al., 2015; Wichers et al., 2018; Vondrick et al., 2016; Vondrick & Torralba, 2017; Denton et al., 2017; Lee et al., 2018). Weissenborn et al. (2019) introduced local self-attention on videos directly for a large scale video precessing. Another active line of investigation is to train stochastic prediction models using VAEs (Denton & Fergus, 2018; Babaeizadeh et al., 2017; Lee et al., 2018). These models predict plausible futures by sampling latent variables and produce long-range future predictions.
The Spatio-temporal LSTM (Wang et al., 2017; 2018a) was introduced to better represent the dynamics of videos. This model is able to learn spatial and temporal representations simultaneously. Byeon et al. (2018) introduced a Multi-Dimensional LSTM-based approach (Stollenga et al., 2015) for video prediction. It contains directional ConvLSTM-like units that efficiently aggregate the entire spatio-temporal contextual information. Wang et al. (2018b) recently proposed a memory recall function with 3D ConvLSTM. This work is the most related to our approach. It uses a set of cell states with an attention mechanism to capture the long-term frame interaction similar to the work of Cheng et al. (2016). With 3D-convolutional operations, the model is able to capture short-term and long-term information flow. In contrast to this work, the attention mechanism in our model is
used for a set of hidden state. We also disentangle the history and update memory cells to better memorize and restore the relevant information. By integrating a double memory LH-STM into the context-aware video prediction (ContextVP) model (Byeon et al., 2018), our networks can capture the entire spatio-temporal context for a long-range video sequence.

heading: 3 METHOD
text: In this section, we first describe the standard ConvLSTM architecture and then introduce the LH-STM. Finally, we explain the ConvLSTM-based network architectures for multi-frame video prediction using the Context-aware Video Prediction model (ContextVP) (Byeon et al., 2018).

heading: 3.1 CONVOLUTIONAL LSTM
text: Let Xn1 = {X1, ..., Xn} be an input sequence of length n. Xk ∈ Rh×w×c is the k-th frame, where k ∈ {1, ..., n}, h is the height, w the width, and c the number of channels. For the input frame Xk, a ConvLSTM unit computes the current cell and hidden states (Ck, Hk) given the cell and hidden states from the previous frame, (Ck−1, Hk−1):
Ck, Hk = ConvLSTM(Xk, Hk-1, Ck-1), (1)
by computing the input, forget, output gates ik, fk, ok, and the transformed cell state Ĉk:
ik = σ(Wi ∗Xk +Mi ∗Hk-1 + bi), fk = σ(Wf ∗Xk +Mf ∗Hk-1 + bf ), ok = σ(Wo ∗Xk +Mo ∗Hk-1 + bo), Ĉk = tanh(Wĉ ∗Xk +Mĉ ∗Hk-1 + bĉ), Ck = fk Ck-1 + ik Ĉk, Hk = ok tanh(Ck),
(2)
where σ is the sigmoid function, W and M are 2D convolutional kernels for input-to-state and state-to-state transitions, (∗) is the convolution operation, and ( ) is element-wise multiplication. The size of the weight matrices depends on the size of convolutional kernel and the number of hidden units.

heading: 3.2 LONG HISTORY SHORT-TERM MEMORY (LH-STM)
text: LH-STM is an extension of the standard ConvLSTM model by integrating a set of history states into the LSTM unit. The history states include the spatio-temporal context of each frame in addition to the pixel level information in the frame itself. Figure 1 illustrates the differences between a standard RNN, Higher-Order RNN (Soltani & Jiang, 2016), and our proposed model (Double and Single LH-STM).
History Soft-Selection Unit (HistSSel): The HistSSel unit computes the relationship between the recent history and the earlier ones using dot-product similar to (Vaswani et al., 2017)1. This mechanism can be formulated as SoftSel(Q,K, V ) = softmax(WQQ ·WQK) ·WQV . It consists of queries (Q), keys (K) and values (V). It computes the dot products of the queries and the keys; and then applies the a softmax function. Finally, the values (V) are weighted by the outputs of softmax function. The queries, keys, and values can be optionally transformed by the WQ, WK , and WV matrices.
Using this mechanism, HistSSel computes the relationship between the last hidden state Hk-1 and the earlier hidden states Hk-m:k-2 at time step k (See Figure 2). Hk-m:k-2 is the set of previous hidden states, (Hk-m, Hk-m-1, · · ·Hk-3, Hk-2). We will show in Section 4.2 the benefit of using history states versus using the past input frames directly. The history soft-selection mechanism can be formulated as follows:
HistSSel(Hk-1, Hk-m:k-2) = n∑
i=2
softmaxi(H̃ Q k-1 · H̃ K k-i) · H̃Vk-i,
H̃ji =W j i Hi + b j i , j ∈ {Q,K, V }. 2
(3)
Single LH-STM: A simple way to employ HistSSel in ConvLSTM (Equation 1) is to add HistSSel(Hk-1, Hk-m:k-2) in addition to the input and the previous states (Xk, Hk-1, Ck-1) in the Equation 6 Figure 1c shows a diagram of the computation. This direct extension is named Single LH-STM: Hk = SingleConvLSTM(Xk, Hk-1, Ck-1,HistSSel(Hk-1, Hk-m:k-2)).
ik = σ(Wi ∗Xk +Mi ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + bi), fk = σ(Wf ∗Xk +Mf ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + bf ), ok = σ(Wo ∗Xk +Mo ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + bo) Ĉk = tanh(Wĉ ∗Xk +Mĉ ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + bĉ), Ck = ik Ĉk + fk Ck-1, Hk = ok tanh(Ck).
(4)
Double LH-STM: To effectively learn dynamics from the history states, we propose Double LH-STM. It contains two ConvLSTM blocks, History LSTM (H-LSTM) and Update LSTM (U-LSTM). The goal of the Double LH-STM is to explicitly separate the long-term history memory and the update memory. By disentangling these, the model can better encode complex long-range history and keep track of the their dependencies. Figure 1d illustrates a diagram of Double LH-STM.
1The purpose of this unit is to compute the importance of each history state. While we used a selfattention-like unit in this paper, this can be achieved with other common layers as well, e.g., fully connected or convolutional layers.
2bji can be omitted.
The H-LSTM block explicitly learns the complex transition function from the (possibly entire) set of past hidden states, Hk-m:k-1, 1 < m < k. If m = k, H-LSTM incorporates the entire history up to the time step k − 1. The U-LSTM block updates the states Hk and Ck for the time step k, given the input Xk, previous cell state Ck−1, and the output of the H-LSTM, H ′k−1.
The History-LSTM (H-LSTM) and Update-LSTM (U-LSTM) can be formulated as:
H-LSTM
i′k-1 = σ(M ′ i ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + b′i), f ′k-1 = σ(M ′ f ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + b′f ), o′k-1 = σ(M ′ o ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + b′o)
Ĉ′k-1 = tanh(M ′ ĉ ∗Hk-1 + HistSSel(Hk-1, Hk-m:k-2) + b′ĉ),
C′k-1 = f ′ k−1 C′k-2 + i′k−1 Ĉ′k−1, H ′k-1 = o ′ k-1 tanh(c′k-1).
(5)
U-LSTM
ik = σ(Wi ∗Xk +Mi ∗H ′k-1 + bi), fk = σ(Wf ∗Xk +Mf ∗H ′′k-1 + bf ), ok = σ(Wo ∗Xk +Mo ∗H ′k-1 + bo),
Ĉk = tanh(Wĉ ∗Xk +Mĉ ∗H ′′k-1 + bĉ),
Ck = fk Ck-1 + ik Ĉk, Hk = ok tanh(Ck).
(6)

heading: 3.3 IMPLEMENTATION DETAILS
text: Model Architectures: ConvLSTM (Xingjian et al., 2015; Stollenga et al., 2015) is a popular building block for spatio-temporal sequence forecasting problems. In this paper, we focused on the recent state-of-the-art ConvLSTM-based architectures proposed by Byeon et al. (2018): the context-aware video prediction model (ContextVP-4). It can capture sufficient spatio-temporal context for video prediction and achieves state-of-the-art results on KTH human actions and BAIR robot pushing dataset. In this paper, the ConvLSTM block is replaced with LH-STM. Additionally, we exploit two types of skip connections to avoid the problem of vanishing gradients (Hochreiter et al., 2001) and allow long-term prediction: (a) between previous and the current recurrent layers (blue dotted line in Figure 9) and (b) across layers (green solid line in Figure 9). See Appendix B for the effectiveness of both skip connections.
ContextVP-4 (Byeon et al., 2018) consists of 4 layers of five directional Parallel Multi-Dimensional LSTM (PMD) units (Stollenga et al., 2015) along the h+, h−, w+, w−, and t− recurrence directions. A PMD unit along the time direction (t−) is mathematically equivalent to a standard ConvLSTM (Equation 6). By using LSTM connectivities across the temporal and spatial dimensions, each processing layer covers the entire context in a video. Following Byeon et al. (2018), we also included weighted blending, directional weight sharing (DWS), and two skip connections between the layers 1 - 3 and 2 - 4 (see Figure 9). See Appendix B for details of the network architectures.
Loss: The loss function to train the networks is the combination ofL1-norm, L2-norm, and perceptual loss (Johnson et al., 2016; Zhang et al., 2018):
L(Y, X̂) = λ1L1(Y, X̂) + λ2L2(Y, X̂) + λplLpl(Y, X̂), (7)
where Y and X̂ are the target and the predicted frames, respectively. λ is a weight for each function. Lp-norm is defined as Lp(Y, X̂) = ||Y − X̂||p. p = 1 for L1-norm, and p = 2 for L2-norm. The perceptual loss computes the cosine distance between the feature maps extracted from a VGG-16 network pre-trained on the ImageNet dataset (Simonyan & Zisserman, 2014) as Lpl(y, x̂) = 1− 1l ∑ l 1 hl×wl ∑ hl,wl(φ(y)l · φ(x̂)l), where φ(y)l and φ(x̂)l are the feature maps of the target and the predicted frames Y and X̂ , respectively at the layer l. The size of the feature map at layer l is hl × wl.

heading: 4 EXPERIMENTS
text: We evaluated our LH-STM model on the KTH human actions (Laptev et al., 2004) and BAIR actionfree robot pushing (Ebert et al., 2017) datasets. Videos in the KTH human actions dataset consist of six human actions including walking, jogging, running, boxing, hand waving, and hand clapping. The BAIR action-free robot pushing dataset includes randomly moving robotic arms, which push objects on a table. The videos have a resolution of 64× 64 pixels. We show quantitative comparisons
by computing frame-wise Structural Similarity (SSIM) (Wang et al., 2004), Peak Signal-to-Noise Ratio (PSNR), Visual Information Fidelity (VIF) (Sheikh & Bovik, 2005), and LPIPS (Zhang et al., 2018) between the ground truth and generated video sequences.

heading: 4.1 KTH HUMAN ACTIONS DATASET
text: We follow the experimental setup of Lee et al. (2018). We evaluate our methods on 128× 128 pixels for 40 frame prediction (Table 2) We train all models to predict the next 10 frames, given 10 prior ones. For testing, all models recursively predict 40 frames.
Our Double LH-STM outperforms all competing methods on all metrics including the base model (ContextVP). Our Double LH-STM model consistently outperforms other methods for the 40 predicted frames. The improvement is larger for longer-term frame prediction into the future (40 frames). Qualitative results can be found in Figure 7 and Appendix E (Figure 14 and Figure 15). These results show that motion and human shape predicted by Double LH-STM are the closest to the ground truth.
Our models are compared with FRNN (Oliu et al., 2018), PredRNN (Wang et al., 2017), PredRNN++ (Wang et al., 2018a), E3D-LSTM (Wang et al., 2018b), SAVP-VAE and SAVP-Deterministic models (Lee et al., 2018)3. Our model outperforms all other methods.

heading: 4.2 STATE HISTORY VS INPUT HISTORY
text: The proposed LH-STM models use past hidden states. Alternatively, input frames can be directly applied to LH-STM as illustrated in Figure 3. We compared the performance of using input and state history with the same LH-STM models and the same model size in Table 1 on the KTH human actions dataset. Figure 7 shows a qualitative comparison of using state history (Double LH-STM) and input history (Double, input history). The results show the importance of using history states with LH-STM and that the state history is more suitable for videos.

heading: 4.3 LONGER PREDICTION
text: To test the performance of longer-term prediction (more than 40 frames), we additionally trained the KTH human actions dataset with a resolution of 64× 64 pixels4. Figure 4 and Figure 5 compares the performance of 80 frame prediction with Double LH-STM and two state-of-the-art models (3SAVPVAE and SAVP-Deterministic (Lee et al., 2018)) on all action classes and three action classes only (hand waving, hand clapping, and boxing), respectively. We compare only these three action classes with all action classes to show that the high performance at longer frames is not due to background copy. The human in other action class videos (running, jogging, and walking) may disappear after certain frames. Additionally, Figure 13 shows the performance comparisons of each action class. The experimental setup is the same as the experiments of a resolution 128 × 128 (Section 4.1). From these comparisons, we can observe that Double LH-STM results are generally better than
3 We note that Lee et al. (2018) reported the performance of VAE+GAN, VAE only, GAN only, and of their deterministic models. We select their best (VAE only) and the most comparable (deterministic) models to compare against. To evaluate the VAE model, we take 100 random input samples for each video following Lee et al. (2018). The reported numbers in this paper are for their method’s “median” output among the 100 randomly generated ones, compared to the ground truth. Please see Appendix C for the discussion of this evaluation setup
4Due to lack of GPU memory, we could not use the original resolution for 80 frame prediction.
SAVP-Deterministic and -VAE on all metrics. These also show that our model is not just copying the background when predicting more than 40 frames.
0 20 40 60 80 Time steps
25
26
27
28
29
30
31
32
PS NR
Double LH-STM (ours) (27.53) SAVP-Det (Lee, 2018) (25.65) SAVP-VAE (Lee, 2018) (25.0)
0 20 40 60 80 Time steps
0.60
0.65
0.70
0.75
0.80
0.85
0.90
SS IM
Double LH-STM (ours) (0.775) SAVP-Det (Lee, 2018) (0.717) SAVP-VAE (Lee, 2018) (0.696)
0 20 40 60 80 Time steps
0.725
0.750
0.775
0.800
0.825
0.850
0.875
0.900
0.925
LP IP
S
Double LH-STM (ours) (0.799) SAVP-Det (Lee, 2018) (0.802) SAVP-VAE (Lee, 2018) (0.78)
Figure 4: Per-frame comparisons on the KTH human actions dataset (all actions, 64× 64, 80 frame prediction). (·): averaged score
0 20 40 60 80 Time steps
22
24
26
28
30
PS NR
Double LH-STM (ours) (25.01) SAVP-Det (Lee, 2018) (23.06) SAVP-VAE (Lee, 2018) (22.68)
0 20 40 60 80 Time steps
0.60
0.65
0.70
0.75
0.80
0.85 0.90 SS IM Double LH-STM (ours) (0.748) SAVP-Det (Lee, 2018) (0.681) SAVP-VAE (Lee, 2018) (0.667)
0 20 40 60 80 Time steps
0.750
0.775
0.800
0.825
0.850
0.875
0.900
0.925
LP IP
S
Double LH-STM (ours) (0.813) SAVP-Det (Lee, 2018) (0.8) SAVP-VAE (Lee, 2018) (0.795)
Figure 5: Per-frame comparisons on the KTH human actions dataset (‘boxing’, ‘waving’, and ‘clapping’ actions, 64× 64, 80 frame prediction). (·): averaged score

heading: 4.4 BAIR ROBOT PUSHING DATASET
text: Our experimental setup follows that of Lee et al. (2018). All models are conditioned on five frames to predict the next ten frames 5. The models are tested to predict 25 frames. We evaluated our LH-STM models (Single and Double) and compared them with the base model (ContextVP), the 3SAVP-VAE, and SAVP-Deterministic models (Lee et al., 2018) in Table 3. Our Double LH-STM achieved better PSNR and SSIM scores than the SAVP-Deterministic and SAVP-VAE models. It can be seen from the per-frame prediction results (Figure 6), that the performance LH-STM is higher over time, especially on the PSNR and SSIM metrics. On the LPIPS metric, the performance of Double LH-STM is comparable to SAVP-Deterministic and better than SAVP-VAE. Qualitative results can be found in Appendix E (Figure 17 and Figure 18). This dataset contains random robot motion which makes long-term prediction extremely hard with deterministic models. However, the overall motion prediction with Double LH-STM and SAVP-Deterministic is still reasonable compared to SAVP-VAE.
5 Lee et al. (2018) used two input frames. We increase the number of input frames as motion in two frames is almost not visible. We increased the number of input frames to provide enough history to the model. We also re-trained the models of Lee et al. (2018) with five input frames.

heading: 5 CONCLUSION
text: In this paper, we have introduced a new RNN structure, LH-STM, which incorporates long history states into the recurrent unit to learn longer range dependencies. This module is combined with the ContextVP architecture allowing context aware long-term video prediction. The resulting model produces more realistic predictions for a longer period of time compared to the recent state-of-the-art methods. The proposed approach can easily be extended to standard LSTM or Gated Recurrent Units (GRU) and applied to tasks that require a long-term memory trace, e.g., language modeling or speech recognition.

heading: A NETWORK DETAILS
text: Each layer of ContextVP-4 contains five Parallel Multi-Dimensional LSTM (PMD) units (see Figure 8). The parameters of opposite directions are shared. We followed the same network architecture as Byeon et al. (2018) except for the number of neurons.
Weighted Blending: This layer learns the relative importance of each direction during training.
Sk = [H h+ k , H h− k , H w+ k , H w− k , H t− k ],
Mk = f(Wk · Sk + bk), (8)
where H is the output of the PMD units from h+, h−, w+, w−, and t− directions, and W is the weight matrix.
Directional Weight-Sharing: Weights and biases of the PMD units in opposite directions (i.e., the h+ and h− directions, w+ and w− directions) are shared.
Skip connection in a recurrent block (R-Skip): A skip connection is commonly used in LSTM layers (Melis et al., 2017; Merity et al., 2017; Wang et al., 2018a) for long-term gradient propagation. Our proposed model concatenates outputs of the previous and current LSTM layers (see Figure 9).
Gated skip connection across layers (L-Skip): Unlike in (Byeon et al., 2018), We added the gated skip connection across layers. In our model, the multiplicative gate is added to control the flow of information across layers. It stabilizes learning as depth increases. A layer is formulated as:
Y = Hl · Tl +Xl−i · (1− Tl), i < l, (9)
where Hl = Xl ·WHl + bH and Tl = σ(Xl ·WTl + bT ). Xl is the input at the lth layer, T is a gate, and Hl is the transformation of the input at the layer l.
We adjusted the number of neurons, so all models have similar model size (≈ 17M). LH-STM Double contains 64-64-128-128 neurons for the input size of 64× 64 pixels and 64-128-128-128 for 128× 128. LH-STM Single contains 128-128-128-128 neurons for the input size of 64× 64 pixels and 64-128-128-256 for 128× 128. The initial learning rate is 1e − 4 found by grid search between 1e − 3 and 1e − 5. The learning rate is decayed every 10 epochs with the decay rate 0.95. Weights are initialized using the Xavier’s normalized initializer (Glorot & Bengio, 2010) and the recurrent states are all initialized to zero.

heading: B ABLATION STUDY
text: We performed a series of ablation studies to evaluate the importance of each component of our proposed model 1) RNN type (standard ConvLSTM, Single LH-STM, or Double LH-STM); 2) different losses (L1, L2, and the perceptual loss); adding skip connections 3) in recurrent blocks; and 4) across layers (see Section 3.3 for details). We evaluated performance on the KTH human actions dataset with a resolution of 64 × 64 pixels. The weights for the loss, λ1, λ2, and λpl, are tested between zero to one. The results of this experiment are summarized in Table 4. We found that performance improves by adding skip connections in recurrent blocks and across layers. Double LH-STM outperforms the standard ConvLSTM and Single LH-STM. Combination of L1, L2, and Lpl also improves the performance (with λ1 = 1, λ2 = 1, λpl = 1).

heading: C EVALUATION OF STOCHASTIC MODELS
text: SAVP-VAE numbers in the paper by Lee et al. (2018) are obtained by first generating 100 output samples per input and then selecting the best generated sample for each input based on the highest score between the output and the ground truth. This evaluation measures the best prediction the model can do, given 100 tries. However, it does not provide an overall picture of the generations from the model. In real world scenarios, the ground truth is not available, so the ‘best’ samples cannot be picked as done in such evaluations. Therefore, we re-computed the scores for stochastic models based on their ‘median’ output sample (instead of best) among the 100 randomly generated ones, compared to the ground truth. This strategy measures how well the model can be expected to perform on average, and so it is a more representative score.

heading: D EVALUATION WITH LARGE AND SMALL MOTION
text: We split the video samples into five bins by motion magnitude computed as averaged L2 norm between target frames similar to the evaluations by Villegas et al. (2017a). Figure 12 shows the comparisons on the KTH human action dataset, and ?? shows on BAIR action free robot pushing dataset. Overall, across all metrics, all models perform worse when the motion is larger. For PSNR and SSIM, Double LH-STM achieves the best performance except for the largest motion. For LPIPS, Double LH-STM performs better for small motions (first two bins). SAVP-Deterministic performs the best on this metric for larger motions.

heading: E QUALITATIVE RESULTS
text: Figure 14 and Figure 15 show prediction results on the KTH human actions dataset, and Figure 17 and Figure 18 presents the results on the BAIR action free robot pushing dataset. Figure 16 and Figure 19 include the samples with one of the largest motions for both datasets. When motion is larger, all models fail to produce reasonable predictions, whether they are deterministic or stochastic.

heading: F RETRAINING PREDRNN++
text: As mentioned in the paper, we re-trained PredRNN++ on the KTH human actions dataset using the provided code by Wang et al. (2018a) (https://github.com/Yunbo426/predrnn-pp). We conducted grid hyper-parameter search for the learning rate and the patch size. We found in the code that there is an option to divide the input frames into smaller patches. The learning rate search was performed in the range between 1e− 3 and 1e− 4. The range of patch size was 1, 2, 4 for both frame resolutions (64× 64 and 128× 128 pixels). The reported results in the paper are for a learning rate of 1e− 4 and the patch size 1 (full resolution).

Review summaries:
The paper proposes a type of recurrent neural network module called Long History Short-Term Memory (LH-STM) for longer-term video generation. This module can be used to replace ConvLSTMs in previously published video prediction models. It expands ConvLSTMs by adding a "previous history" term to the ConvLSTM equations that compute the IFO gates and the candidate new state. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism. As such, it is not clear if there are significant differences between LH-STMs and previously proposed LSTMs with attention on previous hidden states. The authors propose recurrent units that include one or two History Selection (soft-attention) steps, called single LH-STM and double LH-STM respectively. The exact formulation of the double LH-STM is not clear from the paper.  The authors then propose to use models with LH-STM units for longer term video generation. They claim that LH-STM can better reduce error propagation and better model the complex dynamics of videos. To support the claims, they conduct empirical experiments where they show that the proposed model outperforms previous video prediction models on KTH (up to 80 frames) and the BAIR Push dataset (up to 25 frames).

This paper proposes a new LSTM architecture called LH-STM (and Double LH-STM). The main idea deals with having a history selection mechanism to directly extract what information from the past. The authors also propose to decompose the history and update in LH-STM into two networks called Double LH-STM. In experiments, the authors evaluate and compare their two architectures with previously proposed models. They show that their architecture outperforms previous in the PSNR, SSIM and VIF metrics.