Long History Short-Term Memory for Long-Term Video Prediction
Authors: Wonmin Byeon
Jan Kautz
Venue:  (2020)
Keywords: LSTM
video
long-term prediction

Abstract: While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions. To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion flow learning, or adversarial training. In this paper, we propose a new recurrent unit, Long History Short-Term Memory (LH-STM). LH-STM incorporates long history states into a recurrent unit to learn longer range dependencies. To capture spatio-temporal dynamics in videos, we combined LH-STM with the Context-aware Video Prediction model (ContextVP). Our experiments on the KTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the-art methods. 

Under review as a conference paper at ICLR 2020
LONG HISTORY SHORT-TERM MEMORY FOR LONG-TERM VIDEO PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging ambiguous future enror propagation over time yield blurry predictions. To address this challenge, existing algorithms rely on extra supervision (e.g., action or object pose), motion flow learning, or adversarial training. In this paper, we propose a new recurrent unit, Long History Short-Term Memory (LH-STM). LH-STM incorporates long history states into recurrent unit to learn longer range dependencies. To capture spatio- temporal dynamics in videos, we combined L.HSTMM M with the Context-aware Video Prediction model (ContextVP). Our experiments on the KTTH human actions and BAIR robot pushing datasets demonstrate that our approach produces not only sharper near-future predictions, but also farther into the future compared to the state-of-the- methods.
INTRODUCTION
Learning the dynamics of an environment and predicting consequences in the future has become important research problem. A common task is to train a model that accurately predicts pixel-level future frames conditioned on past frames. It can be utilized for intelligent agents guide them to interact with the world, or for other video analysis tasks such as activity recognition. An important component of designing such models is how effectively learn good spatio-temporal representations from video frames. The Convolutional Long Short-'Temn Memory (ConvI.STM) network (Xingjian al. 2015) has been a popular model architecture choice for video prediction. However, recent state- of-the-ane art approaches produce high-qualin predictions only for one or less then ten frames (Lotter cct al. 2016 Villegas et al. 2017 7a Byeon et al. 2018). Learning to predict long-term future video Trames remains challenging due to 1) the presence of complex dynamics in high-dimensional video data, 2) prediction error propagation over time, and 3) inherent uncertainty of the future.
Many recent works (Denton & Fergus 2018 Babaeizadeh al. 2017 Lee et al. 2018) focus on the third issue by introducing stochastic models; this issue is a crucial challenge for long-term prediction. However, the architectures eurrently in use not sufficiently powerful and efficient for long-term prediction, and this is also an important but unsolved problem. The model needs to extract important information from spatio-temporal data and retain this information longer into the future efficiently, Otherwise, its uncertainty about the future will increase even if the future is completely predictable given the past. Therefore, in this paper, we attempt address the issue of learning complex dynamics of videos and minimizing long-term prediction error by fully observing the history.
We propose novel modification of the ConvLSTM structure, Long History Short-Term Memory (LH-STM). LH-STM learns to interconnect history states and the current input by a History Soft- Selection Unit (HistSSel) and double memory modules. The weighted history states computed by our HistSSel units are combined with the history memory and then used to update the current states by the update memory. The proposed method brings the power of higher-order RNNs (Soltani & Jiang 2016) to ConvLSTMs, which have been limited to simple recurrent mechanisms so far. The HistSSel unit acts as a short-cut to the history, so the gradient flow in the LSTM is improved. More powerful RNNs are likely to be necessary solve the hard and unsolved problem of long-term video prediction, which is extremely challenging for current architectures. Moreover, by disentangling the history and update memories, model can fully utilize the long history states.

---

Under review as a conference paper at ICLR 2020
This structure can better model long-tern dependencies in sequential data. In this paper, the proposed modification is integrated into the ConvLSTM--iine -based architectures to solve the long-term video prediction problem: Context-aware Video Prediction (ContextVP) model. The proposed models can fully leverage long-range spatio-temporal contexts in real-world videos. Our experiments on the KTH human actions and the BAIR robot pushing datasets show that model produces sharp and realistic predictions for more frames into the future compared to recent state-of-the-ar long-(erm video prediction methods.
2 RELATED WORK
Term Dependencies with Recurrent Neural Networks (RNN): While Long Short- Term Memory (LSTM) has been successful for sequence prediction, many recent approaches aim to capture longer-term dependencies in sequential data. Several works have proposed to allow dynamic recurrent state updates or to learn more complex transition functions. [Chnng et al. (2016) introduced the hierarchical multiscale RNN that captures hierarchical representation of a sequence by encoding multiple time scales of temporal dependencies, Koutnik et al. (2014) modified the standard RNN to a Clockwork RNN that partitions hidden units and processes them at different clock speeds. Neil et al.) (2016) introduced a new time gate that controls update intervals based on periodic patterns. |Campos) al. (2017) proposed an explicit skipping module for state updates. Zilly al., (2017) increased the recurrent transition depth with highway layers. Fast-Slow Recurrent Neural Networks (Mujika et al.
Learning Long-Term
2017) incorporate ideas from both multiscale (Schmidhuber) 1992. El Hihi & Bengio 1996] Chung
et al. 2016) and deep transition (Pascanu et al. 2013 Zilly al. 2017) RNNs. The advantages of the above approaches are efficient information propagation through time, better long memory traces, and generalization to unseen data.
Alternative solutions include the use of history states, an attention model or skip connections. Soltani & Jiang (016)) investigated higher order RNN to aggregate more history information and showed that it is beneficial for long range sequence modeling. Cheng et al.] (016)) deployed an attention mechanism in LSTM induce relations between input and history states. [Cui et al. (2018) incorporated into an LSTM, dynamic skip connections and reinforcement learning to model long-term dependencies. These approaches use the history states in a single LSTM by directly adding more recurrent connections or adding an attention module in the memory cell. These models are used for one dimensional sequence modeling, whereas our proposed approach separates the history and update memories that learn to encode the long-range relevant history states. Furthermore, our approach is more suitable for high-dimensional (e.g., video) prediction tasks.
Video Prediction: The main issue in long-term pixel-level video prediction is how to capture long- term dynamics and handle uncertainty of the future while maintaining sharpness and realism. [hh et al. (2015) introduced action-conditionn
video prediction using a Convolutional Neural Network (CNN) architecture. Villegas et al. (2017b) and Wichers el al. (2018) focused on a hierarchical model to predict long-term videos. Their model estimates high-level structure before generating pixel-level predictions. However, the approach by Villegas et al. (2017b) requires object pose information as ground truth during training. [Finn al. (2016) used ConvL.ST to explicitly model pixel motions. To generate nigh-quality predictions, many approaches train with an adversarial loss (Mathieu et al.)
2015 Wichers al.) 2018 Vondrick et al. 2016 Vondrick & Torralba 2017, Denton al. 2017 Lee ct al. 2018). Weissenborn et al. (2019) introduced local self-attention on videos directly for large scale video precessing. Another active line of investigation is train stochastic prediction models using VAEs (Denton & Fergus 2018 Babaeizadeh et al. 2017: Lee et al.) 2018). These models predict plausible futures by sampling latent variables and produce long-rau long-range future predictions.
2018a) was introduced to better represent the dynamics of videos. This model able to learn spatial and temporal representations simultaneously. nulti--Dimensooal LSSM- approach (Stollenga et al.. (015))
The Spatio-temporal LSTM (Wang et al., 2017:
Byeon et al. (2018) introduced for video prediction. It contains directional ConvLSTM-like units that efficiently aggregate the entire spatio-temporal contextual information. Wang et al. (2018)) recently proposed memory recall function with 3D ConvLSTM This work is the most related to our approach. It uses a set of cell states with an attention mechanism to capture the long-term frame interaction similar to the work Cheng et al. (2016). With 3D- cD-covvoltionall operations, the model is able to capture short-term and long-term information flow. In contrast this work, the attention mechanism in our model is
2

---

Under review as a conference paper at ICLR 2020
used for of hidden state. We also disentangle the history and update memory cells to better memorize and restore the relevant information. By integrating a double memory LH-STM into the context-aware video prediction (ContextVP) model (Byeon et al. 2018)) our networks can capture the entire spatio-temporal context for a long-range video sequence.
METHOD
In this section, we first describe the standard ConvLSTM, architecture and then introduce the LH-STM. Finally, we explain the ConyLSTM-bases network architectures for multi-frame video prediction using the Context-aware Video Prediction model (ContextVP) (Byeon et al. 2018).
3.1 CONVOLUTIONAL LSTM
Let X = {X1,..., Xn, be an input sequence of length n. X E RXxxx is the k-th frame, where k E (I,...,,)) n). h is the height, the width, and the number of channels. For the input frame X a ConvLSTM unit computes the current cell and hidden states (Ck, H&) given the cell and hidden states from the previous frame, (C Hk-1):
Ck Hy ConeLSTM(X4, HK-1, Ck-1),
by computing the input, forget, output gates ik. fk. Ok, and the transformed cell state Oj:
i a (W; X + M; H--1 + bz),
f a (W, X2 + My H + by),
O₁: a (W, X2 + Mo HK-1 bo),
(2)
CN tanh(We Xx + Ma * H + ba).
C₁ fk @ Ck-1 is Ck.
H₁ o% @ (an)(Ck),
where is the sigmoid function. W and M are 2D convolutional kemnels for input-to-state and stte-to-ssttte transitions, (+) the convolution operation, and (o) is element-wise multiplication. The size of the weight matrices depends on the size of convolutional kernel and the number of hidden units.
3.2 LONG HISTORY SHORT-TERS MEMORY (LH-STM)
LH-STM is extension of the standard ConvLSTM model by integrating set of history states into the LSTM unit. The history states include the spatio-temporal context of each frame in addition to the pixel level information in the frame itself. Figure Iillustrates the differences between a standard RNN, Higher-Order RNN (Soltani & Jiang 2016). and our proposed model (Double and Single LH-STM).
X
X
N
X
X
X
X
X
X
X
H
H
H
He-t
H
H
Ha-r
H
H
H
H
3
Xy
(a) Standard RNN
(b) Higher-Order RNN
(c) Single LH-STM
(d) Double LH-STM
Figure I:
Graphical illustrations of the standard RNN, Higher-Order RNN Soltani & Jiang 2016).
Single and Double LH-STM. X₁ and Xτ, respectively indicate the input and predicted frame at time k. Hx-3. Hx-2, Hx-1 are past hidden states. blue line indicates recurrent connection; green color indicates History Soft-Selection unit, and circle donates memory unit.
3

---

Under review as a conference paper at ICLR 2020
H-m-222
H
H
M
Who
Mi.
Values (V)
Keys (K)
Queries
B
Doe product
Saftmax
Dat-product
HistSSel H A-1 H H
Figure 2: A graphical illustration of the History Soft-Selection Unit (HistSSel). It computes the relationship between the last H and the earlier hidden state H 2-
History Soft-Selection Unit (HistSSel): The HistSSel unit computes relationship between the recent history and the earlier ones using dot-product similar to (Vaswani et al. 2017 This mechanism can be formulated as SoftSel K, V) = softmax ( W WQV V. consists of queries (Q), keys (K) and values (V). It computes the dot products of the queries and the keys; and then applies the softmax function. Finally, the values (V) are weighted by the outputs of softmax function. The queries, keys, and values can be optionally transformed by the Wº, WK and WV matrices.
Using this mechanism, HistSSel computes the relationship between the last hidden state Ha-1 and the earlier hidden states H at time step l (See Figure 2). H Ha-mak-2 is the set of previous hidden states, (Hem Hy-m-1.- H4-3 Hb-2). We will show Section 4.2 the benefit of using history states versus using the past input frames directly. The history soft-selection mechanism can be formulated as follows:
Hist Hx-mak)2)
Σ softmax, (11 ig) ily.
(3)
i = + 1/. jE (2, K,
Single LH-STM: A simple way employ HistSSel in ConvLSTM (Equation 1 is to add
HistSSel( HІ-1. Hs-m-k-2 in addition to the input and the previous states (Xt. Hg-1. Ck-1) in the
Equation Figure Ic) shows a diagram of the computation. This direct extension named Single
LII-STM: HA = SingleComoLST (Xx, Ha-1, Ca-1, HistSSel(H1-1, Hi-mat-2)).
i (W: * Xx + M; Hz11 HistSSel H H b;),
f& = a (W, X + My H+1 HistSSel(H H kmm.k-2) + by):
Ok a (W, X M, Нat + HistSSel H HR-mm:k22) + bo)
(4)
Č = tanh(W: * X + Ma Ha-1 + HistSSel(Hgs1. Hx-msk-2) + ba),
C₁ is @ G₁ + fx o Ca-1,
Hg = o O (Ck)
Double LH-STM: To effectively learn dynamics from the history states, we propose Double LH-STM. contains two ConvL L. blocks. History LSTM (H-LSTM) and Update LSTM (U-LSTM). The goal of the Double LH-STM is to explicitly separate the long-term history memory and the update memory. By disentangling these, the model can better encode complex long-range history and keep track of the their dependencies. Figure Id illustrates diagram of Double LH-STMM.
'The purpose of this unit is to compute the importance of each history state. While used self- attention-like unit in this paper, this can achieved with other common layers as well, egg. fully connected or convolutional layers. 2t. can be omitted.
4

---

Under review as a conference paper at ICLR 2020
The H-LSTM block explicitly learns the complex transition function from the (possibly entire) set of past hidden states, H H 1 < m < k. If m = k. H-LSTM incorporates the entire history up to the time step k 1.
The U-LSTM block updates the states H₃ and Cs for the time step k, given the input X2. previous cell state Ck-1, and the output of the H-LSTM, H Ha-1-
The History-LSTM (H-LSTM) and Update-LSTM, (U-LSTM) can be formulated as:
H-LSTM
U-LSTM
d11 o (M, Hx-1 HII, + 6,),
i = o oW, Xx + M, Hi-1 + b.),
fe1 a (Mý Ha-1 + HistSSel (mmmkk--))) liy);
fa a (W, Xx + My HH by).
1+1 = o (M' Ha-1 + HistSSel(H2-11 H k-2)
1/6)
= σ (W, Xx + Mo Hi1 + bo).
1₂ = tanh( M2 Ha-1 + IlistSSe((Hx-., Hiemeee)
bl)). C2 = tanh (Wa: Xx + Ma * H + ba).
C = fh-1 3 Che2 + 1/212 @ Ch-1
C2 = ft @ C+- + is ① Cr.
Hi 0-- tanb(ce-1).
H₂ O tanh(CC)
(5)
3.3 IMPLEMENTATION DETAILS
Model Architectures: ConvLSTM (Xingjian et al. 2015 Stollenga et al., 2015) is a popular building block for spatio-temporal sequence forecasting problems. In this paper, focused on the recent state-of-the-aat
architectures proposed by [yyon et al. (2018): the context-aware video prediction model (ContextVP-4)) It can capture sufficient spatio-temporal context for video prediction and achieves state-of-the-art results on KTH human actions and BAIR robot pushing dataset. In this paper, the ComVLSTM block is replaced with LH-STM. Additionally, types of skip connections to avoid the problem of vanishing gradients (Hochreiter et al. 2001) and allow long-term prediction: (a) between previous and the current recurrent layers (blue dotted line in Figure 9 and (b) across layers (green solid line in Figure 9p. See Appendix B for the effectiveness of both skip connections.
exploit two
ContextVP-44 (Byeon et al. 2018 consists of 4 layers of five directional Parallel LSTM (PMD) units (Stollenga et al. 2015) along the h+, h- w+,, w-- and t- recurrence directions. PMD unit along the time direction (t is mathematically equivalent to standard ConvLSTM (Equation 6). By using LSTM connectivities across the temporal and spatial dimensions, each processing layer covers the entire context in video. Following [yyon al. (2018), we also included weighted blending, directional weight sharing (DWS), and two skip connections between the layers -3 and 2 4 (see [Figure 9). See Appendix B for details of the network architectures.
Loss: The loss function to train the networks is the combination of Ll-norm, L2-norm, and perceptual loss (Johnson et al. 2016 Zhang et al. 2018):
L(Y X) (Y, X) + X2,2 X) + (Y, &Y,
where Y and X are the target and the predicted frames, respectively. function. Lp-norm is defined as CY, = Y - I11 1-- = for L1-norm, and = for L2-norm. The perceptual loss computes the cosine distance between the feature maps extracted from VGG-16 network pre-trained on the ImageNet dataset (Simonyan & Zisserman, (014) as Lyl 2) = E1 कx ΣM,et (ő(y)) ((t)), where σ(y), and o(+); are the feature maps of the target and the predicted frames Y and X, respectively the layer 1. The size of the feature map at layer is h' w'.
is a weight for each
EXPERIMENTS
We evaluated our LH-STM model on the KTH human actions (Laptev al. 2004) and BAIR action- free robot pushing (Ebert et al. 2017) datasets. Videos in the KTH human actions dataset consist of six human actions including walking, jogging, running, boxing, hand waving, and hand clapping. The BAIR action-free robot pushing dataset includes randomly moving robotic arms, which push objects table. The videos have resolution of 64 64 pixels. We show quantitative comparisons
5

---

Under review as a conference paper at ICLR 2020
by computing frame-wise Structural Similarity (SSIM) Waang al. 2004). Peak Signal-to-Noies Ratio (PSNR), Visual Information Fidelity (VIF) (Sheikh & Bovik 2005), and LPIPS (Zhang et al. 2018 between the ground truth and generated video sequences.
4.1 KTH HUMAN ACTIONS DATASET
We follow the experimental setup of Lee et al. 2018). We evaluate our methods on 128 x 128 pixels for 40 frame prediction (Tble 2) We train all models to predict the next 10 frames, given 10 prior ones. For testing, all models recursively predict 40 frames.
Our Double LH-STM outperforms all competing methods on all metrics including the base model (ContextVP).. Our Double LH-STM model consistently outperforms other methods for the 40 pre- dicted frames. The improvement larger for longer-term frame prediction into the future (40 frames). Qualitative results be found in FFigure and Appendix B (Figure and Figure 13). These results show that motion and human shape predicted by Double LH-STM are the closest to the ground truth.
Our models compared with FRNN (Oliu al.
2018), PredRNN (Wang et al., 2017). Pre-
dRNN++ (Wang et al. 2018a), E3D-LSTM (Wang et al.
2018b), SAVP-VATE and SAVP- Deteemmiiiiiiiicc
models (Lee al. 2018 Our model outperforms all other methods.
Xx-2
2-1
History
LH-STM
PSNR
SSIM
VIF
X
X
X2
Single Double
25.78
0.779
0.302
Input
Ha-a
26.45
0.788
0.308
H4-1
Single 27.83
0.816
0.354
State
Double
28.64
0.826
0.367
&
11'2.
Table 1: Comparison of using input vs state his-
Figure 3: The use of input history with LH-STM
tory with LH-STM Single and Double on the KTH human actions dataset (resolution 128 x 128).
Single (left) and Double (right)
4.2 STATE HISTORY VS INPUT HISTORY
The proposed LH-STM models use past hidden states. Alternatively, input frames can be directly applied to LH-STM as illustrated in Figure 3 We compared the performance of using input and state history with the same LH-STM models and the same model size in Table oon the KTTH human actions dataset. Figure 7 shows a qualitative comparison of using state history (Double LH-STM) and input history (Double, input history). The results show the importance of using history states with LH-STM and that the state history is more suitable for videos.
4.3 LONGER PREDICTION
To test the performance of longer-term prediction (more than 40 frames), we additionally trained the KTH human actions dataset with resolution of 64 x 64 pixel:" Figure 4 and Figure 5 compares the performance of 80 frame prediction with Double LH-STM and two state-of thht--ff models AVV. VAE and SAVP-Deterministie (Lee et al. 2018)) on all action classes and three action classes only (hand waving, hand clapping, and boxing), respectively. We compare only these three action classes with all action classes to show that the high performance at longer frames is not due to background copy. The human in other action class videos (running, jogging, and walking) may disappear after certain frames. Additionally, Figure 13 shows the performance comparisons of each action class. The experimental setup is the same as the experiments of a resolution 128 x 128 (Section [4.1). From these comparisons, we can observe that Double LH-STM results are generally better than
3 We note that Lee et al. (2118) reported the performance of VAE+GAN, VAE only, GAN only, and of their deterministic models. We select their best (VAE only) and the most comparable (deterministic) models compare against. To evaluate the VAE model, we take 100 random input samples for each video following Lee al (2188). The reported numbers this paper are their method's "median" output among the 100 randomly generated ones, compared to the ground truth. Please see Appendix Cfoor the discussion of this evaluation setup
*Due lack of GPU memory, we could use the original resolution for 80 frame prediction.

---

Under review as a conference paper at ICLR 2020
HAVP-Deterministe and -AAE on all metrics. These also show that our model is not just copying the background when predicting more than 40 frames.
32
Double Lat-SI (ours)
SAVP-Dor (Las, (25 65)
Desble LIH-STM (ouro 015.01) SAVP-Du (23.01) SAVP VAA (Lec. 2014) (22.68)
Double LH-STM (vun))
SAYP Dei (Lee, 2018: (15.08) SAYP VAB (Lee. (17.5))
31
SAVP-VAE (Les 2018) (25
25
20
25
20
40
40
40 Time stepe
170
10
13 Time steps
20
25
Time steps
0.90
Doubble LH-STM (ours (17)88)
0.90
Doubie LH-STM (ean) (0.344)
Dowhle LH--TM (nxn) (3)
SAVP-De 2018)
SAWP-Dai (Lec, 2018: (0.681)
083
SAVP-De: (Les, 2013) (..00)
0.85
SAVP-VAB (Len, 2018 (0.000)
0.85
SAVI-VAR (La 201 (0.667)
0.85
SAVP-VAB (1.78)
0.80
084
SSM
55054
000
0.75
0.70
0.70
0.74
0.65
0.65
0.66
0.800
20
40
60
80
0.00
0.74)
10
15 Time stops
20
25
Time steps
Time stops
0.925
Double LE STM (ouEs) (0.799)
01925
Double STM (ours) (0813))
Double LHH-TMM (ours (0 (0.894)
0.903
(Lee. 2019) (UME))
SAVP-Der (Lin, 2018) (U))
0.90
SAVP-DeI dlec, (0875)
SAVP-VAR (L.v, 2018) (177))
0.900
SAVI-VAR (0.795)
0.91
SAYP-VAE (Lee, 2018: (0.865)
0.850
0.50
0.850
6.25
0.99
0.000
0.825
0.58
0.775
0.800
0.87
0.750
0.715
0.56
0.25
0.750
20
201
60
60
0.85
steps
Time dtrp
stops
Figure 5: Per- comparisons the KTH human actions dataset ("boxing', 'waving', and 'clapping' BAIR robot pushing dataset
Figure 4: Per-frame comparisons on the KTH human actions dataset (all actions, 64 x 64, 80 frame prediction). (-): averaged score
Figure 6: Per-frame comparisons
actions, 64 x 64, frame prediction). (-): averaged score
(25 frame prediction)
(-): averaged score
4.4 BAIR ROBOT PUSHING DATASET
Our experimental setup follows that of Lee et al. 2018). All models are conditioned on five frames to predict the next ten frames [] The models are tested to predict 25 frames. We evaluated our LH-STM models (Single and Double) and compared them with the base model (ContextVP), the VAV.-VAE. and Dete-meteriiiistic models (Lee et al. 2018) in Table Our Double LH-STM achieved better PSNR and SSIM scores than the SAVP-Deterministic and SAVP- VAE models. It can be seen from the per-frame prediction results (Figure 6). that the performance LH-STM is higher over time, especially on the PSNR and SSIM metrics. On the LPIPS metric, the performance of Double LH-STM is comparable to SAVe-Dettrmiiisticc and better than SAVP-VAE Qualitative results can be found in Appendix E (Figure 17 and Figure 18). This dataset contains random robot motion which makes long-term fangtterm/in prediction extremely hard with deterministic models. However, the overall motion prediction with Double LH-STM and SAVP AAVe-Detrmministic is still reasonable compared to SAVP-VAE.
5 LLe et al. 2018 used two input frames. We increase the number of input frames as motion in two frames is almost not visihle. We increased the number of input frames to provide enough history to the model. We also re-trained the models Lfe et al. (2018) with five input frames.
7

---

Under review as a conference paper at ICLR 2020
Input
Predicted Frames (top: Ground truth)
15
19
27
35
39
43
47
Double LH-STM (ours)
Single LH-STM (ours)
Double, input history
ContextVP Byeon 2118
Bet-
2018
SAVP- VAP
Figure 7: Qualitative comparison of Double LH-STM, Single LH-STM, Double LH-STM with input history (See Section 4.2 for the details), the base model (ContextVP (Bycon et al. 2018)). and the state-of-the-an model (SAVP-Va) VAE and overall produces the sharper images and greater realism compared to other models. Compare to the Deterministic model (Lee al. 2018), model can capture more precise motion.
(Lee et al. 201 8). Double LH-STM in
Table Evaluation of multi-frame prediction on the KTH human actions dataset (resolution 128 x 128). All models are trained for 10 frame prediction given 10 input frames. They are then tested to recursively predict 40 future frames. The reported metrics are averages across the predicted frames. Higher values indicate better results. (*) We re-trained the model using the provided code by /Wangg ct al. (2018a) as their originally reported numbers are not with the same experimental setup as ours. Appendix F for training details.
See
Method
PSNR
SSIM
VIF
parameters
ConvLSTM (ingrian et 2015)
22.85
0.639
FRNN (Oliu et al.
2018)
23.77
0.678
PredRNN (Wang al.
2017
24.16
0.703
PredRNN++ (original)
Wang et al. 2018 8a
25.21
0.741
15.1M
*PredRNN++ (retrained) Wang et
2018a
26.37
0.734
15.1M
SAVP-VAE (retrained) (Lee et al. 20 g
26.23
0.777
0.289
9.8M
SAVP-Det (retrained) (L (Lee et al.
2018)
27.59
0.796
0.328
8.9M
E3D-LSTM (Wang et 80186
27.24
0.810
ContextVP-4 (base model) Byeon et al.
2018
26.37
0.785
0.320
17.3M
Single LH-STM (ours)
27.83
0.816
0.354
18.6M
Double LH-STM (ours)
28.64
0.826
0.367
16.2M
5 CONCLUSION
In this paper, we have introduced new RNN structure, LH-STM, which incorporates long history states into the recurrent unit to learn longer range dependencies. This module combined with the Co architecture allowing context aware long-term video prediction. The resulting model produces more realistic predictions for a longer period of time compared to the recent sfate-of-the-an methods. The proposed approach easily be extended to standard LSTM or Gated Recurrent Units (GRU) and applied to tasks that require long-term term memory trace, e.g., language modeling or speech recognition.
8

---

Under review as a conference paper at ICLR 2020
Table 3: Evaluation of multi-frame prediction on the BAIR robot pushing dataset. All models are trained for 10 frame prediction given 5 input frames. They are then tested to predict 25 future frames. The reported numbers are averages across the predicted frames, Higher values indicate better results.
Method
PSNR
SSIM
VIF
# parameters
SAVP-VAE Lee et al. 2018
17.32
0.779
0.524
6.7M
AVP-Determimisti (Lee et al. 2018)
18.08
0.801
0.544
6.4M
ContextVP-4 (Bycon al. 2018
18.24
0.817
0.513
17.3M
Single LH-STM (ours) Double LH-STM (ours)
18.64
0.817
0.523
17.6M
18.82
0.825
0.547
16.3M
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint 1eXiv:1710.11200 2017.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 753-769, 2018.
Víctor Campos, Brendan Jou, Xavier Giró-i Nieto, Jordi Torres, and Shih-Fu Chang. Skip rnn: Learning to skip state updates recurrent neural networks. arXiv preprint 1.X.v:1708.06835 2017.
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp- 551-561, 2016.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hicrarchical multiscale recurrent neural networks arXiv preprint 1609v 2016
Emily Denton and Robert Fergus. Stochastic video generation with a learned prior. In 35th Inter- national Conference on Machine Learning, ICML 2018, pp. 1906-1919, International Machine Learning Society (IMLS), 2018.
Emily L Denton al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. 4414-4423, 2017.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. arXiv preprint 170.0217,, 2017.
Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In Advances in neural information processing systems, pp. 499. 1996.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through vidco prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Aistats, volume 9, pp. 249-256, 2010.
Tao Gui, Qi Zhang, Lujun Zhao, Yaosong Lin, Minlong Peng, Jingjing Gong, and Xuanjing Huang. Long short-term memory with dynamic skip connections. arXiv preprint arXiv: 1.03873, 2018.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. A clockwork mnn arXiv preprint 1402 2014.

---

Under review as a conference paper at ICLR 2020
Ivan Laptev, Barbara Caputo, et al. Recognizing human actions: local svm approach. In mill, pp. 22-36.1 IEEE, 2004.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 01523, 2018.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint 1605 00004, 2016.
Michael Mathieu, Camille Couprie, and Yann LeCun, Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.0544/ 2015.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint arXiv:/007.05588, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. arXiv preprint 17X.:v1708.02182.. 2017.
Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 5915-5924, 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network training for long or event-based sequences. In Advances in neural information processing systems, pp. 3882-3890, 2016.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in neural information processing systems, pp. 2863-2871, 2015.
Marc Oliu, Javier Selva. and Sergio Escalera. Folded recurrent neural networks for future video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 716-731, 2018.
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6022 2013.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compres- sion. Neural Computation, 4(2):234-244 1992.
Hamid R Sheikh and Alan C Bovik. A visual information fidelity approach video quality assessment. In The First International Workshop on Video Processing and Quality Metrics for Consumer Electronics, volume 7, pp. 2. sn, 2005.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Rohollah Soltani and Hui Jiang. Higher order recurrent neural networks. arXiv preprint 00004 2016.
Marijn F Stollenga, Wonmin Byeon, Marcus Liwicki, and Juergen Schmidhuber. Parallel multi- dimensional Istm, with application to fast biomedical volumetric image segmentation. In Advances in neural information processing systems, pp. 2998-3006, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint ArXiv:1706.08030 2017a.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn. Xunyu Lin, and Honglak Lee. Learning to generate long-term term future via hierarchical prediction. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 3560-3569. JMLR. org, 2017b.
10

---

Under review as a conference paper at ICLR 2020
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1020-1028. 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics, In Advances In Neural Information Processing Systems, pp. 613-621, 2016.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip. Predrnn: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, pp. 879-888, 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long. Jianmin Wang, and Philip S Yu. Predrnn++: Towards a resolution the deep-in-time dilemma in spatiotemporal predictive learning. arXiv preprint 0rx4vv1804.06300 2018a.
Yunbo Wang, Lu Jiang, Ming- Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d 1stm: A model for video prediction and beyond. 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Fero P Simoncelli, et al. Image quality assessment: from error visibility to structural similarity. IFEE transactions on image processing, 13(4)6600-612, 2004.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv: 1906.0269000644 2019.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision. arXiv preprint 1rXiv:1806.044 04768, 2018.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189 4198. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2020
A NETWORK DETAILS
Each layer of ContextVP-4 contains five Parallel Multi-Dimensional LSTM (PMD) units (see Fig ure 8). The parameters opposite directions are shared. followed the same network architecture
as Byeon et al. 2018) except for the number of neurons.
(h
(h
(h
LH-STM (h-)
(b-)
LH-STM (+-
LH-STM (h+)
v--Blending
Cutput Layer
LH-STM (971)
(T
LIF-STM (w+)
LH-STM (w++
Input
1.-STM
(H-)
SI
Soure
LH-STM (w-)
grediction
[-)
Figure 8: ContextVP-4 network archtecture
Weighted Blending: This layer learns the relative importance of each direction during training.
S. = [lt H Hwt _.1 .HI M₃ f(Wx Sp + bk),
(8)
where H is the output of the PMD units from h+, h-, w+, w-, and t- directions, and W the weight matrix.
Directional Weight-Sharing: Weights and biases of the PMD units in opposite directions (i.e, the h+ and h directions, w+ and w- directions) are shared.
K
LISTN
LHSTTM
LHISTMM
LEST
LEISTM
LHISTM
LHSTT
LESTY
X k-2
X k-1
X
Figure 9; A graphical illustration of skip connections for the direction é-, Blue dotted lines indicate concatenation-base skip connection in the LH-STM block. A green solid line indicates a gated skip connection across the 1-3 and 2-4 layers.
Skip connection in recurrent block (R-Skip): skip connection is commonly used in LSTM layers (Melis et al., [017 Merity et al.) 2017: Wang et al. (2018)) for long -tem gradient propagation. Our proposed model concatenates outputs the previous and current LSTT layers Figure 9p.
Gated skip connection across layers (I-Skip): Unlike in (Byeon et al. 2018), We added the gated skip connection across layers. In our model, the multiplicative gate is added to control the flow of information across layers. It stabilizes learning as depth increases. A layer is formulated as:
Y H Ti X
T)), <1,
(9)
12

---

Under review as a conference paper at ICLR 2020
where H, = X1 W." + and Ti oX, W7" + +b"). X₁ is the input the 1th layer, T is gate, and H, is the transformation of the input at the layer
We adjusted the number of neurons, all models have similar model size ( 17M). LH-STM Double contains 64-64-128-128 neurons for the imput size of 64 x 64 pixels and 64- 14-128-122 1-1-122-222 for 128 x 128. LH-STM Single contains 128- 188-128-18 128-128--111111111 128-12882122 128 neurons for the input size of 64 64 pixels and 14-128-128-256 for 128 x 128.
The initial learning rate is 1e found by grid search between 1e and 1e 5. The learning rate is decayed every 10 epochs with the decay rate 0.95. Weights are initialized using the Xavier's normalized initializer (Glorot & Bengio 20TO) and the recurrent states are all initialized to zero
B ABLATION STUDY
We performed a series of ablation studies evaluate the importance of each component of our proposed model 1) RNN type (standard ConvLSTM. Single LH-STM, Double LH-STM); 2) different losses (L1, C2. and the perceptual loss); adding skip connections 3) in recurrent blocks; the KTH human and 4) across layers (see Section 3.3 for details). We evaluated performance actions dataset with a resolution of 64 x 64 pixels. The weights for the loss, \1. X2, and Apl. are tested between zero one. The results of this experiment are summarized in Table 4 We found that performance improves by adding skip connections in recurrent blocks and across layers. Double LH-STM outperforms the standard ConvLSTM and Single LH-STM. Combination of L1, L2, and Cpt also improves the performance (with \1 = 1, X2 = 1. Apt = 1).
C EVALUATION OF STOCHASTIC MODELS
SAVP- numbers in the paper by Lee et al. (2018) obtained by first generating 100 output samples per input and then selecting the best generated sample for each input based on the highest score between the output and the ground truth. This evaluation measures the best prediction the model can do, given 100 tries. However, it does not provide an overall picture of the generations from the model. In real world scenarios, the ground truth is not available, the 'best' samples cannot be picked as done in such evaluations. Therefore, we re-computed the scores for stochastic models based on their 'median' output sample (instead of best) among the 100 randomly generated ones, compared to the ground truth. This strategy measures how well the model can be expected to perform on average, and so it is more representative score.
Table 4: An ablation study on KTH human actions dataset. The frame resolution is 64 x 64 pixels. All models are trained to predict 10 future frames given 10 input frames. They are then tested to recursively predict 40 future frames. The reported numbers are averages over all the predicted frames.
RNN type
Loss
R-Skip
L-Skip
PSNR
SSIM
L
27.74
0.812
ConvLSTM
L1
27.96
0.812
L1
28.41
0.812
CL
V
28.59
0.818
Single LH-STM
CI+LN
/
28.46
0.817
C
V
V
28.69
0.812
Ci+Lo
V
28.20
0.811
Double LH-STM
Li+L2
V
29.01
0.818
29.05
0.820
13

---

Under review as a conference paper at ICLR 2020
D EVALUATION WITH LARGE AND SMALL MOTION
We split the video samples into five bins by motion magnitude computed as averaged L2 norm between target frames similar to the evaluations by Villegas et al. (2017a). Figure 12 shows the comparisons on the KTH human action dataset, and ?? shows on BAIR action free robot pushing dataset. Overall, across all metrics, all models perform worse when the motion is larger. For PSNR and SSIM, Double LH-STM achieves the best performance except for the largest motion. For LPIPS, Double LH-STM performs better for small motions (first two bins). SAVP-Deterministic performs the best on this metric for larger motions.
E QUALITATIVE RESULTS
show prediction results on the KTH human actions dataset, and Figure 17 and FFiuure 18 presents the results the BAIR action free robot pushing dataset. Figure 16 and Figure include the samples with one of the largest motions for both datasets. When motion is larger, all models fail produce reasonable predictions, whether they are deterministic stochastic.
Figure 14 and Figure 13
F RETRAINING PREDRNN+
As mentioned in the paper, re-trained PredRNN++ the KTH human actions dataset using the provided code by Wang et al. (2018a) (https: / com Yunbo /.tpprddrnn pp). We conducted grid hyper-paramette search for the learning rate and the patch size. We found in the code that there is an option to divide the input frames into smaller patches. The learning rate search was performed the range between 1e and le The range of patch size was 2, for both frame resolutions (64 x 64 and 128 x 128 pixels). The reported results in the paper are for learning rate of le and the patch size (full resolution).
14

---

Under review as a conference paper at ICLR 2020
36
Duutte LIB-TMM (mm)) (31.72)
Double LB (oun) (0.822)
0.925
Duubie LH-STM (wun) (0.442)
SAVP Det (Lee, 2018) (29.99)
090
SAVP Det (Lee, 2018)
01900
SAVP De (Lee. 2018) (0.12))
34
SAVP (2554)
SAWP-VAI (Lee, 2010 (0:44)
SAYP-VAL (Lee, (0.799
0.875
PSNR
22
555M
0.850
080
0.825
30
26
0.70
0.750
0.725
46
40
10
20
40
50
0
20
40 Time steps
40
Time steps
Time dips
32
DoutNo (su))
0.90
Dowble (sun) (1277)
0.825
Dautle LH-STM (sum)
37.
SAVI-De: (Laz. 2018) 14)
SAVP-De (Lax, 2018) (0.71)
0.900
SAVP-Du (Lax, 2018) (0. 795)
30
SAVP-VAE 2015)
0.5
SAMP-VAE (Lee. 2018) (0:592)
0.875
SAVP-VAE (Lee. 2018) (0:779)
29
080
0.850
PSVR
SSIM
0.255
26
0.25
27
0.70
0.750
065
0.725
0
20
40
50
50
X0
40 Time dap
60
50
0
20
40 Time stops
00
80
Time stops
Doulle LH-STM (wu)) (24.))
090
Doobla LHHSTM (win) (0.71)
D LH--TT (muno d.
30
SAVP-Dei 2018) (22.4)
SAWP-Det (Lez, 20
SAVP-Des 2018) (0.783)
SAVP-VAE (Lee. (2177)
085
SAVP-VAE VAE (Lee. 2018) (0:549)
SAVP (Les. 2015) (0.755)
0.0
PSNR
26
0.75
24
0.70
0.75
22
0.70
20
40
50
0.60
20
40
steps
steps
steps
DouNte LH STM (ouns) (19.0))
0.90
Double LH STM (NUS)) (L631)
Double LE STM (ouE)) (2.724)
2018; (18.0))
085
SAVP-De 2018)
(L.), 201 (0747)
SAVI-VAE (Lass, 201 (18.19)
SAVI-VAE (Lax, 2018) (0.591)
SAVR-VAE (lac, 80 (0.799)
0.80
PSNN
2..50
0.70
0.65
0.75
20
0.60
19
18
0.5
Time
Time
Time
25
Douuee LIM (ouns) (19.00)
085
DouNe LH STM (ours) (0,66)
0.90
Double LH STM (ouns) 01 (070))
SAVP-De (Len, 2018) 14)
SAVP-Der (Loz, 2018) (0.5)
SlAVP-Der (las, 201 8) (2.7189)
24
SANP-VAE (Lav, (19.3) (19.38)
0.00
SAVI-VAE (Los. 2013: (0.68)
0.55
SAVP-VAR dLec, 201 (0.514)
025
0.50
8 21
0.75
20
0.70
0.65
19
0.55
18.
0.60
40
0.50
Timee
Timee stops
Time steps
Figure 10: fer-frame comparisons of KTH human actions dataset (80 frame prediction, resolution 64x64) with motion-based test set separation. We split the video samples into five bins by motion magnitude computed as averaged La norm between target frames following Villegas et al. (2017a). The first row is with smallest motion and the bottom row with largest motion. (.) indicates averaged score.
15

---

Under review as a conference paper at ICLR 2020
0.94
Duabie LIBSTM (mm)) (22.5))
26
SAVP Det (Lee, 2018) (21.3))
092
0.98
25
SAVP-NAB (Lee, (2004))
0.0
0.92
24
PSNR
23
0.991
0.90
21
0.4
Double LISSTM (ous) (0.905)
Deubi: LH STM (ous)) (0,929)
SAVP-De (Lax; 2018) (0.553)
SAV dlat, 201 5 (0 9272)
20
OR2
SAVP-VAE (0.857)
SAVP VAE 2018) (0.91))
15
15
10
20
10 Time steps
20
Time steps
Time dips
Doutho (sm) (19.))
23
SAVP-VAE (151)
SAVP-De: (Laz. 2018) (18.64)
0.500
0.90
22
0.50
PSNR
0.990
LPPS
0.552
20
0.88
19
0.975
Douile LH-STM (oum) (0.55)
SAYP-Da (Ler, 2015) (0.22))
0.56
Duabie LHH-TTM (miro (0 895)
DAV (Lez, 2013 5) (0, (9.9
16
SAVP (Lee 2018: (0.83)
SAVP VAE (Lec. 2015) (0.592)
30
15 Time stops
0.790
15
0.54
10 Time stops
20
daps
0.900
Dustic LH-STM (oun) (18.
Double LH-STM (ourd) (b. 7990
Duothic LH-STM (muro di 849
2.1
SAVE (Lee, 2015) (55.15)
SAVP-Dei Dei (Let, 2013) (16.22
0.375
SAVP-Dei (Lay, 201 (2.763)
SAVP-VAR (Laz, 2018) (t
SAVP (Les, 2015) (0.543)
SAVP-Des 2015) (0.85))
0.550
20
0.925
PSNS
0.500
0.56
0.750
0.54
0.725
0.52
25
15 depss
5
10
20
steps
steps
Doubte LH STM (ouns) (1677))
0.975
Double LH STM (ouzs) (U763)
Double LB STM (our)) (7.818)
19
SAVI-VAE (Lass (15.3)
2018; (15.5)
0.550
SAVE-VAF (Lau, 2018) (6 (0.766)
201
SAVP-VAE (lac, 201 (0 (3.34)
201 (0:3)
0.525
0.68
0.400
PSNRR
0.775
0.750
0.84
0.728
36
0.900
0.50
Time
Time
Time
0.90
16
Douue LI-STMM (ouns) (15.0))
0.85
Douhle LH-STM (ouns) (U736)
Double LIKSTM Cours) 7999
SAVP-De (Lee, 2018) (15.0))
SAMP-Der (Lan, (0
0.98
SAVT-DD (Lax: 2018) (0815)
SAVP-VAE (Lax, (13.))
SAVI-VAE (Lac, 2018) (0.546)
SAVP-VAL (Lee. 2018) (0, (1783))
17
0.0
0.56
PSNRR
0.75
2052
0.70
0.40
0.65
0.78
14
0.76
10
Timte
Timte stops
Time steps
Figure 11: Per-frame comparisons of BAIR robot pushing dataset with motion-based test separation. We split the video samples into five bins by motion magnitude computed as averaged La norm between target frames following Villegas et al. (2177a). The first row is with smallest motion and the bottom row with largest motion. (-) indicates averaged score.
16

---

Under review as a conference paper at ICLR 2020
Hand waving
Hand waving
Hand waving
Doutle STM (MIS)) (24.5)
090
Double LB (ounS (L742)
0.925
Double LH (OUNS) (UMIN)
30
SAVP-De (Lee, 2018 (22 95)
SAVP-VAE 2015) (2.47)
085
SAMP-VAR (Las. 2018)
SAVP-I (Lan, 2018) 60,88
0.500
SAVP-DD (Leo, 2018 (1:8)
SAVI-VAE (Las, 2018) (0.7k7)
0.875
0.850
0.825
0.800
0.70
24
0.775
0.5
0.750
23
0
25
50
0.00
X5
40 Tlate stops
10
0.125
20
40
60
20
Tlate stops
Time stops
Hand clapping
Hand clapping
Hand clapping
DoNNNl LH-STM (wm)) (25
0.90
Diahle LH-STM (mwm) (0.77)
0.925
Doustle LHHSTM (wwn) (0 (884))
SAVP-Dei (Lev, 2018) (23.57)
SAWP-Dei (Lan, 2018) (9.68)
SAVP-DeI (Lax, 2018) (0:818)
SAVE-VAE (Lee. (23.21)
SAVP-VAE (Lee. 2013) (0.682)
0.500
SAVP-VAE (Lee, 2018) (0.515)
PSNR
5504
0.800
3.75
0.825
0.70
24
0.5
0.775
23
0.750
60
20
40 Timee steps
Time steps
Timte steps
Boxing
Boxing
Boxing
Double LH-STM (wun) (24.48)
0.90
Doable LH-STM (sun) (8.724)
Double LH-- STM (ouns) 0 783)
SAVP Det (Lee. (2277)
SAMP (Lee, 2015) (0.674)
0.50
SAVP Det CLee. 201 5) (0.794)
28
SAYP (Leo, (2.3)
(22.1)
SAVP-VAL (Lee, 2018) (0A64)
SAYP-VALL (Lee. 20139 (0.785)
27
080
26
1PPS
0025
0..0
24
0.20
0.75
23.
0.60
Time
Time dipps
Time
Figure 12: 80 frame prediction on KTH human actions dataset separated by actions: hand waving, hand clapping, and boxing. (-) indicates averaged score.
17

---

Under review as a conference paper at ICLR 2020
Jogging
Jogging
Jogging
Doutle STM (MIS)) (29.5))
090
Double (1796)
34
SAVP-De (Lee. 2018) (2) 10
SAVK-De (Lon, 2018) (5)
SAVP-DDe dles, 201 (0.5)
SAVP-VAE
085
SAMP-VAR (Las. 2018) (0.725)
SAVI-VAE 2015)
PSNR
LIDS
0.80
0.70
26
0.75
065
0
25
50
80
X5
40 Tlate stops
50
50
0.10
20
Tlate stops
Time steps
Running
Running
Running
36
Doubli LX--TM (oum) (1000)
0.90
Dubiie LH-STM (mum) (0) (0,70)
Duublic LH--TM (mars) di
SAVP-Det (Les, 2018) (28.8))
SAVP-Dei (Lan, (0.18)
0.50
SAVP-De claz, 2015) (0.303 SAVP-V VAA (Lee, 2015) (0.755)
SWVP-VAB (Lee. (27.)
SAVP-VAE (Lee 2018) (0.718)
36
0.15
PSNN
5504
3975
0,.50
30
0.70
0.75
26
065
Time steps
Timte steps
Time steps
Walking
Walking
Walking
Double LHESTM (mun) (29:4))
0.90
Double LH-STT (oun) (0. (8.75)
Double LH- STM (ouns) d 786)
34
SAVP Dei (Lee.
SAMP (Lee, 2015) (0.75))
SAVP Der fLee. 201 (0:515
SAYP-VAU (Lee, 201 (25.5))
SAVP-VAL (Lee, 2018 (0.723)
SAVP-VAU (Lee. 2013 (0.733)
0.55
080
PSNR
1PPS
3025
3,50
0.70
0.75
26
0.60
Time
Time dipps
Time
Figure 13: 80 frame prediction on KTH human actions dataset separated by actions: jogging, running, and
walking. (-) indicates averaged score.
18

---

Under review as a conference paper at ICLR 2020
Input
Predicted Frames (top: Ground truth)
t=1
7
11
15
23
31
35
1
1
1
N
1
Double LH-STM (ours)
1
1
1
1
Single LH-STM (ours)
1
Double, input history
VV Bycon et al. 2018
SAVP-Det
211
SAVP-VAE L. al.
2018
Double LH-STM (ours)
Single LH-STM (ours)
Double, input history
VP
2018
SAVP-De Lee et al. 2018
SAVP-VAE
Figure 14: Qualitative comparison of Double LH-STM. Single LH-STM, Double LH-STM with input history (See Section 4.2 for the details), the base model (ContextVP (Byeon et al. 2018)). and the state-of-the-ae art models actions dataset. Top row shows the ground truth, and the rest is the prediction results. The model is trained for 10 frames and predicted for 40 frames given 10 frames.
and SAVP-VAE et al. (0118) on the KTH human
19

---

Under review as a conference paper at ICLR 2020
Input
Predicted Frames (top: Ground truth)
11
15
19
23
27
31
35
39
43
47
Double LH-STM (ours)
Single LH-STM (ours)
Double, input history
CP Ryoon al
2018
SAVP-Det Lee
SAVP-VAE cl.
2018
Figure 15: Qualitative comparison of Double LH-STM, Single LH-STM, Double LH-STM with input history (See Section 4.2 for the details), the base model (ContextVP (Byeon al.) 2018)), and the state-of-the-art models (S (SVV--Detrmministt and LAVP-VAE et al. (2018) on the KTH human actions dataset. Top row shows the ground truth, and the rest the prediction results. The model is trained for 10 frames and predicted for 40 frames given 10 frames.
Input
Predicted Frames (top: Ground truth)
t=1
11
15
19
23
27
31
35
39
43
47
Double L.H-TMM (ours)
SAVP-Det Lec 2018
SAVP-VAE
Lee et 2018
Figure 16: An example video with of the largest motion the KTH human actions dataset. The motion magnitude is computed as averaged C2 norm between target frames following Villegas et al. (2017a). Top row shows the ground truth, and the rest the prediction results. The model trained for 10 frames and predicted for 40 frames given 10 frames. When motion is larger, all models fail to produce reasonable predictions.
20

---

Under review as a conference paper at ICLR 2020
Input
Predicted Frames (top: Ground truth)
3
5
12
24
Double LH-STM (ours)
SAVP-Det
Lce ct 2018
SAVP- VAE Loe et al. 2018
Double LH-STM (ours)
SAVP-Der Lee 2018
SAVP-VAE
Figure 17: Qualitative comparison of Double LH-STM and the state-of-the-art models (SAVP- Deterministic and SAVP-VAE Lee et al. (2018) on the BAIR robot pushing dataset. Top row shows the ground truth, and the rest is the prediction results. The model is trained for 10 frames and predicted for 25 frames given 5 frames.
21

---

Under review as a conference paper at ICLR 2020
Input
Predicted Frames (top: Ground truth)
Double LH-STM (ours)
SAVP-Det Lee et al. 2018
SAVP-VAF
Lo ctall 2018
Figure 18: Qualitative comparison of Double LH-STM and the state-of-the-ar models (SAVP. Deterministic and SAVP-VAE. Lee et al. (2018) the BAIR robot pushing dataset. Top row shows ground truth, and the rest is the prediction results. The model is trained for 10 frames and predicted for 25 frames given 5 frames.
Input
Predicted Frames (top: Ground truth)
6
12
18
24
30
Double LH-STM (ours)
SAVP-Det Lee et 2015
SAVP-VAE
Loc at 2018
Figure 19: An example video with one of the largest motion on the BAIR robot pushing dataset. The motion magnitude is computed averaged L2 norm between target frames following Villegas et al.) (2017a). Top row shows the ground truth, and the rest is the prediction results. The model trained for 10 frames and predicted for 25 frames given 5 frames. When motion is larger, all models fail to produce reasonable predictions.
22


Reviews:
[{"review_id": "rkgTTMcKKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new RNN unit based on ConvLSTM for long-term video prediction. The proposed method is technically correct but lacks enough originality. I tend to reject this paper due to the following three reasons:\n\n1. The major novelty of this paper is the LH-STM unit, which applies a temporal attention approach to historical hidden states. This module is very similar to the Recall gate of the E3D-LSTM [Wang et al. 2018b]. Besides, the Double LH-STM looks like an incremental extension of the Single LH-STM. As mentioned, it is technically correct, and yet has limited novelty for an ICLR paper.\n\n2. The authors mainly compared the proposed network with the Context-VP model in the experiments (Fig 5, Fig 8, and Table 3), which is not enough. As far as I know, there are other existing methods for long-term video prediction, e.g. [Denton et al. 2017]. \n\n3. Another problem of the experiments is that Lee et al. [2018] proposed a stochastic model for video prediction, but the authors only compared the LH-STM with its deterministic version.\n\n4. In Figure 11, there is no significant improvement by using LH-STM. Also, the authors might include more compared models on the pushing dataset.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1xjMAW0tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a new LSTM architecture called LH-STM (and Double LH-STM). The main idea deals with having a history selection mechanism to directly extract what information from the past. The authors also propose to decompose the history and update in LH-STM into two networks called Double LH-STM. In experiments, the authors evaluate and compare their two architectures with previously proposed models. They show that their architecture outperforms previous in the PSNR, SSIM and VIF metrics.\n\n\nPros:\n+ New architecture that can use all computed state history in a sequence\n+ Outperforms previous methods in the used metrics\n\nWeaknesses / comments:\n- Larger number of parameters in comparison to ContextVP-4\nIn Table 1, the authors present a comparison to previous works and the respective number of parameters used for most of the methods. It is mentioned in the paper that Single LH-STM uses an architecture similar to ContextVP-4. Since the architectures are similar and the proposed method is added to this architecture, can the authors make sure that Context VP-4 and Single LH-STM have around the same number of parameters? This can give a more direct comparison in performance to make sure that the parameter boost is not the reason for performance boost.\n\n\n- Why retrain for longer sequences?\nThe authors have experiments where they attempt to predict past 40 frames into the future. However, they mention that for this experiment, they train another network that takes in 64x64 pixels. Optimally, they should just let the 128x128 network predict past 40 frames. Can the authors comment on why they don’t just do this? Is long-term prediction limited to how many previous states you can store in the GPU?\n\n\n- PSNR, SSIM, and VIF could be biased to blurriness and perfect background reconstruction\nIt has been shown before that PSNR and SSIM can be biased to blurriness and perfectly copying the background (Villegas et al., 2017b). Therefore, these metrics should be complemented with other metrics such as actual humans look at the videos and evaluated them for realism. The fact that these metrics look good in this paper can be due to blurriness. There is clear evidence of blurry predictions in the comparison Figures in the main paper and supplementary material. In addition these metrics, and VIF can also be biased to perfectly copying the background. I suggest the authors separate the data into videos with large motion and little motion similar to Villegas et al., 2017a. This way we can better evaluate this method on how well it predicts videos with large motion in comparison with videos where copying the background is enough.\n\n\n- Testing for 80 frame sequences is not very meaningful in this dataset.\nThe KTH dataset contains the action categories of running, walking, and jogging which most of these videos do not go up up 40 frames while the person still being in the frame. Therefore, after frame 40, the method is just required to copy the background and it will look like the future is perfectly predicted.. Can the authors clarify if they only tested on handwave, handclap and boxing? Handwave, handclap and boxing are the only categories that will still have a human present in the video at frame 80.\n\n\n- No video files are provided.\nFinally, this method does not provide any videos to better evaluate the proposed network. Looking at videos is necessary to observe temporal consistency and blurriness happening in the video. Or humans appearing and disappearing randomly.\n\n\nConclusion:\nThe proposed method is novel and interesting, but the experimental section has many issues as discussed above. If the authors successfully address these issues, I am willing to increase my score.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgpVslatS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a type of recurrent neural network module called Long History Short-Term Memory (LH-STM) for longer-term video generation. This module can be used to replace ConvLSTMs in previously published video prediction models. It expands ConvLSTMs by adding a \"previous history\" term to the ConvLSTM equations that compute the IFO gates and the candidate new state. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism. As such, it is not clear if there are significant differences between LH-STMs and previously proposed LSTMs with attention on previous hidden states. The authors propose recurrent units that include one or two History Selection (soft-attention) steps, called single LH-STM and double LH-STM respectively. The exact formulation of the double LH-STM is not clear from the paper.  The authors then propose to use models with LH-STM units for longer term video generation. They claim that LH-STM can better reduce error propagation and better model the complex dynamics of videos. To support the claims, they conduct empirical experiments where they show that the proposed model outperforms previous video prediction models on KTH (up to 80 frames) and the BAIR Push dataset (up to 25 frames).\n\nOverall I believe there are serious flaws with the paper that prevent acceptance in its current form.\n\nFirst, I believe the paper starts from the wrong assumption, namely that current video prediction models are limited by their capacity to limit the propagation of errors and to capture complex dynamics. Instead, it is well known that the main difficulty for longer term video prediction is to manage the increasing uncertainty in future outcomes. Stochastic models such as SVG-LP or SAVP are currently the state-of-the-art in video generation, with deterministic models not being able to generate more than a few non-blurry frames of video. While the authors mention that they do not focus on future uncertainty here, it is not clear how the proposed model helps to generate better longer-term videos when it does not deal with what actually makes long-term video generation difficult. In addition, it's misleading to claim that current models produce high quality generations for \"only one or less than ten frames\", especially without defining high quality. Models such as SVG [1] or SAVP[2] can produce non-blurry videos for 30-100 frames for the BAIR dataset, for example. \n\nThe experiments are missing 1) SVG as a baseline, 2) metrics that correlate with human perception such as LPIPS or FVD [3] and 3) qualitative samples that compare to stochastic models. Deterministic models can achieve very high PSNR/MSE/SSIM scores but produce very bad samples, as these scores are maximized by blurry predictions that conflate all possible future outcomes. This is highly apparent when looking at samples, and metrics that correlate better with human perception are usually better to compare video prediction methods. Comparisons to SAVP are found in Table 1 and 3 but there are no figures comparing samples from this model to the proposed model. The samples from the proposed model on the BAIR Push dataset for example (found in the appendix) are of significant lower quality than those reported from SAVP or SVG, and at the same time they are not longer-term than the predictions from these models. Consequently, the experimental section does not correctly assess how this model can generate better longer-term prediction than current models and it also does not give an accurate assessment of the model with respect to the current state-of-the-art.\n\nTo sum up, the paper does not adequately address how the proposed model allows for longer-term video generation. It is missing critical qualitative comparisons to state-of-the-art models such as SVG and it is unclear how the proposed model is different from a ConvLSTM with attention on previous hidden states.\n\n[1] Stochastic Video Generation with a Learned Prior. E.Denton and R. Fergus. ICML 2018\n[2] Stochastic Adversarial Video Prediction. Lee et al. Arxiv 2018\n[3] Towards Accurate Generative Models of Video: A New Metric & Challenges. Unterthiner et al. Arxiv 2018\n\n\n--- Post-discussion update ---\nThe authors have addressed a number of points raised by the reviewers and I'm raising my score to a weak reject from a reject. There are important remaining issues with the experimental section and the conclusions reached from their results, and therefore I still think the paper is below the acceptance bar.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new recurrent unit which incorporates long history states to learn longer range dependencies for improved video prediction. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism and can be directly added to ConvLSTM equations that compute the IFO gates and the new state. The authors perform empirical validation on the challenging KTH and BAIR Push datasets and show that their architecture outperforms existing work in terms of SSIM, PSNR, and VIF.
The main issue raised by the reviewers is the incremental nature of the work and issues in the empirical evaluation which do not support the main claims in the paper. After the rebuttal and discussion phase the reviewers agree that these issues were not adequately resolved and the work doesn’t meet the acceptance bar. I will hence recommend the rejection of this paper. Nevertheless, we encourage the authors improve the manuscript by addressing the remaining issues in the empirical evaluation.
Decision: Reject