Convolutional Tensor-Train LSTM for Long-Term Video Prediction
Authors: Jiahao Su
Wonmin Byeon
Furong Huang
Jan Kautz
Animashree Anandkumar
Venue:  (2020)
Keywords: Tensor decomposition
Video prediction

Abstract: Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames.Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations. A potential solution is to extend to higher-order spatio-temporal recurrent models. However, such a model requires  a  large number of parameters and operations, making it intractable  to learn in practice and is prone to overfitting. In this work, we propose convolutional tensor-train LSTM (Conv-TT-LSTM), which  learns higher-orderConvolutional LSTM (ConvLSTM) efficiently using convolutional  tensor-train decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio-temporal information at a small cost of memory and computation by using efficient low-rank tensor representations. We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and better/comparable results to other ConvLSTM-based approaches, but with much fewer parameters.

Under review as a conference paper at ICLR 2020
CONVOLUTIONAL TENSOR-TRAIN LSTM FOR LONG-TERM VIDEO PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across long range of image frames. Standard recurrett models are ineffective since they are prone to eπror propaga- tion and cannot effectively capture higher-order correlations. potential solution is to extend to higher-order recurrent models. However, such model requires a large number of parameters and operations, making intractable to learn and prone to overfitting in practice. In this work, we propose Convolational Tensor- Train LSTM (Conv-TT-LSTM), which learns higher-order Convolutional Long Short-Term Memory (ConvLSTM) efficiently using Convolutional Tensor-Train Decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio- neatio-temporl information with low memory and computational requirements by efficient low-rank tensor-train representations. We evaluate our model on Moving- MNIST and KTT datasets and show improvements over standard ConvLSTM and other -bsed approaches, but with much fewer parameters.
INTRODUCTION
Understanding dynamics of videos and performing long-term predictions of the future highly challenging problem. It entails learning complex representation of real-world environment without external supervision. This arises in a wide range of applications, including autonomous driving. robot control (Finn & Levine, 2017). or other visual perception tasks like action recognition or object tracking (Alahi et al. (016). However, long-term video prediction remains an open problem due to high complexity of the video contents. Therefore, prior works mostly focus next or first few frames prediction (Lotter et al. 2016 Finn et al. 2016 Byeon et al. 2018).
Many recent video models use Convolutional LSTM (ConvLSTM) 2013), where spatio-temporal information is encoded as a tensor explicitly in each cell. In ConvL- STM networks, each cell is first-order recurrent model, where the hidden state is updated based on its immediate previous step. Therefore, they cannot easily capture higher-order temporal corre- lations needed for long-term prediction. Moreover, they are highly prone error propagation.
basic block (Xingjian al.,
Various approaches have been proposed to augment ConvLSTM, either by modifying networks to explicitly modeling motion (Finn al. 2016), or by integrating spatio-tempont interaction in Con- vLSTM cells (Wang et al. 2017: 201 Sa). These approaches are often incapable of capturing long- term dependencies and produce blurry prediction.
Another direction to augment ConvLSTM is to incorporate a higher-order RNNs (Soltani & Jiang, 2016) inside each LSTM cell, where its hidden state is updated using multiple past steps. However, higher-order model for high-dimensional data (e.g. video) requires huge number of model param- eters, and the computation grows exponentially with the order of the RNNs. A principled approach to address the curse of dimensionality is tensor decomposition, where higher-order tensor is com- pressed into smaller core tensors (Anandkumar et al. 2014). Tensor representations are powerful since they retain rich expressivity even with small number of parameters. In this work, propose a novel convolutional tensor decomposition, which allows for compact higher-order ConvLSTM.
modifica- tion of CSmVLSTM, to build a higher-order spatio-temporal model. (1) We introduce Convolutional Tensor- Train Decomposition (CTTD) that factorizes a large convolutional kernel into chain of
Contributions. We propose Convolutional Tensor-Train LSTM

---

Under review as a conference paper at ICLR 2020
FW
output
output y
U
U tare
U work priy Soo
Convolutional Tensor-Train
Convolutional Tenaor-Train (CTT)
CAS
D
output
V
H
153
KB
K
K
2
prodas
comvoltti
selected current pix
2r1
canvolve tensor
Convolutional Tensor-Train (CTT)
H
(a) CTTD
(b) TO--STT-LSTM-FW
(c) Conv- TON-TTT-SSTM-SW
Figure 1: Illustration of (a) convolutional tensor-train (Eqs. and (6) and the difference between
convolutional tensor-train LSTM (b) Fixed window version (Eqs. TIa and (TO) and (c) Sliding window version (Eqs. (b and (TO)). The fixed window version all steps compute each input
to convolutional tensor-train, while sliding window version uses a window of steps for each input.
smaller tensors. (2) We integrate CTTD into ConvLSTM and propose Conv-TT-LSTM, which learns long-term dynamics in video sequence with a small number of model parameters. (3) We propose two versions of Conv-TT-LSTM. Fixed Window (FW) and Sliding Window (SW) (See Figures [bb and Ic), and we found that the SW version performs better than the FW one. (4) We found that trai- ing higher-order tensor models is not straightforward due to gradient instability. We present several approaches to overcome this such as good learning schedules and gradient clipping. (5) In the exper- iments, we show our proposed Conv-TT-LSTY, consistently produces sharp prediction over a long period of time for both Moving-MNIST-- and KTH action datasets. Conv-TT-LSTM outperforms the oftte-o--hh-arr PredRNN+ (Wang et al. 2018a) Sa) in LPIPS (Zhang et al,, 2018) by 0.050 on the Moving MNIST-2 and 0.071 the KTH action dataset, with 5.6 times fewer parameters. Thus, we obtain best of both worlds: better long-term prediction and model compression.
RELATED WORK
Tensor Decomposition In machine learning, tensor decompositions. including CP decomposi- tion (Anandkumar et al.] 2014). Tucker decomposition (Kolda & Bader 2009). and tensor-train decomposition (Oseledets, (201)) widely used for dimensionality reduction (Cichocki et al.) 2016) and learning probabilistic models (Anandkumar et al. 2014). In deep learning, prior works focused on their application in model compression, where the parameters tensors are factorized into smaller tensors. This technique has been used in compressing convolutional networks (Lebedev
al. 2014; Kim al. 2015 Novikov al. 2015 Su et al. 2018 Kossaili al. 2017] Kolbeins- son et al. 20199] Kossaifi et al. 2019), recurrent networks jjndra et al.) 2017 Yang et al., 2017) and transformers (Ma et al. | 2019). Specifically, Yang et al. 2017) demonstrates that the accuracy of video classification increases if the parameters in recurrent networks are compressed by tensor-train decomposition (Oseledets. 201T). eu al.) (2017) used tensor-train decomposition to constrain the complexity of higher-order LSTM, where each next step is computed based the outer product of previous steps. While this work only considers vector input at each step, we extend their approach to higher-order ConVLSTM, where each step also encodes spatial information.
Video Prediction Prior works on video prediction have focused on several directions: predicting short-term video (Lotter et al.] 2016: Byeon et al. 2018). decomposing motion and contents (Finn)
eet al., 2016 Villegas et al.] 2017 Denton et al. 2017 Hsich et al.) 2018), improving the objec-
tive function Mathieu et al. (2015) and handling diversity of the future (Denton & Fergus 2018
2

---

Under review as a conference paper at ICLR 2020
Babaeizadeh et al. [017 Lee al. 2018 Many of these works use Convolutional LSTM (ConvL- STM) (Xingjian et al. 20115) as a base module, which deploys 2D convolutional operations in LSTM to efficiently exploit spatio-temporal information. Finn al. (2016) used ConvLSTM to model pixel motion. Some works modified the standard ConvLSTI' to better capture spatio-temporal correla- tions (Wang 20.7 2018a). Waang et al. (2018b) integrated 3D convolutions into ConvL.STM In addition, current cell states are combined with its historical records using self-attention to efficiently recall the history information. [yeon et al. (2018) applied ConvLSTM in all possible directions to capture full contexts in video and also demonstrated strong performance using deep ConvLSTM network as a baseline. This baseline is adapted to obtain the base architecture in the present paper.
3 MANSOR-TRANE DECOMPOSITION AND SEQUENCE MODELING
The goal of tensor decomposition to represent a higher-order tensor as a set of smaller and lower- order core tensors, with fewer parameters while preserve essential information. In Yu et al. (2017). tensor-train decomposition (Oseledets) 201T) is used to reduce both parameters and computations in higher-order recurrent models, which we review in the first part of this section.
However, the approach in Yu et al. (2017) only considers recurrent models with vector inputs and cannot cope with image inputs directly. In the second part, we extend the standard tensor-train decomposition to convolutional lensor-train decomposition (CTTD). With CTTD, large convolu- tional kernel is factorized into a chain of smaller kernels. We show that such decomposition can reduce both parameters and operations of higher-order spatio-temporal recurrent models.
Standard Tensor-train decomposition Given an m-order tensor T 18/1x--xmm where I the dimension of its (-th order, standard tensor-train decomposition (TTD) (Oseledets) 2011) factorizes the tensor T into set of m core tensors {T())m
ym with T E R X such that
R
Pam
T Am
Σ 7m-1=1
T(2)
(m)
(1)
r==1
where tensor-train ranks {R (with Ro = Rm = 1) control the number of parameters in the
tensor-train format [. With TTD, the original tensor T of size (II= Ii) is compressed to EE 4Rt1R Ri) entries, which grows linearly with the order (assuming Rj's constant). Therefore, TTD is commonly used to approximate higher-order tensors with fewer parameters.
The sequential structure in tensor-train decomposition makes it particularly suitable for sequence modeling (Yu et al.] (0177). Consider a higher-order recurrent model that predicts a scalar output v E IR based on the outer product sequence of input vectors (u() E IR 1m according to:
Jr
Im Tes Am 2
(T,
(m)
(1)
(m) km
This model is intractable in practice since the number of parameters in T E R x (and therefore computational complexity of Eq. (2) grows exponentially with the order m. Now suppose T takes
tensor-train format as in Eq. I we prove in Appendix A that (2) can be efficiently computed
Ri
1 (I (1) ui V [m]
(3)
11-1=1
where the vectors (v()) E R M==1 are the intermediate steps, with (0)) E R initialized as v(0)) = 1, ((m)) Notice that the higher-order tensor T is never reconstructed in the sequential process in Eq. (3). therefore both space and computational complexities grow linearly (not
and final output
exponentially compared to Eq. with the order m assuming all tensor-train ranks constants.
Convolutional Tensor- Train Decomposition A convolutional layer in neural network is typically parameterized by 4-th order tensor T E R K'x R,,, Ro where K is the kernel size, R. and R are the number of input and output channels respectively. Suppose the kernel size K takes the form K m(k 1) + (e.g. K = 7 and m = 3), convolutional tensor-train decomposition
3

---

Under review as a conference paper at ICLR 2020
(CTTD) factorizes T into a set of m tensors {T 11=1 with T E RixxxRixRR-1 such that
R1
Ram
Tatt:Poe
T TTD * =T((2)
1i..2. * *....
(m) m.
(4)
where denotes convolution between 2D-filters, and {Ri]i= are the convolutional tensor-train ranks that control the complexity of the convolutional tensor-train format in Eq. the number of parameters in the decomposed format reduces from K2 Ro to
44 With CTTD, Ri)
Similar to standard TTD, its convolutional counterpart can also be used to compress higher-order spatio-temporal recurrent models with convolutional operations. Consider a model that predicts a 3-rd order feature V E IR. #x x Ro based on sequence of 3-rd features (2/01 E R& x Ra ym (where H, are height/width of the features and R is the number channels in ((()) such that
NATO
Will
Moriro *
u 1A0 with w = CTTD (1T(0)ER)) 14 E [m]]
(5)
where yy() E [ is the corresponding weights tensor for 210)) Suppose
each W() takes convolutional tensor-train format in Eq. 4 we prove in Appendix A that the model Eq. (5) can be computed sequentially similarly without reconstructing the original W(('s 's:
R. T:00 Hamum-
(1-1)) 1.1011
* (v! (ii + +2 vl E [m]
(6)
where (V(0) E RR*WW*R M= are intermediate results of the sequential process, where v(m) E R x W x m is initialized as all zeros and final prediction V = y(0) The operations in Eq. (5) is illus-
trated in Figure la In this paper, we denote the Eq. simply as V CTT((T()) Jmm (27)) )
4 CONVOLUTIONAL. TENSOR-TRAI LSTM NETWORKS
Convolutional LSTM is a basic block for most recent video forecasting models (Xingjian et al. 2013), where the spatial information is encoded explicitly as tensors in the LSTM cells. In Con- vLSTM network, each cell is a first-order Markov model, i.c. the hidden state is updated based on its previous step. In this section, propose convolutional tensor-train LSTM, where convolutional tensor-train is incorporated to model multi-steps spatio-temporal correlation explicitly.
Notations. In this section, the symbol is overloaded to denote convolution between higher-order tensors. For instance, given a 4-th order weights tensor W E IR xxxxxSxx and a 3-rd order input tensor X' Hx y W X computes 3-rd output tensor y E HR
Vi...
Σn=1 W.t.s.c X...... The symbol o is used to denote element- wise product between two tensors, and a represents a function that performs element-wise (nonlinear) transformation on a tensor.
Xingjian et al. (0155) extended fully-comeected LSTM (FC-LSTM) to Convolutional LSTM (ConvLSTMM) to model spatio-temporal structures within each recurrent unit, where all features are encoded as 3-rd order tensors with dimensions (height x width x channels) and matrix multiplications are replaced by convolutions between tensors. In ConvLSTM cell, the parameters are characterized by two 4-th order tensors W E k xxxxxx and T E R x KXCX4C where is the kernel size all convolutions and S and C' are the numbers channels of the input ((() E H xxxx x5 and hidden states H(?) Hx xC respectively. At each time step 4, a Con- vLSTM cell updates its hidden states (()) Hxx xC based on the previous step ((t-1) and the current input (t),, where H and W are the height/width that the same for X(()) and 7(0)
Convolutional LSTM
I(; F(0; 200 o05)
( x()) + +T+ H((111)
c(+) = (()) I(0; H(1) ( e cit)
(8)
where (-) applies sigmoid on the input gate I(t), forget gate F(t), output gate (lt), and hyperbolic tangent memory cell (1)), Note that all tensors ((t), I(t) F(0, (00) xXWX are 3-rd order.
4

---

Under review as a conference paper at ICLR 2020
Convolutional Tensor-Train LSTM In Conv-TT-LSTM, we introduce higher-order recurrent unit to capture multi-steps spatio-temporal correlations in LSTM, where the hidden state H(00 is updated based on its n previous steps (H(t-D) |² with an m-order convolutional lensor-train (CTT) as in Eq. ( Concretely, suppose the parameters CTT are characterized by m tensors of 4-th order (T()) m Jo=1> Conv-TT-LSTM replaces Eq.
7 in ConvLSTM by two equations:
H( f ( (2(--))))
vo E [m]]
IZo; F00; ( 000) = a (w
X100 (1)x(00 + CTT (T()) 1o (M )o-1)
(10)
(1) Since (C ) takes sequence of m tensors as inputs, the first step in Eq. 4 maps the inputs (H((-0)) ( J==1 to intermediate tensors (7((5-0) ja= with a function f. (2) These m tensors (H yo=1 are then fed into (C and compute the gates according Eq. (TO).
We propose two realizations of Eq. ( where the first realization uses fixed window of (H(t-0) N=
to compute each (7(tt.)) while the second one adopts a sliding window strategy. At each step, the Conv-TT-LSTM model computes H(th by replacing Eq. (9) by either Eq.
(Ta) or (TTb).
CoN--TT-LSTM-FW:
H(tt.0) ((()) * H k(() * H (((- H'
(1la)
Con--TT-LSTM-SW:
(((000) k(()) * A((1.0) k(()= *
H
(11b)
Mー= In the fixed window version, the previous steps (7 are concatenated into 3-rd order tensor ((t.,) IR X which is then mapped to tensor H(t,o) IR xxxxxx by 2D-convolution with a kernel XC(1) [B4xXxnCXXR And in the sliding window version, (H()) Ji=1 are concatenated into a 4-th order tensor R((t,0) E xx (with - 1), which is mapped to HZ(..))
R xgH x by 3D-convolution with kernel K(O)) B For later reference, we name the model with Eqs. Ta and (10) as CONV-TT-LSTM-FH and the one with Eqs. (Tb) and (10) as ConV-TT-LSTM-S Figure Ib bbaad]] Figure Ic visualize the difference between these two variants.
EXPERIMENTS
We first evaluate our approach extensively on the synthetic Moving-MNIST--2 dataset (Srivastava) jet al. 2015). In addition, we use KTH human action dataset (Laptev et al. 2004) to test the perfor- mance of our models in more realistic scenario.
Model Architecture All experiments use a stack of 12-layers of ConvLSTM or Conv-TT-LSTM with 32 channels for the first and last 3 layers, and 48 channels for the 6 layers in the middle. A con- volutional layer is applied on top of all LSTM layers to compute the predicted frames. Following added be-
Byeon et al. 2018), two skip connections performing concatenation over channels tween (3, 9) and (6, 12) layers. Illustration of the network architecture is included in the appendix. All parameters are initialized by Xavier's normalized initializer (Glorot & Bengio 2010) and initial states in ConvLSTM or Conv-TT-LSTM are initialized as zeros.
Evaluation Metrics We use two traditional metrics MSE (or PSNR) and SSIM WWag al. 2004),
and recently proposed deep-learning based metric LPIPS (Zhang et al. 2018). which measures the similarity between deep features. Since MSE (or PSNR) is based on pixel-wise difference, it favors vague and blurry predictions, which is not a proper measurement of perceptual similarity. While SSIM was originally proposed to address the problem. Zhang et al. (2018) shows that their proposed LPIPS metric aligns better to human perception.
Learning Strategy All models are trained with ADAM optimizer (Kngmma & Ba, 2014) with L1 + Cg loss. Learning rate decay and scheduled sampling (Bengio et al. 20155) are used to case training. Scheduled sampling is started once the model does not improve in 20 epochs (in term of validation loss), and the sampling ratio is decreased linearly from 1 until it reaches zero (by 2 10-4 each epoch for Moving-MNISTA and 5 x 10 for KTH). Learning rate decay is further activated if the loss does not drop in 20 epochs, and the rate is decreased exponentially by 0.98 every 5 epochs.
Selection We perform a wide range of hyper-parameter parameters search for Conv-TT- LSTM to identify the best model, Table summarizes our search values. The initial learning rate
5

---

Under review as a conference paper at ICLR 2020
Kernel size
Initial learning rate
Tensor order
Tensor rank
Time steps
[3,5}
[1e-4, 1e-3) (1,2,3,5]
14,8, 16)
[1,3,5)
Table Hyper-parameters search values for Conv-TT-LSTM experiments.
of 10-3 3 is found for the models of kernel size 3 and 10-4 for the models of kernel size 5. We found that Conv-TT-LSTM models suffer from exploding gradients when learning rate is high (e.g. 10-3 in our experiments), therefore we also explore various gradient clipping values and select 1 for all Conv-TT-LSTM models. All hyper-parameters are selected using the best validation performance.
5.1 MOVING-MNIST-2 DATASET
The Moving-MNIST-2 dataset is generated by moving two digits of size 28 x 28 in MNIST dataset within a 64 x 64 black canvas. These digits are placed at a random initial location, and move with constant velocity in the canvas and bounce when they reach the boundary. Following Wang et al. (2188)), we generate 10,000 videos for training, 3,000 for validation, and 5,000 for test with default parameters in the generator All our models trained to predict 10 frames given 10 input frames.
(10 10)
(10 30)
Method
params.
MSE
SSIM
LPIPS
MSE
SSIM
LPIPS
ConvLSTM (Xingjian al. 2015
25.22
0.713
38.13
0.595
7.58M
CDNA (Finn et al. 2016)
23.78
0.728
34.74
0.609
VPN KKlchbrrenner el al.
2007
15.65
0.870
31.64
0.620
E3D-LSTM (Wang ct al.
2018b
10.08
0.910
1=5M
PredRNN++ (original)
11.35
0.898
22.24
0.814
15.05M
PredRNN++ (retrained)
10.29
0,913
59.51
20.53
0.834
139.9
ConvLSTTM (baseline)
18.17
0.882
67.13
33.08
0.806
140.1
3.97M
Sony-TT-LSTM-FW (ours)
14.29
0.906
48.29
28.88
0.831
104.1
2.65M
COnV-TT-LSTM-SW (ours)
12.96
0.915
40.54
25.81
0.840
90.38
2.69M
Table 2: Comparison of 10 and 30 frames prediction on Moving- test set, where lower MSE values (in 10 10-3) higher SSIM / lower LPIPS values (in 10 1) indicate better results. All our models use kernel size 5: COn--IT-LSTM-FW has hyperparameters as (order 1, steps 3, ranks 8), and SON--TT-LSTM-SW has hyperparameters (order 3, steps 3, ranks 8).
Crent Swaltar
035
3.30
ProttiNiie
815
2.45
002
3.0
3.90
835
875
Figure/ 2: Frame-wise comparison MSE, SSIM and PIPS on Moving- For MSE and LPIPS, lower curves denote higher quality; while for SSIM, higher curves imply better quality.
Multi-Steps Prediction Table 2 reports the average statistics for 10 and 30 frames prediction, and Figure shows comparison of per-frame statistics for PredRNN+ model, ConvLSTM baseline and our proposed Conv-TT-LSTM models. (1) Our Conv-TT-LSTM models consistently outperform the
ww/nsstteeeeeeeeeeeeeeeeeeeeeeee httpeeerre//ttt/////// wtwpto/wwnnnoootoooooooooooooo ing_mnnitat -t...t-ppp 2 The results are cited from the original paper, where the miscaleulation of MSE corrected in the table.
tttps: //gi thb.. com/ htss /ttp//////avv///// dec-predi
'The results are reproduced from https: / thub, com Yunboo4 / httpp//eeddnnnnn pp with the same
datasets in this paper. The original implementation crops each frame into patches as the input to the model. We find such pre-pracessing is unnecessary and the performance is better than the original paper.

---

Under review as a conference paper at ICLR 2020
12-layer ConvLSTM baseline for both 10 and 30 frames prediction with fewer parameters; (2) The Cony-TT-LSTMA outperform previous approaches in terms of SSIM and LPIPS (especially on 30 frames prediction), with less than one fifth of the model parameters.
input
ground truth (top) / predictions
17
20
23
29
32
35
38
2
2
9
3
3
2
322
3
-2
3
2
2
3
5
_
3
3
PredRNN++
9
3
-
3
32
3
2
3
.2
2
?
-
3
.2
5
ConvLSTM
9
5
9
5
2
&
_
_2
2
2
2
3
3
5
COn-TTT-SSTM-FW
9
9
3
9
COnv-TT-LSTM-SW
5 3
9
32
3
9
3
9
2
9 3
9
?
Figure 3: 30 frames prediction Moving-MNISST given 10 input frames. Every 3 frames are shown.
We reproduce the PredRNN++ model (Wang et al. 2018 8a) from their source code?, and we find
that (1) The PredRNN++ model tends to output vague and blurry results in long-term prediction (especially after 20 steps). (2) and our Conv-TT-LSTMS are able to produce sharp and realistic
digits over all steps. An example of comparison for different models is shown in Figure 3 The visualization is consistent with the results in Table and Figure
(10 -> 30)
Method
parameters
MSE x 10-
SSIM
LPIPS
Baseline ConvLSTM (4-layers model)
37.19
0.791
184.2
11.48M
COn--TTALSTM-FFW (4-layers model)
31.46
0.819
112.5
5.65M
Baseline ConvLSTM (C, loss only)
33.96
0.805
184.4
3.97M
COn--TT-LSTTMFFW (L, loss only)
30.27
0.827
118.2
2.65M
Baseline ConvLSTM (teacher forcing)
36.95
0,802
135.1
3,97M
Con--TT-LSTM-FW (teacher forcing)
34.84
0.807
128.4
2.65M
Baseline ConvLSTM (our strategy) Con--TT-LSTM-FW (our strategy)
33.08 28.88
0,806
140.1
3.97M
0.831
104.1
2.65M
Table Evaluation of ConvLSTM and our Conv-TT-LSTM under the ablated experimental settings.
Ablation Study To understand whether our proposed Cony-TT-LSTM universally improves upon ConvLSTM (i.e. not tied to specific architecture, loss function and learning schedule), we perform three ablation studies: (1) Reduce the number of layers from 12 layers 4 layers (same as Xingjian ct al. (2015) and Wang et al. (2018a)); (2) Change the loss function from L1 + La to L1 only; (3) Disable the scheduled sampling and use teacher forcing during training process. We evaluate the ConvLSTM baseline and our proposed Cony-TT-LSTMN in these three settings, and summarize their comparisons in Table 3 The results show that our proposed Conv-TT-LSTM outperforms ConvLSTM consistently for all settings, i.e. the Conv-TT-LSTM,, model improves upon ConvLSTM in a board range of setups, which is not limited to the certain setting used in our paper. These ablation studies further show that our setup is optimal for predictive learning in MoVIng
5.2 KTH ACTION DATASET
KTH action dataset (Laptev et al. 2004) contains videos of 25 individuals performing 6 types of actions simple background. Our experimental setup follows Wang et al. (2018)), which uses

---

Under review as a conference paper at ICLR 2020
persons 1-16 for training and 17-25 for testing, and each frame is resized to 128 128 pixels. All our models are trained to predict 10 frames given 10 imput frames. During training, we randomly select 20 contiguous frames from the training videos sample and group every 10,000 samples into one epoch to apply the learning strategy as explained at the beginning of this section.
input
ground truth (top) / predictions
16
18
20
22
24
26
28
PredRNN+
A
ConvLSTM
CON-TTT-LSTM-FW
COnV-TT-LSTM-SW
Figure 4: 20 frames prediction KTH given 10 input frames. Every 2 frames shown.
(10 -> 20)
(10 -> 40)
Method
Parameters
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
ConvLSTM (iingiaan et al.
2015
23.58
0.712
22.85
0.639
7.58M
MCNET Villegas
2017)
25.95
0.804
E3D-LSTM Wang et al.
2018b
29.31
0.879
27.24
0.810
15M
ProdRNN++ (original)*
28.46
0.865
25.21
0.741
15.05M
PredRNN++(rctranned))
28.62
0.888
228.9
26.94
0.865
279.0
ConvI_STM-12 (haseline)
28.21
0.903
137.1
26.01
0.876
201.3
3.97M
Con--TT-LSTM-FW (ours)
28.46
0.907
134.8
26.42
0.882
196.0
2.65M
Conv-TT-LSTM-SSW (ours)
28.36
0.907
133.4
26.11
0.882
191.2
2.69M
Table Evaluation of multi-steps prediction on KTH dataset, where higher PSNR SSIM values indicate better predictive results. For Con--TT-LSTM-FW. the reported model has hyperparameters (order 1, steps ranks 8); and CONV-TT-LSTM-SW use hyperparameters (order 3, steps 3, ranks 8).
Results Table we report the evaluation on both 20 and 40 frames prediction. (1) Our models consistently better than the ConvLSTM baseline for both 20 and 40 frames prediction. (2) While our proposed Conv-TT-LSTMs achieve lower SSIM value compared to the state-of-the-as models in 20 frames prediction, they outperform all previous models in LPIPS for both 20 and 40 frames prediction. An example of the predictions by the baseline and Conv-TT--STTW is shown in iigFiee 3
6 CONCLUSION
In this paper, we proposed convolutional tensor-train decomposition to factorize a large convolu- tional kerne into set of smaller core tensors. We applied this technique to efficiently construct convolutional tensor-train LSTM (Conv-TT-LSTM), high-order spatio-temporal recurrent model whose parameters are represented in tensor-train format. We empirically demonstrated that our proposed Conv-TT-- TT-LSTM outperforms standard ConvLSTM and produce better/compprable results compared to other state-of- -fate-of-the-art models with fewer parameters. Utilizing the proposed model for high-resolution videos is still challenging due to gradient vanishing explosion. Future direction will include investigating other training strategies or a model design to ease the training process.
Wang 2018b) mentions that the number of parameters is similar PredRNN++ Wang et al.
2018a
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971, 2016.
Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. The Journal of Machine Learning Research, 15(1):2773-2832, 2014.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint /rXiv: 1252, 2017.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Sys- tems, pp. 1171-1179, 2015.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 753-769, 2018.
Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, Danilo P Mandic, al. Tensor networks for dimensionality reduction and large-scale optimization: Part I low-rank tensor decompositions. Foundations and Trends(K in Machine Learning, 9(4-5):249 4299994299900990 2016.
Emily Denton and Robert Fergus. Stochastic video generation with a learned prior. In 35th Inter- national Conference on Machine Learning, ICML 2018, pp. 1906-1919. International Machine Learning Society (IMLS), 2018.
Emily L. Denton et al. Unsupervised learning of disentangled representations from video. In Ad- vances in neural information processing systems, pp. 4423, 2017.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2786-2793. IEEE, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances neural information processing systems, pp. 64-72, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning de- compose and disentangle representations for video prediction. In Advances Neural Information Processing Systems, pp. 517-526, 2018.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proceedings the 34th Interna- tional Conference on Machine Moaming-Vollume 70, pp. 1771-1779. JMLR. org, 2017
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression deep convolutional neural networks for fast and low power mobile applications. arXiv preprint ww..iv:1111006.... 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1412. 2014.
Arinbjöm Kolbeinsson, Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, Ioanna Tzoulaki, and Paul Matthews. Stochastically regulaegelarized tensor regression networks. arXiv preprint 1rXiv:1902.10758 2019.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.

---

Under review as a conference paper at ICLR 2020
Jean Kossaifi, Zachary Lipton, Aran Khanna, Tommaso Furlanello, and Anima Anandkumar. Tensor regression networks. arXiv, 2017.
Jean Kossaifi, Adrian Bulat, Yannis Panagakis, and Maja Pantic. Efficient n-dimensional convolu- tions via higher-order factorization. arXiv preprint (906v009900661 2019.
Ivan Laptev, Barbara Caputo, al. Recognizing human actions: local svm approach. In mull, pp- 32-36. IEEE, 2004.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned ep-decomposition. arXiv preprint arXiv:1412.6553, 2014.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.00 015233, 2018.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning, arXiv preprint arXiv: 1605.08/04 2016.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou. A tensorized transformer for language modeling. arXiv preprint arXiv: (e0e::906009777 2019.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprini arXiv: 1.05440, 2015.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in neural information processing systems, pp. 442-450, 2015.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295- 2317, 2011.
Rohollah Soltani and Hui Jiang. Higher order recurrent neural networks. arXiv preprint arXiv:1605.00064, 2016.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference machine learning, pp. 843-852, 2015.
Jiahao Su, Jingling Li, Bobby Bhattacharjee, and Furong Huang. Tensorized spectrum preserving compression for neural networks. arxiv preprint arXiv: 10352, 2018.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Compressing recurrent neural network with tensor train. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 4451- 4458. IEEE, 2017.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.08033. 2017.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip. Predrnn: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems. pp. 879-888888 779-888,42 2017.
Yunbo Wang, Zhifeng Gao, Mingsheng Long. Jianmin Wang, and Philip S Yu. Predmn++: Towards resolution of the deep-in-time dilemma in spatiotemporal predictive learning. arXiv preprint 180.0000 2018a.
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. preprint, 2018b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, al. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wii-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, pp. 802-810, 2015.
10

---

Under review as a conference paper at ICLR 2020
Yinchong Yang, Denis Krompass, and Volker Tresp. Tensor-train recurrent neural networks for video classification. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3891-3900. JMLR. org, 2017.
Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor- train mas. arXiv preprini urXiv:171 00111711100075. 2017.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Appendix: Convolutional Tensor-Train LSTM for Long- -Temm Video Prediction
A PROOF OF THE SEQUENTIAL ALGORITHMS IN SECTION 3
In this section, we prove the sequential algorithms in Eq. (3) for tensor-train decomposition (1) and Eq. (6) for convolutional tensor-train decomposition (4) both by induction.
Proof of Eq. (3) For simplicity, we denote the standard tensor-train decomposition in Eq. (1) as
T (TT((T()) )m
)m then Eq. (2) can be rewritten Eq. 12 since Ro and 0 (0) 1.
Ra
4
Am
v =
Σ
TTD 1Z
(1()) (1)
& (m)
(12)
dm
is. Am
To=
1=
8m
Ra
h
Im
Rm-1
Σ
Σ
fm
(m)
Uro Thi
(0)
(m)
(13)
11
1=1
R1
1
Ra
Σ
(12)
12.11.2
...
Am
1m
12=1
1m-1=1
(14)
Σ
(
(1)
(2)
(m)
U7a
Uij
dig
Aro=l1=1
R1
12
Im
TTD (1TW)m)
(11) 11
(2(2)
2 (m)
(15)
11
12
8m
where Ro = 1, 1
(0)
and the sequential algorithm in Eq. (3) is performed at Eq. 14
Proof of Eq. (6) For simplicity, we denote the convolutional tensor-train decomposition Eq. (4)
as T ())) then Eq. (5) can be rewritten as (16) since (mm) an all zeros tensor.
CTTD (T(0))
(0 SU
m=
V ViLo
(16)
Rmn CTTD
(1T(0))1²
(mm) fi.f
CTTD (T(()))
(1())
(17)
Ri
CTTD (17(0)
((m) Vitim
(uerm +
11

---

Under review as a conference paper at ICLR 2020
Note that the second term in Eq. 177) can now be simplified
CTTD
((T()
(ui.. + V(in)
(18)
T'm
Ran
Ri
Rom
Σ
Σ
T())
((m)
*
+ Vi
(19)
I'm
r==
F'm 1=
Rm-1
R
Rm-1
((1)
(1m-1) 1)
No 2m 22tm
Tm--1
n==
Fm-1= ==
(20)
Rm
((m) mm)
* (uim
V((m
+
7m-1
Tm==
Pyn
CTTD ((T()]
((m=l) 5i.mm
(21)
F'm
where the sequential algorithm in Eq. (5) is performed to achieve Eq. 21 from Eq. 20 Eq. (21) into Eq. (17), we reduce Eq. back to the form as Eq. [6
Plugging
1
R4
Σ
CTTD
(T())
1=1
(22)
Rm
CTTD ((
1:1m-1170
T'
which completes the induction.
B SUPPLEMENTARY MATERIAL OF THE EXPERIMENTS
MOnv-TT-LSTM with 32 channels for the first and last layers, and 48 channels for the 6 layers in the middle. A convolutional layer is applied on top all LSTM layers to compute the predicted frames, followed by an optional sigmoid function (In the experiments, we add sigmoid for KTH dataset but not for Moving-MNIST-22, Additionally, two skip connections performing concatenation over channels are added between (3, 9) and (6, 12) layers as shown in Figure
All experiments use stack of laylrs of ConvLSTM
units
48 units
48 wnits
32 units
Conv-(TT) LSTM X3
Conv-(TT) LSTM
Conv-(TT) LSTM 3
Conv-(TT) LSTM X3
Input
Output
Block
Block 2
Block 3
Block 4
Conv
Figure 5: Illustaation of the network architecture for the 12-layers model used in the experiments. In this section, we provide additional results on the visual comparison between our proposed Conv- TT-LSTM and baseline STM.
12

---

Under review as a conference paper at ICLR 2020
input
ground truth (top) / predictions
17
20
23
35
2
7
2
7
27
2
2
2
7
2
7
27
2
7
2
7
2
PredRNN+
27
2
7
2
7
2
27
2
2
7
7
2
ConvLSTM
27
2
7
2
7
2
21
2
7
]
2
1
!
COnY-TT-LSTM-FW
27
2
7
2
7
2
27
2
7
2
7
2
7
2
)
COn--TT-LTT--SW
27
2
7
2
2
7
27
2
7 2
2
1
2
Figure 6: 30 frames prediction on Moving-MNIST given 10 input frames. Every 3 frames are shown.
input
ground truth (top) / predictions
8 2
5
11
14
17
20
23
26
29
32
35
38
2
2 5
2
2
2
2
5-
52
2
3
2
5
5
5
5
2
2
2
2
22
52
2
3
2
2
PredRNN+
5
5
5
5
2
2
2
2
2
7
-
?
ConvLSTM
5"
5"
COn--TT-LTTM-FW
-/
3
3
5.
$
2
9
9
c
2
2
2
2
52
52
52
2
3
2
2
CON--TT-LSTM-SW
5
5
5
5
5
Figure 7: frames prediction Moving-MNIIS given 10 input frames. Every 3 frames are shown.
input
ground truth (top) / predictions
17
20
23
26
32
35
38
4
g
à
3
3
-
8
4
4
4
3
3
4
4
3
3
PredRNN+
g
3
3
4
4
4
4
9
4
4
4
3
3
ConvLSTM
g
4
2
?
4
6
6
6
COn--TT-LSTM-FW
g
4
4
4
4
4
4
4
4
CON--TT-STT--SW
g
4
3
3
3
4
3
4
4
3
3
Figure 8: 30 frames prediction on Moving-MNIST given 10 input frames. Every 3 frames are shown.
13

---

Under review as a conference paper at ICLR 2020
input
ground truth (top) / predictions
17
20
23
35
8
8
&
8
8
8
88
88
8
3
8
2
2
8
2
8
8
8
PredRNN+
88
8
8
3
J
8
J
8
8 7
8
J
ConvLSTM
88
8
4
7
7
y,
COnY-TT-LSTM-FW
83
8
4
R
-
4
1
COn--TT-LTT--SW
8
8
2
8
85
8
3
Figure 9: 30 frames prediction on Moving-MNIST given 10 input frames. Every 3 frames are shown.
input
ground truth (top) / predictions
10
12
14
16
18
20
22
24
26
28
PredRNN+
ConvLSTM
COn--TT-LTTM-FW
COn--TT-LTTM-SW
Figure 10: 20 frames prediction on KTH given 10 input frames. Every 2 frames are shown,
input ground truth (top) / predictions
16
18
20
22
24
26
28
PredRNN+
ConvLSTM
Conv-TT-LSTM-FW
COn-ITT-LSTM-SW
Figure 11: 20 frames prediction on KTH given 10 input frames. Every 2 frames are shown.
14


Reviews:
[{"review_id": "rkgc4-Cy5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a convolutional tensor-train (CTT) format based high-order and convolutional LSTM approach for long-term video prediction. This paper is well-motivated. Video data usually have high dimensional input, and the proposed method aims to explicitly take into account more than one hidden representation of previous frames - both lead to a huge number of parameters. Therefore, some sort of parameter reduction is needed. This paper considers two different types of operations - convolution and tensor-train (TT) decomposition - in an interleaved way. The basic model considered in this paper is a high-order variant of convolutional LSTM (convLSTM).\n \nThere exist several works using tensor decomposition methods including TT to compress a fully connected layer or a convolutional layer in neural nets, to break the memory bottleneck and accelerate computation. This paper takes a different direction - it further embeds convolution into the TT decomposition and thus defines a new type of tensor decomposition, termed convolutional tensor train (CTT) decomposition. CTT is used to represent the huge weight matrices arisen in the high-order convLSTM. To my best knowledge, this combination of convolution and TT decomposition is new.\n \nThe paper is well-written as the literature review is well done. Experimental results demonstrate improved performance over the convolutional LSTM baseline, a fewer number of parameters, and the qualitative results show sharp and clean digits. This improvement could be attributed to multiple causes: the high-order, the tensor decomposition-based compression, or the CTT. The authors also provide an ablation study, but it mainly concerns comparisons with ConvLSTM.   \n \nDespite the promising results, this paper is not ready for ICLR yet. Below is a list of suggested points needed to address:\n(1) Yang et al 2017 claim that TT-RNN without convolution can also capture spatial and temporal dependence patterns in video modeling. This is an important baseline but missing in the current version of the paper. \n(2) The justification of high-order modeling in long-term prediction. The first-order model also implicitly aggregates multiple past steps. It would be good to add more experimental evidence to support the necessity of the high-order.\n(3) There exists some unjustified complexity for the CTT approach. How does it compare to TT for high-order ConvLSTM?\n \nPerhaps, a more complete ablation study should include:\n(1) LSTM with TT but without high-order and convolution\n(2) LSTM with high-order and TT but without convolution\n(3) ConvLSTM with TT\n(4) ConvLSTM with CTT\n(5) ConvLSTM with high-order and TT\n(6) ConvLSTM with high-order and CTT\n \nQuestion:\n• How is the backpropagation done for the CTT core tensors? \n• What is the error propagation issue of first-order methods and how does the high-order one not prone to it? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyegM4fo9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper build a higher-order spatio-temporal model by means of combining Convolutional Tensor-Train Decomposition(CTTD) and ConvLSTM, and utilize the combination method to solve long-term video prediction problems. The CTTD factorizes a large convolutional kernel into a chain of smaller tensors, so as to relieve the difficult convergence and overfitting problems caused by too much model params. \nExperiments on Moving-MNIST and KTH datasets show that the proposed method achieved better results than standard ConvLSTM, and in some way comparable with SOTA model. Ablation Studies are also provided.\n\nAlthough it seems novel combing CTTD with ConvLSTM, the idea of CTTD and the combination mainly comes from [Yu et al.,2017] and [Yang et al.,2017]，this paper use the method in a new problem of video prediction,  I think the theoretical innovation is not enough for ICLR.\nAlthough the experimental results were better than ConvLSTM(2015), but not as good as PredRNN++(2018), especially in terms of the MSE metrics. Since the prediction accuracy has not yet achieved, I don't think the reduction of model params is a matter of primary importance.  What’s more, Moving-MNIST and KTH are relatively simple datasets, video prediction on a more complicated datasets such as UCF101 will be more convincing.\nConclusion:\nThis paper is in some way novel, but not enough for ICLR, and the experiment results seems not enough convincing, so I will give a weak reject.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkxAlRIRYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a method that saves memory and computation in the task of video prediction by low-rank tensor representations via tensor decomposition. The method is able to outperform standard convolutional lstm and other methods by using less parameters when testing it in the Moving MNIST and KTH datasets. The authors also present a proof to validate their method.\n\n\nPros:\n+ Interesting method for decomposing tensors operations in convolutional architectures\n+ Outperforms immediate baseline (Convolutional LSTM)\n\nWeaknesses / comments:\n- Weak experimental section\nThe authors mainly compare against Convolutional LSTM. The performance increase is there but the difference in parameters is not that significant in comparison to the performance. Needing fewer parameters is one of the claims in this paper and I am not fully convinced of the trade-off between the complexity of the model and the gain in parameter reduction / performance. In addition, the show videos do not look that much improved. The paper is also missing baselines from Villegas et al., 2017 and Denton et al., 2017 which both have available models for the KTH dataset.\n\n- No videos provided\nThe paper does not provide any videos which is a must for video prediction papers. Judging the video quality from images in the paper is not easy, and also the used metrics have been shown to not be very objective in terms of video prediction quality or image generation in general.\n\n\nConclusion:\nThe proposed decomposition method is interesting, but the experimental section fails to convince me as to whether the methods performance validates the complicated formulations. My current score is between weak reject and reject so I will give a weak reject.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Conv-TT-LSTM for long-term video prediction. The proposed method saves memory and computation by low-rank tensor representations via tensor decomposition and is evaluated in Moving MNIST and KTH datasets.

All reviews argue that the novelty of the paper does not meet the standard of ICLR. In the rebuttal, the authors polish the experiment design, which fails to change any reviewer’s decision.

Overall, the paper is not good enough for ICLR.
Decision: Reject