Convolutional Tensor-Train LSTM for Long-Term Video Prediction

1 INTRODUCTION . Understanding dynamics of videos and performing long-term predictions of the future is a highly challenging problem . It entails learning complex representation of real-world environment without external supervision . This arises in a wide range of applications , including autonomous driving , robot control ( Finn & Levine , 2017 ) , or other visual perception tasks like action recognition or object tracking ( Alahi et al. , 2016 ) . However , long-term video prediction remains an open problem due to high complexity of the video contents . Therefore , prior works mostly focus on next or first few frames prediction ( Lotter et al. , 2016 ; Finn et al. , 2016 ; Byeon et al. , 2018 ) . Many recent video models use Convolutional LSTM ( ConvLSTM ) as a basic block ( Xingjian et al. , 2015 ) , where spatio-temporal information is encoded as a tensor explicitly in each cell . In ConvLSTM networks , each cell is a first-order recurrent model , where the hidden state is updated based on its immediate previous step . Therefore , they can not easily capture higher-order temporal correlations needed for long-term prediction . Moreover , they are highly prone to error propagation . Various approaches have been proposed to augment ConvLSTM , either by modifying networks to explicitly modeling motion ( Finn et al. , 2016 ) , or by integrating spatio-temporal interaction in ConvLSTM cells ( Wang et al. , 2017 ; 2018a ) . These approaches are often incapable of capturing longterm dependencies and produce blurry prediction . Another direction to augment ConvLSTM is to incorporate a higher-order RNNs ( Soltani & Jiang , 2016 ) inside each LSTM cell , where its hidden state is updated using multiple past steps . However , a higher-order model for high-dimensional data ( e.g . video ) requires a huge number of model parameters , and the computation grows exponentially with the order of the RNNs . A principled approach to address the curse of dimensionality is tensor decomposition , where a higher-order tensor is compressed into smaller core tensors ( Anandkumar et al. , 2014 ) . Tensor representations are powerful since they retain rich expressivity even with a small number of parameters . In this work , we propose a novel convolutional tensor decomposition , which allows for compact higher-order ConvLSTM . Contributions . We propose Convolutional Tensor-Train LSTM ( Conv-TT-LSTM ) , a modification of ConvLSTM , to build a higher-order spatio-temporal model . ( 1 ) We introduce Convolutional Tensor-Train Decomposition ( CTTD ) that factorizes a large convolutional kernel into a chain of smaller tensors . ( 2 ) We integrate CTTD into ConvLSTM and propose Conv-TT-LSTM , which learns long-term dynamics in video sequence with a small number of model parameters . ( 3 ) We propose two versions of Conv-TT-LSTM : Fixed Window ( FW ) and Sliding Window ( SW ) ( See Figures 1b and 1c ) , and we found that the SW version performs better than the FW one . ( 4 ) We found that training higher-order tensor models is not straightforward due to gradient instability . We present several approaches to overcome this such as good learning schedules and gradient clipping . ( 5 ) In the experiments , we show our proposed Conv-TT-LSTM consistently produces sharp prediction over a long period of time for both Moving-MNIST-2 and KTH action datasets . Conv-TT-LSTM outperforms the state-of-the-art PredRNN++ ( Wang et al. , 2018a ) in LPIPS ( Zhang et al. , 2018 ) by 0.050 on the Moving-MNIST-2 and 0.071 on the KTH action dataset , with 5.6 times fewer parameters . Thus , we obtain best of both worlds : better long-term prediction and model compression . 2 RELATED WORK . Tensor Decomposition In machine learning , tensor decompositions , including CP decomposition ( Anandkumar et al. , 2014 ) , Tucker decomposition ( Kolda & Bader , 2009 ) , and tensor-train decomposition ( Oseledets , 2011 ) , are widely used for dimensionality reduction ( Cichocki et al. , 2016 ) and learning probabilistic models ( Anandkumar et al. , 2014 ) . In deep learning , prior works focused on their application in model compression , where the parameters tensors are factorized into smaller tensors . This technique has been used in compressing convolutional networks ( Lebedev et al. , 2014 ; Kim et al. , 2015 ; Novikov et al. , 2015 ; Su et al. , 2018 ; Kossaifi et al. , 2017 ; Kolbeinsson et al. , 2019 ; Kossaifi et al. , 2019 ) , recurrent networks ( Tjandra et al. , 2017 ; Yang et al. , 2017 ) and transformers ( Ma et al. , 2019 ) . Specifically , Yang et al . ( 2017 ) demonstrates that the accuracy of video classification increases if the parameters in recurrent networks are compressed by tensor-train decomposition ( Oseledets , 2011 ) . Yu et al . ( 2017 ) used tensor-train decomposition to constrain the complexity of higher-order LSTM , where each next step is computed based on the outer product of previous steps . While this work only considers vector input at each step , we extend their approach to higher-order ConvLSTM , where each step also encodes spatial information . Video Prediction Prior works on video prediction have focused on several directions : predicting short-term video ( Lotter et al. , 2016 ; Byeon et al. , 2018 ) , decomposing motion and contents ( Finn et al. , 2016 ; Villegas et al. , 2017 ; Denton et al. , 2017 ; Hsieh et al. , 2018 ) , improving the objective function Mathieu et al . ( 2015 ) , and handling diversity of the future ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ) . Many of these works use Convolutional LSTM ( ConvLSTM ) ( Xingjian et al. , 2015 ) as a base module , which deploys 2D convolutional operations in LSTM to efficiently exploit spatio-temporal information . Finn et al . ( 2016 ) used ConvLSTM to model pixel motion . Some works modified the standard ConvLSTM to better capture spatio-temporal correlations ( Wang et al. , 2017 ; 2018a ) . Wang et al . ( 2018b ) integrated 3D convolutions into ConvLSTM . In addition , current cell states are combined with its historical records using self-attention to efficiently recall the history information . Byeon et al . ( 2018 ) applied ConvLSTM in all possible directions to capture full contexts in video and also demonstrated strong performance using a deep ConvLSTM network as a baseline . This baseline is adapted to obtain the base architecture in the present paper . 3 TENSOR-TRAIN DECOMPOSITION AND SEQUENCE MODELING . The goal of tensor decomposition is to represent a higher-order tensor as a set of smaller and lowerorder core tensors , with fewer parameters while preserve essential information . In Yu et al . ( 2017 ) , tensor-train decomposition ( Oseledets , 2011 ) is used to reduce both parameters and computations in higher-order recurrent models , which we review in the first part of this section . However , the approach in Yu et al . ( 2017 ) only considers recurrent models with vector inputs and can not cope with image inputs directly . In the second part , we extend the standard tensor-train decomposition to convolutional tensor-train decomposition ( CTTD ) . With CTTD , a large convolutional kernel is factorized into a chain of smaller kernels . We show that such decomposition can reduce both parameters and operations of higher-order spatio-temporal recurrent models . Standard Tensor-train decomposition Given an m-order tensor T ∈ RI1×···×Im , where Il is the dimension of its l-th order , a standard tensor-train decomposition ( TTD ) ( Oseledets , 2011 ) factorizes the tensor T into a set of m core tensors { T ( l ) } ml=1 with T ( l ) ∈ RIl×Rl×Rl+1 such that Ti1 , ··· , im , R1∑ r1=1 · · · Rm−1∑ rm−1=1 T ( 1 ) i1,1 , r1 T ( 2 ) i2 , r1 , r2 · · · T ( m ) im , rm−1,1 ( 1 ) where tensor-train ranks { Rl } ml=0 ( with R0 = Rm = 1 ) control the number of parameters in the tensor-train format Eq. ( 1 ) . With TTD , the original tensor T of size ( ∏m l=1 Il ) is compressed to ( ∑m l=1 IlRl−1Rl ) entries , which grows linearly with the order m ( assuming Rl ’ s are constant ) . Therefore , TTD is commonly used to approximate higher-order tensors with fewer parameters . The sequential structure in tensor-train decomposition makes it particularly suitable for sequence modeling ( Yu et al. , 2017 ) . Consider a higher-order recurrent model that predicts a scalar output v ∈ R based on the outer product of a sequence of input vectors { u ( l ) ∈ RIl } ml=1 according to : v = 〈 T , ( u ( 1 ) ⊗ · · · ⊗ u ( m ) ) 〉 = I1∑ i1=1 · · · Im∑ im=1 Ti1 , ··· , im u ( 1 ) i1 · · · u ( m ) im ( 2 ) This model is intractable in practice since the number of parameters in T ∈ RI1×···Im ( and therefore computational complexity of Eq . ( 2 ) ) grows exponentially with the order m. Now suppose T takes a tensor-train format as in Eq . ( 1 ) , we prove in Appendix A that ( 2 ) can be efficiently computed as v ( l ) rl = Il∑ il=1 Rl∑ rl−1=1 T ( l ) il , rl−1 , rl v ( l−1 ) rl−1 u ( l ) il , ∀l ∈ [ m ] ( 3 ) where the vectors { v ( l ) ∈ RRl } ml=1 are the intermediate steps , with v ( 0 ) ∈ R initialized as v ( 0 ) = 1 , and final output v = v ( m ) . Notice that the higher-order tensor T is never reconstructed in the sequential process in Eq . ( 3 ) , therefore both space and computational complexities grow linearly ( not exponentially compared to Eq . ( 2 ) ) with the order m assuming all tensor-train ranks are constants . Convolutional Tensor-Train Decomposition A convolutional layer in neural network is typically parameterized by a 4-th order tensor T ∈ RK×K×Rm×R0 , where K is the kernel size , Rm and R0 are the number of input and output channels respectively . Suppose the kernel size K takes the form K = m ( k − 1 ) + 1 ( e.g . K = 7 and m = 3 , k = 3 ) , a convolutional tensor-train decomposition ( CTTD ) factorizes T into a set of m core tensors { T ( l ) } ml=1 with T ( l ) ∈ Rk×k×Rl×Rl−1 such that T : , : ,rm , r0 , R1∑ r1=1 · · · Rm−1∑ rm−1=1 T ( 1 ) : , : ,r1 , r0 ∗ T ( 2 ) : , : ,r2 , r1 ∗ · · · ∗ T ( m ) : , : ,rm , rm−1 ( 4 ) where ∗ denotes convolution between 2D-filters , and { Rl } ml=1 are the convolutional tensor-train ranks that control the complexity of the convolutional tensor-train format in Eq . ( 4 ) . With CTTD , the number of parameters in the decomposed format reduces from K2R0Rm to ( ∑m l=1 k 2Rl−1Rl ) . Similar to standard TTD , its convolutional counterpart can also be used to compress higher-order spatio-temporal recurrent models with convolutional operations . Consider a model that predicts a 3-rd order feature V ∈ RH×W×R0 based on a sequence of 3-rd features { U ( l ) ∈ RH×W×Rl } ml=1 ( where H , W are height/width of the features and Rl is the number of channels in U ( l ) ) such that V : , : ,r0 = m∑ l=1 W ( l ) : , : ,rl , r0 ∗ U ( l ) : , : ,rl , withW ( l ) = CTTD ( { T ( l ) } ml=k ) , ∀l ∈ [ m ] ( 5 ) whereW ( l ) ∈ R [ l ( k−1 ) +1 ] × [ l ( k−1 ) +1 ] ×Rl×R0 is the corresponding weights tensor for U ( l ) . Suppose each W ( l ) takes a convolutional tensor-train format in Eq . ( 4 ) , we prove in Appendix A that the model in Eq . ( 5 ) can be computed sequentially similarly without reconstructing the originalW ( l ) ’ s : V ( l−1 ) : , : ,rl−1 = Rl∑ rl=1 T ( l ) : , : ,rl , rl−1 ∗ ( V ( l ) : , : ,rl + U ( l ) : , : ,rl ) , ∀l ∈ [ m ] ( 6 ) where { V ( l ) ∈ RH×W×Rl } ml=1 are intermediate results of the sequential process , where V ( m ) ∈ RH×W×Rm is initialized as all zeros and final prediction V = V ( 0 ) . The operations in Eq . ( 5 ) is illustrated in Figure 1a . In this paper , we denote the Eq . ( 5 ) simply as V = CTT ( { T ( l ) } ml=1 , { U ( l ) } ml=1 ) .

Summary: This paper proposed a convolutional tensor-train (CTT) format based high-order and convolutional LSTM approach for long-term video prediction. This paper is well-motivated. Video data usually have high dimensional input, and the proposed method aims to explicitly take into account more than one hidden representation of previous frames - both lead to a huge number of parameters. Therefore, some sort of parameter reduction is needed. This paper considers two different types of operations - convolution and tensor-train (TT) decomposition - in an interleaved way. The basic model considered in this paper is a high-order variant of convolutional LSTM (convLSTM).