Convolutional Tensor-Train LSTM for Long-Term Video Prediction

1 INTRODUCTION . Understanding dynamics of videos and performing long-term predictions of the future is a highly challenging problem . It entails learning complex representation of real-world environment without external supervision . This arises in a wide range of applications , including autonomous driving , robot control ( Finn & Levine , 2017 ) , or other visual perception tasks like action recognition or object tracking ( Alahi et al. , 2016 ) . However , long-term video prediction remains an open problem due to high complexity of the video contents . Therefore , prior works mostly focus on next or first few frames prediction ( Lotter et al. , 2016 ; Finn et al. , 2016 ; Byeon et al. , 2018 ) . Many recent video models use Convolutional LSTM ( ConvLSTM ) as a basic block ( Xingjian et al. , 2015 ) , where spatio-temporal information is encoded as a tensor explicitly in each cell . In ConvLSTM networks , each cell is a first-order recurrent model , where the hidden state is updated based on its immediate previous step . Therefore , they can not easily capture higher-order temporal correlations needed for long-term prediction . Moreover , they are highly prone to error propagation . Various approaches have been proposed to augment ConvLSTM , either by modifying networks to explicitly modeling motion ( Finn et al. , 2016 ) , or by integrating spatio-temporal interaction in ConvLSTM cells ( Wang et al. , 2017 ; 2018a ) . These approaches are often incapable of capturing longterm dependencies and produce blurry prediction . Another direction to augment ConvLSTM is to incorporate a higher-order RNNs ( Soltani & Jiang , 2016 ) inside each LSTM cell , where its hidden state is updated using multiple past steps . However , a higher-order model for high-dimensional data ( e.g . video ) requires a huge number of model parameters , and the computation grows exponentially with the order of the RNNs . A principled approach to address the curse of dimensionality is tensor decomposition , where a higher-order tensor is compressed into smaller core tensors ( Anandkumar et al. , 2014 ) . Tensor representations are powerful since they retain rich expressivity even with a small number of parameters . In this work , we propose a novel convolutional tensor decomposition , which allows for compact higher-order ConvLSTM . Contributions . We propose Convolutional Tensor-Train LSTM ( Conv-TT-LSTM ) , a modification of ConvLSTM , to build a higher-order spatio-temporal model . ( 1 ) We introduce Convolutional Tensor-Train Decomposition ( CTTD ) that factorizes a large convolutional kernel into a chain of smaller tensors . ( 2 ) We integrate CTTD into ConvLSTM and propose Conv-TT-LSTM , which learns long-term dynamics in video sequence with a small number of model parameters . ( 3 ) We propose two versions of Conv-TT-LSTM : Fixed Window ( FW ) and Sliding Window ( SW ) ( See Figures 1b and 1c ) , and we found that the SW version performs better than the FW one . ( 4 ) We found that training higher-order tensor models is not straightforward due to gradient instability . We present several approaches to overcome this such as good learning schedules and gradient clipping . ( 5 ) In the experiments , we show our proposed Conv-TT-LSTM consistently produces sharp prediction over a long period of time for both Moving-MNIST-2 and KTH action datasets . Conv-TT-LSTM outperforms the state-of-the-art PredRNN++ ( Wang et al. , 2018a ) in LPIPS ( Zhang et al. , 2018 ) by 0.050 on the Moving-MNIST-2 and 0.071 on the KTH action dataset , with 5.6 times fewer parameters . Thus , we obtain best of both worlds : better long-term prediction and model compression . 2 RELATED WORK . Tensor Decomposition In machine learning , tensor decompositions , including CP decomposition ( Anandkumar et al. , 2014 ) , Tucker decomposition ( Kolda & Bader , 2009 ) , and tensor-train decomposition ( Oseledets , 2011 ) , are widely used for dimensionality reduction ( Cichocki et al. , 2016 ) and learning probabilistic models ( Anandkumar et al. , 2014 ) . In deep learning , prior works focused on their application in model compression , where the parameters tensors are factorized into smaller tensors . This technique has been used in compressing convolutional networks ( Lebedev et al. , 2014 ; Kim et al. , 2015 ; Novikov et al. , 2015 ; Su et al. , 2018 ; Kossaifi et al. , 2017 ; Kolbeinsson et al. , 2019 ; Kossaifi et al. , 2019 ) , recurrent networks ( Tjandra et al. , 2017 ; Yang et al. , 2017 ) and transformers ( Ma et al. , 2019 ) . Specifically , Yang et al . ( 2017 ) demonstrates that the accuracy of video classification increases if the parameters in recurrent networks are compressed by tensor-train decomposition ( Oseledets , 2011 ) . Yu et al . ( 2017 ) used tensor-train decomposition to constrain the complexity of higher-order LSTM , where each next step is computed based on the outer product of previous steps . While this work only considers vector input at each step , we extend their approach to higher-order ConvLSTM , where each step also encodes spatial information . Video Prediction Prior works on video prediction have focused on several directions : predicting short-term video ( Lotter et al. , 2016 ; Byeon et al. , 2018 ) , decomposing motion and contents ( Finn et al. , 2016 ; Villegas et al. , 2017 ; Denton et al. , 2017 ; Hsieh et al. , 2018 ) , improving the objective function Mathieu et al . ( 2015 ) , and handling diversity of the future ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2017 ; Lee et al. , 2018 ) . Many of these works use Convolutional LSTM ( ConvLSTM ) ( Xingjian et al. , 2015 ) as a base module , which deploys 2D convolutional operations in LSTM to efficiently exploit spatio-temporal information . Finn et al . ( 2016 ) used ConvLSTM to model pixel motion . Some works modified the standard ConvLSTM to better capture spatio-temporal correlations ( Wang et al. , 2017 ; 2018a ) . Wang et al . ( 2018b ) integrated 3D convolutions into ConvLSTM . In addition , current cell states are combined with its historical records using self-attention to efficiently recall the history information . Byeon et al . ( 2018 ) applied ConvLSTM in all possible directions to capture full contexts in video and also demonstrated strong performance using a deep ConvLSTM network as a baseline . This baseline is adapted to obtain the base architecture in the present paper . 3 TENSOR-TRAIN DECOMPOSITION AND SEQUENCE MODELING . The goal of tensor decomposition is to represent a higher-order tensor as a set of smaller and lowerorder core tensors , with fewer parameters while preserve essential information . In Yu et al . ( 2017 ) , tensor-train decomposition ( Oseledets , 2011 ) is used to reduce both parameters and computations in higher-order recurrent models , which we review in the first part of this section . However , the approach in Yu et al . ( 2017 ) only considers recurrent models with vector inputs and can not cope with image inputs directly . In the second part , we extend the standard tensor-train decomposition to convolutional tensor-train decomposition ( CTTD ) . With CTTD , a large convolutional kernel is factorized into a chain of smaller kernels . We show that such decomposition can reduce both parameters and operations of higher-order spatio-temporal recurrent models . Standard Tensor-train decomposition Given an m-order tensor T ∈ RI1×···×Im , where Il is the dimension of its l-th order , a standard tensor-train decomposition ( TTD ) ( Oseledets , 2011 ) factorizes the tensor T into a set of m core tensors { T ( l ) } ml=1 with T ( l ) ∈ RIl×Rl×Rl+1 such that Ti1 , ··· , im , R1∑ r1=1 · · · Rm−1∑ rm−1=1 T ( 1 ) i1,1 , r1 T ( 2 ) i2 , r1 , r2 · · · T ( m ) im , rm−1,1 ( 1 ) where tensor-train ranks { Rl } ml=0 ( with R0 = Rm = 1 ) control the number of parameters in the tensor-train format Eq. ( 1 ) . With TTD , the original tensor T of size ( ∏m l=1 Il ) is compressed to ( ∑m l=1 IlRl−1Rl ) entries , which grows linearly with the order m ( assuming Rl ’ s are constant ) . Therefore , TTD is commonly used to approximate higher-order tensors with fewer parameters . The sequential structure in tensor-train decomposition makes it particularly suitable for sequence modeling ( Yu et al. , 2017 ) . Consider a higher-order recurrent model that predicts a scalar output v ∈ R based on the outer product of a sequence of input vectors { u ( l ) ∈ RIl } ml=1 according to : v = 〈 T , ( u ( 1 ) ⊗ · · · ⊗ u ( m ) ) 〉 = I1∑ i1=1 · · · Im∑ im=1 Ti1 , ··· , im u ( 1 ) i1 · · · u ( m ) im ( 2 ) This model is intractable in practice since the number of parameters in T ∈ RI1×···Im ( and therefore computational complexity of Eq . ( 2 ) ) grows exponentially with the order m. Now suppose T takes a tensor-train format as in Eq . ( 1 ) , we prove in Appendix A that ( 2 ) can be efficiently computed as v ( l ) rl = Il∑ il=1 Rl∑ rl−1=1 T ( l ) il , rl−1 , rl v ( l−1 ) rl−1 u ( l ) il , ∀l ∈ [ m ] ( 3 ) where the vectors { v ( l ) ∈ RRl } ml=1 are the intermediate steps , with v ( 0 ) ∈ R initialized as v ( 0 ) = 1 , and final output v = v ( m ) . Notice that the higher-order tensor T is never reconstructed in the sequential process in Eq . ( 3 ) , therefore both space and computational complexities grow linearly ( not exponentially compared to Eq . ( 2 ) ) with the order m assuming all tensor-train ranks are constants . Convolutional Tensor-Train Decomposition A convolutional layer in neural network is typically parameterized by a 4-th order tensor T ∈ RK×K×Rm×R0 , where K is the kernel size , Rm and R0 are the number of input and output channels respectively . Suppose the kernel size K takes the form K = m ( k − 1 ) + 1 ( e.g . K = 7 and m = 3 , k = 3 ) , a convolutional tensor-train decomposition ( CTTD ) factorizes T into a set of m core tensors { T ( l ) } ml=1 with T ( l ) ∈ Rk×k×Rl×Rl−1 such that T : , : ,rm , r0 , R1∑ r1=1 · · · Rm−1∑ rm−1=1 T ( 1 ) : , : ,r1 , r0 ∗ T ( 2 ) : , : ,r2 , r1 ∗ · · · ∗ T ( m ) : , : ,rm , rm−1 ( 4 ) where ∗ denotes convolution between 2D-filters , and { Rl } ml=1 are the convolutional tensor-train ranks that control the complexity of the convolutional tensor-train format in Eq . ( 4 ) . With CTTD , the number of parameters in the decomposed format reduces from K2R0Rm to ( ∑m l=1 k 2Rl−1Rl ) . Similar to standard TTD , its convolutional counterpart can also be used to compress higher-order spatio-temporal recurrent models with convolutional operations . Consider a model that predicts a 3-rd order feature V ∈ RH×W×R0 based on a sequence of 3-rd features { U ( l ) ∈ RH×W×Rl } ml=1 ( where H , W are height/width of the features and Rl is the number of channels in U ( l ) ) such that V : , : ,r0 = m∑ l=1 W ( l ) : , : ,rl , r0 ∗ U ( l ) : , : ,rl , withW ( l ) = CTTD ( { T ( l ) } ml=k ) , ∀l ∈ [ m ] ( 5 ) whereW ( l ) ∈ R [ l ( k−1 ) +1 ] × [ l ( k−1 ) +1 ] ×Rl×R0 is the corresponding weights tensor for U ( l ) . Suppose each W ( l ) takes a convolutional tensor-train format in Eq . ( 4 ) , we prove in Appendix A that the model in Eq . ( 5 ) can be computed sequentially similarly without reconstructing the originalW ( l ) ’ s : V ( l−1 ) : , : ,rl−1 = Rl∑ rl=1 T ( l ) : , : ,rl , rl−1 ∗ ( V ( l ) : , : ,rl + U ( l ) : , : ,rl ) , ∀l ∈ [ m ] ( 6 ) where { V ( l ) ∈ RH×W×Rl } ml=1 are intermediate results of the sequential process , where V ( m ) ∈ RH×W×Rm is initialized as all zeros and final prediction V = V ( 0 ) . The operations in Eq . ( 5 ) is illustrated in Figure 1a . In this paper , we denote the Eq . ( 5 ) simply as V = CTT ( { T ( l ) } ml=1 , { U ( l ) } ml=1 ) .

Summary: This paper proposes a method that saves memory and computation in the task of video prediction by low-rank tensor representations via tensor decomposition. The method is able to outperform standard convolutional lstm and other methods by using less parameters when testing it in the Moving MNIST and KTH datasets. The authors also present a proof to validate their method.