Modeling Variable Space with Residual Tensor Networks for Multivariate Time Series

1 INTRODUCTION . Multivariate time series are ubiquitous in the real world , such as energy collection and consumption ( Rangapuram et al. , 2018 ) , traffic occupancy ( Lai et al. , 2018 ) , financial market trends ( Shih et al. , 2019 ) , and aerospace data ( Zhao et al. , 2020 ) . They mainly consist of sensor data sampled on multiple variables at different timestamps . In order to accurately forecast future time series , it is necessary to learn the laws and the dynamic interaction information between variables that exist in the series history ( Wu et al. , 2020 ; Xu et al. , 2020 ; Cheng et al. , 2020 ) . For illustrating the dependencies in the multivariate time series , we focus on examples through case study . In Figure 1 ( a ) , a certain variable itself has periodic laws , and the dependency relationship in this respect can be well captured by the short-term and long-term model of the modeling sequence . As shown in Figure 1 ( b ) , there are dynamic dependency relations between the variables . The “ arrow ” indicates the trend of data changes . In 2011 , the “ Switzerland ” variable changed significantly earlier than the other in 2013 , the trends of the three variables occurred almost simultaneously . In literature , there have been some approaches to capture interactions in series history from different perspectives . research utilizing traditional deep learning components such as CNNs , RNNs , and attention mechanisms are more inclined to learn long-term and short-term sequence states , and fail to make full use of the entanglement between variables in the latent space ( Lai et al. , 2018 ; Shih et al. , 2019 ; Cheng et al. , 2020 ) . New mathematical tools such as graph neural networks and tensor calculation methods have inspired the research on the spatial Relations in multivariate time series . ( Wu et al. , 2020 ) uses graph neural network to extract one-way relationships between variable pairs in the end-to-end framework . The TLASM ( Xu et al. , 2020 ) adopts tensor calculation to model the trend in the multivariate time series , and combines the core tensor components into the LSTM . Nevertheless , most of the above-mentioned schemes for explicitly modeling spatio-temporal dependencies tend to bulid low-dimensional variable spaces , but this is still incomplete for the variable spaces shown in Figure 1 ( c ) of . In other words , in the high-dimensional variable space evolved over time , the ideal state for multivariate time series is that the model can capture the effective relationship between variables as fully as possible . To address the above challenges , we first define the high-dimensional variable space mathematically through the tensor product operator . Then , the proposed MVSRTN model models the variable space of multivariate time series . In the proposed scheme , N-Order Residual Tensor Network is the core component . The tensor network is based on the idea of low-rank approximation , which refers to the mode of decomposing and truncating the extremely large dependent space through tensor decomposition methods ( Ma et al. , 2019 ; Zhang et al. , 2019 ) in the case of a certain error . For modeling the exponential variable space , we first adopt the Tensor-Train ( Oseledets , 2011 ) algorithm satisfying the demands of processing different timestamps to decompose the variable space representation , and then obtain the space-approximated tensor network by tensor contraction . However , for long-term sequences , the continuous multiplication in the tensor contraction calculation would result in series of gradient problems . Inspired by ordinary differential equations , we extend and introduce N-order Euler discretization method , and the first-order approach can be thought of as residual connection . Finally , the space-approximated tensor network and the proposed N-order residual connection are coupled to model the effective interdependence in the variable space . In addition , Series-Variable Encoder aims at improving the quality of the variable space by reintegrating the time series . In this component , the combination of 1D CNN and self-attention mechanism at series level and variable level effectively enrich the dependency information mapped to the latent space . For making MVSRTN more robust , we use the linear Skip-Connection Layer to directly propagate the input scale and coding scale information to the Output Layer . Our contributions are of three-folds : • We represent the high-dimensional variable space , and through theoretical analysis , we demonstrate the space-approximated tensor network has the ability to model the latent variable space . • We propose the MVSRTN . In this architecture , a designed encoder combines long-term and short-term interactive information to optimize the variable space , a tensor network coupled by N-order residual connection captures the dynamically changing dependencies , and an additional layer propagates the scale features . • We perform extensive experiments on four benchmark datasets and the results demonstrate the effectiveness of MVSRTN in evaluation metrics . 2 RELATED WORK . Multivariate Time Series In the real world , a large number of multivariate time series tasks are derived from the fields of finance , economy , and transportation . The purpose of this type of task is to capture or predict the trend of a set of variables that change dynamically over time . In the research history of multivariate time series , researchers have reached a basic consensus that multiple variables are interdependent . After a period of development , statistical machine learning models such as Vector Auto-Regressive ( Box et al. , 2015 ) , Gaussian Process ( Frigola , 2015 ) algorithm can linearly model multivariate time series data , but the complexity of the model expands dramatically with the increase of time or variables . In recent years , with the rise of deep learning , neural networks implicitly capture the dependence of data in a non-linear mode . LSTNet ( Lai et al. , 2018 ) uses CNN and RNN to extract the local-term dependent mode of the time series , and extracts the long-term mode by recurrent-skip layer . Based on the attention mechanism and the LSTNet model , the TPALSTM ( Shih et al. , 2019 ) model can capture long-term dependencies across multiple time steps . However , algorithms based on neural networks only explicitly model the interaction in the time dimension , and fail to fully analyze the relationships in the latent variable space . The MTCNN ( Wu et al. , 2020 ) model based on graph neural network technology explicitly extracts the relationship between variables and achieves the effect of SOTA . However , only the one-way relationship between variable pairs is processed , which is not enough for the dynamic and complex variable space . Therefore , we consider using the idea of approximation to model the variable space . Tensor Network for Dependent Space The tensor network is a powerful mathematical modeling framework originated from many-body physics , which learns the effective associations in the exponential Hilbert space through the idea of low-rank approximation ( Stoudenmire & Schwab , 2016 ) . Based on the training approach of deep learning , Tensor networks just need to approximate the dependent space by training the decomposed parameter space . In the field of natural language , the TSLM ( Zhang et al. , 2019 ) model proves that the theoretical model obtained by tensor decomposition of the high-dimensional semantic space has a stronger ability to capture semantic dependence than the RNNs network . The u-MPS ( Zauner-Stauber et al. , 2018 ) tensor network based on Tensor-Train decomposition ( Oseledets , 2011 ) is also effective for probabilistic sequence modeling . Tensor networks have natural advantages in the field of modeling high-dimensional dependent spaces ( Miller et al. , 2021 ) . However , due to the gradient problem , the existing tensor network modeling schemes are still difficult to deal with real-world sequence data with long periods . Euler Discretization and Residual Connection The residual connection method can solve the problem of the gradient disappearance of the deep network ( He et al. , 2016 ) . When the gradient is saturated , it can maintain the stability of the network according to the principle of identity mapping . Based on the theory of differential dynamics , the residual connection can be considered as a discretization application of the first-order Euler solver ( Chen et al. , 2018 ) . Higher-order solvers ( for example , the second-order Runge-Kutta ) reduce the approximation error caused by the identity mapping ( Zhu et al. , 2019 ; Li et al. , 2021 ) , and also have better ability to process continuously sampled data ( Demeester , 2020 ) . 3 PROBLEM STATEMENT . In this paper , we denote the multivariate time series task as follows . Formally , given the time series signal of T points of time X = { y1 , y2 , · · · , yT } > , where yt = { vt,1 , vt,2 , · · · , vt , D } > ∈ RD , T is the number of time steps andD is the number of variables . We leverage X ∈ RT×D as time series history to forecast the series of next time stamp yT+h , where h ≥ 0 and it is the desirable horizon which is ahead of the current time stamp . It is worth mentioning that the range of the prediction task h varies on different tasks . The main purpose of this paper is to model the effective variable dependency in latent space . Hence , we formulate the variable space as shown in Definition 1 . Definition 1 . The tensor product operator is adopted to fully interact with the variable series yt indexed by different timestamps . Taking y1 and y2 as an example , the interactive function is : Ty1 , y2 = y1 ⊗ y2 = v1,1v2,1 · · · v1 , Dv2,1 ... . . . ... v1,1v2 , D · · · v1 , Dv2 , D ( 1 ) where Ty1 , y2 ∈ RD×D , and we define the whole variable space T ∈ RD T as : Ty1 , ··· , yT = y1 ⊗ y2 ⊗ · · · ⊗ yT ( 2 ) In the model architecture , we will detail how to model the proposed variable space . 4 MODEL ARCHITECTURE . In this section , we introduce the proposed MVSRTN architecture as shown in Figure 2 . The SeriesVariable Encoder is utilized to optimize the original series data . The N-Order Residual Tensor Network approximates the variable space , captures the dynamic dependence between variables over time , and ensures the ability to model long-term sequences . We leverage Skip-Connection Layer components to memory coding and scale information . Finally , the loss function exploited by MVSRTN is mentioned . 4.1 SERIES-VARIABLE ENCODER . The Series-variable Encoder is the first part of MVSRTN , in this component , 1D CNN layer is first exploited to extract short-term patterns in the time dimension as well as local dependencies between variables : ct = GeLU ( WC ∗ yt + bC ) ( 3 ) where ∗ represents convolution operation , WC ∈ RkD×d and bC ∈ Rd are all the weight parameters , d is the number of filters , k is the kernel size , and GeLU is an activation function ( Hendrycks & Gimpel , 2016 ) . After the above operations and zero-padding , the new sequence representation C = { c1 , · · · , cT } > ∈ RT×d can be obtained . In addition , the proposed model gains the representation that enriches the global interaction information by self-attention mechanism at series level and variable level , denoted as G ∈ RT×d , GS = Softmax ( Tril ( QSK > S√ d ) ) VS GV = Softmax ( QV K > V√ T ) VV G = ReLU ( Concat ( GS ; G > V ) WG + bG ) ( 4 ) where GS is the output of series-level self-attention , QS = CW Q S , KS = CW K S , VS = CW V S , and WS ∈ Rd×d are all weight parameters . GV is from variable-level self-attention , QV = C > WQV , KV = C > WKV , VV = C > W VV , and WV ∈ RT×T . Moreover , ReLU is an activation function ( Nair & Hinton , 2010 ) , Softmax means normalization , Tril refer to lower triangular matrix and WG ∈ R2d×d . A gate structure is leveraged to fuse information : X ′ = Sigmoid ( α ) ×C + Sigmoid ( β ) ×G ( 5 ) where α and β are all scale parameters , and Sigmoid is an function ( Leshno et al. , 1993 ) commonly used as gates . Finally , we achieve the encoded high-quality multivariate time series , and then map it to the latent variable space .

Summary: The paper considers the problem of forecasting of multivariate time series. The authors propose an architecture for such forecasting that incorporates several layers, including a residual tensor network layer. The idea is to tensorize (via outer products/Kronecker products) the features passed on from the encoder layer and then handle these very large tensorized features with a tensor network. The point of the tensorization is to better incorporate the effect of combinations of variables from different time steps. Experiments are done on four benchmark datasets.