Modeling Variable Space with Residual Tensor Networks for Multivariate Time Series

1 INTRODUCTION . Multivariate time series are ubiquitous in the real world , such as energy collection and consumption ( Rangapuram et al. , 2018 ) , traffic occupancy ( Lai et al. , 2018 ) , financial market trends ( Shih et al. , 2019 ) , and aerospace data ( Zhao et al. , 2020 ) . They mainly consist of sensor data sampled on multiple variables at different timestamps . In order to accurately forecast future time series , it is necessary to learn the laws and the dynamic interaction information between variables that exist in the series history ( Wu et al. , 2020 ; Xu et al. , 2020 ; Cheng et al. , 2020 ) . For illustrating the dependencies in the multivariate time series , we focus on examples through case study . In Figure 1 ( a ) , a certain variable itself has periodic laws , and the dependency relationship in this respect can be well captured by the short-term and long-term model of the modeling sequence . As shown in Figure 1 ( b ) , there are dynamic dependency relations between the variables . The “ arrow ” indicates the trend of data changes . In 2011 , the “ Switzerland ” variable changed significantly earlier than the other in 2013 , the trends of the three variables occurred almost simultaneously . In literature , there have been some approaches to capture interactions in series history from different perspectives . research utilizing traditional deep learning components such as CNNs , RNNs , and attention mechanisms are more inclined to learn long-term and short-term sequence states , and fail to make full use of the entanglement between variables in the latent space ( Lai et al. , 2018 ; Shih et al. , 2019 ; Cheng et al. , 2020 ) . New mathematical tools such as graph neural networks and tensor calculation methods have inspired the research on the spatial Relations in multivariate time series . ( Wu et al. , 2020 ) uses graph neural network to extract one-way relationships between variable pairs in the end-to-end framework . The TLASM ( Xu et al. , 2020 ) adopts tensor calculation to model the trend in the multivariate time series , and combines the core tensor components into the LSTM . Nevertheless , most of the above-mentioned schemes for explicitly modeling spatio-temporal dependencies tend to bulid low-dimensional variable spaces , but this is still incomplete for the variable spaces shown in Figure 1 ( c ) of . In other words , in the high-dimensional variable space evolved over time , the ideal state for multivariate time series is that the model can capture the effective relationship between variables as fully as possible . To address the above challenges , we first define the high-dimensional variable space mathematically through the tensor product operator . Then , the proposed MVSRTN model models the variable space of multivariate time series . In the proposed scheme , N-Order Residual Tensor Network is the core component . The tensor network is based on the idea of low-rank approximation , which refers to the mode of decomposing and truncating the extremely large dependent space through tensor decomposition methods ( Ma et al. , 2019 ; Zhang et al. , 2019 ) in the case of a certain error . For modeling the exponential variable space , we first adopt the Tensor-Train ( Oseledets , 2011 ) algorithm satisfying the demands of processing different timestamps to decompose the variable space representation , and then obtain the space-approximated tensor network by tensor contraction . However , for long-term sequences , the continuous multiplication in the tensor contraction calculation would result in series of gradient problems . Inspired by ordinary differential equations , we extend and introduce N-order Euler discretization method , and the first-order approach can be thought of as residual connection . Finally , the space-approximated tensor network and the proposed N-order residual connection are coupled to model the effective interdependence in the variable space . In addition , Series-Variable Encoder aims at improving the quality of the variable space by reintegrating the time series . In this component , the combination of 1D CNN and self-attention mechanism at series level and variable level effectively enrich the dependency information mapped to the latent space . For making MVSRTN more robust , we use the linear Skip-Connection Layer to directly propagate the input scale and coding scale information to the Output Layer . Our contributions are of three-folds : • We represent the high-dimensional variable space , and through theoretical analysis , we demonstrate the space-approximated tensor network has the ability to model the latent variable space . • We propose the MVSRTN . In this architecture , a designed encoder combines long-term and short-term interactive information to optimize the variable space , a tensor network coupled by N-order residual connection captures the dynamically changing dependencies , and an additional layer propagates the scale features . • We perform extensive experiments on four benchmark datasets and the results demonstrate the effectiveness of MVSRTN in evaluation metrics . 2 RELATED WORK . Multivariate Time Series In the real world , a large number of multivariate time series tasks are derived from the fields of finance , economy , and transportation . The purpose of this type of task is to capture or predict the trend of a set of variables that change dynamically over time . In the research history of multivariate time series , researchers have reached a basic consensus that multiple variables are interdependent . After a period of development , statistical machine learning models such as Vector Auto-Regressive ( Box et al. , 2015 ) , Gaussian Process ( Frigola , 2015 ) algorithm can linearly model multivariate time series data , but the complexity of the model expands dramatically with the increase of time or variables . In recent years , with the rise of deep learning , neural networks implicitly capture the dependence of data in a non-linear mode . LSTNet ( Lai et al. , 2018 ) uses CNN and RNN to extract the local-term dependent mode of the time series , and extracts the long-term mode by recurrent-skip layer . Based on the attention mechanism and the LSTNet model , the TPALSTM ( Shih et al. , 2019 ) model can capture long-term dependencies across multiple time steps . However , algorithms based on neural networks only explicitly model the interaction in the time dimension , and fail to fully analyze the relationships in the latent variable space . The MTCNN ( Wu et al. , 2020 ) model based on graph neural network technology explicitly extracts the relationship between variables and achieves the effect of SOTA . However , only the one-way relationship between variable pairs is processed , which is not enough for the dynamic and complex variable space . Therefore , we consider using the idea of approximation to model the variable space . Tensor Network for Dependent Space The tensor network is a powerful mathematical modeling framework originated from many-body physics , which learns the effective associations in the exponential Hilbert space through the idea of low-rank approximation ( Stoudenmire & Schwab , 2016 ) . Based on the training approach of deep learning , Tensor networks just need to approximate the dependent space by training the decomposed parameter space . In the field of natural language , the TSLM ( Zhang et al. , 2019 ) model proves that the theoretical model obtained by tensor decomposition of the high-dimensional semantic space has a stronger ability to capture semantic dependence than the RNNs network . The u-MPS ( Zauner-Stauber et al. , 2018 ) tensor network based on Tensor-Train decomposition ( Oseledets , 2011 ) is also effective for probabilistic sequence modeling . Tensor networks have natural advantages in the field of modeling high-dimensional dependent spaces ( Miller et al. , 2021 ) . However , due to the gradient problem , the existing tensor network modeling schemes are still difficult to deal with real-world sequence data with long periods . Euler Discretization and Residual Connection The residual connection method can solve the problem of the gradient disappearance of the deep network ( He et al. , 2016 ) . When the gradient is saturated , it can maintain the stability of the network according to the principle of identity mapping . Based on the theory of differential dynamics , the residual connection can be considered as a discretization application of the first-order Euler solver ( Chen et al. , 2018 ) . Higher-order solvers ( for example , the second-order Runge-Kutta ) reduce the approximation error caused by the identity mapping ( Zhu et al. , 2019 ; Li et al. , 2021 ) , and also have better ability to process continuously sampled data ( Demeester , 2020 ) . 3 PROBLEM STATEMENT . In this paper , we denote the multivariate time series task as follows . Formally , given the time series signal of T points of time X = { y1 , y2 , · · · , yT } > , where yt = { vt,1 , vt,2 , · · · , vt , D } > ∈ RD , T is the number of time steps andD is the number of variables . We leverage X ∈ RT×D as time series history to forecast the series of next time stamp yT+h , where h ≥ 0 and it is the desirable horizon which is ahead of the current time stamp . It is worth mentioning that the range of the prediction task h varies on different tasks . The main purpose of this paper is to model the effective variable dependency in latent space . Hence , we formulate the variable space as shown in Definition 1 . Definition 1 . The tensor product operator is adopted to fully interact with the variable series yt indexed by different timestamps . Taking y1 and y2 as an example , the interactive function is : Ty1 , y2 = y1 ⊗ y2 = v1,1v2,1 · · · v1 , Dv2,1 ... . . . ... v1,1v2 , D · · · v1 , Dv2 , D ( 1 ) where Ty1 , y2 ∈ RD×D , and we define the whole variable space T ∈ RD T as : Ty1 , ··· , yT = y1 ⊗ y2 ⊗ · · · ⊗ yT ( 2 ) In the model architecture , we will detail how to model the proposed variable space . 4 MODEL ARCHITECTURE . In this section , we introduce the proposed MVSRTN architecture as shown in Figure 2 . The SeriesVariable Encoder is utilized to optimize the original series data . The N-Order Residual Tensor Network approximates the variable space , captures the dynamic dependence between variables over time , and ensures the ability to model long-term sequences . We leverage Skip-Connection Layer components to memory coding and scale information . Finally , the loss function exploited by MVSRTN is mentioned . 4.1 SERIES-VARIABLE ENCODER . The Series-variable Encoder is the first part of MVSRTN , in this component , 1D CNN layer is first exploited to extract short-term patterns in the time dimension as well as local dependencies between variables : ct = GeLU ( WC ∗ yt + bC ) ( 3 ) where ∗ represents convolution operation , WC ∈ RkD×d and bC ∈ Rd are all the weight parameters , d is the number of filters , k is the kernel size , and GeLU is an activation function ( Hendrycks & Gimpel , 2016 ) . After the above operations and zero-padding , the new sequence representation C = { c1 , · · · , cT } > ∈ RT×d can be obtained . In addition , the proposed model gains the representation that enriches the global interaction information by self-attention mechanism at series level and variable level , denoted as G ∈ RT×d , GS = Softmax ( Tril ( QSK > S√ d ) ) VS GV = Softmax ( QV K > V√ T ) VV G = ReLU ( Concat ( GS ; G > V ) WG + bG ) ( 4 ) where GS is the output of series-level self-attention , QS = CW Q S , KS = CW K S , VS = CW V S , and WS ∈ Rd×d are all weight parameters . GV is from variable-level self-attention , QV = C > WQV , KV = C > WKV , VV = C > W VV , and WV ∈ RT×T . Moreover , ReLU is an activation function ( Nair & Hinton , 2010 ) , Softmax means normalization , Tril refer to lower triangular matrix and WG ∈ R2d×d . A gate structure is leveraged to fuse information : X ′ = Sigmoid ( α ) ×C + Sigmoid ( β ) ×G ( 5 ) where α and β are all scale parameters , and Sigmoid is an function ( Leshno et al. , 1993 ) commonly used as gates . Finally , we achieve the encoded high-quality multivariate time series , and then map it to the latent variable space .

Summary: The paper proposes a deep learning architecture for time series forecasting called MVSRTN. The model is composed of 3 blocks with skip-connections in-between them: a 1) "Series-Variable Encoder", 2) an "N-Order Residual Tensor Network", as the authors call it, and 3) the output layer. Out of these, 1) is essentially a 1D CNN combined with (causal and non-causal) self-attention, and 2) seems to be a tensor network built from taking the tensor product of the sequence entries, and then contracting that by a TT-rank-constrained weight tensor. The computation of the tensor network output is formulated as a recursion across time-steps, and motivated by residual networks, an identity mapping with respect to the hidden state at the previous time-stamp is added to the recursive formulation. Then, by the analogy with higher-order solvers for ODEs, a "higher-order residual connection" is introduced. The model is applied to four time series forecasting datasets, where some improvements are achieved, and an ablation study is included to show how the various model blocks contribute to the performance.