Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

1 INTRODUCTION . Various monitoring applications , such as those for air quality ( Zheng et al . ( 2015 ) ) , health-care ( Silva et al . ( 2012 ) ) and traffic ( Jagadish et al . ( 2014 ) ) , widely use networked observation stations to record multivariate , geo-tagged time series data . For example , air quality monitoring systems employ a collection of observation stations at different locations ; at each location , multiple sensors concurrently record different measurements such as PM2.5 and CO over time . Such time series are important for advanced investigation and also are useful for future forecasting . However , due to unexpected sensor damages or communication errors , missing data is unavoidable . It is very challenging to impute the missing data because of the diversity of the missing patterns : sometimes almost random while sometimes following various characteristics . Traditional data imputation methods usually suffer from imposing strong statistical assumptions . For example , Scharf & Demeure ( 1991 ) and Friedman et al . ( 2001 ) fit a smooth curve on observations in either time series ( Ansley & Kohn ( 1984 ) ; Shumway & Stoffer ( 1982 ) ) or spatial distribution ( Friedman et al . ( 2001 ) ; Stein ( 2012 ) ) . Deep learning methods ( Li et al . ( 2018 ) ; Che et al . ( 2018 ) ; Cao et al . ( 2018 ) ; Luo et al . ( 2018a ) ) have been proposed to capture temporal relationship based on RNN ( Cho et al . ( 2014b ) ; Hochreiter & Schmidhuber ( 1997 ) ; Cho et al . ( 2014a ) ) . However , due to the constraint of sequential computation over time , the training of RNN can not be parallelized and thus is usually time-consuming . Moreover , the relationship between each two distant time stamps can not be directly modeled . Recently , the self-attention mechanism as shown in Fig . 1 ( b ) has been proposed by the seminal work of Transformer ( Vaswani et al . ( 2017 ) ) to get rid of the limitation of sequential processing , accelerating the training time substantially and improving the performance significantly on seq-to-seq tasks in Natural Language Processing ( NLP ) because the relevance between each two time stamps is captured explicitly . In this paper , we are the first to adapt the self-attention mechanism to impute missing data in multivariate time series , which cover multiple geo-locations and contain multiple measurements as Under review as a conference paper at ICLR 2020 Completed Value Available ObservationMissing Value Data Imputation Model ( a ) Location ( L ) Time ( T ) Measurement ( M ) Location ( L ) Time ( T ) Measurement ( M ) ( c ) Temporal Only ( Traditional ) Cross‐Dimensional ( Ours ) ( L ) ( T ) ( M ) ( L ) ( T ) ( M ) Query Key Value Weighted Sum ( b ) Attention ( Q , K , V ) Attention Map shown in Fig . 1 ( a ) . In order to impute a missing value in such unique multi-dimensional data , it is very useful to look into available data in different dimensions ( i.e . time , location and measurement ) , as shown in Fig . 1 ( c ) , to capture the intra-correlation individually . To this end , we investigate several choices of modeling self-attention across different dimensions . In particular , we propose a novel Cross-Dimensional Self-Attention ( CDSA ) mechanism to capture the attention crossing all dimension jointly yet in a decomposed manner . In summary , we make the following contributions : ( i ) We are the first to apply the self-attention mechanism to the multivariate , geo-tagged time series data imputation task , replacing the conventional RNN-based models to speed up training and directly model the relationship between each two data values in the input data . ( ii ) For such unique time series data of multiple dimensions ( i.e . time , location , measurement ) , we comprehensively study several choices of modeling self-attention crossing different dimensions . Our proposed CDSA mechanism models self-attention crossing all dimensions jointly yet in a dimension-wise decomposed way , preventing the size of attention maps from being too large to be tractable . We show that CDSA is independent with the order of processing each dimension . ( iii ) We extensively evaluate on two standard benchmarks and our newly collected traffic dataset . Experimental results show that our model outperforms the state-of-the-art models for both data imputation and forecasting tasks . We visualize the learned attention weights which validate the capability of CDSA to capture important cross-dimensional relationships . 2 RELATED WORK . Statistical data imputation methods . Statistical methods ( Ansley & Kohn ( 1984 ) ; Zhang ( 2003 ) ; Shumway & Stoffer ( 1982 ) ; Nelwamondo et al . ( 2007 ) ; Buuren & Groothuis-Oudshoorn ( 2010 ) ) often impose assumptions over data and reconstruct the missed value by fitting a smooth curve to the available values . For instance , Kriging variogram model ( Stein ( 2012 ) ) was proposed to capture the variance in data w.r.t . the geodesic distance . Matrix completion methods ( Acuna & Rodriguez ( 2004 ) ; Yu et al . ( 2016 ) ; Friedman et al . ( 2001 ) ; Cai et al . ( 2010 ) ; Ji & Ye ( 2009 ) ; Ma et al . ( 2011 ) ) usually enforce low-rank constraint . 𝑉 ” 𝑉 ” RNN-based data imputation methods . Li et al . ( 2018 ) proposed DCGRU for seq-to-seq by adopting graph convolution ( Chung & Graham ( 1997 ) ; Shi ( 2009 ) ; Shuman et al . ( 2012 ) ) to model spatial-temporal relationship . Luo et al . ( 2018a ) built GRUI by incorporating RNN into a Generative Adversarial Network ( GAN ) . Nevertheless , the spatiotemporal and measurements correlation are mixed and indistinguishable . so that the mediate back propagation from loss of available observation can contribute to the missing value updating . Nevertheless , these RNN-based models fundamentally suffer from the constraint of sequential processing , which leads to long training time and prohibits the direct modeling of the relationship between two distant data values . Self-attention . Recently , Vaswani et al . ( 2017 ) introduced the Transformer framework which relies on self-attention , learning the association between each two words in a sentence . Then self-attention has been widely applied in seq-to-seq tasks such as machine translation , image generation ( Yang et al . ( 2016 ) ; Zhang et al . ( 2018a ) ) and graph-structured data ( Veličković et al . ( 2017 ) ) . In this paper , we are the first to apply self-attention for multi-dimensional data imputation and specifically we investigate several choices of modeling self-attention crossing different data dimensions . 3 APPROACH . In Sec . 3.1 , we first review the conventional self-attention mechanism in NLP . In Sec . 3.2 , we propose three methods for computing attention map cross different dimension . In Sec . 3.3 and 3.4 , we present details of using CDSA for missing data imputation . 3.1 CONVENTIONAL SELF-ATTENTION . As shown in Fig . 1 ( b ) , for language translation task in NLP , given an input sentence , each word xi is mapped into a Query vector qi of d-dim , a Key vector ki of d-dim , and a Value vector vi of v-dim . The attention from word xj to word xi is effectively the scaled dot-product of qi and kj after Softmax , which is defined as A ( i , j ) = exp ( S ( i , j ) ) ( ∑T j=1 exp ( S ( q , j ) ) ) −1 where Then , vi is updated to v′i as a weighted sum of all the Value vectors , defined as v ′ i = ∑T j=1 A ( i , j ) vj , after which each v′i is mapped to the layer output x ′ i of the same size as xi . In order to adapt the self-attention from NLP to our multivariate , geo-tagged time series data , a straightforward way is to view all data in a time stamp as one word embedding and model the self-attention over time . 3.2 CROSS-DIMENSIONAL SELF-ATTENTION . In order to model Cross-Dimensional Self-Attention ( CDSA ) , in this section we propose three solutions : ( 1 ) model attention within each dimension independently and perform late fusion ; ( 2 ) model attention crossing all dimension jointly ; ( 3 ) model attention crossing all dimension in a joint yet decomposed manner . We assume the input X ∈ RT×L×M has three dimensions corresponding time , location , measurement . X can be reshaped into 2-D matrices ( i.e . XT ∈ RT×LM , XL ∈ RL×MT , XM ∈ RM×TL ) or an 1-D vector ( i.e . X ∈ RTLM×1 ) . Similarly , this subscript may be applied on the Query , Key and Value , e.g. , Q ∈ RT×L×M×d , QL ∈ RL×MTd and Q ∈ RTLM×d . 3.2.1 INDEPENDENT . As shown in Fig . 2 ( a ) , the input X is reshaped into three input matrices XT , XL and XM . Three streams of self-attention layers are built to process each input matrix in parallel . Such as the first layer in stream on XL , each vector XL ( l , : ) of MT -dim is viewed as a word vector in NLP . Following the steps in Sec . 3.1 , XL ( l , : ) is mapped to QL ( l , : ) and KL ( l , : ) of dL-dim , as well as VL ( l , : ) of vL-dim . The output of every stream ’ s last layer are fused through element-wise addition , X ′ = αTX ′T + αLX ′L + αMX ′M , where the weights αT , αL and αM are trainable parameters . Besides , the hyper-parameters for each stream such as the number of layers , are set separately . 3.2.2 JOINT . As shown in Fig . 2 ( b ) , the three-dimensional input X is reshaped as to X . Each unit X ( p ) is mapped to Q ( p , : ) and K ( p , : ) of d-dim as well as V ( p , : ) of v-dim , where p = p ( t , l , m ) denotes the index mapping from the 3-D cube to the vector form . In this way , an attention map of dimension TLM × TLM is built to directly model the cross-dimensional interconnection . 3.2.3 DECOMPOSED . The Independent manner sets multiple attention sub-layers in each stream to model the dimensionspecific attention but fail in modeling cross-dimensional dependency . In contrast , the Joint manner learns the cross-dimensional relationship between units directly but results in huge computation workload . To capture both the dimension-specific and cross-dimensional attention in a distinguishable way , we propose a novel Decomposed manner . As shown in Fig . 2 ( c ) , the input X is reshaped as input matrices XT , XL , XM and X . Each unit X ( p ) is mapped to vector V ( p , : ) of v-dim as in the Joint while XT , XL and XM are used for building attention map AT , AL , AM individually as in the Independent . The attention maps are applied on Value vector in order as , V ′ = AV = ÃMVS,2 = ÃMÃLVS,1 = ÃMÃLÃTV . ( 1 ) The attention map with ˜ is reshaped from the original attention map and consistent with the calculation in ( 1 ) , e.g. , ÃT ∈ RTLM×TLM is reshaped from AT ∈ RT×T . More specifically , ÃT = AT ⊗ IL ⊗ IM , ÃL = IT ⊗AL ⊗ IM , ÃM = IT ⊗ IL ⊗AM , ( 2 ) where ⊗ denotes tensor product and I is the Identity matrix where the subscript indicates the size , e.g. , IT ∈ RT×T . Although the three reshaped attention maps are applied with a certain order , according to ( 2 ) , we show that each unit in A is effectively calculated as A ( p0 , p1 ) = AT ( t0 , t1 ) AL ( l0 , l1 ) AM ( m0 , m1 ) , ( 3 ) where p0 = p ( t0 , l0 , m0 ) , p1 = p ( t1 , l1 , m1 ) . Following the associativity of tensor product , we demonstrate Ãσ ( T ) Ãσ ( L ) Ãσ ( M ) = AT ⊗AL ⊗AM , ( 4 ) where σ = σ ( T , L , M ) denotes the arbitrary arrangement of sequence ( T , L , M ) , e.g. , σ = ( L , T , M ) and σ ( T ) = L. Effectively , the arrangement σ is the order of attention maps to update V . As ( 3 ) - ( 4 ) shows that the weight in A is decomposed as the product of weights in three dimension-specific attention maps , the output and gradient back propagation are order-independent . Furthermore , we show in Supp . B that the cross-dimensional attention map has the following property : TLM∑ p1=1 A ( p0 , p1 ) = T∑ t1=1 L∑ l1=1 M∑ m1=1 AT ( t0 , t1 ) AL ( l0 , l1 ) AM ( m0 , m1 ) = 1 . ( 5 ) In summary , the Independent builds attention stream for each dimension while the Joint directly model the attention map among all the units . Our proposed CDSA is based on the Decomposed , which forms a cross-dimensional attention map , out of three dimension-specific maps . As an alternative of the Decomposed , the Shared maps unit X ( p ) to Q ( p , : ) and K ( p , : ) of d-dim and calculates all three dimension-specific attention map , e.g. , AL = Softmax ( QLK > L / √ MTd ) . As shown in Table 1 , by using Tensorflow profile and fixing the hyper-parameters with detailed explanations in Supp. , the Decomposed significantly decreases the FLoating point OPerations ( FLOPs ) compared to the Joint and requires less variables than the Independent . Detailed comparisons are reported in Sec . 4.3 .

Summary: This paper proposes a Transformer-based model with cross-dimensional self-attention for multivariate time series imputation and forecasting. The authors consider time series with observations collected at different locations (L), for different measurements (M), and at different timestamps (T). The authors describe 4 different self-attention mechanisms based on how the three dimensions, and it turns out the proposed Decomposed approach achieves the best performance and has moderate model complexity among the four. Experiments on several traffic and air quality datasets show the superiority of the proposed model.