Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation
Authors: Jiawei Ma*
Zheng Shou*
Alireza Zareian
Hassan Mansour
Anthony Vetro
Shih-Fu Chang
Venue:  (2020)
Keywords: self-attention
cross-dimensional
multivariate time series
imputation

Abstract: Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages or communication errors. In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps. Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly. In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data. In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner. On three real-world datasets, including one our newly collected NYC-traffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks. 


Under review as a conference paper at ICLR 2020
CROSS-DIMENSIONAL SELF-ATTENTION FOR MULTI- VARIATE, GEO-TAGGEL TIME SERIES IMPUTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Many real-world applications involve multivariate, geo-tagged time series data: each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages communication errors. In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps. Recently, the self-attention mechanism has been proposed for sequence modeling tasks such machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly. In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data. In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose novel approach called Cross-Dimensiona (CDSA) to process each dimension sequentially, yet in an order- indeeeenne manner. On three real-world datasets, including one our newly collected NYC- traffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.
Delf-Attention
INTRODUCTION
Various monitoring applications, such as those for air quality (Zheng et al. (2015)), health-care (Silva al. 2012)) and traffic (Jagadish et al. (2014)), widely use networked observation stations to record multivariate, geo-tagged time series data. For example, air quality monitoring systems employ collection of observvtion stations at different locations; at each location, multiple sensors concurrently record different measurements such PM2.5 and CO over time. Such time series are important for advanced investigation and also are useful for future forecasting. However, due to unexpected sensor damages communication errors, missing data is unavoidable. It is very challenging to impute the missing data because of the diversity of the missing patterns: sometimes almost random while sometimes following various characteristics.
Traditional data imputation methods usually suffer from imposing strong statistical assumptions. For example, [Shharf & Demeure (199I) and Friedman al. (2001) fit a smooth curve on observations in either time series (Ansley& & Kohn 1984); (Shumway & Stoffer (1982)) or spatial distribution (Fried- man et al. (2001): ; Stein (2012)) Deep learning methods (Li et al. (2018); Che al. (2018); Cao
et al. (2018): Luo et al. (2018a)) have been proposed to capture temporal relationship based on RNN
(Cho et al. (2014b); Hochreiter Schmidhuber (1997); Cho al. (2014))). However, due to the constraint of sequential computation over time, the training of RNN cannot be parallelized and thus usually time-consuming. Moreover, the relationship between each two distant time stamps cannot be directly modeled. Recently, the self-attention mechanism shown in Fig. [[bb has been proposed
by the seminal work of Transformer (Vaswani et al. (2017)) to get rid of the limitation of sequential processing, accelerating the training time substantially and improving the performance significantly on seq-to-seq tasks in Natural Language Processing (NLP) because the relevance between each two time stamps is captured explicitly.
In this paper, we are the first to adapt the self-attention mechanism to impute missing data in multivariate time series, which cover multiple geo-locations and contain multiple measurements as

---

Under review as a conference paper at ICLR 2020
Missing Value
Completed Value
Available Observation
(a)
Time (
Time (T)
Temporal Only (Traditional)
(T)
Data Imputation
Location
(L)
Model
Location
(L)
(L)
Measurement (M)
Measurement (M)
(b)
(M)
(T)
Query
Value
Key
Weighted
Cross- Dooss-Dimensionl
Attention Map
Updated Value
(Ours)
[M)
Figure 1:
(a) Illustration of the multivariate, geo-tagged time series imputation task: the input data has three dimensions (i.e. time, location, measurement) with some missing values (indicated by the orange dot); the output is of same shape as the input while the missing values have been imputed (indicated by the red dot). (b) Self-attention mechanism: the Attention Map is first computed using every pair of Query vector and Key vector and then guides the updating of Value vectors via weighted sum to take into account contextual information. (c) Traditional Self-Attention mechanism updates Value vector along the temporal dimension only vs. Proos-DDmenssinal Sell-Attention mechanism updates Value vector according to data across all dimensions.
shown in Fig. [a). order to impute a missing value in such unique muult--dimensionll data, is very useful to look into available data in different dimensions (i.c. time, location and measurement), as shown in Fig. Fig.Te), [Tc, to capture the intra-correlation individually. To this end, we investigate several choices of modeling self-attention across different dimensions. In particular, Self-Attentio (CDSA) mechanism to capture the attention crossing all dimension jointly yet decomposed manner. In summary, we make the following contributions:
propose novel
(i) We are the first to apply the self-attention mechanism to multivariate, geo-tagged time series data imputation task, replacing the conventional RNN-based models to speed up training and directly model the relationship between each two data values the input data.
(ii) For such unique time series data of multiple dimensions (i.e. time, location, measurement), comprehensively study several choices of modeling self-attention crossing different dimensions. Our proposed CDSA mechanism models self-attention crossing all dimensions jointly yet in dimension- sinisiioowiiieeeeee decomposed way, preventing the size of attention maps from being too large to be tractable. We show that CDSA is independent with the order of processing each dimension.
(iii) We extensively evaluate two standard benchmarks and newly collected traffic dataset. Experimental results show that our model outperforms the state-of-th fi--hhe-n art models for both data imputation and forecasting tasks. We visualize the learned attention weights which validate the capability of CDSA to capture important cross-dimensional relationships.
2 RELATED WORK
Statistical data imputation methods. Statistical methods (Ansley & Kohn (1984); Zhang (203):
Shumway & Stoffer (982); Nelwamondo et al. (2007): Buuren & Sobothuis-Oudshood (0100)
often impose assumptions over data and reconstruct the missed value by fitting smooth curve to
the available values. For instance, Kriging variogram model (Stein (2022) ) was proposed to capture
the variance in data w.r.t. the geodesic distance. Matrix completion methods (Acuna & Rodriguez (2004); Yu et al. 2016):/Freddm) et al. (200T): Cai et al. (000)) Ji & Ye (2009); Ma et al. (2011)) usually enforce low-rank constraint.
2

---

Under review as a conference paper at ICLR 2020
Value(T)
AttMap (TxT)
Value (TXL*M*V)
Value
(TxLxMxV)
AttMap (TxT)
Value (TLMxV)
FC Layer
Reshape
Value(L)
AttMap (LxL)
AttMap (LxL)
Addition w/ Weight
V
FC Layer
Value(M)
AttMap (MxM)
V/.2
AttMap (M*M)
AttMap[TLMxTLM)
FC Layer
Vix
FC Layer
V
FC Layer
(a) Independent
(b) Joint
(c) Decomposed
Figure 2: Three choices of implementing our Cross-Dimensional
mechanism
RNN-based data imputation methods. Li et al. 20 8 proposed DCGRU for seq-to-seq by adopting graph convolution (Chung & Graham (1997) ; Shi 2009); Shuman al. (2022) to model spatial-temporal relationship. Luo et al. 2018ap built GRUT by incorporating RNN into a Generative Adversarial Network (GAN). Nevertheless, the spatiotemporal and measurements correlation are mixed and indistinguishable. can contribute to the missing value updating. Nevertheless, these RNN-based models fundamentally suffer from the constraint of sequential processing, which leads to long training time and prohibits the direct modeling of the relationship between two distant data values.
that the mediate back propagation from loss of available observation
Self-attention. Recently, Vaswani al. (017) introduced the Transformer framework which relies on self-attention, learning the association between each two words in a sentence. Then self-attention has been widely applied seq-to-se seo-see tasks such as machine translation, image generation Yangg et al. (2016); Zhang et al. (2018a)) and graph-structured data (Veličković et al. (2017)). In this paper, we are the first apply self-attention for multi--imenniioall data imputation and specifically investigate several choices of modeling self-attention crossing different data dimensions.
APPROACH
In Sec. 3.1 we first review the conventional self-attention mechanism in NLP. In Sec. 3.2 three methods for computing attention map cross different dimension. In Sec. 3.3 and 3.4 we present details of using CDSA for missing data imputation.
we propose
3.1 CONVENTIONAL SELF-ATTENTION
As shown in Fig. bb), for language translation task in NLP, given an input sentence, each word 2; is mapped into Query vector qi of d-dim, a Key vector k; of d-dim, and a Value vector v; of o-dim. The attention from word æ, to word æ; is effectively the scaled dot dot-product of q: and ky after Softmax, which is defined as A (ii,j) = exp(S(i,.))) EJ_ (xp(S(q.))))
where Then, v; updated to o as a weighted sum all the Value vectors, defined as oa - A(i,)), after which each of is mapped to the layer output x of the same size as T, In order to adapt the self-attention from NLP to multivariate, geo-tagged time series data. straightforward way is to view all data in time stamp as one word embedding and model the self-attention over time.
3.2 CROSS-DIMEE DIMENSIONAL SELF-ATTENTION
In order to model mooss-Dimensional Self-Attention (CDSA), in this section we propose three solutions: (1) model attention within each dimension independently and perform late fusion; (2)

---

Under review as a conference paper at ICLR 2020
model attention crossing all dimension jointly; (3) model attention crossing all dimension in joint yet decomposed manner. We assume the input E R I.X has three dimensions corresponding time, location, measurement. X' can be reshaped into 2-D matrices (i.e. XT E RTxLM Xx e R R.XMT MT
XM E TRMxTL or an 1-D vector (i.e. X E RTTMXX x Similarly, this subscript may be applied on the Query, Key and Value, e.g., 2 DERTTR TxLxxxx Qc E RLxMTd and Q E TR
3.2.1 INDEPENDENT
As shown in Fig. 2(a) the input X' is reshaped into three input matrices XT. X and XM. Three streams of self-ateeniion layers are built to process each input matrix in parallel. Such as the first layer in stream on Xc. each vector (l.:)) of Mi'-dim viewed as word vector in NLP. Following the steps in Sec. 3.1 X (1,;)) is mapped to Qz(1;) and K (1,:) of dz-dim, as well as V2 of Ur dr,. The output of every stream 's last layer are fused through element-wise addition, X" ar&Y + ar,22 + & where the weights at, ar and am are trainable parameters. Besides, the hyper- ppprrpparaeeters for each stream such as the number of layers, are set separately.
3.2.2 JOINT
As shown in Fig. 2(b) the firee-imeensiona input X' is reshaped as to X. Each unit ( is mapped to (p,,) and K (p. :) of d-dim as well as (p,:)) of e-dim, where p p(t. I, m) denotes the index mapping from the 3-D cube to the vector form. In this way, an attention map of dimension TLM TLM is built to directly model the vross-dimensional interconnection.
3.2.3 DECOMPOSED
The Independent manner sets multiple attention sub-layers in each stream model the dimension- specific attention but fail in modeling cross- pooss-dimensional dependency. In contrast, the Joint manner learns the diseessiiesi relationship between units directly but results in huge computation workload. To capture both the cemeesion-sspciffe and cross-dimensional attention in a distinguishable way, we propose novel Decomposed manner.
As shown in Fig. 2(c) the input X' is reshaped as input matrices XT. Xx, XM and X. Each unit (p) is mapped vector Vp,,:) :) of u-dim as in the Joint while X,. X c and XM are used for building attention map Ar, Ar.AM individually in the Independent. The attention maps are applied on Value vector in order as,
V AV = AM AM V.,
Å Ar
(1)
The attention map with is reshaped from the original attention map and consistent with the calculation in D. e.g., Ar E TI..MTIM is reshaped from Ar E RTx More specifically,
År Ar & IL & IM.
A, IT @ A X IM,
ÅM AM- IT @ I Am
where @ denotes tensor product and I is the Identity matrix where the subscript indicates the size, e.g., In E RTXT. Although the three reshaped attention maps are applied with a certain order, according show that each unit in A is effectively calculated as
A PI) Ar (lo, 4 A 41)..44)... AM (mo, m1),
where Po p(oo, lo mo) (111,4,m1) Following the associativity of tensor product, demonstrate
Ao( Āo(M; Ar Ar @ Am:
where a = (T.L.M) denotes the arbitrary arrangement of sequence (T.L.M), e.g., (LL,T, and (T) I.. Effectively, the arrangement is the order of attention maps to update V. As B shows that the weight in A is decomposed as the product of weights in three dimension-speeiitc attention maps, the output and gradient back propagation are inder-independent. Furthermore, we show in Supp. B that the coos--dimensionl attention map has the following property:
TLM
Σ P1=1
A(po pI)
Σ 4=14=1 m==1
ΣΣ
Ar (4) m.) 1.
(5)
4

---

Under review as a conference paper at ICLR 2020
(a) Imputation
Missing Value
Completed Value
Transformer Encoder CDSA
[
Forward
(U]
(4)
Available Observation
Loss
(b) Forecasting
(M)
(M)
Transformer Encoder w/ CDSA
Transformer Decoder w/ CDSA
T+1
T+1
L4
Forward
(L)
(L)
[M]
(M)
(M)
Figure 3: The framework employing CDSA for data imputation and forecasting.
In summary, the Independent builds attention stream for each dimension while the Joint directly model the attention map among all the units. Our proposed CDSA is based on the Decomposed, which forms cross-dimensional attention map, out three dimension-speciife maps. As an alternative of the Decomposed, the Shared maps unit X (p) to (p,;) and K (p, :) of d-dim and calculates all three dimension-specti attention map, e.g., A,, = Softmax (Q [ /Tdd). As shown in Table by using Tensorflow profile and fixing the ryper-parameters with detailed explanations in Supp., the Decomposed significantly decreases the Floating point OPerations (FIOPs) compared to the Joint and requires less variables than the Independent. Detailed comparisons are reported in Sec.
Table 1: Computational complexity of several methods to implement CDSA
Methods
Independent
Joint
Shared
Decomposed
FLOPs(x 10")
0.39
3.22
0.21
0.24
Number Variables (x109)
18.15
0.44
0.44
16.09
3.3 FRAMEWORK
Imputation: As shown in Fig. B(a), we apply our CDSA mechanism in Transformer Encoder, stack of N 8 identical layers with residual connection (He et al. (2016) and normalization (Lei Ba et al. (2016)) employed by Vaswani et al. (2017). To reconstruct the missing (along with other) values the input, we apply a fully connected Feed Forward network on the final Value tensor, which trained jointly with the rest of the model.
Forecasting: As shown in Fig. 3)), we apply our CDSA mechanism Transformer framework where N = 9 for both encoder and decoder. Similar to imputation, fully connected feed forward network to generate the predicted values.
3.4 IMPLEMENTATION DETAILS
We normalize each mcasurement of the input by subtracting the mean and dividing by standard deviation across training data. Then the entries with missed value set O. We use the Adam optimizer (Kingma & Ba (014)) to minimize the Root Mean Square Error (RMSE) between the prediction and ground truth. The model is trained single NVIDIA GTX 1080 Ti GPU. More details (e.g., network hyper-paane parameters, learning rate and batch size) can be found in Supp.
EXPERIMENTS
4.1 DATASETS, TASKS, EVALUATION METRICS
NYC-Traffic. New York City Department of Transportation has set up various street cameras] Each camera keeps taking snapshot every a few seconds. The is collected around 1-month data
hrtps///nyttm..rr/

---

Under review as a conference paper at ICLR 2020
for 186 cameras Manhattan from 1 12/03/2015 to 12/26/2015. For each snapshot, we apply trained faster-RCNN (Ren et al. (2013) vehicle detection model to detect the number vehicles (#vehicle) in each snapshot. To aggregate such raw data into time series, for every non-overlapping 5-minute window, we averaged #vehicle from each snapshot to obtain the average #vehicle as the only measurement. Finally, obtained 186 time series and the gap between consecutive time stamps is 5 minutes.
The natural missing rate of the whole dataset 8.43%. In order to simulate experiments for imputation, we further remove some entries and hold them as ground truth for evaluation. The imputation task is estimate values of these removed entries. To mimic the natural data missing pattern, we model our manual removal as a Burst Loss, which means at certain location the data is continuously missed for a certain period. More details about vehicle detection and burst loss are be found in Supp. To simulate various data missing extents, we vary the final missing rate after removal from 20% to 90%. For each missing rate, we randomly select 432 consecutive time slots to train our model and evaluate the average RMSE of 5 trials. The dataset will be released publicly.
KDD-2018 (Cup 2018)) is an Air Quality and Meteorology dataset recorded hourly. As indicated in Luo al. (2018a), 11 locations and 12 measurements are selected. The natural missing rate is 6.83%. In order to simulate experiments for imputation, we follow Luo et al. (2018a) to split the data to every 48 hours, randomly hold values of some available entries and vary the missing rate from 20% to 90%. Mean Squared Error (MSE) is used for evaluation.
METR-LA Jagadish et al. 2014)). We follow Li et al. 2018) to this dataset for traffic speed forecasting. This dataset contains traffic speed at 207 locations recorded every 5 minutes for 4 months. Following Li al. 2018), 80% of data the beginning of these 4 months is used for training and the remaining 20% is for testing. In order to simulate the forecasting scenario, within either training or testing set, every time series of consecutive hours are enumerated. For each time series, data in the first hour is treated as input and data in the second hour is to be predicted. We respectively evaluate the forecasting results at 15-th, 30-th, 60-th minutes in the second I hour and also evalaute the average evaluation results within the total 1 hour. We use RMSE, Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) as evaluation metries.
4.2 COMPARISONS WITH STATE-OF-THE-AN
Table 2: RMSE on dataset NYC-Traffic for comparisons with SOTA
Model \ Missing Rate
20%
30%
40%
50%
60%
70%
80%
90%
Auto Regressive Kriging expo
2.354
2.357
2.359
2.362
2.364
2.652
2.796
3.272
2.142
2.145
2.157
2.152
2.155
2.165
2.182
2.231
Kriging linear
2.036
2.008
2.031
2.038
2.056
2.074
2.111
2.194
MTSI et al. 2018a
1.595
1.597
1.603
1.605
1.608
1.641
1.672
1.834
BRITS Cao et a. 2018
1.337
1.339
1.341
1.355
1.376
1.395
1.408
1.477
DCRNN Li et al. 2018
1.397
1.399
1.401
1.419
1,432
1.443
1,459
1.601
CDSA (ours)
1.203
1.208
1.211
1.214
1.215
1.217
1.234
1.377
Table 3: MSE on dataset KDD-2018 for comparisons with SOTA
Model
Missing Rate
20%
30%
40%
50%
60%
70%
80%
90%
Mean Filling
0.916
0.907
0.914
0.923
0.973
0.935
0.937
1.002
KNN Filling
0.892
0.803
0.776
0.798
0.856
0.852
0.873
1.243
MF Filling
0.850
0.785
0.787
0.772
0.834
0.805
0.860
1.196
MTSI Luo al.
(2018) 8a)
0.844
0.780
0.753
0.743
0.803
0.780
0.837
1.018
BRITS Cao et al.
(2018)
0.455
0.421
0.372
0.409
0.440
0.482
0.648
0.725
DCRNN Li et al. 2018
0.579
0.565
0.449
0.506
0.589
0.622
0.720
0.861
CDSA (ours)
0.373
0.393
0.287
0.291
0.387
0.495
0.521
0.631
Imputation (NYC-Traffic, KDD-2018) Table 2. our CDSA consistently outperforms traditional
methods (i.e., Auto Regressive, Kriging expo. Kriging linear) and recent RNN-based methods (i.e. MTSI, BRITS, DCRNN) over a wide range of missing rate. Because CDSA leverages the self-attention mechanism to avoid sequential processing of RNN and directly model the relationship

---

Under review as a conference paper at ICLR 2020
between distant data. Table shows that method again achieves significant improvements cross-dimensional data imputation task. Detailed overview of baselines can be found Supp.
Forecasting (METR-LA). Table 4 shows that for the forecasting task, our CDSA method outper- forms previous methods in most cases. particular, our method demonstrates clear improvement long-term forecasting such as 60 min. This again confirms that CDSA cthe effectiveness of directly modeling the relationship between every two data values (could from different dimensions and of far distance). But RNN-based methods and methods that sequentially conduct spatial conv and temporal conv fail to model the distant spatio-temporal relationship explicitly.
Table 4: MAE/R.SSEMAPE dataset METR-LA for comparisons with SOTA
15min
30min
Model
MAE
RMSE
MAPE
MAE
RMSE
MAPE
FC-LSTM Sutskever et al. (014))
3.44
6.3
9.6%
3.77
7.23
10.9%
MTSI
Luo al. (2018a)
3.75
7.31
10.52%
3.89
7.73
11.04%
BRITS
Cao et al. (2088
2.86
5.46
7.49%
3.37
6.78
9.13%
DCRNN
al. 2018
2.77
5.38
7.3%
3.15
6.45
8.8%
DST-GCNN
Wang el al. (2018)
2.68
5.35
7.2%
3.01
6.23
8.52%
GaAN
Zhang et al. (20 18b)
2.71
5.24
6.99%
3.12
6.36
8,56%
CDSA(ours)
3.01
5.08
7.82%
3.14
5.38
8.30 %
60min
Mean
Model
MAE
RMSE
MAPE
MAE
RMSE
MAPE
FC-LSTM
Sutskever et al. (014))
4.37
6.89
13.2%
3.86
7.41
11.2%
MTSI Luo al. (2018a
4.22
8.39
12.15%
4.01
7.59
10.85%
BRITS Cao et a. (2018
3.65
7.66
10.55%
3.32
6.96
9.47%
DCRNN Li ci al. 2018
3.60
7.59
10.50%
3.28
6.80
8.87%
DST-GCNN Wang al. (2018)
3.41
7.47
10.25%
GaAN Zhang et al. (2018))
3.6
7.6
10.5%
3.16
6.41
8.72%
CDSA(ours)
3.40
6.27
9.76%
3.16
5.48
8.50%
4.3 DISCUSSIONS
The effects of different training losses: For the forecasting task in METR-LA, we compare the performance by setting different training loss in Table 5 and we can see the performance with RMSE as loss metric achieves the best performance.
Table 5: Comparisons of different losses CDSA on METR-LA
Time
30min
Ave
30min
Ave
30min
Ave
30min
Ave
Metric
Loss
RMSE
MSE
MAE
(RMSE+MAFJ)?
MAE
3.14
3.16
3.43
3.41
3.28
3.33
3.21
3.25
RMSE
5.38
5.48
6.20
6.11
5.67
5.83
5.55
5.70
MAPE
8.30%
8.50 %
9.32%
9.19
8.70%
9.00%
8.53%
8.80%
Ablation study of different cross-dimensional self-attention manners: We compare the perfor- mance for different solutions CDSA mechanism the three datasets listed above. 1) The way of attention modeling determines the computational complexity. As shown in Table [ since the Independent calculates dimension-speeiffc Value vectors parallel, the number of variables and FL.OPs are larger than those of the Decomposed. As the Join! and the Shared both share the variables for each dimension, the number variables small and basically equals with each other. the Joint builds huge attention map, its FLOPs is much larger than others. Since the Decomposed draws attention maps like the Independent but shares Value like the Joint, reduces the computational complexity significantly. 2) As shown in Table 6 - evaluate these methods three datasets and the Decomposed always achieves the best performance thanks to the better learning ability compared to the Joint and Shared. More discussions be found in Supp.
Study of using the imputed time series for forecasting. On WYC-Traffit of missing rate 50%, we impute missing values in historical data (using statistical methods and our CDSA respectively) and
7

---

Under review as a conference paper at ICLR 2020
then feed the imputed historical data into traffic prediction model ARIMA. We compare performances in terms of RMSE: Mean Filling 1.953, Kriging expo 1.681, Kriging linear 1.733, MTSI 1.675, DCRNN 1.666, BRITS 1.579, CDSA 1.536. This indicate that when using the imputed time series for forecasting, our CDSA can achieve significant gains in the downstream forecasting task as well compared to traditional imputation methods. More details can be found in Supp. E.3.
Table 6: Comparisons of different manners to implement CDSA on dataset NYC-Traffic.
Model Missing Rate
20%
30%
40%
50%
60%
70%
80%
90%
CDSA(Independent)
1.327
1.327
1.331
1.355
1.362
1.379
1.393
1.425
CDSA
Not Applicable due to memory usage limitation
CDSA (DSa(Shared))
1.637
1.645
1.651
1.657
1.684
1.729
1.733
1.935
CDSA (Dcompec)o)))
1.204
1.208
1.211
1.214
1.215
1.217
1.235
1.377
Table 7: Comparisons of different manners to implement CDSA on dataset KDD-2018.
Model Missing Rate
20%
30%
40%
50%
60%
70%
80%
90%
CDSA((ndependen))
0.482
0.523
0.351
0.366
0.484
0.573
.608
0.721
CDSA(Joint)
0.451
0.497
0.317
0.336
0.404
0.520
0.558
0.677
CDSA 0.783
0.799
0.672
0.692
0.784
0.793
0.791
0.832
CDSA (Decomposed)
0.373
0.393
0.287
0.291
0.387
0.495
0.521
0.631
Table 8: Comparisons of different manners to implement CDSA on dataset METR-LA.
60 min
Mean
Model
\
Dataset
MAE
RMSE
MAPE
MAE
RMSE
MAPE
CDSA(Independent)
3.54
7.02
10.29%
3.25
6.29
8.81%
CDSA(Joint)
3.63
7.62
10.54%
3.30
6.83
9.43%
CDSA
3.92
7.93
11.17%
3.53
7.33
10.26%
CDSA (Decomopo))))))
3.40
6.27
9.76%
3.16
5.48
8.50
Attention Map Visualization: Fig. shows PM10 imputation example in location fangshan at tg. Since pattern PM2.5 around l2 similar that at (, the attention orange box is high. As see that PM2.5 and PM10 are strongly correlated, in order impute PM10 at fz, model utilizes PM10 at t (green arrow) and PM2.5 at t (blue arrow), which crosses dimensions. More visualization examples can be found in Supp.
PM2.5
PM10
27
PM255
PM2.5
41
Measurements
55
27
PM10
41
100
55
10
20
t 30
40
50
t
27
Timeslots (hour)
(a) Attention Map
(b) Air Quality Time Series Data
Figure 4: Visualization of the cross-dimennional self-attention on KDD-2018. (a) Part of Time- Measurement attention map. (b) Two time series of PM2.5 and PM10. The value purple dot is missing and our model predicts its value based on other values. The arrow in (b) represents attention whose score is highlighted with bounding box in (a) of the same color.
CONCLUSION
In this paper, we have proposed a cross-dimensional self-attention mechanism to impute the missing values in multivariate, geo-tagged time series data. We have proposed and investigated three methods to model the cross-dimen dimeensiol self-attention. Experiments show that our proposed model achieves superior results to the state-of-the of-the-art methods on both imputation and forecasting tasks. Given the encouraging results, in the future we plan to extend our CDSA mechanism from multivariate, geo-tagged time series the input that has higher dimension and involves multiple data modalities.
8

---

Under review as a conference paper at ICLR 2020
BIBLIOGRAPHY
Edgar Acuna and Caroline Rodriguez. The treatment of missing values and its effect on classifier accuracy. In Classification, clustering, and data mining applications, pp. 639-64996777 -47. Springer, 2004.
Hirotugu Akaike. Fitting autoregressive models for prediction. Annals of the institute of Statistical Mathematics, 21(1):243- 1969.
Craig F Ansley and Robert Kohn. On the estimation of arima models with missing values. In Time series analysis of irregularly observed data, pp. 9-37. Springer, 1984.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems, pp. 1171-11799 2015.
van Buuren and Karin Groothuis-Oudshoom mice: Multivariate imputation by chained equations in Journal of statistical software, pp. 1-68, 2010.
Fian-Feng Cai, Emmanuel Candès, and Zuowei Shen, singular value thresholding algorithm for matrix completion. SIAM Journal Optimization, 20(4):1556-1982 2010.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent
imputation for time series. In Advances in Neural Information Processing Systems, pp. 6775-6785, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprini arXiv:1409, 1211914999.2.,,,,,. 2014a.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406. 1078, 2014b.
Fan RK Chung and Fan Chung Graham. Spectral graph theory. American Mathematical Soc., 1997.
KDD Cup. Available on:. http:/ /wwp://wwwwwwoo kdd. wttp:kkdd..rg//dd201111 2018.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning, volume 1. Springer series in statistics New York, 2001.
E Haddadi, M R. Shabghard, M M. Ettefagh, A Bhattacharyya, G Boothroyd, M Castejon, E Alegre, Barreiro, LK Hernandez, SK Choudhury, et al. Modern spectral estimation: Theory and application. Journal of Applied Sciences, 8(21):pp-438, 1998.
Trevor Hastie, Robert Tibshirani, Gavin Sherlock, Michael Eisen, Patrick Brown, and David Botstein. Imputing missing data for gene expression arrays. 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8): 1735-1780, 1997.
Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 2(5):359-366, 1989.
HV Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M Patel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Communications of the ACM, 57(7):86-94, 2014.

---

Under review as a conference paper at ICLR 2020
Shuiwang Ji and Jieping Ye. An accelerated gradient method for trace minimization. In Proceedings of the 26th anmual international conference on machine learning, pp. 457-464. ACM, 2009.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint orXiv:1412.6980 2014.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500, 2016.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference Learning Representations (ICLR 2018.
2018.
Yonghong Luo. Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate time series imputation with generative adversarial networks. In Advances Neural Information Processing Systems, pp- 1607, 2018a. 8a.
Zhiming Luo, Frédéric Cranchaud-Charonn Carl Lemaire, Janusz Konrad, Shaozi Li, Akshaya Mishra, Andrew Achkar, Justin Eichel, and Pierre-Marc Jodoin. Mio-ted: A new benchmark dataset for vehicle classification and localization. IEEE Transactions on Image Processing, 27(10):
5129-5141, 2018b.
Shiqian Ma, Donald Goldfarb, and Lifeng Chen. Fixed point and bregman iterative methods for matrix rank minimization. Mathematical Programming, 128(1-2):321-35) 2011.
Fulufhelo Nelwamondo, Shakir Mohamed, and Tshilidzi Marwala. Missing data: A comparison of neural network and expectation maximization techniques. Current Science, pp. 1514-1521, 2007.
Sophoeles J Orfanidis. Optimum signal processing: an introduction. Macmillan publishing company, 1988.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 2015.
Louis L Scharf and Cédric Demeure. Statistical signal processing: detection, estimation, and time series analysis, volume 63. Addison Wesley Reading, MA, 1991.
Ling Shi. Kalman filtering over graphs: Theory and applications. IEEE transactions on automatic control, 54(9):2230-2234, 2009.
David Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst.
The emerging field of signal processing graphs: Extending high- high-dimensional data analysis to networks and other irregular domains. arXiv preprint arXiv:1211.005: 2012.
Robert H Shumway and David S Stoffer. An approach to time series smoothing and forecasting using the algorithm. Journal of time series analysis, 3(4):253-264, 1982.
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger G Mark. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology, pp. 245-248. IEEE, 2012.
Terry Speed. Statistical analysis of gene expression microarray data. Chapman and Hall/CRC, 2003.
Michael L Stein. Interpolation spatial data: some theory for kriging. Springer Science & Business Media, 2012.
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David Botstein, and Russ B Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 1766)520--225, 2001.
10

---

Under review as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1700:00933003 2017.
Menglin Wang, Baisheng Lai, Zhongming Jin, Xiaojin Gong, Jianqiang Huang, and Xiansheng Hua. Dynamic spatio-temporal graph-based cnns for traffic prediction. arXiv preprint arXiv:1812.02019, 2018.
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 21 29, 2016.
Xiuwen Yi, Yu Zheng, Junbo Zhang, and Tianrui Li. St-mvl: filling missing values in geo-sensory time series data. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, June 2016.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit $ Dhillon. Temporal regularized matrix factorization for high-dimensional time series prediction. In Advances in neural information processing systems, pp. 847-855, 2016.
G Peter Zhang. Time series forecasting using a hybrid arima and neural network model. Neurocom- puling, 10:159-175, 2003.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv: 08338, 2018a.
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung. Gaan: Gated atten- tion networks for learning on large and spatiotemporal graphs. arXiv preprint arXiv:1803.0700 07.09444444 2018b.
Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2267-2276. ACM, 2015.
Jingguang Zhou and Zili Huang. Recover missing sensor data with iterative imputing network. In Workshops at the Seirdy- AAAI Conference on Artificial Intelligence, 2018.

---

Under review as a conference paper at ICLR 2020
A MODEL ARCHITECTURE
A. NORMALIZATION LAYER
Under the NLP scenario, each word is embedded vector and normalized individually. However, in the Cross-Dimensional scenario. the normalization applied on each individual unit will always lead to zero-output. As shown Fig. [ different measurements may exhibit different correlation, i.e. PM2.5 and PM 10 are significantly positively correlated (pPM2.5, PM oо 0.8278) while NO2 and O3 are negatively correlated (pN0,.0, = -0.5117). As discussed in Fig. 4 in the paper, different measurement may be used as reference for imputation of other measurements. As such, the normalization cross multiple measurements is unreasonable and we choose to apply normalization for each measurement in parallel which presumes that the time series inside the spatial network is essentially drawn from a standard normal distribution. We subsequently add the trainable scalar and bias to scale the normalized value and the scalars (biases) for different measurements trained individually.
600
PM10
PM2.5
450
150
160
03
120
40
100
Timeslot (Hour)
200
300
Figure 5: Different correlation between different measurements. The time slots when both of the chosen measurements available selected and the value of the first 300 selected time slots plotted. Upper: PM2.5 & PMIO are significantly positively correlated and Lower: NO, Og are negatively correlated.
A.2 UNIT- WISE
NETWORK
Making use of the approximation property of multi-linear ceed-fowward network (FFNN is applied each unit separately and identically. This FFN consists of three fully connected layer while RuLU is set as the activation function.
et al. (1999. fully connected
(FN = max (max (0, xW1 + b2Wa + b,) + by
(6)
During experiment, since the FFN applied on each unit individually, we found the improvement by simply increasing the size of weight and bias of each layer is not obvious while increasing the depth of FFN will lead to obvious improvement.
12

---

Under review as a conference paper at ICLR 2020
A.3
IMPUTATION MASK
For the self-attention sub-layer in imputation task NYC and KDD-2018, we modify the attention map with mask in (7 to prevent unit of available observation from contributing the estimation of itself.
1-2 lq.K/Va otherwise
i=j
((.
where qi and kn are d-dim vectors. This masking, combined with fact that there is no offset between input and output, ensures that the estimation of unit X (p(t,,, m,) depends on all the units except for itself, including both available and complemented units. In this way, the gradient back-pre preck--pppgationn be used to update the missed value effectively. The Table Table|9showe shows the performance improvement of imputation mask applied our model and demonstrate that mask prevents the estimation of itself and improve the inference ability of the model.
Table 9: Performance Improvement for Imputation Mask
Model
Missing Rate
20%
30%
40%
50%
60%
70%
80%
90%
Masked
0.373
0.393
0.287
0.291
0.387
0.495
0.521
0.631
KDD-2018
No Mask
0.886
0.839
0.838
0.849
0.917
0.903
0.947
1.195
Masked
1.04
1.10
1.32
1.18
1.18
1.28
1.29
1.75
NYC
No Mask
1.69
1.70
1.72
1.77
1.80
1.84
1.91
2.04
A.4 ATTENTION MAP CALCULATION
Time
Time
X(po)
((Po:) R²
SPoo P.)
Location
Location (L)
K(p.:) Rd/
Dot
X(pi) Scaled numerical product
Measurement (M)
product
Measurement (M)
Figure 6: The effective attention map calculation in Joint and Shared.
Joint As shown in Fig. 6 when we build the attention among different units in the Joint, two different kernels will be used to map each input unit ((p) = (, to an 1-D Query vector and an 1-D Key vector individually. As attention map a scaled dot-product between Query and Key (Fig. 6 Left) after Softmax, each value of attention map in Joint is essentially the scaled numerical multiplication between each two units of input (Fig. (Fig.6) Right after Softmax. As such, the multiple parameters inside that kernels only perform Joint is limited.
single scalar and the learning ability/relationshipp representation in
Decomposed According to Sec. 3.2.3 to calculate the dimension-specife. attention map, e.g., the at- tention map Time Ar, the input X will be reshaped into matrix Xy. Thus, the units corresponding to the same timestamp, reshaped into one vector (t,:)), will be mapped into dimension-speciiie Query vector Qr(t,:) and Key vector (t,t,,) As more parameters will be introduced into such vector-vector mapping, each dimensional-specte dimensional attention map can model the relationship among each input units effectively.
map can learn the intra-correlation and the cross-
Shared Same with the Decomposed, the Shared will calculate the dimension-specific attention maps individually. Like Joint, the Shared will map each unit X (p) into vector. To calculate the dimension- specific attention map (c.g., Ar), the evector-vector mapping" is essentially the summation of the units corresponding to the same timestamp while the multiple parameters introduced in the mapping still perform as a single scalar. As result, the learning ability of the intra-correlation limited so that the cross-dimensional attention map cannot model the relationship among input units effectively.
13

---

Under review as a conference paper at ICLR 2020
B
ATTENTION MAP RESHAPE IN THE Decomposed
42
1
1
mi
m
22
22
1
my
m
(a) Attention Map of Time, Location and Measurement
22
11
12
m
m
41
42
11
12
m
m
2
1
22
m
m7
42
121
1,2
m
m2
1
1/2
m m2
Er2
1
122
m,
m
t2
121
1/2
m
m
121
12
1
122
m m.
(b) Time
(c) Location
(d) Measurement
Figure 7: Reshape Attention Map: The original attention map (a) is reshaped into (b-d) respec-
tively. The variable (t, m) denotes the attention map units for Time, Location and Measurement where the subseript labels the attention between two units, i.e., t12 Softmax (1 k_ /Vd) where Qi and l2 are d-dim vectors. Besides, the empty entry indicates 0.
As described in (2) in the paper, the original attention maps Ar, A,, and Am are reshaped to A as an example, we draw the attention maps before and after reshape in Fig. 7. Making use of the matrix structure, have
A' and A₁, By setting T
Σ A(po.Pi)
ΣΣ
Σ
Ar (r) Am mi)
(8)
T
M
Σ
Ar(to,ti)
Σ
Σ ml
A Am m.)
[=11
where Σ Ar(lo,4) 41) En= A (l,, 41) and EM Emy=1 Av (mo, mi) 1. Besides, the
M
reshape operation for attention map on different dimension is only determined by the index mapping function, p p(t,l,m) LML MI m, from the 3-dimensional cube the vector form.
14

---

Under review as a conference paper at ICLR 2020
C CDSA FRAMEWORK FOR DATA FORECASTING
Different from time series imputation task, the time series forecasting task use the current observation estimate the time series in the future. To begin with, We first introduce the framework for time series forecasting and we compare the performance for two different types of input.
C.I PREDICTION FRAMEWORK
As shown in Fig. we apply our CDSA mechanism Transformer framework and use the same Feed Forward structure in Sec. 3.3 in the paper. Notably, we set N 9 for both encoder and decoder and no CDSA module is used to derive a complement input for prediction task where the missing value is replaced with global mean. The architecture detail is shown in Fig.
Transformer Encoder w/ CDSA
Transformer Decoder w/ CDSA
Feed
T+1
1U
Forward
(L)
(M)
(M)
(M)
Figure The framework using Crossing-Dimensionn Self-Attention (CDSA) data forecasting.
Imputation
Prediction
2T
Outputs
T+1
Feed
(L)
Forward
(L)
Outputs
Multi- Head-
(M)
(M)
CMSA
Feed Forward
Multi- Head
Add Norm
(L)
CMSA
(L)
Feed
Forward
(M)
(M)
Add Norm
Add & Norm
Add Norm
Multi- CMSA (Mask L&M)
Feed
Feed
Forward
Forward
xN
Nx
Nx
Add Norm
Add Norm
Add & Norm
Multi-Head CMSA
Multi- Head CMSA
Multi- Multi-Head CMSA (Mask TLM)
Missing Value
Completed Value
Available Observation
Inputs
Inputs
Figure 9: Model Architecture
15

---

Under review as a conference paper at ICLR 2020
C.2 DISCUSSION FOR INPUT FORM
The decoder in NLP task originally sets the shifted oulput as input. Take the German-to-English translation scenario an example where the embedded word vectors of German Encoder input, the model will first send a [GO] vector into the decoder and generate the first word vector of the translated English sequence, then the predicted vector will be sent into the decoder to predict the next word vector and the decoder will complete the sentence translation by repeating this operation until the end.
the
Mapping directly from this model setting in NLP task to our series forecasting scenario, we can also use the shifted ground truth the decoder input, i.e., to forecast the speed of the next T time stamps given the speed of the first T time stamps, the data of T S / I 27 1 are sent into the decoder. Consequently, the Casual Mask Vaswani et al. (0177 need be modified to make sure that leftward information flow prevented.
For data forecasting by CDSA in the Decomposed, the masking on Attention Map on Time is simply masking out (setting to -0o) the values in the input of Softmax which corresponds to the leftward information flow. Same with Vaswani et al. (2017). the masking is only adopted in the Multi-head CDSA layer labeled as (Mask TLM) in Fig. 9 However, shown in Fig. TO to calculate the Attention Map of Location and Measurement for data forecasting at 2T 1. all illegal units of input corresponding
2T 1 have be masked (setting 0). Then, the Masked Impul mapped to Query, Key and Value to build the Attention Map and calculate the Updated Value
2T-1
2T-1x
AttMap (MxM)
(L)
(L)
Mask Out
(M)
(M)
Input
Masked Input
AttMap (LxL)
Figure 10: Casual Mask Design for Attention Map Location
Besides, the decoder generates predictions given previous ground truth observations during training while the ground truth observations are replaced by predictions generated by the model itself during testing. As, the discrepancy between the input distributions of training and testing can cause degraded performance, We adopt the integrated sampling Bengio et al. (2015) as in Li et al. (008)) to mitigate this impact while this method very time-consuming for the Transformer framework. During testing,
Table 10: Comparisons of Prediction Performance on dataset METR-LA
Shifted Output
Shifted Output
Encoder Input (Complemented)
Encoder Input
Time
Metric
(Mean Final)
(Mean Step)
(Mean)
MAE
3.05
3,09
3.15
3.01
15 min
RMSE
5,46
5.50
5.31
5.08
MAPE
8.31%
8.47%
8.18%
7.82%
MAE
3.49
3.54
3.47
3.14
30 min
RMSE
6.62
6.66
5,96
5.38
MAPE
9.56%
9.88%
9.19%
8.30 %
MAE
4.21
4.26
4.17
3.40
60 min
RMSE
8.14
8.19
7.67
6.27
MAPE
11.44%
11.79%
11.94%
9.76 %
MAE
3.51
3.56
3.54
3.16
Mean
RMSE
6.56
6.61
6.14
5.48
MAPE
9.56%
9.84%
9.53%
8.50
In summary, by setting shifted output as the Decoder input, multiple Attention Map are calculated for forecasting value of different time stamps which requires huge memory usage. Still, integrated
16

---

Under review as a conference paper at ICLR 2020
sampling makes this framework suffer from exhausted training time, since need send the predicted output back to decoder (Run) and repeat this Run for T times. During testing, we can use the output corresponding
its own Run (Step) as the predicted result, well the output of the last run (Final.. As shown in the first 2 columns in Table TO the performance of outputs in the last run (Final) is better than that of Step mode, which means the leftward information flow still exists break the auto-regressive property in data forecasting even though the mask is adopted on the input data. For fair comparison, the models for testing are trained one GPU and the training time all less than 50 hours.
Typically, missing value still exists in the original dataset. During experiment, we use the global mean to replace the missing value (Mean). We also compare the prediction performance between the input with Mean Filling and Complemented Inpul of Sec. 3.3 and the results Table showswws the Complemented Input does not lead to performance improvement but increase the training workload. Consequently, we make encoder and decoder share the same input to reduce the memory usage and training time while our model achieves better performance for long-term prediction.
17

---

Under review as a conference paper at ICLR 2020
D DATASET DESCRIPTION
D.1 DATA AGGREGATION ON KDD 2018 DATASET
Table 11: Selection of Common Locations between Air Quality and Meteorology
Air Quality
Meteorology
fengtaihuayuan fangshan
fengtai meo
fangshan meo daxing_ tongzhou_me shunyi_meo pingchang meo
daxing
tongzhou
shunyi
6
pingchang
mentougou
mentougou meo
8
pinggu
pinggu meo huairou_meo
huairou
10
miyun
miyun_meo yanqing_mex
yanqin
The original KDD 2018 dataset consists an Air Quality data of 35 locations and an Meteorology dataset of 18 locations. each dataset contains 6 different measurements. During experiment, [Luo iet al. (2018a) select 11 common locations between the two datasets and the measurements of paired locations are concatenated. The location pairs are described Table II Since the unit of some measurements are label-based, e.g., the measurement weather denotes the types of weather including sunny, rainy and etc, these label are replaced with value such as 2, .... As the range of different measurements varies, Luo et al. (20 first apply Z-score normalization for each measurement and MSE is calculated based on normalized data while the metrics calculation for other dataset are based on the original data.
18

---

Under review as a conference paper at ICLR 2020
D.2 NYC TRAFFIC DATASET
Abnormal Loss
System Loss
Burst Loss
186
sensors
12/03/2015
Time Stamps (5 min)
12/26/2015
00:00
00:00
Figure 11: Loss condition of NYC dataset: The horizontal axis represents time line while the vertical axis represents sensors. Each unit indicates whether the data is missed in a 5-min window. The white area indicates available observation. The blue indicates the Burst Loss. The red area indicates the time slots when the data of all the sensors are missed. The green area denotes as Abnormal for certain location the data is continuously missed for very long period.
Traffic volume extraction: We extract the traffic volume from images using a Faster R-CNN with VGG16 backbone, trained on the MIO-TCD Luo et al.] (2018b) dataset. The dataset contains 110k training images, with bounding box annotation for vehicle categories (articulated truck, bicycle, bus, car, motorcycle, motorized vehicle, non-motorized vehicle, pedestrian, pick-up truck, single-uni unit truck, work van). On the NYC-traffic dataset, we manually annotated bounding boxes for portion of images to evaluate the vehicle detector. Our model achieves 73% precision and 54% recall with IoU of 0.5. To construct the NYC traffic time series, we use the model to extract and then sum up the #cars of 11 different types (articulated truck, bicycle, bus, car, motoreycle, motorized vehicle, non-motorized vehicle, pedestrian, pick-up truck, single-unit truck, work van), in non-overlapping 5-minute intervals.
Burst loss simulation: As shown in Fig. 11 we term the loss area marked as blue as the Burst loss area where for a certain camera, the data continuously missed for ô time slots. After statisties and analysis, we found the the length of time slots 2 < 8 < 134. Then, for those time slots of burst loss, we calculate the mean μ = 6.350773 and standard deviation
9.809643. With the mean and standard deviation, We model the generation of burst loss as Gaussian process.
19

---

Under review as a conference paper at ICLR 2020
E
DISCUSSION
E.1
VISUALIZATION OF DIMENSION-SPECIHI ATTENTION MAP
Wind dirrecion Wind speed Weather
Temperature
Pressure
Humidity
PM2.5
PM10
NO2
CO
03
S02
1
2
4
5
7
8
9
10
11
PM2.5
PM10
CO
03
5
6
Temperature
7
Pressure
8
Humidity
9
Wind direction
10
Wind Wind_speed
Weather
11
(a) Measure
(b) Location
Figure 12: Attention map example of the last CDSA layer
BEDHERSEDH
ACSET
10
MULAA
DMAN HizzAoN
B.B. ST
GINE
Hoy NANURA SCAING AVIN
SEKANDHWHH
SCErIO AECD
LOOK angfend
ScenIE Area
6
UALAN HALANHAANENN
MACHNOSADIR
PUROHE
SEWNCHEN
AOOU TOOO
SEOUAATION
SUI
BONI
APRIEI
MMANSODHNN
SHEN
DETTICT
KANGSHAN
DEMGAEOOIOO CHCOAON
OAO DONEN
YAMUMODHO
XOX SHODEN
SHLING
Beijing AbAT
TIZ
SOP
TONDHIDEDEN
TEOT
MAOHNNMMEEEUA SAANS
12
LECHE CAGUNN
VUNDUE
DUTRECT
CURTUEDDEN
Figure 13: KDD-2015 Visualization of Location Correlation (arrow with darker color indicates higher weight)
20

---

Under review as a conference paper at ICLR 2020
We provide the attention map examples extracted from the last CDSA layer. As shown in Fig. 4(b) correlation exists between different measurement, i.c., PM2.5 and PMI0 are highly correlated. As shown in Fig. [2(a] the estimation of PM2.5 and PM10 is also highly relied on each other. i..., for the estimation of PM2.5, the color in second unit, representing the weight of PM10, is darker than the rest in the first
As shown in Fig. 12(b) and Fig. [3 the arrow/unit with deeper color indicates a higher weight and the index of location can be found in Table According to the map in Fig. [3 in most cases, neighbouring locations often share higher attention weight, e.g., the estimation at location is mainly relied on the available data from location 2, location and location location 11 is not relied on its neighbor (location 6), instead, it is mainly relied on the location 8. We think this relation is induced since both location and location are the center of express way while they are away from the urban area. Thus, the air condition from those two location my highly correlated.
However, the estimation of
E.2 VISUALIZATION OF ATTENTION MAP FOR CROSS-DIMENSIONAL IMPUTATION
150
PM
the width of orange arrow indicates the attention value (wider->stronger)
PM2.5
Measurements
100
A
B
50
D
0
10 5
20
30
Timeslots (hour)
Figure 14: Visualization of prediction missing point A, our model not only attends to available points (e.g. C, D) but also attends to missing points (e.g. B).
Besides the sample in Fig. where the missing value be estimated from the cross- cross-dimensional available data, Fig. 14 visualizes another example and further shows that when predicting missing value A, model pays strong attention to available values C and D while also some attention to another missing value B.
21

---

Under review as a conference paper at ICLR 2020
E.3
RUNNING TIME COMPARISON
Table 12: Computational complexity of several methods to implement CDSA Independent
Methods
Joint 859
Shared
Decomposed
Average running time sample (ms)
218
159
161
Following the model hyper-parameted setting in Tabls. we further compare the average running time for segment during testing. As the way of attention modeling determines the computational efficiency, computation method with higher FL.OPs also leads to longer running time. As shown in Table. 12 the running time of Joint is much higher than the rest methods. Since the computation schemes of Shared is similar with the Decomposed, while the number of trainable variables of Shared much less that of Decomposed, the average processing time of Shared bit smaller than the running time of the Decomposed.
E.4 DETAIL OF FORECASTING BY USING THE IMPUTED TIME SERIES
30%
2.5
1.796
1,069
1.597
1.602
1.5
1.451
0.5
Mean Filling
Kriging expo
Kriging linear
MTSI
DCRNN
BRITS
CDSA
(a) ARIMA
30%
50%
70%
2.5
1.858
1.852
1.5
1.457
1400
0.5
Mean Filling
Kriging
Kriging linear
MTSI
DCRNN
BRITS
CDSA
(b) Random Forest
Figure 15: RMSE comparison for Downstream Forecasting on NYC-Traffic
As described in the main paper, we use the 23-day data of NYC-Traffic for further forecasting. split the data into two segments, one segment contained the data of the first 20 days and the other contained the data of the rest 3 days. We used the imputed data from the first segment to forecast the value the second segment. To provide comprehensive comparison, according to different missing ratio (i.c., 30%, 50% 70%), remove the value of some units in the first segment according to burst loss and then feed the segment with missing value into the data statistical imputation model (i.e., Mean filling, Kriging expo, Kriging linear) and deep learning methods (MTSI, DCRNN, BRITS, CDSA(ours)). Then, we feed the imputed data in to prediction model (ARIMA, Random Forest Friedman et al. (2001)) and evaluate the forecasting performance in of RMSE. According to the [5
we can find our proposed model always outperforms than other method.
22

---

Under review as a conference paper at ICLR 2020
F EXPERIMENT SETTING
F.1 BASELINES
We compare our method with both deep learning based methods and statistical methods while all statistical methods are adopted for each measurement individually.
Mean Filling: Replace the missing data with global mean.
Auto Regressive Akaike (969); Orfanidis (1988); Haddadi et al. (1998): Aggregate both forward and backward auto regressive each time series data by weighted average and replace the missed value, implemented in MATLAB.
Kriging: Fit linear (Exponential) function between data variance and geodesic distance and replace the missing value w.r.t. the geo-location. Applied on each time slot and implemented in PrKrige
Aulti-Imputatinn by Chained Equation (MICE) Buuren Grosthuis-Oudshooon 2010):
Replace missing data by creating multiple imputations with chained equations.
k-nearest neighbor (KNN) Speed (2003): Hastie et al. | Troyanskaaa et al. (2001): )
Use normalized Euclidean distance to find similar samples and impute the missing values
with weighted average of its neighbors.
Matrix Factorization (MF) Yu et al. (2016): Factorizes the incomplete matrix into two
low-rank matrices and fill the missing values by sparsity and l2 penalty.
MTSI et al. (2) Treat each measurement at different location as variables identi-
cally, align the time series all the variables into one matrix and impute the missed data through RNN-based GAN.
ST-MVL Yi et al.
20166: Replace the missed value by using geo-location information
(2018) Treat the missing data as variable of the bidirectional RNN and impute by getting the delayed gradients for missing values in both forward and backward directions.
BRITS Cao et al.
Connect three bi-directional RNN in turn while the output of each RNN are adopted in loss calculation.
IIN Zhou & Huang 2018):
FC-LSTM Sutskever et al. (2014): Apply fully connected LSTM hidden units in Encoder-
Decoder RNN structure model the temporal dependency and forecast the traffic speed.
DCRNN Li al. (2018): Model traffic speed signals diffused over bidirectional graph and modify the gated recurrent unit in encoder-decoder RNN structure to capture the spatiotemporal dependency.
DST-GCNN Wang et al. (2018): Build two-stream framework for model the condition stream and flow evolution while the model use the predicted condition to forecast the flow.
GaAN Zhang et al. 2018b): Use attention mechanism for each location and then graph aggregator to assemble the neighbor nodes and impute the missing value.
23

---

Under review as a conference paper at ICLR 2020
F.2
IMPLEMENTATION DETAILS
Table 13: Statistics of Dataset
Dataset
Task
# location
# meas.
duration
Start
End
KDD-2018
Imputation
36
12
364 days
1/1/2017
12/30/2017
NYC Traffic
Imputation
186
I
23 days
12/03/2015
12/26/2015
METR-LA
Forecasting
207
122 days
3/1/2012
6/30/2012
Dataset
# #Training
#Testing
#Validation
unit duration
seg. length
Loss function
KDD-2018
182
182
2 hours
48 hours
RMSE
NYC Traffic
173
173
5 minutes
36 hours
RMSE
METR-LA
23991
3404
6831
5 minutes
hour
RMSE
The dataset in METR-LA also has missing data while the missing rate is 91%. Thus, the segment sample whose all units are zero, i.e., all-zero sample. exists. During training, the all-zero sample (in
training set) essentially has no contribution for the model training. During testing and validation, the evaluation metric will of such samples will not be counted.
Data Pre-processing We apply Z-score normalization on each measurement as respectively and fill the missing value with
(t, X(1,1) -M
DT ΣL Zi-1 X(t,))
11
TL
EJ El_ (X(t.l) pb)
TL
Optimizer We the Adam optimizer Kingma & Ba (0014) while the initial learning rate in each epoch is set as
(r(e)= = ro x c0i((ma((0,e-d)//
(10)
24

---

Under review as a conference paper at ICLR 2020
F.3 KDD-2015
For KDD-2018, [uo et al. (2018a) adopts content loss in a GAN-based model to train the random noise and then estimate the missing value, i.e., for data segment, according the specified missing rate, some available data will be held to evaluate the imputation performance, while the remaining available data will be as groundtruth to calculate the content loss. Our experiment on NYC-Traffic follows the same experiment setup as KDD-2018 while the noise is replaced with the remaining available data and the model parameter is trained according to the content loss. Thus, there is no division of training, validation or testing since the training loss is not calculated from the held available data.
To comprehensively develop our experiment, we also adopting our method on KDD-2015 and follow experiment setup Yi al. (2016); Cao et al. (2018) while the available data will be trained to predict the held data directly.
KDD-2015 (Zheng et al. (2015). This dataset focuses on air quality and meteorology. contains data recorded hourly, ending up with totally 8,759 time stamps. PM2.5 measurement is recorded at 36 locations and Temperature and Humidity are recorded at 16 locations Beijing from 05/01/2014 to 04/30/2015, with natural missing rate 13.3%, 21.1% and 28.9% respectively. We treat those two subsets as two separate tasks and evaluate our method on each task separately. Following Yi et al. (2016), data in 3rd, 6th 9 and 12th months are for testing and the remaining months for training. We randomly select 36 consecutive time slots to train our model and evaluate Mean Absolute Error (MAE) as well Mean Relative Error (MRE).
In order to simulate experiments for imputation, besides the natural missing data, for PM2.5 we follow the strategy used in (Yi et al. (2016); Cao al. 20188; Zhou & Huang (2018) further manually remove entries and hold the corresponding value as ground truth. The imputation task is to predict values of these manually removed entries. For Temperature and Humidity, we follow Zhou Huang (2018) to randomly hold 20% of available data.
KDD-2015. Table 14 shows that for PM2.5, our method outperforms the traditional methods significantly and achieves comparable MAE as IIN (Zhou & Huang (2018) while better MRE than INN (Zhou & Huang (2018)). For Temperature and Humidity, our method consistently outperforms state-of-the-art methods.
Table 14: MAE/MRE on dataset KDD-2015 for comparisons with SOTA
PM2.5
TEMP
HUM
Model \ Dataset
MAE
MRE
MAE
MRE
MAE
MRE
Mean KNN
55.51
77.97%
9.21
97.56%
20.34
57.85%
29.79
41.85%
1.26
19.83%
7.28
16.22%
MICE
27.42
38.52%
1.23
18.29%
6.97
15.87%
ST-MVL Yi et al 2016
12.12
17.40%
0.68
4.59%
3.37
5.91%
MTSI Luo al. 2018a
13.34
18.01%
0.71
4.67%
3.51
6.21%
BRITS C ao et al. 2018
11.56
16.65%
0.63
4.16%
3.31
5.68%
DCRNN et al. Zhou & Huang
(2018)
12.33
17.82%
0.69
4.59%
2.95
5.12%
IIN
2018
10.63
15.31%
0.63
4.22%
2.90
5.09%
CDSA (ours)
10.67
14.89%
0.61
4.15%
2.81
4.92%
Table 15: Comparisons of different manners implement CDSA on dataset KDD-2015.
PM2.5
TEMP
HUM
Model \ Dataset
MAE
MRE
MAE
MRE
MAE
MRE
(DSA((ndependeet)
11.54
16.01%
0.68
4.40%
3.19
5.42%
CDS CDSA())
11.20
15.52%
0.65
4.27%
3.05
5.37%
CDDSA(Shared)
13.85
19.26%
0.75
5.18%
3.56
6.47%
CDSA(Decomposed)
10.67
14.89%
0.61
4.15%
2.81
4.92%
Since the Decomposed draws attention maps the Independent but shares Value the Joint, it reduces the computational complexity significantly. As shown in Table we also evaluate these methods KDD-2015 datasets and the Decomposed achieves the best performance.
25

---

Under review as a conference paper at ICLR 2020
F.4 METRICS
Suppose a [ii, T2,...NN E RV represents the ground truth and æ [ 22, E R.
represents the predicted value.
Root Mean Square Error (RMSE)
RMSE( (2, .b)
F ),,)²
Mean Squared Error (MSE)
MSE(x, ()
(nn xn
Mean Absolute Percentage Error (MAPE)
N
MAPE(2:. xz, (I,
Σ N
1
In In
Fn
Mean Absolute Error (MAE)
1
MAB(a, .)
In fx
Mean Relative Error (MRE)
MRE(x, 2 10 1pal
NN 2n= In
F,5 MODEL
Since there are missing data (Naturally missing data) in the original dataset, to evaluate the model performance, we manually remove some of the available observation (Manually removed data) and hold those entries' value ground truth for evaluation. The rest data are termed as Available data. Thus, as counterpart of Naturally missing data in the original dataset, the Naturally available data consists of Manually removed data and Available data.
Following the setting in Yi et al. (2016), we split the data into training set and testing set. During training, send the Available data into the model to estimate the missing data while the loss calculated based on the Manually removed data. During testing, we send the Available data into the model and the metric is calculated based the Manually removed data. For model structure Air Quality, we set (d,,, dr, v) as (6, 12, 3) and 12 heads each layer. We set (ro, a. d, batch size) (0.003, 0.2, 60, 23) for at most 100 epoch. The model hyper-parameter Meteorology is same with that for Air quality except for da = 15.
KDD 2015:
KDD-2018: Following the setting in Luo et al. (2018a), don't 't split the data for testing or validation while the imputation task assumes the completed data has no label. Thus, we send the Available data into the model while the loss is calculated based on Available data while the evaluation metrie is calculated based on the Manually removed data. For the model structure, we set (dy. dz, d.s. v) as (30, 6, 14,3 3) and 12 heads in each layer. We set (ro. a, d, batch size) as (0.003, 0.2, 60, 20) for at most 100 epoch.
NYC: Like KDD-2018, the imputation task assumes the completed data has no label. For the estimated data, the loss is calculated on the Available data part while the evaluation metric is calculated on the Manually removed data part.
Following the setting in Li et al. (2018), split the data into training set and testing set where this prediction task assumes the predicted data has labels. For model structure, we set (dr, dr, v) as (14, 6, 3) and there are 16 heads in each layer. During training, we set (ro- a, d, batch size) as (0.008, 0.5, 40, 16).
METR-LA:
26


Reviews:
[{"review_id": "rkx9YbGcYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The classic transformer network is designed to capture the dependencies along one dimension. Applying the transformer to spatiotemporal data creates the challenge of modeling the attention in a computationally efficient way (time, location, variable). This paper investigates different ways of implementing 3D attention and extensively evaluates the performance of them. \n\nThe proposed problem is a practical problem and I think the main contribution of this paper is valuable. However, there is always a trade-off between having access to the entire history and the ability to be used in streaming settings. The need for the processing of the entire history of time series increases the latency of the algorithm. Also, the authors should report the actual run-time of the algorithms on the real data (beyond Table 1 and compared to the baselines).\n\nThe main criticism of the experiments is that the datasets are very small. For example, NYC-Traffic has only 186 time series which is considered to be of the toy-scale. Also, if you have run the experiments and not copied the reported data, make sure to indicate that in the paper. Citing a paper in a table usually means the numbers are reported from the paper.\n\nThe ways that the authors decompose the attention tensor reminds me of the works in different tensor decomposition algorithms in spatiotemporal data [1, 2].\n\nThere are minor typos in the paper too. For example, see the first in-line equation in Section 3.1.\n\n[1] Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455-500.\n[2] Bahadori, M. T., Yu, Q. R., & Liu, Y. (2014). Fast multivariate spatiotemporal analysis via low-rank tensor learning. In NeurIPS.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1loIGITtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically studies the effectiveness of transformer models for time series data imputation.  In particular, the paper studies the effect of generalized forms of self-attention that can attend across dimensions of the input.\n\nGeneralizing self attention to work across dimensions of a multi dimensional time series is a good idea, and the experiments in the paper seem to support its effectiveness.  The paper does provide some ablation results to compare their three forms of modified self attention, which is good.  I believe the primary contribution of the paper is as an empirical study into the effectiveness of generalized self attention in time series datasets.\n\nHowever, I have to vote to reject the paper.\n\nMy primary issue with the paper is its tone.  While generalized forms of self attention are a good idea, the paper strongly emphasizes that it is a novel idea.  In particular, image models that use self attention regularly attend too multiple dimensions.  Consider for instance arXiv:1802.05751, arXiv:1904.09793, arXiv:1712.09763, arXiv:1805.08318.\n\nThere is also several existing generalized self attention discussions: arXiv:1812.01243 or arXiv:1805.00912\n\nThe idea of extending self attention to look at multiple dimensions is fairly obvious.  If the paper changed its tone from purporting to construct a novel method of self-attention (which I do not believe it does), to being an empirical study of the utility of self attention models for doing time series imputation I would be much more willing to accept it, though as a purely empirical study the bar would be high on the standard of the experiments, the reported experiments are on rather small datasets.\n\nAlso, the paper could use an edit for grammar.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "B1lAfWvh9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Transformer-based model with cross-dimensional self-attention for multivariate time series imputation and forecasting. The authors consider time series with observations collected at different locations (L), for different measurements (M), and at different timestamps (T). The authors describe 4 different self-attention mechanisms based on how the three dimensions, and it turns out the proposed Decomposed approach achieves the best performance and has moderate model complexity among the four. Experiments on several traffic and air quality datasets show the superiority of the proposed model.\n\nOverall the problem and the proposed model are well motivated. Handling time series data with missing values is quite important, and the authors design the novel cross-dimensional attention mechanism which is reasonable and performs well. Especially, the authors compare several recent RNN-based models. The proposed method outperforms these strong baselines in imputation and long-term forecasting tasks.\n\nHowever, I do have a few questions and concerns.\n\nIt would be quite helpful to see the training time and model size comparisons with baselines, which is to validate the claim in the introduction that `replacing the conventional RNN-based models to speed up training`.\n\nThe proposed model treats all three dimensions equally, with direct attention of every two variables, and is independent with the order. Though effective and mathematically clean, I am not sure the temporal/spatial smoothness and dependencies in time series are properly modeled in this way -- as time series is not the same as embedded sequences in NLP. This may explain why the performance on short-term forecasting is relatively unsatisfying.\n\nIt seems that the proposed model is designed for missing completely at random (implied by the statement `... due to unexpected sensor damages or communication errors` from the introduction, and the experimental settings on adding missing values). Many missing variables in time series may be missing at random or even not at random.\n\nAbout the experiments:\nTwo datasets of the three mentioned in the main paper have M=1, which degrade the proposed model from 3-dimensional to 2-dimensional.\nWhy forecasting experiment is conducted on METR-LA, while using the imputation for forecasting is conducted on a different dataset and without comparing other forecasting baselines?\nWhat are the metrics used in Tables 6, 7, 9? Several metrics are used (RMSE, MSE, MAPE, MAE, MRE) while results on different datasets are shown in different but not all metrics. Is there any reason to cherry-pick metrics for different experiments?\nIn Table 4, the proposed method's results in RMSE is consistently better than shown in MAE compared with other baselines. Any explanations would be useful.\n\nThe overall idea is relatively easy to follow, while some detailed descriptions should be added or clarified.\nWhen taking health-care data as an example of geo-tagged time series, could you explain or provide references?\nFigure 2 only demonstrates 3 attention mechanisms, and the Shared should also be included.\nS(i,j) is used in Section 3.1 without explicit formal definition.\nPlease clarify the sentence about \\sigma below Equation (4).\nPlease refer to the section number explicitly in Supp if used. (E.g., on Pages 5 and 7.)\nOn which dataset and what settings are the results in Table 1 computed? The numbers are helpful, but it would be better if the results are computed based on the hyperparameters (e.g., T, L, M, d_V, etc).\n\nMinor typos:\nPage 3, Paragraph of RNN-based data imputation methods: `...indistinguishable. so that...`\nPage 3, Section 3.1, `..where Then, ...`\nPage 14, A' is used to denote the reshaped tensor, while \\tilde is used in the main paper.\nPage 16, `During testing,`", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a solution based on self-attention RNN to addressing the missing value in spatiotemporal data. 

I myself read through the paper, followed by a discussion with the reviewers. We agree that the model is reasonable, and the results are promising. However, there is still some room for improvement:
1. The self-attention mechanism is not new. The specific way proposed in the paper is an interesting tweak of existing models, but not brand new per se. Most importantly, it is unclear if the proposed way is the optimal one and where the performance improvement comes from. As the reviewer suggested, more thorough empirical analysis should be performed for deeper insights of the model. 

2. The datasets were adopted from existing work, but most of them do not have such complex models as the one proposed in the paper. Therefore, the suggestion for bigger datasets is valid. 

Given the considerations above, we agree that while the paper has a lot of good materials, the current version is not ready yet. Addressing the issues above could lead to a good publication in the future. 
Decision: Reject