Granger Causal Structure Reconstruction from Heterogeneous Multivariate Time Series
Authors: Yunfei Chu
Xiaowei Wang
Chunyan Feng
Jianxin Ma
Jingren Zhou
Hongxia Yang
Venue:  (2020)
Keywords: causal inference
Granger causality
time series
inductive
LSTM
attention

Abstract: Granger causal structure reconstruction is an emerging topic that can uncover causal relationship behind multivariate time series data. In many real-world systems, it is common to encounter a large amount of multivariate time series data collected from heterogeneous individuals with sharing commonalities, however there are ongoing concerns regarding its applicability in such large scale complex scenarios, presenting both challenges and opportunities for Granger causal reconstruction. To bridge this gap,  we propose a Granger cAusal StructurE Reconstruction (GASER) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series. In particular, we address the problem through a novel attention mechanism, called prototypical Granger causal attention. Extensive experiments, as well as an online A/B test on an E-commercial advertising platform, demonstrate the superior performances of GASER.

Under review as a conference paper at ICLR 2020
GRANGER CAUSAL STRUCTURE RECONSTRUCTION FROM HETEROGENEOUS MULTIVARIATE TIME SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Granger causal structure reconstruction an emerging topic that can uncover causal relationship behind multivariate time series data. In many real-world sys- tems, it is common to encounter large amount of multivariate time series data collected from hetcrogeneous individuals with sharing commonalities, however there are ongoing concerns regarding its applicability plex scenarios, presenting both challenges and opportunities for Granger causal reconstruction. To bridge this gap, we propose a Granger cAusal StructurE Reconstruction (GASER) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series. particular, we address the problem through novel attention mechanism, called prototypical Granger causal attention. Extensive experiments, as well as an on- line A/B test on an E-commercial advertising platform, demonstrate the superior performances of GASER.
such large scale com-
INTRODUCTION
Broadly, machine learning tasks are either predictive descriptive in nature, often addressed by black-box methods (Guo et al.) 2018). With the power of uncovering relationship behind the data and providing explanatory analyses, causality inference has drawn increasing attention in many fields, e.g. marketing, economics, and neuroscience (Pearl 2000 [Pteers et al., 2017 Since the cause generally precedes its effects, known as temporal precedence (Eichlerj (2033), recently, an increasing number of studies have focused on causal discovery from time series data. They are commonly based on the concept of Granger causality (Granger 1969 1980 to investigate the causal relationship with quantification measures.
In many real-world systems, common to encounter large amount of multivariate time series (MTS) data collected from different individuals with shared commonalities, which we define as het- erogeneous multivariate time series. The underlying causal structures of such data often vary (Zhang et al. 2017 Huang et al., 2019). For example, in the financial market, the underlying causal drivers of stock prices are often heterogencous across stocks different plates. Similar phenomenons are also observed in the sales of different products in E-commence. To this situation, most existing methods have to train separate models for MTS of each individual, which suffer from over-fitting especially given limited training samples. Although some works have been proposed solve such problem (Zhang et al., 2017: Huang et al. 2019). they lack the inductive capability to do inferences for unseen samples and fall short of fully exploiting shared causal information among the heteroge- neous data which often exist in practice. For instance, the causal structures of the products belonging the same categories are usually similar. Such shared information presents opportunities for causal reconstruction to alleviate overfitting and to do inductive reasoning. However, it is also challenging to detect common and specific causal structures simultaneously.
In this paper, we propose Granger cAusal StructurE Reconstruction (GASER) framework for inductive Granger causality learning and common causal structure detection on heterogeneous mul- tivariate time series data. Our approach builds on the idea of quantifying the contributions of each variable series into the prediction of target variable via novel designed prototypical Granger causal attention mechanism. In order to ensure that the attention capturing Granger causality, first de- sign an attention mechanism based on Granger causal attribution of the target series and then perform prototype learning that generates both shared and specific prototypes improve the model's 's robust-

---

Under review as a conference paper at ICLR 2020
ness. Extensive experiments demonstrate the superior causal structure reconstruction and prediction performances of GASER. In summary, our specific contributions are as follows:
A novel framework that inductively reconstructs Granger causal structures and uncovers common structures among heterogeneous multivariate time series.
A prototypical Granger causal attention mechanism that summarizes variable-wise contributions towards prediction and generates prototypes representing common causal structures.
Relative extensive experiments on real-world, benchmark and synthetic datasets as well as an on- line A/B test on an E-commercial advertising platform that demonstrate the superior performance on the causal discovery and the prediction performance comparable to state-of-the-an methods.
2 GASER
In this section, we formally define the problem, introduce the architecture of GASER, present the prototypical Granger causal attention with the final objective function.
2.1 PROBLEM DEFINITION
Assuming we have a set of heterogeneous multivariate time series from N individuals, i.c., X
(X,) with each consisting of time series of length T, denoted as X; = (a|. ................................ xx
RSxT, where x! (.rf. -2.2..... " E R represents the s-th time series of individual i, and one of them is taken as the target series yı We aim to train a model that (1) reconstructs Granger causal structures among variables for each individual: (2) generates K common structures among all the N individuals, each structure represented by a prototype pt E R k = I,... K; and (3) learns nonlinear mapping to predict the next value of the target variable series for each individual, i.e. D1,T+1 = F(X.).
2.2 NETWORK ARCHITECTURE
Our GASER framework consists of two parts: a set of parallel encoders, each predicting the target given the past observations, and an attention mechanism that generates prototypical Granger causal attention vectors to quantify variable-wist contributions towards prediction. Figure illlsstaaees the overall framework of GASER. As illustrated in Figure [[a] for input multivariate time series X;, the encoder specific to s-th variable projects the time series æ into a sequence of hidden state, denoted h - H. (Ii,e h. The encoder could be any RNN models, such LSTM (Hochre- iter & Schmidhuber 1997) and GRU (Cho et al. 2014). The last hidden states, ( are used as the hidden embeddings of each variable. Then the predicted next value of the target variable con- ditioned on historical data of variable s, denoted as 0r can be computed by V8 = fs (hf,,)) where (.) denotes the MLP network specific to variable 8. Then we obtain the prediction Di Ti by aggregating the predicted values specific variables through the prototypical Granger causal attention described below.
2.3 PROTOTYPICAL GRANGER CAUSAL ATTENTION
We propose a novel attention mechanism in GASER, namely prototypical Granger causal attention, to reconstruct Granger causal relationships for each individual and uncover common causal struc- tures among heterogeneous individuals. The goal is to learn attentions that can reflect the Granger causal strength between variables for each individual, and generate prototypes among heteroge- neous individuals. As illustrated in Figure [(b), the idea of the prototypical Granger causal attention mechanism is follows. The Granger causal attribution corresponding to each individual is first computed according to the concept of Granger causality, followed by prototype learning that sum- marizes common causal structures for heterogeneous individuals in the training set, and produces the attention vector specific to each individual. The details of these two parts are described below.
2.3.1 GRANGER CAUSAL ATTRIBUTION
Granger causality Granger 1969 1980) is a concept of causality based on prediction, which de- clares that if time series Granger-causes time series y, then be better predicted using all available information than if the information apart from x had been used. Thus, we obtain the Granger causal attributions by comparing the prediction error when using all available information
2

---

Under review as a conference paper at ICLR 2020
Legand:
MEP
attention factor Oimutip a.m O mins e IEeLU
Prototypical Granger Causal Attention
Input: X,
Paraliel encodors
761
91 7+1
H2
H2
HHa
M
Target:
Pl.
MSE Loas
B4 ++
Hidden embeddings
(a) The architecture of the proposed GASER.
Target: 1711
P1
Coalie Seniladlt
Gar Softnes
2
P2
Ouiput
PK
Granger causal attention
Pronttypical
Prototypical Door Doarger consell atterion
N. 1.
altss
Diversty
Granger Causal Attibuttion
Frototype
(b) Prototypical Granger causal attention.
Figure I: The overview of the GASER framework.
with the error when using the information excluding one variable series. In particular, given all the hidden embeddings ( of individual i, we obtain the embedding that encodes all available information and the one that encodes all available information excluding one variable s, denoted as hall and h a respectively, by concatenating the embeddings of corresponding variables:
19 h h
h
all'ss
where [.] represents the concatenation operation. Then we feed them into respective predictors, denoted as gaul (-) and (-()), get the predicted value of target and compute the squared errors:
call (an(h)
sall's alli
go (h (alll)))
aal (0 1 E;
(U 3r,T+1)
(3)
where the predictor (-) and (.) can be MLP networks. Inspired by Schwab et al. (009 we define the Granger causal attribution of the target variable corresponding variable the decrease
in error when adding s-th series to the set of available information, computed as:
a (E all) - eq!l),
(4)
where ReLU(-) is the rectified linear unit. For each individual i, by normalising the Granger causal attribution, we obtain an attention vector that reflects Granger causality, namely Granger causal attention, denoted qi. The attention factor for variable can be computed as:
Aet
(S)
ΣS
Σ
A
2.3.2 PROTOTYPE LEARNING
The Granger causal attention above not robust enough to reconstruct Granger causal structure, given limited data (e.g., very short time series) of each individual in training. We address the problem
3

---

Under review as a conference paper at ICLR 2020
by generating Granger causal prototypes from all the individuals, under the assumption that there should be several common causal structures among heterogeneous individuals.
In particular, we assume there exist K Granger causal prototypes, denoted as {px He and compute the similarity between the Granger causal attention vector q, of individual and each prototype vectors Since the attention can be seen distribution, we use the cosine similarity:
Pk
d
(6)
P
g
Then we output a prototype most similar to q, by sampling from the similarity distribution d; using Gumbel-Softmax (Maddison al. 2017; Jang al., (016). which samples from a reparameterized continuous distribution approximation to the categorical one-hot distribution:
e GumbelSoftmax((4) (d,) = softmax((log(d) + g)/T),
where GumbelSSofmaxx( denotes the Gumbel-Softmax function, e E R.K the sample vector which approaches one-hot, and g is vector of i.i.d. samples drawn from Gumbel(O, 1) distribution. r is the softmax temperature, and the distribution becomes discrete when т goes to O. With the sample vector e, the output prototype can be obtained as:
p P1, P2, ................................ .Pp]
After normalizing the sampled prototype, obtain an attention vector for individual i, denoted as Ts, namely prototypical attention.
The Granger causal attention reflects the Granger causal structure specific to each individual, while the prototypical attention reflects one common Granger causal structure most similar to the Granger causal structure of each individual. To detect the specific and common causal structures simulta- neously, we summarize them together and generate the prototypical Granger causal attention a, as follows:
a oq: + (1 a)r;,
where E [0, 1] is hyperparameter that controls the ratio of the two attention mechanism.
Finally, the prediction of the target variable's next value can be computed as the weighted sum of the predicted values from all variables:
Ú. Σ af
(10)
2.4 LEARNING OBJECTIVE
In order to obtain accurate prediction and Granger causality structure, and generate diverse common causality structures, the objectives of GASER consist of three parts. The first two objective functions are to encourage accurate predictors, including the predictors f(()) to perform final prediction and the auxiliary predictors g(-) to compute Granger attribution, and we adopt the the mean squared error (MSE) the prediction loss function:
1 E -00T++1-WWTT -1.4+1) Laxx
1 N
allla
Cped
(11)
The last objective function to avoid duplicate prototypes by diversity regularization term that penalizes on prototypes that are similar to each other (Ming et al. 2019h:
P: Pj
Laiv
max(")
(12)
|p.
p,
j=1=+
where 7 controls the closeness to a tolerable degree.
To summarize, the loss function, denoted by L, is given by:
L = Lpeed + X1 Laux + X Luiv.
(13)
where A and Xg are hyperparameters that adjust the ratios between the losses.
4

---

Under review as a conference paper at ICLR 2020
3 EXPERIMENTS
In this section, we evaluate the causal structure reconstruction performance on multivariate time series from both single individual and multiple individuals, as well as the prediction performance of GASER. We also conduct an online A/B test on E-commerce advertising platform further test GASER in more practical situations.
3.1
EXPERIMENTAL SETUP
We first evaluate the causal structure reconstruction performances causal benchmark datasets.
Finance (Kleinberg 2009) consists of simulated financial market time series with known underlying causal structures. Each dataset includes 25 variables of length 4,000. For each dataset, choose variables that are related to the most causes as the target variables to test model abilities in the relatively most challenging scenarios.
FMRI (Smith et al. 2011) contains 28 different Blood-oxygen-levé dependent time series datasets with the ground-truth causal structures. In the experiments, we evaluate the first 5 datasets and take the first variable as the target as causal variables distribute relatively evenly in this dataset.
Then, evaluate the causal structure reconstruction performance heterogeneous individuals synthetic data:
Synthetic data: We first obtain the S exogenous time series through the following Non-linear Au- loregressive Moving Average (NARMA) (Atiya & Parlos 2000) generators:
J a. B,zi,e- Tht + 766.1- + 6i,
(14)
where are zero-mean noise terms of 0.01 variance, d is the order of non-linear interactions, and as. Bg and Ys are parameters specific to variable s, generated from (00 0.1). Then, we generate the target series from the generated exogenous series via the formula:
U8.t Σ wi (mt) tanh (wf,t-pa-1) Ei,
(15)
where wf (0, 1} with 0.6 probability of being zero that controls the underlying causal relationship from the 8-th variable the target variable, nf E Rp controls the causal strength sampling from
Uni and -i.--pt-11 = (#f,--p: " ") RPP represents the last p historical
values of variable of sample i. The 0-1 indicator vector w; = "u:..,,.... w E R is the ground-truth causal structure of i-th individual.
For the causal structure reconstruction task, we compare our method with previous causal discovery methods including linear Granger causality (Granger 1969; Lütkepohl] 2005) and TCDF (Nauta) et al.] 2019). as well as the interpretable neural network based prediction method, i.c., IMV- LSTM (uu et al. 2019), using the standard metrics of Area Under the Precision- Reccal Curve (PR-AUC), and Area Under the ROC Curve (ROC-AUC) (Fawceti] 2006).
Since byproduct of GASER the time series prediction, we also evaluate the prediction perfor- mance on the real-world datasets, i.e., PM2.5 and SML::
PM2.5 contains the hourly PM2.5 and meteorological data in Beijing during Jan 2010 to Dec 2014, includes 7 variables (such as PM2.5 concentration, temperature, pressure and wind speed), and forms multivariate time series length 43,824. The PM2.5 concentration the target series. the dataset is split into training (60%), validation (20%) and testing sets (20%).
SMI. a monitoring dataset for the temperature forecasting, collected from a monitor system in domotic house for approximately 40 days. The data are sampled every minute and smoothed with the mean of every minutes, forming the MTS of length 4.137. We predict the dinning-roon temperature with 17 relevant variable series. The first 3,200, the following 400 and the last 537 data points are respectively used for training, validation, and test.
We compare with linear Granger causality (VAR) and the state-of-the-an prediction models includ- ing DUAL (Qin et al.) 2017) and IMV-LSTM (Guo et al.) 2019), and adopt Root Mean Squared Error (RMSE) and Mean Absolute Error (MAB) metrics.

---

Under review as a conference paper at ICLR 2020
Table 1: Causal structure reconstruction results on Finance and FMRI data.
Finance (9 datasets)
FMRI (5 datascts)
Methods
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
IMV-LSTM
0.778=:0.222 0.187-+00003
0.862:+0.172
0.593=:0.399
0.620= 136
lincar Granger
0.652+00.084
0.492+0.310
0.654=0. 126
TCDF
0.478:10.263
0.766:10.145
0.540:10.250
0.664 099
GASER
1.000=0.000
1.000=0,000'
0.740 0.641:10.227
denotes the p-value is less than 1%, and denotes the p-value is less than 5%.
3.2 EXPERIMENTAL RESULTS
3.2.1 CAUSAL. STRUCTURE RECONSTRUCTION PERFORMANCE ON HOMOGENEOUS MULTIVARIATE TIME SERIES
To evaluate the causal discovery performance homogeneous multivariate time series, train individual models for each dataset with the hyper-parameter a equaling 0.5. We report PR-AUC and ROC-AUC averaged across all datasets, with the standard deviation reported in Table As can be seen, the proposed method greatly surpasses other methods. Especially, GASER recovers the ground-truth causal structure with high score on the Finance data.
3.2.2 CAUSAL STRUCTURE RECONSTRUCTION PERFORMANCE ON HETEROGENEOUS MULTIVARIATE TIME SERIES
In this part, we evaluate the causal discovery performance on heterogeneous multivariate time series. We denote the number of common causal structures as C, the number of variables as S and the series length as T. and generate 100 multivariate time series for each common causal structure according to Equation (14 and Equation (15). forming 100C datasets. For the inductive methods GASER and IMV-LSTM, we train one model using all the datasets, while for other methods, we train sepa- rate models for each dataset. We report PR-AUC and ROC-AUC results w.r.t the variable number, the series length and the common structure number in Table to 4 respectively. We observe that GASER outperforms other methods significantly in all cases, and GASER (a 0.5) (with the Pro- totypical Granger causal attention) performs better than GASER (a = 1) (only with Granger causal attention). The observations demonstrate the superior causal discovery performance of GASER, the effectiveness of the prototypical Granger causal attention in GASER, and the ad advantages of utilizing shared commonalities among heterogeneous MTS. Regarding the other competitors, linear Granger performs the best followed by TCDF and IMV-LSTM at most cases. The possible reason is that linear Granger can detect Granger causal relations to some extent, though it utilizes linear model, i.e. Vector autoregression (VAR). TCDF utilizes iitention-based CNN inference potential causals followed by a causal validation step, but the attention it proposed cannot reflect Granger causality, thus achieves unsatisfactory performance. Compare to the performance homogeneous multivari- ate time series, the performance of IMV-LSTM drops dramatically, which indicates that the attention mechanism in IMV-LSTM fails given heterogeneous multivariate time series.
In Table we vary the number of variables to generate datasets of different complexity, and we can that GASER outperforms other competitors consistently across different S, and achieves good performance when S is as large as 20, demonstrating our method's 's capability to infer complex causal structures. Since in practice, the size of collected data is often limited, which poses challenges to recover causal structure, thus we also vary the length of time series to see the model robustness to data of small sizes. As can be seen in Table 3 GASER outperforms other methods across all cases, even when T' is as small as 20, which demonstrates that advantage of using shared information. We also observe that GASER (a 0.5) surpasses GASER (a = 1) by a large margin, which demon-
Table 2: Causal structure reconstruction results w.r.t the variable number (C=3, T=1000)
S=5
S=10
$=20
Methods
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
IMV-LSTM
0.511 102
0.500-10.236
0.53610.056
0.51410.019
0.599:10.087
0.61910.887
linear Granger TCDF
0.666-:0.10
0.822440.05
0.765=10.00
0.889-00063
0.826-10. 106
0.854=0.000
0.523-0.103
0.523:102114
0.548:0.165
0.58710.188
0.584:10.162
0.642:0.152
GASER (a 1)
18866
0.906 1433*
0.975 0.034
0.830:100. 108
0.883=0.069
GASER (a = 0.5)
0.911
0.922-0.122
0.999::0.00
0.858=00.10
0.030
denotes the p-value is less than 1%, and denotes the p-value is less than 5%.

---

Under review as a conference paper at ICLR 2020
Table 3: Causal structure reconstruction results w.r.t the series length (C=3, S=10).
T=20
T=100
T=1000
Methods
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
IMV-LSTM
0.467=:0.025
0.541=0.035
0.503=0.081
0.51140.018
0.536 056
0.514::0.019
linear Granger TCDF
0.400-0.000
0.500=0.000
0.889+0.152
0,943+0.085
0.765++0.109
0.889-
0.518=00.131
0.513=0.112
0,517=0.120
0,544:10.16
0,548:10.165
0,587:+0.180
GASER (a = 1)
1.90010,14""
0.793-1.1100
0.973
0.974_0.038
0.974 0.998=0.009"
0.975
GASER (a = 0.5)
0.824::0.233"
0.833::0.11
0.973=0.040"
0.976=0.036
0.999=10.008
denotes the p-value is less than 1%, and denotes the p-value is less than 5%.
Table 4: Causal structure reconstruction results w.r.t. the common structure number C' (S=10,
T=1000). We set the hyper-parameter of prototype number K in the model as the same as the ground-truth common structure number C.
C=3
C=S
C=7
Methods
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
PR-AUC
ROC-AUC
GASER(0 =
0.974+0.038
0.975=00037
0.891 118
0.883+001228
0.838+0.118
0.850+0.1 113
GASER(o = 0.5)
0.998+0.009
0.999-0.008
0.924+0.091
0.913++01105
0.851+0.113
0.855++0.099
strates that learning prototypical attention can alleviate the over-fitting problem. In Table 4
control the causal heterogeneity by varing the number of common causal structures C = {3. 5, 7)-
We observe that the performance of GASER decreases with increasing C. Figure 2 we map the learned causal attention vectors to a 2D space by the visualization tool t-SNE (Maaten & Hinton, 2008 Individuals of different causal structures are labeled by different colors. From the results, we observe that nodes belonging to the same causal structures are clustered together, which also demonstrates the effectiveness of our method.
3.2.3 PREDICTION PERFORMANCE
We evaluate the prediction performance on the real-world datasets, i.e., PM2.5 and SML. To eval- uate the robustness of prediction and the accuracy of Granger causal attribution, we also build an- other datasets that only contain top 50% important variables towards prediction detected from each method. We report the prediction results in Table 5 As can be seen, GASER achieves the best performance on PM2.5 data with all features, demonstrating its superior prediction performances. We also observe that GASER achieves comparable or even better performance using selected vari- ables, while the others' performances decrease, which indicates that effective variable selection of GASER. Linear Granger achieves the best performance on SML data, because the time series length of SML is short, thus providing limited training samples for neural-netwon based methods.
3.3
ONLINE A/B TESTS
In order to further evaluate the effectiveness of GASER in practice, online A/B test is conducted on an E-commercial platform, and the process is designed as follows:
We first train GASER on the historical MTS of 30,665 items. Each MTS includes 26 variables related to searching. recommending and advertising, such as Page View (PV), Gross Merchandise Volume (GMV) and Impression Position In- In-Page, etc. Here, take the item popularity as the target series, and generate the underlying causal structure for each item.
We randomly sample 100 items whose impression position in-page Granger-causes the item pop- ularity with high confidence, and divide them into two buckets. For Bucket A. we adjust impres- sion positions in page of each item by grid since 201 9/08/19 9 till 201 2019/08/29, and ensure the intervention has little impact on other variables. For Bucket B, we do nothing.
We compare the improvement rate of item popularity week-on- on-kee on the two bucket in Fig
Table 5: Predition results under all variables and top 50% important variables on the PM2.5 and
SML datasets.
PM2.5 (all)
PM2.5 (top 50%)
SML (all)
SML (top 50%)
Methods
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
linear Granger
21.38
11.82
21.39
11.83
0.0187
0.0144
0.0190
0.0151
DUAL
20.93
11.70
20.90
11.90
0.0778
0.0676
0.0887
0.0752
IMV-LSTM
21.60
11.77
21.62
11.80
0.0747
0.0561
0.1180
0.0857
GASER
18.88
11.03
18.22
10.62
0.0670
0.0540
0.0621
0.0492
7

---

Under review as a conference paper at ICLR 2020
0.1
Improvement rate of item popularity
0.05
-0.05
10001390000
Bucket A
Bucket
01001011101111111000100010011110
-0.15
1:000111:00
8/18
8/19
8/20
8/21
8/22
8/23 8/24 8/25
8/26 8/27 8/28 8/29
Time
Figure 2: 2D t-SNE projections of attention vectors from 500 individ- uals. Color of node indicates the underlying causal structures. The causal groundtruth w; is shown in the legend ($=10.T=100).
Figure The result of online A/B test. The interven- tion starts from 08/19, and results in the item popularity improvement rate of Bucket consistently outperforming Bucket B after 8/22, and the gap between the two buckets increases significantly since 08/25.
As shown in Figure four days after the beginning of the intervention, the item popularity im- provement rate of Bucket A consistently outperforms that of Bucket B, and the gap between the two buckets increases significantly since 2019/08/25, which shows that the intervention, i.e., adjusting the impression positions in-page, causes the improvement on item popularities, thus demonstrates the right causal relationships detected by GASER.
RELATED WORK
Recently considerable amount of work has been proposed for causal inference. Classical meth- ods, such as constraint-bent -asedd methods (Pearl 2000 Spprtes et al. 20000 Peters et al.) 22013] Runge al. 2017: Zhang et al. (017), score-based methods (Chickering 2002) and functional causal models (HCM) based methods (Shimizu et al. 2006), mainly focus on Lidd data. Under the scope of time series, causal inference is commonly based on the notion of Granger causality (Granger 1969
1980), and a classical way is to estimate linear Granger causality under the framework of VAR models (Lütkepohl) 2005). However, existing classicial methods fail to uncover causal structures inductively. Neural network based methods that infer causal relationships or relations that approach causality have gained increasing popularity. [Lopez-Paz et (2015) learns probability distribution classifier to unveil causal relations. Kipf et al. (2018) proposes neural relation inference model infer interactions while simultaneously learning the dynamics. Yu et al. (2019) develops deep generative model to recover the underlying DAG from complex data. Attention mechanism has often been adopted to discover relations between variables. For example, Dang et al. (2018) discov- ers dynamic dependencies with multi-level attention. [aauaa et al. (2019) studies causal discovery through anteetioo-bbsed neural networks with a causal validation step. Guo et al. (2019) proposes an interpretable multi-variable LSTM with mixture attention to extract variable importance knowledge. However, these attention mechanisms provide no incentive to yield accurate attributions (Sundarara-
jan al. 2017 Schwab al. 2019)
Since our method utilizes the concept of prototype to detect common causal structures, another line of related research is about prototype learning. Prototype learning form of cased-based reason- ing (Slade 1991}. which solves problems for new inputs based on similarity to prototypical cases. Recently prototype learning has been leveraged in interpretable classification (Bien et al. 201T Kim
et al. 2014 Snell et al., 2017 Li et al. 2018 Chen et al. 2018) and sequence learning (Ming et al.] 2019. We incorporate the concept for Granger causal structure reconstruction on time series data for the first time.
5 CONCLUSION
We formalize the problem of Granger causal structure reconstruction from heterogeneous MTS data and propose inductive framework GASER solve it. In particular, propose novel at- tention mechanism, namely prototypical Granger causal attention, which computes Granger causal attribution combined with prototype learning, reconstruct Granger causal structures and uncover
8

---

Under review as a conference paper at ICLR 2020
common causal structures. The approach has been successfully evaluated by offline experiments real-world and synthetic datasets compared to previous methods, also confirmed by an online A/B test E-commercial platform.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv: 04467, 2016.
Amir Atiya and Alexander G Parlos. New results on recurrent network training: unifying the algorithms and accelerating convergence. IEEE transactions neural networks, 11(3):697-709, 2000.
Jacob Bien, Robert Tibshirani, et al. Prototype selection for interpretable classification. The Annals of Applied Statistics, 5(4):2403-2424, 2011.
Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. arXiv preprint
1rXi::8806.10544.. 2018.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-55) 2002.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dumitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Xuan-Hong Dang, Syed Yousaf Shah, and Petros Zerfos. seq2graph: Discovering dynamic depen- dencies from multivariate time series with multi-level attention. arXiv preprint arXiv:1812.04444 2018.
Michael Eichler. Causal inference with multiple time series: principles and problems. Philosophi- cal Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371 (1997):20110611 100613,,111,1.. 2013.
2013.
Tom Fawcett, An introduction roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
Clive WJ Granger. Investigating causal relations by econometric models and cross-spectral methods.
Econometrica: Journal of the Econometric Society, pp. 424-438. 1969.
Clive WJ Granger. Testing for causality: personal viewpoint. Journal of Economic Dynamics and control, 2:329-352, 1980.
Ruocheng Guo, Lu Cheng, Jundong Li, P Richard Hahn, and Huan Liu. A survey of learning causality with data: Problems and methods. arXiv preprint (tti::/809.0953. 2018.
Tian Guo, Tao Lin, and Nino Antulov-Fantulin Exploring interpretable Istm neural networks over multi-variable data. In International Conference Machine Learning, pp. 2494-2504, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()) 1780, 1997.
Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Bernhard Schölkopf, and Clark Glymour. Causal discovery and hidden driving force estimation from nonstatonaaryheteeggeeeuus data. arXiv preprint arXiv: (90..0/990..7 2019.
Eric Jang, Shixiang Gu. and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv: 1.0/ /44, 2016.
Been Kim, Cynthia Rudin, and Julie Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Proceedings of Neural Information Process- ing Systems (NIPS), 2014.

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2693- 2702, 2018.
Samantha Kleinberg. Causality, probability, and time. In Causality, probability, and time. Cam- bridge University Press, 2009.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of AAAI, 2018.
David Lopez-Pazz Krikamol Muandet, Bernhard Schölkopf. and liya Tolstikhin. Towards learning theory of cause-effect inference. In International Conference on Machine Learning, pp. 1452- 2015.
Helmut Lütkepohl. New introduction multiple time series analysis. Springer Science & Business Media, 2005.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNvv):5579-2605 2008
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations (ICLR), 2017.
Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. KDD, 2019.
Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention based convolu- tional neural networks. Machine Learning and Knowledge Extraction, (1):3312340)) 2019.
Judea Pearl. Causality: models, reasoning and inference, volume 29, Springer, 2000.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Causal inference time series using restricted structural equation models. In Advances in Neural Information Processing Systems, pp- 164-162, 2013.
Jonas Peters, Dominik Janzing, and Berhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
Yao Qin, Dongjin Song, Haifeng Cheng, Wei Cheng, Guofei Jiang, and Garrison W Cottrell. A dual-stage attention- based recurrent neural network for time series prediction. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 2627-2633 AAAI Press, 2017.
Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting causal associations in large nonlinear time series datasets. arXiv preprint 17Xiv:1702.07007, 2017.
Patrick Schwab, Djordje Miladinovic, and Walter Karlen. Granger-causal attentive mixtures of ex- perts: Learning important features with neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 4846 4846-4853, 2019.
Skipper Seabold and Josef Perktold. Statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, and Antti Kerminen. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 70c2)):003--0030, 2006.
Stephen Slade. Case-based reasoning: A research paradigm. Al magazine, 12(1):42 42, 1991.
10

---

Under review as a conference paper at ICLR 2020
Stephen M Smith, Karla L Miller, Gholamreza Beckmann, Thomas E Nichols, Joseph Ramsey, and Mark W Woolrich. Network modelling methods for finri. Neuroinage, 54(2)875-891, 2011.
Matthew Webster, Christian
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087,2011
Peter Spirtes, Clark N Glymour, Richard Scheines, David Heckerman, Christopher Meek, Gregory Cooper, and Thomas Richardson. Causation, prediction, and search. MIT press, 2000.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference Machine Learning-Volle 70, pp. 3319- 3328. JMLR. org, 2017.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning. pp. 7154-7163, 2019.
Kun Zhang, Biwei Huang, Jiji Zhang, Clark Glymour, and Bernhard Schölkopf. Causal discovery data: Skeleton estimation and orientation determination. In
from ponstationary/heeerrggnnn IJCAI: Proceedings of the Conference, volume 2017, pp. 1347. NIH Public Access, 2017.

---

Under review as a conference paper at ICLR 2020
A
TABLE OF NOTATIONS
Table 6: Terms and Notations
Symbol
Definition
N
the number of individuals the number of variables the length of time series
S
T
K
the number of prototypes
C
the number of underlying common causal structures
X
heterogenous multivariate time series multivariate time series of individual the s-th time series individual i the target variable series of individual i the T + 1-step value of the target variable the predicted T 1-step value of the target variable by the main predictor the predicted T + 1-step value of the target variable conditioned on variable the k-th prototype
X,
y
T
U1T+1
9 911+1
ps
a;
the prototypical Granger causal attention vector of individual i the Granger causal attention vector of individual the prototypical attention vector of individual the hyperparameter to control the ratio of different attentions the hyperparameter to control the ratio of loss functions the softmax temperature
qi
r;
a
\1, A2
T
B ALGORITHM PSEUDOCODE
The full algorithm is presented in Algorithm [ The network parameter set θ includes the param- eters of sequence encoders and MLPs. We adopt stochastic gradient descent (SGD) to optimize the network parameters and the prototype parameters. To initialize the prototypes, GASER for several epochs, and then employ k-means with cosine similarity on the Granger causal attentions {q) and finally take the cluster centers the initial prototypes. Note that the Gumbel-softmax function is only adopted in the training phase to backpropagate, and replaced by the argmax function in inference.
first pretrain
Algorithm 1: The algorithm of GASER.
Input:
Input data X [X.] J.W. Number of prototypes K; Maximum iterations Maxlter: Hyperparameters a, 7, Aı and A2.
Output:
Network parameters O; Attention vectors {a, J Prototypes {p) Prediction results
(Us.++1l J.
Pretrain the model by optimizing Epred + X₁ Caxi
2: Employ k-means on Granger causal attention vectors (q:}, N to get initial prototypes
{pi)
3: for iter+ + 1 Marlter do
K 4: Update network parameters and prototypes {
by optimizing
L= Epedd + Xa + XaLavv
5: end for
6: Generate prototypical Granger causal attentions (a: by Equation 1
using the argmax
function instead of the Gumbel-softmax function.
12

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL DETAILS ON THE EXPERIMENTAL SETUP
C.1 DATASETS
In this section, we provide some additional dataset details. Finance data are available at http: 7/ //www. exleinberrg org/ data. .html We the processed FMRI data provide by (Nauta et al. 2019). The source and details of PM2.5 and SML are at at/https://acchhhaaa DDData and htttps://rrchiee. wcw. edu
ics.ucu uci,
eed acdu/ml. httmaa/aatt/////ataama Bei Beweieeiinnnneeeee Ding+ PM2
ml/ datasets/9ML2010 respectively.
C.2 IMPLEMENTATION DETAILS
We implement GASER in Tensorflow (Abadi et al. 2016) by the Adam optimizor (Kingma & Ba 2014) with the learning rate set to 0.001. We adopt LSTMs as the sequence encoders with the hidden states size set to 128 and the window size set to 5. In all experiments, we first pretrain GASER with only the Granger causal attention for 40 epochs. The hyperparameters A and 22 are both set to 1, and the softmax temperature in Gumbel-softmax is set 0.1.
C.3 COMPARED METHOOS
Linear Granger Granger 1969 1980): We conduct Granger causality test in the context Vec-
tor Autoregression (VAR) as described in chapter 7.6.3 in (Lütkepohl) (0055) and implemented by the Statsmodels package (Seabold & Perktold 2010). In detail, we perform F-test at 5% significance level. The maximum number of lags to check for order selection is set to 5, which is larger than the causal order in the ground-truth.
TCDF (Nauta et al.) 2019): TCDF learns causal structure on multivariate time series by attention- based convolutional neural networks combined with a causal validation step. The codes are available at :ttpps:/Aee
com / M Nauta TCDF In all experiments, we follow the default settings as described (Nauta et al. 2019), i.e., the significance number (stating when an increase in loss significant enough label potential cause true) 0.8, the size of kernels 4, dilation coefficient as 4, the learning rate as 0.01, and adopting Adam optimizator.
DUAL QQn et al.) 2017): It is encoder-decoder RNN with input attention mechanism, which forces the model pay more attention on certain driving series rather than treating all the input driving series equally. In the experiment, we use the input-attention factors detect important variables as (Guo et al.] 2019) did. We set the the size of hidden states for encoder and decoder to 64 and the window size 10 as stated in the paper.
IMV-LSTM (Guo et al. 2019: It is a multi-variable attention-based LSTM capable of both pre- diction and variable importance interpretation, with the attention factors reflecting importance of variables in prediction. Thus, we take the learnt attention vectors as the Granger causal weights in the experiment. The codes are available fetttss:////.. / com/ Kete/kurochkk IA nnlexey/ IMV_LSTV LSTM In all experiments, IMV-LSTM is implemented by Adam optimizer with the mini- batch size 64, hidden layer size 128 and learning rate 0.001.
D SUPPLEMENTARY EXPERIMENTAL RESULTS
D.1 VISUALIZATION LEARNED PROTOTYPES
We visualize the the learned prototypes and the ground-truth causal structures in Fig. 4 In this experiments, we set the parameerr of prototype number K equal to the ground-truth common structure number C. From the results, we can see that the learned prototypes are similar to the ground-truth causal structures, which demonstrates the learned prototypes are interpretable.
D.2 VISUALIZATION OF GRANGER CAUSAL ATTENTION
In this part, we visualize the Granger causal attention vectors over epochs during the training phase and the ground-truth causal structures in Fig. From the results, see that for the later epochs,
13

---

Under review as a conference paper at ICLR 2020
Leared
Leattied protatlyas
Ground-rr sirustunes
4
-38
42
93
25
16
45
03
45
66-
02
C=3
(b) C=5
Figure 4: The visualization of prototypes and ground-truth causal structures. Plots are shown for various number of common causal structures C (S=10, T=1000). Each column of the heat map visualizes one structure.
Epoch
Epoch 2
Epoch
Epoch
Epoch
Ground-truth structures
x1
x1
x6
x4
16
x6
X6
86
86
18
X8
X8
Figure 5: The visualization of Granger causal attention vectors epochs during the training phase and the ground-truth causal structures. Each column of the heat map visualizes one structure (C=5, S=10, T=1000).
shown the right of the figure, the Granger causal attention vectors are similar to the ground- truth structures. demonstrates that the proposed Granger causal attention mechanism forces the attention values to correlate with Granger causality.
D.3 RUNNING TIME
We compare the running time including the training time and the inference time with linear Granger method, i.e. VAR. The experiments were carried out on server with 64 Intel(R) Xeon(R) E5-2682 v4 2.50GHz processors and 512 GB RAM. Since the linear Granger method not inductive and has to retrain for new individuals, we set the inference time equal to the training time. As shown in Table 7 although GASER takes more time on the training phase, about 6.4 times slower than linear Granger, it is about 44.5 times faster on the inference phase.
Table The comparison running time on heterogeneous multivariate time series of 1,000 indi- viduals (5=10, T=100, C=10).
Training (s)
Inference (s)
linear Granger
19.59
19.59
GASER
124.53
0.44
14

---

Under review as a conference paper at ICLR 2020
0.84
0.84
0.84
2.00
0.00
0.00
PR-AUC
PR-AUC
PR-AUC
0.76
0.76
0.78
20
13
25
0.7
0.. 022 o' oA
(a)
(b)
(c)
0.82
0.02
ROC-AUC
ROC-AUC
100
(d) X
(e)
Figure 6: Performance results w.r.t. different numbers of prototypes K and different values of the hyper-parameter a, λ1 X2 and 7.
D.4 PARAMETER SENSITIVITY
We investigate the parameter sensitivity this section. Specifically, we evaluate the sensitivity of GASER to different numbers of prototypes K and different values of hyper-parameter o,7,, À and Ag. The results on heterogenous MTS (C=5, $$10, T=100) are shown Fig. 6
We first show how the number of prototypes affects the performance in Fig. 6(a) We can that the performance raises when increases and achieves the best performance when the number of prototypes K is equal the number of underlying common structures C. Then, the performance starts to drop slowly. Overall, the proposed method is not very sensitive to the parameter K.
Then evaluate how the value of affects the performance in Fig. 6(b) The parameter balances the weight of the Granger causal attention and the prototypical attention in our model. We can see that when 0.5, the model reaches the best performance, which demonstrates both the Granger causal attention and the prototypical attention are essential in our model, and it is important to find good balanced point between them.
The parameter controls the closeness of different prototypes. The smaller the the prototypes are more diverse. The result is shown in Fig. 6(C) We can that when y is too small, the result is good. This is intuitive as the ground truth common structures among heterogeneous MTS are not totally different. Too large y also deteriorates the performance, because it will result in number of similar even duplicate prototypes. Thus, it is important to determine the parameter carefully.
Figure ((d) shows the the performance of GASER w.r.t. the parameter Å₁. The parameter X is the weight of the auxiliary prediction loss. From the result, we can see that the performance raises then \ increases initially, and then keeps stable. demonstrates that the auxiliary predictors are essential to detect the Granger causality.
Finally, we show how the value of A2 affects the performance in Fig. 6(e) The parameter X controls the weight of the prototype diversity regularization. From the result, we that the prototype diversity loss is essential our mobel, but we should concentrate more on choosing an appropriate weight.
15


Reviews:
[{"review_id": "Sygod8Oeqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel way of reconstructing Granger causal structures using a differentiable neural network architecture that contains attention modules that are proportional to the Granger causality of the input layers. Furthermore, the architecture blends individual-specific induced causal structures and cross-population prototypical causal structures. The paper has an extensive experimental section on which the proposed method shows impressive improvements in causal discovery performance and predictive performance on par with state-of-the-art.\n\nAs main contributions the paper:\n* proposes a novel architecture \n* shows its values using extensive experiments\n\nOverall, I find the paper well-written with a clear description of the proposed architecture and clear experiments showing the importance of the architectural choices. A possible downside is the relative lack of novelty, since the method seems like a reasonable extension of the existing work. However, I think this counterbalanced by the excellent results on the causal discovery task and the extensive nature of the experiments.\n \nIn terms of suggestions, I think an illustrative example of the granger attention on an artificial / toy example would help a lot to give an intuitive understanding on how the method works and how the causal structure is being built.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx4mNVt9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new way of finding the Granger temporal-causal network based on attention mechanism on the predictions obtained by individual time series. It describes a surprisingly complex procedure for computing the attention vector based on combining Granger-inspired attentions with attentions obtained during a diverse prototype generation process. There are also extensive experiments demonstrating the success of the proposed method in uncovering the underlying temporal-causal graph.\n\nThere is a major theoretical and conceptual issue in this paper: the proposed attention vector depends on all of the time series. Thus, theoretically, it is a mistake to think that the $i$th time series causes the target if its attention value is higher.  In other words, the attention values leak the information about all time series in prediction of the target time series. The following paper is helpful on this topic:\n\nSarthak Jain and Byron C Wallace. \"Attention is not explanation\". In NAACL 2019.\n\nThe authors should be extremely careful in terms of calling Granger causality the actual causality. Granger causality only discovers time order. Also, it is unclear why the authors use terms such as \"inductive\" Granger causality, which is redundant.\n\nIt is unclear for me if the results in the paper are statistically significant, given the provided intervals. \n\nThe authors should expand the motivation for the prototypical design. In the experiments they show that having the prototype part improves robustness and accuracy, but the authors never show what interesting prototypes they learn. Also, this model is super complex. A key reason for popularity of the Granger causality is the simplicity of the underlying VAR model. Finally, the authors need to provide the run-time speed of training and inference for the model.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1e0Fk7A5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--\nComments after reading the reply from the authors:\nThanks for the clarification which resolves most of my concerns, and I have updated the score accordingly.  Besides, it will be more convincing to add more baselines for comparison or provide more explanation why linear Granger (which is proposed decades ago) is a strong baseline.\n\n--\nThis paper investigates the important problem of inferring Granger casual structures from multi-variate time-series data, and propose the Granger casual structure reconstruction (GASER) framework with prototypical Granger causal attention.  The paper is in general well written with clear notations and is easy to follow. The proposed attention mechanism is also  intuitive. Experiments on  both simulated/synthetic and real-world datasets show the proposed approach achieved both improved casual recovery and more accurate predictions.\n\nHere are some concerns, and the reviewer is willing to adjust the rating if these concerns are resolved.  \n\n- Baselines. Have enough strong baseline algorithms been included? \n (1) It seems that GASER outperforms state-of-the-art prediction algorithm IMV-LSTM by a large margin (Table 5) even if it is not designed for the prediction task. \n(2) In Table 1-3, the other two baselines usually perform similarly or worse than the simple linear Granger baseline. Does it indicate the compared baselines are not strong enough? \n(3) Why linear Granger, i.e, VAR with L1 regularization, is not included in Table 5 for prediction? \n\n- Model design and trade-off.  According to Table 1-5, it seems that by adding the proposed attention mechanism, we can achieve both improved results (better than SOTA IMV-LSTM) and better interpretability.\n(1) Is there any trade-off in the design?  \n(2) Is the proposed approach sensitive to different parameters, e.g., 1) the number of prototypes, $K$ (when it is not the same with the ground-truth), 2) $\\alpha$ in Equation 9 (is 0.5 is a generally good default choice?), 3) $\\lambda_1, \\lambda_2$ in Equation 13, 4) $\\gamma$ in Equation 12.\n\n- Prototype learning. It seems that the prototype learning is used to deal with the lack of data. However, adding prototype learning increases the number of parameters to be learnt, i.e., $[p_1, … p_K]$.  It will be helpful to provide more intuition. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a solution to learn Granger temporal-causal network for multivariate time series by adding attention named prototypical Granger causal attention in LSTM. 

The work aims to address an important problem. The proposed solution seems effective empirically. However, two major issues have not been fully addressed in the current version: (1) the connection between Granger causality and the attention mechanism is not fully justified; (2) the complex design overkills the whole concept of Granger causality (since its popularity is due to the simplicity). 

The paper would be a strong publication in the future if the two issues can be addressed in a satisfactory way. 
Decision: Reject