Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality
Authors: Saurabh Khanna
Vincent Y. F. Tan
Venue:  (2020)
Keywords: Recurrent neural networks
Granger causality
Causal inference
Statistical Recurrent Unit

Abstract: Granger causality is a widely-used criterion for analyzing interactions in large-scale networks. As most physical interactions are inherently nonlinear, we consider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements. Our proposed approach relies on modeling the embedded nonlinearities in the measurements using a component-wise time series prediction model based on Statistical Recurrent Units (SRUs). We make a case that the network topology of Granger causal relations is directly inferrable from a structured sparse estimate of the internal parameters of the SRU networks trained to predict the processes’ time series measurements. We propose a variant of SRU, called economy-SRU, which, by design has considerably fewer trainable parameters, and therefore less prone to overfitting. The economy-SRU computes a low-dimensional sketch of its high-dimensional hidden state in the form of random projections to generate the feedback for its recurrent processing. Additionally, the internal weight parameters of the economy-SRU are strategically regularized in a group-wise manner to facilitate the proposed network in extracting meaningful predictive features that are highly time-localized to mimic real-world causal events. Extensive experiments are carried out to demonstrate that the proposed economy-SRU based time series prediction model outperforms the MLP, LSTM and attention-gated CNN-based time series models considered previously for inferring Granger causality. 

Published as a conference paper at ICLR 2020
ECONOMY STATISTICAL RECURRENT UNITS FOR INFERRING NONLINEAR GRANGER CAUSALITY
Saurabh Khanna
Vincent Y. F. Tan
Department of Electrical and Computer Engineering National University of Singapore
Department of Electrical and Computer Engineering Department of Mathematics National University of Singapore
e llesaur@nus eew.eu...... sg
vtar..nus edu. 5g
ABSTRACT
Granger causality is widely-used criterion for analyzing interactions in large- scale networks. As most physical interactions are inherently nonlinear, we con- sider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements. Our proposed approach relies modeling the embedded nonlinearities in the measurements using a component-wise time series prediction model based on Statistical Recurrent Units (SRUs). We make case that the network topology of Granger causal relations is directly inferrable from a structured sparse estimate of the internal parameters of the SRU networks trained to predict the processes' time series measurements. We propose variant of SRU, called economy-SRU, which, by design has considerably fewer trainable parameters, and therefore less prone to overfitting. The economy-SRU computes low-dimensional sketch of its high-dimensional hidden state in the form of random projections to generate the feedback for its recurrent processing. Additionally, the internal weight parameters of the economy-SRL are strategically regularized in a group-wise manner to fa- cilitate the proposed network in extracting meaningful predictive features that are highly time-localized to mimic real-world causal events. Extensive experiments are carried out to demonstrate that the proposed economy-SRU based time series prediction model outperforms the MLP, LSTM and attention-gated CNN-based time series models considered previously for inferring Granger causality.
INTRODUCTION
The physical mechanisms behind the functioning of any large-scale system can be understood in terms of the networked interactions between the underlying system processes. Granger causality one accepte criterion used in building network models of interactions between large en- sembles of stochastic processes. While Granger causality may not necessarily imply true causality, it has proven effective in qualifying pairwise interactions between stochastic processes in a variety of system identification problems, e.g., gene regulatory network mapping (Fujita al. 2007) ), and the mapping of human brain connectome (Seth er al. (2015)). This perspective has given rise to the canonical problem of inferring pairwise Granger causal relationships between set of stochastic processes from their time series measurements. At present, the vast majority of Granger causal in- ference methods adopt a model-based inference approach whereby the measured time series data modeled using with a suitable parameterized data generative model whose inferred parameters ul- timately reveal the true topology of pairwise Granger causal relationships. Such methods typically rely using linear regression models for inference. However, as illustrated in the classical bivariate example by Back & Brock 1992). , linear model-based Granger causality tests can fail catastrophi- cally in the presence of even mild nonlinearities in the measurements, thus making a strong case for our work which tackles the nonlincarities in the measurements by exploring new generative models of the time series measurements based recurrent neural networks.

---

Published as a conference paper at ICLR 2020
2 PROBLEM FORMULATION
Consider multivariate dynamical system whose evolution from an initial state is fully characterized by distinct stochastic processes which can potentially interact nonlinearly among themselves. Our goal here is to unravel the unknown nonlinear system dynamics by mapping the entire network of pairwise interactions between the jystem-defining stochastic processes, using Granger causality as the qualifier of the individual pairwise interactions.
In order detect the pairwise Granger causal relations between the stochastic processes, we as- sume access to their concurrent, mniforml--samples measurements presented as an n-variate time series x {x, : t N} R Let Xt,; denote the 7t component of the n-dimensional vector mea- surement Xt, representing the measured value of process i at time L. Motivated by the framework proposed Tank et al. 2017). we assume that the measurement samples x,, E N are generated sequentially according the following nonlinear, component autoregressive model:
Xt,t fo (xt- X -ppptt---------- 1,t--11221 xt-p:t-1n)) + Cr,b 1, ........ n,
(1)
-ppp. represents the most recent p measurements of the jth component of x in the immediate past relative current time The scalar-valued component generative function fi captures all of the linear and nonlinear interactions between the n stochastic processes up to time f 1 that decide the measured value of the 2th stochastic process at time The residual Ci,, encapsulates the combined effect of all instantaneous and exogenous factors influenc- ing the measurement of process i at time I, as well as any imperfections the presumed model. Equation [ may be viewed as a generalization of the linear vector autoregressive (VAR) model in the sense that the components of x can be nonlinearly dependent on one another across time. The value p is loosely interpreted to be the order of the above nonlinear autoregressive model.
where X 1f- _ 1.3-1.x.(-2,51
2.1 GRANGER CAUSALITY IN NONLINEAR DYNAMICAL SYSTEMS
We now proceed to interpret Granger causality in the context of the above component-wis time series model. Recalling the standard definition by Granger 1969), a time series said to Granger cause another time series u if the past of v contains new information above and beyond the past of u that can improve the predictions of current or future values of u. For x with its n components generated according equation[]] the concept of Granger causality can be extended suggested by Tank et al. (2018) as follows. We say that series j does not Granger cause series if the component- wise generative function fi does not depend on the past measurements in series j, i.e., for all t and all distinct pairs X4 11. and x x-pptt1.1
St (xt-pet-1,1: x(-ptt-. X(-pt-1n) = (X1-p--111- (-pptt1.
From equation]] I it is immediately evident that under the constraint in equation the past of series does not assert any causal influence on series i, in alignment with the core principle behind Granger causality. Based on the above implication of equation 2 the detection of Granger noncausality between the components of x translates to identifying those components of x whose past irrelevant to the functional description of each individual f; featured in equation
Note that any reliable inference of pairwise Granger causality between the components of fea- sible only if there influence x. In this work, we assume that the system of interest is causally sufficient (Spirtes & Zhang (016))), i.e., none of the stochastic processes (whose measurements are available) have common Granger-causing-ancee annessoor that is unobserved.
no unobserved confounding factors in the system which could potentially
2.2 INFERRING GRANGER CAUSALITY USING COMPONENT-WIR RECURRENT MODELS
We undertake model-based inference approach wherein the time series measurements are used as observations to learn an autoregressive model which is anatomically similar to the component- wise generative model described in equation [ except for the unknown functions ft replaced with their respective parameterized approximations denoted by 9:. Let θ,, 1 < i < denote the com- plete set of parameters encoding the functional description of the approximating functions {9: hr Then, the pairwise Granger causality between series i and the components of is deduced from O; which is estimated by fitting 9, 's output to the ordered measurements in series Specifically, the estimated O suggests that g; output is independent of the past measurements in series j, then
2

---

Published as a conference paper at ICLR 2020
declare that series j is Granger noncausal for series i. We aim design the approximation function 9: to be highly expressive and capable of well-approximmaing any intricate causal coupling between the components of x induced by the component wise function fi, while simultaneously being easily identifiable from underdeternined measurements.
By virtue of their universal approximation property (Schäfer & Zimmermann (006)), recurrent neu- ral networks or RNNs are a particularly ideal choice for 9, towards inferring the pairwise Granger causal relationships in x. In this work, we investigate the use of a special type of RNN called the statistical recurrent unit (SRU) for inferring pairwise Granger causality between multiple non- linearly interacting stochastic processes. Introduced by Oliva et al.] (2017). an SRU is a highly expressive recurrent neural network designed specifically for modeling multivariate time series data with complex-nonlinen nonlinear dependencies spanning multiple time lags. Unlike the popular gated RNNs (e.g., long short-term memory (LSTM) (Hochreiter & Schmidhuber (1997)) and gated recurrent unit (GRU)) (Chung et al. (2014)), the SRU's design completely devoid of the highly nonlinear sigmoid gating functions and thus less affected by the vanishing/exploding gradient issue during training. Despite its simpler ungated architecture, an SRU can model both short and long-term tem- poral dependencies in multivariate time series. It does so by maintaining multi-time scale summary statistics of the time series data in the past, which are preferentially sensitive to different older por- tions of the time series x. By taking appropriate linear combinations of the summary statisties at different time scales, an SRU is able to construct predictive causal features which can be both highly component-speciinn and lag-specific dual-specificity of the SRU's predictive features is its most desirable feature, as one would argue that causal effects in reality also tend to be highly localized in both space and time.
the same time. From the causal inference perspective, this
The main contributions of this paper can be summarized as follows:
We propose the use of statistical recurrent units (SRUs) for detecting pairwise Granger causality between the nonlinearly interacting stochastic processes. We show that the entire network of pairwise Granger causal relationships can be inferred directly from the regu- larized block-sparse estimate of the input-layer weight parameters of the SRUs trained to predict the time series measurements of the individual processes.
2.
We propose modified SRU architecture called economy SRU or eSRU in short. The first of the two proposed modifications is aimed at substantially reducing the number of trainable parameters in the standard SRU model without sacrificing its expressiveness. The second modification entails regularizing the SRU's internal weight parameters to enhance the inter- pretability of its learned predictive features. Compared to the standard SRU, the proposed eSRU model is considerably less likely overfit the time series measurements.
3. We conduct extensive numerical experiments to demonstrate that eSRU is a compelling model for inferring pairwise Granger causality. The proposed model is found to outperform the multi-layer perceptron (MLP), LSTM and attention gated convolutional neural network (AG-CNN) based models considered in the earlier works.
3 PROPOSED GRANGER CAUSAL INFERENCE FRAMEWORK
In the proposed scheme, each of the unknown generative functions ft, 1 S S. in the presumed component-wise model of x in (I is individually approximated by a distinct SRU network. The 7thh SRU network sequentially processes the time series measurements x and outputs a next-step prediction sequence X (8.2. ....... 81,+11...)) C RR where ki denotes the predicted value of component series at time t + 1. The prediction 84,41 is computed in a recurrent fashion by combining the current input sample x, at time t with the summary statisties of past samples of up to and including time - as illustrated in Figure
The following update equations describe the sequential processing of the input time series within the tthh SRU network in order to generate a prediction of x,,+11
Feedback: ri,t = h (w
uu + b(th) E ERR.
(3a)
Recurrent statistics: Φi., = h (w)())
in Xt + W r + (000
ERde
(3b)
3

---

Published as a conference paper at ICLR 2020
Multi-scale summary statistics: ui, = (uft)) (ug))* (u. (u.r)
E IRnds aj S.A Vj.
(3c)
Single-scale summary statistics: u,, = (1- -1y 2 + tayasa E ER [0,
(3d)
Output features: Oi, = / ( Uie u 69")) E ER4
(3c)
(1) T Output prediction: x,,++1 = w
o + b R.
(3f)
The function J in the above updates is the elementwise Rectified Linear Unit (ReLU) operator, h(-) = max(-,0) 0), which serves as the nonlinear activation in the three dedicated single layer neural networks that generate the recurent statisties di,,, the feedback F4, and the output fea- tures Oi, in the 7th SRU network. In order to generate the next-step prediction of series i at time I, the 7th SRU network first prepares the feedback ri, by nonlinearly transforming its last hidden state Ui As stated equation Ba single layer ReLU network parameterized by weight matrix (! and bias (3) maps the hidden state u1 to the feedback ri,-- Another single layer ReLU network parameterized by weight matrices w input x, and the feedback ri, and tranforms them into the recurrent statistics Φi,r described in equation 36 Equation 3d describes how the network's multi-timescale hidden states u,,, for E A [a. Q2.......... am C [0. 1] are updated in parallel by taking exponentially weighted moving averages of the recurrent statistics Φi,c corresponding m different scales A. A third single layer ReLU network parameterized by (! and be' transforms the concatenated multi- timescale summary statistics u,, = [uft) (u?²) (ufm) )T to generate the nonlinear causal features Oir which, according to Oliva et al. (2017), are arguably highly sensitive to the input time series measurements at specific lags. Finally, the network generates the next-step prediction se- ries as x, 1+1 by linearly combining the nonlinear output features in O,,t, depicted in equation 3f
in w and bias b takes the
x.
Concatenate
Hag
[x F.cl
dut
K,443
ReLU
ReLU U
Linear
Hay (a)
Figure 1: The 7th SRU approximat- ing ft in equation]
Ho. (2)
Fst
H,(2) = 1- 0)2
ReLU
W
For values of scale 1, the single-scale summary statistic u., in equation 3d is more sensitive to the recent past measurements x. On the other hand, a 0 yields a summary statistic that more representative of the older portions of the input time series. Oliva al. (211)) claborates how the SRU able to generate output features (O,,,, 1 I S n)) that are preferentially sensitive to the measurements from specific past segments of x by taking appropriate linear combinations of the summary statistics corresponding to different values of a in A.
3.1 INFERRING PAIRWISE GRANGER CAUSALITY USING SRUS
Let (1)
w( ( ro.", n19) denote the complete of param- eters of the th SRU network approximating fa the presumed component-wise model of x. From equation 3b we observe that the weight matrix w-(() regulates the influence of the individual com- the generation of the recurrent statistics diis, and ultimately
SRU ( w (1)
(
ponents of the input time series the next-step prediction of series i. In real-world dynamical systems, the networked interactions are typically sparse which implies that very few dimensions of the input time series x actually play role in the generation of its individual components. Bearing this property of the networked interac-
tions in mind, interested in learning the parameters (6) such that the Hth SRU's sequential
4

---

Published as a conference paper at ICLR 2020
output closely matches with series i's measurements, while simultaneously seeking a column-sparse estimate of the weight matrix W
We propose to learn the parameters O SRU of the 5th SRU network by minimizing the penalized mean squared prediction error loss as shown below.
SRL == arg min
1
Σ (Rie x1,0+1) A I (3.))l
(4)
e e
7-1
In the above, the network output &i, depends nonlinearly on W according to the composite relation described by the updates (3a-431 and w (:.j) denotes the jth column in the weight matrix w The f-group norm penalty in the objective is known to promote column sparsity in the estimated w (Simon et al. 2013)). From equation 3b a straightforward implication of the in column vector W being estimated the all-zeros vector is that the past measurements in series j do not influence the predicted future value of series i. In this case, we declare that series j does not Granger-cause series Moreover, the index set supporting the non-zero columns in the estimated weight matrix V enumerates the components of x which are likely to Granger-cause series i. Likewise, the entire network of pairwise Granger causal relationships in
be deduced
in from the non-zero column support of the estimated weight matrices ((()) 1 S S n in the n SRU networks trained to predict the components of x.
The component- wisse wise SRU optimization problem in equation is nonconvex and potentially has mul- tiple local minima. To solve for (1)
SRU we use first-order gradient-based methods such stochastic gradient descent which have been found to be consistently successful in finding good solutions of nonconvex deep neural network optimization problems (Allen-Zhu et al. (2019)). Since our ap- proach of detecting Granger noncausality hinges upon correctly identifying the all-zero columns of W(()) it is important that the first-order gradient based parameter updates used for minimizing in
the penalized SRU loss ensure that majority of the coefficients in w(() iterates become exactly zero after certain number of iterations. Seeking exact column sparsity in the converged solution
of ( we follow the same approach as Tank et al. (018)) and resort to first-order proximal gra- dient descent algorithm to find regularized solution of the SRU optimization. The gradients needed for executing the gradient descent updates of the SRU network parameters are computed efficiently using the backpropagation through time (BPTT) procedure (Jaeger (2002)).
ECONOMY SRU: A REMEDY FOR OVERFITTING
By computing the summary statistics of past measurements at sufficiently granular time scales, an SRU can learn predictive causal features which are highly localized in time. While higher granularity of a in A translates to a more general SRU model that fits better to the time series measurements, it also entails substantial increase in the number of trainable parameters. Since measurement scarcity is typical in causal inference problems, the proposed component-wiss SRU based time series prediction model is usually overparameterized and thus susceptible to overfitting. The typical high dimensionality of the recurrent statistic d, accentuates this issue.
To alleviate the overfitting concerns, we propose two modifications to the standard SRU (Oliva et al.) (217)) aimed primarily at reducing its likelihood of overfitting the time series measurements. The modifications are relevant regardless of the current Granger causal inference context, and henceforth we refer to the modified SRU Economy-SRU (eSRU).
4.1 MODIFICATION-I: GENERATING FEEDBACK FROM A LOW-DIMENSIONAL SKETCH OF SUMMARY STATISTICS
We propose to reduce the number of trainable parameters in the th SRU network by substitut- ing the feedback ReLU network parameterized by !" and ( with the two stage network shown in Fig. 2 The first stage implements the linear matrix-vector multiplication operation
((() Uze to generate the output vú, RRo where ( Xdxdd is fixed, full row-rank
5

---

Published as a conference paper at ICLR 2020
matrix with df < mdg. The d' d'-dimensional output of the first stage can be viewed low- dimensional, stable embedding of the multi-timescale summary statistics u,e. The entries of the constant matrix D are drawn independently from zero mean Gaussian distribution with vari- ance x. The stage-l processing is based on the premise that for most real-world systems and the associated time series measurements, their high- dimensional summary statistics learned by the SRU network us tend to be highly structured, and thus 4j,t has significantly fewer degrees of free- dom relative its ambient dimension. Thus, by projecting the mdd-ddimessional Ua, onto the d mdg) rows of D we obtain its liw-dimensional embedding Vi, which nonetheless retains most of the contextual information conveyed by the un- compressed u. Johnson & Lindenstrauss, (1984); Dirksen (2014) The second stage of the proposed feedback network is singlc/multi-layer ReLU net- work which maps the sketched summary statis- ties di to the feedback vector ri,e- The second stage ReLU network is parameterized by weight matrix w/:00) E 1d,, and bias B/-(0) E RR4. Compared to the standard SRU's feedback whose generation is controlled by mdgd, + de trainable parameters, the proposed feedback network has only d,d, + d, trainable parameters, which is substantially fewer when d' < mdg. Consequently, the modified SRU less susceptible to overfitting.
Stage-1 (Encoder)
Stage-2 (Decoder)
Vi--1
Linear
RéLU
D
w (
Figure Proposed two-stage feedback in economy-SRU.
4.2 MODIFICATION-II: GROUPED-SPARSE MIXING OF MULTI-SCALE SUMMARY STATISTICS FOR LEARNING TIME-LOCALIZED PREDICTIVE FEATURES
In the standard SRU proposed by Oliva et al. 2017). there are no restrictions on the weight ma- trix ( parameterizing the ReLU network that maps the summary statistics u,, the final predic- tive features in o, Noting that the number of parameters in the md. x d,, sized weight matrix w usually dominates the overall number of trainable parameters in the SRU, any meaningful effort wards addressing the model overfitting concerns must consider regularizing the weights in ! In this spirit, we propose the following penalized optimization problem estimate the parameters e @iSRU proposed in Section 4.1
dSRU = (e (w((") }) U (W.")) of the cSRU model equipped with the two-stage feedback
T-1
d
de
( eSRU
arg min
(Rig xi,441) )2 +
Σ j=
w (j. 9j.x)) ||12
7
ISRU
(5) Here a and X2 are positive constants that bias the group sparse penalizations against the eSRU's fit to the measurements in the 2h component series. The term Wi G,,,) (1 و S do, 1 S S do)
denotes the subvector of the jth row in ( obtained by extracting the weight coefficients indexed by set Gj,A- As shown via an example in Fig. 3 the index set Gj.A enumerates the m weight coefficients in the vector w(') (j,:) which are multiplied to the exponentially weighted running averages of 4thh recurrent statistic (k,((k) corresponding to the m timescales in A prior to being transformed by the neural unit generating the jih predictive feature in O Compared to equation the second penalty term in equation promotes a group-sparse solution for ( the effect that each predictive feature in o,, depends on only a few components of the recurrent statistic di,t via their linearly mixed multi-scale exponentially weighted averages. We opine that the learned linear
mixtures, represented by the intermediate products ( (j. (.,,))))))) are highly sensitive to certain past segments of the input time series x. Consequentl, the output features in o,, are both time-localized and component-speciffe
common trait of real-world causal effects.
"Gaussian random matrices of appropriate dimensions are approximately isometries with overwhelming probability (Johnson& & Lindenstrauss (1944). However, instead of using n independent instantiations of
Gaussian random matrix for initializing D. 1 i < n, we recommend initializing them with the same random matrix, as the latter strategy reduces the probability that any one of them is spurious encoder by n-fold.

---

Published as a conference paper at ICLR 2020
Vn summarr statisties evaluated using
different scales
W
3.3
26
15(m-11+11
Node-j
Wa
o,(
(0,
Figure 3: An illustration of the proposed group-wise mixing of the multi-timesee summary statis-
ties u in the 1h SRU (with da = 5) towards generating the jth predictive feature in Or, The weights corresponding to the same colored connections belong the same group.
The above group-sparse regularization of the weight coefficients in W. combined with the column-sparsity of w is pivotal enforcing that the occurrence of any future pattern in a time series can be attributed to the past occurrences of a few highly time-localized patterns in the an- cestral time series. The results of our numerical experiments further confirm that by choosing X and X2 appropriately, the proposed group-wise sparsity inducing regularization of w! ameliorates overfitting, and the optimization in equation 5 yields an estimate of w whose column support closely reflects the true pairwise Granger causal relationships between the components of x.
5 EXPERIMENTS
We evaluate the performance of the proposed SRU- and eSRU-based component-wist time series models in inferring pairwise Granger causal relationships in a multivariate time series. The proposed models are compared the existing MLP- and LSTM-based models in Tank et al. (2018) and the attention-gaten ENN-based model (referred hereafter as Temporal Causal Discovery Framework (TCDF)) in Nauua al. (2019). To ensure parity between the competing models, the maximum size of all the inpu//hidden/output layers in the different NN/RNN time series models is fixed to 10, unless specified otherwise. The complete list tuned hyperparameters of the considered models used for different datasets is provided in Appendix G The performance of each method is qualified in terms of its AUROC (Area Under the Receiver Operating Characteristic curve). Here, the ROC curve illustrates the trade off between the true- siee-positive rate (TPR) and the false- calse-positive rate (FPR) achieved by the methods towards the detection n² pairwise Granger causal relationships between the n measured processes in the experiment. The ROC curves of SRU and eSRU models obtained by sweeping through different values of the regularization parameter X in equation[aand equation respectively. Likewise, the ROCs of component-wer varying A1 's counterpart in Tank et al. (2018). For TCDF, the ROC curve is obtained by varying the threshold that is applied attention scores of the trained AG-CNN model Nauta al. (2019).
MLP and LSTM models are obtained by
5.1 LORENZ-96 SIMULATIONS
In the first set of experiments, the time series measurements x intended for Granger causal inference are generated according to the Lorenz-96 model which has been extensively used in climate science for modeling and prediction purposes (Schneider et al. (2017)). In the Lorenz-96 model of an n- variable system, the individual state trajectories of the n variables are governed by the following set of odinary differential equations:
Ox,, Ot
X1,1-1 (X1,6-2 x1,411) X1. FF, 1sisn.
(6)
where the first and the second terms on the RHS represent the advection and the diffusion in the system, respectively, and the third term F is the magnitude of the external forcing. The system dynamics becomes increasingly chaotic for higher values of F (Karimi & Paul (0100). We evaluate and compare the accuracy of the proposed methods in inferring pairwise Granger causal relation- ships between n = 10 variables with Lorenz-96 dynamics. We consider two settings: F = 10 and F = 40 in order to simulate two different strengths of nonlinearity in the causal interactions between

---

Published as a conference paper at ICLR 2020
Table Averaged AUROC for 5 independent Lorenz-96 datasets
(a) F= 10
(b) F = 40
MODEL
AVERAGE AUROC
MODEL
AVERAGE AUROC
T = 250
T = 500
T 250
T= 500
MLP
0.93 0.02
0.96 0.03
MLP
0.85
0.94 0.03
LSTM
0.90 + 0.02
0.95 + 0.05
LSTM
0.78
0.90 0.00
TCDF
0.70 + 0.01
0.72 0.04
TCDF
0.62
0.68 + 0.04
SRU
0.84
0.90 0.02
SRU
1.0
1.0
eSRU
0.95 0.02
0.98 0.01
eSRU
0.99 0.0
1.0
the variables. Here, the ground truth is straightforward i.c., for any 1 1 n, the 2th component of time series x is Granger caused by its components with time indices from i to
In the case of weak nonlinear interactions (F 10), from Table Ta we observe that eSRU achieves the highest AUROC among all competing models. The gap in performance is more pronounced when fewer time series measurements
case of stronger nonlinear inter- actions (F = 40), we observe that both SRU and eSRU are the only models that are able to perfectly recover the true Granger causal network (Table [b]. Surprisingly, the SRU and eSRU models per- form poorer when F is small. This could be attributed to the proposed models not sufficiently regularized when fitted to weakly-interacting time series measurements that are less nonlinear.
250) are available.
5.2
VAR SIMULATIONS
In the second set of simulations, we consider the time series measurements x be generated ac- cording to a 3rd order linear VAR model:
X, = A (1),111 A (2) X1-2 A x
x1-3 We: t 22
where the matrices A = 1,2,3 contain the regression coefficients which model the linear in- teractions between its n 10 components. The noise term W; is Gaussian distributed with zero mean and covariance 0.01I. We consider sparse network of Granger causal interactions with only 30% of the regression coefficients in A selected uniformly being non-zero and the regression
matrices A' being collectively joint sparse (same setup as in Bolstad et al. (2011)) All non-zero regression coefficients are set equal to 0.0994 which guarantees the stability the simulated VAR process.
From Table 2 we observe that all time series models generally achieve a higher AUROC as the number of measurements available increases. For T 500, the component-win MLP and the proposed eSRU are statistically tied when comparing their average AUROCs. For = 1000, eSRU significantly outperforms the rest of the time series models and is able recover the true Granger causal network almost perfectly.
Table 2: Averaged AUROC for indepen- dently generated VAR datasets
Table 3: Average AUROC corresponding to ferred brain connectivity for 5 human subjects
MODEL
AVERAGE AUROC
MODEL AVERAGE AUROC
T' 500
T' 1000
T 200
MLP
0.94 + 0.03
0.93 + 0.02
MLP
0.81 1: 0.04
LSTM
0.79 0.12
0.89
LSTM
0.70 1: 0.03
TCDF
0.77
0.78
TCDF
0.75 +.05
SRU
0.82 - 0.06
0.91 +: 0.04
SRU
0.78 0.02
<SRU
0.93
0.98 - 0.01
cSRU
0.84 + 0.03
5.3 IN SILICO ESTIMATION OF BRAIN CONNECTIVITY USING BOLD SIGNALS
In the third set of experiments, nections in the human brain from simulated blood oxygenation level dependent (BOLD) imag- ing data. Here, the individual components of comprise T 200 time-ordered samples of
apply the different learning methods to estimate the
8

---

Published as a conference paper at ICLR 2020
the BOLD signals simulated for 15 different brain regions of interest (ROIs) in human subject. To conduct the experiments, we use simulated BOLD time series measurements corre- sponding to the five different human subjects (labelled /eeeeests/tttsseeeeee s m/ indx. html] The generation of
to 6) in the Sim-3.mat file shared at
https: /ttp://wwww. fmrib.r........ b. ox. ac. the Sim3 dataset described in Smith et al. (201T). The goal here is detect the directed connec- tivity between different brain ROIs in the form of pairwise Granger causal relationships between the components of x.
From 3 it is evident that eSRU is more robust to overfitting compared to the standard SRU and detects the true Granger causal relationships more reliably. Interestingly, single-layer cMLP model is found to outperform more complex cLSTM and attention gated-CNN (TCDF) models; however we expect the latter models to perform better when more time series measurements are available.
5.4 DREAM-3 IN SILICO NETWORK INFERENCE CHALLENGE
In the final set of experiments, we evaluate the performance of the different time series models in inferring gene regulation networks synthesized for the DREAM-3 In Silico Network Challenge (Prill et al.] (2010);)) Marbach et al. (2009).. Here, the time series represents the in silico measurements of 100 genes, available for estimating the gene regulatory networks of
the gene expression levels E.coli and yeast. A total of five gene regulation networks are to be inferred (two for E.coli and three for yeast) from the networks' gene expression level trajectories recorded while they recover from 46 different perturbations (each trajectory has 21 time points). All NN/RNN models are implemented with 10 neurons per layer, except for the componentwise MLP model which has 5 neurons per layer. From Table we observe that the proposed SRU and eSRU models are generally more accurate
Table 4: AUROCs for the inferred gene regulatory networks
MODEL
AUROC
E.coli-1
E.coli-2
Yeast-1 1
Yeast-2
Yeast-3
MLP
0.644
0.568
0.585
0.506
0.528
LSTM
0.629
0.609
0.579
0.519
0.555
TCDF
0.614
0.647
0.581
0.556
0.557
SRU
0.657
0.666
0.617
0.575
0.55
eSRU
0.66
0.629
0.627
0.557
0.55
compared to the MLP, LSTM, and attention-gated CNN (TCDF) models in inferring the true gene- gene interactions. For four out of the five gene regulatory networks, either SRU or eSRU was the best performing model among the competing ones.
6 CONCLUSION
In this work, we addressed the problem of inferring pairwise Granger causal relationships between stochastic processes that interact nonlinearly. We showed that the such causality between the pro- cesses can be robustly inferred from the regularized internal parameters of the proposed eSRU-based recurrent models trained to predict the time series measurements of the individal processes. Future work includes:
Investigating the use of other loss functions besides the square error loss which can capture the exogenous and instantancous causal effects in more realistic way.
ii Incorporating unobserved confounding variables/processes in recurrent models.
ii Inferring Granger causality from multi-rate time series measurements.
ACKNOWLEDGEMENTS
This work is supported by a Singapore Ministry of Education (MOE) Tier 2 Grant (R-263-000-C83. 112).

---

Published as a conference paper at ICLR 2020
REFERENCES
Zahra Abbasvandi and Ali Motie Nasrabadi. self-organized recurrent neural network for esti- mating the effective connectivity and its application to EEG data. Computers in Biology and Medicine, 110:93 - 107, 2019.
Zeyuan Allen- Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In Proceedings of the 36th International Conference Machine Learning. volume 97, pp. 242-252, Jun 2019.
E. Baek and W. Brock. A general test for Granger causality: Bivariate model. In Technical Report. Korean Development Institute and University of Lisconsin-Madisonn 1992.
M. T. Bahadori and Y. Liu. An examination of practical Granger causality inference. In Proc. SIAM Int. Conf. Data Min., pp. 467-475, 2013.
Zhidong Bai, Wing-Keung Wong, and Bingzhi Zhang. Multivariate linear and nonlinear causality tests. Mathematics and Computers in Simulation, 81(1):5-17, September 2010.
A. Bolstad, B. D. Van Veen, and R. Nowak. Causal network inference via group sparse regulariza- tion. IEEE Transactions on Signal Processing, 59(6)22628-2411 June 2011.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 2014.
Cees Diks and Valentyn Panchenko. A new statistic and practical guidelines for nonparametric Granger causality testing. Journal of Economic Dynamics and Control, 30(9):1647 1669, 2006.
Cees Diks and Marcin Wolski. Nonlinear Granger causality: Guidelines for multivariate analysis.
Journal of Applied Econometrics, 31(7):1333-1351, 2016.
Sjoerd Dirksen. Dimensionality reduction with subgaussian matrices: A unified theory. Foundations of Computational Mathematics, 16:1367-1336, 2014.
Andrea Duggento, Maria Guerrisi, and Nicola Toschi. Echo state network models for nonlinear Granger causality. bioRxiv, 2019.
André Fujita, Joāo R Sato, Miguel Garay, Rui Yamaguchi, Satoru Miyano, Mari Sogayar, and Carlos E Ferreira. Modeling gene expression regulatory networks with the sparse vector autoregressive model. BMC Systems Biology, 1, Feb 2007.
C. W. Granger. Investigatiig causal relations by econometric models and cross-spectral methods. Econometrica, 37(3):4244338 1969.
Craig Hiemstra and Jonathan D. Jones. Testing for linear and nonlinear Granger causality in the stock price- volume relation. The Journal of Finance, 49(5)::639-1164 1994.
Sepp Hochreiter and Jargen Schmidhuber. Long short-term memory. Neural Computation, 9()): 1735-1780) 1780, 1997.
Herbert Jaeger. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the echo state network approach. GGMDDfoorscunngzzemme Informationstechnik, 5, Jan 2002.
William Johnson and Joram Lindenstrauss. Extensions of lipschitz maps into hilbert space. Con- temporary Mathematics, 26: 06:1892006, 01 1984.
A. Karimi and M. R. Paul. Extensive chaos in the Lorenz-96 model. Chaos: An Interdisciplinary Journal of Nonlinear Science, 20(4):043105, 2010.
Néhémy Lim, Florence d'Acché Buc, Cédric Auliac, and George Michailidis. Operator-valued kernel-based vector autoregressive models for network inference. Machine Learning, 99:489- 513, 2014.
Daniel Marbach, Thomas Schaffter, Claudio Mattiussi, and Dario Floreano. Generating realistic in silico gene networks for performance assessment of reverse engineering methods. Journal of Computational Biology, 16(2):229-2)) 239, 2009.
10

---

Published as a conference paper at ICLR 2020
D. Marinazzo, M. Pellicoro, and S. Stramaglia. Kemne-Grranger causality and the analysis of dy- namical networks. Physical Review E, 77, May 2008.
Meike Nauta, Doina Bucur, and Christin Seifert. Machine Learning and Knowledge Extraction, (1):312-340, Jan 2019.
Junier B. Oliva, Barnabás Póczos, and Jeff Schneider, The Statistical Recurrent Unit. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 2671-2680, Aug 2017.
Robert J. Prill, Daniel Marbach, Julio Saez- Bhez-Rodrig,ez, Peter K. Sorger, Leonidas G. Alexopoulos, Xiaowei Xue, Neil D. Clarke, Gregoire Altan-Bonnet, and Gustavo Stolovitzky. Towards a rig- orous assessment of systems biology models: The DREAM3 challenges. PLOS ONE, 5(2):1-18, Feb 2010.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In Proceedings of the 16th International Conference Artificial Neural Networks Volume Part I, pp. 632-640, 2006.
Tapio Schneider, Shiwei Lan, Andrew Stuart, and João Teixeira. Earth system modeling 2.0: A blueprint for models that learn from observations and targeted high-resolution simulations. Geo- physical Research Letters, 44(24))12,39--12241,, 2017
Anil K. Seth, Adam B. Barrett, and Lionel Barnett. Granger causality analysis in neuroscience and neuroimaging. Journal of Neuroscience, 35(8):3293-3297, 2015.
Yanning Shen, Brian Baingana, and Georgios B. Giannakis. Nonlinear structural vector autoregres- sive models for inferring effective brain network connectivity, arXiv preprint arXiv:1610.06551,,
Oct 2016.
Noah Simon, Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A sparse-group LASSO. Journal of Computational and Graphical Statistics, 22(2))231-244 2013.
Vikas Sindhwani, Ha Quang Minh, and Aurélie C. Lozano. Scalable matrix-valued kernel learning for high-dimensional nonlinear multivariate regression and Granger causality. In Proceedings of the Twenty-Ninth Conference on Uncertainty Artificial Intelligence, pp. 586-595, 2013.
Stephen M. Smith, Karla Miller, Gholamreza Salimi-Khorsien KKorshidii Matthew Webster, Christian F. Beckmann, Thomas E. Nichols, Joseph D. Ramsey, and Mark W. Woolrich. Network modelling methods for FMRI. Neurolmage, 54(2):875 891, 2011.
Peter Spirtes and Kun Zhang. Causal discovery and inference: concepts and recent methodological advances. Applied Informatics, 3(1):3, Feb 2016.
Alex Tank, Ian Cover, Nicholas Foti, Ali Shojaie, and Emily Fox. An interpretable and sparse neural network model for nonlinear Granger causality discovery. NIPS Time Series Workshop, Nov 2017.
Alex Tank, Ian Cover, Nicholas Foti, Ali Shojaie, and Emily Fox. Neural Granger causality for nonlinear time series. urXiv:1802.05844 2018.
Y. Wang, K. Lin. Y. Qi. Q. Lian, Feng. Wu, and G. Pan. Estimating brain connectivity with verytng-length time lags using a recurrent neural network. IEEE Transactions on Biomedical Engineering, 6559)11953 Sep 2018
A REVIEW OF RELATED WORK
Initial efforts in testing for nonlinear Granger causality focused mostly on the nonparameteric ap- proach. Back & Brock (1992) proposed a general statistical test to detect nonlinear Granger causal- ity between two variables under the assumption that the linear VAR modeling of their time series measurements results in i.idd residual errors. The test involves computing correlation integral es- timators of the conditional probabilities concerning distances between carefully selected lead-lag subsequences of the input bivariate time series. Successive works Hieestraa & Jones (1994): Diks & Panchenkol (2006); Bai al. (2010); Diks & Wolski (2016) proposed their improved variants of

---

Published as a conference paper at ICLR 2020
the Back-Brock test. Hiemstra & Jones (1994) modified and extended the original Back-Brock test to allow for weakly dependent residual etrors in linear VAR modeling of the time series measure- ments. Diks & Panchenko (2006) proposed rectified version of the Heimstra-Jones' test statistic, making it unbiased while fixing the issue of overrejection of the null hypothesis. Later works by Bai et al. (20TO); Diks & Wolski (2016) extended the bi-variate test in DDis & Panchenko (2006) to the multivariate setting. The biggest common drawback of these nonparameteric tests is the large sample sizes required to robustly estimate the conditional probabilities that constitute the test statis- tic. Furthermore, the prevalent strategy in these methods of testing each one of the variable-pan -airs individually to detect pairwise Granger causality is unappealing from a computational standpoint, especially when very large number of variables are involved.
In the model driven approach, the Granger causal relationships are inferred directly from the pa- rameters of a data generative model fitted to the time series measurements. Compared to the non- parameteric approach, the model-based inference approach is considerably more sample efficient, however the scope of inferrable causal dependencies is dictated by the choice of data generative model. Nonlinear kernel based regression models have been found to be reasonably effective in test- ing of nonlinear Granger causality. Kernel methods rely on linearization of the causal interactions in a -indull high dimensional feature space; the linearized interactions are subsequently mod- eled using linear VAR model in the feature space. Based this idea, Marinazzo al. (2008) proposes a kernel Granger causality index to detect pairwise nonlinear Granger causality in the mul- tivariate case. In Sindhwani al. (2013); Lim et al. (2014), the nonlinear dependencies in the time series measurements are modeled using nonlinear functions expressible as sums of vector valued functions in the induced reproducing kernel Hilbert space (RKHS) of a matrix-valucs kernel. In Lim et al. (2014), additional smoothness and structured sparsity constraints are imposed on the ker- nel parameters to promote consistency of the time series fitted nonlinear model. [Shen et al. (2016) proposes nonlinear kernel-based structural VAR model to capture instantaneous nonlinear inter- actions. The existing kernel based regression models are restrictive as they consider only additive linear combinations of the RKHS functions to approximate the nonlinear dependencies in the time series. Furthermore, deciding the optimal order of kernel based regression models is difficult as it requires prior knowledge of the mimimum time delay beyond which the causal influences negligible.
By virtue of their universal approximation ability, RNNs offer a pragmatic way forward in modeling of complex nonlinear dependencies in the time series measurements for the purpose of inferring Granger causality. Mutiple recent works by Wang et al. (2018); DDggento et al. (2019); Abbasvandi & Nasrabadi] (2019) have investigated the use of different types of RNNs for inferring nonlinear Granger causal relationships. However, they all adopt the same naive strategy whereby each pair- wise causal relationship is tested individually by estimating its causal connection strength. The strength of the causal connection from series j to series is determined by the ratio of mean- uquan- prediction errors incurred by unrestricted and restricted RNN models towards predicting series i us- ing the past measurement sequences of all n component including and excluding the jHH component alone, respectively. The pairwise testing strategy however does not scale well computationally as the number of component series becomes very large. This strategy also fails to exploit the typical sparse connectivity of networked interactions between the processes which has unlocked significant performance gains in the existing linear methods (Bahadori & Liu (2013: : Bolstad et al. (21TT).
In a recent work by Tank et al. (2018), the pairwise Granger causal relationships are inferred di- rectly from the weight parameters of component-wise MLP or LSTM networks fitted to the time series measurements. By enforcing volumn-sparsitn of the input-layer weight matrices in the fitted MLP/LSTM models, their proposed approach returns a sparsely connected estimate of the underly- ing Granger causal network. Due to its feedforward architecture, traditional MLP network is not well-suited for modeling ordered data such as time series. Tank et al. (2018) demonstrated that the MLP network can learn short range temporal dependencies spanning few time delays by letting the network's 's input stage process multi-lag time series data over sliding windows. However, modeling long-range temporal dependencies using the same approach requires a larger sliding window size which entails an inconvenient increase in the number of trainable parameters. The simulation results in Tank et al. (2018) indicate that MLP models are generally outperformed by LSTM models in extracting the true topology of pairwise Granger causality, especially when the processes interact in a highly nonlinear and intricate manner. While purposefully designed for modeling short and long term temporal dependencies in time series, the LSTM (Hochreiter & Schmidhuber (1997)) is very
12

---

Published as a conference paper at ICLR 2020
general and often too much overparameterized and thus prone overfitting. While using overparam- eterized models for inference is preferable when there is abundant training data available to leverage upon, there several applications where the data available for causal inference is extremely scarce. our opinion that using a simpler RNN model combined with meaningful regularization of the model parameters is the best way forward in inferring Granger causal relationships from underde- termined time series measurements, Building on the ideas put forth by Tank et al. (2018). this paper investigates the use of Statistical Recurrent Units (SRUs) towards inferring Granger causality.
B PROXIMAL GRADIENT DESCENT UPDATES FOR ESTIMATING THE REGULARIZED WEIGHT PARAMETERS IN THE SRU AND eSRU MODELS
Noting that the proximal operator corresponding to mixed f norm (group-norm) is the group- wise soft-thresoolding operator, we use the following proximal gradient-descent updates minimize the 7thh SRU's regularized loss in equation 4
W (1),t+1 (i, (; Sxin (wol) (1,)) -V wi G.. (( Vj [n]]
(8)
Here, ((9)) 1 T-11 (Ri, - xx,+11 is the unregularized SRU loss function, 1) is the gradient- T-1 descent stepsize and SAy7 is the elementwise soft-thresholding operator defined below.
[w A w
|| > Au || ||
(w))
Ven E R...
0,
\1n
The columns of weight matrix w in the 7th eSRU model are also updated in exactly the same fashion as above.
Likewise, the 1th row of the group-norm regularized weight matrix ( in the eSR optimization
equation 5 is updated as shown below.
W(0++ (j.9,,x) Sxnn (w. (0. (I v/V wT (GG, ( ((0())))) Vj 1,2,...dd do (10)
The gradient of the unregularized loss function 4, 1 S sis S n associated with the SRU and eSRU models used in the above updates is evaluated via the backpropagation through time (BPTT) proce- dure (Jaeger (2002) )
C ABLATION STUDY
C.1 CHOICE OF ENCODER FOR eSRU FEEDBACK'S 'S STAGE-1: FIXED OR DATA DEPENDENT
As a possible further enhancement of the proposed eSRU time series model, one may consider learning the encoding map, ( in the feedback path, as trainable parameters of the 5th eSRU. In Table we compare the Granger causality detection performance of this particular eSRU variant and the proposed design wherein ( is taken to be a random matrix with i.i.d. Gaussian entries.
The experimental setup is kept the same as in Section 5 and the entries of D (t) in the eSRU variant are fo-m penalized during training.
We observe that the performance of these two models is statistically tied, which indicates that the randomly constructed ( is able to distill the necessary information from the nigh-dimensional summary statistics -5,-1 required for generating the feedback Tı, Based on these results, we
recommend using the proposed eSRU design with its randomly constructed encoding map D because of its simpler design and reduced training complexity.
C.2 IMPACT OF GROUP-SPARSE REGULARIZATION OF W!
In order to highlight the importance learning tice-localized predictive features in detecting Granger causality, compare the following time series models:
13

---

Published as a conference paper at ICLR 2020
Table 5: Average AUROC for eSRU variants
DATASET
Average AUROC
Randomly constructed D
D. as trainable parameters
Lorenz (T' 250, F 10)
0.95 + 0.02
0.97 + 0.01
Lorenz (I' 500, F' 10)
0.98 0.01
0.99 0.0
Lorenz 250, 40)
0.99
0.98 0.01
Lorenz (T 500, 40)
1.0
1.0
VAR (T- = 500)
0.93 0.05
0.91 + 0.04
VAR (T 1000)
0.98 0.01
0.98 + 0.01
NetSim
0.84 0.03
0.80 0.02
i. proposed eSRU (with group-sparse regularization of W described in Section 4.2
ii eSRU variant with ridge-regularized (
Once again, we use the same experimental settings as mentioned in Section 5 From Table 6 we observe that barring the Lorenz 99 (T-250/500, F=40) datasets, for which nearly perfect re- covery of the Granger causal network is achieved, the average AUROC improves consistently for the other datasets by switching from unstructured ridge regularization to the proposed group-sparse regularization of the output weight matrix ('
Table 6: Performance of eSRU variants with different regularizations of w!
DATASET
Average AUROC
Proposed group-sparse
Ridge regularization of W!
regularization of W:(
Lorenz (T 250, 10)
0.95 0.02
0.93 0.03
Lorenz (T 500, F = 10)
0.98 + 0.01
0.94 + 0.04
Lorenz (T 250, F 40)
0.99 0.0
0.99 0.0
Lorenz (T 500, =40)
1.0 0.0
1.0
VAR 500)
0.93 0.05
0.90 + 0.03
VAR (T' 1000)
0.98 + 0.01
0.96 0.01
NetSim
0.84 + 0.03
0.83 + 0.03
D IMPLEMENTATION DETAILS
Activation function for SRU and eSRU models
While the standard SRU proposed by Oliva et al. (011)) uses ReLU neurons, we found in our numerical experiments that using the Exponential Linear Unit (ELU) activation resulted in better performance. The ELU activation function is defined as
f
0
ELU(x) = aaee- (a(=))=
0.
(11)
x 50""
In our simulations, the constant is set equal one.
Number of neural layers in SRU model To approximate the generative functions f; in equation we consider the simplest ar- chitecture for the SRU networks, whereby the constituent ReLU networks generating the recurrent features, output features and feedback have single layer feedforward design with equal number of neurons.
Number of neural layers in Economy-SRU model
The ReLU networks used for generating the recurrent and output features in the proposed
14

---

Published as a conference paper at ICLR 2020
eSRU model have single-layer feedforward design. However, the second stage of eSRU's modified feedback can either single or multi-layered feedforward network. Provided that d' < md multi multi-layer implementation of the second stage of eSRU's feedback can still have fewer trainable parameters overall compared to the SRU's single layer feedback network. The simulation results in Section 5 are obtained using two-layer ReLU network in the second stage of eSRU's feedback for the DREAM-3 experiments, and while using three-layer design for the Lorenz-96, VAR and NetSim experiments.
Self-interactions in Dream-3 gene networks
The in-silico gene networks synthesized for the DREAM-3 challenge have self- connections. Noting that none of the Granger causal inference methods evaluated in our experiments intentionally suppress the self-interactions the reported AUROC values computed by ignoring any self-connections in the inferred Granger causal networks.
E PYTHON CODES
cMLP & cLSTM models
Pytorch implementation of the componentwise MLP and LSTM models taken from https:
//github. com/Iccc2.1//eerralla
Temporal Causal Discovery Framework (TCDF)
Pytorch implementation of the attention-gated CNN based Temporal Causal Discovery Framework (TCDF) is taken from https: h thub. comtb//aoaMMMa TCDD
Proposed SRU and Economy-SRU models
Pytorch implementations of the proposed componentwise SRU and eSRU models are shared
http:://gthhuo..cmmssshaaeee/// ffr_aaar_RaS__o_____
F
ROC PLOTS
The receiver operating characteristics (ROC) of different Granger causal inference methods are com- pared in Figures 7 Here, ROC curve represents the trade-off between the true-positive rate (TPR) and the false-positive rate (FPR) achieved by given method while inferring the underlying pairwise Granger causal relationships.
15

---

Published as a conference paper at ICLR 2020
0.8
0.5
0.5
1PM
0.4
KLLP
0.4
(NLP
CLTT
<LSTM
TCDF
02
0.2
65RU
eSRU
00
0
0.6
0.8
FPR
FPR
F 10, T 250
(b) F 10, T 500
155
0.8
0.6
1PP
7P%
CLSTH
<LSTN
TCDF
TC.3F
SRU 6SRU
SRU
FPR
(c) F = 40, T 250
(d) F 40, T 500
Figure 4: Average ROC curves for Lorenz-96 datasets
0.6
TPPR
CHLP
<NLP
SRU
SRU
25RU
eSRU
0.2
0.4
0.6
0.
0
0.2
0.4
0.5
0.8
FPR
FPR
(a) 30% sparsity, T 500
(b) 30% spansity, T 1000
Figure 5: Average ROC curves for VAR datasets
16

---

Published as a conference paper at ICLR 2020
0.6
0.4
CNLP
0.2
4SRU
0.2
0.4
0.5
0.8
FPR
Figure 6: Average ROC curves for the NetSim experiment.
0.7
0.6
0.6
0.4
0.3
SHLP
0.3
GMLP
CLSTM
<LSTM
TCDF
TCDF
0.1
FPR
FPR
(a) E.coli-1
(b) E.coli-2
0.6
0.5
0.4
1PR
0.3
0.2
7CDF
TCDF
SRU
0.1
SRU
25RU
0.1
0.2
0.3
0.4
0.5
0.1
FPR
(c) Yeast-1 1
(d) Yeast-2
0.5
0.4
198
0.3
CMLP
CLSTM
4SRU
05
0.1
0.2
0.3
0.4
FPR
(e) Yeast-3
Figure 7: ROC curves for Dream-3 datasets
17

---

Published as a conference paper at ICLR 2020
G TUNED HYPERPARAMETERS
Tables 7 to 11 summarize the chosen hyperparameters and configurations of the different NN/RNN
models used for generating the results reported Section For the Dream-3 experiments, the model configurations used for inferring the E.coli-1 gene regulatory network Prill et al. (010) have been provided in the tables.
Dataset
Parameters
Tuning range
Lorenz (F 10/40)
Dream-3
NetSim
# Neural units per layer
10
10
10
Batch size
250/500/100 10000
500/1000
200
Learning rate
Two-fold cross-
0.0005 (F 10),
0.1
0.0005
0.0005
validation
0.001 10)
[5e-5, Te-1]
Ridge regularization bias
Two-fold
0.232079 (F
0.002
5.0
0.4644159
validation acmos
10.0 40)
[5e-5, 10
Block-sparse regu-
10 (F 10),
[0.0001, 0.01]
[0.1, 100]
[0.1, 3.12)
larization bias (in-
[1, 100 (F 40)
put layer)
# Lags
Training epochs
2000
2000
2000
2000
Table Componentwise MLP model configuration (Refer Tank et al. (2017) for detailed description of the model parameters).
Dataset
Parameters
Tuning strategy
Lorenz 10/40)
VAR
Dream-3
NetSim
# unils per layer
Batch size
250/500/
500/1000
21
200
Learning rile
Two-fold cruss [Ge-5, le-1]
0.0005 (F 10).
0.1
0.0005
0.001
40)
Ridge regularization blas
Two-fold validation across [e-55 10]
0.021544 (F 10),
0.0005
5.0
0.010772
(F = 10)
Group-spanse regu-
[1, 36.234]
00.03162, 0.01]
[o.1, 17.52)
[0.1, 3.166]
larization hias
Truncation
No
# Training epochs
2000
2000
2000
4000
Table 8: Componentwise LSTM model configuration (Refer Tank al. (017)) for detailed descrip- tion of the model parameters).
Dataset
Parameters
Toning strategy
Lorenz (F 10)
Lorenz (F
40)
VAR
Dream-3
NetSin
Kernel
2
Batch size
250/500 11000
/50/500/10
550/500/1000 1000
500/1000
21
200
Layers
[2. 3, 43
2
2
3
3
2
Learning rate
{10 1, 10
10 "y
0.01
0.01
0.001
0.01
0.001
Dilation
(1, 2,44
4
Significance
Training apochs
{1000, 2000, 4000)
2000
2000
2000
4000
2000
Table 9: TCDF's Attention-gale gated CNN model parameters (Refer Nauta al. 2019 for detailed description of the model parameters).
18

---

Published as a conference paper at ICLR 2020
Dataset
Parameters
Tuning strategy
Lorenz 10/40)
(F
VAR
Dream-3
NetSim
units layer
10
10
10
10
(0.0, 0.1,
(0.0. 0.01, 0.1, 0.99)
{0.0, 0.01, 0.1,
(0.0, 0.01, 0.1,
0.99)
0.5, 0.99}
0.99)
I Leaming rate
Two-fold cross-
0.005 (F 10),
0.04
0.005
0.001
validation across
(P== 40)
[5e-4, le-1]
Ridge regularization bias
Two-fold
U.U21541
0.021541
0.461159
validation across
0.464159 (F))
[0.01. 10
Group-spe saann-ppasee regu-
0.1, 1] (F 10),
[0.001, 1]
[0.01, 1.0]
1, 3.12)
larization bias
0.00631 (F
(iaput layer)
Batch size
125
125
Training epochs
1000
Table 10: Componentwise SRU model configuration
Parameters
Tuning strategy
Lorenz
Dream-3
10/40)
# per layer
10
A
{0.0, 0.01, 0.1, 0.99)
{0.0, 0.01, 0.1, 0.99}
(0.05, 0.1,
{0.0, 0.01, 0.1,
0.99)
0.99)
Learning
Two-fold
0.01
0.01
0.001
0.001
validation
across
[5e-4,
Ridge regularization parameter
Two-fold validation 00.01, 10]
cross
0.001 (f000(=100) 10),
0.021544
0.1
0.232
0.043088 (F 40)
Group-sparse regu-
[0.03162, 0.11)
0.03166, 0.3162]
[0.1, 3.102)
(0.1, 3.12)
larization bias A (input
Group-spass spurse reg-
Two-fold validation [0.01, 10]
cross
0.232079 10), 0.928318 (F 40)
0.464159
ularization bias Aa coutput feature
across
# layers ond stage of feed- back network
2
Batch size
125
21
5
# Training epachs
2000
2000
2000
2000
Table 11: Economy-Sii model configuration
19


Reviews:
[{"review_id": "rJgVWTWJ5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "the paper attempts to infer Granger causality between nonlinearly interacting stochastic processes from their time series measurements. instead of using MLP/LSTM etc to to model time series measurement, the paper proposed to use component-wise time series prediction model with Statistical Recurrent Units to model the measurements. they consider a low-dimensional version of SRU, which they call economy-SRU. in particular, they use group-wise regularizing to accompany the particular structure of the model to aid interpretability. they compared the performance with existing models with MLP/LSTM and show some gains in a few examples (but not all.) the proposal is interesting, but the experiment section might need further strengthening. currently, the experimental results do not immediately pop out as showing eSRU particularly useful.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gRwkYpFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed to use SRU for inferring nonlinear granger causality. It also provided two extensions of SRU with regularization to alleviate the issue of overfitting.\n\nSRU was proposed in a previous paper. This paper extended this algorithm for inferring granger causality through applying group sparse regularization, which is a pretty smart design to me. Another major innovation comes from the two modifications to combat the possible overfitting when using very fine-grained scale parameters in SRU.\n\nThe paper was well-written in general. I can follow the paper without problem.\nThe effectiveness of the algorithm depends on the sparse regularization, which lacks theoretical guarantee for non-convex optimization. Based on the experimental results, it doesn't seem a big problem though. But I'd like to see some analysis of the actual sparsity of the weight with the proposed group sparse regularization.\n\nThe experiments seem convincing to me and I really appreciate the authors provided the tuned hyper-parameters in the appendix.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxuKq0CtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose using Statistical Recurrent Units to predict the network for Granger causality. They motivate this choice by the high representation power of SRUs for multivariate time series, the good performance they usually enjoy and as a way to alleviate the vanishing gradient problem. More importantly the particular form of the SRUs gives a very simple predictor and therefore explanability for Granger causality: the authors propose to simply mark serie $i$ as Granger caused by $j$ if the $j$th column of the input mixing matrix of the $g_i$ is non-zero.\nIn order to force whole columns of the input matrix to be negative, the authors use a group regularization on the columns of the input matrix $W_{\\text{in}}$. The resulting problem is then optimized by proximal gradient descent.\nThe second contribution of the authors is eSRU, a smaller (in terms of parameters) variant of SRU in order to prevent overfitting.\nFirst the authors introduce a dimensionality reduction layer before the SRU units by using fixed non-trainable random projections. Then the authors propose adding a regularization term for the output mixing matrix, which represents the bulks of the parameters of the model.\nThe eSRU and SRU causal models presented are then compared to the previous state of the art on several datasets, both synthetic and real, where they manage to reach a new state of the art.\nRegarding the two improvements in eSRU proposed by the authors, I have several questions:\nFirst, regarding the random projections, have the authors tried learning the projections ? While it increases the risk of overfitting it is possible, with enough regularization, that it may help learn good representations. Secondly, as I understand it a single projection is drawn at random for every component. While we know from Johnson-Lindenstrauss's Lemma that this will in average be a good strategy, how stable is the model to spurious projections ?\nRegarding the scarification of the output matrix: the number of parameters is effectively reduced but the computational requirements are still the same. Did the authors try any explicit methods, maybe matrix factorization, to exploit the very specific sparse structure of $W_o$ and reduce the number of operations and parameters ?\nFinally, while the results are undeniable, just observing the non-zero columns of $W_\\text{in}$ is as I understand it a mostly ad hoc rule. It would have been interesting to empirically verify its validity by contrasting it with the results given by following equation 2.\nOverall the paper presents an interesting model for inferring Granger causality. The authors clearly present their two main contributions and verify them using varied datasets.\nThe authors also made a clear and appreciated effort toward reproducibility by including all hyperparameters and implementation details as well as the code, including those of competing techniques.\nThe inclusion of the literature review in the appendix is also greatly appreciated.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a modification of the statistical recurrent unit for modelling mutliple time series and show that it can be very useful in practice for identifying granger causality when the time series are non-linearly related. The contributions are primarily conceptual and empirical. The reviewers agree that this is a useful contribution in the causality literature.
Decision: Accept (Poster)