Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality

Granger causality is a widely-used criterion for analyzing interactions in largescale networks . As most physical interactions are inherently nonlinear , we consider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements . Our proposed approach relies on modeling the embedded nonlinearities in the measurements using a component-wise time series prediction model based on Statistical Recurrent Units ( SRUs ) . We make a case that the network topology of Granger causal relations is directly inferrable from a structured sparse estimate of the internal parameters of the SRU networks trained to predict the processes ’ time series measurements . We propose a variant of SRU , called economy-SRU , which , by design has considerably fewer trainable parameters , and therefore less prone to overfitting . The economy-SRU computes a low-dimensional sketch of its high-dimensional hidden state in the form of random projections to generate the feedback for its recurrent processing . Additionally , the internal weight parameters of the economy-SRU are strategically regularized in a group-wise manner to facilitate the proposed network in extracting meaningful predictive features that are highly time-localized to mimic real-world causal events . Extensive experiments are carried out to demonstrate that the proposed economy-SRU based time series prediction model outperforms the MLP , LSTM and attention-gated CNN-based time series models considered previously for inferring Granger causality . 1 INTRODUCTION . The physical mechanisms behind the functioning of any large-scale system can be understood in terms of the networked interactions between the underlying system processes . Granger causality is one widely-accepted criterion used in building network models of interactions between large ensembles of stochastic processes . While Granger causality may not necessarily imply true causality , it has proven effective in qualifying pairwise interactions between stochastic processes in a variety of system identification problems , e.g. , gene regulatory network mapping ( Fujita et al . ( 2007 ) ) , and the mapping of human brain connectome ( Seth et al . ( 2015 ) ) . This perspective has given rise to the canonical problem of inferring pairwise Granger causal relationships between a set of stochastic processes from their time series measurements . At present , the vast majority of Granger causal inference methods adopt a model-based inference approach whereby the measured time series data is modeled using with a suitable parameterized data generative model whose inferred parameters ultimately reveal the true topology of pairwise Granger causal relationships . Such methods typically rely on using linear regression models for inference . However , as illustrated in the classical bivariate example by Baek & Brock ( 1992 ) , linear model-based Granger causality tests can fail catastrophically in the presence of even mild nonlinearities in the measurements , thus making a strong case for our work which tackles the nonlinearities in the measurements by exploring new generative models of the time series measurements based on recurrent neural networks . 2 PROBLEM FORMULATION . Consider a multivariate dynamical system whose evolution from an initial state is fully characterized by n distinct stochastic processes which can potentially interact nonlinearly among themselves . Our goal here is to unravel the unknown nonlinear system dynamics by mapping the entire network of pairwise interactions between the system-defining stochastic processes , using Granger causality as the qualifier of the individual pairwise interactions . In order to detect the pairwise Granger causal relations between the stochastic processes , we assume access to their concurrent , uniformly-sampled measurements presented as an n-variate time series x = { xt : t ∈ N } ⊂ Rn . Let xt , i denote the ith component of the n-dimensional vector measurement xt , representing the measured value of process i at time t. Motivated by the framework proposed in Tank et al . ( 2017 ) , we assume that the measurement samples xt , t ∈ N are generated sequentially according to the following nonlinear , component-wise autoregressive model : xt , i = fi ( xt−p : t−1,1 , xt−p : t−1,2 , . . . , xt−p : t−1 , n ) + et , i , i = 1 , 2 , . . . n , ( 1 ) where xt−p : t−1 , j , { xt−1 , j , xt−2 , j , . . . , xt−p , j } represents the most recent p measurements of the jth component of x in the immediate past relative to current time t. The scalar-valued component generative function fi captures all of the linear and nonlinear interactions between the n stochastic processes up to time t− 1 that decide the measured value of the ith stochastic process at time t. The residual ei , t encapsulates the combined effect of all instantaneous and exogenous factors influencing the measurement of process i at time t , as well as any imperfections in the presumed model . Equation 1 may be viewed as a generalization of the linear vector autoregressive ( VAR ) model in the sense that the components of x can be nonlinearly dependent on one another across time . The value p is loosely interpreted to be the order of the above nonlinear autoregressive model . 2.1 GRANGER CAUSALITY IN NONLINEAR DYNAMICAL SYSTEMS . We now proceed to interpret Granger causality in the context of the above component-wise time series model . Recalling the standard definition by Granger ( 1969 ) , a time series v is said to Granger cause another time series u if the past of v contains new information above and beyond the past of u that can improve the predictions of current or future values of u . For x with its n components generated according to equation 1 , the concept of Granger causality can be extended as suggested by Tank et al . ( 2018 ) as follows . We say that series j does not Granger cause series i if the componentwise generative function fi does not depend on the past measurements in series j , i.e. , for all t ≥ 1 and all distinct pairs xt−p : t−1 , j and x′t−p : t−1 , j , fi ( xt−p : t−1,1 , . . . , xt−p : t−1 , j , . . . , xt−p : t−1 , n ) = fi ( xt−p : t−1,1 , . . . , x ′ t−p : t−1 , j , . . . , xt−p : t−1 , n ) . ( 2 ) From equation 1 , it is immediately evident that under the constraint in equation 2 , the past of series j does not assert any causal influence on series i , in alignment with the core principle behind Granger causality . Based on the above implication of equation 2 , the detection of Granger noncausality between the components of x translates to identifying those components of x whose past is irrelevant to the functional description of each individual fi featured in equation 1 . Note that any reliable inference of pairwise Granger causality between the components of x is feasible only if there are no unobserved confounding factors in the system which could potentially influence x . In this work , we assume that the system of interest is causally sufficient ( Spirtes & Zhang ( 2016 ) ) , i.e. , none of the n stochastic processes ( whose measurements are available ) have a common Granger-causing-ancestor that is unobserved . 2.2 INFERRING GRANGER CAUSALITY USING COMPONENT-WISE RECURRENT MODELS . We undertake a model-based inference approach wherein the time series measurements are used as observations to learn an autoregressive model which is anatomically similar to the componentwise generative model described in equation 1 except for the unknown functions fi replaced with their respective parameterized approximations denoted by gi . Let Θi , 1 ≤ i ≤ n denote the complete set of parameters encoding the functional description of the approximating functions { gi } ni=1 . Then , the pairwise Granger causality between series i and the components of x is deduced from Θi which is estimated by fitting gi ’ s output to the ordered measurements in series i . Specifically , if the estimated Θi suggests that gi ’ s output is independent of the past measurements in series j , then we declare that series j is Granger noncausal for series i . We aim to design the approximation function gi to be highly expressive and capable of well-approximating any intricate causal coupling between the components of x induced by the component-wise function fi , while simultaneously being easily identifiable from underdetermined measurements . By virtue of their universal approximation property ( Schäfer & Zimmermann ( 2006 ) ) , recurrent neural networks or RNNs are a particularly ideal choice for gi towards inferring the pairwise Granger causal relationships in x . In this work , we investigate the use of a special type of RNN called the statistical recurrent unit ( SRU ) for inferring pairwise Granger causality between multiple nonlinearly interacting stochastic processes . Introduced by Oliva et al . ( 2017 ) , an SRU is a highly expressive recurrent neural network designed specifically for modeling multivariate time series data with complex-nonlinear dependencies spanning multiple time lags . Unlike the popular gated RNNs ( e.g. , long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber ( 1997 ) ) and gated recurrent unit ( GRU ) ) ( Chung et al . ( 2014 ) ) , the SRU ’ s design is completely devoid of the highly nonlinear sigmoid gating functions and thus less affected by the vanishing/exploding gradient issue during training . Despite its simpler ungated architecture , an SRU can model both short and long-term temporal dependencies in a multivariate time series . It does so by maintaining multi-time scale summary statistics of the time series data in the past , which are preferentially sensitive to different older portions of the time series x . By taking appropriate linear combinations of the summary statistics at different time scales , an SRU is able to construct predictive causal features which can be both highly component-specific and lag-specific at the same time . From the causal inference perspective , this dual-specificity of the SRU ’ s predictive features is its most desirable feature , as one would argue that causal effects in reality also tend to be highly localized in both space and time . The main contributions of this paper can be summarized as follows : 1 . We propose the use of statistical recurrent units ( SRUs ) for detecting pairwise Granger causality between the nonlinearly interacting stochastic processes . We show that the entire network of pairwise Granger causal relationships can be inferred directly from the regularized block-sparse estimate of the input-layer weight parameters of the SRUs trained to predict the time series measurements of the individual processes . 2 . We propose a modified SRU architecture called economy SRU or eSRU in short . The first of the two proposed modifications is aimed at substantially reducing the number of trainable parameters in the standard SRU model without sacrificing its expressiveness . The second modification entails regularizing the SRU ’ s internal weight parameters to enhance the interpretability of its learned predictive features . Compared to the standard SRU , the proposed eSRU model is considerably less likely to overfit the time series measurements . 3 . We conduct extensive numerical experiments to demonstrate that eSRU is a compelling model for inferring pairwise Granger causality . The proposed model is found to outperform the multi-layer perceptron ( MLP ) , LSTM and attention-gated convolutional neural network ( AG-CNN ) based models considered in the earlier works . 3 PROPOSED GRANGER CAUSAL INFERENCE FRAMEWORK . In the proposed scheme , each of the unknown generative functions fi , 1 ≤ i ≤ n in the presumed component-wise model of x in ( 1 ) is individually approximated by a distinct SRU network . The ith SRU network sequentially processes the time series measurements x and outputs a next-step prediction sequence x̂+i = { x̂i,2 , x̂i,3 , . . . , x̂i , t+1 , . . . } ⊂ R , where x̂i , t+1 denotes the predicted value of component series i at time t + 1 . The prediction x̂i , t+1 is computed in a recurrent fashion by combining the current input sample xt at time t with the summary statistics of past samples of x up to and including time t− 1 as illustrated in Figure 1 . The following update equations describe the sequential processing of the input time series x within the ith SRU network in order to generate a prediction of xi , t+1 . Feedback : ri , t = h ( W ( i ) r ui , t−1 + b ( i ) r ) ∈ Rdr . ( 3a ) Recurrent statistics : φi , t = h ( W ( i ) in xt + W ( i ) f ri , t−1 + b ( i ) in ) ∈ Rdφ . ( 3b ) Multi-scale summary statistics : ui , t = [ ( uα1i , t ) T ( uα2i , t ) T . . . ( uαmi , t ) T ] T ∈ Rmdφ , αj ∈ A , ∀j . ( 3c ) Single-scale summary statistics : uαji , t = ( 1− αj ) u αj i , t−1 + αjφi , t , ∈ R dφ , αj ∈ [ 0 , 1 ] . ( 3d ) Output features : oi , t = h ( W ( i ) o ui , t + b ( i ) o ) ∈ Rdo . ( 3e ) Output prediction : x̂i , t+1 = ( w ( i ) y ) T oi , t + b ( i ) y ∈ R. ( 3f ) The function h in the above updates is the elementwise Rectified Linear Unit ( ReLU ) operator , h ( · ) : = max ( · , 0 ) , which serves as the nonlinear activation in the three dedicated single layer neural networks that generate the recurrent statistics φi , t , the feedback ri , t and the output features oi , t in the ith SRU network . In order to generate the next-step prediction of series i at time t , the ith SRU network first prepares the feedback ri , t by nonlinearly transforming its last hidden state ui , t−1 . As stated in equation 3a , a single layer ReLU network parameterized by weight matrix W ( i ) r and bias b ( i ) r maps the hidden state ui , t−1 to the feedback ri , t . Another single layer ReLU network parameterized by weight matrices W ( i ) in , W ( i ) f and bias b ( i ) in takes the input xt and the feedback ri , t and tranforms them into the recurrent statistics φi , t as described in equation 3b . Equation 3d describes how the network ’ s multi-timescale hidden states uαi , t for α ∈ A = { α1 , α2 , . . . , αm } ⊂ [ 0 , 1 ] are updated in parallel by taking exponentially weighted moving averages of the recurrent statistics φi , t corresponding to m different scales in A . A third single layer ReLU network parameterized by W ( i ) o and b ( i ) o transforms the concatenated multi- timescale summary statistics ui , t = [ ( uα1i , t ) T ( uα2i , t ) T . . . ( uαmi , m ) T ] T to generate the nonlinear causal features oi , t which , according to Oliva et al . ( 2017 ) , are arguably highly sensitive to the input time series measurements at specific lags . Finally , the network generates the next-step prediction of series i as x̂i , t+1 by linearly combining the nonlinear output features in oi , t , as depicted in equation 3f . For values of scale α ≈ 1 , the single-scale summary statistic uαi , t in equation 3d is more sensitive to the recent past measurements in x . On the other hand , α ≈ 0 yields a summary statistic that is more representative of the older portions of the input time series . Oliva et al . ( 2017 ) elaborates on how the SRU is able to generate output features ( oi , t , 1 ≤ i ≤ n ) that are preferentially sensitive to the measurements from specific past segments of x by taking appropriate linear combinations of the summary statistics corresponding to different values of α in A .

Summary: the paper attempts to infer Granger causality between nonlinearly interacting stochastic processes from their time series measurements. instead of using MLP/LSTM etc to to model time series measurement, the paper proposed to use component-wise time series prediction model with Statistical Recurrent Units to model the measurements. they consider a low-dimensional version of SRU, which they call economy-SRU. in particular, they use group-wise regularizing to accompany the particular structure of the model to aid interpretability. they compared the performance with existing models with MLP/LSTM and show some gains in a few examples (but not all.) the proposal is interesting, but the experiment section might need further strengthening. currently, the experimental results do not immediately pop out as showing eSRU particularly useful.