Granger Causal Structure Reconstruction from Heterogeneous Multivariate Time Series

Granger causal structure reconstruction is an emerging topic that can uncover causal relationship behind multivariate time series data . In many real-world systems , it is common to encounter a large amount of multivariate time series data collected from heterogeneous individuals with sharing commonalities , however there are ongoing concerns regarding its applicability in such large scale complex scenarios , presenting both challenges and opportunities for Granger causal reconstruction . To bridge this gap , we propose a Granger cAusal StructurE Reconstruction ( GASER ) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series . In particular , we address the problem through a novel attention mechanism , called prototypical Granger causal attention . Extensive experiments , as well as an online A/B test on an E-commercial advertising platform , demonstrate the superior performances of GASER . 1 INTRODUCTION . Broadly , machine learning tasks are either predictive or descriptive in nature , often addressed by black-box methods ( Guo et al. , 2018 ) . With the power of uncovering relationship behind the data and providing explanatory analyses , causality inference has drawn increasing attention in many fields , e.g . marketing , economics , and neuroscience ( Pearl , 2000 ; Peters et al. , 2017 ) . Since the cause generally precedes its effects , known as temporal precedence ( Eichler , 2013 ) , recently , an increasing number of studies have focused on causal discovery from time series data . They are commonly based on the concept of Granger causality ( Granger , 1969 ; 1980 ) to investigate the causal relationship with quantification measures . In many real-world systems , it is common to encounter a large amount of multivariate time series ( MTS ) data collected from different individuals with shared commonalities , which we define as heterogeneous multivariate time series . The underlying causal structures of such data often vary ( Zhang et al. , 2017 ; Huang et al. , 2019 ) . For example , in the financial market , the underlying causal drivers of stock prices are often heterogeneous across stocks of different plates . Similar phenomenons are also observed in the sales of different products in E-commerce . To this situation , most existing methods have to train separate models for MTS of each individual , which suffer from over-fitting especially given limited training samples . Although some works have been proposed to solve such problem ( Zhang et al. , 2017 ; Huang et al. , 2019 ) , they lack the inductive capability to do inferences for unseen samples and fall short of fully exploiting shared causal information among the heterogeneous data which often exist in practice . For instance , the causal structures of the products belonging to the same categories are usually similar . Such shared information presents opportunities for causal reconstruction to alleviate overfitting and to do inductive reasoning . However , it is also challenging to detect common and specific causal structures simultaneously . In this paper , we propose a Granger cAusal StructurE Reconstruction ( GASER ) framework for inductive Granger causality learning and common causal structure detection on heterogeneous multivariate time series data . Our approach builds on the idea of quantifying the contributions of each variable series into the prediction of target variable via a novel designed prototypical Granger causal attention mechanism . In order to ensure that the attention capturing Granger causality , we first design an attention mechanism based on Granger causal attribution of the target series and then perform prototype learning that generates both shared and specific prototypes to improve the model ’ s robust- ness . Extensive experiments demonstrate the superior causal structure reconstruction and prediction performances of GASER . In summary , our specific contributions are as follows : • A novel framework that inductively reconstructs Granger causal structures and uncovers common structures among heterogeneous multivariate time series . • A prototypical Granger causal attention mechanism that summarizes variable-wise contributions towards prediction and generates prototypes representing common causal structures . • Relative extensive experiments on real-world , benchmark and synthetic datasets as well as an online A/B test on an E-commercial advertising platform that demonstrate the superior performance on the causal discovery and the prediction performance comparable to state-of-the-art methods . 2 GASER . In this section , we formally define the problem , introduce the architecture of GASER , present the prototypical Granger causal attention with the final objective function . 2.1 PROBLEM DEFINITION . Assuming we have a set of heterogeneous multivariate time series from N individuals , i.e. , X = { Xi } Ni=1 , with each consisting of S time series of length T , denoted as Xi = ( x1i , x2i , . . . , xSi ) T ∈ RS×T , where xsi = ( xsi,1 , xsi,2 , . . . , xsi , T ) T ∈ RT represents the s-th time series of individual i , and one of them is taken as the target series yi . We aim to train a model that ( 1 ) reconstructs Granger causal structures among variables for each individual ; ( 2 ) generates K common structures among all the N individuals , each structure represented by a prototype pk ∈ RS , k = 1 , ... , K ; and ( 3 ) learns a nonlinear mapping to predict the next value of the target variable series for each individual , i.e. , ŷi , T+1 = F ( Xi ) . 2.2 NETWORK ARCHITECTURE . Our GASER framework consists of two parts : a set of parallel encoders , each predicting the target given the past observations , and an attention mechanism that generates prototypical Granger causal attention vectors to quantify variable-wise contributions towards prediction . Figure 1 illustrates the overall framework of GASER . As illustrated in Figure 1 ( a ) , for an input multivariate time series Xi , the encoder specific to s-th variable projects the time series xsi into a sequence of hidden state , denoted as hsi , t = Hs ( xi , t , hsi , t−1 ) . The encoder could be any RNN models , such as LSTM ( Hochreiter & Schmidhuber , 1997 ) and GRU ( Cho et al. , 2014 ) . The last hidden states , { hsi , T } Ss=1 , are used as the hidden embeddings of each variable . Then the predicted next value of the target variable conditioned on historical data of variable s , denoted as ŷsi , T+1 , can be computed by ŷ s i , T+1 = fs ( h s i , T ) , where fs ( · ) denotes the MLP network specific to variable s. Then we obtain the prediction ŷi , T+1 by aggregating the predicted values specific to variables through the prototypical Granger causal attention described below . 2.3 PROTOTYPICAL GRANGER CAUSAL ATTENTION . We propose a novel attention mechanism in GASER , namely prototypical Granger causal attention , to reconstruct Granger causal relationships for each individual and uncover common causal structures among heterogeneous individuals . The goal is to learn attentions that can reflect the Granger causal strength between variables for each individual , and generate prototypes among heterogeneous individuals . As illustrated in Figure 1 ( b ) , the idea of the prototypical Granger causal attention mechanism is as follows . The Granger causal attribution corresponding to each individual is first computed according to the concept of Granger causality , followed by prototype learning that summarizes common causal structures for heterogeneous individuals in the training set , and produces the attention vector specific to each individual . The details of these two parts are described below . 2.3.1 GRANGER CAUSAL ATTRIBUTION . Granger causality ( Granger , 1969 ; 1980 ) is a concept of causality based on prediction , which declares that if a time series x Granger-causes a time series y , then y can be better predicted using all available information than if the information apart from x had been used . Thus , we obtain the Granger causal attributions by comparing the prediction error when using all available information with the error when using the information excluding one variable series . In particular , given all the hidden embeddings { hsi , T } Ss=1 of individual i , we obtain the embedding that encodes all available information and the one that encodes all available information excluding one variable s , denoted as halli and h all\s i respectively , by concatenating the embeddings of corresponding variables : halli = [ h j i , T ] S j=1 , h all\s i = [ h j i , T ] S j=1 , j 6=s , ( 1 ) where [ · ] represents the concatenation operation . Then we feed them into respective predictors , denoted as gall ( · ) and gs ( · ) , to get the predicted value of target and compute the squared errors : ŷalli , T+1 = gall ( h all i ) , ŷ all\s i , T+1 = gs ( h all\s i ) , ( 2 ) εalli = ( ŷ all i , T+1 − yi , T+1 ) 2 , εall\si = ( ŷ all\s i , T+1 − yi , T+1 ) 2 , ( 3 ) where the predictor gall ( · ) and gs ( · ) can be MLP networks . Inspired by Schwab et al . ( 2019 ) , we define the Granger causal attribution of the target variable corresponding to variable s as the decrease in error when adding s-th series to the set of available information , computed as : ∆εsi = ReLU ( ε all\s i − εalli ) , ( 4 ) where ReLU ( · ) is the rectified linear unit . For each individual i , by normalising the Granger causal attribution , we obtain an attention vector that reflects Granger causality , namely Granger causal attention , denoted as qi . The attention factor for variable s can be computed as : qsi = ∆εsi∑S j=1 ∆ε j i . ( 5 ) 2.3.2 PROTOTYPE LEARNING . The Granger causal attention above is not robust enough to reconstruct Granger causal structure , given limited data ( e.g. , very short time series ) of each individual in training . We address the problem by generating Granger causal prototypes from all the individuals , under the assumption that there should be several common causal structures among heterogeneous individuals . In particular , we assume there existK Granger causal prototypes , denoted as { pk } Kk=1 , and compute the similarity between the Granger causal attention vector qi of individual i and each prototype vectors pk . Since the attention can be seen as a distribution , we use the cosine similarity : dk , i = pk · qi ‖pk‖‖qi‖ , ( 6 ) Then we output a prototype most similar to qi by sampling from the similarity distribution di using Gumbel-Softmax ( Maddison et al. , 2017 ; Jang et al. , 2016 ) , which samples from a reparameterized continuous distribution approximation to the categorical one-hot distribution : e = GumbelSoftmax ( di ) = softmax ( ( log ( di ) + g ) /τ ) , ( 7 ) where GumbelSoftmax ( · ) denotes the Gumbel-Softmax function , e ∈ RK is the sample vector which approaches one-hot , and g is a vector of i.i.d . samples drawn from Gumbel ( 0 , 1 ) distribution . τ is the softmax temperature , and the distribution becomes discrete when τ goes to 0 . With the sample vector e , the output prototype p̂ can be obtained as : p̂ = [ p1 , p2 , . . . , pK ] · e. ( 8 ) After normalizing the sampled prototype , we obtain an attention vector for individual i , denoted as ri , namely prototypical attention . The Granger causal attention reflects the Granger causal structure specific to each individual , while the prototypical attention reflects one common Granger causal structure most similar to the Granger causal structure of each individual . To detect the specific and common causal structures simultaneously , we summarize them together and generate the prototypical Granger causal attention ai as follows : ai = αqi + ( 1− α ) ri , ( 9 ) where α ∈ [ 0 , 1 ] is a hyperparameter that controls the ratio of the two attention mechanism . Finally , the prediction of the target variable ’ s next value can be computed as the weighted sum of the predicted values from all variables : ŷi , T+1 = S∑ s=1 asi ŷ s i , T+1 . ( 10 )

Summary: The paper proposes a novel way of reconstructing Granger causal structures using a differentiable neural network architecture that contains attention modules that are proportional to the Granger causality of the input layers. Furthermore, the architecture blends individual-specific induced causal structures and cross-population prototypical causal structures. The paper has an extensive experimental section on which the proposed method shows impressive improvements in causal discovery performance and predictive performance on par with state-of-the-art.