How to Measure Statistical Causality: A Transfer Entropy Approach with Financial Applications | by Thársis Souza, PhD | Towards Data Science

We’ve all heard the say “correlation does not imply causation”, but how can we quantify causation? This is an extremely difficult and often misleading task, particularly when trying to infer causality from observational data and we cannot perform controlled trials or A/B testing.
Take for example the 2-dimensional system from Fig. 4.1.
At a first glance, one could say that there is no clear relationship or causality between random variables x1 and x2. However, this apparent random system presents a very simple causal relationship defined by the following equations:
A simple non-linearity introduced in the relationship between x2 and x1 was enough to introduce complexity into the system and potentially mislead a naive human.
Fortunately, we can take advantage of statistics and information theory to uncover complex causal relationships from observational data (remember, this is still a very challenging task).
The objectives of this Article are the following:
Introduce a prediction-based definition of causality and its implementation using a vector auto-regression formulation.
Introduce a probabilistic-definition of causality and its implementation using an information-theoretical framework.
Simulate linear and nonlinear systems and uncover causal links with the proposed methods.
Quantify information flow among global equity indexes further uncovering which indexes are driving the global financial markets.
Discuss further applications including the impact of social media sentiment in financial and crypto markets.
We also provide code to reproduce results as part of our Open Source Live Book Initiative.
We quantify causality by using the notion of the causal relation introduced by Granger (Wiener 1956; Granger 1969), where a signal X is said to Granger-cause Y if the future realizations of Y can be better explained using the past information from X and Y rather than Y alone.
The most common definitions of Granger-causality (G-causality) rely on the prediction of a future value of the variable Y by using the past values of X and Y itself. In that form, X is said to G-cause Y if the use of X improves the prediction of Y.
Let 






 be a random variable associated at time 



 while 






 represents the collection of random variables up to time 



.
We consider 















 and 






 to be three stochastic processes. 
Let 















 be a predictor for the value of the variable 



 at time 









.
We compare the expected value of a loss function 












 with the error 




































 of two models: 

The expected value of the prediction error given only 





















































































































The expected value of the prediction error given 






 and 











































































































































The expected value of the prediction error given only 





















































































































The expected value of the prediction error given 






 and 










































































































































In both models, the functions f1(.) and f2(.) are chosen to minimize the expected value of the loss function. In most cases, these functions are retrieved with linear and, possibly, with nonlinear regressions, neural networks etc. Typical forms for g(.) are the l1- or l2-norms.
We can now provide our first definition of statistical causality under the Granger causal notion as follows:
Standard Granger-causality tests assume a functional form in the relationship among the causes and effects and are implemented by fitting autoregressive models (Wiener 1956; Granger 1969).
Consider the linear vector-autoregressive (VAR) equations:
where k is the number of lags considered. Alternatively, you can choose your DL/SVM/RF/GLM method of choice to fit the model.
From Def. 4.1, X does not G-cause Y if and only if the prediction errors of X in the restricted Eq. (4.1)and unrestricted regression models Eq. (4.2) are equal (i.e., they are statistically indistinguishable). A one-way ANOVA test can be utilized to test if the residuals from Eqs. (4.1) and (4.2) differ from each other significantly. When more than one lag k is tested, a correction for multiple hypotheses testing should be applied, e.g. False Discovery Rate (FDR) or Bonferroni correction.
A more general definition than Def. 4.1 that does not depend on assuming prediction functions can be formulated by considering conditional probabilities.
A probabilistic definition of G-causality assumes that 












 and 






 are independent given the 
past information 





















 if and only if



























































































, 
where 


















 represents the conditional probability distribution.
In other words, omitting past information from 



 does not change the probability distribution of 



. This leads to our second definition of statistical causality as follows:
Def. 4.2 does not assume any functional form in the coupling between X and Y. Nevertheless, it requires a method to assess their conditional dependency. In the next section, we will leverage an Information-Theoretical framework for that purpose.
To compute G-Causality, we use the concept of Transfer Entropy. Since its introduction (Schreiber 2000), Transfer Entropy has been recognized as an important tool in the analysis of causal relationships in nonlinear systems (Hlavackovaschindler et al. 2007). It detects directional and dynamical information (Montalto 2014) while not assuming any particular functional form to describe interactions among systems.
One can interpret this quantity as a measure of the dominant direction of the information flow. In other words, a positive result indicates a dominant information flow from X to Y compared to the other direction or, similarly, it indicates which system provides more predictive information about the other system (Michalowicz, Nichols, and Bucholtz 2013).
It has been shown (Barnett, Barrett, and Seth 2009) that linear G-causality and Transfer Entropy are equivalent if all processes are jointly Gaussian. In particular, by assuming the standard measure (l2-norm loss function) of linear G-causality for the bi-variate case as follows:
the following can be proved (Barnett, Barrett, and Seth 2009):
This result provides a direct mapping between the Transfer Entropy and the linear G-causality implemented in the standard VAR framework. Hence, it is possible to estimate the TE both in its general form and with its equivalent form for linear G-causality.
In this section, we construct simulated systems to couple random variables in a causal manner. We then quantify information flow using the methods studied in this Article.
We first assume a linear system, where random variables have linear relationships defined as follow:
We simulate this linear system with the following code:
Fig. 4.2 represents the dependencies of the simulated linear system.
We first define a function that calculates a bi-variate measure for G-causality as defined in Eq. (4.4) as follows:
We use the function calc_te from the R package RTransferEntropy and the previously defined function Linear.GC to calculate pairwise information flow among the simulated variables as follows:
The function FApply.Pairwise is an auxiliary function that simply applies a given function D.Func to all possible pairs of columns from a given matrix X as follows:
Fig. 4.3 A) and B) show Granger-causality and Transfer Entropy among the system’s variables, respectively. A cell (x,y) presents the information flow from variable y to variable x. We observe that both the Granger-causality (linear) and Transfer Entropy (nonlinear) approaches presented similar results, i.e., both methods captured the system’s dependencies similarly. This result is expected as the system is purely linear and Transfer Entropy is able to capture both the linear and nonlinear interactions.
We define a second system by introducing nonlinear interactions between x1 and the variables x2 and x5 as follows:
We simulate this nonlinear system with the following code:
Fig. 4.5 represents the dependencies of the simulated nonlinear system.
We calculate Granger-causality and Transfer Entropy of the simulated nonlinear system as follows:
From Fig. 4.6 A) and B), we observe that the nonlinear interactions introduced were not captured by the linear formulation of Granger-causality. While all linear interactions presented similar linear and nonlinear information flows, the two nonlinear interactions introduced into the system presented relatively higher nonlinear information flow compared to the linear formulation.
The world’s financial markets form a complex, dynamic network in which individual markets interact with one another. This multitude of interactions can lead to highly significant and unexpected effects, and it is vital to understand precisely how various markets around the world influence one another (Junior, Mullokandov, and Kenett 2015).
In this section, we use Transfer Entropy for the identification of dependency relations among international stock market indices. First, we select some of the major global indices for our analysis, namely the S&P 500, the FTSE 100, the DAX, the EURONEXT 100 and the IBOVESPA, which track the following markets, respectively, the US, the UK, Germany, Europe and Brazil. They are defined by the following tickers:
tickers<-c("^GSPC", "^FTSE", "^GDAXI", "^N100", "^BVSP")
Next, we will load log-returns of daily closing adjusted prices for the selected indices as follows (check here to have access to this dataset):
library(xts)dataset<-as.xts(read.zoo('./data/global_indices_returns.csv',                  header=TRUE,                  index.column=1, sep=","))head(dataset)##                        X.GSPC   X.FTSE  X.GDAXI   X.N100   X.BVSP## 2000-01-02 19:00:00  0.000000       NA  0.00000  0.00000  0.00000## 2000-01-03 19:00:00 -0.039099  0.00000 -0.02456 -0.04179 -0.06585## 2000-01-04 19:00:00  0.001920 -0.01969 -0.01297 -0.02726  0.02455## 2000-01-05 19:00:00  0.000955 -0.01366 -0.00418 -0.00842 -0.00853## 2000-01-06 19:00:00  0.026730  0.00889  0.04618  0.02296  0.01246## 2000-01-09 19:00:00  0.011128  0.01570  0.02109  0.01716  0.04279
The influence that one market plays in another is dynamic. Here, we will consider the time period from 01/01/2014 until 08/19/2019 and we will omit days with invalid returns due to missing data using the function NARV.omit from the package IDPmisc as follows:
library(IDPmisc)dataset.post.crisis <- NaRV.omit(as.data.frame(dataset["2014-01-01/"]))
We will calculate pairwise Transfer Entropy among all indices considered and construct a matrix such that each value in the position (i,j) will contain the Transfer Entropy from tickers[i] to tickers[j] as follows:
# Calculate pairwise Transfer Entropy among global indicesTE.matrix<-FApply.Pairwise(dataset.post.crisis, calc_ete)rownames(TE.matrix)<-colnames(TE.matrix)<-tickers
Fig. 4.8 displays the resulting Transfer Entropy matrix. We normalize the Transfer Entropy values by dividing it by the maximum value in the matrix such that all values range from 0 to 1. We observe that the international indices studied are highly interconnected in the period analyzed with the highest information flow going from the US market to the UK market (^GSPC -> ^FTSE). The second highest information flow is going in the opposite direction, i.e., from the UK market to the US market. That’s a result we would expect as the US and the UK markets are strongly coupled, historically.
We also calculate the marginal contribution of each market to the total Transfer Entropy in the system by calculating the sum of Transfer Entropy for each row in the Transfer Entropy matrix, which we also normalize such that all values range from 0 to 1:
TE.marginal<-apply(TE.matrix, 1, sum)TE.marginal.norm<-TE.marginal/sum(TE.marginal)print(TE.marginal.norm)##  ^GSPC  ^FTSE ^GDAXI  ^N100  ^BVSP ##  0.346  0.215  0.187  0.117  0.135
We observe that the US is the most influential market in the time period studied with 34.6% of the total Transfer Entropy followed by the UK and Germany with 21.4% and 18.6%, respectively. Japan and Brazil are the least influential markets with normalized Transfer Entropies of 11.7% and 13.4%, respectively.
An experiment left to the reader is to build a daily trading strategy that exploits information flow among international markets. The proposed thesis is that one could build a profitable strategy by placing bets on futures of market indices that receive significant information flow from markets that observed unexpected returns/movements.
For an extended analysis with a broader set of indices see (Junior, Mullokandov, and Kenett 2015). The authors develop networks of international stock market indices using an information theoretical framework. They use 83 stock market indices of a diversity of countries, as well as their single day lagged values, to probe the correlation and the flow of information from one stock index to another taking into account different operating hours. They find that Transfer Entropy is an effective way to quantify the flow of information between indices, and that a high degree of information flow between indices lagged by one day coincides to same day correlation between them.
Investors’ decisions are modulated not only by companies’ fundamentals but also by personal beliefs, peers influence and information generated from news and the Internet. Rational and irrational investor’s behavior and their relation with the market efficiency hypothesis (Fama 1970) have been largely debated in the economics and financial literature (Shleifer 2000). However, it was only recently that the availability of vast amounts of data from online systems paved the way for the large-scale investigation of investor’s collective behavior in financial markets.
A research paper (Souza and Aste 2016) used some of the methods studied in this Article to uncover that information flows from social media to stock markets revealing that tweets are causing market movements through a nonlinear complex interaction. The authors provide empirical evidence that suggests social media and stock markets have a nonlinear causal relationship. They take advantage of an extensive data set composed of social media messages related to DJIA index components. By using information-theoretic measures to cope for possible nonlinear causal effects between social media and the stock market, the work points out stunning differences in the results with respect to linear coupling. Two main conclusions are drawn: First, social media significant causality on stocks’ returns are purely nonlinear in most cases; Second, social media dominates the directional coupling with stock market, an effect not observable within linear modeling. Results also serve as empirical guidance on model adequacy in the investigation of sociotechnical and financial systems.
Fig. 4.9 shows the significant causality links found between social media and stocks’ returns considering both cases: nonlinear (Transfer Entropy) and linear G-causality (linear VAR framework). The linear analysis discovers only three stocks with significant causality: INTEL CORP., NIKE INC. and WALT DISNEY CO. The Nonlinear analysis discovers that several other stocks have significant causality. In addition to the 3 stocks identified with significant linear causality, other 8 stocks presented purely nonlinear causality.
The low level of causality obtained under linear constraints is in-line with results from similar studies in the literature, where it was found that stocks’ returns show weak causality links (Alanyali, Moat, and Preis 2013, Antweiler and Frank (2004)) and social media sentiment analytics, at least when taken alone, have small or no predictive power (Ranco 2015) and do not have significant lead-time information about stock’s movements for the majority of the stocks (Zheludev, Smith, and Aste 2014). Contrariwise, results from the nonlinear analyses unveiled a much higher level of causality indicating that linear constraints may be neglecting the relationship between social media and stock markets.
In summary, (Souza and Aste 2016) is a good example on how causality can be not only complex but also misleading further highlighting the importance of choice in the methodology used to quantify it.
In (Keskin and Aste 2019), the authors use information-theoretic measures studied in this Article for non-linear causality detection applied to social media sentiment and cryptocurrency prices.
Using these techniques on sentiment and price data over a 48-month period to August 2018, for four major cryptocurrencies, namely bitcoin (BTC), ripple (XRP), litecoin (LTC) and ethereum (ETH), the authors detect significant information transfer, on hourly timescales, in directions of both sentiment to price and of price to sentiment. The work reports the scale of non-linear causality to be an order of magnitude greater than linear causality.
The information-theoretic investigation detected a significant non-linear causal relationship in BTC, LTC and XRP, over multiple timescales and in both the directions sentiment to price and price to sentiment. The effect was strongest and most consistent for BTC and LTC. Fig. 4.10 shows Transfer Entropy results between BTC sentiment and BTC prices.
All analysis for this paper was performed using a Python package (PyCausality), which is available at https://github.com/ZacKeskin/PyCausality.
Untangling cause and effect can be devilishly difficult. However, statistical tools can help us tell correlation from causation.
In this Article, we introduced the notion of Granger-causality and its traditional implementation in a linear vector-autoregressive framework. We then defined information theoretical measures to quantify Transfer Entropy as a method to estimate statistical causality in nonlinear systems.
We simulated linear and nonlinear systems further showing that the traditional linear G-causality approach failed to detect simple non-linearities introduced into the system while Transfer Entropy successfully detected such relationships.
Finally, we showed how Transfer Entropy can be used to quantify relationships among global equity indexes. We also discussed further applications from the literature where Information Theoretical measures were used to quantify causal links between investor sentiment and movements in the equity and crypto markets.
We hope you enjoyed this casual causal journey and remember: Quantify causality, responsibly.