Partial Correlation Vs. Interaction Information | by Tomer Nahshon | Towards Data Science

Finding relationships between different variables or features in a dataset during a data analysis task is one of the key and fundamental skills that a data scientist should possess.
In this short article, we are going to cover a phenomenon that you may (or may not) have encountered during your research and analysis problems. We are going to create a simple dataset and apply our metrics to spot this phenomenon and discuss it.
Finding the relationships between variables in our dataset is essential to get rid of redundant variables for prediction, inferring causality (if possible!) and have a general sense of how rich our dataset is in explaining our hypotheses.
One of the main issues rising concerns the dependency of different random variables. We can account for this issue using a correlation matrix, linear fit and other different methods to find the relationship and use as we see fit. But what if we have two variables that we believe have a strong correlation between them but the truth is they are both tightly related to a third variable that increases the strength of this relationship?
This issue could ruin our assumptions and force us to tell lies to ourselves (and our superiors...) about our findings.
One of the possible techniques to account for this problem is to use a measure called “partial correlation”.
Partial Correlation, in contrast to Pearson Correlation, takes into account the existence of a third variable and “controlling” it.
What does it mean controlling for a variable?
In causal models, controlling for a variable means binning data according to the measured values of the variable. This is typically done so that the variable can no longer act as a confounder, for example in an observational study or experiment.
The formal way to do this is to conduct a linear regression between both X and Z and Y and Z, collecting the error terms and use the standard Pearson correlation between the error terms. When we think about this, if X and Y are correlated regardless to Z, then the errors of their respective linear regression to Z will be correlated to one another (higher partial correlation). If the error terms are different (lower partial correlation), that means that X and Y have different relationships with the confounding factor (Z) and are influenced by it in two different ways, so the relationship between themselves (X and Y) should be lower when taking Z into account.
Let’s see an example.
First, we are going to import our relevant libraries and create a simple dataset.
import numpy as npfrom scipy import stats, linalgimport matplotlib.pyplot as pltimport randomrandom.seed( 30 )mu1 = 2sigma1 = 2x = np.arange(100)p1 =  np.random.normal(mu1,sigma1,100)p2 = np.square(p1) + 3p3 = np.power(p1,3) + 2plt.figureplt.plot(x, p1, 'r--',x, p2, 'g--',x, p3, 'b--')plt.xlim(0, 40)plt.ylim(-1, 50)
As you can see, I created p2 and p3 such that they are based on p1. p2 is the square of p1 plus a constant and p3 is the power of 3 of p1 plus a constant. Let’s naively check the Pearson Correlation between p2 and p3.
np.corrcoef(p2,p3)array([[1.        , 0.91533677],       [0.91533677, 1.        ]])
So we have a 0.91 correlation, makes sense if we are looking at the blue and green lines in the upper graph, both variables are trending together.
Let’s check the partial correlation of p2 and p3 considering p1 as the control variable using the partial correlation formula we described earlier.
def partial_corr(p1,p2,p3):    p1k = np.expand_dims(p1, axis=1)    p2k = np.expand_dims(p2, axis=1)    p3k = np.expand_dims(p3, axis=1)        beta_p2k = linalg.lstsq(p1k, p2k)[0]  # Calculating Weights (W*)    beta_p3k = linalg.lstsq(p1k, p3k)[0]  # Calculating Weights(W*)    res_p2k = p2k - p1k.dot(beta_p2k) # Calculating Errors    res_p3k = p3k - p1k.dot(beta_p3k) # Calculating Errorscorr = stats.pearsonr(np.squeeze(res_p2k), np.squeeze(res_p3k))[0]return corrpartial_corr(p1,p2,p3)0.7327481408542633
The correlation between p2 and p3 is 0.73. As we can see this correlation is lower than the one we thought before (0.91) since the variable p1 correlates to each one of them in a different way. This, in turn, causes the error terms to be different for both variables and as a result, the partial correlation will be lower.
There could be numerous examples of using this correlation metric. You can see them in behavioral science and in climate change analysis.
Let’s now discuss another similar technique in the case of discrete random variables, the Conditional Mutual Information.
Mutual Information (MI) in information theory describes the mutual dependency between two random variables. It is more general than the Pearson Correlation coefficient in the sense it doesn’t demand linear relationships and real-valued random variables. The idea of MI is closely related to entropy more familiar from information theory.
Now assume we have a third variable, that we would like to take into account in our mutual information estimation between the two variables. We would like to use a technique that, similarly to partial correlation, will modify our estimation and give us a hint about the interaction between the three of them.
Let’s define first another measure called the Conditional Mutual Information(CMI):
We will use a measure called interaction information that is a generalization of MI. The formal definition of interaction information between p1, p2, and p3 is :
I will not get into the rigorous mathematical definition of MI and Conditional MI in this post, but you can check the links I attached before.
Let’s have a short example to understand the intuition:
Say that P1 from our previous example was the number of clouds in the sky measured for 100 days. Let’s also assume that P2 is the level of darkness measured by a discrete lightmeter (amount of darkness is influenced by the amount of clouds) measured for 100 days and P3 is the amount of rain measured in milliliters for 100 days.
Let’s use our variables to calculate MI, Conditional MI and Interaction Information to understand better.
from pyitlib import discrete_random_variable as drvp1 = np.round(p1) # to get integer valuesp2 = np.round(p2) # to get integer valuesp3 = np.round(p3) # to get integer valuesMI = drv.information_mutual(p2,p3)MI3.676603187392168CMI = drv.information_mutual_conditional(p2,p3,p1)CMI1.35947929InteractionInformation = MI - CMIInteractionInformation2.317123892473724
So we got a MI (3.67) larger than the Conditional MI ( 1.35) and as a result a positive interaction information (2.31).
Can we explain it?
As we know clouds(p1) cause both rain(p2) and darkness(p3), so the relationship (or the amount of information in our case) between darkness and rain will be influenced by the clouds. so:
This will cause a positive Interaction Information as we defined it earlier!
One ought to be careful with the interpretation of Interaction information since it can also have negative values and the conclusions from it could be tricky (as in correlation/ partial correlation). This is also why it is not often used in the Data Science community as a common metric for relationships evaluation between random variables.
We introduced here two different methods for evaluating interactions between confounding random variables: partial correlation and interaction information. I used both discrete and continuous variables to show application under different settings. These methods are more “subtle” since they demand the underlying familiarity with your data, so be conscious about it.
You can send me comments for this article through zeogod@gmail.com or follow my Github page.
References:
Partial Correlation function implementation from Fabian PedregosaWikipediaComparison of Correlation, Partial Correlation, and Conditional Mutual Information for Interaction Effects Screening in Generalized Linear Models, Ji Li, University of Arkansas, Fayetteville 2018— Linkhttps://pypi.org/project/pyitlib/ — python package for handling discrete random variables and information theory functions
Partial Correlation function implementation from Fabian Pedregosa
Wikipedia
Comparison of Correlation, Partial Correlation, and Conditional Mutual Information for Interaction Effects Screening in Generalized Linear Models, Ji Li, University of Arkansas, Fayetteville 2018— Link
https://pypi.org/project/pyitlib/ — python package for handling discrete random variables and information theory functions