RIP correlation. Introducing the Predictive Power Score
By: Florian Wetschoreck
Tags: Data Science, Open Source, Python, Exploratory Data Analysis, Correlation

Too many problems with the correlation

It is Friday afternoon and your boss tells you that the data delivery surprisingly arrived early — after only 4 weeks of back and forth. This was the missing piece for your predictive model. You’re excited but also a little bit anxious because you know what’s next: exploring the data. All. 45. Columns. This will take many hours but you know it’s worth it because without data understanding you are walking blind. One obvious step is to have a look at all the univariate column distributions. But this won’t be enough.

You ask yourself: what relationships exist between columns?

To answer this question, you just repeat the typical drill: calculate a correlation matrix and check for some surprising relationships. Whenever you are surprised, you take a moment to plot a scatterplot of the two columns at hand and see if you can make any sense of it. Hopefully you can but too often you cannot because you don’t even know what the columns mean in the first place. But this is a story for another day.

After inspecting the correlation matrix you move on and you don’t even know what you don’t know (scary).

Let’s take a moment to review the correlation. The score ranges from -1 to 1 and indicates if there is a strong linear relationship — either in a positive or negative direction. So far so good. However, there are many non-linear relationships that the score simply won’t detect. For example, a sinus wave, a quadratic curve or a mysterious step function. The score will just be 0, saying: “Nothing interesting here”. Also, correlation is only defined for numeric columns. So, let’s drop all the categoric columns. In my last project more than 60% of the columns were categoric, but hey. Never mind. And no, I won’t convert the columns because they are not ordinal and OneHotEncoding will create a matrix that has more values than there are atoms in the universe.

Too many scenarios where the correlation is 0. This makes me wonder if I missed something… (Excerpt from the image by Denis Boigelot)

If you are a little bit too well educated you know that the correlation matrix is symmetric. So you basically can throw away one half of it. Great, we saved ourselves some work there! Or did we? Symmetry means that the correlation is the same whether you calculate the correlation of A and B or the correlation of B and A. However, relationships in the real world are rarely symmetric. More often, relationships are asymmetric. Here is an example: The last time I checked, my zip code of 60327 tells strangers quite reliably that I am living in Frankfurt, Germany. But when I only tell them my city, somehow they are never able to deduce the correct zip code. Pff … amateurs. Another example is this: a column with 3 unique values will never be able to perfectly predict another column with 100 unique values. But the opposite might be true. Clearly, asymmetry is important because it is so common in the real world.

Thinking about those shortcomings of correlation, I started to wonder: can we do better?

The requirements: One day last year, I was dreaming about a score that would tell me if there is any relationship between two columns — no matter if the relationship is linear, non-linear, gaussian or only known by aliens. Of course, the score should be asymmetric because I want to detect all the weird relationships between cities and zip codes. The score should be 0 if there is no relationship and the score should be 1 if there is a perfect relationship. And as the icing on the cake, the score should be able to handle categoric and numeric columns out of the box. Summing it up for all my academic friends: an asymmetric, data-type-agnostic score for predictive relationships between two columns that ranges from 0 to 1.