Correlation Is Simple With Seaborn And Pandas
By: Jeremiah Lutes
Tags: Pandas, Python, Correlation, Data Visualization, Data Science

Correlation Is Simple With Seaborn And Pandas

How to quickly find strong correlations in data using Python, Pandas, and Seaborn’s heatmap function

Photo by NeONBRAND on Unsplash

Datasets can tell many stories. A great place to start, to see these stories unfold, is checking for correlations between the variables. One of the first tasks I perform when exploring a dataset to see which variables have correlations. This gives me a better understanding of the data I’m working with. It’s also a great way to develop an interest in the data and establish some initial questions to try to answer. Simply put, correlations are awesome.

Luckily Python has some amazing libraries which give us the tools we need to quickly and efficiently look at correlations. Let’s take a brief look at what correlation is and how to find strong correlations in a dataset using a heat map.

What is Correlation?

Correlation is a way to determine if two variables in a dataset are related in any way. Correlations have many real-world applications. We can see if using certain search terms are correlated to views on youtube. Or, we can see if ads are correlated to sales. When building machine learning models correlations are an important factor in determining features. Not only can this help us to see which features are linear related, but if features are strongly correlated we can remove them to prevent duplicating information.

How Do You Measure Correlation?

In data science we can use the r value, also called Pearson’s correlation coefficient. This measures how closely two sequences of numbers( i.e., columns, lists, series, etc.) are correlated.

The r value is a number between -1 and 1. It tells us whether two columns are positively correlated, not correlated, or negatively correlated. The closer to 1, the stronger the positive correlation. The closer to -1, the stronger the negative correlation (i.e., the more “opposite” the columns are). The closer to 0, the weaker the correlation.

The rvalue formula is:

Pearson Correlation Coefficient

We aren’t going to explain the math behind the r value, but if you are curious, this youtube video does a great job. Instead, let’s visualize correlations with a simple dataset

The dataset below shows data on seven children. It has the following columns, weight, age(in months), amount of baby teeth, and eye color. The eye color column has been categorized where 1 = blue, 2 = green, and 3 = brown.

Sample dataset

Let’s make 3 scatter plots using the above data. We’ll look at the following 3 relationships: age and weight, age and baby teeth, and age and eye color.

Age and Weight

When we look at the correlation between age and weight the plot points start to form a positive slope. When we calculate the r value we get 0.954491. With the r value so close to 1, we can come to the conclusion that age and weight have a strong positive correlation. Intuitively, this should check out. In a growing child, as they get older and grow they start to weigh more.

Age and Baby Teeth

Conversely, the plot points on the age and baby teeth scatter plot start to form a negative slope. The r value of this correlation is -0.958188. This signifies a strong negative correlation. Intuitively, this also makes sense. As a child gets older they lose their baby teeth.

Age and Eye Color

On our last scatterplot, we see some plot points with no clear slope. This correlation has an r value of -0.126163. There is no significant correlation between age and eye color. This should also make sense as eye color shouldn’t change as a child gets older. If this relationship showed a strong correlation we would want to examine the data to find out why.

Using Python to Find Correlation

Let’s look at a larger dataset and see how easy it is to find correlations using Python.

The data we’ll look at comes from a Kaggle dataset about movies on streaming platforms. This dataset contains data on which movies are what streaming platforms. It also includes a few various qualifiers about each movie such as name, runtime, IMDB score, etc.

Importing and a little cleaning

We’ll first import the dataset and turn it into a DataFrame using pandas.

import pandas as pd

movies = pd.read_csv("MoviesOnStreamingPlatforms_updated.csv")

The Rotten Tomatoes column is a string, let’s change the data type to a float.

movies['Rotten Tomatoes'] =

movies['Rotten Tomatoes'].str.replace("%" , "").astype(float)

The Type column doesn’t seem to be entered properly, let’s drop it.

movies.drop("Type", inplace=True, axis=1)

Alright now for the fun stuff!

Using the corr() method

Using the Pandas correlation method we can see correlations for all numerical columns in the DataFrame. Since this is a method, all we have to do is call it on the DataFrame. The return value will be a new DataFrame showing each correlation.

*the corr() method has a parameter that allows you to choose which method to find the correlation coefficient. The Pearson method is the default, but you can also choose the Kendall or Spearman method.

correlations = movies.corr()

print(correlations) \\ ID Year IMDb Rotten Tomatoes Netflix \

ID 1.000000 -0.254391 -0.399953 -0.201452 -0.708680

Year -0.254391 1.000000 -0.021181 -0.057137 0.258533

IMDb -0.399953 -0.021181 1.000000 0.616320 0.135105

Rotten Tomatoes -0.201452 -0.057137 0.616320 1.000000 0.017842

Netflix -0.708680 0.258533 0.135105 0.017842 1.000000

Hulu -0.219737 0.098009 0.042191 0.020373 -0.107911

Prime Video 0.554120 -0.253377 -0.163447 -0.049916 -0.757215

Disney+ 0.287011 -0.046819 0.075895 -0.011805 -0.088927

Runtime -0.206003 0.081984 0.088987 0.003791 0.099526 Hulu Prime Video Disney+ Runtime

ID -0.219737 0.554120 0.287011 -0.206003

Year 0.098009 -0.253377 -0.046819 0.081984

IMDb 0.042191 -0.163447 0.075895 0.088987

Rotten Tomatoes 0.020373 -0.049916 -0.011805 0.003791

Netflix -0.107911 -0.757215 -0.088927 0.099526

Hulu 1.000000 -0.255641 -0.034317 0.033985

Prime Video -0.255641 1.000000 -0.298900 -0.067378

Disney+ -0.034317 -0.298900 1.000000 -0.019976

Runtime 0.033985 -0.067378 -0.019976 1.000000

Yikes! That is a lot of numbers. The output has too many columns making it hard to read. This is only the correlations for 9 variables which results in a 9x9 grid. Can you imagine looking at 20 or 30? It would be very difficult.

Output with Jupyter Notebook

Things get a little better if we don’t call print and just let Jupyter notebook format the return.

movies.corr()

Jupyter notebook return

Examining one variable

We could also check each variable individually by slicing using the column name.

print(correlations["Year"])

//

ID -0.254391

Year 1.000000

IMDb -0.021181

Rotten Tomatoes -0.057137

Netflix 0.258533

Hulu 0.098009

Prime Video -0.253377

Disney+ -0.046819

Runtime 0.081984

This is a little easier to read and sufficient if only looking at the correlations for 1 variable. But, there must be an easier way to look at a whole dataset.

Seaborn to the Rescue

Lucky for us, seaborn gives us the ability to quickly generate a heat map.

We simply import seaborn and matplotlib and use seaborn’s heatmap() function.

#always remember your magic function if using Jupyter

% matplotlib inline import seaborn as sns

import matplotlib.pyplot as plt sns.heatmap(correlations)

plt.show()

Wow! How cool is that?

Correlations we found

Now we can quickly see a few correlations;

A strong positive correlation between IMDb and Rotten Tomatoes. As well as a strong positive correlation between Prime Video and ID. A slight positive correlation between Netflix and Year. Strong negative correlations between Netflix and ID, Netflix and Prime Video Slight negative correlations between Year and Prime Video, Disney Plus and Prime Video, Hulu and Prime Video, and Netflix and ID. No correlation between runtime and any of the streaming platforms. No correlation between Netflix and year

Some observations

With that information, we can make a few observations.

With strong positive and negative correlations between ID and two of the platforms it appears, the data was added sequentially with Netflix first and Prime Video last. If we are going to use this data to build a model it would be best to shuffle it before splitting into test and training data. It looks like Netflix has newer movies. This could be a hypothesis to explore. Netflix and Amazon appear to have the highest amount of unique movies when compared to the other streaming platforms. Another hypothesis to explore. The different platforms don’t appear to choose movies based on scores by critics or runtime. Yet another cool hypothesis we can explore.

In a matter of seconds, we were able to see how the data was entered and get at least 3 ideas to explore.

Conclusion

Correlations are useful to help explore a new dataset. By using seaborn’s heatmap we easily saw where the strongest correlations are. Now you can go to Kaggle and check out a few more datasets to see what other correlations might spark your interest!