Correlation Is Simple With Seaborn And Pandas | by Jeremiah Lutes | Towards Data Science

Datasets can tell many stories. A great place to start, to see these stories unfold, is checking for correlations between the variables. One of the first tasks I perform when exploring a dataset to see which variables have correlations. This gives me a better understanding of the data I’m working with. It’s also a great way to develop an interest in the data and establish some initial questions to try to answer. Simply put, correlations are awesome.
Luckily Python has some amazing libraries which give us the tools we need to quickly and efficiently look at correlations. Let’s take a brief look at what correlation is and how to find strong correlations in a dataset using a heat map.
Correlation is a way to determine if two variables in a dataset are related in any way. Correlations have many real-world applications. We can see if using certain search terms are correlated to views on youtube. Or, we can see if ads are correlated to sales. When building machine learning models correlations are an important factor in determining features. Not only can this help us to see which features are linear related, but if features are strongly correlated we can remove them to prevent duplicating information.
In data science we can use the r value, also called Pearson’s correlation coefficient. This measures how closely two sequences of numbers( i.e., columns, lists, series, etc.) are correlated.
The r value is a number between -1 and 1. It tells us whether two columns are positively correlated, not correlated, or negatively correlated. The closer to 1, the stronger the positive correlation. The closer to -1, the stronger the negative correlation (i.e., the more “opposite” the columns are). The closer to 0, the weaker the correlation.
The rvalue formula is:
We aren’t going to explain the math behind the r value, but if you are curious, this youtube video does a great job. Instead, let’s visualize correlations with a simple dataset
The dataset below shows data on seven children. It has the following columns, weight, age(in months), amount of baby teeth, and eye color. The eye color column has been categorized where 1 = blue, 2 = green, and 3 = brown.
Let’s make 3 scatter plots using the above data. We’ll look at the following 3 relationships: age and weight, age and baby teeth, and age and eye color.
When we look at the correlation between age and weight the plot points start to form a positive slope. When we calculate the r value we get 0.954491. With the r value so close to 1, we can come to the conclusion that age and weight have a strong positive correlation. Intuitively, this should check out. In a growing child, as they get older and grow they start to weigh more.
Conversely, the plot points on the age and baby teeth scatter plot start to form a negative slope. The r value of this correlation is -0.958188. This signifies a strong negative correlation. Intuitively, this also makes sense. As a child gets older they lose their baby teeth.
On our last scatterplot, we see some plot points with no clear slope. This correlation has an r value of -0.126163. There is no significant correlation between age and eye color. This should also make sense as eye color shouldn’t change as a child gets older. If this relationship showed a strong correlation we would want to examine the data to find out why.
Let’s look at a larger dataset and see how easy it is to find correlations using Python.
The data we’ll look at comes from a Kaggle dataset about movies on streaming platforms. This dataset contains data on which movies are what streaming platforms. It also includes a few various qualifiers about each movie such as name, runtime, IMDB score, etc.
We’ll first import the dataset and turn it into a DataFrame using pandas.
import pandas as pdmovies = pd.read_csv("MoviesOnStreamingPlatforms_updated.csv")
The Rotten Tomatoes column is a string, let’s change the data type to a float.
movies['Rotten Tomatoes'] = movies['Rotten Tomatoes'].str.replace("%" , "").astype(float)
The Type column doesn’t seem to be entered properly, let’s drop it.
movies.drop("Type", inplace=True, axis=1)
Alright now for the fun stuff!
Using the Pandas correlation method we can see correlations for all numerical columns in the DataFrame. Since this is a method, all we have to do is call it on the DataFrame. The return value will be a new DataFrame showing each correlation.
*the corr() method has a parameter that allows you to choose which method to find the correlation coefficient. The Pearson method is the default, but you can also choose the Kendall or Spearman method.
correlations = movies.corr()print(correlations)\\ID      Year      IMDb  Rotten Tomatoes   Netflix  \ID               1.000000 -0.254391 -0.399953        -0.201452 -0.708680   Year            -0.254391  1.000000 -0.021181        -0.057137  0.258533   IMDb            -0.399953 -0.021181  1.000000         0.616320  0.135105   Rotten Tomatoes -0.201452 -0.057137  0.616320         1.000000  0.017842   Netflix         -0.708680  0.258533  0.135105         0.017842  1.000000   Hulu            -0.219737  0.098009  0.042191         0.020373 -0.107911   Prime Video      0.554120 -0.253377 -0.163447        -0.049916 -0.757215   Disney+          0.287011 -0.046819  0.075895        -0.011805 -0.088927   Runtime         -0.206003  0.081984  0.088987         0.003791  0.099526Hulu  Prime Video   Disney+   Runtime  ID              -0.219737     0.554120  0.287011 -0.206003  Year             0.098009    -0.253377 -0.046819  0.081984  IMDb             0.042191    -0.163447  0.075895  0.088987  Rotten Tomatoes  0.020373    -0.049916 -0.011805  0.003791  Netflix         -0.107911    -0.757215 -0.088927  0.099526  Hulu             1.000000    -0.255641 -0.034317  0.033985  Prime Video     -0.255641     1.000000 -0.298900 -0.067378  Disney+         -0.034317    -0.298900  1.000000 -0.019976  Runtime          0.033985    -0.067378 -0.019976  1.000000
Yikes! That is a lot of numbers. The output has too many columns making it hard to read. This is only the correlations for 9 variables which results in a 9x9 grid. Can you imagine looking at 20 or 30? It would be very difficult.
Things get a little better if we don’t call print and just let Jupyter notebook format the return.
movies.corr()
We could also check each variable individually by slicing using the column name.
print(correlations["Year"])//ID                -0.254391Year               1.000000IMDb              -0.021181Rotten Tomatoes   -0.057137Netflix            0.258533Hulu               0.098009Prime Video       -0.253377Disney+           -0.046819Runtime            0.081984
This is a little easier to read and sufficient if only looking at the correlations for 1 variable. But, there must be an easier way to look at a whole dataset.
Lucky for us, seaborn gives us the ability to quickly generate a heat map.
We simply import seaborn and matplotlib and use seaborn’s heatmap() function.
#always remember your magic function if using Jupyter%matplotlib inlineimport seaborn as snsimport matplotlib.pyplot as pltsns.heatmap(correlations)plt.show()
Wow! How cool is that?
Now we can quickly see a few correlations;
A strong positive correlation between IMDb and Rotten Tomatoes. As well as a strong positive correlation between Prime Video and ID.A slight positive correlation between Netflix and Year.Strong negative correlations between Netflix and ID, Netflix and Prime VideoSlight negative correlations between Year and Prime Video, Disney Plus and Prime Video, Hulu and Prime Video, and Netflix and ID.No correlation between runtime and any of the streaming platforms. No correlation between Netflix and year
A strong positive correlation between IMDb and Rotten Tomatoes. As well as a strong positive correlation between Prime Video and ID.
A slight positive correlation between Netflix and Year.
Strong negative correlations between Netflix and ID, Netflix and Prime Video
Slight negative correlations between Year and Prime Video, Disney Plus and Prime Video, Hulu and Prime Video, and Netflix and ID.
No correlation between runtime and any of the streaming platforms. No correlation between Netflix and year
With that information, we can make a few observations.
With strong positive and negative correlations between ID and two of the platforms it appears, the data was added sequentially with Netflix first and Prime Video last. If we are going to use this data to build a model it would be best to shuffle it before splitting into test and training data.It looks like Netflix has newer movies. This could be a hypothesis to explore.Netflix and Amazon appear to have the highest amount of unique movies when compared to the other streaming platforms. Another hypothesis to explore.The different platforms don’t appear to choose movies based on scores by critics or runtime. Yet another cool hypothesis we can explore.
With strong positive and negative correlations between ID and two of the platforms it appears, the data was added sequentially with Netflix first and Prime Video last. If we are going to use this data to build a model it would be best to shuffle it before splitting into test and training data.
It looks like Netflix has newer movies. This could be a hypothesis to explore.
Netflix and Amazon appear to have the highest amount of unique movies when compared to the other streaming platforms. Another hypothesis to explore.
The different platforms don’t appear to choose movies based on scores by critics or runtime. Yet another cool hypothesis we can explore.
In a matter of seconds, we were able to see how the data was entered and get at least 3 ideas to explore.
Correlations are useful to help explore a new dataset. By using seaborn’s heatmap we easily saw where the strongest correlations are. Now you can go to Kaggle and check out a few more datasets to see what other correlations might spark your interest!