The Best Exploratory Data Analysis with Pandas Profiling | by Matt Przybyla | Towards Data Science

IntroductionOverviewVariablesInteractionsCorrelationsMissing ValuesSampleSummaryReferences
Introduction
Overview
Variables
Interactions
Correlations
Missing Values
Sample
Summary
References
There are countless ways to perform exploratory data analysis (EDA) in Python (and in R). I do most of mine in the popular Jupyter Notebook. Once I realized there was a library that could summarize my dataset with just one line of code, I made sure to utilize it for every project, reaping countless benefits from the ease of this EDA tool. The EDA step should be performed first before executing any Machine Learning models for all Data Scientists, therefore, the kind and intelligent developers from Pandas Profiling [2] have made it easy to view your dataset in a beautiful format, while also describing the information well in your dataset.
The Pandas Profiling report serves as this excellent EDA tool that can offer the following benefits: overview, variables, interactions, correlations, missing values, and a sample of your data. I will be using randomly generated data to serve as an example of this useful tool.
The overview tab in the report provides a quick glance at how many variables and observations you have or the number of rows and columns. It will also perform a calculation to see how many of your missing cells there are compared to the whole dataframe column. Additionally, it will point out duplicate rows as well and calculate that percentage. This tab is most similar to part of the describe function from Pandas, while providing a better user-interface (UI) experience.
The overview is broken into dataset statistics and variable types. You can also refer to warnings and reproduction for more specific information on your data.
I will be discussing variables, which are also referred to as columns or features of your dataframe
To achieve more granularity in your descriptive statistics, the variables tab is the way to go. You can look at distinct, missing, aggregations or calculations like mean, min, and max of your dataframe features or variables. You can also see the type of data you are working with (i.e., NUM). Not pictured is when you click on ‘Toggle details’. This toggle prompts a whole plethora of more usable statistics. The details include:
Statistics — quantile and descriptive
quantile
Minimum5th percentileQ1MedianQ395th percentileMaximumRangeInterquartile range (IQR)
descriptive
Standard deviationCoefficient of variation (CV)KurtosisMeanMedian Absolute Deviation (MAD)SkewnessSumVarianceMonotonicity
These statistics also provide similar information from the describe function I see most Data Scientists using today, however, there are a few more and it presents them in an easy-to-view format.
Histograms
The histograms provide for an easily digestible visual of your variables. You can expect to see the frequency of your variable on the y-axis and fixed-size bins (bins=15 is the default) on the x-axis.
Common Values
The common values will provide the value, count, and frequency that are most common for your variable.
Extreme Values
The extreme values will provide the value, count, and frequency that are in the minimum and maximum values of your dataframe.
The interactions feature of the profiling report is unique in that you can choose from your list of columns to either be on the x-axis or y-xis provided. For example, pictured above is variable A against variable A, which is why you see overlapping. You can easily switch to other variables or columns to achieve a different plot and an excellent representation of your data points.
Sometimes making fancier or colorful correlation plots can be time-consuming if you make them from line-by-line Python code. However, with this correlation plot, you can easily visualize the relationships between variables in your data, which are also nicely color-coded. There are four main plots that you can display:
Pearson’s r
Spearman’s ρ
Kendall’s τ
Phik (φk)
You may only be used to one of these correlation methods, so the other ones may sound confusing or not usable. Therefore, the correlation plot also comes provided with a toggle for details onto the meaning of each correlation you can visualize — this feature really helps when you need a refresher on correlation, as well as when you are deciding between which plot(s) to use for your analysis
As you can see from the plot above, the report tool also includes missing values. You can see how much of each variable is missing, including the count, and matrix. It is a nice way to visualize your data before you perform any models with it. You would preferably want to see a plot like the above, meaning you have no missing values.
Sample acts similarly to the head and tail function where it returns your dataframe’s first few rows or last rows. In this example, you can see the first rows and last rows as well. I use this tab when I want a sense of where my data started and where it ended — I recommend ranking or ordering to see more benefit out of this tab, as you can see the range of your data, with a visual respective representation.
I hope this article provided you with some inspiration for your next exploratory data analysis. Being a Data Scientist can be overwhelming and EDA is often forgotten or not practiced as much as model-building. With the Pandas Profiling report, you can perform EDA with minimal code, providing useful statistics and visualizes as well. That way, you can focus on the fun part of Data Science and Machine Learning, the model process.
To summarize, the main features of Pandas Profiling report include overview, variables, interactions, correlations, missing values, and a sample of your data.
Here is the code I used to install and import libraries, as well as to generate some dummy data for the example, and finally, the one line of code used to generate the Pandas Profile report based on your Pandas dataframe [10].
# install library #!pip install pandas_profilingimport pandas_profilingimport pandas as pdimport numpy as np# create data df = pd.DataFrame(np.random.randint(0,200,size=(15, 6)), columns=list('ABCDEF'))# run your report!df.profile_report()# I did get an error and had to reinstall matplotlib to fix
Please feel free to comment down below if you have any questions or have used this feature before. There is still some information I did not describe, but you can find more of that information on the link I provided from above.
Thank you for reading, I hope you enjoyed!
[1] M.Przybyla, Screenshot of Pandas Profile Report correlations example, (2020)
[2] pandas-profiling, GitHub for documentation and all contributors, (2020)
[3] M.Przybyla, Screenshot of Overview example, (2020)
[4] M.Przybyla, Screenshot of Variables example, (2020)
[5] M.Przybyla, Screenshot of Interactions example, (2020)
[6] M.Przybyla, Screenshot of Correlations example, (2020)
[7] M.Przybyla, Screenshot of Missing Values example, (2020)
[8] M.Przybyla, Screenshot of Sample example, (2020)
[9] Photo by Elena Loshina on Unsplash, (2018)
[1] M.Przybyla, Pandas Profile report code from example, (2020)