Modern Exploratory Data Analysis. Review of 4 libraries for automatic EDA | by ChiefHustler | Towards Data Science

EDA stands for Exploratory Data Analysis and is probably the most important yet most overlooked step in any data science project.
My goal is to share with you some tools that will instantly 10x your productivity when exploring data.
It’s during the exploration phase that the data scientists get close and intimate with the data set before moving on to data cleaning, feature engineering and modeling.
Yet too many of us rush through data exploration and jump prematurely to the next steps.
It used to be that you had to write home-cooked functions for EDA, and there has been a lot of reinventing the wheel and/or copy-pasting in this area. But with the advent of advanced techniques such as Automatic Machine Learning, it has become obvious that EDA could benefit from automation as well.
In this post, I first show an example of the old fashioned way of doing EDA. Then I review the following packages that focus on making EDA as easy and efficient as possible:
pandas-profiling (python)
summarytools (R)
explore (R)
dataMaid (R)
The data sets we’ll be using are:
regression: health insurance, where the goal is to predict the insurance premium.
classification: titanic, where the goal is to predict who survives.
Install the R packages:
install.packages(c("summarytools", "explore" "dataMaid"))
Install the python pandas-profiling package:
pip install pandas-profiling
To use the package from Jupyter notebooks, we also need to run the following from the command-line:
jupyter nbextension enable --py widgetsnbextension
If you prefer Jupyter Lab, the command is instead:
jupyterlab: jupyter labextension install @jupyter-widgets/jupyterlab-manager
In the olden days, one would either use a visualization library or pandas built-in visualization functions. Let’see a few examples.
import pandas as pd import numpy as np data = pd.read_csv('insurance.csv')
This is probably the 1st function call in every data science notebook, used to peek at the data:
data.head()
This function gets us some summary statistics on each variable. I like to split the numeric and the categorical variables in two separate calls:
data.describe(include=np.number)
data.describe(include=np.object)
The two most common plots for one-way distributions are bar charts for categorical variables and histograms for continuous variables. Both are easy to use directly from pandas:
data['region'].value_counts().plot.barh(title='Region')
data['age'].plot.hist(title='Age distribution')
Finally, we have the ubiquitous correlation matrix:
data.corr().style.background_gradient(cmap='coolwarm').set_precision(2)
That concludes our quick review of old-school exploratory data analysis. As you can see, we have to explicitly write down every step. If the number of columns gets high, this process will either become tedious or we will have to write down our own functions to automate it.
In the rest of the post, I review existing successful packages that can perform that automation for us.
summarytools might be the easiest way to get 1-way information for every column in the dataset.
library(summarytools) library(readr) df <- read_csv("insurance.csv") view(dfSummary(df))
The output is shown directly inside RStudio’s viewer, but you can also save it as an html file with the file parameter. Take a look::
Athough not comprehensive, I find this output extremely efficient in packing a lot of useful information in very little space. The package has a few other functions for univariate statistics and cross-tabulations, but the real gem in my opinion is the dfSummary function above.
The explore package is quite incredible. For 1 line of code, you get a complete shiny application that lets you interactively explore your data. Let's see an example with the titanic data set. Below, I show one and two-way plots for 2 variables, a categorical one (Sex) and a continuous one (Age). We instantly see that women were much more likely to survive, and that young people were slightly more likely to survive:
library(explore) df <- read_csv("titanic_train.csv") explore(df)
Another nice feature is the “explain” tab where we can visualize a simple decision tree to build our intuition on the predictive variables:
library('dataMaid')  makeDataReport(df,                               render = FALSE,               file = "output_file.Rmd",               replace = TRUE)
dataMaid is probably the most comprehensive option so far. You can output an Rmd file and then knit it into an HTML report from RStudio, or you can use render=TRUE which will output a nice Word document for you!
The report starts with a summary table:
Followed by a section for each variable. Here’s the section for bmi, which according to the summary table above might have a problem, which turns out to be a problem of potential outliers:
The makeDataReport function has a lot of parameters to tweak the output. Overall, I'm very impressed with this package.
Last but not least, pandas-profiling is probably the most popular package for automatic EDA in the python world. You can use it from within a notebook and thanks to the fantastic ipywidgets library, you get an interactive and very rich visualization to explore your data.
from pandas_profiling import ProfileReport profile = ProfileReport(data,                         title='Pandas Profiling Report',                         html={'style':{'full_width':True}}) profile.to_widgets()
First, you get an overview of the data:
Along with a few warnings not caught by the previous packages:
We also, we get a detailed report for each variable:
And we can even toggle details for deeper exploration:
Also available are interactions plots between the variables:
The other tabs (Correlations, Missing values and Sample) are what you expect them to be. I invite you to play with this wonderful package.
I hope I convinced you to stop using home-made functions or expensive visualization tools and switch to open-source dedicated packages.
The 4 packages described above are my go-to tools for data exploration in a real-case setting. I start with the non-interactive ones, first summarytools followed by dataMaid. I then play around with explore, and finally, I spend the most time with pandas-profiling.
This is how I get a familiar as possible with my data without wasting time writing boiler-plate code. Depending on the findings I come away with, I will use other packages for anomaly detection, imputation, feature engineering, etc. Each of these will be the subject of future posts. Thanks for reading!