Exploratory Data Analysis: DataPrep.eda vs Pandas-Profiling | by Brandon Lockhart | Towards Data Science

Exploratory data analysis (EDA) is part and parcel of every data science project. The purpose of EDA is to achieve an understanding of the data and to gain insights about phenomena the data represents.
Pandas-profiling (2016) has been lauded as an exemplary tool for doing EDA [1, 2, 3]. However, a significant downside of pandas-profiling is that it gives a dataset’s profile! EDA is an iterative process where the data scientist will question, understand, process, transform the data, and repeat [4, 5, 6]. The rigid structure of pandas-profiling is contrary to current EDA best practices.
DataPrep.eda (2020) is a Python library for doing EDA produced by SFU’s Data Science Research Group. DataPrep.eda enables iterative and task-centric analysis — as EDA is meant to be done. (see this article for a comprehensive introduction to DataPrep.eda)
Better API DesignUp to 100x FasterSmart VisualizationHandles Large Data
Better API Design
Up to 100x Faster
Smart Visualization
Handles Large Data
“‘Exploratory data analysis’ is an attitude, a state of flexibility, a willingness to look for those things that we believe are not there, as well as those we believe to be there.”
— John Tukey, Author of Exploratory Data Analysis
We will use the plot() function from DataPrep.eda. To understand how to perform EDA effectively with this function, the following gives the syntax of the function call with the intent of the data scientist:
plot(df): “I want an overview of the dataset”
plot(df, “col_1”): “I want to understand the column col_1”
plot(df, “col_1”, “col_2”): “I want to understand the relationship between columns col_1 and col_2”
To see this in action, we will use a dataset consisting of records of COVID-19 patients in South Korea. Let’s start with an overview of the dataset:
from dataprep.eda import plotimport pandas as pddf = pd.read_csv("PatientInfo.csv")df["confirmed_date"] = pd.to_datetime(df["confirmed_date"])plot(df)
Notice the column birth_year seemingly has a bimodal distribution, let’s learn more about this column:
# transform birth_year to age in 2020 for simplicitydf["age"] = 2020 - df["birth_year"]plot(df, "age", bins=26)
Using the tooltip we can inspect the bounds of the modes, and use the various plots to attain a comprehensive understanding of this distribution.
Next, let’s investigate the age distribution of men and women for contracting COVID-19. To do this, we simply add the column sex to the previous function call:
plot(df, "age", "sex", bins=26)
We see a large disparity in the age distributions for contracting COVID-19 between men and women — all with one simple line of code!
This exemplifies how EDA is meant to be performed — question, visualize, understand, repeat. Taking under 0.5 seconds to finish each above command, DataPrep.eda is an efficient and effective tool for doing EDA.
We run the following code where it takes pandas-profiling over 50 seconds to produce a report:
from pandas_profiling import ProfileReportProfileReport(df).to_widgets()
To analyze the univariate distributions, pandas-profiling has the following output:
The user needs to toggle each column to see its information. Although we can find the bimodal distribution of birth_year, pandas-profiling has no support for further investigation of this insight, and no support for analyzing the relationship between age and sex.
For this simple EDA scenario, DataPrep.eda enabled the discovery of significant insights, but pandas-profiling was inadequate.
User interaction with the data is imperative for effective data understanding [7]. DataPrep.eda creates all plots with Bokeh, an interactive visualization library, and incorporates a tooltip which enables an exact reading of each component of a visualization. Pandas-profiling, however, does not support a tooltip.
Recall that in Section 1 it took DataPrep.eda under 0.5 seconds to complete each task, yet it took pandas-profiling over 50 seconds to produce a report.
DataPrep.eda is faster than pandas-profiling for two reasons:
DataPrep.eda uses Dask, a parallel computing library, for all data processing. Pandas-profiling, however, uses Pandas.DataPrep.eda avoids unnecessary computation by creating visualizations relevant to the current EDA task, whereas pandas-profiling only profiles the entire dataset.
DataPrep.eda uses Dask, a parallel computing library, for all data processing. Pandas-profiling, however, uses Pandas.
DataPrep.eda avoids unnecessary computation by creating visualizations relevant to the current EDA task, whereas pandas-profiling only profiles the entire dataset.
DataPrep.eda can be used to generate components of pandas-profiling’s report, enabling a direct performance comparison. The following figure shows the results of running pandas-profiling’s ProfileReport against three components of DataPrep.eda: univariate analysis (plot(df)), correlation matrices (plot_correlation(df)) and missing value plots (plot_missing(df)).
Using Dask instead of Pandas is the main reason DataPrep.eda is faster than pandas-profiling. More specific factors affecting the performance are
DataPrep.eda parallelizes univariate analysis, whereas pandas-profiling computes univariate statistics sequentially.DataPrep.eda using Dask supports block-wise computations, whereas Pandas-profiling performs computations over the whole dataset (significant for large datasets).
DataPrep.eda parallelizes univariate analysis, whereas pandas-profiling computes univariate statistics sequentially.
DataPrep.eda using Dask supports block-wise computations, whereas Pandas-profiling performs computations over the whole dataset (significant for large datasets).
Some of the intelligent features of DataPrep.eda include
selecting the right plots to visualize the data for each EDA task;
column type inference (numerical, categorical, and datetime);
finding an appropriate time unit for each plot (the user can also specify it);
outputting the categorical values with the highest count for visual clarity (the user can also specify).
To see these features in action, let’s understand how people are contracting COVID-19 over time, i.e., the relationship between the columns confirmed_date and infection_case. We run
plot(df, "confirmed_date", "infection_case")
We can easily see which methods of contraction are significant and during which periods!
No! Pandas-profiling only supports interactions in the form of correlation matrices (also supported by DataPrep.eda) and a heat map for bivariate analysis of two continuous variables. Effective bivariate analysis is too expensive in a dataset-profiling framework since it must be computed for each pair of columns, even though only a small subset of relationships is likely interesting to a user.
DataPrep.eda using Dask works with larger than memory datasets. Dask supports out-of-core and parallel processing so computations on very large datasets can be evaluated efficiently.
Pandas-profiling using Pandas only has data structures for in-memory analytics; pandas-profiling suffers from significant performance degradation on large datasets.
Exploratory data analysis is an iterative cycle with steps including
Questioning the dataAnswering the questions by processing and visualizing the dataRefining previous questions after achieving a new understanding, or creating new questions
Questioning the data
Answering the questions by processing and visualizing the data
Refining previous questions after achieving a new understanding, or creating new questions
There is no one-size-fits-all data profile that is suitable for comprehensive EDA.
DataPrep.eda is a better tool for doing EDA than pandas-profiling for four reasons:
Better API design DataPrep.eda’s APIs are designed for EDA rather than data profilingUp to 100x FasterDataPrep.eda executes computations in parallelSmart VisualizationDataPrep.eda will automatically select the right plots to visualize the dataHandles Large DataDataPrep.eda supports out-of-core processing
Better API design DataPrep.eda’s APIs are designed for EDA rather than data profiling
Up to 100x FasterDataPrep.eda executes computations in parallel
Smart VisualizationDataPrep.eda will automatically select the right plots to visualize the data
Handles Large DataDataPrep.eda supports out-of-core processing
It’s time to move on from generating a data profile, and perform EDA in the manner it’s meant to be done with DataPrep.eda.
A notebook with the code from this article can be found here. To install DataPrep.eda, and for information about contributing to the project, visit here. A DataPrep.eda tutorial video can be found here. Don’t forget to star the project on GitHub ★.
[1] M. Deep, Quick Exploratory Data Analysis: Pandas Profiling (2020), Medium
[2] L. Frei, Speed Up Your Exploratory Data Analysis With Pandas-Profiling (2019), Towards Data Science
[3] R. Rei, EDA Using Panda’s Profiling (2020), Towards Data Science
[4] D. Bourke, A Gentle Introduction to Exploratory Data Analysis (2019), Towards Data Science
[5] J. Wei, Exploratory Data Analysis: A Practical Guide and Template for Structured Data (2019), Towards Data Science
[6] G. Grolemund and H. Wickham, R for Data Science (December 2016), Online Book
[7] W. Koehrsen, The Next Level of Data Visualization in Python (2019), Towards Data Science