Exploratory Data Analysis: DataPrep.eda vs Pandas-Profiling
By: Brandon Lockhart
Tags: Data Science, Python, Eda, Data Analysis, Exploratory Data Analysis

1. Better API Design

We will use the plot() function from DataPrep.eda. To understand how to perform EDA effectively with this function, the following gives the syntax of the function call with the intent of the data scientist:

plot(df) : “I want an overview of the dataset”

: “I want an overview of the dataset” plot(df, “col_1”) : “I want to understand the column col_1 ”

: “I want to understand the column ” plot(df, “col_1”, “col_2”) : “I want to understand the relationship between columns col_1 and col_2 ”

To see this in action, we will use a dataset consisting of records of COVID-19 patients in South Korea. Let’s start with an overview of the dataset:

from dataprep.eda import plot

import pandas as pd df = pd.read_csv("PatientInfo.csv")

df["confirmed_date"] = pd.to_datetime(df["confirmed_date"])

plot(df)

Frequency distribution for each column

Notice the column birth_year seemingly has a bimodal distribution, let’s learn more about this column:

# transform birth_year to age in 2020 for simplicity

df["age"] = 2020 - df["birth_year"]

plot(df, "age", bins=26)

Distribution of one column plotted in several ways

Using the tooltip we can inspect the bounds of the modes, and use the various plots to attain a comprehensive understanding of this distribution.

Next, let’s investigate the age distribution of men and women for contracting COVID-19. To do this, we simply add the column sex to the previous function call:

plot(df, "age", "sex", bins=26)

Plots comparing the age distribution by sex

We see a large disparity in the age distributions for contracting COVID-19 between men and women — all with one simple line of code!

This exemplifies how EDA is meant to be performed — question, visualize, understand, repeat. Taking under 0.5 seconds to finish each above command, DataPrep.eda is an efficient and effective tool for doing EDA.

Can pandas-profiling support this simple EDA task?

We run the following code where it takes pandas-profiling over 50 seconds to produce a report:

from pandas_profiling import ProfileReport

ProfileReport(df).to_widgets()

To analyze the univariate distributions, pandas-profiling has the following output:

Pandas-profiling API in a Jupyter Notebook

The user needs to toggle each column to see its information. Although we can find the bimodal distribution of birth_year , pandas-profiling has no support for further investigation of this insight, and no support for analyzing the relationship between age and sex .

For this simple EDA scenario, DataPrep.eda enabled the discovery of significant insights, but pandas-profiling was inadequate.

Interactive functionality — tooltip

User interaction with the data is imperative for effective data understanding [7]. DataPrep.eda creates all plots with Bokeh, an interactive visualization library, and incorporates a tooltip which enables an exact reading of each component of a visualization. Pandas-profiling, however, does not support a tooltip.

2. Up to 100x Faster

DataPrep.eda is 100x faster than pandas-profiling for EDA

Recall that in Section 1 it took DataPrep.eda under 0.5 seconds to complete each task, yet it took pandas-profiling over 50 seconds to produce a report.

DataPrep.eda is faster than pandas-profiling for two reasons:

DataPrep.eda uses Dask, a parallel computing library, for all data processing. Pandas-profiling, however, uses Pandas. DataPrep.eda avoids unnecessary computation by creating visualizations relevant to the current EDA task, whereas pandas-profiling only profiles the entire dataset.

DataPrep.eda is 5x+ faster than pandas-profiling even for data profiling

DataPrep.eda can be used to generate components of pandas-profiling’s report, enabling a direct performance comparison. The following figure shows the results of running pandas-profiling’s ProfileReport against three components of DataPrep.eda: univariate analysis ( plot(df) ), correlation matrices ( plot_correlation(df) ) and missing value plots ( plot_missing(df) ).

Wating time of DataPrep.eda vs pandas-profiling (used the Automobile dataset, enlarged by duplication)

Using Dask instead of Pandas is the main reason DataPrep.eda is faster than pandas-profiling. More specific factors affecting the performance are

DataPrep.eda parallelizes univariate analysis, whereas pandas-profiling computes univariate statistics sequentially. DataPrep.eda using Dask supports block-wise computations, whereas Pandas-profiling performs computations over the whole dataset (significant for large datasets).

3. Smart Visualization

Some of the intelligent features of DataPrep.eda include

selecting the right plots to visualize the data for each EDA task;

column type inference (numerical, categorical, and datetime);

finding an appropriate time unit for each plot (the user can also specify it);

outputting the categorical values with the highest count for visual clarity (the user can also specify).

To see these features in action, let’s understand how people are contracting COVID-19 over time, i.e., the relationship between the columns confirmed_date and infection_case . We run

plot(df, "confirmed_date", "infection_case")

Plots depicting the relationship between a datetime and categorical column

We can easily see which methods of contraction are significant and during which periods!

Can we do this task with pandas-profiling?

No! Pandas-profiling only supports interactions in the form of correlation matrices (also supported by DataPrep.eda) and a heat map for bivariate analysis of two continuous variables. Effective bivariate analysis is too expensive in a dataset-profiling framework since it must be computed for each pair of columns, even though only a small subset of relationships is likely interesting to a user.

4. Handles Large Data

DataPrep.eda using Dask works with larger than memory datasets. Dask supports out-of-core and parallel processing so computations on very large datasets can be evaluated efficiently.

Pandas-profiling using Pandas only has data structures for in-memory analytics; pandas-profiling suffers from significant performance degradation on large datasets.