Dataprep.eda: Accelerate your EDA | by Slavvy Coelho | Towards Data Science

Authors: Slavvy Coelho, Ruchita Rozario
Mentor: Dr. Jiannan Wang, Director, SFU’s Professional Master’s Programs (Big Data and Cybersecurity and Visual Computing)
“Numbers have an important story to tell. They rely on you to give them a clear and convincing voice” — Stephen Few
Most of the industries today have recognized data as a valuable asset. However, what you do with the data and how you utilize it is what helps you get those additional profit figures or that new discovery that is going to create a revolution.
When you start working with a dataset most of the trends and patterns are not apparent. Exploratory data analysis helps one to carefully analyze data through an analytical lens. It helps us draw conclusions to get an overall sense of what’s happening with the data. Uncovering these hidden relationships and patterns are critical to build analytical and learning models on the top of the data.
The general workflow of EDA looks as follows:
Dataprepare is an initiative by SFU Data Science Research Group to speed up Data Science. Dataprep.eda attempts to simplify the entire EDA process with very minimal lines of code. Since we know that EDA is a very essential and time-consuming part of the data science pipeline, having a tool that eases the process is a boon.
This blog intends on providing you an easy and hands-on experience of everything you can do with dataprepare.eda. So let’s dive into it, shall we?
#installing dataprep.eda#open your terminalpip install dataprep
In order to simplify things, we have worked on a Fraud Detection dataset. The data has columns like amount, original balance, origin account, destination account, etc. and finally a label column that indicates if this transaction was in fact a fraudulent transaction or not.
#importing all the librariesimport pandas as pdfrom dataprep.eda import plot, plot_correlation, plot_missing#importing data and dropping missing valuesdf = pd.read_csv("capdata.csv")
The data looks as follows:
The dataprep.eda package has 4 submodules. We shall explain each one of them:
plot(): analyze distributions
plot(): analyze distributions
We provide an API plot() to allow users to analyze the basic characteristics of the dataset. It plots the distribution or bar chart for each column to give the user a basic sense of the dataset.
#API Plotplot(df) 
If a user is interested in one or two specific columns, it provides a more detailed plot for the specific columns by passing column names as the parameter.
plot(df, "newbalanceDest", bins=20)
If x contains categorical values, a bar chart and pie chart are plotted.
plot(df,"type")
If x contains numerical values, a histogram, kernel density estimate plot, box plot, and qq plot are plotted.
plot(df, "newbalanceDest", "oldbalanceDest")
If x and y contain numerical values, a scatter plot, hexbin plot, and binned box plot are plotted. If one of x and y contain categorical values and the other contains numerical values, a box plot and multi-line histogram are plotted. If x and y contain categorical vales, a nested bar chart, stacked bar chart, and heat map are plotted.
2. plot_correlation(): analyze correlations
We provide an API plot_correlation to analyze the correlation between columns. It plots the correlation matrix between columns. If a user is interested in the correlated columns for a specific column, e.g the most correlated columns to column ‘A’, the API can provide a more detailed analysis by passing column names as the parameter.
#API Correlationplot_correlation(df)
plot_correlation(df, k=1)
Top ‘k’ attributes’ correlation heatmap (here, k=1)
plot_correlation(df, "newbalanceOrig") 
Correlation of specified element with all other attributes. (ie. newbalanceOrig with everything else).
plot_correlation(df, "newbalanceOrig", value_range=[-1, 0.3])
All the correlation values that lie within the given range. (-1, 0.3) for newbalanceOrig will appear in the plot.
plot_correlation(df, x="newbalanceDest", y="oldbalanceDest", k=5)
Correlation between two attributes with line of best fit and most influential points.
3. plot_missing(): analyze missing values
We provide an API plot_missing to analyze the pattern and impact of missing values. At the first glance, it shows the position of missing values, which allows the user to be aware of data quality for each column or find any underlying pattern of missing values.
#API Missing Valueplot_missing(df) 
To understand the impact of missing values from a specific column, users can pass the column name into the parameter. It will compare the distribution of each column with and without missing values from the given column, such that the user could understand the impact of the missing values.
plot_missing(df, "isFraud", "type") #count of rows with and without dropping the missing values
4. create_report: generate profile reports from a Pandas dataframe.
The goal of create_report is to generate profile reports from a pandas DataFrame. create_report utilizes the functionalities and formats the plots from dataprep. It provides information like overview, variables, quantile statistics (minimum value, Q1, median, Q3, maximum, range, interquartile range), descriptive statistics (mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness), text analysis for length, sample and letter, correlations and missing values.
from dataprepare.eda import create_reportdf = pd.read_csv("https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv")create_report(df)
The above 3 APIs in combination work towards easing the EDA process.
To support this, we conducted a survey to compare dataprep.eda to traditional python EDA implementation. We surveyed 10 people (a mix of real-world Data Scientists and University students studying Data Science). The survey was based on 3 key performance indicators:
Need: Here we analyzed the number of minutes data professionals spend on EDA per project.
Need: Here we analyzed the number of minutes data professionals spend on EDA per project.
As we can see, most people spend about an hour and half over EDA on an average. This result establishes the fact that we need tools to speed up EDA process.
2. Readability: Here we made a comparison of the two approaches to see how readable they are.
Clearly, dataprepare.eda provides a more lucid and readable code.
3. Lines of code: Does dataprep.eda really reduce the length of code?
In addition to the above key performance indicators, dataprep.eda also proves to be better at several other aspects compared to normal python implementation and even existing data profiling libraries like pandas profiling. Let’s take a look at how these three approaches perform when pitted against each other.
When we compare the efficiency of the other two approaches versus Dataprep.eda, it proves to be more efficient.
When we compare the efficiency of the other two approaches versus Dataprep.eda, it proves to be more efficient.
2. When we talk about these methods and their capabilities to support out of memory data, Dataprepare.eda is the only one that supports out-of-memory data processing. This proves to be very useful in today’s era of Big Data.
3. The plots generated by dataprep.eda are all interactive but matplotlib and seaborn only create static plots. This facilitates deeper understanding of the plot.
We asked several industry professionals and university students to give their feedback on this amazing tool.
Here’s what some of them have to say:
“The library looks really good. Eases the work by a great margin. All the best for the venture” — Dhruva Gaidhani, Software Developer at Amazon
“Spotting missing values is efficient. I usually spend a lot of time on EDA. The library has definitely eased a few functionalities.” — Manan Parasher, Alumni of SFU
Data science is adapting every second as we speak. We are exploring the lengths and breadths of almost all the domains right from sport analysis to medical imaging. These times call for us to channel the power of AI and Machine Learning into developing things that will make an impact and save those few seconds. Having tools like data prepare at our disposal enables us to carry out the preparatory tasks more efficiently.
Dataprepare.eda excels at tasks like checking the data-distribution, correlation, missing values and the EDA process in general. The ease of coding and readability also facilitates novices to use dataprepare library. All in all, Dataprepare.eda serves as your one-stop library for carrying out all the preparatory analysis tasks. It’s an upcoming library with a promising future!