Exploratory data analysis guide
By: Zolzaya Luvsandorj
Tags: Data Analysis, Python, Getting Started, Visualisation, Data Sience

Step 1: Basic exploration🐇

By basic exploration, you may recall that we are referring to any exploration that are common to most data. To streamline this step, we will prepare a template by leveraging automation tool as well as custom functions. Therefore, let’s split the basic exploration step into two parts:

Part 1: Leverage automation tool

Leverage automation tool Part 2: Use custom functions for the remaining basic exploration

The idea behind the template is that we just have to update a few parts in the template when we have new data and can get basic exploration with little effort. Once basic exploration is completed, it’s important to carefully analyse all the charts and aggregate statistics, and create a summary of key findings. This analysis will help shape step 2 of the exploration.

Part 1. Leverage automation tool

We will use Pandas Profiling package to automate parts of the basic exploration. We can create Pandas Profiling report (just report from here onwards) in .html file like this:

# Create profile report

profile_train = ProfileReport(train) # Export it to html file

profile_train.to_file("training_data_report.html")

With just 2 lines of code, we get an awesome report containing data overview, descriptive statistics, univariate and some bivariate distributions, correlations and missing value summary. Output report will look similar to this. You will notice that you can use the tabs at the top right corner to jump to sections and look at more detailed information by clicking on toggle buttons.

You can change the title that appears on the top left corner with title and turn on dark mode with dark_mode arguments. For instance, when creating a report, extending the first line to below will get you a report titled ‘Training Data Report’ in darker mode (which I find nice):

profile_train = ProfileReport(train, title="Training Data Report",

dark_mode=True)

While I think that creating html report is the best way to use the report, it can be accessed in Jupyter Notebook using one of the following options:

# Option 1

train.profile_report() # Option 2

profile = ProfileReport(train)

profile.to_widgets() # Option 3

profile = ProfileReport(train)

profile.to_notebook_iframe()

I recommend try running these options to find your preference. If you want to learn more about the package, see their documentation page.

Part 2. Use custom functions for the remaining basic exploration

It’s likely that automation tools like Pandas Profiling may not cover all parts of basic exploration. If you find yourself doing the same exploration over and over again when working on different projects, and they are not included in the report, then create custom functions or use pandas built-in functions or methods to cater for the gap.

One key trick to keep the template as clean as possible is to abstract away certain functionalities into custom functions and save it in a separate module. Examples in this post use functions from custom module called ‘helpers’. The code used for each function is available at the end of this post.

In this section, we will look at a few example explorations one can do to complement the report. Let’s start with summary for all variables:

helpers.summarise(train)

Although this output may seem like a duplicate of what the report shows, looking this information side by side for all variables can be useful. From this output, we can see that deck is missing in roughly 4 in 5 records and age is missing in about 1 in 5 records. Let’s split features into two groups:

# Define feature groups

continuous = train[features].select_dtypes(['float']).columns

discrete = train[features].columns.difference(continuous) # Convert to category dtype

train[discrete] = train[discrete].astype('category')

Checking out summary statistics for continuous variables is also useful. The following function extends pandas DataFrame’s .describe() method:

helpers.describe_more(train, continuous)

Here, we can see the descriptive statistics and the summary of outliers. Outlier here is defined using John Tukey’s rule. 12% of fare values are outliers.

Visualising the distribution of continuous features for each target class can give us insights on their usefulness in discriminating target classes:

for feature in continuous:

helpers.plot_continuous(train, feature, target)

Similar summaries and visualisations can be done for discrete values:

train.describe(exclude=['number'])

We can see the number of non-missing and unique values in the first two rows. In the last two rows, we can see the most frequent value and its frequency count.

Now, let’s visualise the discrete features. By imputing with ‘missing’, we can see if those records with missing values are more or less likely to have survived relative to the other categories in the feature:

# Fill missing

for feature in discrete:

if train[feature].isnull().sum()>0:

train[feature] = train[feature].cat\

.add_categories('missing')

train[feature] = train[feature].fillna('missing') # Visualise

for feature in discrete:

helpers.plot_discrete(train, feature, target)

In this section, we have looked at summary statistics side by side for features and visualised features’ relationship with target to gain some insights on their usefulness in discriminating target classes. I hope you will create your own template for basic exploration using some or all of the ideas mentioned supplemented by your own favourite ways to explore data.

Once basic exploration is completed, it’s important to carefully analyse output and create a summary of key findings. Here are a few examples:

There were substantially more third class passengers and their survival rate is much lower.

There were substantially more male passengers and their survival rate is much lower.

Understanding the context of the problem you are trying to solve with data is valuable and can help you sense check the data or interpret the findings. Reading about the problem or talking to relevant stakeholders regarding the problem are good ways gain that understanding. In Titanic’s example, we could google around information to understand the disaster better. After doing a little bit of reading, we will soon find out that there was a ‘women and children first’ protocol during an emergency which played a role in who was more likely to have survived. Contextualising data this way is helpful in making sense of the data exploration findings and understanding the problem better.