Comprehensive Guide to Machine Learning (Part 1 of 3)
By: Tapas Das
Tags: Machine Learning, Data Cleansing, Web Scraping, Outlier Detection, Exploratory Data Analysis

3) Exploratory Data Analysis

This is one of the most crucial step of developing a machine learning model. In this step, we will take a deep-dive into the training data and get ourselves accustomed with different data relations and data dependencies.

Matplotlib and Seaborn python libraries are our best-friends here in getting insight into the training data. I’ll show few of the basis EDA steps in this post.

Continuous Variables Analysis

For continuous numerical fields, we can get the distribution of data using “distplot” functionality of Seaborn library.

As we can see from above images, the data in “length” and “height” fields are more-or-less distributed in a uniform way.

Categorical Variables Analysis

Typically, categorical data attributes represents discrete values which belong to a specific finite set of categories or classes. These discrete values can be text or numeric in nature. Examples: movie, music and video game genres, country names, food and cuisine types, etc.

We can use “countplots” to understand the data distribution for categorical fields. This functionality is readily available in Seaborn library.

In below images, we can see the data distribution for “pet_size” and “condition” fields.

Outliers Analysis

An outlier is an observation that deviates significantly from rest of the data observations. Outliers can have many causes, such as:

Measurement or input error Data corruption True outlier observation (e.g. Michael Jordan in basketball)

We can use “boxplots” to detect outliers in data. You can refer the below link to understand more about boxplots and their functionalities.

As we can see from below image, the dots towards the right of the plot are indicators of outliers in the data.

There are several methods to detect and treat outliers, which are explained in great details in below post. I’d recommend everyone to go through this to have better grasp at outliers handling.

For our purposes, we can use “cube-root” method of Numpy library to reduce the number of outliers, as shown in below image.

Data Correlation Analysis

In layman terms, correlation is a measure of how strongly one variable depends on another.

Consider a hypothetical dataset containing information about IT professionals. We might expect a strong relationship between age and salary, since senior project managers will tend to be paid better than young pup engineers. On the other hand, there is probably a very weak, if any, relationship between shoe size and salary.

Correlations can be positive or negative. Our age and salary example is a case of positive correlation. Individuals with a higher age would also tend to have a higher salary. An example of negative correlation might be age compared to outstanding student loan debt: typically older people will have more of their student loans paid off.

We can use the “heatmap” functionality of Seaborn library to display the correlation of different input features in training data.

As we can see in the image, there is high “positive” correlation between “X1” and “X2” fields. Also there is high “negative” correlation between “condition” and “breed category” fields.

I’d recommend to go through the below post for better understanding of correlation analysis and it’s impact on machine learning models.