How to do some basic EDA- A guide for dummies
By: Raghav Sharma
Tags: Exploratory Data Analysis, Data Science, Machine Learning, Data Visualization

“The greatest value of a picture is when it forces us to notice what we never expected to see” — John Tukey

Introduction

The first step before doing any kind of data-related task, be it machine learning, data analytics, etc. is to see and analyze the data that is contained inside the data set that you have. Getting a strong idea of its behavior and trends will allow you to make more educated decisions so that your solution at the end of your data-related task is an optimal one. For instance, if you have a strong idea about which input variables most affect the response variable in a machine learning model, it will not only simplify your model but save on the computation cost of training the model.

How can you analyze the data to make such crucial decisions? This is done by drawing various graphs/charts/plots or other data visualization methods between the various variables in your data-set and drawing inferences from them. This procedure is termed Exploratory Data Analysis or EDA. However, for someone starting out in their data science journey, EDA can seem a daunting task. You are not really sure where to start, which charts to choose, and how to interpret the visualizations that you get. EDA is more like an art, like writing or building a machine learning model. There is no one way and each analyst has their own methodology that they like to follow. But the good thing is, like any art form, you get better the more you practice and develop your own heuristics.

Types of Data

Before doing any EDA, you should know the kind of data you may encounter. Data can be categorized as follows:

Categories of Data

Dichotomous Variable: A dichotomous variable is a variable that takes only one out of two possible values when measured. For eg. Gender: male/female.

A dichotomous variable is a variable that takes only one out of two possible values when measured. For eg. Gender: male/female. Polynomic Variable: A polynomic variable is a variable that has multiple values to choose from. For eg. Educational Qualifications: Uneducated/ Undergraduate/ Postgraduate/ Doctoral, etc.

A polynomic variable is a variable that has multiple values to choose from. For eg. Educational Qualifications: Uneducated/ Undergraduate/ Postgraduate/ Doctoral, etc. Discrete Variable: Discrete variables are countable variables but can take on a large range of values. For eg. your bank balance, no. of employees in an organization, etc.

Discrete variables are countable variables but can take on a large range of values. For eg. your bank balance, no. of employees in an organization, etc. Continuous Variable: A continuous variable is a variable that has an infinite no. of possible values. Any kind of measure is a continuous variable. For eg. Temperature is a continuous variable. The temperature of a particular area can be described as 30 °C, 30.1 °C, 30.22 °C, 30.221 °C, and so on.

Categorizing your data will allow you to short-list the kind of charts that are most applicable to your requirements from a plethora of data visualization tools.

To demonstrate the EDA process, I have chosen the California Housing Prices data-set from the StatLib repository. Here I am doing an EDA as a part of the process of building a machine learning model to predict the median housing price.

Types of EDA analyses

EDA can be broadly divided into two types of analyses:

Univariate Analysis: Various visualizations are done by taking only one variable into account. Some of the most common univariate data visualization tools are:

(a) Bar-Graphs: Bar graphs are used to get an idea as to how a categorical/qualitative variable is distributed in a data set.

#Count-plot (Bar-plots) for the categorical variable

sns.countplot(housing.ocean_proximity)

Distribution of categorical variable- ocean_proximity

Ocean_proximity tells us the location of the houses categorized to where they are from the ocean. This bar graph tells us the majority of the houses in the data set are located about less than an hour’s drive from the ocean. While there are hardly any houses that are located on islands.

(b) Histograms: All the inputs in the data set, except ocean_proximity are quantitative in nature. Histograms are a really useful tool in visualizing such data. Histograms can tell you about how the inputs are distributed, whether they are centered or skewed, the ranges of the data, the distribution is unimodal or multimodal, etc. A histogram distribution can give you a strong indication if you need to do some transformation to a given variable before you apply it to a Machine Learning algorithm.

#Plot histograms of numerical data

%matplotlib inline

import matplotlib.pyplot as plt

housing.hist(bins = 50, figsize = (20,15))

plt.show

Histograms of all continuous inputs

It can be seen that:

longitude, latitude, and housing_median_age are multi-modal distributions.

total_rooms, total_bedrooms, population, households, and median_income are unimodal but skewed to the left.

Median_house_value is bimodal.

2. Multivariate Analysis: Various visualizations are done by taking two or more variables into account. Some of the most common multivariate data visualization tools are:

(a) Scatter-Plots: Scatter plots are made between two variables to observe whether a trend exists as well as observe an existing trend.

housing.plot(kind = "scatter", x = "median_income", y = "median_house_value", alpha = 0.1)

Scatter plot between median_house_value and median_income

This scatter plot between median_house_value and median_income shows that there is a positive trend between these two quantities. It can be seen that as the median_income rises, so does the median_house_value. This strongly suggests that median_income will be an important variable while developing our machine learning model.

(b) Correlation Heat-Maps: Correlation plots or heat-maps tell us whether a correlation exists between any pair of variables. The correlation is quantified by Pearson’s r factor, also called the correlation coefficient. The value of this coefficient varies from -1 to 1. If the correlation coefficient is close to 1 or -1, we say that the correlation is highly positive or highly negative. A positive correlation means that the pair of variables are highly dependent and as one increases, so does the other. On the other hand, a negative correlation means that the pair of variables are highly dependent but have an opposite trend. i.e as one increases, the other decreases. If the value of the correlation coefficient is around 0, we say there is no correlation. The degree of correlation between variables is dependent on the range between 0 to 1 or 0 to -1 in the positive or negative direction respectively.

Correlation plots help in simplifying models. Suppose you have a pair of inputs that are highly correlated. By removing one of those variables there will not be much loss of information in the overall data set. This is very useful while determining the input variables that need to be considered for a machine learning model.

Inputs that are highly correlated with the response tell us that the particular input variable has a strong effect on the response variable and hence, has a lot of importance. These are the kind of variables that we wish to keep for our machine learning model.

corr_matrix = housing.corr()

ax = sns.heatmap(

corr_matrix,

vmin=-1, vmax=1, center=0,

cmap=sns.diverging_palette(20, 220, n=200),

square=True

)

Correlation heatmap between different fields of the data set

This correlation heatmap tells us latitude and longitude are strongly correlated. Hence, considering only one of them as inputs in our machine learning model is sufficient.

We can also see that the median_house_value, our response variable, is strongly correlated to the median_income. Hence, median_income has established itself as an important decision variable. This observation can be corroborated by the scatter plot drawn earlier.

Conclusion

These are just basic visualizations that are done in all most all data-related tasks. This is barely scratching the surface. I hope you get some insight into the EDA process. I would suggest picking up a data set and doing your own EDA. Happy learning!

My jupyter notebook containing the full EDA demonstrated in the blog can be found at: https://github.com/sharma2409/EDA_blog/blob/main/EDA.ipynb

References