Exploratory Data Analysis (EDA) with Python & Matplotlib
Tags: Data Visualization, Data Science, Matplotlib, Python, Data Analysis

“In statistics, exploratory data analysis (EDA) is an approach of analyzing datasets to summarize their main characteristics, often with visual methods.” — Wikipedia

Graphical techniques

There are a number of tools that are useful for EDA, but EDA is characterized more by the attitude taken than by particular techniques. Typical graphical techniques used in EDA are:

Dimensionality reduction:

Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable.

Typical quantitative techniques are:

Univariate, Bivariate and Multivariate Analysis:

a) Univariate Analysis

Univariate analysis is the simplest form of data analysis where the data being analyzed contains only one variable. Since it’s a single variable it doesn’t deal with causes or relationships. The main purpose of univariate analysis is to describe the data and find patterns that exist within it.



You can think of the variable as a category that your data falls into. One example of a variable in univariate analysis might be “age”. Another might be “height”. Univariate analysis would not look at these two variables at the same time, nor would it look at the relationship between them.

Some ways you can describe patterns found in univariate data include looking at mean, mode, median, range, variance, maximum, minimum, quartiles, and standard deviation. Additionally, some ways you may display univariate data include frequency distribution tables, bar charts, histograms, frequency polygons, and pie charts.

b) Bivariate Analysis

Bivariate analysis is used to find out if there is a relationship between two different variables. Something as simple as creating a scatterplot by plotting one variable against another on a Cartesian plane (think X and Y axis) can sometimes give you a picture of what the data is trying to tell you. If the data seems to fit a line or curve then there is a relationship or correlation between the two variables. For example, one might choose to plot caloric intake versus weight.

c) Multivariate Analysis

Multivariate analysis is the analysis of three or more variables. There are many ways to perform multivariate analysis depending on your goals. Some of these methods include Additive Tree, Canonical Correlation Analysis, Cluster Analysis, Correspondence Analysis / Multiple Correspondence Analysis, Factor Analysis, Generalized Procrustean Analysis, MANOVA, Multidimensional Scaling, Multiple Regression Analysis, Partial Least Square Regression, Principal Component Analysis / Regression / PARAFAC, and Redundancy Analysis.

In Data Analysis, we will analyze to find out the following:

Dataset’s shape and overview Missing values All numerical variables Distribution of the numerical variables Outliers Categorical variables Cardinality of categorical variables Relationship between independent and dependent feature (We will plot and check distributions in each section).

1. Dataset shape & overview

2. Missing values

Here we will check the percentage of NaN values present in each feature:

1) make the list of features which has missing values

2) print the feature name and the percentage of missing values

We can use heatmap and visualize missing values.

We need to find the relationship between missing values and the target feature. Let’s plot some diagram for this relationship.

Here, the relation between the missing values and the dependent variable is clearly visible. So we need to replace these NaN values with something meaningful which we will do in Feature Engineering. From the above dataset some of the features might not be required, which we will drop.

3. Numerical Variables

Get the list of Numerical variables and visualize columns.

Temporal Variables(Eg: Datetime Variables): From a Dataset, we may have year variables. We can extract information from the datetime variables like number of years or number of days.

4. Distribution of the numerical variables

Numerical variables are usually of 2 types:

1) Discrete Variable and 2) Continuous variable.

5. Outliers

Let’s find outliers in continuous variables using Boxplot.

6. Categorical Variables

Let’s find out the relationship between categorical variable and dependent feature.

7. Cardinality of categorical variables

Many machine learning algorithms e.g. Logistic Regression, Support Vector Machine can not handle categorical variables and expect all variables to be numeric. Those that can, categorical data can pose a serious problem if they have high cardinality i.e too many unique values.

The popular approach is to convert them to n dummy numerical variables if the cardinality of the variable is n. It’s called one hot encoding, because after a categorical variable value is converted to a vector of size n, only one of the vector elements will have a value 1 and the rest 0, assuming that the vector elements are assigned value of 1 or 0. Although a simple, this approach may cause a significant increase number of dimensions, which is not a good for machine learning problems.

Even if the Machine Learning algorithms is tolerant of categorical variables, e.g., Decision Tree, high cardinality could be taxing in terms of computing resources. Decision tree splits the feature space into sub spaces such that data population in the sub spaces is as homogeneous as possible. It iteratively splits and considers different partitions created. For categorical variables , the number of possible splits grows non linearly with cardinality. If we are splitting the categorical values into 2 sub sets for example, it has to consider all possible such pair of sub sets, e.g Zip code. With significant increase in number of dimensions, you will be impacted by so called curse of dimensionality.

In the supervised Machine Learning context, where class or target variables are available, high cardinality categorical attribute values can be can be converted to numerical values. The encoding algorithms are based on correlation of such categorical attributes to the target or class variables. As an added bonus, we get some feature engineering done when using these algorithms. When the categorical attribute is highly correlated with the target attribute, the numerical values will have high variance, with most values concentrated around the extremes.

In the supervised ratio algorithm, the numerical value is a function of number of records with the categorical value in question and how they break down between positive and negative class attribute values as follows.

vi = pi / ti

where,

vi = numerical value for ith value of some categorical attribute

pi = number of records with positive class value for the categorical attribute value in question

ti = total number of records with the categorical attribute value in question

In the weight of evidence algorithm, additionally total number of records with the positive and negative class labels are also taken into account as follows. For class imbalanced data, this algorithm works better.

vi = log((pi / p) / (ni / n)

where,

pi = number of records with positive class value for the categorical attribute value in question

ni = number of records with negative class value for the categorical attribute value in question

p = total number of records with positive class value

n = total number of records with negative class value