Exploratory Data Analysis (EDA)
By: Mukul Kirti Verma
Tags: Data Transformation, Eda, Data Preprocessing, Exploratory Data Analysis, Data Visualization

Here we are going to Learn everything regarding EDA ( Exploratory Data Analysis) analysis, so let's go:

What is EDA?

Exploratory data analysis (EDA) is employed by data scientists to research and investigate data sets and summarize their main characteristics, typically using data visual image methods. It facilitates verify however best to govern data sources to urge the answers you need, creating it easier for data scientists to get patterns, spot anomalies, check a hypothesis, or check assumptions. EDA is primarily wont to see what data can reveal on the far side of the formal modelling or hypothesis testing task and provides a much better understanding of information set variables and therefore the relationships between them. It also can help determine if the applied math techniques you’re considering for knowledge analysis are appropriate.

Exploratory Data Analysis All about the:

How to explore the data with your domain knowledge. How to ensure you are ready to use machine learning algorithms in a project? How to choose the most suitable algorithms for your data set? How to define the feature variables that can potentially be used for machine learning?

Why EDA is very much important in data science?

The idea of EDA is to help you better to understand the data before making any assumptions. It can help you to identify some common errors, better understand patterns in the data, finding the outliers or unusual events like any kind of novelty, and interestingness between the variables you are looking for correlation and relationships.



Data scientists can use exploratory analysis to ensure that their findings are valid and applicable to their desired business outcomes and goals. EDA also helps stakeholders by ensuring that they ask the right questions. EDA can help answer questions about standard deviation, Null values, time-series data & categorical Features. EDA ends & outputs the information, and its functions can be used for more complex modelling or data analysis, including machine learning.

The Major goal of EDA

Put forward hypotheses about the reasons for the observed phenomena

Estimate the assumptions on which the statistical conclusions are based

helps to find appropriate statistical methods & tools

gives a basis for the process of collecting data through experiments

To see how exactly data is distributed

Data Cleaning, like handle Missing values

Identify outliers and remove them

Drop the Duplicate data

Data Normalization

Data Scaling

Data Dimension Reduction

Some Feature Engineering

The Terminology:

2. Features/Variables/input-variables/Dependent-Variables : see this

3. Label/Output-Variables/Class-Labels/Response Label: see this

4. Vectors-1D,2D: see this

I hope you are familiar with this simple terminology, if you don’t then please go through the link given respectively before going ahead.

Types of exploratory data analysis

There are four primary types of EDA:

Univariate (non-graphical) This one is the simplest one, where we are analyzing the data with a single feature. But it doesn't deal with the relationship because we are dealing with a single feature the purpose of the univariate analysis is to describe the data, finding nulls, finding anomalies and finding patterns in that feature.

This one is the simplest one, where we are analyzing the data with a single feature. But it doesn't deal with the relationship because we are dealing with a single feature the purpose of the univariate analysis is to describe the data, finding nulls, finding anomalies and finding patterns in that feature. Univariate graphical. It provides a full picture of the dataset and gives better visuals of the data.For Univerient types of EDA techniques are: Stem-and-leaf plots, Histograms, Box plots, and many more in this blog we will discuss each and every technique.

It provides a full picture of the dataset and gives better visuals of the data.For Univerient types of EDA techniques are: Stem-and-leaf plots, Histograms, Box plots, and many more in this blog we will discuss each and every technique. Multivariate nongraphical: Multivariate data arises from more than one variable. Multivariate non-graphical EDA techniques generally show the relationship between two or more variables of the data through cross-tabulation or statistics.

Multivariate data arises from more than one variable. Multivariate non-graphical EDA techniques generally show the relationship between two or more variables of the data through cross-tabulation or statistics. Multivariate graphical: Multivariate data uses graphics to display relationships between two or more sets of data. The most used graphic is a grouped bar plot or bar chart with each group representing one level of one of the variables and each bar within a group representing the levels of the other variable.

Basically, EDA includes Data Analysis by, Visualization, Quantitative techniques, Dimension reduction, Data scaling and transformation.Here are some tools and techniques (Graphical and non Graphical) for EDA which is categorized into 4 categories:

The above table lists out all the techniques of EDA

Now let's discuss one by one

1. Graphical Technique(for Data Visualization)

1.1 Box Plot:

It is a small rectangular box is drawn and a line representing the median, while the top and bottom line of the box represent the 75th percentiles and 25th percentiles (3rd and 1st quartiles), respectively. If the median is in the middle of the box then the data is normally distributed otherwise skewed. If the median is closer to the 1st quantile, then the distribution is +ve skewed and if shifted to words 3rd quantile the negatively skewed. And the point outside represented by the circle are the outliers

Box Plot showing the distribution of data

Purpose:

Box plot tells you the central tendency of the data, median, outliers, skewness of the data, quintiles variability, overall box plot tells many things about the data.

1.2 Histogram

Histogram Summarize the Univariate Data Set. The purpose of a plotting histogram is to graphically tells the distribution of a data set (Univariate).

Vertical axis: Frequency count, horizontal axis: Response variable

Purpose:

The histogram graphically shows the following:

a. The centre tendency of the data;

b. Spread of the dataset

c. Whether the data is skewed or not

d. Is there any outliers

1. 3 Run Chart

Mainly Run sequence plots are an easy way to graphically summarize a univariate data set. But here are a few assumptions of univariate datasets, that they should behave like:

random drawings; from a fixed distribution; with a common location; and with a common scale.

Purpose :

Using run sequence plots, we can find the shifts in location and scale are typically quite evident. So outlier detection will be too easy. It helps to answer :

a. Is there any shifts in variation?

b. Is there any shifts in location?

c. Is there any outliers?

1.4 Pareto Chart

A Pareto chart is a kind of bar chart but with one extra advantage that, all the frequency count bars are arranged in decreasing order of their height. these charts are good to do the analysis of the very bulk data.

This is the Pareto chart of a restaurant, which is showing max customer ordering a dish on the basis of Taste.

Purpose:

To pick any specific data and work on it

Any specific Data Explanation,

To describe the population growth in a sample over the time

Pointing the global problems & resolve them.

to check the max data point tending to which value

1.5 Scatter Plot:

Scatter plot helps for bivariate feature analysis, its matrix visualizes the bivariate relationships b/w several pairs of Features. The scatter plots from individual features are stacked such that every variable is in on the x-axis and at the y-axis. When implemented to standardized variables (with mean value 0 and standard deviation one), it’s exact counterpart of a correlation matrix.

This is the scatter plot of the iris data set, and this plot clearly says that on the basis of petal length and petal width we can easily classify the Iris-setosa flower.

Purpose:

Scatter plots’ describe the relationship b/w the two variables. The dots in the scatter plot report the values of individual variables as well it reports the pattern if existed. I general we find the correlational relationships b/w variables.

1.6 Stem and leaf

Stem and leaf plots are used to distinguish discrete or continuous variables. Stem and leaf plot is used to organize data as it is collected. It very much looks like a bar chart. Each number in the data is divided into stem and leaf, hence the name.

Once the information has been received, it may be best to summarize the details. We can simply summarize the data using a histogram. The main drawback of using a histogram to summarize data is that actual data is not shown on the graph. The stem-and-leaf graph, on the other hand, summarizes data and also showing the data simultaneously.

The basic idea behind a stem-and-leaf plot is to divide each data point into a stem and a leaf.

Purpose:

Is the data distribution of IQs is bell-shaped

Also clearly show the data point with min occurrence

1.7 Parallel Coordinate

A parallel Coordinate Plot (PCP) is a way to test multiple relationships. Each conceivable variance is drawn as a parallel line where coordinates of the observations are recorded as points. The same points for each sample are connected and form a line. As a result, there are many lines as seen in the PCP. PCP can be used to find “clusters” between observations where their lines show the same patterns. In addition, the typical pattern on the slopes of the linked lines on the various Features axes indicates the nature of the “correlation” between those Features (positive or negative, or no pattern).

Here is the parallel graph of the IRIS data set which shows the correlation b/w the petal length and petal width.

Petal length and Petal width variables are very close

Purpose:

a. Finding the Pattern in the data Set

b. Finding Correlated variables

c. Comparing the variance of variables

1.8 Targeted projection pursuit

Targeted projection pursuit is the type of mathematical process used for the analysis of EDA, visualization, and feature selection. Allows the user to collaborate on the most complex data (usually with tens to hundreds of symbols) to determine potential features or patterns of interest.

See the full gif animation: https://upload.wikimedia.org/wikipedia/commons/d/d3/Example_of_Targeted_Projection_Pursuit.gif

in the given example targeted projection pursuit is used to explore the projections of a gene expression in the data set. here colour of data points represents the 4 categories of the cancer tumour. For every single sample, the expression level of 100 genes was recorded (that represented by the axes). The GIF shows that TPP is able to clearly separate 2 of the classes (red and purple), but 2 others can not be separated (blue and green). position of the axes then shows the activation of which genes are most associated with each class.

Purpose:

projection pursuit, finds the most “interesting” guess in multi-dimensional data, using some search algorithm that optimizes some fixed criterion of “interestingness” — such as deviation from Gaussian distribution. In contrast, targeted projection allows the user to explore the Space of projection by manipulating data points directly into the scatter graph.

2. Quantitative techniques For EDA

2.1 Odd Ratio

It is the measure of the association b/w the exposure and the outcome. The odd ratio defines the odd that an outcome will occur for a given particular exposure, compared to the odds of the outcome, occurring in the absence of that exposure. this method is most commonly used in case-control studies, cross-sectional and cohort study designs.

Image shows that how to calculate odd Ratio

Purpose:

OR are used to compare, relative odds of occurrence of the outcome of interest, given exposure to the variable of interest. The OR can be used to find whether a particular exposer is risky for any particular outcome, and compare the magnitude of various risk factors for that outcome.

If OR=1 then Exposure does not affect the odds of the outcome

If OR>1 then Exposure is associated with higher odds of the outcome

If OR<1 then Exposure is associated with lower odds of the outcome

OR uses confidence interval (CI) for precision, it used CI to estimate the precision of the odd ratio. A large value of CI indicates a low level of the precision of the odd ratio, and a small value of CI indicates the higher precision of the OR.

2.2 Median Polish

Medium polishing is a simple and robust data analysis process proposed by mathematician John Tukey. The purpose of the central polish is to obtain an adequate data entry model in the two-way table (frequency, results from the factorial experiment) for the row effect + column effect + overall median.

Purpose:

Median Polish algorithm is used in the RMA normalization of microarrays. As you may be aware, microarray data is quite noisy, therefore they need a more robust way of estimating the probe intensities taking into account observations for all the probes and microarrays. This is a typical model used for normalizing the intensities of probes across arrays.

2.3 Ordination

In a standard set of data, however, there are many types of species and samples. It is impossible for the human mind at the same time to think of any dimension. The purpose of the anointing is to assist in the implementation of Ockham’s Razor: a few dimensions are very easier to understand than many dimensions. A good ordination method will be able to specify the most important feature in the data set, and regardless of “noise” or variability.

Both direct and indirect gradient analyzes have the potential to reduce the size of the data set. However, reducing the size is not the only reason to use Ordination. Prior to the development of the CCA, the most widely used Ordination methods were not specific, and the original purpose of ordination was considered “Exploratory”.

Purpose:

a. To find the importance of features in the data set.

b. Regardless of noise it gives a better result.

3. Data Dimension Reduction

3.1 PCA (Principal Component Analysis)

In the case of large datasets, it is really hard to to interpret.even with hi dimension dataset some machine learning models take too much time to train. Principal component analysis (PCA) helps to reducing the dimensionality of such type of datasets,PCA increasing interpretability but also it minimize the information loss. And it has been done by creating new uncorrelated variables that successively maximize the variance. To Finding such new variables, principal components, reduces to solving an eigenvalue/eigenvector problem, & the new variables are then defined by the dataset at hand, not a priori, hence making Principal component analysis an adaptive data analysis technique

How PCA works?

Here are the following steps

1. Get the dataset

2. Calculate the covariance matrix of data

3. Calculate the eigenvalues and eigenvectors over covariance matrix

4. Choose the principal components

5. Construct a new featured dataset from chosen components

Purpose:

The main purpose of PCA is to reduce the dimension of the data so that it will reduce the time and space complexity of the training process.

3.2 MPCA (Multilinear PCA)

Traditional techniques for reducing the size of subspace dimensionality such as PCA can convert the image matrix into high vectors in only one mode, which can not meet the need to reduce the dimension. Such strategies are therefore incapable of managing multidimensional objects effectively and obtaining satisfactory results. Therefore, to reduce the size, a reduction algorithm that can work directly on a high-order tensor object is preferred.

Purpose:

The MPCA performs feature extraction by determining a multilinear projection that captures most of the original tensor input variation. The solution is iterative in nature and it proceeds by decomposing the original problem into a series of multiple projection subproblems.

3.3 NLDR

NLDR is also a type of dimension reduction algorithm, but the whole algorithm has its advantages and disadvantages, in fact reducing dimension assumes that the data is still distributed near the low dimension manifold. In that case, one can choose PCA if the manifold (approximately) is linear, and NLDR if the manifold is not in linear. Therefore, other questions to consider: is the data very low dimension and, if so, is it still distributed near non-line manifold?

Purpose :

NLDR is used to reduce the dimension of the data where the manifold is non-linear.

4. Data Scaling and Transformation

4.1 Normalization

Normalization is the data preparation technique to normalize the numeric Features into the defined range. The goal of normalization is to change the values of numeric columns in the dataset to use a common scale, without distorting differences in the ranges of values or losing information.

Purpose:

a. To Reduce the space and time complexity of the model

b. To bring all variables on the same scale

c. Reduce the computation

4.2 Data standardization

The process of rescaling one or more attributes so that they have a mean value of 0 and a standard deviation of Standardization assumes that your data has a Gaussian (bell curve) distribution.

Purpose:

a. To shift the mean of the variable to 0.

b. To make the STD to 1 of the data

c. To make sure that all the parametric algorithms perform well on Dataset

4.3 Data Transformation

In machine learning or in-depth learning, some models such as Linear Regression, Logistic Regression, Artificial Neural Networks think that features are often distributed and would do much better if the features given to them during modelling are Gaussian distributed.

But the information provided does not follow the Gaussian distribution. So, what should we do in such a situation? in that case some transformers are available to transform any distribution to normal distribution like power transformer, Quantile transformer, and many.

Purpose:

a. To convert Skewed distribution to normally distributed data.

b. To transform log or any other distribution to Gaussian Distribution.

c. to make sure that all the parametric algorithms perform well on Dataset.

Some ALGO of Data Scaling and Transformation

2.1) Min Max Scaler

2.2) Standard Scaler

2.3) Max Abs Scaler

2.4) Robust Scaler

2.5) Quantile Transformer Scaler

2.6) Power Transformer Scaler

2.7) Unit Vector Scaler

EDA also Includes Outliers detection

Outliers are the data points ​​that deviate from other observations in the data, may show variations in measurement, test errors or new ones. In other words, the outlier views are different from the complete pattern in the sample, so they are very dangerous for parametric machine learning algorithms we must detect and delete those outliers.

List of Algo to handle Outliers:

3.1 Z-Score or Extreme Value Analysis (parametric)

3.2 Isolation Forest

3.3 DBSCAN

3.4 Elliptical Envelop

3.5 One class SVM

3.6 Local Outlier Factor

3.7 Boxplot (IQR based)

Conclusion:

In this post I try to cove many EDA techniques, I did not include the code in it If you are really interested in the code, then you can follow the Kaggle notebooks:

1. Data Visualization: 2D see here, 3D see here.

2. EDA on IRIS data Set, see here

3. Data Cleaning: see here.

4. Anomalies Detection in Dataset: see here.

5. Data Preparation: see here.

6. Data Transformation: see here