Look before you leap - Know how and why exploring data might be crucial for your success in modeling
By: Viswanatha Krishnamurthy
Tags: Eda, Data Science, Exploration, Profiler

Look before you leap - Know how and why exploring data might be crucial for your success in modeling Viswanatha Krishnamurthy Follow Jun 8 · 8 min read

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so” — Mark Twain

A tendency to believe what we think is fact and acting on it can go spectacularly wrong, especially when our belief turns out to be not only opinion but wrong.

Solving complex business problems necessitates a thorough understanding of the data and such in-depth analysis would require more than just the basic techniques.

The Modeling process can be broadly classified into 5 major phases. Though feature engineering, hyper parameter tuning, algorithm selections are some of the intrinsic concepts to deal with, it’s the phase of Exploratory Data Analysis that would demand significant time and attention of an analyst.

Did you know that Collecting Datasets, Cleaning, and organizing data account for almost 80% of the work for Data Scientists? *source -https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#46ac962f6f63

Deducing latent patterns from the data is the cornerstone of any modeling activity or any analysis. It’s imperative to identify and isolate noise from signals in the data.

In this article, we would take a peek into the strategies applied through the EDA (Exploratory Data Analysis).

Exploratory data analysis is a set of actions or analysis which are performed to familiarize oneself with the data and ensure its in expected shape and form to be processed further for consumption.

Data Extraction:

Once we have analyzed the business case and have drafted all possible hypotheses, It is time for us to gather data to either approve or reject the hypothesis.

Sources of data can be multiple therefore during the Data Extraction phase, we would tap into relevant systems to glean necessary information. During the process, we might come out with some new hypotheses, which will need to be added to the list of hypotheses we are considering.

Data Exploration:

In-depth knowledge of the data being considered for analysis ensues efficient and accurate modeling.Clarity on the data enables the analyst in deciding techniques to be adopted to tune the data to suit the business requirement.

Following are the steps in the course of Data Exploration -

Before we delve deep into the process, let’s have a quick look at the metrics evaluated -

For the sake of explanation, we would be using the datasets from Mobility Analytics — Analytics Vidhya Hackathon.

First things first - Lets get acquainted with the data.

Business Context - A cab aggregator service operates in multiple cities. Their model of operation is to collaborate with locally available car service vendors to provide most appropriate option available, considering customer demographics. They have been in the play for about an year and have gleaned sufficient data about surge_pricing_type from vendors.

Problem Statement - Predict surge_pricing_type in order to help cab service provider arrange appropriate cab (considering demographics) timely and efficiently.

A quick look at the attributes and its context

Note: I would be using dataprep package to better represent the variables we intend to analyze. Dataprep lets you prepare your data using a single library with a few lines of code. Refer to https://pypi.org/project/dataprep/ for further details.

Step 1- Variable Type Identification — Raw data can consist of either numeric / string or can be a good mix of both. However, the input to the model should always be numeric, therefore we will need to preprocess the data to make it suitable for the model to learn.

Following visual would help you to –

A) Understand % of missing values in each attribute

B) Identify attribute type

C) Visualize the distribution of the data points in an attribute

Charts highlighted in darker shade are categorical and the rest are numeric

Also, describe method can be used populate measures of central tendency and dispersion metrics

Step 2 - Uni variate Analysis — Evaluating the significance of attributes in isolation

Let’s discover Trip_Distance. Below plots would reveal the distribution of data points

Step 3 - Bi-variate Analysis — Permutation combination of attributes to explore latent patterns or behaviors. It can be done at three levels –

For instance, let’s explore Trip_Distance and Customer_Rating as a combination.

Categorical & Categorical

Could there be some interesting pattern if you analyze Destination_Type from a Gender perspective?

Numeric & Categorical — will Cancellation_Last_1Month have any to do with Destination_type? Let’s find out.

Step 4 - Missing Value Treatment

It is improbable not to have attributes with missing records. Having enough data in each attribute does influence the efficacy of the model, therefore its imperative for us to know the degree of absent data and decide whether to retain or to impute.

The reasons for such scenarios depend on the nature of the data. For e.g.: in certain instances, gather data about Gender, Salary, Religion may be a challenge. Else, the subject may refuse to divulge certain details citing privacy issues (during the survey)

Did you know — even missing data can reveal a certain pattern in the data?

Types of missing data –

Identify missing values in each attribute –

Strategies to deal with missing data –

A) Dropping null values from data set — identify and delete rows or columns which has null values. dropna() function can be utilized to drop null values from all rows and columns. Also,you can utilize “any” or “all” parameter to option available in dropna() function to prune the data to suit your needs.

Note: Deleting records (row-wise or column-wise) ensues loss of information which may be critical therefore its recommended to try out imputation techniques before dropping

Imputation Techniques –

A) Imputation with mean or median — The below example uses the average of the rest of the values in the “Var1” attribute to impute for the missing values. Similar imputation can be done using the Median as well.

B) Simple Imputer — Another most frequently used imputation technique from sklearn. Mean or Median can be used for continuous variable and “most frequent” or “constant” can be used for Categorical variables. Please refer to the sklearn documentation for details.

You might also want to explore the below two imputation techniques which are based on regression models

C) KNN Imputer

D)Iterative Imputer

Step 5 - Outlier Treatment - Any data point which stands distinct or does not conform to the pattern of the majority are deemed to be outliers.

What causes outliers?

A) Data entry errors — For e.g.: adding one extra “0” to remuneration number does cause considerable

B) Measurement errors — For e.g.: Some of the data points in Height Attribute (ideal scale of measurement — Feet) populated in meters

C) Processing errors — Instances of the file was not read accurately from the source file. E.g.: alphanumeric numbers or text messages

Type of outliers?

A) Uni-variate outliers

B) Bi-variate outliers

How to Identify outliers?

A) Uni-variate Outliers — Box plot

B) Bi-variate outliers — scatter plot

Formula –

< Q1–1.5 * IQR or > Q3 + 1.5* IQR

Where IQR = Q3 — Q1

Q3 = Value at 3rd quartile

Q1 = Value at 2nd quartile

Different methods to deal with outliers?

A) Exclusion - Delete the observations from the data set

B) Transforming and binning outliers - Adopt techniques such as Logarithmic, square root, cube root transformation to transform the data

C) Impute outliers - Treat them like missing values and impute them using the imputation techniques discussed in the earlier section

D) Treat them in isolation - Consider them as a separate set of data. Try applying different algorithms and techniques on it as compared to the other set of data. Later, you can club it to the main part of the data set.

Variable transformation -

Primary reasons to use transformation -

A) Change the scale of the variable Ensures consistency in measuring the scale of an attribute. For e.g.: weight recorded in both Kilo grams and pounds.

B) Transform non-linear relationships into Linear relationships - Makes it easy to comprehend the linear relationship compared to a non-linear relationship

D) Create symmetric distribution whenever there is skewed distribution- easier to interpret and generate inferences. It also facilitates the normal distribution.

Techniques available -

Apart from the techniques mentioned above, you may want to try out “Pandas profiling”. Output from profiling report host most of the critical information such as correlation matrix, information on missing values, interactions etc. in a single frame. Its findings are exhaustive and appropriate.

Concepts and techniques mentioned in this article should definitely get you started in discovering data. However, there is potential to delve deeper, for e.g: Identifying outliers is a complex process and box plot or IQR formula.