Exploratory Data Analysis — Part I
By: Priyanka Banerjee
Tags: Exploratory Data Analysis, Data Science, Data Analysis, Pandas

In this article, we will understand how to explore NUMERIC DATA with Pandas.

What is a data set used in data analysis?

A data set used in data analysis is a combination of certain numbers or string or both which may or may not be in any particular order and from which we need to predict an outcome that might happen in the future.

Now when we have the raw data set from which we have to make a prediction, several questions will arise to take such a decision. For example, if I want to invest my money for best return in 5 years, I would first study several types of investment schemes in public and private enterprises and compare them, contact customer services of the enterprises to clear my doubts, read reviews by other customers to understand them at a deeper level, thoroughly go through their terms and conditions and many more such investigations. These investigating measures are nothing but “Exploratory Data Analysis” for a data scientist while working with such raw data and inferring predictions from it.

The questions that are important to understand before starting with the process of exploratory data analysis:

What is the question that I’m looking for in the given data set

Which type of data does my data set contain

To predict my target variable, which features are needed to be added or removed from the data set

Are there any missing values present, if so, how to deal with it

If there are any outliers, how do we remove them

In this article, we will only explore the process of EDA on a Numerical data set.

Let’s read the given data set (using read_csv) , and take a look at the first 5 lines using the head method considering that the given data set that is to be trained is in .csv format.

import pandas as pd

import numpy as np

df = pd.read_csv("train.csv")

df.head(5)

Note: If the original raw data is separated by delimiter “;” then our command will be:

df = pd.readcsv(“train.csv”, sep = “;”)

Here we need to predict the “fare_amount”

We have used a data set which will predict the fare of a cab ride based on the distance traveled inside New York City. The head() and tail() functions return first and last observations on the data set.

df.shape() enables to get the no. of rows and columns in our data set.

enables to get the no. of rows and columns in our data set. df.describe() gives us the mean, standard deviation and other features of our data set of each and every variables.

Here notably large difference exists between the 75% and max values of our target variable “fare_amount” and other variables like “passenger_count” and “pickup_latitude”. These values clearly imply that our data set contains Outliers .

. We can also notice here that the mean value is much higher than the value of the median of the variable “fare_amount”.

df.dtypes gives us the data types of each and every variable in the df dataframe.

df.info() can help us to see if there are any missing values

In our data set, the variable columns “fare_amount” and “passenger_count” contain missing values.

Our target variable “fare_amount” is numerical in nature.

missing_value = pd.DataFrame(df.isnull().sum())

We need to impute our missing values. But before performing the same, we need to remove the outliers.

Why are we removing the outliers before doing missing value imputation?

Because missing value imputation has different methods; one of them is replacing the missing values of a particular column with the mean or median or mode of that column. Now, if the data set contains these outliers while performing the imputation process, then the missing values will get replaced wrongly. Hence, we need to remove the outliers.

Outlier Removal

We will go ahead with constructing boxplots for each variable to understand the outliers of each variable.

import seaborn as sns

import matplotlib.pyplot as plt

%matplotlib inline

sns.boxplot(df['passenger_count'])

Clearly here we can see that there are some data points which state that the “passenger_count” is more than 5000 which we will remove. In fact, no cab can carry more than 8 passengers hence, we will delete those data points that have values >8 and <1.

df=df.drop(((df[df['passenger_count']>8])).index,axis=0

df=df.drop(((df[df['passenger_count']<1])).index,axis=0)

df

Once the rows with outliers are deleted, we can once again check our boxplot.

We also checked if any latitude or longitude is out of the range. If so, we have dropped it along with the outliers present in every column.

Sorting :

Here, we can sort our data frame based on a particular variable either in ascending or descending order.

df.sort_values(by='passenger_count', ascending =True).head(5)

df

Now if we require to sort more than one column :

df.sort_values(by=['passenger_count','fare_amount'], ascending =[True,True] ).head(5)

Above we can see that our codes are almost same but the outputs are different. In this way we can obtain our required data frame to perform deeper analysis on it.

Now, if I need to know average values of all the other variables when my passenger_count is 1 or 2, I can obtain the same :

df[df['passenger_count'] == 1].mean()

Missing Value Analysis

Now, we need to check the number of missing values in all the variables.

missing_value = pd.DataFrame(df.isnull().sum())

missing_value

I always think it is important to impute missing values rather than deleting or removing them. Here I have used the method of “most frequently occurring data” for each column to replace the respective missing values.

imp = SimpleImputer(missing_values=np.nan, strategy=’most_frequent’)

train_data=pd.DataFrame(imp.fit_transform(df), columns=df.columns)

print(f’Row count after imputing missing values — {train_data.shape[0]}’)

Checking Correlation among variables through diagram

It is very important to check if any variable is highly correlated to each other or not. If so, then we shouldn’t consider those variables instead we will select only one variable between them for the required data frame to predict the dependent variable.

df.corr(method=’pearson’).style.format(“{:.2}”).background_gradient(cmap=plt.get_cmap(‘coolwarm’), axis=1)

We have used the corr() function and a diagrammatic representation of all our variables to observe if they are highly correlated with any other variable in our data set.

After cleaning and removing certain data points, we get the required data frame on which we can perform a little deeper analysis.

Here, we compare two variables through a contingency table :

pd.crosstab(df['passenger_count'], df['fare_amount'], normalize = True)

Let’s also take a look at the average numbers of person taking ride in a particular area measured by latitudes and longitudes:

df.pivot_table(['pickup_latitude', 'dropoff_latitude', 'pickup_longitude','dropoff_longitude'], ['passenger_count'], aggfunc='mean')

Now our data set is ready for machine learning!