Data Exploration and Analysis Using Python
By: Raji Rai
Tags: Python, Data Analysis, Data Science, Data Visualization, Data Exploration

Data Exploration and Analysis Using Python

Simple ways to make your data talk

Data exploration is a key aspect of data analysis and model building. Without spending significant time on understanding the data and its patterns one cannot expect to build efficient predictive models. Data exploration takes major chunk of time in a data science project comprising of data cleaning and preprocessing.

In this article, I will explain the various steps involved in data exploration through simple explanations and Python code snippets. The key steps involved in data exploration are:

> Load data

> Identify variables

> Variable analysis

> Handling missing values

> Handling outliers

> Feature engineering

Load data and Identify variables:

Data sources can vary from databases to websites. Data sourced is known as raw data. Raw data cannot be directly used for model building, as it will be inconsistent and not suitable for prediction. It has to be treated for anomalies and missing values. Variable can be of different types such as character, numeric, categorical, and continuous.

Variable Type

Identifying the predictor and target variable is also a key step in model building. Target is the dependent variable and predictor is the independent variable based on which the prediction is made. Categorical or discrete variables are those that cannot be mathematically manipulated. It is made up of fixed values such as 0 and 1. On the other hand, continuous variables can be interpreted using mathematical functions like finding the average or sum of all values. You can use a series of Python codes to understand the types of variables in your dataset.

#Import required libraries

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns #Load the data

titan=pd.read_csv("../input/titan.csv") #get an overview of the data

titan.head()

titan.tail()

titan.sample(10)



#identify variable type

titan.dtypes

titan.info()

titan.describe()

Variable Analysis:

Variable analysis can be done in three ways, univariate analysis, bivariate analysis, and multivariate analysis.

Variable Analysis

Univariate analysis is used to highlight missing and outlier values. Here each variable is analysed on its own for range and distribution. Univariate analysis differs for categorical and continuous variables. For categorical variables, you can use frequency table to understand distribution of each category. For continuous variables, you have to understand the central tendency and spread of the variable. It can be measured using mean, median, mode, etc. It can be visualized using box plot or histogram.

#Understand various summary statistics of the data

include =['object', 'float', 'int']

titan.describe(include=include)

titan.describe() #Get count of values in a categorical variable

titan.survived.value_counts() titan.age.hist(figsize=(10,5))

Histogram

Bivariate Analysis is used to find the relationship between two variables. Analysis can be performed for combination of categorical and continuous variables. Scatter plot is suitable for analyzing two continuous variables. It indicates the linear or non-linear relationship between the variables. Bar charts helps to understand relation between two categorical variables. Certain statistical tests are also used to effectively understand bivariate relationship. Scipy library has extensive modules for performing these tests in Python.

Bivariate Analysis

Matplotlib and Seaborn libraries can be used to plot different relational graphs that help visualizing bivariate relationship between different types of variables.

Scatter Plot

iris = sns.load_dataset("iris")

sns.relplot(x = 'sepal_length', y = 'petal_length', hue='species',data = iris) relplot = sns.catplot(x="pclass", hue="who", col="survived",

data=titan, kind="count",

height=4, aspect=.7);

relplot

Handling Missing Values:

Missing values in the dataset can reduce model fit. It can lead to a biased model as the data cannot be analysed completely. Behavior and relationship with other variables cannot be deduced correctly. It can lead to wrong prediction or classification. Missing values may occur due to problems in data extraction or data collection, which can be categorized as MCAR, MAR, and NMAR.

Missing Values

Missing values can be treated by deletion, mean/mode/median imputation, KNN imputation, or using prediction models.

Handling Missing Values

You can visually analyse the missing data using a library called as Missingno in Python.

import missingno as msno

msno.bar(titan)

msno.heatmap(titan) np.mean(titan['age']) from scipy import stats

stats.mode(titan['embarked']) titancopy['age'].fillna(29,inplace=True)

titancopy['embarked'].fillna("S", inplace=True)

Handling Outliers:

Outliers can occur naturally in a data or can be due to data entry errors. They can drastically change the results of the data analysis and statistical modeling. Outliers are easily detected by visualization methods, like box-plot, histogram, and scatter plot. Outliers are handled like missing values by deleting observations, transforming them, binning or grouping them, treating them as a separate group, or imputing values.

Box Plot

import plotly.express as px

fig = px.box(titan,x='survived',y='age', color='pclass')

fig.show() px.box(titan, y='age') px.box(titan,x='survived',y='fare', color='pclass') #Adding trendline to the data

x=iris.sepal_length

y=iris.petal_width

plt.scatter(x, y)



z = np.polyfit(x, y, 1)

p = np.poly1d(z)

plt.plot(x,p(x),"y--")



plt.show()

Feature Engineering:

Feature engineering is the process of extracting more information from existing data. Feature selection also can be part of it. Two common techniques of feature engineering are variable transformation and variable creation. In variable transformation existing variable is transformed using certain functions. For example, a number can be replaced by its logarithmic value. Another technique is to create a new variable from the existing variable. For example, breaking the date field in the format of dd/mm/yy to date, month and year columns.

Variable Transformation

titancopy = titan.copy() #variable transformation

titancopy['alive'].replace({'no':0,'yes':1}, inplace=True) #Convert boolean to integer

titancopy["alone"]=titancopy["alone"].astype(int)

Two other data transformation techniques are encoding categorical variables and scaling continuous variables to normalize the data. This depends on the model that is used for evaluation, as some models accept categorical variables.

Irrelevant features can decrease the accuracy of the model. Feature selection can be done automatically or manually. A correlation matrix is used to visualize how the features are related to each other or with the target variable.

Correlation Matrix

titancopy.corr() plt.figure(figsize=(10,10))

corr = titan.corr()

ax = sns.heatmap(

corr,

vmin=-1, vmax=1, center=0,

cmap=sns.diverging_palette(20, 220, n=200),

square=True, annot=True

)

ax.set_xticklabels(

ax.get_xticklabels(),

rotation=45,

horizontalalignment='right'

)

ax.set_yticklabels(

ax.get_yticklabels(),

rotation=45,



);

The scikit-learn library provides few good classes such as SelectBest to select specific number of features from the given dataset. The tree-based classifier in the same library can be used to get the feature importance scores.

This covers some of the key steps involved in data exploration. Each of these steps can be reiterated depending on the size of the data and the requirement of the model. Data scientists spend the maximum amount of time in data preprocessing as data quality directly impacts the success of the model.

All the code snippets shown here are executed in the Exploratory Data Analysis and Visualization Kaggle notebook.