Exploratory Data Analysis on Diabetes dataset
By: Saurav Anand
Tags: Machine Learning, Data Science, Data Visualization, Data Analysis

“If you torture the data long enough, it will confess.”

― Ronald H. Coase, Essays on Economics and Economists

Background

Every data has lot of hidden information. These hidden information was required to be investigated to find out the hidden patterns . These patterns can be helpful in making decisions on the procedure , removal of any ambiguity and also in getting key business insights. To solve all this questions, exploratory data analysis was introduced.

Introduction

Exploratory data analysis is all about getting and overall understanding of data. It is mainly done to find it’s properties , patterns and visualizations. It helps us to assure that our data is correct and ready to use for machine learning algorithms.

In this blog we are using python as our programming language for the analysis purpose. Python has a wide variety of libraries like pandas , seaborn , numpy , matplotlib which can be used for this purpose. We are using the Pima Indians Dataset where it shows the various diagnostic factors influencing the diabetes.

Brief Introduction to the used libraries

As discussed above , we are going to use the following libraries to perform different operations on the data

Understanding the data :-

This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective of the dataset is to diagnostically predict whether or not a patient has diabetes, based on certain diagnostic measurements included in the dataset. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage.

Content

The datasets consists of several medical predictor variables and one target variable, Outcome. Columns are following :-

Pregnancies :- Number of times pregnant Glucose:- Plasma glucose concentration a 2 hours in an oral glucose tolerance test BloodPressure:- Diastolic blood pressure (mm Hg) SkinThickness:- Triceps skin fold thickness (mm) Insulin:- 2-Hour serum insulin (mu U/ml) BMI:- Body mass index (weight in kg/(height in m)²) DiabetesPedigreeFunction:- Diabetes pedigree function Age:-Age in years Outcome:- Class variable (0 or 1) 268 of 768 are 1, the others are 0

Importing the necessary libraries:-

In order to use the libraries , we need to import the libraries .

Loading the dataset

Meet and Greet the data

In this section , we will explore the data and try to find out it’s characteristics

Collecting Basic Information about the Data

It shows that there are eight independent variables (Pregnancies,Glucose,Blood Pressure , SkinThickness ,Insulin, BMI, DiabetesPedigreeFunction, Age) and one dependent variable (Outcome).

It shows the number of rows, number of columns, data types information, Memory usage, number of null values in each column.

Descriptive statistics using describe method as shown below

Pandas Profiling to get the overview of data

The above steps can be replaced by simply using pandas profiling library which generated profile report from a dataframe

Handling the missing values

We need to check the presence of the missing values and need to replace them with mean, median and mode. Sometimes we have null values in the form of 0 , so we need to convert them to NaN and then replace them accordingly. The missing values can be removed also but it should be less than 5 percent of the whole dataset.

Data Visualization

Visualizing data in different type of graphs will provide us with greater insights into our data. We will explore different options on visualizing our data and find out any patterns between within it.

Visualizing the outcome column

Visualizing the Outliers

Pairplot, Correlation matrix and HeatMap

The plot shows that there is some relationship between parameters. Outcome is added as hue. We see that blue and orange dots are overlap. Also,

Pregnancies and age have some kind of a linear line. BloodPressure and age have little relation. Most of the aged people have BloodPressure. Insulin and Glucose have some relation.

The correlation plot shows the relation between the parameters.

Glucose,Age,BMI and Pregnancies are the most correlated parameters with the Outcome. Insulin and DiabetesPedigreeFunction have little correlation with the outcome. BloodPressure and SkinThickness have tiny correlation with the outcome. There is a little correlation between Age and Pregnancies,Insulin and Skin Thickness, BMI and Skin Thickness,Insulin and Glucose

Analysis of each parameter w.r.t outcome parameter

“Uni” means “one”, so in other words your data has only one variable. It doesn’t deal with causes or relationships. It’s major purpose is to describe; it takes data, summarizes that data and finds patterns in the data. This means here we deal and with only one variable or column of the data and try to find out it’s nature

Analysis of ‘Pregnancies’ parameter

Similarly we can analyze the each parameter with respect to the outcome of the dataset

Analysis of each parameter with each respect to each other

In the above section we have drawn the heatmap and the correlation matrix.From there we can find out which variables are highly related to each other .

Analysis of Age and pregnancies

Following link contains the full code in kaggle which I have written :-

Conclusion

Hence we can say from the above that EDA is very important as it helps us in achieving the following :-

It helps in detection of mistakes (like missing values and outliers) It determines relationships between explanatory variables Assessing the direction and rough size of relationships between explanatory and outcome variables. It makes our data ready for machine learning algorithm

Happy reading,

Saurav Anand