Exploratory Data Analysis(EDA) on Diabetes Data Set
By: Maruthi Ram
Tags: Exploratory Data Analysis, Diabetes, Machine Learning, Artificial Intelligence, Python

EDA plays a crucial role in building any data model. There is no single way to perform EDA on a data set. As the name says we have to explore different means to find the patterns in the data and arrive at meaningful inferences.

Let us start doing EDA on a very serious problem the world is facing today.

DIABETES -More than 88 million US adults — over a third — have prediabetes, and more than 84% of them don’t know they have it

We have taken the diabetes data set from Kaggle for our analysis today. You can download it by following the above link.

The first and foremost important thing in EDA is setting a goal. In the journey, you may see a lot of other information but remember you should never deviate from your goal. With the kind of analysis we perform, it is very easy to get lost in the swamp of information.

Let’s get started

1. Setting the Goal:

Our goal today is to draw meaningful inferences which in turn will be used in identifying Diabetes in a person.

2. Reading the Data

Import all the related libraries

Python Libraries

3. Understanding the Data

Spend enough time in understanding the data. If possible, try gathering some basic domain knowledge on the dataset you are working on. This will be a huge added advantage in your data analysis.

# 3.1. Reviewing the data frame info

This operation helps us understand some of the key information like names of the columns, data type of each column, the total number of records, no. of columns, etc.,

This helps us sense check the tidiness of data. For example, the age field should always be int type, it cannot be of string type.

# 3.2. See the top 5 records to better understand the data

Observations:

When we see the above table, values for Glucose, Blood Pressure, SkinThickness, Insulin are 0, which is not possible in the real world. So, as part of our data cleansing exercise(covered in the next steps), we have to drop all such records from our data frame.

# 3.3. Describe various properties of the data

# 3.4. Try to understand the outcomes

Observations:

There are only two outcomes 0 and 1, 0 refers to No Diabetes and 1 means the person is having diabetes

The data set looks quite imbalanced as there are 1316 people who are healthy and just 684 people who have diabetes.

# 3.5. To check if there are any null values in the data set

There are no null values, we are good with it. If there are null values in the data set based on what is more meaningful to you, you can either drop that particular record or replace the null value with mean or median.

4. Cleansing the Data

The hard fact is that real-world data is never perfect. You might need to spend a lot of time cleaning the data and bringing it to a near-perfect state.

# 4.1. Based on our observation from the above analysis we have to drop a few records which are having 0 values. With the simple below code snippet, we drop all the records from the data frame where either Insulin or Blood Pressure or Skin Thickness or BMI, or Glucose levels are 0

# 4.2. Let us try to once again see the data after dropping the records with 0 values for Insulin, Blood Pressure, Skin Thickness, BMI, and glucose levels

We can see from the above table that the “min” values for Insulin, Blood Pressure, Skin Thickness, BMI, and glucose levels are no more 0’s. Our data cleansing worked. :-)

5. Analyzing the Data

Bivariate Analysis:

In Bivariate analysis, we analyze the outcomes w.r.t two variables using scatter and pair plots

# 5.1. Scatter Plot

Let us draw a simple scatterplot by projecting outcomes based on two features Glucose and Insulin

The outcomes are all over the place. We need some superpowers to classify the outcomes based on these two features. Let's plot similar graphs for all the features.

Wait!, What? I have to do this 8C2 times = 28 graphs :-(

# 5.2. Pair Plots

Some good news! a pair plot can do it all for you with just a single line of code

sns.pairplot(df,hue=’Outcome’)

Pair Plot

The screenshot is so big, so I placed just a part of it. (Try to plot the above graph in your machine).

Once you plot the graphs, you will be able to see that (Glucose, Insulin),(SkinThickness, BMI) are useful in classifying the outcomes.

x-Insulin, y-Glucose

x- BMI, y- Skin Thickness

Although not so accurate, these features are better than others. To support this analysis let us also plot a Spearman Rank correlation coefficient heatmap to find the relation between the variables.

# 5.3. Spearman Rank correlation coefficient

Close to 1 indicates a very good relationship

Close to -1 indicates a very poor relationship

From the above graph, we can see that the relationship is very strong for the below features

Age-Pregnancies

SkinThickness-BMI

Glucose-Insulin

In addition to what we have observed in Pair plot analysis, we also found a strong relationship between age and pregnancy. Let us take all the 6 features and try to analyze them individually using Univariate analysis

Univariate Analysis

In Univariate analysis, we analyze the outcomes w.r.t one variable at a time.

# 5.4. Density Plot or Distribution Plot

A simple density plot to understand how Glucose is affecting the outcomes. We can see there is a lot of overlap.

The same plot can be generated for all other features to identify the patterns.

Observations:

The curve is right-skewed which indicates there are outliers.

# 5.5. Splitting Data Frames based on Outcomes

Now, we split the data into two different data frames based on the outcomes, this will be helpful in performing univariate analysis.

All the records with diabetes are copied to data frame Diab_Yes

All the records with no diabetes are copied to data frame Diab_No

# 5.6. PDF and CDF

Now, we plot PDF and CDF of the features to identify the underlying patterns.

Not very useful. The same plot can be generated for other features like Insulin, Pregnancies, BMI, Skin Thickness, Age to find the patterns

# 5.7. Box Plots

With the below simple code snippet we can generate box plots for all the features.

Below is a screenshot of the box plot for the Glucose feature.

Box Plot

Observations:

There are outliers in the Glucose feature data.

The Median of Glucose levels for Healthy people is ~110, whereas for a person with Diabetes is ~145

If Glucose levels are <=~80, no diabetes

If Glucose levels are> ~80 and <~125, then there is a very low probability of having diabetes

If Glucose levels >= ~125 and <~175, then there is a high probability of having diabetes

If Glucose levels are >~175, then the person has diabetes

Conclusion

The glucose levels may not alone help us in diagnosing diabetes. If you continue to do this analysis on other features as well, the predictions can be more accurate. Not all plots work for all the data, we have to try and test to find which fits the best!

References

https://www.appliedroots.com/

https://www.kaggle.com/uciml/pima-indians-diabetes-database

https://www.analyticsvidhya.com/blog/2021/02/introduction-to-exploratory-data-analysis-eda/