A-Z Of Exploratory Data Analysis Under 10 mins
By: Ramya Vidiyala
Tags: Machine Learning, Technology, Artificial Intelligence, Education, Data Science

3. UNIVARIATE ANALYSIS

Univariate analysis, as the name says, simply means analysis using a single variable. This analysis gives the frequency/count of occurrences of the variable and lets us understand the distribution of that variable at various values.

3.1. PROBABILITY DENSITY FUNCTION (PDF) :

In PDF plot, X-axis is the feature on which analysis is done and the Y-axis is the count/frequency of occurrence of that particular X-axis value in the data. Hence the term “Density” in PDF.

import seaborn as sns

sns.set_style("whitegrid")

Seaborn is the library that provides various types of plots for analysis.

sns.FacetGrid(haberman_data,hue='surv_status',height=5).map(sns.distplot,'age').add_legend()

Output :

PDF of Age

Observations :

Major overlapping is observed, so we can not clearly say about the dependency of age on survival. A rough estimate that patients age 20–50 have a slightly higher rate of survival and patients age 75–90 have a lower rate of survival. Age can be considered as a dependent variable.

sns.FacetGrid(haberman_data,hue='surv_status',height=5).map(sns.distplot,'op_year').add_legend()

Output :

PDF of Operation Year

Observations:

The overlap is huge. Operation year alone is not a highly dependent variable.

sns.FacetGrid(haberman_data,hue='surv_status',height=5).map(sns.distplot,'axil_nodes').add_legend()

Output :

PDF of Axillary nodes

Observations:

Patients with 0 nodes have a high probability of survival. Axillary nodes can be used as a dependent variable.

The disadvantage of PDF: In PDF, we can’t say exactly how many data points are in a range/ lower to a value/ higher than a particular value.

3.2. CUMULATIVE DENSITY FUNCTION (CDF) :

To know the number of data points below/above a particular value, CDF is very useful.

Let’s start with segregating data according to the class of survival rate.

survival_yes = haberman_data[haberman_data['surv_status']==1]

survival_no = haberman_data[haberman_data['surv_status']==2]

Now, let us analyze these segregated data sets.

import numpy as np

import matplotlib.pyplot as plt

count, bin_edges = np.histogram(survival_no['age'], bins=10, density = True)

#count : the number of data points at that particular age value

#bin_edges :the seperation values of the X-axis (the feature under analysis)

#bins = the number of buckets of seperation

pdf = count/sum(count)

print(pdf)

# To get cdf, we want cumulative values of the count. In numpy, cumsum() does cumulative sum

cdf = np.cumsum(pdf)

print(cdf) count, bin_edges = np.histogram(survival_yes['age'], bins=10, density = True)

pdf2 = count/sum(count)

cdf2 = np.cumsum(pdf2) plt.plot(bin_edges[1:],pdf,label='yes')

plt.plot(bin_edges[1:], cdf,label='yes')

plt.plot(bin_edges[1:],pdf2,label='no')

plt.plot(bin_edges[1:], cdf2,label='no')

plt.legend()

#adding labels

plt.xlabel("AGE")

plt.ylabel("FREQUENCY")

Output :

[0.03703704 0.12345679 0.19753086 0.19753086 0.13580247 0.12345679

0.09876543 0.04938272 0.02469136 0.01234568]

[0.03703704 0.16049383 0.35802469 0.55555556 0.69135802 0.81481481

0.91358025 0.96296296 0.98765432 1. ] Text(0, 0.5, 'FREQUENCY')

CDF, PDF of segregated data on age

Observations:

There are around 80% of data points have age values less than or equal to 60

count, bin_edges = np.histogram(survival_no['axil_nodes'], bins=10, density = True)

pdf = count/sum(count)

print(pdf) cdf = np.cumsum(pdf)

print(cdf) count, bin_edges = np.histogram(survival_yes['axil_nodes'], bins=10, density = True)

pdf2 = count/sum(count)

cdf2 = np.cumsum(pdf2) plt.plot(bin_edges[1:],pdf,label='yes')

plt.plot(bin_edges[1:], cdf,label='yes')

plt.plot(bin_edges[1:],pdf2,label='no')

plt.plot(bin_edges[1:], cdf2,label='no')

plt.legend()

plt.xlabel("AXIL_NODES")

plt.ylabel("FREQUENCY")

Output :

[0.56790123 0.14814815 0.13580247 0.04938272 0.07407407 0.

0.01234568 0. 0. 0.01234568]

[0.56790123 0.71604938 0.85185185 0.90123457 0.97530864 0.97530864

0.98765432 0.98765432 0.98765432 1. ] Text(0, 0.5, 'FREQUENCY')

CDF, PDF of segregated data on axillary nodes

Observations:

There are around 90% of data points have axil_node values less than or equal to 10

3.3. BOX PLOTS

Before exploring box plots, few commonly used statistics terms are,

median (50th quartile) is the middlemost value of the sorted data

25th quartile is the value in sorted data which has 25% of the data less than it and 75% of the data above it

75th quartile is the value in sorted data which has 75% of the data less than it and 25% of the data above it.

In the box plot, the lower line represents the 25th quartile, the middle line represents the Median/50th quartile, the upper line represents the 75th quartile. And the whiskers represent the minimum and maximum in most of the plots or some complex statistical values. Whiskers are not min and max values, when seaborn is used.

sns.boxplot(x='surv_status',y='age', data=haberman_data)

sns.boxplot(x='surv_status',y='axil_nodes', data=haberman_data)

sns.boxplot(x='surv_status',y='op_year', data=haberman_data)

Output :