ABC of EDA with Haberman’s Survival Dataset
By: Mahavir Dabas
Tags: Machine Learning, Data Visualization, Data Science, Exploratory Data Analysis

Before continuing the analysis let me familiarise you with some keywords related to a dataset which I will be using from here on. Any dataset has the following characteristics-

Datapoint/Vector/Observation- Each row in the data frame is a data point or an observation. It is one of the instances that has been recorded. If there are n features in a data point, we call it an n-dimensional vector. Feature/ Input Variable/ Independent Variable- It is that component of the data point that is used as an indicator for prediction. Our dataset has 3 features- the age of the patient, the year of the surgery, and the number of axillary/lymph nodes present in the body. Label/ Class/ Output variable/ Dependent Variable- The one thing for which we are ready to enter this mess for- the value/result that we need to predict after we are done with everything. The label is predicted on the basis of the features and thus is named as the dependent variable. Here, we need to predict the chances of survival of a cancer patient which acts as a label in our case.

To identify how many datapoints of each class are present, we run a simple command-

g=haberman.groupby(“survival_status”).size() g

We can observe that in our dataset, there are 225 instances of the patient surviving and 81 instances of the patient succumbing to the disease. We can conclude that our dataset is an unbalanced dataset, which is a common observation in the case of medical records.

Note that the total observations present ( 306) can be found out with this simple command-

haberman.shape

Now that we are done familiarising ourselves with our data, let’s move forward to the real stuff.

Multivariate Analysis-3-D Scatter Plot

Since our dataset has 3 features, why not quickly see how a 3-d scatter plot of the same would look like- maybe we’ll be done with our problem in this very step?

import plotly.express as px fig=px.scatter_3d(data_frame=haberman,x=”age”,y=”year”,z=”axillary_nodes”,color=”survival_status”)

fig.show()

As you can see for yourself that no procedure ever defined by humanity can help us in making sense out of this mess. So to answer the previous question-No, we won’t be done with our problem in the very first step!

Bivariate Analysis

It refers to finding the relationship between 2 feature variables and using it to find the target variable of interest. The most used technique in the bivariate analysis is that of the 2-d scatter plots. Pair plots are the collection of 2-d scatter plots of all possible pairs that can be generated from our features.

The motive behind using the pair plots would be to identify those features which result in the best distinction between the 2 class labels-survival status: yes and survival status: no. I used seaborn for plotting the pair plots-

sns.set_style(“whitegrid”) sns.pairplot(haberman,hue=”survival_status”,size=3)

plt.show()

Note: Only consider the graphs above the diagonal as the ones below it are the same graphs with a reversal of axes which won’t affect our observation.

Note: The general workflow while dealing with pair plots is to select the graphs with the least overlap between the various classes and then choosing the features involved in these graphs for further classification to achieve the best possible results.

Observations from Pair Plots-

Age vs Year of Operation- Highly overlapping and thus discarded. Year of Operation vs Number of Axillary Nodes-Highly overlapping and thus discarded. Age vs Number of Axillary Nodes- Overlapping but better than the abovementioned cases. Therefore we’ll select these 2 features for univariate analysis.

Limitation of Pair Plots- When the number of features reaches a very large value, it is not possible to use Pair Plots for classification in an efficient manner as we would have to go through a large number of plots to identify our best features. In such cases, we use certain mathematical techniques like PCA and t-SNE ( which will be the topic of discussion of the next article!) for dimensionality reduction.

Conclusion- The age of the patient and the Number of Axillary Nodes are the best features to predict the chances of survival.

Univariate Analysis

Univariate analysis is perhaps the simplest form of statistical analysis. The key fact is that only one variable is involved.

One thing that has become extremely relevant in today’s world is the concept of diplomacy, which is also ingrained in the bodies and minds of all the data scientists around the world. Instead of actually predicting what will happen, we always prefer talking about the chances ( diplomacy!). And the moment we start talking about chances is when probability kicks in.

Probability Density Function- It represents the density of data points over a continuous range helping us determine the likelihood of a particular outcome. It is formed by smoothing the edges of a histogram.

PDF for Age of Patient

sns.FacetGrid(haberman,hue=”survival_status”,size=5)\ .map(sns.distplot,”age”)\.add_legend();

plt.title(“pdf for survival status labelling on the basis of age”) plt.show();

Observations-

We can see that the PDF of survival status on the basis of age is highly overlapping for both classes between ages 40 and 65 years (the area under the curves is almost equal). Therefore between ages 40 and 65 years, the percentage of patients with survival status as yes and the percentage of patients with survival status no is almost the same-thus this feature is not suitable for classification.

PDF for Year of Operation

sns.FacetGrid(haberman,hue="survival_status",size=5)\ .map(sns.distplot,"year")\.add_legend();\

plt.title("pdf for survival status labelling on the basis of year of operation")

plt.show();

Observations-

We can see that the pdf for survival status on the basis of the year of operation is again highly overlapping for both the classes between the years 1958 and 1968. Therefore between the years 1958 and 1968, the percentage of patients with survival status as yes and the percentage of patients with survival status no is almost the same-thus this feature is also not suitable for classification.

PDF for Number of Axillary Nodes

sns.FacetGrid(haberman,hue="survival_status",size=5)\ .map(sns.distplot,"axillary_nodes")\.add_legend();

plt.title("pdf for survival status labelling on the basis of number of axillary nodes")

plt.show();

Observations-

The PDF for the survival status on the basis of the number of axillary nodes has the least overlap for the 2 label classes and is our best shot at making the best possible classification. Thus we can make use of the following if-else conditions to get a fairly accurate result-

if no of axillary nodes>=0 and <=3 →survival status=high if no of axillary nodes>=3 →survival status →low

Detailed Univariate Analysis on the basis of Number of Axillary Nodes

Now knowing our best feature for classification, we have the luxury to go for a more detailed analysis to get some concrete results.