Tutorial: Exploratory Data Analysis (EDA) with Categorical Variables
By: Erin Hoffman
Tags: Python, Exploratory Data Analysis, Matplotlib, Tutorial, Data Science

Tutorial: Plotting EDA with Matplotlib and Seaborn

Code to load in the Titanic dataset (CSV file located in this GitHub repo):

import pandas as pd

import numpy as np



import matplotlib.pyplot as plt

from matplotlib.patches import Patch

from matplotlib.lines import Line2D

import seaborn as sns df = pd.read_csv("titanic.csv")



# PassengerId is a dataset artifact, not something useful for analysis

df.drop("PassengerId", axis=1, inplace=True)



# We want to use Age as one of the main examples, drop rows that are missing Age values

df.dropna(subset=["Age"], inplace=True)

Note that we have not performed a train-test split or even separated X (features) from y (target) — everything is in one dataframe. This is for compatibility with Seaborn, but in a real machine learning context you will need to perform additional steps to ensure you avoid leakage and other undesired side effects of sloppy data preparation.

Note also that in all instances we use blue to represent “survived”, green to represent “did not survive”, yellow to represent “1st class”, orange to represent “2nd class”, and red to represent “3rd class”. When showing different representations of the same data, it’s important to use meaningful colors to help the reader follow the meaning of the visualization.

Numeric vs. Categorical EDA

These code snippets represent alternatives for the first scatter plot shown above, plotting Age (a numeric value) against the target Survived (a categorical value).

Multiple Histograms

Rather than using the y -axis to represent the two categories, let’s use two different colors. That means that we can use the y -axis to represent counts rather than trying to discern this information from the density of dots.

fig, ax = plt.subplots()



ax.hist(df[df["Survived"]==1]["Age"], bins=15, alpha=0.5, color="blue", label="survived")

ax.hist(df[df["Survived"]==0]["Age"], bins=15, alpha=0.5, color="green", label="did not survive")



ax.set_xlabel("Age")

ax.set_ylabel("Count of passengers")



fig.suptitle("Age vs. Survival for Titanic Passengers")



ax.legend();

Multiple Density Estimate Plots

This is showing largely the same information as the histograms, except that it’s a density estimate (estimate of the probability density function) rather than a count across bins. Seaborn has nice built-in functionality for this.

fig, ax = plt.subplots()



sns.kdeplot(df[df["Survived"]==1]["Age"], shade=True, color="blue", label="survived", ax=ax)

sns.kdeplot(df[df["Survived"]==0]["Age"], shade=True, color="green", label="did not survive", ax=ax)



ax.set_xlabel("Age")

ax.set_ylabel("Density")



fig.suptitle("Age vs. Survival for Titanic Passengers");

Multiple Box Plots

Here we lose some of the information about the distribution overall in order to focus in on particular summary statistics of the distribution.

Dcbmariano / CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)

fig, ax = plt.subplots()



sns.boxplot(x="Age", y="Survived", data=df, orient="h", palette={1:"blue", 0:"green"}, ax=ax)



ax.get_yaxis().set_visible(False)



fig.suptitle("Age vs. Survival for Titanic Passengers")



color_patches = [

Patch(facecolor="blue", label="survived"),

Patch(facecolor="green", label="did not survive")

]

ax.legend(handles=color_patches);

Categorical vs. Categorical EDA

These code snippets represent alternatives for the second scatter plot shown above, plotting Pclass (a categorical value) against the target Survived (a categorical value).

Grouped Bar Charts

This shows the distribution across the categories, similar to the “multiple histograms” example for numeric vs. categorical.

First, an example where Pclass is represented along the x -axis and Survival is represented by color, essentially answering the question for each passenger class, how many passengers survived vs. did not survive?

fig, ax = plt.subplots()



sns.catplot("Pclass", hue="Survived", data=df, kind="count",

palette={1:"blue", 0:"green"}, ax=ax)



plt.close(2) # catplot creates an extra figure we don't need



ax.set_xlabel("Passenger Class")



color_patches = [

Patch(facecolor="blue", label="survived"),

Patch(facecolor="green", label="did not survive")

]

ax.legend(handles=color_patches)



fig.suptitle("Passenger Class vs. Survival for Titanic Passengers");

Second, an example where Survival is represented along the x -axis and Pclass is represented by color, essentially answering the question for passengers who survived vs. did not survive, how many were in each passenger class?

fig, ax = plt.subplots()



sns.catplot("Survived", hue="Pclass", data=df, kind="count",

palette={1:"yellow", 2:"orange", 3:"red"}, ax=ax)



plt.close(2) # catplot creates an extra figure we don't need



ax.legend(title="Passenger Class")

ax.set_xticklabels(["did not survive", "survived"])

ax.set_xlabel("")



fig.suptitle("Passenger Class vs. Survival for Titanic Passengers");

Stacked Bar Charts

These can be used for counts (same as grouped bar charts) but if you use percentages rather than counts, they show proportions. Basically like a pie chart but better, because you are visually comparing heights rather than angles.

This requires a bit of dataframe manipulation to prepare for plotting:

# Create a dataframe with the counts by passenger class and survival

# ["Name"] could be any column within the data; it is the equivalent

# of COUNT(id) in SQL counts_df = df.groupby(["Pclass", "Survived"])["Name"].count().unstack()

First, an example where Pclass is represented along the x -axis and Survival is represented by color, essentially answering the question for each passenger class, what proportion of passengers survived vs. did not survive?

# Transpose so survived + did not survive add up to 1, divide by the total number (transposed), then transpose one more time for plotting

survived_percents_df = counts_df.T.div(counts_df.T.sum()).T fig, ax = plt.subplots()



survived_percents_df.plot(kind="bar", stacked=True, color=["green", "blue"], ax=ax)



ax.set_xlabel("Passenger Class")

ax.set_xticklabels([1, 2, 3], rotation=0)

ax.set_ylabel("Proportion")



color_patches = [

Patch(facecolor="blue", label="survived"),

Patch(facecolor="green", label="did not survive")

]

ax.legend(handles=color_patches)



fig.suptitle("Passenger Class vs. Survival for Titanic Passengers");

Second, an example where Survival is represented along the x -axis and Pclass is represented by color, essentially answering the question for passengers who survived vs. did not survive, what proportion were in each passenger class?

# Divide by the total number and transpose for plotting

pclass_percents_df = counts_df.div(counts_df.sum()).T fig, ax = plt.subplots()



pclass_percents_df.plot(kind="bar", stacked=True, color=["yellow", "orange", "red"], ax=ax)



ax.legend(title="Passenger Class")

ax.set_xticklabels(["did not survive", "survived"], rotation=0)

ax.set_xlabel("")

ax.set_ylabel("Proportion")



fig.suptitle("Passenger Class vs. Survival for Titanic Passengers");

Numeric vs. Numeric vs. Categorical EDA

Sometimes it’s interesting to see the relationship between two different numeric features and the target, not just one at a time. This code snippet represents an alternative for the third scatter plot shown above, plotting Age (a numeric feature) vs. Fare (a numeric feature) vs. Survived (a categorical target).

fig, ax = plt.subplots(figsize=(10, 5))



ax.scatter(df[df["Survived"]==1]["Age"], df[df["Survived"]==1]["Fare"], c="blue", alpha=0.5)

ax.scatter(df[df["Survived"]==0]["Age"], df[df["Survived"]==0]["Fare"], c="green", alpha=0.5)



ax.set_xlabel("Age")

ax.set_ylabel("Fare")



color_patches = [

Line2D([0], [0], marker='o', color='w', label='survived', markerfacecolor='b', markersize=10),

Line2D([0], [0], marker='o', color='w', label='did not survive', markerfacecolor='g', markersize=10)

]

ax.legend(handles=color_patches)



fig.suptitle("Survival by Age and Fare for Titanic Passengers");

Summary

Most of the time if your target is a categorical variable, the best EDA visualization isn’t going to be a basic scatter plot. Instead, consider:

Numeric vs. Categorical (e.g. Survived vs. Age )

Multiple histograms

Multiple density estimate plots

Multiple box plots

Categorical vs. Categorical (e.g. Survived vs. Pclass )

Grouped bar charts

Stacked bar charts

Numeric vs. Numeric vs. Categorical (e.g. Age vs. Fare vs. Survived )