Marvel vs DC: Comparative Gender Analysis using Python
By: Nehla Shajahan
Tags: Exploratory Data Analysis, Gender, Marvel, Comics, Dc

Performing Exploratory Data Analysis techniques using Python to compare the characters of Marvel and DC with a focus on their gender.

“You know what’s cooler than magic? Math” — Peter Parker (Spider-Man)

Photo by Waldemar Brandt on Unsplash

If you happen to be a comic geek, it is pretty sure that you might have had ‘the Marvel vs DC’ talk with fellow geeks. Although this rivalry is assumed to last till eternity, let’s look at this from a Machine Learning perspective to gain some insights regarding how these comics evolved in terms of gender and other factors.

Exploratory Data Analysis or EDA in short is one of the first things you do when you encounter a dataset. Wikipedia defines EDA as follows:

“Exploratory Data Analysis is an approach of analyzing datasets to summarize their main characters, often using statistical graphs and other data visualization methods”

EDA is considered to be an investigative tool used to delve further into the data and look for hidden insights. It is widely advisable to perform multiple EDA methods to explore the data to get a sense of the story it is aiming to tell.

Introduction

The analysis was inspired by the article Comic Books are still Made by Men, For Men And About Men. The data set used can be downloaded here.

The data was copied was from two public fan websites and both the datasets, Marvel and DC contain the following features.

Feature description of the datasets

To begin with, our analysis, let us import all the required libraries for our Exploratory Data Analysis.

#Importing the required libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

%matplotlib inline import plotly.graph_objs as go

import plotly.express as px

import plotly

plotly.offline.init_notebook_mode(connected=True)

We load the required two datasets into our python environment using the following code and store them in two variables namely, ‘marvel’ and ‘dc’:

#Loading the dataset

marvel = pd.read_csv("marvel-wikia-data.csv")

dc = pd.read_csv("dc-wikia-data.csv")

First, five entries of the marvel dataset using ‘marvel.head()’

First, five entries of the DC dataset using ‘dc.head()’

Performing some rudimentary analysis, we get the following results,

print(marvel.shape)

print(dc.shape)

The above code will give us (16376, 13); (6896, 13) as the output. The Marvel universe has twice the number of characters as the DC universe.

To get basic information about the two datasets:

marvel.info()

dc.info()

Basic info about Marvel and DC datasets respectively

From the output, we can see that both the datasets consist of a significant number of null values in the ‘ID’, ‘ALIGN’, ‘EYE’, ‘HAIR’, ‘SEX’, ‘GSM’, etc. Furthermore, we see that there are 2 features of the float datatype, 1 with integer datatype and 10 of the features with object datatype.

Data Pre-processing

It is very crucial to perform the step of data pre-processing before we move ahead with the datasets. It is the process of transforming raw data into an understandable and clean format because real-world data is likely to be incomplete, inconsistent, and is likely to contain several errors.

Dropping unnecessary variables

An initial look at the head of the datasets tells us that feature ‘YEAR’ gives us the first appearing year of the mentioned character (essentially, the same information as ‘FIRST APPEARANCE’ ). So, it is advisable to drop the feature ‘YEAR’ from both datasets.

Moreover, using the function ‘.isnull().sum()’ to determine the number of missing values in each of the features, we see that ‘GSM’ has a significant proportion of missing entries in both ‘Marvel’ and ‘DC’

marvel.isnull().sum()

dc.isnull().sum()

Number of missing entries in each of the features of ‘Marvel’ and ‘DC’ respectively

So, we will be dropping the feature ‘GSM’ altogether from our analysis.

The features ‘page_id’ and ‘urlslug’ do not contribute to our analysis in any manner. We will be dropping the said features from our datasets as well.

Now, to drop multiple features from our datasets, we will be creating a list with all the unnecessary features and subsequently dropping them from data.

cols = ['page_id','urlslug', 'GSM', 'Year']

marvel = marvel.drop(cols,axis=1)

cols = ['page_id','urlslug', 'GSM', 'YEAR']

dc = dc.drop(cols,axis=1)

Printing the first five entries of both datasets:

Unifying the format of the features

We will initially be looking at how similar features in the datasets are recorded in different formats and will try and unify them.

The heads of the data indicate that ‘FIRST APPEARANCE’ is recorded in a different format in both datasets.

Due to the inconsistencies in the way it is recorded, we will be splitting the said feature into ‘MONTH’ and ‘YEAR’ in both the data. But before that, we will be dropping the entries which have a null value for ‘FIRST APPEARANCE’

marvel = marvel.dropna(subset=['FIRST APPEARANCE'])

dc = dc.dropna(subset=['FIRST APPEARANCE']) marvel[['MONTH','YEAR']] = marvel['FIRST APPEARANCE'].str.split('-', expand=True)

dc[['YEAR','MONTH']] = dc['FIRST APPEARANCE'].str.split(', ', expand=True)

However, we can still see that the months and years are entered in varied formats. In the Marvel dataset, months are of ‘Aug’, ‘Mar’, ‘Oct’… format, and the years are of ‘62’, ‘41’, ‘74’ format. We want them to be of ‘August’, ‘March’,’ October’….. and ‘1962’, ‘1941’, ‘1974’… formats.

A simple google search indicated that the Marvel comics did not exist before the 1921 s. Therefore, we will be adding the prefix ‘19’ to all the entries of the Marvel dataset greater than ‘21’ else we will be prefixing the entries with ‘20’. We will use ‘marvel.head()’ to help us see the changes we just made.

marvel['YEAR'] = marvel.YEAR.apply(lambda x: '19' + x if int(x) > 21 else '20' + x)

marvel.head()

Now, changing the format of the months of the Marvel dataset

marvel["MONTH"] = marvel["MONTH"].replace({"Jan":"January", "Feb":"February", "Mar": "March", "Apr": "April", "Jun": "June",

"Jul": "July", "Aug": "August", "Sep": "September", "Oct": "October", "Nov": "November", "Dec": "December"})

Now, we will be dropping the ‘FIRST APPREANCE’ feature as mentioned previously and will be displaying the entries of the two datasets.

marvel = marvel.drop('FIRST APPEARANCE',axis=1)

dc = dc.drop('FIRST APPEARANCE',axis=1)

Clean datasets after pre-processing

Exploratory Data Analysis

EDA or Exploratory Data Analysis helps us to arrive at basic inferences, patterns, anomalies in the dataset using various techniques mainly visualizations. The art of storytelling using data can be easily made effective using this.

Introduction of new characters over time

To begin with, we will be taking a look at how characters were introduced in both the comics over the due course of time. But before that, let's see when was the first character appearance in both Marvel and DC.

year_m_first = marvel.sort_values('YEAR').iloc[0]

print("The first character of MARVEL appeared in the year ", year_m_first['YEAR']) year_d_first = dc.sort_values('YEAR').iloc[0]

print("The first character of DC appeared in the year ", year_d_first['YEAR'])

The output showed us that the first character of MARVEL appeared in the year 1939 and the first character of DC appeared in the year 1935.

Now to understand how these characters were introduced along time, we will initially be looking at the distribution of ‘YEAR’ from the datasets.

fig,(ax1) = plt.subplots(figsize=(15,8))

sns.distplot(marvel['YEAR'], label='Marvel', color='red')

sns.distplot(dc['YEAR'], label='DC',color='blue')

plt.title('Distribution of Appearance of heroes in comic in years')

plt.show()

Here, the distribution of Marvel indicated by the red line shows that the character introduction of Marvel has been significantly different from that of DC. From the initial analysis, we could see that Marvel had almost twice the number of characters as DC.

To get a better understanding, we will be plotting a bar graph of ‘YEAR’ from both datasets.

plt.figure(figsize=(15,8))

m_bar = marvel.groupby('YEAR')

m_bar['YEAR'].value_counts(sort=False).plot(kind='bar',color='red',title='Marvel First Appearances by Year'); plt.figure(figsize=(15,8))

d_bar = dc.groupby('YEAR')

d_bar['YEAR'].value_counts(sort=False).plot(kind='bar',color='blue',title='DC First Appearances by Year');

The peak character introduction year was 1993 for Marvel and 2006 for DC. Except for the late 1950s to early 1960s, Marvel has been almost consistent in introducing characters over the years. Meanwhile, DC started upping its game in the 1980s. This dataset also indicated that DC stopped introducing new characters after 2011.

Furthermore, we will be looking at how these characters were introduced along time concerning gender by looking at the respective distribution.

Initially, we will be segregating the characters according to the gender they belong

marvel_female_characters = marvel[marvel.SEX == 'Female Characters']

dc_female_characters = dc[dc.SEX == 'Female Characters'] marvel_male_characters = marvel[marvel.SEX == 'Male Characters']

dc_male_characters = dc[dc.SEX == 'Male Characters'] marvel_gf_characters = marvel[marvel.SEX == 'Genderfluid Characters']

dc_gf_characters = dc[dc.SEX == 'Genderless Characters'] marvel_ag_characters = marvel[marvel.SEX == 'Agender Characters']

dc_tg_characters = dc[dc.SEX == 'Transgender Characters']

Now, we will be plotting each of its distributions.

f, axes = plt.subplots(2,2, figsize=(20, 20), sharex=True) sns.distplot(marvel_female_characters['YEAR'].dropna(),label='Marvel',color='red',ax=axes[0,0]).set_title('Ratio of Female characters created over the years')

sns.distplot(dc_female_characters['YEAR'].dropna(),label='DC',color='blue',ax=axes[0,0]) sns.distplot(marvel_male_characters['YEAR'].dropna(),label='Marvel',color='red',ax=axes[0,1]).set_title('Ratio of Male characters created over the years')

sns.distplot(dc_male_characters['YEAR'].dropna(),label='DC',color='blue',ax=axes[0,1]) sns.distplot(marvel_ag_characters['YEAR'].dropna(),label='Marvel',color='red',ax=axes[1,0]).set_title('Ratio of Agender/Genderless characters created over the years')

sns.distplot(dc_gf_characters['YEAR'].dropna(),label='DC',color='blue',ax=axes[1,0]) sns.distplot(marvel_gf_characters['YEAR'].dropna(),label='Marvel',color='red',ax=axes[1,1]).set_title('Ratio of Genderfluid/Transgender characters created over the years')

sns.distplot(dc_tg_characters['YEAR'].dropna(),label='DC',color='blue',ax=axes[1,1])

Distribution of characters of both datasets concerning their gender

We will now be looking at the year of introduction of gender minorities in these comics.

marvel_female_characters[‘YEAR’].min()

dc_female_characters['YEAR'].min() marvel_gf_characters[‘YEAR’].min()

dc_gf_characters[‘YEAR’].min() marvel_ag_characters[‘YEAR’].min()

dc_tg_characters[‘YEAR’].min()

The output of the above codes gave us ‘1936’, ‘1939’, ’1949', ‘1961’, ‘1964’, ‘2009’.

So we can conclude that both Marvel and DC introduced their first female characters during the first or second year of producing comics as DC Comics as a company started in 1934 and Marvel started in 1939.

Yet another noticeable fact is that Marvel introduced its first genderfluid character in 1949 which is almost a decade after its first character introduction. However, it took DC a massive 74 years to introduce its first transgender character in 2009!

The proportion of gender minorities

There is no denial in the fact that comics have been a male-dominated industry in all terms. From the people who write them, to the artists who draw them to the characters within them all are overwhelmingly men. This vast underrepresentation of other genders was taken into notice by both Marvel and DC and they have made efforts to curb this disparity. Now, we will be using various EDA tools to help us understand how diverse the offerings of both these comic giants are in terms of gender.

First of all, to know the proportion of each gender in both datasets, we will be plotting a pie chart using the Plotly library.

sex_m = marvel.SEX.value_counts()

sex_dc = dc.SEX.value_counts() m_sex = sex_m.sort_values(ascending=False)

data = [go.Pie(labels = m_sex.index,

values = m_sex.values,

title = 'Gender diversity in Marvel',

hoverinfo = 'label+value')]

plotly.offline.iplot(data) dc_sex = sex_dc.sort_values(ascending=False)

data = [go.Pie(labels = dc_sex.index,

values = dc_sex.values,

title = 'Gender diversity in DC',

hoverinfo = 'label+value')]

plotly.offline.iplot(data)

From the pie charts, we can say that DC is doing comparatively better than Marvel in terms of representation as 30% of its characters are female while at Marvel, it is only almost 25%. Concerning queer characters, the proportion is almost negligible in both cases.

We will now see which among the characters are queer specifically genderfluid and transgender.

So we can conclude that out of the massive 16376 characters of Marvel, only Loki and Xavin are genderfluid and out of the 6876 characters of DC, only Daystar is transgender.

Bivariate analysis for Gender and Alignment

Now, we will be performing some bivariate analysis that is, we will be analyzing two different variables to find out the patterns between them.

Alignment of a character usually refers to their moral and ethical outlook on everything in general. Here, we have 4 unique alignments — good, bad, neutral, and reformed criminals. A grouped bar graph of the ‘Sex’ with rest to their alignment is as follows:

f,(ax1,ax2) = plt.subplots(1, 2, figsize=(10, 6), sharex=True) for tick in ax1.get_xticklabels():

tick.set_rotation(45)

for tick in ax2.get_xticklabels():

tick.set_rotation(45) sns.countplot(x="ALIGN",hue="SEX",data=marvel,ax=ax1).set_title('Marvel Sex vs Align')

sns.countplot(x="ALIGN", hue="SEX",data=dc,ax=ax2).set_title('DC Sex vs Align')

Gender with rest to their respective Character Alignment

Here we can see that although each of the four alignments is dominated by male characters, Neutral Characters in Marvel and Bad Characters in DC tend to be overwhelmingly dominated by male characters.

Bivariate analysis for Gender and Identity

Now, we will be plotting a grouped bar graph of ‘Sex’ concerning their character identity. A character's identity is how the character is known to the general populace. It could be secret, public, unknown, etc.

f,(ax1,ax2) = plt.subplots(1, 2, figsize=(10, 6), sharex=True) for tick in ax1.get_xticklabels():

tick.set_rotation(45)

for tick in ax2.get_xticklabels():

tick.set_rotation(45) sns.countplot(x="ID",hue="SEX",palette="plasma",data=marvel,ax=ax1).set_title('Marvel Sex vs Identity')

sns.countplot(x="ID",hue="SEX",palette="plasma",data=dc,ax=ax2).set_title('DC Sex vs Identity')

Gender with rest to their respective Character Identity

Once again, all the identities are dominated by men. But noticeably in DC, there are only very few characters with their identity unknown.

Bivariate analysis for Gender and Dead/Alive status

Furthermore, a character could either be living or deceased. Now, we will be plotting a grouped bar chart to see how gender is related to the living or deceased status of the characters

Gender with rest to their respective living/deceased status

There is a significantly larger proportion of living characters in both comics.

Analysis concerning physical features like hair and eye color

Now we will be looking at the 5 most occurring eye and hair colors of female and male characters from both datasets using bar plots.

Starting with eye color:

f,ax = plt.subplots(2, 2, figsize=(15, 17), sharex=True) plt.subplot(2, 2, 1)

marvel_female_characters['EYE'].value_counts(normalize=True).head(5).plot(kind='bar',title="Marvel Female character Eyes",

color='red')

plt.subplot(2, 2, 2)

dc_female_characters['EYE'].value_counts(normalize=True).head(5).plot(kind='bar',title="DC Female Eyes",color='blue') plt.subplot(2, 2, 3)

marvel_male_characters['EYE'].value_counts(normalize=True).head(5).plot(kind='bar',title="Marvel Male Eyes",color='red') plt.subplot(2, 2, 4)

dc_male_characters['EYE'].value_counts(normalize=True).head(5).plot(kind='bar',title="DC Male Eyes",color='blue') fig.tight_layout()

plt.show()

So it is evident that the most occurring eye colors are blue and brown respectively. Female characters from both comics tend to have blue as the most occurring eye color while for marvel male characters, it is brown and for dc males, it is blue.

Looking at the top 5 most occurring hair colors similarly,

The most occurring hair color is black in all categories. We can also notice that there are bald as well as no hair characters present amongst Marvel males.

Top Characters

Now we will be looking at the topmost appearing characters in each of the universes separately and then together.

We define the top 10 most appearing characters initially into two different variables

top_10_appearances_m = marvel.sort_values('APPEARANCES', ascending=False).head(10)

top_10_appearances_dc = dc.sort_values('APPEARANCES', ascending=False).head(10)

Now we will be plotting a pie chart using the plotly library.

top_10_appearances_m = top_10_appearances_m.sort_values('APPEARANCES', ascending=False)

data = [go.Pie(labels = top_10_appearances_m['name'],

values = top_10_appearances_m['APPEARANCES'],

hoverinfo = 'label+value')]

plotly.offline.iplot(data) top_10_appearances_dc = top_10_appearances_dc.sort_values('Appearances', ascending=False)

data = [go.Pie(labels = top_10_appearances_dc['Name'],

values = top_10_appearances_dc['Appearances'],

hoverinfo = 'label+value')]

plotly.offline.iplot(data)

Top 10 most appearing characters in the Marvel comics

Top 10 most appearing characters in the DC comics

Amongst the top 10 most appearing characters of Marvel, all 10 of them are identified as male characters. While in DC, there are two female characters, Wonder Woman (Diana Prince) and Dinah Laurel Lance (New Earth) among the top 10 characters in terms of their appearance.

Now we will be concatenating the two and checking for the combined representation of gender.

dc_marvel = pd.concat([dc, marvel])

top_10_appearances_dc_marvel = dc_marvel.sort_values('APPEARANCES', ascending=False).head(10) top_10_appearances_dc_marvel = top_10_appearances_dc_marvel.sort_values('APPEARANCES', ascending=False)

data = [go.Pie(labels = top_10_appearances_dc_marvel['name'],

values = top_10_appearances_dc_marvel['APPEARANCES'],

hoverinfo = 'label+value')]

plotly.offline.iplot(data)

Top 10 most appearing characters irrespective of the universe

Not surprisingly, all of these characters are male with Spider-Man (Peter Parker) being the most appeared character.

Conclusion

To recapitulate, we can see that both the comics are far more or less dominated by males in regard to their characters. Although these comics are trying to curb this disparity of vastly underrepresented women and queer characters, they still have a long way to go.

The entire code for the above, as well as the dataset, can be accessed at my Github repository.