Exploring the Pantheon Project: Historical Popularity Index
By: Zuha Ansari
Tags: Data Science, Data Analysis, History

This blogpost aims to explore the data collected by Pantheon, a project developed by the Macro Connections group at the MIT Media Lab. An overview of the dataset prompted several questions that we will explore over the course of this article:

What careers make a figure more popular? How have particular events influenced the popularity of these careers? What is the relationship between popularity and gender? What role does nationality and ethnicity play in determining popularity? How do wikipedia statistics affect a person’s popularity?

Part 1: Data Cleaning — Exploring the Data

The data was downloaded from the Pantheon’s repository on Kaggle

The original dataset contains 11341 rows and 17 columns

Key Features:

sex — Male or Female — string

Male or Female — string birth_year — Year figure was born in — string

Year figure was born in — string continent - Continent of native country — string

Continent of native country — string domain - Career domain of figure — string

Career domain of figure — string historical_popularity_index - Measure of popularity of the figure — float

Part 2: Data Cleaning — Feature Cleaning

Checked the number of missing values for every column of the dataset. Columns with missing values included: city, country, continent, longitude, latitude, birth_year

If the proportion of missing values was <10%, rows were dropped

Birth_year converted to integer from string

Regular expressions were used on all string attributes to get rid of unwanted characters (e.g: !?/)

Columns not needed for analysis of key questions were dropped. These include State, City, and Occupation

Post Cleaning

Column names were also improved to be more concise and uniform

The only remaining columns with missing values are longitude and latitude, which will not affect any analysis. They are only used to show for generalized visualisations.

The cleaned dataset contains 10873 rows and 14 columns

Part 3: Data Analysis & Statistical Inference

Data analysis was done individually on key questions using a common cleaned dataset.

1. Which careers are most popular?

The dataset contains a column ‘domain’ with 8 distinct values: Institutions, Arts, Sports, Science & Technology, Humanities, Public Figure, Business & Law, Exploration

We started by visualizing the popularity of each domain. The plots below imply that Institutions, Arts and Humanities are the 3 most popular domains. Considering both was important because a domain that features a lot in the dataset may have a relatively lower HPI score consistently. This was exactly the case with Sports which has greater prevalence than Humanities in Figure 1, but has a considerably lower HPI scores as shown by Figure 2.

Figure 1: Bar Plot that shows us the number times each domain occurs in the dataset

Figure 2: Box Plot that shows us the variation in HPI (popularity) scores.

We then moved to visualizing the popularity of each domain across continents, and gender. We observed that 4 continents (Asia, Africa, Europe, South America) have their peak at Institutions and the remaining 2 (North America and Oceania) have peaks at Arts. This further indicated greater popularity of the domains Arts and Institutions.

Figure 3: Grouped Line Plot which allows us to observe relative counts of each domain in each continent.

Statistical Inference:

To test our claim of popularity, we used hypothesis testing for Arts and Institution separately. The following were the Null and Alternative hypotheses:

Null: Arts/Institution domain is significantly more popular than other domains

Alternative: There is no significant difference between the popularity of the Arts/Institution domain and the other domains, and any considerable difference is due to chance.

The test-statistic that we used for this hypothesis was the difference between the proportion of Arts/Institution domain and the domain with the highest proportion(excluding Arts/Institution).

Institutions:

Observed Test Statistic came out to be 0.1164320

Next, we simulated the test statistic 10000 times by generating a random sample of size 10000 in each simulation.

The distribution of our statistic is plotted in the axes below. The dotted red line signifies our observed test statistic.

The P-Value was calculated to be 0.520, which is fairly high, and conclusive enough to support the Null hypothesis. Hence we conclude that the apparent popularity of the Institution domain is not by chance.

Arts:

This exact process was then repeated with the Arts domain. The results were the following:

Observed statistic came out to be 0.10235272323082625 and the distribution of simulated statistics was as follows:

The P-Value was calculated to be 0.502 which again is high enough to accept the Null hypothesis and hence conclude that the Arts domain is more popular than the rest.

The results of both these hypotheses seem sensible as most of the popular personalities throughout history have been political figures, actors and musicians.

Figure 5: World Cloud of industries for the entire dataset

Figure 6: World Cloud of industries of the Top 1000 historical figures

-

Our dataset also has a column named industry, with 27 unique values — each corresponding to a specific domain. We then visualized the popularity of these industries. We see that Government, Music, Language, Film and Team Sports are dominant when we take the whole dataset. But when we consider the top1000, we see that Philosophy and Religion overpower Film and Team Sports.

-

-

Finally, we compared industry popularity between the 19th and 20th centuries. The plot below shows that a much larger percentage of historical figures belong to Music, Film & Theatre and Sports in the 20th century than in the 19th century.

Figure 7

2. How have particular events influenced popularity?

To analyze how particular events in history can lead to a difference in popularity among various domains, we looked for various possible events in history that might have caused any changes in the popularity of personalities in several domains.

One major event was the Birth of Jesus Christ(1 A.D), which is quite significant as it changed the calendar system of the world. To see if this event changed the popularity of several domains, we divided our dataset into 2 smaller datasets, one containing personalities born before 1 A.D and one containing personalities born after 1 A.D. Then we compared the percentage of personalities in various domains in both the dataframes. The results are summarized in the figure below:

Figure 8

The percentage of people in Arts and Sports increased considerably, and the percentage of people across the Institutions domain decreased by more than 20%. By this we can partially conclude that Arts and Sports are relatively new phenomena.

This analysis has it’s own limitations as there have been several other events in history which might have popularised several industries and domains and those are not taken into account. Additionally, our dataframe contained most of the people born after 1 A.D hence less people before 1 A.D, and a small sample may have resulted in incorrect inferences.

Another event that we believed could have caused a change in popularity among several career domains was the Renaissance. Sources suggest that this was not a single event in history, rather a series of events that happened between the 15th and 16th centuries. We have arbitrarily taken 1500 A.D to notice the changes that Renaissance might have caused in terms of popularity of domains. We divided our dataframe into two dataframes. One containing popular personalities between 1300 A.D and 1500 A.D and the other containing personalities between 1500 A.D and 1700 A.D. The 200 year bounds were incorporated to limit the effects of other important events in history.

We predicted that there will be an increase in popularity in Science & Technology, Arts and Humanities as the renaissance period led to an increase of awareness among people and led to major developments in the field of science and technology. Other than this, people started appreciating art more and the event was a rebirth of learning and wisdom.

The figure below summarizes our findings for this event:

Figure 9

Our predictions came out to be true as all three of our suspected categories saw an increase in popularity relative to other categories. Although with limitations, we can still support the common claim(through data analysis) that renaissance was a major event in history which led to many important developments in Sciences, arts and humanities(hence led to an increase in popularity of personalities in these domains.

Another series of events we considered were the major breakthroughs and innovations in technologies like Television, Radio, Internet, Cameras etc. (20th century)

The sports industry experienced major growth in percentage over the past century or so(at time time these technologies evolved). This was because of the increased exposure of sports due to games getting telecasting and recorded, it became a global phenomenon and sportsmen started becoming more popular and influential. Additionally, some industries in the ‘Arts’ domain like the Film-making industry gained popularity with the advancement in these technologies.

The following figure shows our findings, which prove the above written claims, using data:

Figure 10

Permutation Test

The sports and the arts domain experienced major breakthroughs and growth due to the advancement of technology. Is this claim true?

To test this inference, we conducted a hypothesis test. The following are the Null and Alternative hypothesis:

Null: There is no considerable difference in the popularity of sports and arts domains before and after the advent of technology.

Alternative: There is considerable difference in the popularity of sports and arts domains before and after the advent of technology.

We computed the proportion(or the percentage) of personalities belonging to both of these domains(before and after the 19th century) and used the difference between these proportions as our test statistic.

To simulate under the NULL hypothesis, we combined our dataframes and shuffled it, and sampled data based on it. Then computed the statistic after the re-arrangement of the data and simulated the process 5000 times.

Result:

The histogram of the distribution of these 5000 computed values is shown below, and the vertical green line shows the value of our observed statistic.

As the observed statistic is well out of the range of distribution of the histogram, we can safely conclude that the data favours the alternative hypothesis over the null hypothesis, and that advancement in technology did bring about a significant difference(increase) in the popularity of the sports and arts domains.

3. What is the relationship between popularity and gender?

To get a sense of how the female figures in our dataset compare with the male figures we first observed KDE plots of their distributions in the dataset. The plot below shows that average values of HPI are similar. This might suggest a certain equality but we needed to go deeper.

Figure 11

Figure 12

The bar plot on the left shows that there are very few women in the dataset, which is not a surprise considering society and the general role of males and females. Men have generally been much more popular than women.

-

Statistical Inference: A/B Testing:

In our EDA we observed a clear difference between the popularity of males and females, with almost 90 % figures being male, which is not a surprise given the norms of our society.

We also wanted to see if the start of feminist movements in early 20th century had a positive effect on popularity of females. We conducted an A/B hypothesis test for this purpose.

Null: Any difference in proportions of females in the 20th century from the 19th is due to chance i.e both groups have the same underlying distribution.

Alternative: Female popularity (proportion of females) significantly increased in the 20th century.

Test statistic — proportion of females in 20th century — proportion of females in 19th century

Results:

Observed Test Statistic came out to be 0.1214593889879

Next, we simulated the test statistic under our Null Model 10000 times each time with a randomly permuted sex column. If Null Model is correct then random permutations of the sex should give simulated statistic values similar to the observed value. This, however, is not supported by the empirical distribution of simulated statistics and P-value.

The distribution of our statistic is plotted in the axes below. The dotted red line signifies our observed test statistic:

The P-Value was calculated to be 0, and hence we conclude that the 20th century saw a significant increase in female popularity, indicating the positive effect of various feminist movements that started in the 20th century.

What more can we find out about female historical figures?

To get a sense of what domains our figures belonged to, categorized by gender we drew up another visualization. Figure 4 implies that most historical female figures belong to the Arts, while most historical male figures belong to Institutions.

Figure 4: Grouped Line Plot which allows us to observe relative counts of each domain by sex

Finally we wanted to get an idea of where the women in our dataset are from. For this, we used a heat map to observe the distribution of women historical figures across continents. We observed that a very high percentage of women in our dataset belonged to either North America or Europe. There is barely any representation of women in Africa, Asia and South America.

Figure 13

4. What role does ethnicity play in determining popularity?

Continuing on from the last question, we can also observe that the distribution of women across continents indicated that some ethnicities achieved more popularity than others. Namely, a European or American background as opposed to Asian, Black, Latino, etc. We decided to look into this further by visualizing a heatmap of all historical figures in the dataset to show their distribution across continents.

Figure 14: Heatmap showing distribution across Continents

The above heatmap reaffirms our previous observation that most historical figures belonged to ‘White’ ethnicities. The following bar plot presents a more numerical representation of this.

Figure 15

These observations led us to question why certain ethnicities might have a higher HPI than others. The dataset estimates a figures ‘popularity’ based on certain wikipedia statistics. By intuition, we can argue that certain first world countries (or better developed continents) have maintained control over historical narratives and thus are more likely to be known.

Statistical Inference:

We decided to conduct a hypothesis test to see if our claim about ‘white’ ethnic figures being in majority.

5. How do Wikipedia statistics affect a person’s popularity?

Following up on that last note, we decided to analyze how exactly different Wikipedia statistics affect an individuals calculated Historical Popularity Index. To do this, we first visualized a heatmap of a variable correlation matrix. The map shows that the number of translations into different languages affects the popularity of a figure the most.

Figure 16

The following regression plot reaffirms this observation.

Concluding Remark

The Pantheon data set was an interesting data set and provided answers to several key questions that we initially set out to answer. We were able to extract some useful information and trends among popular personalities and their domains, ethnicity, gender etc. However, there are endless possibilities when it comes to exploring and drawing inferences from data. A lot of our analysis was based on certain assumptions hence there are limitations of the analyses presented above.

Despite these limitations, performing all the steps of the data-science lifecycle on this particular data set was quite insightful and a great learning experience.