Star Wars Data Science. Network Analysis, Topic Modeling, and a…
By: Dennis Bakhuis
Tags: Star Wars, Data Science, Programming, Machine Learning, Python

Star Wars Data Science

Star Wars 🌌 is the most epic fantasy space adventure of all times (strongly biased). Why only save one world when you can rescue whole galaxies! Each year, millions of fans celebrate the Star Wars day on May the fourth. Last year I had some fun and created a blog post that could decipher a secret message from Mustafar using a neural network (build from scratch).

Figure 1: The wordcloud and network graph created in this article.

To start a tradition, this year, I combined Star Wars with Data Science yet again. An infamous source of Star Wars information is collected on the Wookieepedia, a Fandom site with thousands of pages. An amazing source to investigate using Data Science tools such as topic modeling and network analysis.

As it is a rather long article, I have divided it in a number of topics. In this way you can easily skim down to the topics you are interested in the most.

An overview of topics:

Scraping and building dataset Wookieepedia Data exploration We need a wordcloud! Topic modeling Wookieepedia network analysis

The github repository with all notebooks and the dataset can be found here.

Other useful links are my 10-minute-a-day Python course and a tutorial to work with Python environments.

1. Scraping and building the dataset

There is an incredible amount of information on Star Wars online. One of my favorite sources is the so called Wookieepedia, a wiki with a crazy amount of Star Wars knowledge build by fans.

All data is available in the Github repository and Kaggle so there is no need to scrape it yourself and generate high amount of traffic for Wookieepedia.

This section describes the process to scrape all data used. A wiki is a collection of pages and each topic has its own page. To scrape this information, we need to visit each page and download its contents. There is a clever way to scrape such websites and that is by using Sitemaps. It is a special file that webmasters can provide that will help web crawlers indexing the website. We can make use of the sitemap to get a list of all the pages that are available.

Currently (April 2021) there are 219.900 pages that could be scraped. This is however a bit too much. Therefore, I decided to only scrape the pages that are considered canon. Luckily, Wookieepedia gave canon articles their own category. When we click on the category, we get a paginated index of all pages that are considered canon. This needs some additional work to scrape all the topics.

Alright, we now have a list of 29k pages that are considered canon which still need to be scraped. These pages have a typical format consisting of a title, a description often with subsections, and a sidebar with properties. To reduce the amount of information, I will only scrape the the first paragraph, the full sidebar, and all links that point towards other canon pages. A typical Wookieepedia page is shown in figure 2:

Figure 2: From each page the description, sidebar information, and all links are scraped (page).

Next, we scrape each page and save the partitioned to disk:

It took a little bit more than an hour to scrape all information and all is stored in pickle files that are split in sections of max 5000 pages. This is the raw dataset and we can always fall back to this.

Next we will split the raw data in two sections: characters and raw text sentences. The characters are identified by a property called ‘species’ that is available in the sidebar. We will collect all characters in a strongly structured Pandas DataFrame and this means that we need to select properties beforehand. We have a total of 5334 characters that are marked as canon. More details on the selected properties are in the scrape notebook that can be found in the Github repository.

The raw text sentences are extracted from the descriptions. Each description is split into sentences and collected in a single list. Details of this split is can also be found in scrape notebook on Github.

2. Wookieepedia Data exploration

The first step with a new dataset is always data exploration. It is a way to get acquainted with the dataset and a first step to understand what information is in it. Details of the analysis can be found in the data exploration notebook on Github.

Species

If you ask a non-fanboy about Star Wars something you hear often is that it is full of weird creatures. Especially the famous `Cantina scene` is full with different species that exist in the galaxy. Therefore, it is interesting to have a first look on the amount of species and how often they are represented in the canon dataset.

There are a total of 530 species mentioned, too much to make a nice visualization, Therefore, we only select species that have at least 40 mentions in the dataset. All others are grouped as other.

Figure 3: Humans are by far the dominant species.

Humans are by far the dominant species with more than half of the characters or 2770 mentions. The next species are Twi’leks that are only 2.5% of the dataset. There are only a few tenths for the other characters as the number is dropping quickly. We grouped 519 other species in the others group which gives an average of less than four characters each. If we would meet a random Star Wars character, there is a 50% chance that he/she is human. If it is not a human, it an be one of the many diverse species available.

Home world

As the species are very diverse, it might also be interesting to have a look at the home world of each character. Looking at the unique counts, there are 463 worlds mentioned in the dataset. Again, due to the high diversity, we threshold these to a minimum of 8 charters that have to mention the planet as a home world.

Figure 4: Kamino is the most mentioned planed.

Interestingly, Kamino is the most mentioned world which is famous for their cloning technology. It was first mentioned in episode 2 — attack of the clones where the protagonists found a hidden clone army. But the main reason why this planet is so prominent is because of the animated series of Star Wars called the clone wars. It had seven seasons and a total of 133 episodes and was centralized around many characters that were created on Kamino. The second planet is Naboo which was under rule of Queen Amidala and famous throughout the series. What is maybe a surprise is the high mention of Mandalore, but that is probably due to The Mandalorial, which is a great spin-off series on the streaming service Disney+.

Gender

While there are many species, most of them seem to be divided in the same genders we know on earth. Lets have a quick peek on how the distribution is:

Figure 5: Since 2016 Star Wars has non-binary or genderqueer characters (inset: wookieepedia) .

Because Star Wars started in the late seventies, I expected that there would be a strong representation of male characters. Indeed, two-third of all characters are males but I somehow expected more. Maybe Disney started to increase the amount of females to finally bring balance to the force.

Another thing that is pretty cool is that there are a few non-binary characters in the official Star Wars lore. The author Chuck Wendig confirmed that a humanoid character from his books called Eleodie Maracavanya was indeed non-binary and thereby the first official non-binary canon character. Now there are a total of four, including Keo Venzee.

For some species, it is not very clear if they are male or female and about 8% of all characters do not have a gender.

The tallest and smallest bunch

As Qui-Gon Jinn once remarked: “there is always a bigger fish’ and therefore, it is interesting to see what sizes are recorded in the dataset. Unfortunately, only for about 12% of the characters, the height is known. Lets have a look at the distribution:

Figure 6: The Star Wars characters spike at 1.83m which is close to the Dutch average. (inset: Wookieepedia)

Most characters have heights that are pretty typical to humanoids with a large spike at 1.83m which is about the average height of Dutch males. Looking at the extremes we find that Babu Frik (appeared in episode 9) is by far the smallest intelligent creature with only 22cm. Also nice to see is that Grogu has to grow about 20cm more to be equal in size to Yoda.

The largest creature is Omi, the one-eyed beast living in the trash compactor of the Death Star and apparently is 10 meters tall. Also Jabba the Hutt is pretty large with 3.90m but I guess they measure him from nose till the end of his tail.

Eye color

We have also logged the color of they eyes and there are 97 colors in the dataset. I guess that there are some typos and also some descriptive colors like ‘Bluish green’. Still for almost half the eye color was registered.

Figure 7: the third most common eye color is black.

Brown eyes is the most common in the Star Wars universe, very similar to the real world where the vast majority of the world’s population also has brown eyes. A bit more disturbing is that black is apparently the third most common eye color.

Other properties such as skin color and hair color are also available. Feel free to check them out in the notebook.

3. We need a wordcloud!

When doing data analysis on a fandom topic, we have to make a wordcloud. For this we are going to use the description dataset and the Wordcloud Python Package from Andreas Mueller. Even cooler, the documentation even has an example that uses a Stormtrooper!

The Wordcloud package could not have made it easier. All frequency calculations are done under the hood and a beautiful image is generated:

Figure 8: A wordcloud of all the descriptions of the canon articles (all data used).

The wordcloud is generated using the text of all canon articles, so not only the characters. As with all wordclouds, first all stopwords are removed and after a frequency analysis we see the most common words that definitely make sense in a Star Wars way.

4. Topic modeling

Until now we have covered the basics with some data exploration and we have created the mandatory wordcloud. Therefore, we are now ready to go into the more advanced analysis and the first I want to give a try in topic modeling. The method described here is based on the work by Maarten Grootendorst, the creator of BerTopic.

Topic modeling is an unsupervised learning technique that can answer the following question: I have this bunch texts, what are the most common topics these text talk about.

For topic modeling we will make use of a Python package called Sentence-Transformers which is as the name suggests based on the transformers architecture and is able to convert a full sentence into a vector. To find similar topics, we need to find vectors that are grouped together. Lets first create the sentence embeddings:

The sentence vectors have a length of 768 which is very large to to the cluster analysis, therefore, we will apply a dimension reduction. There are many choices, such as LDA or NMF, but here we will use a method called Umap which has the benefit of keeping the local structure in tact.

After the dimension reduction we can try to identify the clusters using Hdbscan. This will find the clusters that are grouped together in the reduced parameter space.

We have reduced the 768 features to only 6 features, however this reduction is not enough if we want to plot this to the screen. For this we need to reduce it at least to 3 dimensions and we will go even down to 2 for better visibility. For this we are again using Umap and reduce the original dataset from 768 all the way to 2 and combine the earlier defined label with the coordinates.

And now we can plot the result:

Figure 9: Colorful overview of all the topics.

The result is very pretty but to get some idea if these clusters make any sense we need to combine the clusters with their original sentences. Also, we want to find the most common keywords for each cluster and therefore, we need to analyze each cluster separately. To solve this, Maarten Grootendorst came up with a class based TF_IDF which works pretty neat!

We need to apply c_tf_idf for each cluster and treat the collection of documents as a single document. The result is a list of most frequent terms and hopefully they make some sense on our Wookieepedia dataset.

When looking at the results there are actually some that make sense but are not that cool. There is a full cluster that has the many colors, which is still pretty cool, knowing that it got it from an unsupervised way. There are however a couple of clusters that are indeed have story-based topics. For example, a cluster containing Emperor Palpatine contained lots of political terms such as senate, supreme chancellor, and constitution. Palatine also had links to clones which is the master plan he created to destroy the Jedi.

However my most favorite cluster combines almost all star characters together with the terms missions and battle of Yavin. Of course, the battle of Yavin is one of the most important events in Star Wars and it makes sense that it is often referenced. Still, finding this using topic modeling is pretty awesome.

Figure 10: My favorite cluster with terms that are most frequent in the cluster.

5. Wookieepedia network analysis

We collected a total of 5334 characters and wouldn’t it be great to analyze the relations between each character. As a first attempt to work with graph networks, I want to visualize the network around characters. For this we need to have nodes, which are the characters, and their relations, which are called edges in a graph. For example, Anakin Skywalker would have a relation called ‘father of’ to Luke Skywalker. To extract the various relations would mean extensive natural language processing to reduce the corpus to forms of node-edge-node, which is far from trivial.

To make it a bit easier, we will sum all relations to a single kind of relation which we call ‘connected to’. To find out if a character is connected to another, we will look if there is a link on the page. We expect that on the page of Anakin Skywalker there will be a link to the page of Luke Skywalker. All these links are collected during the scraping process as a list which we call crosslinks.

Find the full notebook in the Github repository. Lets start with preparing the raw data:

Now we can build a Graph network and we will be using the excellent networkx package. Networks are geometrical structures to organize data into nodes and edges. In our case, nodes are the characters and the relations between the nodes are edges. Normally you also provide a weight to an edge. This can be seen as how strong a bond is, but also as a distance. These weights enable many methods for analysis such as the shortest path. As we cannot add a measure to our relation (at least not that I know of from this data), all the weights will be set to unity.

When building the network, we will scan the crosslinks of each character. These can point to other characters, but also to any other page. Therefore, we need to check if the crosslink is a character and ignore non-characters:

After the creation we end up with 4794 nodes. Apparently, about 500 characters do not have a single relation to another character. From the characters that do have relations (edges), this is a total of almost 20k or 4 relations on average.

Lets first have a look at the degree distribution, which show the amount of connections to other nodes. A node of degree 1 has only one connection, while a node with degree 10 has 10 connections to other nodes. The code is available in the notebook.

Figure 11: The degree distribution of the Wookieepedia dataset.

Most of the nodes only have a few connections and with a dataset that consists of more 5334 characters it is also no surprise. Many characters only have a single appearance. Characters like Anakin Skywalker have connections to more than 100 other characters and this is probably also the case of other stars from Star Wars.

To reduce the network a bit in size and to remove the less important characters, we will filter out all nodes that have degree 2 or less. We have to do this iteratively because when we remove nodes, there will also be edges removed which might set a node to a lower degree.

This brings the characters down to just below 3000. The amount of edges almost did not reduce and we still have 17k of those. This makes of course sense as we only removed low degree nodes.

The power of graphs is that you can use a bag of tools to investigate a nodes importance. One of these tools is called Betweenness centrality. It is a method to find nodes that serve as a bridge between other nodes, i.e. as a sort of relay station for information. In our example, we use the Betweenness centrality to find which nodes are the link to various other persons and hopefully find some key characters. The result if plotted in figure12.

Figure 12: Most important nodes in the graph according to betweenness centrality.

No surprises here. I had to look up Ezra Bridger but he was very prominent in the Clone War series. There are many other algorithms such as closeness centrality or page rank but these resulted in similar results.

To have a bit of more play play with the graph network, we will visualize it using PyVis. PyVis is build around VisJS and makes it extremely easy to plot these graphs and play around with the result. There is however one thing we need to take care off: we have much to much nodes and edges. While it seems that PyVis has no problem with large amounts, due to overlap of all the edges, it does not so interesting. Therefore, we need to make an algorithm that selects a subgroup from our network.

The next piece of code needs three parameters: a starting node, a maximum level penetration, and maximum crosslinks. The algorithm will, as the name suggest, start from the starting node and selects the maximum amount of neighboring nodes. If there are more nodes available than the maximum set crosslinks, it will select the nodes with the larges degree. Next, the algorithm will jump to the selected nodes (this is a level increase) and repeat the process until the maximum penetration is achieved. The amount of nodes selected will increase exponentially with maximum level penetration so be careful with that parameter.

The result is an interactive website that is shown directly in the Jupyter notebook or any other web browser.

Figure 13: A screenshot of the interactive graph network.

Each node can be dragged with the mouse and will pull its connections along with it. When hovering over a node, the first paragraph of the Wookieepedia page is shown and makes it easy to read who that character is. The color of the node indicate the level. The color of the edge is the color from its source. Feel free to investigate other characters as well!

Round up

I had a lot of fun playing around with this dataset. There is still a lot possible as I barely touched the surface. But even with some straight forward methods we found quite some cool things.

‘this is the way’ — Din Djarin

Please let me know if you have any comments! Feel free to connect on LinkedIn.