Construct the Matrix interaction network based on the movie script | by Tomaz Bratanic | Towards Data Science

Christmas is just around the corner, and with it, comes the newest Matrix movie. I can’t think of a better way to wait till the movie is released than to perform a network analysis of the first Matrix movie.
This blog post will present how to combine web scraping, OCR, and NLP techniques to construct the Matrix interaction network.
Scraping Matrix fandom page with Selenium
Using PyTesseract to read the Matrix movie script PDF
Extract characters in each scene by using the SpaCy’s rule-based matcher
Construct and analyze the character’s co-occurrence network in Neo4j
I have already performed a similar analysis based on the Harry Potter book, and this time we will be using the Matrix movie script.
As always, I have prepared a Google Collab notebook that you can use to follow the examples in this post.
We will begin by scraping the Matrix Fandom page to get the list of characters that appeared in the movie. As mentioned, we will be using the Selenium library to achieve this. The content of the fandom page is available under the CC BY 4.0 license.
In the first step, we will extract the names and the links of characters that appeared in the first Matrix movie.
Just because we can, we will also extract detailed information from personal pages of characters.
Before continuing, we will store the character information into Neo4j. If you are using the Colab notebook, then it would be easiest to create either a free Neo4j Sandbox or free Aura database instance to store the results.
Once you have created the Sandbox or the Aura environment, simply copy the connection details into the notebook.
Now that you have defined the connection to your Neo4j instance, you can go ahead and import the characters’ information.
To get a feeling for the information we just stored in the Neo4j database, I’ve prepared the following visualization made in Neo4j Bloom.
At this moment, there are no connections in the database, just lonely and isolated nodes. If you wanted to, you could refactor some of the node properties such as Spouse to a relationship. However, we will skip this part and move on to constructing a co-occurrence network based on the movie script.
The movie script is available on the Daily Script web page in PDF format. While no explicit license is stated, the web page says that the scripts are available for educational purposes, so we are good to go.
We will use PyTesseract library to transform the PDF into a text format.
This process takes around 15 minutes, so you can use this time to take a break and perhaps stretch your legs.
Before moving on to the character extraction step, we will perform a simple text cleanup and split the script by scenes.
Now that we have preprocessed the text, we will go ahead and identify all the characters that appear in a particular scene. Since we already know which characters to expect from our fandom scraping process, we’ll use SpaCy’s rule-based matcher to identify characters.
A character name can appear in two forms. First, if the character is talking, its name is uppercased in the text. The second form is the title-cased version, where a character is mentioned by other persons or mentioned in the scene description. SpaCy makes it really easy to describe these two patterns.
We will also omit the The word from the pattern definition. For example, the fandom page contains the character The Oracle. Therefore, we will skip the word The and only search for Oracle pattern.
The following code will construct the SpaCy’s matcher object used in the next step to identify characters.
We have the text and the entity matcher ready. We will iterate over scenes, identify all the characters that appear in it, and store the results directly into Neo4j in one step.
That’s how easy it is to extract information from the movie script and store the output into Neo4j.
Now we will move on to the network analysis part of this blog. First, we will evaluate all the characters that didn’t appear in any scene.
MATCH (n:Character) WHERE NOT (n)--()RETURN n.name AS character
Results
Twelve characters weren’t identified in any scene. I don’t remember any of the characters except the Woman in Red. Probably she is mentioned as the red woman or something similar in the text, so our pattern matcher didn’t identify her. Of course, we could fine-tune the patterns to include these types of exceptions, but overall, it seems that the main characters were identified.
Next, we will examine the characters that appeared in most scenes.
MATCH (s:Scene)WITH count(s) as countOfScenesMATCH (n:Character)WITH  n.name AS character,      size((n)-[:IN_SCENE]->()) as scenes,      countOfScenesRETURN character,       scenes,       round(toFloat(scenes) / countOfScenes * 100, 2) AS scenePercentageORDER BY scenes DESCLIMIT 5
Results
Nothing shocking here. Neo appeared in more than half of the scenes, followed by Morpheus and Trinity. Only Emil Eifrem is surprisingly missing from this list.
We can define the co-occurrence event as a pair of characters that appeared in the same scene. In this context, co-occurrence can also be understood as an interaction. The more scenes a duo of characters appeared in, the more they interacted in the movie.
We can evaluate which characters interacted the most by executing the following Cypher statement.
MATCH (n1:Character)-[:IN_SCENE]->()<-[:IN_SCENE]-(n2:Character)WHERE id(n1) < id(n2)RETURN n1.name AS character1,       n2.name AS character2,       count(*) AS countORDER BY count DESCLIMIT 5
Results
Most interactions occurred between Neo, Morpheus, Trinity, and Tank. If you watched the movie, this all makes sense.
Lastly, we can infer the co-occurrence network between characters and perform a network analysis of it. We will simply count the number of interactions between a pair of characters and store the information as a relationship.
MATCH (n1:Character)-[:IN_SCENE]->()<-[:IN_SCENE]-(n2:Character)WHERE id(n1) < id(n2)WITH n1, n2, count(*) AS countMERGE p=(n1)-[r:INTERACTS]-(n2)SET r.weight = countRETURN p
Results
Neo4j features a Graph Data Science library with more than 50 graph algorithms ranging from centrality, community detection, and node embedding categories.
We will use PageRank to evaluate node importance and Louvain to determine the community structure of the inferred co-occurrence network. Instead of inspecting each algorithm result separately, we will store the results and construct a network visualization that visualizes both node importance and community structure.
First, we have to project an in-memory graph in order to be able to execute graph algorithms on it. Notice that we project the co-occurrence relationship as undirected. For example, if Neo interacted with Trinity, this directly implies that Trinity also interacted with Neo.
CALL gds.graph.create("matrix", "Character",   {INTERACTS: {orientation:"UNDIRECTED", properties:"weight"}})
Now we can go ahead and execute the weighted PageRank algorithm and store the results back to Neo4j.
CALL gds.pageRank.write("matrix", {relationshipWeightProperty:"weight", writeProperty:"pagerank"})
Lastly, we execute the weighted Louvain algorithm to deduce the community structure and store the results in the database.
CALL gds.louvain.write("matrix", {relationshipWeightProperty:"weight", writeProperty:"louvain"})
You can use Neo4j Bloom to produce a network visualization that colors the nodes based on their community id and calculates the node size based on its PageRank score.
Check the official documentation to learn how to develop a rule-based visualization in Neo4j Bloom.
Results are better than I expected. The yellow community is the Nebuchadnezzar gang, with Neo, Trinity, and Morpheus being the most important members. The purple community consists of the Matrix agents, which the Nebuchadnezzar gang is fighting against. We would have to introduce some sort of timeline to observe how Cypher went from being one of the good guys to switching sides and joining the dark side. Lastly, the orange community consists of the Oracle, Priestess, and the Spoon Boy, who helped Neo transcend reality.
I hope that we can all agree that using simple assumptions as the co-occurrence interaction can produce incredible results that can help explain the underlying 150 pages of PDF in single network visualization. So open Neo4j Sandbox today and try it out for yourself. Let me know if you find any tweaks in the data pipeline that produces better or more accurate results.
Happy holidays!
As always, the code is available on GitHub.