Network Analysis of the Romance of Three Kingdoms | by Dimitris Manolidis | Towards Data Science

Recently, I posted an article about the social networks of the 14th-century novel Romance of Three Kingdoms. In the article, I discuss the social networks which emerge when analyzing the interactions of the characters of the novel. Furthermore, with the use of centrality measures, I determine who the most influential characters are. The article is non-technical and some of the details of my analysis are left out.
Here, I present the code behind the analysis and discuss further the technical details of the social networks of the novel. Moreover, this code can be used (with minor modifications) to apply a similar analysis to other novels or pieces of text.
My analysis was inspired by the network analysis of Game of Thrones, by Andrew Beveridge and Jie Shan. Both works of literature have an intricate plot and a large number of characters. This makes Romance of Three Kingdoms a good candidate for such analysis.
Before starting my analysis of Romance of Three Kingdoms, there were some initial decisions I had to make.
Characters: The first decision was which characters to include in my analysis. There are over a thousand characters appearing in the novel. For the reader to be able to follow the characters more closely, only 70 characters were selected. To select the characters, I used the following procedure. First, I selected 100 characters whose names appear the most throughout the novel. Then, I made a list of characters that appear the most in websites and articles about the novel, the characters that appeared in the series and even some characters I personally considered important. I compared the lists and manually selected which characters I was going to keep for my final list.
Sections: Another decision I made, was to analyze the novel in sections. The volume of the novel is great. It includes 120 chapters and the English translated version contains more than 500,000 words. I decided that, to better follow the interactions of the characters, I should divide the novel into groups of chapters and analyze each group separately, before analyzing the full text. Since chapters are similar in length, I decided to create groups of 30 chapters, making a total of 4 groups. In the end, I also processed the whole novel, to get the grand network of all characters.
With the use of bash scripts, I concatenated the chapters in groups of 30 in separate files, plus one file with the full text. Next, I created the python script which would take a text file as input and produce a graph object as output. Then, I could further process the graph with a tool like Gephi to create beautiful network visualizations.
We start by importing our file and removing any special characters, as the initial text had a lot of them
import rewith open(infile, 'r') as file:    data = file.read().replace('\n', ' ')text = re.sub('[^A-Za-z0–9]+', ' ', data)
Next, we need to create a list of tokens from our text. Python’s nltk library can do that for us easily. Also, we remove stop words from our tokens
import nltkstopWords = nltk.corpus.stopwords.words('english')def wordTokens(text, stop_words):     wtokens = nltk.word_tokenize(text.lower())    wtokens = [w for w in wtokens if w not in stop_words]return wtokens
Since the names we are looking for in the text all consist of two words, we want to create bigrams. This is again really easy with the help of nltk
bigrm = nltk.bigrams(tokens)bigrms = list(bigrm)
If there were one-word names or names with more than two words, we could generalize the code by including the initial tokens, as well as trigrams or any other n-grams.
Finally, we create tuples from our characters list, to be directly comparable to the bigrams
def char_tuple_f(chars_list):    char_tuples_list = []    for char in chars:        tup = tuple(char.split(" "))        char_tuples_list.append(tup)return char_tuples_list
Our ultimate goal is to analyze the interactions the characters have with each other. But what counts as an interaction? Here, I followed Beveridge and Shan and counted an interaction between two characters when their names appear in the novel within 15 words of each other. Actually, the number of words, the “word distance” of the characters, can be a variable itself. After experimenting with various values, 15 seemed as a value which gave good results.
We create a dictionary with the names of the characters as keys. The values of each key are a numpy array of all the indexes where the character’s name appears in the bigrams list.
def indices_dic(char_tuples, bigr):    dic = {}    for tup in char_tuples:        char_name = " ".join(tup)        indices = [i for i, x in enumerate(bigr) if x == tup]        dic[char_name] = np.array(indices)        return dic
Then, for each character in the above dictionary, we take the difference of the character’s indexes with every other character indexes. If that difference is lower than some threshold (usually 15), we count that as one interaction between the characters. Here, the vectorization of numpy significantly speeds up the code. Finally, we add all the interactions of each character to a dictionary.
We only consider characters that have more than 3 interactions. This is done for the following reason. Sometimes, the name of a character at the end of a paragraph/line and the name of the character at the beginning of the next paragraph/line may appear within our threshold, even though the characters don’t really interact with each other. Such occurrences usually happen randomly and the probability of them happening for the same characters more than three times is low. On the other hand, if two characters truly interact with each other, they will have more than 3 interactions
def links_dic_f(indices_dic, threshold):        link_dic = {}    for first_char, ind_arr1 in indices_dic.items():        dic = {}        for second_char, ind_arr2 in indices_dic.items():            if first_char == second_char:                continuematr = np.abs(ind_arr1[np.newaxis].T -                          ind_arr2) <= threshold            s = np.sum(matr)            if s > 3:                dic[second_char] = s        link_dic[first_char] = dic        return link_dic
This part of the code could probably be optimized further. Nevertheless, for the size of the text I was analyzing, the code ran almost instantly and I didn’t feel the need for further optimization.
Since we are using the same list of characters for all sections of the novel, it is possible that some characters don’t appear, or don’t interact with anyone in certain sections. This is already taken care of in the interactions dictionary, but we would like to remove them from the characters list as well. This way, we will not have isolated nodes with no edges in our graph.
def remove_zero_link_chars(inter_dic, chars_list):    rem_set = set()    for key in inter_dic:        if inter_dic[key] == {}:            rem_set.add(key)     fin_list = [char for char in chars_list if char not in rem_set]    return fin_list
Now that we have a dictionary with the number of interactions of all characters, we are ready to create a graph representing our characters’ network. We are going to use the networkx library to create the graph. networkx expects the nodes of the graph as a list. We already have that, it is the list of our characters. Also, it expects the edges of the graph to be given as a tuple of two nodes plus a value for the weight of the edge. We create those tuples as follows
def edge_tuples_f(link_dic):    edges_tuples = []    for key in link_dic:        for item, value in link_dic[key].items():            tup = (key.title(), item.title(), value)            edges_tuples.append(tup)return edges_tuples
When we have both lists, creating the graph is extremely simple with networkx
import networkx as nxG = nx.Graph()G.add_nodes_from(node_chars)G.add_weighted_edges_from(edges_tuples)
Finally, we want to detect communities that exist within our graph. Communities are subsets of our graph, the members of which interact more with each other than with members of other communities. The algorithm behind community detection is beyond the scope of this article. Nevertheless, we want to show that the python module community can accomplish this, with just one line of code. We add this partition as an attribute to the nodes of the graph. This can be used later by Gephi, to group the nodes.
import communitypartition = community.best_partition(G)nx.set_node_attributes(G, partition, ‘group’)
We export the graph as a .gexf file
nx.write_gexf(G, outfile)
Using Gephi, we can produce beautiful visualizations of our graphs. The main point when creating the network in Gephi is that I use the Force Atlas layout. Also, I use the community groups for the color of the nodes and the number of edges to determine the size of the nodes.
In order to find the most influential character of the novel, we look at the centrality measures. We calculate three different centralities.
Degree centrality is defined as the number of edges of each node. A node, or a character in our case, with more edges, is considered more influential
Betweenness centrality shows how many times a node is found within the shortest path of two other nodes. Practically, it identifies the character
Eigenvector centrality measures the importance of nodes based on their connections, but connections to other important nodes contribute more than connections to unimportant nodes. It identifies the characters that interact with powerful characters.
We use all three centralities to calculate the importance of a character. Since we cannot directly compare different centralities, at the end, our assertion will be qualitative. Nevertheless, for characters of a novel, we consider someone who is connected with more important characters to be more influential than someone who has connections with minor characters. Thus, we consider the eigenvector centrality more decisive of a character’s importance than the other two centralities. Moreover, in our calculations, the eigenvector centrality takes into account the weight of the edges.
The networkx library includes algorithms to calculate all three centralities. We create a pandas data frame with the centralities of each node.
def calc_centralities(graph):        dgc = nx.degree_centrality(graph)    dgc = pd.DataFrame.from_dict(dgc, orient='index', columns=["DGC"])    btc = nx.betweenness_centrality(graph)    btc = pd.DataFrame.from_dict(btc, orient='index', columns=["BTC"])    evc = nx.eigenvector_centrality(graph, weight='weight')    evc = pd.DataFrame.from_dict(evc, orient='index', columns=["EVC"])df = pd.concat([dgc, btc, evc], axis=1)        return dfdf = calc_centralities(G)
Finally, we create a plot with the top 10 characters for all three centralities
import matplotlib.pyplot as pltdef plot_centrality(centr, df, title, n, col_list):        ax = plt.subplot(1, 3, n)    s = df.sort_values(centr, ascending=False)[:10]    x = list(s[centr].index)[::-1]    y = list(s[centr])[::-1]        for i, v in enumerate(y):        bars = plt.barh(x[i], v, color=col_list[n-1])        plt.title(title, size=22)    ax.get_xaxis().set_visible(False)    ax.spines['top'].set_visible(False)    ax.spines['bottom'].set_visible(False)    ax.spines['right'].set_visible(False)    ax.spines['left'].set_visible(False)    ax.tick_params(axis='y', length = 0, labelsize=14)col_list = ["peachpuff", "plum", "orange"]fig, ax = plt.subplots(1,3, figsize=(15, 10))plot_centrality("DGC", df, 'Degree Centrality', 1, col_list)plot_centrality("BTC", df, 'Betweeness Centrality', 2, col_list)plot_centrality("EVC", df, 'Eigenvector Centrality', 3, col_list)plt.savefig(figfile_centr, dpi=300)
Of course, for one of the colors of the graph, we use “peachpuff”, which reminds me of the oath at the peach garden.
The full jupyter notebook can be found here. The results of the analysis can be found in this article.