Building a Social Network from the News using Graph Theory | by Marcell Ferencz | Towards Data Science

Who are the most influential individuals on the news? What does the sprawling web of politicians, companies and celebrities really look like? How is Meghan Markle related to Argos?
If you’ve ever found yourself lying in bed sleeplessly, wondering any of the above, you’ve clicked on the right article. For we will explore a novel way of representing the vast information fed to us every minute of every hour of every day through news channels and opinion pieces: we are going to build a social network of people on the news.
We live in an age of media and information, and the importance of understanding the intricacies of the News cannot be overstated. In a previous article I have outlined the value of utilising social media to the benefit of any organisation, and demonstrated one approach using data from Twitter. Well, if Twitter is the mouthpiece of the consumer, its earpiece is listening to the news. In fact, many Tweeters these days simply share news articles in an effort to signal their agreement with (or outrage over) it.
Knowing if your organisation, product or boss was on the news is not difficult, but knowing in whose company they were while on the news can give you valuable insight into the kind of crowd they’re being banded with by those pesky journalists.
On a more technical note, building graph representations (i.e. social networks) of data opens up a whole host of possibilities for data science applications, such as finding the most central nodes (influential people) or identifying clusters of nodes (cliques of people) based on their edges (connections).
On a far less technical note, network graphs look really cool.
The fundamental premise behind building our social network will be two-fold and quite simple:
If two people are mentioned in the same article, they are friends.The more articles mention the same two people, the closer they are as friends.
If two people are mentioned in the same article, they are friends.
The more articles mention the same two people, the closer they are as friends.
Let’s take an example: if an article mentions Donald and Boris , and two other, separate articles mention Donald and Mike, we’ll say that Donald is friends with Boris, and Donald is also friends with Mike, only twice as much. We’ll therefore construct a social network like the following:
We not only get a pictorial representation of the friendship group, we can also start seeing hidden relationships: although Boris wasn’t mentioned in the same article as Mike, we can guess with some certainty that the two are related (and that they are related via their mutual friend, Donald). We can also tell that Donald is the alpha male in the group, having influence over both Mike and Boris.
This is just a bare-bones example of three (very short) articles on the news. You can imagine the kinds of insights we can draw from hundreds of documents writing about thousands of people. To achieve this, we’ll use two techniques, which I’ll briefly touch upon now.
Our first order of business is identifying individuals of interest within articles, which is not so easily done, if we don’t know whom we’re looking for ahead of time (otherwise we could use a simple string search). Luckily for us, Named Entity Recognition exists.
Named Entity Recognition is a Natural Language Processing task for extracting information from text; as the name suggests, it recognises entities, such as proper nouns within unstructured data. It achieves this using statistical models trained on large corpora of documents; the model learns to recognise and categorise entities based on the context in which they appear as words. We’ll use one of these models to list, for each article, words tagged as persons and organisations by the model.
Once we have our list of entities for each article, we’ll organise them into a graph structure. Graphs are, by definition, a set of vertices and edges:
Where G is our graph, made up of a set of vertices V (or nodes) and a set of edges E (or links).
This article isn’t meant as a primer on Graph Theory, but I do want to highlight a few important properties of graphs, which we’ll find useful further along the exercise:
Graphs can be directed or undirected — a directed graph’s nodes are linked with a direction (surprisingly), whereas the direction of links are irrelevant for an undirected graph. In our case, we will set up our network as an undirected graph, because Mike and Don being mentioned in the same article alone does not give any indication of direction of information between the two entities.Graphs can be connected or unconnected — a graph is connected if you can trace a path from any node to any other node. Our graph from above is connected, because all nodes are somehow connected to all others; if I were to introduce a fourth node, Jeremy, who wasn’t mentioned in any article together with Don, Mike or Boris, he would have no connection to any of the existing nodes, and our graph would become unconnected.Nodes have centrality measures — these are metrics to describe how important a node is within a network. One of these measures (which we’ll use in the upcoming analysis) is eigenvector centrality, which assigns a score to each node based on how many other important nodes to which it’s connected. A famous use case of eigenvector centrality is Google’s PageRank algorithm.Graphs can have cliques — cliques are a subset of graphs (or sub-graphs) in which all pairs of nodes are connected. These are the mathematical representations of friendship groups from your school days. In our example, if Mike also had a connection to Boris, they would form a clique.
Graphs can be directed or undirected — a directed graph’s nodes are linked with a direction (surprisingly), whereas the direction of links are irrelevant for an undirected graph. In our case, we will set up our network as an undirected graph, because Mike and Don being mentioned in the same article alone does not give any indication of direction of information between the two entities.
Graphs can be connected or unconnected — a graph is connected if you can trace a path from any node to any other node. Our graph from above is connected, because all nodes are somehow connected to all others; if I were to introduce a fourth node, Jeremy, who wasn’t mentioned in any article together with Don, Mike or Boris, he would have no connection to any of the existing nodes, and our graph would become unconnected.
Nodes have centrality measures — these are metrics to describe how important a node is within a network. One of these measures (which we’ll use in the upcoming analysis) is eigenvector centrality, which assigns a score to each node based on how many other important nodes to which it’s connected. A famous use case of eigenvector centrality is Google’s PageRank algorithm.
Graphs can have cliques — cliques are a subset of graphs (or sub-graphs) in which all pairs of nodes are connected. These are the mathematical representations of friendship groups from your school days. In our example, if Mike also had a connection to Boris, they would form a clique.
I’ve rambled on for long enough about entities and graphs — let’s get to the nitty gritty now.
So far, I’ve not touched on where I actually intend to get all these articles I was bragging about. There are, of course, many News APIs you can use to extract information from the news. The source I used was from the GDELT Project, which is a free, open platform of all world events, monitored in real time from across the globe.
Supported by Google Jigsaw, the GDELT Project monitors the world’s broadcast, print, and web news from nearly every corner of every country in over 100 languages...
They carry out a number of very interesting activities and analyses (seriously, check them out), but we’ll be interested in their raw data, which they also make available completely free of charge (thank you GDELT). They publish daily CSVs with thousands of events which occurred during that day, but more importantly, they include the URL of the news source which reported on the event.
I used the file containing the events that ocurred on the 26th of March, and filtered on news published by some of the UK’s most read news providers. I limited my search in such a way to somewhat narrow down the volume of entities we’ll be dealing with and also to focus on what’s important for the British mainstream reader. Doing so still netted me whopping 805 articles from the following sources:
Once I had my 805 URLs, I used Goose to extract the content from each web page. Again, this is not the focus of this article, so I won’t go into the detail. There are countless resources out there which will tell you how to do that (including the Goose documentation), if you’re interested in following along.
Extracting the title and the content of each article allowed me to organise my data into a nice pandas DataFrame:
If you’ve read my previous article (that I already shamelessly linked to earlier), you’ll know of my love affair with the Flair library, providing state-of-the-art NLP solutions in a few lines of code. Flair uses a neural language model (Akbik et al., 2018) to assign tags to text data, beating most previous models’ accuracy in the process. Flair is more than good enough for us. Flair is a blessing. Let’s take it for a spin.
We’ll tell Flair’s sentence tagger to predict entities within the sentence:
Boris went to Seattle with Donald to meet Microsoft.
!pip install flairimport torchfrom flair.data import Sentencefrom flair.models import SequenceTaggertagger = SequenceTagger.load('ner')sentence = Sentence('Boris went to Seattle with Donald to meet Microsoft')tagger.predict(sentence)for entity in sentence.get_spans('ner'):  print(entity)
Which yields four entities (as expected):
PER-span [1]: “Boris”LOC-span [4]: “Seattle”PER-span [6]: “Donald”ORG-span [10]: “Microsoft”
It successfully identified all four entities, and was able to recognise the two people (PER), one location (LOC) and one organisation (ORG).
Flair’s Sentence objects have a useful .to_dict() method which allows us to extract the entities in a more robust way:
sentence.to_dict(tag_type='ner')
Giving us a dictionary with all the juicy properties of each entity in the sentence:
{'entities': [{'confidence': 0.999889612197876,  'end_pos': 5,  'start_pos': 0,  'text': 'Boris',  'type': 'PER'}, {'confidence': 0.9988295435905457,  'end_pos': 21,  'start_pos': 14,  'text': 'Seattle',  'type': 'LOC'}, {'confidence': 0.9999843835830688,  'end_pos': 33,  'start_pos': 27,  'text': 'Donald',  'type': 'PER'}, {'confidence': 0.8911125063896179,  'end_pos': 56,  'start_pos': 47,  'text': 'Microsoft',  'type': 'ORG'}], 'labels': [], 'text': 'Boris went to Seattle with Donald to meet with Microsoft'}
We stated above that we only care about people and organisations, so we’ll organise the dict into a nice DataFrame and filter for those two:
sent_dict = sentence.to_dict(tag_type='ner')df_ner = pd.DataFrame(data={  'entity': [entity['text'] for entity in sent_dict['entities']],  'type': [entity['type'] for entity in sent_dict['entities']] })df_ner[df_ner['type'].isin(['PER', 'ORG'])]
To build the actual social network, we’ll use the tried and trusted NetworkX package. Its Graph() class needs (at least) a list of edges for the graph, so we’ll massage our list of entities into a list of paired connections.
We’ll use the combinations functionality from itertools to, well, find all possible combinations given a list of items. First, we’ll sort our entities alphabetically — this is to ensure that in every pair we find (now and thereafter), the alphabetically superior entity appears on the left hand side, and we don’t duplicate pairs of A-B and B-A, for instance:
from itertools import combinationsdf_ner = df_ner.sort_values('entity')combs = list(combinations(df_ner['entity'], 2))df_links = pd.DataFrame(data=combs, columns=['from', 'to'])df_links
We are ready to create our graph and plot it!
import networkx as nxG = nx.Graph()for link in df_links.index:  G.add_edge(df_links.iloc[link]['from'],             df_links.iloc[link]['to'])nx.draw(G, with_labels=True)
All of the above can be wrapped in a nice function for us to apply to each of our articles. You’ll notice a couple of additions — first, I am doing some basic cleaning in the process, removing newlines and the like, and I’m also removing pronouns, which sometimes get recognised as named entities. Another key step here is that given an entity (say, Boris Johnson), I’m taking their last name only, as often individuals are referred to by their last name only. This step ensures that Boris Johnson and Johnson won’t be counted as two different entities.
def get_ner_data(paragraph):  # remove newlines and odd characters  paragraph = re.sub('\r', '', paragraph)  paragraph = re.sub('\n', ' ', paragraph)  paragraph = re.sub("’s", '', paragraph)  paragraph = re.sub("“", '', paragraph)  paragraph = re.sub("”", '', paragraph)  # tokenise sentences  sentences = tokenize.sent_tokenize(paragraph)  sentences = [Sentence(sent) for sent in sentences]  # predict named entities  for sent in sentences:    tagger.predict(sent)  # collect sentence NER's to list of dictionaries  sent_dicts = [sentence.to_dict(tag_type='ner') for sentence in sentences]  # collect entities and types  entities = []  types = []  for sent_dict in sent_dicts:    entities.extend([entity['text'] for entity in sent_dict['entities']])    types.extend([entity['type'] for entity in sent_dict['entities']])  # create dataframe of entities (nodes)  df_ner = pd.DataFrame(data={'entity': entities, 'type': types})  df_ner = df_ner[df_ner['type'].isin(['PER', 'ORG'])]  df_ner = df_ner[df_ner['entity'].map(lambda x: isinstance(x, str))]  df_ner = df_ner[~df_ner['entity'].isin(df_contraptions['contraption'].values)]  df_ner['entity'] = df_ner['entity'].map(lambda x: x.translate(str.maketrans('', '', string.punctuation)))  df_ner['entity'] = df_ner.apply(lambda x: x['entity'].split(' ')[len(x['entity'].split(' '))-1] if x['type']=='PER' else x['entity'], axis=1)  df_ner = df_ner.drop_duplicates().sort_values('entity')  # get entity combinations  combs = list(combinations(df_ner['entity'], 2))  # create dataframe of relationships (edges)  df_links = pd.DataFrame(data=combs, columns=['from', 'to'])  return df_ner, df_links
We’re ready to apply our function to each article’s content. Note that this took me about 40–50 minutes to run on Google Colab for my 800 articles:
df_ner = pd.DataFrame()df_links = pd.DataFrame()for content in tqdm(df_day['content']):  try:    df_ner_temp, df_links_temp = get_ner_data(content)    df_ner = df_ner.append(df_ner_temp)    df_links = df_links.append(df_links_temp)  except:    continue
I put the try-catch in there just in case no entities are recognised in an article and the DataFrame.append() method fails. I know it’s not good practice. I did it anyway.
Armed with our massive list of entity links, we can start building our graph. To make things more manageable, I’ll only keep links which have appeared in at least two separate articles:
df_links = df_links.groupby(['from', 'to']).size().reset_index()df_links.rename(columns={0: 'weight'}, inplace=True)df_links = df_links[df_links['weight'] > 1]df_links.reset_index(drop=True, inplace=True)df_links.sort_values('weight', ascending=False).head(10)
By far the most connections found were between Boris Johnson and the NHS, followed by Matt Hancock and the NHS. Further honourable mentions go to Prince Harry and Meghan and Andrew Cuomo and Donald Trump.
Let’s build a more comprehensive picture, though. We’ll take the links which have appeared in over 6 articles (trust me, it’s the lowest number where you can still make something out of a plot) and draw the graph as before:
df_plot = df_links[df_links['weight']>6]df_plot.reset_index(inplace=True, drop=True)G_plot = nx.Graph()for link in tqdm(df_plot.index):  G_plot.add_edge(df_plot.iloc[link]['from'],                  df_plot.iloc[link]['to'],                  weight=df_plot.iloc[link]['weight'])pos = nx.kamada_kawai_layout(G_plot)nodes = G_plot.nodes()fig, axs = plt.subplots(1, 1, figsize=(15,20))el = nx.draw_networkx_edges(G_plot, pos, alpha=0.1, ax=axs)nl = nx.draw_networkx_nodes(G_plot, pos, nodelist=nodes, node_color='#FAA6FF', with_labels=True, node_size=50, ax=axs)ll = nx.draw_networkx_labels(G_plot, pos, font_size=10, font_family=’sans-serif’)
Each dot represents an entity — a person or organisation — and a link between two dots mean they appeared in at least 2 separate articles together.
We can immediately see some groups of interest — the NHS and Johnson are indeed very much in the centre of things, but we can spot members of the Royal Family to the upper right corner as well as US politicians on the bottom half of the map.
And there you have it — we have built a social network from real news data. Let’s extract some insight from it!
Now that we’ve done the heavy lifting, the fun part can begin. We will ask our graph some questions.
Let’s start simple — we’ll use the nodes() and edges() methods to find the number of entities and connections in our social network:
n_nodes = len(G.nodes())n_edges = len(G.edges())print(f'There were {n_nodes} entities and {n_edges} connections found in the network.')
Which tells us that there were 2287 entities and 8276 connections found in the network.
In other words, is our graph connected? If not, how big is each subgraph?
nx.is_connected(G)
This command returns False, telling us that not all nodes are connected to each other in the graph.
subgraphs = [G.subgraph(c) for c in nx.connected_components(G)]subgraph_nodes = [sg.number_of_nodes() for sg in subgraphs]df_subgraphs = pd.DataFrame(data={  'id': range(len(subgraph_nodes)),  'nodes': subgraph_nodes})df_subgraphs['percentage'] = df_subgraphs['nodes'].map(lambda x: 100*x/sum(df_subgraphs['nodes']))df_subgraphs = df_subgraphs.sort_values('nodes', ascending=False).reset_index(drop=True)df_subgraphs
Over 95% of our nodes belong to one big connected cluster. Interestingly, there don’t seem to be separate, equally large graphs in our network; the vast majority of entities are connected to one another in some way.
We will use a shortest-path algorithm to find this out, but first we’ll need to do some housekeeping. We’ll recreate our graph using only the nodes which belong in the 95% who all know each other (we want to make sure our graph is connected) and we’ll also add a new attribute, inverse_weight to our edges. This will be the reciprocal of our original weight (i.e. the number of articles mentioning the two entities), which will help our shortest path algorithm prioritise high weights (more common connections) over low ones.
sg = subgraphs[np.argmax(np.array(subgraph_nodes))]df_links_sg = nx.to_pandas_edgelist(sg)df_links_sg['inverse_weight'] = df_links_sg['weight'].map(lambda x: 1/x)G = nx.Graph()for link in tqdm(df_links_sg.index):  G.add_edge(df_links_sg.iloc[link]['source'],   df_links_sg.iloc[link]['target'],  weight = df_links_sg.iloc[link]['weight'],  inverse_weight = df_links_sg.iloc[link]['inverse_weight'])
We can now compute the shortest path between any two entities, finally — finally — giving us the long-awaited answer to the question of how Meghan Markle is related to the retail store Argos.
source='Markle'target='Argos'path = nx.shortest_path(G,  source=source,  target=target,  weight='inverse_weight')path
Which gives us the chain of entities leading from Markle to Argos: ‘Markle’, ‘Charles’, ‘Johnson’, ‘Sainsbury’, ‘Argos’.
So Meghan Markle and Prince Charles appeared in an article, Prince Charles and Boris Johnson appeared in another article, and so on...
Let’s take a look at which articles these were:
df_path = pd.DataFrame([(path[i-1], path[i]) for i in range(1, len(path))], columns=['ent1', 'ent2'])def get_common_title(ent1, ent2):  df_art_path = df_articles[(df_articles['content'].str.contains(ent1)) & (df_articles['content'].str.contains(ent2))]  df_art_path.sort_values('date', ascending=False).head(1)  return df_art_path.iloc[0]['title']df_path['titles'] = df_path.apply(lambda x: get_common_title(x['ent1'], x['ent2']), axis=1)for title in df_path['titles']:  print(title)
The figure above illustrates our route from Meghan all the way to Argos, one article at a time:
Meghan was in an article with Prince Charles about her forbidding Prince Harry to visit the latter.Prince Charles appeared in an article with Boris Johnson on testing positive for Coronavirus.Boris was in an article with Sainsbury’s on the topic of panic buying in shops.Sainsbury’s appeared in an article with Argos on stores remaining open during the lockdown.
Meghan was in an article with Prince Charles about her forbidding Prince Harry to visit the latter.
Prince Charles appeared in an article with Boris Johnson on testing positive for Coronavirus.
Boris was in an article with Sainsbury’s on the topic of panic buying in shops.
Sainsbury’s appeared in an article with Argos on stores remaining open during the lockdown.
Onto more serious waters then. We can find each node’s centrality measure to find out how influential they are on the network. In other words, we will assign a score to each person or organisation based on how many other influential people or organisations they have appeared together with. We recall from an earlier paragraph that this can be achieved using the eigenvector centrality measure:
nodes = []eigenvector_cents = []ec_dict = nx.eigenvector_centrality(G, max_iter=1000, weight='weight')for node in tqdm(G.nodes()):  nodes.append(node)  eigenvector_cents.append(ec_dict[node])df_centralities = pd.DataFrame(data={  'entity': nodes,  'centrality': eigenvector_cents})
Visualising the 20 most influential entities in the network gives:
The longer (and greener) the bar for an entity, the more influential they have been on the news.
The NHS and Boris Johnson lead the pack in terms of influence on British news, which is not surprising, considering the situation we’re all in with COVID-19.
In other words, can we identify cliques in our network? Remember, cliques are a set of nodes in a graph in which every pair of nodes is connected. The algorithm we’ll use to do this is called k-clique communities (Palla et al., 2005), which allows us to find closely connected clusters in the network by defining k, or the minimum number of nodes required for a clique to be formed. It’s worth noting that one node can be very popular and belong to multiple cliques.
Setting k = 12 gives us 9 nice and manageable cliques:
from networkx.algorithms.community.kclique import k_clique_communitiescliques = list(k_clique_communities(G, 12))
As before, we’ll also calculate the eigenvector centrality in each subgraph formed by the cliques (this, again, will be based on edge weights) and visualise the results:
Each pair of graphs represents a clique. On the left plot, we see the position of the nodes on the network, and on the right, we see the most influential nodes in the clique.
This allows us to identify core topics on the news via the groups of people who appeared together a lot:
Clique 0: US political and commercial entities
Clique 1: Australian academic institutes
Clique 2: UK military entities
Clique 3: UK retailers
Clique 4: UK political organisations
Clique 5: London transport companies
Clique 6: British politicians
Clique 7: more British politicians
Clique 8: the British Royal Family
We collected over 800 articles published on the 26th or March by some of the most read British newspapers
We used a state-of-the-art Named Entity Recognition algorithm to extract people and organisations that appeared together in articles
We structured the data in graph form, resulting in a social network of over 2,000 entities and over 8,000 connections
We found that over 95% of entities in our network were connected to each other
We found that Meghan Markle can be linked to Argos
We found that the NHS, Boris Johnson and Prince Charles were the most influential entities on the news on the day
We identified 9 distinct groups of entities, mainly politicians and government bodies who regularly all appeared together on the news
This took some time and writing, so I commend you if you’ve made it this far — I hope it was worth it. This exercise merely scratched the surface of what can be achieved with graph data structures. I intend to go a bit further with this data in a follow-up article, but for now, I’ll leave it here.
Did I do something wrong? Could I have done something better? Did I do something well?
Please don’t hesitate to reach out to me on LinkedIn; I’m always happy to be challenged or just have a chat if you’re interested in my work.
If you want to play with the code yourself, please follow the link to my Google Colab notebooks: