Recommending Trump’s Next Friend with Machine Learning
By: Marcell Ferencz
Tags: Link Prediction, Data Science, Imbalanced Class, Python, Graph Theory

The How

We’ll demonstrate the concepts I outlined above with real life data. This time we’ll be working front-to-back.

Training Data

In my previous post I have shown how I constructed ‘social’ links by identifying pairs of entities which appeared in the same news article, on the 26th of March. I repeated the same exercise for the following 6 days, all the way up to the 1st of April. This netted me over 400,000 (non-unique) links, distributed across the week like so:

Total number of entity links by day

I deliberately didn’t visualise the 7th day, as we won’t be creating a graph from that, we’ll only check newly formed links compared to day 6.

We’ll do a walk through the exercise for the first day for demonstration’s sake.

Let’s build the training data set by importing the CSV with all the links, filtering on the first day and grouping the same links together, assigning weights to them based on the count. I’ll also be discarding any links which only occurred once to focus on the more prominent ones:

import pandas as pd

import numpy as np df_links = pd.read_csv('/content/drive/My Drive/News_Graph/links.csv') date = '2020-03-31' df_links_train = df_links[df_links['date']==date] df_links_train = df_links_train.groupby(['from', 'to']).size().reset_index() df_links_train.rename(columns={0: 'weight'}, inplace=True) df_links_train = df_links_train[df_links_train['weight'] > 1]

Next, we’ll build a graph structure from the resultant links, then find the largest connected subgraph — this is a requirement for calculating our features. We’ll find that over 95% of nodes are connected on all days, anyway.

import networkx as nx G = nx.Graph() for link in tqdm(df_links_train.index):

G.add_edge(df_links_train.iloc[link]['from'],

df_links_train.iloc[link]['to'],

weight=df_links_train.iloc[link]['weight']) subgraphs = [G.subgraph(c) for c in nx.connected_components(G)] subgraph_nodes = [sg.number_of_nodes() for sg in subgraphs] sg = subgraphs[np.argmax(np.array(subgraph_nodes))] df_links_sg = nx.to_pandas_edgelist(sg) G = nx.Graph() for link in tqdm(df_links_sg.index):

G.add_edge(df_links_sg.iloc[link]['source'],

df_links_sg.iloc[link]['target'],

weight = df_links_sg.iloc[link]['weight'])

The above steps yield us a graph, G, which contains a weighted and undirected network that encapsulates the largest connected sub-graph representing the ‘social network’ of the news on the 26th of March. Specifically, it contains 8,045 edges between 2,183 nodes.

Network plot of the news on the 26th of March

Beautiful, isn’t it? Not very useful, though. Let’s see if we can find out which of these pink guys will make friends the next day.

Similarly to before, we define a data set of links weighted by occurrence, but this time we also filter to links where both entities appeared in the network on the previous day. After that, we will remove all links from the network on the 27th which have already appeared on the 26th and therefore aren’t newly formed edges. Finally, we’ll label the remaining links with a 1, to represent a link that was newly formed:

date_target = '2020-03-27' df_links_tar = df_links[df_links['date']==date_target]

df_links_tar = df_links_tar.groupby(['from', 'to']).size().reset_index()

df_links_tar.rename(columns={0: 'weight'}, inplace=True)

df_links_tar = df_links_tar[df_links_tar['count'] > 1]

df_links_tar.reset_index(drop=True, inplace=True) # filter to only include nodes which exist in training data all_nodes = G.nodes() df_links_tar = df_links_tar[(df_links_tar['from'].isin(all_nodes)) & (df_links_tar['to'].isin(all_nodes))] # create edge columns and filter out those who also appear in training data df_links_tar['edge'] = df_links_tar.apply(lambda x: x['from'] + ' - ' + x['to'], axis=1) df_links_sg['edge'] = df_links_sg.apply(lambda x: x['source'] + ' - ' + x['target'], axis=1) # remove edges which exist in training data df_links_tar = df_links_tar[~df_links_tar['edge'].isin(df_links_sg['edge'])] # label remaining edges with 1 df_links_tar['label'] = 1

With the above steps we have produced a dataframe with 2,183 pairs of entities which have not yet formed a link on the 26th, but will form one on the 27th, and labelled them with a 1. What about those which won’t form a link on the 27th?

Well, theoretically any unconnected pair in our graph could form a link the next day, for all we know. So we need to find all possible combinations of two nodes in the graph from the 26th, which aren’t already included in the pairs we identified as forming a link on the 27th, and label them 0.

Let’s pause here for a second and consider something.

We have 2,382 entities in our graph and we know 2,183 pairs of them will form a new link. How many won’t then? There is an enormous number of possibilities: 2382-choose-2, to be exact. That number is nearly 3 million, which would mean that the ratio of positive to negative samples would be under 1:1000! That’s a very imbalanced data set to train a classifier on. We have a few options to deal with this, but for this exercise I opted for down-sampling my data, because I knew I was likely to have a fair number of rows by the time I finished the whole week.

Note: this may not be best practice — I primarily went with down-sampling for demonstrative purposes and to reduce feature engineering and model training times. Something like SMOTE might be more appropriate here.

We will find all possible combinations of nodes in the network, remove those which have a link on the 26th and those which will form a link on the 27th. We’ll then randomly take a sample of 9x2,183 (nine times the number of newly formed links) from these unconnected pairs, to give us a training ratio of 1:10 — much better.

from itertools import combinations combs = list(combinations(all_nodes, 2)) all_combinations = pd.DataFrame(data=combs, columns=['from', 'to']) all_combinations['edge'] = all_combinations.apply(lambda x: x['from'] + ' - ' + x['to'], axis=1) all_combinations = all_combinations[~all_combinations['edge'].isin(df_links_sg['edge'])] all_combinations = all_combinations[~all_combinations['edge'].isin(df_links_tar['edge'])] all_combinations.reset_index(inplace=True, drop=True) sample_size = 9*len(df_links_tar.index) all_combinations = all_combinations.iloc[np.random.choice(len(all_combinations.index),

size=sample_size,

replace=False)] all_combinations['label'] = 0

Finally, we concatenate the two dataframes:

edges_to_predict = all_combinations.append(df_links_tar[['from', 'to', 'edge', 'label']])

The dataframe edges_to_predict will be home to the list of all new links, labelled with a 1, and a sample of edge pairs which won’t form a link, labelled with a 0.

Feature Engineering

We’ll recall that to create features we need to calculate the number of common neighbours, the Jaccard Coefficient, the Adamic/Adar Index and the Preferential Attachment Score for each pair of our training data. The NetworkX library will be our best friend for this. All we need to do is create a bunch of node pairs, that is, a list of tuples to identify the source and target of each:

bunch = [(source, target) for source, target in zip(edges_to_predict['from'].values, edges_to_predict['to'].values)]

Feature calculation is then laughably easy:

# Preferential Attachment preferential_attachment = nx.preferential_attachment(G, bunch) pref_attach = [] for u, v, p in preferential_attachment:

pref_attach.append(p) edges_to_predict['preferential_attachment'] = pref_attach # Jaccard Coefficient jaccard_coefficient = nx.jaccard_coefficient(G, bunch) jaccard = [] for u, v, p in jaccard_coefficient:

jaccard.append(p) edges_to_predict['jaccard_coefficient'] = jaccard # Common Neighbours cn = [] for source, target in zip(edges_to_predict['from'].values, edges_to_predict['to'].values):

cn.append(len(list(nx.common_neighbors(G, source, target)))) edges_to_predict['common_neighbours'] = cn # Adamic Adar Index adamic_adar = nx.adamic_adar_index(G, bunch) aa = [] for u, v, p in adamic_adar:

aa.append(p) edges_to_predict['adamic_adar_index'] = aa

Finally, we have real features and real targets! The dataframe now looks like this:

Classification

We’re ready to train our machine learning model. Off-screen, we have repeated the above process for 5 subsequent days, with a slight modification for the 6th day, our training data: on the 31st of March, I did not down-sample the data, as it won’t be a part of our training set. We need to pretend like we don’t know what links will be formed on the 1st of April, therefore we need to use all the data available.

We’ve seen in our sample training data that our feature columns assume widely varying ranges of values. To enhance our learning model, we’ll scale them to values with a zero mean and a standard deviation of 1, using scikit-learn:

from sklearn.preprocessing import StandardScaler df_test = pd.read_csv('/content/drive/My Drive/News_Graph/edges_test.csv') df_train= pd.read_csv('/content/drive/My Drive/News_Graph/edges_train.csv') features = ['preferential_attachment',

'jaccard_coefficient',

'common_neighbours',

'adamic_adar_index'] X_train = df_train[features].to_numpy()

Y_train = df_train['label'].to_numpy() X_test = df_test[features].to_numpy()

Y_test = df_test['label'].to_numpy() scaler = StandardScaler() X_train = scaler.fit_transform(X_train)

X_test = scaler.fit_transform(X_test)

As a reminder, we are training a model on a slightly imbalanced data set and testing on a realistic, very imbalanced data set:

Ratio of positive samples in training set: 0.1.

Ratio of positive samples in test set: 0.00092.

We’re ready to initialise our model: logistic regression.

Yes, logistic regression. Not what you expected when I touted machine learning in my title, is it? I promise I tried — I really did. I tested a Random Forest model, a Support Vector Machine, I spent a week tuning Light Gradient Boosting and XGBoost models. I grid-searched a deep neural network. None of them could beat the ROC of logistic regression. So there you have it. I suppose it’s quite fitting for my old-school features.

from sklearn.linear_model import LogisticRegression clf = LogisticRegression(class_weight='balanced') clf.fit(X_train, Y_train) lr_prob = clf.predict_proba(X_test)[:, 1] lr_pred = clf.predict(X_test)

The class_weight parameter ensures that the error penalties in the model are proportional to the ratio of positive/negative training samples in our data, making the model more sensitive to getting a positive prediction wrong, and increasing performance overall.

Let’s see how we did on our test data.

ROC Curve of logistic regression model

We have a very respectable ROC AUC score of 0.9312. Let’s plot the confusion matrix.

Logistic regression confusion matrix

This allows us to calculate our precision and recall:

Precision: 0.0076

Recall: 0.8394

Not bad — not great, but not bad. We managed to identify almost 84% of future links. However, we identified so many potential links that less than 1% of our positive predictions actually came true the next day.