Link Prediction in Bipartite Graph | by Edward Elson Kosasih | Towards Data Science

Answer: They can all be mathematically formulated as a graph link prediction problem!
In short, given a graph G (V, E) with |V| vertices and |E| edges, our task is to predict the existence of a previously unknown edge e_12 ∉ E between vertices v_1, v_2 ∈ V. We can then use the link prediction model to, for instance, recommend the two vertices to each other. In Linkedin/Facebook this is equivalent to recommending “people/friends you may know”, in Tinder this would be a new person for you to swipe, in Amazon this is the “customer also bought” recommendation, in Netflix this is the “Top Picks for you”, in Spotify this is the personalised “Based on your recent listening / Shows you might like” playlist and in supply chain this is a new supplier that you could try to source from.
In this article, we will focus on a particular type of network called bipartite graph. Mathematically speaking, this means we can split the vertices V into two disjoint sets, V_left and V_right, such that edges can only exist between vertex v_left ∈ V_left and v_right ∈ V_right. No two vertices from the same set are connected to each other. Bipartite graph can be used to model user-product network in a recommendation system e.g. V_left could be users and V_right products e.g. songs in Spotify, movies in Netflix, or items in Amazon. Users in these networks will only receive a recommendation about products and not other users, hence there are no edges formed between the same set.
Such constraints in edges make link prediction in bipartite graphs tricky. As we will show later, some commonly used link prediction algorithms will no longer work. Therefore, we need to consider different methods. In this article, we will consider spectral transformation techniques proposed by Kunegis (2011) [1].
We will also discuss some issues related to measuring the performance of link prediction algorithms, as highlighted by Yang et. al. (2015) [2].
Disclaimer: The aim of this article to show a typical pipeline that one goes through when doing link prediction. We will work with a random bipartite graph, hence we don’t expect any pattern to emerge really. In practice, you could replace the graph G with your problem set.
First, we import several python packages that we need to run our program. We use networkx as our underlying graph representation package.
import networkx as nximport randomimport numpy as npimport scipy as scimport matplotlib.pyplot as pltfrom sklearn import metricsfrom scipy.optimize import curve_fitfrom tqdm import tqdm
First, we create a random bipartite graph with 25 nodes and 50 edges (arbitrarily chosen). Looking at the adjacency matrix, we can tell that there are two independent block of vertices at the diagonal (upper-right to lower-left).
# create random graphG = nx.bipartite.gnmk_random_graph(15, 10, 50, seed=123)# get layouttop = nx.bipartite.sets(G)[0]pos = nx.bipartite_layout(G, top)# get adjacency matrixA = nx.adjacency_matrix(G)A = A.toarray()# plot adjacency matrixplt.title(“Adjacency Matrix”)plt.imshow(A, cmap=’Greys’)plt.show()# plot graph visualisationnx.draw(G, pos, with_labels=True)
Next, we split the graph into training and testing set by holding out 30% of the original edges [3]. Note that there are ongoing discussions about the importance of choosing test sets properly and the drastic impact it could have upon the performance result [2].
“””TRAINref: https://maelfabien.github.io/machinelearning/graph_4/#i-link-prediction“””# Remove 30% of the edgesproportion_edges = 0.3# this is our test setedge_subset = random.sample(G.edges(), int(proportion_edges * G.number_of_edges()))# Create a copy of the graph and remove the edgesG_train = G.copy()G_train.remove_edges_from(edge_subset)# adjacency matrixA_train = nx.adjacency_matrix(G_train)A_train = A_train.toarray()# visualise the train graphplt.title(“Train Graph”)nx.draw(G_train, pos, with_labels=True)plt.show()“””TEST“””G_test = nx.Graph()G_test.add_edges_from(edge_subset)# visualise the test graphplt.title(“Test Graph”)nx.draw(G_test, pos, with_labels=True)plt.show()
There are two commonly used metrics to evaluate the performance of link prediction algorithms: Mean Average Precision (MAP) and Receiver Operating Characteristics (ROC) & Precision-Recall Curves. There are certain caveats to each (see [2]) for further discussion, as this is beyond the scope of this article.
Mean Average Precision / MAP is defined as a weighted mean of the average precision across all nodes. A node’s average precision is obtained by ranking all predicted edges that are attached to it and calculating the accumulative mean of the precision score as we include more edges (see [4] for more derivation details). The formulas are provided below.
"""Mean Average Precision"""def MAP(G_test, G_pred, thres = 0):    # calculate avePrecision for each node and its neighbors    avePs = []        # loop through every node    for node in tqdm(G_test.nodes()):        # get predicted edges sorted in ranking order        rankedPredWeights = sorted(G_pred[node].items(), key=lambda x: -x[1]['weight'])        # only include edges that exist i.e. predicted rank / weight > threshold        rankedPred = filter(lambda x: x[1]['weight'] > thres, rankedPredWeights)        # get the rank        pred = [x[0] for x in rankedPred]        # calculate rel (existence of predicted edge in the groundtruth/actual set of edges)        # get groundtruth neighbors        gt = set(G_test[node])        rel = np.array([x in gt for x in pred])        # calculate P accumulative average of precision        predLength = len(pred)        P = np.array([            sum(rel[:i+1])/len(rel[:i+1]) for i in range(predLength)        ])        # calculate aveP        aveP = (rel @ P)/len(gt)        # keep track of results        avePs.append(aveP)    MAPvalue = sum(avePs) / len(avePs)    print("MAP: {}".format(MAPvalue))    return MAPvalue
However, in order to use MAP, we’ll need to determine a fixed threshold of edge existence beforehand (e.g. in the above example, we set the threshold to be 0. In other words, if the link prediction model outputs a non-zero positive score to a pair of nodes, then we consider that there might be an edge there). Yang et al [2] have identified several issues with this approach. Therefore, we implement another approach, where we sweep across several thresholds in order to obtain the Receiver-Operating Characteristic Curve and Precision-Recall Curve. We can then measure the performance by looking at the Area Under Curve (AUC). This is a common evaluation metric used in classification tasks.
"""Visualise Receiver-Operating Characteristic Curve and Precision-Recall Curve"""def ROC_PRC(pred, G):    # prediction score    y_score = [p[2] for p in pred]    # groundtruth label    y_true = [G.has_edge(p[0], p[1]) for p in pred]    fig, (ax1, ax2) = plt.subplots(1, 2)    # precision-recall curve    fpr, tpr, thresholds = metrics.precision_recall_curve(y_true,  y_score)     ax1.plot(fpr, tpr)    ax1.set_title("Precision-Recall Curve")       # receiver-operating characteristic curve    fpr, tpr, thresholds = metrics.roc_curve(y_true, y_score)     ax2.plot(fpr, tpr)    ax2.set_title("ROC Curve, AUC = {:.2f}".format(metrics.roc_auc_score(y_true, y_score)))        plt.show()
Having defined the training-testing set and evaluation metrics, we can now move on and discuss some algorithms that we can use for link prediction. If we forget about our bipartite graph for a while and consider general undirected graphs, one of the most common link prediction methods are neighbours-based techniques. The idea here is that two disconnected vertices who share common neighbours will have a higher likelihood of being linked by an edge. The exact likelihood score varies in practice, depending on the exact implementation (see Resource Allocation index, Jaccard Coefficient and Adamic-Adar [5]).
Unfortunately, these algorithms don’t work for bipartite graphs, because two vertices from different sets will never share common neighbours. If we insist on applying these algorithms, they will predict the existence of edges between the same sets, hence resulting in inferior performance, i.e. zero MAP.
for algo in [    nx.resource_allocation_index,    nx.jaccard_coefficient,    nx.adamic_adar_index]:    print(algo)    pred = list(algo(G_train))    # create graph    G_pred = nx.Graph()    G_pred.add_weighted_edges_from(pred)    # visualise adjacency matrix    Apred = nx.adjacency_matrix(G_pred)    Apred = Apred.toarray()    plt.imshow(Apred, cmap='Greys')    plt.show()    # evaluation    ROC_PRC(pred, G)    MAP(G_test, G_pred)
Researchers have identified this problem with neighbours-based approaches for a while. Several alternative algorithms such as odd paths counting have been proposed (odd because only odd length of paths matter in a bipartite graph eg even paths mean we’re connecting vertices from two different sets). Interestingly, Kunegis [1] proved that many of these algorithms can actually be generalised as the same spectral curve-fitting approach but with different kernel function.
The main idea lies in the following equation. Consider performing an eigenvalue decomposition on the adjacency matrix A of graph G. We’ll then obtain the eigenvector matrix U and diagonal eigenvalue matrix Λ. Kunegis [1] proposed that one can transform a graph with kernel function F by either applying it directly to the adjacency matrix F(A) or to its’ eigenvalue matrix F(Λ). The assumption here is that the eigenvectors stay the same, because we assume that the original and transformed graph are not vastly different. This is reasonable for our case, where the original graph is our train set while the transformed graph is our test set. We assume that the two comes from the same underlying distribution profile. If they differ, then this is an out-of-distribution task which is another research topic beyond the scope of this article.
Not all kernel functions F could work. For reference, Kunegis [1] has written down several of such working kernels as listed below.
Now consider our link prediction task, where we have our training graph G (adjacency matrix A) and testing graph G’ (adjacency matrix A’). The idea here is to find a function F that can map our original A to A’. We can formulate this as the following objective function. We can swap U around from F(Λ) to A’ because the objective is calculated under a Frobenius norm that is invariant under multiplication by an orthogonal matrix (we assume full rank eigenvector matrix hence orthogonal).
Kunegis [1] equated the above objective with a sum of one-dimensional least square regression problems as following. The function f (applied to eigenvalue) corresponds to its’ F version (applied to adjacency matrix) based on Table 1.
In order to visualise what type of regression model would we need, we can do a scatter plot from each input-output pair of
to see the pattern.
"""Kernel as Curve Fitting Problem"""# eigenvalue decompositionV_train, U_train = np.linalg.eig(A_train)# U.T * Atest * Utarget_V = U_train.T @ A @ U_train# take only the diagonalstarget_V = np.diag(target_V)# plot the patternplt.figure(figsize=(5, 3))plt.xlabel("V_train")plt.ylabel("V_test")plt.scatter(V_train, target_V, c='b')plt.show()
We can then try to fit several kernel functions that are specifically designed for bipartite graphs (with odd path countings), as follows.
We perform regression on the input-output data in order to learn the kernel parameters e.g. in this case α and β.
# fit kernel functionfor kernel in [OddPathCountingKernel(), SinhPseudokernel(), OddNeumannPseudokernel()]:    print(kernel)    # fit the kernel to the data        kernel.fit(V_train, target_V)    # predict the output    V_pred = kernel.pred(V_train)    # assume our function is exponential V_train with alpha = 0.6    plt.figure(figsize=(5, 3))    plt.xlabel("V_train")    plt.ylabel("V_test")    plt.scatter(V_train, target_V, c='b', label="target_V")    plt.scatter(V_train, V_pred, c='r', label="predicted_V")    plt.legend()    plt.show()    # transformation    Apred = U_train @ np.diag(V_pred) @ U_train.T    Apred = Apred.real    # make edges of prediction    pred = [(i, j, Apred[i, j]) for i in range(Apred.shape[0]) for j in range(Apred.shape[1])]    # create graph    G_pred = nx.Graph()    G_pred.add_weighted_edges_from(pred)    # evaluate MAP and precision    ROC_PRC(pred, G)    MAP(G_test, G_pred)
The first kernel that we use is odd path counting. Here we look at weighted sum of several odd powers of the original adjacency matrix (recall that k-th power of an adjacency matrix refers to numbers of k-hop neighbours from every node). The higher the value, the more likely it is for two vertices to be connected.
# odd path counting kernel functionclass OddPathCountingKernel:    def __init__(self):        self.a1 = 0        self.a3 = 0        self.a5 = 0        self.a7 = 0            def func(self, V, a1, a3, a5, a7):           return V * a1 + V**3 * a3 + V**5 * a5 + V**7 * a7    def fit(self, V_train, target_V):        # do curve fitting        popt, pcov = curve_fit(self.func, V_train, target_V)        self.a1, self.a3, self.a5, self.a7 = popt    def pred(self, V_train):        return self.func(V_train, self.a1, self.a3, self.a5, self.a7)
As shown by Kunegis [1], hyperbolic sine pseudokernel is a matrix exponential operation that’s basically equivalent to a weighted sum of all odd-power adjacency matrix. Smaller weights are assigned to terms with higher powers (using some clever tricks involving factorial terms), hence we can estimate this sum with a matrix exponential instead of having to enumerate all (infinitely many) powers of adjacency matrix.
# sinh pseudokernel functionclass SinhPseudokernel:    def __init__(self):        self.alpha = 0            def func(self, V, alpha):           return np.array([            alpha * (np.exp(lamb) - np.exp(-lamb)) for lamb in V        ])    def fit(self, V_train, target_V):        # do curve fitting        popt, pcov = curve_fit(self.func, V_train, target_V)        self.alpha, = popt    def pred(self, V_train):        return self.func(V_train, self.alpha)
This has similar flavours to hyperbolic sinh pseudokernel, as it involves an infinite weighted sum of odd-powered adjacency matrix. However, the weights are designed in a different way hence resulting in a different form of kernel.
# odd neumann pseudokernel functionclass OddNeumannPseudokernel:    def __init__(self):        self.alpha = 0            def func(self, V, alpha):           return np.array([            alpha * (1/(1-lamb) - 1/(1+lamb)) for lamb in V        ])    def fit(self, V_train, target_V):        # do curve fitting        popt, pcov = curve_fit(self.func, V_train, target_V)        self.alpha, = popt    def pred(self, V_train):        return self.func(V_train, self.alpha)
We can see that Odd Neumann pseudokernel doesn’t seem to fit the input-output data well (from the scatter plot), resulting in a worse AUC score compared to Odd Path Counting and Hyperbolic Sine Pseudokernel.
In this article we’ve seen how to (1) define the link prediction task and split the train-test set, (2) write down two evaluation metrics: one with a fixed threshold (MAP) and another with sweeping thresholds (ROC and PR), (3) use spectral curve fitting methods to predict the existence of edges in a bipartite graph. For more rigorous mathematical derivation, please refer to the two papers from [1] and [2].
Find this article in a jupyter notebook version in my github repo here.
[1] Kunegis, J. 2011. On the Spectral Evolution of Large Networks. Accessed from https://github.com/kunegis/phd.
[2] Yang, Y., Lichtenwalter, R. N., Chawla, N. V. 2015. Evaluating Link Prediction Methods. Accessed from https://arxiv.org/abs/1505.04094.
[3] Fabien, M. Graph Learning. Accessed from https://maelfabien.github.io/machinelearning/graph_4/#i-link-prediction.
[4] Accessed from https://www.csie.ntu.edu.tw/~r01922164/SNA/Problems.pdf.
[5] Accessed from https://networkx.github.io/documentation/stable/reference/algorithms/link_prediction.html.