Title: Mixture of Link Predictors on Graphs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Link Prediction

Graph Neural Networks

Mixture of Experts
Abstract: Link prediction, which aims to forecast unseen connections in graphs, is a fundamental task in graph machine learning. Heuristic methods, leveraging a range of different pairwise measures such as common neighbors and shortest paths, often rival the performance of vanilla Graph Neural Networks (GNNs). Therefore, recent advancements in GNNs for link prediction (GNN4LP) have primarily focused on integrating one or a few types of pairwise information. 
In this work,  we reveal that different node pairs within the same dataset necessitate varied pairwise information for accurate prediction and models that only apply the same pairwise information uniformly could achieve suboptimal performance.
As a result, we propose a simple mixture of experts model Link-MoE for link prediction.  Link-MoE utilizes various GNNs as experts and  strategically selects the appropriate expert for each node pair based on various types of pairwise information. Experimental results across diverse real-world datasets demonstrate substantial performance improvement from Link-MoE. Notably,  Link-Mo achieves a relative improvement of 18.71% on the MRR metric for the Pubmed dataset and 9.59% on the Hits@100 metric for the ogbl-ppa dataset, compared to the best baselines. The code is available at https://github.com/ml-ml/Link-MoE/.

Mixture of Link Predictors on Graphs

Abstract

Link prediction, which aims to forecast unseen connections in graphs, is a fundamental task in graph machine learning. Heuristic methods, leveraging a range of different pairwise measures such as common neighbors and shortest paths, often rival the performance of vanilla Graph Neural Networks (GNNs). Therefore, recent advancements in GNNs for link prediction (GNN4LP) have primarily focused on integrating one or a few types of pairwise information. In this work, we reveal that different node pairs within the same dataset necessitate varied pairwise information for accurate prediction and models that only apply the same pairwise information uniformly could achieve suboptimal performance. As a result, we propose a simple mixture of experts model Link-MoE for link prediction. Link-MoE utilizes various GNNs as experts and strategically selects the appropriate expert for each node pair based on various types of pairwise information. Experimental results across diverse real-world datasets demonstrate substantial performance improvement from Link-MoE. Notably, Link-MoE achieves a relative improvement of 18.71% on the MRR metric for the Pubmed dataset and 9.59% on the Hits@100 metric for the ogbl-ppa dataset, compared to the best baselines. The code is available at https://github.com/ml-ml/Link-MoE/ .

1 Introduction

Link prediction (LP) is a central challenge in graph analysis with many real-world applications, such as recommender systems [1, 2], drug discovery [3], and knowledge graph completion [4, 5]. Specifically, LP attempts to predict unseen edges in a graph. Unlike node-level tasks where Graph Neural Networks (GNNs) excel in modeling individual node representations [6], LP demands the use of pairwise node representations to model the existence of a link where vanilla GNNs often fall short [7, 8]. Traditionally, various heuristic methods [9] were used to identify new links by encapsulating the pairwise relationship between two nodes. For instance, the Common Neighbors (CN) heuristic [10] counts the number of shared neighbors between a node pair, postulating that the number of common neighbors can indicate the likelihood of a connection. The Katz index [11] considers the total number of paths between two nodes, assigning a higher weight to those shorter in length. Node feature similarity-based methods [12] assumes that nodes with similar features tend to connect. Despite their simplicity, these heuristic-based methods are still considered as strong baselines in LP tasks.

To leverage both the representational power of GNNs and the effectiveness of heuristics, recent GNN4LP works have sought to incorporate pairwise information into GNN frameworks, thereby

enhancing their expressiveness for better link prediction. For example, NCN/NCNC [13] exploit the common neighbor information into GNNs. Neo-GNN [14] further incorporates multi-hop neighbor overlap information. SEAL [15] and NBFNet [16] leverage the full and partial labeling trick, respectively, to indict the target node pair, which has been proven to learn heuristic patterns like common neighbors and the Katz index. These GNN4LP methods mark significant progress in link prediction [17].

However, both traditional heuristic approaches and GNN4LP models typically adopt a one-size-fits-all solution, uniformly applying the same strategy to all target node pairs. There are several limitations with this one-size-fits-all solution: (1) Limited Use of Heuristics: These methods utilize only one or a few heuristics. Our preliminary studies in Section 3 have shown that different heuristics tend to complement each other, and employing multiple heuristics within the same dataset can lead to improved link prediction performance. Therefore, methods relying on a single type of heuristic might not be optimal. (2) Uniform Application of Heuristics : In Section 3, we also find that different node pairs within the same dataset often require distinct heuristics for predictions. Consequently, these methods that uniformly apply the same heuristic across all node pairs lack this adaptability, potentially leading to suboptimal performance. These findings underscore the pressing need for an approach that can adaptively apply a range of pairwise information specific to node pairs. Inspired by the superior performance of existing GNN4LP, we delved deeper into understanding different GNN4LP models. Our investigation reveals that these models are highly complementary and they excel under specific conditions, often correlated with particular heuristics. For example, the NCN model tends to perform well in scenarios with a high number of common neighbors. These observations motivate us to ask: can we design a strategy that can simultaneously enjoy the strengths of various GNN4LP models and correspondingly enhance link prediction?

In response to this question, we introduce a simple yet remarkably effective mixture of experts model for link prediction Link-MoE . This model operates by utilizing a range of existing link predictors as experts. A gating function is learned to assign different node pairs to different experts based on various types of pairwise information. Extensive experimental results showcase the surprisingly effective performance of Link-MoE. For instance, it surpasses the best baseline on Pubmed and ogbl-ppa dataset by 18.71% on the MRR and 9.59% on the Hits@100, respectively.

2 Related Work

2.1 Link Prediction

Link prediction aims to predict unseen links in a graph. There are mainly three classes of methods for the link prediction task. An overview of each is given below.

Heuristic Methods : Heuristic methods have been traditionally used for link prediction. They attempt to explicitly model the pairwise information between a node pair via hand-crafted measures. Several classes of heuristics exist for link prediction [17] including: local structural proximity, global structural proximity, and feature proximity. Local Structural Proximity (LSP): These method extract the information in the local neighborhood of a node pair. Common Neighbors (CN) [10], AdamicAdar (AA) [18], and Resource Allocation [19] are popular measures that consider the number of shared 1-hop neighbors between the node pair. Global Structural Proximity (GSP): These methods attempt to model the interaction of a node pair by extracting the global graph information. The Shortest Path Distance assumes that a shorter distance between nodes results in a higher likelihood of them connecting. Both Katz Index [11] and Personalized Pagerank (PPR) [20] consider the paths of disparate length that connect both nodes, giving a higher weight to those shorter paths. Feature Proximity (FP): FP measures the similarity of the node features for both nodes in the pair, positing that nodes with similar features are more likely to form edges. Previous work [21, 22] has derived heuristics algorithms to measure the feature proximity.

GNN-based Methods : Recent work has looked to move beyond pre-defined heuristic measures and model link prediction through the use of graph neural networks (GNNs). Earlier work [23, 24, 25] has sought to first use a GNN to learn node representations, and the representations for both nodes in a pair are then used to predict whether they link. However, multiple works [7, 8] have shown that node-based representations are unable to properly model link prediction. As such, newer methods, which we refer to as GNN4LP, attempt to learn pairwise representations to facilitate link prediction. Both SEAL [7] and NBFNet [16] condition the GNN aggregation on either both or one of the two

nodes in the pair. While expressive, both methods tend to be prohibitively expensive as they require aggregating messages separately for each node pair. Recent methods [14, 13, 26, 27] have attempts to devise more efficient ways of learning pairwise representations by injecting pairwise information in the score function, bypassing the need of customizing the GNN aggregation to each node pair. These methods typically attempt to exploit different structural patterns on the local and global scales.

(a) Citeseer (K=3)

(b) ogbl-collab (K=20)

FCS

0.33

0.52

0.58

FCS

0.19

0.52

0.56

2

0.52

0.35

0.53

2

0.52

0.5

0.52

0.58

0.53

0.53

0.56

0.52

0.46

FCS

CN

SP

FCS

N

SP

(a) Citeseer

(b) ogbl-collab

FCS

1

0.23

0.25

FCS

0.29

0.29

2

0.23

1

0.66

0.29

0.68

0.25

0.66

1

0.29

0.68

1

FCS

CN

SP

FCS

CN

SP

Ensemble Methods : Ensemble-based methods for link prediction primarily fall into two categories: bagging and stacking. Bagging methods create multiple base learners trained on varied data subsets and integrate their predictions for final output [28, 29]. Stacking methods train multiple models on the entire graph and use a meta-model to integrate their predictions ( [30, 31, 32, 33]). However, these methods directly combine the predictions of base learner without considering the specific patterns and heuristics of base learners.

2.2 Mixture of Experts

The use of Mixture of Experts (MoE) [34, 35], which is based on the divide-and-conquer principle to divide problem to different experts, has been explored across various domains [36]. Recent researches mainly focus on the efficiency of leveraging the MoE in the NLP [37, 38] or Computer Vision [39] domains. Chen et al. [40] attribute the success of MoE to the cluster structure of the underlying problem. In the graph domain, GMoE [41] integrates the MoE model with GNNs, enabling nodes to learn from information across different hops. Wu et al. [42] leverage MoE to address the distribution shift issue in GNNs. To the best of our knowledge, Link-MoE represents the first instance of MoE specifically tailored for the link prediction task, showcasing exceptionally effective performance.

3 Preliminary

In this section, we begin by analyzing the relationship between various heuristics employed in link prediction, aiming to uncover the complex patterns that exist within the same dataset. Subsequently, we explore the relationship between the performance of different GNN4LP models and these heuristics. This exploration aims to identify the most suitable scenarios for each model, thereby enhancing our understanding of how different approaches can be optimally applied in varying link prediction contexts. Before that, we first introduce key notations and experimental settings.

Notations and experimental settings. Let G = ( V , E ) be a graph with n nodes, where V is the node set and E is the edge set. N i denotes the neighborhood node set for node v i . The graph can be denoted as an adjacency matrix A ∈ R n × n , and each node v i may be associated with a d -dimensional feature x i and we use X = [ x 1 , . . . , x n ] ⊤ ∈ R n × d to denote the node feature matrix. We conduct analysis on the Cora, Citeseer, Pubmed, ogbl-collab, and ogbl-ppa datasets. We use Hits@K as a metric to measure the ratio of positive samples ranked among the top K against a set of negative samples. The details on each dataset and the evaluation setting can be found in Appendix A. Due to the limited space, we only illustrate partial results in the following subsections. More results can be found in Appendix B.

3.1 Exploring Heuristics in Link Prediction

In this subsection, we focus on three vital types of pairwise factors used in link prediction identified by Mao et al. [17]: (a) local structure proximity, (b) global structure proximity, and (c) feature

proximity. For each type, we adopt a single widely used heuristic as a representative metric including Common Neighbors (CN) [10] for local structure proximity, Shortest Path (SP) [43] for global structural proximity, and Feature Cosine Similarity (FCS) for feature proximity. We initially evaluate the performance of each heuristic individually and then assess their combinations by simply adding their normalized values. Specifically, we normalize each heuristic value ( h ) to the range of [0, 1] using h -h max h max -h min , where h max and h min are the maximum and minimum heuristic values in the dataset. One exception is the calculation of SP, where a smaller SP indicates a higher likelihood that two nodes are connected. Therefore, we first calculate 1 SP , and then normalize it in the same way as other heuristics. For this evaluation, we employ Hits@3 as the metric for smaller datasets and Hits@20 for larger OGB datasets. The results for Citeseer and ogbl-collab are illustrated in Figure 1, with diagonal values representing the individual performance of each heuristic. We can have two observations: (1) combining different heuristics generally enhances overall performance, indicating that reliance on a single heuristic may be inadequate for accurate link prediction; and (2) the performance of each heuristic varies across datasets. For instance, in the Citeseer dataset, FCS and CN exhibit comparable performance. However, in the ogbl-collab dataset, CN significantly outperforms FCS.

We further investigate the overlap in correctly predicted sample pairs by each heuristic. We use the Jaccard Coefficient to calculate the overlapping ratio between each pair of heuristics. For the calculation of the Jaccard coefficient, we use the Hits@K metric for each edge. Specifically, we choose Hits@3 for small datasets and Hits@20 for the OGB datasets. We first rank the prediction scores of each method for both positive and negative edges. If the prediction score of a positive edge is in the Top-K, we label this positive edge as 'present' and add it to the correct prediction set. In this way, we can calculate the Jaccard coefficient by comparing the correct prediction sets for each pair of methods. The results for the Citeseer and ogbl-collab datasets are presented in Figure 2. We observe that the overlapping ratio between certain heuristics is notably low. This implies that the sets of node pairs correctly predicted by different heuristics have a minimal intersection. Therefore, different node pairs require distinct heuristics for accurate prediction even on the same dataset .

From the previous analysis, it is enticing to think that simply considering multiple heuristics should result in superior link prediction performance. We test this hypothesis by learning to classify links using multiple popular heuristic methods. For a single link, the individual heuristic scores are concatenated together and passed to an MLP, where the output is then used to classify the link. The full set of heuristic considered can be found in Section 5.1. The results on Citeseer and ogbl-collab can be found in Table 1. We report the MRR for Citeseer and Hits@50

for ogbl-collab. We find that ensembling multiple heuristics can modestly improve the performance. However, it still noticeably lags behind GNN4LP methods in performance. Based on this observation, we are motivated to investigate whether different GNN4LP models can be used to model a wider variety of links.

3.2 Exploring GNN4LP Models and Heuristics

In this subsection, we move beyond analyzing the performance of only heuristic measures and further consider the capabilities of different GNN4LP methods. To better understand the abilities of different GNN4LP models, we first evaluate the overlapping ratio between different models using the Jaccard Coefficient. We also include the MLP and different heuristics in the analysis. The overlapping ratio of different methods on ogbl-collab dataset is shown in Figure 3. These results reveal that the overlapping ratios among different GNN4LP models are rel-

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

1

0.47

0.34

0.39

0.39

0.39

0.39

0.53

0.39

0.39

0.39

0.38

0.4

0.47

1

0.52

0.64

0.64

0.65

0.62

0.35

0.64

0.63

0.64

0.64

0.62

0.34

0.52

1

0.68

0.58

0.65

0.64

0.24

0.65

0.58

0.7

0.66

0.63

0.39

0.64

0.68

1

0.72

0.83

0.8

0.27

0.81

0.73

0.84

0.85

0.82

0.39

0.64

0.58

0.72

1

0.79

0.69

0.29

0.79

0.75

0.76

0.79

0.67

0.39

0.65

0.65

0.83

0.79

1

0.79

0.28

0.85

0.78

0.84

0.92

0.8

0.39

0.62

0.64

0.8

0.69

0.79

1

0.28

0.8

0.74

0.82

0.83

0.77

0.53

0.35

0.24

0.27

0.29

0.28

0.28

1

0.28

0.29

0.28

0.27

0.29

0.39

0.64

0.65

0.81

0.79

0.85

0.8

0.28

1

0.89

0.91

0.87

0.74

0.39

0.63

0.58

0.73

0.75

0.78

0.74

0.29

0.89

1

0.82

0.79

0.68

0.39

0.64

0.7

0.84

0.76

0.84

0.82

0.28

0.91

0.82

1

0.86

0.76

0.38

0.64

0.66

0.85

0.79

0.92

0.83

0.27

0.87

0.79

0.86

1

0.8

0.4

0.62

0.63

0.82

0.67

0.8

0.77

0.29

0.74

0.68

0.76

0.8

1

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

atively low, suggesting that each model is capable of predicting a unique set of links. Furthermore, different GNN4LP models have varying degrees of overlap with different heuristics. These observations lead us to an intriguing question: Are the unique sets of links correctly predicted by different GNN4LP models related to specific heuristics?

To answer this question, we first categorize node pairs into 5 groups based on each heuristic and evaluate the performance of different GNN4LP models within these groups. Additionally, we also include the MLP and GCN in our analysis. The performance of these models across different Common Neighbors (CN) groups for Cora and ogbl-collab is depicted in Figure 4. The x-axis represents different groups, along with the proportion of node pairs in each group. From the results, we can find that no single model consistently outperforms others across all groups on either dataset. Interestingly, when there are no common neighbors, MLP and GCN tend to excel in both the Cora and ogbl-collab datasets. In situations with a few common neighbors, SEAL shows better performance in the Cora dataset, while BUDDY tends to lead in the ogbl-collab dataset. With an increase in the number of common neighbors, methods that encode CN information, such as NCNC, generally exhibit strong performance. A similar phenomena can be found on other datasets and heuristics in Appendix B.

(b) ogbl-collab

1.0

mlp

1.0

mlp

0.8

0.8

neognn

neognn

H

m

0.6

buddy

Hit@2

0.6

buddy

seal

seal

0.4

0.4

0.2

0.2

0.0

0 - 0 (57.31%)

1 - 2 (37.76%)

3 - 4 (3.42%)

5 - 6 (0.76%)

7 - 9 (0.57%)

0.0

0 -0 (34.40%)

1 - 24 (42.49%)

25 - 48 (8.92%)

49 - 72 (4.15%) 73 - 3785 (10.04%)

In conclusion, our analysis underscores that accurate link prediction necessitates the use of multiple heuristics. Different node pairs require distinct heuristics for optimal prediction. Furthermore, the overlapping ratio of different GNN4LP models is relatively low. The performance of powerful GNN4LP models is closely tied to the specific heuristics they encode. And there is no single model that can uniformly achieve the best performance across all scenarios. These findings pave us a way to adaptively select the most suitable model for each node pair to achieve better overall performance in link prediction task.

4 Method

The investigations conducted in Section 3 reveal that various heuristics complement each other for the task on link prediction. This implies that different node pairs may need different heuristics to properly predict the existence of a link. Therefore, one approach to potentially achieve better performance in link prediction is to integrate all these heuristics into a single, unified model. However, as shown in Table 1, this strategy fails to outperform existing GNN4LP methods. This suggests that these GNN4LP models are already quite effective. Our findings further suggest that different GNN4LP models demonstrate unique strengths in different scenarios, which are related to specific heuristics. Therefore, leveraging these diverse heuristics as a guidance could help identify the most suitable GNN4LP models for specific node pairs. Based on the above intuitions, we aim to design a framework that can harness the unique strengths of each model to enhance the performance of link prediction.

4.1 Link-MoE - A General Framework

To leverage the strengths of various existing models, we introduce a novel mixture-of-experts (MoE) method tailored for link prediction, which we term Link-MoE . An overview of Link-MoE is depicted in Figure 5(a). There are two major components: the gating model and the multiple expert models. The gating function can be implemented using any neural network and each expert can be any method used for link prediction. When predicting whether a node pair ( i, j ) are linked, the gating function utilizes their heuristic information to produce normalized weights for each expert.

w1 =0.04

Feature

→w1*model1

Branch

IHeuristics

Model Weights

1

w2=0.36

w2*model2

MLP

W1

2

9()

M

Sigmoid

y

g()

MLP

w3=0.22

w3*model3

K

GatingModel

MLP

Wm

Structural

wm=0.1

Branch

(a)Link-MoE

(b)GatingModel

These weights dictate the level of contribution each expert model has towards the final prediction. Each expert model processes the graph and node pair information, estimating the likelihood of a connection between the two nodes. The individual expert predictions are then aggregated according to the weights assigned by the gating function. The final prediction is made using the sum of the weighted scores, effectively leveraging the strengths of multiple experts to determine the probability of a link. Formally, the prediction of Link-MoE can be expressed as:

where m denotes the number of expert models incorporated, pairwise node features x ij and structural heuristics s ij serves as the input to the gating function, G ( · ) represents the gating model function, E o refers to the o -th expert model, and σ is a sigmoid activation function. This configuration makes the Link-MoE remarkably flexible and easily adaptable, allowing for the seamless integration of different expert models as required. We will detail our implementation in the following subsections.

4.2 The Design of the Gating Model

As highlighted in our preliminary studies (Section 3), it's evident that different GNN4LP models excel in varying contexts and that their strengths can be indicated by different heuristics. Therefore, we incorporate a broad spectrum of heuristics as inputs to the gating model to leverage these strengths. Specifically we consider CN [10], AA [18] and RA [19] to model the local structural proximity and Shortest Path, Katz index [11], and PPR [44] for the global structural proximity. For the feature proximity, to ensure the score is invariant to the ordering of the node pair, we utilize the element-wise product for deriving the feature heuristic of node pair ( i, j ) . This is defined as x ij = x i ⊙ x j . We also use s ij = [ s 1 ij , s 2 ij , ..., s k ij ] to represent all the k structural heuristics, and the dimension of each structural heuristics is typically very small. However, the node pair feature heuristic is equal to the input node feature dimension and can span hundreds or even thousands of dimensions. This significant disparity in dimensionality could hinder the model's ability to effectively learn from the structural heuristics. To address this challenge, we design a two-branch gating model, as illustrated in Figure 5(b). Each branch is a simple MLP. One branch is dedicated to encoding the structural heuristics, while the other focuses on processing the node pair features. After that, these two branches are merged via concatenation. Finally, an MLP with the softmax function is applied to this combined output to generate the final weight predictions. This design ensures a balanced consideration of both structural and feature-based heuristics. Formally, the gating function is defined as follows:

where f is a MLP and || denotes the concatenation operator.

4.3 Optimization of Link-MoE

Given the experts chosen in Section 5.1, there are several training strategies used by MoE models to combine them. This includes end-to-end training [42] and the EM algorithm [45]. Given the multiple options, we are then tasked with the question: how do we optimize Link-MoE?

In this work, we employ a two-step training strategy, as detailed in Algorithm 1 in Appendix E. Initially, we train each expert individually using their respective optimal hyperparameters and perform inference on the dataset to obtain the prediction scores for each link. Subsequently, we focus on the training of the gating model. Specifically, we adopt the cross entropy to train the gating function:

where y ij = 1 when a link exists between node v i and v j in the graph, and y ij = 0 otherwise. P and N denote the set of positive/negative links in the graph, respectively.

There are several benefits to our two-step training strategy. (1) Efficiency : This approach eliminates the need to load every expert into memory simultaneously, as each expert is trained and infers independently; For time efficiency, tuning the gating model to identify the best hyperparameters is more efficient since it involves training only MLPs. (2) Effectiveness : The two-step training strategy helps to avoid the 'collapse problem' often encountered in MoE models [37]. This issue arises when only a single expert is consistently selected, leading to the under-utilization and inadequate learning of the other experts. By training the experts individually first, we mitigate this risk, ensuring a more balanced and effective utilization of all experts. We compare two-step and end-to-end training strategies in Appendix G. (3) Flexibility : When introducing new experts into the Link-MoE, it's only necessary to train these new experts and the gating model. All previously trained experts can be seamlessly integrated without the need for retraining.

5 Experiments

In this section, we conduct comprehensive experiments to validate the effectiveness of the proposed Link-MoE. Specifically, we aim to address the following research questions: RQ1: How does Link-MoE perform when compared to other baseline models? RQ2: Are the heuristics effective in aiding the selection of experts? RQ3: Can Link-MoE adaptively select suitable experts for different node pairs?

5.1 Experimental Settings

Datasets . We evaluate our proposed method on eight datasets including homophilous graphs: Cora, Citeseer, Pubmed [46], ogbl-ppa, ogbl-collab, and ogbl-citation2 [47] and heterophilic graphs: Chameleon and Squirrel [48]. Please see Appendix A for more details on each dataset.

Baselines . We consider a diverse set of baselines include heuristics, embedding methods, GNNs, and GNN4LP methods. This includes: CNs [10], AA [18], RA [19], Shortest Path [43], Katz [11], Node2Vec [49], Matrix Factorization (MF) [50], MLP, GCN [23], GAT [51], SAGE [24], GAE [25], SEAL [15], BUDDY [26], Neo-GNN [14], NCN and NCNC [13], NBFNet [16], PEG [52], LPFormer [27].

Additionally, to comprehensively evaluate the proposed Link-MoE, which integrates multiple link predictors, we design two ensemble baseline methods for comparison. The first method, MeanEnsemble , combines all expert models with uniform weight, ensuring each expert contributes equally to the final prediction. The second method, Global-Ensemble , learns a global weight for each expert that is applied when predicting all node pairs. In this approach, each expert contributes to the final prediction based on the learned global weight for all node pairs, allowing for a differentiated influence of each expert based on their performance. See Appendix A.2 for more details on these two methods. Additionally, we compare our method with two other ensemble methods [30, 31], with the results provided in Appendix H.

Link-MoE Settings . In this study, we incorporate both node features and a variety of different heuristics as input features for the gating model. These heuristics include node degree, CN, AA, RA, Shortest Path, Katz, and Personalized PageRank (PPR) [20]. Furthermore, our approach uses a wide range of experts, including NCN, NCNC, Neo-GNN, BUDDY, MLP, Node2Vec, SEAL, GCN, NBFNet, and PEG. NBFNet and PEG are only used for smaller datasets, as they often run into out-of-memory issues on the larger OGB datasets. For the two baselines, Mean-Ensemble and GlobalEnsemble, we use the same experts as with Link-MoE. More setting details are in Appendix A.2. For evaluation, we report several ranking metrics including the Hits@K and Mean Reciprocal Rank (MRR). In the main paper, we report the MRR for Cora, Citeseer, and Pubmed and for OGB we

use the evaluation metric used in the original study [47]. Results for other metrics are shown in Appendix C.

5.2 Main Results

We present the main results of link prediction for the small datasets and OGB datasets in Table 2. Reported results are mean and standard deviation over 10 seeds. We use 'Improv.' to denote the relative improvement of Link-MoE over the second best model. From the table, we can have the following observations:

Link-MoE consistently outperforms all the baselines by a significant margin. For example, it achieves a relative improvement of 18.71% on the MRR metric for the Pubmed dataset and 9.59% on the Hit@100 metric for the ogbl-ppa dataset, compared to the best-performing baseline methods.

While both the Mean-Ensemble and Global-Ensemble methods also incorporate all the experts used in Link-MoE, their performance is generally subpar in most cases. Although the Global-Ensemble, which learns different weights for each expert, usually outperforms the MeanEnsemble, it still falls short of the performance of single baseline methods in some scenarios. We attribute this to their inability to adaptively apply different experts to specific node pairs, which demonstrates the effectiveness of the gating model in Link-MoE.

We further compare against LPFormer [27], a recent method that attempts to adaptively customize pairwise information to each node pair, resulting in strong performance. We find that our model is able to considerably outperform LPFormer on all datasets. From this we conclude that Link-MoE is better than LPFormer at customizing the pairwise information to each node pair.

Additionally, instead of using all experts, we conducted experiments with only a few experts (i.e., 3 or 4 experts). Furthermore, we also explored a sparse gating strategy [37], which selectively activates only the Top-K experts for each sample's prediction. The results, presented in Appendix D, demonstrate that these two variants can achieve comparable performance with using all the experts.

We further evaluate the proposed Link-MoE on heterophilic graphs, with results in Appendix F indicating that Link-MoE achieves strong performance. Additionally, we assess Link-MoE in the more challenging HeaRT setting [53], with results in Appendix J further validating its effectiveness.

Hits@3

Hits@10

40

45

50

55

60

65

70

75

80

All

OnlyFeat

OnlyStruct

OnlyLocalStruct

OnlyGlobalSturct

(a) Pubmed

Hits@50

Hits@100

40

45

50

55

60

65

70

75

80

All

OnlyFeat

OnlyStruct

OnlyLocalStruct

OnlyGlobalSturct

(b) ogbl-ppa

All

-MLP

-GCN

-NCNC -NeoGNN-BUDDY

-SEAL-Node2Vec

40

50

60

70

80

Hits@50

ogbl-collab

ogbl-ppa

(a) ogbl-collab

(b) ogbl-ppa

1.0

mlp

1.0

mlp

u36

u36

0.8

ncn

ncnc

0.8

ncn

ncnc

neognn

neognn

Appnq

Appnq

芸

Weigh

0.6

AZu

seal

Weigh

0.6

AZu

seal

0.4

0.4

0.2

0.2

0.0

0-0

1 - 24

25 - 48

49 - 72

73-3786

0.0

0-0

1 - 41

42 - 82

83 -124

125 -2241

5.3 The Effectiveness of Different Heuristics in Gating

The addition of the gating model enables Link-MoE to significantly surpass both the Mean-Ensemble and Global-Ensemble methods. This improvement underscores the pivotal role of the gating model, which intelligently leverages heuristic information to enhance the overall performance. In this subsection, we delve into the impact of various heuristics utilized by the gating model in Link-MoE. Specifically, there are three types of heuristics: local structure proximity (e.g., CN, AA, RA), global structure proximity (e.g., Shortest Path, Katz), and feature proximity. We design experiments to isolate the effect of specific types of heuristics by including them either individually or in groups. For instance, utilizing only feature proximity is referred to as 'OnlyFeat', combining local and global structure proximities is labeled 'OnlyStruct', employing only local structure proximity is denoted 'OnlyLocalStruct', and using only global structure proximity is marked 'OnlyGlobalStruct'. Additionally, the 'All' is used to indicate leveraging all heuristics. The results on Pubmed and ogbl-ppa datasets are shown in Figure 6. From the experimental results, several observations can be made: (1) Using only the feature proximity (OnlyFeat) or only the global structural proximity (OnlyGlobalStruct) typically yields lower performance. (2) Combining the local and global structure proximity (OnlyStruct) tends to outperform using either one alone. (3) The effectiveness of feature proximity varies between datasets. On Pubmed, it is beneficial to include it but not on ogbl-ppa. This discrepancy could stem from the relative importance of feature information in each dataset, as suggested by the very poor performance of MLP on ogbl-ppa, whose Hits@100 is only around 0.45. Moreover, we explore different inputs for the gating model, the results are presented in Appendix I, which highlights the rationality of the design of our gating model.

5.4 The Importance of Different Experts

In the previous sections, we incorporated all the selected experts in Link-MoE. This subsection aims to explore the impact of each expert's removal on the framework's performance. We conducted experiments on the ogbl-collab and ogbl-ppa datasets, utilizing Hits@50 as the metric for evaluation. The results are presented in Figure 7, where the x-axis indicates the experts removed for each trial. We observe that the removal of individual experts does not significantly impact the performance of Link-MoE on both the ogbl-collab and ogbl-ppa datasets. This phenomenon suggest that Link-MoE possesses an adaptive capability to compensate for the absence of certain models by effectively utilizing other available experts.

5.5 Analysis of the Gating Weights

In this subsection, we explore the mechanism by which the gating model allocates weights to experts based on the heuristics of different node pairs. Similar to Section 3, we categorize the test node pairs into distinct groups according to different heuristics. Subsequently, for each group, we compute the average weights assigned to each expert by the gating model. The results based on Common Neighbors for ogbl-collab and ogbl-ppa are shown in Figure 8. We can have some interesting findings: (1) : For the ogbl-collab dataset, when the node pair doesn't have common neighbor, the gating model usually assigns a large weight to the MLP model, which aligns the analysis in Section 3 that the MLP can perform well when there is no CN on ogbl-collab. However, For the ogbl-ppa dataset, when there is no common neighbor, the gating model would assign a high weight to NCNC. This preference arises because node features hold less significance in the ogbl-ppa dataset and NCNC can leverage multi-hop common neighbor information. (2) : As the number of common neighbors increases, the gating model increasingly allocates more weight to the NCN. This shift reflects the NCN's capability to efficiently capture and utilize common neighbor information. (3) : Not every expert model contributes much to the prediction. This selective engagement is attributed to the overlapping capabilities of certain models. For instance, both Neo-GNN and NCNC are capable of exploiting multi-hop information. In such cases, the gating model opts for one over the other to avoid redundancy and optimize prediction efficacy. This phenomenon is also consistent with the results in Section 5.4 that removing one expert doesn't affect the overall performance. The analysis of gating weights on heterophilic datasets can be found in Appendix F.

While Link-MoE can successfully leverage heuristics to select appropriate experts for different node pairs, we recognize substantial room for further enhancements. For instance, the observed overlapping ratio between Neo-GNN and NCNC on the ogbl-collab dataset is not very high, as shown in Figure 3, even if they exploit similar heuristics. But in Link-MoE, Neo-GNN has very low weights on the ogbl-collab dataset. This observation underscores the vast potential for advancing MoE applications in link prediction, a direction we intend to explore in future work.

6 Conclusion

In this study, we explored various heuristics and GNN4LP models for link prediction. Based on the analysis, a novel MoE model Link-MoE is designed to capitalize on the strengths of diverse expert models for link prediction. The extensive experiments underscore the exceptional performance of Link-MoE in link prediction, validating the rationale behind its design. Furthermore, we also showcase the substantial potential of MoE models for link prediction.

References

Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based recommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 346-353, 2019.

Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi. A survey of recommendation systems: recommendation models, techniques, and application fields. Electronics, 11(1):141, 2022.

Jurgen Drews. Drug discovery: a historical perspective. science, 287(5460):1960-1964, 2000.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014.

Seiji Maekawa, Koki Noda, Yuya Sasaki, et al. Beyond real-world benchmark datasets: An empirical study of node classification with gnns. Advances in Neural Information Processing Systems, 35:5562-5574, 2022.

Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34:9061-9073, 2021.

Balasubramaniam Srinivasan and Bruno Ribeiro. On the equivalence between positional node embeddings and structural graph representations. In International Conference on Learning Representations, 2019.

Ajay Kumar, Shashank Sheshar Singh, Kuldeep Singh, and Bhaskar Biswas. Link prediction techniques, applications, and performance: A survey. Physica A: Statistical Mechanics and its Applications, 553:124289, 2020.

Mark EJ Newman. Clustering and preferential attachment in growing networks. Physical review E, 64(2):025102, 2001.

Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18(1):39-43, 1953.

Yohsuke Murase, Hang Hyun Jo, János Török, János Kertész, Kimmo Kaski, et al. Structural transition in social networks. 2019.

Xiyuan Wang, Haotong Yang, and Muhan Zhang. Neural common neighbor with completion for link prediction. arXiv preprint arXiv:2302.00890, 2023.

Seongjun Yun, Seoyoon Kim, Junhyun Lee, Jaewoo Kang, and Hyunwoo J Kim. Neo-gnns: Neighborhood overlap-aware graph neural networks for link prediction. Advances in Neural Information Processing Systems, 34:13683-13694, 2021.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems, 34:29476-29490, 2021.

Haitao Mao, Juanhui Li, Harry Shomer, Bingheng Li, Wenqi Fan, Yao Ma, Tong Zhao, Neil Shah, and Jiliang Tang. Revisiting link prediction: A data perspective. arXiv preprint arXiv:2310.00793, 2023.

Lada A Adamic and Eytan Adar. Friends and neighbors on the web. Social networks, 25(3):211230, 2003.

Tao Zhou, Linyuan Lü, and Yi-Cheng Zhang. Predicting missing links via local information. The European Physical Journal B, 71:623-630, 2009.

Hanzhi Wang, Zhewei Wei, Junhao Gan, Sibo Wang, and Zengfeng Huang. Personalized pagerank to a target node, revisited. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 657-667, 2020.

Jiliang Tang, Huiji Gao, Xia Hu, and Huan Liu. Exploiting homophily effect for trust prediction. In Proceedings of the sixth ACM international conference on Web search and data mining, pages 53-62, 2013.

He Zhao, Lan Du, and Wray Buntine. Leveraging node attributes for incomplete relational data. In International conference on machine learning, pages 4072-4081. PMLR, 2017.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.

N Kipf Thomas and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2(10), 2016.

Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M Bronstein, and Max Hansmire. Graph neural networks for link prediction with subgraph sketching. arXiv preprint arXiv:2209.15486, 2022.

Harry Shomer, Yao Ma, Haitao Mao, Juanhui Li, Bo Wu, and Jiliang Tang. Lpformer: An adaptive graph transformer for link prediction. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2686-2698, 2024.

Liang Duan, Shuai Ma, Charu Aggarwal, Tiejun Ma, and Jinpeng Huai. An ensemble approach to link prediction. IEEE Transactions on Knowledge and Data Engineering, 29(11):2402-2416, 2017.

Yi-Ling Chen, Ming-Syan Chen, and S Yu Philip. Ensemble of diverse sparsifications for link prediction in large-scale networks. In 2015 IEEE International Conference on Data Mining, pages 51-60. IEEE, 2015.

Amir Ghasemian, Homa Hosseinmardi, Aram Galstyan, Edoardo M Airoldi, and Aaron Clauset. Stacking models for nearly optimal link prediction in complex networks. Proceedings of the National Academy of Sciences, 117(38):23393-23400, 2020.

Yen-Liang Chen, Chen-Hsin Hsiao, and Chia-Chi Wu. An ensemble model for link prediction based on graph embedding. Decision Support Systems, 157:113753, 2022.

Kuanyang Li, Lilan Tu, and Lang Chai. Ensemble-model-based link prediction of complex networks. Computer Networks, 166:106978, 2020.

Shruti Pachaury, Nilesh Kumar, Ayush Khanduri, and Himanshu Mittal. Link prediction method using topological features and ensemble model. In 2018 Eleventh International Conference on Contemporary Computing (IC3), pages 1-6. IEEE, 2018.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.

Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275-293, 2014.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017.

Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.

Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208.02813, 2022.

Haotao Wang, Ziyu Jiang, Yan Han, and Zhangyang Wang. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling. arXiv preprint arXiv:2304.02806, 2023.

Shirley Wu, Kaidi Cao, Bruno Ribeiro, James Zou, and Jure Leskovec. Graphmetro: Mitigating complex distribution shifts in gnns via mixture of aligned experts. arXiv preprint arXiv:2312.04693, 2023.

David Liben-Nowell and Jon Kleinberg. The link prediction problem for social networks. In Proceedings of the twelfth international conference on Information and knowledge management, pages 556-559, 2003.

Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. Computer networks and ISDN systems, 30(1-7):107-117, 1998.

Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural networks, 12(9):1229-1252, 1999.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR, 2016.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2):cnab014, 2021.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016.

Aditya Krishna Menon and Charles Elkan. Link prediction via matrix factorization. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part II 22, pages 437-452. Springer, 2011.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.

Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. arXiv preprint arXiv:2203.00199, 2022.

Juanhui Li, Harry Shomer, Haitao Mao, Shenglai Zeng, Yao Ma, Neil Shah, Jiliang Tang, and Dawei Yin. Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking. Advances in Neural Information Processing Systems, 36, 2023.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

Shijie Zhou, Zhimeng Guo, Charu Aggarwal, Xiang Zhang, and Suhang Wang. Link prediction on heterophilic graphs via disentangled representation learning. arXiv preprint arXiv:2208.01820, 2022.

A Datasets and Experimental Settings

A.1 Datasets

The statistics for each dataset is shown in Table 3. We adopt the single fixed train/validation/test split with percentages 85/5/10% for Cora, Citeseer, and Pubmed as used in [53]. For the OGB datasets, we use the fixed splits provided by the OGB benchmark [54]. Note that we omit ogbl-ddi due to observations made by Li et al. [53] showing a weak correlation between validation and test performance. For heterophilic graphs Chameleon and Squirrel, we use the same split ratio as Cora.

A.2 Experimental Settings

Training Settings . We use the binary cross entropy loss to train each model. The loss is optimized using the Adam optimizer [55]. At first, we train all of the expert models by using the hyperparameters suggested in this repository * . We then do the inference to obtain the prediction score for each link. Secondly, in order to train Link-MoE, we split the original validation dataset into a new training set and validation set. Thirdly, we train the gating model until it converges and choose the model weights associated with the best validation performance. The rationale for utilizing a portion of the validation set to train the gating model is as follows: Given that each expert model is finely tuned on the training data, there exists a significant disparity between the prediction scores for positive and negative edges within this dataset. Should the original training set be employed to train our gating model, the outcome would be skewed-regardless of the gating model's outputs, predicted scores for positive pairs would invariably remain substantially higher than those for negative pairs. Therefore, to train the gating model effectively, we repurpose the original validation set as our training data, dividing it into new training and validation subsets. Unlike baseline models, which are trained exclusively on the original training set, our method benefits from incorporating a very small validation set compared to the training set into the training process for the gating model, yielding notable performance improvements. Despite this modification, the comparison remains relatively fair. The original validation set is relatively small, so using a portion of it for training does not significantly alter the amount of data available. Thus, the baseline models remain essentially equivalent, even with this adjustment. Moreover, it is common for validation set is to be used for the search of model architectures in neural architecture search. We train both experts and gating models on NVIDIA RTX A6000 GPU with 48GB memory.

Training Data Split . In the original datasets, we are given a fixed train, validation and test dataset. These splits are used to train each individual expert. Once each expert model is fully trained, we perform inference on the validation and test sets, thereby obtaining the prediction score for each link. As noted earlier, we split the original validation set into a new training and validation split for training and validating Link-MoE. Note that the original test dataset is still used solely for testing. When splitting the validation set, we use different ratios for different datasets. For ogbl-citation2, ogbl-ppa, ogbl-collab, Citeseer, Chameleon and Squirrel, the ratios are 0 . 8 , and for Cora and Pubmed, the ratios aer 0 . 9 .

Hyperparameter Settings . The hyperparameter ranges are shown in Table 4. Since our gating model has a low complexity, we can efficiently search over a large hyperparameter space on all eight datasets.

Evaluation Setting . The evaluation is conducted by ranking each positive sample against a set of negative samples. The same set of negatives are shared among all positive samples with the exception of ogbl-citation2, which customizes 1000 negatives to each positive sample. The set of negative samples are fixed and are taken from Li et al. [53] and Hu et al. [47] for their respective datasets.

Mean-Ensemble . After training the experts, we can obtain the prediction score on the test set from each expert. Then we take the mean across all experts to get a final score for link in the test set. The purpose of this model is to ascertain whether naively combining the different experts is itself enough for good performance. More formally, the Mean-Ensemble can be defined as follows:

Global-Ensemble . For the Global-Ensemble, we learn a weight vector w = [ w 1 , w 2 , ..., w m ] to combine the experts which can be defined as follows:

Notably, the w is uniform for all node pairs. Therefore, this method is not able to flexibility adjust the weight of different experts to each node pair. Rather, a single weight is used across all samples. The purpose of this method is to test whether it is necessary to customize the weight of the experts to each link as is done in Link-MoE.

B Additional Results for the Preliminary Study

In this section, we extend the preliminary study (Section 3) with additional results, including model and heuristic overlaps on Cora, Citeseer, Pubmed, and ogbl-ppa datasets. We also analyze the performance of various models across node pair groups categorized by Commen Neighbors (CN), Shortest Path (SP), and Feature Cosine Similarity (FCS).

B.1 Model Overlapping Results

Following Section 3, for the Cora, CiteSeer, PubMed, we use Hits@3. And we use Hits@20 as the metric for ogbl-ppa dataset. The results are shown in Figure 9 and Figure 10. From the results, we can have the following findings:

For most datasets, the overlapping between different GNN4LP models is not very high, which consistent with the finding in Section 3.

The feature cosine similarity usually has much smaller overlapping with other methods.

B.2 The Performance of Different Models on Different Heuristic Groups

In this section, we showcase the varied performances of GNN4LP models across groups categorized by shortest path and feature cosine similarity within the Cora and ogbl-collab datasets. The results for these heuristics are depicted in Figure 11 for shortest path groups and in Figure 12 for cosine similarity groups. We have the following observations:

For the Cora dataset, SEAL excels over other models at shorter path lengths, while MLP shows superior performance for longer paths. In the ogbl-collab dataset, BUDDY is effective at shorter distances, whereas Neo-GNN, which integrates multi-hop common neighbor information, performs better as the shortest path (SP) length increases.

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

1

0.4

0.39

0.38

0.37

0.39

0.38

0.29

0.38

0.21

0.38

0.33

0

0.4

1

0.55

0.51

0.49

0.51

0.53

0.2

0.56

0.28

0.56

0.43

0

0.39

0.55

1

0.57

0.54

0.52

0.59

0.18

0.65

0.29

0.65

0.48

0

0.38

0.51

0.57

1

0.48

0.47

0.47

0.18

0.47

0.27

0.47

0.39

0

0.37

0.49

0.54

0.48

1

0.45

0.53

0.17

0.55

0.31

0.55

0.43

0

0.39

0.51

0.52

0.47

0.45

1

0.5

0.21

0.56

0.33

0.56

0.48

0

0.38

0.53

0.59

0.47

0.53

0.5

1

0.19

0.73

0.29

0.73

0.5

0

0.29

0.2

0.18

0.18

0.17

0.21

0.19

1

0.19

0.11

0.19

0.16

0

0.38

0.56

0.65

0.47

0.55

0.56

0.73

0.19

1

0.34

1

0.62

0

0.21

0.28

0.29

0.27

0.31

0.33

0.29

0.11

0.34

1

0.34

0.47

0

0.38

0.56

0.65

0.47

0.55

0.56

0.73

0.19

1

0.34

1

0.62

0

0.33

0.43

0.48

0.39

0.43

0.48

0.5

0.16

0.62

0.47

0.62

1

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0.0

0.2

0.4

0.6

0.8

1.0

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

1

0.72

0.73

0.73

0.68

0.72

0.56

0.36

0.48

0.48

0.48

0.65

0.66

0.72

1

0.82

0.81

0.78

0.79

0.64

0.33

0.54

0.54

0.54

0.74

0.75

0.73

0.82

1

0.86

0.77

0.84

0.61

0.36

0.52

0.52

0.52

0.74

0.73

0.73

0.81

0.86

1

0.74

0.86

0.58

0.37

0.48

0.48

0.48

0.7

0.7

0.68

0.78

0.77

0.74

1

0.74

0.66

0.3

0.57

0.57

0.57

0.77

0.78

0.72

0.79

0.84

0.86

0.74

1

0.59

0.35

0.49

0.49

0.49

0.71

0.7

0.56

0.64

0.61

0.58

0.66

0.59

1

0.24

0.6

0.6

0.6

0.78

0.79

0.36

0.33

0.36

0.37

0.3

0.35

0.24

1

0.23

0.23

0.23

0.24

0.25

0.48

0.54

0.52

0.48

0.57

0.49

0.6

0.23

1

1

1

0.64

0.66

0.48

0.54

0.52

0.48

0.57

0.49

0.6

0.23

1

1

1

0.64

0.66

0.48

0.54

0.52

0.48

0.57

0.49

0.6

0.23

1

1

1

0.64

0.66

0.65

0.74

0.74

0.7

0.77

0.71

0.78

0.24

0.64

0.64

0.64

1

0.97

0.66

0.75

0.73

0.7

0.78

0.7

0.79

0.25

0.66

0.66

0.66

0.97

1

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

(b) Citeseer

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

1

0.24

0.28

0.3

0.3

0.25

0.28

0.097

0.26

0.27

0.24

0.28

0

0.24

1

0.39

0.43

0.29

0.33

0.24

0.1

0.23

0.21

0.23

0.23

0

0.28

0.39

1

0.58

0.48

0.37

0.49

0.089

0.38

0.32

0.36

0.51

0

0.3

0.43

0.58

1

0.45

0.38

0.42

0.098

0.3

0.28

0.28

0.37

0

0.3

0.29

0.48

0.45

1

0.33

0.54

0.085

0.33

0.3

0.31

0.44

0

0.25

0.33

0.37

0.38

0.33

1

0.32

0.1

0.28

0.26

0.28

0.34

0

0.28

0.24

0.49

0.42

0.54

0.32

1

0.071

0.35

0.29

0.34

0.52

0

0.097

0.1

0.089 0.098 0.085

0.1

0.071

1

0.08

0.09

0.075 0.078

0

0.26

0.23

0.38

0.3

0.33

0.28

0.35

0.08

1

0.81

0.82

0.54

0

0.27

0.21

0.32

0.28

0.3

0.26

0.29

0.09

0.81

1

0.65

0.51

0

0.24

0.23

0.36

0.28

0.31

0.28

0.34

0.075

0.82

0.65

1

0.5

0

0.28

0.23

0.51

0.37

0.44

0.34

0.52

0.078

0.54

0.51

0.5

1

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0.0

0.2

0.4

0.6

0.8

1.0

(a) Pubmed

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

mlp

gcn

ncn

ncnc

neognn

buddy

seal

FCS

AA

CN

RA

SP

1

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0.17

0.16

0.25

0.25

0.22

0

0.26

0.26

0.26

0.26

0

0

0.17

1

0.64

0.41

0.45

0.35

0

0.23

0.21

0.44

0.21

0

0

0.16

0.64

1

0.35

0.4

0.31

0

0.18

0.17

0.38

0.17

0

0

0.25

0.41

0.35

1

0.68

0.44

0

0.42

0.36

0.74

0.36

0

0

0.25

0.45

0.4

0.68

1

0.39

0

0.39

0.34

0.81

0.34

0

0

0.22

0.35

0.31

0.44

0.39

1

0

0.42

0.37

0.43

0.37

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0.26

0.23

0.18

0.42

0.39

0.42

0

1

0.88

0.44

0.88

0

0

0.26

0.21

0.17

0.36

0.34

0.37

0

0.88

1

0.39

1

0

0

0.26

0.44

0.38

0.74

0.81

0.43

0

0.44

0.39

1

0.39

0

0

0.26

0.21

0.17

0.36

0.34

0.37

0

0.88

1

0.39

1

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0.0

0.2

0.4

0.6

0.8

1.0

(b) ogbl-ppa

For the Cora dataset, the seal excels over other models at smaller feature cosine similarity, while MLP outperforms when the feature cosine similarity is high. For the ogbl-collab dataset, NCNC is effective at low feature cosine similarity, but BUDDY can works well when the feature similarity is high.

Different datasets exhibit distinct patterns, even within the same heuristic groups. There is no single model consistently outperforms others across all groups.

These observations further validate the rationale behind employing heuristics as inputs to the gating model, effectively leveraging the strengths of various GNN4LP models.

C Additional Results on Benchmark Datasets

We present additional results of Cora, Citeseer, Pubmed, and OGB datasets in Table 5, 6, 7, and 8, respectively. We use' > 24h' to denote methods that require more than 24 hours for either training one epoch or evaluation. OOM indicates that the algorithm requires over 50Gb of GPU memory [53]. Note that LPFormer [27] is omitted from these tables as it doesn't report the results on these additional metrics. The data presented in these tables indicate that Link-MoE consistently achieves the best results in most cases, demonstrating its superior effectiveness in link prediction task. Although it does not always achieve the best performance across all metrics for the Cora and Citeseer datasets, it usually achieves the second or third best results. This slightly diminished performance can likely be

(b) ogbl-collab

mlp

1.0

mlp

0.7

gcn

0.8

ncn

0.6

ncnc

ncnc

H

m

0.5

人ppnq

neognn

0.6

neognn

ppnq

0.4

seal

seal

0.3

0.4

0.2

0.2

T'0

0°0

sp=3 (20.11%)

sp=4 (8.54%)

Sp=5 (4.74%)

sp>=6 (23.91%)

0.0

sp=1 (46.49%)

sp=3 (14.46%)

sp=4 (10.19%)

sp>=5 (8.53%)

(b) ogbl-collab

1.0

mlp

0.6

mlp

8'0

0.5

H

m

0.6

neognn

人ppnq

Hit@20

0.4

neognn

人ppnq

seal

0.3

seal

0.4

0.2

0.2

0.1

0.0

0.00 - 0.18 (60.53%) 0.18 - 0.36 (32.26%) 0.36 - 0.54 (4.74%)

0.54 - 0.72 (1.52%)

0.72 - 0.90 (0.76%)

0.0

0.40 - 0.52 (0.02%)

0.52 -0.64 (0.08%)0.64 - 0.76 (0.85%)0.76 - 0.88 (12.20%) 0.88 - 1.00 (86.84%)

attributed to the limited number of validation edges available in these datasets, i.e., 263 in Cora and 227 in Citeseer, restricting the amount of information Link-MoE can leverage for optimal learning and performance.

D Results for Top-K Experts and a few Experts

We conduct experiments on the ogbl-collab and Pubmed datasets, we limit the activation to the Top-3 experts. And we also conduct experiments using 3 or 4 experts. Specifically, for the ogbl-collab, we use MLP, NCNC, BUDDY (3 experts) and Neo-GNN (4 experts); for the Pubmed datasets, we use NCN, SEAL, NCN (3 experts) and MLP (4 experts). The performance metrics used to evaluate this approach are also Hits@50 for ogbl-collab and MRR for Pubmed. The result are shown in Table 9.

From the results, we can find that only 3 or 4 experts can achieve comparable performance with using all experts. Notably, these results don't use the computationally intensive SEAL for the ogbl-collab datasets. Furthermore, the inclusion of the less effective MLP expert in the Pubmed dataset still results in performance improvement, highlighting the complementary nature of the experts and the effectiveness of the proposed method. From the result, a sparse gating mechanism can also achieve promising performance on both datasets.

E Algorithm

The full algorithm is detailed in Algorithm 1. Notably, line 1-3 trains the experts individally. Line 4 performs the inference on links to obtain the prediction scores using the obtained experts and get the heuristic features for each link. Line 5-7 updates the gating model using Equation 3.

F Results for Heterophilic Datasets

We conduct experiments on two widely used heterophilic graphs, i.e., Chameleon and Squirrel. We follow the same setting with [56] and use the MRR for the evaluation metrics. The results are shown in Table 10. From the results, we can find the proposed Link-MoE outperforms the baselines by a large margin. These results demonstrate the proposed Link-MoE works well for both the homophilous and heterophilic graphs.

We also analyzed the weights generated by the gating mechanism on the two heterophilic graphs. The results are shown in Figure 13. We observed that feature proximity-based experts, such as MLP and GCN, are rarely employed for both datasets. This is consistent with the characteristics of heterophilic graphs, where connect nodes tend to have dissimilar features. These findings demonstrate the effectiveness of our gating design, as it accurately selects the appropriate experts for different types of graphs.

Algorithm 1 Two-step Training

input Input graph G , Node feature X , Expert models E = { E 1 , E 2 , ..., E m } , Positive set P and Negative set N

output Converged Link-MoE

1: for i = 1, 2, . . . , m do

2: Train each individual expert E i

3: end for

4: Get heuristics and prediction scores for positive and negative links

5: repeat

6: Update gating parameters by optimizing Eq. (3)

7: until Gating model converge

(a) Chameleon

0.8

mlp

1.0

mlp

0.6

0.7

gcn

ncn

0.8

gcn

ncn

Wei

0.5

0.4

0.3

ncnc

neognn

buddy

AZU

Weigh

0.6

0.4

n2v

neognn

buddy

0.2

0.2

0.1

0.0

0-0

1 - 25

26 -50

51 - 76

77 - 397

0.0

0-0

1 -186

187-372

373-558

559 -937

G End-to-end Results

In this section, we explore the end-to-end training of the experts and gating models, as shown in Table 11 and Figure 14. Our results show that the convergence speed of different experts varies significantly, which often leads to the model collapsing to a single expert. As a result, the performance of end-to-end training is not as good as the proposed Link-MoE. Despite this, it remains an interesting and challenging idea to explore the effective end-to-end training.

0

10

20

30

40

50

60

70

Epochs

0

10

20

30

40

50

60

70

Hits@50

Each expert's and overall result on collab (end2end)

MLP

GCN

NCN

NCNC

Overall

0

10

20

30

40

50

60

70

Epochs

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Gating Weights

Gating weights of each expert on collab (end2end)

MLP

GCN

NCN

NCNC

(a) Results of overall & each expert.

(b) The gating weights of each expert.

H Results for Additional Ensemble Methods

Wecompare our methods with two ensemble methods [30, 31] specifically designed for link prediction tasks. Specifically, [30] found that individual link prediction algorithms exhibit a broad diversity of prediction errors on different graphs and utilized a random forest to ensemble various link predictors; Similarly, [31] employs different graph embedding techniques to train multiple link predictors,

using the outputs of these predictors as inputs for training a subsequent link predictor. We conduct experiments on ogbl-collab and Pubmed dataset.The evaluation metrics are Hits@50 and MRR for ogbl-collab and Pubmed, respectively. The results, shown in Table 12, demonstrate that the proposed Link-MoE model significantly outperforms the ensemble-based methods by a large margin. This can be attributed to the dynamic nature of our gating module, which assigns customized weights to each expert for every node pair.

I Results for Different Gating Input

We compare our Link-MoE with different gating inputs on ogbl-collab and Pubmed datasets. The evaluation metrics are Hits@50 and MRR for ogbl-collab and Pubmed, respectively. Specifically, we design two different gating inputs.

The first one is traditional gating, which only leverages the node features in gating. The results are shown in Table 13 (Traditional Gating). Traditional Gating only results in comparable performance to the best single experts, while our approach yields superior performance. This phenomenon demonstrates the effectiveness and rationality of the designed gating model.

To investigate the impact of involving expert model predictions as input to the gating model, we conducted experiments on the ogbl-collab and Pubmed datasets by concatenating the prediction results of experts with the heuristic features. The results are shown in Table 13 (With Experts as Input). We observed that involving the experts' prediction results as additional input did not lead to improvement. In fact, this approach may result in lower performance compared to using heuristics alone. This phenomenon suggests that the outputs of the expert models may not effectively reflect their importance to specific node pairs, highlighting the effectiveness and rationality of using heuristics as the gating input.

J Results for HeaRT Setting

We conducted additional experiments under the HeaRT setting [53] with OGB datasets, and the results are shown in 14. In the HeaRT setting, hard negative samples are selected for positive samples based on specific heuristics, restricting the negatives to include one of the two nodes from the original positive pair. The results in Table 14 demonstrate that Link-MoE still significantly outperforms the best baseline models, even in this more challenging evaluation scenario.

K Limitations

In our current work, we explore various heuristics and GNN4LP models for link prediction and demonstrate the potential of MoE models in this context. The training of the gating model in

Link-MoE relies on pre-trained experts and heuristic features generated from the graphs. Although we can select a few experts, training these pre-trained experts remains time-consuming, especially for complex models. Additionally, since real-world graphs come from diverse domains and the graph generation process might be quite different, the heuristics used in this paper might not be comprehensive to other domains. Future work should focus on exploring more heuristics to better accommodate diverse domains.

L Impact Statement

In this paper, we explore the use of a mixture of experts (MoE) model for use in link prediction. We view the impact of this work as positive, as it can help improve performance of link prediction in many real-world applications including drug discovery and recommender systems. In general, we don't envision any specific negative societal consequences of our work. However, it is possible that the choice of experts used in our model may introduce some negative effects that stem from those individual models.

M Dataset Licenses

The license for each dataset can be found in Table 15.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 3 and 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is discussed in Section K.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See https://github.com/ml-ml/Link-MoE/ .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See https://github.com/ml-ml/Link-MoE/ .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See details in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There was no ethical concerns in either the research process or in potential impacts of our research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section L.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We introduce no new datasets in our paper, as all datasets used are already public. Furthermore, we don't envision any specific harm that can be caused by the framework introduced in our paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Table 15.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All data and models used in our paper are properly cited and attributed, including their license information (see Table 15. Also, we release the code for our model at https://github.com/ml-ml/Link-MoE/ .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an ensemble model of different link prediction methods. The authors find that different node pairs on the graph can form a link due to different pairwise representations, and there is no single link prediction model that can capture all of them. Then the author proposes to combine the outputs of different link prediction models into one ensemble model, weighted by the heuristics information of the node pair. Experiments on graph benchmarks show the improvement brought by ensembling.\n\nSTRENGTHS:\n1. The paper is clear and well-written.\n\n2. The motivation of the study is clearly stated. In the preliminary study section, the authors give solid examples of why individual link prediction models can fail, which motivates a combination of such models.\n\n3. The experiment is convincing and comprehensive, showing the competitive performance improvement with a mixture of link prediction models.\n\nWEAKNESSES:\n1. The novelty of the paper is limited. This study proposes a mixture of existing link prediction models, which brings minor research values to the community. The overall contribution of the paper makes it more like a technical report, rather than a research paper. While this paper has its own value, it is more suitable to be submitted to other venues.\n\n2. The baselines considered in the experiment are not comprehensive. For example, [1], which is an ensemble of link prediction models, is not evaluated as a baseline.\n\n3. The overall computational efficiency of the mixture method is concerning. On each graph, a set of link prediction models needs to be trained first, which typically involves an extensive process of hyperparameter tuning like in NCN [2]. This step can be too costly for most real-world use cases. The idea of MoE, especially conditional computation, is not leveraged at all to reduce the computational cost.\n\n4. The authors can further benchmark the Link-MoE on [3], which introduces more challenging link prediction tasks.\n\n[1] Ghasemian, A., Hosseinmardi, H., Galstyan, A., Airoldi, E. M., & Clauset, A. (2020). Stacking models for nearly optimal link prediction in complex networks. Proceedings of the National Academy of Sciences.\n\n[2] Wang, X., Yang, H., & Zhang, M. (2023). Neural common neighbor with completion for link prediction.\n\n[3] Li, J., Shomer, H., Mao, H., Zeng, S., Ma, Y., Shah, N., ... & Yin, D. (2024). Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking. Advances in Neural Information Processing Systems.\n\nQUESTIONS:\n1. For the gating model, only the heuristics information is considered as the input. However, this can limit the expressiveness of gating models, because the inherent limitation of the heuristics models themselves. Do authors consider involving the expert models as inputs to the gating models?\n\n2. Following up on the question above, is there any ablation study on training the gating model and expert models jointly? It may introduce higher memory consumption, but one can consider remove the computational heavy models like SEAL or NBFNet.\n\nLIMITATIONS:\nSee the weakness and question parts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a mixture of experts model, Link-MoE, for link prediction on graphs. The motivation behind the proposed approach is that different node pairs within the same dataset necessitate varied pairwise information for accurate prediction, while existing methods consider the same pairwise information uniformly across all pairs. To address this, Link-MoE consists of multiple expert models (existing link prediction methods) and a gating model whose goal is to produce importance scores to weight the different expert models according to their contribution towards the final prediction. Experiments on several real-world datasets show that the proposed Link-MoE outperforms existing competing methods, validating the rationale behind its design.\n\nSTRENGTHS:\nS1) The source code is provided with the submission\n\nS2) The paper is overall well-written\n\nS3) The proposed approach is compared against several competing methods and baselines.\n\nS3) Strong experimental results: the proposed method outperforms all the considered competing methods.\n\nS4) The proposed approach is well motivated\n\nS5) The proposed method was assessed on standard benchmark datasets for link prediction\n\nWEAKNESSES:\nW1) The figures are hardly readable sometimes (e.g., Fig. 8), the font size (in the legend) should be increased.\n\nW2) Notation problem: in Eq. 1 the gating function G has only one parameter while in Eq. 2 G takes both x_ij and s_ij. Please uniform it as this may be a little confusing for the reader.\n\nW3) The difference between the proposed Link-MoE and the considered baseline Global-Ensemble should be further discussed in the main paper given the strong similarity between the two approaches. As a suggestion, I recommend the authors to move lines 572-575 to the main paper since it describes the main differences between the two methods, i.e. the importance weight vector is uniform in Global-Ensemble while it is adaptable to the pair in Link-MoE.\n\nW4) To evaluate the effectiveness of the proposed method different metrics are reported for different datasets. \n\nMinor:\n- Eq. 3: final extra parenthesis\n- Line 220: “consits” -> “consists”\n- Line 234: extra ”the” in “represent the all the”\n\nQUESTIONS:\nQ1) Why reporting different metrics for different datasets? As an example, line 155, the authors state that they report MRR for Citeseer and Hits@50 for ogbl-collab. Similarly, in Table 2, different metrics are reported for different datasets. have seen this approach used in other papers on link prediction, but I am not convinced that it is the correct way to proceed because, as shown in the tables in the appendix (e.g., Tables 5-7), the proposed method is not always the best when the evaluation metrics are changed.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLink prediction in graphs is a fundamental task in graph machine learning and multiple heuristics and ML algorithms have been designed in research to leverage the pairwise and structural information to predict links between nodes. This work takes inspiration from the success of MoE models across various verticals and proposes a new method that outperforms the SoTA baselines on this task on several standard datasets. The authors demonstrate that heuristics and algorithms for LP are very diverse in their capabilities to predict links depending on the node structure and pairwise features (i.e. when one would perform well, the other might not). Due to this little overlap in their abilities the authors propose a dynamically weighted ensemble (MoE) like approach - such that a gating network predicts the weight given to each expert's prediction and the overall value is the weighted sum. \nThey also provide a very practical method to train the experts and the gating network achieving SoTA performance beating the second best by significant margins.\n\nSTRENGTHS:\nThe paper is very well written with the motivations being made clear and experiments being clearly able to prove what the authors set out to. The number of experiments and metrics being covered are large enough to justify the claims in a border sense - with all the obvious baselines being addressed. I particularly liked the comparisons with the Mean and Global weight ensembles, and LPFormers.\nThe authors also provide an efficient and effective training strategy to build this E2E - which in my opinion can be understood and used easily by the community.\nThe paper also provides reasonable explanations for the results and conducts ablations (in section 5) required to improve understanding and readability for a potential user.\nThis result greatly improves on the SoTA baselines and should be generalisable to multiple graph use cases.\n\nWEAKNESSES:\n- The method being used is fairly obvious and intuitive.\n- It solves for a very specific task - Link Prediction in an inductive setting only. Not clear to me how it could be leveraged for a transductive setting as well\n\nQUESTIONS:\n- Did you face any convergence issues with training the gating network ?\n- What happens when you use a graph network which is transductive in nature ? Can that information be leveraged in some manner ?\n- In the situations when the graph is a KG on language. Can LLM's outperform this method ?\n\nLIMITATIONS:\nThe method is pretty generic and the societal impacts are not discussed in a lot of detail as it wont be different from existing methods.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper presents a mixture of experts model, termed Link-MoE, for link prediction on graphs. Link-MoE individually trains various link prediction models as experts and selects the most appropriate experts for different node pairs. The prediction results from the selected experts are then weighted to produce the final predication. Experiments and analyses validate the effectiveness of the proposed method.

STRENGTHS:
1. This paper is generally well-written and easy to follow. Necessary analyses are conducted to well motivate the design of the proposed method. 
2. The proposed method is presented as a general framework, so that various link prediction models (i.e., experts) can be combined via different weights generated by the gating function.
3. The experimental design can largely validate the efficacy of the proposed method.

WEAKNESSES:
1. This paper presents a straightforward application of the MoE model to the problem of link prediction. Despite its empirical effectiveness, the technical novelty appears limited. The methodological contributions could be enhanced by addressing specific problems associated with MoE models, e.g., the collapse problem, in the context of link prediction. 
2. The experimental evaluation could provide more insights and deeper analyses on important issues, for example,
- How does Link-MoE determine the appropriate number of experts?
- On heterophilic graphs, how the experts selected by Link-MoE differ from those selected on homophiles graphs?
3. The complexity of the proposed method can be high. The nature of using MoE makes the proposed method difficult to scale up.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
The authors have discussed the limitations in terms of the scalability.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08