Mixture of Link Predictors on Graphs
Authors: Li Ma
Haoyu Han
Juanhui Li
Harry Shomer
Hui Liu
Xiaofeng Gao
Jiliang Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Link Prediction
Graph Neural Networks
Mixture of Experts
Primary area: graph_neural_networks

Abstract: Link prediction, which aims to forecast unseen connections in graphs, is a fundamental task in graph machine learning. Heuristic methods, leveraging a range of different pairwise measures such as common neighbors and shortest paths, often rival the performance of vanilla Graph Neural Networks (GNNs). Therefore, recent advancements in GNNs for link prediction (GNN4LP) have primarily focused on integrating one or a few types of pairwise information. 
In this work,  we reveal that different node pairs within the same dataset necessitate varied pairwise information for accurate prediction and models that only apply the same pairwise information uniformly could achieve suboptimal performance.
As a result, we propose a simple mixture of experts model Link-MoE for link prediction.  Link-MoE utilizes various GNNs as experts and  strategically selects the appropriate expert for each node pair based on various types of pairwise information. Experimental results across diverse real-world datasets demonstrate substantial performance improvement from Link-MoE. Notably,  Link-Mo achieves a relative improvement of 18.71% on the MRR metric for the Pubmed dataset and 9.59% on the Hits@100 metric for the ogbl-ppa dataset, compared to the best baselines. The code is available at https://github.com/ml-ml/Link-MoE/.

Mixture of Link Predictors on Graphs
Li Ma Haoyu Han Juanhui Li Harry Shomer Hui Liu²,
Xiaofeng Gao Jillinng Tang²
Shanghai Jiao Tong University, 'Michigan State University cali-csssjtu edu. en, gao-xfêcs sjtu. edu. cn (hanhaoy1,1ijus shomerha, liuhui7, tangj (angjll1@mmmss edu
Abstract
Link prediction, which aims to forecast unseen connections in graphs, is funda- mental task in graph machine learning. Heuristic methods, leveraging range different pairwise measures such as common neighbors and shortest paths, often rival the performance of vanilla Graph Neural Networks (GNNs). Therefore, recent advancements in GNNs for link prediction (GNN4LP) have primarily focused on integrating one or a few types of pairwise information. this work, we reveal that different node pairs within the same dataset necessitate varied pairwise information for accurate prediction and models that only apply the same pairwise information uniformly could achieve suboptimal performance. As result, we propose simple mixture of experts model Link-MoE for link prediction. Link-MoE utilizes various GNNs experts and strategically selects the appropriate expert for each node pair based on various types of pairwise information. Experimental results across diverse real-world datasets demonstrate substantial performance improvement from Link-MoE. Notably, Link-MoB achieves a relative improvement of 18.7 88.71% % on the MRR metric for the Pubmed dataset and 9.59% on the Hits @ 100 metric for the ogbl-ppa dataset, compared to the best baselines. The code is available at https://githhin com/ml m1mmmm///aaa Li/Link-Monn MoE/
Introduction
Link prediction (LP) is a central challenge in graph analysis with many real-world applications, such as recommender systems [ [2, drug discovery [3], and knowledge graph completion 4 ()) Specifically, I.P attempts to predict unseen edges in graph. Unlike node-level tasks where Graph Neural Networks (GNNs) excel in modeling individual node representations (6], LP demands the use of pairwise node representations to model the existence of link where vanilla GNNs often fall short 7 g Traditionally, various heuristic methods ( were used to identify new links by encapsulating the pairwise relationship between
nodes. For instance, the Common Neighbors (CN) heuristic [10] counts the number of shared neighbors between a node pair, postulating that the number of common neighbors indicate the likelihood of connection. The Katz index (II) considers the total number of paths between two nodes, assigning a higher weight to those shorter in length. Node feature maillartty-aseed methods [12] assumes that nodes with similar features tend to connect. Despite their simplicity, these heuristic-based methods are still considered as strong baselines in LP tasks.
To leverage both the representational power of GNNs and the effectiveness heuristics, recent GNN4LP works have sought to incorporate pairwise information into GNN frameworks, thereby
"Equal Contribution.
'Li Ma and Xiaofeng Gao are in the MoE Key Lab of Artificial Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong University.
*This work is done when Li Ma is a visiting student at Michigan State University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

enhancing their expressiveness for better link prediction. For example, NCN/NCNC [3] exploit the common neighbor information into GNNs. Neo-GNN [4 further incorporates multi-hop neighbor overlap information. SEAL [5] and NBFNet [6] leverage the full and partial labeling trick, respectively, to indict the target node pair, which has been proven to learn heuristic patterns like common neighbors and the Katz. index. These GNN4LP methods mark significant progress in link prediction [17].
However, both traditional heuristic approaches and GNN4LP models typically adopt one-size-fits solution, uniformly applying the same strategy to all target node pairs. There are several limitations with this one-size-fits-all solution: (1) Limited Use of Heuristics: These methods utilize only one or a few heuristics. Our preliminary studies in Section 3 have shown that different heuristics tend to complement each other, and employing multiple heuristics within the same dataset can lead to improved link prediction performance. Therefore, methods relying on a single type of heuristic might not be optimal. (2) Uniform Application of Heuristics: In Section 3 we also find that different node pairs within the same dataset often require distinet heuristics for predictions. Consequently. these methods that uniformly apply the same heuristic across all node pairs lack this adaptability, potentially leading to suboptimal performance. These findings underscore the pressing need for an approach that can adaptively apply range of pairwise information specific to node pairs. Inspired by the superior performance of existing GNN4LP, we delved deeper into understanding different GNN4LP models. Our investigation reveals that these models are highly complementary and they excel under specific conditions, often correlated with particular heuristics. For example, the NCN model tends to perform well in scenarios with a high number of common neighbors. These observations motivate us ask: can we design a strategy that can simultaneously enjoy the strengths of various GNN4LP models and correspondingly enhance link prediction?
In response to this question, we introduce simple yet remarkably effective mixture of experts model for link prediction - Link-MoE, This model operates by utilizing a range of existing link predictors as experts. A gating function is learned to assign different node pairs to different experts based on various types of pairwise information. Extensive experimental results showcase the surprisingly effective performance of Link-MoE. For instance, it surpasses the best baseline on Pubmed and ogbl-ppa dataset by 18.7 % on the MRR and 9.59% on the Hits @ 100, respectively.
2 Related Work
2.1 Link Prediction
Link prediction aims to predict unseen links in a graph. There are mainly three classes of methods for the link prediction task. An overview of each is given below.
Heuristic methods have been traditionally used for link prediction. They attempt to explicitly model the pairwise information between a node pair via hand-crafted measures. Several classes of heuristics exist for link prediction [17] including: local structural proximity, global structural proximity, and feature proximity. Local Structural Proximity (LSP): These method extract the information in the local neighborhood of a node pair. Common Neighbors (CN) [1O]. Adamic- Adar (AA) 18 and Resource Allocation [99 are popular measures that consider the number of shared h-hop neighbors between the node pair. Global Structural Proximity (GSP): These methods attempt to model the interaction of a node pair by extracting the global graph information. The Shortest Path Distance assumes that shorter distance between nodes results in a higher likelihood of them connecting. Both Katz Index [1] and Personalized Pagerank (PPR) 20 consider the paths of disparate length that connect both nodes, giving a higher weight to those shorter paths. Feature Proximity (FP): FP measures the similarity of the node features for both nodes in the pair, positing that nodes with similar features more likely to form edges. Previous work [2] 22 has derived heuristics algorithms to measure the feature proximity.
Heuristic Methods:
GNN-based Methods: Recent work has looked to move beyond pre-defined heuristic measures and model link prediction through the use of graph neural networks (GNNs). Earlier work 23 24 25 has sought to first use GNN to learn node representations, and the representations for both nodes in a pair are then used to predict whether they link. However, multiple works 7 have shown that node-based representations are unable to properly model link prediction. As such, newer methods, which we refer to as GNN4LP, attempt to learn pairwise representations to facilitate link prediction. Both SEAL 7 and NBFNet [16] condition the GNN aggregation either both one of the two
2

---

nodes in the pair. While expressive, both methods tend to be prohibitively expensive as they require aggregating messages separately for each node pair. Recent methods [3 26 27 have attempts
to devise more efficient ways of learning pairwise representations by injecting pairwise information in the score function, bypassing the need of customizing the GNN aggregation to each node pair. These methods typically attempt to exploit different structural patterns on the local and global scales.
0:33
0
0.19
6.25
0.29
0.58
0.53
0.53
0.36
0.52
0.06
5
0.25
0:60
0.29
0.63
RB
LM
CM
(a) Citeseer (K=3)
(b) ogbl-collab (K=20)
(a) Citeseer
(b) ogbl-collab
Figure 1: Hits@ of combined heuristics.
Figure 2: The overlapping ratio heuristics.
Ensemble-based methods for link prediction primarily fall into two categories: bagging and stacking. Bagging methods create multiple base learners trained varied data subsets and integrate their predictions for final output [28, 29). Stacking methods train multiple models on
Ensemble Methods:
the entire graph and use meta-model to integrate their predictions [30 31 32 33)) However, these methods directly combine the predictions of base learner without considering the specific patterns and heuristics of base learners.
2.2 Mixture of Experts
The use Mixture of Experts (MoE) 34 35,, which is based on the divide-and-conquer principle to divide problem to different experts, has been explored across various domains [36]. Recent researches
mainly focus on the efficiency of leveraging the MoB the NLP [37 38 or Computer Vision [39] domains. Chen et al. [40] attribute the success of MoE to the cluster structure of the underlying problem. In the graph domain, GMoE [41] integrates the MoE model with GNNs, enabling nodes to learn from information across different hops. Wu et al. [42) leverage MoE to address the distribution shift issue GNNs. To the best of our knowledge, Link-MoB represents the first instance of MoE
specifically tailored for the link prediction task, showcasing exceptionally effective performance.
3 Preliminary
In this section, begin by analyzing the relationship between various heuristics employed in link prediction, aiming to uncover the complex patterns that exist within the same dataset. Subsequently. we explore the relationship between the performance of different GNN4LP models and these heuris- tics. This exploration aims to identify the most suitable scenarios for each model, thereby enhancing our understanding of how different approaches can be optimally applied in varying link prediction contexts. Before that, we first introduce key notations and experimental settings.
Notations and experimental settings. Let g E), be graph with nodes, where V is the node set and & is the edge set. N; denotes the neighborhood node set for node u, The graph can be denoted an adjacency matrix A Raxn and each node ; may be associated with d-dimensional feature x, and we use X analysis on the Cora, Citeseer, Pubmed, ogbl-collab, and ogbl-ppa datasets. We use Hits@ metric to measure the ratio of positive samples ranked among the top K against set of negative samples. The details on each dataset and the evaluation setting can be found in Appendix A Due to the limited space, we only illustrate partial results in the following subsections. More results can be found in Appendix
.x...... |x........
x,] E Rand to denote the node feature matrix. We conduct
3.1 Exploring Heuristics in Link Prediction
In this subsection, we focus on three vital types of pairwise factors used link prediction identified by Mao et al. [[7]: (a) local structure proximity, (b) global structure proximity, and (c) feature
3

---

proximity. For each type, we adopt a single widely used heuristic as a representative metric including Common Neighbors (CN) [T]] for local structure proximity, Shortest Path (SP) [43 for global structural proximity, and Feature Cosine Similarity (FCS) for feature proximity. We initially evaluate the performance of each heuristic individually and then assess their combinations by simply adding their normalized values. Specifically, we normalize each heuristic value (h) to the range of [0, 1] using A A-Amaa
Amm=- Momnn where lemar and hmin are the maximum and minimum heuristic values in the dataset. One exception is the calculation of SP, where a smaller SP indicates a higher likelihood that two nodes are connected. Therefore, we first calculate SP. and then normalize it in the same way as other heuristics. For this evaluation, we employ Hits @ as the metric for smaller datasets and Hits @ 20 for larger OGB datasets. The results for Citeseer and ogbl-collab are illustrated in Figure [ with diagonal values representing the individual performance of each heuristic. We can have two observations: (1) combining different heuristics generally enhances overall performance, indicating that reliance
single heuristic may be inadequate for accurate link prediction; and (2) the performance of each heuristic varies across datasets. For instance, in the Citeseer dataset, FCS and CN exhibit comparable performance. However, in the ogbl-collab dataset, CN significantly outperforms FCS.
We further investigate the overlap in correctly predicted sample pairs by each heuristic. We use the Jaccard Coefficient to calculate the overlapping ratio between each pair of heuristics. For the calculation of the Jaccard coefficient, we use the Hits @ K metric for each edge. Specifically, we choose @its@3 for small datasets and Hits@20 for the OGB datasets. We first rank the prediction scores of each method for both positive and negative edges. If the prediction score of a positive edge is the Top-K, we label this positive edge as ppresent' and add it to the correct prediction set. In this way, we can calculate the Jaccard coefficient by comparing the correct prediction sets for each pair of methods. The results for the Citeseer and ogbl-collab datasets are presented in Figure [ We observe that the overlapping ratio between certain heuristics is notably low. This implies that the sets of node pairs correctly predicted by different heuristics have a minimal intersection. Therefore, different node pairs require distinet heuristics for accurate prediction even the same dataset.
From the previous analysis, it is enticing to think that simply considering multiple heuristics should result in superior link prediction perfor- mance. We test this hypothesis by learning to classify links using multiple popular heuristic methods. For single link, the individual heuris- tic scores are concatenated together and passed to an MLP, where the output is then used to clas- sify the link. The full set of heuristic considered can be found in Section 5.1 The results on Cite- seer and ogbl-collab can be found in Table [ We report the MRR for Citeseer and Hits Tits@55 50 for ogbl-collab. We find that ensembling multiple heuristics can modestly improve the performance. However, it still noticeably lags behind GNN4LP methods in performance. Based on this observation, we are motivated to investigate whether different GNN4LP models can be used to model a wider variety of links.
Method
Citeseer
ogbl-collab
CN
28.34
61.37
Heuristic
Shortest Path
31.82
46.49
Katz
36.16
64.33
Feature Similarity
31.82
26.27
Ensemble
44.08 4 0.18
64.44 = 0.21
Neo-GNN
53.97 +: 5.88
66.13 0.61
GNNALP
NCNC
64.03 4: 3.67
65.97 1.03
Table 1: Performance of ensembling heuristics.
3.2 Exploring GNN4LP Models and Heuristics
mlp
CRT
004
C32
C27
1.26
3.26
In this subsection, we move beyond analyz- ing the performance of only heuristic measures and further consider the capabilities of differ- ent GNN4LP methods. To better understand the abilities of different GNN4LP models, we first evaluate the overlapping ratio between differ- ent models using the Jaccard Coefficient. We also include the MLP and different heuristics in the analysis. The overlapping ratio of different methods on ogbl-collab dataset is shown in Fig- ure(3 These results reveal that the overlapping ratios among different GNN4LP models are rel-
24n
4.47
XX2
3/4
0.5
1/0
115
1.44
nc0
4.3
5:38
324
314
ne2gn0
buddy
smal
6.42
D04
rC5
4.59
633
X24
327
62
A2X
3
C22
629
927
46
Kate
A62
CaF
0.15
22
8
Figure 3: The overlapping ratio on ogbl-collab.
4

---

atively low, suggesting that each model is capable of predicting unique set of links. Furthermore, different GNN4LP models have varying degrees of overlap with different heuristics. These observa- tions lead us to intriguing question: Are the unique sets of links correctly predicted by different GNN4LP models related to specific heuristics?
To answer this question, we first categorize node pairs into 5 groups based on each heuristic and evaluate the performance of different GNN4LP models within these groups. Additionally, we also include the MLP and GCN in our analysis. The performance of these models across different Common Neighbors (CN) groups for Cora and ogbl-collab is depicted in Figure The x-axis represents different groups, along with the proportion of node pairs in each group. From the results, we can find that no single model consistently outperforms others across all groups on either dataset. Interestingly, when there are no common neighbors, MLP and GCN tend to excel in both the Cora and ogbl-collab datasets. In situations with few common neighbors, SEAL shows better performance in the Cora dataset, while BUDDY tends to lead in the ogbl-collab dataset. With an increase in the number of common neighbors, methods that encode CN information, such as NCNC, generally exhibit strong performance. A similar phenomena can be found on other datasets and heuristics in Appendix
FTS
busby
Duiiy
2,00 10.0791
Cora
ogbl-collab
Figure 4: The performance of different models each CN group.
In conclusion, our analysis underscores that accurate link prediction necessitates the use of multiple heuristics. Different node pairs require distinct heuristies for optimal prediction. Furthermore, the overlapping ratio of different GNN4LP models is relatively low. The performance of powerful GNN4LP models is closely tied the specific heuristics they encode. And there is no single model that can uniformly achieve the best performance across all scenarios. These findings pave us way to adaptively select the most suitable model for each node pair to achieve better overall performance in link prediction task.
4 Method
The investigations conducted in Section Dectionnrrvved reveal that various heuristics complement each other for the task on link prediction. This implies that different node pairs may need different heuristics to properly predict the existence of a link. Therefore, one approach to potentially achieve better performance in link prediction is to integrate all these heuristics into single, unified model. However, as shown in Table T] this strategy fails to outperform existing GNN4LP methods. This suggests that these GNN4LP models are already quite effective. Our findings further suggest that different GNN4LP models demonstrate unique strengths in different scenarios, which are related to specific heuristics. Therefore, leveraging these diverse heuristics
guidance could help identify the most suitable GNN4LP models for specific node pairs. Based on the above intuitions, we aim to design a framework that can harness the unique strengths of each model to enhance the performance of link prediction.
4.1 Link-MoE - A General Framework
To leverage the strengths of various existing models, we introduce a novel mixture-of-expertt (MoE) method tailored for link prediction, which we term Link-MoE. An overview of Link-MoE is depicted Figu There are two major components: the gating model and the multiple expert models. The gating function can be implemented using any neural network and each expert can be any method used for link prediction. When predicting whether a node pair (i,j) are linked, the gating function utilizes their heuristic information to produce normalized weights for each expert.
5

---

model
Heurtetios
Model Weigtt
0:36
mi model
MLP
x
Σ
Sigmald
9
s(-)
MLP
by 0.28 model
Gating Model
Me
Structural Branch
811
the model..
(a) Link-MoE
(b) Gating Model
Figure 5: overview of the proposed Link-MoE.
These weights dictate the level of contribution each expert model has towards the final prediction. Each expert model processes the graph and node pair information, estimating the likelihood of connection between the two nodes. The individual expert predictions are then aggregated according to the weights assigned by the gating function. The final prediction is made using the sum of the weighted scores, effectively leveraging the strengths of multiple experts to determine the probability of link. Formally, the prediction of Link-MoE can be expressed as:
YJ Σ G(xij, B:)
(I)
where m denotes the number of expert models incorporated. pairwise node features Xiÿ and structural heuristics Sij serves as the input to the gating function, G(-) represents the gating model function, E, refers the o-th expert model, and is sigmoid activation function. This configuration makes the Link-MoB remarkably flexible and easily adaptable, allowing for the seamless integration of different expert models required. We will detail our implementation the following subsections.
4.2 The Design of the Gating Model
As highlighted in our preliminary studies (Section B), it's evident that different GNN4LP models excel in varying contexts and that their strengths can be indicated by different heuristics. Therefore, we incorporate broad spectrum heuristics inputs the gating model leverage these strengths. Specifically we consider CN [0,, AA [18] and RA 19 to model the local structural proximity and Shortest Path, Katz index I1. and PPR 44 for the global structural proximity. For the feature proximity, to ensure the score is invariant to the ordering of the node pair, we utilize the element-wise product for deriving the feature heuristic of node pair (i, This defined as X,j = X; @ Xj. We also use 8;j [sty ....... s to represent all the k structural heuristies, and the dimension each structural heuristics is typically very small. However, the node pair feature heuristic is equal to the input node feature dimension and can span hundreds or even thousands of dimensions. This significant disparity in dimensionality could hinder the model's ability effectively learn from the structural heuristics. To address this challenge, we design a two-branch gating model, as illustrated in Figure b)... Each branch is simple MLP. One branch is dedicated to encoding the structural heuristics, while the other focuses on processing the node pair features. After that, these two branches merged via concatenation. Finally, an MLP with the softmax function is applied this combined output to generate the final weight predictions. This design ensures balanced consideration both structural and feature-based heuristics. Formally, the gating function is defined as follows:
G(Xij, Stj) = softmax softma(((Ufxx)l)lff (f((()))lI(ss.)))
(2)
where / is MLP and || denotes the concatenation operator.
4.3 Optimization of Link-MoE
Given the experts chosen in Section 5.1 there are several training strategies used by MoE models to combine them. This includes end-to-end training [42] and the EM algorithm 45). Given the multiple options, are then tasked with the question: how do optimize Link-MoE?

---

In this work, we employ a two-step training strategy, as detailed in Algorithm I in Appendix E Initially, we train each expert individually using their respective optimal hyperparameters and perform inference on the dataset to obtain the prediction scores for each link. Subsequently, we focus on the training of the gating model. Specifically, we adopt the cross entropy to train the gating function:
Σ Vij log Yis + vis) log(1 - Yig)),
= -
(i.))PUU
where Vij 1 when a link exists between node and v in the graph, and Bii = 0 otherwise. P and N denote the set of positive/negative links in the graph, respectively.
There are several benefits to our two-step training strategy. (1) Efficiency: This approach eliminates the need to load every expert into memory simultaneously, as each expert trained and infers independently; For time efficiency, tuning the gating model to identify the best hyperparameters is more efficient since it involves training only MLPs. (2) Effectiveness: The two-step training strategy helps to avoid the "collapse problem' often encountered in MoE. models [37]. This issue arises when only a single expert consistently selected, leading to the under-utilization and inadequate learning of the other experts. By training the experts individually first, we mitigate this risk, ensuring more balanced and effective utilization of all experts. We compare two-step and end-to and training strategies in Appendix G (3) Flexibility; When introducing new experts into the Link-MoE, it's only necessary to train these new experts and the gating model. All previously trained experts can be seamlessly integrated without the need for retraining.
5 Experiments
In this section, we conduct comprehensive experiments to validate the effectiveness of the proposed Link-MoE. Specifically, aim to address the following research questions: RQ1: How does Link-MoE perform when compared to other baseline models? RQ2: Are the heuristics effective in aiding the selection of experts? RQ3: Can Link-MoE adaptively select suitable experts for different node pairs?
5.1 Experimental Settings
Datasets.
We evaluate our proposed method on eight datasets including homophilous graphs: Cora, Citeseer, Pubmed 446. ogbl-ppa, ogbl-collab, and ogbl-citation2 [47] and heterophilic graphs:
Chameleon and Squirrel
48 Please see Appendix A for more details on each dataset.
Baselines. We consider diverse set of baselines include heuristics, embedding methods, GNNs,
and GNN4LP methods. This includes: CNs [10. AA [88,, RA [1),, Shortest Path [43 Katz [I]
Vec [4), Matrix Factorization (MF) [5)]. MLP, GCN (33, GAT [51], SAGE [24] GAE 25
SEAL 15 BUDDY 26). Neo-GNN 14 NCN and NCNC 13 NBFNet [6. PEG [2) L.P
Former 27
Additionally, comprehensively evaluate the proposed Link-MoE, which integrates multiple link predictors, we design two ensemble baseline methods for comparison. The first method, Mean- Ensemble, combines all expert models with uniform weight, ensuring each expert contributes equally to the final prediction. The second method, -HobaleEnsemmbe, learns a global weight for each expert that is applied when predicting all node pairs. In this approach, each expert contributes to the final prediction based on the learned global weight for all node pairs, allowing for differentiated influence of each expert based on their performance. See Appendix AA2 more details on these two methods. Additionally, we compare our method with two other ensemble methods 30 31, with the results provided in Appendix H
Link-MoE Settings. In this study, we incorporate both node features and variety of different heuristics as input features for the gating model. These heuristics include node degree, CN, AA, RA, Shortest Path, Katz, and Personalized PageRank (PPR) (20). Furthermore, our approach uses a wide range of experts, including NCN, NCNC, Neo-GNN, BUDDY, MLP, Node2Vee. SEAL, GCN, NBFNet, and PEG. NBFNet and PEG are only used for smaller datasets, as they often run into out-of-memory issues on the larger OGB datasets. For the two baselines, deenn-Eneemble and Global- Ensemble, we use the same experts with Link-MoE. More setting details are in Appendix A.2 For evaluation, we report several ranking metrics including the Hits @ and Mean Reciprocal Rank (MRR). In the main paper, report the MRR for Cora, Citeseer, and Pubmed and for OGB we
7

---

use the evaluation metric used in the original study 47 Results for other metrics are shown in Appendix
Table 2: Main results link prediction (%). Highlighted the results ranked first, second, and third. We use to highlight the experts we used Link-MoE. Notably, NBFNet and PEG are not used experts OGB datasets due to their OOM issues.
Cora MRR
Citeseer MRR
ogbl-collab Hits
oghl-ppa Hits@ 100
ogbl-citation2 MRR
Metric
MRR
CN
20.99
28.34
14.02
61.37
27.65
74.3
AA
31.87
29.37
16.66
64.17
32.45
75.96
Heuristic
30.79
27.61
15.63
49.33
76.04
Shortest Puth
12.45
31.82
7.15
46.49
0
>-24
Katz
27.4
38.16
21.44
64.33
27.65
74.3
Node2Vec*
37.29 8.82
44,33 a 8.99
34.61 a: 2.48
49.06 1.04
26.24 0.96
45.04 a 0.10
Embedding
MF
14.29
24.80 4.71
19.29 6.29
41.81 1.67
28.4 4.62
50.57 12.14
MLP
1
43.53 à 7.26
16.52 a 4.14
35.81 a: 1.08
0.45 a 0.04
38.07 a 0.09
GCN'
3
50.01 * 6.04
19.94 + 4.24
54,96 3.18
29.57 2,90
84.85 + 0.07
31.86
48.69 7.53
18.63 7.75
55.00 3.28
OOM
OOM
GNN
SAGE
7.75
47.84 6.39
22.74 + 5.47
39.44 1.37
41,02 1.94 0OM
83,06 0.09 OOM
GAE
29.98
63.33 4 3.14
16.67 +: 0.19
OOM
SEAL"
26.69 5:889
39.36 + 4.99
38.06 5,18
63.37 0.69
48,80 # 5.61
86.93 = 0.43
BUDDY"
26.40
59.48 8.96
23.98 5.11
0,46
1,96
0.18
Neo-GNN
2.66
53.97 5.88
3..45
6 0.61
48.45 1.01
83.54 0.32
GNNALP
NCN'
32,93
54,97 * 6.03
35.65 + 4.60
63.86 0,51
62.63 1.15
89.27 * 0.05
NCNC
65,97
89.82 0.43
NBFNet
37.69+3..3
38.17 3.06
44.73 +: 2.12
OOM
OOM
OOM
PEG
22.76 1.84
56.12 6.62
21.05 2.85
49.02 2.99
0OM
OOM
LPFormer
3.74
65.42 4.65
40.17
68.14 0.51
63.32 = 0.63
89.81 0.13
Ensembbe
4.77
53.73 2.83
38.54 5.40
66.82
26.70 3.92
89.55 0.55
Ensemble
Global Ensemble
4..6
53.96 *: 2,79
37.63 +: 6.54
67.08 0.34
60.67 1.44
90.72 0.72
Link MoE
44.03 2.28
67.49 a: 0,30
53.10 d: 0,24
71.32 a: 0.99
69.39 0.61
91,25 a 0.02
Improv.
10.80%
3.16%
18.71%
4.67%
9.59%
0.58%
5.2
Main Results
We present the main results of link prediction for the small datasets and OGB datasets in Table 2 Reported results are mean and standard deviation over 10 seeds. We use "Improv." to denote the relative improvement of Link-MoE over the second best model. From the table, we can have the following observations:
Link-MoE consistently outperforms all the baselines by significant margin. For example, it achieves a relative improvement of 18.71% on the MRR metric for the Pubmed dataset and 9.59% on the Hit@ 100 metric for the ogbl-ppa dataset, compared to the best rest-performing baseline methods.
While both the perts used in Link-MoE, their performance is generally subpar in most cases. Although the Global-Ensemblee which learns different weights for each expert, usually outperforms the Mean- Ensemble, it still falls short of the performance of single baseline methods in some scenarios. We attribute this their inability to adaptively apply different experts specific node pairs, which demonstrates the effectiveness the gating model in Link-MoE.
and Global-Ensemiee methods also incorporate all the
We further compare against LPFormer (27). recent method that attempts to adaptively customize pairwise information to each node pair, resulting in strong performance. We find that our model is able considerably outperform LPFormer on all datasets. From this we conclude that Link-MoE is better than LPFormer at customizing the pairwise information to each node pair.
Additionally, instead of using all experts, conducted experiments with only few experts (i.e., or 4 experts). Furthermore, we also explored a sparse gating strategy [37], which selectively activates only the Top-K experts for each sample's prediction. The results, presented in Appendix D demonstrate that these two variants can achieve comparable performance with using all the experts.
We further evaluate the proposed Link-MoB on heterophilic graphs, with results in Appendix indicating that Link-MoF: achieves strong performance. Additionally, we assess Link-MoE in more challenging HeaRT setting [53), with results in Appendix furtther validating its effectiveness.

---

80
ogbi oget ppa
70
70-
60-
55
55
45
40 H
Hils@10
40
Hilsgr5o
Hit=@100
(a) Pubmed
(b) ogbl-ppa
Figure 7: Performance impact of removing single expert.
Figure 6: Performance of Link-MoB variants.
1.21
0.8
0.5
0.4
0.7
03
0.0
0-0
1:24
24-48 40
41-12
73 3756
00
0-0
1-41
43
10
E3-124
125-2241
(a) ogbl-collab
(b) ogbl-ppa
Figure 8: Expert weights for ogbl-collab and ogbl-ppa dataset. The groups are split based on CN.
5.3 The Effectiveness of Different Heuristics in Gating
The addition of the gating model enables Link-MoE to significantly surpass both the Mean Ensemble and Ensemble methods. This improvement underscores the pivotal role of the gating model, which intelligently leverages heuristic information to enhance the overall performance. In this subsection, we delve into impact of various heuristics utilized by the gating model Link-MoE. Specifically, there are three types of heuristics: local structure proximity (e.g., CN, AA, RA), global structure proximity (e.g., Shortest Path, Katz), and feature proximity. We design experiments to isolate the effect of specific types of heuristics by including them either individually or in groups. For instance, utilizing only feature proximity is referred to as 'OnlyFeat', combining local and global structure proximities is labeled 'OnlyStruct', employing only local structure proximity is denoted OollLoocaStrrutt' and using only global structure proximity is marked 'OnlyGlobalStruet"" Additionally, the 'All' is used to indicate leveraging all heuristics. The results on Pubmed and ogbl-ppa datasets are shown in Figure 6) From the experimental results, several observations can be made: (1) Using only the feature proximity (OnlyFeat) only the global structural proximity (OntyGlobalStrruet) typically yields lower performance. (2) Combining the local and global structure proximity (OnlyStruct) tends to outperform using either alone.
(3) The effectiveness of feature proximity varies between datasets. On Pubmed, it is beneficial to include it but not on ogbl-ppa. This discrepancy could stem from the relative importance of feature information in each dataset, as suggested by the very poor performance of MLP on ogbl-ppa, whose Hits@ 100 is only around 0,45. Moreover, explore different inputs for the gating model, the results are presented in Appendix which highlights the rationality of the design of our gating model.
5.4 The Importance of Different Experts
In the previous sections, we incorporated all the selected experts Link-MoE. This subsection aims to explore the impact of each expert's removal on the framework's performance. We conducted experiments on the ogbl-collab and ogbl-ppa datasets, utilizing H0 as the metric for evaluation. The results are presented in Figure 7 where the x-axis indicates the experts removed for each trial. We observe that the removal individual experts does not significantly impact the performance of Link-MoE on both the ogbl-collab and ogbl-ppa datasets. This phenomenon suggest that Link-MoE possesses an adaptive capability to compensate for the absence of certain models by effectively utilizing other available experts.

---

5.5 Analysis of the Gating Weights
In this subsection, we explore the mechanism by which the gating model allocates weights to experts based on the heuristics of different node pairs. Similar to Section we categorize the test node pairs into distinct groups according to different heuristics. Subsequently, for each group, we compute the average weights assigned to each expert by the gating model. The results based Common Neighbors for ogbl-collab and ogbl-ppa are shown in Figure g
We can have some interesting findings: (1): For the ogbl-collab dataset, when the node pair doesn't have common neighbor, the gating model usually assigns a large weight to the MLP model, which aligns the analysis in Section 3 that the MLP can perform well when there is CN ogbl-collab. However, For the ogbl-ppe dataset, when there is no common neighbor, the gating model would assign a high weight to NCNC. This preference arises because node features hold less significance in the ogbl-ppa dataset and NCNC can leverage multi-hop common neighbor information. (2): As the number of common neighbors increases, the gating model increasingly allocates more weight to the NCN. This shift reflects the NCN's capability to efficiently capture and utilize common neighbor information. (3): Not every expert model contributes much to the prediction. This selective engagement is attributed to the overlapping capabilities of certain models. For instance, both Nco-GNN and NCNC are capable of exploiting multi-hop information. In such cases, the gating model opts for one over the other avoid redundancy and optimize prediction efficacy. This phenomenon is also consistent with the results in Section 5.4 that removing one expert doesn't affect the overall performance. The analysis of gating weights on heterophilic datasets can be found in Appendix
While Link-MoB successfully leverage heuristics select appropriate experts for different node pairs, we recognize substantial room for further enhancements. For instance, the observed overlapping ratio between Nco-GNN and NCNC on the ogbl-collab dataset not very high, as shown in Figure [3 even if they exploit similar heuristics. But in Link-MoE, Nco-GNN has very low weights on the ogbl-collab dataset. This observation underscores the vast potential for advancing MoE applications in link prediction, a direction we intend to explore in future work.
6 Conclusion
In this study, explored various heuristics and GNN4LP models for link prediction. Based on the analysis, a novel MoE model Link-MoE is designed to capitalize on the strengths of diverse expert models for link prediction. The extensive experiments underscore the exceptional performance of Link-MoE in link prediction, validating the rationale behind its design. Furthermore, we also showcase the substantial potential of MoE models for link prediction.
References
[1] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session- recommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 346-353, 2019.
[2] Hyeyoung Ko, Suyeon Lee, Yoonseo Park, and Anna Choi. A survey of recommendation systems: recommendation models, techniques. and application fields. Electronics, 11(1):141, 2022.
[3] Jurgen Drews. Drug discovery: historical perspective. science, 287(5460):1900-1166 2000.
[4] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[5] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. 2014.
[6] Seiji Maekawa, Koki Noda, Yuya Sasaki, et al. Beyond roal-world benchmark datasets: An empirical study of node classification with gnns. Advances in Neural Information Processing Systems, 35:5562-5574, 2022.
10

---

[7] Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 14:9061-90733 2021.
[8] Balasubramaniam Srinivasan and Bruno Ribeiro. On the equivalence between positional node embeddings and structural graph representations. International Conference on Learning Representations, 2019.
[9] Ajay Kumar, Shashank Sheshar Singh, Kuldeep Singh, and Bhaskar Biswas. Link prediction techniques, applications, and performance: A survey. Physica A: Statistical Mechanics and its Applications, 253:194289, 2020.
[10] Mark EJ Newman. Clustering and preferential attachment in growing networks. Physical review
E, 64(2):025102, 2001.
[11] Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18(1):39-44 43, 1953.
[12] Yohsuke Murase, Hang Hyun Jo, János Török, János Kertész, Kimmo Kaski, et al. Structural transition in social networks. 2019.
[13] Xiyuan Wang, Haotong Yang, and Muhan Zhang. Neural common neighbor with completion for link prediction. arXiv preprint arXiv:2302.00880 2023.
[14] Seongjun Yun, Seoyoon Kim, Junhyun Lee, Jaewoo Kang, and Hyunwoo Kim. Neo-gnns: Neighborhood overlap-aware graph neural networks for link prediction. Advances in Neural Information Processing Systems, 34: 101111683111333 13694, 2021.
[15] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
[16] Zhaocheng Zhu, Zuobai Zhang. Louis-Pascal Xhonneux, and Jian Tang. Neural bellman- ford networks: A general graph neural network framework for link prediction. Advances Neural Information Processing Systems, 34:29476-29490, 2021.
[17] Haitao Mao, Juanhui Li, Harry Shomer, Bingheng Li, Wenqi Fan, Yao Ma, Tong Zhao, Neil Shah, and Jiliang Tang. Revisiting link prediction: A data perspective. arXiv preprint mtXiv:2310.00999 2023.
[18] Lada A Adamic and Eytan Adar. Friends and neighbors on the web. Social networks, 25(3)2111- 230, 2003.
[19] Tao Zhou, Linyuan Lü and Yi-Cheng Zhang. Predicting missing links via local information. The European Physical Journal B. 71:623-630, 2009.
[20] Hanzhi Wang, Zhewei Wei, Junhao Gan, Sibo Wang. and Zengfeng Huang. Personalized pagerank target node, revisited. In Proceedings the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 657-667, 2020.
[21] Jiliang Tang, Huiji Gao, Xia Hu, and Huan Liu. Exploiting homophily effect for trust prediction. In Proceedings of the sixth ACM international conference on Web search and data mining, pages 53 53-62,44 2013.
[22] He Zhao, Lan Du, and Wray Buntine. Leveraging node attributes for incomplete relational data. In International conference machine learning, pages 4072-4081. PMLR, 2017.
[23] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint
2016.
[24] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[25] Kipf Thomas and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.0722 2(10), 2016.

---

[26] Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M Bronstein, and Max Hansmire. Graph neural networks for link prediction with subgraph sketching. arXiv preprint arXiv:2209. 15486, 2022.
[27] Harry Shomer, Yao Ma. Haitao Mao, Juanhui Li, Bo Wu, and Jiliang Tang. Lpformer: An adaptive graph transformer for link prediction. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2686-2698, 2024.
[28] Liang Duan, Shuai Ma, Charu Aggarwal, Tiejun Ma, and Jinpeng Huai. An ensemble approach link prediction. IEEE Transactions on Knowledge and Data Engineering, 29(11):2402-241 2017.
[29] Yi-Ling Chen, Ming-Syan Chen, and S Yu Philip. Ensemble of diverse sparsifications for link prediction in large-scale networks. In 2015 IEEE International Conference on Data Mining, pages 51-60. IEEE, 2015.
[30] Amir Ghasemian, Homa Hosseinmardi, Aram Galstyan, Edoardo M Airoldi, and Aaron Clauset. Stacking models for nearly optimal link prediction in complex networks. Proceedings of the National Academy of Sciences, 1 (1333)))3333-223400 2020.
[31] Yen-Liang Chen, Chen-Hsin Hsiao, and Chia-Chi Wu. An ensemble model for link prediction based on graph embedding. Decision Support Systems, 157:113753, 2022.
[32] Kuanyang Li, Lilan Tu, and Lang Chai. Ensemble-model-baae link prediction of complex networks. Computer Networks, 166: 106978, 2020.
[33] Shruti Pachaury, Nilesh Kumar, Ayush Khanduri, and Himanshu Mittal. Link prediction method using topological features and ensemble model. In 2018 Eleventh International Conference on Contemporary Computing (IC3), pages 1-6. IEEE, 2018.
[34] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[35] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-2144 1994.
[36] Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275-22 2014.
[37] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expprts layer. arXiv preprint arXiv: 70666388,, 2017.
[38] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Efficient scaling of language models with mixture-of-exppers. In International Conference on Machine Learning, pages 5547-5569. PMLR, 2022.
[39] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595 2021.
[40] Zixiang Chen. Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208.028: 028113, 2022.
[41] Haotao Wang. Ziyu Jiang, Yan Han, and Zhangyang Wang. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling. arXiv preprint arXiv:2304.02806, 2023.
[42] Shirley Wu, Kaidi Cao, Bruno Ribeiro, James Zou, and Jure Leskovec. Graphmetro: Mit- igating complex distribution shifts in gnns via mixture of aligned experts. arXiv preprint arXiv:2312.04699. 2023.
[43] David Liben-Nowell and Jon Kleinberg. The link prediction problem for social net- works. In Proceedings of the twelfth international conference on Information and knowledge management, pages 556-559, 2003.
12

---

[44] Sergey Brin and Lawrence Page. The anatomy of a large-scake scale hypertextual web search engine. Computer networks and ISDN systems, 30(1-7):107-11) 1998.
[45] Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture experts in multiclass classification. Neural networks, 12(9): (29):1229-12535 1999.
[46] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR, 2016.
[47] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133 2020.
[48] Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2):nnab014 2021.
[49] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 355-864,77 2016.
[50] Aditya Krishna Menon and Charles Elkan. Link prediction via matrix factorization. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part II 22, pages 437-452. Springer, 2011.
[51] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint (i.Kii:1110.199000 2017.
[52] Haorui Wang. Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more powerful graph neural networks. arXiv preprint arXiv:2003.01199 2022.
[53] Juanhui Li, Harry Shomer, Haitao Mao, Shenglai Zeng, Yao Ma, Neil Shah, Jiliang Tang, and Dawei Yin. Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking. Advances Neural Information Processing Systems, 36, 2023.
[54] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs Advances in neural information processing systems, 33:22118-22133 2020.
[55] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
[56] Shijie Zhou, Zhimeng Guo, Charu Aggarwal, Xiang Zhang, and Suhang Wang. Link prediction on heterophilic graphs via disentangled representation learning. arXiv preprint arXiv:2208.00 01820, 2022.
13

---

A Datasets and Experimental Settings
A.1 Datasets
The statistics for each dataset is shown in Table 3 We adopt the single fixed train/validaatinntess split with percentages 85/5/10% for Cora, Citeseer, and Pubmed as used in [5)] For the OGB datasets, use the fixed splits provided by the OGB benchmark [54]. Note that we omit ogbl-ddi
due to observations made by Li et al. [53] showing a weak correlation between validation and test
performance. For heterophilic graphs Chameleon and Squirrel, we use the same split ratio Cora.
Table 3: Statistics of datasets. The split ratio is for train/validationntess.
Cora
Citeseer
Pubmed
ogbl-collab
ogbl-ppa
ogbl-citation2
Chameleon
Squirel
#Nodes
2,708
3,327
18.717
235.868
576,289
2,927.963
2,277
5,201
#Edges
5,278
4,676
44,327
1,285,465
30,326,233
30,561.187
36,101
217,037
Mcan Degree
3.9
2.81
4.74
10,90
105.25
20.88
31.71
83.46
Split Ratio
85/5/10
85/5/10
85/5/10
92/4/4
70/20/10
98/1/1
85/5/10
85/5/10
A.2 Experimental Settings
Training Settings. We use the binary cross entropy loss to train each model. The loss is optimized using the Adam optimizer [55]. At first, we train all of the expert models by using the hyperparameters suggested in this repository We then do the inference to obtain the prediction score for each link. Secondly, in order to train Link-MoE, we split the original validation dataset into a new training set and validation set. Thirdly, weights associated with the best validation performance. The rationale for utilizing a portion of the validation set to train the gating model is follows: Given that each expert model is finely tuned on the training data, there exists a significant disparity between the prediction scores for positive and negative edges within this dataset. Should the original training set be employed to train our gating model, the outcome would be skewed reeerddeegasdlesss of the gating model's outputs, predicted scores for positive pairs would invariably remain substantially higher than those for negative pairs. Therefore, to train the gating model effectively, we repurpose the original validation set as our training data, dividing into new training and validation subsets. Unlike baseline models, which are trained exclusively on the original training set, our method benefits from incorporating very small validation set compared to the training set into the training process for the gating model. yielding notable performance improvements. Despite this modification, the comparison remains relatively fair. The original validation set is relatively small, so using a portion of it for training does not significantly alter the amount of data available. Thus, the baseline models remain essentially equivalent, even with this adjustment. Moreover, it is common for validation set is to be used for the search of model architectures in neural architecture search. We train both experts and gating models on NVIDIA RTX A6000 GPU with 48GB memory.
train the gating model until it converges and choose the model
Training Data Split. In the original datasets, we are given a fixed train, validation and test dataset. These splits are used to train each individual expert. Once each expert model fully trained, we perform inference on the validation and test sets, thereby obtaining the prediction score for each link. As noted carlier, we split the original validation set into a new training and validation split for training and validating Link-MoF. Note that the original test dataset is still used solely for testing. When splitting the validation set, we use different ratios for different datasets. For ogbl-citation2. ogbl-ppa, ogbl-collab, Citeseer, Chameleon and Squirrel, the ratios are 0.8, and for Cora and Pubmed, the ratios aer 0.9.
Hyperparameter Settings. The hyperparameter ranges are shown in Table 4 Since our gating model has a low complexity, we can efficiently search over a large hyperparameter space on all eight datasets.
Evaluation Setting. The evaluation is conducted by ranking each positive sample against set of negative samples. The same set of negatives are shared among all positive samples with the exception of ogbl-citation2, which customizes 1000 negatives to each positive sample. The set of negative samples are fixed and are taken from Li et al. (33 and Hu et al. (47 for their respective datasets.
14

---

Table Hyperparameter Search Ranges
Dataset
Learning Rate
Dropout
Weight Decay
Model Layers
Hidden Dim
Cora
(0.001, 0.0001, 0.0001)
0.,0,3, 0.5, 0.8)
(le-2, le-4, le-7, 0)
(1. 2, 3)
(8. 16, 32, 64)
Citeseer
(0.001, 0.0001, 0.0001)
(0, 0.3, 0.5, 0.8)
(le-2, le-4, le-7, 0)
(1, 2,
(8. 16, 32, 64)
Pubmed
(0.001, 0.0001. 0.0001)
(0. 0.30 0.5, 0.8)
(le-2, le-4, 1c-7.
(1.2.3
(8. 16, 64)
ogbl-collab
(0.01, 0.001, 0.0001)
10.00.00.5
(1c-7.0)
(2,3.4)
(32, 128)
ogbl-ppa
(0.01. 0.001, 0.0001)
(0.0..0.55
(le-7.0) 0)
(,,3
(32, 64, 128)
ogbl-citation2
(0.01. 0.001. 0.0001)
0.3,
(1e-7. 0)
(2.3.4)
(32, 64, 128)
Chameleon
(0.001, 0.0001)
(0, 0.3, 0.5)
(le-4, le-7, 0)
(1,
(8, 16, 32, 64)
Squirrel
(0.01, 0.001)
(0, 0.3, 0.5)
(lc-7.0) 0)
3,
(32, 64, 128)
Mean-Ensemble. After training the experts, we can obtain the prediction score on the test set from each expert. Then we take the mean across all experts to final score for link in the test set. The purpose of this model is to ascertain whether naively combining the different experts is itself enough for good performance. More formally, the Mcan-Ensemble can be defined as follows:
YU
(4)
Global-Ensemble. For the Global-Ensemine Ensemble, we learn weight vector w [W], W2, Wm] to combine the experts which can be defined as follows:
YJ
W. E, X)ii)
(5)
Notably, the is uniform for all node pairs. Therefore, this method is not able to flexibility adjust the weight of different experts to each node pair. Rather, single weight is used across all samples. The purpose of this method is to test whether it is necessary to customize the weight of the experts to each link is done in Link-MoE.
B Additional Results for the Preliminary Study
In this section, we extend the preliminary study (Section 3 with additional results, including model and heuristic overlaps Cora, Citeseer, Pubmed, and ogbl-ppa datasets. We also analyze the performance of various models across node pair groups categorized by Commen Neighbors (CN), Shortest Path (SP), and Feature Cosine Similarity (FCS).
B.1 Model Overlapping Results
Following Section 3 for the Cora, CiteSeer, PubMed, we use Hits@3. And we use Hits@ as the metric for ogbl-ppa dataset. The results are shown in Figure and Figure I0 From the results, we can have the following findings:
For most datasets, the overlapping between different GNN4LP models is not very high, which consistent with the finding Section
The feature cosine similarity usually has much smaller overlapping with other methods.
B.2 The Performance of Different Models on Different Heuristic Groups
In this section, we showcase the varied performances of GNN4LP models across groups categorized by shortest path and feature cosine similarity within the Cora and ogbl-collab datasets. The results for these heuristics are depicted Figure TT for shortest path groups and Figure 12 for cosine similarity groups. We have the following observations:
For the Cora dataset, SEAL excels over other models at shorter path lengths, while MLP shows superior performance for longer paths. In the ogbl-collab dataset, BUDDY is effective at shorter distances, whereas Neo-GNN, which integrates multi-hop common neighbor information, performs better the shortest path (SP) length increases.
15

---

1:00
031
0.38
<12
C75
C95
E35
6:41
10
0:3
BAY
D47
3.41
5:44
5:48
no0gin
5:37
5:M0
500gmn
5:72
5:52
6:3
5C31
FCS
2:24
9:
F22
3.29
2:21
0.21
A6
2-28
0.54
8:55
SAT
AA
5
55
6.6
222
0.99
L2T
343
1a77
122
0/2
A/D
KX2z
3.24
3.4
6.4
6.7
SP
6-12
E
3.8
mt
90
1000
PCL
M
badd)
FC5
(a) Cora
(b) Citesee
Figure 9: The overlapping ratio of different methods on Cora and CiteSeer dataset.
mp
0.2
5.55
022
0.80
321
2:28
mip
E12
0:43
9:37
nCSc
0.
0:29
4:48
4-4
830
054
DUNS
0.20
6.3
0.30
140gm
622
441
X22
DCE
C4
644
3X
2.55
25
0:30
8:30
0.28
buddy
622
445
0.34
4.4
0.000
6:00
0.79
5.24
AA
E26
CN
A2T
8:K
CN
2:28
FM1
5P
5P
mn
g20
bat
KC
RCCC
(a)
Pubmed
(b) ogbl-ppa
Figure 10: The overlapping ratio of different methods on Pubmed and ogbl- -ppa dataset.
For the Cora dataset, the seal excels over other models at smaller feature cosine similarity, while MLP outperforms when the feature cosine similarity is high. For the ogbl-collab dataset, NCNC is effective low feature cosine similarity, but BUDDY can works well when the feature similarity is high.
Different datasets exhibit distinct patterns, even within the same heuristic groups. There is no single model consistently outperforms others across all groups
These observations further validate the rationale behind employing heuristics inputs the gating model, effectively leveraging the strengths of various GNN4LP models.
C Additional Results on Benchmark Datasets
We present additional results of Cora, Citeseer, Pubmed, and OGB datasets in Table respectively. We use">>2h" to denote methods that require more than 24 hours for either training one epoch or evaluation. OOM indicates that the algorithm requires over 50Gb of GPU memory [53]. Note that LPFormer (27] is omitted from these tables as it doesn't 't report the results on these additional metrics. The data presented in these tables indicate that Link-MoE consistently achieves the best results in most cases, demonstrating its superior effectiveness in link prediction task. Although it does not always achieve the best performance across all metrics for the Cora and Citeseer datasets, usually achieves the second or third best results. This slightly diminished performance can likely be
and g
16

---

PCEC
20.3
(a) Cora
(b) ogbl-collab
Figure 11: The performance of different models on each shortest path group.
buddy
buddy
Cora
ogbl-collab
Figure 12: The performance of different models on each feature cosine similarity group.
attributed to the limited number of validation edges available in these datasets, i.c., 263 in Cora and 227 in Citeseer, restricting the amount information Link-MoE can leverage for optimal learning and performance.
Table Additional results on Cora(%). Highlighted are the results ranked first, second, and third. We use to highlight the experts used in Link-MoE.
Models
Hise@
Hise
H0
Hits@ 100
13.47
13.47
42.69
42.69
AA
22.2
39.47
42.69
42.69
Houristic
20.11
39.47
42.69
Shortest Path
0
0
42.69
71.35
28.46
Node2Vec*
11.76
41.63 # 10.5
62.34 2.35
84.88 = 0.96
Embedding
MLP
18.79
10.71 35.35
53.59 3.57
85.52 : 1.44
GCN
1
GNN
GAT
18.02 8.96
42.28 a: 6.37
63.82 2.72
90.70 1.03
SAGE
44.51 6.57
GAE
17.57.4..37
24.82 a: 4.91
70.29 2.75
92.75 0.95
BUDDY"
12.62
29.64 5.7
+
91.42 = 1.26
INo
1.96
33.36
64.1
87.76 1.57
GNNHLP
19.34
7.01
74.38 3.115
95.56 0.79
NCNC
9.79
34.31 + 8.87
75.07 1.95
95.62 0.84
PEG
5.88 2: 1.65
30.53 +: 6.42
62.49
91.42 0.8
Mean- Mean-Ensemble
2.934
4.92
Elobal-Ensemble Link-MoB
2.65
36.7 1 6.66
76.81
96.9 : 0.59
32.12
38.81 1.09
75.84 0.28
96.26 0.09
D Results for Top-K Experts and few Experts
We conduct experiments on the ogbl-collab and Pubmed datasets, we limit the activation to the Top-3 experts. And we also conduct experiments using 3 or 4 experts. Specifically, for the ogbl-collab, we use MLP. NCNC, BUDDY (3 experts) and Neo-GNN (4 experts); for the Pubmed datasets, we use NCN, SEAL, NCN (3 experts) and MI.P (4 experts). The performance metrics used evaluate this approach are also Hits @ 50 for ogbl-collab and MRR for Pubmed. The result are shown in Table 9
17

---

Table 6: Additional results on Citeseer(%). Highlighted are the results ranked first, second, and third. We use to highlight the experts we used in Link-MoE.
Models
His@1
Hit@#
Hise 10
Hite 100
AA
21.98
35.16
35.16
35.16
Houristic
35.16
35.16
Shortest Path
53.41
56.92
62.64
Katz
24.18
54,95
57.36
62.64
Node2Voc"
30.24 16.37
54.15 1 6.96
68.79
3.05
89.89 +: 1.48
Embedding
MF
6.71
29.03 4.82
38.99
59.47 2.69
ML.P
30.22 10.78
56.42 + 7.90
2.19 69:74
91.25 1.90
GCN"
62.77
1.24
GAT
34,00 11.14
62.72 4,60
74.99 1,78
91 6969 2.11
GNN
SAGE
10.27 27.08
65.52
78.06 2.26
96.50 0.53
GAE
54,06 5.8
65.3 + 2.54
81.72 + 2.62
95.17 0.5
SEAL"
8.12
46.04
60.02
85.6
BUDDY"
49.01 15007
67.01 6.22
80.04 2:22
95.4 0.63
Neo GNN
41.01
59.87 6.33
69.25
+: 0.97
GNN-LP
NCN'
3
66.83 4.06
79.12 1.73
96.17 1.06
NONC
53.21
69.65 3.19
82.64 6:44::100
97.54 +: 0.59
17.25
0.42
PEG
39.19
70.15 4.3
77.06 3:553
94.82 0.81
Mean-Ensemble
32.50:+6.21
70.53 +
85.45
97.27 # 0.40
Global- Inseml-Fnsemble
6.223
=
85.09 2,11
97.39 + 0.34
Table Additional results on Pubmed(%) %). Highlighted are the results ranked first, second, and third. We use * to highlight the experts we used in Link-MoE.
Models
Hitse
Hitse:
Hits 10
Hits@ 100
7.06
12.95
27.93
27.93
AA
12.95
16
27.93
27.93
Heuristic
15.21
Shortest Path
27.93
60,34
Katz
12.88
25.38
42.17
61.8
Node2Vee
29.76 4.00
34.08 2,43
44,29 2.62
63.07 = 0.34
Embedding
MF
12.58 6.08
22.51 5.6
32.05
2.44
53.75 2.06
MLP
7,83 6.40
17.23 + 2.79
34.01 4.94
84.19 = 1.33
GCN
4.22
7.59
0.65
GNN
GAT
6.45 10.37
23.02 10.49
4.03 46.77
80.95 0.72
SAGE
11.26:66.86
27.23 7,48
48.18 +
90.02 0.70
1.99
1.13 31.75
0.31
SEAT."
30.93
40.58 6.79
48.45 2.67
76,06 4.12
BUDDY"
15.31
29.79 6.76
46.62 4.58
83.21 0.59
Noo-GNN"
+9.95
34.85 + 4.43
56.25 3.42
86.12 1.18
GNNALP
NCNC
9.74
33.01 a: 6.28
61.89 3.54
91.93 0.6
NBFNot
40.47
PEG
8.52
24.46 6.94
45.11 4.00
76.45 3.83
Ensemble
Global Ensemble
24.36 11.01
6.32
58.85 3.14
93.14 0.36
Link-MolE
4..33
52.57 2,27
61.11 1.03
90.38 0.24
From the results, we can find that only or 4 experts can achieve comparable performance with using all experts. Notably, these results don't 't use the computationally intensive SEAL for the ogbl-collab datasets. Furthermore, the inclusion of the less effective MLP expert in the Pubmed dataset still results in performance improvement, highlighting the complementary nature of the experts and the effectiveness of the proposed method. From the result, sparse gating mechanism can also achieve promising performance on both datasets.
E
Algorithm
The full algorithm is detailed in Algorithm Notably, line 1-3 trains the experts individally. Line 4 performs the inference on links to obtain the prediction scores using the obtained experts and get the heuristic features for each link. Line 5-7 updates the gating model using Equation
18

---

Table 8: Additional results on OGB datasets(%)) Highlighted the results ranked first, second, and third. We use to highlight the experts we used in Ank.
oghi-collah
ogbl
ngbl-citation2 Hits
Hits @
Hits
Hits@
Hits 50
Hits 20
Hits@
65.6
77.99
77.99
AA
53.79
65.6
14.96
21.83
77.99
77.99
77.99
RA
55.01
65.6
25.64
38.81
77.99
77.99
77.99
Shortest Path
46.49
66.82
>24h
>24h
24hh
Katz
58.11
71.04
13.26
19.67
78
78
78
Noce2Voc*
40.68 1.75
55.58 a 0.77
11.22 a: 1.91
19.22 +: 1.69
82.8 0.13
92.33 1: 0.1
96.44 :: 0.03
39,99 1.25
4 1.94
9.33
21.08 + 3.92
70.8 12.0
74.48 10.42
75.5 10.13
MLP
27.66
42.13 1.09
0,16 + 0.0
0.26 0.03
74.16
86.59 0.08
93.14 0.06
GCN
3,72
62.67 2.14
11,17 2.93
2.
98.01 0.04
99.03 = 0.02
99:48 = 0.02
GAT
43.59
62.24 + 2.29
OOM
OOM
OOM
OOM
OOM
SAGE
50.77
65.36 1.05
19.37 2.65 0OM
3
98.75
99.3 OOM
GAE
0OM
OOM
OOM
0OM
OOM
SEAL
1.57
69.94 0.72
21.81 #.4.3
36.88 +: 4.06
94.61
0.11
9 0.12
95.37 #: 0.14
BUDDY
0.55
0.87
26.33 2.63
38.18 1.32
97.79 0.07
98.
99.38 0.03
Nco- GNN
$7,05 a 1.56
71.76 0.55
26.16 1.24
37.95 1.45
97.05 0.07
98.75 a: 0.03
99.41 0.02
NCN'
50.27
67.58 = 0.09
40.29 + 2.22
$3.35 + 1.77
97.97 0.03
99.02 0.02
99.5 0.01
NCNC
54.91 54.91 22884
70.91 0.25
40.1
$2.09 +
97.22
98.2 0.71
98.77 0.6
NBFNet
0OM
0OM
OOM
OOM
0OM
OOM
OOM
PEG
7.440
55.14 2,10
OOM
OOM
0OM
OOM
OOM
Mean- Ensemble
57.96
71.65 = 0.30
5.22
13.94 4.26
98.51 = 0.03
99.21 = 0.03
99.57 # 0.01
Global-Ensemble
57.91 1.57
71,24
27.3
47.27 5.62
98.7 0,04
99.37 = 0.02
99.69 0.01
Link-MoBi
63.83 0.65
75.16 1.64
48.36 # 1.37
59.87 +: 0,80
98.59 = 0.02
99.28 = 0.01
99.63 a 0.02
Table 9: Results for using Top-K experts or a few experts on ogbl-collab and Pubmed.
ogbl-collab
Pubmed
Best Expert
66.13
44.73
Top-3 Experts
71.94
51.13
Experts
71.25
52.03
4 Experts
72.75
52.30
All Experts
71.32
53.10
Results for Heterophilic Datasets
We conduct experiments on two widely used heterophilic graphs, i.e., Chameleon and Squirrel. We follow the same setting with (56) and use the MRR for the evaluation metrics. The results are shown in Table I0, From the results, we can find the proposed Link-MoE outperforms the baselines by large margin. These results demonstrate the proposed Link-MoE works well for both the homophilous and heterophilic graphs.
Table 10: Results on heterophilic datasets. The metric is MRR.
Node2Vec
MLP
GCN
BUDDY
Neo-GNN
NCN
NCNC
Link-MoE
Chumeleon Squirell
18.1
34.65
18.44
10.96
21.63
35.31
30.87
41.20
10.60
13.66
25.22
3.80
8.05
26.97
22.25
31.98
We also analyzed the weights generated by the gating mechanism on the two heterophilic graphs. The results are shown in Figure 13] We observed that feature proximity-based experts, such as MIP and GCN, are rarely employed for both datasets. This consistent with the characteristics of heterophilic graphs, where connect nodes tend to have dissimilar features. These findings demonstrate the effectiveness our gating design, accurately selects the appropriate experts for different types of graphs.
19

---

Algorithm Two-step Training
input Input graph G, Node feature X, Expert models E = (E1, Ez, ...; Em Positive set P and Negative set
N
output Converged Link-MoE
1: for i= 1, 2, .... m do
2: Train each individual expert E,
3: end for
4: Get heuristics and prediction scores for positive and negative links
5: repeat
Update gating parameters by optimizing Eq.
7: until Gating model converge
0.8
06
reagen
02
00
0
25
26-10
51-76
77 397
0.6
0
100
167
372
373 753
559 937
(a) Chameleon
(b) Squirrel
Figure 13: Expert weights for chameleon and squirrel dataset. The groups are split based on CN.
G End-to-end Results
In this section, we explore the end-to-end training of the experts and gating models, as shown in Table [1 and Figure 14 Our results show that the convergence speed of different experts varies significantly, which often leads to the model collapsing to a single expert. As a result, the performance of end-to-end training is not good as the proposed Link-MoE, Despite this, it remains an interesting and challenging idea to explore the effective end-to-end training.
Table 11: Results of the end-to-end training on ogbl-collab and Cora.
ogbl-collab
Cora
end-to-end Link-MoE
67.43
21.97
70.86
44.03
expert's and on collab (endZend)
Gating weights each expert (ena2end)
0.8
0.7-
MLP
# 0.6
GCN
MLP
NCN
40
GON
0.5
NCNC
0.4
NCNC
0.3
Overall
0.2-
10
0.0
10
20
30
40
50
60
70
0
10
20
30
40
50
60
70
Epochs
Epochs
(a) Results of overall each expert.
(b) The gating weights of each expert.
Figure 14: End-to-end training of experts and gating on ogbl-collab.
H
Results for Additional Ensemble Methods
We compare our methods with two ensemble methods (30 specifically designed for link prediction
tasks. Specifically, 30 found that individual link prediction algorithms exhibit broad diversity of
prediction errors on different graphs and utilized random forest to ensemble various link predictors; Similarly,
31 employs different graph embedding techniques to train multiple link predictors,
20

---

using the outputs of these predictors as imputs for training a subsequent link predictor. We conduct experiments on ogbl-collab and Pubmed dataset.The evaluation metrics are Hits@ and MRR for ogbl-collab and Pubmed, respectively. The results, shown in Table T2, demonstrate that the proposed Link-MoF. model significantly outperforms the ensemble-based methods by a large margin. This can be attributed to the dynamic nature of our gating module, which assigns customized weights to each expert for every node pair.
Table 12: Results for ensemble baselines ogbl-collab and Pubmed.
ogbl-collab
Pubmed
Best Expert
66.13
44.73
Mean-Ensemble
66.82
38.54
Global-Ensemble
67.05
37.63
Ensemble 30
69.65
41.21
Ensemble
31
65.11
43.92
Link-MoB
71.32
53.10
I Results for Different Gating Input
We compare our Link-MoE with different gating inputs on ogbl-collab and Pubmed datasets. The evaluation metries are Hits @ 50 and MRR for ogbl-collab and Pubmed, respectively. Specifically, design two different gating inputs.
The first one is traditional gating, which only leverages the node features in gating. The results are shown in Table 13 (Traditional Gating). Traditional Gating only results in comparable performance to the best single experts, while our approach yields superior performance. This phenomenon demonstrates the effectiveness and rationality of the designed gating model.
To investigate the impact of involving expert model predictions as input to the gating model, we conducted experiments on the oobl-collab and Pubmed datasets by concatenating the prediction results of experts with the heuristic features. The results are shown in Table 13 (With Experts as Input). We observed that involving the experts' prediction results as additional input did not lead to improvement. In fact, this approach may result in lower performance compared to using heuristics alone. This phenomenon suggests that the outputs of the expert models may not effectively reflect their importance specific node pairs, highlighting the effectiveness and rationality of using heuristics as the gating input.
Table 13: Results for different gating inputs ogbl-collab and Pubmed.
ogbl-collab
Pubmed
Best Expert
66.13
44.73
Traditional Gating
66.59
42.15
With Experts Input
71.04
51.36
Link-MoE
71.32
53.10
J Results for HeaRT Setting
We conducted additional experiments under the HeaRT setting [53] with OGB datasets, and the results are shown in 14 In the HeaRT setting, hard negative samples are selected for positive samples based on specific heuristics, restricting the negatives to include one of the two nodes from the original positive pair. The results in Table 14 demonstrate that Link-MoE still significantly outperforms the best baseline models, even this more challenging evaluation scenario.
K Limitations
In our current work, we explore various heuristics and GNN4LP models for link prediction and demonstrate the potential of MoE models in this context. The training of the gating model in
21

---

Table 14: Results on OGB datasets (%) under HeaRT. Highlighted the results ranked first, second,
and third.
ogbl-collab
ogbl-ppa
ogbl-citation2
Models
MRR
Hits 20
MRR
Hits@20
MRR
Hits
CN
4.20
16.46
25.70
68.25
17.11
41.73
AA
5,07
19.59
26.85
70.22
17.83
43.12
RA
6.29
24.29
28.34
71.50
17.79
43.34
Shortest Path
2.66
15.98
0.54
1.31
>24h
>24h
Katz
6.31
24.34
25.70
68.25
14.10
35.55
Node2Vec
4.68 = 0.08
16.84 0.17
18.33 0.10
53.42 0.11
14.67 0.188
42.68
0.20
MF
4.89 0.25
18,86 0.40
22.47 1.53
4.82 70.71
2.60
29.64 7.30
MI.P
5.37 = 0.14
16.15 + 0.27
0.98 0.00
0.00
16.32 = 0.07
43.15 0.10
GCN
22.48 + 0.81
26.94 + 0.48
68.38 0.73
19.98 0.35
51.72 0.46 OOM
GAT
4.18
18.30 # 1.42
OOM
OOM
OOM
SAGE
5.53
21.26 + 1.32
27.27 = 0.30
69.49 0.43 OOM
22.05 0.12 0OM
4
GAE
0OM
0OM
OOM
OOM
SEAL
6.43
21.57 0.38
29.71
76.77 0.94
20.60
48.62 1.93
BUDDY
5.67+0.36
23.35
27.70
71.50
19.17
47.81 0.37
Nco-GNN
5.23 5.29
21.03 3.39
21.68 + 1.14
64.81 2.26
16.12 0.25
43.17 0.53
NCN
5.09 0.38
+
35.06 + 0.26
81.89 + 0.31
23.35 + 0.28
53.76 + 0.20
NCNC
0.86
20.4943.97
33.52 + 0.26
82.24 = 0.40
19.61 = 0.54
51.69 + 1.48
NBFNet
OOM
OOM
OOM
OOM
OOM
OOM
PEG
4.83 0.21
18.29 # 1.06
OOM
OOM
OOM
OOM
Link-MoE
15.11 # 8.28
29.80 + 4.43
62.11 + 3.54
88.49 = 0.56 24.07 = 1.77
57.71 + 0.19
Link-MoE relies on pre-trained experts and heuristic features generated from the graphs. Although we can select few experts, training these pre-trained experts remains time- tinecconuming, especially for complex models. Additionally, since real-world graphs come from diverse domains and the graph generation process might be quite different, the heuristics used in this paper might not be comprehensive to other domains. Future work should focus on exploring more heuristics to better accommodate diverse domains.
L Impact Statement
In this paper, we explore the use of mixture experts (MoE) model for in link prediction. We view the impact of this work as positive, as it can help improve performance of link prediction in many real-world applications including drug discovery and recommender systems. In general, we don't envision any specific negative societal consequences of our work. However, it is possible that the choice of experts used in our model may introduce some negative effects that stem from those individual models.
M Dataset Licenses
The license for each dataset can be found in Table 15
Table 15: Dataset Licenses.
Datasets
Com
Citeseer
Pubmed
ogbl-collab
ogbl-ppa
oghl-citation2
Chameleon
Squirrel
License
NLM License
NLM Liceuse
NLM License
MIT License
MIT Liceuse
MIT License
GPLv3
GPLv3
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section S and S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This is discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See http://gtthub..comm/mm/////aaa MoE/
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See Seeltttss://githhi http:////ggtttub..oommmaaa//////
Guidelines:
The answer NA means that paper does not include experiments requiring code.

---

Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/nips.cc//i cttplss.ccppublccguuuuc guides/ htccppuubb//ggiiiee//eeeeaaaaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See details in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
25

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p::///aarpppssccccpp//t////
Answer: [Yes]
Justification: There was no ethical concerns in either the research process or in potential impacts of our research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section L
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
26

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We introduce no new datasets in our paper, as all datasets used are already pub- lic. Furthermore, don't envision any specific harm that can be caused by the framework introduced in our paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Table 15
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
27

---

Answer: [Yes]
Justification; All data and models used in our paper are properly cited and attributed. including their license information (see Table 15 Also, release the code for model /ttp//ggtttbb..t./
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "54YZDKC7JQ", "reviewer": "Reviewer_bwf6", "summary": "This paper presents a mixture of experts model, termed Link-MoE, for link prediction on graphs. Link-MoE individually trains various link prediction models as experts and selects the most appropriate experts for different node pairs. The prediction results from the selected experts are then weighted to produce the final predication. Experiments and analyses validate the effectiveness of the proposed method.", "questions": "See weaknesses above.", "limitations": "The authors have discussed the limitations in terms of the scalability.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is generally well-written and easy to follow. Necessary analyses are conducted to well motivate the design of the proposed method. \n2. The proposed method is presented as a general framework, so that various link prediction models (i.e., experts) can be combined via different weights generated by the gating function.\n3. The experimental design can largely validate the efficacy of the proposed method.", "weaknesses": "1. This paper presents a straightforward application of the MoE model to the problem of link prediction. Despite its empirical effectiveness, the technical novelty appears limited. The methodological contributions could be enhanced by addressing specific problems associated with MoE models, e.g., the collapse problem, in the context of link prediction. \n2. The experimental evaluation could provide more insights and deeper analyses on important issues, for example,\n- How does Link-MoE determine the appropriate number of experts?\n- On heterophilic graphs, how the experts selected by Link-MoE differ from those selected on homophiles graphs?\n3. The complexity of the proposed method can be high. The nature of using MoE makes the proposed method difficult to scale up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bwf6,\n\nWe appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.\n\n**W1:Despite its empirical effectiveness, the technical novelty appears limited.**\n\n**A1:** The novelty of Link-MoE. Our Link-MoE model stands out by intelligently leveraging the complementary strengths of different GNN4LP models, resulting in substantial performance gains. The core of our model's success lies in the following findings and innovations:\n\n*1. GNN4LP Models are Complementary:* Our preliminary studies show that the overlapping between different GNN4LP models is notably low, shown in Figure 3, 9 and 10 in our paper, indicating these models offer complementary insights for link prediction. Different node pairs may be best addressed by different models. The critical challenge, then, is determining the most effective way to assign these node pairs to the appropriate GNN4LP models.\n\n*2. Heuristic-Informed Gating Mechanism:* Our preliminary studies revealed that different GNN4LP models perform differently across different heuristic groups, shown in Figure 4, 11 and 12 in our paper. By designing a gating model that intelligently applies these heuristics, we facilitate the optimal assignment of node pairs to their most suitable predictors. This strategic use of heuristics marks a significant departure from traditional MoE applications, which use the same input features for gating and experts [1]. We compare the MoE with traditional gating with Link-MoE. The results are shown in Table 1 (Traditional MoE) in the global response. Traditional MoE only results in comparable performance to the best single experts, while our approach yields superior performance. \n\n[1] Towards Understanding Mixture of Experts in Deep Learning. NeurIPS'22\n\n**The Collapse problem.** In traditional MoE models, the collapse problem [37] can occur when a single expert is consistently selected, resulting in the under-utilization and inadequate learning of other experts. However, we employ a two-step training strategy for the Link-MoE. First, we train each expert separately. Then we train the gating model to leverage the strengths of each expert effectively. This approach ensures that each expert is well-trained before the gating model is introduced. We empirically observe that there does not exist collapse problem in our model. As illustrated in Figure 8 of this paper, different experts are activated to model different node pairs.\n\nWe also tested the end2end training of the experts and gating models, as shown in Table 3 and Figure 1 in the global response. Our results show that the convergence speed of different experts varies significantly, which often leads to the model collapsing to a single expert. As a result, the performance of end2end training is not as good as the proposed Link-MoE. Despite this, it remains an interesting and challenging idea to explore the effective end2end training.\n\n**W2.1:How does Link-MoE determine the appropriate number of experts?**\n\n**A2:** There are mainly three types of pairwise structural information in link-prediction tasks: local structure proximity (LSP), global structure proximity (GSP), and feature proximity (FP). Different GNN4LP models leverages different heuristic information. Based on these, we classify these methods into three groups: LSP (NeoGNN, NCN, NCNC, n2v), GSP (Seal, Buddy, NbfNet, PEG), and FP (MLP, GCN).\n\nFor the selection of experts, it is suggested to include methods that cover all groups due to the complexity of connection patterns across different datasets. To verify this, we conducted an experiment on the ogbl-collab dataset. When using all the models, the Hits@50 score is 71.32. However, if we remove all LSP experts, the Hits@50 score drops to 67.93. Interestingly, if we include only one LSP expert, the Hits@50 score can reach 71.25. This demonstrates the necessity of local structure information for the ogbl-collab dataset, but it also indicates that only one model may be sufficient to capture this information effectively.\n\nFurthermore, our experiments show that using just a few experts (3 or 4) can achieve similar performance levels to using all experts, as shown in A4.\n\n**W2.2:On heterophilic graphs, how the experts selected by Link-MoE differ from those selected on homophiles graphs?**\n\n**A3:** We analyzed the weights generated by the gating mechanism on two heterophilic graphs, Chameleon and Squirrel. The results are shown in Figure 3 in the global response. We observed that feature proximity-based experts, such as MLP and GCN, are rarely employed for both datasets. This is consistent with the characteristics of heterophilic graphs, where connect nodes tend to have dissimilar features. These findings demonstrate the effectiveness of our gating design, as it accurately selects the appropriate experts for different types of graphs.\n\n**W3:The complexity of the proposed method can be high.**\n\n**A4:** The proposed Link-MoE model employs a two-step training strategy, which first trains the single experts and then trains a lightweight gating model. The most time-consuming part is the training of the single experts. However, as discussed in A2, there is no need to use all available experts, which can effectively reduce complexity and enhance scalability. we conduct experiments on ogbl-collab and Pubmed by only using 3 or 4 experts (please refer to Different number of experts for the expert selection details in the global response). The results shown in Table 1 (3 Experts & 4 Experts).\n\nFrom the results, we can find that only 3 or 4 experts can achieve comparable performance with using all experts in the paper. Additionally, the two-step training strategy of our Link-MoE model introduces another layer of efficiency by allowing the integration of pre-trained models. In real-world scenarios where models are continually updated, this approach is highly efficient: when a new model arrives, we only need to train the gating model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t87PTjj5C2", "reviewer": "Reviewer_fyCt", "summary": "Link prediction in graphs is a fundamental task in graph machine learning and multiple heuristics and ML algorithms have been designed in research to leverage the pairwise and structural information to predict links between nodes. This work takes inspiration from the success of MoE models across various verticals and proposes a new method that outperforms the SoTA baselines on this task on several standard datasets. The authors demonstrate that heuristics and algorithms for LP are very diverse in their capabilities to predict links depending on the node structure and pairwise features (i.e. when one would perform well, the other might not). Due to this little overlap in their abilities the authors propose a dynamically weighted ensemble (MoE) like approach - such that a gating network predicts the weight given to each expert's prediction and the overall value is the weighted sum. \nThey also provide a very practical method to train the experts and the gating network achieving SoTA performance beating the second best by significant margins.", "questions": "- Did you face any convergence issues with training the gating network ?\n- What happens when you use a graph network which is transductive in nature ? Can that information be leveraged in some manner ?\n- In the situations when the graph is a KG on language. Can LLM's outperform this method ?", "limitations": "The method is pretty generic and the societal impacts are not discussed in a lot of detail as it wont be different from existing methods.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well written with the motivations being made clear and experiments being clearly able to prove what the authors set out to. The number of experiments and metrics being covered are large enough to justify the claims in a border sense - with all the obvious baselines being addressed. I particularly liked the comparisons with the Mean and Global weight ensembles, and LPFormers.\nThe authors also provide an efficient and effective training strategy to build this E2E - which in my opinion can be understood and used easily by the community.\nThe paper also provides reasonable explanations for the results and conducts ablations (in section 5) required to improve understanding and readability for a potential user.\nThis result greatly improves on the SoTA baselines and should be generalisable to multiple graph use cases.", "weaknesses": "- The method being used is fairly obvious and intuitive.\n- It solves for a very specific task - Link Prediction in an inductive setting only. Not clear to me how it could be leveraged for a transductive setting as well", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fyCt,\n\nWe sincerely appreciate your recognition of our framework and insightful comments. We are pleased to provide detailed responses to your questions.\n\n**W2: It solves for a very specific task - Link Prediction in an inductive setting only. Not clear to me how it could be leveraged for a transductive setting as well.**\n\n**A1:** We'd like to clarify that our paper actually focuses on the transductive setting, where the same graph is used during both the training and testing phases. In contrast, the inductive setting involves unseen nodes and connections in the test graphs.  Since the vast majority of methods for link prediction tasks are designed specifically for the transductive setting (please refer to  [14, 15, 25, 26, 13, 16, 27] in our paper), we have limited our focus to it as well.\n\nHowever, due to the generality of our Link-MoE method, it should have no issue adapting to the inductive setting when the individual expert models are themselves inductive.  In the inductive setting, we would train the gating model based on the heuristics calculated from the training graph. During inference, we would simply recalculate the heuristics based on the test graph before applying Link-MoE. We believe this approach would allow Link-MoE to perform effectively in inductive settings, and we intend to explore this in future work.\n\n**Q1: Did you face any convergence issues with training the gating network?**\n\n**A2:** We did not encounter any convergence issues with training the gating network in our Link-MoE model. Our model successfully converges on the datasets we used, as evidenced by the smooth training loss curves on Pubmed presented in Figure 2 in the global response.\n\n**Q2: What happens when you use a graph network which is transductive in nature? Can that information be leveraged in some manner?**\n\n**A3:** As discussed in A1, while our Link-MoE focuses on the transductive setting, it can be generalized to the inductive setting. During the training phase, we train the gating model based on the heuristics calculated from the training graph. During inference, we would recalculate the heuristics based on the test graph before applying Link-MoE.\n\n**Q3: In the situations when the graph is a KG on language. Can LLM's outperform this method?**\n\n**A4:** We appreciate the inspiring question.\n\nDue to the uniqueness of KGs, the methods used for KGs differ from those used on non-KGs (i.e., the graphs used in our paper). This is because the type of structural information needed to accurately predict new links depends on the graph. On KGs, research shows that path-based information that connects the two nodes are necessary for strong performance [16]. On the other hand, for link prediction on non-KGs, heuristics such as common neighbors or feature similarity tend to be more important [17]. **Because of the disparity in inductive bias, in our paper we limit our focus to link prediction on non-KGs only**. However, due to the generality of our method Link-MoE, it should have no issue adapting to the link prediction on KGs, as we only need to use KG-specific methods as the experts. \n\nLastly, as of now there is no consensus on whether LLMs can outperform graph-based methods for link prediction on KGs. However, recent work has shown that for inductive link prediction, LLM-based methods can potentially outperform SOTA graph-based methods on KGs (see [1] below). \n\n[1] Wang, Kai, et al. \"LLM as Prompter: Low-resource Inductive Reasoning on Arbitrary Knowledge Graphs.\" arXiv preprint arXiv:2402.11804 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U1OyXwf3yK", "reviewer": "Reviewer_758a", "summary": "The paper proposes a mixture of experts model, Link-MoE, for link prediction on graphs. The motivation behind the proposed approach is that different node pairs within the same dataset necessitate varied pairwise information for accurate prediction, while existing methods consider the same pairwise information uniformly across all pairs. To address this, Link-MoE consists of multiple expert models (existing link prediction methods) and a gating model whose goal is to produce importance scores to weight the different expert models according to their contribution towards the final prediction. Experiments on several real-world datasets show that the proposed Link-MoE outperforms existing competing methods, validating the rationale behind its design.", "questions": "Q1) Why reporting different metrics for different datasets? As an example, line 155, the authors state that they report MRR for Citeseer and Hits@50 for ogbl-collab. Similarly, in Table 2, different metrics are reported for different datasets. have seen this approach used in other papers on link prediction, but I am not convinced that it is the correct way to proceed because, as shown in the tables in the appendix (e.g., Tables 5-7), the proposed method is not always the best when the evaluation metrics are changed.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1) The source code is provided with the submission\n\nS2) The paper is overall well-written\n\nS3) The proposed approach is compared against several competing methods and baselines.\n\nS3) Strong experimental results: the proposed method outperforms all the considered competing methods.\n\nS4) The proposed approach is well motivated\n\nS5) The proposed method was assessed on standard benchmark datasets for link prediction", "weaknesses": "W1) The figures are hardly readable sometimes (e.g., Fig. 8), the font size (in the legend) should be increased.\n\nW2) Notation problem: in Eq. 1 the gating function G has only one parameter while in Eq. 2 G takes both x_ij and s_ij. Please uniform it as this may be a little confusing for the reader.\n\nW3) The difference between the proposed Link-MoE and the considered baseline Global-Ensemble should be further discussed in the main paper given the strong similarity between the two approaches. As a suggestion, I recommend the authors to move lines 572-575 to the main paper since it describes the main differences between the two methods, i.e. the importance weight vector is uniform in Global-Ensemble while it is adaptable to the pair in Link-MoE.\n\nW4) To evaluate the effectiveness of the proposed method different metrics are reported for different datasets. \n\nMinor:\n- Eq. 3: final extra parenthesis\n- Line 220: “consits” -> “consists”\n- Line 234: extra ”the” in “represent the all the”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 758a,\n\nThank you so much for your support and recognition of our framework. We are pleased to provide detailed responses to address your concerns.\n\n**W1: The figures are hardly readable sometimes (e.g., Fig. 8), the font size (in the legend) should be increased.**\n\n**A1:** Thank you for pointing out this issue. We will enlarge the font size of the axis labels, legend, and axis ticks for all figures to ensure they are more readable.\n\n**W2: Notation problem: in Eq. 1 the gating function G has only one parameter while in Eq. 2 G takes both $x_{ij}$ and $s_{ij}$. Please uniform it as this may be a little confusing for the reader.**\n\n**A2:** In Eq. 1, we initially used $h_{ij}$ to represent all the heuristics, including both the pairwise node features $x_{ij}$ and the structural heuristics $s_{ij}$.\nTo alleviate potential confusion, we will rewrite Eq. 1 to match Eq. 2 as follows:\n$$\nY\\_{ij} = \\sigma \\left( \\sum\\_{o=1}^m G(\\mathbf{x}\\_{ij}, \\mathbf{s}\\_{ij})\\_o E\\_o(\\mathbf{A}, \\mathbf{X})\\_{ij} \\right)\n$$\n\n**W3: The difference between the proposed Link-MoE and the considered baseline Global-Ensemble should be further discussed in the main paper given the strong similarity between the two approaches.**\n\n**A3:** We will further discuss the Global-Ensemble method in the main text. Specifically, we will include a discussion of this method at the end of Section 3. We will first include the definition of the Global-Ensemble method (given by lines 570-575 in the appendix). We will then discuss the weaknesses of this method, mainly that the weights are non-adaptive and are the same for all links. Our empirical results in Section 3.1 and 3.2 also show that this kind of design is unable to model all links. The deficiency of this formulation serves as a strong motivation for our proposed method Link-MoE in Section 4.\n\n**W4: To evaluate the effectiveness of the proposed method different metrics are reported for different datasets.**\n\n**A4:** In Table 1, we use MRR for evaluating Cora, Citeseer, and Pubmed datasets, while Hits@50, Hits@100, and MRR are used for the ogbl-collab, ogbl-ppa, and ogbl-citation2 datasets. This approach is consistent with prior research, where notable methods [13, 14, 15, 16, 27, 53] have been designed for link prediction tasks. Additionally, we provide the results of other metrics in the Appendix, specifically in Tables 5, 6, 7, and 8.\n\n**Q1: Why reporting different metrics for different datasets?**\n\n**A5:** For the training of Link-MoE, we adhere to the experimental settings outlined in prior research [53], which serves as a benchmark for the link prediction task. \nDuring the training, we select and save the best model based on the validation performance. However, there are multiple evaluation metrics, which might be not aligned well. In this paper, we select the best model based on MRR for the Cora, Citeseer, and Pubmed datasets. For the ogbl-collab, ogbl-ppa, and ogbl-citation2 datasets, we used Hits@50, Hits@100, and MRR as selection metrics following [53].\nAs a result, our method might not perform optimally on all individual metrics due to the misalignment of different metrics. Nevertheless, we consistently rank within the top 3 across almost all datasets and metrics. These observations indicate that Link-MoE is capable of delivering strong performance across various metrics.\n\nIn the revision, we plan to fix all the typos and add another experiment that selects the best results based on each specific metric to provide a more comprehensive evaluation.\n\nWe hope that we have addressed the concerns in your comments, and please kindly let us know if there is any further concern, and we are happy to clarify.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gpxWumgm9H", "reviewer": "Reviewer_SDXC", "summary": "This paper proposes an ensemble model of different link prediction methods. The authors find that different node pairs on the graph can form a link due to different pairwise representations, and there is no single link prediction model that can capture all of them. Then the author proposes to combine the outputs of different link prediction models into one ensemble model, weighted by the heuristics information of the node pair. Experiments on graph benchmarks show the improvement brought by ensembling.", "questions": "1. For the gating model, only the heuristics information is considered as the input. However, this can limit the expressiveness of gating models, because the inherent limitation of the heuristics models themselves. Do authors consider involving the expert models as inputs to the gating models?\n\n2. Following up on the question above, is there any ablation study on training the gating model and expert models jointly? It may introduce higher memory consumption, but one can consider remove the computational heavy models like SEAL or NBFNet.", "limitations": "See the weakness and question parts.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The paper is clear and well-written.\n\n2. The motivation of the study is clearly stated. In the preliminary study section, the authors give solid examples of why individual link prediction models can fail, which motivates a combination of such models.\n\n3. The experiment is convincing and comprehensive, showing the competitive performance improvement with a mixture of link prediction models.", "weaknesses": "1. The novelty of the paper is limited. This study proposes a mixture of existing link prediction models, which brings minor research values to the community. The overall contribution of the paper makes it more like a technical report, rather than a research paper. While this paper has its own value, it is more suitable to be submitted to other venues.\n\n2. The baselines considered in the experiment are not comprehensive. For example, [1], which is an ensemble of link prediction models, is not evaluated as a baseline.\n\n3. The overall computational efficiency of the mixture method is concerning. On each graph, a set of link prediction models needs to be trained first, which typically involves an extensive process of hyperparameter tuning like in NCN [2]. This step can be too costly for most real-world use cases. The idea of MoE, especially conditional computation, is not leveraged at all to reduce the computational cost.\n\n4. The authors can further benchmark the Link-MoE on [3], which introduces more challenging link prediction tasks.\n\n[1] Ghasemian, A., Hosseinmardi, H., Galstyan, A., Airoldi, E. M., & Clauset, A. (2020). Stacking models for nearly optimal link prediction in complex networks. Proceedings of the National Academy of Sciences.\n\n[2] Wang, X., Yang, H., & Zhang, M. (2023). Neural common neighbor with completion for link prediction.\n\n[3] Li, J., Shomer, H., Mao, H., Zeng, S., Ma, Y., Shah, N., ... & Yin, D. (2024). Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking. Advances in Neural Information Processing Systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SDXC,\n\nWe appreciate your constructive feedback. We are pleased to provide detailed responses to address your concerns.\n\n**W1:The novelty of the paper is limited.**\n\n**A1:** The novelty of Link-MoE: Our Link-MoE model stands out due to its innovative approach, leveraging the complementary strengths of different GNN4LP models, resulting in substantial performance gains. The core of our model's success lies in the following findings and innovations:\n\n*1. GNN4LP Models are Complementary:* Our preliminary studies show that the overlapping between different GNN4LP models is notably low, shown in Figure 3, 9 and 10 in our paper, indicating these models offer complementary insights for link prediction. Different node pairs may be best addressed by different models. The critical challenge, then, is determining the most effective way to assign node pairs to the appropriate GNN4LP models.\n\n*2. Heuristic-Informed Gating Mechanism:* Our preliminary studies revealed that different GNN4LP models perform differently across different heuristic groups. By designing a gating model that intelligently applies these heuristics, we facilitate the optimal assignment of node pairs to their most suitable predictors, which is a significant departure from traditional MoE applications, which use the same input features for the gating and experts. We compare the Link-MoE with traditional gating, which only leverages the node features in gating. The results are shown in Table 1 (Traditional MoE) in the global response. Traditional MoE only results in comparable performance to the best single experts, while our approach yields superior performance. This phenomenon demonstrates the effectiveness and rationality of the designed gating model. \n\n**W2:The baselines considered in the experiment are not comprehensive.**\n\n**A2:** Thank you for pointing out the related papers. Despite the Mean-Ensemble and Global-Ensemble method in our paper, we test two more ensemble methods. [1] and [2]  utilized a random forest and MLP to ensemble various link predictors, respectively. The results are presented in Table 2 in the global response. Link-MoE outperforms all ensemble methods. The superior performance of our approach is attributed to the dynamic nature of our gating module, which assigns customized weights to each expert for every node pair.\n\n[1] Stacking Models for Nearly Optimal Link Prediction in Complex Networks, PNAS'20\n\n[2] An Ensemble Model for Link Prediction based on Graph Embedding, Decision Support Systems'22\n\n**W3:The overall computational efficiency of the mixture method is concerning. The idea of MoE, especially conditional computation, is not leveraged at all to reduce the computational cost.**\n\n**A3:** We would like to highlight that our framework differs from traditional MoE methods. Our goal is to leverage different GNN4LP models to address various node pairs effectively. We utilize a two-step training strategy to train the experts and a lightweight gating model separately. While training multiple experts does increase computational cost, our approach only needs a few experts to achieve superior performance compared to the baselines. \n\nTo demonstrate this, we conducted experiments on ogbl-collab and Pubmed datasets using only 3 or 4 experts. The detailed settings and results are shown in Table 1 (3 Experts \\& 4 Experts) in the global response. \n\nWe observe that using only 3 or 4 experts can achieve comparable performance to using all experts, indicating the computational cost of Link-MoE is not prohibitively high.\n\n**W4:The authors can further benchmark the Link-MoE on [3].**\n\n**A4:** We conduct experiments on ogbl-collab and PubMed under HeaRT setting [3]. The results shown in Table 4 of global response demonstrate that our model significantly outperforms all individual experts on both datasets. This highlights the effectiveness of our approach in leveraging the strengths of multiple experts.\n\n**Q1:Do authors consider involving the expert models as inputs to the gating models?**\n\n**A5:** We use the heuristic as input because the GNN4LP models are aligned with the heuristic methods, as investigated in Section 3.2 in our paper. To investigate the impact of involving expert model predictions as input to the gating model, we conducted experiments on the ogbl-collab and Pubmed datasets by concatenating the prediction results of experts with the heuristic features. The results are shown in Table 1 (With Experts as Input) of global response.\n\nWe observed that involving the experts' prediction results as additional input did not lead to improvement. In fact, this approach may result in lower performance compared to using heuristics alone. This phenomenon suggests that the outputs of the expert models may not effectively reflect their importance to specific node pairs, highlighting the effectiveness and rationality of using heuristics as the gating input.\n\n**Q2:Is there any ablation study on training the gating and expert models jointly?**\n\n**A6:** We explored an end2end training strategy that trains several experts alongside the gating module from scratch. We conducted experiments on Cora and ogbl-collab to evaluate this approach. The results and experimental settings are detailed in Table 3 of global response.\n\nThe end2end results were significantly worse than  Link-MoE. A potential reason is that different experts have varying convergence rates, leading to the collapse problem. Figure 1(a) of global response illustrates some strong experts, such as NCN and NCNC, converge faster and dominate the performance. Strong models are assigned higher gating weights while MLP and GCN are assigned zero weights, as shown in Figure 1(b) of global response,  which limits its ability to leverage additional information from MLP and GCN. While our end2end approach did not yield improved results, it remains an interesting and challenging idea to explore the conditional computation of MoE for efficient end2end training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We thank the reviewers for the valuable comments and suggestions. In this global response, we are willing to provide information about tables and figures in the rebuttal pdf file. 

**Table 1** presents the results of traditional MoE, a few experts, and experts used as input features for the ogbl-collab and Pubmed datasets. The evaluation metrics are Hits@50 and MRR for ogbl-collab and Pubmed, respectively.

1. Traditional MoE. It uses the same input features for the gating model and experts. Traditional MoE only results in comparable performance to the best single expert, while our approach yields superior performance.

2. Different number of experts. For the ogbl-collab, we use MLP, NCNC, BUDDY (3 experts) and Neo-GNN (4 experts); for Pubmed, we use NCN, SEAL, NCNC (3 experts) and MLP (4 experts). From the results, we can find that only 3 or 4 experts can achieve comparable performance with using all experts in the paper. Notably, we don't use the computationally intensive SEAL for the ogbl-collab datasets. Furthermore, the inclusion of the less effective MLP expert in the Pubmed dataset still results in performance improvement, highlighting the complementary nature of the experts and the effectiveness of the proposed method.

3. With Experts as Input. We concatenate the prediction results of experts with the heuristic features as the input feature of gating model. Results show that this additional input did not lead to improvement.

**Table 2** compares the results of ensemble methods with our model. The evaluation metrics are Hits@50 and MRR for ogbl-collab and Pubmed, respectively. Ensemble [1] and Ensemble [2] refer to two ensemble methods specifically designed for link prediction tasks. As shown, our model outperforms these ensemble methods.

[1] Stacking Models for Nearly Optimal Link Prediction in Complex Networks, PNAS'20

[2] An Ensemble Model for Link Prediction based on Graph Embedding, Decision Support Systems'22

**Figure 1** is derived from  results when training experts and gating in an end2end way on ogbl-collab. Figure 1(a) shows the overall and each expert’s performance. Figure 1(b) shows the gating weights of each expert.

**Figure 2** illustrates the convergence process of training for our Link-MoE model on the Pubmed dataset, demonstrating a smooth training loss curve.

**Table 3** shows the results of end2end training on the ogbl-collab and Cora datasets. The evaluation metrics are Hits@50 and MRR for ogbl-collab and Cora.

1. Cora: we employed eight models (MLP, node2vec, GCN, NeoGNN, NCN, NCNC, SEAL, and BUDDY) and used the gating model to select the top-3 experts for each node pair. 

2. ogbl-collab: we used four models (MLP, GCN, NCN, and NCNC) and selected the top-2 experts.

**Figure 3** displays the weight distribution generated by our gating model for two heterophilic graphs:  Chameleon and Squirrel datasets. We can see that feature proximity-based experts, such as MLP and GCN, are rarely employed for both datasets, which is consistent with the characteristics of heterophilic graphs, where connect nodes tend to have dissimilar features.

**Table 4** presents the results of our Link-MoE model under the HeaRT setting [53] for the ogbl-collab and Pubmed datasets. It is evident that our model achieves better performance even under this challenging HeaRT setting. This highlights the effectiveness of our approach in leveraging the strengths of multiple experts to achieve superior performance.

Meta-review: The paper proposes an ensemble method for link prediction, where the mixing coefficients are predicted *separately* for each pair of nodes based on a set of heuristic features.
The strengths of the paper are
1. A clear motivation, including detailed and convincing preliminary analyses (`bwf6`, `fyCt`, `758a`, `SDXC`).
2. Strong experimental results (`bwf6`, `fyCt`, `758a`, `SDXC`) and evaluation on standard benchmark datasets (`758a`).
3. The availability of the source code (`bwf6`) and the simplicity of rebuilding the system in the community (`fyCt`), as well as being an intuitive method (`fyCt` - this was mentioned as a weakness, but to me it is a strength).
4. Clear writing (`bwf6`, `fyCt`, `758a`, `SDXC`).

The weaknesses are
1. Its limited technical novelty (`bwf6`, `SDXC`)
2. Its limitation to a single use case, inductive connection prediction (`fyCt`)
3. Its computational complexity (`bwf6`, `SDXC`)
4. The lack of additional analyses, e.g. the number of experts selected or qualitative differences in homophilic vs. heterophilic graphs.
5. Some missing baselines, namely [0] (`SDXC`) - during the rebuttal additional results for [1, 2] were provided instead.
6. The lack of a clear distinction again mixture of experts.

There were discussions with most reviewers, some over several rounds, in which some issues were resolved while others remained controversial. 

[0] Ghasemian, A., Hosseinmardi, H., Galstyan, A., Airoldi, E. M., & Clauset, A. (2020). Stacking models for nearly optimal link prediction in complex networks. Proceedings of the National Academy of Sciences.  
[1] Stacking Models for Nearly Optimal Link Prediction in Complex Networks, PNAS'20  
[2] An Ensemble Model for Link Prediction based on Graph Embedding, Decision Support Systems'22
Decision: Accept (poster)